Information processing apparatus and method

Abstract

The present disclosure relates to an information processing device and method capable of suppressing a decrease in encoding efficiency. The method comprises the following steps of: decoding coded data of a basic grid; decoding encoded data of a shifted video having, as a frame thereof, a 2D image in which local coordinates are stored as a shift vector; deriving a vertex normal vector that is a normal vector of a vertex of the subdivided base grid; and using a local coordinate system corresponding to the vertex normal vector to apply the local coordinate as a displacement vector to the vertex of the subdivided base grid. The present disclosure can be applied to, for example, an information processing device, an electronic apparatus, an information processing method, and a program.

Description

Technical Field

[0001] The present disclosure relates to an information processing device and method, and more particularly to an information processing device and method capable of suppressing a decrease in encoding efficiency. Background Art

[0002] As a method for encoding a mesh (which is 3D data that represents the three-dimensional structure of an object using vertices and connections), there has conventionally been video-based dynamic mesh coding (V-DMC) (for example, see non-patent document 1). In V-DMC, a mesh to be encoded is represented by a coarse base mesh and displacement vectors of subdivided points obtained by subdividing the base mesh, and the base mesh and the displacement vectors are encoded. The displacement vectors are stored (packed) in a two-dimensional image and are encoded as a moving image (displacement video) including such a two-dimensional image as a frame.

[0003] For example, as a coordinate system for representing a displacement vector when the displacement vector is stored in a two-dimensional image, there is a local coordinate system set for each vertex (for each displacement vector). A common coordinate system needs to be adopted in the encoder and decoder, but the local coordinate system is not explicitly sent. The encoder and decoder conventionally set the local coordinate system using a similar method with reference to the normal vector of each vertex used for shadow processing, etc. at rendering.

[0004] This normal vector is obtained by an "interpolation method" to be suitable for shadow processing during rendering, etc. In the interpolation method, the normal vectors of the vertices of the base mesh are first obtained, and then the normal vectors of the subdivision points are obtained using the normal vectors. At this time, the normal vectors of the vertices of the base mesh are weighted according to the positions of the subdivision points (the number of subdivisions), and the normal vectors of the subdivision points are obtained by weighted averaging. By this method, even if a plurality of subdivision points are formed between the vertices of the base mesh, the direction of the normal vector of each subdivision point can be changed. Therefore, by adopting the normal vector in shadow processing, etc., an independent processing result can be obtained for each vertex, and a higher definition processing result can be obtained.

[0005] Reference List

[0006] Non-patent literature

[0007] Non-Patent Literature 1: Khaled Mammou, Jungsun Kim, Alexis Tourapis, DimitriPodborski, Krasimir Kolarov, “[V-CG] Apple's Dynamic Mesh Coding CfP Response,” ISO / IEC JTC 1 / SC 29 / WG 7m59281, April 2022 Summary of the invention

[0008] Problems to be solved by the present invention

[0009] Incidentally, this local coordinate system controls the quantized value of the displacement vector for each axis. That is, what kind of coordinate system is used as the local coordinate system affects the quality of the displacement vector (i.e., the mesh to be reconstructed). In addition, by determining the local coordinate system to be adopted, the displacement vector (local coordinate) represented by the local coordinate system is determined. That is, the amount of information (i.e., the amount of encoding) of the displacement video (frame image) may also vary depending on what kind of local coordinate system is adopted. That is, what kind of local coordinate system is adopted affects the encoding efficiency.

[0010] However, what local coordinate system should be adopted to improve encoding efficiency depends on various factors, including, for example, the shape of the object. Therefore, as described above, the normal vector obtained by the "interpolation method" is not always optimal for deriving a local coordinate system for representing a displacement vector when encoding. In other words, by transferring the normal vector used for rendering to the derivation of the local coordinate system, the quality of the displacement vector may be reduced or the amount of information of the displacement video may increase, and therefore, the encoding efficiency may be reduced.

[0011] The present disclosure has been made in view of such circumstances, and an object of the present disclosure is to make it possible to suppress a decrease in encoding efficiency.

[0012] Solution to the problem

[0013] An information processing device according to one aspect of the present technology is an information processing device including: a base mesh decoding unit that decodes encoded data about a base mesh; a displacement video decoding unit that decodes encoded data about a displacement video, the displacement video including a 2D image storing local coordinates as displacement vectors as a frame; a vertex normal vector deriving unit that derives a vertex normal vector as a normal vector of a vertex of a subdivided base mesh; and a displacement vector applying unit that applies the local coordinates of the vertex as a displacement vector to the vertex of the subdivided base mesh using a local coordinate system corresponding to the vertex normal vector. The base mesh is a mesh that is coarser than the original mesh and is generated by extracting vertices of an original mesh to be encoded that includes vertices representing a three-dimensional structure of an object and connections. The displacement vector is a difference in position between a vertex of the subdivided base mesh and a vertex of the original mesh. The local coordinate system is a coordinate system for each vertex of the subdivided base mesh that is set based on the vertex normal vector. The local coordinate is a coordinate of a displacement vector representing a vertex of the subdivided base mesh in a local coordinate system corresponding to the vertex normal vector of the vertex.

[0014] An information processing method according to one aspect of the present technology is an information processing method as follows, the information processing method comprising:

[0015] Decoding the encoded data about the base mesh; decoding the encoded data about the displacement video, the displacement video including a 2D image storing the local coordinates as a displacement vector as a frame; deriving a vertex normal vector as a normal vector of a vertex of the subdivided base mesh; and using a local coordinate system corresponding to the vertex normal vector, applying the local coordinates of the vertex as a displacement vector to the vertex of the subdivided base mesh. The base mesh is a mesh coarser than the original mesh generated by extracting vertices of the original mesh to be encoded including vertices representing a three-dimensional structure of an object. The displacement vector is the difference in position between the vertex of the subdivided base mesh and the vertex of the original mesh. The local coordinate system is a coordinate system for each vertex of the subdivided base mesh set based on the vertex normal vector. The local coordinate is the coordinate of the displacement vector representing the vertex of the subdivided base mesh in the local coordinate system corresponding to the vertex normal vector of the vertex.

[0016] An information processing device according to another aspect of the present technology is an information processing device including: a method of deriving a vertex normal vector setting unit that sets a method of deriving a vertex normal vector as a normal vector of a vertex of a subdivided base mesh; a vertex normal vector deriving unit that derives a vertex normal vector by the set deriving method; a local coordinate deriving unit that derives a local coordinate representing a displacement vector in a local coordinate system corresponding to the vertex normal vector; a displacement video encoding unit that encodes a displacement video, the displacement video including a 2D image storing local coordinates as a displacement vector as a frame; a method designation flag generating unit that generates a method designation flag that specifies the set deriving method; and a method designation flag encoding unit that encodes the method designation flag. The base mesh is a mesh coarser than the original mesh generated by extracting vertices including a three-dimensional structure representing an object and connected vertices of the original mesh to be encoded. The displacement vector is the difference in position between the vertex of the subdivided base mesh and the vertex of the original mesh. The local coordinate system is a coordinate system for each vertex of the subdivided base mesh set based on the vertex normal vector.

[0017] An information processing method according to another aspect of the present technology is an information processing method, comprising: setting a method for deriving a vertex normal vector as a normal vector of a vertex of a subdivided base mesh; deriving the vertex normal vector by the set deriving method; deriving a local coordinate representing a displacement vector in a local coordinate system corresponding to the vertex normal vector; encoding a displacement video, the displacement video including a 2D image storing the local coordinate as a displacement vector as a frame; generating a method designation flag specifying the set deriving method; and encoding the method designation flag. The base mesh is a mesh coarser than the original mesh generated by extracting vertices including a three-dimensional structure representing an object and connected vertices of the original mesh to be encoded. The displacement vector is the difference in position between the vertex of the subdivided base mesh and the vertex of the original mesh. The local coordinate system is a coordinate system for each vertex of the subdivided base mesh set based on the vertex normal vector.

[0018] In an information processing device and method according to one aspect of the present technology, encoded data about a base mesh is decoded, encoded data about a displacement video including a 2D image storing local coordinates as a displacement vector as a frame is decoded, vertex normal vectors as normal vectors of vertices of the subdivided base mesh are obtained, and local coordinates of the vertices of the subdivided base mesh are applied to the vertices as displacement vectors using a local coordinate system corresponding to the vertex normal vectors.

[0019] In an information processing device and method according to another aspect of the present technology, a method for deriving vertex normal vectors as normal vectors of vertices of a subdivided base mesh is set, the vertex normal vectors are derived by the set deriving method, local coordinates representing displacement vectors are derived in a local coordinate system corresponding to the vertex normal vectors, a displacement video including a 2D image storing the local coordinates as a displacement vector as a frame is encoded, a method designation flag specifying the set deriving method is generated, and the method designation flag is encoded. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a diagram for explaining V-DMC.

[0021] Figure 2 It is a diagram for explaining displacement vectors.

[0022] Figure 3 A diagram for explaining a displacement video.

[0023] Figure 4 is a diagram for explaining an example of a method for deriving a normal vector;

[0024] Figure 5 is another diagram for explaining an example of a method for deriving a normal vector.

[0025] Figure 6 is a diagram showing an example of an encoding method.

[0026] Figure 7 is a diagram for explaining an example of the non-interpolation method.

[0027] Figure 8 is a diagram for explaining another example of the non-interpolation method.

[0028] Fig. 9 is a diagram for explaining an example of a method for selecting a derivation method.

[0029] Fig.10 is a diagram for explaining an example of weight values.

[0030] Fig.11 is a diagram for explaining an example of clustering.

[0031] Fig.12 is a diagram for explaining an example of quantization.

[0032] Fig.13 is a block diagram showing a main configuration example of an encoding device.

[0033] Fig.14 : is a block diagram showing a main configuration example of a local coordinate system setting section.

[0034] Fig.15 is a flowchart showing an example of the flow of an encoding process.

[0035] Fig.16 is a flowchart showing an example of the flow of a process for setting a local coordinate system.

[0036] Fig.17 is a flowchart showing another example of the flow of a process for setting a local coordinate system.

[0037] Fig.18 is a flowchart showing another example of the flow of a process for setting a local coordinate system.

[0038] Fig.19 is a block diagram showing a main configuration example of a decoding device.

[0039] Fig. 20 is a block diagram showing a main configuration example of another local coordinate system setting section.

[0040] Fig.21 is a flowchart for explaining an example of the flow of a decoding process.

[0041] Fig. 22 is a flowchart showing another example of the flow of a process for setting a local coordinate system.

[0042] Fig.23 is a flowchart showing another example of the flow of a process for setting a local coordinate system.

[0043] Fig.24 is a flowchart showing another example of the flow of a process for setting a local coordinate system.

[0044] Fig.25 is a block diagram showing another main configuration example of the local coordinate system setting section.

[0045] Fig.26 is a flowchart showing another example of the flow of a process for setting a local coordinate system.

[0046] Fig. 27 is a block diagram showing another main configuration example of another local coordinate system setting section.

[0047] Fig.28 is a flowchart showing another example of the flow of a process for setting a local coordinate system.

[0048] Fig.29 is a block diagram showing another main configuration example of the local coordinate system setting section.

[0049] Fig.30 is a flowchart showing another example of the flow of a process for setting a local coordinate system.

[0050] Fig.31 is a block diagram showing another main configuration example of another local coordinate system setting section.

[0051] Fig.32 is a flowchart showing another example of the flow of a process for setting a local coordinate system.

[0052] Fig.33 is a block diagram showing another main configuration example of the local coordinate system setting section.

[0053] Fig.34 is a flowchart showing another example of the flow of a process for setting a local coordinate system.

[0054] Fig.35 is a block diagram showing another main configuration example of another local coordinate system setting section.

[0055] Fig.36 is a flowchart showing another example of the flow of a process for setting a local coordinate system.

[0056] Fig.37 is a block diagram showing a main configuration example of a computer. DETAILED DESCRIPTION

[0057] Modes for carrying out the present disclosure (hereinafter, referred to as embodiments) will be described below. Note that the description will be made in the following order.

[0058] 1. Documents supporting technical content and technical terminology, etc.

[0059] 2. Local Coordinate System

[0060] 3. Local coordinate system for encoding

[0061] 4. Implementation Method

[0062] 5. Appendix

[0063] <1. Documents supporting technical content and technical terminology, etc.>

[0064] The scope disclosed in the present technology includes, in addition to the contents described in the embodiments, contents described in the following non-patent documents and the like known at the time of filing, contents of other documents mentioned in the following non-patent documents, and the like.

[0065] Non-patent document 1: (as mentioned above)

[0066] That is, the contents described in the above-mentioned non-patent literature, the contents of other documents mentioned in the above-mentioned non-patent literature, and the like are also used as a basis for determining the support requirements.

[0067] <2. Local Coordinate System>

[0068] <v-dmc>

[0069] As 3D data representing the three-dimensional structure of a solid structure (an object having a three-dimensional shape), a mesh representing the three-dimensional shape of the surface of an object by forming polygons using vertices and connections is conventionally known. As a method for encoding the mesh, for example, there is video-based dynamic mesh coding (V-DMC) as disclosed in non-patent document 1.

[0070] In V-DMC, a mesh to be encoded is represented by a coarse base mesh and displacement vectors of subdivided points obtained by subdividing the base mesh, and the base mesh and the displacement vectors are encoded.

[0071] For example, suppose there is Figure 1 The original mesh is shown in the upper part of Figure 1 In FIG. 1 , black dots represent vertices, and lines connecting the black dots represent connections. As described above, a mesh is originally formed into polygons using vertices and connections, but here, for ease of description, a mesh is described as a set of vertices connected linearly (in series) to each other.

[0072] By extracting the vertices of the original mesh, Figure 1 The coarse mesh shown second from the top. This is used as the base mesh.

[0073] By subdividing each grid of the base grid, such as Figure 1 Vertices are added as shown in the third part from the top in FIG. It is assumed here that as many vertices are added by subdivision as those vertices thinned out from the original mesh. Thus, a mesh having the same number of vertices as the original mesh is obtained. In this specification, the added vertices will also be referred to as subdivision points.

[0074] However, since the connections are updated due to the extraction of vertices of the original mesh and the subdivision points are formed on the updated connections, the shape of the subdivided base mesh is different from that of the original mesh. More specifically, Figure 1 As shown in the bottom part of , the positions of the subdivision points (on the dotted lines) are different from those in the original mesh. In this specification, the difference between the positions of the subdivision points and the positions of the vertices of the original mesh will be referred to as displacement vectors.

[0075] For example, suppose the original mesh 21 and the subdivided base mesh 22 are Figure 2 The positional relationship shown in FIG. 2 exists. It is also assumed that there are vertices 23 and 24 of the subdivided base mesh 22. In this case, Figure 2 As shown, the position of vertex 23 and the position of vertex 23' of original mesh 21 corresponding to vertex 23 are different from each other. The difference is represented as a displacement vector (displacement vector 25). Similarly, the difference between the position of vertex 24 and the position of vertex 24' of original mesh 21 corresponding to vertex 24 is represented as displacement vector 26. Therefore, a displacement vector is set for each of the vertices of the subdivided base mesh.

[0076] Since the original mesh is known in the encoder, the base mesh can be generated and such a displacement vector can also be derived. The decoder can generate (restore) the original mesh (the mesh corresponding to the original mesh) by subdividing the base mesh and applying the displacement vector to each vertex.

[0077] By reducing the number of vertices of the original mesh and encoding the original mesh as a base mesh, the amount of encoding can be reduced. In addition, the encoding efficiency can be improved by storing the displacement vector in a two-dimensional image and encoding the displacement vector via 2D encoding. In this specification, storing data in a two-dimensional image will also be referred to as packing.

[0078] Note that V-DMC supports scalable decoding. Figure 3 As shown, the displacement vectors are layered for different resolution levels (number of subdivisions of the underlying mesh) and packed as data for each level in a two-dimensional image 31. Figure 3 In the example of , LoD0 packed in the two-dimensional image 31 indicates data on displacement vectors of vertices of the uppermost layer (lowest definition) among displacement vectors of vertices layered for different definition levels. Similarly, LoD1 indicates data on displacement vectors of vertices of the layer immediately above LoD0. LoD2 indicates data on displacement vectors of vertices of the layer immediately above LoD1. The displacement vectors are thus layered (put together as data of different levels) and packed.

[0079] Note that the displacement vectors can be packed in a two-dimensional image as transform coefficients by a coefficient transform such as a wavelet transform. In addition, the displacement vectors can be quantized. For example, the displacement vectors can be converted into transform coefficients by a wavelet transform, the transform coefficients can be quantized, and the quantized transform coefficients (quantized coefficients) can be packed.

[0080] In addition, the three-dimensional shape of the object can change in the time direction. In this specification, the variability in the time direction will be referred to as "dynamic". Therefore, the grid (i.e., the base grid and the displacement vector) is dynamic. For this purpose, the displacement vector is encoded as a motion image including a two-dimensional image as a frame. In this specification, the motion image will also be referred to as displacement video.

[0081] <Set local coordinate system>

[0082] The displacement vector is represented as a coordinate in an arbitrary coordinate system, and the coordinate value is stored in a two-dimensional image. As such a coordinate system for representing the displacement vector, for example, there is a local coordinate system set for each vertex (for each displacement vector). That is, the displacement vector is represented in the local coordinate system for each displacement vector. In this specification, the coordinate representing the displacement vector in the local coordinate system will also be referred to as the local coordinate.

[0083] The same local coordinate system needs to be applied to the same displacement vectors in the encoder and decoder, but the local coordinate system is not explicitly sent from the encoder to the decoder. The encoder and decoder conventionally use a similar method to set the local coordinate system with reference to the normal vector of each vertex used for shading, etc. when rendering. For example, a Cartesian coordinate system with the normal vector as one of the axes is set as the local coordinate system. In this specification, the normal vector of the vertex of the mesh will also be referred to as the vertex normal vector.

[0084] This vertex normal vector is obtained by an "interpolation method" to be suitable for shadow processing, etc. during rendering. In the interpolation method, the vertex normal vector of the base mesh is first obtained, and then the vertex normal vector of the subdivision point is obtained using the vertex normal vector. At this time, the vertex normal vector of the base mesh is weighted according to the position of the subdivision point (the number of subdivisions), and the vertex normal vector of the subdivision point is obtained by weighted averaging. Through this method, even if multiple subdivision points are formed between the vertices of the base mesh, the direction of the vertex normal vector of each subdivision point can be changed.

[0085] For example, Figure 4 As shown, it is assumed that a face (polygon) whose edge is a connection connecting vertex 42-1 to vertex 42-3 of the base mesh is subdivided, and subdivision points 44-1 to subdivision points 44-3 are formed on the edge between vertex 42-1 and vertex 42-2. The vertex normal vectors of these vertices are derived based on the normal vectors of the face including the vertex (also referred to as the surrounding face). In the "interpolation method", as described above, the normal vectors of the surrounding faces are weighted according to the positions of the respective subdivision points (the number of subdivisions of the base mesh), and the vertex normal vector is derived by weighted averaging or the like.

[0086] Therefore, if Figure 5 As shown, vertex normal vector 51-1 of vertex 42-1, vertex normal vector 52-1 of subdivision point 44-1, vertex normal vector 52-2 of subdivision point 44-2, vertex normal vector 52-3 of subdivision point 44-3, and vertex normal vector 51-2 of vertex 42-2 can be oriented in different directions. Therefore, by adopting the vertex normal vectors obtained in this way in shadow processing, etc., an independent processing result can be obtained for each vertex, and a processing result with higher definition can be obtained.

[0087] By the way, this local coordinate system controls the quantized value of the displacement vector for each axis. Therefore, the quality of the displacement vector (i.e., the mesh to be reconstructed) may vary depending on the orientation of the local coordinate system adopted. In addition, the local coordinates representing the displacement vector may vary depending on the orientation of the local coordinate system adopted. That is, the amount of information (i.e., the amount of encoding) of the displacement video (frame image) may also vary depending on the orientation of the local coordinate system adopted. That is, the encoding efficiency is affected by the orientation of the local coordinate system adopted.

[0088] However, the orientation of the local coordinate system to be adopted in order to improve coding efficiency depends on various factors, including, for example, the shape of the object. At least, adopting a local coordinate system based on vertex normal vectors derived by the "interpolation method" described above does not always produce the best coding efficiency. For example, since weighting is performed based on the position of the vertex, the difference between the directions of the derived normal vectors of each vertex depends on, for example, Figure 5 The weighted method in the example in . However, the difference between the directions of the normal vectors of each vertex is not necessarily optimal for representing the displacement vectors.

[0089] In other words, by transferring the normal vector used for rendering to derivation of the local coordinate system, the quality of the displacement vector may be reduced or the amount of information of the displacement video may be increased, and thus, the encoding efficiency may be reduced.

[0090] <3. Local Coordinate System for Encoding>

[0091] <Method 1>

[0092] Therefore, the encoder and decoder derive the vertex normal vectors for setting the local coordinate system without using the normal vectors used for rendering. The derivation method is the same between the encoder and the decoder. That is, Figure 6 As shown in the top row of the table in , the decoder derives the vertex normal vector and applies the displacement vector to the subdivided base mesh using the local coordinate system corresponding to the derived vertex normal vector (method 1).

[0093] For example, an information processing device (also referred to as a first information processing device) may include: a base mesh decoding unit that decodes encoded data about a base mesh; a displacement video decoding unit that decodes encoded data about a displacement video, the displacement video including a 2D image storing local coordinates as displacement vectors as a frame; a vertex normal vector deriving unit that derives vertex normal vectors as normal vectors of vertices of a subdivided base mesh; and a displacement vector applying unit that applies local coordinates corresponding to the vertices as displacement vectors to the vertices of the subdivided base mesh using a local coordinate system corresponding to the vertex normal vector.

[0094] For example, in an information processing method performed by a first information processing device, encoded data about a base mesh can be decoded, encoded data about a displacement video including a 2D image storing local coordinates as displacement vectors as frames can be decoded, vertex normal vectors as normal vectors of vertices of the subdivided base mesh can be obtained, and local coordinates corresponding to the vertices of the subdivided base mesh can be applied to the vertices as displacement vectors using a local coordinate system corresponding to the vertex normal vectors.

[0095] Note that the base mesh is a mesh coarser than the original mesh generated by extracting the vertices of the original mesh to be encoded including the vertices representing the three-dimensional structure of the object and the connections. The displacement vector is the difference in position between the vertex of the subdivided base mesh and the vertex of the original mesh. The local coordinate system is a coordinate system for each vertex of the subdivided base mesh set based on the vertex normal vector. The local coordinate is the coordinate of the displacement vector representing the vertex of the subdivided base mesh in the local coordinate system corresponding to the vertex normal vector of the vertex.

[0096] By applying the local coordinates (displacement vectors) to the vertices of the base mesh, a mesh corresponding to the original mesh is reconstructed. That is, by doing so, the local coordinate system can be set independently of the vertex normal vectors used for rendering. Therefore, it is possible to suppress a reduction in encoding efficiency. In addition, since there is no need to change the method for deriving the vertex normal vectors used for rendering, it is possible to suppress a reduction in encoding efficiency without reducing the quality of rendering such as shading processing.

[0097] <Method 1-1>

[0098] Any method can be used to derive the vertex normal vector used to set the local coordinate system. For example, Figure 6 As shown in the second row from the top of the table, the vertex normal vector can be derived by the "non-interpolation method" (method 1-1). For example, in the first information processing device, the vertex normal vector deriving unit can derive the vertex normal vector of the processing target point using the normal vector of the surface surrounding the processing target point, and the processing target point is the processing target vertex of the subdivided base mesh.

[0099] The non-interpolation method is a method for deriving a vertex normal vector without performing weighting according to the position of the vertex in the above interpolation method. That is, the vertices and subdivision points of the base mesh are derived in a similar manner using the normal vectors of the surrounding faces. For example, as in Figure 7 In the virtual program shown in the rectangle 101 in FIG. 1 , the normal vector n of each face is derived from the positions of the three vertices of the face, and the average value of the normal vectors n of the surrounding faces is derived as the vertex normal vector. That is, the above-mentioned vertex normal vector deriving unit can derive the average value of the normal vectors of the surrounding faces and use the value as the vertex normal vector of the processing target point.

[0100] like Figure 4 As shown in FIG. 1 , the orientations of the faces obtained by subdividing the faces of the base mesh are all the same as the orientations of the faces of the base mesh. The subdivision points located on the edge of the base mesh (such as Figure 4 The orientation of the faces around each of the subdivision points 44-1 to 44-3 in the same manner, and the vertex normal vectors are also the same. Figure 5 In the example in , the directions of vertex normal vectors 52 - 1 to 52 - 3 are the same.

[0101] That is, by adopting the "non-interpolation method", it is possible to obtain a vertex normal vector (i.e., a local coordinate system) in a direction different from that in the case of the "interpolation method". Therefore, by adopting the "non-interpolation method", it is possible to at least suppress a reduction in encoding efficiency caused by the difference between the directions of the vertex normal vectors caused by adopting the "interpolation method".

[0102] Note that as long as the weighting method does not depend on the position of the subdivision point, the weighted average of the normal vectors of the surrounding faces can be used. That is, the above-mentioned vertex normal vector deriving unit can derive the weighted average of the normal vectors of the surrounding faces and use this value as the vertex normal vector of the processing target point.

[0103] For example, the normal vectors of the surrounding faces can be weighted according to the area of ​​the surrounding faces. In this case, for example, as in Figure 7 In the virtual program shown in the rectangle 102 in FIG. 1 , a weight value v is set for each face according to the size of the face, and the vertex normal vector is obtained by performing a weighted average on the normal vectors n of the surrounding faces using the weight value v. In this case, too, since the faces of the base mesh are uniformly (equally) subdivided, the subdivision points located on the edge of the base mesh (such as Figure 4 The faces around the subdivision points 44-1 to 44-3 in the figure have the same size and the vertex normal vectors are in the same direction. That is, the vertex normal vector deriving unit may derive a weighted average using a weight value based on the area of ​​the surrounding faces.

[0104] Furthermore, the normal vectors can be weighted according to the angles (differences in orientation) between the surrounding faces. In this case, for example, Figure 8 In the virtual program shown in the rectangle 103 in FIG. 1 , weight values ​​ai, aj, ak are set according to the angles (differences in orientation) between the faces, and the vertex normal vector is obtained by performing a weighted average on the normal vectors n of the surrounding faces using the weight values. In this case, too, the orientations of the faces obtained by subdividing the faces of the base mesh are all the same as the orientations of the faces of the base mesh. Therefore, each subdivision point located on the edge of the base mesh (such as Figure 4 The differences in orientation between the surrounding faces of the subdivision points 44-1 to 44-3 in the figure are the same, and the vertex normal vectors are in the same direction. That is, the vertex normal vector deriving unit can derive a weighted average using a weight value based on the differences in orientation between the surrounding faces.

[0105] Furthermore, the normal vectors may be weighted according to both the area of ​​the surrounding faces and the angle (difference in orientation) between the surrounding faces. In this case, for example, Figure 8 In the virtual program shown by the rectangle 104 in FIG. 1 , a weight value v is set for each face according to the size of the face, weight values ​​ai, aj, ak are set according to the angles (orientation differences) between the faces, and the vertex normal vector is obtained by performing a weighted average on the normal vectors n of the surrounding faces using these weight values. In this case, each subdivision point (such as Figure 4 The vertex normal vectors of the subdivision points 44-1 to 44-3 in the above example are in the same direction as in the above example. That is, the above vertex normal vector deriving unit can derive a weighted average using a weight value based on the difference in the area of ​​the surrounding faces and the orientation between the surrounding faces.

[0106] <Method 1-2>

[0107] In addition, if Figure 6 As shown in the third row from the top of the table, "interpolation method" or "non-interpolation method" can also be selected in the decoder, and the selected method can be used to derive the vertex normal vector (method 1-2). For example, in the first information processing device, the vertex normal vector deriving unit can select the first method (i.e., interpolation method) or the second method (i.e., non-interpolation method), and use the selected method to derive the vertex normal vector. In the first method, weighting is performed according to the position of the processing target point of the processing target vertex as the subdivided base mesh, and the normal vectors of the surrounding faces of the processing target point are used to derive the vertex normal vector of the processing target point. In the second method, the vertex normal vector of the processing target point is derived using the normal vectors of the surrounding faces without performing such weighting. This derivation method will also be referred to as the "decoder selection method" hereinafter.

[0108] In this case, the decoder and the encoder only need to select the "interpolation method" or the "non-interpolation method" by a similar method (by some predetermined method). That is, in this case, there is no need to explicitly inform the decoder about which method the encoder has selected. Therefore, the increase in the amount of encoding caused thereby can be suppressed. In addition, since the encoder and the decoder can adopt a more appropriate one of the "interpolation method" and the "non-interpolation method", the reduction in encoding efficiency can be further suppressed.

[0109] Such a selection method (a method for determining which of the "interpolation method" and the "non-interpolation method" is to be selected) may be any method. For example, it may be determined which of the "interpolation method" and the "non-interpolation method" is to be selected based on the relationship (difference in orientation) between the orientations of the faces of the base mesh (i.e., the directions of the normal vectors). For example, the above-mentioned vertex normal vector derivation unit may select the first method or the second method for each face of the base mesh based on the difference in orientation between the processing target face and the faces surrounding the processing target face (also referred to as surrounding faces).

[0110] For example, it is generally believed that in a portion where the surface of the object is flat, the original mesh and the base mesh also become flat, and the displacement vector is likely to be in the same direction (the change in orientation is small). Therefore, it is believed that if the "interpolation method" is adopted and the directions of the vertex normal vectors (that is, the orientations of the local coordinates) become different from each other, the encoding efficiency tends to decrease. For this reason, for example, when the difference in orientation between the faces of the base mesh is small, the "non-interpolation method" can be selected. For example, the above-mentioned vertex normal vector derivation unit can perform a threshold determination on the difference in orientation between the processing target face and the surrounding faces, and if the difference is less than the threshold (or less than or equal to the threshold), the "non-interpolation method" is selected, and if not, the "interpolation method" is selected.

[0111] The vertex normal vector of each vertex is derived based on the normal vectors of the faces surrounding the vertex. In other words, the normal vectors of the faces surrounding the target face are indicated by the vertex normal vectors of the three vertices of the target face. Fig. 9 In FIG. 1 , the processing target surface 111 indicated in gray is set as the processing target, and the surrounding surface 114 indicated by the dotted box is set as the surface surrounding the processing target surface 111. The normal vector 112 is the normal vector of the processing target surface 111. The normal vector 115 is the normal vector of the surrounding surface 114. Note that in Fig. 9 , only one surrounding face and its normal vector are given a reference numeral, but all faces (triangles) defined by the dotted box are surrounding faces 114, and their normal vectors are normal vectors 115. In addition, vertex normal vector 113-1, vertex normal vector 113-2, and vertex normal vector 113-3 are vertex normal vectors of respective vertices of the processing target face 111. In this case, vertex normal vector 113-1 is derived using normal vector 115 of surrounding faces 114 that share the corresponding vertex (for example, by taking an average, etc.). Similarly, vertex normal vector 113-2 and vertex normal vector 113-3 are each derived using normal vector 115 of surrounding faces 114 that share the corresponding vertex (for example, by taking an average, etc.). That is, it can be said that vertex normal vector 113-1 to vertex normal vector 113-3 represent Fig. 9 The normal vector 115 of the surrounding surface 114 is shown.

[0112] The normal vector of the processing target surface and the vertex normal vectors of the three vertices of the processing target surface can be used to obtain the difference in orientation between the processing target surface and the surrounding surfaces. For example, the inner product of the normal vector of the processing target surface and the vertex normal vector of each vertex of the processing target surface can be obtained, and the minimum value of the inner product can be compared with a threshold value, and if the minimum value of the inner product is small (or if the minimum value of the inner product is equal to the threshold value), the first method can be selected, and if not (if the minimum value of the inner product is large or if the minimum value of the inner product is equal to the threshold value), the second method can be selected.

[0113] In this way, it is expected that the improvement in coding efficiency can be suppressed.

[0114] Note that the threshold may be predetermined, or may be variable. That is, the encoder and decoder may have a common threshold in advance, or the threshold adopted by the encoder may be sent to the decoder (and the decoder may adopt the threshold). For example, the above-mentioned vertex normal vector derivation unit may compare the threshold sent from the encoder with the minimum value of the inner product. When the threshold is sent like this, the decoder does not need to maintain the threshold. In addition, the threshold can be easily made variable.

[0115] <Method 1-3>

[0116] In addition, if Figure 6 As shown in the fourth row from the top of the table, the vertex normal vector obtained by the "interpolation method" and the vertex normal vector obtained by the "non-interpolation method" can be combined together (methods 1-3). For example, in the first information processing device, the vertex normal vector deriving unit can perform weighting according to the position of the processing target point which is the processing target vertex of the subdivided base mesh, use the normal vectors of the surrounding faces of the processing target point to derive the vertex normal vector (first vertex normal vector) of the processing target point or use the normal vectors of the surrounding faces to derive the vertex normal vector (second vertex normal vector) of the processing target point without performing weighting, combine the derived first vertex normal vector and the second vertex normal vector, and use the combined result (composite vector) as the vertex normal vector of the processing target point. Hereinafter, this deriving method will also be referred to as a "combination method".

[0117] In this way, there is a possibility that a vertex normal vector in a direction different from both the case where the "interpolation method" is adopted and the case where the "non-interpolation method" is adopted can be obtained. That is, a local coordinate system with an orientation different from the orientations in these cases can be set. Therefore, by adopting this "combination method", it is possible to suppress at least a reduction in encoding efficiency due to the difference between the directions of the vertex normal vectors caused by adopting the "interpolation method" and a reduction in encoding efficiency due to the alignment of the directions of the vertex normal vectors caused by adopting the "non-interpolation method".

[0118] Note that the first vertex normal vector and the second vertex normal vector can be combined together by weighted averaging. For example, the above-mentioned vertex normal vector derivation unit can combine the derived first vertex normal vector and the second vertex normal vector by weighted averaging. For example, a weight value w can be set, and a weighted average of the first vertex normal vector (interpolation vector) and the second vertex normal vector (non-interpolation vector) can be derived as in the following expression (1), and the weighted average can be used as a composite vector (updated normal vector).

[0119] Updated normal vector = w*(non-interpolated vector)+(1-w)*(interpolated vector)...(1)

[0120] At this time, the weight value may be changed between a vertex located on the edge of the face (hereinafter also referred to as an edge vertex) and a vertex located in a portion other than the edge of the face (hereinafter also referred to as a non-edge vertex). Fig.10 In the face 121 of the basic mesh shown, the gray vertex 122 is an edge vertex, and the black vertex 123 is a non-edge vertex. For example, the above-mentioned vertex normal vector deriving unit can combine the first vertex normal vector and the second vertex normal vector corresponding to the vertices located on the edge of the face of the basic mesh by weighted averaging using a first weight value, and combine the first vertex normal vector and the second vertex normal vector corresponding to the vertices located in the portion other than the edge by weighted averaging using a second weight value different from the first weight value.

[0121] Note that the weight values ​​can vary per face, per edge, or per vertex of the base mesh.

[0122] Note that the weight value may be predetermined or may be variable. That is, the encoder and decoder may have a common weight value in advance, or the weight value adopted by the encoder may be sent to the decoder (the decoder may adopt the weight value). For example, the above-mentioned vertex normal vector derivation unit may combine the derived first vertex normal vector and second vertex normal vector by weighted averaging using the weight value sent from the encoder. When the weight value is sent like this, the decoder does not need to maintain the weight value. In addition, the weight value can be easily made variable.

[0123] <Method 1-4>

[0124] In addition, if Figure 6 As shown in the fifth row from the top of the table, a method designation flag indicating a derivation method adopted in the encoder may be sent from the encoder to the decoder (methods 1-4). For example, the first information processing device may further include a derivation method setting unit that sets a method for deriving vertex normal vectors based on a method designation flag that specifies a method for deriving vertex normal vectors, and the vertex normal vector derivation unit may derive the vertex normal vectors by the set derivation method. Such a derivation method will also be referred to as a "flag method".

[0125] In this way, the encoder can select and adopt a more suitable method from a plurality of derivation methods, and can further suppress the reduction of coding efficiency. In addition, since the decoder can adopt the derivation method based on the flag of the method, the decoder can more easily adopt the same derivation method as the encoder.

[0126] The encoder can adopt any selection method. In addition, the method used to derive the selection candidate can be any method, and any number of candidates can be prepared.

[0127] For example, the above-mentioned "interpolation method", "non-interpolation method", "decoder selection method" and "combination method" can be used as candidates. That is, the encoder can adopt any of these derivation methods. In the decoder, for example, the first method (interpolation method), the second method (non-interpolation method), the third method (decoder selection method) or the fourth method (combination method) can be selected and set as a method for deriving a vertex normal vector. In the first method, the above-mentioned derivation method setting unit performs weighting according to the position of the processing target point of the processing target vertex of the subdivided base mesh according to the method designation flag and uses the normal vectors of the surrounding faces of the processing target point to derive the vertex normal vector of the processing target point. In the second method, the derivation method setting unit uses the normal vectors of the surrounding faces to derive the vertex normal vector of the processing target point without performing weighting. In the third method, the first method or the second method is selected by a predetermined method. In the fourth method, the first vertex normal vector derived by adopting the first method and the second vertex normal vector derived by adopting the second method are combined together.

[0128] Note that the method designation flag can be sent in any data unit. For example, the method designation flag can be sent for each sequence of the original mesh, the method designation flag can be sent for each frame, the method designation flag can be sent for each base mesh, the method designation flag can be sent for each face of the base mesh, or the method designation flag can be sent for each vertex of the base mesh. In other words, the method designation flag can specify the method for deriving the vertex normal vector for each sequence of the original mesh, each frame, each base mesh, each face of the base mesh, or each vertex of the base mesh. In addition, the above-mentioned deriving method setting unit can set the deriving method for each sequence of the original mesh, the deriving method can be set for each frame, the deriving method can be set for each base mesh, the deriving method can be set for each face of the base mesh, or the deriving method can be set for each vertex of the base mesh based on the method designation flag.

[0129] Note that in the case of this flag method, the encoder sets a method for deriving vertex normal vectors, derives vertex normal vectors by adopting the derivation method and sets the local coordinate system. The encoder then generates a method designation flag that specifies the derivation method, and sends the method designation flag to the decoder.

[0130] For example, an information processing device (also referred to as a second information processing device) may include: a deriving method setting unit, which sets a method for deriving a vertex normal vector as a normal vector of a vertex of a subdivided base mesh; a vertex normal vector deriving unit, which derives the vertex normal vector by the set deriving method; a local coordinate deriving unit, which derives local coordinates representing a displacement vector in a local coordinate system corresponding to the vertex normal vector; a displacement video encoding unit, which encodes a displacement video including a 2D image in which the local coordinates are stored as a displacement vector as a frame; a method designation flag generating unit, which generates a method designation flag that specifies the set deriving method; and a method designation flag encoding unit, which encodes the method designation flag.

[0131] In addition, in the information processing method executed by the second information processing device, a method for deriving vertex normal vectors as normal vectors of vertices of a subdivided base mesh can be set, the vertex normal vectors can be derived by the set deriving method, local coordinates representing displacement vectors can be derived in a local coordinate system corresponding to the vertex normal vectors, a displacement video including a 2D image storing the local coordinates as displacement vectors as a frame can be encoded, a method specifying flag specifying the set deriving method can be generated, and the method specifying flag can be encoded.

[0132] Note that the base mesh is a mesh coarser than the original mesh generated by extracting the vertices of the original mesh to be encoded including the vertices representing the three-dimensional structure of the object and the connections. The displacement vector is the difference in position between the vertex of the subdivided base mesh and the vertex of the original mesh. The local coordinate system is a coordinate system for each vertex of the subdivided base mesh set based on the vertex normal vector. The local coordinate is the coordinate of the displacement vector representing the vertex of the subdivided base mesh in the local coordinate system corresponding to the vertex normal vector of the vertex.

[0133] In addition, in the second information processing device, the first method (interpolation method), the second method (non-interpolation method), the third method (decoder selection method) or the fourth method (combination method) can be selected and set as a method for deriving a vertex normal vector. In the first method, a deriving method setting unit performs weighting according to the position of a processing target point which is a processing target vertex of a subdivided base mesh and uses the normal vectors of the surfaces surrounding the processing target point to derive the vertex normal vector of the processing target point. In the second method, the deriving method setting unit uses the normal vectors of the surrounding surfaces to derive the vertex normal vector of the processing target point without performing weighting. In the third method, the decoder selects which of the first method and the second method to adopt. In the fourth method, the first vertex normal vector obtained by adopting the first method and the second vertex normal vector obtained by adopting the second method are combined together and the result of the combination (composite vector) is used as the vertex normal vector of the processing target point.

[0134] Note that when the third method (decoder selection method) is selected, the above-mentioned vertex normal vector derivation unit can select the first method or the second method by the same method as the method adopted by the decoder, and derive the vertex normal vector by adopting the selected method. For example, the vertex normal vector derivation unit can select the first method or the second method for each face of the base mesh based on the difference in orientation between the processing target face and the surrounding faces. For example, the vertex normal vector derivation unit can obtain the inner product of the normal vector of the processing target face and the vertex normal vector of each vertex of the processing target face, compare the minimum value of the inner product with the threshold, and if the minimum value of the inner product is smaller (or if the minimum value of the inner product is equal to the threshold), the first method is selected, and if not (if the minimum value of the inner product is larger or if the minimum value of the inner product is equal to the threshold), the second method is selected. Note that the threshold can be sent to the decoder. For example, in the second information processing device, the method designation flag encoding unit can further encode the threshold.

[0135] In addition, if the fourth method (combination method) is selected, the above-mentioned vertex normal vector deriving unit can derive the first vertex normal vector by adopting the first method, derive the second vertex normal vector by adopting the second method, and combine the derived first vertex normal vector and second vertex normal vector together by weighted averaging. For example, the vertex normal vector deriving unit can combine the first vertex normal vector and the second vertex normal vector corresponding to the vertices located on the edge of the face of the base mesh by weighted averaging using a first weight value. Then, the vertex normal vector deriving unit can combine the first vertex normal vector and the second vertex normal vector corresponding to the vertices located in the part other than the edge by weighted averaging using a second weight value different from the first weight value. Note that the threshold value can be sent to the decoder. For example, in the second information processing device, the above-mentioned method specifies that the flag encoding unit can further encode the weight value used for weighted averaging.

[0136] As described above, the method designation flag can be sent in any data unit. For example, the method designation flag can be sent for each sequence of the original mesh, the method designation flag can be sent for each frame, the method designation flag can be sent for each base mesh, the method designation flag can be sent for each face of the base mesh, or the method designation flag can be sent for each vertex of the base mesh. In other words, in the second information processing device, the method designation setting unit can set the method for deriving the vertex normal vector for each sequence of the original mesh, for each frame, for each base mesh, for each face of the base mesh, or for each vertex of the base mesh. In addition, in other words, in the second information processing device, the method designation flag generation unit can generate the method designation flag for each sequence of the original mesh, for each frame, for each base mesh, for each face of the base mesh, or for each vertex of the base mesh.

[0137] <Method 1-5>

[0138] In addition, if Figure 6 As shown in the sixth row from the top of the table, in the encoder, the derived vertex normal vectors can be clustered using the pre-prepared vertex normal vector candidates (methods 1-5). For example, the first information processing device may further include a clustering unit that clusters the derived vertex normal vectors with the pre-prepared predetermined normal vector candidates. In addition, the second information processing device may further include a clustering unit that clusters the derived vertex normal vectors with the pre-prepared predetermined normal vector candidates.

[0139] like Fig.11 As shown, for example, when six directions along the respective coordinate axes of the Cartesian coordinate system are set as vertex normal vector candidates and the vertex normal vector 131 is derived by a certain deriving method, the clustering unit replaces the vertex normal vector 131 with a candidate (vertex normal vector 132) having the closest orientation. Then, the local coordinate system is set using the vertex normal vector 132 after the replacement.

[0140] By clustering the vertex normal vectors like this, the directions of the vertex normal vectors are limited to the directions of the candidates prepared in advance. That is, it is possible to suppress the increase in the variation in the direction of each vertex normal vector. In other words, by such clustering, the orientation of the local coordinate system is corrected and limited to the orientation of the candidates prepared in advance. That is, it is possible to suppress the increase in the variation in the orientation of each local coordinate system.

[0141] Method 1-5 can be used in combination with any of the above methods 1-1 to 1-4. That is, in any of the above methods 1-1 to 1-4, the obtained vertex normal vectors can be clustered.

[0142] Notice, Fig.11 The examples in are examples, and any number of candidates can be used. In addition, the direction of the candidate vector can be any direction and is not limited to Fig.11 Examples in .

[0143] <Method 1-6>

[0144] In addition, if Figure 6 As shown in the bottom row of the table in , the obtained vertex normal vector (orientation) can be quantized in the encoder (methods 1-6). For example, the first information processing device may further include a quantization unit for quantizing the obtained vertex normal vector. In addition, the second information processing device may further include a quantization unit for quantizing the obtained vertex normal vector.

[0145] For example, Fig.12 As shown on the left side of , when the vertex normal vector 141 is derived by a certain deriving method, the quantization unit moves the vertex normal vector 131 to a predetermined grid (quantizes the coordinates). Then, the local coordinate system is set using the vertex normal vector 142 obtained as a result of the quantization.

[0146] By quantizing the vertex normal vectors like this, the directions of the vertex normal vectors are limited. That is, it is possible to suppress the increase in the variation in the direction of each vertex normal vector. In other words, by such quantization, the orientation of the local coordinate system is corrected and limited to the orientation prepared in advance. That is, it is possible to suppress the increase in the variation in the orientation of each local coordinate system.

[0147] Note that any quantization method can be used as long as the direction of the vertex normal vector (i.e., the local coordinate) can be corrected. Fig.12 As shown on the right side of , a spherical coordinate system can be used to quantify the vertex normal vector (direction). For example, in the case of this spherical coordinate system, θ, r, etc. Of course, the quantification method is not limited to Fig.12 Examples in .

[0148] Method 1-6 can be used in combination with any of the above methods 1-1 to 1-4. That is, in any of the above methods 1-1 to 1-4, the obtained vertex normal vector can be quantized. Of course, method 1-5 and method 1-6 can be used in combination. That is, both quantization and clustering can be performed on the vertex normal vector.

[0149] Note that the above methods (methods 1-1 to 1-6) can be used in combination with other methods. For example, the method to be adopted can be switched in the middle of the sequence. For example, method 1-1 can be adopted in a certain frame, method 1-2 can be adopted in another frame, and method 1-3 can be adopted in another frame.

[0150] <4. Implementation Method>

[0151] <Encoding device>

[0152] The present technology can be applied to an encoding device for encoding a grid. Fig.13 : is a block diagram showing an example of the configuration of an encoding device as one aspect of an information processing device to which the present technology is applied. Fig.13 The encoding device 200 shown in FIG. 1 is a device that encodes a lattice. The encoding device 200 encodes a lattice by a method basically similar to the V-DMC described in Non-Patent Document 1.

[0153] However, the encoding device 200 encodes the mesh by adopting the method 1 described above in <3. Local coordinate system for encoding> In addition, the encoding device 200 may adopt one or more of the above-described methods 1-1 to 1-3.

[0154] Notice, Fig.13 The main processing parts, main data flows, etc. are shown, and the processing parts, data flows, etc. are not limited to Fig.13 That is, in the encoding device 200, there may be Fig.13 The processing section is not shown as a block, and may exist Fig.13 Processing units and data flows such as arrows are not shown.

[0155] like Fig.13 As shown, the encoding device 200 includes a basic grid encoding unit 211, a local coordinate system setting unit 212, a local coordinate derivation unit 213, a displacement vector correction unit 214, a packing unit 215, a displacement video encoding unit 216, a grid reconstruction unit 217, an attribute map correction unit 218, an attribute video encoding unit 219, a header encoding unit 220 and a combination unit 221.

[0156] The base mesh, displacement vectors and property maps are provided for each frame to the encoding device 200. This data may be generated from the original mesh, for example, in pre-processing.

[0157] The base grid encoding unit 211 encodes the base grid and provides the encoded data to the combining unit 221. This encoding method may be any method. For example, the base grid of each frame may be encoded independently (intra-frame encoding). In addition, the base grid of the target frame may be encoded as a difference (motion vector) from the base grid of the reference frame (inter-frame encoding).

[0158] In addition, the base mesh encoding unit 211 can decode the generated encoded data to generate (restore) the base mesh. The generated (restored) base mesh includes encoding distortion. The base mesh encoding unit 211 provides the base mesh to the displacement vector correction unit 214 and the mesh reconstruction unit 217.

[0159] The local coordinate system setting unit 212 obtains a base mesh and sets a local coordinate system for each vertex of the subdivided base mesh. The local coordinate system setting unit 212 sets the local coordinate system by adopting the above-mentioned method 1. Note that the local coordinate system setting unit 212 can set the local coordinate system by adopting one or more of the above-mentioned methods 1-1 to 1-3. The local coordinate system setting unit 212 provides the set local coordinate system to the local coordinate derivation unit 213. Note that the local coordinate system setting unit 212 can adopt method 1-2 and provide the threshold used in the decoder selection method to the header encoding unit 220. Alternatively, the local coordinate system setting unit 212 can adopt method 1-3 and provide the weight value used in the combination method to the header encoding unit 220.

[0160] The local coordinate deriving section 213 obtains the displacement vector and the local coordinate system provided from the local coordinate system setting section 212, and derives local coordinates representing the displacement vector in the local coordinate system. The local coordinate deriving section 213 derives the local coordinates by adopting the above-mentioned method 1. The local coordinate deriving section 213 provides the derived local coordinates to the displacement vector correcting section 214 as a displacement vector.

[0161] The displacement vector correction section 214 obtains the base mesh including the encoding distortion provided from the base mesh encoding section 211. The displacement vector correction section 214 also obtains the local coordinates (displacement vectors) provided from the local coordinate derivation section 213. The displacement vector correction section 214 subdivides the base mesh including the encoding distortion, and corrects the local coordinates (displacement vectors) using the subdivided base mesh. The displacement vector correction section 214 provides the corrected local coordinates (displacement vectors) to the packing section 215. Note that this correction may be omitted.

[0162] The packing unit 215 obtains the local coordinates (displacement vectors) provided from the displacement vector correction unit 214. The packing unit 215 packs the local coordinates (displacement vectors) into a two-dimensional image (also referred to as a frame image) of the current frame. At this time, the packing unit 215 may perform a coefficient transformation (e.g., a wavelet transformation) on the local coordinates (displacement vectors) and pack the transformation coefficients into the two-dimensional image. In addition, the packing unit 215 may quantize the local coordinates (or transformation coefficients) and pack the quantized coefficients into the two-dimensional image. The packing unit 215 provides the two-dimensional image in which the local coordinates (or information corresponding thereto) are packed as displacement vectors to the displacement video encoding unit 216.

[0163] The displacement video encoding section 216 obtains the two-dimensional image provided from the packing section 215. The displacement video encoding section 216 sets the two-dimensional image as a frame image, and encodes the frame image as a motion image (displacement video). The displacement video encoding section 216 encodes the displacement video by adopting the above-mentioned method 1. The displacement video encoding section 216 provides the encoded data on the generated displacement video to the mesh reconstruction section 217 and the combination section 221.

[0164] The mesh reconstruction section 217 decodes the encoded data about the displacement video provided from the displacement video encoding section 216, and derives the local coordinates (displacement vectors) by, for example, unpacking the two-dimensional image. In addition, the mesh reconstruction section 217 subdivides the base mesh (including encoding distortion) provided from the base mesh encoding section 211, and reconstructs the mesh by applying the derived displacement vectors. The mesh includes encoding distortion. The mesh reconstruction section 217 provides the reconstructed mesh to the attribute map correction section 218.

[0165] The property map correction section 218 obtains a property map such as texture, and corrects the property map using the mesh (including encoding distortion) supplied from the mesh reconstruction section 217. The property map correction section 218 supplies the corrected property map to the property video encoding section 219. Note that this correction may be omitted.

[0166] The attribute video encoding section 219 sets the attribute map supplied from the attribute map correction section 218 as a frame image, and encodes the frame image into a moving image (attribute video). The attribute video encoding section 219 supplies encoded data on the generated attribute video to the combining section 221.

[0167] The header encoding unit 220 adopts the above-mentioned method 1 and encodes the information stored in the header. For example, the header encoding unit 220 may adopt the above-mentioned method 1-2 and encode the threshold used in the decoder selection method. Alternatively, the header encoding unit 220 may adopt the above-mentioned method 1-3 and encode the weight value used in the combination method. That is, the threshold value and the weight value may be stored in the header and transmitted. The header encoding unit 220 provides the encoded data about the generated header to the combination unit 221.

[0168] The combining section 221 combines (multiplexes) the coded data on the header, the coded data on the base grid, the coded data on the displacement video, and the coded data on the attribute video provided thereto to generate a bit stream. The combining section 221 outputs the generated bit stream to the outside of the encoding device 200. The bit stream is sent to the decoding device via any transmission medium or any storage medium.

[0169] <Local Coordinate System Setting Section>

[0170] Fig.14 2 is a block diagram showing a main configuration example of the local coordinate system setting section 212. Fig.14 As shown, the local coordinate system setting unit 212 includes a vertex normal vector deriving unit 241 and a local coordinate system setting unit 242 .

[0171] The vertex normal vector deriving unit 241 adopts the above-mentioned method 1 to subdivide the provided base mesh and derive the vertex normal vectors of the vertices of the subdivided base mesh. Note that the vertex normal vector deriving unit 241 can derive the vertex normal vector by adopting one or more of the above-mentioned methods 1-1 to 1-3. For example, the vertex normal vector deriving unit 241 can adopt method 1-1 and derive the vertex normal vector by a non-interpolation method.

[0172] In addition, the vertex normal vector deriving unit 241 may adopt method 1-2 and derive the vertex normal vector by the decoder selection method. In this case, the vertex normal vector deriving unit 241 may provide the threshold used in the decoder selection method as a parameter to the header encoding unit 220. In addition, the vertex normal vector deriving unit 241 may adopt method 1-3 and derive the vertex normal vector by the combination method. In this case, the vertex normal vector deriving unit 241 may provide the weight value used in the combination method as a parameter to the header encoding unit 220. The vertex normal vector deriving unit 241 provides the derived vertex normal vector to the local coordinate system setting unit 242.

[0173] The local coordinate system setting unit 242 adopts the above-mentioned method 1, and sets the local coordinate system based on the vertex normal vector provided from the vertex normal vector deriving unit 241. For example, the local coordinate system setting unit 242 sets each vertex normal vector as a coordinate axis, and sets two other coordinate axes (bi-tangent, tangent, and bi-tangent) perpendicular to the vertex normal vector to set the local coordinate system as a Cartesian coordinate system. The local coordinate system setting unit 242 sets the local coordinate system for each vertex normal vector. The local coordinate system setting unit 242 provides the set local coordinate system to the local coordinate deriving unit 213.

[0174] With such a configuration, the encoding device 200 can suppress a decrease in encoding efficiency.

[0175] <Flow of the encoding process>

[0176] Will refer to Fig.15 The flowchart describes an example of the flow of the encoding process performed by the encoding device 200.

[0177] When the encoding process is started, in step S201, the base mesh encoding section 211 encodes the base mesh. In addition, the base mesh encoding section 211 decodes the generated encoded data to generate (restore) the base mesh including the encoding distortion.

[0178] In step S202, the local coordinate system setting section 212 performs a process for setting a local coordinate system and sets the local coordinate system using the above-described method 1. Note that the local coordinate system setting section 212 may perform the process by using one or more of the above-described methods 1-1 to 1-3.

[0179] In step S203 , the local coordinate deriving section 213 adopts the above-described method 1 and derives the local coordinates representing the displacement vector in the local coordinate system set in step S202 .

[0180] In step S204 , the displacement vector correction unit 214 uses the basic mesh (including encoding distortion) generated in step S201 to correct the local coordinates (displacement vectors) obtained in step S203 .

[0181] In step S205 , the packing unit 215 packs the local coordinates (displacement vectors) (or the corrected local coordinates if correction has been performed in step S204 ) into a frame (two-dimensional image).

[0182] In step S206 , the displacement video encoding unit 216 adopts the above-mentioned method 1 to set the two-dimensional image in which the local coordinates are packed as a frame image, and encodes the frame image into a displacement video.

[0183] In step S207 , the mesh reconstruction unit 217 reconstructs a mesh (including encoding distortion) using the base mesh (including encoding distortion) generated in step S201 and the encoded data on the displacement video generated in step S206 .

[0184] In step S208 , the attribute map correction unit 218 corrects the attribute map using the reconstructed mesh.

[0185] In step S209 , the attribute video encoding section 219 sets the corrected attribute map as a frame image, and encodes the frame image as an attribute video.

[0186] In step S210, the header encoding unit 220 encodes the information stored in the header. For example, the header encoding unit 220 may adopt the above-mentioned method 1-2 and encode the threshold used in the decoder selection method. Alternatively, the header encoding unit 220 may adopt the above-mentioned method 1-3 and encode the weight value used in the combination method.

[0187] In step S211, the combining unit 221 multiplexes the encoded data about the base grid generated in step S201, the encoded data about the displacement video generated in step S206, the encoded data about the attribute video generated in step S209, and the encoded data about the header generated in step S210 to generate a bit stream.

[0188] When the processing in step S211 ends, the encoding process ends.

[0189] <Flow 1 of the process for setting the local coordinate system during encoding>

[0190] Next, we will describe Fig.15 The process for setting the local coordinate system is performed in step S202 of Fig.16 The flowchart of describes an example of the flow of a process for setting a local coordinate system when method 1-1 is adopted (ie, when vertex normal vectors are derived by a non-interpolation method).

[0191] When the process for setting the local coordinate system is started, in step S231 , the vertex normal vector deriving section 241 subdivides the base mesh to generate subdivision points.

[0192] In step S232, the vertex normal vector deriving section 241 derives a vertex normal vector for each vertex of the subdivided base mesh by the “non-interpolation method” described above in <3. Local coordinate system for encoding>.

[0193] In step S233, the local coordinate system setting unit 242 sets each vertex normal vector obtained in step S232 as a coordinate axis, and sets two other coordinate axes (bitangent, tangent and bitangent) perpendicular to the vertex normal vector to set the local coordinate system as a Cartesian coordinate system.

[0194] When the processing in step S233 ends, the process for setting the local coordinate system ends, and the process returns to Fig.15 .

[0195] <Flow 2 of the process for setting the local coordinate system during encoding>

[0196] Next, we will refer to Fig.17 The flowchart of describes an example of the flow of a process for setting a local coordinate system when method 1-2 is adopted (ie, when vertex normal vectors are derived by a decoder selection method).

[0197] When the process for setting the local coordinate system is started, in step S251, the vertex normal vector deriving section 241 determines (selects) whether to derive the vertex normal vector by the “interpolation method”.

[0198] This selection method may be any method. For example, it may be determined which of the "interpolation method" and the "non-interpolation method" to select based on the relationship (difference in orientation) between the orientations of the faces of the base mesh (i.e., the directions of the normal vectors). For example, the vertex normal vector derivation unit 241 may perform a threshold determination on the difference in orientation between the processing target face and the surrounding faces, and if the difference is less than the threshold (or less than or equal to the threshold), the "non-interpolation method" is selected, and if not, the "interpolation method" is selected. For example, the vertex normal vector derivation unit 241 may obtain the inner product of the normal vector of the processing target face and the vertex normal vector of each vertex of the processing target face, compare the minimum value of the inner product with the threshold, and if the minimum value of the inner product is smaller (or if the minimum value of the inner product is equal to the threshold), the first method is selected, and if not (if the minimum value of the inner product is larger or if the minimum value of the inner product is equal to the threshold), the second method is selected.

[0199] Note that the threshold may be predetermined, or may be variable. That is, the encoder and decoder may have a common threshold in advance, or the threshold adopted by the encoder may be sent to the decoder (and the decoder may adopt the threshold).

[0200] If it is determined that the vertex normal vector is to be obtained by the “interpolation method”, the process proceeds to step S252.

[0201] In step S252 , the vertex normal vector deriving unit 241 derives the vertex normal vectors of the base mesh.

[0202] In step S253 , the vertex normal vector deriving unit 241 subdivides the base mesh.

[0203] In step S254, the vertex normal vector deriving section 241 derives the vertex normal vector of the subdivision point by the “interpolation method.” When the processing in step S254 ends, the procedure proceeds to step S257.

[0204] On the other hand, if it is determined in step S251 that the vertex normal vector is to be derived by the “non-interpolation method”, the process proceeds to step S255.

[0205] In step S255 , the vertex normal vector deriving unit 241 subdivides the base mesh.

[0206] In step S256, the vertex normal vector derivation section 241 derives the vertex normal vector for each vertex of the subdivided base mesh by the “non-interpolation method” described above in <3. Local coordinate system for encoding> When the processing in step S256 ends, the processing proceeds to step S257.

[0207] In step S257, the local coordinate system setting section 242 sets each vertex normal vector obtained as described above as a coordinate axis, and sets two other coordinate axes (bitangent, tangent and bitangent) perpendicular to the vertex normal vector to set the local coordinate system as a Cartesian coordinate system.

[0208] When the processing in step S257 ends, the process for setting the local coordinate system ends, and the process returns to Fig.15 .

[0209] <Flow 3 of the process for setting the local coordinate system during encoding>

[0210] Next, we will refer to Fig.18 The flowchart of describes an example of the flow of a process for setting a local coordinate system when methods 1-3 are adopted (ie, when vertex normal vectors are derived by a combination method).

[0211] When the process for setting the local coordinate system is started, in step S271 , the vertex normal vector deriving section 241 derives the vertex normal vectors of the base mesh.

[0212] In step S272 , the vertex normal vector deriving unit 241 subdivides the base mesh.

[0213] In step S273, the vertex normal vector deriving section 241 derives the vertex normal vector of the subdivided point by the “interpolation method.” That is, through the processing in steps S271 to S273, the vertex normal vector of each vertex of the subdivided base mesh is derived by the “interpolation method.”

[0214] In step S274, the vertex normal vector derivation section 241 derives a vertex normal vector for each vertex of the base mesh subdivided in step S272 by the “non-interpolation method” described above in <3. Local coordinate system for encoding>.

[0215] In step S275, the vertex normal vector deriving unit 241 combines the vertex normal vector derived by the "interpolation method" and the vertex normal vector derived by the "non-interpolation method" to derive a composite vector. At this time, the vertex normal vector deriving unit 241 can combine the vertex normal vector derived by the "interpolation method" and the vertex normal vector derived by the "non-interpolation method" by weighted averaging. The weight value applied to the weighted average may be different between edge vertices and non-edge vertices. In addition, the weight value can be changed for each face, can be changed for each edge, and can be changed for each vertex of the base mesh. Note that the weight value can be predetermined, or can be variable. That is, the encoder and decoder can have a common weight value in advance, or the weight value adopted by the encoder can be sent to the decoder (the decoder can adopt the weight value).

[0216] In step S276, the local coordinate system setting unit 242 sets each vertex normal vector (composite vector) obtained as described above as a coordinate axis, and sets two other coordinate axes (bitangent, tangent and bitangent) perpendicular to the vertex normal vector to set the local coordinate system as a Cartesian coordinate system.

[0217] When the processing in step S276 ends, the process for setting the local coordinate system ends, and the process returns to Fig.15 .

[0218] By performing each process as described above, the encoding device 200 can suppress a decrease in encoding efficiency.

[0219] <Decoding device>

[0220] The present technology can be applied to a decoding device that decodes encoded data on a grid. Fig.19 : is a block diagram showing an example of the configuration of a decoding device as one aspect of an information processing apparatus to which the present technology is applied. Fig.19 The illustrated decoding device 300 is a device that decodes coded data on a mesh. The decoding device 300 decodes coded data on a mesh by a method basically similar to that of V-DMC described in Non-Patent Document 1.

[0221] However, the decoding device 300 decodes the encoded data on the mesh by adopting the method 1 described above in <3. Local coordinate system for encoding> In addition, the decoding device 300 may adopt one or more of the above-described methods 1-1 to 1-3.

[0222] That is, the decoding device 300 is Fig.13 The decoding device corresponds to the encoding device 200 in the figure, and can decode the bit stream generated by the encoding device 200 to reconstruct the grid.

[0223] Notice, Fig.19 The main processing parts, main data flows, etc. are shown, and the processing parts, data flows, etc. are not limited to Fig.19 That is, in the decoding device 300, there may be Fig.19 The processing section is not shown as a block, and may exist Fig.19 Processing units or data flows such as arrows are not shown.

[0224] like Fig.19 As shown, the decoding device 300 includes a demultiplexing unit 311, a header decoding unit 312, a basic grid decoding unit 313, a subdivision unit 314, a displacement video decoding unit 315, an unpacking unit 316, a local coordinate system setting unit 317, a displacement vector application unit 318 and an attribute video decoding unit 319.

[0225] A bit stream generated by an encoding device (eg, the encoding device 200 ) that encodes a trellis by the V-DMC method is provided to the decoding device 300 .

[0226] The demultiplexing section 311 demultiplexes the bitstream and generates various coded data included in the bitstream. For example, the demultiplexing section 311 extracts coded data about the header from the bitstream and provides the coded data to the header decoding section 312. In addition, the demultiplexing section 311 extracts coded data about the base grid from the bitstream and provides the coded data about the base grid to the base grid decoding section 313. In addition, the demultiplexing section 311 extracts coded data about the displacement video from the bitstream and provides the coded data to the displacement video decoding section 315. In addition, the demultiplexing section 311 extracts coded data about the attribute video from the bitstream and provides the coded data to the attribute video decoding section 319.

[0227] The header decoding section 312 adopts the above-mentioned method 1, decodes the encoded data about the header provided from the demultiplexing section 311, and generates (restores) the information stored in the header. The header decoding section 312 appropriately provides the generated information to the base grid decoding section 313, the subdivision section 314, the displacement video decoding section 315, the unpacking section 316, the local coordinate system setting section 317, and the attribute video decoding section 319. For example, the header decoding section 312 may adopt the above-mentioned method 1-2, and provide the threshold value (threshold value used in the encoder) generated by decoding the encoded data to the local coordinate system setting section 317. In addition, the header decoding section 312 may adopt the above-mentioned method 1-3, and provide the weight value (weight value used in the encoder) generated by decoding the encoded data to the local coordinate system setting section 317.

[0228] The base mesh decoding unit 313 decodes the encoded data about the base mesh provided from the demultiplexing unit 311, and generates (restores) the base mesh. Note that the encoded data about the base mesh may be intra-frame encoded or inter-frame encoded. That is, the base mesh decoding unit 313 may perform intra-frame decoding or inter-frame decoding on the encoded data. The base mesh decoding unit 313 adopts the above-mentioned method 1 to decode the encoded data. The base mesh decoding unit 313 provides the generated base mesh to the subdivision unit 314 and the local coordinate system setting unit 317.

[0229] The subdivision section 314 subdivides the base mesh supplied from the base mesh decoding section 313 , and supplies the subdivided base mesh to the displacement vector application section 318 .

[0230] The displacement video decoding unit 315 decodes the coded data on the displacement video supplied from the demultiplexing unit 311 to generate (restore) the displacement video. The displacement video decoding unit 315 decodes the coded data by adopting the above-mentioned method 1. The displacement video decoding unit 315 supplies the generated displacement video (the current frame) to the depacketizing unit 316.

[0231] The unpacking section 316 unpacks the local coordinates (displacement vector) of the current frame of the displacement video provided from the displacement video decoding section 315. At this time, the unpacking section 316 can unpack the transform coefficients and perform coefficient transform (e.g., wavelet transform) on the transform coefficients to obtain the local coordinates (displacement vector). Alternatively, the unpacking section 316 can unpack the quantized coefficients and inversely quantize the quantized coefficients to obtain the local coordinates (displacement vector). Alternatively, the unpacking section 316 can unpack the quantized coefficients, inversely quantize the quantized coefficients to obtain the transform coefficients, and perform coefficient transform (e.g., wavelet transform) on the transform coefficients to obtain the local coordinates (displacement vector). The unpacking section 316 provides the unpacked local coordinates (displacement vector) to the displacement vector application section 318.

[0232] The local coordinate system setting unit 317 obtains the base mesh provided from the base mesh decoding unit 313, and sets the local coordinate system for each vertex of the subdivided base mesh. The local coordinate system setting unit 317 sets the local coordinate system by adopting the above-mentioned method 1. Note that the local coordinate system setting unit 317 can set the local coordinate system by adopting one or more of the above-mentioned methods 1-1 to 1-3. Note that the local coordinate system setting unit 317 sets the local coordinate system by the same method as the method adopted by the encoder (for example, the local coordinate system setting unit 212 of the encoding device 200). The local coordinate system setting unit 317 provides the set local coordinate system to the displacement vector application unit 318. Note that the local coordinate system setting unit 317 can adopt method 1-2 to derive the vertex normal vector by the decoder selection method using the threshold value provided from the header decoding unit 312, and set the local coordinate system. Alternatively, the local coordinate system setting unit 317 can adopt method 1-3 to derive the vertex normal vector by the combination method using the weight value provided from the header decoding unit 312, and set the local coordinate system.

[0233] The displacement vector application unit 318 adopts the above-mentioned method 1, and applies the local coordinates (displacement vectors) provided from the unpacking unit 316 to the vertices of the subdivided base mesh provided from the subdividing unit 314 to reconstruct the mesh. That is, the displacement vector application unit 318 applies the local coordinates (displacement vectors) to the vertices of the subdivided base mesh using the local coordinate system provided from the local coordinate system setting unit 317. In this specification, the reconstructed mesh will also be referred to as a decoded mesh. That is, it can also be said that the displacement vector application unit 318 generates a decoded mesh. The displacement vector application unit 318 outputs the generated decoded mesh to the outside of the decoding device 300.

[0234] The attribute video decoding section 319 decodes the encoded data about the attribute video supplied from the demultiplexing section 311 and generates (the current frame of) the attribute video. The attribute video decoding section 319 outputs the generated current frame of the attribute video (ie, the attribute map corresponding to the decoded grid) to the outside of the decoding device 300.

[0235] <Local Coordinate System Setting Section>

[0236] Fig. 20 317 is a block diagram showing a main configuration example of the local coordinate system setting section 317. Fig. 20 As shown, the local coordinate system setting unit 317 includes a vertex normal vector deriving unit 341 and a local coordinate system setting unit 342 .

[0237] The vertex normal vector deriving unit 341 adopts the above-mentioned method 1 to subdivide the provided base mesh and derive the vertex normal vectors of the vertices of the subdivided base mesh. Note that the vertex normal vector deriving unit 341 can derive the vertex normal vector by adopting one or more of the above-mentioned methods 1-1 to 1-3. For example, the vertex normal vector deriving unit 341 can adopt method 1-1 and derive the vertex normal vector by a non-interpolation method. In addition, the vertex normal vector deriving unit 341 can adopt method 1-2 and derive the vertex normal vector by a decoder selection method. In this case, the vertex normal vector deriving unit 341 can derive the vertex normal vector using the provided threshold. In addition, the vertex normal vector deriving unit 341 can adopt method 1-3 and derive the vertex normal vector by a combination method. In this case, the vertex normal vector deriving unit 341 can derive the vertex normal vector using the provided weight value. The vertex normal vector deriving unit 341 derives the vertex normal vector by the same method as that adopted by the encoder (e.g., the vertex normal vector deriving unit 241 of the encoding device 200). The vertex normal vector deriving section 341 provides the derived vertex normal vector to the local coordinate system setting section 342 .

[0238] The local coordinate system setting unit 342 adopts the above-mentioned method 1, and sets the local coordinate system based on the vertex normal vector provided from the vertex normal vector deriving unit 341. For example, the local coordinate system setting unit 342 sets each vertex normal vector as a coordinate axis, and sets two other coordinate axes (bitangent, tangent, and bitangent) perpendicular to the vertex normal vector to set the local coordinate system as a Cartesian coordinate system. The local coordinate system setting unit 342 sets the local coordinate system for each vertex normal vector. The local coordinate system setting unit 342 provides the set local coordinate system to the displacement vector application unit 318.

[0239] With such a configuration, the decoding device 300 can suppress a decrease in encoding efficiency.

[0240] <Decoding Process Flow>

[0241] Will refer to Fig.21 The flowchart of describes an example of the flow of the decoding process performed by the decoding device 300.

[0242] When the decoding process is started, in step S301 , the demultiplexing section 311 demultiplexes a bit stream and extracts various types of encoded data.

[0243] In step S302, the header decoding section 312 decodes the encoded data on the header extracted in step S301 using the above-described method 1, and generates (restores) the information stored in the header.

[0244] In step S303 , the base mesh decoding section 313 decodes the encoded data on the base mesh extracted in step S301 using the above-described method 1, and generates (restores) the base mesh.

[0245] In step S304 , the subdivision unit 314 subdivides the basic mesh.

[0246] In step S305 , the displacement video decoding section 315 applies the above-described method 1 to decode the encoded data on the displacement video extracted in step S301 , and generates (restores) (the current frame of) the displacement video.

[0247] In step S306, the unpacking unit 316 unpacks the local coordinates (displacement vectors) from the current frame (two-dimensional image). At this time, the unpacking unit 316 may unpack the transform coefficients and perform coefficient transformation (e.g., wavelet transform) on the transform coefficients to obtain the local coordinates (displacement vectors). Alternatively, the unpacking unit 316 may unpack the quantized coefficients and perform inverse quantization on the quantized coefficients to obtain the local coordinates (displacement vectors). Alternatively, the unpacking unit 316 may unpack the quantized coefficients, perform inverse quantization on the quantized coefficients to obtain the transform coefficients, and perform coefficient transformation (e.g., wavelet transform) on the transform coefficients to obtain the local coordinates (displacement vectors).

[0248] In step S307, the local coordinate system setting unit 317 adopts the above-mentioned method 1, and performs a process for setting a local coordinate system to set the local coordinate system. Note that the local coordinate system setting unit 317 can perform the process by adopting one or more of the above-mentioned methods 1-1 to 1-3. For example, the local coordinate system setting unit 317 can adopt method 1-2 to derive the vertex normal vector by a decoder selection method using a threshold value provided from the header decoding unit 312, and set the local coordinate system. Alternatively, the local coordinate system setting unit 317 can adopt method 1-3 to derive the vertex normal vector by a combination method using the weight value provided from the header decoding unit 312, and set the local coordinate system. Note that the local coordinate system setting unit 317 sets the local coordinate system by the same method as the method adopted by the encoder (e.g., the local coordinate system setting unit 212 of the encoding device 200).

[0249] In step S308, the displacement vector application section 318 adopts the above-mentioned method 1, applies the local coordinates (displacement vectors) unpacked in step S306 to the vertices of the base mesh subdivided in step S304 using the local coordinate system set in step S307, and generates a decoded mesh.

[0250] In step S309 , the attribute video decoding section 319 decodes the encoded data on the attribute video extracted in step S301 , and generates (restores) the current frame of the attribute video (ie, attribute map).

[0251] When the processing in step S309 ends, the decoding process ends.

[0252] <Flow 1 of process for setting local coordinate system during decoding>

[0253] Next, we will describe Fig.21 The process for setting the local coordinate system is performed in step S307 of Fig. 22 The flowchart of describes an example of the flow of a process for setting a local coordinate system when method 1-1 is adopted (ie, when vertex normal vectors are derived by a non-interpolation method).

[0254] When the process for setting the local coordinate system is started, in step S331 , the vertex normal vector deriving section 341 subdivides the base mesh to generate subdivision points.

[0255] In step S332, the vertex normal vector deriving section 341 derives a vertex normal vector for each vertex of the subdivided base mesh by the “non-interpolation method” described above in <3. Local coordinate system for encoding>.

[0256] In step S333, the local coordinate system setting unit 342 sets each vertex normal vector obtained in step S332 as a coordinate axis, and sets two other coordinate axes (bitangent, tangent and bitangent) perpendicular to the vertex normal vector to set the local coordinate system as a Cartesian coordinate system.

[0257] When the processing in step S333 ends, the process for setting the local coordinate system ends, and the processing returns to Fig.21 .

[0258] <Flow 2 of process for setting local coordinate system during decoding>

[0259] Next, we will refer to Fig.23 The flowchart of describes an example of the flow of a process for setting a local coordinate system when method 1-2 is adopted (ie, when vertex normal vectors are derived by a decoder selection method).

[0260] When the process for setting the local coordinate system is started, in step S351, the vertex normal vector deriving section 341 determines (selects) whether to derive the vertex normal vector by the “interpolation method”.

[0261] The selection method may be any method as long as the selection method is the same as the method adopted by the encoder. For example, it may be determined which of the "interpolation method" and the "non-interpolation method" to select based on the relationship (difference in orientation) between the orientations of the faces of the base mesh (i.e., the directions of the normal vectors). For example, the vertex normal vector derivation unit 341 may perform a threshold determination on the difference in orientation between the processing target face and the surrounding faces, and if the difference is less than the threshold (or less than or equal to the threshold), the "non-interpolation method" is selected, and if not, the "interpolation method" is selected. For example, the vertex normal vector derivation unit 341 may obtain the inner product of the normal vector of the processing target face and the vertex normal vector of each vertex of the processing target face, compare the minimum value of the inner product with the threshold, and if the minimum value of the inner product is smaller (or if the minimum value of the inner product is equal to the threshold), the first method is selected, and if not (if the minimum value of the inner product is larger or if the minimum value of the inner product is equal to the threshold), the second method is selected.

[0262] Note that the threshold may be predetermined or may be variable. That is, the encoder and decoder may have a common threshold in advance, or the threshold used by the encoder may be sent to the decoder (the vertex normal vector derivation unit 341 may use the threshold sent from the encoder).

[0263] If it is determined that the vertex normal vector is to be obtained by the "interpolation method", the process proceeds to step S352.

[0264] In step S352 , the vertex normal vector deriving unit 341 derives the vertex normal vectors of the base mesh.

[0265] In step S353 , the vertex normal vector deriving unit 341 subdivides the base mesh.

[0266] In step S354, the vertex normal vector derivation section 341 derives the vertex normal vector of the subdivision point by the “interpolation method.” When the processing in step S354 ends, the procedure proceeds to step S357.

[0267] On the other hand, if it is determined in step S351 that the vertex normal vector is to be derived by the “non-interpolation method”, the process proceeds to step S355.

[0268] In step S355 , the vertex normal vector deriving unit 341 subdivides the base mesh.

[0269] In step S356, the vertex normal vector derivation section 341 derives the vertex normal vector for each vertex of the subdivided base mesh by the “non-interpolation method” described above in <3. Local coordinate system for encoding> When the processing in step S356 ends, the procedure proceeds to step S357.

[0270] In step S357, the local coordinate system setting section 342 sets each vertex normal vector obtained as described above as a coordinate axis, and sets two other coordinate axes (bitangent, tangent and bitangent) perpendicular to the vertex normal vector to set the local coordinate system as a Cartesian coordinate system.

[0271] When the processing in step S357 ends, the process for setting the local coordinate system ends, and the process returns to Fig.21 .

[0272] <Flow 3 of the process for setting the local coordinate system during decoding>

[0273] Next, we will refer to Fig.24 The flowchart of describes an example of the flow of a process for setting a local coordinate system when methods 1-3 are adopted (ie, when vertex normal vectors are derived by a combination method).

[0274] When the process for setting the local coordinate system is started, in step S371 , the vertex normal vector deriving section 341 derives the vertex normal vectors of the base mesh.

[0275] In step S372 , the vertex normal vector deriving unit 341 subdivides the base mesh.

[0276] In step S373, the vertex normal vector deriving section 341 derives the vertex normal vector of the subdivided point by the “interpolation method.” That is, through the processing in steps S371 to S373, the vertex normal vector of each vertex of the subdivided base mesh is derived by the “interpolation method.”

[0277] In step S374, the vertex normal vector derivation section 341 derives a vertex normal vector for each vertex of the base mesh subdivided in step S372 by the “non-interpolation method” described above in <3. Local coordinate system for encoding>.

[0278] In step S375, the vertex normal vector deriving unit 341 combines the vertex normal vector derived by the "interpolation method" and the vertex normal vector derived by the "non-interpolation method" to derive a composite vector. At this time, the vertex normal vector deriving unit 341 can combine the vertex normal vector derived by the "interpolation method" and the vertex normal vector derived by the "non-interpolation method" by weighted averaging. The weight value applied to the weighted average may be different between edge vertices and non-edge vertices. In addition, the weight value can be changed for each face, can be changed for each edge, and can be changed for each vertex of the base mesh. Note that the weight value can be predetermined or can be variable. That is, the encoder and decoder can have a common weight value in advance, or the weight value used by the encoder can be sent to the decoder (the vertex normal vector deriving unit 341 can use the weight value sent from the encoder).

[0279] In step S376, the local coordinate system setting unit 342 sets each vertex normal vector (composite vector) obtained as described above as a coordinate axis, and sets two other coordinate axes (bitangent, tangent and bitangent) perpendicular to the vertex normal vector to set the local coordinate system as a Cartesian coordinate system.

[0280] When the processing in step S376 ends, the process for setting the local coordinate system ends, and the process returns to Fig.21 .

[0281] By performing each process as described above, the decoding device 300 can suppress a decrease in encoding efficiency.

[0282] <Use of Methods 1-4>

[0283] <Local Coordinate Setting Section of Encoding Device>

[0284] The encoding device 200 may adopt the above methods 1 to 4. That is, the encoding device 200 may obtain the vertex normal vector by the "sign method", use the obtained vertex normal vector to set the local coordinate system, and use the local coordinate system to obtain the local coordinates of the displacement vector.

[0285] In this case, too, the encoding device 200 has the same Fig.13 Each processing unit performs the same Fig.13 . However, the local coordinate system setting unit 212 sets the local coordinate system by adopting the above-mentioned methods 1-4. That is, the local coordinate system setting unit 212 derives the vertex normal vector by the flag method, and sets the local coordinate system using the derived vertex normal vector. Therefore, the local coordinate system setting unit 212 generates a method designation flag indicating the deriving method adopted at this time, provides the method designation flag to the header encoding unit 220, and stores the method designation flag in the header. The header encoding unit 220 encodes the header including the method designation flag. That is, the header encoding unit 220 encodes the method designation flag. The combining unit 221 stores the encoded data about the header (method designation flag) in a bit stream. The bit stream is sent to the decoder via any transmission path or recording medium. That is, the method designation flag is sent to the decoder. Note that the method designation flag can be sent in any data unit. For example, the method specifying flag may be sent for each sequence of the original mesh, the method specifying flag may be sent for each frame, the method specifying flag may be sent for each base mesh, the method specifying flag may be sent for each face of the base mesh, or the method specifying flag may be sent for each vertex of the base mesh.

[0286] Fig.25 FIG. 2 shows a main configuration example of the local coordinate system setting section 212 in this case. Fig.25 As shown, in this case, the local coordinate system setting unit 212 has Fig.14 The illustrated configuration (the vertex normal vector derivation section 241 and the local coordinate system setting section 242 ) further includes a derivation method setting section 411 and a method designation flag generation section 412 .

[0287] The derivation method setting unit 411 adopts the above-mentioned methods 1-4, and sets the method for deriving the vertex normal vector. For example, the derivation method setting unit 411 can set "interpolation method", "non-interpolation method", "decoder selection method" and "combination method" as candidates, select one of these candidates, and adopt the selected candidate as the method for deriving the vertex normal vector. Note that the derivation method can be selected by any method. In addition, the derivation method setting unit 411 can set the method for deriving the vertex normal vector in any data unit. For example, the derivation method setting unit 411 can set the method for deriving the vertex normal vector for each sequence of the original mesh, for each frame, for each base mesh, for each face of the base mesh, or for each vertex of the base mesh. The derivation method setting unit 411 notifies the vertex normal vector derivation unit 241 and the method designation flag generation unit 412 of the set derivation method.

[0288] The vertex normal vector deriving section 241 derives the vertex normal vector by the deriving method set by the deriving method setting section 411. For example, the vertex normal vector deriving section 241 may derive the vertex normal vector by adopting an "interpolation method", a "non-interpolation method", a "decoder selection method" or a "combination method". Each deriving method is as described above.

[0289] The vertex normal vector deriving unit 241 provides the derived vertex normal vector to the local coordinate system setting unit 242. Fig.14 As in the case of , the local coordinate system setting unit 242 sets the local coordinate system using the vertex normal vector.

[0290] The method designation flag generation unit 412 generates a method designation flag, which is flag information indicating the derivation method set by the derivation method setting unit 411. The method designation flag generation unit 412 can generate the method designation flag in any data unit. For example, the method designation flag generation unit 412 can generate the method designation flag for each sequence of the original mesh, for each frame, for each base mesh, for each face of the base mesh, or for each vertex of the base mesh. The method designation flag generation unit 412 provides the generated method designation flag to the header encoding unit 220.

[0291] With such a configuration, the encoding device 200 can derive vertex normal vectors by adopting methods 1 to 4. Therefore, the encoding device 200 can suppress a decrease in encoding efficiency.

[0292] <Flow 4 of the process for setting the local coordinate system during encoding>

[0293] When using methods 1-4, Fig.15 The encoding process is performed in a similar process to the process in the case of the flowchart of . However, in step S202, the local coordinate system setting unit 212 adopts the above-mentioned methods 1-4 and performs the process for setting the local coordinate system to set the local coordinate system. In step S321, the header encoding unit 220 then encodes the header including the method designation flag generated in step S202.

[0294] Will refer to Fig.26 The flowchart of FIG. 20 describes an example of the flow of a process for setting the local coordinate system (step S202 ) in this case.

[0295] When the process for setting the local coordinate system is started, in step S401, the derivation method setting unit 411 adopts the above-mentioned methods 1-4, and sets the method for deriving the vertex normal vector. For example, the derivation method setting unit 411 can set "interpolation method", "non-interpolation method", "decoder selection method" and "combination method" as candidates, select one of these candidates, and adopt the selected candidate as the method for deriving the vertex normal vector. Note that the derivation method can be selected by any method. In addition, the derivation method setting unit 411 can set the method for deriving the vertex normal vector in any data unit. For example, the derivation method setting unit 411 can set the method for deriving the vertex normal vector for each sequence of the original mesh, for each frame, for each base mesh, for each face of the base mesh, or for each vertex of the base mesh.

[0296] In step S402, the vertex normal vector deriving unit 241 derives the vertex normal vector by the deriving method set in step S401. For example, the vertex normal vector deriving unit 241 may derive the vertex normal vector by "interpolation method", "non-interpolation method", "decoder selection method" or "combination method".

[0297] In step S403 , the local coordinate system setting section 242 sets each vertex normal vector obtained in step S402 as a coordinate axis, and sets two other coordinate axes (bitangent, tangent, and bitangent) perpendicular to the vertex normal vector to set the local coordinate system as a Cartesian coordinate system.

[0298] In step S404 , the method designation flag generation section 412 generates a method designation flag indicating the derivation method set in step S401 .

[0299] When the processing in step S404 ends, the process for setting the local coordinate system ends, and the process returns to Fig.15 .

[0300] By performing each process as described above, the encoding device 200 can derive the vertex normal vector by adopting methods 1 to 4. Therefore, the encoding device 200 can suppress a decrease in encoding efficiency.

[0301] <Local Coordinate Setting Section of Decoding Device>

[0302] Similarly, the decoding device 300 can also adopt the above-mentioned methods 1-4. That is, the decoding device 300 can derive the vertex normal vector by the "flag method", set the local coordinate system using the derived vertex normal vector, and derive the local coordinates of the displacement vector using the local coordinate system. That is, the decoding device 300 can set the method for deriving the vertex normal vector based on the method designation flag sent from the encoder, derive the vertex normal vector by the deriving method, set the local coordinate system using the derived vertex normal vector, and derive the local coordinates of the displacement vector using the local coordinate system.

[0303] In this case, too, the decoding device 300 has the same Fig.19 Each processing unit performs the same Fig.19 However, the header decoding unit 312 decodes the encoded data about the header including the method designation flag, and generates (restores) the method designation flag sent from the encoder. Then, the header decoding unit 312 provides the generated (restored) method designation flag to the local coordinate system setting unit 317.

[0304] However, the local coordinate system setting unit 317 sets the local coordinate system by adopting the above-mentioned methods 1 to 4. That is, the local coordinate system setting unit 317 derives the vertex normal vector by the flag method, and sets the local coordinate system using the derived vertex normal vector. That is, the local coordinate system setting unit 317 sets the method for deriving the vertex normal vector based on the method designation flag (the method designation flag sent from the encoder) provided from the header decoding unit 312, derives the vertex normal vector by the deriving method, and sets the local coordinate system using the derived vertex normal vector.

[0305] Note that the method-specified flag may be sent in any data unit, for example, the method-specified flag may be sent for each sequence of the original mesh, the method-specified flag may be sent for each frame, the method-specified flag may be sent for each base mesh, the method-specified flag may be sent for each face of the base mesh, or the method-specified flag may be sent for each vertex of the base mesh.

[0306] Fig. 27 FIG. 3 shows a main configuration example of the local coordinate system setting section 317 in this case. Fig. 27 As shown, in this case, the local coordinate system setting unit 317 has Fig. 20 The illustrated configuration (the vertex normal vector derivation section 241 and the local coordinate system setting section 242 ) further includes a derivation method setting section 421 .

[0307] The derivation method setting unit 421 adopts the above-mentioned methods 1-4, and sets a method for deriving a vertex normal vector. For example, the derivation method setting unit 421 obtains a method designation flag provided from the header decoding unit 312. The derivation method setting unit 421 sets the derivation method indicated by the method designation flag as the method for deriving a vertex normal vector. The derivation method setting unit 421 provides the set derivation method to the vertex normal vector derivation unit 341. Note that the derivation method setting unit 421 can obtain other parameters (e.g., a threshold value, a weight value, etc. sent from the encoder) as needed, and provide the parameters to the vertex normal vector derivation unit 341.

[0308] The vertex normal vector deriving section 341 derives the vertex normal vector by the deriving method set by the deriving method setting section 421. For example, the vertex normal vector deriving section 341 may derive the vertex normal vector by adopting an "interpolation method", a "non-interpolation method", a "decoder selection method" or a "combination method". Each deriving method is as described above.

[0309] The vertex normal vector deriving unit 341 provides the derived vertex normal vector to the local coordinate system setting unit 342. Fig. 20 As in the case of , the local coordinate system setting section 342 sets the local coordinate system using the vertex normal vector, and provides the local coordinate system to the displacement vector application section 318.

[0310] With such a configuration, the decoding device 300 can derive vertex normal vectors by adopting methods 1 to 4. Therefore, the decoding device 300 can suppress a decrease in encoding efficiency.

[0311] <Flow 4 of the process for setting the local coordinate system during decoding>

[0312] When using methods 1-4, Fig.21 The decoding process is performed in a similar process to the process in the case of the flowchart of . However, in step S302, the header decoding unit 312 decodes the encoded data about the header including the method designation flag, etc. sent from the encoder. That is, the method designation flag sent from the encoder is generated (restored).

[0313] In step S307 , the local coordinate system setting section 317 then adopts the above-described methods 1 to 4 and performs a process for setting a local coordinate system to set a local coordinate system.

[0314] Will refer to Fig.28 The flowchart of FIG. 1 describes an example of the flow of the process (step S307 ) for setting the local coordinate system in this case.

[0315] When the process for setting the local coordinate system is started, in step S421, the method setting unit 421 adopts the above-mentioned methods 1-4 and sets the method for deriving the vertex normal vector. For example, the method setting unit 421 sets the method for deriving the vertex normal vector based on the method designation flag sent from the encoder. For example, the method setting unit 421 can select the method specified by the method designation flag from "interpolation method", "non-interpolation method", "decoder selection method" and "combination method", and adopt the selected method as the method for deriving the vertex normal vector. Note that the method for deriving can be selected by any method. In addition, the method setting unit 411 can set the method for deriving the vertex normal vector in any data unit. For example, the method setting unit 411 can set the method for deriving the vertex normal vector for each sequence of the original mesh, for each frame, for each base mesh, for each face of the base mesh, or for each vertex of the base mesh.

[0316] In step S422, the vertex normal vector deriving unit 341 derives the vertex normal vector by the deriving method set in step S421. For example, the vertex normal vector deriving unit 341 may derive the vertex normal vector by "interpolation method", "non-interpolation method", "decoder selection method" or "combination method".

[0317] In step S423, the local coordinate system setting unit 342 sets each vertex normal vector obtained in step S422 as a coordinate axis, and sets two other coordinate axes (bitangent, tangent and bitangent) perpendicular to the vertex normal vector to set the local coordinate system as a Cartesian coordinate system.

[0318] When the processing in step S423 ends, the process for setting the local coordinate system ends, and the processing returns to Fig.21 .

[0319] By performing each process as described above, the decoding device 300 can derive the vertex normal vector by adopting methods 1 to 4. Therefore, the decoding device 300 can suppress a decrease in encoding efficiency.

[0320] <Adoption of Methods 1-5>

[0321] <Local Coordinate Setting Section of Encoding Device>

[0322] The encoding device 200 may adopt the above methods 1 to 5. That is, the encoding device 200 may cluster the obtained vertex normal vectors using the pre-prepared vertex normal vector candidates.

[0323] In this case, too, the encoding device 200 has the same Fig.13 Each processing unit performs the same Fig.13 However, the local coordinate system setting unit 212 sets the local coordinate system by adopting the above-mentioned methods 1 to 5. That is, the local coordinate system setting unit 212 clusters the obtained vertex normal vectors using the pre-prepared vertex normal vector candidates.

[0324] Fig.29 FIG. 2 shows a main configuration example of the local coordinate system setting section 212 in this case. Fig.29 As shown, in this case, the local coordinate system setting unit 212 is Fig.14 The vertex normal vector clustering section 431 is included between the vertex normal vector deriving section 241 and the local coordinate system setting section 242 .

[0325] The vertex normal vector deriving unit 241 derives the vertex normal vector. The deriving method may be any method. For example, the vertex normal vector deriving unit 241 may derive the vertex normal vector by adopting an "interpolation method", a "non-interpolation method", a "decoder selection method" or a "combination method". Each deriving method is as described above. The vertex normal vector deriving unit 241 provides the derived vertex normal vector to the vertex normal vector clustering unit 431. In addition, the vertex normal vector deriving unit 241 provides the parameters (such as a threshold value or a weight value) used to derive the vertex normal vector to the header encoding unit 220 as needed, stores the parameters in the header and encodes the parameters.

[0326] The vertex normal vector clustering section 431 clusters the vertex normal vectors supplied from the vertex normal vector deriving section 241. For example, the vertex normal vector clustering section 431 performs clustering using predetermined normal vector candidates prepared in advance. The vertex normal vector clustering section 431 supplies the vertex normal vectors that replace the derived vertex normal vectors as a result of the clustering to the local coordinate system setting section 242.

[0327] By such a configuration, the encoding device 200 can limit the orientation of the local coordinate system to the orientations of the candidates prepared in advance by adopting the methods 1 to 5. Therefore, the encoding device 200 can suppress a decrease in encoding efficiency.

[0328] <Flow 5 of the process for setting the local coordinate system during encoding>

[0329] When using methods 1-5, Fig.15 The encoding process is performed in a similar process as in the case of the flowchart of . However, in step S202, the local coordinate system setting unit 212 adopts the above-mentioned method 1-5 and performs the process for setting the local coordinate system to set the local coordinate system. That is, the local coordinate system setting unit 212 limits the orientation of the local coordinate system to the orientation of the candidate prepared in advance.

[0330] Will refer to Fig.30 The flowchart of FIG. 20 describes an example of the flow of a process for setting the local coordinate system (step S202 ) in this case.

[0331] When the process for setting the local coordinate system is started, the vertex normal vector deriving section 241 derives the vertex normal vector in step S441. The deriving method may be any method. For example, an "interpolation method", "non-interpolation method", "decoder selection method" or "combination method" may be used.

[0332] In step S442, the vertex normal vector clustering section 431 clusters the vertex normal vectors obtained in step S441. For example, the vertex normal vector clustering section 431 performs clustering using predetermined normal vector candidates prepared in advance.

[0333] In step S443, the local coordinate system setting unit 242 sets each vertex normal vector obtained in step S442 as a coordinate axis, and sets two other coordinate axes (bitangent, tangent and bitangent) perpendicular to the vertex normal vector to set the local coordinate system as a Cartesian coordinate system.

[0334] When the processing in step S443 ends, the process for setting the local coordinate system ends, and the process returns to Fig.15 .

[0335] By performing each process as described above, the encoding device 200 can derive the vertex normal vector by adopting methods 1 to 5. Therefore, the encoding device 200 can suppress a decrease in encoding efficiency.

[0336] <Combination with Methods 1-4>

[0337] Note that although the case where method 1-5 is adopted in the encoding device 200 capable of adopting method 1-1 to method 1-3 has been described in the above description, method 1-5 may be adopted in an encoding device capable of adopting method 1-4. In this case, Fig.25 In the local coordinate system setting section 212 in the configuration example of , the above-mentioned vertex normal vector clustering section 431 may be provided between the vertex normal vector deriving section 241 and the local coordinate system setting section 242. That is, in this case as well, the vertex normal vector clustering section 431 clusters the vertex normal vectors derived by the vertex normal vector deriving section 241, and provides the clustered vertex normal vectors to the local coordinate system setting section 242.

[0338] With such a configuration, the encoding device 200 can derive vertex normal vectors by adopting methods 1 to 5. Therefore, the encoding device 200 can suppress a decrease in encoding efficiency.

[0339] Furthermore, in this case, Fig.26 In the process for setting the local coordinate system in the example of , only the processing in the above-mentioned step S442 needs to be performed between step S402 and step S403.

[0340] By performing each process like this, the encoding device 200 can derive the vertex normal vector by adopting methods 1 to 5. Therefore, the encoding device 200 can suppress a decrease in encoding efficiency.

[0341] <Local Coordinate Setting Section of Decoding Device>

[0342] Similarly, the decoding device 300 may also adopt the above methods 1 to 5. That is, the decoding device 300 may cluster the obtained vertex normal vectors using the pre-prepared vertex normal vector candidates.

[0343] In this case, too, the decoding device 300 has the same Fig.19 Each processing unit performs the same Fig.19 The processing is similar to the processing in the case of . However, the local coordinate system setting unit 317 sets the local coordinate system by adopting the above-mentioned methods 1 to 5. That is, the local coordinate system setting unit 317 clusters the obtained vertex normal vectors using the pre-prepared vertex normal vector candidates.

[0344] Fig.31 FIG. 4 shows a main configuration example of the local coordinate system setting section 317 in this case. Fig.31 As shown, in this case, the local coordinate system setting unit 317 is Fig. 20 The vertex normal vector clustering section 441 is included between the vertex normal vector deriving section 341 and the local coordinate system setting section 342 .

[0345] The vertex normal vector deriving unit 341 derives the vertex normal vector. The deriving method may be any method. For example, the vertex normal vector deriving unit 341 may derive the vertex normal vector by adopting an "interpolation method", a "non-interpolation method", a "decoder selection method" or a "combination method". Each deriving method is as described above. The vertex normal vector deriving unit 341 provides the derived vertex normal vector to the vertex normal vector clustering unit 441. In addition, the vertex normal vector deriving unit 341 may obtain parameters such as a threshold value or a weight value provided from the encoder as needed, and use the parameters to derive the vertex normal vector.

[0346] The vertex normal vector clustering section 441 clusters the vertex normal vectors supplied from the vertex normal vector deriving section 341. For example, the vertex normal vector clustering section 441 performs clustering using predetermined normal vector candidates prepared in advance. The vertex normal vector clustering section 441 supplies the vertex normal vectors that replace the derived vertex normal vectors as a result of the clustering to the local coordinate system setting section 342.

[0347] By such a configuration, the decoding device 300 can limit the orientation of the local coordinate system to the orientations of the candidates prepared in advance by adopting the methods 1 to 5. Therefore, the decoding device 300 can suppress a decrease in encoding efficiency.

[0348] <Flow 5 of the process for setting the local coordinate system during decoding>

[0349] When using methods 1-5, Fig.21 The decoding process is performed in a similar process as in the case of the flowchart of . However, in step S307, the local coordinate system setting unit 317 adopts the above-mentioned method 1-5 and performs the process for setting the local coordinate system to set the local coordinate system. That is, the local coordinate system setting unit 317 limits the orientation of the local coordinate system to the orientation of the candidate prepared in advance.

[0350] Will refer to Fig.32 The flowchart of FIG. 1 describes an example of the flow of the process (step S307 ) for setting the local coordinate system in this case.

[0351] When the process for setting the local coordinate system is started, the vertex normal vector deriving section 341 derives the vertex normal vector in step S461. The deriving method may be any method. For example, an "interpolation method", "non-interpolation method", "decoder selection method" or "combination method" may be used.

[0352] In step S462, the vertex normal vector clustering section 441 clusters the vertex normal vectors obtained in step S461. For example, the vertex normal vector clustering section 441 performs clustering using predetermined normal vector candidates prepared in advance.

[0353] In step S463, the local coordinate system setting unit 342 sets each vertex normal vector obtained in step S462 as a coordinate axis, and sets two other coordinate axes (bitangent, tangent and bitangent) perpendicular to the vertex normal vector to set the local coordinate system as a Cartesian coordinate system.

[0354] When the processing in step S463 ends, the process for setting the local coordinate system ends, and the process returns to Fig.21 .

[0355] By performing each process as described above, the decoding device 300 can derive the vertex normal vector by adopting methods 1 to 5. Therefore, the decoding device 300 can suppress a decrease in encoding efficiency.

[0356] <Combination with Methods 1-4>

[0357] Note that although the case where method 1-5 is adopted in the decoding device 300 capable of adopting method 1-1 to method 1-3 has been described in the above description, method 1-5 may be adopted in the decoding device 300 capable of adopting method 1-4. In this case, Fig. 27 In the local coordinate system setting section 317 in the configuration example of , the above-mentioned vertex normal vector clustering section 441 may be provided between the vertex normal vector deriving section 341 and the local coordinate system setting section 342. That is, in this case as well, the vertex normal vector clustering section 441 clusters the vertex normal vectors derived by the vertex normal vector deriving section 341, and provides the clustered vertex normal vectors to the local coordinate system setting section 342.

[0358] With such a configuration, the decoding device 300 can derive a vertex normal vector by adopting methods 1 to 5. Therefore, the decoding device 300 can suppress a decrease in encoding efficiency.

[0359] Furthermore, in this case, Fig.28 In the process for setting the local coordinate system in the example of , it is only necessary to perform the processing in the above-mentioned step S462 between step S422 and step S423.

[0360] By performing each process like this, the decoding device 300 can derive the vertex normal vector by adopting methods 1 to 5. Therefore, the decoding device 300 can suppress a decrease in encoding efficiency.

[0361] <Use of Methods 1-6>

[0362] <Local Coordinate Setting Section of Encoding Device>

[0363] The encoding device 200 may adopt the above methods 1 to 6. That is, the encoding device 200 may quantize the obtained vertex normal vector (orientation).

[0364] In this case, too, the encoding device 200 has the same Fig.13 Each processing unit performs the same Fig.13 However, the local coordinate system setting unit 212 sets the local coordinate system by adopting the above-mentioned methods 1-6. That is, the local coordinate system setting unit 212 quantizes the obtained vertex normal vector (orientation), for example, as shown in FIG. Fig.12 described.

[0365] Fig.33 FIG. 2 shows a main configuration example of the local coordinate system setting section 212 in this case. Fig.33 As shown, in this case, the local coordinate system setting unit 212 is Fig.14 A vertex normal vector quantization unit 451 is included between the vertex normal vector derivation unit 241 and the local coordinate system setting unit 242 .

[0366] The vertex normal vector deriving unit 241 derives the vertex normal vector. The deriving method may be any method. For example, the vertex normal vector deriving unit 241 may derive the vertex normal vector by adopting an "interpolation method", a "non-interpolation method", a "decoder selection method" or a "combination method". Each deriving method is as described above. The vertex normal vector deriving unit 241 provides the derived vertex normal vector to the vertex normal vector quantization unit 451.

[0367] Furthermore, the vertex normal vector derivation section 241 provides parameters (such as a threshold value or a weight value) for deriving the vertex normal vector to the header encoding section 220 as necessary, stores the parameters in the header, and encodes the parameters.

[0368] The vertex normal vector quantization unit 451 quantizes the vertex normal vector (the direction) provided by the vertex normal vector derivation unit 241. Quantization limits the orientation of the vertex normal vector. That is, it is possible to suppress the increase in the change in the direction of each vertex normal vector. In other words, quantization corrects and limits the orientation of the local coordinate system. That is, it is possible to suppress the increase in the change in the orientation of each local coordinate system. The vertex normal vector quantization unit 451 provides the vertex normal vector whose direction has been corrected by quantization to the local coordinate system setting unit 242.

[0369] With such a configuration, the encoding device 200 can restrict the orientation of the local coordinate system by adopting methods 1 to 6. Therefore, the encoding device 200 can suppress a decrease in encoding efficiency.

[0370] <Flow 6 of the process for setting the local coordinate system during encoding>

[0371] When using methods 1-6, Fig.15 The encoding process is performed in a similar process as in the case of the flowchart of . However, in step S202, the local coordinate system setting unit 212 adopts the above-mentioned method 1-6 and performs the process for setting the local coordinate system to set the local coordinate system. That is, the local coordinate system setting unit 212 limits the orientation of the local coordinate system to the orientation of the candidate prepared in advance.

[0372] Will refer to Fig.34 The flowchart of FIG. 20 describes an example of the flow of the process (step S202 ) for setting the local coordinate system in this case.

[0373] When the process for setting the local coordinate system is started, the vertex normal vector deriving section 241 derives the vertex normal vector in step S481. The deriving method may be any method. For example, an "interpolation method", "non-interpolation method", "decoder selection method" or "combination method" may be used.

[0374] In step S482 , the vertex normal vector quantization unit 451 quantizes (the direction of) the vertex normal vector obtained in step S481 .

[0375] In step S483, the local coordinate system setting unit 242 sets each vertex normal vector obtained in step S482 as a coordinate axis, and sets two other coordinate axes (bitangent, tangent and bitangent) perpendicular to the vertex normal vector to set the local coordinate system as a Cartesian coordinate system.

[0376] When the processing in step S483 ends, the process for setting the local coordinate system ends, and the process returns to Fig.15 .

[0377] By performing each process as described above, the encoding device 200 can derive the vertex normal vector by adopting methods 1 to 6. Therefore, the encoding device 200 can suppress a decrease in encoding efficiency.

[0378] <Combination with Methods 1-4>

[0379] Note that although the case where method 1-6 is adopted in the encoding device 200 capable of adopting method 1-1 to method 1-3 has been described in the above description, method 1-6 may be adopted in the encoding device 200 capable of adopting method 1-4. In this case, Fig.25 In the local coordinate system setting section 212 in the configuration example in , the above-mentioned vertex normal vector quantization section 451 may be provided between the vertex normal vector deriving section 241 and the local coordinate system setting section 242. That is, in this case as well, the vertex normal vector quantization section 451 quantizes (the direction of) the vertex normal vector derived by the vertex normal vector deriving section 241, and provides the quantized vertex normal vector to the local coordinate system setting section 242.

[0380] With such a configuration, the encoding device 200 can derive vertex normal vectors by adopting methods 1 to 6. Therefore, the encoding device 200 can suppress a decrease in encoding efficiency.

[0381] Furthermore, in this case, Fig.26 In the process for setting the local coordinate system in the example of , only the processing in the above-mentioned step S482 needs to be performed between step S402 and step S403.

[0382] By performing each process like this, the encoding device 200 can derive the vertex normal vector by adopting methods 1 to 6. Therefore, the encoding device 200 can suppress a decrease in encoding efficiency.

[0383] <Local Coordinate Setting Section of Decoding Device>

[0384] Similarly, the decoding device 300 may also adopt the above methods 1 to 6. That is, the decoding device 300 may quantize the obtained vertex normal vector (the direction thereof).

[0385] In this case, too, the decoding device 300 has the same Fig.19 Each processing unit performs the same Fig.19 The processing is similar to the processing in the case of . However, the local coordinate system setting unit 317 sets the local coordinate system by adopting the above-mentioned methods 1 to 6. That is, the local coordinate system setting unit 317 quantizes the (direction of) the obtained vertex normal vector.

[0386] Fig.35 FIG. 4 shows a main configuration example of the local coordinate system setting section 317 in this case. Fig.35 As shown, in this case, the local coordinate system setting unit 317 is Fig. 20 A vertex normal vector quantization unit 461 is included between the vertex normal vector derivation unit 341 and the local coordinate system setting unit 342 .

[0387] The vertex normal vector deriving unit 341 derives the vertex normal vector. The deriving method may be any method. For example, the vertex normal vector deriving unit 341 may derive the vertex normal vector by adopting an "interpolation method", a "non-interpolation method", a "decoder selection method" or a "combination method". Each deriving method is as described above. The vertex normal vector deriving unit 341 provides the derived vertex normal vector to the vertex normal vector quantization unit 461.

[0388] In addition, the vertex normal vector derivation section 341 may obtain a parameter such as a threshold value or a weight value provided from the encoder as needed, and derive a vertex normal vector using the parameter.

[0389] The vertex normal vector quantization unit 461 quantizes the vertex normal vector (the direction) provided by the vertex normal vector derivation unit 341. The vertex normal vector quantization unit 461 corrects the derived vertex normal vector (the direction) by quantization. Quantization limits the orientation of the vertex normal vector. That is, it is possible to suppress the increase in the change in the direction of each vertex normal vector. In other words, quantization corrects and limits the orientation of the local coordinate system. That is, it is possible to suppress the increase in the change in the orientation of each local coordinate system. The vertex normal vector quantization unit 461 provides the quantized vertex normal vector to the local coordinate system setting unit 342.

[0390] With such a configuration, the decoding device 300 can restrict the orientation of the local coordinate system by adopting methods 1 to 6. Therefore, the decoding device 300 can suppress a decrease in encoding efficiency.

[0391] <Flow 6 of the process for setting the local coordinate system during decoding>

[0392] When using methods 1-6, Fig.21 The decoding process is performed in a similar process as in the case of the flowchart. However, in step S307, the local coordinate system setting unit 317 adopts the above-mentioned method 1-6 and performs the process for setting the local coordinate system to set the local coordinate system. That is, the local coordinate system setting unit 317 can limit the orientation of the local coordinate system.

[0393] Will refer to Fig.36 The flowchart of FIG. 1 describes an example of the flow of the process (step S307 ) for setting the local coordinate system in this case.

[0394] When the process for setting the local coordinate system is started, the vertex normal vector deriving section 341 derives the vertex normal vector in step S501. The deriving method may be any method. For example, an "interpolation method", "non-interpolation method", "decoder selection method" or "combination method" may be used.

[0395] In step S502 , the vertex normal vector quantization unit 461 quantizes (the direction of) the vertex normal vector obtained in step S501 .

[0396] In step S503 , the local coordinate system setting section 342 sets each vertex normal vector obtained in step S502 as a coordinate axis, and sets two other coordinate axes (bitangent, tangent, and bitangent) perpendicular to the vertex normal vector to set the local coordinate system as a Cartesian coordinate system.

[0397] When the processing in step S503 ends, the process for setting the local coordinate system ends, and the process returns to Fig.21 .

[0398] By performing each process as described above, the decoding device 300 can derive the vertex normal vector by adopting methods 1 to 6. Therefore, the decoding device 300 can suppress a decrease in encoding efficiency.

[0399] <Combination with Methods 1-4>

[0400] Note that although the case where method 1-6 is adopted in the decoding device 300 capable of adopting method 1-1 to method 1-3 has been described in the above description, method 1-6 may be adopted in the decoding device 300 capable of adopting method 1-4. In this case, Fig. 27 In the local coordinate system setting section 317 in the configuration example in , the above-mentioned vertex normal vector quantization section 461 may be provided between the vertex normal vector deriving section 341 and the local coordinate system setting section 342. That is, in this case as well, the vertex normal vector quantization section 461 quantizes (the direction of) the vertex normal vector derived by the vertex normal vector deriving section 341, and provides the quantized vertex normal vector to the local coordinate system setting section 342.

[0401] With such a configuration, the decoding device 300 can derive a vertex normal vector by adopting methods 1 to 6. Therefore, the decoding device 300 can suppress a decrease in encoding efficiency.

[0402] Furthermore, in this case, Fig.28 In the process for setting the local coordinate system in the example of , only the processing in the above-mentioned step S502 needs to be performed between step S422 and step S423.

[0403] By performing each process like this, the decoding device 300 can derive the vertex normal vector by adopting methods 1 to 6. Therefore, the decoding device 300 can suppress a decrease in encoding efficiency.

[0404] <5. Appendix>

[0405] <Computer>

[0406] The above series of processes can be performed by hardware or software. When a series of processes are performed by software, the program included in the software is installed on a computer. Here, the computer includes, for example, a computer incorporated into dedicated hardware, a general-purpose personal computer capable of performing various functions by installing various programs, etc.

[0407] Fig.37 : is a block diagram showing a configuration example of hardware of a computer that executes the above-described series of processes by a program.

[0408] exist Fig.37 In a computer 900 shown in FIG. 9 , a central processing unit (CPU) 901 , a read only memory (ROM) 902 , and a random access memory (RAM) 903 are connected to one another via a bus 904 .

[0409] Furthermore, an input / output interface 910 is also connected to the bus 904. An input section 911, an output section 912, a storage section 913, a communication section 914, and a drive 915 are connected to the input / output interface 910.

[0410] The input unit 911 includes, for example, a keyboard, a mouse, a microphone, a touch panel, an input terminal, etc. The output unit 912 includes, for example, a display, a speaker, an output terminal, etc. The storage unit 913 includes, for example, a hard disk, a RAM disk, and a nonvolatile memory, etc. The communication unit 914 includes, for example, a network interface. The drive 915 drives a removable medium 921 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0411] In the computer configured as described above, for example, the CPU 901 loads the program stored in the storage unit 913 into the RAM 903 via the input / output interface 910 and the bus 904 and executes the program, thereby performing the above-described series of processing. In addition, the RAM 903 also appropriately stores data and the like necessary for the CPU 901 to perform various types of processing.

[0412] The program executed by the computer can be applied by being recorded on the removable medium 921 as, for example, a package medium, etc. In this case, by attaching the removable medium 921 to the drive 915 , the program can be installed in the storage section 913 through the input / output interface 910 .

[0413] In addition, the program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting. In this case, the program can be received by the communication section 914 and installed in the storage section 913.

[0414] Furthermore, the program may be installed in the ROM 902 or the storage section 913 in advance.

[0415] <Targets of application of this technology>

[0416] The present technology can be applied to any configuration. For example, the present technology can be applied to various electronic devices.

[0417] In addition, for example, the present technology can also be implemented as a partial configuration of a device, such as a processor as a system large-scale integration (LSI) (e.g., a video processor), a module using multiple processors (e.g., a video module), a unit using multiple modules (e.g., a video unit), or a collection obtained by further adding other functions to the unit (e.g., a video collection).

[0418] In addition, for example, the present technology can also be applied to a network system including multiple devices. For example, the present technology can be implemented as cloud computing that is shared and collaboratively processed by multiple devices via a network. For example, the present technology can be implemented in a cloud service that provides services related to images (moving images) to any terminal such as a computer, an audio-visual (AV) device, a portable information processing terminal, or an Internet of Things (IoT) device.

[0419] Note that in this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), and it does not matter whether all the components are in the same housing. Therefore, multiple devices stored in different housings and connected via a network and one device in which multiple modules are stored in one housing are both systems.

[0420] <Fields and uses of this technology>

[0421] For example, systems, devices, processing units, etc. to which the present technology is applied can be used in any field such as transportation, medical care, crime prevention, agriculture, animal husbandry, mining, beauty care, factories, household appliances, weather, and natural monitoring. In addition, its application is also arbitrary.

[0422] <Others>

[0423] Note that in this specification, a "flag" is information for identifying multiple states, and includes not only information for identifying two states of true (1) and false (0), but also information capable of identifying three or more states. Therefore, the value that a "flag" can take can be binary or ternary or more, such as 1 / 0. That is, the number of bits forming the "flag" is any number, and can be one bit or more bits. In addition, it is assumed that the identification information (including the flag) includes not only its identification information in the bit stream, but also the difference information of the identification information relative to a certain reference information in the bit stream, and therefore, in this specification, "flag" and "identification information" include not only its information, but also the difference information relative to the reference information.

[0424] In addition, various information (such as metadata) related to the coded data (bitstream) can be sent or recorded in any form, as long as the information is associated with the coded data. Here, the term "association" means, for example, allowing the use (linking) of other data when processing one data. That is, data associated with each other can be collected as one data, or can be made into separate data. For example, information associated with the coded data (image) can be sent on a transmission path different from the transmission path of the coded data (image). In addition, for example, information associated with the coded data (image) can be recorded in a recording medium different from the recording medium of the coded data (image) (or other recording areas of the same recording medium). Note that the "association" may not be the entire data, but a part of the data. For example, an image and information corresponding to the image can be associated with each other in any unit such as multiple frames, a frame, or a part within a frame.

[0425] Note that in this specification, for example, terms such as "combine", "multiplex", "add", "integrate", "include", "store", "put in", "introduce" and "insert" mean combining multiple objects into one, for example, combining encoded data and metadata into one data, and mean a method of the above-mentioned "association".

[0426] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications may be made without departing from the scope of the present technology.

[0427] For example, a configuration described as one device (or processing unit) may be divided and configured as a plurality of devices (or processing units). Conversely, a configuration described as a plurality of devices (or processing units) may also be configured as one device (or processing unit) in a combined manner. Furthermore, it goes without saying that, in addition to the above configurations, other configurations may be added to the configuration of each device (or each processing unit). Furthermore, when the configuration and operation as the entire system are substantially the same, a portion of the configuration of a certain device (or processing unit) may be included in the configuration of another device (or another processing unit).

[0428] In addition, for example, the above-mentioned program can be executed in any device. In this case, it is only necessary for the device to have necessary functions (functional blocks, etc.) and obtain necessary information.

[0429] In addition, for example, each step in a flowchart may be performed by one device, or may be performed by being shared by a plurality of devices. In addition, when a plurality of processes are included in one step, the plurality of processes may be performed by one device, or may be shared and performed by a plurality of devices. In other words, the plurality of processes included in one step may be performed as a plurality of steps. Conversely, the processes described as a plurality of steps may also be performed together as one step.

[0430] In addition, for example, in a program executed by a computer, the processing of the steps describing the program may be performed in chronological order in the order described in this specification, or may be performed individually or in parallel at a desired timing such as when a call is made. That is, as long as there is no contradiction, the processing of each step may be performed in an order different from the above order. In addition, the processing in the steps describing the program may be performed in parallel with the processing of other programs, or may be performed in combination with the processing of other programs.

[0431] In addition, for example, multiple technologies related to the present technology can be independently implemented as a single entity as long as there is no contradiction. Of course, multiple arbitrary technologies can be implemented in combination. For example, part or all of the present technology described in any embodiment can be implemented in combination with part or all of the present technology described in other embodiments. In addition, part or all of any of the above-mentioned present technologies can be implemented together with another technology not described above.

[0432] Note that the present technology may also have the following configurations.

[0433] (1) An information processing device comprising:

[0434] a derivation method setting section for setting a method for deriving vertex normal vectors as normal vectors of vertices of a subdivided base mesh;

[0435] A vertex normal vector deriving unit, which derives the vertex normal vector by a set deriving method;

[0436] A local coordinate deriving unit that derives a local coordinate representing a displacement vector in a local coordinate system corresponding to the vertex normal vector;

[0437] a displacement video encoding unit, the displacement video encoding unit encoding a displacement video, the displacement video including a 2D image storing the local coordinates as the displacement vector as a frame;

[0438] a method designation flag generating section that generates a method designation flag that designates the set derivation method; and

[0439] A method designation flag encoding unit is configured to encode the method designation flag, wherein:

[0440] The base mesh is a mesh coarser than the original mesh generated by extracting vertices of the original mesh to be encoded including the vertices representing the three-dimensional structure of the object and the connections,

[0441] The displacement vectors are the differences in position between the vertices of the subdivided base mesh and the vertices of the original mesh, and

[0442] The local coordinate system is a coordinate system for each vertex of the subdivided base mesh set based on the vertex normal vector.

[0443] (2) The information processing device according to (1), wherein:

[0444] The derivation method setting unit selects a first method, a second method, a third method or a fourth method, and sets the selected method as a method for deriving a vertex normal vector. In the first method, weighting is performed according to the position of a processing target point which is a processing target vertex of the subdivided base mesh and the vertex normal vector of the processing target point is derived using the normal vectors of the surfaces surrounding the processing target point. In the second method, the vertex normal vector of the processing target point is derived using the normal vectors of the surrounding surfaces without performing the weighting. In the third method, a decoder is caused to select which of the first method and the second method to adopt. In the fourth method, the vertex normal vector derived by adopting the first method and the vertex normal vector derived by adopting the second method are combined together.

[0445] (3) The information processing device according to (2), wherein:

[0446] When the third method is selected, the vertex normal vector deriving section selects the first method or the second method by the same method as that adopted by the decoder, and derives the vertex normal vector by adopting the selected method.

[0447] (4) The information processing device according to (3), wherein:

[0448] The vertex normal vector deriving section selects the first method or the second method for each face of the base mesh based on a difference in orientation between a processing target face and surrounding faces.

[0449] (5) The information processing device according to (4), wherein:

[0450] The vertex normal vector derivation unit obtains the inner product of the normal vector of the processing target surface and the vertex normal vector of each vertex of the processing target surface, compares the minimum value of the inner product with a threshold, and if the inner product is smaller, selects the first method, and if the inner product is larger, selects the second method.

[0451] (6) The information processing device according to (5), wherein:

[0452] The method specifies that the flag encoding unit further encodes the threshold value.

[0453] (7) The information processing device according to any one of (2) to (6), wherein:

[0454] When the fourth method is selected, the vertex normal vector deriving unit adopts the first method to derive a first vertex normal vector, adopts the second method to derive a second vertex normal vector, and combines the derived first vertex normal vector and the derived second vertex normal vector together by weighted averaging.

[0455] (8) The information processing device according to (7), wherein:

[0456] The vertex normal vector deriving unit combines the first vertex normal vector and the second vertex normal vector corresponding to vertices located on the edge of the face of the base mesh by using a weighted average of a first weight value, and combines the first vertex normal vector and the second vertex normal vector corresponding to vertices located in a portion other than the edge by using a weighted average of a second weight value different from the first weight value.

[0457] (9) The information processing device according to (7) or (8), wherein:

[0458] The method designation flag encoding unit further encodes a weight value used for the weighted average.

[0459] (10) The information processing device according to any one of (1) to (9), wherein:

[0460] The derivation method setting section sets the derivation method for each sequence of the original mesh, each frame, each base mesh, each face of the base mesh, or each vertex of the base mesh.

[0461] (11) The information processing device according to any one of (1) to (10), further comprising:

[0462] A clustering unit clusters the obtained vertex normal vectors using predetermined normal vector candidates prepared in advance.

[0463] (12) The information processing device according to any one of (1) to (11), further comprising:

[0464] A quantization unit is used to quantize the obtained vertex normal vector.

[0465] (13) An information processing method comprising:

[0466] Sets the method used to derive the vertex normal vectors that are the normal vectors of the vertices of the subdivided base mesh;

[0467] Obtain the vertex normal vector by using the set obtaining method;

[0468] Determining local coordinates representing a displacement vector in a local coordinate system corresponding to the vertex normal vector;

[0469] encoding a displacement video, the displacement video comprising a 2D image storing the local coordinates as the displacement vector as a frame;

[0470] Generates a method-specified flag specifying the derive method to be set; and

[0471] The method specifies the flag to be encoded, wherein:

[0472] The base mesh is a mesh coarser than the original mesh generated by extracting vertices of the original mesh to be encoded including the vertices representing the three-dimensional structure of the object and the connections,

[0473] The displacement vectors are the differences in position between the vertices of the subdivided base mesh and the vertices of the original mesh, and

[0474] The local coordinate system is a coordinate system for each vertex of the subdivided base mesh set based on the vertex normal vector.

[0475] (21) An information processing device comprising:

[0476] a base grid decoding unit configured to decode encoded data on the base grid;

[0477] a displacement video decoding unit that decodes coded data about a displacement video including, as a frame, a 2D image in which local coordinates are stored as displacement vectors;

[0478] a vertex normal vector deriving section that derives vertex normal vectors as normal vectors of vertices of a subdivided base mesh; and

[0479] a displacement vector application unit, which uses a local coordinate system corresponding to the vertex normal vector to apply the local coordinates of the vertex as the displacement vector to the vertex of the subdivided base mesh, wherein

[0480] The base mesh is a mesh coarser than the original mesh generated by extracting vertices of the original mesh to be encoded including the vertices representing the three-dimensional structure of the object and the connections,

[0481] The displacement vector is the difference in position between the vertices of the subdivided base mesh and the vertices of the original mesh,

[0482] The local coordinate system is a coordinate system for each vertex of the subdivided base mesh set based on the vertex normal vector, and

[0483] The local coordinates are coordinates representing a displacement vector of a vertex of the subdivided base mesh in the local coordinate system corresponding to a vertex normal vector of the vertex.

[0484] (22) The information processing device according to (21), wherein:

[0485] The vertex normal vector deriving section derives a vertex normal vector of a processing target point, which is a processing target vertex of the subdivided base mesh, using normal vectors of surfaces surrounding the processing target point.

[0486] (23) The information processing device according to (22), wherein:

[0487] The vertex normal vector deriving section derives an average of normal vectors of surrounding surfaces and uses the average as a vertex normal vector of the processing target point.

[0488] (24) The information processing device according to (22), wherein:

[0489] The vertex normal vector deriving section derives a weighted average of normal vectors of surrounding surfaces and uses the weighted average as the vertex normal vector of the processing target point.

[0490] (25) The information processing device according to (24), wherein:

[0491] The vertex normal vector deriving unit derives the weighted average using a weight value based on the areas of the surrounding faces.

[0492] (26) The information processing device according to (24), wherein:

[0493] The vertex normal vector deriving unit derives the weighted average using a weight value based on a difference in orientation between the surrounding faces.

[0494] (27) The information processing device according to (24), wherein:

[0495] The vertex normal vector deriving section derives the weighted average using a weight value based on areas of the surrounding surfaces and differences in orientations between the surrounding surfaces.

[0496] (28) The information processing device according to any one of (21) to (27), wherein:

[0497] The vertex normal vector deriving unit selects a first method or a second method, and uses the selected method to derive the vertex normal vector. In the first method, weighting is performed according to the position of a processing target point which is a processing target vertex of the subdivided base mesh, and the vertex normal vector of the processing target point is derived using the normal vectors of the faces surrounding the processing target point. In the second method, the vertex normal vector of the processing target point is derived using the normal vectors of the surrounding faces without performing the weighting.

[0498] (29) The information processing device according to (28), wherein:

[0499] The vertex normal vector deriving section selects the first method or the second method for each face of the base mesh based on a difference in orientation between a processing target face and surrounding faces.

[0500] (30) The information processing device according to (29), wherein:

[0501] The vertex normal vector derivation unit obtains the inner product of the normal vector of the processing target surface and the vertex normal vector of each vertex of the processing target surface, compares the minimum value of the inner product with a threshold, and if the inner product is smaller, selects the first method, and if the inner product is larger, selects the second method.

[0502] (31) The information processing device according to (30), wherein:

[0503] The vertex normal vector deriving section compares the threshold value sent from the encoder with the minimum value of the inner product.

[0504] (32) The information processing device according to any one of (21) to (31), wherein:

[0505] The vertex normal vector derivation unit performs the following processing:

[0506] deriving a first vertex normal vector of the processing target point by performing weighting according to the position of the processing target point as a processing target vertex of the subdivided base mesh, using normal vectors of faces surrounding the processing target point,

[0507] using the normal vectors of the surrounding faces to derive a second vertex normal vector of the processing target point without performing the weighting, and

[0508] The derived normal vector for the first vertex and the derived normal vector for the second vertex are combined together.

[0509] (33) The information processing device according to (32), wherein:

[0510] The vertex normal vector deriving unit combines the derived first vertex normal vector and the derived second vertex normal vector by weighted averaging.

[0511] (34) The information processing device according to (33), wherein:

[0512] The vertex normal vector deriving unit combines the first vertex normal vector and the second vertex normal vector corresponding to vertices located on the edge of the face of the base mesh by using a weighted average of a first weight value, and combines the first vertex normal vector and the second vertex normal vector corresponding to vertices located in a portion other than the edge by using a weighted average of a second weight value different from the first weight value.

[0513] (35) The information processing device according to (33) or (34), wherein:

[0514] The vertex normal vector deriving section combines the derived first vertex normal vector and the derived second vertex normal vector by weighted average using the weight value transmitted from the encoder.

[0515] (36) The information processing device according to any one of (21) to (35), further comprising:

[0516] a deriving method setting section that sets the method for deriving the vertex normal vector based on a method designation flag that designates a method for deriving the vertex normal vector, wherein:

[0517] The vertex normal vector deriving unit derives the vertex normal vector by using a set deriving method.

[0518] (37) The information processing device according to (36), wherein:

[0519] The derivation method setting unit selects a first method, a second method, a third method or a fourth method according to the method designation flag, and sets the selected method as the derivation method. In the first method, weighting is performed according to the position of the processing target point which is the processing target vertex of the subdivided base mesh, and the vertex normal vector of the processing target point is derived using the normal vector of the surface surrounding the processing target point. In the second method, the vertex normal vector of the processing target point is derived using the normal vector of the surrounding surface without performing the weighting. In the third method, the first method or the second method is selected by a predetermined method. In the fourth method, the first vertex normal vector derived by adopting the first method and the second vertex normal vector derived by adopting the second method are combined.

[0520] (38) The information processing device according to (36) or (37), wherein:

[0521] The method specifying flag specifies the derivation method for each sequence of the original mesh, each frame, each base mesh, each face of the base mesh, or each vertex of the base mesh.

[0522] (39) The information processing device according to any one of (21) to (38), further comprising:

[0523] A clustering unit clusters the derived vertex normal vectors using pre-prepared predetermined normal vector candidates.

[0524] (40) The information processing device according to any one of (21) to (39), further comprising:

[0525] A quantization unit is used to quantize the obtained vertex normal vector.

[0526] (41) An information processing method comprising:

[0527] Decoding the encoded data about the base grid;

[0528] decoding encoded data about a displacement video, the displacement video including, as a frame, a 2D image storing local coordinates as a displacement vector;

[0529] deriving vertex normal vectors as normal vectors of vertices of the subdivided base mesh; and

[0530] Using a local coordinate system corresponding to the vertex normal vector, the local coordinates of the vertex are applied as the displacement vector to the vertices of the subdivided base mesh, wherein,

[0531] The base mesh is a mesh coarser than the original mesh generated by extracting vertices of the original mesh to be encoded including the vertices representing the three-dimensional structure of the object and the connections,

[0532] The displacement vector is the difference in position between the vertices of the subdivided base mesh and the vertices of the original mesh,

[0533] The local coordinate system is a coordinate system for each vertex of the subdivided base mesh set based on the vertex normal vector, and

[0534] The local coordinates are coordinates representing a displacement vector of a vertex of the subdivided base mesh in the local coordinate system corresponding to a vertex normal vector of the vertex.

[0535] Reference numerals list

[0536] 200 encoding devices

[0537] 211 Basic Grid Coding Department

[0538] 212 Local coordinate system setting section

[0539] 213 Local coordinates are obtained

[0540] 214 Displacement Vector Correction Unit

[0541] 215 Packaging Department

[0542] 216 Video Coding Department

[0543] 217 Grid Reconstruction Department

[0544] 218 Attribute map correction unit

[0545] 219 attribute video encoding unit

[0546] 220 Header encoding unit

[0547] 221 Combination Department

[0548] 241 Vertex Normal Vector Derivation

[0549] 242 Local coordinate system setting section

[0550] 300 decoding equipment

[0551] 311 Demultiplexing Unit

[0552] 312 header decoding unit

[0553] 313 Basic Grid Decoding Unit

[0554] 314 Subdivision

[0555] 315 displacement video decoding unit

[0556] 316 Unpacking Department

[0557] 317 Local Coordinate System Setting Section

[0558] 318 Displacement Vector Application Department

[0559] 319 attribute video decoding unit

[0560] 341 Vertex Normal Vector Derivation

[0561] 342 Local Coordinate System Setting Section

[0562] 411 method setting section

[0563] 412 Method Designation Flag Generator

[0564] 421 method setting section

[0565] 431 Vertex Normal Vector Clustering

[0566] 441 Vertex Normal Vector Clustering

[0567] 451 Vertex Normal Vector Quantization Unit

[0568] 461 Vertex Normal Vector Quantization Unit

[0569] 900 Computer

Claims

1. An information processing device, comprising: a base grid decoding unit configured to decode encoded data on the base grid; a displacement video decoding unit that decodes coded data about a displacement video including, as a frame, a 2D image in which local coordinates are stored as displacement vectors; a vertex normal vector deriving unit that derives vertex normal vectors as normal vectors of vertices of a subdivided base mesh; as well as A displacement vector application unit, wherein the displacement vector application unit uses a local coordinate system corresponding to the vertex normal vector to apply the local coordinates of the vertex as the displacement vector to the vertex of the subdivided base mesh, wherein The base mesh is a mesh coarser than the original mesh generated by extracting vertices of the original mesh to be encoded including the vertices representing the three-dimensional structure of the object and the connections, The displacement vector is the difference in position between the vertices of the subdivided base mesh and the vertices of the original mesh, The local coordinate system is a coordinate system for each vertex of the subdivided base mesh set based on the vertex normal vector, and The local coordinates are coordinates representing a displacement vector of a vertex of the subdivided base mesh in the local coordinate system corresponding to a vertex normal vector of the vertex.

2. The information processing device according to claim 1, wherein: The vertex normal vector deriving section derives the vertex normal vector of a processing target point using normal vectors of surfaces surrounding the processing target point, the processing target point being a processing target vertex of the subdivided base mesh.

3. The information processing device according to claim 2, wherein: The vertex normal vector deriving section derives an average of normal vectors of surrounding surfaces and uses the average as a vertex normal vector of the processing target point.

4. The information processing device according to claim 2, wherein: The vertex normal vector deriving section derives a weighted average of normal vectors of surrounding surfaces and uses the weighted average as the vertex normal vector of the processing target point.

5. The information processing device according to claim 4, wherein: The vertex normal vector deriving unit derives the weighted average using a weight value based on the areas of the surrounding faces.

6. The information processing device according to claim 4, wherein: The vertex normal vector deriving unit derives the weighted average using a weight value based on a difference in orientation between the surrounding faces.

7. The information processing device according to claim 4, wherein: The vertex normal vector deriving section derives the weighted average using a weight value based on areas of the surrounding surfaces and differences in orientations between the surrounding surfaces.

8. The information processing device according to claim 1, wherein: The vertex normal vector deriving unit selects a first method or a second method, and uses the selected method to derive the vertex normal vector. In the first method, weighting is performed according to the position of a processing target point which is a processing target vertex of the subdivided base mesh, and the vertex normal vector of the processing target point is derived using the normal vectors of the faces surrounding the processing target point. In the second method, the vertex normal vector of the processing target point is derived using the normal vectors of the surrounding faces without performing the weighting.

9. The information processing device according to claim 8, wherein: The vertex normal vector deriving section selects the first method or the second method for each face of the base mesh based on a difference in orientation between a processing target face and the surrounding faces.

10. The information processing device according to claim 9, wherein: The vertex normal vector derivation unit obtains the inner product of the normal vector of the processing target surface and the vertex normal vector of each vertex of the processing target surface, compares the minimum value of the inner product with a threshold, and if the inner product is smaller, selects the first method, and if the inner product is larger, selects the second method.

11. The information processing device according to claim 10, wherein: The vertex normal vector deriving section compares the threshold value sent from the encoder with the minimum value of the inner product.

12. The information processing device according to claim 1, wherein: The vertex normal vector derivation unit performs the following processing: deriving a first vertex normal vector of the processing target point by performing weighting according to the position of the processing target point as a processing target vertex of the subdivided base mesh, using normal vectors of faces surrounding the processing target point, using normal vectors of surrounding faces to derive a second vertex normal vector of the processing target point without performing the weighting, and The derived normal vector for the first vertex and the derived normal vector for the second vertex are combined together.

13. The information processing device according to claim 12, wherein: The vertex normal vector deriving unit combines the derived first vertex normal vector and the derived second vertex normal vector by weighted averaging.

14. The information processing device according to claim 13, wherein: The vertex normal vector deriving section combines the derived first vertex normal vector and the derived second vertex normal vector by weighted average using the weight value transmitted from the encoder.

15. The information processing device according to claim 1, further comprising: a deriving method setting section that sets the method for deriving the vertex normal vector based on a method designation flag that designates a method for deriving the vertex normal vector, wherein: The vertex normal vector deriving unit derives the vertex normal vector by using a set deriving method.

16. The information processing device according to claim 1, further comprising: A clustering unit clusters the obtained vertex normal vectors using predetermined normal vector candidates prepared in advance.

17. The information processing device according to claim 1, further comprising: A quantization unit quantizes the obtained vertex normal vector.

18. An information processing method, comprising: Decoding the encoded data about the base grid; decoding encoded data about a displacement video, the displacement video including, as a frame, a 2D image storing local coordinates as a displacement vector; deriving vertex normal vectors that are normal vectors of vertices of the subdivided base mesh; and Using a local coordinate system corresponding to the vertex normal vector, the local coordinates of the vertex are applied as the displacement vector to the vertices of the subdivided base mesh, wherein, The base mesh is a mesh coarser than the original mesh generated by extracting vertices of the original mesh to be encoded including the vertices representing the three-dimensional structure of the object and the connections, The displacement vector is the difference in position between the vertices of the subdivided base mesh and the vertices of the original mesh, The local coordinate system is a coordinate system for each vertex of the subdivided base mesh set based on the vertex normal vector, and The local coordinates are coordinates representing a displacement vector of a vertex of the subdivided base mesh in the local coordinate system corresponding to a vertex normal vector of the vertex.

19. An information processing device, comprising: a derivation method setting section for setting a method for deriving vertex normal vectors as normal vectors of vertices of a subdivided base mesh; A vertex normal vector deriving unit, which derives the vertex normal vector by a set deriving method; A local coordinate deriving unit that derives a local coordinate representing a displacement vector in a local coordinate system corresponding to the vertex normal vector; a displacement video encoding unit, the displacement video encoding unit encoding a displacement video, the displacement video including a 2D image storing the local coordinates as the displacement vector as a frame; a method designation flag generating section that generates a method designation flag that designates the set derivation method; and A method designation flag encoding unit is configured to encode the method designation flag, wherein: The base mesh is a mesh coarser than the original mesh generated by extracting vertices of the original mesh to be encoded including the vertices representing the three-dimensional structure of the object and the connections, The displacement vectors are the differences in position between the vertices of the subdivided base mesh and the vertices of the original mesh, and The local coordinate system is a coordinate system for each vertex of the subdivided base mesh set based on the vertex normal vector.

20. An information processing method, comprising: Sets the method used to derive the vertex normal vectors that are the normal vectors of the vertices of the subdivided base mesh; Obtain the vertex normal vector by using the set obtaining method; Determining local coordinates representing a displacement vector in a local coordinate system corresponding to the vertex normal vector; encoding a displacement video, the displacement video comprising a 2D image storing the local coordinates as the displacement vector as a frame; Generates a method-specified flag specifying the derive method to be set; and The method specifies the flag to be encoded, wherein: The base mesh is a mesh coarser than the original mesh generated by extracting vertices of the original mesh to be encoded including the vertices representing the three-dimensional structure of the object and the connections, The displacement vectors are the differences in position between the vertices of the subdivided base mesh and the vertices of the original mesh, and The local coordinate system is a coordinate system for each vertex of the subdivided base mesh set based on the vertex normal vector.

Images