Information processing apparatus and method

Abstract

The present disclosure pertains to an information processing device and method that make it possible to improve encoding / decoding performance. The present disclosure relates to layering, according to fineness, displacement values of division points obtained by subdividing a base mesh formed by vertices and connections of a 3-dimensional structure representing an object, packing the displacement values of the layered division points into a 2D image according to a partial region of the object, and encoding a displacement video in which the 2D image is a frame. The present disclosure can be applied to, for example, an information processing device, an electronic device, an information processing method, a program, etc.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus and method, and more particularly, to an information processing apparatus and method that enable improvement in encoding / decoding performance. Background Art

[0002] From the past, there has been V-DMC (Video-based Dynamic Mesh Coding) as a coding method for a mesh, which is 3D data that represents the 3D structure of an object through vertices and connections (see, for example, non-patent document 1). In V-DMC, a mesh as a coding target is represented by a coarse base mesh and displacement values ​​of segmentation points obtained by subdividing the base mesh, and the base mesh and the displacement values ​​are encoded. The displacement values ​​are stored (packed) in a 2D image and encoded as an image. Since an object may change in the time direction (is dynamic), the mesh (i.e., the base mesh and the displacement value) is also dynamic. Therefore, the displacement value is encoded as a moving image (displacement video) using a 2D image as a frame.

[0003] Reference List

[0004] Non-patent literature

[0005] Non-Patent Literature 1: Khaled Mammou, Jungsun Kim, Alexis Tourapis, DimitriPodborski, Krasimir Kolarov, “[V-CG] Apple's Dynamic Mesh Coding CfP Response,” ISO / IEC JTC 1 / SC 29 / WG 7m59281, April 2022. Summary of the invention

[0006] Technical issues

[0007] However, in this method, the displacement video is only used to pack the displacement values, and in recent years, there has been a demand for further utilizing the displacement video.

[0008] The present disclosure has been made in view of the above circumstances, and aims to enable improvement of encoding / decoding performance by extending the use of displacement video.

[0009] Solution to the problem

[0010] An information processing device according to one aspect of the present technology is an information processing device including: a packing unit which layers displacement values ​​of segmentation points obtained by subdividing a base grid for each fineness, and packs the layered displacement values ​​of the segmentation points in a 2D image for each partial area of ​​a partial area of ​​an object; and a displacement video encoding unit which encodes a displacement video in which the 2D image is set as a frame, wherein the base grid is generated by thinning out vertices from an original grid as an encoding target composed of vertices and connections representing a 3D structure of an object, and the base grid is a grid coarser than the original grid, and the displacement value is a positional difference between the segmentation point and the vertex of the original grid.

[0011] An information processing method according to one aspect of the present technology is an information processing method, comprising: for each fineness, layering the displacement values ​​of the segmentation points obtained by subdividing the base mesh, and packing the layered displacement values ​​of the segmentation points in a 2D image for each partial area of ​​the object; and encoding a displacement video in which the 2D image is set as a frame, wherein the base mesh is generated by thinning out vertices from an original mesh as an encoding target composed of vertices and connections representing a 3D structure of the object, and the base mesh is a mesh coarser than the original mesh, and the displacement value is a positional difference between the segmentation point and the vertex of the original mesh.

[0012] An information processing device according to another aspect of the present technology is an information processing device, comprising: a displacement video decoding unit, which decodes a bit stream and generates a displacement video, in which a 2D image is set as a frame, wherein the displacement values ​​of the segmentation points obtained by subdividing the base grid are layered for each fineness and packed in the 2D image for each partial area of ​​the object; an unpacking unit, which unpacks the displacement values ​​of the segmentation points of the desired layer of the desired partial area that have been packed in the 2D image of the current frame of the displacement video; a subdivision unit, which subdivides the base grid of the current frame; and a displacement value application unit, which applies the displacement values ​​of the segmentation points to the segmentation points of the subdivided base grid, wherein the base grid is generated by sparse vertices from an original grid as an encoding target composed of vertices and connections representing the 3D structure of the object, and the base grid is a grid coarser than the original grid, and the displacement value is the position difference between the segmentation point and the vertex of the original grid.

[0013] An information processing method according to another aspect of the present technology is an information processing method, comprising: decoding a bit stream and generating a displacement video, in which a 2D image is set as a frame, wherein the displacement values ​​of the segmentation points obtained by subdividing the base grid are layered for each fineness and packed in the 2D image for each partial area of ​​the object; unpacking the displacement values ​​of the segmentation points of the desired level of the desired partial area that have been packed in the 2D image of the current frame of the displacement video; subdividing the base grid of the current frame; and applying the displacement values ​​of the segmentation points to the segmentation points of the subdivided base grid, wherein the base grid is generated by thinning out vertices from an original grid as an encoding target composed of vertices and connections representing a 3D structure of an object, and the base grid is a grid coarser than the original grid, and the displacement values ​​are the positional differences between the segmentation points and the vertices of the original grid.

[0014] An information processing device according to another aspect of the present technology is an information processing device, including: a base mesh vertex displacement value deriving unit, which derives base mesh vertex displacement values ​​as displacement values ​​of vertices of a base mesh; a packing unit, which, for each fineness, layers the displacement values ​​of a segmentation point obtained by subdividing the base mesh, and packs the base mesh vertex displacement values ​​and the layered displacement values ​​of the segmentation point in a 2D image; and a displacement video encoding unit, which encodes a displacement video in which the 2D image is set as a frame, wherein the base mesh is generated by thinning out vertices from an original mesh as an encoding target composed of vertices and connections representing a 3D structure of an object, and the base mesh is a mesh coarser than the original mesh, the displacement value of the segmentation point is a position difference between the segmentation point and a vertex of the original mesh, and the base mesh vertex displacement value is a position difference between the vertices of the base mesh and the vertices of the original mesh.

[0015] An information processing method according to another aspect of the present technology is an information processing method, comprising: deriving base mesh vertex displacement values ​​as displacement values ​​of vertices of a base mesh; layering the displacement values ​​of segmentation points obtained by subdividing the base mesh for each fineness, and packing the base mesh vertex displacement values ​​and the layered displacement values ​​of the segmentation points in a 2D image; and encoding a displacement video in which the 2D image is set as a frame, wherein the base mesh is generated by thinning out vertices from an original mesh as an encoding target composed of vertices and connections representing a 3D structure of an object, and the base mesh is a mesh coarser than the original mesh, the displacement value of the segmentation point is a position difference between the segmentation point and a vertex of the original mesh, and the base mesh vertex displacement value is a position difference between the vertices of the base mesh and the vertices of the original mesh.

[0016] According to another aspect of the present technology, an information processing device is an information processing device, comprising: a displacement video decoding unit, which decodes a bit stream and generates a displacement video, in which a 2D image is set as a frame, wherein base mesh vertex displacement values ​​as displacement values ​​of vertices of a base mesh and displacement values ​​of segmentation points obtained by subdividing the base mesh are packed in the 2D image; an unpacking unit, which unpacks the base mesh vertex displacement values ​​and the displacement values ​​of the segmentation points that have been packed in the 2D image of the current frame of the displacement video; a base mesh vertex position application unit, which applies the base mesh vertex displacement values ​​to the 2D image of the current frame of the displacement video; applied to vertices of a base mesh; a subdivision unit that subdivides the base mesh in which base mesh vertex displacement values ​​are applied to the vertices; and a displacement value applying unit that applies the displacement values ​​of the segmentation points to the segmentation points of the subdivided base mesh, wherein the base mesh is generated by thinning out vertices from an original mesh as an encoding target composed of vertices and connections representing a 3D structure of an object, and the base mesh is a mesh coarser than the original mesh, the displacement values ​​of the segmentation points are positional differences between the segmentation points and vertices of the original mesh, and the base mesh vertex displacement values ​​are positional differences between vertices of the base mesh and vertices of the original mesh.

[0017] An information processing method according to another aspect of the present technology is an information processing method, comprising: decoding a bit stream and generating a displacement video in which a 2D image is set as a frame, wherein base mesh vertex displacement values ​​as displacement values ​​of vertices of a base mesh and displacement values ​​of segmentation points obtained by subdividing the base mesh are packed in the 2D image; unpacking the base mesh vertex displacement values ​​and displacement values ​​of segmentation points that have been packed in the 2D image of a current frame of the displacement video; applying the base mesh vertex displacement values ​​to the vertices of the base mesh; subdividing the base mesh to which the base mesh vertex displacement values ​​are applied to the vertices; and applying the displacement values ​​of the segmentation points to the segmentation points of the subdivided base mesh, wherein the base mesh is generated by thinning out vertices from an original mesh as an encoding target composed of vertices and connections representing a 3D structure of an object, and the base mesh is a mesh coarser than the original mesh, the displacement value of the segmentation point is a position difference between the segmentation point and the vertex of the original mesh, and the base mesh vertex displacement value is a position difference between the vertex of the base mesh and the vertex of the original mesh.

[0018] In an information processing device and method according to one aspect of the present technology, the displacement values ​​of the segmentation points obtained by subdividing the base grid are layered for each fineness, the layered displacement values ​​of the segmentation points are packed in a 2D image for each partial area of ​​the object, and the 2D image is set as a displacement video of the frame to be encoded.

[0019] In an information processing device and method according to another aspect of the present technology, a displacement video is generated in which a 2D image is set as a frame, wherein the displacement values ​​of the segmentation points obtained by subdividing the base grid are layered for each fineness and packed in the 2D image for each partial area of ​​the object, the displacement values ​​of the segmentation points of the desired level of the desired partial area that have been packed in the 2D image of the current frame of the displacement video are unpacked, the base grid of the current frame is subdivided, and the displacement values ​​of the segmentation points are applied to the segmentation points of the subdivided base grid.

[0020] In an information processing device and method according to another aspect of the present technology, base mesh vertex displacement values ​​as displacement values ​​of vertices of a base mesh are derived, displacement values ​​of segmentation points obtained by subdividing the base mesh are layered for each degree of refinement, the base mesh vertex displacement values ​​and the layered displacement values ​​of the segmentation points are packed in a 2D image, and a displacement video in which the 2D image is set as a frame is encoded.

[0021] In an information processing device and method according to another aspect of the present technology, a bit stream is decoded to generate a displacement video in which a 2D image is set as a frame, wherein base mesh vertex displacement values ​​as displacement values ​​of vertices of a base mesh and displacement values ​​of segmentation points obtained by subdividing the base mesh are packed in the 2D image, the base mesh vertex displacement values ​​and displacement values ​​of the segmentation points that have been packed in the 2D image of a current frame of the displacement video are unpacked, the base mesh vertex displacement values ​​are applied to the vertices of the base mesh, the base mesh to which the base mesh vertex displacement values ​​are applied to the vertices is subdivided, and the displacement values ​​of the segmentation points are applied to the segmentation points of the subdivided base mesh. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] [ Figure 1 ]A diagram used to illustrate V-DMC.

[0023] [ Figure 2 ] Figure used to illustrate the packaging example.

[0024] [ Figure 3 ] Figure used to illustrate the packaging example.

[0025] [ Figure 4 ]A diagram showing an example of an encoding method.

[0026] [ Figure 5 ] Figure used to illustrate the packaging example.

[0027] [ Figure 6 ]A block diagram showing a main configuration example of an encoding device.

[0028] [ Figure 7 ]A flowchart illustrating an example of the flow of encoding processing.

[0029] [ Figure 8 ]A block diagram showing a main configuration example of an encoding device.

[0030] [ Fig. 9 ]A flowchart illustrating an example of the flow of encoding processing.

[0031] [ Fig.10 ]A block diagram showing an example of the main configuration of a decoding device.

[0032] [ Fig.11 ]A flowchart illustrating an example of the flow of decoding processing.

[0033] [ Fig.12 ]A block diagram showing an example of the main configuration of a decoding device.

[0034] [ Fig.13 ]A flowchart illustrating an example of the flow of decoding processing.

[0035] [ Fig.14 ]A diagram of an example used to illustrate the decoding process.

[0036] [ Fig.15 ]A diagram showing an example of an encoding method.

[0037] [ Fig.16 ]A diagram of an example used to illustrate the decoding process.

[0038] [ Fig.17 ] Figure used to illustrate the packaging example.

[0039] [ Fig.18 ]A block diagram showing a main configuration example of an encoding device.

[0040] [ Fig.19 ]A flowchart illustrating an example of the flow of encoding processing.

[0041] [ Fig. 20 ]A block diagram showing a main configuration example of an encoding device.

[0042] [ Fig.21 ]A flowchart illustrating an example of the flow of encoding processing.

[0043] [ Fig. 22 ]A block diagram showing an example of the main configuration of a decoding device.

[0044] [ Fig.23 ]A flowchart illustrating an example of the flow of decoding processing.

[0045] [ Fig.24 ]A block diagram showing an example of the main configuration of a decoding device.

[0046] [ Fig.25 ]A flowchart illustrating an example of the flow of decoding processing.

[0047] [ Fig.26 ]A block diagram showing an example of the main configuration of a computer. DETAILED DESCRIPTION

[0048] Hereinafter, a configuration for executing the present disclosure (hereinafter, will be referred to as an embodiment) will be described. Note that the description will be given in the following order.

[0049] 1. Documents supporting technical content and technical terminology, etc.,

[0050] 2. Grid coding,

[0051] 3. Packaging for each partial area,

[0052] 4. Packing of base mesh vertex displacement values,

[0053] 5. Notes.

[0054] <1. Documents supporting technical content and technical terminology, etc.>

[0055] The scope disclosed in the present technology includes not only the contents described in the embodiments but also the contents described in the following non-patent documents and the like known at the time of application, the contents of other documents cited in the following non-patent documents, and the like.

[0056] Non-patent document 1: (mentioned above)

[0057] In other words, the contents described in the above non-patent documents, the contents of other documents cited in the above non-patent documents, etc. also become the basis for determining the support conditions.

[0058] <2. Grid Coding>

[0059] <v-dmc>

[0060] In the past, there has been a mesh as 3D data representing the 3D structure of a 3D structure object (3D shaped object), in which polygons are formed by vertices and connections, thereby representing the 3D shape of the object surface. As a method for encoding this mesh, for example, there is V-DMC (Video-based Dynamic Mesh Coding) as disclosed in Non-Patent Document 1.

[0061] In V-DMC, a mesh as an encoding target is represented by a coarse base mesh and displacement values ​​of division points obtained by subdividing the base mesh, and the base mesh and the displacement values ​​are encoded.

[0062] For example, suppose there is a Figure 1 The original grid is shown in the top row of . Figure 1 In FIG. 1 , black dots indicate vertices, and lines connecting the black dots indicate connections. Although a polygon is initially formed by vertices and connections in a mesh as described above, for ease of description, this document will give a description in terms of groups of vertices connected in a line (in a row).

[0063] like Figure 1 The coarse mesh shown in the second row from the top is formed by decimate some of the vertices of the original mesh. This becomes the base mesh.

[0064] By subdividing each mesh of the base mesh, such as Figure 1 vertices are added as shown in the third row from the top in . In this document, through this splitting, vertices are added only in a sparse number from the original mesh. Therefore, a mesh having the same number of vertices as the original mesh is obtained. In this specification, the added vertices are also referred to as splitting points.

[0065] However, since the connections are updated when the vertices of the original mesh are thinned, and the segmentation points are formed on the updated connections, the shape of the segmented base mesh is different from that of the original mesh. More specifically, the positions of the segmentation points are different from those of the original mesh. Since the original mesh is known, the displacement of the segmentation points can be obtained.

[0066] In other words, if the above-mentioned sparse method and segmentation method are known, the original grid can be represented by the displacement values ​​of the base grid and the segmentation points. In the case of V-DMC, the encoder performs encoding while dividing the original grid into the base grid and the displacement values ​​of the segmentation points as described above. The displacement values ​​of the segmentation points are stored in the 2D image and encoded as an image. In this specification, storing data in the 2D image is also called packing.

[0067] Note that V-DMC supports scalable decoding. Figure 2 As shown, the displacement values ​​of the segmentation points are layered for each level of refinement and packaged into data for each level. Figure 2 In the example shown, LoD0 packed in the 2D image 11 indicates data of displacement values ​​of the segmentation points of the uppermost level (lowest level of fineness) among the displacement values ​​of the segmentation points that have been layered for each level of fineness. Similarly, LoD1 indicates data of displacement values ​​of the segmentation points of a level one level higher than LoD0. LoD2 indicates data of displacement values ​​of the segmentation points of a level one level higher than LoD1. In this way, the displacement values ​​of the segmentation points are packed separately for each level (compiled into data for each level).

[0068] Note that the displacement values ​​of the segmentation points may be packed in the 2D image as, for example, transformation coefficients obtained by performing coefficient transformation such as wavelet transformation. Alternatively, the displacement values ​​of the segmentation points may be quantized. For example, the displacement values ​​of the segmentation points may be converted into transformation coefficients by wavelet transformation, and the transformation coefficients may be quantized so that the quantized transformation coefficients (quantized coefficients) are packed.

[0069] In addition, the object may change in the time direction. In this specification, being variable in the time direction will also be expressed as "being dynamic". Therefore, the grid (i.e., the displacement values ​​of the base grid and the segmentation points) is also dynamic. Therefore, the displacement values ​​are encoded as a moving image in which the 2D image is set as a frame. In this specification, this moving image will also be referred to as displacement video.

[0070] As base mesh encoding methods, there are intra-frame encoding and inter-frame encoding. In the case of intra-frame encoding, the base mesh is encoded for each frame. In contrast, in the case of inter-frame encoding, the base mesh is encoded as a motion vector using the vertices of the base mesh of the reference frame as a reference. In other words, in this case, on the decoding side, the base mesh of the processing target frame can be reconstructed by adding the motion vector to the vertices of the base mesh of the reference frame. Note that in this specification, the base mesh of the reference frame will also be referred to as the reference base mesh. In addition, the processing target frame will also be referred to as the current frame. In addition, the base mesh of the current frame will also be referred to as the current base mesh.

[0071] <Partial access>

[0072] However, in the method described in Non-Patent Document 1, displacement video is simply used for packing displacement values, but further utilization of displacement video has been required in recent years.

[0073] For example, a function that enables decoding to be performed for each partial area in the decoder (so-called partial access function) is required. For example, when reproducing an image of visual sense seen from a viewpoint position, a grid with higher fineness is required at a portion of the object that is close to the viewpoint position and subjectively prominent. In contrast, a grid with low fineness is sufficient at a portion that is far from the viewpoint position or a portion that is not subjectively prominent. In addition, a grid is not required at a portion that is visually blocked by an object and cannot be seen from the viewpoint position. In this way, during reproduction, the fineness required for each partial area may be different, or information of some areas may become unnecessary.

[0074] The partial function enables reproduction control to be performed for each partial area. Therefore, by the partial function of V-DMC, also in the case of reproduction as described above, reproduction can be performed for each partial area with the desired fineness, or only some areas can be reproduced. Therefore, an increase in the processing amount of the decoding process can be suppressed.

[0075] However, since in the method described in Non-Patent Document 1, the displacement value of the entire object is Figure 2 As shown in FIG. 1 , the grid is collectively packed, and therefore, even in the case of reproducing only the grid of a certain area or changing the fineness of each partial area as described above, it is necessary to reconstruct the entire grid. In other words, complex tasks such as reconstructing the base grid of the entire grid, subdividing the entire base grid, and applying displacement values ​​to corresponding division points are necessary. Therefore, there is a concern that the processing amount of the decoding process may increase unnecessarily.

[0076] In order to implement such a partial access function, it is conceivable that, for example, Figure 3 The method shown is to separate the 2D image for packing for each partial area (ie, to generate a displacement video for each partial area). Figure 3 In the example of , the displacement values ​​of the division points of region A are packed in 2D image 21, the displacement values ​​of the division points of region B are packed in 2D image 22, and the displacement values ​​of the division points of region C are packed in 2D image 23. With such a configuration, since only the displacement video in which the desired region is packed can be decoded, a partial access function can be implemented.

[0077] However, with this method, it is necessary to generate a plurality of bit streams of displacement video, so there is a concern that the amount of information will increase unnecessarily and the encoding efficiency will decrease compared to the case where the number of bit streams of displacement video is one. Although the convenience is more improved as the area for each of which the partial access function is implemented becomes smaller, this requires an increase in the number of bit streams of displacement video, and therefore there is a concern that the encoding efficiency will further decrease.

[0078] In addition, a 2D decoder that decodes the encoded data of the displacement video needs to start an instance for each displacement video. Therefore, there is a concern that when the bitstreams of multiple areas are decoded, the amount of decoding processing will increase unnecessarily compared to the case where the number of bitstreams of the displacement video is one. Although convenience is further improved as the area for each of which the partial access function is implemented becomes smaller, this requires an increase in the number of bitstreams of the displacement video, and in this case, there is a concern that a large number of bitstreams will be very likely to be decoded and therefore the amount of decoding processing will further increase.

[0079] <3. Packaging for each partial area>

[0080] <Method 1>

[0081] In this regard, Figure 4 As shown in the top row of the table shown, the displacement values ​​can be packed for each partial region of the object (method 1). Figure 5 In the case of the 2D image 51 shown, as indicated by the bold box, the displacement values ​​of the segmentation points of the corresponding level of region A, the displacement values ​​of the segmentation points of the corresponding level of region B, and the displacement values ​​of the segmentation points of the corresponding level of region C are collectively packaged. Regions A to C are partial regions of the object. In other words, for each of the partial regions of the object, the displacement values ​​of the segmentation points of the corresponding level are packaged separately. With such a configuration, the decoder can extract the displacement value of the segmentation point from the 2D image 51 for each of the partial regions of the object. In addition, the decoder can extract data of the desired fineness and the desired partial region from the 2D image 51.

[0082] In other words, a partial access function can be realized by applying method 1. Therefore, an increase in the processing amount of the decoding process can be suppressed. In addition, in the case of method 1, since the displacement values ​​of the segmentation points of all regions can be packed in one displacement video, a decrease in encoding efficiency can be suppressed compared to a case where a displacement video is generated for each partial region. In addition, an increase in the processing amount of the decoding process can be suppressed. In other words, the encoding / decoding performance can be improved.

[0083] For example, the information processing device may include: a packing unit that layers the displacement values ​​of the segmentation points obtained by subdividing the base grid for each fineness, and packs the layered displacement values ​​of the segmentation points in a 2D image for each partial area of ​​the object; and a displacement video encoding unit that encodes the displacement video in which the 2D image is set as a frame. Note that the base grid can be generated by thinning the vertices from the original grid as the encoding target, which is composed of vertices and connections representing the 3D structure of the object. In other words, the base grid can be a grid coarser than the original grid. In addition, the displacement value of the segmentation point can be the position difference between the segmentation point and the vertex (corresponding to the segmentation point) of the original grid.

[0084] For example, the information processing device including the above-mentioned packing unit and the displacement video encoding unit may also include a base grid encoding unit for encoding the base grid of the current frame. In addition, the information processing device including the above-mentioned packing unit and the displacement video encoding unit may also include: a motion vector encoding unit, which encodes the motion vectors of the vertices of the base grid of the current frame using the base grid of the reference frame as a reference.

[0085] For example, the information processing method may include: for each fineness, layering the displacement values ​​of the segmentation points obtained by subdividing the base mesh, and packing the layered displacement values ​​of the segmentation points in a 2D image for each partial area of ​​the object; and encoding the displacement video in which the 2D image is set as a frame. Note that the base mesh can be generated by thinning the vertices from the original mesh as the encoding target, which is composed of vertices and connections representing the 3D structure of the object. In other words, the base mesh can be a mesh coarser than the original mesh. In addition, the displacement value of the segmentation point can be the position difference between the segmentation point and the vertex (corresponding to the segmentation point) of the original mesh.

[0086] Note that in this specification, extracting data packed in a 2D image from a 2D image will also be referred to as unpacking. For example, when the encoder packs the displacement value of the segmentation point in the 2D image, the decoder unpacks the displacement value of the segmentation point from the 2D image. In addition, when the displacement value of the segmentation point is subjected to coefficient transformation (e.g., wavelet transformation) and its transformation coefficient is packed by the encoder, the decoder unpacks the transformation coefficient from the 2D image and performs an inverse coefficient transformation (e.g., inverse wavelet transformation), thereby deriving the displacement value of the segmentation point. In addition, when the displacement value of the segmentation point is quantized and its quantization coefficient is packed by the encoder, the decoder unpacks the quantization coefficient from the 2D image and performs inverse quantization, thereby deriving the displacement value of the segmentation point. In addition, when the displacement value of the segmentation point is subjected to coefficient transformation (e.g., wavelet transformation), its transformation coefficient is quantized, and its quantization coefficient is packed by the encoder, the decoder unpacks the quantization coefficient from the 2D image, performs inverse quantization to derive the transformation coefficient, and performs an inverse coefficient transformation (e.g., inverse wavelet transformation) on the transformation coefficient, thereby deriving the displacement value of the segmentation point.

[0087] <Method 1-1>

[0088] When applying method 1, such as Figure 4 As shown in the second row from the top of the table shown, the displacement value of the segmentation point can be subjected to wavelet transformation for each partial area (method 1-1). By performing coefficient transformation such as wavelet transformation on the displacement value of the segmentation point for each partial area, the independence of the data of each partial area can be maintained. In other words, the decoder can unpack the data of the desired partial area without the need for data of other partial areas (and further perform inverse wavelet transformation on the unpacked data). In other words, a partial access function can be implemented. Therefore, an effect similar to that of the case of the above-mentioned method 1 can be obtained.

[0089] For example, in an information processing device including the above-mentioned packing unit and displacement video encoding unit, the packing unit may perform wavelet transform on the displacement value of the segmentation point for each partial area, and pack the obtained transform coefficients in a 2D image.

[0090] <Method 1-2>

[0091] When applying method 1, such as Figure 4 As shown in the bottom row of the table shown, partial zone information can be sent. Figure 5 In the example of the 2D image 51 shown, when the displacement value of the segmentation point is individually packed for each partial area, the decoder needs to grasp which partial area of ​​the displacement value of the segmentation point in the 2D image the desired partial area in the grid corresponds to. In this regard, information indicating the range corresponding to the partial area in the grid for each partial area of ​​the displacement value of the segmentation point packed in the 2D image can be sent from the encoder to the decoder (hereinafter, also referred to as grid range designation information). With such a configuration, the decoder can more easily grasp which partial area of ​​the displacement value of the segmentation point in the 2D image the desired partial area in the grid corresponds to.

[0092] In the mesh range designation information, for example, when a mesh is formed independently for each tile, the range of the mesh corresponding to each partial area can be indicated by using a tile ID. In addition, when one mesh is formed in the entire area, the range of the mesh corresponding to each partial area can be indicated by using a vertex index or a mesh face index. In other words, the mesh range designation information can be a list of tile IDs, vertex indices, face indices, etc. corresponding to each partial area. Note that when a decoding order is given and the number of faces corresponding to the corresponding partial areas is the same as each other, the number of faces can be sent as mesh range designation information. The decoder can group the corresponding faces for each of the multiple faces in the decoding order so that the groups correspond to the corresponding partial areas.

[0093] In addition, the decoder needs to grasp the position where the data of the desired partial area corresponding to the desired range in the grid is packed in the 2D image 51. In this regard, information indicating the range of each partial area of ​​the displacement value of the division point in the 2D image (hereinafter, also referred to as displacement intra-video range designation information) can be sent from the encoder to the decoder. With such a configuration, the decoder can more easily grasp where the desired partial area of ​​the displacement value of the division point in the 2D image is packed in the 2D image.

[0094] In the displacement video intra-range designation information, the starting position of the partial area can be indicated by using a pixel index. In addition, the range in which the data of each partial area is packed can be indicated by the number of pixels, or can be indicated by block units instead of pixels. For example, the starting position of the partial area can be indicated by using a block index. In addition, the range in which the data of each partial area is packed can be indicated by the number of blocks. In other words, the displacement video intra-range designation information can be a list of pixel indices, pixel numbers, block indices, block numbers, etc. corresponding to each partial area. Note that when the sizes of the ranges in which the data of the corresponding partial areas are packed are the same as each other (fixed values ​​known to the decoder), the sending of the displacement video intra-range designation information can be omitted. In addition, when the decoder does not know the fixed value, the fixed value can be sent as the displacement video intra-range designation information.

[0095] In other words, these grid range designation information and displacement intra-video range designation information can be sent from the encoder to the decoder as information related to each of the partial regions of the displacement value of the segmentation point in the 2D image (hereinafter, also referred to as partial region information). With such a configuration, the decoder can more easily unpack the displacement value of the segmentation point of the desired partial region.

[0096] For example, the mesh range specifying information and the shifted in-video range specifying information are indicated as follows as an information list corresponding to each partial area.

[0097] The grid range is specified as [1, 3, 4, 6],

[0098] The displacement in the video is specified in the range [0, 7, 15, 30].

[0099] In either information, the elements are arranged in the order of the partial areas. Therefore, the decoder can easily establish a correspondence between the grid range specifying information and the displacement video range specifying information based on the number of elements. For example, in the case of the above list, the displacement values ​​indicating the division points of the range indicated by "index 3" in the grid are packed in the area indicated by "index 7" in the 2D image.

[0100] For example, in an information processing device including the above-mentioned packaging unit and the displacement video encoding unit, the displacement video encoding unit may further encode partial region information. The partial region information may include grid range designation information indicating the range of the partial region in the grid and displacement video intra-range designation information indicating the range of the partial region in the 2D image of the displacement video.

[0101] Note that the encoding method of the partial region information is arbitrary, and differential encoding or entropy encoding may be used. Alternatively, the partial region information does not need to be encoded.

[0102] In addition, the transmission method of the partial area information is arbitrary. For example, the partial area information can be stored in the encoded data of the displacement video for each frame. Alternatively, the partial area information can be stored in the header information of the bit stream that stores not only the displacement video but also the attributes, etc.

[0103] <Encoding Device (Intra-frame)>

[0104] The present technology can be applied to an encoding device for encoding a grid. Figure 6 : is a block diagram showing an example of the configuration of an encoding device as an embodiment of an information processing device to which the present technology is applied. Figure 6 The encoding device 100 shown is a device that encodes a mesh. The encoding device 100 basically encodes the mesh by a method similar to the method of V-DMC described in Non-Patent Document 1. The encoding device 100 performs intra-frame encoding on a base mesh.

[0105] Note that the encoding device 100 encodes the grid by applying the above-mentioned method 1. Alternatively, the encoding device 100 may apply the above-mentioned method 1-1 or method 1-2.

[0106] Notice, Figure 6 The main configuration including processing units, data flow, etc. is shown, but not all configurations are Figure 6 In other words, in the encoding device 100, there may be Figure 6 The processing units shown in the box may exist Figure 6 Processing and data flow indicated by arrows etc.

[0107] like Figure 6 As shown, the encoding device 100 includes a base grid encoding unit 111, a base grid decoding unit 112, a displacement value correction unit 113, a packing unit 114, a displacement video encoding unit 115, a grid reconstruction unit 116, an attribute map correction unit 117, an attribute video encoding unit 118 and a synthesis unit 119.

[0108] In the encoding device 100, a base mesh, a displacement value of a segmentation point, and an attribute map are provided for each frame. Note that the base mesh can be generated by thinning out vertices from an original mesh as an encoding target, which is composed of vertices and connections representing the 3D structure of an object. In other words, the base mesh can be a mesh coarser than the original mesh. In addition, the displacement value of the segmentation point can be the position difference between the segmentation point and the vertex (corresponding to the segmentation point) of the original mesh. For example, these data can be generated from the original mesh in preprocessing.

[0109] The base grid encoding unit 111 encodes the current base grid and provides the encoded data to the base grid decoding unit 112 and the synthesis unit 119.

[0110] The base grid decoding unit 112 decodes the coded data of the provided current base grid to generate (restore) the current base grid. The generated current base grid contains coding distortion. The base grid decoding unit 112 provides the current base grid to the displacement value correction unit 113 and the grid reconstruction unit 116.

[0111] The displacement value correction unit 113 uses the provided current base grid (including coding distortion) to correct the displacement value of the segmentation point. More specifically, the displacement value correction unit 113 subdivides the provided current base grid (including coding distortion) and obtains the segmentation point. Then, the displacement value correction unit 113 uses the obtained segmentation point to correct the displacement value of the segmentation point provided to the encoding device 100. The displacement value correction unit 113 provides the corrected displacement value of the segmentation point to the packing unit 114.

[0112] The packing unit 114 packs the provided displacement values ​​of the segmentation points in a 2D image of the current frame (also referred to as a frame image). At this time, the packing unit 114 may perform coefficient transformation (e.g., wavelet transformation) on the displacement values ​​of the segmentation points and pack the transformation coefficients in the 2D image. Alternatively, the packing unit 114 may quantize the displacement values ​​(or transformation coefficients) of the segmentation points and pack the quantized coefficients in the 2D image.

[0113] In addition, the packing unit 114 applies method 1 in the packing. In other words, the packing unit 114 packs the displacement values ​​of the segmentation points for each of the partial regions of the object. For example, the packing unit 114 may layer the displacement values ​​of the segmentation points obtained by subdividing the base grid for each fineness, and pack the layered displacement values ​​of the segmentation points in the 2D image for each of the partial regions of the object.

[0114] Alternatively, the packing unit 114 may apply method 1-1. In other words, the packing unit 114 may perform wavelet transform on the displacement values ​​of the division points for each partial region and pack the obtained transform coefficients in the 2D image.

[0115] Alternatively, the packing unit 114 may apply the method 1-2. In other words, the packing unit 114 may also generate partial region information including the mesh range specifying information and the displacement intra-video range specifying information.

[0116] The packing unit 114 supplies the 2D image in which the displacement values ​​of the division points (or information corresponding thereto) are packed to the displacement video encoding unit 115. In addition, the packing unit 114 supplies the generated partial region information to the displacement video encoding unit 115.

[0117] The displacement video encoding unit 115 sets the supplied 2D image as a frame image and encodes it as a moving image (displacement video). The displacement video encoding unit 115 supplies the generated encoded data of the displacement video to the mesh reconstruction unit 116 and the synthesis unit 119.

[0118] Alternatively, the displacement video encoding unit 115 may apply method 1-2. In other words, the displacement video encoding unit 115 may also encode partial area information including grid range designation information and displacement video internal range designation information. In this case, the displacement video encoding unit 115 provides the generated encoded data of the partial area information to the synthesis unit 119.

[0119] The mesh reconstruction unit 116 decodes the coded data of the provided displacement video and unpacks data from the 2D image, etc., to obtain the displacement values ​​of the segmentation points. In addition, the mesh reconstruction unit 116 subdivides the provided current base mesh and applies the obtained displacement values ​​of the segmentation points to reconstruct the mesh. The mesh contains coding distortion. The mesh reconstruction unit 116 provides the reconstructed mesh to the attribute map correction unit 117.

[0120] The property map correction unit 117 corrects the property map of texture or the like using the supplied mesh (including encoding distortion), and supplies the corrected property map to the property video encoding unit 118 .

[0121] The attribute video encoding unit 118 sets the supplied attribute map as a frame image, and encodes it as a moving image (attribute video). The attribute video encoding unit 118 supplies the generated encoded data of the attribute video to the synthesis unit 119.

[0122] The synthesis unit 119 synthesizes (multiplexes) the encoded data of the base grid, the encoded data of the displacement video, and the encoded data of the attribute video that have been supplied thereto to generate one bit stream. Note that when the encoded data of the partial area information is supplied, the synthesis unit 119 also stores the encoded data of the partial area information in the bit stream. The synthesis unit 119 outputs the generated bit stream to the outside of the encoding device 100. The bit stream is sent to the decoding device via any transmission medium or any recording medium.

[0123] With such a configuration, the encoding device 100 can implement a partial access function. Therefore, the encoding device 100 can suppress an increase in the amount of processing of the decoding process and suppress a decrease in encoding efficiency. In other words, the encoding device 100 can improve encoding / decoding performance.

[0124] <Encoding Process Flow>

[0125] Will refer to Figure 7 The flowchart describes an example of the flow of the encoding process performed by the encoding device 100.

[0126] When the encoding process starts, the base grid encoding unit 111 encodes the current base grid in step S101.

[0127] In step S102 , the base grid decoding unit 112 decodes the encoded data of the base grid.

[0128] In step S103 , the displacement value correction unit 113 corrects the displacement values ​​of the division points using the base grid (including the encoding distortion) generated in step S102 .

[0129] In step S104, the packing unit 114 packs the displacement value of the segmentation point of each LoD in the frame (2D image) for each partial area of ​​the object. At this time, the packing unit 114 can perform coefficient transformation (for example, wavelet transformation) on the displacement value of the segmentation point for each partial area. In addition, the packing unit 114 can generate partial area information.

[0130] In step S105, the displacement video encoding unit 115 sets the packed 2D images as frame images and encodes them into displacement videos. In addition, the displacement video encoding unit 115 can also encode partial region information.

[0131] In step S106 , the mesh reconstruction unit 116 reconstructs a mesh using the base mesh (including the encoding distortion) generated in step S102 and the encoded data of the displacement video generated in step S105 .

[0132] In step S107 , the property map correction unit 117 corrects the property map using the reconstructed mesh.

[0133] In step S108 , the attribute video encoding unit 118 sets the corrected attribute map as a frame image, and encodes it as an attribute video.

[0134] In step S109, the synthesis unit 119 multiplexes the encoded data of the base grid generated in step S101, the encoded data of the displacement video generated in step S105, and the encoded data of the attribute video generated in step S108 to generate one bit stream. Note that when the encoded data of the partial area information is additionally generated in step S105, the synthesis unit 119 can also store the encoded data in the bit stream.

[0135] When the process of step S109 ends, the encoding process ends.

[0136] By performing the corresponding processing as described above, the encoding device 100 can realize the partial access function. Therefore, the encoding device 100 can suppress the increase in the processing amount of the decoding process and suppress the reduction in encoding efficiency. In other words, the encoding device 100 can improve the encoding / decoding performance.

[0137] <Encoding Device (Inter-frame)>

[0138] Figure 8 : is a block diagram showing an example of the configuration of an encoding device as an embodiment of an information processing device to which the present technology is applied. Figure 8 The encoding device 150 shown is a device that encodes a grid. The encoding device 150 basically encodes the grid by a method similar to the method of V-DMC described in Non-Patent Document 1. The encoding device 150 performs inter-frame encoding on a base grid.

[0139] Note that the encoding device 150 encodes the grid by applying the above-mentioned method 1. Alternatively, the encoding device 150 may apply the above-mentioned method 1-1 or method 1-2.

[0140] Notice, Figure 8 The main configuration including processing units, data flow, etc. is shown, but not all configurations are Figure 8 In other words, in the encoding device 150, there may be Figure 8 The processing units shown in the box may exist Figure 8 Processing and data flow indicated by arrows etc.

[0141] like Figure 8 As shown, the encoding device 150 includes a motion vector encoding unit 161, a base grid reconstruction unit 162, a displacement value correction unit 163, a packing unit 164, a displacement video encoding unit 165, a grid reconstruction unit 166, an attribute map correction unit 167, an attribute video encoding unit 168 and a synthesis unit 169.

[0142] In the encoding device 150, the current base mesh, the displacement value of the segmentation point, and the attribute map are provided for each frame. Note that the (current) base mesh can be generated by sparse vertices from the original mesh as the encoding target, which is composed of vertices and connections representing the 3D structure of the object. In other words, the base mesh can be a mesh coarser than the original mesh. In addition, the displacement value of the segmentation point can be the position difference between the segmentation point and the vertex of the original mesh (corresponding to the segmentation point). For example, these data can be generated from the original mesh in preprocessing. In addition, the encoding device 150 can obtain the base mesh of the reference frame (reference base mesh). The reference frame is a frame that can be referenced from the current frame and has been processed in the past.

[0143] Using the current base mesh and the reference base mesh, the motion vector encoding unit 161 generates a motion vector using the vertices of the reference base mesh as a reference. In other words, the motion vector encoding unit 161 generates a motion vector indicating the displacement of the corresponding vertices between the current base mesh and the reference base mesh. The motion vector encoding unit 161 encodes the generated motion vector and provides the encoded data to the base mesh reconstruction unit 162 and the synthesis unit 169.

[0144] The base grid reconstruction unit 162 decodes the coded data of the provided motion vector to generate (restore) the motion vector. Then, the base grid reconstruction unit 162 reconstructs the current base grid using the reference base grid and the motion vector. The reconstructed current base grid contains coding distortion. The base grid reconstruction unit 162 provides the current base grid to the displacement value correction unit 163 and the grid reconstruction unit 166.

[0145] The displacement value correction unit 163 uses the provided current base grid (including coding distortion) to correct the displacement value of the segmentation point. More specifically, the displacement value correction unit 163 subdivides the provided current base grid (including coding distortion) and obtains the segmentation point. Then, the displacement value correction unit 163 uses the obtained segmentation point to correct the displacement value of the segmentation point provided to the encoding device 150. The displacement value correction unit 163 provides the corrected displacement value of the segmentation point to the packing unit 164.

[0146] The packing unit 164 packs the displacement values ​​of the provided segmentation points in a 2D image of the current frame (also referred to as a frame image). At this time, the packing unit 164 may perform coefficient transformation (e.g., wavelet transformation) on the displacement values ​​of the segmentation points and pack the transformation coefficients in the 2D image. Alternatively, the packing unit 164 may quantize the displacement values ​​(or transformation coefficients) of the segmentation points and pack the quantized coefficients in the 2D image.

[0147] In addition, the packing unit 164 applies method 1 in the packing. In other words, the packing unit 164 packs the displacement values ​​of the segmentation points for each of the partial regions of the object. For example, the packing unit 164 may layer the displacement values ​​of the segmentation points obtained by subdividing the base grid for each fineness, and pack the layered displacement values ​​of the segmentation points in the 2D image for each of the partial regions of the object.

[0148] Alternatively, the packing unit 164 may apply the method 1-1. In other words, the packing unit 164 may perform wavelet transform on the displacement values ​​of the division points for each partial region and pack the obtained transform coefficients in the 2D image.

[0149] Alternatively, the packing unit 164 may apply the method 1-2. In other words, the packing unit 164 may also generate partial region information including the mesh range specifying information and the displacement intra-video range specifying information.

[0150] The packing unit 164 supplies the 2D image in which the displacement values ​​of the division points (or information corresponding thereto) are packed to the displacement video encoding unit 165. In addition, the packing unit 164 supplies the generated partial region information to the displacement video encoding unit 165.

[0151] The displacement video encoding unit 165 sets the supplied 2D image as a frame image and encodes it as a moving image (displacement video). The displacement video encoding unit 165 supplies the generated encoded data of the displacement video to the mesh reconstruction unit 166 and the synthesis unit 169.

[0152] Alternatively, the displacement video encoding unit 165 may apply method 1-2. In other words, the displacement video encoding unit 165 may also encode partial area information including grid range designation information and displacement video internal range designation information. In this case, the displacement video encoding unit 165 provides the generated encoded data of the partial area information to the synthesis unit 169.

[0153] The mesh reconstruction unit 166 decodes the coded data of the provided displacement video and unpacks data from the 2D image, etc., to obtain the displacement values ​​of the segmentation points. In addition, the mesh reconstruction unit 166 subdivides the provided current base mesh and applies the obtained displacement values ​​of the segmentation points to reconstruct the mesh. The mesh contains coding distortion. The mesh reconstruction unit 166 provides the reconstructed mesh to the attribute map correction unit 167.

[0154] The property map correction unit 167 corrects the property map of texture or the like using the supplied mesh (containing encoding distortion), and supplies the corrected property map to the property video encoding unit 168 .

[0155] The attribute video encoding unit 168 sets the supplied attribute map as a frame image, and encodes it as a moving image (attribute video). The attribute video encoding unit 168 supplies the generated encoded data of the attribute video to the synthesis unit 169.

[0156] The synthesis unit 169 synthesizes (multiplexes) the coded data of the motion vector, the coded data of the displacement video, and the coded data of the attribute video that have been supplied thereto to generate a bit stream. Note that when the coded data of the partial area information is supplied, the synthesis unit 169 also stores the coded data of the partial area information in the bit stream. The synthesis unit 169 outputs the generated bit stream to the outside of the encoding device 150. The bit stream is sent to the decoding device via an arbitrary transmission medium or an arbitrary recording medium.

[0157] With such a configuration, the encoding device 150 can realize the partial access function. Therefore, the encoding device 150 can suppress the increase in the processing amount of the decoding process and suppress the reduction in encoding efficiency. In other words, the encoding device 150 can improve the encoding / decoding performance.

[0158] <Encoding Process Flow>

[0159] Will refer to Fig. 9 The flowchart describes an example of the flow of the encoding process performed by the encoding device 150.

[0160] When the encoding process starts, the motion vector encoding unit 161 generates a motion vector using the reference base grid and the current base grid and encodes the motion vector in step S151.

[0161] In step S152 , the base mesh reconstruction unit 162 decodes the encoded data of the motion vector, and reconstructs the base mesh using the generated (restored) motion vector and the reference base mesh.

[0162] In step S153 , the displacement value correction unit 163 corrects the displacement values ​​of the division points using the base grid (containing the encoding distortion) reconstructed in step S152 .

[0163] In step S154, the packing unit 164 packs the displacement value of the segmentation point of each LoD in the frame (2D image) for each partial region of the object. At this time, the packing unit 164 can perform coefficient transformation (for example, wavelet transformation) on the displacement value of the segmentation point for each partial region. In addition, the packing unit 164 can generate partial region information.

[0164] In step S155, the displacement video encoding unit 165 sets the packed 2D images as frame images and encodes them into displacement videos. In addition, the displacement video encoding unit 165 can also encode partial area information.

[0165] In step S156 , the mesh reconstruction unit 166 reconstructs a mesh using the base mesh reconstructed in step S152 (including encoding distortion) and the encoded data of the displacement video generated in step S155 .

[0166] In step S157 , the property map correction unit 167 corrects the property map using the reconstructed mesh.

[0167] In step S158 , the attribute video encoding unit 168 sets the corrected attribute map as a frame image, and encodes it as an attribute video.

[0168] In step S159, the synthesis unit 169 multiplexes the encoded data of the motion vector generated in step S151, the encoded data of the displacement video generated in step S155, and the encoded data of the attribute video generated in step S158 to generate one bit stream. Note that when the encoded data of the partial area information is additionally generated in step S155, the synthesis unit 169 can also store the encoded data in the bit stream.

[0169] When the processing of step S169 ends, the encoding processing ends.

[0170] By performing the corresponding processing as described above, the encoding device 150 can realize the partial access function. Therefore, the encoding device 150 can suppress the increase in the processing amount of the decoding process and suppress the reduction in encoding efficiency. In other words, the encoding device 150 can improve the encoding / decoding performance.

[0171] <Method 1, Method 1-1 and Method 1-2>

[0172] Note that the above-described method 1, method 1-1, and method 1-2 can also be applied to a decoding device.

[0173] For example, the information processing device may include: a displacement video decoding unit that decodes a bit stream and generates a displacement video in which a 2D image is set as a frame, wherein the displacement values ​​of the segmentation points obtained by subdividing the base grid are layered for each fineness and packed in the 2D image for each partial area of ​​the object; an unpacking unit that unpacks the displacement values ​​of the segmentation points of the desired layer of the desired partial area that have been packed in the 2D image of the current frame of the displacement video; a subdivision unit that subdivides the base grid of the current frame; and a displacement value application unit that applies the displacement values ​​of the segmentation points to the segmentation points of the subdivided base grid. Note that the base grid can be generated by thinning out vertices from an original grid as an encoding target composed of vertices and connections representing the 3D structure of the object. In other words, the base grid can be a grid coarser than the original grid. In addition, the displacement value of the segmentation point can be a position difference between the segmentation point and the vertex (corresponding to the segmentation point) of the original grid.

[0174] For example, the information processing device including the above-mentioned displacement video decoding unit, unpacking unit and displacement value application unit may further include: a base mesh decoding unit, which decodes the bit stream and generates a base mesh of the current frame. In addition, the information processing device including the above-mentioned displacement video decoding unit, unpacking unit and displacement value application unit may further include: a motion vector generation unit, which decodes the bit stream and generates a motion vector; and a base mesh reconstruction unit, which applies the motion vector to the vertices of the base mesh of the reference frame to reconstruct the base mesh of the current frame.

[0175] For example, in an information processing device including the above-mentioned displacement video decoding unit, unpacking unit and displacement value application unit, the unpacking unit can unpack the transformation coefficients of the desired level of the desired partial area and subject the transformation coefficients to an inverse wavelet transform to generate the displacement value of the segmentation point.

[0176] For example, the information processing device including the above-mentioned displacement video decoding unit, unpacking unit and displacement value application unit may also include: a header decoding unit, which decodes the bit stream and generates partial area information, the partial area information including grid range designation information indicating the range of the partial area in the grid and displacement video intra-range designation information indicating the range of the partial area in the 2D image of the displacement video. Then, the unpacking unit can unpack the displacement value of the segmentation point of the desired level of the desired partial area based on the displacement video intra-range designation information. In addition, the subdivision unit can subdivide the desired partial area of ​​the base grid based on the grid range designation information.

[0177] For example, the information processing method may include: decoding a bit stream and generating a displacement video in which a 2D image is set as a frame, wherein the displacement values ​​of the segmentation points obtained by subdividing the base grid are layered for each fineness and packed in the 2D image for each partial area of ​​the object; unpacking the displacement values ​​of the segmentation points of the desired layer of the desired partial area that have been packed in the 2D image of the current frame of the displacement video; subdividing the base grid of the current frame; and applying the displacement values ​​of the segmentation points to the segmentation points of the subdivided base grid. Note that the base grid can be generated by sparse vertices from an original grid as an encoding target composed of vertices and connections representing the 3D structure of the object. In other words, the base grid can be a grid coarser than the original grid. In addition, the displacement value of the segmentation point can be the position difference between the segmentation point and the vertex (corresponding to the segmentation point) of the original grid.

[0178] <Decoding Device (Intra-frame)>

[0179] The present technology can be applied to a decoding device that decodes coded data of a grid. Fig.10 : is a block diagram showing an example of the configuration of a decoding device as an embodiment of an information processing device to which the present technology is applied. Fig.10 The decoding device 200 shown is a device that decodes the coded data of the mesh. The decoding device 200 basically decodes the coded data of the mesh by a method similar to the method of V-DMC described in Non-Patent Document 1. The decoding device 200 decodes the coded data obtained by intra-coding the base mesh (intra-frame decoding).

[0180] Note that the decoding device 200 decodes the encoded data of the grid by applying the above-mentioned method 1. Alternatively, the decoding device 200 may apply the above-mentioned method 1-1 or method 1-2.

[0181] In other words, the decoding device 200 is Figure 6 The encoding device 100 shown corresponds to a decoding device, and can decode the bit stream generated by the encoding device 100 to reconstruct the grid.

[0182] Notice, Fig.10 The main configuration including processing units, data flow, etc. is shown, but not all configurations are Fig.10 In other words, in the decoding device 200, there may be Fig.10 The processing units shown in the box may exist Fig.10 Processing and data flow indicated by arrows etc.

[0183] like Fig.10 As shown, the decoding device 200 includes a demultiplexing unit 211, a header decoding unit 212, a base grid decoding unit 213, a subdivision unit 214, a displacement video decoding unit 215, an unpacking unit 216, a displacement value application unit 217 and an attribute video decoding unit 218.

[0184] A bit stream generated by an encoding device (for example, the encoding device 100 ) that intra-encodes a base grid is provided to the decoding device 200 .

[0185] The demultiplexing unit 211 demultiplexes the bitstream and extracts the corresponding encoded data included in the bitstream. For example, the demultiplexing unit 211 extracts the header information of the bitstream from the bitstream and provides it to the header decoding unit 212. Note that it is assumed that the encoded data of the partial area information is stored in the header information. In addition, the demultiplexing unit 211 extracts the encoded data of the base grid from the bitstream and provides it to the base grid decoding unit 213. In addition, the demultiplexing unit 211 extracts the encoded data of the displacement video from the bitstream and provides it to the displacement video decoding unit 215. In addition, the demultiplexing unit 211 extracts the encoded data of the attribute video from the bitstream and provides it to the attribute video decoding unit 218.

[0186] The header decoding unit 212 decodes the encoded data of the supplied header information to generate (restore) the header information. The header decoding unit 212 supplies the generated header information to the base grid decoding unit 213, the subdivision unit 214, the displacement video decoding unit 215, the depacketization unit 216, and the attribute video decoding unit 218.

[0187] The base grid decoding unit 213 decodes the encoded data of the supplied base grid based on the supplied header information to generate (restore) the current base grid. The base grid decoding unit 213 supplies the generated current base grid to the subdivision unit 214.

[0188] The subdivision unit 214 subdivides the supplied base grid based on the supplied header information, and supplies the subdivided base grid to the displacement value application unit 217 .

[0189] The subdivision unit 214 applies method 1 in the subdivision. In other words, the subdivision unit 214 subdivides the desired partial area of ​​the base grid. At this time, the subdivision unit 214 may alternatively apply method 1-2. In other words, the subdivision unit 214 may specify a partial area corresponding to the desired range in the reconstructed grid based on the partial area information included in the header information (the grid range designation information included in the subdivision unit 214), and subdivide the specified partial area of ​​the base grid.

[0190] The displacement video decoding unit 215 decodes the provided coded data of the displacement video based on the provided header information to generate (restore) (the current frame of) the displacement video. The displacement video decoding unit 215 provides (the current frame of) the generated displacement video to the depacketizing unit 216.

[0191] The unpacking unit 216 unpacks the displacement value of the segmentation point from the current frame of the provided displacement video based on the provided header information. At this time, the unpacking unit 216 can unpack the transform coefficients and perform coefficient transformation (e.g., wavelet transform) on the transform coefficients to obtain the displacement value of the segmentation point. Alternatively, the unpacking unit 216 can unpack the quantization coefficients and inversely quantize the quantization coefficients to obtain the displacement value of the segmentation point. Alternatively, the unpacking unit 216 can unpack the quantization coefficients, inversely quantize the quantization coefficients to derive the transform coefficients, and perform coefficient transformation (e.g., wavelet transform) on the transform coefficients to obtain the displacement value of the segmentation point.

[0192] In addition, the unpacking unit 216 applies method 1 in the unpacking. In other words, the unpacking unit 216 unpacks the displacement values ​​of the segmentation points of the desired partial region. For example, the unpacking unit 216 may unpack the displacement values ​​of the segmentation points of the desired level of the desired partial region.

[0193] Alternatively, the unpacking unit 216 may apply method 1-1. In other words, the unpacking unit 216 may unpack the transform coefficients of the desired level of the desired partial area and perform inverse wavelet transform on the transform coefficients, thereby generating displacement values ​​of the division points.

[0194] Alternatively, the unpacking unit 216 may apply method 1-2. In other words, the unpacking unit 216 may specify the location where data of the desired partial area is stored in the 2D image based on the partial area information included in the header information (the displacement video range designation information included therein), and unpack the displacement value of the segmentation point of the desired level from the specified area.

[0195] The unpacking unit 216 supplies the unpacked displacement values ​​of the division points to the displacement value application unit 217 .

[0196] The displacement value application unit 217 applies the displacement values ​​of the provided segmentation points to the segmentation points of the provided base grid (subdivided base grid) to reconstruct the grid. In this specification, the grid will also be referred to as a decoding grid. In other words, it can also be said that the displacement value application unit 217 generates a decoding grid. The displacement value application unit 217 outputs the generated decoding grid to the outside of the decoding device 200.

[0197] The attribute video decoding unit 218 decodes the encoded data of the provided attribute video based on the provided header information, thereby generating (the current frame of) the attribute video. The attribute video decoding unit 218 outputs the generated current frame of the attribute video (i.e., the attribute map corresponding to the decoding grid) to the outside of the decoding device 200.

[0198] With such a configuration, the decoding device 200 can realize the partial access function. Therefore, the decoding device 200 can suppress the increase in the processing amount of the decoding process and suppress the reduction in encoding efficiency. In other words, the decoding device 200 can improve the encoding / decoding performance.

[0199] <Decoding Process Flow>

[0200] Will refer to Fig.11 The flowchart describes an example of the flow of the decoding process performed by the decoding device 200.

[0201] When the decoding process starts, the demultiplexing unit 211 demultiplexes a bit stream and extracts various types of encoded data in step S201 .

[0202] In step S202, the header decoding unit 212 decodes the encoded data of the header information extracted in step S201 to generate (restore) the header information.

[0203] In step S203 , the base grid decoding unit 213 decodes the encoded data of the base grid extracted in step S201 to generate (restore) the current base grid.

[0204] In step S204, the subdivision unit 214 subdivides the desired partial area of ​​the current base grid. At this time, the subdivision unit 214 can specify the partial area corresponding to the desired range in the grid based on the partial area information included in the header information (the grid range designation information included in the subdivision unit 214), and subdivide the specified partial area of ​​the base grid.

[0205] In step S205 , the displacement video decoding unit 215 decodes the encoded data of the displacement video extracted in step S201 to generate (restore) (the current frame of) the displacement video.

[0206] In step S206, the unpacking unit 216 unpacks the displacement value of the desired LoD of the desired partial area from the current frame (2D image). At this time, the unpacking unit 216 may unpack the transform coefficients obtained by performing coefficient transform (e.g., wavelet transform) for each partial area, and perform inverse coefficient transform (e.g., inverse wavelet transform) on the transform coefficients for each partial area, thereby obtaining the displacement value of the segmentation point. Alternatively, the unpacking unit 216 may specify the location where the data of the desired partial area is stored in the 2D image based on the partial area information included in the header information (the displacement intra-video range designation information included therein), and unpack the displacement value of the segmentation point of the desired level from the specified area.

[0207] In step S207 , the displacement value application unit 217 applies the displacement values ​​of the division points unpacked in step S206 to the division points of the base grid subdivided in step S204 to generate a decoded grid.

[0208] In step S208, the attribute video decoding unit 218 decodes the encoded data of the attribute video extracted in step S201 to generate (restore) the current frame of the attribute video, that is, the attribute map.

[0209] When the process of step S208 ends, the decoding process ends.

[0210] By performing the corresponding processing as described above, the decoding device 200 can realize the partial access function. Therefore, the decoding device 200 can suppress the increase in the processing amount of the decoding process and suppress the reduction in encoding efficiency. In other words, the decoding device 200 can improve the encoding / decoding performance.

[0211] <Decoding Device (Inter-frame)>

[0212] Fig.12 : is a block diagram showing an example of the configuration of an encoding device as an embodiment of an information processing device to which the present technology is applied. Fig.12 The decoding device 250 shown is a device that decodes the coded data of the grid. The decoding device 250 basically decodes the coded data of the grid by a method similar to the method of V-DMC described in Non-Patent Document 1. The decoding device 250 decodes the coded data obtained by inter-coding the base grid (inter-frame decoding).

[0213] Note that the decoding device 250 decodes the encoded data of the mesh by applying the above-mentioned method 1. Alternatively, the decoding device 250 may apply the above-mentioned method 1-1 or method 1-2.

[0214] In other words, the decoding device 250 is Figure 8 The encoding device 150 shown corresponds to a decoding device, and can decode the bit stream generated by the encoding device 150 to reconstruct the grid.

[0215] Notice, Fig.12 The main configuration including processing units, data flow, etc. is shown, but not all configurations are Fig.12 In other words, in the decoding device 250, there may be Fig.12 The processing units shown in the box may exist Fig.12 Processing and data flow indicated by arrows etc.

[0216] like Fig.12 As shown, the decoding device 250 includes a demultiplexing unit 261, a header decoding unit 262, a motion vector decoding unit 263, a base grid reconstruction unit 264, a subdivision unit 265, a displacement video decoding unit 266, an unpacking unit 267, a displacement value application unit 268 and an attribute video decoding unit 269.

[0217] A bit stream generated by an encoding device (eg, the encoding device 150 ) that inter-encodes a base grid is provided to the decoding device 250 .

[0218] The demultiplexing unit 261 demultiplexes the bit stream and extracts the corresponding encoded data included in the bit stream. For example, the demultiplexing unit 261 extracts the header information of the bit stream from the bit stream and provides it to the header decoding unit 262. Note that it is assumed that the encoded data of the partial area information is stored in the header information. In addition, the demultiplexing unit 261 extracts the encoded data of the motion vector from the bit stream and provides it to the motion vector decoding unit 263. In addition, the demultiplexing unit 261 extracts the encoded data of the displacement video from the bit stream and provides it to the displacement video decoding unit 266. In addition, the demultiplexing unit 261 extracts the encoded data of the attribute video from the bit stream and provides it to the attribute video decoding unit 269.

[0219] The header decoding unit 262 decodes the encoded data of the supplied header information to generate (restore) the header information. The header decoding unit 262 supplies the generated header information to the motion vector decoding unit 263, the subdivision unit 265, the displacement video decoding unit 266, the depacketization unit 267, and the attribute video decoding unit 269.

[0220] The motion vector decoding unit 263 decodes the supplied coded data of the motion vector based on the supplied header information to generate (restore) the motion vector. The motion vector decoding unit 263 supplies the generated motion vector to the base mesh reconstruction unit 264.

[0221] The base grid reconstruction unit 264 applies the provided motion vector to the reference base grid to reconstruct the current base grid. The base grid reconstruction unit 264 provides the reconstructed current base grid to the subdivision unit 265. Note that the base grid reconstruction unit 264 may provide the current base grid to the outside of the decoding device 250 or the like to store it therein so that it can be reused as a reference base grid in a subsequent frame. Note that the reference base grid may alternatively be generated from a decoded grid.

[0222] The subdivision unit 265 subdivides the supplied base mesh based on the supplied header information, and supplies the subdivided base mesh to the displacement value application unit 268 .

[0223] The subdivision unit 265 applies method 1 in the subdivision. In other words, the subdivision unit 265 subdivides the desired partial area of ​​the base grid. At this time, the subdivision unit 265 may alternatively apply methods 1-2. In other words, the subdivision unit 265 may specify a partial area corresponding to the desired range in the grid based on the partial area information included in the header information (the grid range designation information included in the subdivision unit 265), and subdivide the specified partial area of ​​the base grid.

[0224] The displacement video decoding unit 266 decodes the provided coded data of the displacement video based on the provided header information to generate (restore) (the current frame of) the displacement video. The displacement video decoding unit 266 provides (the current frame of) the generated displacement video to the depacketizing unit 267.

[0225] The unpacking unit 267 unpacks the displacement value of the segmentation point from the current frame of the provided displacement video based on the provided header information. At this time, the unpacking unit 267 can unpack the transform coefficients and perform coefficient transformation (e.g., wavelet transform) on the transform coefficients to obtain the displacement value of the segmentation point. Alternatively, the unpacking unit 267 can unpack the quantization coefficients and inversely quantize the quantization coefficients to obtain the displacement value of the segmentation point. Alternatively, the unpacking unit 267 can unpack the quantization coefficients, inversely quantize the quantization coefficients to derive the transform coefficients, and perform coefficient transformation (e.g., wavelet transform) on the transform coefficients to obtain the displacement value of the segmentation point.

[0226] In addition, the unpacking unit 267 applies method 1 in the unpacking. In other words, the unpacking unit 267 unpacks the displacement values ​​of the segmentation points of the desired partial region. For example, the unpacking unit 267 may unpack the displacement values ​​of the segmentation points of the desired level of the desired partial region.

[0227] Alternatively, the unpacking unit 267 may apply method 1-1. In other words, the unpacking unit 267 may unpack the transform coefficients of the desired level of the desired partial area and perform inverse wavelet transform on the transform coefficients, thereby generating displacement values ​​of the division points.

[0228] Alternatively, the unpacking unit 267 may apply method 1-2. In other words, the unpacking unit 267 may specify the location where data of the desired partial area is stored in the 2D image based on the partial area information included in the header information (the displacement video range designation information included therein), and unpack the displacement value of the segmentation point of the desired level from the specified area.

[0229] The unpacking unit 267 supplies the unpacked displacement values ​​of the division points to the displacement value application unit 268 .

[0230] The displacement value application unit 268 applies the provided displacement values ​​of the segmentation points to the provided segmentation points of the base grid (subdivided base grid) to reconstruct the grid (generate a decoded grid) and outputs the generated decoded grid to the outside of the decoding device 250.

[0231] The attribute video decoding unit 269 decodes the encoded data of the provided attribute video based on the provided header information, thereby generating (the current frame of) the attribute video. The attribute video decoding unit 269 outputs the generated current frame of the attribute video (i.e., the attribute map corresponding to the decoding grid) to the outside of the decoding device 250.

[0232] With such a configuration, the decoding device 250 can realize the partial access function. Therefore, the decoding device 250 can suppress the increase in the processing amount of the decoding process and can suppress the reduction in encoding efficiency. In other words, the decoding device 250 can improve the encoding / decoding performance.

[0233] <Decoding Process Flow>

[0234] Will refer to Fig.13 The flowchart describes an example of the flow of the decoding process performed by the decoding device 250.

[0235] When the decoding process starts, the demultiplexing unit 261 demultiplexes the bit stream and extracts various types of encoded data in step S251.

[0236] In step S252, the header decoding unit 262 decodes the encoded data of the header information extracted in step S251 to generate (restore) the header information.

[0237] In step S253 , the motion vector decoding unit 263 decodes the encoded data of the motion vector extracted in step S251 to generate (restore) the motion vector.

[0238] In step S254, the base grid reconstruction unit 264 reconstructs the current base grid using the motion vector and the reference base grid.

[0239] In step S255, the subdivision unit 265 subdivides the desired partial area of ​​the current base grid. At this time, the subdivision unit 265 can specify the partial area corresponding to the desired range in the grid based on the partial area information included in the header information (the grid range designation information included in the subdivision unit 265), and subdivide the specified partial area of ​​the base grid.

[0240] In step S256 , the displacement video decoding unit 266 decodes the encoded data of the displacement video extracted in step S251 to generate (restore) (the current frame of) the displacement video.

[0241] In step S257, the unpacking unit 267 unpacks the displacement value of the desired LoD of the desired partial area from the current frame (2D image). At this time, the unpacking unit 267 may unpack the transform coefficients obtained by performing coefficient transform (e.g., wavelet transform) for each partial area, and perform inverse coefficient transform (e.g., inverse wavelet transform) on the transform coefficients for each partial area, thereby obtaining the displacement value of the segmentation point. Alternatively, the unpacking unit 267 may specify the location where the data of the desired partial area is stored in the 2D image based on the partial area information included in the header information (the displacement intra-video range designation information included therein), and unpack the displacement value of the segmentation point of the desired level from the specified area.

[0242] In step S258 , the displacement value application unit 268 applies the displacement values ​​of the division points unpacked in step S257 to the division points of the base grid subdivided in step S255 to generate a decoded grid.

[0243] In step S259, the attribute video decoding unit 269 decodes the encoded data of the attribute video extracted in step S251 to generate (restore) the current frame of the attribute video, that is, the attribute map.

[0244] When the processing of step S259 ends, the decoding processing ends.

[0245] By performing the corresponding processing as described above, the decoding device 250 can realize the partial access function. Therefore, the decoding device 250 can suppress the increase in the processing amount of the decoding process and suppress the reduction in encoding efficiency. In other words, the decoding device 250 can improve the encoding / decoding performance.

[0246] <4. Packing of base mesh vertex displacement values>

[0247] <Overall Outline of Decoding Process>

[0248] In the case of the method described in Non-Patent Document 1, according to Fig.14 For example, when the base grid is intra-coded, the decoder decodes the bit stream to generate the base grid, subdivides the base grid, and applies the displacement value to the segmentation points of the subdivided base grid, such as Fig.14 Alternatively, when the base grid is inter-coded, the decoder decodes the bitstream to generate motion vectors, applies the motion vectors to the reference base grid to reconstruct the current base grid, subdivides the current base grid, and applies the displacement values ​​to the segmentation points of the subdivided base grid, as shown in FIG. Fig.14 as shown on the lower side of .

[0249] However, when intra-coding a base mesh using such a method, subdivision is performed in the base mesh containing coding distortion, and thus there is a concern that displacement of division points will increase and coding efficiency will decrease.

[0250] In addition, when the base grid is inter-coded using such a method, the motion vector is entropy-coded for each frame, and thus it is difficult to apply inter-coding. Therefore, there is a concern that the coding efficiency will be reduced.

[0251] <Method 2>

[0252] In this regard, Fig.15 As shown in the top row of the table shown, the displacement values ​​of the vertices of the base mesh can be sent and applied before segmentation (method 2).

[0253] For example, the information processing device may include: a base mesh vertex displacement value deriving unit that derives base mesh vertex displacement values ​​as displacement values ​​of vertices of the base mesh; a packing unit that, for each fineness, layers the displacement values ​​of the segmentation points obtained by subdividing the base mesh, and packs the base mesh vertex displacement values ​​and the layered displacement values ​​of the segmentation points in a 2D image; and a displacement video encoding unit that encodes a displacement video in which the 2D image is set as a frame. Note that the base mesh may be generated by thinning out vertices from an original mesh as an encoding target, which is composed of vertices and connections representing a 3D structure of an object. In other words, the base mesh may be a mesh coarser than the original mesh. In addition, the displacement value of the segmentation point may be a positional difference between the segmentation point and the vertex (corresponding to the segmentation point) of the original mesh. In addition, the base mesh vertex displacement value may be a positional difference between the vertex of the base mesh and the vertex of the original mesh.

[0254] In addition, the information processing method may include: deriving base mesh vertex displacement values ​​as displacement values ​​of vertices of the base mesh; layering the displacement values ​​of the segmentation points obtained by subdividing the base mesh for each fineness, and packing the base mesh vertex displacement values ​​and the layered displacement values ​​of the segmentation points in a 2D image; and encoding a displacement video in which the 2D image is set as a frame. Note that the base mesh may be generated by thinning out vertices from an original mesh as an encoding target, which is composed of vertices and connections representing a 3D structure of an object. In other words, the base mesh may be a mesh coarser than the original mesh. In addition, the displacement value of the segmentation point may be a position difference between the segmentation point and the vertex of the original mesh (corresponding to the segmentation point). In addition, the base mesh vertex displacement value may be a position difference between the vertex of the base mesh and the vertex of the original mesh.

[0255] With such a configuration, the base mesh can be corrected using the base mesh vertex displacement values, thereby suppressing a decrease in encoding efficiency.

[0256] <Method 2-1>

[0257] When applying method 2, the displacement values ​​of the vertices of the current base mesh can be sent, such as Fig.15 As shown in the second row from the top of the table (Method 2-1). In other words, Method 2 can also be applied to intra-coding the base grid.

[0258] For example, the information processing device including the base mesh vertex displacement value deriving unit, the packing unit, and the displacement video encoding unit may further include: a base mesh encoding unit that encodes the base mesh of the current frame; and a base mesh decoding unit that decodes the encoded data of the base mesh of the current frame, and the base mesh vertex displacement value deriving unit may derive the displacement of the vertices between the base mesh of the current frame obtained before encoding and the base mesh of the current frame that has been encoded and decoded as the base mesh vertex displacement value. In other words, the displacement of the vertex caused by the encoding distortion may be used as the base mesh vertex displacement value.

[0259] In this case, if Fig.16 As shown on the upper side of , the decoder decodes the bitstream to generate a base mesh, applies the base mesh vertex displacement values ​​to the base mesh, subdivides the base mesh, and applies the displacement values ​​to the segmentation points of the subdivided base mesh.

[0260] With such a configuration, an increase in displacement of the division points due to encoding distortion can be suppressed, and a decrease in encoding efficiency can be suppressed.

[0261] <Method 2-2>

[0262] When applying method 2, you can send the displacement values ​​of the vertices of the reference base mesh, such as Fig.15 As shown in the bottom row of the table (Method 2-2). In other words, Method 2 can also be applied to inter-frame coding of the base grid.

[0263] For example, in an information processing device including a base mesh vertex displacement value deriving unit, a packing unit, and a displacement video encoding unit, the base mesh vertex displacement value deriving unit may derive the displacement of vertices between a base mesh of a reference frame and a base mesh of a current frame as base mesh vertex displacement values. In other words, the displacement of vertices between frames (i.e., motion vectors) may be used as base mesh vertex displacement values.

[0264] In this case, if Fig.16 As shown on the lower side of , the decoder applies the base mesh vertex displacement values ​​to the reference base mesh to reconstruct the current base mesh, subdivides the current base mesh, and applies the displacement values ​​to the segmentation points of the subdivided base mesh.

[0265] With such a configuration, the information of the motion vector can be packed in the 2D image for encoding. In other words, inter-frame encoding for 2D encoding can be applied to the motion vector, and the reduction in encoding efficiency can be suppressed.

[0266] Note that in this case, the encoder packs the base mesh vertex displacement values ​​separately from the segmentation point displacement values ​​in the 2D image. Fig.17 In the 2D image 310 shown, the base mesh vertex displacement values ​​311 can be packed at a position before the position of the displacement value of the segmentation point for each fineness. The position where the base mesh vertex displacement values ​​are packed can be predetermined, or can be an arbitrary position. When the base mesh vertex displacement values ​​are packed at an arbitrary position of the 2D image, the encoder only needs to provide information indicating the position to the decoder side.

[0267] <Encoding Device (Intra-frame)>

[0268] The present technology can be applied to an encoding device for encoding a grid. Fig.18 : is a block diagram showing an example of the configuration of an encoding device as an embodiment of an information processing device to which the present technology is applied. Figure 6 The situation shown is similar to that Fig.18 The encoding device 100 shown is a device that encodes a mesh. Note that in this case, the encoding device 100 encodes the mesh by applying the above-mentioned method 2, method 2-1, or method 2-2.

[0269] Notice, Fig.18 The main configuration including processing units, data flow, etc. is shown, but not all configurations are Fig.18 In other words, in the encoding device 100, there may be Fig.18 The processing units shown in the box may exist Fig.18 Processing and data flow indicated by arrows etc.

[0270] like Fig.18 As shown, Figure 6 Compared to the illustrated configuration, the encoding device 100 includes a base mesh vertex displacement value deriving unit 411 and a packing unit 412 instead of the packing unit 114 .

[0271] The base mesh vertex displacement value deriving unit 411 derives base mesh vertex displacement values ​​based on the current base mesh provided to the encoding device 100 and the base mesh (including encoding distortion) generated (restored) by the base mesh decoding unit 112. In other words, the base mesh vertex displacement value deriving unit 411 derives the displacement of the vertex due to the encoding distortion as the base mesh vertex displacement value. The base mesh vertex displacement value deriving unit 411 provides the derived base mesh vertex displacement values ​​to the packing unit 412.

[0272] The packing unit 412 packs the displacement values ​​of the division points provided by the displacement value correction unit 113 and the base mesh vertex displacement values ​​provided by the base mesh vertex displacement value deriving unit 411 in a 2D image.

[0273] At this time, the packing unit 412 can perform coefficient transformation (e.g., wavelet transformation) on the displacement values ​​of the segmentation points and the base mesh vertex displacement values, and pack the transformation coefficients in the 2D image. Alternatively, the packing unit 412 can quantize the displacement values ​​of the segmentation points and the base mesh vertex displacement values ​​(or the corresponding transformation coefficients), and pack the quantized coefficients in the 2D image.

[0274] The packing unit 412 provides the 2D image in which the displacement values ​​of the division points and the base mesh vertex displacement values ​​(or information corresponding thereto) are packed to the displacement video encoding unit 115 .

[0275] Other processing units and Figure 6 The case of the illustrated example is performed similarly.

[0276] With such a configuration, the encoding device 100 can suppress an increase in displacement of the division point due to encoding distortion, and can suppress a decrease in encoding efficiency. In other words, the encoding device 100 can improve encoding / decoding performance.

[0277] <Encoding Process Flow>

[0278] Will refer to Fig.19 The flowchart describes an example of the flow of the encoding process performed by the encoding device 100 in this case.

[0279] At the beginning of the encoding process, Figure 7 The corresponding processing of steps S101 to S103 shown is similarly performed to the processing of steps S401 to S403.

[0280] In step S404, the base mesh vertex displacement value deriving unit 411 derives base mesh vertex displacement values ​​based on the current base mesh obtained before and after encoding. In other words, the base mesh vertex displacement value deriving unit 411 derives the displacement of the vertex caused by encoding distortion as the base mesh vertex displacement value.

[0281] In step S405 , the packing unit 114 packs the base mesh vertex displacement values ​​derived in step S404 and the displacement values ​​of the segmentation points of each LoD in a frame (2D image).

[0282] and Figure 7 The processing of steps S406 to S410 is similarly performed to the corresponding processing of steps S105 to S109 shown.

[0283] When the process of step S410 ends, the encoding process ends.

[0284] By performing the corresponding processing as described above, the encoding device 100 can suppress the increase of the displacement of the division point due to the encoding distortion and can suppress the reduction of the encoding efficiency. In other words, the encoding device 100 can improve the encoding / decoding performance.

[0285] <Encoding Device (Inter-frame)>

[0286] Fig. 20 : is a block diagram showing an example of the configuration of an encoding device as an embodiment of an information processing device to which the present technology is applied. Figure 8 The situation shown is similar to that Fig. 20 The encoding device 150 shown is a device that encodes a mesh. The encoding device 150 basically encodes the mesh by a method similar to the method of V-DMC described in Non-Patent Document 1. Note that the encoding device 150 encodes the mesh by applying the above-mentioned method 2, method 2-1, or method 2-2.

[0287] Notice, Fig. 20 The main configuration including processing units, data flow, etc. is shown, but not all configurations are Fig. 20 In other words, in the encoding device 150, there may be Fig. 20 The processing units shown in the box may exist Fig. 20 Processing and data flow indicated by arrows etc.

[0288] like Fig. 20 As shown, Figure 8 Compared to the configuration shown, the encoding device 150 includes a base mesh vertex displacement value derivation unit 461, a current base mesh encoding unit 462, a current base mesh decoding unit 463 and a packing unit 464 instead of the motion vector encoding unit 161, the base mesh reconstruction unit 162 and the packing unit 164.

[0289] The base mesh vertex displacement value deriving unit 461 derives base mesh vertex displacement values ​​based on the current base mesh and the reference base mesh provided to the encoding device 150. In other words, the base mesh vertex displacement value deriving unit 461 derives the inter-frame displacements of the vertices of the base mesh as base mesh vertex displacement values. The base mesh vertex displacement value deriving unit 461 provides the derived base mesh vertex displacement values ​​to the packing unit 464.

[0290] The current base grid encoding unit 462 encodes the current base grid provided to the encoding device 150 and provides the encoded data to the current base grid decoding unit 463.

[0291] The current base grid decoding unit 463 decodes the coded data of the provided current base grid to generate (restore) the current base grid. The generated current base grid contains coding distortion. The current base grid decoding unit 463 provides the generated current base grid (containing coding distortion) to the displacement value correction unit 163.

[0292] The packing unit 464 packs the displacement values ​​of the division points supplied by the displacement value correction unit 163 and the base mesh vertex displacement values ​​supplied by the base mesh vertex displacement value deriving unit 461 in a 2D image.

[0293] At this time, the packing unit 464 can perform coefficient transformation (e.g., wavelet transformation) on the displacement values ​​of the segmentation points and the base mesh vertex displacement values, and pack the transformation coefficients in the 2D image. Alternatively, the packing unit 464 can quantize the displacement values ​​of the segmentation points and the base mesh vertex displacement values ​​(or the transformation coefficients corresponding thereto), and pack the quantized coefficients in the 2D image.

[0294] The packing unit 464 provides the 2D image in which the displacement values ​​of the division points and the base mesh vertex displacement values ​​(or information corresponding thereto) are packed to the displacement video encoding unit 165 .

[0295] Other processing units and Figure 6 The case of the illustrated example is performed similarly.

[0296] With such a configuration, the encoding device 150 can pack the information of the motion vector in the 2D image for encoding. In other words, the encoding device 150 can apply the inter-frame encoding used for 2D encoding to the motion vector. Therefore, the encoding device 150 can suppress the reduction of the encoding efficiency. In other words, the encoding device 150 can improve the encoding / decoding performance.

[0297] <Encoding Process Flow>

[0298] Will refer to Fig.21 The flowchart describes an example of the flow of the encoding process performed by the encoding device 150 in this case.

[0299] When the encoding process starts, in step S451, the base mesh vertex displacement value deriving unit 461 derives base mesh vertex displacement values ​​based on the current base mesh and the reference base mesh. In other words, the base mesh vertex displacement value deriving unit 461 derives the inter-frame displacement of the vertices of the base mesh as the base mesh vertex displacement values.

[0300] In step S452, the current base grid encoding unit 462 encodes the current base grid.

[0301] In step S453, the current base grid decoding unit 463 decodes the encoded data of the current base grid generated in step S452 to generate (restore) the current base grid.

[0302] In step S454 , the displacement value correction unit 163 corrects the displacement values ​​of the division points using the base grid (containing the encoding distortion) generated (restored) in step S453 .

[0303] In step S455 , the packing unit 464 packs the displacement values ​​of the segmentation points of each LoD and the base mesh vertex displacement values ​​into a 2D image.

[0304] and Fig. 9 The processing of steps S456 to S460 is similarly performed to the corresponding processing of steps S155 to S159 shown.

[0305] When the process of step S460 ends, the encoding process ends.

[0306] By performing the corresponding processing as described above, the encoding device 150 can pack the information of the motion vector in the 2D image for encoding. In other words, the encoding device 150 can apply the inter-frame encoding for 2D encoding to the motion vector. Therefore, the encoding device 150 can suppress the reduction of the encoding efficiency. In other words, the encoding device 150 can improve the encoding / decoding performance.

[0307] <Method 2, Method 2-1 and Method 2-2>

[0308] Note that the above-mentioned Method 2, Method 2-1, and Method 2-2 can also be applied to a decoding device.

[0309] For example, the information processing device may include: a displacement video decoding unit that decodes a bit stream and generates a displacement video in which a 2D image is set as a frame, wherein base mesh vertex displacement values ​​as displacement values ​​of vertices of a base mesh and displacement values ​​of a segmentation point obtained by subdividing the base mesh are packed in the 2D image; an unpacking unit that unpacks the base mesh vertex displacement values ​​and the displacement values ​​of the segmentation point that have been packed in the 2D image of the current frame of the displacement video; a base mesh vertex position application unit that applies the base mesh vertex displacement values ​​to the vertices of the base mesh; a subdivision unit that subdivides the base mesh to which the base mesh vertex displacement values ​​are applied to the vertices; and a displacement value application unit that applies the displacement values ​​of the segmentation point to the segmentation points of the subdivided base mesh. Note that the base mesh may be generated by thinning out vertices from an original mesh as an encoding target composed of vertices and connections representing a 3D structure of an object. In other words, the base mesh may be a mesh coarser than the original mesh. In addition, the displacement value of the segmentation point may be a position difference between the segmentation point and the vertex (corresponding to the segmentation point) of the original mesh. Furthermore, the base mesh vertex displacement values ​​may be positional differences between vertices of the base mesh and vertices of the original mesh.

[0310] In addition, the information processing method may include: decoding a bit stream and generating a displacement video in which a 2D image is set as a frame, wherein base mesh vertex displacement values ​​as displacement values ​​of vertices of a base mesh and displacement values ​​of segmentation points obtained by subdividing the base mesh are packed in the 2D image; unpacking the base mesh vertex displacement values ​​and the displacement values ​​of the segmentation points that have been packed in the 2D image of the current frame of the displacement video; applying the base mesh vertex displacement values ​​to the vertices of the base mesh; subdividing the base mesh to which the base mesh vertex displacement values ​​are applied to the vertices; and applying the displacement values ​​of the segmentation points to the segmentation points of the subdivided base mesh. Note that the base mesh may be generated by thinning out vertices from an original mesh as an encoding target composed of vertices and connections representing a 3D structure of an object. In other words, the base mesh may be a mesh coarser than the original mesh. In addition, the displacement value of the segmentation point may be a position difference between the segmentation point and the vertex (corresponding to the segmentation point) of the original mesh. In addition, the base mesh vertex displacement value may be a position difference between the vertex of the base mesh and the vertex of the original mesh.

[0311] With such a configuration, the base mesh can be corrected using the base mesh vertex displacement values, thereby suppressing a decrease in encoding efficiency.

[0312] In addition, for example, the information processing device including a displacement video decoding unit, an unpacking unit, a base mesh vertex position application unit, a subdivision unit and a displacement value application unit may also include: a base mesh decoding unit that decodes the bit stream and generates a base mesh of the current frame, and a base mesh vertex position application unit that can apply base mesh vertex displacement values ​​to the vertices of the generated base mesh of the current frame.

[0313] With such a configuration, an increase in displacement of the division points due to encoding distortion can be suppressed, and a decrease in encoding efficiency can be suppressed.

[0314] In addition, for example, in an information processing device including a displacement video decoding unit, a depacketizing unit, a base mesh vertex position application unit, a subdivision unit, and a displacement value application unit, the base mesh vertex position application unit may apply base mesh vertex displacement values ​​to vertices of a base mesh of a reference frame.

[0315] With such a configuration, the information of the motion vector can be packed in the 2D image for encoding. In other words, inter-frame encoding for 2D encoding can be applied to the motion vector, and the reduction in encoding efficiency can be suppressed.

[0316] <Decoding Device (Intra-frame)>

[0317] The present technology can be applied to a decoding device that decodes coded data of a grid. Fig. 22 : is a block diagram showing an example of the configuration of a decoding device as an embodiment of an information processing device to which the present technology is applied. Fig.10 The situation shown is similar to that Fig. 22 The illustrated decoding device 200 is a device that decodes the coded data of the mesh. Note that in this case, the decoding device 200 decodes the coded data of the mesh by applying the above-mentioned method 2, method 2-1, or method 2-2.

[0318] Notice, Fig. 22 The main configuration including processing units, data flow, etc. is shown, but not all configurations are Fig. 22 In other words, in the decoding device 200, there may be Fig. 22 The processing units shown in the box may exist Fig. 22 Processing and data flow indicated by arrows etc.

[0319] like Fig. 22 As shown, Fig.10 Compared to the illustrated configuration, the decoding apparatus 200 includes an unpacking unit 511 and a base mesh vertex displacement value application unit 512 instead of the unpacking unit 216 .

[0320] The unpacking unit 511 unpacks the base mesh vertex displacement values ​​and the displacement values ​​of the segmentation points from the current frame of the displacement video provided by the displacement video decoding unit 215. At this time, the unpacking unit 511 can unpack the transformation coefficients and perform coefficient transformation (e.g., wavelet transformation) on the transformation coefficients, thereby obtaining the displacement values ​​of the segmentation points and the base mesh vertex displacement values. Alternatively, the unpacking unit 511 can unpack the quantization coefficients and perform inverse quantization on the quantization coefficients, thereby obtaining the displacement values ​​of the segmentation points and the base mesh vertex displacement values. Alternatively, the unpacking unit 511 can unpack the quantization coefficients, perform inverse quantization on the quantization coefficients to derive the transformation coefficients, and perform coefficient transformation (e.g., wavelet transformation) on the transformation coefficients, thereby obtaining the displacement values ​​of the segmentation points and the base mesh vertex displacement values.

[0321] The unpacking unit 511 supplies the unpacked base mesh vertex displacement values ​​to the base mesh vertex displacement value application unit 512. In addition, the unpacking unit 511 supplies the unpacked displacement values ​​of the segmentation points to the displacement value application unit 217.

[0322] The base mesh vertex displacement value application unit 512 applies the base mesh vertex displacement values ​​provided by the unpacking unit 511 to the vertices of the current base mesh provided by the base mesh decoding unit 213. Therefore, encoding distortion is reduced. The base mesh vertex displacement value application unit 512 provides the processed base mesh to the subdivision unit 214.

[0323] Other processing units and Fig.10 The case of the illustrated example is performed similarly.

[0324] With such a configuration, the decoding device 200 can suppress an increase in displacement of the division point due to coding distortion, and can suppress a decrease in coding efficiency. In other words, the decoding device 200 can improve coding / decoding performance.

[0325] <Decoding Process Flow>

[0326] Will refer to Fig.23 The flowchart of FIG. 2 describes an example of the flow of the decoding process performed by the decoding device 200 in this case.

[0327] At the beginning of the encoding process, Fig.11 The corresponding processing of steps S201 to S203 shown is similarly performed to the processing of steps S501 to S503.

[0328] In step S504 , the unpacking unit 511 unpacks the vertex positions of the base mesh.

[0329] In step S505 , the base mesh vertex displacement value application unit 512 applies the base mesh vertex displacement values ​​to the vertices of the base mesh.

[0330] and Figure 7 The processing of steps S506 to S510 is similarly performed to the corresponding processing of steps S204 to S208 shown.

[0331] When the process of step S510 ends, the encoding process ends.

[0332] By performing the corresponding processing as described above, the decoding device 200 can suppress the increase of the displacement of the division point due to the coding distortion and can suppress the reduction of the coding efficiency. In other words, the decoding device 200 can improve the encoding / decoding performance.

[0333] <Decoding Device (Inter-frame)>

[0334] Fig.24 : is a block diagram showing an example of the configuration of an encoding device as an embodiment of an information processing device to which the present technology is applied. Fig.12 The situation shown is similar to Fig.24 The decoding device 250 shown is a device that decodes the coded data of the mesh. Note that the decoding device 250 decodes the coded data of the mesh by applying the above-mentioned method 2, method 2-1 or method 2-2.

[0335] Notice, Fig.24 The main configuration including processing units, data flow, etc. is shown, but not all configurations are Fig.24 In other words, in the decoding device 250, there may be Fig.24 The processing units shown in the box may exist Fig.24 Processing and data flow indicated by arrows etc.

[0336] like Fig.24 As shown, Fig.12 Compared to the configuration shown, the decoding device 250 includes a depacketizing unit 561 and a base grid reconstruction unit 562 instead of the motion vector decoding unit 263 , the base grid reconstruction unit 264 , and the depacketizing unit 267 .

[0337] The unpacking unit 561 unpacks the base mesh vertex displacement values ​​and the displacement values ​​of the segmentation points from the current frame of the displacement video provided by the displacement video decoding unit 266. At this time, the unpacking unit 561 can unpack the transform coefficients and perform coefficient transformation (e.g., wavelet transform) on the transform coefficients, thereby obtaining the displacement values ​​of the segmentation points and the base mesh vertex displacement values. Alternatively, the unpacking unit 561 can unpack the quantization coefficients and perform inverse quantization on the quantization coefficients, thereby obtaining the displacement values ​​of the segmentation points and the base mesh vertex displacement values. Alternatively, the unpacking unit 561 can unpack the quantization coefficients, perform inverse quantization on the quantization coefficients to derive the transform coefficients, and perform coefficient transformation (e.g., wavelet transform) on the transform coefficients, thereby obtaining the displacement values ​​of the segmentation points and the base mesh vertex displacement values.

[0338] The unpacking unit 561 supplies the unpacked base mesh vertex displacement values ​​to the base mesh reconstruction unit 562. In addition, the unpacking unit 561 supplies the unpacked displacement values ​​of the segmentation points to the displacement value application unit 268.

[0339] The base mesh reconstruction unit 562 applies the base mesh vertex displacement values ​​provided by the unpacking unit 561 to the vertices of the reference base mesh. Thus, the current base mesh is reconstructed. The base mesh reconstruction unit 562 provides the reconstructed current base mesh to the subdivision unit 265.

[0340] Other processing units and Fig.12 The case of the illustrated example is performed similarly.

[0341] With such a configuration, the decoding device 250 can pack the information of the motion vector in the 2D image for encoding. In other words, the decoding device 250 can apply the inter-frame encoding used for 2D encoding to the motion vector. Therefore, the decoding device 250 can suppress the reduction of the encoding efficiency. In other words, the decoding device 250 can improve the encoding / decoding performance.

[0342] <Decoding Process Flow>

[0343] Will refer to Fig.25 The flowchart describes an example of the flow of the decoding process performed by the decoding device 250 in this case.

[0344] At the beginning of the decoding process, Fig.13 The corresponding processing of steps S251, S252 and S256 shown is similarly performed to the processing of steps S551 to S553.

[0345] In step S554 , the unpacking unit 561 unpacks the vertex positions of the base mesh.

[0346] In step S555 , the base mesh reconstruction unit 562 applies the base mesh vertex displacement values ​​to the vertices of the reference base mesh to reconstruct the current base mesh.

[0347] and Fig.13 The processing of steps S556 to S559 is similarly performed to the corresponding processing of steps S255 and S257 to S259 shown.

[0348] When the process of step S560 is completed, the decoding process is completed.

[0349] By performing the corresponding processing as described above, the decoding device 250 can pack the information of the motion vector in the 2D image for encoding. In other words, the decoding device 250 can apply the inter-frame encoding for 2D encoding to the motion vector. Therefore, the decoding device 250 can suppress the reduction of the encoding efficiency. In other words, the decoding device 250 can improve the encoding / decoding performance.

[0350] <5. Notes>

[0351] <Computer>

[0352] The above series of processes may be performed by hardware or may be performed by software. When a series of processes are performed by software, a program configuring the software is installed in a computer. In this article, a computer includes a computer incorporated into dedicated hardware, a computer such as a general-purpose personal computer in which various programs can be installed to perform various functions, and the like.

[0353] Fig.26 : is a block diagram showing a hardware configuration example of a computer that executes the above-described series of processes by a program.

[0354] exist Fig.26 In the computer 900 shown, a CPU (Central Processing Unit) 901 , a ROM (Read Only Memory) 902 , and a RAM (Random Acess Memory) 903 are connected to each other via a bus 904 .

[0355] An input / output interface 910 is also connected to the bus 904. An input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915 are connected to the input / output interface 910.

[0356] The input unit 911 is composed of, for example, a keyboard, a mouse, a microphone, a touch panel, an input terminal, etc. The output unit 912 is composed of, for example, a display, a speaker, an output terminal, etc. The storage unit 913 is composed of, for example, a hard disk, a RAM disk, a nonvolatile memory, etc. The communication unit 914 is composed of, for example, a network interface. The drive 915 drives a removable medium 921 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0357] In the computer configured as described above, for example, the CPU 901 loads a program stored in the storage unit 913 into the RAM 903 via the input / output interface 910 and the bus 904, and executes the program to perform the above-described series of processing. Data required for the CPU 901 to perform various types of processing, etc., is also appropriately stored in the RAM 903.

[0358] For example, the program executed by the computer can be applied by being recorded on the removable medium 921 as a package medium, etc. In this case, by loading the removable medium 921 into the drive 915 , the program can be installed in the storage unit 913 via the input / output interface 910 .

[0359] In addition, the program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, and digital satellite broadcasting. In this case, the program can be received by the communication unit 914 to be installed in the storage unit 913.

[0360] Alternatively, the program may be installed in the ROM 902 or the storage unit 913 in advance.

[0361] <Application target of this technology>

[0362] The present technology can be applied to any configuration. For example, the present technology can be applied to various electronic devices.

[0363] In addition, for example, the present technology can be implemented as a partial configuration of an apparatus, such as a processor as a system LSI (Large Scale Integration) or the like (e.g., a video processor), a module using multiple processors or the like (e.g., a video module), a unit using multiple modules or the like (e.g., a video unit), or a device obtained by adding other functions to the unit (e.g., a video device).

[0364] In addition, for example, the present technology can also be applied to a network system composed of multiple devices. For example, the present technology can be implemented as cloud computing, in which multiple devices share and collaborate to process via a network. For example, the present technology can be implemented in, for example, a cloud service that provides image (moving image) related services for any terminal such as a computer, an AV (Audio Visual) device, a mobile information processing terminal, and an IoT (Internet of Things) device.

[0365] Note that in this specification, a system refers to a collection of multiple constituent elements (devices, modules (components), etc.), and whether all the constituent elements are in the same housing is irrelevant. Therefore, multiple devices housed in different housings and connected via a network and a single device in which multiple modules are housed in a single housing are both systems.

[0366] <Fields / Applications Where This Technology Can Be Applied>

[0367] Systems, devices, processing units, etc. to which the present technology is applied can be used in any field such as transportation, medical care, crime prevention, agriculture, animal husbandry, mining, beauty care, factories, home appliances, weather, nature monitoring, etc. In addition, its use is also arbitrary.

[0368] <Others>

[0369] Note that the "flag" used in this specification is information used to identify multiple states, and includes not only information used when identifying two states, namely true (1) and false (0), but also information with which three or more states can be identified. Therefore, the value that the "flag" can take can be, for example, a binary value such as 1 / 0 or a ternary or higher-order value. In other words, the number of bits configuring the "flag" is arbitrary and can be 1 bit or more. In addition, since the identification information (including the flag) can take not only the form in which the identification information is incorporated into the bit stream, but also the form in which the difference information of the identification information relative to a certain information that serves as a reference is incorporated into the bit stream, in this specification, the "flag" or "identification information" includes not only the information, but also the difference information relative to the information that serves as a reference.

[0370] In addition, various types of information (metadata, etc.) related to the coded data (bitstream) can be sent or recorded in any form, as long as the information is associated with the coded data. In this article, the term "association" means that when processing one of the data, other data can be used (linked). In other words, the data associated with each other can be compiled into a piece of data, or can be regarded as separate data. For example, information associated with the coded data (image) can be transmitted on a transmission path different from the transmission path of the coded data (image). In addition, for example, the information associated with the coded data (image) can be recorded on a recording medium different from the recording medium of the coded data (image) (or in different recording areas in the same recording medium). Note that this "association" can be applied not only to the entire data but also to a part of the data. For example, an image and information corresponding to the image can be associated with each other in arbitrary units including multiple frames, one frame, a part of a frame, etc.

[0371] Note that in this specification, for example, the terms "synthesize", "multiplex", "add", "integrate", "merge", "store", "put in", "introduce" and "insert" mean compiling multiple objects into one, such as compiling encoded data and metadata into one piece of data, and refer to a method of the above-mentioned "association".

[0372] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications may be made without departing from the gist of the present technology.

[0373] For example, a configuration described as one device (or processing unit) may be divided and configured as a plurality of devices (or processing units). Conversely, a configuration described above as a plurality of devices (or processing units) may be configured together as one device (or processing unit). In addition, configurations other than the above configurations may of course be added to the configuration of the corresponding device (or corresponding processing unit). In addition, as long as the configuration and operation of the entire system are substantially the same, a portion of the configuration of a certain device (or processing unit) may be incorporated into the configuration of another device (or another processing unit).

[0374] In addition, for example, the above-mentioned program can be executed in any device. In this case, the device only needs to have necessary functions (functional blocks, etc.) to be able to acquire necessary information.

[0375] In addition, for example, the corresponding steps of a single flowchart may be performed by a single device, or may be shared by a plurality of devices for execution. In addition, when a plurality of processes are included in a single step, the plurality of processes may be performed by a single device, or may be shared by a plurality of devices for execution. In other words, the plurality of processes included in a single step may be executed as processes of a plurality of steps. Conversely, the processes described as a plurality of steps may be executed as a single step as a whole.

[0376] Furthermore, for example, with respect to a program executed by a computer, the processing of the steps describing the program may be performed in time series in the order described in this specification, or may be performed separately in parallel at necessary timings such as when called. In other words, the processing of the corresponding steps may be performed in an order different from the above order as long as no contradiction is caused. In addition, the processing of the steps describing the program may be performed in parallel with the processing of other programs, or may be performed in combination with the processing of other programs.

[0377] In addition, for example, as long as no contradiction is caused, multiple technologies related to the present technology can be performed independently and separately. Of course, multiple arbitrary present technologies can be performed in combination. For example, part or all of the present technology described in any one of the embodiments can be performed in combination with part or all of the present technology described in other embodiments. In addition, part or all of any of the above-mentioned present technologies can be performed in combination with other technologies not described above.

[0378] Note that the present technology can also take the following configurations.

[0379] (1) An information processing device comprising:

[0380] a packing unit that, for each fineness, layers the displacement values ​​of the segmentation points obtained by subdividing the base grid, and packs the layered displacement values ​​of the segmentation points in the 2D image for each of the partial regions of the object; and

[0381] A displacement video encoding unit is used to encode the displacement video in which the 2D image is set as a frame, wherein:

[0382] The base mesh is generated by thinning out the vertices from an original mesh as an encoding target composed of vertices and connections representing a 3D structure of the object, and the base mesh is a mesh coarser than the original mesh, and

[0383] The displacement value is the position difference between the segmentation point and the vertex of the original mesh.

[0384] (2) The information processing device according to (1), wherein:

[0385] The packing unit performs wavelet transformation on the displacement values ​​of the division points for each of the partial regions, and packs transformation coefficients obtained thereby into the 2D image.

[0386] (3) The information processing device according to (1) or (2), wherein:

[0387] The displacement video encoding unit further encodes partial area information, and

[0388] The partial region information includes mesh range designation information indicating a range of the partial region in the mesh and in-displacement video range designation information indicating a range of the partial region in the 2D image of the displacement video.

[0389] (4) The information processing device according to any one of (1) to (3), further comprising:

[0390] A base grid encoding unit encodes the base grid of the current frame.

[0391] (5) The information processing device according to any one of (1) to (3), further comprising:

[0392] A motion vector encoding unit encodes motion vectors of vertices of the base mesh of a current frame using the base mesh of a reference frame as a reference.

[0393] (6) An information processing method comprising:

[0394] hierarchical displacement values ​​of segmentation points obtained by subdividing a base grid for each fineness, and packing the hierarchical displacement values ​​of the segmentation points in a 2D image for each of the partial regions of the object; and

[0395] The 2D image is set as a displacement video of a frame, wherein:

[0396] The base mesh is generated by thinning out the vertices from an original mesh as an encoding target composed of vertices and connections representing a 3D structure of the object, and the base mesh is a mesh coarser than the original mesh, and

[0397] The displacement value is the position difference between the segmentation point and the vertex of the original mesh.

[0398] (11) An information processing device comprising:

[0399] a displacement video decoding unit that decodes the bit stream and generates a displacement video in which a 2D image is set as a frame, wherein displacement values ​​of a segmentation point obtained by subdividing a base grid are layered for each fineness and packed in the 2D image for each partial region of a partial region of an object;

[0400] an unpacking unit, which unpacks the displacement values ​​of the segmentation points of the desired level of the desired partial area that have been packed in the 2D image of the current frame of the displacement video;

[0401] a subdivision unit, which subdivides the base grid of the current frame; and

[0402] A displacement value application unit, which applies the displacement value of the segmentation point to the segmentation point of the subdivided base grid, wherein:

[0403] The base mesh is generated by thinning out the vertices from an original mesh as an encoding target composed of vertices and connections representing a 3D structure of the object, and the base mesh is a mesh coarser than the original mesh, and

[0404] The displacement value is the position difference between the segmentation point and the vertex of the original mesh.

[0405] (12) The information processing device according to (11), wherein:

[0406] Transformation coefficients obtained by wavelet transforming the displacement values ​​of the segmentation points for each of the partial regions are packed in the 2D image, and

[0407] The unpacking unit unpacks the transform coefficients of the desired level of the desired partial area and subjects the transform coefficients to inverse wavelet transform, thereby generating displacement values ​​of the segmentation points.

[0408] (13) The information processing device according to (11) or (12), further comprising:

[0409] A header decoding unit, which decodes the bit stream and generates partial area information, wherein the partial area information includes grid range designation information indicating the range of the partial area in the grid and displacement video intra-range designation information indicating the range of the partial area in the 2D image of the displacement video, wherein

[0410] The unpacking unit unpacks the displacement values ​​of the segmentation points of the desired level of the desired partial area based on the displacement video intra-range designation information, and

[0411] The subdividing unit subdivides a desired partial area of ​​the base grid based on the grid range specifying information.

[0412] (14) The information processing device according to any one of (11) to (13), further comprising:

[0413] A base grid decoding unit decodes a bit stream and generates the base grid of the current frame.

[0414] (15) The information processing device according to any one of (11) to (13), further comprising:

[0415] a motion vector generation unit that decodes the bit stream and generates a motion vector; and

[0416] A base mesh reconstruction unit applies the motion vector to vertices of the base mesh of a reference frame to reconstruct the base mesh of the current frame.

[0417] (16) An information processing method comprising:

[0418] decoding the bitstream and generating a displacement video in which a 2D image is set as a frame, wherein displacement values ​​of a segmentation point obtained by subdividing a base grid are layered for each fineness and packed in the 2D image for each partial region of a partial region of an object;

[0419] unpacking the displacement values ​​of the segmentation points of the desired level of the desired partial area that have been packed in the 2D image of the current frame of the displacement video;

[0420] subdividing the base grid of the current frame; and

[0421] The displacement value of the segmentation point is applied to the segmentation point of the subdivided base grid, wherein:

[0422] The base mesh is generated by thinning out the vertices from an original mesh as an encoding target composed of vertices and connections representing a 3D structure of the object, and the base mesh is a mesh coarser than the original mesh, and

[0423] The displacement value is the position difference between the segmentation point and the vertex of the original mesh.

[0424] (21) An information processing device comprising:

[0425] a base mesh vertex displacement value deriving unit, which derives base mesh vertex displacement values ​​as displacement values ​​of vertices of the base mesh;

[0426] a packing unit that, for each fineness, layers displacement values ​​of segmentation points obtained by subdividing the base mesh, and packs the base mesh vertex displacement values ​​and the layered displacement values ​​of the segmentation points in a 2D image; and

[0427] A displacement video encoding unit is used to encode the displacement video in which the 2D image is set as a frame, wherein:

[0428] The base mesh is generated by thinning out the vertices from an original mesh as an encoding target composed of vertices and connections representing a 3D structure of an object, and the base mesh is a mesh coarser than the original mesh,

[0429] The displacement value of the segmentation point is the position difference between the segmentation point and the vertex of the original mesh, and

[0430] The base mesh vertex displacement value is a position difference between the vertex of the base mesh and the vertex of the original mesh.

[0431] (22) The information processing device according to (21), further comprising:

[0432] a base grid encoding unit, which encodes the base grid of the current frame; and

[0433] A base grid decoding unit is configured to decode the encoded data of the base grid of the current frame, wherein:

[0434] The base mesh vertex displacement value deriving unit derives, as the base mesh vertex displacement values, displacements of the vertices between the base mesh of the current frame obtained before encoding and the base mesh of the current frame that has been encoded and decoded.

[0435] (23) The information processing device according to (21), wherein:

[0436] The base mesh vertex displacement value deriving unit derives the displacement of the vertex between the base mesh of a reference frame and the base mesh of a current frame as the base mesh vertex displacement value.

[0437] (24) An information processing method comprising:

[0438] deriving base mesh vertex displacement values ​​as displacement values ​​of vertices of the base mesh;

[0439] For each fineness, layering the displacement values ​​of the segmentation points obtained by subdividing the base mesh, and packing the base mesh vertex displacement values ​​and the layered displacement values ​​of the segmentation points in a 2D image; and

[0440] The 2D image is set as a displacement video of a frame, wherein:

[0441] The base mesh is generated by thinning out the vertices from an original mesh as an encoding target composed of vertices and connections representing a 3D structure of an object, and the base mesh is a mesh coarser than the original mesh,

[0442] The displacement value of the segmentation point is the position difference between the segmentation point and the vertex of the original mesh, and

[0443] The base mesh vertex displacement value is a position difference between the vertex of the base mesh and the vertex of the original mesh.

[0444] (31) An information processing device comprising:

[0445] a displacement video decoding unit that decodes the bit stream and generates a displacement video in which a 2D image is set as a frame, wherein base mesh vertex displacement values ​​that are displacement values ​​of vertices of a base mesh and displacement values ​​of division points obtained by subdividing the base mesh are packed in the 2D image;

[0446] an unpacking unit, which unpacks the base mesh vertex displacement values ​​and the segmentation point displacement values ​​that have been packed in the 2D image of the current frame of the displacement video;

[0447] a base mesh vertex position application unit, which applies the base mesh vertex displacement values ​​to the vertices of the base mesh;

[0448] a subdivision unit that subdivides the base mesh in which the base mesh vertex displacement values ​​are applied to the vertices; and

[0449] A displacement value application unit, which applies the displacement value of the segmentation point to the segmentation point of the subdivided base grid, wherein:

[0450] The base mesh is generated by thinning out the vertices from an original mesh as an encoding target composed of vertices and connections representing a 3D structure of an object, and the base mesh is a mesh coarser than the original mesh,

[0451] The displacement value of the segmentation point is the position difference between the segmentation point and the vertex of the original mesh, and

[0452] The base mesh vertex displacement value is a position difference between the vertex of the base mesh and the vertex of the original mesh.

[0453] (32) The information processing device according to (31), further comprising:

[0454] A base grid decoding unit, which decodes the bit stream and generates the base grid of the current frame, wherein

[0455] The base mesh vertex position application unit applies the base mesh vertex displacement values ​​to the generated vertices of the base mesh of the current frame.

[0456] (33) The information processing device according to (31), wherein:

[0457] The base mesh vertex position application unit applies the base mesh vertex displacement values ​​to vertices of the base mesh of a reference frame.

[0458] (34) An information processing method comprising:

[0459] decoding the bitstream and generating a displacement video in which a 2D image is set as a frame, wherein base mesh vertex displacement values ​​that are displacement values ​​of vertices of a base mesh and displacement values ​​of segmentation points obtained by subdividing the base mesh are packed in the 2D image;

[0460] Unpacking the base mesh vertex displacement values ​​and the segmentation point displacement values ​​that have been packed in the 2D image of the current frame of the displacement video;

[0461] Applying the base mesh vertex displacement values ​​to the vertices of the base mesh;

[0462] subdividing the base mesh with the base mesh vertex displacement values ​​applied to the vertices; and

[0463] The displacement value of the segmentation point is applied to the segmentation point of the subdivided base grid, wherein:

[0464] The base mesh is generated by thinning out the vertices from an original mesh as an encoding target composed of vertices and connections representing a 3D structure of an object, and the base mesh is a mesh coarser than the original mesh,

[0465] The displacement value of the segmentation point is the position difference between the segmentation point and the vertex of the original mesh, and

[0466] The base mesh vertex displacement value is a position difference between the vertex of the base mesh and the vertex of the original mesh.

[0467] Reference numerals list

[0468] 100 encoding device, 111 base grid encoding unit, 112 base grid decoding unit, 113 displacement value correction unit, 114 packing unit, 115 displacement video encoding unit, 116 grid reconstruction unit, 117 attribute map correction unit, 118 attribute video encoding unit, 119 synthesis unit, 150 encoding device, 161 motion vector encoding unit, 162 base grid reconstruction unit, 163 displacement value correction unit, 164 packing unit, 165 displacement video encoding unit, 166 grid reconstruction unit, 167 attribute map correction unit, 168 attribute video encoding unit, 169 synthesis unit, 200 decoding device, 211 demultiplexing unit, 212 header decoding unit, 213 base grid decoding unit, 214 subdivision unit, 215 displacement video decoding unit code unit, 216 unpacking unit, 217 displacement value application unit, 218 attribute video decoding unit, 250 decoding device, 261 demultiplexing unit, 262 header decoding unit, 263 motion vector decoding unit, 264 base grid reconstruction unit, 265 subdivision unit, 266 displacement video decoding unit, 267 unpacking unit, 268 displacement value application unit, 269 attribute video decoding unit, 411 base grid vertex displacement value deriving unit, 412 packing unit, 461 base grid vertex displacement value deriving unit, 462 current base grid encoding unit, 463 current base grid decoding unit, 464 packing unit, 511 unpacking unit, 512 base grid vertex displacement value application unit, 561 unpacking unit, 562 base grid reconstruction unit, 900 computer

Claims

1. An information processing device, comprising: a packing unit that, for each fineness, layers the displacement values ​​of the segmentation points obtained by subdividing the base grid, and packs the layered displacement values ​​of the segmentation points in the 2D image for each partial region of the object; as well as A displacement video encoding unit is used to encode the displacement video in which the 2D image is set as a frame, wherein: The base mesh is generated by thinning out the vertices from an original mesh as an encoding target composed of vertices and connections representing a 3D structure of the object, and the base mesh is a mesh coarser than the original mesh, and The displacement value is the position difference between the segmentation point and the vertex of the original mesh.

2. The information processing device according to claim 1, wherein: The packing unit performs wavelet transformation on the displacement values ​​of the division points for each of the partial regions, and packs transformation coefficients obtained thereby into the 2D image.

3. The information processing device according to claim 1, wherein: The displacement video encoding unit further encodes partial area information, and The partial region information includes mesh range designation information indicating a range of the partial region in the mesh and in-displacement video range designation information indicating a range of the partial region in the 2D image of the displacement video.

4. The information processing device according to claim 1, further comprising: A base grid encoding unit encodes the base grid of the current frame.

5. The information processing device according to claim 1, further comprising: A motion vector encoding unit encodes motion vectors of vertices of the base mesh of a current frame using the base mesh of a reference frame as a reference.

6. An information processing method, comprising: For each fineness, the displacement values ​​of the segmentation points obtained by subdividing the base grid are layered, and for each of the partial regions of the object, the layered displacement values ​​of the segmentation points are packed in a 2D image; as well as The 2D image is set as a displacement video of a frame, wherein: The base mesh is generated by thinning out the vertices from an original mesh as an encoding target composed of vertices and connections representing a 3D structure of the object, and the base mesh is a mesh coarser than the original mesh, and The displacement value is the position difference between the segmentation point and the vertex of the original mesh.

7. An information processing device, comprising: a displacement video decoding unit that decodes the bit stream and generates a displacement video in which a 2D image is set as a frame, wherein displacement values ​​of a segmentation point obtained by subdividing a base grid are layered for each fineness and packed in the 2D image for each partial region of a partial region of an object; an unpacking unit, which unpacks the displacement values ​​of the segmentation points of the desired level of the desired partial area that have been packed in the 2D image of the current frame of the displacement video; a subdivision unit, which subdivides the base grid of the current frame; and A displacement value application unit, which applies the displacement value of the segmentation point to the segmentation point of the subdivided base grid, wherein: The base mesh is generated by thinning out the vertices from an original mesh as an encoding target composed of vertices and connections representing a 3D structure of the object, and the base mesh is a mesh coarser than the original mesh, and The displacement value is the position difference between the segmentation point and the vertex of the original mesh.

8. The information processing device according to claim 7, wherein: Transformation coefficients obtained by wavelet transforming the displacement values ​​of the segmentation points for each of the partial regions are packed in the 2D image, and The unpacking unit unpacks the transform coefficients of the desired level of the desired partial area and subjects the transform coefficients to inverse wavelet transform, thereby generating displacement values ​​of the segmentation points.

9. The information processing device according to claim 7, further comprising: a header decoding unit, which decodes the bit stream and generates partial area information, wherein the partial area information includes grid range designation information indicating the range of the partial area in the grid and displacement video intra-range designation information indicating the range of the partial area in the 2D image of the displacement video, wherein The unpacking unit unpacks the displacement values ​​of the segmentation points of the desired level of the desired partial area based on the displacement video intra-range designation information, and The subdividing unit subdivides a desired partial area of ​​the base grid based on the grid range specifying information.

10. The information processing device according to claim 7, further comprising: A base grid decoding unit decodes a bit stream and generates the base grid of the current frame.

11. The information processing device according to claim 7, further comprising: A motion vector generation unit, which decodes the bit stream and generates a motion vector; as well as A base mesh reconstruction unit applies the motion vector to vertices of the base mesh of a reference frame to reconstruct the base mesh of the current frame.

12. An information processing method, comprising: decoding the bitstream and generating a displacement video in which a 2D image is set as a frame, wherein displacement values ​​of a segmentation point obtained by subdividing a base grid are layered for each fineness and packed in the 2D image for each partial region of a partial region of an object; unpacking the displacement values ​​of the segmentation points of the desired level of the desired partial area that have been packed in the 2D image of the current frame of the displacement video; subdividing the base grid of the current frame; and The displacement value of the segmentation point is applied to the segmentation point of the subdivided base grid, wherein: The base mesh is generated by thinning out the vertices from an original mesh as an encoding target composed of vertices and connections representing a 3D structure of the object, and the base mesh is a mesh coarser than the original mesh, and The displacement value is the position difference between the segmentation point and the vertex of the original mesh.

13. An information processing device, comprising: a base mesh vertex displacement value deriving unit, which derives base mesh vertex displacement values ​​as displacement values ​​of vertices of the base mesh; a packing unit, which, for each fineness, layers the displacement values ​​of the segmentation points obtained by subdividing the base mesh, and packs the base mesh vertex displacement values ​​and the layered displacement values ​​of the segmentation points in a 2D image; as well as A displacement video encoding unit is used to encode the displacement video in which the 2D image is set as a frame, wherein: The base mesh is generated by thinning out the vertices from an original mesh as an encoding target composed of vertices and connections representing a 3D structure of an object, and the base mesh is a mesh coarser than the original mesh, The displacement value of the segmentation point is the position difference between the segmentation point and the vertex of the original mesh, and The base mesh vertex displacement value is a position difference between the vertex of the base mesh and the vertex of the original mesh.

14. The information processing device according to claim 13, further comprising: A base grid encoding unit, which encodes the base grid of the current frame; as well as A base grid decoding unit is configured to decode the encoded data of the base grid of the current frame, wherein: The base mesh vertex displacement value deriving unit derives, as the base mesh vertex displacement values, displacements of the vertices between the base mesh of the current frame obtained before encoding and the base mesh of the current frame that has been encoded and decoded.

15. The information processing device according to claim 13, wherein: The base mesh vertex displacement value deriving unit derives the displacement of the vertex between the base mesh of a reference frame and the base mesh of a current frame as the base mesh vertex displacement value.

16. An information processing method, comprising: deriving base mesh vertex displacement values ​​as displacement values ​​of vertices of the base mesh; For each fineness, the displacement values ​​of the segmentation points obtained by subdividing the base mesh are layered, and the base mesh vertex displacement values ​​and the layered displacement values ​​of the segmentation points are packed in a 2D image; as well as The 2D image is set as a displacement video of a frame, wherein: The base mesh is generated by thinning out the vertices from an original mesh as an encoding target composed of vertices and connections representing a 3D structure of an object, and the base mesh is a mesh coarser than the original mesh, The displacement value of the segmentation point is the position difference between the segmentation point and the vertex of the original mesh, and The base mesh vertex displacement value is a position difference between the vertex of the base mesh and the vertex of the original mesh.

17. An information processing device, comprising: a displacement video decoding unit that decodes the bit stream and generates a displacement video in which a 2D image is set as a frame, wherein base mesh vertex displacement values ​​that are displacement values ​​of vertices of a base mesh and displacement values ​​of division points obtained by subdividing the base mesh are packed in the 2D image; an unpacking unit, which unpacks the base mesh vertex displacement values ​​and the segmentation point displacement values ​​that have been packed in the 2D image of the current frame of the displacement video; a base mesh vertex position application unit, which applies the base mesh vertex displacement values ​​to the vertices of the base mesh; a subdivision unit that subdivides the base mesh in which the base mesh vertex displacement values ​​are applied to the vertices; and A displacement value application unit, which applies the displacement value of the segmentation point to the segmentation point of the subdivided base grid, wherein: The base mesh is generated by thinning out the vertices from an original mesh as an encoding target composed of vertices and connections representing a 3D structure of an object, and the base mesh is a mesh coarser than the original mesh, The displacement value of the segmentation point is the position difference between the segmentation point and the vertex of the original mesh, and The base mesh vertex displacement value is a position difference between the vertex of the base mesh and the vertex of the original mesh.

18. The information processing device according to claim 17, further comprising: A base grid decoding unit, which decodes the bit stream and generates the base grid of the current frame, wherein The base mesh vertex position application unit applies the base mesh vertex displacement values ​​to the generated vertices of the base mesh of the current frame.

19. The information processing device according to claim 17, wherein: The base mesh vertex position application unit applies the base mesh vertex displacement values ​​to vertices of the base mesh of a reference frame.

20. An information processing method, comprising: decoding the bitstream and generating a displacement video in which a 2D image is set as a frame, wherein base mesh vertex displacement values ​​that are displacement values ​​of vertices of a base mesh and displacement values ​​of segmentation points obtained by subdividing the base mesh are packed in the 2D image; Unpacking the base mesh vertex displacement values ​​and the segmentation point displacement values ​​that have been packed in the 2D image of the current frame of the displacement video; Applying the base mesh vertex displacement values ​​to the vertices of the base mesh; subdividing the base mesh with the base mesh vertex displacement values ​​applied to the vertices; and The displacement value of the segmentation point is applied to the segmentation point of the subdivided base grid, wherein: The base mesh is generated by thinning out the vertices from an original mesh as an encoding target composed of vertices and connections representing a 3D structure of an object, and the base mesh is a mesh coarser than the original mesh, The displacement value of the segmentation point is the position difference between the segmentation point and the vertex of the original mesh, and The base mesh vertex displacement value is a position difference between the vertex of the base mesh and the vertex of the original mesh.

Images