Information processing device and method
The information processing device and method dynamically control the reference mesh for inter encoding and decoding, addressing the inefficiencies in existing 3D mesh encoding methods by allowing flexible adjustment of the processing range, thus enhancing encoding efficiency and reducing submeshing-related losses.
Patent Information
- Application Number
- PCT/JP2024/023279
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-07-10
- Filing Date
- 2024-06-27
- Publication Date
- 2025-01-16
AI Technical Summary
Existing methods for encoding 3D meshes, such as V-DMC, face efficiency issues due to rigid submeshing that limits flexible control over inter encoding and decoding, leading to reduced encoding efficiency.
An information processing device and method that includes a control information encoding unit, a reference mesh updating unit, and inter encoding/decoding units to dynamically control the processing range of inter encoding and decoding by updating the reference mesh based on control information, allowing for flexible adjustment of the encoding and decoding process.
This approach enhances encoding efficiency by allowing for flexible control over the processing range, reducing unnecessary sub-meshing and increasing the applicability of inter coding, thereby maintaining high encoding efficiency without the need for rigid submeshing.
Smart Images

Figure JP2024023279_16012025_PF_FP_ABST
Abstract
Description
Information processing device and method
[0001] The present disclosure relates to an information processing device and method, and more particularly to an information processing device and method that can suppress a decrease in coding efficiency.
[0002] Conventionally, V-DMC (Video-based Dynamic Mesh Coding) has been used as a method for encoding meshes, which are 3D data that represent the three-dimensional structure of an object using vertices and connections (see, for example, Non-Patent Document 1). In V-DMC, the mesh to be encoded is represented by a coarse base mesh and displacement vectors at division points obtained by subdividing the base mesh, and the base mesh and displacement vectors are encoded. The displacement vectors are stored (packed) in two-dimensional images, and the two-dimensional images are used as frames to encode a moving image (displacement video).
[0003] There are two methods for encoding base meshes: intra-coding, which encodes each frame independently, and inter-coding, which encodes by taking advantage of the correlation between frames. Inter-coding has high coding efficiency, but in order to be applicable, the topology (graph structure) of the mesh, i.e., the number of vertices and the connections between the vertices, must be consistent.
[0004] Recently, a method has been proposed in which a frame is divided into multiple sub-meshes (patches) and coded independently (see, for example, Non-Patent Documents 2 and 3). By dividing parts with the same topology into sub-meshes, it has become easier to apply inter-coding.
[0005] Khaled Mammou, Jungsun Kim, Alexis Tourapis, Dimitri Podborski, Krasimir Kolarov, "[V-CG] Apple's Dynamic Mesh Coding CfP Response", ISO / IEC JTC 1 / SC 29 / WG 7 m59281, April 2022Alexis Tourapis, Jungsun Kim, Dimitri Podborski, Khaled Mammou, "Base mesh data substream format "VDMC support in the V3C framework", ISO / IEC JTC 1 / SC 29 / WG 7 m60363, July 2022
[0006] However, because these sub-meshes are set for the entire sequence, it is difficult to flexibly control the range of inter-coding and inter-decoding processing in the time direction, which can lead to a decrease in coding efficiency due to unnecessary sub-meshes or an increase in the number of sub-meshes.
[0007] The present disclosure has been made in light of such circumstances, and makes it possible to suppress a decrease in coding efficiency.
[0008] An information processing device according to one aspect of the present technology is an information processing device that includes a control information encoding unit that encodes control information that controls the processing range of inter-encoding and inter-decoding of a mesh that represents a three-dimensional structure of an object, a reference mesh update unit that updates a reference mesh, which is the mesh that is referenced during the inter-encoding, based on the control information, and an inter-encoding unit that inter-encodes a current mesh, which is the mesh of a current frame, using an updated reference mesh, which is the updated reference mesh.
[0009] An information processing method according to one aspect of the present technology is an information processing method that encodes control information that controls the processing range of inter-encoding and inter-decoding of a mesh that represents a three-dimensional structure of an object, updates a reference mesh, which is the mesh that is referenced during the inter-encoding, based on the control information, and inter-encodes a current mesh, which is the mesh of a current frame, using the updated reference mesh, which is the updated reference mesh.
[0010] Another aspect of the information processing device of the present technology is an information processing device that includes a control information decoding unit that decodes encoded data and generates control information that controls the processing range of inter-encoding and inter-decoding of a mesh that represents the three-dimensional structure of an object, a reference mesh update unit that updates a reference mesh, which is the mesh that is referenced during the inter-decoding, based on the control information, and an inter-decoding unit that inter-decodes the encoded data using the updated reference mesh, which is the updated reference mesh, and generates a current mesh, which is the mesh of a current frame.
[0011] Another aspect of the information processing method of the present technology is an information processing method that decodes encoded data, generates control information that controls the processing range of inter-encoding and inter-decoding of a mesh that represents the three-dimensional structure of an object, updates a reference mesh, which is the mesh that is referenced during the inter-decoding, based on the control information, inter-decodes the encoded data using the updated reference mesh, and generates a current mesh, which is the mesh of a current frame.
[0012] In an information processing device and method according to one aspect of the present technology, control information that controls the processing range of inter-encoding and inter-decoding of a mesh that represents the three-dimensional structure of an object is encoded, and a reference mesh, which is a mesh that is referenced during inter-encoding, is updated based on the control information, and the current mesh, which is a mesh of the current frame, is inter-encoded using the updated reference mesh, which is the updated reference mesh.
[0013] In an information processing device and method according to another aspect of the present technology, encoded data is decoded, control information is generated that controls the processing range of inter-encoding and inter-decoding of a mesh that represents the three-dimensional structure of an object, a reference mesh that is a mesh that is referenced during inter-decoding is updated based on the control information, the encoded data is inter-decoded using the updated reference mesh, and a current mesh that is a mesh of the current frame is generated.
[0014] 1 is a diagram illustrating a mesh. FIG. 1 is a diagram illustrating V-DMC. FIG. 1 is a diagram illustrating an example configuration of a bitstream. FIG. 1 is a diagram illustrating sub-meshing. FIG. 1 is a diagram illustrating an example configuration of a bitstream. FIG. 2 is a diagram illustrating an example of a mesh encoding / decoding method. A block diagram showing an example of the main configuration of an encoding device. A diagram showing an example of a method of specifying a control target. A diagram showing an example of a method of specifying a control target. A diagram showing an example of a method of specifying a control target. A diagram showing an example of an update of a mesh. A diagram showing an example of a mesh update interval. A diagram showing an example configuration of a bitstream. A flowchart illustrating an example of the flow of an encoding process. A block diagram showing an example of the main configuration of a decoding device. A flowchart illustrating an example of the flow of a decoding process. A block diagram showing an example of the main configuration of an inter encoding unit. A flowchart illustrating an example of the flow of inter encoding process. A block diagram showing an example of the main configuration of an inter decoding unit. A flowchart illustrating an example of the flow of inter decoding process. A diagram illustrating an application example of the present technology. A diagram illustrating an application example of the present technology. A diagram illustrating an application example of the present technology. A diagram illustrating an application example of the present technology. A block diagram showing an example of the main configuration of a computer.
[0015] Hereinafter, modes for carrying out the present disclosure (hereinafter referred to as embodiments) will be described. The description will be made in the following order: 1. Literature supporting technical content and technical terminology 2. Inter-encoding of meshes 3. Updating of reference meshes 4. Application examples of the present technology 5. Supplementary notes
[0016] <1. Literature, etc. supporting technical content and technical terminology> The scope of disclosure of the present technology includes not only the content described in the embodiments, but also the content described in the following non-patent documents, etc. that were publicly known at the time of filing, and the content of other documents referenced in the following non-patent documents.
[0017] Non-patent document 1: (described above) Non-patent document 2: (described above) Non-patent document 3: (described above)
[0018] In other words, the contents of the above-mentioned non-patent documents and the contents of other documents referenced in the above-mentioned non-patent documents are also used as the basis for determining the support requirements.
[0019] <2. Inter-coding of Meshes> <V-DMC> Conventionally, 3D data representing the three-dimensional structure of a three-dimensional structure (an object with a three-dimensional shape) has been available as a mesh, which expresses the three-dimensional shape of the object surface by forming polygons with vertices and connections.
[0020] As shown in the upper left of Figure 1, in a mesh, vertices 11 and connections 12 connecting these vertices 11 form polygonal planes (polygons). These polygons (also called faces) represent the surface of a three-dimensional object, i.e., the three-dimensional shape of the object. A texture 13 can be applied to each face of this mesh.
[0021] Mesh data is composed of information such as that shown in the lower part of Figure 1. Vertex information 14, shown first from the left in the lower part of Figure 1, is information indicating the three-dimensional position (three-dimensional coordinates (X, Y, Z)) of each vertex 11 that constitutes the mesh. Connection information 15, shown second from the left in the lower part of Figure 1, is information indicating each connection 12 that constitutes the mesh. A texture image 16, shown third from the left in the lower part of Figure 1, is map information for the texture 13 that is applied to each face. A UV map 17, shown fourth from the left in the lower part of Figure 1, is information indicating the correspondence between the vertices 11 and the texture 13. The UV map 17 indicates the coordinates (UV coordinates) of each vertex 11 in the texture image 16.
[0022] As an example of such a mesh coding method, there is V-DMC (Video-based Dynamic Mesh Coding) as disclosed in Non-Patent Document 1.
[0023] In V-DMC, a mesh to be coded is represented by a coarse base mesh and displacement vectors at division points obtained by subdividing the base mesh, and the base mesh and displacement vectors are coded.
[0024] For example, suppose there is an original mesh as shown in the top row of Figure 2. In Figure 2, black dots represent vertices, and lines connecting the black dots represent connections. As mentioned above, a mesh essentially forms polygons by vertices and connections, but for the sake of convenience, it is explained here as a group of vertices connected linearly (in series).
[0025] By decimating some of the vertices of the original mesh, a coarse mesh is formed as shown in the second row from the top of Figure 2. This is called the base mesh.
[0026] By subdividing each mesh of this base mesh, vertices are added, as shown in the third row from the top of Figure 2. Here, this subdivision adds vertices equal to the number of vertices thinned out from the original mesh. This results in a mesh with the same number of vertices as the original mesh. In this specification, these added vertices are also referred to as division points.
[0027] However, because the connections are updated when the vertices of the original mesh are thinned out, and the division points are formed on these updated connections, the shape of this subdivided base mesh differs from the shape of the original mesh. More specifically, as shown in the bottom part of Figure 2, the positions of the division points (on the dotted lines) differ from those of the original mesh. In this specification, the difference between the positions of these division points and the positions of the vertices of the original mesh is referred to as a displacement vector. A displacement vector is set for each vertex of the subdivided base mesh.
[0028] Since the original mesh is known in the encoder, it can generate a base mesh and derive such a displacement vector. The decoder can generate (reconstruct) the original mesh (or a mesh equivalent to it) by subdividing the base mesh and applying the displacement vector to each vertex.
[0029] In this way, by reducing the number of vertices of the original mesh and encoding it as a base mesh, the amount of code can be reduced. Also, by storing the displacement vector in a two-dimensional image and encoding it using 2D encoding, the coding efficiency can be improved. In this specification, storing data in a two-dimensional image is also referred to as packing.
[0030] In this specification, being variable in the time direction is also referred to as "dynamic." Being unchanging in the time direction or having no concept of time direction is also referred to as "static." For example, a mesh representing the three-dimensional shape of an object at one time can be said to be static. When a sequence consists of multiple meshes representing the three-dimensional shapes of an object at different times, the meshes in the sequence can be said to be dynamic. In other words, a dynamic mesh is made up of static meshes at multiple times. In this specification, information at each time is also referred to as a frame. In other words, a dynamic mesh is made up of meshes of multiple frames.
[0031] In this specification, the three-dimensional shape of an object is dynamic, and the mesh, which is the 3D data of the object, is also dynamic. In other words, the object to be coded is a dynamic mesh. In other words, the base mesh and displacement vectors are also dynamic, and the displacement vectors are coded as a moving image with the two-dimensional images as frames. In this specification, this moving image is also referred to as a displacement video.
[0032] The displacement vectors may be packed into a two-dimensional image as transform coefficients by coefficient transformation such as wavelet transform. The displacement vectors may also be quantized. For example, the displacement vectors may be transformed into transform coefficients by wavelet transform, the transform coefficients may be quantized, and the quantized transform coefficients (quantized coefficients) may be packed.
[0033] <Base Mesh Coding> Base mesh coding methods include intra-coding, which encodes each frame independently, and inter-coding, which encodes using the correlation between frames. Generally, inter-coding has higher coding efficiency than intra-coding. However, to apply inter-coding, the mesh topology (graph structure), i.e., the number of vertices and the connections between vertices, must match between the current frame and the reference frame. It is difficult to maintain the mesh topology (hereinafter simply referred to as topology) of the entire frame consistent over a long period of time. Therefore, inter-coding may not be applied, resulting in reduced coding efficiency. Even if inter-coding were applied, inter-prediction may be completed in a short period of time, as in the bitstream 31 shown in FIG. 3, resulting in reduced coding efficiency.
[0034] <Sub-Meshing> Recently, methods have been proposed in which one frame is divided into multiple sub-meshes (patches) and the sub-meshes are coded independently of each other, as disclosed in, for example, Non-Patent Document 2 and Non-Patent Document 3. For example, as shown in FIG. 4, a mesh 41 is divided into sub-meshes 42 and 43, and each of the sub-meshes is coded independently of each other. In this way, a coding method can be selected for each sub-mesh. Therefore, a coding method that is more suitable for the shape characteristics of the mesh can be selected.
[0035] For example, assume that there is a base mesh 50 representing a person's face, as shown on the left side of Figure 5 . This base mesh 50 includes a head 51, which has a relatively simple three-dimensional shape, and a hair 52, which has a relatively complex three-dimensional shape. Generally, if the correlation between frames is sufficiently high, inter-coding tends to achieve higher coding efficiency than intra-coding. Because the head 51 has a relatively simple three-dimensional shape, it is likely to have high correlation between frames, making it suitable for inter-coding. In contrast, because the hair 52 has a relatively complex three-dimensional shape, it is likely to have low correlation between frames, making it unsuitable for inter-coding. Therefore, when encoding this base mesh 50, the influence of the hair 52 reduces the correlation between frames, making it more likely that intra-coding will be applied, resulting in a risk of reduced coding efficiency. Furthermore, even if inter-coding were applied, it would not last long, resulting in a switch between inter-coding and intra-coding every few frames, potentially reducing coding efficiency.
[0036] In such a case, as shown on the right side of Figure 5, by sub-meshing the head 51 and hair 52 and encoding them independently, it is possible to stably select inter-coding for the head 51 and suppress a decrease in encoding efficiency.
[0037] 6, submesh A and submesh B are coded independently of each other, so that bitstreams such as bitstream 61 and bitstream 62 can be generated independently of each other. Therefore, coding methods can be selected independently of each other. Therefore, for example, by submeshing parts with long-term topology matching, long-term inter-coding can be realized, and a decrease in coding efficiency can be suppressed.
[0038] However, since this submesh is set for the entire sequence, it is difficult to flexibly control the processing range of inter-encoding and inter-decoding in the time direction. For example, in the case of Figure 6, the configuration of submesh A and submesh B is maintained throughout the entire sequence. In other words, both the bit stream 61 and the bit stream 62 of each submesh are bit streams for each sequence.
[0039] Therefore, for example, it was not possible to divide regions only in a certain section (a certain frame section). Even in such a case, it was necessary to form submeshes throughout the sequence. Furthermore, it was not possible to change the portions to be submeshes within the sequence. For example, if it was desired to change the mesh division position within the sequence, it was necessary to submesh each division pattern. Therefore, there was a risk of a decrease in coding efficiency due to submeshes in unnecessary sections, an increase in the number of submeshes, the occurrence of overlapping regions, etc. Furthermore, since there is no submesh that can be referenced at the beginning of a submesh, intra prediction was required, which could also decrease coding efficiency.
[0040] <3. Updating the Reference Mesh> <Method 1> Therefore, as shown in the top row of the table in Figure 7, the reference mesh is updated based on control information that controls the processing range of the dynamic mesh inter-coding and inter-decoding, and the updated reference mesh is applied to the inter-coding and inter-decoding of the current mesh (Method 1). Note that in this specification, the current mesh refers to the mesh of the current frame (i.e., the mesh to be encoded). Also, the reference mesh refers to the mesh of another frame (reference frame) that is referenced in the inter-coding and inter-decoding of the current mesh. Also, the reference mesh updated based on this control information is also referred to as the updated reference mesh.
[0041] <Encoding device> Fig. 8 is a block diagram showing an example of the configuration of an encoding device, which is one aspect of an information processing device to which the present technology (the above-mentioned method 1) is applied. The encoding device 100 shown in Fig. 8 is a device that encodes a dynamic mesh. The encoding device 100 acquires a current mesh of the dynamic mesh, encodes it, generates a bitstream, and outputs it.
[0042] Note that Fig. 8 shows the main processing units, data flows, etc., and is not necessarily all that is shown in Fig. 8. In other words, in the encoding device 100, there may be processing units that are not shown as blocks in Fig. 8, and there may be processing and data flows that are not shown as arrows, etc. in Fig. 8.
[0043] As shown in FIG. 8, the encoding device 100 has a control unit 101, an encoding method selection unit 111, a control information supply unit 112, a control information encoding unit 113, a reference mesh update unit 114, an inter-encoding unit 115, an intra-encoding unit 116, and a multiplexing unit 117.
[0044] The control unit 101 controls each processing unit (from the coding method selection unit 111 to the multiplexing unit 117) in the encoding device 100. In other words, the coding method selection unit 111 to the multiplexing unit 117 are appropriately controlled by the control unit 101 to execute processing.
[0045] The coding method selection unit 111 performs processing related to the selection of a mesh coding method. For example, the coding method selection unit 111 acquires a current mesh (i.e., a target for coding) of dynamic meshes supplied from outside the coding device 100. The coding method selection unit 111 selects a coding method for the current mesh using a predetermined method. That is, the coding method selection unit 111 selects whether the current mesh is to be inter-coded or intra-coded. Note that any method may be used for this selection. If inter-coding is selected, the coding method selection unit 111 supplies the current mesh to the inter-coding unit 115. If intra-coding is selected, the coding method selection unit 111 supplies the current mesh to the intra-coding unit 116.
[0046] The control information supply unit 112 performs processing related to the supply of control information. This control information, which will be described in detail later, is information that controls the processing range of inter-encoding and inter-decoding of meshes. This control information may be generated or supplied from another device. If the control information is generated, the control information supply unit 112 may also be referred to as a control information generation unit. Furthermore, if the control information is acquired from outside the encoding device 100, the control information supply unit 112 may also be referred to as a control information acquisition unit. The control information may be generated by any method. For example, the control information may be generated manually by a user (e.g., a mesh creator), based on the mesh to be encoded, based on the content of the content (such as the occurrence status of a scene or action), based on the load status of the encoding device 100, or based on conditions outside the encoding device 100, such as the load of the playback process or the bandwidth of the network used as the transmission path. Note that the control information is associated with any frame of the dynamic mesh to be encoded. In other words, control information may be provided for any frame of the dynamic mesh. The control information supply unit 112 supplies the control information to the control information encoding unit 113 and the reference mesh updating unit 114 at an appropriate timing.
[0047] The control information encoder 113 performs processing related to encoding of control information. For example, the control information encoder 113 encodes the control information supplied from the control information supply unit 112. The control information encoder 113 encodes the control information to generate encoded control information data. Any encoding method may be used. The control information encoder 113 supplies the generated encoded control information data to the multiplexer 117.
[0048] The reference mesh update unit 114 performs processing related to updating the reference mesh. For example, when inter-coding is selected by the coding method selection unit 111, the reference mesh update unit 114 acquires and stores control information provided by the control information supply unit 112. The control information supply unit 112 supplies control information corresponding to the current mesh (current frame). In other words, the control information supply unit 112 sets the control information corresponding to the current mesh in the reference mesh update unit 114. The reference mesh update unit 114 also acquires a reference mesh corresponding to the current mesh. In other words, the reference mesh update unit 114 acquires a mesh of a reference frame referenced by the current frame in inter-coding. This reference mesh may be acquired from anywhere. For example, it may be a current mesh acquired from outside the coding device 100 (i.e., a mesh previously input to the coding device 100) or a locally decoded mesh previously coded. The reference mesh update unit 114 updates the reference mesh based on the control information. The reference mesh update unit 114 supplies the updated reference mesh to the inter-encoding unit 115 .
[0049] The inter-encoding unit 115 performs processing related to inter-encoding. For example, the inter-encoding unit 115 obtains a current mesh supplied from the encoding method selection unit 111. The inter-encoding unit 115 also obtains an updated reference mesh supplied from the reference mesh update unit 114. The inter-encoding unit 115 uses the updated reference mesh to inter-encode the current mesh. In this case, the inter-encoding unit 115 inter-encodes only a portion of the current mesh that corresponds to the updated reference mesh (has the same topology). For example, if the updated reference mesh corresponds to the entire current mesh, the inter-encoding unit 115 inter-encodes the entire current mesh. If the updated reference mesh corresponds to a portion of the current mesh, the inter-encoding unit 115 inter-encodes that portion of the current mesh. In this case, the remaining portion of the current mesh is coded separately. For example, the inter-encoding unit 115 may inter-encode the remaining portion using a reference mesh other than the updated reference mesh. If the remaining portion is to be intra-encoded, the inter-encoding unit 115 may supply the remaining portion to the intra-encoding unit 116. The inter-encoding unit 115 supplies the inter-encoded data, which is encoded data generated by inter-encoding (all or part of) the current mesh, to the multiplexing unit 117 .
[0050] The intra-encoding unit 116 performs processing related to intra-encoding. For example, the intra-encoding unit 116 obtains the current mesh supplied from the encoding method selection unit 111. The intra-encoding unit 116 intra-encodes the current mesh. The intra-encoding unit 116 may also obtain a mesh (a part of the current mesh) supplied from the inter-encoding unit 115 and intra-encode it. Note that any method may be used for intra-encoding the mesh. The intra-encoding unit 116 supplies intra-encoded data, which is encoded data generated by intra-encoding the current mesh (part or all), to the multiplexing unit 117.
[0051] The multiplexing unit 117 performs processing related to bitstream generation. For example, the multiplexing unit 117 acquires control information coded data supplied from the control information coding unit 113. The multiplexing unit 117 acquires inter-coded data supplied from the inter-coder 115. The multiplexing unit 117 acquires intra-coded data supplied from the intra-coder 116. The multiplexing unit 117 multiplexes the acquired control information coded data, inter-coded data, and intra-coded data to generate a bitstream. The multiplexing unit 117 outputs the generated bitstream to an external device from the coding device 100. This bitstream may be stored in an external storage device or transmitted to a decoder or the like via any transmission path such as a network or a computer.
[0052] The present technology (the above-described method 1) may be applied to the encoding device 100 configured as described above. For example, the control information encoding unit 113 of the encoding device 100 may encode control information that controls the processing range of inter-encoding and inter-decoding of a mesh that represents the three-dimensional structure of an object, the reference mesh update unit 114 may update the reference mesh that is a mesh that is referenced during inter-encoding based on the control information, and the inter encoding unit 115 may inter-encode the current mesh that is a mesh of the current frame using the updated reference mesh that is the updated reference mesh.
[0053] With this configuration, the encoding device 100 can perform inter-coding on a part of the current mesh without sub-meshing, thereby preventing a decrease in encoding efficiency.
[0054] In this case, the reference mesh update unit 114 may update the reference mesh in an update interval specified by the control information. For example, the reference mesh update unit 114 may set a frame associated with the control information as the start frame of the update interval for updating the reference mesh, and update the reference mesh in that update interval. As described above, the control information can be associated with any frame of the dynamic mesh. Therefore, this update interval can start from any frame. In this way, the reference mesh update unit 114 can update the reference mesh in a partial interval within the sequence. In other words, the inter-encoding unit 115 can inter-encode a portion of the current mesh in that partial interval. Therefore, the encoding device 100 can suppress a decrease in encoding efficiency.
[0055] The reference mesh may be updated by any process. For example, it may be the deletion of a face from the reference mesh. For example, the control information may include information specifying a deletion target face, which is a face to be deleted. The reference mesh update unit 114 may delete the deletion target face specified by the control information from the reference mesh during the update interval. In this way, the reference mesh update unit 114 can reduce the number of faces in the reference mesh.
[0056] The faces to be deleted may be specified in any manner. For example, the control information may include information indicating a reference vertex and the number of connections from the reference vertex to specify the faces to be deleted. For example, suppose that the control information for the mesh 201 shown in FIG. 9 specifies P10 as the reference point and the number of connections as 1. In this case, the reference mesh update unit 114 interprets that P7, P8, P9, and P10, indicated by white dots in the mesh 201, are specified as target vertices, and the faces corresponding to these target vertices are specified as faces to be deleted. Also, suppose that the control information for the mesh 202 shown in FIG. 9 specifies P10 as the reference point and the number of connections as 2. In this case, the reference mesh update unit 114 interprets that P4, P5, P6, P7, P8, P9, and P10, indicated by white dots in the mesh 201, are specified as target vertices, and the faces corresponding to these target vertices are specified as faces to be deleted. In this way, by using the (reference) vertex and the number of connections, faces to be deleted can be specified according to the structure of the mesh. For example, vertices that are physically close but structurally far from each other (the number of connections between the reference vertex and the vertex is large) may have little relevance to each other in the object (belonging to faces that are far from each other). Such vertices can be excluded from the target vertices.
[0057] Note that the correspondence between the faces to be deleted and the target vertices may be any relationship. For example, a face whose vertices are all made up of target vertices may be a face to be deleted. That is, the reference mesh update unit 114 may determine a vertex connected to a reference vertex specified by control information within the connectivity specified by the control information as a target vertex, and a face consisting only of that target vertex as a face to be deleted. For example, suppose that the control information for the mesh 203 shown in FIG. 10 specifies P10 as the reference point and 1 as the connectivity. In this case, the reference mesh update unit 114 determines P7, P8, P9, and P10, indicated by white dots, as target vertices, and determines the faces surrounded by the connections connecting target vertices P7, P8, and P10, and the faces surrounded by the connections connecting target vertices P8, P9, and P10, as faces to be deleted. That is, the faces indicated by gray in the mesh 203 are determined as faces to be deleted.
[0058] Alternatively, a face having one or more vertices formed by the target vertex may be a face to be deleted. In other words, the reference mesh update unit 114 may set a vertex connected to a reference vertex specified by the control information within the connectivity specified by the control information as a target vertex, and set a face formed by vertices including the target vertex as a face to be deleted. For example, suppose that the control information for the mesh 204 shown in Figure 10 specifies the reference point as P10 and the connectivity as 1. In this case, the reference mesh update unit 114 sets the target vertices to P7, P8, P9, and P10, indicated by white dots, and sets the following faces to be deleted: faces surrounded by connections connecting target vertices P7, P8, and P10; faces surrounded by connections connecting target vertices P8, P9, and P10; faces surrounded by connections connecting target vertices P5, P7, and P8; faces surrounded by connections connecting target vertices P6, P8, and P9; faces surrounded by connections connecting target vertices P4, P5, and P7; and faces surrounded by connections connecting target vertices P5, P6, and P8. In other words, the faces indicated by gray in the mesh 204 are set as faces to be deleted.
[0059] Furthermore, the control information may include information indicating a three-dimensional spatial region as information specifying faces to be deleted. For example, the control information may indicate a rectangular region 211 shown in FIG. 11 as information specifying faces to be deleted. Vertices within this rectangular region 211 are specified as target vertices. Any method for specifying this rectangular region 211 may be used. For example, this rectangular region 211 may be specified using the coordinates of two diagonal vertices (e.g., vertices Q1 and Q2). Alternatively, this rectangular region 211 may be specified using the coordinates of vertex Q1 and the lengths of the three sides (X, Y, Z) of the rectangular region 211. Of course, other methods are also acceptable. The shape of the three-dimensional spatial region specifying faces to be deleted is arbitrary and may be other than the rectangular shape shown in FIG. 11 . For example, it may be spherical, cylindrical, or another shape. By specifying target vertices using a three-dimensional spatial region in this manner, target vertices can be specified based on their physical distance, regardless of the mesh configuration. Therefore, for example, a group of vertices whose distance (connectivity) from a reference vertex in a mesh structure is not constant can be specified as target vertices.
[0060] When specifying faces to be deleted (target vertices) using a three-dimensional spatial region in this manner, as in the case of using the reference vertex and connectivity number described above, any relationship between the faces to be deleted and the target vertices may be used. For example, a face whose vertices are all made up of target vertices may be the face to be deleted. That is, the reference mesh update unit 114 may designate vertices located within a three-dimensional spatial region designated by control information as target vertices, and designate faces consisting only of those target vertices as faces to be deleted. For example, suppose that the rectangular region 221 shown in FIG. 11 is designated by control information. In this case, the reference mesh update unit 114 designates the vertices within the rectangular region 221 as target vertices, and designates face 222, surrounded by the connections connecting those target vertices, as the face to be deleted. In this case, face 223 is not included in the faces to be deleted.
[0061] Alternatively, a face having one or more vertices constituted by the target vertex may be a face to be deleted. That is, the reference mesh update unit 114 may determine a vertex located within a three-dimensional spatial region specified by control information as a target vertex, and determine a face constituted by vertices including the target vertex as a face to be deleted. For example, assume that the rectangular region 211 shown in FIG. 11 is specified by control information. In this case, the reference mesh update unit 114 determines a vertex within the rectangular region 211 as a target vertex, and determines faces 212 and 213 surrounded by connections connecting vertices including the target vertex as faces to be deleted.
[0062] When a face to be deleted as described above is set, the reference mesh update unit 114 updates the reference mesh by deleting the face to be deleted. For example, when a face shown in gray in the mesh 203 in Fig. 10 is set as a face to be deleted, the reference mesh update unit 114 deletes the face to be deleted and generates an updated reference mesh such as the mesh 231 shown in Fig. 12, which is made up of vertices P1 to P9 and the connections between them.
[0063] The inter-encoding unit 115 performs inter-encoding of the current mesh using the updated reference mesh. For example, assume that the current mesh is a mesh consisting of vertices P1' to P10', such as mesh 232 shown in FIG. 12 . If vertices P1' to P9' of mesh 232 correspond to vertices P1 to P9 of mesh 231, respectively, the topology of mesh 231 matches that of the portion formed by vertices P1' to P9' of mesh 232 and the connections between them. Therefore, the inter-encoding unit 115 uses the updated reference (mesh 231) to inter-encode the portion formed by vertices P1' to P9' of mesh 232 and the connections between them. In this way, even if the topology of the entire mesh does not match, the encoding device 100 can easily extract portions with matching topologies between frames by updating the reference mesh (deleting some faces), making inter-encoding easier. This reduces the reduction in encoding efficiency.
[0064] Note that updating the reference mesh is not limited to deleting faces. For example, it may involve adding faces. For example, information specifying a target face to be added, which is a face to be added, may be included in the control information. The reference mesh update unit 114 may add the target face to be added specified by the control information to the reference mesh during the update interval. For example, assume that the reference mesh 241 shown in FIG. 13 is composed of vertices P1 to P10 and the connections between them. Faces shown in gray may be added to this reference mesh 241. In this way, the reference mesh update unit 114 can increase the number of faces in the reference mesh.
[0065] Any method may be used to specify the face to be added. For example, the control information may include information indicating an additional vertex, which is a vertex to be added, and information indicating an additional connection, which is a connection to be added, as information specifying the face to be added. In the example of the reference mesh 241 in FIG. 13 , the control information may include information indicating P11 as the additional vertex, and information indicating the connection (connection) between vertex P7 and vertex P11 and the connection (connection) between vertex P10 and vertex P11 as the additional connections.
[0066] The reference mesh may also be updated by deleting and adding faces. For example, the control information may include information specifying faces to be deleted, which are faces to be deleted, and information specifying faces to be added, which are faces to be added. The reference mesh update unit 114 may delete faces to be deleted specified by the control information from the reference mesh during the update interval described above, and may add faces to be added specified by the control information to the reference mesh. The method for specifying faces to be deleted is as described above. The method for specifying faces to be added is as described above. In other words, the reference mesh may be updated by combining the above-described deletion and addition. For example, as in the reference mesh 242 of FIG. 13, a face including vertex P10 may be deleted and a new face including vertex P11 may be added. In this way, the reference mesh update unit 114 can change the faces of the reference mesh.
[0067] The control information may also specify an end frame of the update interval. As described above, the start frame of the update interval is specified as the frame to which the control information belongs. Therefore, it can be said that the length of the update interval is specified by specifying the end frame in the control information. Furthermore, since the start frame and end frame are specified, it can also be said that the update interval is specified from which point in the sequence to which frame interval (which frame interval) (i.e., the update interval itself or the position of the update interval) is specified. The reference mesh update unit 114 specifies the update interval from the frame associated with the control information to the end frame specified by the control information, and updates the reference mesh in the update interval as described above. Therefore, as shown in FIG. 14, any face or vertex of the reference mesh can be updated only in any frame interval.
[0068] For example, as shown in the bitstream 251 in Fig. 15, even if mesh A is not divided into sub-meshes, it is possible to easily exclude a portion in any frame from inter-coding (at least to avoid referencing the same reference mesh). Also, in any frame, it is possible to return the reference destination of that portion to the original reference mesh.
[0069] That is, it is possible to update only the necessary portions of the reference mesh both temporally and spatially. In other words, it is possible to perform more flexible control of the reference mesh. By controlling the reference mesh in this way, it is possible to increase the number of frames that refer to a common reference mesh (the number of frames with matching topology can be increased), making it easier to apply inter-coding. Furthermore, since sub-meshing is not performed, there is no need to perform intra-coding in unnecessarily narrow areas, and it is possible to suppress a decrease in the coding efficiency of intra-coding.
[0070] In addition, like the bit stream 252 of submesh B (meaning a partial region, not actually submeshed) and the bit stream 253 of submesh C (similar to submesh B), there is no need to align their sequences with the sequence of mesh A, and it is sufficient to generate a bit stream of only the necessary parts, so it is possible to suppress a decrease in coding efficiency. Also, even if submeshing is not performed, it is possible to omit decoding of the area of mesh A corresponding to this submesh B (only the information to be reproduced needs to be decoded), so it is possible to suppress an increase in the load of the reproduction process.
[0071] In addition, as shown in Figure 15, during the inter-coding section (without the need to end the inter-coding of mesh A and perform intra-coding), such control (changing the coding method of some areas) can be performed. Therefore, it is possible to suppress the reduction of coding efficiency.
[0072] In the above, face deletion and addition (or a combination thereof) have been shown as examples of updating the reference mesh, but any updating method may be used and is not limited to these examples. For example, a face may be rotated or flipped. A face may be enlarged or reduced. A face may be moved or deformed. A face may be subdivided, or multiple faces may be combined into a single face. Furthermore, multiple updating methods may be applied in combination. Vertices and connections may be updated to achieve such face updates.
[0073] As described above, the inter-encoding unit 115 performs inter-encoding of the current mesh using the updated reference mesh. In this case, inter-encoding is mesh encoding, so the inter-encoding unit 115 may derive motion vectors of corresponding vertices between frames and encode the motion vectors. In other words, the inter-encoding unit 115 may derive motion vectors of corresponding vertices between the updated reference mesh and the current mesh and encode the motion vectors. In other words, the motion vectors indicate how the vertices have moved from the reference frame to the current frame. The motion vectors are derived for each vertex. Any encoding method may be used to encode the motion vectors.
[0074] The inter-encoding unit 115 inter-encodes the portion of the current mesh that corresponds to the updated reference mesh. Therefore, if the updated reference mesh corresponds to a portion of the current mesh, a portion of the current mesh that does not correspond to the updated reference mesh (a non-corresponding portion) will be generated. This non-corresponding portion will not be subject to inter-encoding for the portion that corresponds to the updated reference mesh. Therefore, the encoding device 100 may further include a non-corresponding portion encoding unit that encodes the portion of the current mesh that does not correspond to the updated reference mesh. In this way, the encoding device 100 can encode the entire current mesh. Therefore, a decrease in encoding efficiency can be suppressed.
[0075] Note that this non-corresponding portion may be intra-coded. That is, the non-corresponding portion encoding unit may intra-code the portion of the current mesh that does not correspond to the updated reference mesh. In other words, the intra-encoding unit 116 may be the non-corresponding portion encoding unit.
[0076] In addition, this non-corresponding portion may be inter-coded. That is, the non-corresponding portion coding unit may inter-code the portion of the current mesh that does not correspond to the updated reference mesh. In other words, the inter-coding unit 115 may be a non-corresponding portion coding unit, and may inter-code the non-corresponding portion independently of the corresponding portion.
[0077] That is, in the example of Figure 15, the first frames of bit stream 252 and bit stream 253 are intra-coded, but they may also be inter-coded. In that case, the mesh of any frame decoded before mesh A may be used as the reference mesh. In the example of Figure 15, bit stream 252 and bit stream 253 are shown separately from bit stream 251, but they may also be multiplexed into a single bit stream.
[0078] <Flow of Encoding Process> An example of the flow of encoding process executed by the encoding device 100 will be described with reference to the flowchart of FIG.
[0079] When the encoding process starts, the encoding method selection unit 111 selects the encoding method for the current frame (that is, whether to use intra-encoding or inter-encoding) in step S101.
[0080] In step S102, the control unit 101 determines whether or not to perform inter-coding on the current mesh. If it is determined that inter-coding is to be performed, the process proceeds to step S103.
[0081] In step S103, the control information supply unit 112 sets the control information associated with the current frame.
[0082] In step S104, the control information encoding unit 113 encodes the control information associated with the current frame.
[0083] In step S105, the reference mesh update unit 114 updates the reference mesh with update content corresponding to the current mesh based on the control information.
[0084] In step S106, the inter-encoding unit 115 performs inter-encoding on the current mesh using the updated reference mesh.
[0085] In step S107, the inter-encoding unit 115 or the intra-encoding unit 116 encodes the remaining faces of the current mesh.
[0086] When the process of step S107 ends, the process proceeds to step S109. If it is determined in step S102 that inter-coding is not to be performed (intra-coding is to be performed), the process proceeds to step S108.
[0087] In step S108, the intra-encoding unit 116 intra-encodes the current mesh. When the process of step S108 ends, the process proceeds to step S109.
[0088] In step S109, the multiplexing unit 117 multiplexes the data to generate a bit stream. In step S110, the multiplexing unit 117 outputs the bit stream. When the process of step S110 ends, the encoding process ends.
[0089] The present technology (method 1 described above) may be applied to such encoding processing. For example, in step S104, the control information encoding unit 113 encodes control information that controls the processing range of inter-encoding and inter-decoding of a mesh representing a three-dimensional structure of an object. In step S105, the reference mesh update unit 114 updates a reference mesh, which is a mesh referenced during inter-encoding, based on the control information. In step S106, the inter encoding unit 115 inter-encodes a current mesh, which is a mesh of the current frame, using the updated reference mesh, which is the updated reference mesh.
[0090] By performing each process in this manner, the encoding device 100 can inter-code a part of the current mesh without sub-meshing it. Therefore, the encoding device 100 can suppress a decrease in encoding efficiency.
[0091] <Decoding device> Fig. 17 is a block diagram showing an example of the configuration of a decoding device, which is one aspect of an information processing device to which the present technology (the above-described method 1) is applied. The decoding device 300 shown in Fig. 17 is a device that decodes coded data (bit stream) and generates (reconstructs) a dynamic mesh. The decoding device 300 acquires the bit stream, decodes it, generates a dynamic mesh, and outputs it.
[0092] Note that Fig. 17 shows the main processing units, data flows, etc., and does not necessarily show everything. In other words, in the decoding device 300, there may be processing units that are not shown as blocks in Fig. 17, and there may be processing and data flows that are not shown as arrows, etc. in Fig. 17.
[0093] As shown in FIG. 17, the decoding device 300 includes a control unit 301 , a demultiplexing unit 311 , a control information decoding unit 312 , a reference mesh updating unit 313 , an inter decoding unit 314 , an intra decoding unit 315 , and a synthesizing unit 316 .
[0094] The control unit 301 controls each processing unit (demultiplexing unit 311 to combining unit 316) in the decoding device 300. In other words, the demultiplexing unit 311 to combining unit 316 are appropriately controlled by the control unit 301 to perform processing.
[0095] The demultiplexing unit 311 performs processing related to demultiplexing. For example, the demultiplexing unit 311 acquires a bitstream supplied from outside the decoding device 300. This bitstream is a bitstream generated by the encoding device 100 and is obtained by encoding dynamic meshes. The demultiplexing unit 311 demultiplexes this bitstream to extract control information encoded data, inter-encoded data, and intra-encoded data. This demultiplexing method may be any method that corresponds to the multiplexing method applied by the multiplexing unit 117 of the encoding device 100. The demultiplexing unit 311 supplies the control information encoded data to the control information decoding unit 312. The demultiplexing unit 311 supplies the inter-encoded data to the inter decoding unit 314. The demultiplexing unit 311 supplies the intra-encoded data to the intra decoding unit 315.
[0096] The control information decoding unit 312 performs processing related to the decoding of the control information coded data. For example, the control information decoding unit 312 acquires the control information coded data supplied from the demultiplexing unit 311. The control information decoding unit 312 decodes the control information coded data to generate (restore) control information. This decoding method may be any method as long as it corresponds to the method applied by the control information coding unit 113 of the coding device 100 to code the control information. The control information decoding unit 312 supplies the generated control information to the reference mesh updating unit 313.
[0097] The reference mesh update unit 313 is a processing unit similar to the reference mesh update unit 114 of the encoding device 100, and performs processing related to updating the reference mesh. For example, the reference mesh update unit 313 acquires control information supplied from the control information decoding unit 312. The control information supplied from the control information decoding unit 312 corresponds to the current mesh (current frame). In other words, it can be said that the control information decoding unit 312 sets the control information corresponding to the current mesh in the reference mesh update unit 313. The reference mesh update unit 313 also acquires a reference mesh corresponding to the current mesh. In other words, the reference mesh update unit 313 acquires a mesh of a reference frame referenced by the current frame in inter-coding. This reference mesh may be acquired from anywhere. For example, it may be a mesh decoded in the past. The reference mesh update unit 313 updates the reference mesh based on the control information. The reference mesh update unit 313 supplies the updated reference mesh to the inter-decoding unit 314.
[0098] The inter decoding unit 314 performs processing related to inter decoding. For example, the inter decoding unit 314 obtains inter coded data supplied from the demultiplexing unit 311. The inter decoding unit 314 also obtains an updated reference mesh supplied from the reference mesh update unit 313. The inter decoding unit 314 uses the updated reference mesh to inter decode the inter coded data and generate (reconstruct) a current mesh. However, the inter decoding unit 314 generates (reconstructs) only a portion of the current mesh that corresponds to the updated reference mesh (has the same topology). For example, if the updated reference mesh corresponds to the entire current mesh, the inter decoding unit 314 generates (reconstructs) the entire current mesh by inter decoding the inter coded data. If the updated reference mesh corresponds to a portion of the current mesh, the inter decoding unit 314 generates (reconstructs) that portion of the current mesh by inter decoding the inter coded data. In this case, the remaining portion of the current mesh is decoded separately. For example, the inter decoding unit 314 may use a reference mesh other than the updated reference mesh to inter decode the remaining portion. The remaining part may be intra-decoded by the intra-decoding unit 315. The inter-decoding unit 314 decodes the inter-encoded data and generates (reconstructs) the current mesh (part or all of it), and supplies this to the synthesis unit 316.
[0099] The intra-decoding unit 315 performs processing related to intra-decoding. For example, the intra-decoding unit 315 obtains intra-coded data supplied from the demultiplexing unit 311. The intra-decoding unit 315 intra-decodes the intra-coded data to generate a current mesh. Note that this intra-decoding method may be any method as long as it is a decoding method that corresponds to the encoding method applied by the intra-encoding unit 116 of the encoding device 100. The intra-decoding unit 315 supplies the generated current mesh (part or all) to the synthesis unit 316.
[0100] The synthesis unit 316 performs processing related to the generation of dynamic meshes. For example, the synthesis unit 316 acquires the current mesh supplied from the inter-decoding unit 314. The synthesis unit 316 also acquires the current mesh supplied from the intra-decoding unit 315. The synthesis unit 316 generates a mesh sequence (i.e., a dynamic mesh) by synthesizing (arranging in the time direction) the meshes of each frame. The synthesis unit 316 outputs the generated dynamic mesh to the outside of the decoding device 300 as a decoding result. The output dynamic mesh may be stored in a storage medium, or may be supplied to a playback device or the like via any transmission path such as a network or computer, and may be rendered and displayed by the playback device. It may also be used for other processing.
[0101] The present technology (the above-described method 1) may be applied to a decoding device 300 configured as described above. For example, the control information decoding unit 312 of the decoding device 300 may decode the control information encoded data and generate control information that controls the processing range of inter-encoding and inter-decoding of a mesh representing a three-dimensional structure of an object. Furthermore, the reference mesh updating unit 313 may update a reference mesh, which is a mesh referenced during inter-decoding, based on the control information. Furthermore, the inter-decoding unit 314 may perform inter-decoding of the encoded data using the updated reference mesh, which is the updated reference mesh, and generate a current mesh, which is a mesh of the current frame.
[0102] With this configuration, the decoding device 300 can inter-decode inter-coded data and generate (restore) a part of the current mesh without the need for sub-meshing. Therefore, the decoding device 300 can suppress a decrease in coding efficiency.
[0103] In this case, the reference mesh update unit 313 may update the reference mesh in an update interval specified by the control information, similar to the reference mesh update unit 114. For example, the reference mesh update unit 313 may set a frame associated with the control information as the start frame of the update interval for updating the reference mesh, and update the reference mesh in that update interval. As described above, the control information can be associated with any frame of the dynamic mesh. Therefore, this update interval can start from any frame. In this way, the reference mesh update unit 313 can update the reference mesh in a partial interval within the sequence. In other words, the inter decoding unit 314 can generate (restore) a portion of the current mesh for that partial interval by inter decoding. Therefore, the decoding device 300 can suppress a decrease in coding efficiency.
[0104] As in the case of the encoding device 100, the reference mesh may be updated by any process. For example, it may be the deletion of a face from the reference mesh. For example, the control information may include information specifying a deletion target face, which is a face to be deleted. The reference mesh update unit 313 may delete the deletion target face specified by the control information from the reference mesh in the above-mentioned update interval. In this way, the reference mesh update unit 313 can reduce the faces in the reference mesh.
[0105] As with the encoding device 100, the faces to be deleted may be specified in any manner. For example, the control information may include information indicating a base vertex and the number of connections from the base vertex as information specifying the faces to be deleted. In this way, by using the base vertex and the number of connections, it is possible to specify faces to be deleted according to the mesh structure. For example, even if they are physically close, vertices that are far apart in the mesh structure (the number of connections between the base vertex and the vertex is high) may have little relevance to each other in the object (belong to faces that are far from each other). Such vertices can be excluded from the target vertices.
[0106] As in the case of the encoding device 100, the correspondence between the face to be deleted and the target vertex may be any relationship. For example, a face whose all vertices are made up of target vertices may be the face to be deleted. In other words, the reference mesh update unit 313 may set a vertex that is connected to a reference vertex specified by control information within the number of connections specified by the control information as the target vertex, and may set a face that is made up only of that target vertex as the face to be deleted.
[0107] Furthermore, a face to be deleted may be a face having one or more vertices constituted by a target vertex, as in the case of the encoding device 100. That is, the reference mesh update unit 313 may set a vertex connected to a reference vertex specified by control information within the number of connections specified by the control information as a target vertex, and may set a face constituted by vertices including the target vertex as a face to be deleted.
[0108] Furthermore, as described above in the description of the encoding device 100, information indicating a three-dimensional spatial region may be included in the control information as information specifying the faces to be deleted. For example, the control information may indicate a rectangular region 211 shown in FIG. 11 as information specifying the faces to be deleted. A vertex within this rectangular region 211 is specified as the target vertex. Any method for specifying this rectangular region 211 may be used. For example, this rectangular region 211 may be specified using the coordinates of two diagonal vertices (e.g., vertex Q1 and vertex Q2). Alternatively, this rectangular region 211 may be specified using the coordinates of vertex Q1 and the lengths of the three sides (X, Y, Z) of the rectangular region 211. Of course, other methods are also acceptable. Furthermore, the shape of the three-dimensional spatial region specifying the faces to be deleted is arbitrary and may be other than the rectangular shape shown in FIG. 11. For example, it may be spherical, cylindrical, or some other shape. By specifying the target vertex using a three-dimensional spatial region in this manner, it is possible to specify the target vertex based on physical distance, regardless of the mesh configuration. Therefore, for example, a group of vertices whose distance (connectivity) from a reference vertex in a mesh structure is not constant can be specified as target vertices.
[0109] When specifying a face to be deleted (target vertex) using a three-dimensional spatial region in this way, the correspondence between the face to be deleted and the target vertex may be any relationship, as in the case of using the reference vertex and connectivity number described above. For example, as in the case of the encoding device 100, a face whose vertices are all made up of target vertices may be the face to be deleted. In other words, the reference mesh update unit 114 may set a vertex located within the three-dimensional spatial region specified by the control information as the target vertex, and set a face made up only of that target vertex as the face to be deleted.
[0110] Furthermore, a face to be deleted may be a face having one or more vertices constituted by a target vertex, as in the case of the encoding device 100. That is, the reference mesh update unit 313 may determine a vertex located within a three-dimensional spatial region specified by control information as a target vertex, and may determine a face constituted by vertices including the target vertex as a face to be deleted.
[0111] When a face to be deleted as described above is set, the reference mesh update unit 313 updates the reference mesh by deleting the face to be deleted. For example, when a face shown in gray in the mesh 203 in Fig. 10 is set as a face to be deleted, the reference mesh update unit 114 deletes the face to be deleted and generates an updated reference mesh such as the mesh 231 shown in Fig. 12, which is made up of vertices P1 to P9 and the connections between them.
[0112] The inter-decoding unit 314 performs inter-decoding of inter-coded data using the updated reference mesh. For example, assume that the current mesh is a mesh consisting of vertices P1' to P10', such as mesh 232 shown in FIG. 12. If vertices P1' to P9' of mesh 232 correspond to vertices P1 to P9 of mesh 231, respectively, the topology of mesh 231 matches the topology of the portion formed by vertices P1' to P9' of mesh 232 and the connections between them. Therefore, the inter-decoding unit 314 performs inter-decoding using the updated reference (mesh 231) and generates (restores) the portion formed by vertices P1' to P9' of mesh 232 and the connections between them. In this way, even if the topology of the entire mesh does not match, the decoding device 300 can easily extract portions with matching topology between frames by updating the reference mesh (deleting some faces). This allows the decoding device 300 to more easily apply inter-decoding. This reduces the reduction in coding efficiency.
[0113] As in the case of the encoding device 100, the updating of the reference mesh is not limited to the deletion of a face. For example, it may be the addition of a face. For example, information specifying a target face to be added, which is a face to be added, may be included in the control information. The reference mesh updating unit 313 may add the target face to be added specified by the control information to the reference mesh in the above-mentioned update interval. In this way, the reference mesh updating unit 313 can increase the number of faces in the reference mesh.
[0114] As in the case of the encoding device 100, any method may be used to specify the face to be added. For example, as information specifying the face to be added, information indicating an additional vertex, which is a vertex to be added, and information indicating an additional connection, which is a connection to be added, may be included in the control information.
[0115] As in the case of the encoding device 100, the reference mesh update may also involve the deletion and addition of faces. For example, the control information may include information specifying a deletion target face, which is a face to be deleted, and information specifying an addition target face, which is a face to be added. The reference mesh update unit 313 may delete the deletion target face specified by the control information from the reference mesh during the above-mentioned update interval, and may also add the addition target face specified by the control information to the reference mesh. The method for specifying the deletion target face is as described above. The method for specifying the addition target face is as described above. In other words, the above-mentioned deletion and addition may be combined to update the reference mesh. In this way, the reference mesh update unit 313 can change the faces of the reference mesh.
[0116] Also, as in the case of the encoding device 100, the control information may specify an end frame of the update interval. As described above, the start frame of the update interval is specified as the frame to which the control information belongs. Therefore, it can be said that the length of the update interval is specified by specifying the end frame in the control information. Furthermore, since the start frame and end frame are specified, it can also be said that the update interval is specified from which point in the sequence to which frame interval (i.e., the update interval itself or the position of the update interval) is specified. The reference mesh update unit 313 specifies the update interval from the frame associated with the control information to the end frame specified by the control information, and updates the reference mesh in the update interval as described above. Therefore, as shown in FIG. 14, any face or vertex of the reference mesh can be updated only in any frame interval.
[0117] As in the case of the encoding device 100, the reference mesh update unit 313 can update only the necessary portions of the reference mesh both temporally and spatially. In other words, the reference mesh update unit 313 can perform more flexible control of the reference mesh. By controlling the reference mesh in this way, the decoding device 300 can increase the number of frames that refer to a common reference mesh (increase the number of frames with matching topology), making it easier to apply inter-decoding. Furthermore, since sub-meshing is not performed, there is no need to perform intra-decoding in an unnecessarily narrow area, and a decrease in the coding efficiency of intra-decoding can also be suppressed.
[0118] In addition, like the bit stream 252 of submesh B (meaning a partial region, not actually submeshed) and the bit stream 253 of submesh C (similar to submesh B), there is no need to align their sequences with the sequence of mesh A, and it is sufficient to generate a bit stream of only the necessary parts, so it is possible to suppress a decrease in coding efficiency. Also, even if submeshing is not performed, it is possible to omit decoding of the area of mesh A corresponding to this submesh B (only the information to be reproduced needs to be decoded), so it is possible to suppress an increase in the load of the reproduction process.
[0119] In addition, such control (changing the decoding method of some areas) can be performed during the inter-decoding section (without the need to end the inter-decoding of mesh A and perform intra-decoding). Therefore, it is possible to suppress the reduction in coding efficiency.
[0120] As with the encoding device 100, the above examples of updating the reference mesh include deleting and adding faces (or a combination thereof), but any update method may be used and is not limited to these examples. For example, a face may be rotated or flipped. A face may be enlarged or reduced. A face may be moved or deformed. A face may be subdivided, or multiple faces may be combined into a single face. Furthermore, multiple update methods may be combined and applied. Vertices and connections may be updated to achieve such face updates.
[0121] As described above, the inter decoding unit 314 performs inter decoding on the inter coded data using the updated reference mesh. Since the inter decoding in this case is mesh decoding, the inter decoding unit 314 may decode the coded data, generate motion vectors for corresponding vertices between the updated reference mesh and the current mesh, and generate the current mesh by adding the motion vectors to the updated reference mesh. In other words, the motion vectors indicate how the vertices have moved between the reference frame and the current frame. The motion vectors are derived for each vertex. The method for decoding the motion vectors may be any decoding method that is compatible with the inter coding applied by the inter coding unit 115.
[0122] The inter-decoding unit 314 decodes the inter-encoded data to generate (restore) a portion of the current mesh that corresponds to the updated reference mesh. Therefore, if the updated reference mesh corresponds to a portion of the current mesh, a portion of the current mesh that does not correspond to the updated reference mesh (a non-corresponding portion) will be generated. This non-corresponding portion will not be subject to inter-decoding for the portion that corresponds to the updated reference mesh. Therefore, the decoding device 300 may further include a non-corresponding portion decoding unit that decodes the encoded data and generates (restores) a portion of the current mesh that does not correspond to the updated reference mesh. In this manner, the decoding device 300 can decode the entire current mesh. Therefore, a reduction in coding efficiency can be suppressed.
[0123] Note that this non-corresponding portion may be intra-decoded. That is, the non-corresponding portion encoding unit may decode the encoded data to generate (restore) the portion of the current mesh that does not correspond to the updated reference mesh. In other words, the intra-decoding unit 315 may be the non-corresponding portion decoding unit.
[0124] In addition, this non-corresponding portion may be inter-decoded. That is, the non-corresponding portion encoding unit may decode the encoded data to generate (restore) the portion of the current mesh that does not correspond to the updated reference mesh. In other words, the inter-decoding unit 314 may be a non-corresponding portion decoding unit, and may inter-decode the non-corresponding portion independently of the corresponding portion.
[0125] That is, in the example of Figure 15, the first frames of bit stream 252 and bit stream 253 are intra-decoded, but they may also be inter-decoded. In this case, the mesh of any frame that is decoded before mesh A may be used as the reference mesh. In the example of Figure 15, bit stream 252 and bit stream 253 are shown separately from bit stream 251, but they may also be multiplexed into one bit stream.
[0126] <Flow of Decoding Process> An example of the flow of encoding process executed by the decoding device 300 will be described with reference to the flowchart of FIG.
[0127] When the decoding process starts, the demultiplexer 311 demultiplexes the bit stream in step S301 and extracts the control information coded data, inter-coded data, and intra-coded data.
[0128] In step S302, the control unit 301 determines whether or not to perform inter-coding. If it is determined that inter-coding is to be performed, the process proceeds to step S303.
[0129] In step S303, the control information decoder 312 decodes the control information coded data of the current frame to generate (restore) the control information.
[0130] In step S304, the reference mesh update unit 313 updates the reference mesh with update contents corresponding to the current mesh based on the control information.
[0131] In step S305, the inter decoding unit 314 performs inter decoding on the inter encoded data using the updated reference mesh, and generates (restores) the part of the current mesh that corresponds to the updated reference mesh.
[0132] In step S306, the inter-decoding unit 314 inter-decodes the inter-coded data, or the intra-decoding unit 315 intra-decodes the intra-coded data to generate the remaining faces of the current mesh. When the processing of step S306 is completed, the processing proceeds to step S308. Also, if it is determined in step S302 that inter-coding is not to be performed (intra-coding is to be performed), the processing proceeds to step S307.
[0133] In step S307, the intra-decoding unit 315 intra-decodes the intra-coded data. After the process of step S307 ends, the process proceeds to step S308.
[0134] In step S308, the synthesizing unit 316 synthesizes the frames (meshes) obtained by inter-decoding or intra-decoding to generate a sequence in which the frames are arranged. That is, the synthesizing unit 316 generates a dynamic mesh.
[0135] In step S309, the synthesis unit 316 outputs the dynamic mesh to the outside of the decoding device 300. When the process of step S309 ends, the decoding process ends.
[0136] By performing each process in this manner, the decoding device 300 can inter-decode part of the current mesh without dividing it into sub-meshes, thereby preventing a decrease in coding efficiency.
[0137] 4. Application Examples of the Present Technology Method 1-1 The present technology described above may be applied to any encoding method and decoding method. For example, as shown in the second row from the top of the table in FIG. 7, the present technology may be applied to inter-encoding and inter-decoding of V-DMC base meshes (Method 1-1).
[0138] <Inter-encoding unit> Fig. 19 is a diagram showing an example of the main configuration of the inter-encoding unit 115 (Fig. 7) in this case. Note that Fig. 19 shows the main processing units, data flows, etc., and is not limited to what is shown in Fig. 19. In other words, the inter-encoding unit 115 may include processing units that are not shown as blocks in Fig. 19, and may include processing and data flows that are not shown as arrows, etc. in Fig. 19.
[0139] As shown in FIG. 19, in this case, the inter-encoding unit 115 has a motion vector encoding unit 411, a base mesh reconstruction unit 412, a displacement vector correction unit 413, a packing unit 414, a displacement video encoding unit 415, a mesh reconstruction unit 416, an attribute map correction unit 417, an attribute video encoding unit 418, and a synthesis unit 419.
[0140] The motion vector encoding unit 411 uses the current base mesh and the updated reference base mesh to generate motion vectors based on the vertices of the updated reference base mesh. That is, the motion vector encoding unit 411 generates motion vectors that indicate the displacement of each vertex between the current base mesh and the updated reference base mesh. The motion vector encoding unit 411 encodes the generated motion vectors and supplies the encoded data to the base mesh reconstruction unit 412 and the synthesis unit 419.
[0141] The base mesh reconstruction unit 412 decodes the coded data of the supplied motion vector and generates (reconstructs) the motion vector. Then, the base mesh reconstruction unit 412 reconstructs a current base mesh using the updated reference base mesh and its motion vector. This reconstructed current base mesh includes coding distortion. The base mesh reconstruction unit 412 supplies the current base mesh to the displacement vector correction unit 413 and the mesh reconstruction unit 416.
[0142] The displacement vector correction unit 413 corrects the displacement vectors of the segmentation points using the supplied current base mesh (including encoding distortion). More specifically, the displacement vector correction unit 413 subdivides the supplied current base mesh (including encoding distortion) to derive segmentation points. Then, the displacement vector correction unit 413 corrects the displacement vectors of the segmentation points supplied to the encoding device 100 using the derived segmentation points. The displacement vector correction unit 413 supplies the corrected displacement values of the segmentation points to the packing unit 414.
[0143] The packing unit 414 packs the supplied displacement vectors of the division points into a two-dimensional image of the current frame (also referred to as a frame image). At this time, the packing unit 414 may perform coefficient transformation (e.g., wavelet transform) on the displacement values of the division points and pack the transformation coefficients into the two-dimensional image. Alternatively, the packing unit 414 may quantize the displacement vectors (or transformation coefficients) of the division points and pack the quantized coefficients into the two-dimensional image.
[0144] The packing unit 414 supplies the two-dimensional image in which the displacement vectors of the division points (or information equivalent thereto) are packed to the displacement video encoding unit 415. The packing unit 414 also supplies the generated partial region information to the displacement video encoding unit 415.
[0145] The displacement video encoding unit 415 encodes the supplied two-dimensional images as frame images into a moving image (displacement video). The displacement video encoding unit 415 supplies the generated encoded data of the displacement video to the mesh reconstruction unit 416 and the synthesis unit 419.
[0146] The mesh reconstruction unit 416 decodes the coded data of the supplied displacement video and derives displacement vectors of the division points by, for example, unpacking the two-dimensional image. The mesh reconstruction unit 416 also subdivides the supplied current base mesh and reconstructs a mesh by applying the derived displacement vectors of the division points. This mesh contains coding distortion. The mesh reconstruction unit 416 supplies the reconstructed mesh to the attribute map correction unit 417.
[0147] The attribute map correction unit 417 corrects an attribute map such as a texture using the supplied mesh (including encoding distortion), and supplies the corrected attribute map to the attribute video encoding unit 418 .
[0148] The attribute video encoding unit 418 converts the supplied attribute map into frame images and encodes them as moving images (attribute video). The attribute video encoding unit 418 supplies the generated encoded data of the attribute video to the synthesis unit 419.
[0149] The synthesizing unit 419 synthesizes (multiplexes) the supplied coded data of the motion vector, coded data of the displacement video, and coded data of the attribute video to generate inter-coded data, and outputs the generated inter-coded data to the outside of the inter-coding unit 115 (to the multiplexing unit 117 (FIG. 7)).
[0150] The present technology may be applied to a coding device 100 having an inter-coding unit 115 configured as described above. For example, the mesh may be a base mesh with lower resolution than an original mesh to be coded, generated by thinning vertices from the original mesh. The reference mesh update unit 114 may then update the reference base mesh, which is a base mesh referenced during inter-coding and inter-decoding, based on the control information. The inter-coding unit 115 may then inter-code the current base mesh, which is a base mesh of the current frame, using the updated reference base mesh.
[0151] In the inter-encoding unit 115, the displacement video encoding unit 415 may encode a displacement video consisting of frames packed with displacement vectors indicating the positional differences between division points, which are vertices of the subdivided base mesh, and the vertices of the original mesh. Also, the attribute video encoding unit 418 may encode an attribute video consisting of frames packed with attributes corresponding to the mesh.
[0152] With this configuration, the inter-encoding unit 115 can apply the present technology to inter-encoding of V-DMC base meshes. Therefore, the encoding device 100 can suppress a decrease in encoding efficiency.
[0153] <Flow of Inter-Encoding Process> An example of the flow of the inter-encoding process executed by the inter-encoding unit 115 in step S106 of FIG. 16 in this case will be described with reference to the flowchart of FIG.
[0154] When the inter-coding process starts, in step S411, the motion vector coding unit 411 generates and codes a motion vector using an updated reference mesh.
[0155] In step S412, the base mesh reconstruction unit 412 decodes the motion vector and reconstructs the current base mesh.
[0156] In step S413, the displacement vector correction unit 413 corrects the displacement vector.
[0157] In step S414, the packing unit 414 packs the displacement vectors into a frame.
[0158] In step S415, the displaced video encoding unit 415 encodes the displaced video.
[0159] In step S416, the mesh reconstructing unit 416 reconstructs the mesh.
[0160] In step S417, the attribute map corrector 417 corrects the attribute map.
[0161] In step S418, the attribute video encoding unit 418 encodes the attribute video.
[0162] In step S419, the synthesizing unit 419 multiplexes each piece of coded data to generate inter-coded data.
[0163] When the process of step S419 ends, the inter-encoding process ends and the process returns to FIG.
[0164] By performing each process in this manner, the inter-encoding unit 115 can apply the present technology to inter-encoding of V-DMC base meshes. Therefore, the encoding device 100 can suppress a decrease in encoding efficiency.
[0165] <Inter decoding unit> Fig. 21 is a block diagram showing an example of the configuration of the inter decoding unit 314 (Fig. 17), which is one aspect of an information processing device to which the present technology is applied. Note that Fig. 21 shows main processing units, data flows, etc., and does not necessarily show everything. In other words, in the inter decoding unit 314, there may be processing units that are not shown as blocks in Fig. 21, or there may be processing or data flows that are not shown as arrows, etc. in Fig. 21.
[0166] As shown in FIG. 21, in this case, the inter-decoding unit 314 has a demultiplexing unit 511, a motion vector decoding unit 512, a base mesh reconstruction unit 513, a subdivision unit 514, a displacement video decoding unit 515, an unpacking unit 516, a displacement vector application unit 517, and an attribute video decoding unit 518.
[0167] The demultiplexing unit 511 demultiplexes the inter-coded data and extracts each piece of coded data included in the inter-coded data. For example, the demultiplexing unit 511 extracts coded data of motion vectors from the inter-coded data and supplies it to the motion vector decoding unit 512. The demultiplexing unit 511 also extracts coded data of displacement video from the inter-coded data and supplies it to the displacement video decoding unit 515. The demultiplexing unit 511 also extracts coded data of attribute video from the bitstream and supplies it to the attribute video decoding unit 518.
[0168] The motion vector decoding unit 512 decodes the supplied coded data of the motion vector to generate (restore) the motion vector, and supplies the generated motion vector to the base mesh reconstructing unit 513.
[0169] The base mesh reconstruction unit 513 applies the supplied motion vector to the updated reference base mesh to reconstruct a current base mesh (a portion corresponding to the updated reference base mesh). The base mesh reconstruction unit 513 supplies the reconstructed current base mesh to the subdivision unit 514. Note that the base mesh reconstruction unit 513 may supply the current base mesh to an external device such as the decoding device 300, where it may be stored and reused as a reference base mesh in a subsequent frame. Note that the reference base mesh may be generated from the decoded mesh.
[0170] The subdivision unit 514 subdivides the supplied base mesh based on the supplied header information, and supplies the subdivided base mesh to the displacement vector application unit 517 .
[0171] The displacement video decoding unit 515 decodes the supplied encoded data of the displacement video to generate (restore) the displacement video (current frame). The displacement video decoding unit 515 supplies the generated displacement video (current frame) to the unpacking unit 516.
[0172] The unpacking unit 516 unpacks displacement vectors of segmentation points from the current frame of the supplied displacement video. In this case, the unpacking unit 516 may unpack transform coefficients and perform coefficient transformation (e.g., wavelet transform) on the transform coefficients to derive displacement values of the segmentation points. Alternatively, the unpacking unit 516 may unpack quantized coefficients and dequantize the quantized coefficients to derive displacement values of the segmentation points. Alternatively, the unpacking unit 516 may unpack quantized coefficients and dequantize them to derive transform coefficients, and perform coefficient transformation (e.g., wavelet transform) on the transform coefficients to derive displacement values of the segmentation points. The unpacking unit 516 supplies the unpacked displacement vectors of the segmentation points to the displacement vector application unit 517.
[0173] The displacement vector application unit 517 applies the displacement values of the division points provided to the division points of the base mesh (subdivided base mesh) provided to reconstruct the mesh (generate a decoded mesh). The displacement vector application unit 517 outputs the generated decoded mesh to the synthesis unit 316 ( FIG. 17 ).
[0174] The attribute video decoding unit 518 decodes the supplied coded data of the attribute video to generate (the current frame of) the attribute video. The attribute video decoding unit 518 outputs the generated current frame of the attribute video, i.e., the attribute map corresponding to the decoded mesh, to the synthesis unit 316 ( FIG. 17 ).
[0175] The present technology may be applied to a decoding device 300 having an inter-decoding unit 314 configured as described above. For example, the mesh may be a base mesh with lower resolution than an original mesh to be encoded, generated by thinning vertices from the original mesh. The reference mesh update unit 313 may then update the reference base mesh, which is a base mesh referenced during inter-encoding and inter-decoding, based on the control information. The inter-decoding unit 314 may then perform inter-decoding on the encoded data using the updated reference base mesh to generate a current base mesh, which is a base mesh for the current frame.
[0176] In the inter-decoding unit 314, the displacement video decoding unit 515 may decode the encoded data and generate a displacement video consisting of frames packed with displacement vectors indicating the positional differences between the division points, which are the vertices of the subdivided base mesh, and the vertices of the original mesh. Also, the attribute video decoding unit 518 may generate an attribute video consisting of frames packed with attributes corresponding to the mesh.
[0177] With this configuration, the inter decoding unit 314 can apply the present technology to inter decoding of V-DMC base meshes. Therefore, the decoding device 300 can suppress a decrease in coding efficiency.
[0178] <Flow of Inter-Decoding Process> An example of the flow of the inter-decoding process executed in step S305 of FIG. 18 will be described with reference to the flowchart of FIG.
[0179] When the inter-decoding process starts, the demultiplexing unit 511 demultiplexes the inter-coded data in step S511.
[0180] In step S512, the motion vector decoding unit 512 decodes the coded data of the motion vector to generate (restore) the motion vector.
[0181] In step S513, the base mesh reconstructor 513 reconstructs the current base mesh using the updated reference base mesh.
[0182] In step S514, the subdivision unit 514 subdivides the current base mesh.
[0183] In step S515, the displaced video decoding unit 515 decodes the encoded data of the displaced video to generate (restore) the displaced video.
[0184] In step S516, the unpacking unit 516 unpacks the displacement vector.
[0185] In step S517, the displacement vector application unit 517 applies the displacement vector to the division point.
[0186] In step S518, the attribute video decoding unit 518 decodes the coded data of the attribute video and generates an attribute map.
[0187] When the process of step S518 ends, the decoding process ends and the process returns to FIG.
[0188] By performing the process in this manner, the inter decoding unit 314 can apply the present technology to inter decoding of the V-DMC base mesh. Therefore, the decoding device 300 can suppress a decrease in coding efficiency.
[0189] <Method 1-2> As shown in the third row from the top of the table in FIG. 7, partial mesh replacement (independent encoding and decoding) may be applied (Method 1-2).
[0190] For example, when encoding 3D data (meshes) of a person, a torso portion 601 and a face portion 602 may be encoded independently of each other by updating a reference base mesh, as shown in FIG. 23 . By applying this technology instead of sub-meshing, the face portion 602 can be easily replaced with a face portion 603 with a different expression midway through a sequence. In other words, in this case, the face portion 602 and the face portion 603 can be encoded independently of each other, allowing switching between the face portion 602 and the face portion 603 at any timing. Furthermore, depending on the timing, the face portion 602 and the torso portion 601 can also be inter-coded together. Furthermore, as described above, a decrease in encoding efficiency can be suppressed.
[0191] For example, if such replacement is performed using the method described in Non-Patent Document 1 without sub-meshing, the only option is to decode the body part 601, the face part 602, and the face part 603, and then replace the face part 602 with the face part 603 through image processing. This method requires decoding the face part 602, which is not necessary for playback, increasing the processing load. By applying the present technology to update the reference mesh and performing inter-encoding and inter-decoding, it is possible to decode the body part 601 and the face part 603 without decoding the face part 602. This makes it possible to suppress an increase in the playback processing load.
[0192] Furthermore, by applying this technology, it is possible to independently encode non-face parts at different times, which means that the parts to be replaced can be changed over time.
[0193] In this way, by applying this technology, more flexible playback control can be achieved.
[0194] <Method 1-3> As shown in the fourth row from the top of the table in Fig. 7, the present technology may be applied to a dynamic mesh generated by capturing a real object (Method 1-3). Methods for generating dynamic meshes include capturing a real object in each frame to generate 3D data (mesh), and tracking each vertex of a static mesh (e.g., a mesh at a certain time) (also referred to as mesh tracking).
[0195] In the case of dynamic meshes generated by capture, the meshes for each frame are generated independently, making it difficult to match the topology across the entire mesh. This makes it difficult to apply inter-frame coding, which can reduce coding efficiency.
[0196] Therefore, by applying the present technology, inter-coding may be performed by deleting the portion of the reference mesh where the topology is inconsistent. For example, as shown by the thick line in FIG. 24 , assume that there is a topology inconsistency between the reference mesh 611 and the current mesh 612. In this case, the reference mesh 611 and the current mesh 612 may be updated so as to leave the portion where the topology of the reference mesh 611 and the current mesh 612 is inconsistent (so as to delete the inconsistent portion), and an updated reference mesh 613 and an updated current mesh 614 may be generated. Then, the updated reference mesh 613 may be used to encode the updated current mesh 614. In this way, it is possible to suppress a decrease in encoding efficiency.
[0197] 25, a portion of a reference mesh 621 that does not match the topology of a current mesh 622 may be converted to the same topology as the current mesh 622. For example, a mesh 623 may be generated by deleting the mismatched portion of the reference mesh 621, and then a face (vertex 10) identical to that of the current mesh 622 may be added to the deleted portion. In other words, the reference mesh may be updated to be closer to the current mesh. This can prevent a decrease in coding efficiency.
[0198] <Method 1-4> As shown in the bottom row of the table in FIG. 7, this technique may be applied to a dynamic mesh generated by mesh tracking (Method 1-4).
[0199] In the case of mesh tracking, the topology is consistent between frames. Therefore, inter-coding and inter-decoding are applicable, but there are cases where the current mesh has low quality compared to the real object. Such low-quality portions (where the difference is large compared to the shape of the real object) may be excluded from inter-coding and inter-decoding. For example, in FIG. 26 , the portion of the current mesh 632 indicated by the dashed-dotted line frame may be low-quality. Therefore, to exclude this portion, a portion corresponding to this portion may be deleted from the reference mesh 631 to generate an updated reference mesh 633. Similarly, the current mesh 632 may be updated to generate an updated current mesh 634. By performing inter-coding in this manner, it is possible to remove low-quality portions and perform inter-coding. Therefore, not only is a reduction in quality suppressed, but a reduction in coding efficiency is also suppressed.
[0200] <5. Supplementary Notes> <Encoded Data> In the above, a mesh has been described as an example of the encoding target, but the encoding target may be any kind of information. For example, it may be 3D mesh data such as a point cloud. The encoding target may also be a 2D image. The encoding target is not limited to visual information, but may also be auditory information such as audio data, or vibration data (tactile information) such as haptic data. The encoding target may also be any other information, or multiple types of information may be encoded.
[0201] <Computer> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs that make up the software are installed on a computer. Here, the term "computer" includes computers built into dedicated hardware, and general-purpose personal computers, etc., that can execute various functions by installing various programs.
[0202] FIG. 27 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.
[0203] In a computer 900 shown in FIG. 27, a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903 are interconnected via a bus 904.
[0204] An input / output interface 910 is also connected to the bus 904. To the input / output interface 910, an input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915 are connected.
[0205] The input unit 911 includes, for example, a keyboard, a mouse, a microphone, a touch panel, and an input terminal. The output unit 912 includes, for example, a display, a speaker, and an output terminal. The storage unit 913 includes, for example, a hard disk, a RAM disk, and a non-volatile memory. The communication unit 914 includes, for example, a network interface. The drive 915 drives removable media 921 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.
[0206] In a computer configured as described above, the CPU 901 performs the above-described series of processes by, for example, loading a program stored in the storage unit 913 into the RAM 903 via the input / output interface 910 and the bus 904 and executing the program. The RAM 903 also stores data necessary for the CPU 901 to execute various processes as appropriate.
[0207] The program executed by the computer can be applied by recording it on, for example, a removable medium 921 such as a package medium. In this case, the program can be installed in the storage unit 913 via the input / output interface 910 by inserting the removable medium 921 into the drive 915.
[0208] This program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, digital satellite broadcasting, etc. In this case, the program can be received by the communication unit 914 and installed in the storage unit 913.
[0209] Alternatively, this program can be installed in advance in the ROM 902 or the storage unit 913 .
[0210] <Application of the Present Technology> The present technology can be applied to any configuration. For example, the present technology can be applied to various electronic devices.
[0211] Furthermore, for example, the present technology can also be implemented as part of an apparatus, such as a processor (e.g., a video processor) as a system LSI (Large Scale Integration), a module using multiple processors (e.g., a video module), a unit using multiple modules (e.g., a video unit), or a set in which other functions are added to a unit (e.g., a video set).
[0212] Furthermore, for example, the present technology can also be applied to a network system configured with multiple devices. For example, the present technology may be implemented as cloud computing in which multiple devices share and collaborate on processing via a network. For example, the present technology may be implemented in a cloud service that provides image (video)-related services to any terminal, such as a computer, an AV (Audio Visual) device, a portable information processing terminal, or an IoT (Internet of Things) device.
[0213] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are housed in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.
[0214] <Fields and uses to which this technology can be applied> Systems, devices, processing units, etc. to which this technology is applied can be used in any field, for example, transportation, medical care, crime prevention, agriculture, livestock farming, mining, beauty, factories, home appliances, weather, nature monitoring, etc. In addition, the uses thereof are also arbitrary.
[0215] <Others> In this specification, a "flag" refers to information for identifying multiple states, and includes not only information used to identify two states, true (1) or false (0), but also information capable of identifying three or more states. Therefore, the value that this "flag" can take may be, for example, two values, 1 / 0, or three or more values. That is, the number of bits constituting this "flag" is arbitrary, and may be one bit or multiple bits. Furthermore, identification information (including flags) can be included not only in a bitstream, but also in a bitstream that includes differential information of the identification information relative to certain reference information. Therefore, in this specification, "flag" and "identification information" encompass not only the information itself, but also differential information relative to the reference information.
[0216] Furthermore, various types of information (e.g., metadata) related to the coded data (bitstream) may be transmitted or recorded in any form as long as they are associated with the coded data. Here, the term "associate" means, for example, that one piece of data can be used (linked) when processing the other piece of data. That is, the associated pieces of data may be combined into one piece of data or may be separate pieces of data. For example, information associated with coded data (image) may be transmitted over a transmission path separate from that of the coded data (image). Furthermore, for example, information associated with coded data (image) may be recorded on a recording medium separate from that of the coded data (image) (or on a different recording area of the same recording medium). Note that this "association" may refer not to the entire data, but to only part of the data. For example, an image and information corresponding to that image may be associated with each other in any unit, such as multiple frames, one frame, or a portion of a frame.
[0217] In this specification, terms such as "composite," "multiplex," "add," "integrate," "include," "store," "embed," "insert," and the like refer to combining multiple items into one, such as combining encoded data and metadata into one piece of data, and refer to one method of "associating" as described above.
[0218] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present technology.
[0219] For example, a configuration described as one device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, configurations described above as multiple devices (or processing units) may be combined and configured as one device (or processing unit). Of course, configurations other than those described above may be added to the configuration of each device (or each processing unit). Furthermore, as long as the configuration and operation of the entire system are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).
[0220] Furthermore, for example, the above-described program may be executed in any device, as long as the device has the necessary functions (functional blocks, etc.) and is able to obtain the necessary information.
[0221] Also, for example, each step of a single flowchart may be executed by a single device, or may be shared and executed by multiple devices. Furthermore, when a single step includes multiple processes, the multiple processes may be executed by a single device, or may be shared and executed by multiple devices. In other words, multiple processes included in a single step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as a single step.
[0222] For example, the steps of a program executed by a computer may be executed in chronological order in the order described herein, or may be executed in parallel or individually at the required timing, such as when a call is made. In other words, as long as no contradiction occurs, the steps may be executed in an order different from the order described above. Furthermore, the steps of this program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.
[0223] Furthermore, for example, multiple technologies related to the present technology can be implemented independently and independently, as long as no contradiction occurs. Of course, any multiple technologies can also be implemented in combination. For example, part or all of the present technology described in any embodiment can be implemented in combination with part or all of the present technology described in another embodiment. Furthermore, part or all of any of the above-described present technologies can be implemented in combination with other technologies not described above.
[0224] The present technology may also be configured as follows. (1) An information processing device including: a control information encoding unit that encodes control information controlling a processing range for inter-encoding and inter-decoding of a mesh that represents a three-dimensional structure of an object; a reference mesh updating unit that updates a reference mesh that is the mesh referenced during the inter-encoding based on the control information; and an inter-encoding unit that inter-encodes a current mesh that is the mesh of a current frame using the updated reference mesh that is the updated reference mesh. (2) The information processing device described in (1), in which the inter-encoding unit derives motion vectors of corresponding vertices between the updated reference mesh and the current mesh and encodes the motion vectors. (3) The information processing device described in (2), further including a non-corresponding portion encoding unit that encodes a portion of the current mesh that does not correspond to the updated reference mesh. (4) The information processing device described in (3), in which the non-corresponding portion encoding unit intra-encodes a portion of the current mesh that does not correspond to the updated reference mesh. (5) The information processing device according to (3) or (4), wherein the non-corresponding portion encoding unit inter-encodes a portion of the current mesh that does not correspond to the updated reference mesh. (6) The information processing device according to any of (1) to (5), wherein the reference mesh updating unit sets a frame associated with the control information as a start frame of an update interval for updating the reference mesh, and updates the reference mesh in the update interval. (7) The information processing device according to (6), wherein the control information includes information specifying a deletion target face that is a face to be deleted, and the reference mesh updating unit deletes the deletion target face specified by the control information from the reference mesh in the update interval. (8) The information processing device according to (7), wherein the control information includes information indicating a base vertex and the number of connections from the base vertex as information specifying the deletion target face.(9) The information processing device according to (8), wherein the reference mesh updating unit sets a vertex connected from the base vertex within the connectivity number as a target vertex, and sets a face constituted only by the target vertex as the face to be deleted. (10) The information processing device according to (8) or (9), wherein the reference mesh updating unit sets a vertex connected from the base vertex within the connectivity number as a target vertex, and sets a face constituted by vertices including the target vertex as the face to be deleted. (11) The information processing device according to any of (7) to (10), wherein the control information includes information indicating a three-dimensional spatial region as information specifying the face to be deleted. (12) The information processing device according to (11), wherein the reference mesh updating unit sets a vertex located within the three-dimensional spatial region as a target vertex, and sets a face constituted only by the target vertex as the face to be deleted. (13) The information processing device according to (11) or (12), wherein the reference mesh updating unit sets a vertex located within the three-dimensional spatial region as a target vertex, and sets a face constituted by vertices including the target vertex as the face to be deleted. (14) The information processing device according to any of (6) to (13), wherein the control information includes information specifying an addition target face that is a face to be added, and the reference mesh update unit adds the addition target face specified by the control information to the reference mesh in the update interval. (15) The information processing device according to any of (14), wherein the control information includes, as information specifying the addition target face, information indicating an addition vertex that is a vertex to be added, and information indicating an addition connection that is a connection to be added. (16) The information processing device according to any of (6) to (15), wherein the control information includes information specifying a deletion target face that is a face to be deleted, and information specifying an addition target face that is a face to be added, and the reference mesh update unit deletes the deletion target face specified by the control information from the reference mesh in the update interval, and further adds the addition target face specified by the control information to the reference mesh.(17) The information processing device according to any of (6) to (16), wherein the control information includes information specifying an end frame of the update interval, and the reference mesh update unit sets the update interval from a frame associated with the control information to the end frame specified by the control information, and updates the reference mesh in the update interval. (18) The information processing device according to any of (1) to (17), wherein the mesh is a base mesh with lower resolution than the original mesh generated by thinning vertices from an original mesh to be coded, and the reference mesh update unit updates a reference base mesh that is the base mesh referenced during the inter coding and the inter decoding based on the control information, and the inter coding unit inter codes a current base mesh that is the base mesh of a current frame, using the updated reference base mesh that is the updated reference base mesh. (19) The information processing device according to (18), further comprising a displacement video coding unit that codes a displacement video configured by frames in which displacement vectors that indicate positional differences between division points that are vertices of the subdivided base mesh and vertices of the original mesh are packed. (20) The information processing device according to (18) or (19), further comprising an attribute video encoding unit that encodes an attribute video configured by frames in which attributes corresponding to the meshes are packed. (21) An information processing method, comprising: encoding control information that controls a processing range of inter-coding and inter-decoding of meshes that represent a three-dimensional structure of an object; updating a reference mesh that is the mesh referred to during the inter-coding based on the control information; and inter-coding a current mesh that is the mesh of a current frame using the updated reference mesh that is the updated reference mesh.
[0225] (31) An information processing device comprising: a control information decoding unit that decodes coded data and generates control information that controls a processing range for inter-coding and inter-decoding a mesh that represents a three-dimensional structure of an object; a reference mesh updating unit that updates a reference mesh that is the mesh referenced during the inter-decoding based on the control information; and an inter decoding unit that inter-decodes coded data using the updated reference mesh that is the updated reference mesh and generates a current mesh that is the mesh of a current frame. (32) The information processing device described in (31), in which the inter decoding unit decodes the coded data and generates motion vectors of corresponding vertices between the updated reference mesh and the current mesh, and adds the motion vectors to the updated reference mesh to generate the current mesh. (33) The information processing device described in (32), further comprising a non-corresponding portion decoding unit that decodes coded data and generates a portion of the current mesh that does not correspond to the updated reference mesh. (34) The information processing device described in (33), in which the non-corresponding portion decoding unit intra-decodes the coded data and generates a portion of the current mesh that does not correspond to the updated reference mesh. (35) The information processing device according to (33) or (34), wherein the non-corresponding portion decoding unit inter-decodes the encoded data and generates a portion of the current mesh that does not correspond to the updated reference mesh. (36) The information processing device according to any of (31) to (35), wherein the reference mesh updating unit sets a frame associated with the control information as a start frame of an update interval in which the reference mesh is updated, and updates the reference mesh in the update interval. (37) The information processing device according to (36), wherein the control information includes information specifying a deletion target face that is a face to be deleted, and the reference mesh updating unit deletes the deletion target face specified by the control information from the reference mesh in the update interval.(38) The information processing device according to (37), wherein the control information includes information indicating a base vertex and a connectivity number from the base vertex as information specifying the faces to be deleted. (39) The information processing device according to (38), wherein the reference mesh updating unit sets a vertex connected to the base vertex within the connectivity number as a target vertex, and sets a face consisting only of the target vertex as the faces to be deleted. (40) The information processing device according to (38) or (39), wherein the reference mesh updating unit sets a vertex connected to the base vertex within the connectivity number as a target vertex, and sets a face consisting of vertices including the target vertex as the faces to be deleted. (41) The information processing device according to any of (37) to (40), wherein the control information includes information indicating a three-dimensional space region as information specifying the faces to be deleted. (42) The information processing device according to (41), wherein the reference mesh updating unit sets a vertex located in the three-dimensional space region as a target vertex, and sets a face consisting only of the target vertex as the faces to be deleted. (43) The information processing device according to (41) or (42), wherein the reference mesh updating unit sets a vertex located within the three-dimensional spatial region as a target vertex, and sets a face constituted by vertices including the target vertex as the face to be deleted. (44) The control information includes information specifying an addition target face that is a face to be added, and the reference mesh updating unit adds the addition target face specified by the control information to the reference mesh in the update interval. (45) The information processing device according to (44), wherein the control information includes, as information specifying the addition target face, information indicating an addition vertex that is a vertex to be added, and information indicating an addition connection that is a connection to be added.(46) The information processing device according to any of (36) to (45), wherein the control information includes information specifying a deletion target face which is a face to be deleted, and information specifying an addition target face which is a face to be added, and the reference mesh update unit deletes the deletion target face specified by the control information from the reference mesh in the update interval, and further adds the addition target face specified by the control information to the reference mesh. (47) The information processing device according to any of (36) to (46), wherein the control information includes information specifying an end frame of the update interval, and the reference mesh update unit sets the update interval from a frame associated with the control information to the end frame specified by the control information, and updates the reference mesh in the update interval. (48) The information processing device according to any one of (31) to (47), wherein the mesh is a base mesh of lower resolution than the original mesh generated by thinning vertices from the original mesh to be encoded, the reference mesh update unit updates a reference base mesh that is the base mesh referenced during the inter-encoding and the inter-decoding based on the control information, and the inter-decoding unit inter-decodes the encoded data using the updated reference base mesh that is the updated reference base mesh to generate a current base mesh that is the base mesh of a current frame. (49) The information processing device according to (48), further comprising a displacement video decoding unit that decodes the encoded data and generates a displacement video configured by frames in which displacement vectors indicating positional differences between division points that are vertices of the subdivided base mesh and vertices of the original mesh are packed. (50) The information processing device according to (48) or (49), further comprising an attribute video decoding unit that decodes the encoded data and generates an attribute video configured by frames in which attributes corresponding to the mesh are packed.(51) An information processing method, comprising: decoding encoded data; generating control information for controlling the processing range of inter-encoding and inter-decoding of a mesh representing a three-dimensional structure of an object; updating a reference mesh, which is the mesh referred to during the inter-decoding, based on the control information; inter-decoding the encoded data using the updated reference mesh, which is the updated reference mesh; and generating a current mesh, which is the mesh of a current frame.
[0226] REFERENCE SIGNS LIST 100 Encoding device, 101 Control unit, 111 Encoding method selection unit, 112 Control information supply unit, 113 Control information encoding unit, 114 Reference mesh update unit, 115 Inter encoding unit, 116 Intra encoding unit, 117 Multiplexing unit, 300 Decoding device, 301 Control unit, 311 Demultiplexing unit, 312 Control information decoding unit, 313 Reference mesh update unit, 314 Inter decoding unit, 315 Intra decoding unit, 316 Synthesis unit, 900 Computer
Claims
1. An information processing device comprising: a control information encoding unit that encodes control information that controls the processing range of inter-encoding and inter-decoding of a mesh that represents a three-dimensional structure of an object; a reference mesh update unit that updates a reference mesh, which is the mesh referenced during the inter-encoding, based on the control information; and an inter-encoding unit that inter-encodes a current mesh, which is the mesh of a current frame, using the updated reference mesh, which is the updated reference mesh.
2. The information processing device according to claim 1, wherein the inter-encoding section derives motion vectors of corresponding vertices between the updated reference mesh and the current mesh, and encodes the motion vectors.
3. The information processing device according to claim 1, wherein the reference mesh update unit sets a frame associated with the control information as a start frame of an update interval in which the reference mesh is updated, and updates the reference mesh in the update interval.
4. The information processing device of claim 3, wherein the control information includes information specifying a face to be deleted, which is a face to be deleted, and the reference mesh update unit deletes the face to be deleted specified by the control information from the reference mesh during the update interval.
5. The information processing device according to claim 4, wherein the control information includes information indicating a base vertex and a number of connections from the base vertex as information for specifying the face to be deleted.
6. The information processing device according to claim 4, wherein the control information includes information indicating a three-dimensional space area as information for specifying the face to be deleted.
7. The information processing device according to claim 3, wherein the control information includes information specifying a face to be added, which is a face to be added, and the reference mesh update unit adds the face to be added specified by the control information to the reference mesh during the update interval.
8. The information processing device of claim 3, wherein the control information includes information specifying an end frame of the update interval, and the reference mesh update unit defines the update interval as the frame associated with the control information to the end frame specified by the control information, and updates the reference mesh in the update interval.
9. The information processing device of claim 1, wherein the mesh is a base mesh with lower resolution than an original mesh to be encoded, generated by thinning vertices from the original mesh, the reference mesh update unit updates a reference base mesh, which is the base mesh referenced during the inter-encoding and the inter-decoding, based on the control information, and the inter-encoding unit inter-encodes a current base mesh, which is the base mesh of a current frame, using an updated reference base mesh, which is the updated reference base mesh.
10. An information processing method comprising: encoding control information for controlling a processing range of inter-encoding and inter-decoding of a mesh that represents a three-dimensional structure of an object; updating a reference mesh, which is the mesh that is referenced during the inter-encoding, based on the control information; and inter-encoding a current mesh, which is the mesh of a current frame, using the updated reference mesh, which is the updated reference mesh.
11. An information processing device comprising: a control information decoding unit that decodes encoded data and generates control information that controls the processing range of inter-encoding and inter-decoding of a mesh that represents a three-dimensional structure of an object; a reference mesh update unit that updates a reference mesh, which is the mesh referenced during the inter-decoding, based on the control information; and an inter-decoding unit that inter-decodes the encoded data using the updated reference mesh, which is the updated reference mesh, and generates a current mesh, which is the mesh of a current frame.
12. The information processing device according to claim 11, wherein the inter-decoding unit decodes the encoded data, generates motion vectors of corresponding vertices between the updated reference mesh and the current mesh, and adds the motion vectors to the updated reference mesh to generate the current mesh.
13. The information processing device according to claim 11, wherein the reference mesh update unit sets a frame associated with the control information as a start frame of an update interval in which the reference mesh is updated, and updates the reference mesh in the update interval.
14. The information processing device according to claim 13, wherein the control information includes information specifying a face to be deleted, which is a face to be deleted, and the reference mesh update unit deletes the face to be deleted specified by the control information from the reference mesh during the update interval.
15. The information processing device according to claim 14, wherein the control information includes information indicating a base vertex and a number of connections from the base vertex as information for specifying the face to be deleted.
16. The information processing device according to claim 14, wherein the control information includes information indicating a three-dimensional spatial region as information for specifying the face to be deleted.
17. The information processing device according to claim 13, wherein the control information includes information specifying a face to be added, which is a face to be added, and the reference mesh update unit adds the face to be added specified by the control information to the reference mesh during the update interval.
18. The information processing device of claim 13, wherein the control information includes information specifying an end frame of the update interval, and the reference mesh update unit defines the update interval as a period from a frame associated with the control information to the end frame specified by the control information, and updates the reference mesh in the update interval.
19. The information processing device described in claim 11, wherein the mesh is a base mesh with lower resolution than an original mesh to be encoded, which is generated by thinning vertices from the original mesh, the reference mesh update unit updates a reference base mesh, which is the base mesh referenced during the inter-encoding and the inter-decoding, based on the control information, and the inter-decoding unit inter-decodes the encoded data using the updated reference base mesh, which is the updated reference base mesh, to generate a current base mesh, which is the base mesh of a current frame.
20. An information processing method comprising: decoding encoded data; generating control information for controlling the processing range of inter-encoding and inter-decoding of a mesh that represents a three-dimensional structure of an object; updating a reference mesh, which is the mesh referred to during the inter-decoding, based on the control information; inter-decoding the encoded data using the updated reference mesh, which is the updated reference mesh; and generating a current mesh, which is the mesh of a current frame.
Citation Information
Patent Citations
Mesh-based video compression using domain transform
JP2009545931A
Methods and apparatuses for dynamic mesh compression
WO2023004241A1
