Encoding method, decoding method, encoding apparatus, and decoding apparatus

Abstract

The encoding method of one embodiment of the present disclosure generates a plurality of pieces of encoded data by encoding a plurality of sub-grids (S721), generates threshold information indicating a threshold value determined for each combination of two sub-grids selected from the plurality of sub-grids, on the basis of the positions of the plurality of three-dimensional points included in the plurality of sub-grids (S722), and generates a bitstream including the plurality of pieces of encoded data and the threshold information (S723).

Description

Technical Field

[0001] This disclosure relates to encoding methods, etc. Background Technology

[0002] Patent document 1 proposes a method and apparatus for encoding and decoding three-dimensional mesh data.

[0003] Existing technical documents Patent documents Patent Document 1: Japanese Patent Application Publication No. 2006-187015 Summary of the Invention

[0004] The problem that the invention aims to solve There is a desire to further improve the encoding or decoding processing related to three-dimensional data. The purpose of this disclosure is to improve the encoding or decoding processing related to three-dimensional data.

[0005] Methods used to solve problems One aspect of the present disclosure is an encoding method that generates multiple encoded data by encoding multiple subgrids; generates threshold information based on the positions of multiple three-dimensional points contained in the multiple subgrids, the threshold information representing a threshold determined by a combination of two subgrids selected from the multiple subgrids for each; and generates a bitstream containing the multiple encoded data and the threshold information.

[0006] Furthermore, these general or specific methods can be implemented either by a system, apparatus, method, integrated circuit, computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or by any combination of systems, apparatus, methods, integrated circuits, computer programs, and recording media.

[0007] Invention Effects This disclosure can help improve coding processes related to three-dimensional data. Attached Figure Description

[0008] Figure 1 This is a conceptual diagram representing a three-dimensional mesh related to the implementation method.

[0009] Figure 2 This is a conceptual diagram representing the basic elements of a three-dimensional mesh for a particular implementation method.

[0010] Figure 3 This is a conceptual diagram representing a mapping of implementation methods.

[0011] Figure 4 This is a block diagram illustrating an example of the configuration of an encoding / decoding system according to an implementation method.

[0012] Figure 5This is a block diagram illustrating an example of the configuration of an encoding device according to an embodiment.

[0013] Figure 6 This is a block diagram illustrating another configuration example of the encoding device according to the relevant implementation method.

[0014] Figure 7 This is a block diagram illustrating an example of the configuration of a decoding device according to an embodiment.

[0015] Figure 8 This is a block diagram illustrating another configuration example of a decoding device according to an embodiment.

[0016] Figure 9 This is a conceptual diagram illustrating a configuration example of a bitstream for a specific implementation method.

[0017] Figure 10 This is a conceptual diagram illustrating another example of a bitstream implementation.

[0018] Figure 11 This is a conceptual diagram representing another example of the configuration of a bitstream in relation to an implementation method.

[0019] Figure 12 This is a block diagram illustrating a specific example of an encoding / decoding system according to an implementation method.

[0020] Figure 13 This is a conceptual diagram representing an example of the configuration of point group data related to the implementation method.

[0021] Figure 14 This is a conceptual diagram representing a data file example of point group data related to an implementation method.

[0022] Figure 15 This is a conceptual diagram illustrating an example of the structure of grid data for a given implementation method.

[0023] Figure 16 This is a conceptual diagram representing a data file example of grid data related to an implementation method.

[0024] Figure 17 This is a conceptual diagram representing the types of three-dimensional data related to the implementation method.

[0025] Figure 18 This is a block diagram illustrating an example of the configuration of a three-dimensional data encoder according to an implementation method.

[0026] Figure 19 This is a block diagram illustrating an example of the configuration of a three-dimensional data decoder according to an implementation method.

[0027] Figure 20 This is a block diagram illustrating another configuration example of a three-dimensional data encoder with respect to an implementation method.

[0028] Figure 21 This is a block diagram illustrating another configuration example of a three-dimensional data decoder with respect to an implementation method.

[0029] Figure 22 This is a conceptual diagram representing a specific example of the coding process related to the implementation method.

[0030] Figure 23 This is a conceptual diagram illustrating a specific example of the decoding process in a particular implementation.

[0031] Figure 24 This is a block diagram illustrating an installation example of the encoding device according to an embodiment.

[0032] Figure 25 This is a block diagram illustrating an installation example of the decoding device according to an embodiment.

[0033] Figure 26 This is a block diagram illustrating another configuration example of an encoding / decoding system according to an embodiment.

[0034] Figure 27 This is a block diagram illustrating another configuration example of the encoding device according to the relevant implementation method.

[0035] Figure 28 This is a block diagram illustrating another configuration example of a decoding device according to an embodiment.

[0036] Figure 29 This is a block diagram illustrating yet another configuration example of an encoding device according to an embodiment.

[0037] Figure 30 This is a block diagram illustrating yet another configuration example of a decoding device according to an embodiment.

[0038] Figure 31 This is a flowchart illustrating the processing of the encoding device in relation to the implementation method.

[0039] Figure 32 This is an explanatory diagram that conceptually represents the encoding of a grid frame in a relevant implementation method.

[0040] Figure 33 This is a flowchart illustrating the processing of the decoding device according to the implementation method.

[0041] Figure 34 This is a conceptual illustration of the decoding of a grid frame in a relevant implementation method.

[0042] Figure 35 This is a block diagram illustrating an example of the configuration of a decoding device according to an embodiment.

[0043] Figure 36 This is a block diagram illustrating an example of the configuration of an encoding device according to an embodiment.

[0044] Figure 37 This is a block diagram illustrating an example of the configuration of a decoding device according to an embodiment.

[0045] Figure 38 This is an explanatory diagram showing an example of sub-segmentation related to the implementation method.

[0046] Figure 39 This is an explanatory diagram showing an example of the displacement of a vertex after subdivision and displacement in a relevant implementation method.

[0047] Figure 40 This is an explanatory diagram showing an example of the vertices of the original mesh in a relevant implementation method.

[0048] Figure 41 This is an explanatory diagram showing an example of a grid for a particular implementation method.

[0049] Figure 42 This is an explanatory diagram illustrating an example of grid-to-subgrid division in a relevant implementation method.

[0050] Figure 43 This is the first explanatory figure, which shows an example of packing displacement information of an image frame in relation to an implementation method.

[0051] Figure 44 This is the second explanatory figure, which shows an example of packing displacement information of an image frame in relation to an implementation method.

[0052] Figure 45 This is the third explanatory figure, which shows an example of packing displacement information of an image frame in relation to an implementation method.

[0053] Figure 46 This is a block diagram illustrating another example of the structure of the encoding device according to an embodiment.

[0054] Figure 47 This is a block diagram illustrating an example of the configuration of a preprocessor for a specific implementation.

[0055] Figure 48 This is a block diagram illustrating another example of the structure of the decoding device according to an embodiment.

[0056] Figure 49 This is a block diagram illustrating a specific example of the structure of a decoding device according to an embodiment.

[0057] Figure 50 This is a diagram illustrating an example of a subgrid with two sub-divided sections having boundary edges in a relevant implementation.

[0058] Figure 51 This is a flowchart illustrating the processing of the decoding device according to the implementation method.

[0059] Figure 52This is a diagram used to illustrate an example of the boundary edges and non-boundary edges of the relevant implementation method.

[0060] Figure 53 This is a diagram used to illustrate other examples of boundary edges and non-boundary edges related to the implementation method.

[0061] Figure 54 This is a diagram illustrating an example of the syntax for signaling different sub-segment types and sub-segment numbers in the header, according to an implementation method.

[0062] Figure 55 This is a diagram illustrating an example of the syntax for signaling the subsegment type and subsegment number using a sequence parameter set in relation to an implementation method.

[0063] Figure 56 This is a diagram illustrating an example of a syntax for using a sequence parameter set to determine the sub-segment type and the number of sub-segments, and to check whether an edge is located on a boundary, in relation to an implementation method.

[0064] Figure 57 This is a flowchart illustrating an example of a process for dividing multiple edges that constitute a subgrid in a relevant implementation.

[0065] Figure 58 This is a diagram illustrating a first example of the process of dividing multiple edges that constitute a subgrid in relation to an implementation method.

[0066] Figure 59 This is a second example of a process for dividing the multiple edges that constitute a subgrid in relation to an implementation method.

[0067] Figure 60 This is a diagram illustrating a third example of the process of dividing the multiple edges constituting a subgrid in relation to an implementation method.

[0068] Figure 61 This is a fourth example of a process for dividing the multiple edges that constitute a subgrid in relation to an implementation method.

[0069] Figure 62 This is a diagram illustrating the fifth example of the process of dividing the multiple edges that constitute a subgrid in relation to an implementation method.

[0070] Figure 63 This is a diagram illustrating the sixth example of the process of dividing the multiple edges that constitute a subgrid in relation to an implementation method.

[0071] Figure 64 This is a flowchart illustrating the boundary vertex determination process performed by the encoding device in the relevant implementation method.

[0072] Figure 65This is a flowchart illustrating the subgrid overlap determination process performed by the decoding device in the relevant implementation method.

[0073] Figure 66 This is a flowchart illustrating the vertex correction process performed by the decoding device in the relevant implementation method.

[0074] Figure 67 This is a diagram illustrating a specific example of the correction processing of the vertices in the implementation method.

[0075] Figure 68 This is a diagram illustrating a specific example of the correction processing of the vertices in the implementation method.

[0076] Figure 69 This is a diagram showing a specific example of the location of the storage flag information related to the implementation method.

[0077] Figure 70 This is a diagram illustrating an example of the syntax for signaling notification of flag information in an implementation method.

[0078] Figure 71 This is another example of the syntax used to illustrate the signaling notification of flag information in the relevant implementation method.

[0079] Figure 72 This is a diagram showing a specific example of the location of information about the storage method for implementing the method.

[0080] Figure 73 This is a diagram illustrating an example of the syntax for signaling information about a method implementation.

[0081] Figure 74 This is another example of the syntax used to illustrate the signaling notification of method information related to the implementation method.

[0082] Figure 75 This is a diagram illustrating a specific example of a method for modifying the vertices of a relevant implementation.

[0083] Figure 76 This is a diagram illustrating an example of the relationship between the number of sub-segments of multiple subgrids and the flag information in an embodiment.

[0084] Figure 77 This is another example of a diagram used to illustrate the relationship between the number of sub-segments of multiple subgrids in an embodiment and the flag information.

[0085] Figure 78 This is a diagram illustrating an example of the syntax for signaling information about the types of overlap of multiple subgrids in an implementation.

[0086] Figure 79This is a diagram used to illustrate the LoD (Location of Implementation) related to the implementation method.

[0087] Figure 80 This is a flowchart illustrating the vertex combination process in a decoding device according to an embodiment.

[0088] Figure 81 This is an explanatory diagram illustrating an example of the vertex combination process in a relevant implementation method.

[0089] Figure 82 This is an explanatory diagram illustrating another example of the vertices joining process in relation to an implementation method.

[0090] Figure 83 This is an explanatory diagram illustrating another example of the vertices joining process in relation to an implementation method.

[0091] Figure 84 This is an explanatory diagram illustrating another example of the vertices joining process in relation to an implementation method.

[0092] Figure 85 This is a flowchart illustrating the combination process of each LoD vertex in the decoding apparatus of the relevant implementation.

[0093] Figure 86 This is a flowchart illustrating the process of combining the vertices of LoD0 in a decoding device according to an embodiment.

[0094] Figure 87 This is another example of the syntax used to illustrate the signaling notification of method information related to the implementation method.

[0095] Figure 88 This is an explanatory diagram illustrating another example of the vertices joining process in relation to an implementation method.

[0096] Figure 89 This is an explanatory diagram illustrating another example of the vertices joining process in relation to an implementation method.

[0097] Figure 90 This is a block diagram illustrating an example of the configuration of an encoding device and a decoding device in an embodiment.

[0098] Figure 91 This is a flowchart illustrating the matching point search process in the implementation method.

[0099] Figure 92 This is a diagram used to illustrate the positional relationship of multiple subgrids in the implementation method.

[0100] Figure 93 This is a diagram used to illustrate the search process for matching points of multiple subgrids in an implementation method.

[0101] Figure 94This is a diagram used to illustrate the search process for matching points of multiple subgrids in an implementation method.

[0102] Figure 95 This is a flowchart illustrating the matching point search process in the implementation method.

[0103] Figure 96 This is a diagram used to illustrate the search process for matching points of multiple subgrids in an implementation method.

[0104] Figure 97 This is a diagram used to illustrate the search process for matching points of multiple subgrids in an implementation method.

[0105] Figure 98 This is a diagram illustrating an example of the syntax for signaling notification of subgrids using metadata in conjunction with an implementation method.

[0106] Figure 99 This is a diagram illustrating an example of the syntax for signaling notification of subgrids using metadata in conjunction with an implementation method.

[0107] Figure 100 This is a diagram illustrating an example of the syntax for signaling notification of subgrids using metadata in conjunction with an implementation method.

[0108] Figure 101 This is a diagram illustrating an example of signaling information about the combination of subgrids to be combined in an implementation.

[0109] Figure 102 This is a flowchart illustrating an example of basic encoding processing in an implementation method.

[0110] Figure 103 This is a flowchart illustrating an example of basic decoding processing in an implementation method. Detailed Implementation

[0111] <Introduction> For example, three-dimensional (3D) meshes are used in computer graphics. For example, computer graphics can also consist of multiple frames that are different in time, and each frame is represented by a three-dimensional mesh.

[0112] Furthermore, a 3D mesh consists of vertex information representing the positions of multiple vertices in 3D space, connection information representing the connections between these vertices, and attribute information representing the properties of each vertex or face. Each face is constructed according to the connection relationships between multiple vertices. Such a 3D mesh can be used to represent a wide variety of computer graphics and images.

[0113] Furthermore, efficient encoding and decoding of 3D meshes are desired for transmission and storage. Arithmetic encoding and decoding can be used for efficient encoding and decoding of 3D meshes.

[0114] There is a desire to further improve the encoding or decoding processing related to three-dimensional data. The purpose of this disclosure is to improve the encoding or decoding processing related to three-dimensional data.

[0115] The inventions obtained from the disclosure of this specification will be illustrated below, and the effects obtained from the inventions will be explained.

[0116] The encoding method in Example 1 generates multiple encoded data by encoding multiple subgrids. Based on the positions of multiple three-dimensional points contained in the multiple subgrids, threshold information is generated. This threshold information represents a threshold determined by the combination of two subgrids selected from the multiple subgrids. A bit stream containing the multiple encoded data and the threshold information is then generated.

[0117] A threshold is used, for example, in a decoding device that has acquired a bitstream to combine 3D points from multiple subgrids. Here, if the threshold is set solely based on the distance to the 3D point, for example, when multiple subgrids exist that need to be combined with the subgrid containing that 3D point, it's possible to combine 3D points from a subgrid that is different from the subgrid containing the 3D point that should be combined. Therefore, by determining the threshold for each combination of subgrids, the 3D points can be appropriately combined. Furthermore, for two uncombined subgrids, for example, by setting the threshold information to a specific value such as 0, the decoding device can omit the process of determining whether to combine each 3D point for those two subgrids. Therefore, the processing load of the decoding device can be reduced.

[0118] The encoding method in Example 2 can also be the same as the encoding method in Example 1, where the combination is determined based on the segmentation method that divides the three-dimensional mesh into the multiple sub-mesh.

[0119] Depending on the segmentation method (specifically, the segmentation approach), sometimes multiple subgrids are combined only with specific subgrids. In such cases, even without setting a threshold for all combinations of multiple subgrids, it is possible to appropriately combine multiple subgrids. Therefore, the processing required to determine the combination can be reduced. Additionally, the amount of information in the bitstream can be reduced.

[0120] The encoding method in Example 3 can also be the same as the encoding method in Example 2, where the combination is determined based on whether the segmentation method is a first segmentation method or a second segmentation method other than the first segmentation method. The first segmentation method is to segment between two subgrids that are adjacent subgrid identification numbers among the subgrid identification numbers that have been determined to be consecutively determined for the plurality of subgrids.

[0121] When a 3D mesh is divided into multiple submeshes, sometimes the 3D mesh is repeatedly divided at predetermined intervals in a predetermined direction, for example, by repeatedly dividing the 3D mesh at predetermined intervals in a direction perpendicular to a line drawn in one stroke, thereby generating multiple submeshes. In such cases, for multiple submeshes, submesh identification numbers are determined consecutively, for example, in the order they are generated from the division of the 3D mesh. According to such a division method, the submesh associated with a certain submesh is only the submesh adjacent to that submesh among the multiple submeshes. That is, if such a division method is used, the submesh associated with a certain submesh is only the submesh among the multiple submeshes whose identification number has been determined to be adjacent to the identification number determined for that submesh.

[0122] The encoding method in Example 4 can also be, in any of the encoding methods in Examples 1 to 3, calculating the distance between the two three-dimensional points for each group of two three-dimensional points, wherein the two three-dimensional points are obtained by selecting one from each of the multiple three-dimensional points possessed by the first sub-grid and the multiple three-dimensional points possessed by the second sub-grid in the two sub-grids of the combination, and calculating the longest distance in each group as the threshold of the combination.

[0123] Therefore, the first and second subgrids can be appropriately combined using a threshold.

[0124] The decoding method in Example 5 obtains a bitstream containing multiple encoded data and threshold information. The multiple encoded data are generated by encoding multiple subgrids. The threshold information represents a threshold determined by the combination of two subgrids selected from the multiple subgrids. The multiple encoded data are decoded, and a combination process is performed based on the threshold information to combine multiple three-dimensional points contained in the multiple subgrids with each other.

[0125] Therefore, by determining the threshold for each combination of subgrids, 3D points can be appropriately combined with each other. Furthermore, for two uncombined subgrids, for example, by setting the threshold information to a specific value such as 0, the process of determining whether to combine each 3D point can be omitted for those two subgrids. Thus, the processing workload can be reduced.

[0126] The decoding method of Example 6 can also be, in the decoding method of Example 5, in the combination process, according to each combination, a search process is performed under the condition that the specified conditions are met. The search process is the process of searching for a group of two three-dimensional points to be combined among the multiple three-dimensional points contained in the multiple sub-grids. Based on the threshold information, the two three-dimensional points of the searched group are combined.

[0127] Therefore, through search processing, it is possible to combine two appropriate three-dimensional points.

[0128] The decoding method in Example 7 can also be, in the decoding method in Example 6, in the combination process, for each combination, it is determined whether there is a group of two three-dimensional points to be combined among the multiple three-dimensional points contained in the two sub-grids of the combination. If it is determined that there is a group of two three-dimensional points to be combined among the multiple three-dimensional points contained in the two sub-grids of the combination, the search process is performed. If it is determined that there is no group of two three-dimensional points to be combined among the multiple three-dimensional points contained in the two sub-grids of the combination, the search process is not performed.

[0129] Therefore, when search processing is not required, it can be omitted, thus reducing the amount of processing required.

[0130] The decoding method in Example 8 can also be, in the decoding method in Example 7, based on the threshold information, determining whether there is a group of two three-dimensional points to be combined among the multiple three-dimensional points contained in the two sub-grids of the combination.

[0131] For two subgrids that are not combined, for example, if the threshold information is determined to be a specific value of 0, then it is possible to appropriately determine whether to combine the two subgrids based on whether the threshold represented by the threshold information is a specific value.

[0132] The encoding apparatus of Example 9 includes: a processor; and a memory, the processor using the memory to perform the following processing: generating multiple encoded data by encoding multiple subgrids; generating threshold information based on the positions of multiple three-dimensional points contained in the multiple subgrids, the threshold information representing a threshold determined by a combination of two subgrids selected from the multiple subgrids for each; and generating a bitstream containing the multiple encoded data and the threshold information.

[0133] This achieves the same effect as the encoding method in Technique 1.

[0134] The decoding apparatus of Example 10 includes: a processor; and a memory, the processor using the memory to perform the following processes: acquiring a bit stream containing multiple encoded data and threshold information, the multiple encoded data being generated by encoding multiple subgrids, the threshold information representing a threshold determined by a combination of two subgrids selected from the multiple subgrids for each; decoding the multiple encoded data; and performing a combination process based on the threshold information for combining multiple three-dimensional points contained in the multiple subgrids with each other.

[0135] This achieves the same effect as the encoding method in Technique 5.

[0136] Furthermore, these general or specific methods can be implemented by non-transitory recording media such as systems, devices, methods, integrated circuits, computer programs, or computer-readable CD-ROMs, or by any combination of systems, devices, methods, integrated circuits, computer programs, and recording media.

[0137] <Performance and Terminology> Here, the following expressions and terms are used.

[0138] (1) Three-dimensional mesh A 3D mesh is a collection of faces, representing, for example, a 3D object. Furthermore, a 3D mesh primarily consists of vertex information, connectivity information, and attribute information. 3D meshes sometimes appear as polygonal meshes or grids. Additionally, 3D meshes can exhibit temporal changes. A 3D mesh can contain metadata related to vertex, connectivity, and attribute information, as well as other additional information.

[0139] (2) Vertex information Vertex information represents information about vertices. For example, vertex information indicates the position of a vertex in three-dimensional space. Furthermore, vertices correspond to the vertices of the faces that make up a three-dimensional mesh. Vertex information is sometimes represented as "Geometry information." Additionally, vertex information can sometimes be represented as positional information.

[0140] (3) Connection information Connection information represents the connections between vertices. For example, connection information represents the connections between faces or edges used to form a 3D mesh. Connection information is sometimes expressed as "Connectivity". Furthermore, connection information can sometimes be expressed as face information.

[0141] (4) Attribute information Attribute information represents the properties of a vertex or face. For example, attribute information represents attributes such as color, image, and normal vector that correspond to a vertex or face. Attribute information is sometimes represented as "Texture".

[0142] (5) face A face is an element that makes up a 3D mesh. Specifically, a face is a polygon on a plane in 3D space. For example, a face can be defined as a triangle in 3D space.

[0143] (6) Plane A plane is a two-dimensional plane in three-dimensional space. For example, a polygon can be formed on a plane, and multiple polygons can be formed on multiple planes.

[0144] (7) Bit stream A bitstream corresponds to encoded information. A bitstream can also be represented as a stream, an encoded bitstream, a compressed bitstream, or an encoded signal.

[0145] (8) Encoding and Decoding Encoding can be replaced by other representations such as storing, containing, writing, recording, signaling, sending, notifying, saving, or compressing, and these representations can also be interchanged. For example, encoding information can also mean including information in a bitstream. Furthermore, encoding information into a bitstream can also mean encoding information to generate a bitstream containing the encoded information.

[0146] Furthermore, the act of decoding can be replaced by acts such as reading, interpreting, reading, inputting, exporting, obtaining, receiving, extracting, restoring, reconstructing, decompressing, or expanding, and these acts can also be interchanged. For example, decoding information can also mean obtaining information from a bitstream. Additionally, decoding information from a bitstream can also refer to decoding a bitstream to obtain the information contained within it.

[0147] (9) Ordinal number In the description, constituent elements are sometimes assigned ordinal numbers such as 1st and 2nd. These ordinal numbers can also be changed appropriately. Furthermore, constituent elements can be either newly assigned ordinal numbers or removed from their ordinal numbers. In addition, regarding these ordinal numbers, sometimes they are assigned to elements for identification purposes, and sometimes they do not correspond to a meaningful order.

[0148] 3D Mesh Figure 1 This is a conceptual diagram illustrating a three-dimensional mesh according to this embodiment. The three-dimensional mesh is composed of multiple faces. For example, each face is a triangle. The vertices of these triangles are defined in three-dimensional space. Furthermore, the three-dimensional mesh represents a three-dimensional object. Each face may also have color or an image.

[0149] Figure 2 This is a conceptual diagram representing the basic elements of a three-dimensional mesh according to this embodiment. The three-dimensional mesh consists of vertex information, connection information, and attribute information. Vertex information represents the position of the vertices of a face in three-dimensional space. Connection information represents the connections between vertices. Faces can be determined from vertex information and connection information. That is, a colorless three-dimensional object is formed in three-dimensional space using vertex information and connection information.

[0150] Attribute information can be mapped to vertices or faces. Attribute information mapped to vertices is sometimes represented as "Attribute Per Point". Attribute information mapped to vertices can represent the attributes of the vertex itself or the attributes of the faces connected to the vertex.

[0151] For example, color can be used as an attribute to establish a correspondence with vertices. The color corresponding to a vertex can be either the vertex's color or the color of the face connected to that vertex. The color of a face can also be the average of multiple colors corresponding to multiple vertices of that face. Furthermore, normal vectors can also be used as attribute information to establish a correspondence with vertices or faces. Such normal vectors can represent the front and back faces of a face.

[0152] Alternatively, a 2D image can be mapped to a face as attribute information. The 2D image mapped to a face is also represented as a texture image or an "Attribute Map". Furthermore, information representing the mapping between a face and a 2D image can also be mapped to a face as attribute information. This mapping information is sometimes represented as mapping information, vertex information of the texture image, texture coordinates, or "Attribute UV Coordinates".

[0153] Furthermore, information such as color, image, and motion image used as attribute information is sometimes represented as "ParametricSpace".

[0154] This attribute information allows textures to be reflected in three-dimensional objects. That is, by using vertex information, connectivity information, and attribute information, colored three-dimensional objects can be formed in three-dimensional space.

[0155] In addition, as described above, attribute information is associated with vertices or faces, but it can also be associated with edges.

[0156] Figure 3 This is a conceptual diagram illustrating the mapping method used in this embodiment. For example, a region of a two-dimensional image in a two-dimensional plane can be mapped onto the surface of a three-dimensional mesh in three-dimensional space. Specifically, the coordinate information of a region in the two-dimensional image is mapped to the surface of the three-dimensional mesh. Thus, the image of the region mapped in the two-dimensional image is reflected onto the surface of the three-dimensional mesh.

[0157] By using mapping, a two-dimensional image used as attribute information can be separated from a three-dimensional mesh. For example, in the encoding of a three-dimensional mesh, a two-dimensional image can also be encoded using image encoding or video encoding methods.

[0158] <System Composition> Figure 4 This is a block diagram illustrating an example of the configuration of the encoding / decoding system according to this embodiment. Figure 4 The encoding and decoding system includes an encoding device 100 and a decoding device 200.

[0159] For example, encoding device 100 acquires a three-dimensional mesh and encodes it into a bitstream. Furthermore, encoding device 100 outputs the bitstream to network 300. For example, the bitstream contains the encoded three-dimensional mesh and control information used to decode the encoded three-dimensional mesh. By encoding the three-dimensional mesh, the information of the three-dimensional mesh is compressed.

[0160] Network 300 transmits the bit stream from encoding device 100 to decoding device 200. Network 300 can be the Internet, a wide area network (WAN), a local area network (LAN), or a combination thereof. Network 300 is not necessarily limited to two-way communication; it can also be a one-way communication network used for terrestrial digital broadcasting or satellite broadcasting, etc.

[0161] Alternatively, the Network 300 can be replaced by recording media such as DVD (Digital Versatile Disc) or BD (Blu-Ray Disc (registered trademark)).

[0162] Decoding device 200 acquires a bitstream and decodes the 3D mesh from the bitstream. Through decoding the 3D mesh, the information of the 3D mesh is decompressed. For example, decoding device 200 decodes the 3D mesh according to a decoding method corresponding to the encoding method used by encoding device 100 to encode the 3D mesh. That is, encoding device 100 and decoding device 200 encode and decode according to mutually corresponding encoding and decoding methods.

[0163] Furthermore, the 3D mesh before encoding can also be represented as the original 3D mesh. Additionally, the 3D mesh after decoding can also be represented as a reconstructed 3D mesh.

[0164] <Encoding device> Figure 5 This is a block diagram illustrating a configuration example of the encoding apparatus 100 according to this embodiment. For example, the encoding apparatus 100 includes a vertex information encoder 101, a connection information encoder 102, and an attribute information encoder 103.

[0165] Vertex information encoder 101 is an electrical circuit that encodes vertex information. For example, vertex information encoder 101 encodes vertex information into a bit stream according to a specified format for vertex information.

[0166] The connection information encoder 102 is an electrical circuit that encodes connection information. For example, the connection information encoder 102 encodes the connection information into a bit stream according to a format specified for the connection information.

[0167] The attribute information encoder 103 is an electrical circuit that encodes attribute information. For example, the attribute information encoder 103 encodes attribute information into a bit stream according to a format specified for the attribute information.

[0168] In encoding vertex information, connectivity information, and attribute information, both variable-length and fixed-length encoding can be used. Variable-length encoding can also correspond to Huffman coding or context-adaptive binary arithmetic coding (CABAC).

[0169] The vertex information encoder 101, the connection information encoder 102, and the attribute information encoder 103 can also be integrated. Alternatively, the vertex information encoder 101, the connection information encoder 102, and the attribute information encoder 103 can each be further refined into multiple constituent elements.

[0170] Figure 6 This is a block diagram illustrating another configuration example of the encoding device 100 according to this embodiment. For example, the encoding device 100, in addition to... Figure 5 In addition to the configuration shown, it also includes a preprocessor 104 and a postprocessor 105.

[0171] The preprocessor 104 is an electrical circuit that processes vertex information, connectivity information, and attribute information before encoding. For example, the preprocessor 104 can also perform transformation, separation, or reuse processing on the 3D mesh before encoding. More specifically, the preprocessor 104 can also separate vertex information, connectivity information, and attribute information from the 3D mesh before encoding.

[0172] The post-processor 105 is an electrical circuit that processes the encoded vertex information, connectivity information, and attribute information. For example, the post-processor 105 can also perform transformation, separation, or multiplexing processing on the encoded vertex information, connectivity information, and attribute information. More specifically, for example, the post-processor 105 can also multiplex the encoded vertex information, connectivity information, and attribute information into a bitstream. Additionally, for example, the post-processor 105 can further perform variable-length encoding on the encoded vertex information, connectivity information, and attribute information.

[0173] <Decoding device> Figure 7 This is a block diagram illustrating a configuration example of the decoding apparatus 200 according to this embodiment. For example, the decoding apparatus 200 includes a vertex information decoder 201, a connection information decoder 202, and an attribute information decoder 203.

[0174] Vertex information decoder 201 is an electrical circuit that decodes vertex information. For example, vertex information decoder 201 decodes vertex information from a bitstream according to a specified format for vertex information.

[0175] The connection information decoder 202 is an electrical circuit that decodes connection information. For example, the connection information decoder 202 decodes connection information from a bitstream according to a format specified for the connection information.

[0176] The attribute information decoder 203 is an electrical circuit that decodes attribute information. For example, the attribute information decoder 203 decodes attribute information from a bitstream according to a format specified for the attribute information.

[0177] In decoding vertex information, connectivity information, and attribute information, both variable-length decoding and fixed-length decoding can be used. Variable-length decoding can also correspond to Huffman coding or context-adaptive binary arithmetic coding (CABAC).

[0178] The vertex information decoder 201, the connection information decoder 202, and the attribute information decoder 203 can also be integrated. Alternatively, the vertex information decoder 201, the connection information decoder 202, and the attribute information decoder 203 can each be further refined into multiple constituent elements.

[0179] Figure 8 This is a block diagram illustrating another configuration example of the decoding apparatus 200 according to this embodiment. For example, the decoding apparatus 200, in addition to... Figure 7 In addition to the configuration shown, it also includes a preprocessor 204 and a postprocessor 205.

[0180] The preprocessor 204 is an electrical circuit that processes vertex information, connectivity information, and attribute information before decoding. For example, the preprocessor 204 can also perform transformation, separation, or multiplexing processing on the bitstream before decoding vertex information, connectivity information, and attribute information.

[0181] More specifically, for example, preprocessor 204 can also separate sub-bitstreams corresponding to vertex information, connection information, and attribute information from the bitstream. Furthermore, for example, preprocessor 204 can also perform variable-length decoding on the bitstream before decoding the vertex information, connection information, and attribute information.

[0182] The post-processor 205 is an electrical circuit that processes vertex information, connectivity information, and attribute information after decoding. For example, the post-processor 205 can also perform transformation, separation, or multiplexing processing on the decoded vertex information, connectivity information, and attribute information. More specifically, the post-processor 205 can also multiplex the decoded vertex information, connectivity information, and attribute information into a 3D mesh.

[0183] <Bitstream> Vertex information, connectivity information, and attribute information are encoded and stored in a bitstream. The following illustrates the relationship between this information and the bitstream.

[0184] Figure 9 This is a conceptual diagram illustrating a configuration example of a bitstream according to this embodiment. In this example, connection information, vertex information, and attribute information are integrated within the bitstream. For example, connection information, vertex information, and attribute information may also be contained in a single file.

[0185] Alternatively, multiple parts of this information can be stored sequentially, such as the first part of the connection information, the first part of the vertex information, the first part of the attribute information, the second part of the connection information, the second part of the vertex information, the second part of the attribute information, and so on. These multiple parts can correspond to multiple parts that are different in time, multiple parts that are different in space, or multiple different faces.

[0186] Furthermore, the storage order of connection information, vertex information, and attribute information is not limited to the examples above, and a different storage order can also be used.

[0187] Figure 10 This is a conceptual diagram illustrating another configuration example of the bitstream according to this embodiment. In this example, the bitstream contains multiple files, and connection information, vertex information, and attribute information are stored in different files. Here, a file containing connection information, a file containing vertex information, and a file containing attribute information are shown, but the storage format is not limited to this example. For example, two types of information—connection information, vertex information, and attribute information—may be contained in one file, and the remaining type of information may be contained in other files.

[0188] Alternatively, this information can be split and stored in more files. For example, multiple parts of connection information, multiple parts of vertex information, and multiple parts of attribute information can be stored in multiple files. These multiple parts can correspond to multiple parts that differ in time, multiple parts that differ in space, or multiple faces.

[0189] Furthermore, the storage order of connection information, vertex information, and attribute information is not limited to the examples above, and a different storage order can also be used.

[0190] Figure 11 This is a conceptual diagram illustrating another configuration example of the bitstream according to this embodiment. In this example, the bitstream is composed of multiple separable sub-bitstreams, with connection information, vertex information, and attribute information stored in different sub-bitstreams.

[0191] Here, sub-bit streams containing connection information, sub-bit streams containing vertex information, and sub-bit streams containing attribute information are represented, but the storage format is not limited to this example.

[0192] For example, two types of information—connection information, vertex information, and attribute information—can be contained in one sub-bitstream, while the remaining type is contained in other sub-bitstreams. Specifically, the attribute information of a two-dimensional image, etc., can be stored in a sub-bitstream based on the image encoding method, different from the sub-bitstreams containing connection information and vertex information.

[0193] Furthermore, each sub-bitstream can contain multiple files. For example, multiple parts of connection information, multiple parts of vertex information, or multiple parts of attribute information can be stored in multiple files.

[0194] Furthermore, the storage order of connection information, vertex information, and attribute information is not limited to... Figure 9 , Figure 10 as well as Figure 11 The examples above can also use a different storage order. For instance, vertex information, connection information, and attribute information can be stored in the bitstream in the order of vertex information, connection information, and attribute information. Alternatively, any other order can be used in the bitstream, such as connection information, attribute information, and vertex information; vertex information, attribute information, and connection information; attribute information, connection information, and vertex information; or attribute information, vertex information, and connection information.

[0195] Alternatively, connection information, vertex information, and attribute information can be divided into multiple data segments, and these multiple data segments can be stored in a bitstream in a periodic or random order.

[0196] <Specific example> Figure 12 This is a block diagram illustrating a specific example of the encoding / decoding system according to this embodiment. Figure 12 The encoding and decoding system includes a three-dimensional data encoding system 110, a three-dimensional data decoding system 210, and an external connector 310.

[0197] The 3D data encoding system 110 includes a controller 111, an input / output processor 112, a 3D data encoder 113, a 3D data generator 115, and a system multiplexer 114. The 3D data decoding system 210 includes a controller 211, an input / output processor 212, a 3D data decoder 213, a system demultiplexer 214, a prompter 215, and a user interface 216.

[0198] In the 3D data encoding system 110, sensor data is input from the sensor terminal to the 3D data generator 115. The 3D data generator 115 generates 3D data, such as point group data or mesh data, based on the sensor data and inputs it to the 3D data encoder 113.

[0199] For example, the 3D data generator 115 generates vertex information and corresponding connection and attribute information. The 3D data generator 115 can also process vertex information while generating connection and attribute information. For example, the 3D data generator 115 can reduce the amount of data by deleting duplicate vertices or perform transformations on vertex information (position shifting, rotation, or normalization, etc.). Furthermore, the 3D data generator 115 can also render attribute information.

[0200] In addition, the 3D data generator 115 in Figure 12 The middle part is a component of the three-dimensional data encoding system 110, but it can also be configured independently outside the three-dimensional data encoding system 110.

[0201] Sensor terminals that provide sensor data for generating 3D data can be, for example, mobile objects such as cars, flying objects such as airplanes, portable terminals, or cameras. Additionally, distance sensors such as LiDAR, millimeter-wave radar, infrared sensors or rangefinders, stereo cameras, or combinations of multiple monocular cameras can also be used as sensor terminals.

[0202] Sensor data can also include the distance (position) of an object, images from a single-lens reflex camera, images from a stereo camera, color, reflectivity, sensor pose, orientation, gyroscope, sensed position (GPS information or altitude), speed, acceleration, sensed time, temperature, air pressure, humidity, or magnetism, etc.

[0203] 3D data encoder 113 corresponds to Figure 5 The encoding device 100 is shown in the figure. For example, the three-dimensional data encoder 113 encodes three-dimensional data to generate encoded data. In addition, the three-dimensional data encoder 113 generates control information in the encoding of the three-dimensional data. Furthermore, the three-dimensional data encoder 113 inputs the encoded data and the control information together to the system multiplexer 114.

[0204] Encoding 3D data can be done using either geometric methods or video codecs. Geometric encoding can be further categorized as geometry-based encoding, while video codec encoding can be categorized as video-based encoding.

[0205] The system multiplexer 114 multiplexes the encoded data and control information input from the 3D data encoder 113, generating multiplexed data using a prescribed multiplexing method. The system multiplexer 114 can also multiplex other media such as images, audio, subtitles, application data, or document files, or reference timing information, along with the encoded data and control information of the 3D data. Furthermore, the system multiplexer 114 can also multiplex attribute information associated with sensor data or 3D data.

[0206] For example, multiplexed data can be in file form for storage or packet form for transmission. ISOBMFF or ISOBMFF-based methods can also be used. Additionally, MPEG-DASH, MMT, MPEG-2 TSSystems, or RTP can also be used.

[0207] Furthermore, the input / output processor 112 outputs the multiplexed data as a transmission signal to the external connector 310. The multiplexed data, as a transmission signal, can be transmitted via wired or wireless means. Alternatively, the multiplexed data can be stored in internal memory or a storage device. The multiplexed data can also be transmitted to a cloud server via the Internet or stored in an external storage device.

[0208] For example, the transmission or storage of multiplexed data is carried out through methods corresponding to the medium used for transmission or storage, such as broadcasting or communication. As communication protocols, HTTP, FTP, TCP, UDP, IP, or combinations thereof can also be used. Furthermore, PULL-type communication methods and PUSH-type communication methods can also be used.

[0209] Wired transmission can utilize Ethernet (registered trademark), USB, RS-232C, HDMI (registered trademark), or coaxial cables. Wireless transmission can utilize 3GPP (registered trademark), IEEE-standard 3G / 4G / 5G, wireless LAN, Wi-Fi, Bluetooth, or millimeter wave. Furthermore, broadcast methods can include DVB-T2, DVB-S2, DVB-C2, ATSC 3.0, or ISDB-S3.

[0210] Alternatively, sensor data can be input to the 3D data generator 115 or the system multiplexer 114. Furthermore, 3D data or encoded data can be output as a transmission signal to the external connector 310 via the input / output processor 112. The transmission signal output from the 3D data encoding system 110 is input to the 3D data decoding system 210 via the external connector 310.

[0211] Furthermore, the various actions of the three-dimensional data encoding system 110 can also be controlled by the controller 111 that executes the application program.

[0212] In the 3D data decoding system 210, a transmission signal is input to the input / output processor 212. The input / output processor 212 decodes multiplexed data in file or packet form from the transmission signal and inputs the multiplexed data to the system demultiplexer 214. The system demultiplexer 214 obtains encoded data and control information from the multiplexed data and inputs it to the 3D data decoder 213. The system demultiplexer 214 can also obtain other media or reference time information from the multiplexed data.

[0213] 3D data decoder 213 corresponds to Figure 7 The decoding device 200 is shown in the figure. For example, the 3D data decoder 213 decodes 3D data from encoded data based on a predefined encoding method. Then, the prompter 215 displays the 3D data to the user.

[0214] In addition, additional information such as sensor data can be input to the prompter 215. The prompter 215 can also prompt 3D data based on the additional information. Furthermore, user instructions can be input from the user terminal to the user interface 216. And the prompter 215 can also prompt 3D data based on the input instructions.

[0215] In addition, the input / output processor 212 can also obtain three-dimensional data and encoded data from the external connector 310.

[0216] Furthermore, the various actions of the 3D data decoding system 210 can also be controlled by the controller 211 that executes the application program.

[0217] Figure 13 This is a conceptual diagram illustrating an example of the configuration of point group data in this embodiment. Point group data is data representing a group of points of a three-dimensional object.

[0218] Specifically, a point cloud consists of multiple points, possessing positional information representing the three-dimensional coordinates of each point, as well as attribute information representing the attributes of each point. The positional information is also represented by geometric information.

[0219] The categories of attribute information can be, for example, color or reflectivity. A mapping can be established between a single point and attribute information of one category, multiple different categories, or multiple values ​​for the same category for a single point.

[0220] Figure 14This is a conceptual diagram illustrating an example of a data file representing point group data according to this embodiment. In this example, there is a one-to-one correspondence between location information items and attribute information items, showing the location and attribute information of N points constituting the point group data. In this example, the location information is information representing three-dimensional coordinate positions using the x, y, and z axes, and the attribute information is information representing colors using RGB. A representative data file for point group data could be a PLY file, etc.

[0221] Figure 15 This is a conceptual diagram illustrating an example of the structure of the mesh data in this embodiment. The mesh data is data used in CG (Computer Graphics) and the like, and is three-dimensional mesh data that represents the three-dimensional shape of an object using multiple faces. Each face is also represented as a polygon, having a polygonal shape such as a triangle or quadrilateral.

[0222] Specifically, a 3D mesh consists of multiple points forming a point group, as well as multiple edges and faces. Each point is represented as a vertex or position. Each edge corresponds to a line segment connecting two vertices. Each face corresponds to a region enclosed by three or more edges.

[0223] In addition, 3D meshes possess positional information representing the 3D coordinates of vertices. This positional information is also represented as vertex information or geometric information. Furthermore, 3D meshes possess connectivity information, which represents the relationships between multiple vertices that constitute an edge or face. This connectivity information is also represented as connectivity. Moreover, 3D meshes possess attribute information representing properties for vertices, edges, or faces. This attribute information of 3D meshes is also represented as texture.

[0224] For example, attribute information can also represent the color, reflectivity, or normal vector of a vertex, edge, or face. The orientation of the normal vector can represent the front and back faces of a face.

[0225] As a data file format for grid data, object files, etc., can be used.

[0226] Figure 16 This is a conceptual diagram representing an example of a data file containing mesh data according to this embodiment. In this example, the data file contains position information G(1) to G(N) of N vertices constituting the 3D mesh and attribute information A1(1) to A1(N) of the N vertices. Furthermore, in this example, it contains M attribute information A2(1) to A2(M). The attribute information items may not correspond one-to-one with vertices or one-to-one with faces. Furthermore, attribute information may not exist at all.

[0227] Connection information is represented by a combination of vertex indices. n[1, 3, 4] represents the face of a triangle formed by three vertices: n=1, n=3, and n=4. Furthermore, m[2, 4, 6] represents the attribute information corresponding to m=2, m=4, and m=6 for the three vertices respectively.

[0228] In addition, the actual content of the attribute information can be recorded in other files. Furthermore, pointers to its content can be mapped to vertices or faces. For example, the attribute information representing the image of a face can be stored in a two-dimensional attribute map file. Furthermore, the filename of the attribute map and the two-dimensional coordinate values ​​in the attribute map can be recorded in attribute information A2(1) to A2(M). The methods for specifying attribute information for faces are not limited to these methods; any method can be used.

[0229] Figure 17 This is a conceptual diagram representing the types of three-dimensional data related to this embodiment. Point group data and mesh data can represent both static and dynamic objects. Static objects are objects that do not change over time, while dynamic objects are objects that change over time. Static objects can also correspond to three-dimensional data at any point in time.

[0230] For example, point group data at any given time point is sometimes represented as a PCC frame. Similarly, grid data at any given time point is sometimes represented as a grid frame. Furthermore, both PCC frames and grid frames are sometimes represented simply as frames.

[0231] Furthermore, the target area can be limited to a certain range, as with typical image data, or it can be unrestricted, as with map data. Additionally, the density of points or polygons can be set in various ways. Either sparse point clusters or sparse grid data, or dense point clusters or dense grid data, can be used.

[0232] Next, the encoding and decoding of point groups or 3D meshes will be described. The apparatus, processing, or syntax disclosed herein for encoding and decoding vertex information of 3D meshes can also be applied to the encoding and decoding of point groups.

[0233] Furthermore, the apparatus, processing, or syntax disclosed herein for encoding and decoding attribute information of point groups can also be applied to the encoding and decoding of connection information or attribute information of three-dimensional meshes.

[0234] Furthermore, at least some of the processing in the encoding and decoding of point group data and grid data can be shared. This allows for the reduction of the size of the circuit and software program.

[0235] Figure 18 This is a block diagram illustrating a configuration example of the 3D data encoder 113 according to this embodiment. In this example, the 3D data encoder 113 includes a vertex information encoder 121, an attribute information encoder 122, a metadata encoder 123, and a multiplexer 124. The vertex information encoder 121, the attribute information encoder 122, and the multiplexer 124 may also correspond to... Figure 6 Vertex information encoder 101, attribute information encoder 103, and post-processor 105, etc.

[0236] Furthermore, in this example, the 3D data encoder 113 encodes the 3D data using a geometry-based encoding method. In this geometry-based encoding method, the 3D structure is taken into account. Additionally, in this geometry-based encoding method, attribute information is encoded using the compositional information obtained from encoding the vertex information.

[0237] Specifically, the vertex information, attribute information, and metadata contained in the 3D data generated from the sensor data are first input to the vertex information encoder 121, attribute information encoder 122, and metadata encoder 123, respectively. Here, the connectivity information contained in the 3D data can be processed in the same way as the attribute information. Furthermore, in the case of point group data, the position information can also be processed as vertex information.

[0238] Vertex information encoder 121 encodes vertex information into compressed vertex information, and outputs the compressed vertex information as encoded data to multiplexer 124. Furthermore, vertex information encoder 121 generates metadata of the compressed vertex information and outputs it to multiplexer 124. Additionally, vertex information encoder 121 generates composition information and outputs it to attribute information encoder 122.

[0239] The attribute information encoder 122 uses the composition information generated by the vertex information encoder 121 to encode the attribute information into compressed attribute information, and outputs the compressed attribute information as encoded data to the multiplexer 124. In addition, the attribute information encoder 122 generates metadata of the compressed attribute information and outputs it to the multiplexer 124.

[0240] Metadata encoder 123 encodes compressible metadata into compressed metadata, and outputs the compressed metadata as encoded data to multiplexer 124. The metadata encoded by metadata encoder 123 can also be used for encoding vertex information and attribute information.

[0241] Multiplexer 124 multiplexes compressed vertex information, compressed vertex information metadata, compressed attribute information, compressed attribute information metadata, and compressed metadata into a bitstream. Furthermore, multiplexer 124 inputs the bitstream to the system layer.

[0242] Figure 19 This is a block diagram illustrating a configuration example of the 3D data decoder 213 according to this embodiment. In this example, the 3D data decoder 213 includes a vertex information decoder 221, an attribute information decoder 222, a metadata decoder 223, and a demultiplexer 224. The vertex information decoder 221, the attribute information decoder 222, and the demultiplexer 224 may also correspond to... Figure 8 Vertex information decoder 201, attribute information decoder 203, and preprocessor 204, etc.

[0243] Furthermore, in this example, the 3D data decoder 213 decodes the 3D data using a geometry-based encoding method. The 3D structure is considered in the geometry-based decoding. Additionally, in the geometry-based decoding, the attribute information is decoded using the compositional information obtained from decoding the vertex information.

[0244] Specifically, first, the bitstream is input from the system layer to the demultiplexer 224. The demultiplexer 224 separates compressed vertex information, compressed vertex information metadata, compressed attribute information, compressed attribute information metadata, and compressed metadata from the bitstream. The compressed vertex information and compressed vertex information metadata are input to the vertex information decoder 221. The compressed attribute information and compressed attribute information metadata are input to the attribute information decoder 222. The metadata is input to the metadata decoder 223.

[0245] Vertex information decoder 221 uses metadata from the compressed vertex information to decode vertex information. Furthermore, vertex information decoder 221 generates composition information and outputs it to attribute information decoder 222. Attribute information decoder 222 uses the composition information generated by vertex information decoder 221 and metadata from the compressed attribute information to decode attribute information. Metadata decoder 223 decodes metadata from the compressed metadata. The metadata decoded by metadata decoder 223 can also be used for decoding vertex information and attribute information.

[0246] Then, the vertex information, attribute information, and metadata are output as 3D data from the 3D data decoder 213. Additionally, this metadata, for example, is metadata of vertex information and attribute information, which can be used in the application.

[0247] Figure 20This is a block diagram illustrating another configuration example of the 3D data encoder 113 according to this embodiment. In this example, the 3D data encoder 113 includes a vertex image generator 131, an attribute image generator 132, a metadata generator 133, an image encoder 134, a metadata encoder 123, and a multiplexer 124. The vertex image generator 131, the attribute image generator 132, and the image encoder 134 may also correspond to... Figure 6 Vertex information encoder 101 and attribute information encoder 103, etc.

[0248] Furthermore, in this example, the 3D data encoder 113 encodes the 3D data using a video-based coding method. In this video-based coding method, multiple 2D images are generated from the 3D data, and these images are then encoded using an image coding method. Here, the image coding method could also be HEVC (High Efficiency Video Coding) or VVC (Versatile Video Coding), etc.

[0249] Specifically, the vertex and attribute information contained in the 3D data generated from the sensor data are first input into the metadata generator 133. Furthermore, the vertex and attribute information are input into the vertex image generator 131 and the attribute image generator 132, respectively. Additionally, the metadata contained in the 3D data is input into the metadata encoder 123. Here, the connectivity information contained in the 3D data can be processed in the same way as the attribute information. Furthermore, in the case of point group data, the position information can also be processed as vertex information.

[0250] Metadata generator 133 generates mapping information for multiple two-dimensional images based on vertex information and attribute information. Furthermore, metadata generator 133 inputs the mapping information into vertex image generator 131, attribute image generator 132, and metadata encoder 123.

[0251] Vertex image generator 131 generates vertex images based on vertex information and mapping information and inputs them to image encoder 134. Attribute image generator 132 generates attribute images based on attribute information and mapping information and inputs them to image encoder 134.

[0252] The image encoder 134 encodes the vertex image and attribute image into compressed vertex information and compressed attribute information respectively according to the image encoding method, and outputs the compressed vertex information and compressed attribute information as encoded data to the multiplexer 124. In addition, the image encoder 134 generates metadata of compressed vertex information and metadata of compressed attribute information, and outputs them to the multiplexer 124.

[0253] Metadata encoder 123 encodes compressible metadata into compressed metadata, and outputs the compressed metadata as encoded data to multiplexer 124. The compressible metadata contains mapping information. In addition, the metadata encoded by metadata encoder 123 can also be used for encoding vertex information and attribute information.

[0254] Multiplexer 124 multiplexes compressed vertex information, compressed vertex information metadata, compressed attribute information, compressed attribute information metadata, and compressed metadata into a bitstream. Furthermore, multiplexer 124 inputs the bitstream to the system layer.

[0255] Figure 21 This is a block diagram illustrating another configuration example of the 3D data decoder 213 according to this embodiment. In this example, the 3D data decoder 213 includes a vertex information generator 231, an attribute information generator 232, an image decoder 234, a metadata decoder 223, and a demultiplexer 224. The vertex information generator 231, the attribute information generator 232, and the image decoder 234 may also correspond to... Figure 8 Vertex information decoder 201 and attribute information decoder 203, etc.

[0256] Furthermore, in this example, the 3D data decoder 213 decodes the 3D data according to a video-based encoding method. In this video-based decoding, multiple 2D images are decoded according to an image encoding method to generate 3D data from the multiple 2D images. Here, the image encoding method could also be HEVC (High Efficiency Video Coding) or VVC (Versatile Video Coding), etc.

[0257] Specifically, first, the bitstream is input from the system layer to the demultiplexer 224. The demultiplexer 224 separates compressed vertex information, compressed vertex information metadata, compressed attribute information, compressed attribute information metadata, and compressed metadata from the bitstream. The compressed vertex information and compressed vertex information metadata, compressed attribute information, and compressed attribute information metadata are then input to the image decoder 234. The compressed metadata is then input to the metadata decoder 223.

[0258] The image decoder 234 decodes the vertex image according to the image encoding method. At this time, the image decoder 234 uses the metadata of the compressed vertex information to decode the vertex image from the compressed vertex information. Furthermore, the image decoder 234 inputs the vertex image to the vertex information generator 231. In addition, the image decoder 234 decodes the attribute image according to the image encoding method. At this time, the image decoder 234 uses the metadata of the compressed attribute information to decode the attribute image from the compressed attribute information. Furthermore, the image decoder 234 inputs the attribute image to the attribute information generator 232.

[0259] Metadata decoder 223 decodes metadata from compressed metadata. The metadata decoded by metadata decoder 223 contains mapping information used in the generation of vertex information and attribute information. In addition, the metadata decoded by metadata decoder 223 can also be used for the decoding of vertex images and attribute images.

[0260] Vertex information generator 231 reconstructs vertex information from the vertex image according to the mapping information contained in the metadata decoded by metadata decoder 223. Attribute information generator 232 reconstructs attribute information from the attribute image according to the mapping information contained in the metadata decoded by metadata decoder 223.

[0261] Then, the vertex information, attribute information, and metadata are output as 3D data from the 3D data decoder 213. Additionally, this metadata, for example, is metadata of vertex information and attribute information, which can be used in the application.

[0262] Figure 22 This is a conceptual diagram illustrating a specific example of the encoding process related to this embodiment. Figure 22 The diagram illustrates a 3D data encoder 113 and a descriptive encoder 148. In this example, the 3D data encoder 113 includes a 2D data encoder 141 and a mesh data encoder 142. The 2D data encoder 141 includes a texture encoder 143. The mesh data encoder 142 includes a vertex information encoder 144 and a connectivity information encoder 145.

[0263] Vertex information encoder 144, connection information encoder 145, and texture encoder 143 can also correspond to Figure 6 Vertex information encoder 101, connection information encoder 102, and attribute information encoder 103, etc.

[0264] For example, the two-dimensional data encoder 141 acts as the texture encoder 143, generating a texture file by encoding the texture corresponding to the attribute information as two-dimensional data according to the image encoding method or video encoding method.

[0265] Furthermore, the mesh data encoder 142 operates as a vertex information encoder 144 and a connectivity information encoder 145, generating a mesh file by encoding vertex information and connectivity information. The mesh data encoder 142 can also encode texture mapping information. Furthermore, the encoded mapping information can be included in the mesh file.

[0266] Furthermore, the description encoder 148 can also generate a description file by encoding descriptions corresponding to metadata, such as text data. The description encoder 148 can also encode the description at the system level. For example, the description encoder 148 can also be included in... Figure 12 In the system multiplexer 114.

[0267] The above steps generate a bitstream containing texture files, mesh files, and description files. These files can also be multiplexed into the bitstream in formats such as glTF (Graphics Language Transmission Format) or USD (Universal Scene Description).

[0268] Alternatively, the 3D data encoder 113 can also have two mesh data encoders, which can function as mesh data encoders 142. For example, one mesh data encoder encodes the vertex and connection information of a static 3D mesh, while the other mesh data encoder encodes the vertex and connection information of a dynamic 3D mesh.

[0269] Furthermore, two mesh files can be included in the bitstream accordingly. For example, one mesh file corresponds to a static 3D mesh, and the other mesh file corresponds to a dynamic 3D mesh.

[0270] Furthermore, a static 3D mesh can also be an intra-frame 3D mesh using intra-frame predictive coding, and a dynamic 3D mesh can also be an inter-frame 3D mesh using inter-frame predictive coding. Additionally, as information for a dynamic 3D mesh, the difference between the vertex or connectivity information of the intra-frame 3D mesh and the vertex or connectivity information of the inter-frame 3D mesh can also be used.

[0271] Figure 23 This is a conceptual diagram illustrating a specific example of the decoding process described in this embodiment. Figure 23 The code represents a 3D data decoder 213, a description decoder 248, and a prompter 247. In this example, the 3D data decoder 213 includes a 2D data decoder 241, a mesh data decoder 242, and a mesh reconstructor 246. The 2D data decoder 241 includes a texture decoder 243. The mesh data decoder 242 includes a vertex information decoder 244 and a connectivity information decoder 245.

[0272] Vertex information decoder 244, connection information decoder 245, texture decoder 243, and mesh reconstructor 246 can also correspond to Figure 8 The vertex information decoder 201, connection information decoder 202, attribute information decoder 203, and post-processor 205, etc. The prompter 247 can also correspond to... Figure 12 The prompt device 215, etc.

[0273] For example, the two-dimensional data decoder 241 acts as the texture decoder 243, decoding the texture corresponding to the attribute information from the texture file into two-dimensional data according to the image encoding method or video encoding method.

[0274] Furthermore, the mesh data decoder 242 operates as both the vertex information decoder 244 and the connectivity information decoder 245, decoding vertex and connectivity information from the mesh file. The mesh data decoder 242 can also decode texture mapping information from the mesh file.

[0275] Furthermore, the description decoder 248 decodes descriptions corresponding to metadata, such as text data, from the description file. The description decoder 248 can also decode descriptions at the system layer. For example, the description decoder 248 may also be included in... Figure 12 In the system demultiplexer 214.

[0276] Mesh Reconstructor 246, as described, reconstructs the 3D mesh from vertex information, connectivity information, and textures. Hint 247, as described, renders and outputs the 3D mesh.

[0277] Through the above actions, a 3D mesh is reconstructed and output from the bitstream containing texture files, mesh files, and description files.

[0278] Alternatively, the 3D data decoder 213 can also have two mesh data decoders, which can function as mesh data decoders 242. For example, one mesh data decoder decodes the vertex and connection information of a static 3D mesh, while the other mesh data decoder decodes the vertex and connection information of a dynamic 3D mesh.

[0279] Furthermore, two mesh files can be included in the bitstream in a corresponding manner. For example, one mesh file corresponds to a static 3D mesh, and the other mesh file corresponds to a dynamic 3D mesh.

[0280] Furthermore, a static 3D mesh can also be an intra-frame 3D mesh encoded using intra-frame prediction, and a dynamic 3D mesh can also be an inter-frame 3D mesh encoded using inter-frame prediction. Additionally, as information for a dynamic 3D mesh, differential information between the vertex or connectivity information of the intra-frame 3D mesh and the vertex or connectivity information of the inter-frame 3D mesh can be used.

[0281] The coding method for dynamic 3D meshes is sometimes called DMC (Dynamic Mesh Coding). In addition, the video-based coding method for dynamic 3D meshes is sometimes called V-DMC (Video-based Dynamic Mesh Coding).

[0282] Point cloud compression is sometimes referred to as PCC. Additionally, video-based point cloud compression is sometimes called V-PCC (Video-based Point Cloud Compression). Furthermore, geometry-based point cloud compression is sometimes called G-PCC (Geometry-based Point Cloud Compression).

[0283] <Installation Example> Figure 24 This is a block diagram illustrating an installation example of the encoding device 100 according to this embodiment. The encoding device 100 includes circuitry 151 and a memory 152. For example, Figure 5 The multiple components of the encoding device 100 shown are composed of... Figure 24 The circuit 151 and memory 152 shown are installed.

[0284] Circuit 151 is a circuit that performs information processing and is capable of accessing memory 152. For example, circuit 151 is a dedicated or general-purpose electrical circuit that encodes a three-dimensional mesh. Circuit 151 can also be a processor like a CPU. Furthermore, circuit 151 can also be a collection of multiple electrical circuits.

[0285] Memory 152 is a dedicated or general-purpose memory for storing information used by circuit 151 to encode a three-dimensional mesh. Memory 152 can also be an electrical circuit and can be connected to circuit 151. Furthermore, memory 152 can also be included within circuit 151. Additionally, memory 152 can be an assembly of multiple electrical circuits. Furthermore, memory 152 can also be a magnetic disk or optical disk, or can be manifested as a storage device or recording medium. Furthermore, memory 152 can be either non-volatile or volatile memory.

[0286] For example, memory 152 can store both a three-dimensional mesh and a bit stream. Furthermore, memory 152 can also store a program for circuit 151 to encode the three-dimensional mesh.

[0287] Furthermore, in the encoding device 100, it is not necessary to install... Figure 5 All of the multiple constituent elements shown here may also be exempt from the multiple processes indicated here. Figure 5A portion of the multiple constituent elements shown may also be included in other devices, and a portion of the multiple processes shown herein may also be executed by other devices. Furthermore, in the encoding device 100, the multiple constituent elements of this disclosure may be arbitrarily combined and installed, and the multiple processes of this disclosure may be arbitrarily combined and performed.

[0288] Figure 25 This is a block diagram illustrating an installation example of the decoding device 200 according to this embodiment. The decoding device 200 includes circuitry 251 and a memory 252. For example, Figure 7 The decoding device 200 shown in the figure consists of multiple components. Figure 25 The circuit 251 and memory 252 shown are installed.

[0289] Circuit 251 is a circuit that performs information processing and is capable of accessing memory 252. For example, circuit 251 is a dedicated or general-purpose electrical circuit that decodes a 3D mesh. Circuit 251 can also be a processor like a CPU. Furthermore, circuit 251 can also be a collection of multiple electrical circuits.

[0290] Memory 252 is a dedicated or general-purpose memory that stores information used by circuit 251 to decode the three-dimensional mesh. Memory 252 can be an electrical circuit or connected to circuit 251. Furthermore, memory 252 can also be included within circuit 251. Additionally, memory 252 can be an assembly of multiple electrical circuits. Furthermore, memory 252 can be a disk or optical disk, or it can be represented as a storage device or recording medium. Furthermore, memory 252 can be either non-volatile or volatile memory.

[0291] For example, memory 252 can store either a three-dimensional grid or a bit stream. Furthermore, memory 252 can also store a program for circuit 251 to decode the three-dimensional grid.

[0292] Alternatively, it is not necessary to install it in the decoding device 200. Figure 7 All of the multiple constituent elements shown here may also be exempt from the multiple processes indicated here. Figure 7 A portion of the multiple constituent elements shown may also be included in other devices, and a portion of the multiple processes shown herein may also be executed by other devices. Furthermore, in the decoding device 200, the multiple constituent elements of this disclosure may be arbitrarily combined and installed, and the multiple processes of this disclosure may be arbitrarily combined and performed.

[0293] The encoding and decoding methods, which include the steps performed by the constituent elements of the encoding apparatus 100 and decoding apparatus 200 of this disclosure, can also be executed by any device or system. For example, part or all of the encoding and decoding methods can be executed by a computer equipped with a processor, memory, and input / output circuits. In this case, the encoding and decoding methods can also be executed by the computer executing a program used to cause the computer to perform the encoding and decoding methods.

[0294] Furthermore, non-transitory computer-readable recording media such as CD-ROMs can record both programs and bit streams.

[0295] An example of a program can also be a bitstream. For instance, a bitstream containing an encoded 3D mesh includes syntactic elements that enable the decoding device 200 to decode the 3D mesh. Furthermore, the bitstream enables the decoding device 200 to decode the 3D mesh according to the syntactic elements contained in the bitstream. Therefore, a bitstream can function similarly to a program.

[0296] The bitstream mentioned above can be either an encoded bitstream containing the encoded three-dimensional grid or a multiplexed bitstream containing the encoded three-dimensional grid and other information.

[0297] Furthermore, the components of the encoding device 100 and the decoding device 200 can be constructed from dedicated hardware, general-purpose hardware that executes the aforementioned programs, or a combination thereof. Additionally, the general-purpose hardware can consist of a memory storing the program and a general-purpose processor that reads the program from the memory and executes it. Here, the memory can be a semiconductor memory or a hard disk, and the general-purpose processor can be a CPU.

[0298] Alternatively, dedicated hardware can consist of memory and a dedicated processor. For example, a dedicated processor can also execute encoding and decoding methods with reference to the memory used to record data.

[0299] Furthermore, the constituent elements of the encoding device 100 and the decoding device 200 can also be electrical circuits as described above. These electrical circuits can be configured as a single electrical circuit or as separate electrical circuits. Moreover, these electrical circuits can correspond to dedicated hardware or general-purpose hardware that executes the aforementioned programs. Furthermore, the encoding device 100 and the decoding device 200 can also be integrated as integrated circuits.

[0300] Alternatively, the encoding device 100 can also be a transmitting device for transmitting a three-dimensional mesh. The decoding device 200 can also be a receiving device for receiving a three-dimensional mesh.

[0301] <Displacement Encoding and Decoding> Here, as an example, the following terminology will be used.

[0302] (1) Image An image is a unit of data consisting of a collection of pixels, containing either a picture or a block smaller than a picture. Images include both moving images and still images.

[0303] (2) Pictures An image is a unit of image processing consisting of a collection of pixels, also known as a frame or field.

[0304] (3) blocks A block is a processing unit consisting of a specific number of pixels. Blocks are also referred to using the terms shown in the examples below. The shape of a block is not particularly limited. A block can be, for example, an M×N pixel rectangle or an M×M pixel square. Blocks can also be triangular, circular, or other shapes. Examples of blocks are described below.

[0305] • Slices, tiles, or bricks • CTU, superblock, or basic partitioning unit • VPDU, Hardware Processing Division Unit • CU, Processing Block Unit, Prediction Block Unit (PU), or Orthogonal Transform Block Unit (TU) ·Sub-block (4) Pixels or samples A pixel or sample is the smallest point in an image; in other words, it is the smallest unit. A pixel or sample includes not only pixels at integer positions but also pixels at sub-pixel positions generated based on those integer-position pixels.

[0306] (5) Pixel value or sample value Pixel values ​​or sample values ​​are the inherent values ​​of a pixel. Pixel values ​​or sample values ​​include luminance values, chrominance values, RGB grayscale levels, and may also include depth values ​​or binary values ​​of 0 or 1.

[0307] (6) Sign A flag represents more than one bit. For example, a flag may be a parameter or index represented by two or more bits. Flags not only represent values ​​expressed in binary numbers, but sometimes also values ​​expressed in numeric values ​​other than binary numbers.

[0308] (7) Signal Signals are symbolized or encoded to convey information. Signals include discrete digital signals or continuous analog signals.

[0309] (8) Stream or bitstream A stream or bitstream is a sequence of digital data that represents the flow of digital data. A stream or bitstream can be a single stream or can be structured as multiple streams with multiple levels. Streams or bitstreams can be transmitted using a single transmission path via serial communication or using multiple transmission paths via packet communication.

[0310] (9) Difference In the case of scalars, differences can include simple differences (xy) and difference calculations. Differences can include the absolute value of the difference (|xy|), the square of the difference (x^2-y^2), the square root of the difference (√(xy)), the weighted difference (ax-by, where a and b are constants), or the offset difference (x-y+a, where a is the offset), etc.

[0311] (10) and In the case of scalars, sums can include simple sums (x+y) and addition calculations. Sums can include the absolute value of the sum (|x+y|), the sum of squares (x^2+y^2), the square root of the sum (√(x+y)), the weighted sum (ax+by, where a and b are constants), or the offset sum (x+y+a, where a is the offset), etc.

[0312] (11) "based on" The phrase "based on" implies that objects other than "that" can also be considered. Additionally, "based on" is sometimes used when a direct result is obtained, or when the result is obtained through intermediate steps.

[0313] (12) "Being used" or "Using" Expressions like "what was used" or "what was used" imply that things other than "what" can also be considered. Furthermore, expressions like "was used" or "was used" are sometimes used when a direct result is obtained, or when a result is obtained through intermediate steps.

[0314] (13) Prohibition "Prohibition" can be translated as "not permitted". In addition, "not prohibited / prohibited" or "permitted / permitted" does not necessarily mean "obligation".

[0315] (14) “Restriction” or “Limitation” "Restriction" or "limitation" can be translated as "not permitted / not allowed" or "not allowed / permitted." Furthermore, "prohibited / not prohibited" or "not permitted / permitted" does not necessarily imply an "obligation." Additionally, what is prohibited in quantity / quality can be partial or complete.

[0316] (15) Chroma The term chroma, represented by the symbols Cb or Cr, is an adjective that indicates one of two color difference signals associated with a primary color in a sample arrangement or a single sample. The term chroma is sometimes used in place of chrominance.

[0317] (16) Luminance The term "luma" is an adjective denoted by the symbol or subscript Y or L, representing the monochromatic signal of a sample arrangement or a single sample that is related to the primary color. The term "luma" is sometimes used in place of "luminance".

[0318] The encoding and decoding system of this embodiment will be described below.

[0319] A typical 3D model (also known as a 3D model) represents an object digitally, allowing users to search the model in 3D using scaling, translation, and rotation while rendering over time. One method of constructing such a representation is to use triangles to build a 3D mesh. In the model described above, the positions of the triangle vertices, the connectivity between the triangle vertices, and the properties associated with them (normals or UV mapping, etc.) are stored.

[0320] To store all this information in uncompressed form, a very large storage area is required, thus requiring a very large bandwidth for transmission. The triangles forming the mesh, especially in temporal and spatial proximity, mostly exhibit repeating patterns and similar properties. Utilizing these repetitions, efficient encoding and decoding methods can be developed for storage and transmission. One such encoding and decoding method is Video-based Dynamic Mesh Coding (V-DMC).

[0321] Figure 26 This is a block diagram illustrating another configuration example of the encoding / decoding system according to this embodiment. For example... Figure 26 As shown, the encoding and decoding system includes an encoding device 100 and a decoding device 200.

[0322] The encoding / decoding system accepts a 3D mesh (also known as a 3D mesh) as input in the form of vertices' 3D coordinates (vertices information), connectivity (connectivity information), and associated attributes (attribute information). Furthermore, a 3D mesh can include not only geometric information but also texture maps.

[0323] The encoding device 100 takes the input 3D mesh (also called the input 3D mesh or input grid) in the form of the three-dimensional coordinates of the vertices, connectivity, and associated attributes. The encoding device 100 encodes all associated information into a stream. The stream can consist of a single bit stream or multiple bit streams.

[0324] Network 300 transmits the stream generated by encoding device 100 to decoding device 200. Network 300 can be the Internet, WAN (Wide Area Network), LAN (Local Area Network), or any combination thereof. Furthermore, network 300 is not necessarily limited to a two-way communication network; it can also be a one-way communication network transmitting broadcast waves such as terrestrial digital broadcasts or satellite broadcasts. Alternatively, recording media such as DVD (Digital Versatile Disc) or BD (Blu-ray Disc) can be used instead of network 300.

[0325] The bitstream is sent to the decoding device 200 via network 300. The decoding device 200 decodes the bitstream and uses the 3D coordinates, connectivity, and association properties of the decoded vertices to generate a 3D mesh. The decoding device 200 outputs the generated 3D mesh (also known as the output 3D mesh or output grid).

[0326] Figure 27 This is a diagram showing another configuration example of the encoding device 100.

[0327] like Figure 27 As shown, the encoding device 100 includes a preprocessor 1103 and a compressor 1106.

[0328] Encoding device 100 reads input mesh 1101 and attribute map 1102 and passes them to preprocessor 1103. Preprocessor 1103 processes the input mesh to extract base mesh 1104 and displacement data 1105. Attribute map 1102, along with the extracted base mesh 1104 and displacement data 1105, is then transmitted to compressor 1106.

[0329] Furthermore, the compressor 1106 compresses the base mesh 1104, displacement data 1105, and attribute map 1102 to generate a bitstream 1107. By also including metadata 1108 in the bitstream 1107, the compressor 1106 can send additional information to the decoding device 200.

[0330] Figure 28 This is a diagram showing another configuration example of the decoding device 200.

[0331] like Figure 28As shown, the decoding device 200 includes a decompressor 2102 and a post-processor 2106.

[0332] Decoding device 200 reads bitstream 2101 and passes it to decompressor 2102. Decompressor 2102 decompresses base mesh 2103, displacement data 2104, and attribute map 2108 from bitstream 2101 and passes them to postprocessor 2106. An example of displacement data 2104 is a displacement vector.

[0333] Furthermore, the post-processor 2106 generates the output mesh 2107 by processing the base mesh 2103 according to the displacement data 2104 and the attribute map 2108. The post-processor 2106 can also use information from the metadata 2105 to generate the output mesh 2107.

[0334] Figure 29 This is a block diagram illustrating yet another configuration example of the encoding device 100 according to this embodiment.

[0335] In this example, the encoding device 100 includes a volume capture device 511, a projector 512, a basic mesh encoder 513, a displacement encoder 514, an attribute encoder 515, and one or more other types of encoders 516 as options.

[0336] Volume capturer 511 captures the content and outputs the captured content to projector 512.

[0337] Projector 512 projects content onto an input mesh (3D mesh frame) containing geometric coordinates (vertex coordinates representing the position of vertices), texture coordinates, and connectivity (connection information). This data is output to a basic mesh encoder 513, a translation encoder 514, an attribute encoder 515, and one or more other types of encoders 516 as options. Each encoder compresses the data into a bitstream.

[0338] Figure 30 This is a block diagram illustrating yet another configuration example of the decoding device 200 according to this embodiment.

[0339] In this example, the decoding device 200 includes a basic mesh decoder 613, a displacement decoder 614, an attribute decoder 615, one or more other types of decoders 616, and a 3D reconstructor 617.

[0340] The bitstream is sent to a base mesh decoder 613, a displacement decoder 614, an attribute decoder 615, and one or more other types of decoders 616 as options. These decoders decode the bitstream to generate data (decoded data) containing geometric coordinates, texture coordinates, and connectivity. The decoded data is then sent to a 3D reconstructor 617 to reconstruct the output mesh (3D mesh frame).

[0341] The encoding process performed by the encoding device 100 will be described in detail below.

[0342] Figure 31 This is a flowchart illustrating the processing of the encoding device 100. Figure 32 This is a conceptual illustration of the encoding of a grid frame. (See reference...) Figure 31 and Figure 32 Explain the processing of the encoding device 100.

[0343] In step S101, the encoding device 100 reads a 3D mesh frame and its attributes as an input mesh frame. The input mesh frame is the mesh frame input to the encoding device 100. An example of a 3D mesh frame as an input mesh frame is shown as mesh frame 1301 (see reference). Figure 32 ).

[0344] In step S102, the encoding device 100 performs decimation processing on the input mesh frame read in step S101 to generate a base mesh frame with fewer vertices than the input mesh frame. The base mesh frame generated by decimating mesh frame 1301 is represented as base mesh frame 1302 (see reference). Figure 32 ).

[0345] In step S103, the encoding device 100 calculates the displacement information used by the decoding device 200 to reconstruct the mesh frame. The displacement information is equivalent to the displacement vector from the vertices of the base mesh frame generated in step S102 towards the vertices of the input mesh frame. One method for calculating the displacement information is to subtract the coordinates of the vertices of the base mesh frame from the coordinates of the vertices of the input mesh frame. The displacement information calculated based on mesh frame 1301 and base mesh frame 1302 is represented as displacement information 1303 (see reference). Figure 32 The displacement information 1303 is in vector form; in other words, it is represented as a displacement vector.

[0346] In step S104, the encoding device 100 encodes the base grid frame generated in step S102, the displacement information generated in step S103, and the attributes of the input grid frame into a bitstream (equivalent to a compressed bitstream). An example of this bitstream is shown as bitstream 1304 (see reference). Figure 32 ).

[0347] Specifically, bitstream 1304 contains vertex coordinates and connectivity information of vertices A, C, E, and F, displacement information, a video bitstream containing texture data, and a compressed attribute map (see reference). Figure 32 The displacement information contains displacement information used to displace vertices based on vertex coordinates obtained from the base mesh frame after sub-segmentation. The compressed attribute map is the texture coordinates used to apply texture data to the mesh frame reconstructed using the base mesh frame and displacement information.

[0348] The decoding process performed by the decoding device 200 will be described below.

[0349] Figure 33 This is a flowchart illustrating the processing of the decoding device 200. Figure 34 This is a conceptual illustration of the decoding of a mesh frame (3D mesh). (See reference...) Figure 33 and Figure 34 Explain the processing of the decoding device 200.

[0350] In step S201, the decoding device 200 decodes the base grid frame and attributes from the bitstream (equivalent to a compressed bitstream). An example of a decoded base grid frame (equivalent to a decoded base grid frame) is shown as decoded base grid frame 2301 (see reference). Figure 34 ).

[0351] In step S202, the decoding device 200 generates sub-segmented vertices by sub-segmenting the base mesh frame decoded in step S201. An example of a base mesh frame including sub-segmented vertices is represented as base mesh frame (mesh frame) 2302 (see reference). Figure 34 ).

[0352] In step S203, the decoding device 200 decodes the displacement information from the bitstream (equivalent to a compressed bitstream). An example of the decoded displacement information is represented as displacement information 2303 (refer to...). Figure 34 The displacement information 2303 is in vector form; in other words, it is represented as a displacement vector.

[0353] In step S204, the decoding device 200 reconstructs the shape of the mesh frame by moving the vertices of the base mesh frame, including the sub-segmented vertices, to new positions using displacement information, and then restores the mesh frame by applying attribute information. One example of an attribute is texture. An example of the reconstructed mesh frame is represented as mesh frame 2304 (see reference). Figure 34 ).

[0354] Figure 35 This is a block diagram illustrating an example of the configuration of the decoding device according to this embodiment.

[0355] Figure 35 This is an example of a block diagram representing a typical intra-frame decoding.

[0356] Figure 35 The decoding device shown includes a demultiplexer 1231, a switch 1232, a static grid decoder 1233, a grid buffer 1234, a motion decoder 1235, a basic grid reconstructor 1236, an inverse quantizer 1237, a video decoder 1238, an image unpacker 1239, an inverse quantizer 1240, an inverse wavelet transform 1241, a reconstructor 1242, a video decoder 1243, and a color converter 1244.

[0357] Demultiplexer 1231 obtains the compressed bitstream and separates the compressed data related to the base grid, the image containing displacement data (also known as the displacement bitstream), and the image containing attribute data (also known as the attribute bitstream). The compressed data related to the base grid is transmitted to switch 1232. Switch 1232 determines whether to perform intra-frame decoding or inter-frame decoding based on the parameters in the bitstream.

[0358] When intra-frame decoding is selected, the bitstream is transmitted to a static mesh decoder 1233 that generates a quantized base mesh. The static mesh decoder 1233 is, for example, a decoder that uses the Edgebreaker algorithm in decoding 3D mesh data. The static mesh decoder 1233 generates a quantized base mesh from the bitstream. The quantized base mesh generated by the static mesh decoder 1233 is stored in a mesh buffer 1234 for reference when inter-frame decoding is selected.

[0359] When inter-frame decoding is selected, switch 1232 passes compressed data related to the base mesh to motion decoder 1235. Motion decoder 1235 receives the previously decoded quantized base mesh and decodes the motion data, which represents the difference in vertex coordinates between the quantized base mesh stored in mesh buffer 1234 and the current quantized base mesh. The motion data and the quantized base mesh stored in mesh buffer 1234 are used by base mesh reconstructor 1236 to reconstruct the current quantized base mesh. The quantized base mesh obtained from inter-frame or intra-frame decoding is then passed to inverse quantizer 1237 to obtain the decoded base mesh.

[0360] Since the bitstream contains displacement data in image format with two chromaticity information points and one luminance information point, the image containing the displacement data is passed to video decoder 1238. Video decoder 1238 decodes the data using video frame decompression. In another example, an arithmetic decoder is used to decode the displacement data. The decompressed data is sent to image unpacker 1239, which extracts wavelet coefficients associated with each vertex from the decompressed data in image form. Inverse quantizer 1240 inverse quantizes the quantized wavelet coefficients using the three components associated with each vertex. Inverse wavelet transform 1241 performs an inverse transform on the result to obtain the final decoded displacement data. The decoded displacement data and the decoded base mesh are passed to reconstructor 1242. Reconstructor 1242 subdivides the edges of the decoded base mesh and uses the decoded displacement data to shift the vertices, obtaining the decoded mesh.

[0361] To obtain the decoded attribute bitstream, the image containing the attribute data is transmitted to another video decoder 1243. For color space and color format conversion, the decoded attribute bitstream is further processed by the color converter 1244 to obtain the decoded attribute map.

[0362] Figure 36 This is a block diagram illustrating an example of the configuration of the encoding device according to this embodiment.

[0363] First, the encoding device obtains the base grid bit stream, the displacement bit stream, and the attribute bit stream obtained from the preprocessing step of the 3D grid.

[0364] Figure 36 The encoding device shown includes a quantizer 1261, a switch 1262, a static mesh encoder 1263, a mesh buffer 1264, a motion encoder 1265, a basic mesh reconstructor 1266, a displacement data updater 1267, a wavelet transform 1268, a quantizer 1269, an image packer 1270, a video encoder 1271, a color converter 1272, and a video encoder 1273.

[0365] The base mesh (specifically, the positional information of the multiple vertices constituting the base mesh) is first quantized by quantizer 1261. The quantized base mesh (base mesh data) is output to switch 1262, which determines whether to use intra-frame coding or inter-frame coding. If intra-frame coding is selected, the quantized base mesh (base mesh bitstream) is output to static mesh encoder 1263, which generates the quantized base mesh. An encoder using the Edgebreaker algorithm for encoding 3D mesh data is exemplified as static mesh encoder 1263. This encoded (quantized) base mesh is stored in mesh buffer 1264 for reference when inter-frame coding is selected. If inter-frame coding is selected, switch 1262 outputs compressed data associated with the base mesh to motion encoder 1265. The motion data and static 3D mesh in mesh buffer 1264 are used in base mesh reconstructor 1266 to reconstruct the current quantized base mesh.

[0366] The displacement data is output to the displacement data updater 1267, where it is updated based on the quantized static base grid or the reconstructed inter-frame coding base grid. Next, the wavelet transformer 1268 performs the transform, followed by quantization by the quantizer 1269. The quantized displacement data is then packed into an image by the image packer 1270, and finally encoded by the video encoder 1271. The encoded displacement data is output to the multiplexer 1274.

[0367] Attribute information (e.g., attribute maps) is output to color converter 1272 for color space and color format transformation. The transformed attribute information is encoded by video encoder 1273 and output to multiplexer 1274.

[0368] The multiplexer 1274 acquires encoded data related to the base mesh (compressed data related to the base mesh), image data containing encoded displacement data, and image data containing attribute information such as encoded attribute maps, and generates a bitstream (compressed bitstream) containing the acquired data. The generated compressed bitstream is output to the decoding device, for example.

[0369] Figure 37 This is a block diagram illustrating an example of the configuration of the decoding device according to this embodiment.

[0370] Figure 37 This is a diagram illustrating an example of a reconstructor that generates a decoded 3D mesh 1256 based on the decoded base mesh 1251 and the decoded displacement data 1254.

[0371] The decoded base grid 1251 is transmitted to the sub-segmenter 1252.

[0372] Subsegmenter 1252 performs subsegmentation by appending a new vertex between any two connected vertices of the entire 3D mesh. To generate a predefined number of vertices, this process can be repeated multiple times, including vertices generated in previous subsegmentation steps. Repeated subsegmentation across the entire 3D mesh generates a new level of detail (LoD). The subsegmented mesh 1253 and the decoded displacement data 1254 are then passed to displacement unit 1255. Displacer 1255 moves each vertex to a new position according to the corresponding displacement data, thereby generating the decoded 3D mesh 1256.

[0373] The following describes sub-segmentation. Sub-segmentation is performed, for example, by sub-segmenter 1252.

[0374] Figure 38 This is an illustrative diagram representing an example of sub-segmentation.

[0375] Figure 38 The basic mesh shown in (a) contains vertices A, B and C, as well as connection information representing their connectivity.

[0376] exist Figure 38 (b) shows the mesh generated by the first sub-segmentation, in other words, the mesh after the first sub-segmentation. The sub-segmenter generates vertices D, E, and F, as well as connection information representing their connectivity, in the first sub-segmentation. The mesh generated by the sub-segmenter is also referred to as LoD1 or the first LoD.

[0377] Vertex D of the mesh after the first sub-segmentation is a vertex generated by sub-segmenting vertices A and B. Similarly, vertex F is a vertex generated by sub-segmenting vertices B and C. Vertex E is a vertex generated by sub-segmenting vertices A and C.

[0378] Furthermore, as an example, vertex D can be the midpoint of line segment AB (in other words, edge AB) connecting vertices A and B, which will form the basis of this generation. Similarly, vertex E can be the midpoint of line segment AC. Vertex F can be the midpoint of line segment BC.

[0379] exist Figure 38 Figure (c) shows the mesh generated by the second sub-segmentation, in other words, the mesh after the second sub-segmentation. In the second sub-segmentation, the sub-segmenter generates vertices G, H, I, J, K, L, M, N, and O, as well as connection information representing their connectivity. The mesh generated by the sub-segmenter is also referred to as LoD2 or the second LoD.

[0380] Vertex G of the mesh after the second sub-segmentation is generated by sub-segmenting vertices A and D. Similarly, vertex H is generated by sub-segmenting vertices A and E. Vertex I is generated by sub-segmenting vertices B and D. Vertex J is generated by sub-segmenting vertices D and F. Vertex K is generated by sub-segmenting vertices E and F. Vertex L is generated by sub-segmenting vertices C and E. Vertex M is generated by sub-segmenting vertices B and F. Vertex N is generated by sub-segmenting vertices C and F. Vertex O is generated by sub-segmenting vertices D and E.

[0381] Furthermore, as an example, vertex G can be the midpoint of line segment AD (in other words, edge AD) connecting vertices A and D, which will serve as the basis for this generation. Similarly, vertex H can be the midpoint of line segment AE. Vertex I can be the midpoint of line segment BD. Vertex J can be the midpoint of line segment DF. Vertex K can be the midpoint of line segment EF. Vertex L can be the midpoint of line segment CE. Vertex M can be the midpoint of line segment BF. Vertex N can be the midpoint of line segment CF. Vertex O can be the midpoint of line segment DE.

[0382] The following is for reference Figure 39 and Figure 40 The displacement of vertices is described. Vertex displacement is performed by the reconstructor.

[0383] Figure 39 This is an explanatory diagram showing an example of the displacement of vertices after sub-segmentation and displacement. Figure 40 This is an illustrative diagram representing an example of the vertices of the original mesh.

[0384] Figure 39 The basic mesh shown in (a) includes vertices A, B, C and Z, as well as connection information representing their connectivity.

[0385] exist Figure 39 (b) shows the mesh generated by the first sub-segmentation, in other words, the mesh after the first sub-segmentation (i.e., the first LoD). The sub-segmenter generates vertices S, T, U, X, or Y, and connection information representing their connectivity, in the first sub-segmentation. Regarding vertices S, T, U, X, or Y, and... Figure 38 The vertices D, E, and F shown in (b) are the same.

[0386] exist Figure 39(c) shows the mesh generated by the second sub-segmentation; in other words, it shows the mesh after the second sub-segmentation (i.e., the second LoD). The sub-segmenter generates vertices D, E, F, G, and H in the second sub-segmentation, along with connection information representing their connectivity. Regarding vertices D, E, F, G, and H, and... Figure 38 The vertices G, H, I, J, K, L, M, N, or O shown in (c) are the same.

[0387] Figure 39 (d) represents the mesh containing the vertices after the subdivision and displacement. Figure 39 The vertices A, B, C, D, E, F, G, H, S, T, U, X, Y, and Z shown in (d) are located using displacement information from Figure 39 The position of the vertex shown in (c) after displacement.

[0388] Figure 40 The original grid shown is an example of the grid input to the encoding device 100, i.e., the grid before encoding.

[0389] Figure 39 The grid shown has the same Figure 40 The shape shown is close to that of the original mesh. The displacement information is generated by the displacement vector calculator 1207 of the encoding device 100 as information representing the displacement from the vertices of the base mesh to the vertices of the original mesh. Therefore, by using the displacement information generated in this way, the mesh is reconstructed to generate a mesh with a shape close to that of the original mesh.

[0390] Decoding device 200 can output Figure 39 The grid shown in (d) is a grid.

[0391] Next, refer to Figure 41 and Figure 42 This describes the division of a grid into subgrids.

[0392] The mesh is divided into multiple smaller parts, each of which can be encoded. During mesh division, the vertices of the mesh are segmented in a way that allows the coordinates and connectivity of the vertices within each segment to be encoded independently.

[0393] Figure 41 This is an illustrative diagram representing an example of a grid. Figure 42 This is an illustrative diagram showing an example of dividing a grid into subgrids.

[0394] Figure 41 The grid shown is the original grid, sometimes referred to as the full grid in contrast to the subgrid.

[0395] Figure 42 Indicates will Figure 41The diagram shows the case where the entire mesh is divided into two sub-mesh segments. Regarding vertices A, B, and C of the entire mesh (refer to...). Figure 41 Vertex A is copied to vertices A1 and A2, vertex B is copied to vertices B1 and B2, and vertex C is copied to vertices C1 and C2, thus creating two sub-meshes (i.e., sub-mesh 1 and sub-mesh 2) from the full mesh. Sub-mesh 1 and sub-mesh 2 are meshes that can be decoded independently.

[0396] The following is for reference Figure 43 , Figure 44 and Figure 45 This indicates that displacement information is packaged into the image frame.

[0397] Figure 43 , Figure 44 and Figure 45 This is an illustrative diagram showing an example of packing displacement information into an image frame. Additionally, an image frame can also be referred to as a video frame.

[0398] Vertex displacement data can be encoded into image frame data, for example, by mapping to each component of a YUV format image frame (i.e., each of the Y component (Y plane), U component (U plane), and V component (V plane). This will be explained below as an example. Alternatively, vertex displacement data can also be encoded into image frame data by mapping to each component of an RGB format image frame (each of the R component, G component, and B component).

[0399] The decoding device 200 can use an image encoding module to extract displacement data. The displacement data can be the X, Y, or Z components in a global coordinate system (e.g., a Cartesian coordinate system), or the shape of normal, tangent, or two tangent components in a local coordinate system. Methods for mapping displacement data to image frames include the following.

[0400] For example, in method 1, displacement data is configured in the image frame according to the scan order. An example of the packing of displacement data in this case is as follows: Figure 43 As shown, displacement data is directly mapped to image frames according to a predefined scanning order.

[0401] Additionally, since the height and width of an image frame are fixed, sometimes the displacement data cannot be fully contained within the frame. In such cases, the remaining portion of the image frame is filled with padding data (also known as padded data) (see [reference]). Figure 43 ).

[0402] For example, in method 2, the displacement data is separated into multiple LoDs and mapped to the Y, U, and V components of the image frame. An example of this type of displacement data packing is shown below. Figure 44As shown. Here, the displacement data of the next LoD image frame begins immediately after the end of the displacement data of the previous LoD. Similar to method 1, if the displacement data does not fit perfectly into the image frame, the end of the image frame is padded (see [reference]). Figure 44 ).

[0403] For example, in method 3, the displacement data corresponding to the LoD is mapped to the Y, U, and V components of the image frame in a manner different from that in method 2. An example of the packing of displacement data in this case is as follows... Figure 45 As shown. Therefore, each LoD can be decoded independently. In method 3, the displacement data of each LoD is padded in the middle, and CTU alignment is performed together with the padding at the end of the video frame (see [reference]). Figure 45 ).

[0404] Next, the encoding device 100 and the decoding device 200 when the grid is divided into multiple sub-grids will be described.

[0405] Figure 46 This is a diagram illustrating another example of the configuration of the encoding device 100 according to the relevant implementation. Specifically, Figure 46 This diagram illustrates the configuration of a subgrid encoding apparatus for encoding processing when the input grid 1101 is divided into subgrids (multiple subgrids). For example, the subgrid encoding apparatus includes multiple encoding devices 100.

[0406] The input grid 1101 (full grid) input to the subgrid encoding device is divided into multiple grids (subgrids). These multiple subgrids are input to multiple encoding devices 100, for example. Each of the multiple subgrids can be input to any one of the multiple encoding devices 100. For example, the subgrid encoding device divides the input grid 1101 into multiple subgrids and inputs the divided multiple subgrids to the multiple encoding devices 100.

[0407] In addition, after dividing into multiple sub-grids, encoding processing related to the boundaries of the sub-grids is performed (processing of overlapping sub-grids).

[0408] For example, for each subgrid, preprocessing is performed by preprocessor 1103 to generate the base grid, displacement data, and metadata, and then encodes them.

[0409] Alternatively, the encoding device 100 can also be implemented using a subgrid encoding device configuration. That is, the encoding device 100 can be configured to include multiple preprocessors 1103 and compressors 1106, and can perform predetermined processing on the subgrid according to each group of multiple preprocessors 1103 and compressors 1106. Furthermore, the number of groups of preprocessors 1103 and compressors 1106 included in the encoding device 100 can be arbitrary and is not particularly limited.

[0410] Figure 47 This is a diagram illustrating an example of the configuration of the preprocessor 1103 in relation to an implementation method.

[0411] The preprocessor 1103 may include a basic mesh generator 1401, a sub-segmenter 1402, and a displacement data generator 1403.

[0412] First, in the preprocessor 1103, the base mesh is generated by the base mesh generator 1401.

[0413] Next, the sub-segmenter 1402 subdivides the base mesh using a prescribed method to generate a subdivided mesh (or subdivided base mesh) that serves as the subdivided base mesh.

[0414] Displacement data is generated by displacement data generator 1403 based on the sub-segmented grid and the sub-grid that serves as the input grid 1101 after segmentation.

[0415] Displacement data, for example, is the difference vector between the input grid 1101 and the subdivided grid.

[0416] In addition, the same sub-segmentation method is used in decoding as in encoding.

[0417] Alternatively, for example, the encoding device 100 may send the method of sub-segmentation in the encoding and the parameters used in the sub-segmentation to the decoding device 200.

[0418] Figure 48 This is a diagram illustrating another example of the configuration of the decoding device 200 according to the relevant implementation. Specifically, Figure 48 This diagram illustrates the configuration of a subgrid decoding apparatus, which performs decoding processing when the bitstream 2101 includes multiple subgrids. For example, the subgrid decoding apparatus includes multiple decoding units 200. For example, the subgrid decoding apparatus includes multiple decoding units 200 and a combiner 2109.

[0419] The encoded data of each subgrid contained in the bitstream 2101 is input to the decompressor 2102 of each decoding device 200.

[0420] Additionally, for example, the refactoring process described above is performed in the post-processor 2106.

[0421] In the post-processor 2106, the base grid is subdivided according to each decoded sub-grid, and a displacement vector is added to the subdivided base grid to form a complex atomic grid.

[0422] That is, the post-processor 2106 performs the refactoring process described above on each subgrid.

[0423] The combiner 2109 combines (merges) the multiple sub-grids restored by the multiple decoding devices 200 respectively, and reconstructs the full grid before the segmentation (output grid 2107).

[0424] Alternatively, the decoding device 200 can also be implemented using a subgrid decoding device configuration. That is, the decoding device 200 can be configured to include multiple decompressors 2102 and post-processors 2106, or it can perform prescribed processing on the encoded data of each subgrid according to each group of multiple decompressors 2102 and post-processors 2106. Furthermore, the number of groups of decompressors 2102 and post-processors 2106 included in the decoding device 200 can be arbitrary and is not particularly limited.

[0425] Figure 49 This is a diagram illustrating a specific example of the configuration of the decoding device 200 according to the relevant implementation method. Specifically, Figure 49 This indicates the specific configuration of the post-decoder 2306 in the decoder 2305 and post-decoder 2306 of the decoding device 200.

[0426] The post decoder 2306 includes a pre reconstructor 2307, a reconstructor 2308, a post reconstructor 2309, and an adapter 2310.

[0427] The processing in the post-decoder 2306 is arbitrary depending on the application. Examples of post-decoding processing in the post-decoder 2306 include transforming the decoded data to the nominal format, similar to video conversion from YUV to RGB space. Post-decoding processing can be encapsulated into multiple processes such as pre-reconstruction, reconstruction, post-reconstruction, and adaptation.

[0428] The pre-reconstructor 2307 performs pre-reconstruction processing. For example, in the context of video-based dynamic mesh coding, the pre-reconstructor 2307 amplifies the normalized texture coordinates to match the dimensions of the texture image.

[0429] Reconstructor 2308 performs reconstruction processing. This reconstruction processing may involve, for example, calling upon decoded atlas frames, decoded base grid frames, decoded video frames, and syntactic elements associated with the same grid sequence. The output of the reconstruction processing is a series of grid frames reconstructed prior to the subsequent reconstruction processing.

[0430] The post-reconstructor 2309 performs post-reconstruction processing. For example, in the context of video-based dynamic mesh coding, the post-reconstructor 2309 performs multiple smoothing processes on the reconstructed mesh frame. Smoothing processes include, for example, collapsing the edges contained in the mesh, or appending new vertices to the mesh.

[0431] Adapter 2310 performs adaptation processing. Adaptation processing may involve adapting the reconstructed mesh to a specified scene through certain applications. For example, the vertices of the reconstructed mesh are transformed from the 3D model coordinate system to the 3D world coordinate system. Adapter 2310 outputs the final reconstructed mesh frame (final 3D mesh frame 2311).

[0432] The following is a detailed explanation of sub-segmentation.

[0433] <sub-partition> Figure 50 This is a diagram illustrating an example of two sub-grids with boundary edges in a relevant implementation method. Specifically, Figure 50 (a) represents an example of a subgrid. Figure 50 (b) represents another example of a subgrid. Figure 50 (c) indicates that Figure 50 The subgrid shown in (a) and Figure 50 The three-dimensional mesh shown in (b) is formed by merging (i.e., combining) the sub-mesh.

[0434] The decoding device 200 performs sub-segmentation on each edge constituting the sub-mesh based on the position information of each vertex constituting the sub-mesh and the connection information representing the connection relationships between each vertex, i.e., the information of the multiple edges constituting the sub-mesh. In the sub-segmentation, for example, new vertices (3D points) are generated on the edges. The generated vertices are connected, for example, by the new edges. Thus, for example, multiple faces contained in the sub-mesh are each divided into multiple faces through sub-segmentation. For example, the face of a triangle surrounded by three vertices contained in the sub-mesh is divided into four faces through sub-segmentation.

[0435] The decoding device 200 subdivides each subgrid and merges the subdivided subgrids to reconstruct a three-dimensional mesh corresponding to the original mesh.

[0436] Here, the encoding device 100 generates multiple sub-meshes from the original mesh and encodes the position and displacement information of the vertices constituting each sub-mesh. Therefore, the position and displacement information of the vertices constituting each sub-mesh may be encoded using encoding parameters that differ for each sub-mesh. Consequently, when the decoding device 200 performs sub-segmentation based on this encoded information, edges shared by two or more sub-meshes (also called boundary edges) may be sub-segmented at different locations within each sub-mesh, or sub-segmented a different number of times within each sub-mesh. This may lead to the problem of inappropriately merging the sub-meshes.

[0437] For example, in Figure 50 In the example shown, edge CD is the boundary edge. When Figure 50 When the submesh shown in (a) is subdivided, for example, vertex F is generated on edge CD. Additionally, when Figure 50 When the submesh shown in (b) is subdivided, for example, vertex M is generated on edge CD. If the positions of vertex F and vertex M are different, or the number of vertices generated on edge CD is different, that is, the number of times edge CD is subdivided is different, then in Figure 50 The subgrid shown in (a) and Figure 50 When the subgrids shown in (b) are merged, gaps are generated around the edge CD, making proper merging impossible.

[0438] To address the problem arising from using different sub-segmentation schemes in adjacent sub-mesh (i.e., sub-mesh with the same boundary edges), this application, for example, enforces the following constraint: for all boundary edges, the same type of sub-segmentation scheme and the same number of iterations are used. That is, for example, in the sub-segmentation of boundary edges, sub-segmentation is performed in each sub-mesh using the same sub-segmentation method and the same number of sub-segments. Thus, if the displacement information (i.e., the value of the displacement vector) of vertex F and vertex M are the same, then the vertex generated by the sub-segmentation undergoes displacement, and the sub-mesh is merged, for example, as... Figure 50 As shown in (c), it is possible to make it so that there are no gaps around the boundary edge.

[0439] Furthermore, in the sub-segmentation at the boundary edge, a prescribed sub-segmentation method and / or a prescribed number of sub-segments can be used, or the encoding device 100 can determine the prescribed sub-segmentation method and / or the prescribed number of sub-segments, and signal the determined information through a bit stream. Additionally, for example, the alignment of vertex F with vertex M can be performed during the sub-segmentation of the subgrid, or it can be performed when merging multiple subgrids.

[0440] [Method 1] Next, the first method of sub-segmentation will be explained.

[0441] Figure 51 This is a flowchart illustrating the processing of the decoding device 200 according to the relevant implementation method. Figure 51 This is a flowchart illustrating an example of using a pre-determined sub-segmentation method.

[0442] First, the decoding device 200 decodes the first and second vertices connected by edges from the bitstream (S301). Specifically, the decoding device 200 obtains the position information of the first and second vertices and the connection information indicating whether the first and second vertices are connected from the bitstream.

[0443] Next, the decoding device 200 determines whether the edge connecting the first vertex and the second vertex is a boundary edge (S302).

[0444] When the decoding device 200 determines that the edge connecting the first vertex and the second vertex is a boundary edge ("Yes" in S302), it derives the third vertex based only on the first vertex and the second vertex (S303). That is, it derives the position of the third vertex based only on the position of the first vertex and the position of the second vertex, thereby supplementing (adding) the third vertex.

[0445] On the other hand, if the decoding device 200 determines that the edge connecting the first vertex and the second vertex is not a boundary edge ("No" in S302), it derives the fourth vertex based on the first vertex, the second vertex, and at least one fifth vertex (S304). That is, the position of the fourth vertex is derived based on the position of the first vertex, the position of the second vertex, and the position of at least one fifth vertex, thereby supplementing the fourth vertex.

[0446] Furthermore, the number of fifth vertices can be one or more. That is to say, the fourth vertex can also be derived from three or more vertices, including the first and second vertices.

[0447] Furthermore, the processes in steps S303 and S304 are just one example; other methods could also be used.

[0448] Figure 52 and Figure 53 These are figures illustrating an example of the boundary edge and non-boundary edge of the relevant implementation method.

[0449] Boundary edges are, for example, edges shared by multiple sub-grids.

[0450] On the other hand, a non-boundary edge is an edge that is not shared by multiple sub-meshes. That is, a non-boundary edge is, for example, an edge that is contained only in one of the multiple sub-meshes included in the base mesh.

[0451] exist Figure 52In the example shown, only one face Y, formed by edge AB and vertex F, is connected to edge AB. Therefore, edge AB is a boundary edge. Similarly, for example, only one face Z, formed by edge EF and vertex D, is connected to edge EF. Therefore, edge EF is a boundary edge.

[0452] Furthermore, for example, edge BD connects two faces: face W, formed by edge BD and vertex C, and face X, formed by edge BD and vertex F. Therefore, edge BD is not a boundary edge, but a non-boundary edge. Similarly, for example, edge DF connects two faces: face X, formed by edge DF and vertex B, and face Z, formed by edge DF and vertex E. Therefore, edge DF is not a boundary edge, but a non-boundary edge.

[0453] For example, in step S303, when the edge connecting the first vertex and the second vertex is a boundary edge, the decoding device 200 derives the third vertex only based on the first vertex and the second vertex. For example, in Figure 52 In this scenario, assume vertex A is the first vertex and vertex C is the second vertex. Furthermore, vertices A and C lie on the boundary edge. In this case, vertex B, as the third vertex, is derived solely from the first and second vertices. As an example of deriving the coordinates (position) of B, one method is to assign the midpoint between vertex A (the first vertex) and vertex C (the second vertex) to these coordinates. That is, the coordinates of the midpoint between vertex A and vertex C can also be calculated as the coordinates of the intersection point B.

[0454] Additionally, for example, in step S304, if the edge connecting the first vertex and the second vertex is not a boundary edge, the decoding device 200 derives the fourth vertex based on the first vertex, the second vertex, and at least one fifth vertex. For example, in Figure 53 In this example, assume vertex B is the first vertex and vertex D is the second vertex. Furthermore, vertices B and D are not on the boundary edge; that is, edge BD is a non-boundary edge. In this case, vertex H, as the fourth vertex, is derived from vertex B (the first vertex), vertex D (the second vertex), and vertex F (the fifth vertex). As an example of deriving the coordinates of vertex H, there is a method that assigns the coordinates of vertex H to the vertex F when orthogonally projected onto edge BD.

[0455] For example, one or more parameters can be decoded from the bitstream. Furthermore, one or more parameters can be decoded from the header of the bitstream. Additionally, one or more parameters may include boundary edge information representing edges that form the boundary edges among the multiple edges constituting the subgrid. The decoding device 200 can also use the decoded boundary edge information to determine the boundary edges.

[0456] Figure 54This is a diagram illustrating an example of the syntax used in the implementation of a method for signaling different sub-segmentation types (also called sub-segmentation methods) and sub-segmentation times in the header.

[0457] Encoding device 100 generates, for example, a bit stream that includes in the header of the sub-grid information (subdivision_type) indicating the method of subdividing each edge of the sub-grid (also referred to as the subdivision method of the sub-grid) and information (subdivision_num_iteration) indicating the number of subdivisions of each edge of the sub-grid (also referred to as the number of subdivisions of the sub-grid).

[0458] Figure 55 This is a diagram illustrating an example of the syntax for signaling the sub-segment type and sub-segment number using a sequence parameter set (SPS) in relation to an implementation method.

[0459] Encoding device 100 generates, for example, a bit stream that contains information in the sequence parameter set indicating whether an edge is a boundary edge (boundary_subdivision_flag), information indicating the method of subdividing the boundary edge (also simply referred to as the boundary edge subdivision method) (boundary_subdivision_type), and information indicating the number of times the boundary edge is subdivided (also simply referred to as the boundary edge subdivision number) (boundary_subdivision_num_iteration).

[0460] Figure 56 This is a diagram illustrating an example of a syntax for determining the sub-segment type and the number of sub-segments using a sequence parameter set, and checking whether an edge is on a boundary, in relation to an implementation method.

[0461] For example, when the edge is a boundary edge, the decoding device 200 performs sub-segmentation on the edge based on information indicating the sub-segmentation method and the number of sub-segments of the boundary edge contained in the SPS of the bitstream obtained from the encoding device 100. On the other hand, when the edge is not a boundary edge, the decoding device 200 performs sub-segmentation on the edge based on information indicating the sub-segmentation method and the number of sub-segments of the sub-grid contained in the header of the sub-grid of the bitstream obtained from the encoding device 100.

[0462] Additionally, information representing the sub-segmentation method of a sub-mesh and / or the number of sub-segments of a sub-mesh can be stored in a common parameter set across frames if they are common across frames, and in a common parameter set across sequences if they are common across sequences.

[0463] Furthermore, if this information is common across all subgrids, it may not be signaled to the subgrid header. Additionally, if the subgrid's sub-segmentation method and / or sub-segmentation count is not signaled in the headers of any adjacent subgrids, it can be determined that the subgrid's sub-segmentation method and / or sub-segmentation count is common across all subgrids, and this method is not used.

[0464] Additionally, the above example shows an example where information representing the sub-segmentation method of the boundary edge and information representing the number of sub-segments of the boundary edge are stored in a common parameter set in the sequence (specifically, stored in SPS), but this information can also be stored in a common parameter set of the frames.

[0465] Furthermore, the sub-segmentation method and / or sub-segmentation number represented by information stored in a common parameter set in the sequence can also be used for the sub-segmentation method and / or sub-segmentation number at the boundary edges. That is, the decoding device 200 can also use the value of the sub-grid header in the sub-segmentation of the sub-grid and the value of the common parameter set in the sequence in the sub-segmentation of the boundary edges, provided that the sub-segmentation method and / or sub-segmentation number are signaled in both the common parameter set in the sequence and the sub-grid header.

[0466] For example, if at least one of the edges constituting a submesh is a boundary edge, and the number of sub-segments for the boundary edge differs from the number of sub-segments for the submesh, it is necessary to define how to sub-segment the multiple edges constituting the submesh, including the boundary edge, and how to connect vertices to generate a new mesh. Furthermore, this could be a case where the number of sub-segments for the boundary edge of the submesh differs from the number of sub-segments for the submesh itself, or a case where the number of sub-segments for the boundary edge of a submesh containing an object differs from the number of sub-segments for the submesh itself.

[0467] Figure 57 This is a flowchart illustrating an example of the process of dividing multiple edges constituting a subgrid in a relevant implementation method.

[0468] First, the decoding device 200 determines whether at least one of the multiple edges constituting the grid (specifically, the subgrid) is a boundary edge (S401).

[0469] When the decoding device 200 determines that at least one of the multiple edges constituting the sub-mesh is a boundary edge ("Yes" in S401), it compares the number of sub-segments of the boundary edge with the number of sub-segments of the sub-mesh (S402).

[0470] Next, the decoding device 200 performs sub-segmentation based on the comparison result in step S402 (S403).

[0471] For example, the number of subdivisions for boundary edges is set to A, and the number of subdivisions for sub-grids is set to B.

[0472] For example, when A=B, the decoding apparatus 200 performs subdivision using Method A described later. Further, for example, when A<B, the decoding apparatus 200 performs subdivision using Method B (Method B1 or Method B2) described later. Further, for example, when A>B, the decoding apparatus 200 performs subdivision using Method C (Method C1 or Method C2) described later.

[0473] In addition, when it is determined that at least one of the multiple edges constituting the sub-grid is not a boundary edge ("No" in S401), the decoding apparatus 200 performs subdivision on the multiple edges using the sub-grid subdivision method and the number of sub-grid subdivisions (S404).

[0474] [Method A] When performing subdivision using Method A, that is, when the number of subdivisions for boundary edges (A times) and the number of subdivisions for sub-grids (B times) are the same (A=B), the decoding apparatus 200 subdivides each edge constituting the sub-grid, connects the vertices generated by the subdivision to each other, and thereby generates a new grid. The decoding apparatus 200 repeats this process A (= B) times.

[0475] When performing subdivision using Method B, that is, when the number of subdivisions for boundary edges (A times) is less than the number of subdivisions for sub-grids (B times) (A<B), the decoding apparatus 200 performs subdivision using Method B1 or Method B2 below.

[0476] [Method B1] Figure 58 is a diagram illustrating a first example of the process of dividing multiple edges constituting a sub-grid according to the embodiment. In addition, in Figure 58 the illustrated example, polygon ABCD is a sub-grid, edge CD (edge CF' and edge F'D) and edge EC are boundary edges, and the other edges are non-boundary edges. In addition, in Figure 58 the illustrated example, the number of sub-grid subdivisions is 2, and the number of boundary edge subdivisions is 1.

[0477] In the example shown in step S303, the decoding apparatus 200 decodes a parameter from the bitstream, and derives the number of boundary edge subdivisions based on this parameter. The parameter may be a value indicating the exact number of boundary edge subdivisions, or a value indicating the difference from the number of sub-grid subdivisions.

[0478] When the number of boundary edge subdivisions is less than the number of sub-grid subdivisions, the decoding apparatus 200, for example, as shown in Figure 58 As shown, the boundary edge (edge ​​CD) is sub-segmented a number of times. In this example, for instance, the decoding device 200 generates vertex F' by performing one sub-segment on edge CD. Then, the decoding device 200 performs sub-segmentation on edges that are not boundary edges (e.g., edges AB, BC, AD, and BD). Furthermore, the decoding device 200 performs sub-segmentation on edges that are not boundary edges, as well as on edges newly generated by connecting newly generated vertices through sub-segmentation. Thus, edges that are not boundary edges are sub-segmented twice. Additionally, the decoding device 200 creates edge CW by connecting vertex C to the newly generated vertex W, and creates edge DT by connecting vertex D to the newly generated vertex T. Then, the decoding device 200 generates edge TL.

[0479] Thus, the decoding device 200 determines, for example, whether the number of sub-segments of the subgrid is different from the number of sub-segments of the boundary edge, and (the number of sub-segments of the subgrid) > (the number of sub-segments of the boundary edge).

[0480] If the determination is "yes", the decoding device 200 first uses the conventional method (connecting the divided vertices to generate a new mesh) to subdivide each edge constituting the submesh until the number of subdivisions that become boundary edges is reached. Then, if the number of subdivisions that become boundary edges is exceeded, the decoding device 200 repeats the following process until the number of subdivisions that become submesh is reached.

[0481] The decoding device 200 determines whether at least one edge of the mesh constituting the sub-segmentation object (e.g., the face of a polygon contained in the sub-mesh) is a boundary edge of the sub-mesh. If there is no boundary edge, the decoding device 200 performs normal sub-segmentation; if there is a boundary edge, it performs sub-segmentation on edges that are not boundaries.

[0482] Furthermore, when the number of boundary edges in the subdivided object's mesh is 1 and the number of non-boundary edges is 2, the decoding device 200 connects the vertices generated by subdividing at least the non-boundary edges, and uses either of the two edges obtained by connecting the generated vertex to the two vertices constituting the boundary edge to generate a new mesh. Additionally, for example, the decoding device 200 determines the priority based on whether the edge connecting the vertices in the two edges has been subdivided, and determines which edge to use based on the determined priority.

[0483] Alternatively, two edges can be used to generate a new mesh.

[0484] Additionally, for example, among the multiple edges that make up a subgrid, there may also be edges that are not subdivided. For example, information representing the edges that are not subdivided may also be included in the bitstream.

[0485] [Method B2] In method B1, when performing sub-segmentation, if the number of sub-segments exceeds the boundary edge, the normal sub-segmentation method is implemented when there is no boundary edge, and the non-boundary edge is sub-segmented when there is a boundary edge.

[0486] In contrast, in method B2, the usual sub-segmentation method is implemented when there are no boundary edges, and no sub-segmentation is performed when there are boundary edges.

[0487] Figure 59 This is a second example of the process for dividing the multiple edges constituting a sub-mesh in relation to an implementation method. Furthermore, in Figure 59 In the example shown, polygon ABCD is a sub-mesh, edge CD (edge ​​CF' and edge F'D) and edge EC are boundary edges, and the other edges are non-boundary edges. Additionally, in Figure 59 In the example shown, the submesh is subdivided twice, and the boundary edge is subdivided once.

[0488] In another example of step S303, the decoding device 200 decodes parameters from the bitstream and derives the number of sub-segments for the boundary edges based on these parameters. The parameters may be, for example, a value representing the exact number of sub-segments for the boundary edges, or a value representing the difference between the number of sub-segments for the subgrid and the number of sub-segments for the subgrid. If the value representing the number of sub-segments for the boundary edges is less than the value representing the number of sub-segments for the subgrid, such as... Figure 59 As shown, the decoding device 200 performs sub-segmentation on each edge constituting the sub-mesh a number of times equivalent to the number of sub-segments representing the boundary edges. Then, the decoding device 200 does not perform explicit displacement of vertices W and T. Instead, the decoding device 200, for example, displaces vertex F based on displacement information, then projects vertex W onto edge EF and vertex T onto edge GF, thereby sub-segmenting edges EF and GF. Thus, after the face (specifically, the vertices constituting the face) is displaced, vertices E, W, and F' and each of the vertices G, T, and F' are also on a straight line. Next, the decoding device 200 creates edges CW and DT, and then creates edge TL.

[0489] [Method C1] Figure 60 This is a diagram illustrating a third example of the process for dividing the multiple edges constituting a sub-mesh in relation to an implementation method. Furthermore, in Figure 60 In the example shown, polygon CDU is a submesh, and edge CD is a boundary edge. Additionally, edges CU and DU are non-boundary edges. Furthermore, in... Figure 60 In the example shown, the submesh is subdivided once, and the boundary edge is subdivided twice.

[0490] When the number of sub-divisions at the boundary edge is greater than the number of sub-divisions in the submesh, firstly, as... Figure 60 As shown, the decoding device 200 performs sub-segmentation on all edges, including boundary edges, a number of times the submesh is sub-segmented. Next, the decoding device 200 performs sub-segmentation on the boundary edges until the number of times the boundary edges are sub-segmented equals the number of times the boundary edges are sub-segmented.

[0491] Furthermore, when the number of sub-segments of the submesh is less than the number of sub-segments of the boundary edge, the number of sub-segments of the boundary edge can also be considered as the number of sub-segments of the submesh. For example, when (the number of sub-segments of the submesh) < (the number of sub-segments of the boundary edge), the decoding device 200 considers (the number of sub-segments of the submesh) = (the number of sub-segments of the boundary edge). Thus, the number of sub-segments of the boundary edge can be defined as not exceeding the number of sub-segments of the submesh.

[0492] [Method C2] Figure 61 This is a diagram illustrating the fourth example of the process for dividing the multiple edges constituting the sub-mesh in relation to the implementation method. Furthermore, in Figure 61 In the example shown, the polygon CDU is the submesh, and the edge CD is the boundary edge. Additionally, in Figure 61 In the example shown, edges CU and DU are non-boundary edges. Additionally, in... Figure 61 In the example shown, the submesh is subdivided once, and the boundary edge is subdivided three times.

[0493] When the number of sub-segments at the boundary edge is greater than the number of sub-segments at the sub-mesh, firstly, the decoding device 200, in order to generate edge UQ, edge UM, and edge UR, performs sub-segmentation on the boundary edge by subtracting the number of sub-segments at the sub-mesh from the number of sub-segments at the boundary edge. Then, as... Figure 61 As shown, the decoding device 200 performs sub-segmentation of the sub-mesh a number of times on all edges, including boundary edges.

[0494] Figure 62 This is a diagram illustrating the fifth example of the process for dividing the multiple edges constituting a sub-mesh in relation to an implementation method. Specifically, Figure 62 This is a diagram illustrating an example of a method for sub-segmenting an edge BD that is a non-boundary edge.

[0495] In the example of step S304, the decoding device 200 determines edge BD as a non-boundary edge. Furthermore, the decoding device 200 derives vertex G using vertices B, D, and A. For example, the decoding device 200 sets vertex G as the intersection of the bisecting line of angle BAD and edge BD. The decoding device 200 performs sub-segmentation on edge BD by generating vertex G in this way, for example.

[0496] Furthermore, when performing sub-segmentation on edges, it can be determined that the sub-segmentation method and the number of sub-segments are common.

[0497] Additionally, for example, the decoding device 200 may determine whether the sub-mesh of the object to be sub-segmented contains a boundary edge. If the sub-mesh does not contain a boundary edge, it may perform sub-segmentation using the sub-mesh sub-segmentation method and the number of sub-segments of the sub-mesh. On the other hand, for example, if the sub-mesh contains a boundary edge, the decoding device 200 may also perform sub-segmentation using either (i) the sub-mesh sub-segmentation method and the number of sub-segments of the sub-mesh, or (ii) the boundary edge sub-segmentation method and the number of boundary edge sub-segments of the boundary edge.

[0498] Subsegmentation can also use a predetermined subsegmentation method and number of subsegments. In addition, the encoding device 100 can also determine the subsegmentation method and number of subsegments, and include the determined subsegmentation method and number of subsegments in the bit stream for transmission.

[0499] Alternatively, for example, one of the information related to the sub-segmentation of the boundary edge (e.g., the sub-segmentation method and the number of sub-segments of the boundary edge) and the information related to the sub-segmentation of the sub-mesh (e.g., the sub-segmentation method and the number of sub-segments of the sub-mesh) can be set to a predetermined method, and the other can be notified via signaling.

[0500] Furthermore, sub-segmentation methods suitable for the sub-mesh can be used in the sub-segmentation of the sub-mesh, and sub-segmentation methods suitable for the boundary edges can be used in the sub-segmentation of the boundary edges. If suitable sub-segmentation methods exist for both boundary edges and non-boundary edges, different sub-segmentation methods can be used; if no suitable sub-segmentation methods exist for either boundary edges or non-boundary edges, the same sub-segmentation method can be used.

[0501] The sub-segmentation method can also be switched based on the comparison between the number of sub-segments at the boundary edge and the number of sub-segments at the sub-mesh.

[0502] In sub-segments within a mesh that becomes a sub-segment, the boundary edge sub-segment count can be used for boundary edges, and the sub-mesh sub-segment count can be used for non-boundary edges. Alternatively, either the boundary edge sub-segment count or the sub-mesh sub-segment count can be used in sub-segments within a mesh that becomes a sub-segment.

[0503] Furthermore, the order in which the above-mentioned judgment and processing are performed can be changed arbitrarily.

[0504] [Effect of Method 1] In the structure of the first method, sub-mesh that have been encoded using different sub-segmentation methods or sub-segmentation times can be merged, thus improving the subjective quality of the whole mesh (the 3D mesh after merging multiple sub-mesh).

[0505] [Combination with other methods] The decoding apparatus 200 of this method can be implemented in combination with at least a portion of other methods in this disclosure. Furthermore, this method can also be implemented by combining a portion of the process shown in a flowchart following this method, a portion of the structure of a device, and / or a portion of the syntax, etc., with other methods.

[0506] Furthermore, the aforementioned processing of the decoding device 200 can also be performed in the encoding device 100 in the same way. Additionally, not all of the constituent elements shown in this embodiment are always necessary; it is permissible to include only a portion of the constituent elements of the first embodiment.

[0507] [Method 2] Figure 63 This is a diagram illustrating the sixth example of the process for dividing the multiple edges constituting a sub-mesh in relation to an implementation method. Specifically, Figure 63 This is another example of the process used to illustrate step S303.

[0508] In step S303, for example, the encoding device 100 encodes a parameter for the number of sub-segments of the boundary edge used by the decoding device 200 to derive the parameter into a bitstream. The parameter can be a value representing the exact number of sub-segments of the boundary edge, or a value representing the difference between the number of sub-segments and the number of sub-segments of the subgrid. If the value representing the number of sub-segments of the boundary edge (derived value) is greater than the value representing the number of sub-segments of the subgrid, the encoding device 100 performs sub-segmentation of the boundary edge corresponding to that derived value. For example, if the derived value is 3, such as... Figure 63 As shown, three vertices (vertices Q, M, and R) are generated along the boundary edge, and their connectivity is added to the base mesh of the submesh. The decoding device 200 performs sub-segmentation equivalent to the number of sub-segments of the submesh. Thus, the decoding device 200 performs uniform sub-segmentation along all edges including the boundary edge in the modified base mesh, which is equivalent to a submesh that has been sub-segmented a number of times.

[0509] [Effect of Method 2] In the second approach, subgrids encoded with different sub-segmentation methods or sub-segmentation numbers can be merged, thus improving the subjective quality of the entire grid.

[0510] [Combination with other methods] This approach of the encoding device 100 can be implemented in combination with at least a portion of other approaches in this disclosure. Furthermore, this approach can also be implemented by combining a portion of the process shown in a flowchart following this approach, a portion of the structure of a device, and / or a portion of the syntax, etc., with other approaches.

[0511] Furthermore, the aforementioned processing of the encoding device 100 can also be performed in the decoding device 200. Additionally, not all the constituent elements described in this method are always necessary; it is permissible to include only a portion of the constituent elements of the first method.

[0512] [Method 3] In the third method, the encoding device 100 does not perform the process of making the vertices of the edges (also known as the boundary edges between sub-mesh) shared by multiple sub-meshes the same. On the other hand, in the third method, the encoding device 100 sends the metadata for the decoding device 200 to correct the vertices (vertices located on the boundary edges) in the boundary edges.

[0513] For example, the encoding device 100 sends a signaling notification (e.g., a flag, etc.) via metadata to the decoding device 200, indicating that the number of sub-segments or the sub-segmentation methods between sub-grids (i.e., multiple adjacent sub-grids) are different, or that the vertices after sub-segmentation are different (or may be different) as a result.

[0514] The decoding device 200 determines whether to correct the vertices in the boundary edges between sub-mesh based on metadata. If it determines that correction is necessary, the decoding device 200 corrects the number of vertices in the boundary edges to be the same in each sub-mesh containing the boundary edges by increasing or decreasing the number of vertices in the boundary edges.

[0515] Furthermore, the encoding device 100 may signal information in the metadata indicating whether the decoding device 200 should correct vertices in the boundary edge, and / or the method of correcting the vertex, instead of signaling a prompt message. In this case, for example, the decoding device 200 decides whether to correct the vertex based on this information.

[0516] Additionally, the encoding device 100 can signal information indicating whether multiple sub-mesh overlaps and / or how multiple sub-mesh overlaps. Furthermore, the encoding device 100 can also perform overlap-related processing (e.g., vertex correction processing) when multiple sub-mesh overlaps is possible.

[0517] Figure 64 This is a flowchart illustrating the boundary vertex determination process performed by the encoding device 100 in the relevant implementation method.

[0518] First, the encoding device 100 performs sub-segmentation based on the number of sub-segments for each sub-grid (S501).

[0519] Next, the encoding device 100 determines whether the boundaries between sub-grids overlap (S502). For example, the encoding device 100 determines whether a sub-grid has a boundary edge (i.e., an edge shared with other sub-grids).

[0520] When the encoding device 100 determines that there is boundary overlap between sub-grids ("Yes" in step S503), it indicates the boundary overlap between sub-grids in the metadata (S504). That is, the encoding device 100 includes information indicating boundary overlap between sub-grids in the metadata. In other words, the encoding device 100 generates metadata containing information indicating boundary overlap between sub-grids.

[0521] Next, the encoding device 100 determines whether the boundary vertices after sub-segmentation (specifically, the vertices in the boundary edge, in other words, the vertices located on the boundary edge) are the same in the boundary edge (overlapping edge between sub-mesh) (S505).

[0522] If the encoding device 100 determines that the boundary vertices after sub-segmentation are different ("No" in S505), the metadata indicates that the boundary vertices after sub-segmentation are different in each sub-grid (S506). That is, the encoding device 100 includes the information indicating that the boundary vertices are different in the metadata. In other words, the encoding device 100 generates metadata that also includes information indicating that the boundary vertices are different.

[0523] On the other hand, if the encoding device 100 determines that the boundary vertices after sub-segmentation are the same ("Yes" in S505), the metadata indicates that the boundary vertices after sub-segmentation are the same in each sub-grid (S507). That is, the encoding device 100 includes the information indicating that the boundary vertices are the same in the metadata. In other words, the encoding device 100 generates metadata that also includes information indicating that the boundary vertices are the same.

[0524] Furthermore, if the encoding device 100 determines that the boundaries between sub-grids do not overlap ("No" in S503), it indicates in the metadata that the boundaries between sub-grids do not overlap (S508). That is, the encoding device 100 includes information indicating that the boundaries between sub-grids do not overlap in the metadata. In other words, the encoding device 100 generates metadata containing information indicating that the boundaries between sub-grids do not overlap.

[0525] In addition, the encoding device 100 can also determine how each subgrid overlaps and display the determination result in the metadata.

[0526] The encoding device 100, for example, checks the number of sub-segments of two or more sub-grids, determines whether they are the same, and sets a flag indicating the determination result. For example, based on the determination result, the encoding device 100 generates a flag (flag information) indicating whether they are the same or different.

[0527] For example, the encoding device 100 checks the number of sub-segments for each of two or more sub-meshes. For example, if the number of sub-segments for two or more sub-meshes is different, the encoding device 100 sets a flag (correction flag) to be on. On the other hand, for example, if the number of sub-segments for two or more sub-meshes is the same, the encoding device 100 sets the flag to be off. That is, for example, the flag indicates that the number of sub-segments between two or more sub-meshes is different. In other words, it can also be called a flag indicating that the vertices at the boundary edges of the sub-segments are different or potentially different.

[0528] Alternatively, for example, the encoding device 100 compares the coordinates of all boundary vertices after subdivision of two or more subgrids, determines whether the boundary vertices are the same, and sets a flag based on the determination result.

[0529] Additionally, for example, the encoding device 100 compares the coordinates of the boundary vertices of two or more sub-grids and determines the value of a flag based on the comparison result. For example, if there is at least one non-common boundary vertex in two or more sub-grids, the encoding device 100 sets the flag to "on". On the other hand, for example, if all the boundary vertices of two or more sub-grids are the same, the encoding device 100 sets the flag to "off". That is, the flag is a flag indicating whether the boundary vertices of two or more sub-grids after subdividing n are the same.

[0530] This flag can also be set for the entire mesh frame. When the flag is set for the entire mesh frame, it is applied to the boundary edges of all sub-mesh areas associated with the current frame during boundary vertex correction processing. Alternatively, the flag can be set for each sub-mesh. When the flag is set for each sub-mesh, it is applied to the sub-mesh associated with the current sub-mesh ID during correction processing.

[0531] [Method for signaling notification of the correction method] In addition to the flag, the encoding device 100 can also determine how the decoding device 200 corrects the vertices (boundary vertices) and signal the type of the correction method. The decoding device 200 can also correct vertices based on the type of correction method.

[0532] Figure 65 This is a flowchart illustrating the subgrid overlap determination process performed by the decoding device 200 in the relevant implementation.

[0533] First, the decoding device 200 determines whether the boundaries between sub-grids overlap by parsing metadata (S511).

[0534] If the decoding device 200 determines that the boundaries between subgrids overlap ("Yes" in S512), it determines whether the decoding device 200 or the application program used to perform decoding-related processing performs overlapping-related processing (processing of overlapping subgrids) (S513).

[0535] If the decoding device 200 determines that it or the application is performing overlapping sub-mesh processing ("Yes" in S513), it performs overlapping sub-mesh processing (S514). As overlapping sub-mesh processing, the decoding device 200 performs, for example, sub-segmentation, detects the boundaries between sub-meshes (specifically, boundary edges), and performs prescribed processing such as boundary vertex correction on the boundaries between the sub-segmented sub-meshes.

[0536] On the other hand, for example, if the decoding device 200 determines that the boundaries between subgrids do not overlap ("No" in S512), or if it determines that the decoding device 200 or the application does not perform the process of overlapping subgrids ("No" in S513), it does not perform the process of overlapping subgrids, that is, the process of overlapping subgrids, and ends the process.

[0537] Figure 66 This is a flowchart illustrating the vertex correction processing performed by the decoding device 200 according to the relevant implementation method. Specifically, Figure 66 This represents a specific example of the processing in step S514.

[0538] Additionally, for example, the decoding device 200 at the beginning Figure 66 Prior to the processing shown, the data and metadata are decoded and parsed from the bitstream, and the multiple subgrids are sub-segmented. Then, the decoding device 200 performs... Figure 66 The processing is shown.

[0539] First, the decoding device 200 decodes two or more subgrids, flag information, and method information from the bit stream (encoded bit stream) (S521).

[0540] Method information is information that indicates the correction method (the way or type of correction method) for the boundary vertices.

[0541] Next, the decoding device 200 determines whether the flag information indicates an application correction (S522). That is, the decoding device 200 determines whether the flag is on or off.

[0542] If the flag information indicates that a correction is to be applied ("Yes" in S522), the decoding device 200 determines whether the method information indicates that a new vertex is to be added (S523). Specifically, the decoding device 200 determines whether the method information indicates that a new vertex is to be added to the boundary edge of at least one of two or more sub-meshes. In this way, the decoding device 200 performs the processing indicated by the method information on two or more sub-meshes based on the method information.

[0543] For example, when the decoding device 200 determines that the method information indicates the addition of a new vertex ("Yes" in S523), it imports one or more vertices into the boundary edge based on the method information (S524). Specifically, the decoding device 200 adds one or more vertices to at least one boundary edge of two or more sub-mesh units based on the method information. In this way, the decoding device 200 applies boundary vertex corrections to two or more sub-mesh units based on the method information.

[0544] On the other hand, for example, if the decoding device 200 determines that the method information does not indicate the addition of a new vertex ("No" in S523), it removes at least one vertex from one or more vertices on the boundary edge based on the method information (S525). Specifically, the decoding device 200 deletes at least one vertex from one or more vertices added (generated) through sub-segmentation on the boundary edge of at least one of two or more sub-meshes based on the method information.

[0545] Furthermore, if the flag information does not indicate that a correction is to be applied ("No" in S522), the decoding device 200 does not perform correction of the vertices in the boundary edge and ends the processing.

[0546] In addition, the decoding device 200 may perform processing based on method information, or it may not perform processing at all.

[0547] Alternatively, the decoding device 200 can also determine whether to add or delete vertices in the boundary edge using methods other than those described above, such as by analyzing the number of sub-segments of the submesh.

[0548] As described above, in the third method, flag information can also be decoded from the bitstream. The flag information indicates whether to correct the number of vertices in the boundary edge when combining more than two subgrids.

[0549] For example, the decoding device 200 applies sub-segmentation processing to all edges of a specified sub-mesh according to a specified number of sub-segments.

[0550] Additionally, for example, if the decoded flag information indicates that the number of vertices on the boundary edge is equal, the decoding device 200 may add vertices to the submesh or delete vertices from the submesh.

[0551] For example, the flag information is signaled as an SEI message. The decoding device 200 completes the sub-segmentation process, combines two or more adjacent sub-grids, and adjusts (corrects) the number of vertices. If the number of vertices changes, the topology of each sub-grid is updated accordingly.

[0552] In addition, triangles in the submesh can be reconstructed after correcting (adding or removing) vertices in the boundary edges.

[0553] Figure 67 This diagram illustrates a specific example of the correction processing for the vertices in the implementation method. Specifically, Figure 67 This is a diagram used to illustrate the process of adding vertices to the boundary edges based on method information.

[0554] Furthermore, whether to add or delete vertices can be determined without using method information, such as by any method, like the number of sub-segments of the submesh being parsed by the decoding device 200.

[0555] For example, the decoding device 200 appends (generates) new vertices at the boundary edge by copying the coordinates of vertices from adjacent sub-mesh.

[0556] In this example, we use submesh 1 with vertices A, B, C, D, and E, and submesh 2 with vertices B1, C1, X, Y, and Z for illustration. Submesh 1 and submesh 2 are adjacent submesh in the base mesh. Vertex B and vertex B1 are the same vertex in the base mesh, and vertex C and vertex C1 are the same vertex in the base mesh. The edges connecting vertices B and C in submesh 1 and connecting vertices B1 and C1 in submesh 2 are boundary edges.

[0557] Additionally, internal edges refer to edges that are not shared (or overlap) with other sub-grids.

[0558] Furthermore, in this example, subgrid 1 is subdivided 0 times, and subgrid 2 is subdivided 1 time. For example, the decoding device 200 by... Figure 67 (a) shows the submesh 1 subdivided 0 times to generate Figure 67 Subgrid 1 is shown in (b). That is, when the number of sub-segments is 0, the decoding device 200, as shown in (b), Figure 67 As shown in (b), it is incorrect. Figure 67 The subgrid 1 shown in (a) is subdivided. Additionally, for example, the decoding device 200 performs sub-segmentation by... Figure 67 Subgrid 2 shown in (e) is subdivided only once to generate Figure 67 Subgrid 2 is shown in (f). As... Figure 67As shown in (f), for submesh 2, new vertices Q1, R, S, T, U, V, and W are added (generated) by sub-segmentation, and new edges (added edges) connecting these vertices are added (generated). In this way, the decoding device 200 creates triangles inside submesh 2 by sub-segmenting submesh 2.

[0559] Decoding device 200, for example, in cases where the flag information indicates that no vertex modification is performed (i.e., the flag is off or flag=no), such as Figure 67 As shown in (c), it is incorrect. Figure 67 Submesh 1 shown in (b) undergoes vertex correction processing. On the other hand, the decoding device 200, for example, when the flag information indicates vertex correction (i.e., flag is ON or flag is yes) and the method information indicates adding vertices (i.e., method = ADD_VERTEX), performs the following processing: Figure 67 As shown in (d), a new vertex Q is added (generated) to submesh 1.

[0560] In addition, specific examples of method information will be described later (for example, see [reference]). Figure 75 ).

[0561] Furthermore, the decoding device 200 uses the new vertex Q to create triangles contained within subgrid 1. Specifically, the decoding device 200 uses the new vertex Q to divide triangle ABC into two triangles: triangle BAQ and triangle QAC.

[0562] For example, the decoding device 200 appends the vertex Q corresponding to vertex Q1 to the boundary edge in submesh 1 by comparing submesh 1 and submesh 2.

[0563] For example, the encoding device 100 determines whether the vertices in the boundary edges shared by a first sub-mesh (such as sub-mesh 1) and a second sub-mesh (such as sub-mesh 2) are the same in the first and second meshes, where the first mesh is formed by subdividing the first sub-mesh a first number of times, and the second mesh is formed by subdividing the second sub-mesh a second number of times. Furthermore, based on the determination result, the encoding device 100 encodes combination information related to the processing of combining the first and second meshes, such as flag information and / or method information, into a bitstream. Flag information, for example, indicates whether it is necessary to correct the vertices (specifically, the number of vertices) in the boundary edges of at least one of the first and second meshes. Additionally, method information indicates the correction method for the vertices in the boundary edges of at least one of the first and second meshes. In this example, the encoding device 100 encodes method information, including information indicating the addition of vertices to the boundary edges of at least one of the first and second meshes (ADD_VERTEX), into the bitstream. The decoding device 200 decodes the combined information from the bitstream. Based on the decoded combined information, it determines whether the vertices in the boundary edges are the same in the first grid and the second grid, where the first grid is formed by subdividing the first subgrid a first number of times, and the second grid is formed by subdividing the second subgrid a second number of times. For example, if the decoding device 200 determines that they are not the same, it corrects the vertices in the boundary edges of at least one of the grids based on the method information contained in the combined information. Based on the corrected vertices, it corrects the triangles contained in at least one of the grids, and then combines the first grid and the second grid after correcting the triangles contained in at least one of the grids.

[0564] Figure 68 This diagram illustrates a specific example of the correction processing for the vertices in the implementation method. Specifically, Figure 68 This is a diagram used to illustrate the process of removing vertices from boundary edges based on method information.

[0565] In this example, with Figure 67 The example shown is illustrated using subgrid 1 and subgrid 2.

[0566] Furthermore, in this example, subgrid 1 is subdivided only once, and subgrid 2 is subdivided only 0 times. For example, the decoding device 200 by... Figure 68 The submesh shown in (a) is generated by subdividing the submesh 1 once. Figure 68 Subgrid 1 is shown in (b). As... Figure 68As shown in (b), for submesh 1, new vertices Q, R, S, T, U, V, and W are added (generated) by sub-segmentation, and new edges (added edges) connecting these vertices are added (generated). Thus, the decoding device 200 creates triangles inside submesh 1 by sub-segmenting submesh 1. Additionally, for example, the decoding device 200... Figure 68 The submesh shown in (e) is generated by subdividing 2 sub-grids 0 times. Figure 68 Subgrid 2 is shown in (f). That is, when the number of sub-segments is 0, the decoding device 200, as shown in (f), Figure 68 As shown in (f), it is incorrect. Figure 68 Sub-segmentation is performed on sub-grid 1 as shown in (b).

[0567] For example, if the flag information indicates that no vertex modification is performed (i.e., the flag is OFF or flag=no), then the decoding device 200 may... Figure 68 As shown in (c), it is incorrect. Figure 68 The submesh 1 shown in (b) undergoes vertex correction processing. On the other hand, the decoding device 200 removes vertices from the boundary edges, for example, when the flag information indicates vertex correction (i.e., flag=ON or flag=yes) and the method information indicates vertex removal (i.e., method=REMOVE_VERTEX). In this example, the decoding device 200 generates [the following] by removing vertex Q. Figure 68 Subgrid 1 is shown in (d).

[0568] Thus, for example, the decoding device 200 removes vertices on the boundary edge by merging two or more vertices into one. In this example, the decoding device 200 removes the vertex Q in submesh 2 that does not have a corresponding vertex from multiple vertices (vertices B, C, and Q) on the boundary edge of submesh 1.

[0569] Furthermore, the decoding device 200 reconstructs the triangle contained in submesh 1 based on the removed vertex Q. That is, since vertex Q is removed, the decoding device 200 corrects submesh 1 by deleting the edges connected to vertex Q and constructing new triangles in submesh 1. Specifically, as... Figure 68 As shown in (d), the decoding device 200 creates triangle UTB and triangle BTC by adding an edge connecting vertex B and vertex T. Thus, for example, triangles UQT, BUQ, and QTC are replaced with triangle UTB and triangle BTC.

[0570] For example, the decoding device 200 removes vertices Q from the boundary edges of sub-mesh 1 that do not have corresponding vertices in sub-mesh 2 by comparing sub-mesh 1 and sub-mesh 2.

[0571] In addition, in this example, the encoding device 100 encodes method information containing information (REMOVE_VERTEX) into a bitstream, which indicates the removal of at least one vertex from the boundary edge of at least one of the first and second grids.

[0572] Figure 69 This is a diagram showing a specific example of the location of the storage flag information related to the implementation method.

[0573] like Figure 69 As shown, for example, the flag information is signaled in the SEI message of the bitstream.

[0574] Additionally, flag information can also be signaled in the header of the bitstream.

[0575] Figure 70 This is a diagram illustrating an example of the syntax for signaling notification of flag information in a relevant implementation method.

[0576] like Figure 70 As shown, the value represented by the flag information is, for example, a binary value. Furthermore, the value represented by the flag information is signaled at the frame level. For example, if the value represented by the flag information is equal to 1, the flag information indicates that correction processing is applied to all subgrids related to the current frame. On the other hand, if the value represented by the flag information is equal to 0, the flag information indicates that no correction processing is applied. In the absence of flag information, the value of the part of the flag information signaled is set to 0.

[0577] Figure 71 This is another example of the syntax for signaling notification of flag information in relation to an implementation method.

[0578] like Figure 71 As shown, flag information can also be signaled for each submesh. When flag[i] equals 1, it indicates that correction processing is applied to the submesh associated with submesh_id. Conversely, when flag[i] equals 0, it indicates that correction processing is not applied to the submesh associated with submesh_id. If flag[i] does not exist, it is set to 0.

[0579] Furthermore, the flag information can be set to indicate whether correction processing is applied, but it can also be set to indicate whether the vertices in the boundary edge are consistent. The decoding device 200 can also determine whether to apply correction processing based on this flag.

[0580] Alternatively, the encoding device 100 may not send the submesh_id. In this case, it can also be specified that the order of the loop is determined by the submesh_id.

[0581] Figure 72 This is a diagram showing a specific example of the location of information about the storage method for implementing the method.

[0582] like Figure 72 As shown, for example, method information is signaled within the SEI message of the bitstream.

[0583] In addition, method information can also be signaled in the header of the bitstream.

[0584] Figure 73 This is a diagram illustrating an example of the syntax for signaling method information in relation to an implementation method.

[0585] like Figure 73 As shown, the method information represents the identifier of the correction processing applied to the subgrid associated with the current frame.

[0586] Figure 74 This is another example of the syntax used to illustrate the signaling notification of method information related to the implementation method.

[0587] like Figure 74 As shown, method information can also represent identifiers for correction processes applied to submesh associated with submesh_id.

[0588] Figure 75 This diagram illustrates a specific example of a method for modifying the vertices of a particular implementation. Specifically, Figure 75 It is a list of examples of the relationship between the identifier represented by the method information and the type of correction method (the name of the correction method).

[0589] For example, if the method information is represented as 0, it means that the decoding device 200 does not perform correction processing. Alternatively, if the method information is represented as 1, it means that the decoding device 200 performs vertex addition processing during correction processing. For example, if the method information is represented as 2, it means that the decoding device 200 performs vertex removal processing during correction processing.

[0590] Alternatively, the correction process can also be performed without signaling the method information in the bitstream, and instead be derived by the decoding device 200.

[0591] For example, the decoding device 200 can also compare the number of sub-segments of two sub-grids by calculating the difference in the number of sub-segments, and decide whether to add or remove vertices. For example, if the number of sub-segments of sub-grid 1 is less than that of sub-grid 2, a correction process of adding vertices is applied to sub-grid 1. On the other hand, if the number of sub-segments of sub-grid 1 is more than that of sub-grid 2, a correction process of removing vertices is applied to sub-grid 1. Furthermore, if sub-grid 1 has the same number of sub-segments as sub-grid 2, neither vertices are added nor removed.

[0592] Additionally, for example, if the method information indicates that a vertex addition process is being performed in the correction process, and the number of sub-segments for submesh 1 is less than the number of sub-segments for submesh 2, then the vertex addition correction process is applied to submesh 1. On the other hand, for example, if the method information indicates that a vertex deletion process is being performed in the correction process, and the number of sub-segments for submesh 1 is more than the number of sub-segments for submesh 2, then the vertex removal correction process is applied to submesh 1.

[0593] also, Figure 69 The examples shown and Figure 72 The examples shown can also be combined to achieve the same result.

[0594] in addition, Figure 70 The example shown Figure 71 The example shown Figure 73 The examples shown and Figure 74 The examples shown can also be combined to achieve the same result.

[0595] Next, we will illustrate an example of a subgrid that is adjacent to multiple subgrids.

[0596] Figure 76 This is a diagram illustrating an example of the relationship between the number of sub-segments of multiple subgrids and the flag information in an embodiment.

[0597] exist Figure 76 In the example shown, submesh 3 is adjacent to submesh 1 and submesh 2, and their boundaries overlap. Furthermore, submesh 1 is subdivided once (repeated once), submesh 2 is subdivided twice, and submesh 3 is subdivided twice. Therefore, the vertices on the shared boundary edges of submesh 1 and submesh 3 are highly likely to be different from the vertices generated by subdividing submesh 1 and submesh 3. Conversely, the vertices on the shared boundary edges of submesh 2 and submesh 3 are highly likely to be the same as the vertices generated by subdividing submesh 2 and submesh 3.

[0598] For example, sub-mesh 1 only shares a boundary edge with sub-mesh 3 and the number of sub-segments is different, so the encoding device 100 sets flag=ON for the flag information corresponding to sub-mesh 1.

[0599] Furthermore, for example, sub-mesh 2 shares a boundary edge with sub-mesh 3 and has the same number of sub-segments, so the encoding device 100 sets flag=OFF for the flag information corresponding to sub-mesh 2.

[0600] Furthermore, submesh 3 shares a boundary edge with submesh 1 and submesh 2. The number of sub-segments of the boundary edge of submesh 3 is the same as that of submesh 2, but different from that of submesh 1.

[0601] For example, if the number of sub-segments of any one of the multiple sub-grids with overlapping boundaries (in other words, multiple sub-grids sharing a common boundary edge) is different, or if the vertices of the sub-grids after sub-segmentation are not the same, the encoding device 100 sets flag=ON.

[0602] Figure 77 This is another example of a diagram used to illustrate the relationship between the number of sub-segments of multiple subgrids in an embodiment and the flag information.

[0603] exist Figure 77 In the example shown, submesh 3 is adjacent to submesh 1 and submesh 2, and their boundaries overlap. Furthermore, submesh 1 is subdivided once (repeated once), submesh 2 is subdivided three times, and submesh 3 is subdivided twice. Therefore, the vertices on the shared boundary edges of submesh 1 and submesh 3 are highly likely to be different from the vertices generated by subdividing submesh 1 and submesh 3. Similarly, the vertices on the shared boundary edges of submesh 2 and submesh 3 are highly likely to be different from the vertices generated by subdividing submesh 2 and submesh 3.

[0604] For example, sub-mesh 1 shares a boundary only with sub-mesh 3 and the number of sub-segments is different, so the encoding device 100 sets the flag=ON corresponding to sub-mesh 1.

[0605] Furthermore, for example, sub-mesh 2 only shares a boundary edge with sub-mesh 3 and the number of sub-segments is different, so the encoding device 100 sets the flag=ON corresponding to sub-mesh 2.

[0606] Furthermore, for example, subgrid 3 shares a boundary edge with subgrid 1 and subgrid 2, and the number of sub-segments is different from the number of sub-segments of subgrid 1 and subgrid 2. Therefore, the encoding device 100 sets the flag=ON corresponding to subgrid 3.

[0607] Furthermore, when the encoding device 100 represents method information in metadata, the method information of each subgrid can also be represented by representing the information of multiple adjacent subgrids for a subgrid.

[0608] For example, the encoding device 100 can, in the loop of subgrid 3 in the syntax, indicate that flag=ON and method=add for subgrid 1, and indicate that flag=ON and method=remove for subgrid 2.

[0609] As described above, for example, the encoding device 100 determines whether the vertices in the boundary edges (i.e., the edges shared by the first sub-grid (e.g., sub-grid 1) and the second sub-grid (e.g., sub-grid 2) are the same in the first and second grids, where the first grid is formed by subdividing the first sub-grid a first number of times, and the second grid is formed by subdividing the second sub-grid a second number of times. Here, for example, if the first and second number of times are the same, the encoding device 100 determines that the vertices in the boundary edges are the same in the first and second grids; if the first and second number of times are different, it determines that the vertices in the boundary edges are different in the first and second grids. Furthermore, for example, if the encoding device 100 determines that the vertices in the boundary edges are different in the first and second grids, it encodes the combination information containing method information into the bitstream. Additionally, for example, if the encoding device 100 determines that the vertices in the boundary edges are the same in the first and second grids, it may generate a bitstream that does not contain method information.

[0610] In addition, the encoding device 100 may also represent in the bit stream (specifically, the metadata contained in the bit stream) flags indicating whether the boundaries between subgrids overlap, and the type of overlap if the boundaries between subgrids overlap.

[0611] Figure 78 This is a diagram illustrating an example of the syntax for signaling information indicating the types of overlap of multiple subgrids in a relevant implementation.

[0612] `submesh_duplicate_flag` is a flag indicating whether submeshes that make up a mesh (e.g., the base mesh) overlap with other submeshes. For example, `submesh_duplicate_flag=1` indicates overlap. On the other hand, `submesh_duplicate_flag=0` indicates no overlap.

[0613] `submesh_duplicate_type` indicates the type (how) of submesh overlap. For example, if `submesh_duplicate_type=0`, it means that the outer boundary edges and vertices of the submesh overlap, while edges, vertices, and faces located inside the outer boundary do not overlap. On the other hand, if `submesh_duplicate_type=1`, it means that not only the outer boundary of the submesh overlaps, but also edges, vertices, and faces located inside the outer boundary overlap.

[0614] Alternatively, the encoding device 100 may use a flag in the bitstream (more specifically, the metadata contained in the bitstream) to indicate that subgrids may overlap, instead of a flag indicating subgrid overlap.

[0615] Furthermore, such flags (parameter sets) can be stored in sequence-level parameter sets, frame-level parameter sets, or both parameter sets. In the case where they are stored in both parameter sets, for example, the encoding device 100 uses the frame-level parameter set to override the sequence-level parameter set.

[0616] In addition, such a set of parameters can also be shown in SEI.

[0617] Furthermore, regarding the flags indicating whether the boundary vertices after sub-segmentation are identical at the boundary edges between submesh areas, or the flags indicating whether to adjust the boundary vertices after sub-segmentation of the submesh, it can be determined that the encoding device 100 performs signaling notification when `submesh_duplicate_flag` is ON, and does not perform signaling notification when it is OFF. Alternatively, it can be determined that the encoding device 100 cannot perform signaling notification when it is OFF. In such a case, it can also be determined that the flag set through signaling notification is invalid when signaling notification is performed.

[0618] Furthermore, when submesh_duplicate_flag is ON, the decoding device 200 can also perform a determination process to determine whether to perform correction of the boundary vertices after sub-segmentation; when OFF, it is determined that no correction is performed.

[0619] Therefore, by including the aforementioned flags in the higher-level metadata through the encoding device 100, the decoding device 200 can determine whether there is a possibility of overlap between subgrids and other subgrids. Thus, the decoding device 200 can determine, based on the flags, whether a prescribed process (correction process) needs to be applied to the overlapping boundary edges.

[0620] When the decoding device 200 does not need to perform such processing, it does not need to detect boundary information or adjust the boundary vertices after sub-segmentation of the boundary, nor does it need to receive, decode, or parse associated information. Therefore, the processing load can be reduced.

[0621] Alternatively, by having the encoding device 100 send only the aforementioned flag, the decoding device 200 determines that the boundaries may overlap, and then analyzes the number of sub-segments of each sub-grid to determine whether the boundary vertices after sub-segmentation are the same, and whether the boundary vertices need to be corrected.

[0622] [Effect of Method 3] According to the structure of the third approach, the subjective quality of the entire mesh is improved by merging sub-mesh encoded using different methods of sub-segmentation number or sub-segmentation method (sub-segmentation scheme).

[0623] [Combination with other methods] The encoding device 100 of this method can be implemented by combining at least a portion of other methods disclosed herein. Furthermore, this method can also be implemented by combining a portion of the process represented by a flowchart following this method, a portion of the structure of a device, and / or a portion of the syntax, etc., with other methods.

[0624] Furthermore, the above-described processing of the encoding device 100 can also be performed in the decoding device 200 in the same way.

[0625] Furthermore, it is not always necessary to include all the constituent elements described in this method; it is also possible to include only a portion of the constituent elements of the third method.

[0626] <Hierarchical Structure Based on Sub-Segmentation> Next, the LoD (Level of Detail) level in subdivision will be explained.

[0627] Figure 79 This is a diagram used to illustrate the LoD (Location of Implementation) related to the implementation method.

[0628] Figure 79 The triangle ABC shown is the mesh before subdivision. Specifically, triangle ABC is the base mesh. Triangle ABC (i.e., the base mesh) is defined as having a LoD level of 0. In other words, each vertex of triangle ABC is defined as a vertex belonging to LoD0.

[0629] Next, the vertices generated after the basic mesh is subdivided once are set to LoD1. Figure 79In the example shown, the vertices represented by quadrilaterals (vertices D, E, and F) are vertices belonging to LoD1.

[0630] Therefore, the vertices generated after the second sub-segmentation are designated as LoD2. Figure 79 In the example shown, the vertices (vertex G, etc.) are represented by triangles.

[0631] For example, a vertex F in LoD1 is generated using vertices A and C in LoD0. Here, vertices A and C are sometimes referred to as the parent vertices of vertex F. A parent vertex is a vertex in the LoD level that belongs to the next higher level. That is, for example, vertices A and C belonging to LoD0 are the parent vertices of vertex F belonging to LoD1.

[0632] For example, vertex A belonging to LoD0 and vertex F belonging to LoD1 are the parent vertices of vertex G. In other words, vertex G is a child vertex of vertex A belonging to LoD0 and vertex F belonging to LoD1. Thus, it is sufficient if at least one of the parent vertices has a LoD level one level higher than the child vertex.

[0633] Here, the LoD level is sometimes referred to as the LoD index. For example, the LoD index of LoD0 is 0.

[0634] When the number of sub-segments between sub-mesh is different, that is, when the number of LoD levels between sub-mesh is different, the above method shows an example of generating vertices and combining them by adding and / or removing vertices based on information about whether to add or remove vertices when combining (merging) vertices at the overlapping boundaries of multiple sub-mesh.

[0635] As another approach, one could consider combining vertices at the overlapping boundaries of multiple subgrids by calculating the distance to adjacent vertices (neighboring points) and combining vertices whose calculated distances are within a specified threshold.

[0636] Furthermore, this section explains how points whose distance to adjacent points is within a specified threshold are combined. However, it also allows for the creation of a correspondence between vertices of one sub-grid on an overlapping boundary and the nearest vertex of the other sub-grid, and the combination of these two corresponding vertices. The specified threshold can be arbitrarily determined and is not specifically limited.

[0637] However, these methods sometimes generate vertices incorrectly, combining incorrect vertices may result in high-density points, or holes may be made in the mesh, making it impossible to generate a mesh of the appropriate shape.

[0638] Therefore, in order to solve the above problems, the encoding device 100 and the decoding device 200 hereby perform processing that takes into account the LoD level of the vertices (also called boundary vertices) at the boundaries between multiple subgrids.

[0639] Hereinafter, in this embodiment, an example of a method for combining multiple decoded segmented subgrids in a subgrid bonding section of a decoding device is shown.

[0640] For example, the boundary vertices after sub-segmentation are only joined if their "parent vertex" has already been joined.

[0641] In addition, for example, the encoding device 100 may signal in the bitstream a flag indicating the removal of boundary vertices of adjacent subgrids that do not have corresponding LoDs, or a flag indicating the artificial generation of corresponding new boundary vertices for adjacent subgrids.

[0642] In addition, for example, the encoding device 100 and / or the decoding device 200 create new triangles from the three-dimensional geometry and UV mapping of adjacent sub-mesh.

[0643] Additionally, for example, the decoding device 200 combines vertices with the same LoD level on overlapping edges (edges) between adjacent subgrids of the combining object (i.e., the vertex to be combined with a certain vertex), without combining points with different LoD levels.

[0644] Alternatively, for example, the two vertices can be combined if they are at the same LoD level and the distance between them is within a specified threshold.

[0645] In addition, for example, when the encoding device 100 and the decoding device 200 combine two vertices that are far apart, they calculate the position coordinates of the combination point (the vertex generated by combining two vertices) using a prescribed method.

[0646] Alternatively, the calculated position coordinates (position coordinates of the junction) can be the midpoint or centroid of the two points, or the position coordinates can be made closer to one of the vertices by using weighting coefficients.

[0647] Furthermore, for example, when a vertex at a higher level (i.e., a level whose LoD index is smaller than a certain vertex) has already been combined, the encoding device 100 and the decoding device 200 determine whether to combine a vertex at a lower level (i.e., a level whose LoD index is larger than that higher level).

[0648] Additionally, for example, when the number of LoD levels differs between sub-mesh, the decoding device 200 makes the number of vertices (LoD levels) consistent between sub-mesh by generating vertices on the boundary of one sub-mesh (the boundary edge of the sub-mesh) (e.g., further implementing sub-segmentation), and then combines the vertices.

[0649] Alternatively, for example, the decoding device 200 makes the number of vertices in the sub-mesh consistent with the number of vertices in the sub-mesh by removing vertices from the boundary of one sub-mesh, and then combines the vertices.

[0650] Furthermore, the decision of whether to add or remove vertices can be based on information notified by the bitstream signaling, or it can be made by the decoding device 200 through a prescribed method.

[0651] Based on the conditions described above, it may be possible to solve the problems of generating high-density vertices or generating meshes of appropriate shapes, such as those with mesh openings.

[0652] Figure 80 This is a flowchart illustrating the vertex combining process in the decoding apparatus 200 of the relevant implementation. Furthermore, the combining process is sometimes also referred to as merging or zipping. Figure 80 The flowchart shown illustrates a specific example of the process in which the decoding device 200 obtains boundary vertices (specifically, encoded information related to boundary vertices, such as position information) from the bitstream, decodes the boundary vertices of the subgrid (specifically, the encoded information related to boundary vertices), sub-segments the two subgrids, and then combines the two sub-segments.

[0653] First, the decoding device 200 obtains a flag (flag information) from the bitstream indicating the addition or removal of vertices with a LoD that do not correspond to adjacent subgrids (S601). Specifically, the decoding device 200 obtains the encoded flag information from the bitstream and performs decoding.

[0654] Next, the decoding device 200 determines for each boundary vertex of one sub-mesh whether there exists a LoD index corresponding to an adjacent sub-mesh (the other sub-mesh) (S602). That is, the decoding device 200 determines whether the LoD level numbers of the two sub-mesh are the same.

[0655] When the decoding device 200 determines that a corresponding LoD index exists ("Yes" in S602), it combines the boundary vertices of one sub-mesh with the boundary vertices of the adjacent other sub-mesh (more specifically, the boundary vertices of the same LoD) (S603). That is, since the number of LoD levels (more specifically, the number of sub-segments) is the same, the decoding device 200 combines vertices without adding or removing vertices.

[0656] On the other hand, if the decoding device 200 determines that there is no corresponding LoD index ("No" in S602), it determines whether the decoded flag indicates the addition of a new vertex (S604).

[0657] When the decoding device 200 determines that the decoded flag indicates the addition of a new vertex ("Yes" in S604), for each boundary vertex, if the parent vertex of that boundary vertex has been combined with one or more vertices of the adjacent sub-mesh, it generates a new boundary vertex in the adjacent sub-mesh (S605). Furthermore, if the parent vertex of that boundary vertex has not been combined with one or more vertices of the adjacent sub-mesh, the decoding device 200 may not generate a new boundary vertex in the adjacent sub-mesh.

[0658] Next, the decoding device 200 combines the boundary vertices of one sub-mesh generated based on the flags with the vertices generated by the sub-segmentation of the other sub-mesh (S606).

[0659] On the other hand, if the decoding device 200 determines that the decoded flag does not indicate the addition of new vertices ("No" in S604), it considers it to indicate the removal of vertices and removes a portion of the multiple boundary vertices of the current submesh using a prescribed method (S607). Furthermore, this prescribed method can be arbitrarily determined and is not particularly limited.

[0660] Next, the decoding device 200 does not use the removed vertices to combine the boundary vertices (S608).

[0661] In addition, the decoding device 200 can also determine at the beginning of the process whether the LoD level between sub-mesh is consistent. If it is consistent, the decoding and parsing of the flags are not performed to combine the vertices.

[0662] Alternatively, the decoding device 200 may first determine whether the LoD level is 0 (basic grid). If it is a basic grid, this process is skipped. If the LoD level is 1 or higher, this process is implemented.

[0663] Furthermore, for example, when the number of levels is the same, since there are corresponding levels, the decoding device 200 does not use the addition or removal of vertices to combine vertices. On the other hand, for example, when the number of levels is inconsistent, the decoding device 200 adds or removes vertices from either of the two sub-mesh grids to be combined, and combines the vertices after making the number of vertices at the boundary edges consistent.

[0664] Next, the method for combining vertices between sub-mesh elements with LoD-related constraints will be explained.

[0665] Figure 81This is an explanatory diagram illustrating an example of the vertex combination process in a relevant implementation method. Specifically, Figure 81 This illustrates an example of merging the boundaries of two sub-mesh grids (zipper merge) by combining (merging) vertices with the same LoD in two different sub-mesh grids. Additionally, the vertices are located at... Figure 81 The two ends of each line segment in subgrid 1 and subgrid 2 are shown. Vertices belonging to LoD0 are located at the two ends of the line segments represented by solid lines, vertices belonging to LoD1 are located at the two ends of the line segments represented by dashed lines, and vertices belonging to LoD2 are located at the two ends of the line segments represented by dotted lines. In the following... Figures 82-84 The same applies to China.

[0666] Figure 81 This is an example of combining two subgrids with different numbers of sub-divisions (specifically, subgrid 1 obtained by sub-division 2 and subgrid 2 obtained by sub-division 1). Additionally, in Figure 81 In the example described, vertices are combined if they share the same LoD, and are not combined if they do not share the same LoD.

[0667] Vertices A, B, C, D, E, and F are the vertices of the base mesh of LoD0, vertices X, Y, Z, L, M, and N are the vertices of LoD1, and the remaining vertices are the vertices of LoD2.

[0668] In addition, each vertex is a vertex that has been corrected by displacement vectors after sub-segmentation.

[0669] When combining the boundaries of two sub-mesh, vertices of the same LoD in different sub-mesh are combined with each other.

[0670] In this example, the vertices LoD0 and LoD1 of submesh 1 are combined with the vertices LoD0 and LoD1 of submesh 2, respectively. The coordinates of the combined vertices (combination points) between two vertices are calculated, for example, as a weighted sum of their averages. For example, the coordinates of the combination point are calculated by M(A, D) = w1 × A + w2 × D.

[0671] Furthermore, M(A, D) represents the coordinates of the point where vertices A and B intersect, generated between vertices A and D. A is the coordinate of vertex A. D is the coordinate of vertex D. w1 is the weighting factor multiplied by the coordinates of vertex A. w2 is the weighting factor multiplied by the coordinates of vertex D.

[0672] For example, vertices of LoD2 in submesh 1 are not bound to vertices of submesh 2. The position coordinates of the binding point can also be determined based on the position coordinates of the two vertices before binding. For example, it could be the midpoint of the coordinates of the two vertices, or a weight could be applied to the coordinates of either of the two vertices.

[0673] Furthermore, the coordinates of the fusion point can be determined not based on the two vertices before fusion, but based on the position information of vertices at a higher level, or based on the coordinates of the two vertices before fusion and the vertices at a higher level above those two vertices. The same applies to the coordinates of the fusion point's position (3D position) and the coordinates of its attribute map (UV map) (2D coordinates). For example, the position coordinates of the fusion point can also be determined not based on the two vertices before fusion, but based on the position information of the two vertices before fusion and the vertices at a higher level above those two vertices. Similarly, the coordinates of the fusion point's attribute map (e.g., attribute information) can also be determined not based on the two vertices before fusion, but based on the coordinates of the attribute maps of the two vertices before fusion and the vertices at a higher level above those two vertices (e.g., attribute information).

[0674] Furthermore, if the number of sub-segments of two sub-mesh is 0 or greater, this method can be applied even to combinations of sub-segmentation numbers. For example, if the difference in the number of sub-segments of two sub-mesh is 2 or greater, it can also be applied to combinations of sub-mesh with the same difference.

[0675] Figure 82 This is an explanatory diagram illustrating another example of the vertex combination process in a relevant implementation method. Specifically, Figure 82 This example illustrates how removing vertices of the LoD (non-matching LoD) from the subgrid with more sub-segments that are not present in the other subgrid can equalize the number of vertices at the boundary edge. In other words, in this example, the decoding device 200 removes vertices with inconsistent LoDs from the two subgrids.

[0676] When the decoding device 200 combines two subgrids with different sub-segments (specifically, subgrid 1 obtained by sub-segmenting twice and subgrid 2 obtained by sub-segmenting once), it combines vertices if vertices with the same LoD exist, and does not combine vertices if vertices with the same LoD do not exist.

[0677] In addition, if vertices of the same LoD do not exist, the decoding device 200 removes the vertices of the LoD from the subgrid with more sub-segments in two subgrids, and reconstructs the mesh by combining the two subgrids.

[0678] After combining the vertices of the same LoD from two different submeshes, this method removes the vertices of inconsistent LoDs from the submesh. In this example, vertices R and Q on the boundary edges of inconsistent LoDs in submesh 1 are removed. Then, the vicinity of the removed vertices is remeshed. For example, after removing vertex R, triangles PAR, PRS, and SRX are replaced with triangles PAS and SAX. Similarly, after removing vertex Q, triangles TXQ, TQK, and KQB are replaced with triangles TXK and KXB.

[0679] In the presence of an attribute map, the decoding device 200 also performs the same remeshing on the triangles of the attribute map. The attribute map is, for example, a UV map. Therefore, the number of vertices in the boundary edges is equalized, and artifacts are removed after combining. UV coordinates (two-dimensional UV coordinates) are coordinates representing the position of the texture corresponding to the mesh within the UV map (two-dimensional map). For each point, there is one UV coordinate. When vertices are removed, the UV coordinates (information representing UV coordinates) are also removed along with the vertices. Furthermore, the information representing the newly generated mesh by removing vertices includes information representing the group and connection relationships of the three three-dimensional points, and information representing the texture located at the three UV coordinates corresponding to the three three-dimensional points becomes information representing the new texture.

[0680] Furthermore, even if vertices are not removed during vertices combining, the original UV coordinates can still be used as is. For example, when combining vertices A and D, even if the position coordinates of vertices A and D are shifted, their UV coordinates can still be used as is. Alternatively, UV coordinates can be corrected using a defined method.

[0681] Furthermore, even when combining two vertices, the decoding device 200 can also exist independently of combining the information of those two vertices, possessing their respective position information and UV coordinate information. In other words, the decoding device 200 can also set the combining points as vertices A and D with the same position information but different UV coordinate information.

[0682] Figure 83 This is an explanatory diagram illustrating another example of the vertex combination process in a relevant implementation method. Specifically, Figure 83 This illustrates an example of equalizing the number of vertices at a boundary edge by generating vertices in the submesh with fewer sub-segments between two submeshes. That is, in Figure 83 In this process, vertices are generated for non-matching LoD.

[0683] When the decoding device 200 combines two subgrids with different sub-segments (specifically, subgrid 1 obtained by sub-segmenting twice and subgrid 2 obtained by sub-segmenting once), it combines vertices if they have vertices with the same LoD, and does not combine vertices if they do not have vertices with the same LoD.

[0684] In addition, when vertices of the same LoD do not exist, the decoding device 200 adds the vertices of the LoD to a subgrid with fewer sub-segments, for example, and reconstructs the mesh by combining the two subgrids.

[0685] After combining vertices with the same LoD from different submesh, this method generates new vertices for inconsistent LoDs in the submesh with fewer subsegments. For example, new vertices G and H are generated on the boundary edge of submesh 2 in a manner corresponding to vertices R and Q of submesh 1.

[0686] As an example, in this method, the parent vertex of vertex R is first found. In this example, the parent vertices of vertex R are vertex A and vertex X. Furthermore, the vertices of submesh 2, which is associated with vertex A and vertex X, are vertex D and vertex L, respectively.

[0687] Next, in this method, vertex G is generated between vertex D and vertex L of submesh 2. The coordinates of the vertex located between vertex R and vertex G, generated by combining vertex R and vertex G, are calculated, for example, as the average weighted sum of the coordinates of vertex R and the coordinates of vertex G.

[0688] Subsequently, the vicinity of the generated vertices is remeshed. For example, after generating vertex G, triangle DML is replaced with triangles DMG and GML. Similarly, after generating vertex H, triangle LNF is replaced with triangles LNH and HNF. In the presence of attribute maps, triangles with attribute maps are also remeshed in the same way. Attribute maps are, for example, UV maps. Therefore, the number of vertices in the boundary edges is equalized, and artifacts are removed after combining.

[0689] Furthermore, for example, when generating vertices, the decoding device 200 generates UV coordinates (UV coordinate information) for the texture of the vertices simultaneously with generating the vertex position coordinates. For instance, when generating vertex G at the midpoint of vertex D and vertex L, the decoding device 200 uses the midpoint of the UV coordinates of vertex D and vertex L as the new UV coordinates of vertex G. The calculation of these UV coordinates can also use points other than the midpoint. By using the same formula as the method for generating the position coordinates of vertices D and L to calculate the UV coordinates, it is possible to improve the quality. Additionally, when combining vertex G and vertex R, even if the position of vertex G is corrected, the previously calculated UV coordinates can be used as is without correction. That is, for example, the decoding device 200 can generate UV coordinates when generating vertices within the same submesh, and not correct the UV coordinates when combining them between different submeshes.

[0690] Figure 84 This is an explanatory diagram illustrating another example of the vertex combination process in a relevant implementation method. Specifically, Figure 84 This illustrates an example of zipping together the boundaries of two partial meshes by combining vertices based on parent-vertex relationships. That is, Figure 84 This represents an example of a combination of vertices that uses parent-child relationships.

[0691] In this example, the boundary zipper method is used. In the boundary zipper method, vertices of LoD0 from different subgrids are first combined. As an example, the decoding device 200 uses the nearest neighbor principle to find the vertices to be combined. Next, vertices with higher LoD indices are combined only if their parent vertices are already combined. For example, vertices D and F, which are the parent vertices of vertex L, are already combined with vertices A and B, which are the parent vertices of vertex X; therefore, vertex L is combined with vertex X. For example, even though vertex W is the closest vertex to vertex L among the vertices of LoD1 in subgrid 1, vertex W is not combined with vertex L because the parent vertices of vertex W and vertex L are not combined with each other. Instead, vertex W is combined with vertex K.

[0692] Thus, for example, the decoding device 200 determines whether the parent vertex of the vertices of the vertices to be combined has already been combined. If it has, the child vertices of the parent vertex to be combined are determined as the vertices to be combined. Through this process, combining can be performed without deriving distance information or performing distance-based search processing. Therefore, the processing load can be reduced.

[0693] In addition to the above processing, the decoding device 200 can also use the vertices of the same LoD as the merging object to search for distances, and determine whether to set it as a merging object based on the searched distance using a prescribed method. Therefore, although the processing load increases, it is possible to achieve high-quality and high-precision mesh merging.

[0694] Figure 85 This is a flowchart illustrating the combination process of each LoD vertex in the decoding device 200 of the relevant implementation.

[0695] First, the decoding device 200 obtains metadata representing the number of sub-segments of each of the two subgrids from the bitstream, extracts the number of sub-segments of each of the two subgrids from the obtained metadata, and compares them (S611). Here, the number of sub-segments of one of the two subgrids is set to N, and the number of sub-segments of the other subgrid is set to M. N and M are both integers greater than or equal to 0.

[0696] Next, the decoding device 200 performs sub-segmentation on the two sub-grids based on the extracted sub-segmentation number, and combines the vertices of LoD0 of the two sub-grids with each other (i.e., the vertices of the base grids of the two sub-grids with each other) (S612).

[0697] Next, the decoding device 200 combines the vertices from LoD1 to LoDX of the two sub-grids with each other (S613). Here, X is min(N, M). That is, the decoding device 200 performs the combination process up to the vertices of the common LoD level in the two sub-grids. In other words, when there is a LoD level to be combined, the decoding device 200 performs the combination process directly without adding or removing vertices.

[0698] Next, the decoding device 200 combines the vertices from LoDX+1 to LoDY of the two sub-meshes with each other (S614). Here, Y is max(N, M). That is, the decoding device 200 performs the combining process for vertices of LoD levels that do not exist in one of the two sub-meshes. Specifically, when there is no LoD level for combining objects, the decoding device 200 performs the addition or removal of vertices and the combining process as described above. The decoding device 200 determines whether to add or remove vertices, for example, by parsing metadata (flags), i.e., based on the metadata.

[0699] Figure 86 This is a flowchart illustrating the vertices combining of LoD0 in the decoding apparatus 200 according to an embodiment. Specifically, Figure 86 This is a flowchart showing the details of the processing in step S612.

[0700] First, the decoding device 200 searches for one or more vertices of LoD0 from multiple boundary vertices of the other sub-mesh (the corresponding sub-mesh) that is combined with one of the two sub-meshes. Among the searched vertices of LoD0, the vertex that is closest to the LoD0 vertex of one sub-mesh is determined as the combination object of the LoD0 vertex of that sub-mesh (S621). That is, the decoding device 200 combines the LoD0 vertices of the two closest sub-meshes with each other.

[0701] Next, the decoding device 200 determines the position coordinates of the joining point based on the position coordinates of the two joined vertices (S622).

[0702] Next, the decoding device 200 determines the UV coordinates of the junction point based on the UV coordinates of the two vertices to be joined (S623).

[0703] Next, the decoding device 200 reconstructs the mesh (the mesh formed by combining two sub-mesh) based on the position coordinates of the junction point and the UV coordinates (S624).

[0704] The same processing is applied to vertices other than LoD0.

[0705] In addition, in the above, X is set to the minimum value of N and M, and Y is set to the maximum value of N and M, but X and Y can also use other values.

[0706] In addition, in the processing of LoD1~LoDX, the fusion process (determining the position coordinates of the fusion point, shifting the position coordinates of the fusion point, determining the UV coordinates of the fusion point, and reconstructing the mesh) can be performed on each LoD, or the fusion process can be performed uniformly in multiple LoDs.

[0707] In addition, in the processing of LoDX+1~LoDY, the fusion process (adding vertices, removing vertices, determining the position coordinates of the fusion point, shifting the position coordinates of the fusion point, determining the UV coordinates of the fusion point, and reconstructing the mesh) can be performed on each LoD, or the fusion process can be performed uniformly in multiple LoDs.

[0708] Furthermore, the sub-segmentation described above, which is assumed to be performed by the decoding device 200, can also be performed by the encoding device 100. Additionally, the encoding device 100 may or may not perform sub-segmentation of the basic grid. Furthermore, after sub-segmenting multiple subgrids, the encoding device 100 may or may not combine the sub-segmented subgrids.

[0709] Method information, for example, is encoded into a bitstream. Figure 72 As shown, method information is signaled within the SEI message of the bitstream.

[0710] Figure 87 This is another example of the syntax used to illustrate signaling notification of method information related to an implementation method. Specifically, Figure 87 Examples of syntax for setting parameters for a grid frame.

[0711] Regarding method information, for example, encoding is performed using ue(v) exponential Golomb coding per frame. For instance, if the value represented by the method information (method value) is equal to 0, the method information indicates that inconsistent LoD correction is not performed. Alternatively, for example, if the value represented by the method information (method value) is equal to 1, the method information indicates that new vertices are generated on the submesh by performing fewer subsegments (i.e., appending vertices). Additionally, for example, if the value represented by the method information (method value) is equal to 2, the method information indicates that vertices with inconsistent LoD are removed. In the absence of method information, for example, the method value is considered 0.

[0712] For example, such as Figure 76 As shown, when the method information is represented as 0, it indicates that the decoding device 200 does not perform correction processing. Alternatively, for example, when the method information is represented as 1, it indicates that the decoding device 200 performs vertex addition processing during correction processing. For example, when the method information is represented as 2, it indicates that the decoding device 200 performs vertex removal processing during correction processing.

[0713] Alternatively, during the correction process, the method information may not be signaled in the bitstream, but may be derived by the decoding device 200.

[0714] Additionally, for example, such as Figure 75 As shown, method information can also represent identifiers for correction processes applied to submesh associated with the current frame. For example, method information can also represent identifiers for correction processes applied to submesh associated with submesh_id.

[0715] Next, the variations of the bonding method will be explained.

[0716] Furthermore, the following explanation uses the midpoint of the two combined vertices as an example, but weights can also be added, and other calculation formulas can be used.

[0717] Figure 88 This is an explanatory diagram illustrating another example of the vertices joining process in relation to an implementation method.

[0718] Furthermore, in this example, the decoding device 200 generates vertex I of the LoDN by combining vertex A of the LoDN of submesh 1 and vertex X of the LoDN of submesh 2. Additionally, the decoding device 200 generates vertex K of the LoDN by combining vertex C of the LoDN of submesh 1 and vertex Z of the LoDN of submesh 2. Furthermore, the decoding device 200 generates vertex J by combining vertex B of the LoDN+1 of submesh 1 and vertex Y of submesh 2. Vertices Y and J are equivalent to vertices of LoDN+1.

[0719] Decoding device 200, for example, uses two vertices at the same LoD level to determine the position coordinates of the junction point of LoDN and LoDN+1. Decoding device 200 determines the position coordinates of the junction point of LoDN, for example, by calculating I = (A + X) / 2 and K = (C + Z) / 2. I, A, and X are the position coordinates of vertices I, A, and X, respectively.

[0720] In addition, the decoding device 200 determines the position coordinates of vertex J by, for example, calculating J = (B + Y) / 2. J, B, and Y are the position coordinates of vertices J, B, and Y, respectively.

[0721] Here, in the absence of Y in submesh 2, the decoding device 200 calculates the position coordinates of vertex Y, for example, based on the position coordinates of the vertices in submesh 2 (e.g., the two parent vertices of vertex Y). For example, the decoding device 200 calculates the position coordinates of vertex Y by calculating Y = (X + Z) / 2.

[0722] In addition, the position coordinates of vertex J can also be determined based on the position coordinates of vertices in the LoD (e.g., LoDN) of the higher level of vertex J.

[0723] Figure 89 This is an explanatory diagram illustrating another example of the vertices joining process in relation to an implementation method.

[0724] exist Figure 89 In the example shown, the decoding device 200 does not generate vertex Y, but calculates the position coordinates of vertex J by shifting the position coordinates of vertex B. Here, the vertex that serves as the starting point of the vector representing the shift amount is, for example, the vertex of the subgrid with more sub-segments in subgrid 1 and subgrid 2 (in this example, subgrid 1).

[0725] First, the decoding device 200 calculates vector AI = IA (e.g., I is the midpoint between A and X). Next, the decoding device 200 calculates vector CK = KC (e.g., K is the midpoint between C and Z). Then, using vectors AI and CK, the decoding device 200 calculates the displacement of vertex B (vector BJ) used to calculate the position coordinates of vertex J. Specifically, the decoding device 200 calculates vector BJ = (vector AI + vector CK) / 2, and J = B + vector BJ.

[0726] According to this method, the decoding device 200 can determine the position coordinates of the junction point without using the position coordinates of the vertices (vertices X and Z) of LoDN in LoDN+1 for calculation.

[0727] Furthermore, by using information from already calculated vectors, the computational load can be reduced.

[0728] Furthermore, if we compare the shapes of subgrid 1 and subgrid 2, which has fewer sub-segments than subgrid 1, the shape of subgrid 1 is more likely to be more accurate than that of subgrid 2. Therefore, the decoding device 200 takes the subgrid with more sub-segments of the two combined subgrids as the starting point (in this case, subgrid 1) and shifts the vertices to the vectors obtained from the higher LoD level, thereby more accurately representing the shifted shape.

[0729] Furthermore, while this embodiment uses the example of combining subgrids as an example of multiple grids after segmentation, it can also be applied to grid data other than subgrids. For example, this embodiment can also be applied to units after further subdividing subgrids (e.g., patches). Moreover, the method of this embodiment can be applied when the number of subdivisions differs between patches. In this case, information on whether to add or remove vertices at boundary edges with inconsistent LoD levels can also be signaled in the bitstream on a patch-by-pattern basis.

[0730] Furthermore, the above describes the method by which the encoding device 100 determines whether to add or remove vertices from a submesh. Information indicating this method is stored in metadata and signaled in the bitstream. The decoding device 200 then determines the method of adding or removing vertices based on this metadata. Alternatively, the encoding device 100 may not signal this information in the bitstream and may pre-determine which method the decoding device 200 will perform for adding or removing vertices. Furthermore, a method for switching between the encoding device 100 and the decoding device 200 using a predetermined method may be pre-set. Alternatively, the decoding device 200 may determine this predetermined method.

[0731] For example, the decoding device 200 can also measure the quality of the combined mesh and decide whether to add or remove vertices based on the measured quality. Quality refers to geometry-related qualities such as vertex positions, mesh shape, and whether there are holes in the mesh, as well as the quality measured based on the overall evaluation of the mesh with added texture.

[0732] In addition, the decoding device 200 can combine multiple subgrids based on metadata, or it can output the metadata to the application without performing the combination process, so that the application can combine the multiple subgrids.

[0733] <Summary of this disclosure> Next, an outline of the technology derived from the disclosure in this specification will be shown.

[0734] For example, in the decoding method disclosed herein, regarding multiple sub-mesh constituting a three-dimensional mesh, the position information of vertices contained in the polygon constituting the sub-mesh and connection information related to the connection relationship of the vertices are obtained from the encoded bitstream (i.e., information decoding), the polygon is generated using the position information and the connection information (i.e., face decoding), it is determined whether the edge constituting the polygon is the boundary of the sub-mesh (i.e., edge condition determination), the edge segmentation process is determined based on the determination result (i.e., sub-segmentation process determination based on the determination result), and the edge is segmented using the segmentation process (sub-segmentation).

[0735] The following are specific examples or variations of the segmentation process.

[0736] In the segmentation process, the method for segmenting the edges (segmentation method) can also be determined.

[0737] In the segmentation process, the number of times the edge is segmented (segmentation number) can also be determined.

[0738] In the segmentation process, the method and number of times the edge is segmented can also be determined.

[0739] The segmentation process can also be a process of generating new vertices based on the positional information of multiple vertices that constitute the edge (other definitions of segmentation process).

[0740] In the decoding process, parameters of the segmentation process (segmentation signaling) can also be specified from the encoded bitstream.

[0741] At least one of the segmentation processes can also be predetermined (the predetermined segmentation process).

[0742] The segmentation process includes the case where the edges are not segmented (the option of not segmenting).

[0743] The following are specific examples or variations of the judgment and handling.

[0744] In the determination process, it is also possible to determine whether one side of the processing object among the edges constituting the polygon is the boundary of the sub-mesh (determination process based on edges).

[0745] In the determination process, it can also be determined whether the edges constituting the polygon include edges that serve as the boundaries of the sub-mesh (determination of whether the polygon contains boundary edges).

[0746] The sub-mesh boundary can be an edge (the definition of a boundary) that contains multiple vertices that constitute multiple sub-meshes at both ends.

[0747] The following are specific examples illustrating the relationship between the judgment result and the segmentation process.

[0748] In the segmentation process, when one side of the processed object is the boundary of the sub-mesh, the first segmentation process can be used to segment the side.

[0749] In the segmentation process, when one side of the processed object is not the boundary of the sub-mesh, a second segmentation process can be used to segment the side.

[0750] In the segmentation process, when the polygon contains an edge that serves as the boundary of the sub-mesh, a first segmentation process can be used to segment the edge that serves as the boundary of the sub-mesh, and a second segmentation process can be used to segment the edge that does not serve as the boundary of the sub-mesh.

[0751] In the segmentation process, when the polygon does not contain any edges that serve as the boundaries of the sub-mesh, a second segmentation process can be used to segment all the edges contained in the polygon.

[0752] In the segmentation process, when the polygon contains an edge that serves as the boundary of the sub-mesh, it is determined whether the first segmentation process and the second segmentation process have a prescribed relationship, and the segmentation process is decided based on the determination result. For example, the segmentation process can also be switched based on the result of comparing the number of segmentations determined by the first segmentation process and the second segmentation process.

[0753] The following are specific examples of the first segmentation process (segmentation of the boundary) and the second segmentation process (segmentation of the non-boundary).

[0754] The first segmentation process and the second segmentation process can be different processes (selecting different segmentation processes).

[0755] Alternatively, the first segmentation process can be selected from the first segmentation processing group, and the second segmentation process can be selected from the second segmentation processing group. The first segmentation processing group and the second segmentation processing group may include different segmentation processes (selected from multiple segmentation processes with different options).

[0756] In the first segmentation process, the same common process can be selected from multiple sub-grids. Alternatively, in the segmentation process, processes can be selected from the same group in multiple sub-grids.

[0757] The first segmentation process can be determined in sequence units or frame units. Alternatively, the parameters used in the first segmentation process can be encoded into the encoded bitstream.

[0758] The second segmentation process can also be determined on a per-subgrid basis. Alternatively, the parameters used in the second segmentation process can be encoded into an encoded bitstream.

[0759] In addition, "different treatments" can also refer to different treatments in terms of at least one of the number of splits or the splitting method.

[0760] Additionally, for example, the encoding apparatus of this disclosure includes a circuit and a memory connected to the circuit. In operation, the circuit encodes a first sub-grid to generate first encoded data, encodes a second sub-grid to generate second encoded data, generates control information for selecting a group of vertices in a decoding apparatus, including a first vertex contained in the first sub-grid and a second vertex contained in the second sub-grid, and generates a bitstream containing the first encoded data, the second encoded data, and the control information, wherein the first vertex and the second vertex have the same LoD (Local Detail) index value.

[0761] Additionally, for example, the decoding device of this disclosure includes a circuit and a memory connected to the circuit, wherein the circuit, in operation, generates a first sub-mesh and a second sub-mesh, selects a group of vertices including a first vertex contained in the first sub-mesh and a second vertex contained in the second sub-mesh, and corrects the position of the first vertex and the position of the second vertex to a corrected position generated according to the group of vertices, wherein the first vertex and the second vertex have the same detail (LoD) index value.

[0762] Additionally, for example, the encoding method of this disclosure encodes a first sub-mesh to generate first encoded data, encodes a second sub-mesh to generate second encoded data, generates control information for selecting a group of vertices in a decoding device, including a first vertex contained in the first sub-mesh and a second vertex contained in the second sub-mesh, and generates a bitstream containing the first encoded data, the second encoded data, and the control information, wherein the first vertex and the second vertex have the same detail (LoD) index value.

[0763] Additionally, for example, the decoding method of this disclosure generates a first sub-mesh and a second sub-mesh, selects a group of vertices including a first vertex contained in the first sub-mesh and a second vertex contained in the second sub-mesh, and corrects the position of the first vertex and the position of the second vertex to a corrected position generated according to the group of vertices, wherein the first vertex and the second vertex have the same detail (LoD) index value.

[0764] In this disclosure, in order to address the problems arising from using different sub-segmentation methods in adjacent subgrids, such as Figure 50 As shown, the mandatory constraint is that all boundary edges have the same type of sub-segmentation method and the same number of sub-segments. Therefore, the vertex displacements after sub-segmentation, and the combined submesh, are as follows: Figure 50 As shown in (c), there are no holes around the boundary edge. Here, the displacement vector values ​​of vertex F and vertex M are the same.

[0765] In the sub-segmentation at the boundary edge, a prescribed sub-segmentation method and / or a prescribed number of sub-segments can be used, or the sub-segmentation method and / or the number of sub-segments can be determined by the encoding device, and the information representing the determined sub-segmentation method and / or the number of sub-segments can be signaled through a bit stream.

[0766] In multimedia data encoding technology, it is hoped that new methods can be proposed to improve encoding efficiency, image quality, and reduce circuit size.

[0767] The various embodiments, components, and methods disclosed herein can achieve at least one of the following: improved encoding efficiency, improved image quality, reduced encoding / decoding processing load, reduced circuit size, and increased encoding / decoding processing speed. Alternatively, the various embodiments, components, and methods disclosed herein can appropriately select any one of elements or operations such as filters, blocks, sizes, motion vectors, reference images, and reference blocks during encoding and decoding. This disclosure includes disclosures regarding structures and methods that can provide advantages beyond those described above. Examples of such structures and methods include those that improve encoding efficiency while suppressing an increase in processing load.

[0768] The additional value and advantages of this disclosure become apparent from the specification and accompanying drawings. Its value and / or advantages can be obtained from the various embodiments and features described and included in the specification and drawings, without the need to provide all of them to obtain more than one such value and / or advantage.

[0769] These general or specific methods can be implemented using systems, integrated circuits, computer programs, or computer-readable recording media such as CD-ROMs, or any combination of systems, methods, integrated circuits, computer programs, or recording media.

[0770] <Threshold for subgrid combination> In the zippering process of multiple sub-grids, when combining sub-grids with each other, specifically when combining the vertices (more specifically, boundary vertices) of the sub-grids with each other, the decoding device 200 searches the vertices of each sub-grid to determine whether they are vertices of the zippered objects (also called matching points). At this time, for a vertex of one sub-grid, the decoding device 200 searches multiple vertices of another sub-grid to determine whether the distance from the vertex of the first sub-grid is within a specified threshold range.

[0771] Furthermore, a boundary vertex refers to a vertex within a boundary edge; in other words, it is a vertex located on the boundary edge. For example, a boundary vertex is a vertex located at either end of a boundary edge. A boundary edge refers to an overlapping edge (edge) between sub-mesh units. Specifically, a boundary edge is the edge shared by two sub-mesh units when a 3D mesh is divided into two sub-mesh units.

[0772] The threshold can be set per sequence, per frame, per subgrid, and / or per vertex.

[0773] Here, for example, the distance to a vertex of a certain subgrid is set as a threshold.

[0774] However, when an object is combined with more than two sub-meshes, i.e., when a sub-mesh is combined with multiple sub-meshes, it is sometimes impossible to set the optimal threshold. For example, sometimes the optimal threshold differs for each vertex. In such cases, for example, if the threshold is set for each sub-mesh, the decoding device 200 may fail to properly search for the vertices of the combined object and make incorrect combinations, thus failing to obtain a high-quality 3D mesh shape.

[0775] Therefore, for example, the encoding device 100 sends information (combination information) representing the submesh-pairs to be combined to the decoding device 200. Furthermore, the encoding device 100 sends, for example, information (threshold information) representing the threshold for each combination of the submesh-pairs to be combined to the decoding device 200.

[0776] By sending information representing the threshold of each combination of submeshes to be combined to the decoding device 200, even when a submesh is combined with multiple submeshes, the decoding device 200 is able to properly search the vertices of the combined objects, thereby preventing incorrect combinations and obtaining a high-quality 3D mesh shape.

[0777] Specifically, for example, when the encoding parameters for basic mesh quantization, displacement vector quantization, and lifting transform differ for each sub-mesh, the distance between sub-meshes varies, and therefore, the optimal threshold varies, for example, for each vertex. Even in such cases, it is possible to reproduce 3D meshes with good overall quality.

[0778] In addition, by setting a threshold for each combination, it is not necessary to set a threshold for each vertex, thus reducing the amount of information.

[0779] In addition, when there are many subgrids combined with a subgrid, information representing the optimal threshold for each combination is sent to the decoding device 200, thus enabling the formation of a high-quality 3D mesh.

[0780] Furthermore, by sending the combination information of the subgrids to the decoding device 200, the transmission of threshold information corresponding to subgrids that are not combined with any subgrids can be omitted, thereby reducing the amount of threshold information.

[0781] Furthermore, by sending sub-mesh combination information to the decoding device 200, the decoding device 200 can only use the vertices of the sub-mesh of the combination object as search objects. Therefore, the processing workload of vertex search performed by the decoding device 200 can be reduced. Additionally, incorrect combination with sub-mesh other than the combination object can be avoided.

[0782] The transmission of threshold information and the search process for matching points using thresholds in the combination of subgrids are explained.

[0783] Figure 90 This is a block diagram illustrating an example of the configuration of the encoding device 100 and the decoding device 200 in the embodiment.

[0784] The encoding device 100 includes, for example, a subgrid divider 2401 and a subgrid encoder 2402.

[0785] The submesh divider 2401 divides the input 3D mesh into multiple submeshes. Furthermore, the submesh divider 2401 generates cue information (submesh cue information metadata) that indicates clues beneficial for high-quality integration of the submeshes in the decoding device 200. The cue information may be, for example, metadata containing threshold information (distance information).

[0786] The submesh divider 2401 calculates, for example, a threshold for searching vertices between submeshes when combining submeshes, and stores the threshold information representing the calculated threshold in metadata such as SEI.

[0787] The subgrid encoder 2402 encodes the SEI, and includes (stores) the encoded SEI together with the encoded data obtained by encoding the relevant subgrid information in the bit stream, and sends it to the decoding device 200.

[0788] The decoding device 200 includes, for example, a subgrid decoder 2403 and a subgrid combiner 2404.

[0789] The subgrid decoder 2403 decodes the encoded data contained in the bitstream (more specifically, information related to the encoded subgrids).

[0790] The submesh combiner 2404 obtains threshold information from metadata such as SEI contained in the bitstream, searches for vertices of a combination object that can be combined with the vertices of a certain submesh based on the threshold information, and combines the searched vertices with the vertices of a certain submesh.

[0791] Figure 91 This is a flowchart illustrating the matching point search process of the implementation method. Before performing the search process, the decoding device 200 obtains multiple subgrids (specifically, the position information of each boundary vertex of the multiple subgrids) by decoding the encoded data contained in the bitstream, for example. Furthermore, in this example, the case where a threshold is determined for each boundary vertex, and the threshold information for each boundary vertex is included in the bitstream, is described. Additionally, in this example, the threshold for boundary vertices that are not combined with other boundary vertices is determined to be 0.

[0792] First, the decoding device 200 determines whether the maximum value of the threshold represented by each threshold information contained in the bitstream, i.e., maxDistance, is 0 (S701). In other words, based on maxDistance, the decoding device 200 determines whether there is a matching point among the boundary vertices of the multiple sub-grids. For example, if maxDistance = 0, the decoding device 200 determines that there is no matching point among the boundary vertices of the multiple sub-grids. In this case, the decoding device 200 does not perform the matching process described later. On the other hand, if maxDistance is not 0, the decoding device 200 determines that at least one of the boundary vertices of the multiple sub-grids has a matching point.

[0793] If the decoding device 200 determines that the maxDistance contained in the bit stream is 0 ("Yes" in S701), it determines that there is no matching point (S709) and ends the processing.

[0794] On the other hand, if the decoding device 200 determines that the maxDistance contained in the bitstream is not 0 ("No" in S701), it determines that a matching point exists and begins the matching process (S702). Specifically, the processing after step S703 is performed for each boundary vertex of the multiple subgrids.

[0795] The decoding device 200 calculates the threshold of each boundary vertex within a frame (e.g., multiple subgrids) based on the SEI contained in the bitstream (S703).

[0796] Next, the decoding device 200 performs processing on each boundary vertex within the frame (S704).

[0797] Next, the decoding device 200 calculates the distances to all boundary vertices of all sub-mesh except the sub-mesh containing the boundary vertices (object boundary vertices) of the searched matching point (S705).

[0798] Next, the decoding device 200 determines whether the distance between the object boundary vertex and each boundary vertex is less than a threshold (S706).

[0799] If the decoding device 200 determines that the distance between the object boundary vertex and each boundary vertex is not less than the threshold ("No" in S706), that is, if the distance between the object boundary vertex and each boundary vertex is all above the threshold, it determines that there is no matching point at the object boundary vertex (S709).

[0800] On the other hand, when the decoding device 200 determines that the distance between the object boundary vertex and each boundary vertex is less than a threshold ("Yes" in S706), that is, when it determines that the distance between the object boundary vertex and each boundary vertex includes distances less than the threshold, the boundary vertex with the smallest distance among the distances less than the threshold is determined as the matching point (S707).

[0801] In addition, the decoding device 200 decides not to perform a matching point search for the boundary vertices that have been determined to be matching points (S708).

[0802] After finding matching points for object boundary vertices, the decoding device 200 performs a combination process between the object boundary vertices and the matching points. For example, when combining two boundary vertices that are far apart, the decoding device 200 calculates the position coordinates of the combination point (a new vertex generated by combining the two boundary vertices) using a prescribed method. The calculated position coordinates (position coordinates of the combination point) can be, for example, the midpoint or centroid of the two points, or the position coordinates can be made closer to a boundary vertex using a weighting coefficient. For example, a 3D point is generated on the calculated position coordinates, and the two boundary vertices are deleted, thereby combining the two boundary vertices.

[0803] Figure 92 This is a diagram used to illustrate the positional relationship of multiple subgrids in the implementation method.

[0804] exist Figure 92 In the example shown, multiple sub-mesh are generated by cutting and dividing the 3D mesh in a certain direction (horizontal in this case) (linear division). With this division method, a sub-mesh can be adjacent to a maximum of two other sub-meshes. For example, sub-mesh 1 is not adjacent to sub-mesh 3.

[0805] During the encoding process, the positions (coordinates) of the vertices of the sub-mesh may sometimes shift. For example, through the division of a 3D mesh, the maximum distance between the boundary vertices of sub-mesh 1 and sub-mesh 2 becomes 5 units, and the maximum distance between the boundary vertices of sub-mesh 2 and sub-mesh 3 becomes 20 units. Furthermore, the unit of distance can be arbitrary and is not particularly limited.

[0806] For example, when setting a threshold for each subgrid, for Figure 92 The three sub-mesh shown are: sub-mesh 1, sub-mesh 2, and sub-mesh 3. The threshold of sub-mesh 1 is set to 5 (Threshold(submesh_1) = 5), the threshold of sub-mesh 2 is set to 20 (Threshold(submesh_2) = 20), and the threshold of sub-mesh 3 is set to 20 (Threshold(submesh_3) = 20).

[0807] In use such Figure 92In the example shown, the segmentation method ensures that, for example, subgrids can be processed in the encoding order. Since subgrids 1 and 3 are significantly separated from subgrids 1 and 2, even when sending threshold information representing the threshold for each subgrid, for example, the threshold for subgrid 1 becomes the threshold for combined processing of subgrids 1 and 2, and the threshold for subgrid 2 becomes the threshold for combined processing of subgrids 2 and 3. Therefore, even if a threshold is set for each subgrid, the set threshold may be used as the threshold for each combination of subgrids.

[0808] However, if submesh 1 and submesh 3 are close in position, they may be combined even if submesh 1 is not the object to be combined with submesh 3.

[0809] Here, for example, the thresholds for subgrids 1 and 2 are set to 5, the thresholds for subgrids 2 and 3 are set to 20, and the thresholds for subgrids 1 and 3 are set to 0. That is, it is assumed that the thresholds are determined based on the combination of each subgrid. For example, when the combined threshold is 0, the decoding device 200 does not combine the subgrids of that combination with each other. In this example, the decoding device 200 does not combine subgrids 1 and 3.

[0810] In this way, by including information indicating that subgrid 1 is not a combination object of subgrid 3 in the bit stream, erroneous combination can be prevented.

[0811] Figure 93 This is a diagram used to illustrate the search process for matching points of multiple subgrids in an implementation method.

[0812] In this example, submesh 1 contains boundary vertices A, B, C, L, M, and N; submesh 2 contains boundary vertices D, E, F, G, H, and K; and submesh 3 contains boundary vertices X, Y, and Z.

[0813] In the algorithm for combining boundary vertices (merging algorithm), a set threshold (threshold distance) is used to search for matching points of boundary vertices. Figure 93 The search radius associated with the search for matching points at each boundary vertex of submesh 2 is shown. The search radius corresponds, for example, to a threshold determined for each boundary vertex. Other boundary vertices whose distance (radius) from the object boundary vertex is within the circle of the threshold become the search range for matching points at that object boundary vertex.

[0814] also, Figure 93 The double-dotted line represents the search radius of the boundary vertex E. Additionally, Figure 93 The dashed line represents the search radius of the boundary vertex D. Figure 93 The single-dot dashed line represents the search radius of the boundary vertex H. Figure 93The dotted line represents the search radius of the boundary vertex F.

[0815] Boundary vertices D, H, F, and E are boundary vertices of submesh 2 that are connected to the boundary vertices of other submeshes. Boundary vertex D is connected to boundary vertex A, boundary vertex H is connected to boundary vertex L, boundary vertex F is connected to boundary vertex B, and boundary vertex E is connected to boundary vertex X. On the other hand, boundary vertices G and K of submesh 2 are not connected to any boundary vertices of other submeshes.

[0816] For example, when setting a threshold for each submesh, the maximum distance between each object boundary vertex and each matching point in the submesh is set as the threshold for the submesh. Therefore, a threshold is set for submesh 2 that is associated with the boundary vertex E with the largest search radius.

[0817] In this example, although subgrid 1 is close to subgrid 2, subgrid 2 is far from subgrid 3. Therefore, in order to combine boundary vertex E with boundary vertex X, the distance between subgrid 2 and subgrid 3 (i.e., the search radius of boundary vertex E) needs to be set to the threshold.

[0818] Figure 94 This is a diagram illustrating the search process for matching points in multiple sub-grids used to explain the implementation method. Furthermore, Figure 94 The double-dotted line represents the search radius of the boundary vertex G.

[0819] Decoding device 200, using a threshold set by encoding device 100, searches for matching points for all boundary vertices of submesh 2.

[0820] Since the set threshold (distance) is greater than or equal to the threshold (distance) that would normally be sufficient for boundary vertices D, H, F, and E, boundary vertices D, H, F, and E are combined with appropriate matching points.

[0821] For example, if a threshold (search radius) is set for boundary vertex G, then boundary vertex A of submesh 1 is found within the search range. Therefore, boundary vertex G is combined with boundary vertex A. Subsequently, boundary vertices A, D, and G are combined, potentially forming a degenerate triangle of face DGH.

[0822] Furthermore, for example, even when using a threshold (search radius) set for the boundary vertex K, sometimes the boundary vertex of subgrid 1 and subgrid 3 cannot be found within the range of the search matching point.

[0823] Therefore, for example, the decoding device 200 performs the following processing in the search for matching points.

[0824] Figure 95This is a flowchart illustrating the matching point search process of the implementation method. Before performing the search process, the decoding device 200 obtains multiple subgrids (specifically, the position information of each boundary vertex of the multiple subgrids) by decoding the encoded data contained in the bitstream, for example. Furthermore, in this example, the case where a threshold is determined for each combination of subgrids and the threshold information for each combination is included in the bitstream is described. Additionally, in this example, the threshold for combinations of non-combined subgrids is determined to be 0.

[0825] First, the decoding device 200 determines whether the maximum value of the threshold represented by each threshold information contained in the bitstream, i.e., maxDistance, is 0 (S711). In other words, based on maxDistance, the decoding device 200 determines whether there is a matching point among the boundary vertices of the multiple sub-grids. For example, if maxDistance = 0, the decoding device 200 determines that there is no matching point among the boundary vertices of the multiple sub-grids. In this case, the decoding device 200 does not perform the matching process described later. On the other hand, if maxDistance is not 0, the decoding device 200 determines that at least one of the boundary vertices of the multiple sub-grids has a matching point.

[0826] If the decoding device 200 determines that the maxDistance contained in the bit stream is 0 ("Yes" in S711), it determines that there is no matching point (S719) and ends the processing.

[0827] On the other hand, if the decoding device 200 determines that the maxDistance contained in the bitstream is not 0 ("No" in S711), it determines that a matching point exists and begins the matching process (S712). Specifically, the processing after step S713 is performed for each boundary vertex of the multiple subgrids.

[0828] The decoding device 200 calculates the threshold of each combination (combination pair of subgrids) of subgrids within a frame (e.g., multiple subgrids) based on the SEI contained in the bitstream, and calculates the threshold of each boundary vertex based on the threshold of each combination (S713).

[0829] Next, the decoding device 200 performs processing on each boundary vertex within the frame (S714).

[0830] Next, the decoding device 200 calculates the distances to all boundary vertices of all sub-mesh groups that have been determined (S715).

[0831] Next, the decoding device 200 determines whether the distance between the object boundary vertex and each boundary vertex is less than a threshold (S716).

[0832] If the decoding device 200 determines that the distance between the object boundary vertex and each boundary vertex is not less than the threshold ("No" in S716), that is, if the distance between the object boundary vertex and each boundary vertex is all above the threshold, it determines that there is no matching point of the object boundary vertex (S719).

[0833] On the other hand, when the decoding device 200 determines that the distance between the object boundary vertex and each boundary vertex is less than a threshold ("Yes" in S716), that is, when it determines that the distance between the object boundary vertex and each boundary vertex includes distances less than the threshold, it determines the boundary vertex with the smallest distance among the distances less than the threshold as the matching point (S717).

[0834] In addition, the decoding device 200 decides not to perform a matching point search for the boundary vertices that have been determined to be matching points (S718).

[0835] Figure 96 This is a diagram illustrating the search process for matching points in multiple sub-grids used to explain the implementation method. Furthermore, Figure 96 The double-dotted line represents the search radius of the boundary vertex E. Additionally, Figure 96 The dashed line represents the search radius of the boundary vertex D. Figure 96 The single-dot dashed line represents the search radius of the boundary vertex H. Figure 96 The dashed line represents the search radius of the boundary vertex F.

[0836] As mentioned above, if a threshold is set for each combination (pair) of the subgrid, then as Figure 96 As shown, when the search radius of boundary vertex H is greater than the search radii of boundary vertices D and F, the search radius of boundary vertex H is set as the threshold of the combination of subgrid 1 and subgrid 2.

[0837] Figure 97 This is a diagram illustrating the search process for matching points in multiple sub-grids used to explain the implementation method. Additionally, Figure 97 The dashed line in the diagram represents the search radius for the boundary vertex G used in submesh 1. Additionally, Figure 97 The double-dotted line in the diagram represents the search radius of the boundary vertex G used for submesh 3.

[0838] The decoding device 200 uses, for example, a threshold associated with the combination of subgrid 1 and subgrid 2 to search for boundary vertices in the boundary vertices of subgrid 1 that are within the search range of matching points of boundary vertex G.

[0839] Furthermore, the decoding device 200 uses, for example, a threshold associated with the combination of subgrid 2 and subgrid 3 to search for boundary vertices in the boundary vertices of subgrid 3 that are within the search range of matching points of boundary vertex G.

[0840] Boundary vertex A is contained in submesh 1, and boundary vertex G is contained in submesh 2. Furthermore, the distance between boundary vertex G and boundary vertex A is closer than the search radius of boundary vertex G used for submesh 3, but farther than the search radius of boundary vertex G used for submesh 1. Therefore, boundary vertex G is not bound to boundary vertex A.

[0841] Figure 98 This is a diagram illustrating an example of the syntax for signaling notification of subgrids using metadata in conjunction with an implementation method. Specifically, Figure 98 It is a diagram representing the syntax used to send threshold information (a threshold representing the maximum distance for searching matching points in the combination process) to the decoding device 200. Figure 98 This is the first example of this syntax.

[0842] Subgrid combined metadata is a syntax that exists for each instance and represents the k-th (an integer greater than 0) metadata element. That is, subgrid combined metadata has k elements in each syntax. Furthermore, in this example, the index k is omitted in each syntax.

[0843] In this example, the submesh combined metadata includes distance_per_submesh_flag (a flag indicating whether a threshold is represented for each submesh (flag information)), and when distance_per_submesh_flag=1 (i.e., in the case of representing a threshold for each submesh), it includes information representing the threshold for each submesh.

[0844] On the other hand, when distance_per_submesh_flag=0 (i.e., without representing the threshold for each submesh), the submesh combination includes distance_per_submesh_pair_flag (a flag indicating whether to represent the threshold for each combination of submesh) in the metadata.

[0845] When distance_per_submesh_pair_flag=1 (i.e., when representing the threshold for each combination of submesh), the submesh combination metadata contains threshold information representing the threshold for each combination of submesh.

[0846] Here, if the threshold represented by the threshold information is a specified value (e.g., 0), the subgrids can be considered as not being adjacent to each other.

[0847] Figure 99 This is a diagram illustrating an example of the syntax for signaling notification of subgrids using metadata in conjunction with an implementation method. Figure 99 This is the second example of this syntax.

[0848] This example also illustrates the types of subgrid segmentation methods (segmentation_type).

[0849] For example, when the segmentation method for a 3D mesh is linear segmentation, it means that the 3D mesh is cut and segmented in a certain direction (e.g., horizontally).

[0850] Furthermore, segmentation methods can also be represented by flags instead of segmentation types.

[0851] Furthermore, in cases where there is a segmentation method other than linear segmentation, the method for transmitting threshold information can be switched according to the segmentation method. For example, in the case where the segmentation method of the three-dimensional mesh is linear segmentation, the encoding device 100 excludes the threshold information representing the threshold of each combination of sub-mesh from the bitstream, but includes the threshold information of the number of sub-mesh (or the number of sub-mesh) - 1 in the bitstream.

[0852] Alternatively, distance_per_submesh_flag and distance_per_submesh_pair_flag can be included independently in the bitstream.

[0853] In addition, distance_per_submesh_pair can also be included in the bitstream in a one-dimensional arrangement instead of a two-dimensional arrangement.

[0854] In the loop of the syntax representing the combination of subgrids, the information of the subgrids after the p-th (p is an integer greater than 1) subgrid of the first loop is represented by the second loop, thereby reducing the amount of information.

[0855] Additionally, for example, `distance_per_submesh_flag[k]=1` means that for multiple submesh instances (zippering instances) corresponding to the metadata of index k, the threshold for each combination of submesh instances is set to the boundary vertex. Furthermore, for example, `distance_per_submesh_flag[k]=0` means that for multiple submesh instances (zippering instances) corresponding to the metadata of index k, the threshold for each combination of submesh instances is not set to the boundary vertex. The default value of `distance_per_submesh_pair_flag[k]` is, for example, 0.

[0856] Furthermore, for example, `segmentation_type[k]=1` indicates that a submesh with an index (submeshIdx, e.g., the identification number of the submesh) ranging from 1 to `MaxNumSubmeshes[frameIdx]-2` can only have mutual boundary vertices with submeshIdx-1 and submeshIdx+1 (i.e., have boundary vertices that can be combined). For example, when `segmentation_type[k]=0`, the submesh with index 0 and the submesh with index `MaxNumSubmeshes[frameIdx]-1` can only have mutual boundary vertices with the submesh with index 1 and the submesh with index `MaxNumSubmeshes[frameIdx]-2`, respectively. Furthermore, for example, `segmentation_type[k]=0` indicates that each submesh can have mutual boundary vertices with submesh different from the aforementioned submesh. That is, when segmentation_type[k]=0, it means that the submesh with index submeshIdx may have boundary vertices that are not related to the submesh with index submeshIdx-1 and index submeshIdx+1. The default value of segmentation_type[k] is, for example, 1.

[0857] Furthermore, for example, `segmentation_type[k]=1` indicates that a submesh with an index (submeshIdx) ranging from 0 to `MaxNumSubmeshes[frameIdx]-1` can have mutual boundary vertices with submeshIdx-1 and submeshIdx+1 if `submeshIdx-1` is above 0 and `submeshIdx+1` is below `MaxNumSubmeshes[frameIdx]-2`. When `segmentation_type[k]=0`, it indicates that each submesh may have mutual boundary vertices with submesh different from the aforementioned submesh. The default value of `segmentation_type[k]` is, for example, 1.

[0858] When using the zippering process, distance_per_submesh_pair_flag[k][p][n] represents the value of the variable (zipperingMaxMatchDistancePerPatchPair[k][p][n]), which is used to process combinations of submesh at index p and submesh at index n+p+1 (different from the submesh at index p) that are derived from the same 3D mesh as the submesh at index [k], for multiple submesh (zippering instances) corresponding to the metadata at index [k]. For example, the length of the syntactic element of distance_per_submesh_pair_flag[k][p][n] is Ceil(Log2(max_match_distance[k])) bits.

[0859] This section describes a specific example of the syntax setting for using the threshold determination method in the encoding device 100.

[0860] For example, suppose the values ​​of the variables are given as follows.

[0861] 1.distance_per_submesh_pair_flag[k]=True 2.distance_per_submesh_flag[k]=False 3. number_of_submesh[k]=4 4.segmentation_type[k]=1 (Linear Segmentation) Using these variables, execute the following code.

[0862] for (p=0;p <numSubmeshes-1;p++){ distance_per_submesh_pair_flag[k][p][0] } If numSubmeshes is replaced with a signed value, the above code can be written as follows.

[0863] for (p=0;p<3;p++){ distance_per_submesh_pair_flag[k][p][0] } Then, from the bitstream, obtain (i) distance_per_submesh_pair_flag[k][0][0] representing the threshold (distance) between submesh 0 and submesh 1, (ii) distance_per_submesh_pair_flag[k][1][0] representing the threshold (distance) between submesh 1 and submesh 2, and (iii) distance_per_submesh_pair_flag[k][2][0] representing the threshold (distance) between submesh 2 and submesh 3.

[0864] Here, since the threshold for the combination of subgrid 1 and subgrid 2 is the same as the threshold for the combination of subgrid 2 and subgrid 1, there is no need to include two pieces of information in the bitstream for the same value.

[0865] Next, another example of the syntactic setting of using the threshold determination method in the encoding device 100 will be described.

[0866] For example, suppose the values ​​of the variables are given as follows.

[0867] 1.distance_per_submesh_pair_flag[k]=True 2.distance_per_submesh_flag[k]=False 3. number_of_submesh[k]=4 4. segmentation_type[k]=0 Using these variables, execute the following code.

[0868] for (p=0;p <numSubmeshes-1;p++){ for (n=p+1;n) <numSubmeshes;n++){ distance_per_submesh_pair_flag[k][p][np-1] } } If numSubmeshes is replaced with a signed value, the above code can be written as follows.

[0869] for (p=0;p<3;p++){ for (n=p+1;n<4;n++){ distance_per_submesh_pair_flag[k][p][np-1] } } Then, from the bitstream, obtain (i) distance_per_submesh_pair_flag[k][0][0] representing the threshold (distance) between submesh 0 and submesh 1, (ii) distance_per_submesh_pair_flag[k][0][1] representing the threshold (distance) between submesh 0 and submesh 2, (iii) distance_per_submesh_pair_flag[k][0][2] representing the threshold (distance) between submesh 0 and submesh 3, (iv) distance_per_submesh_pair_flag[k][1][0] representing the threshold (distance) between submesh 1 and submesh 2, (v) distance_per_submesh_pair_flag[k][1][1] representing the threshold (distance) between submesh 1 and submesh 3, and (vi) distance_per_submesh_pair_flag[k][2][0] representing the threshold (distance) between submesh 2 and submesh 3.

[0870] Here, since the threshold for the combination of subgrid 1 and subgrid 3 is the same as the threshold for the combination of subgrid 3 and subgrid 0, there is no need to include two pieces of information in the bitstream for the same value.

[0871] Next, another example of the syntax for combining subgrids with metadata is shown.

[0872] Figure 100 This is a diagram illustrating an example of the syntax for signaling notification of subgrids using metadata in conjunction with an implementation method. Figure 101 This is a diagram illustrating an example of signaling notification of the syntax for information representing the combination of subgrids to be combined in an implementation method. Specifically, in Figure 101 The syntax for connecting submesh information (connected_submesh_info) is shown in the diagram, indicating whether multiple submeshes have been connected.

[0873] The information represented by connected_submesh_info can be included in the submesh integration metadata or in a different SEI and sent to the decoding device 200 via a bitstream.

[0874] Alternatively, based on combination information indicating whether subgrids have been combined, threshold information representing the threshold of the combination can be sent if the combination is to be combined, and not if the combination is not combined.

[0875] As mentioned above, threshold information representing the threshold of each subgrid can be included in the bitstream, or threshold information representing the threshold of each combination of subgrids can be included in the bitstream.

[0876] Information indicating the threshold can be signaled in the bitstream for each subgrid, or for each combination of vertices to be combined (specifically, boundary vertices). Information regarding whether a vertex is a boundary vertex can also be signaled in the bitstream.

[0877] Encoding device 100, for example, determines a method for setting a threshold, includes information representing the determined method and threshold information in metadata, and sends a bit stream containing the metadata to decoding device 200.

[0878] The decoding device 200 performs a search for matching points in the combination of subgrids, for example, based on information representing the method and threshold information contained in the bitstream.

[0879] When sending threshold information representing the threshold for each combination of subgrids, the decoding device 200 may, for example, determine that the subgrid is not combined if the threshold represented by the threshold information is 0, and skip the search process for matching points using the threshold. That is, in this case, the search process for matching points may not be performed.

[0880] Furthermore, the decoding device 200 determines that a subgrid is combined, for example, when the threshold represented by the threshold information is other than 0, and performs a search process using the threshold.

[0881] Furthermore, the encoding device 100 may, for example, send a bitstream containing information indicating whether subgrids are combined to the decoding device 200 for each combination of subgrids. Additionally, the encoding device 100 may also send a bitstream containing threshold information to the decoding device 200 when subgrids are combined.

[0882] Furthermore, in the search process of matching points using a threshold, the decoding device 200 performs the search process using the threshold corresponding to the combination in the two sub-grids of the combined object.

[0883] Furthermore, the decoding device 200 determines whether a sub-mesh is combined, for example, based on threshold information or information indicating whether a sub-mesh is combined. It performs processing on combinations of sub-mesh that are determined to be combined, and does not perform processing on combinations of sub-mesh that are determined not to be combined. Additionally, the processing described here can be a matching point search process, other processing in the combination of sub-mesh elements, or a combination process of sub-mesh attribute information and sub-mesh face attribute information.

[0884] Furthermore, the encoding device 100 can also switch between sending a bit stream containing threshold information for each sub-grid and sending a bit stream containing threshold information for each combination of sub-grids, depending on the sub-grid segmentation method.

[0885] Alternatively, a third threshold sending method can be used depending on the subgrid segmentation method.

[0886] <Representative Example> Figure 102 This is a flowchart illustrating an example of the basic encoding process in this embodiment. For example, Figure 24 The encoding device 100 shown includes a circuit 151 and a memory 152 connected to the circuit 151. In the encoding device 100, the circuit 151 performs the following processing during operation.

[0887] First, the encoding device 100 generates multiple encoded data by encoding multiple sub-meshes (S721). The encoding device 100 encodes multiple sub-meshes, for example, by encoding information related to the multiple sub-meshes, such as vertex information and connection information (specifically, base mesh information of the multiple sub-meshes) and displacement vector information of each of the multiple sub-meshes.

[0888] Next, the encoding device 100 generates threshold information based on the positions of multiple 3D points contained in the multiple submesh, which represents a threshold determined by the combination of two submesh points selected from the multiple submesh (S722). The 3D points are, for example, the boundary vertices described above. The threshold information is, for example, the distance_per_submesh_pair described above.

[0889] Next, the encoding device 100 generates a bit stream containing multiple encoded data and threshold information (S723).

[0890] For example, a threshold is used by the decoding device 200, which has acquired a bitstream, to combine three-dimensional points of multiple subgrids with each other. Here, if the threshold is set only based on the distance to the three-dimensional point, for example, if there are multiple subgrids combined with the subgrid containing that three-dimensional point, it is possible to combine three-dimensional points that belong to a subgrid different from the subgrid containing the three-dimensional point that should have been combined. Therefore, by determining the threshold for each combination of subgrids, the three-dimensional points can be appropriately combined with each other. Furthermore, for two uncombined subgrids, for example, by setting the threshold information to a specific value such as 0, the decoding device 200 can omit the process of determining whether to combine each three-dimensional point for those two subgrids. Therefore, the processing load of the decoding device 200 can be reduced.

[0891] Furthermore, for example, the encoding device 100 determines the combination based on a segmentation method that divides a three-dimensional mesh into multiple sub-meshes.

[0892] Depending on the segmentation method (specifically, the segmentation approach), sometimes multiple subgrids are combined only with specific subgrids. In such cases, even without setting a threshold for all combinations of multiple subgrids, it is possible to appropriately combine multiple subgrids. Therefore, the processing required to determine the combination can be reduced. Additionally, the amount of information in the bitstream can be reduced.

[0893] Furthermore, for example, the encoding device 100 determines the combination based on whether the segmentation method is a first segmentation method or a second segmentation method other than the first segmentation method, wherein the first segmentation method is to segment between two subgrids that have adjacent subgrid identification numbers among the subgrid identification numbers that have been determined to be consecutively determined for multiple subgrids.

[0894] The sub-mesh identification number is, for example, the sub-mesh ID mentioned above. The first segmentation method is, for example, the linear segmentation mentioned above.

[0895] When a 3D mesh is divided into multiple submeshes, sometimes the 3D mesh is repeatedly divided at predetermined intervals in a predetermined direction, for example, by repeatedly dividing the 3D mesh at predetermined intervals in a direction perpendicular to a line drawn in one stroke, thereby generating multiple submeshes. In such cases, for multiple submeshes, submesh identification numbers are determined consecutively, for example, in the order they are generated from the division of the 3D mesh. According to such a division method, the submesh associated with a certain submesh is only the submesh adjacent to that submesh among the multiple submeshes. That is, if such a division method is used, the submesh associated with a certain submesh is only the submesh among the multiple submeshes whose identification number has been determined to be adjacent to the identification number determined for that submesh.

[0896] Furthermore, for example, the encoding device 100 calculates the distance between two three-dimensional points for each group of two three-dimensional points, which are obtained by selecting one three-dimensional point from each of the multiple three-dimensional points in the first sub-grid and the multiple three-dimensional points in the second sub-grid of the two sub-grids in the combination; and calculates the longest distance in each group as the threshold of the combination.

[0897] Therefore, the first and second subgrids can be appropriately combined using a threshold.

[0898] Figure 103 This is a flowchart illustrating an example of the basic decoding process in this embodiment. For example, Figure 25 The decoding device 200 shown includes a circuit 251 and a memory 252 connected to the circuit 251. In the decoding device 200, the circuit 251 performs the following processing during operation.

[0899] First, the decoding device 200 acquires a bit stream containing multiple encoded data and threshold information, wherein the multiple encoded data are generated by encoding multiple subgrids, and the threshold information represents a threshold determined by a combination of two subgrids selected from the multiple subgrids for each (S731).

[0900] Next, the decoding device 200 decodes multiple encoded data (S732).

[0901] Next, the decoding device 200 performs a combination process (S733) based on the threshold information to combine the multiple three-dimensional points contained in the multiple sub-grids with each other.

[0902] Therefore, by determining the threshold for each combination of sub-grids, 3D points can be appropriately combined with each other. Furthermore, for two uncombined sub-grids, for example, by setting the threshold information to a specific value such as 0, the process of determining whether to combine each 3D point can be omitted for those two sub-grids. Thus, the processing workload can be reduced.

[0903] Furthermore, for example, in the combination process (S733), the decoding device 200 performs a search process for a group of two 3D points to be combined among multiple 3D points contained in multiple subgrids, according to each combination, provided that predetermined conditions are met. Based on threshold information, the two 3D points of the searched group are combined. The predetermined conditions can be arbitrarily determined and are not particularly limited.

[0904] Therefore, through search processing, it is possible to combine two appropriate three-dimensional points.

[0905] Furthermore, for example, in the combination process, the decoding device 200 determines, for each combination, whether there exists a set of two 3D points to be combined among the multiple 3D points contained in the two sub-grids of the combination. If it is determined that there is a set of two 3D points to be combined among the multiple 3D points contained in the two sub-grids of the combination, a search process is performed; if it is determined that there is no set of two 3D points to be combined among the multiple 3D points contained in the two sub-grids of the combination, no search process is performed. That is, for example, if it is determined that there is a set of two 3D points to be combined among the multiple 3D points contained in the two sub-grids of the combination, it determines that the predetermined conditions are met and performs a search process.

[0906] Therefore, when search processing is not required, it can be omitted, thus reducing the amount of processing required.

[0907] Furthermore, for example, the decoding device 200 determines, based on threshold information, whether there exists a group of two three-dimensional points to be combined among the multiple three-dimensional points contained in the two combined sub-grids.

[0908] For two subgrids that are not combined, for example, if the threshold information is determined to be a specific value such as 0, it is possible to appropriately determine whether to combine the two subgrids based on whether the threshold represented by the threshold information is a specific value.

[0909] <Other examples> The encoding device 100 and the decoding device 200 have been described above according to the embodiments, but the manner in which the encoding device 100 and the decoding device 200 are described is not limited to the embodiments. Modifications that can be conceived by those skilled in the art can be implemented in the embodiments, and the various constituent elements of the embodiments can be arbitrarily combined.

[0910] For example, in an implementation, a process that is performed by a specific component may be performed by other components instead of that specific component. Furthermore, the order of multiple processes may be changed, or multiple processes may be performed in parallel.

[0911] Furthermore, as described above, at least a portion of the various configurations of this disclosure can be mounted as an integrated circuit. At least a portion of the various processes of this disclosure can also be used as an encoding method or a decoding method. A program for causing a computer to execute the encoding method or the decoding method can also be used. Furthermore, a non-transitory computer-readable recording medium containing the program can also be used. Additionally, a bitstream for causing the decoding device 200 to perform decoding processing can also be used.

[0912] Furthermore, at least a portion of the various configurations and processes disclosed herein can be used as a transmitting device, a receiving device, a transmitting method, and a receiving method. A program for causing a computer to execute the transmitting method or the receiving method can also be used. Additionally, a non-transitory computer-readable recording medium containing the program can also be used.

[0913] Industrial availability This disclosure includes, for example, encoding devices, decoding devices, transmitting devices, and receiving devices for three-dimensional meshes, which can be applied to computer graphics systems and three-dimensional data display systems.

[0914] Label Explanation 100 encoding device Vertex Information Encoder (101, 121, 144) 102, 145 Connect information encoder Attribute Information Encoder 103, 122 104, 204, 1103 Preprocessors Post-processors 105, 205, and 2106 110 Three-dimensional data encoding system 111, 211 controllers 112, 212 Input / Output Processors 113 Three-dimensional data encoder 114 System Multiplexer 115 3D Data Generator 123 Metadata Encoder 124 and 1274 multiplexers 131 Vertex Image Generator 132 Attribute Image Generator 133 Metadata Generator 134 Image Encoder 141 Two-dimensional data encoder 142 Grid Data Encoder 143 Texture Encoder 148. Description of the encoder 151, 251 circuits 152, 252 memory 200 Decoding Device Vertex information decoder 201, 221, 244 202, 245 Connection Information Decoder 203, 222 Attribute Information Decoder 210 Three-Dimensional Data Decoding System 213 3D Data Decoder 214 System Demultiplexer 215, 247 prompts 216 User Interface 223 Metadata Decoder 224 Demultiplexer 231 Vertex Information Generator 232 Attribute Information Generator 234 Image Decoder 241 Two-dimensional data decoder 242 Grid Data Decoder 243 Texture Decoder 246 Mesh Reconstructor 248 Describe the decoder 300 Network 310 External Connector 511 Volume Capturer 512 Projector 513 Basic Mesh Encoder 514 Displacement Encoder 515 Attribute Encoder 516 Other types of encoders 613 Basic Mesh Decoder 614 Displacement Decoder 615 Attribute Decoder 616 Other Types of Decoders 617 3D Reconstructor 1101 Input Grid Attribute maps 1102 and 2108 1104, 2103 Basic Mesh Displacement data 1105, 2104 1106 Compressor 1107, 1304, 2101 bitstreams 1108, 2105 metadata 1231 Demultiplexer Switches 1232 and 1262 1233 Static Mesh Decoder 1234, 1264 grid buffer 1235 Motion Decoder 1236, 1266 Basic Mesh Reconstructor 1237, 1240 Inverse Quantizer 1238, 1243 video decoders 1239 Image Unpacker 1241 Inverse Wavelet Transform 1242 Reconstructor 1244, 1272 Color Converters 1251 Decoded base mesh 1252, 1402 sub-segments 1253 Sub-divided Mesh 1254 Decoded displacement data 1255 displacement device 1256 Decoded 3D Mesh 1261 and 1269 quantizers 1263 Static Mesh Encoder 1265 Motion Encoder 1267 Displacement Data Updater 1268 Wavelet Transform 1270 Image Packer 1271, 1273 video encoders Frames 1301 and 2304 1302, 2301, 2302 Basic grid frames Displacement information for 1303 and 2303 1401 Basic Mesh Generator 1403 Displacement Data Generator 2102 decompressor 2107 Output Mesh 2109 Connector 2305 decoder 2306 Post-decoder 2307 Pre-reconstructor 2308 Reconfigurator 2309 Post-Reconstructor 2310 Adapter 2311 Final 3D Mesh Frame 2401 Submesh Splitter 2402 Subgrid Encoder 2403 Submesh Decoder 2404 Submesh Connector

Claims

1. An encoding method, wherein, Multiple encoded data are generated by encoding multiple subgrids. Based on the positions of multiple 3D points contained in the plurality of subgrids, threshold information is generated. This threshold information represents a threshold determined for each combination of two subgrids selected from the plurality of subgrids. Generate a bitstream containing the plurality of encoded data and the threshold information.

2. The encoding method according to claim 1, wherein, The combination is determined based on the segmentation method that divides the 3D mesh into the multiple sub-mesh.

3. The encoding method according to claim 2, wherein, The combination is determined based on whether the segmentation method is the first segmentation method or a second segmentation method other than the first segmentation method. The first segmentation method is to segment between two subgrids with adjacent subgrid identification numbers that have been determined from the subgrid identification numbers that are consecutively determined for the plurality of subgrids.

4. The encoding method according to any one of claims 1 to 3, wherein, For each pair of two 3D points, the distance between the two 3D points is calculated. These two 3D points are selected from multiple 3D points in the first sub-grid and multiple 3D points in the second sub-grid of the two sub-grids in the combination. The longest distance among the calculated distances of each group is used as the threshold for that group.

5. A decoding method, wherein, A bitstream containing multiple encoded data and threshold information is obtained. The multiple encoded data are generated by encoding multiple subgrids, and the threshold information represents a threshold determined by a combination of two subgrids selected from the multiple subgrids for each subgrid. Decode the multiple encoded data. Based on the threshold information, a combination process is performed to combine the multiple 3D points contained in the multiple sub-grids with each other.

6. The decoding method according to claim 5, wherein, In the said combination process, according to each of the combinations, Under specified conditions, a search process is performed. This search process involves finding a group of two 3D points to be combined from among the multiple 3D points contained in the multiple sub-grids. Based on the threshold information, the two three-dimensional points of the searched group are combined.

7. The decoding method according to claim 6, wherein, In the said combination process, according to each of the combinations, Determine whether there exists a set of two 3D points to be combined among the multiple 3D points contained in the two sub-mesh of the combination. If it is determined that there exists a group of two 3D points to be combined among the multiple 3D points contained in the two sub-grids of the combination, then the search process is performed. If it is determined that there is no set of two 3D points to be combined among the multiple 3D points contained in the two sub-grids of the combination, the search process is not performed.

8. The decoding method according to claim 7, wherein, Based on the threshold information, it is determined whether there exists a group of two 3D points to be combined among the multiple 3D points contained in the two sub-grids of the combination.

9. An encoding device, wherein, have: processor; as well as memory, The processor uses the memory to perform the following processes: Multiple subgrids are encoded to generate multiple encoded data; Based on the positions of multiple 3D points contained in the multiple subgrids, threshold information is generated, which represents a threshold determined by the combination of two subgrids selected from the multiple subgrids for each. Generate a bitstream containing the plurality of encoded data and the threshold information.

10. A decoding device, wherein, have: Processor; and memory, The processor uses the memory to perform the following processes: A bitstream containing multiple encoded data and threshold information is obtained, wherein the multiple encoded data are generated by encoding multiple subgrids, and the threshold information represents a threshold determined by a combination of two subgrids selected from the multiple subgrids for each subgrid; Decode the multiple encoded data; Based on the threshold information, a combination process is performed to combine the multiple 3D points contained in the multiple sub-grids with each other.

Images