Encoding method, decoding method, encoding device, and decoding device
By predictively encoding and decoding the vertices of a 3D mesh, reducing the number of bits for fractional precision, and integrating multiple predicted values, the problem of low efficiency in 3D mesh encoding and decoding is solved, achieving efficient data processing and precision preservation.
Patent Information
- Application Number
- CN202580027230.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-06-14
- Filing Date
- 2025-04-16
- Publication Date
- 2026-11-10
AI Technical Summary
Existing technologies have low efficiency in encoding and decoding three-dimensional meshes, making it difficult to achieve efficient data transmission and storage.
By predictively encoding multiple vertices of a 3D mesh, reducing the number of bits for fractional precision to calculate a second predicted value, and combining the integration and rounding of multiple predicted values, the encoding and decoding process is optimized.
It improves the processing efficiency of encoding and decoding, reduces the amount of data, and at the same time maintains or improves the accuracy of encoding and decoding, reducing prediction errors.
Smart Images

Figure CN123076897A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to encoding methods, etc. Background Technology
[0002] Patent document 1 discloses a method and apparatus for encoding and decoding three-dimensional mesh data.
[0003] Existing technical documents Patent documents Patent Document 1: Japanese Patent Application Publication No. 2006-187015 Summary of the Invention
[0004] The problem that the invention aims to solve Further improvements are desired in the processing of encoding or decoding related to 3D meshes. The purpose of this disclosure is to improve the processing of encoding or decoding related to 3D meshes.
[0005] Methods for solving problems One aspect of the encoding method disclosed herein is an encoding method performed by an encoding device, wherein a plurality of vertices contained in a three-dimensional mesh are obtained, and predictive encoding is performed on one of the plurality of vertices. In the predictive encoding, a first predicted value of the vertex is calculated with fractional precision, a second predicted value is calculated by reducing the number of bits of the fractional part of the first predicted value, and the second predicted value is used to predictively encode the vertex.
[0006] One aspect of the decoding method disclosed herein is a decoding method performed by a decoding device, wherein a plurality of vertices contained in a three-dimensional mesh are obtained, and predictive decoding is performed on one of the plurality of vertices. In the predictive decoding, a first predicted value of the vertex is calculated with fractional precision, a second predicted value is calculated by reducing the number of bits of the fractional part of the first predicted value, and the second predicted value is used to perform predictive decoding on the vertex.
[0007] Furthermore, these general or specific methods can be implemented through systems, devices, integrated circuits, computer programs, or recording media such as computer-readable CD-ROMs, or through any combination of systems, devices, integrated circuits, computer programs, and recording media.
[0008] Invention Effects This disclosure helps to improve coding processes related to 3D meshes, etc. Attached Figure Description
[0009] Figure 1 This is a conceptual diagram representing a three-dimensional mesh used in the implementation.
[0010] Figure 2 This is a conceptual diagram representing the basic elements of a three-dimensional mesh used in the implementation method.
[0011] Figure 3 This is a conceptual diagram representing the mapping of implementation methods.
[0012] Figure 4 This is a block diagram illustrating a structural example of an encoding / decoding system according to an implementation method.
[0013] Figure 5 This is a block diagram illustrating a structural example of an encoding device according to an implementation method.
[0014] Figure 6 This is a block diagram illustrating another structural example of the encoding device in an embodiment.
[0015] Figure 7 This is a block diagram illustrating a structural example of a decoding device according to an implementation method.
[0016] Figure 8 This is a block diagram illustrating another structural example of the decoding device in an embodiment.
[0017] Figure 9 This is a conceptual diagram representing a structural example of a bitstream implementation.
[0018] Figure 10 This is a conceptual diagram representing another structural example of a bitstream implementation.
[0019] Figure 11 This is a conceptual diagram representing another structural example of a bitstream implementation.
[0020] Figure 12 This is a block diagram illustrating a specific example of an encoding / decoding system implemented in this way.
[0021] Figure 13 This is a conceptual diagram representing a structural example of point group data in an implementation method.
[0022] Figure 14 This is a conceptual diagram representing a data file example of point group data in an implementation method.
[0023] Figure 15 This is a conceptual diagram representing a structural example of grid data in an implementation method.
[0024] Figure 16 This is a conceptual diagram representing a data file example of grid data in an implementation method.
[0025] Figure 17 This is a conceptual diagram representing the types of three-dimensional data used in the implementation.
[0026] Figure 18 This is a block diagram illustrating a structural example of a three-dimensional data encoder according to an implementation method.
[0027] Figure 19This is a block diagram illustrating a structural example of a three-dimensional data decoder implementation.
[0028] Figure 20 This is a block diagram illustrating another structural example of a three-dimensional data encoder according to an implementation method.
[0029] Figure 21 This is a block diagram illustrating another structural example of a three-dimensional data decoder implementation.
[0030] Figure 22 This is a conceptual diagram representing a specific example of the encoding process in the implementation method.
[0031] Figure 23 This is a conceptual diagram representing a specific example of the decoding process in the implementation method.
[0032] Figure 24 This is a block diagram illustrating an implementation example of the encoding device in the embodiment.
[0033] Figure 25 This is a block diagram illustrating an implementation example of the decoding device in the embodiment.
[0034] Figure 26 This is a block diagram illustrating another structural example of an encoding / decoding system according to an implementation method.
[0035] Figure 27 This is a block diagram illustrating another structural example of the encoding device in an embodiment.
[0036] Figure 28 This is a block diagram illustrating another structural example of the decoding device in an embodiment.
[0037] Figure 29 This is a block diagram illustrating another structural example of the encoding device in an embodiment.
[0038] Figure 30 This is a block diagram illustrating another structural example of the decoding device in an embodiment.
[0039] Figure 31 This is a flowchart illustrating the processing of the encoding device in the implementation method.
[0040] Figure 32 This is an explanatory diagram that conceptually represents the encoding of a grid frame in an implementation method.
[0041] Figure 33 This is a flowchart illustrating the processing of the decoding device in the implementation method.
[0042] Figure 34 This is an explanatory diagram conceptually representing the decoding of a grid frame in an implementation method.
[0043] Figure 35 This is a block diagram illustrating a structural example of a decoding device according to an implementation method.
[0044] Figure 36 This is a block diagram illustrating a structural example of a decoding device according to an implementation method.
[0045] Figure 37 This is an explanatory diagram showing an example of sub-segmentation in an implementation method.
[0046] Figure 38 This is an explanatory diagram illustrating an example of the displacement of vertices after subdivision and displacement in an implementation method.
[0047] Figure 39 This is an explanatory diagram showing an example of the vertices of the original mesh in the implementation method.
[0048] Figure 40 This is an explanatory diagram showing an example of a grid used in an implementation method.
[0049] Figure 41 This is an explanatory diagram illustrating an example of grid-to-subgrid division in an implementation method.
[0050] Figure 42 This is a first explanatory diagram illustrating an example of packing displacement information into an image frame in an embodiment.
[0051] Figure 43 This is a second explanatory diagram illustrating an example of packing displacement information into an image frame in an embodiment.
[0052] Figure 44 This is a third explanatory diagram illustrating an example of packing displacement information into an image frame in an embodiment.
[0053] Figure 45 This is a diagram illustrating an example of attribute information in grid data used to explain an implementation method.
[0054] Figure 46 This is a diagram illustrating an example of attribute information in grid data used to explain an implementation method.
[0055] Figure 47 This is a diagram illustrating an example of attribute information in grid data used to explain an implementation method.
[0056] Figure 48 This is a diagram illustrating a specific example of an attribute information encoding method.
[0057] Figure 49 This is a diagram illustrating a specific example of an attribute information encoding method.
[0058] Figure 50 This is a diagram illustrating a specific example of an attribute information encoding method.
[0059] Figure 51This is a diagram illustrating a specific example of an attribute information encoding method.
[0060] Figure 52 This is a diagram illustrating a specific example of an attribute information encoding method.
[0061] Figure 53 This is a diagram illustrating a specific example of an attribute information encoding method.
[0062] Figure 54 This is an example of a flowchart illustrating the encoding of attribute information of three-dimensional points in an implementation method.
[0063] Figure 55 This is a flowchart illustrating an example of the calculation process for the predicted value of the third prediction mode in the implementation method.
[0064] Figure 56 This is an example of a flowchart illustrating the decoding of attribute information of three-dimensional points in an implementation method.
[0065] Figure 57 This is a diagram illustrating an example of the syntax for mesh_attribute_coding in an implementation.
[0066] Figure 58 This is a diagram illustrating an example of the syntax of the mesh_attribute_header in the implementation method.
[0067] Figure 59 This is a diagram illustrating an example of the syntax of mesh_attribute_data in an implementation.
[0068] Figure 60 This is a diagram illustrating an example of attribute information for a variation of the implementation method.
[0069] Figure 61 This is a diagram illustrating an example of the syntax of mesh_attribute_header, representing a variation of the implementation method.
[0070] Figure 62 This is a diagram illustrating an example of the geometry of an implementation method.
[0071] Figure 63 This is an example of a texture map representing an implementation method.
[0072] Figure 64 This is a diagram illustrating an example of the syntax for mesh_position_coding in an implementation.
[0073] Figure 65 This is a diagram illustrating an example of the syntax of mesh_position_header in the implementation.
[0074] Figure 66 This is a diagram illustrating an example of the syntax of mesh_position_data in the implementation method.
[0075] Figure 67 This is a diagram illustrating an example of the structure of the encoding device in the embodiment.
[0076] Figure 68 This is a flowchart illustrating an example of an encoding method using an encoding device in an embodiment.
[0077] Figure 69 This is a diagram illustrating an example of the structure of the decoding device in the embodiment.
[0078] Figure 70 This is a flowchart illustrating an example of a decoding method using a decoding device in an embodiment.
[0079] Figure 71 This is a flowchart illustrating another example of the calculation process for the predicted value of the third prediction mode in the implementation method.
[0080] Figure 72 This is a diagram illustrating an example of a vector in three-dimensional space when a detailed prediction is made in the implementation method.
[0081] Figure 73 This is a diagram illustrating an example of a vector on a two-dimensional plane in the case of detailed prediction of an implementation method.
[0082] Figure 74A This is a diagram illustrating an example of calculating the predicted value when the absolute value of dot(gNgP, gNgC) in the implementation exceeds the threshold TH.
[0083] Figure 74B This is a graph showing an example of calculating the predicted value when the absolute value of dot(gNgP, gNgC) in the implementation method is below the threshold TH.
[0084] Figure 75 This is a diagram illustrating another example of the calculation of predicted values in the implementation method.
[0085] Figure 76 This is a flowchart illustrating an example of the calculation process for the predicted value in an implementation method.
[0086] Figure 77 This is a diagram illustrating an example of a vector in three-dimensional space in the case of making detailed predictions for a modified embodiment.
[0087] Figure 78 This is a diagram illustrating an example of a vector on a two-dimensional plane in the case of detailed prediction of a modified embodiment.
[0088] Figure 79 This is a diagram illustrating another example of the calculation of predicted values in the implementation method.
[0089] Figure 80 This is a flowchart illustrating another example of the calculation process for the predicted value in the implementation method.
[0090] Figure 81 This is a diagram illustrating another specific example of the square root calculation process (sqrt) used to explain the implementation method.
[0091] Figure 82 This is a diagram illustrating another specific example of the square root calculation process (sqrt) used to explain the implementation method.
[0092] Figure 83 This is a diagram illustrating another specific example of the square root calculation process (sqrt) used to explain the implementation method.
[0093] Figure 84 This is a diagram illustrating another specific example of the square root calculation process (sqrt) used to explain the implementation method.
[0094] Figure 85 This is a diagram illustrating another specific example of the square root calculation process (sqrt) used to explain the implementation method.
[0095] Figure 86 This is a diagram illustrating another specific example of the square root calculation process (sqrt) used to explain the implementation method.
[0096] Figure 87 This is a diagram illustrating another specific example of the square root calculation process (sqrt) used to explain the implementation method.
[0097] Figure 88 This is a diagram illustrating another specific example of the square root calculation process (sqrt) used to explain the implementation method.
[0098] Figure 89 This is a diagram illustrating another specific example of the square root calculation process (sqrt) used to explain the implementation method.
[0099] Figure 90 This is a diagram illustrating another specific example of the square root calculation process (sqrt) used to explain the implementation method.
[0100] Figure 91 This is a diagram illustrating another specific example of the square root calculation process (sqrt) used to explain the implementation method.
[0101] Figure 92 This is a diagram illustrating another specific example of the square root calculation process (sqrt) used to explain the implementation method.
[0102] Figure 93This is a diagram illustrating another specific example of the square root calculation process (sqrt) used to explain the implementation method.
[0103] Figure 94 This is a diagram illustrating another example of the structure of the encoding device in the embodiment.
[0104] Figure 95 This is a flowchart illustrating another example of an encoding method using an encoding apparatus according to an embodiment.
[0105] Figure 96 This is a diagram illustrating another example of the structure of the decoding device in the embodiment.
[0106] Figure 97 This is a flowchart illustrating another example of a decoding method using a decoding apparatus according to an embodiment. Detailed Implementation
[0107] <Summary of this disclosure> For example, three-dimensional (3D) meshes are used in computer graphics. For instance, a computer graphics image might consist of multiple frames that are distinct in time, each represented by a 3D mesh.
[0108] Furthermore, a 3D mesh consists of vertex information representing the positions of multiple vertices in 3D space, connection information representing the connections between these vertices, and attribute information representing the properties of each vertex or face. Each face is constructed based on the connection relationships between multiple vertices. Such a 3D mesh can be used to represent various computer graphics and images.
[0109] Furthermore, efficient encoding and decoding of 3D meshes are desired for transmission and storage. Arithmetic encoding and decoding can be used for efficient encoding and decoding of 3D meshes.
[0110] Further improvements are desired in the processing of encoding or decoding related to three-dimensional data. The purpose of this disclosure is to improve the processing of encoding or decoding related to three-dimensional data.
[0111] The inventions obtained from the disclosure of this specification will be illustrated below, and the effects obtained by the inventions will be explained.
[0112] The first aspect of the encoding method disclosed herein is an encoding method performed by an encoding device, wherein a plurality of vertices contained in a three-dimensional mesh are obtained, and predictive encoding is performed on one of the plurality of vertices. In the predictive encoding, a first predicted value of the vertex is calculated with fractional precision, a second predicted value is calculated by reducing the number of bits of the fractional part of the first predicted value, and the second predicted value is used to predictively encode the vertex.
[0113] Therefore, by reducing the number of bits in the fractional part of the predicted value calculated with fractional precision, the amount of data can be reduced while controlling the precision of the operation. By reducing the number of bits, the amount of processing required for encoding is reduced, thus improving processing efficiency.
[0114] The second encoding method disclosed herein is as follows: in the encoding method of the first method, in the calculation of the first predicted value, based on each of the two or more vertices that have been processed, two or more third predicted values are calculated with decimal precision, and the first predicted value is calculated by integrating the two or more third predicted values.
[0115] Therefore, by integrating multiple predicted values, a highly accurate predicted value can be obtained. This reduces prediction error and improves coding efficiency. Furthermore, integrating multiple predicted values allows for greater flexibility in selecting appropriate predicted values.
[0116] The third-party encoding method disclosed herein is that, in the second encoding method, the two or more third prediction values are integrated with decimal precision during the calculation of the first prediction value.
[0117] Therefore, by performing integration processing with decimal precision followed by integerization, encoding processing can be performed while maintaining the precision of the integration process. By performing high-precision integration followed by integerization, the increase in prediction error can be suppressed.
[0118] The fourth encoding method disclosed herein is as follows: in the second or third-party encoding method, the two or more third predicted values and the first predicted value have a predetermined number of bits as a fractional part; in the calculation of the second predicted value, the second predicted value is calculated by reducing the predetermined number of bits of the fractional part of the first predicted value.
[0119] Therefore, by reducing the fractional part by a specified number of bits, integerization can be performed easily. Reducing the number of bits reduces the amount of encoding and improves the efficiency of computation.
[0120] The fifth encoding method disclosed herein is that, in any one of the encoding methods from the second to the fourth method, the first predicted value is the average of the two or more third predicted values.
[0121] Therefore, by averaging multiple predictions, the overall prediction error can be reduced. Through averaging, noise components and prediction errors can potentially be canceled out, thus improving coding accuracy.
[0122] The sixth encoding method of this disclosure is, in the fourth encoding method, in the predictive encoding, the predetermined number of bits is not reduced before the calculation of the second predicted value.
[0123] Therefore, by not reducing the number of bits before integerization, the accuracy of the predicted value can be maintained. Since no reduction in precision occurs during computation, high-precision encoding can be achieved.
[0124] The seventh encoding method disclosed herein is as follows: in any of the encoding methods of the first to sixth methods, in the predictive encoding of the vertex using the second predicted value, the second predicted value is rounded to calculate a fourth predicted value, and the fourth predicted value is used to predictively encode the vertex.
[0125] Therefore, by performing rounding, errors below the decimal point can be suppressed. This improves the precision of the encoded data and minimizes errors when reducing the number of bits.
[0126] The eighth encoding method disclosed herein is as follows: in the seventh encoding method, during the rounding process, the second predicted value is rounded in different directions depending on whether the second predicted value is positive or negative.
[0127] Therefore, by switching the rounding direction based on whether the predicted value is positive or negative, coding accuracy can be improved. In particular, optimal rounding processing corresponding to the magnitude and sign of the predicted value can be achieved.
[0128] The ninth decoding method disclosed herein is a decoding method executed by a decoding device, wherein a plurality of vertices contained in a three-dimensional mesh are obtained, and predictive decoding is performed on one of the plurality of vertices. In the predictive decoding, a first predicted value of the vertex is calculated with fractional precision, a second predicted value is calculated by reducing the number of bits of the fractional part of the first predicted value, and the second predicted value is used to perform predictive decoding on the vertex.
[0129] Therefore, by reducing the number of bits in the fractional part of the predicted value calculated with fractional precision, the amount of data can be reduced while controlling the precision of the calculation. By reducing the number of bits, the amount of processing required for decoding is reduced, thus improving processing efficiency.
[0130] The tenth decoding method disclosed herein is as follows: in the ninth decoding method, in the calculation of the first predicted value, based on each of the two or more processed vertices, two or more third predicted values are calculated with decimal precision, and the first predicted value is calculated by integrating the two or more third predicted values.
[0131] Therefore, by integrating multiple predicted values, a highly accurate predicted value can be obtained. This reduces decoding errors and improves decoding accuracy.
[0132] The eleventh decoding method of this disclosure is, in the tenth decoding method, to integrate the two or more third prediction values with decimal precision during the calculation of the first prediction value.
[0133] Therefore, by integrating fractional precision, higher-precision predicted values can be calculated. Subsequent integerization enables efficient decoding.
[0134] The twelfth decoding method of this disclosure is as follows: in the tenth or eleventh decoding method, the two or more third predicted values and the first predicted value have a predetermined number of bits in a fractional part; in the calculation of the second predicted value, the second predicted value is calculated by reducing the predetermined number of bits in the fractional part of the first predicted value.
[0135] Therefore, by reducing the number of bits in the fractional part and converting it to integers, the burden of decoding can be reduced, and high-precision predictions can be utilized.
[0136] The thirteenth decoding method of this disclosure is, in any of the tenth to twelfth decoding methods, the first predicted value is the average of the two or more third predicted values.
[0137] Therefore, by averaging multiple predicted values, prediction errors can be reduced. Averaging can suppress deviations in predicted values, achieving high-precision decoding.
[0138] The fourteenth decoding method of this disclosure is, in the twelfth decoding method, in the predictive decoding, the predetermined number of bits is not reduced before the calculation of the second predicted value.
[0139] Therefore, no reduction in precision due to bit reduction occurs during the decoding process before integerization, thus maintaining high-precision decoding.
[0140] The fifteenth decoding method of this disclosure is as follows: in any of the nine to fourteenth decoding methods, in the prediction decoding of the vertex using the second prediction value, the second prediction value is rounded to calculate a fourth prediction value, and the fourth prediction value is used to perform prediction decoding on the vertex.
[0141] Therefore, by applying rounding, the error during bit reduction can be minimized. This results in highly accurate predictions during decoding.
[0142] The sixteenth decoding method of this disclosure is, in the fifteenth decoding method, in the rounding process, the second predicted value is rounded in different directions according to whether the second predicted value is positive or negative.
[0143] Therefore, by adjusting the rounding direction based on whether the predicted value is positive or negative, the accuracy of the predicted value can be improved. Furthermore, it can suppress the generation of rounding errors.
[0144] The seventeenth aspect of the encoding apparatus disclosed herein includes: a circuit; and a memory connected to the circuit, wherein, in operation, the circuit acquires a plurality of vertices contained in a three-dimensional mesh, performs predictive encoding on one of the plurality of vertices, and in the predictive encoding, calculates a first predicted value of the vertex with fractional precision, calculates a second predicted value by reducing the number of bits of the fractional part of the first predicted value, and uses the second predicted value to perform predictive encoding on the vertex.
[0145] Therefore, by reducing the number of bits in the fractional part of the predicted value calculated with fractional precision, the amount of data can be reduced while controlling the precision of the operation. By reducing the number of bits, the amount of processing required for encoding is reduced, thus improving processing efficiency.
[0146] The decoding apparatus of the eighteenth aspect of this disclosure includes: a circuit; and a memory connected to the circuit, wherein the circuit, in operation, acquires a plurality of vertices contained in a three-dimensional mesh, performs predictive decoding on one of the plurality of vertices, and in the predictive decoding, calculates a first predicted value of the vertex with fractional precision, calculates a second predicted value by reducing the number of bits of the fractional part of the first predicted value, and uses the second predicted value to perform predictive decoding on the vertex.
[0147] Therefore, by reducing the number of bits in the fractional part of the predicted value calculated with fractional precision, the amount of data can be reduced while controlling the precision of the calculation. By reducing the number of bits, the amount of processing required for decoding is reduced, thus improving processing efficiency.
[0148] Furthermore, these general or specific methods can be implemented by systems, devices, integrated circuits, computer programs, or computer-readable recording media such as CD-ROMs, or by any combination of systems, devices, integrated circuits, computer programs, or recording media.
[0149] The embodiments will now be described in detail with reference to the accompanying drawings.
[0150] Furthermore, the embodiments described below are general or specific examples. The numerical values, shapes, materials, constituent elements, arrangement positions of constituent elements, connection methods, steps, and order of steps shown in the following embodiments are examples and are not intended to limit the present invention. In addition, any constituent elements in the following embodiments that are not described in the independent claims representing the highest-level concept are described as arbitrary constituent elements.
[0151] (Implementation Method) In this embodiment, the encoding method and the decoding method are described.
[0152] <Performance and Terminology> Here, the following expressions and terms are used.
[0153] (1) Three-dimensional mesh A 3D mesh is a collection of faces, representing, for example, a 3D object. Furthermore, a 3D mesh primarily consists of vertex information, connectivity information, and attribute information. 3D meshes are sometimes represented as polygon meshes or grids. Additionally, 3D meshes can vary over time. A 3D mesh can contain metadata related to vertex, connectivity, and attribute information, as well as other additional information.
[0154] (2) Vertex information Vertex information represents information about vertices. For example, vertex information indicates the position of a vertex in three-dimensional space. Additionally, vertices correspond to the vertices of the faces that make up a three-dimensional mesh. Vertex information is sometimes represented as "geometry." It can also sometimes be represented as positional information.
[0155] (3) Connection information Connection information represents the connections between vertices. For example, connection information represents the connections between faces or edges used to form a 3D mesh. Connection information is sometimes expressed as "Connectivity". Additionally, connection information can sometimes be expressed as face information.
[0156] (4) Attribute information Attribute information represents the properties of a vertex or face. For example, attribute information represents the color, image, and normal vectors corresponding to a vertex or face. Attribute information is sometimes represented as "Texture".
[0157] (5) face A face is an element that constitutes a three-dimensional mesh. Specifically, a face is a polygon on a plane in three-dimensional space. For example, a face can be defined as a triangle in three-dimensional space.
[0158] (6) Plane A plane is a two-dimensional plane in three-dimensional space. For example, a polygon can be formed on a plane, and multiple polygons can be formed on multiple planes.
[0159] (7) Bit stream A bitstream corresponds to encoded information. Bitstreams can also be represented as streams, encoded bitstreams, compressed bitstreams, or encoded signals.
[0160] (8) Encoding and Decoding Encoding can be replaced by other representations such as storing, containing, writing, recording, signaling, sending, notifying, saving, or compressing, and these representations can also be interchanged. For example, encoding information can mean including the information in a bitstream. Furthermore, encoding information into a bitstream can mean encoding the information to generate a bitstream containing the encoded information.
[0161] Furthermore, the concept of decoding can be replaced by reading, interpreting, reading, inputting, exporting, obtaining, receiving, extracting, restoring, reconstructing, compressing / decompressing, etc., and these concepts can also be interchanged. For example, decoding information can mean obtaining information from a bitstream. Additionally, decoding information from a bitstream can mean decoding the bitstream itself to obtain the information contained within it.
[0162] (9) Ordinal number In the description, first and second ordinal numbers are sometimes attached to constituent elements, etc. These ordinal numbers can be changed as appropriate. In addition, new ordinal numbers can be assigned to constituent elements, etc., or ordinal numbers can be removed. Furthermore, regarding these ordinal numbers, they are sometimes assigned to elements for identification purposes, and sometimes they do not correspond to a meaningful order.
[0163] 3D Mesh Figure 1 This is a conceptual diagram representing a three-dimensional mesh according to this embodiment. The three-dimensional mesh is composed of multiple faces. For example, each face is a triangle. The vertices of these triangles are defined in three-dimensional space. Furthermore, the three-dimensional mesh represents a three-dimensional object. Each face may have color or an image.
[0164] Figure 2 This is a conceptual diagram representing the basic elements of the three-dimensional mesh in this embodiment. The three-dimensional mesh consists of vertex information, connection information, and attribute information. Vertex information represents the position of the vertices of a face in three-dimensional space. Connection information represents the connections between vertices. A face can be determined by vertex information and connection information. That is, through vertex information and connection information, a colorless three-dimensional object is formed in three-dimensional space.
[0165] Attribute information can be mapped to vertices or faces. Attribute information corresponding to vertices is sometimes expressed as "Attribute Per Point". Attribute information corresponding to vertices can represent the attributes of the vertex itself or the attributes of the faces connected to the vertex.
[0166] For example, color can be used as attribute information to establish a correspondence with vertices. The color corresponding to a vertex can be the color of the vertex itself, or the color of the face connected to that vertex. The color of a face can be the average of multiple colors corresponding to multiple vertices of the face. Furthermore, normal vectors can be used as attribute information to establish a correspondence with vertices or faces. Such normal vectors can represent the front and back faces of a face.
[0167] Alternatively, a 2D image can be used as attribute information to establish a correspondence with a face. The 2D image corresponding to a face is also represented as a texture image or an "Attribute Map". Alternatively, information representing the mapping between a face and a 2D image can also be used as attribute information to establish a correspondence with the face. Information representing such a mapping is sometimes represented as mapping information, vertex information of the texture image, texture coordinates, or "Attribute UV Coordinates".
[0168] Furthermore, information such as color, image, and motion graphics used as attribute information is sometimes represented as "Parametric Space".
[0169] This attribute information allows textures to be reflected in three-dimensional objects. That is, by using vertex information, connectivity information, and attribute information, colored three-dimensional objects can be formed in three-dimensional space.
[0170] In addition, as described above, attribute information corresponds to vertices or faces, but it can also correspond to edges.
[0171] Figure 3 This is a conceptual diagram illustrating the mapping in this embodiment. For example, a region of a two-dimensional image in a two-dimensional plane can be mapped to a surface of a three-dimensional mesh in three-dimensional space. Specifically, the coordinate information of a region in the two-dimensional image is mapped to a surface of the three-dimensional mesh. Thus, an image reflecting the mapped region of the two-dimensional image is generated within the surface of the three-dimensional mesh.
[0172] By using mapping, it is possible to separate the two-dimensional image used as attribute information from the three-dimensional mesh. For example, in the encoding of the three-dimensional mesh, the two-dimensional image can be encoded using image encoding or video encoding methods.
[0173] <System Structure> Figure 4This is a block diagram illustrating a structural example of the encoding / decoding system according to this embodiment. Figure 4 The encoding and decoding system includes an encoding device 100 and a decoding device 200.
[0174] For example, encoding device 100 acquires a three-dimensional mesh and encodes it into a bitstream. Then, encoding device 100 outputs the bitstream to network 300. For example, the bitstream contains the encoded three-dimensional mesh and control information for decoding the encoded three-dimensional mesh. By encoding the three-dimensional mesh, its information is compressed.
[0175] Network 300 transmits the bit stream from encoding device 100 to decoding device 200. Network 300 can be the Internet, a wide area network (WAN), a local area network (LAN), or a combination thereof. Network 300 is not necessarily limited to two-way communication; it can also be a one-way communication network used for terrestrial digital broadcasting or satellite broadcasting, etc.
[0176] Alternatively, the Network 300 can be replaced by recording media such as DVD (Digital Versatile Disc) or BD (Blu-Ray Disc (registered trademark)).
[0177] Decoding device 200 acquires the bitstream and decodes the 3D mesh from the bitstream. Through decoding the 3D mesh, its information is decompressed. For example, decoding device 200 decodes the 3D mesh using a decoding method corresponding to the encoding method used by encoding device 100 to encode the 3D mesh. That is, encoding device 100 and decoding device 200 perform encoding and decoding according to their respective encoding and decoding methods.
[0178] Furthermore, the 3D mesh before encoding can also be represented as the original 3D mesh. Additionally, the 3D mesh after decoding can also be represented as a reconstructed 3D mesh.
[0179] <Encoding device> Figure 5 This is a block diagram illustrating a structural example of the encoding apparatus 100 according to this embodiment. For example, the encoding apparatus 100 includes a vertex information encoder 101, a connection information encoder 102, and an attribute information encoder 103.
[0180] Vertex information encoder 101 is a circuit that encodes vertex information. For example, vertex information encoder 101 encodes vertex information into a bit stream according to a specified format for vertex information.
[0181] The connection information encoder 102 is a circuit that encodes connection information. For example, the connection information encoder 102 encodes the connection information into a bit stream according to a specified format.
[0182] The attribute information encoder 103 is a circuit that encodes attribute information. For example, the attribute information encoder 103 encodes the attribute information into a bit stream according to a specified format.
[0183] Vertex information, connectivity information, and attribute information can be encoded using either variable-length encoding or fixed-length encoding. Variable-length encoding can correspond to Huffman coding or context-adaptive binary arithmetic coding (CABAC), etc.
[0184] Vertex information encoder 101, connection information encoder 102, and attribute information encoder 103 can be integrated into one unit. Alternatively, vertex information encoder 101, connection information encoder 102, and attribute information encoder 103 can be further subdivided into multiple constituent elements.
[0185] Figure 6 This is a block diagram illustrating another structural example of the encoding device 100 according to this embodiment. For example, the encoding device 100, in addition to... Figure 5 In addition to the structure shown, it also includes a preprocessor 104 and a postprocessor 105.
[0186] The preprocessor 104 is a circuit that processes vertex information, connectivity information, and attribute information before encoding. For example, the preprocessor 104 may perform transformation, separation, or reuse processing on the 3D mesh before encoding. More specifically, for example, the preprocessor 104 may separate vertex information, connectivity information, and attribute information from the 3D mesh before encoding.
[0187] The post-processor 105 is a circuit that processes the encoded vertex information, connectivity information, and attribute information. For example, the post-processor 105 may perform conversion, separation, or multiplexing processing on the encoded vertex information, connectivity information, and attribute information. More specifically, for example, the post-processor 105 may multiplex the encoded vertex information, connectivity information, and attribute information in a bitstream. Furthermore, for example, the post-processor 105 may further perform variable-length encoding on the encoded vertex information, connectivity information, and attribute information.
[0188] <Decoding device> Figure 7 This is a block diagram illustrating a structural example of the decoding apparatus 200 according to this embodiment. For example, the decoding apparatus 200 includes a vertex information decoder 201, a connection information decoder 202, and an attribute information decoder 203.
[0189] Vertex information decoder 201 is a circuit that decodes vertex information. For example, vertex information decoder 201 decodes vertex information from a bitstream according to a specified format for vertex information.
[0190] The connection information decoder 202 is a circuit that decodes connection information. For example, the connection information decoder 202 decodes the connection information from the bitstream according to a specified format.
[0191] The attribute information decoder 203 is a circuit that decodes attribute information. For example, the attribute information decoder 203 decodes the attribute information from the bitstream according to the format specified for the attribute information.
[0192] Decoding vertex information, connectivity information, and attribute information can use either variable-length decoding or fixed-length decoding. Variable-length decoding can correspond to Huffman coding or context-adaptive binary arithmetic coding (CABAC), etc.
[0193] The vertex information decoder 201, the connection information decoder 202, and the attribute information decoder 203 can be integrated into one unit. Alternatively, the vertex information decoder 201, the connection information decoder 202, and the attribute information decoder 203 can be further subdivided into multiple constituent elements.
[0194] Figure 8 This is a block diagram illustrating another structural example of the decoding apparatus 200 according to this embodiment. For example, the decoding apparatus 200, in addition to... Figure 7 In addition to the structure shown, it also includes a preprocessor 204 and a postprocessor 205.
[0195] The preprocessor 204 is a circuit that processes vertex information, connectivity information, and attribute information before decoding. For example, the preprocessor 204 may perform conversion, separation, or multiplexing processing on the bitstream before decoding the vertex information, connectivity information, and attribute information.
[0196] More specifically, for example, the preprocessor 204 may separate the sub-bitstream corresponding to vertex information, the sub-bitstream corresponding to connection information, and the sub-bitstream corresponding to attribute information from the bitstream. Furthermore, for example, the preprocessor 204 may pre-decode the bitstream using variable-length decoding before decoding the vertex information, connection information, and attribute information.
[0197] The post-processor 205 is a circuit that processes the vertex information, connectivity information, and attribute information after decoding. For example, the post-processor 205 may perform conversion, separation, or multiplexing processing on the decoded vertex information, connectivity information, and attribute information. More specifically, for example, the post-processor 205 may reuse the decoded vertex information, connectivity information, and attribute information in a 3D mesh.
[0198] <Bitstream> Vertex information, connectivity information, and attribute information are encoded and stored in a bitstream. The relationship between this information and the bitstream is shown below.
[0199] Figure 9 This is a conceptual diagram illustrating a structural example of the bitstream in this embodiment. In this example, connection information, vertex information, and attribute information are integrated within the bitstream. For example, the connection information, vertex information, and attribute information may be contained in a single file.
[0200] Alternatively, multiple parts of this information can be stored sequentially, such as the first part of connection information, the first part of vertex information, the first part of attribute information, the second part of connection information, the second part of vertex information, the second part of attribute information, and so on. These multiple parts can correspond to multiple parts that are different in time, multiple parts that are different in space, or multiple faces.
[0201] Furthermore, the order in which connection information, vertex information, and attribute information are saved is not limited to the examples above; a different saving order can also be used.
[0202] Figure 10 This is a conceptual diagram illustrating another structural example of the bitstream in this embodiment. In this example, multiple files are contained within the bitstream, with connection information, vertex information, and attribute information stored in different files. Here, a file containing connection information, a file containing vertex information, and a file containing attribute information are shown, but the storage format is not limited to this example. For instance, two types of information—connection information, vertex information, and attribute information—could be contained in one file, and the remaining type of information could be contained in other files.
[0203] Alternatively, this information can be divided into more files for storage. For example, multiple parts of connection information, multiple parts of vertex information, and multiple parts of attribute information can be stored in multiple files. These multiple parts can correspond to different parts in time, different parts in space, or different faces.
[0204] Furthermore, the order in which connection information, vertex information, and attribute information are saved is not limited to the examples above; a different saving order can also be used.
[0205] Figure 11 This is a conceptual diagram illustrating another structural example of the bitstream in this embodiment. In this example, the bitstream is composed of multiple separable sub-bitstreams, with connection information, vertex information, and attribute information stored in different sub-bitstreams.
[0206] Here, sub-bitstreams containing connection information, sub-bitstreams containing vertex information, and sub-bitstreams containing attribute information are shown, but the storage format is not limited to these examples.
[0207] For example, two types of information—connection information, vertex information, and attribute information—can be contained in one sub-bitstream, while the remaining type is contained in other sub-bitstreams. Specifically, attribute information, such as 2D image data, can be stored separately from the sub-bitstreams containing connection information and vertex information, in a sub-bitstream that follows an image encoding scheme.
[0208] Furthermore, each sub-bitstream can contain multiple files. Also, multiple parts of connection information, multiple parts of vertex information, and multiple parts of attribute information can be stored in multiple files.
[0209] Furthermore, the order in which connection information, vertex information, and attribute information are saved is not limited to... Figure 9 , Figure 10 and Figure 11 The example can also use a different saving order than the one described above. For instance, they can be saved in the bitstream in the order of vertex information, connectivity information, and attribute information. Alternatively, they can be saved in the bitstream in any other order, such as connectivity information, attribute information, and vertex information; vertex information, attribute information, and connectivity information; attribute information, connectivity information, and vertex information; or attribute information, vertex information, and connectivity information.
[0210] Alternatively, connection information, vertex information, and attribute information can be divided into multiple data segments, and these multiple data segments can be stored in a bitstream in a periodic or random order.
[0211] <Specific example> Figure 12 This is a block diagram illustrating a specific example of the encoding / decoding system of this embodiment. Figure 12 The encoding and decoding system includes a three-dimensional data encoding system 110, a three-dimensional data decoding system 210, and an external connector 310.
[0212] The 3D data encoding system 110 includes a controller 111, an input / output processor 112, a 3D data encoder 113, a 3D data generator 115, and a system multiplexer 114. The 3D data decoding system 210 includes a controller 211, an input / output processor 212, a 3D data decoder 213, a system demultiplexer 214, a prompter 215, and a user interface 216.
[0213] In the 3D data encoding system 110, sensor data is input from the sensor terminal to the 3D data generator 115. The 3D data generator 115 generates 3D data, such as point group data or mesh data, based on the sensor data and inputs it to the 3D data encoder 113.
[0214] For example, the 3D data generator 115 generates vertex information and corresponding connection and attribute information. The 3D data generator 115 can process the vertex information while generating the connection and attribute information. For example, the 3D data generator 115 can reduce the amount of data by deleting duplicate vertices, or perform vertex information transformations (position offset, rotation, or normalization, etc.). Additionally, the 3D data generator 115 can also render the attribute information.
[0215] In addition, the 3D data generator 115 in Figure 12 The middle part is a component of the three-dimensional data encoding system 110, but it can also be configured externally independently of the three-dimensional data encoding system 110.
[0216] Sensor terminals that provide sensor data for generating 3D data can be, for example, mobile objects such as cars, flying objects such as airplanes, portable terminals, or cameras. Alternatively, distance sensors such as LiDAR, millimeter-wave radar, infrared sensors or rangefinders, stereo cameras, or combinations of multiple monocular cameras can also be used as sensor terminals.
[0217] Sensor data can include the distance (position) of an object, images from a single-lens reflex camera, images from a stereo camera, color, reflectivity, sensor pose, orientation, gyroscope, sensing position (GPS information or altitude), speed, acceleration, sensing time, temperature, air pressure, humidity, or magnetism, etc.
[0218] 3D data encoder 113 corresponds to Figure 5 The encoding device 100 is shown in the figure. For example, the three-dimensional data encoder 113 encodes three-dimensional data to generate encoded data. In addition, the three-dimensional data encoder 113 generates control information in the encoding of the three-dimensional data. Furthermore, the three-dimensional data encoder 113 inputs the encoded data and the control information together into the system multiplexer 114.
[0219] 3D data can be encoded using either geometric methods or video codecs. Geometric encoding can be further categorized as geometry-based encoding, while video codec encoding can be categorized as video-based encoding.
[0220] System multiplexer 114 multiplexes the encoded data and control information input from 3D data encoder 113, generating multiplexed data using a prescribed multiplexing method. Alternatively, system multiplexer 114 may multiplex other media such as images, audio, subtitles, application data, or document files, or reference timing information, together with the encoded data and control information of the 3D data. Furthermore, system multiplexer 114 can multiplex attribute information associated with sensor data or 3D data.
[0221] For example, multiplexed data can be in file form for storage or packet form for transmission. These methods can utilize ISOBMFF or ISOBMFF-based methods. Additionally, MPEG-DASH, MMT, MPEG-2 TS Systems, or RTP can also be used.
[0222] Furthermore, the multiplexed data is output as a transmission signal by the input / output processor 112 to the external connector 310. The multiplexed data can be transmitted as a transmission signal via wired or wireless means. Alternatively, the multiplexed data can be stored in internal memory or a storage device. The multiplexed data can be transmitted to a cloud server via the Internet or stored in an external storage device.
[0223] For example, the transmission or storage of multiplexed data is carried out through methods corresponding to the medium used for transmission or storage, such as broadcasting or communication. As communication protocols, HTTP, FTP, TCP, UDP, IP, or combinations thereof can be used. Furthermore, both pull-type and push-type communication methods can be used.
[0224] Wired transmission can utilize Ethernet (registered trademark), USB, RS-232C, HDMI (registered trademark), or coaxial cable, among others. Wireless transmission can utilize 3GPP (registered trademark), IEEE-standard 3G / 4G / 5G, wireless LAN, Wi-Fi, Bluetooth, or millimeter wave. Furthermore, broadcast methods can include DVB-T2, DVB-S2, DVB-C2, ATSC 3.0, and ISDB-S3, among others.
[0225] In addition, sensor data can be input to the 3D data generator 115 or the system multiplexer 114. Furthermore, 3D data or encoded data can be directly output as a transmission signal to the external connector 310 via the input / output processor 112. The transmission signal output from the 3D data encoding system 110 is input to the 3D data decoding system 210 via the external connector 310.
[0226] In addition, the various actions of the three-dimensional data encoding system 110 can be controlled by the controller 111 that executes the application program.
[0227] In the 3D data decoding system 210, the transmission signal is input to the input / output processor 212. The input / output processor 212 decodes the multiplexed data, which is in file or packet form, from the transmission signal and inputs the multiplexed data to the system demultiplexer 214. The system demultiplexer 214 obtains the encoded data and control information from the multiplexed data and inputs it to the 3D data decoder 213. The system demultiplexer 214 can also extract other media or reference time information from the multiplexed data.
[0228] 3D data decoder 213 corresponds to Figure 7 The decoding device 200 is shown in the figure. For example, the 3D data decoder 213 decodes the 3D data from the encoded data based on a predefined encoding method. Then, the 3D data is presented to the user by the prompter 215.
[0229] Additionally, sensor data and other supplementary information can be input to the prompter 215. The prompter 215 can then provide 3D data based on this supplementary information. Furthermore, user instructions can be input from the user terminal to the user interface 216. The prompter 215 can then provide 3D data based on these input instructions.
[0230] In addition, the input / output processor 212 can obtain three-dimensional data and encoded data from the external connector 310.
[0231] In addition, the various actions of the 3D data decoding system 210 can be controlled by the controller 211 that executes the application program.
[0232] Figure 13 This is a conceptual diagram illustrating a structural example of point group data in this embodiment. Point group data is data representing a group of points of a three-dimensional object.
[0233] Specifically, a point cloud consists of multiple points, possessing positional information representing the three-dimensional coordinates of each point, as well as attribute information representing the attributes of each point. The positional information is also represented by geometry.
[0234] The categories of attribute information can be, for example, color or reflectivity. A point can be mapped to attribute information of one category, to attribute information of multiple different categories, or to attribute information that has multiple values for the same category.
[0235] Figure 14 This is a conceptual diagram representing an example of a data file for point group data in this embodiment. In this example, a one-to-one correspondence is shown between location information items and attribute information items, illustrating the location and attribute information of N points constituting the point group data. In this example, the location information is information representing three-dimensional coordinate positions using the x, y, and z axes, and the attribute information is information representing colors using RGB. A PLY file or similar can be used as a representative data file for point group data.
[0236] Figure 15 This is a conceptual diagram illustrating a structural example of the mesh data in this embodiment. Mesh data is data used in CG (Computer Graphics) and the like, and is three-dimensional mesh data that represents the three-dimensional shape of an object using multiple faces. Each face is also represented as a polygon, having a polygonal shape such as a triangle or quadrilateral.
[0237] Specifically, in addition to the multiple points that make up the point group, the 3D mesh also consists of multiple edges and multiple faces. Each point is also represented as a vertex or position. Each edge corresponds to a line segment connecting two vertices. Each face corresponds to a region enclosed by three or more edges.
[0238] In addition, 3D meshes possess positional information representing the 3D coordinates of vertices. This positional information is also represented as vertex information or geometry. Furthermore, 3D meshes possess connectivity information representing the relationships between multiple vertices that constitute an edge or face. This connectivity information is also represented as connectivity. Additionally, 3D meshes possess attribute information representing properties specific to vertices, edges, or faces. This attribute information in a 3D mesh is also represented as texture.
[0239] For example, attribute information can represent the color, reflectivity, or normal vector for vertices, edges, or faces. The direction of the normal vector can represent the front and back faces of a face.
[0240] As a data file format for grid data, an object file can also be used.
[0241] Figure 16This is a conceptual diagram representing an example of a data file for mesh data in this embodiment. In this example, the data file contains position information G(1) to G(N) of N vertices constituting a 3D mesh, and attribute information A1(1) to A1(N) of the N vertices. Additionally, in this example, it contains M attribute information A2(1) to A2(M). The items in the attribute information may not correspond one-to-one with vertices or one-to-one with faces. Furthermore, the attribute information may not exist.
[0242] Connection information is represented by a combination of vertex indices. n[1, 3, 4] represents the face of the triangle formed by vertices n=1, n=3, and n=4. Additionally, m[2, 4, 6] indicates that the attribute information for m=2, m=4, and m=6 corresponds to the three vertices respectively.
[0243] Furthermore, the actual content of the attribute information can be recorded in other files. And, a pointer to this content can be associated with vertices or faces. For example, the attribute information representing the image of a face can be stored in a two-dimensional attribute map file. Furthermore, the filename of the attribute map and the two-dimensional coordinate values in the attribute map can be recorded in attribute information A2(1) to A2(M). The method of specifying attribute information for a face is not limited to these methods; any method can be used.
[0244] Figure 17 This is a conceptual diagram representing the types of three-dimensional data in this embodiment. Point cluster data and mesh data can represent static objects or dynamic objects. Static objects are objects that do not change over time, while dynamic objects are objects that change over time. Static objects can correspond to three-dimensional data for any point in time.
[0245] For example, point group data at any given time point is sometimes represented as a PCC frame. Similarly, grid data at any given time point is sometimes represented as a grid frame. Furthermore, both PCC frames and grid frames can sometimes be represented simply as frames.
[0246] Furthermore, the area of the object can be restricted to a certain range, as with typical image data, or it can be unrestricted, as with map data. Additionally, the density of points or areas can be determined in various ways. Sparse point clusters or sparse grid data, or dense point clusters or dense grid data, can be used.
[0247] Next, the encoding and decoding of point groups or 3D meshes will be described. The apparatus, processing, or syntax for encoding and decoding vertex information of 3D meshes in this disclosure can be applied to the encoding and decoding of point groups.
[0248] Furthermore, the apparatus, processing, or syntax for encoding and decoding attribute information of point groups in this disclosure can be applied to the encoding and decoding of connection information or attribute information of three-dimensional meshes.
[0249] Furthermore, at least a portion of the processing can be shared in the encoding and decoding of point group data and grid data. This reduces the size of the circuitry and software programs.
[0250] Figure 18 This is a block diagram illustrating a structural example of the three-dimensional data encoder 113 according to this embodiment. In this example, the three-dimensional data encoder 113 includes a vertex information encoder 121, an attribute information encoder 122, a metadata encoder 123, and a multiplexer 124. The vertex information encoder 121, the attribute information encoder 122, and the multiplexer 124 can correspond to... Figure 6 Vertex information encoder 101, attribute information encoder 103, and post-processor 105, etc.
[0251] Furthermore, in this example, the 3D data encoder 113 encodes the 3D data using a geometry-based encoding method. In this geometry-based encoding method, the 3D structure is taken into account. Moreover, in this geometry-based encoding method, the structural information obtained from the encoding of vertex information is used to encode the attribute information.
[0252] Specifically, firstly, the vertex information, attribute information, and metadata contained in the 3D data generated from the sensor data are input to the vertex information encoder 121, attribute information encoder 122, and metadata encoder 123, respectively. Here, the connectivity information contained in the 3D data can be processed in the same way as the attribute information. Furthermore, in the case of point group data, the position information can be processed as vertex information.
[0253] Vertex information encoder 121 encodes vertex information into compressed vertex information and outputs the compressed vertex information as encoded data to multiplexer 124. In addition, vertex information encoder 121 generates metadata for the compressed vertex information and outputs it to multiplexer 124. Furthermore, vertex information encoder 121 generates structural information and outputs it to attribute information encoder 122.
[0254] The attribute information encoder 122 uses the structural information generated by the vertex information encoder 121 to encode the attribute information into compressed attribute information, and outputs the compressed attribute information as encoded data to the multiplexer 124. In addition, the attribute information encoder 122 generates metadata of the compressed attribute information and outputs it to the multiplexer 124.
[0255] Metadata encoder 123 encodes compressible metadata into compressed metadata and outputs the compressed metadata as encoded data to multiplexer 124. The metadata encoded by metadata encoder 123 can be used for encoding vertex information and attribute information.
[0256] Multiplexer 124 multiplexes compressed vertex information, compressed vertex information metadata, compressed attribute information, compressed attribute information metadata, and compressed metadata into a bitstream. Furthermore, multiplexer 124 inputs the bitstream to the system layer.
[0257] Figure 19 This is a block diagram illustrating a structural example of the 3D data decoder 213 according to this embodiment. In this example, the 3D data decoder 213 includes a vertex information decoder 221, an attribute information decoder 222, a metadata decoder 223, and a demultiplexer 224. The vertex information decoder 221, the attribute information decoder 222, and the demultiplexer 224 can correspond to... Figure 8 Vertex information decoder 201, attribute information decoder 203, and preprocessor 204, etc.
[0258] In this example, the 3D data decoder 213 decodes the 3D data using a geometry-based encoding method. In this geometry-based decoding, the 3D structure is considered. Furthermore, in this geometry-based decoding, the structural information obtained from decoding the vertex information is used to decode the attribute information.
[0259] Specifically, first, the bitstream is input from the system layer to demultiplexer 224. Demultiplexer 224 separates compressed vertex information, compressed vertex information metadata, compressed attribute information, compressed attribute information metadata, and compressed metadata from the bitstream. The compressed vertex information and compressed vertex information metadata are input to vertex information decoder 221. The compressed attribute information and compressed attribute information metadata are input to attribute information decoder 222. The metadata is input to metadata decoder 223.
[0260] Vertex information decoder 221 uses the metadata of compressed vertex information to decode vertex information from compressed vertex information. Furthermore, vertex information decoder 221 generates structure information and outputs it to attribute information decoder 222. Attribute information decoder 222 uses the structure information generated by vertex information decoder 221 and the metadata of compressed attribute information to decode attribute information from compressed attribute information. Metadata decoder 223 decodes metadata from compressed metadata. The metadata decoded by metadata decoder 223 can be used for decoding vertex information and attribute information.
[0261] Subsequently, vertex information, attribute information, and metadata are output as 3D data from the 3D data decoder 213. Additionally, this metadata, for example, is metadata of vertex information and attribute information, which can be used in the application.
[0262] Figure 20 This is a block diagram illustrating another structural example of the 3D data encoder 113 according to this embodiment. In this example, the 3D data encoder 113 includes a vertex image generator 131, an attribute image generator 132, a metadata generator 133, an image encoder 134, a metadata encoder 123, and a multiplexer 124. The vertex image generator 131, the attribute image generator 132, and the image encoder 134 can correspond to... Figure 6 Vertex information encoder 101 and attribute information encoder 103, etc.
[0263] Furthermore, in this example, the 3D data encoder 113 encodes the 3D data using a video-based coding method. In this video-based coding method, multiple 2D images are generated from the 3D data, and these images are then encoded using an image coding method. Here, the image coding method can be HEVC (High Efficiency Video Coding) or VVC (Versatile Video Coding), etc.
[0264] Specifically, firstly, the vertex information and attribute information contained in the 3D data generated from the sensor data are input to the metadata generator 133. Furthermore, the vertex information and attribute information are input to the vertex image generator 131 and the attribute image generator 132, respectively. Additionally, the metadata contained in the 3D data is input to the metadata encoder 123. Here, the connectivity information contained in the 3D data can be processed in the same way as the attribute information. Furthermore, in the case of point group data, the position information can be processed as vertex information.
[0265] Metadata generator 133 generates mapping information for multiple two-dimensional images based on vertex information and attribute information. Furthermore, metadata generator 133 inputs the mapping information into vertex image generator 131, attribute image generator 132, and metadata encoder 123.
[0266] Vertex image generator 131 generates vertex images based on vertex information and mapping information and inputs them to image encoder 134. Attribute image generator 132 generates attribute images based on attribute information and mapping information and inputs them to image encoder 134.
[0267] The image encoder 134 encodes the vertex image and attribute image into compressed vertex information and compressed attribute information respectively according to the image encoding method, and outputs the compressed vertex information and compressed attribute information as encoded data to the multiplexer 124. In addition, the image encoder 134 generates metadata of compressed vertex information and metadata of compressed attribute information and outputs them to the multiplexer 124.
[0268] Metadata encoder 123 encodes compressible metadata into compressed metadata and outputs the compressed metadata as encoded data to multiplexer 124. The compressible metadata contains mapping information. Furthermore, the metadata encoded by metadata encoder 123 can be used for encoding vertex information and attribute information.
[0269] Multiplexer 124 multiplexes compressed vertex information, compressed vertex information metadata, compressed attribute information, compressed attribute information metadata, and compressed metadata in the bitstream. Furthermore, multiplexer 124 inputs the bitstream to the system layer.
[0270] Figure 21 This is a block diagram illustrating another structural example of the 3D data decoder 213 according to this embodiment. In this example, the 3D data decoder 213 includes a vertex information generator 231, an attribute information generator 232, an image decoder 234, a metadata decoder 223, and a demultiplexer 224. The vertex information generator 231, the attribute information generator 232, and the image decoder 234 can be used with... Figure 8 The vertex information decoder 201 and attribute information decoder 203 correspond to each other.
[0271] In this example, the 3D data decoder 213 decodes the 3D data using a video-based encoding method. In this video-based decoding, multiple 2D images are decoded according to an image encoding method, and 3D data is generated from these images. Here, the image encoding method can be HEVC (High Efficiency Video Coding) or VVC (Versatile Video Coding), etc.
[0272] Specifically, first, the bitstream is input from the system layer to demultiplexer 224. Demultiplexer 224 separates compressed vertex information, compressed vertex information metadata, compressed attribute information, compressed attribute information metadata, and compressed metadata from the bitstream. The compressed vertex information, compressed vertex information metadata, compressed attribute information, and compressed attribute information metadata are input to image decoder 234. The compressed metadata is input to metadata decoder 223.
[0273] The image decoder 234 decodes the vertex image according to the image encoding method. At this time, the image decoder 234 uses the metadata of the compressed vertex information to decode the vertex image from the compressed vertex information. Furthermore, the image decoder 234 inputs the vertex image to the vertex information generator 231. Additionally, the image decoder 234 decodes the attribute image according to the image encoding method. At this time, the image decoder 234 uses the metadata of the compressed attribute information to decode the attribute image from the compressed attribute information. Furthermore, the image decoder 234 inputs the attribute image to the attribute information generator 232.
[0274] Metadata decoder 223 decodes metadata from compressed metadata. The metadata decoded by metadata decoder 223 contains mapping information used in the generation of vertex information and attribute information. Furthermore, the metadata decoded by metadata decoder 223 can be used for decoding vertex images and attribute images.
[0275] Vertex information generator 231 reconstructs vertex information from the vertex image according to the mapping information contained in the metadata decoded by metadata decoder 223. Attribute information generator 232 reconstructs attribute information from the attribute image according to the mapping information contained in the metadata decoded by metadata decoder 223.
[0276] Subsequently, vertex information, attribute information, and metadata are output as 3D data from the 3D data decoder 213. Additionally, this metadata, for example, is metadata of vertex information and attribute information that can be used in applications.
[0277] Figure 22 This is a conceptual diagram illustrating a specific example of the encoding process in this embodiment. Figure 22 A 3D data encoder 113 and a description encoder 148 are shown. In this example, the 3D data encoder 113 includes a 2D data encoder 141 and a mesh data encoder 142. The 2D data encoder 141 includes a texture encoder 143. The mesh data encoder 142 includes a vertex information encoder 144 and a connectivity information encoder 145.
[0278] Vertex information encoder 144, connection information encoder 145, and texture encoder 143 can correspond to Figure 6 Vertex information encoder 101, connection information encoder 102, and attribute information encoder 103, etc.
[0279] For example, the two-dimensional data encoder 141 acts as a texture encoder 143, encoding the texture corresponding to the attribute information as two-dimensional data according to the image encoding method or video encoding method, thereby generating a texture file.
[0280] Furthermore, the mesh data encoder 142 operates as both the vertex information encoder 144 and the connectivity information encoder 145, generating a mesh file by encoding vertex and connectivity information. The mesh data encoder 142 can further encode texture mapping information. The encoded mapping information can then be included within the mesh file.
[0281] Furthermore, the description encoder 148 generates a description file by encoding descriptions corresponding to metadata, such as text data. The description encoder 148 can encode the description at the system layer. For example, the description encoder 148 can be included in... Figure 12 In the system multiplexer 114.
[0282] The above steps generate a bitstream containing texture files, mesh files, and description files. These files can be multiplexed in the bitstream as glTF (Graphics Language Transmission Format) or USD (Universal Scene Description) files.
[0283] Furthermore, the 3D data encoder 113 may have two mesh data encoders as mesh data encoder 142. For example, one mesh data encoder encodes the vertex and connection information of a static 3D mesh, while the other mesh data encoder encodes the vertex and connection information of a dynamic 3D mesh.
[0284] Furthermore, the bitstream can contain two mesh files. For example, one mesh file corresponds to a static 3D mesh, and the other mesh file corresponds to a dynamic 3D mesh.
[0285] Furthermore, a static 3D mesh can be a 3D mesh of an intraframe encoded using intra-frame prediction, while a dynamic 3D mesh can be a 3D mesh of an interframe encoded using inter-frame prediction. Additionally, as information for a dynamic 3D mesh, the difference between the vertex or connectivity information of the intraframe's 3D mesh and the vertex or connectivity information of the interframe's 3D mesh can be used.
[0286] Figure 23 This is a conceptual diagram illustrating a specific example of the decoding process in this embodiment. Figure 23The image shows a 3D data decoder 213, a description decoder 248, and a prompter 247. In this example, the 3D data decoder 213 includes a 2D data decoder 241, a mesh data decoder 242, and a mesh reconstructor 246. The 2D data decoder 241 includes a texture decoder 243. The mesh data decoder 242 includes a vertex information decoder 244 and a connectivity information decoder 245.
[0287] Vertex information decoder 244, connection information decoder 245, texture decoder 243, and mesh reconstructor 246 can be used with Figure 8 The vertex information decoder 201, connection information decoder 202, attribute information decoder 203, and post-processor 205 correspond to these. The prompter 247 can be associated with... Figure 12 The corresponding prompts are 215, etc.
[0288] For example, the two-dimensional data decoder 241 acts as a texture decoder 243, and decodes the texture corresponding to the attribute information as two-dimensional data according to the texture file, based on the image encoding method or video encoding method.
[0289] Furthermore, the mesh data decoder 242 acts as both the vertex information decoder 244 and the connectivity information decoder 245, decoding vertex and connectivity information from the mesh file. The mesh data decoder 242 can further decode texture-specific mapping information from the mesh file.
[0290] Furthermore, the description decoder 248 decodes the description corresponding to metadata such as text data from the description file. The description decoder 248 can decode the description at the system layer. For example, the description decoder 248 can be included in... Figure 12 In the system demultiplexer 214.
[0291] Mesh Reconstructor 246, as described, reconstructs the 3D mesh based on vertex information, connectivity information, and texture. Hint 247, as described, renders and outputs the 3D mesh.
[0292] Through the above actions, a 3D mesh is reconstructed and output from a bitstream including texture files, mesh files, and description files.
[0293] Furthermore, the 3D data decoder 213 may have two mesh data decoders as mesh data decoder 242. For example, one mesh data decoder decodes the vertex information and connectivity information of a static 3D mesh, while the other mesh data decoder decodes the vertex information and connectivity information of a dynamic 3D mesh.
[0294] Furthermore, the bitstream can contain two mesh files. For example, one mesh file corresponds to a static 3D mesh, and the other mesh file corresponds to a dynamic 3D mesh.
[0295] Furthermore, a static 3D mesh can be an intra-frame 3D mesh encoded using intra-frame prediction, while a dynamic 3D mesh can be an inter-frame 3D mesh encoded using inter-frame prediction. Additionally, as information for a dynamic 3D mesh, the difference between the vertex or connectivity information of the intra-frame 3D mesh and the vertex or connectivity information of the inter-frame 3D mesh can be used.
[0296] The coding method for dynamic 3D meshes is sometimes referred to as DMC (Dynamic Mesh Coding). Additionally, video-based coding of dynamic 3D meshes is sometimes referred to as V-DMC (Video-based Dynamic Mesh Coding).
[0297] Point cloud compression is sometimes referred to as PCC. Additionally, video-based point cloud compression is sometimes referred to as V-PCC. Furthermore, geometry-based point cloud compression is sometimes referred to as G-PCC.
[0298] <Implementation Example> Figure 24 This is a block diagram illustrating an implementation example of the encoding device 100 according to this embodiment. The encoding device 100 includes circuitry 151 and a memory 152. For example, Figure 5 The multiple components of the encoding device 100 shown are transmitted through... Figure 24 The circuit 151 and memory 152 shown are used to implement this.
[0299] Circuit 151 is a circuit that performs information processing and can access memory 152. For example, circuit 151 is a dedicated or general-purpose circuit for encoding a three-dimensional mesh. Circuit 151 can be a processor like a CPU. Alternatively, circuit 151 can be an assembly of multiple circuits.
[0300] Memory 152 is a dedicated or general-purpose memory that stores the information used by circuit 151 to encode the three-dimensional mesh. Memory 152 can be a circuit and can be connected to circuit 151. Alternatively, memory 152 can be contained within circuit 151. Alternatively, memory 152 can be a collection of multiple circuits. Furthermore, memory 152 can be a disk or optical disk, or it can be a storage device or recording medium. Memory 152 can be non-volatile memory or volatile memory.
[0301] For example, memory 152 can store a three-dimensional mesh or a bit stream. Additionally, memory 152 can store the program used by circuit 151 to encode the three-dimensional mesh.
[0302] Furthermore, in the encoding device 100, it is not necessary to implement Figure 5 All of the multiple constituent elements shown here may be exempt from the multiple processes shown here. Figure 5 A portion of the multiple constituent elements shown may be included in other devices, and a portion of the multiple processes shown herein may be executed by other devices. Furthermore, in the encoding device 100, the multiple constituent elements of this disclosure may be implemented in any combination, and the multiple processes of this disclosure may be performed in any combination.
[0303] Figure 25 This is a block diagram illustrating an implementation example of the decoding device 200 according to this embodiment. The decoding device 200 includes circuitry 251 and a memory 252. For example, Figure 7 The multiple components of the decoding device 200 shown are transmitted through... Figure 25 The circuit 251 and memory 252 shown are used to implement this.
[0304] Circuit 251 is a circuit that performs information processing and can access memory 252. For example, circuit 251 is a dedicated or general-purpose circuit for decoding a three-dimensional mesh. Circuit 251 can be a processor like a CPU. Alternatively, circuit 251 can be an assembly of multiple circuits.
[0305] Memory 252 is a dedicated or general-purpose memory that stores the information used by circuit 251 to decode the three-dimensional mesh. Memory 252 can be a circuit and can be connected to circuit 251. Alternatively, memory 252 can be contained within circuit 251. Alternatively, memory 252 can be an assembly of multiple circuits. Alternatively, memory 252 can be a disk or optical disk, or it can take the form of a storage device or recording medium. Memory 252 can be non-volatile memory or volatile memory.
[0306] For example, memory 252 can store a three-dimensional mesh or a bit stream. Additionally, memory 252 can also store the program used by circuit 251 to decode the three-dimensional mesh.
[0307] Furthermore, in the decoding device 200, it is not necessary to implement... Figure 7 All of the multiple constituent elements shown here may be exempt from the multiple processes shown here. Figure 7 A portion of the multiple constituent elements shown may be included in other devices, and a portion of the multiple processes shown herein may be executed by other devices. Furthermore, in the decoding device 200, the multiple constituent elements of this disclosure may be arbitrarily combined to implement the present disclosure, and the multiple processes of this disclosure may be arbitrarily combined to perform the present disclosure.
[0308] The encoding and decoding methods, which include steps performed by the constituent elements of the encoding apparatus 100 and decoding apparatus 200 of this disclosure, can be executed by any device or system. For example, part or all of the encoding and decoding methods can be executed by a computer equipped with a processor, memory, and input / output circuits. In this case, the encoding and decoding methods can be executed by the computer executing a program for causing the computer to perform the encoding and decoding methods.
[0309] In addition, programs and bit streams can be recorded on non-transitory computer-readable recording media such as CD-ROMs.
[0310] An example of a program could be a bitstream. For instance, a bitstream containing an encoded 3D mesh includes syntactic elements for the decoding device 200 to decode the 3D mesh. Furthermore, the bitstream causes the decoding device 200 to decode the 3D mesh according to the syntactic elements contained within it. Therefore, a bitstream can function similarly to a program.
[0311] The aforementioned bitstream can be an encoded bitstream containing the encoded three-dimensional grid, or a multiplexed bitstream containing the encoded three-dimensional grid and other information.
[0312] Furthermore, the components of the encoding device 100 and the decoding device 200 can be constructed from dedicated hardware, general-purpose hardware that executes the aforementioned programs, or a combination thereof. Additionally, the general-purpose hardware can consist of a memory storing programs and a general-purpose processor that reads and executes the programs from the memory. Here, the memory can be a semiconductor memory or a hard disk, and the general-purpose processor can be a CPU.
[0313] In addition, dedicated hardware can consist of memory and a dedicated processor. For example, a dedicated processor can refer to the memory used to record data to execute encoding and decoding methods.
[0314] Furthermore, as described above, each component of the encoding device 100 and the decoding device 200 can be a circuit. These circuits can be configured as a single circuit or they can be separate circuits. Additionally, these circuits can correspond to dedicated hardware or general-purpose hardware that executes the aforementioned programs. Moreover, the encoding device 100 and the decoding device 200 can be implemented as integrated circuits.
[0315] Alternatively, the encoding device 100 can be a transmitting device for transmitting a three-dimensional mesh. The decoding device 200 can be a receiving device for receiving a three-dimensional mesh.
[0316] <Displacement Encoding and Decoding> Here, as an example, the following terminology will be used.
[0317] (1) Image An image is a unit of data consisting of a collection of pixels, containing either a picture or a block smaller than a picture. Images include both moving images and still images.
[0318] (2) Pictures An image is a unit of image processing consisting of a collection of pixels, also known as a frame or field.
[0319] (3) blocks A block is a processing unit consisting of a specific number of pixels. Blocks are also referred to using the terms shown in the examples below. The shape of a block is not particularly limited. A block can be, for example, an M×N pixel rectangle or an M×M pixel square. Blocks can also be triangular, circular, or other shapes. Examples of blocks are described below.
[0320] • Slices, tiles, or bricks • CTU, superblock, or basic partitioning unit • VPDU, Hardware Processing Division Unit • CU, Processing Block Unit, Prediction Block Unit (PU), or Orthogonal Transform Block Unit (TU) ·Sub-block (4) Pixels or samples A pixel or sample is the smallest point in an image; in other words, it is the smallest unit. A pixel or sample includes not only pixels at integer positions but also pixels at sub-pixel positions generated based on those integer-position pixels.
[0321] (5) Pixel value or sample value Pixel values or sample values are the inherent values of a pixel. Pixel values or sample values include luminance values, chrominance values, RGB grayscale levels, and may also include depth values or binary values of 0 or 1.
[0322] (6) Sign A flag represents more than one bit. For example, a flag may be a parameter or index represented by two or more bits. Flags not only represent values expressed in binary numbers, but sometimes also values expressed in numeric values other than binary numbers.
[0323] (7) Signal Signals are symbolized or encoded to convey information. Signals include discrete digital signals or continuous analog signals.
[0324] (8) Stream or bit stream A stream or bitstream is a sequence of digital data that represents the flow of digital data. A stream or bitstream can be a single stream or can be structured as multiple streams with multiple levels. Streams or bitstreams can be transmitted using a single transmission path via serial communication or using multiple transmission paths via packet communication.
[0325] (9) Difference In the case of scalars, differences can include simple differences (xy) and difference calculations. Differences can include the absolute value of the difference (|xy|), the square of the difference (x^2-y^2), the square root of the difference (√(xy)), the weighted difference (ax-by, where a and b are constants), or the offset difference (x-y+a, where a is the offset), etc.
[0326] (10) and In the case of scalars, sums can include simple sums (x+y) and addition calculations. Sums can include the absolute value of the sum (|x+y|), the sum of squares (x^2+y^2), the square root of the sum (√(x+y)), the weighted sum (ax+by, where a and b are constants), or the offset sum (x+y+a, where a is the offset), etc.
[0327] (11) "based on" The phrase "based on" implies that objects other than "that" can also be considered. Additionally, "based on" is sometimes used when a direct result is obtained, or when the result is obtained through intermediate steps.
[0328] (12) "Being used" or "Using" Expressions like "what was used" or "what was used" imply that things other than "what" can also be considered. Furthermore, expressions like "was used" or "was used" are sometimes used when a direct result is obtained, or when a result is obtained through intermediate steps.
[0329] (13) Prohibition "Prohibition" can be translated as "not permitted". In addition, "not prohibited / prohibited" or "permitted / permitted" does not necessarily mean "obligation".
[0330] (14) “Restriction” or “Limitation” "Restriction" or "limitation" can be translated as "not permitted / not allowed" or "not allowed / permitted." Furthermore, "prohibited / not prohibited" or "not permitted / permitted" does not necessarily imply an "obligation." Additionally, what is prohibited in quantity / quality can be partial or complete.
[0331] (15) Chroma The term chroma, represented by the symbols Cb or Cr, is an adjective that indicates one of two color difference signals associated with a primary color in a sample arrangement or a single sample. The term chroma is sometimes used in place of chrominance.
[0332] (16) Luminance The term "luma" is an adjective denoted by the symbol or subscript Y or L, representing the monochromatic signal of a sample arrangement or a single sample that is related to the primary color. The term "luma" is sometimes used in place of "luminance".
[0333] The encoding and decoding system of this embodiment will be described below.
[0334] A typical 3D model (also known as a 3D model) represents an object digitally, allowing users to search the model in 3D using scaling, translation, and rotation while rendering over time. One method of constructing such a representation is to use triangles to build a 3D mesh. In the model described above, the positions of the triangle vertices, the connectivity between the triangle vertices, and the properties associated with them (normals or UV mapping, etc.) are stored.
[0335] To store all this information in uncompressed form, a very large storage area is required, thus requiring a very large bandwidth for transmission. The triangles forming the mesh, especially in temporal and spatial proximity, largely possess properties similar to repeating patterns. Utilizing these repetitions, efficient encoding and decoding methods can be developed for storage and transmission. One such method is Video-based Dynamic Mesh Coding (V-DMC).
[0336] Figure 26 This is a block diagram illustrating another structural example of the encoding / decoding system of this embodiment. For example... Figure 26 As shown, the encoding and decoding system includes an encoding device 100 and a decoding device 200.
[0337] The encoding / decoding system accepts a 3D mesh (also known as a 3D mesh) as input in the form of vertices' 3D coordinates (vertices information), connectivity (connection information), and associated attributes (attribute information). Furthermore, a 3D mesh can include not only geometry but also texture maps.
[0338] The encoding device 100 takes the input 3D mesh (also called the input 3D mesh or input grid) in the form of the three-dimensional coordinates of the vertices, connectivity, and associated attributes. The encoding device 100 encodes all associated information into a stream. The stream can consist of a single bit stream or multiple bit streams.
[0339] Network 300 transmits the stream generated by the encoding device to the decoding device 200. Network 300 can be the Internet, WAN (Wide Area Network), LAN (Local Area Network), or any combination thereof. Furthermore, network 300 is not necessarily limited to a two-way communication network; it can also be a one-way communication network transmitting broadcast waves such as terrestrial digital broadcasts or satellite broadcasts. Alternatively, recording media such as DVDs (Digital Versatile Discs) or BDs (Blu-ray Discs) can be used instead of network 300.
[0340] The bitstream is sent to the decoding device 200 via network 300. The decoding device 200 decodes the bitstream and uses the 3D coordinates, connectivity, and association properties of the decoded vertices to generate a 3D mesh. The decoding device 200 outputs the generated 3D mesh (also known as the output 3D mesh or output grid).
[0341] Figure 27 This is a diagram showing another structural example of the encoding device 100.
[0342] like Figure 27 As shown, the encoding device 100 includes a preprocessor 1103 and a compressor 1106.
[0343] Encoding device 100 reads in input mesh 1101 and property map 1102 and passes them to preprocessor 1103. Preprocessor 1103 processes the input mesh to extract base mesh 1104 and displacement data 1105. Property map 1102, along with the extracted base mesh 1104 and displacement data 1105, is then transmitted to compressor 1106.
[0344] Furthermore, compressor 1106 compresses the base mesh 1104, displacement data 1105, and attribute map 1102 to generate bitstream 1107. By also including metadata 1108 in the bitstream 1107, compressor 1106 can send additional information to decoding device 200.
[0345] Figure 28 This is a diagram showing another structural example of the decoding device 200.
[0346] like Figure 28 As shown, the decoding device 200 includes a decompressor 2102 and a post-processor 2106.
[0347] Decoding device 200 reads bitstream 2101 and passes it to decompressor 2102. Decompressor 2102 decompresses the base mesh 2103, displacement data 2104, and property graph 2108 from bitstream 2101 and passes them to postprocessor 2106. An example of displacement data 2104 is a displacement vector.
[0348] Furthermore, the post-processor 2106 generates the output mesh 2107 by processing the base mesh 2103 according to the displacement data 2104 and the property map 2108. The post-processor 2106 can also use information from the metadata 2105 to generate the output mesh 2107.
[0349] Figure 29 This is a block diagram illustrating yet another structural example of the encoding device 100 of this embodiment.
[0350] In this example, the encoding device 100 includes a volumetric capturer 511, a projector 512, a basic mesh encoder 513, a displacement encoder 514, an attribute encoder 515, and one or more other types of encoders 516 as options.
[0351] Volume capturer 511 captures content and outputs the captured content to projector 512.
[0352] Projector 512 projects content onto a 3D mesh frame containing vertex geometric coordinates (vertex coordinates representing the position of vertices), texture coordinates, and connectivity data (connection information). This data is output to a base mesh encoder 513, a translation encoder 514, an attribute encoder 515, and one or more other types of encoders 516 as options. Each encoder compresses the data into a bitstream.
[0353] Figure 30 This is a block diagram illustrating another structural example of the decoding device 200 of this embodiment.
[0354] In this example, the decoding device 200 includes a basic mesh decoder 613, a displacement decoder 614, an attribute decoder 615, one or more other types of decoders 616, and a 3D reconstructor 617.
[0355] The bitstream is sent to a base mesh decoder 613, a displacement decoder 614, an attribute decoder 615, and one or more other types of decoders 616 as options. These decoders decode the bitstream to generate decoded data containing vertex geometric coordinates, texture coordinates, and connectivity data. The decoded data is then sent to a 3D reconstructor 617 to reconstruct a 3D mesh frame.
[0356] The encoding process performed by the encoding device 100 will be described in detail below.
[0357] Figure 31 This is a flowchart illustrating the processing of the encoding device 100. Figure 32 This is a conceptual illustration of the encoding of a grid frame. (See reference...) Figure 31 and Figure 32 Explain the processing of the encoding device 100.
[0358] In step S101, the encoding device 100 reads a 3D mesh frame and its attributes as an input mesh frame. The input mesh frame is the mesh frame input to the encoding device 100. An example of a 3D mesh frame as an input mesh frame is shown as mesh frame 1301 (see reference). Figure 32 ).
[0359] In step S102, the encoding device 100 performs decimation processing on the input mesh frame read in step S101 to generate a base mesh frame with fewer vertices than the input mesh frame. The base mesh frame generated by decimating mesh frame 1301 is represented as base mesh frame 1302 (see reference). Figure 32 ).
[0360] In step S103, the encoding device 100 calculates the displacement information used by the decoding device 200 to reconstruct the mesh frame. The displacement information is equivalent to the displacement vector from the vertices of the base mesh frame generated in step S102 towards the vertices of the input mesh frame. One method for calculating the displacement information is to subtract the coordinates of the vertices of the base mesh frame from the coordinates of the vertices of the input mesh frame. The displacement information calculated based on mesh frame 1301 and base mesh frame 1302 is represented as displacement information 1303 (see reference). Figure 32 The displacement information 1303 is in vector form; in other words, it is represented as a displacement vector.
[0361] In step S104, the encoding device 100 encodes the base grid frame generated in step S102, the displacement information generated in step S103, and the attributes of the input grid frame into a bitstream (equivalent to a compressed bitstream). An example of this bitstream is shown as bitstream 1304 (see reference). Figure 32 ).
[0362] Specifically, bitstream 1304 contains vertex coordinates and connectivity information of vertices A, C, E, and F, displacement information, a video bitstream containing texture data, and a compressed attribute graph (see reference). Figure 32 The displacement information contains displacement information used to displace vertices based on vertex coordinates obtained from the base mesh frame after sub-segmentation. The compressed attribute map is the texture coordinates used to apply texture data to the mesh frame reconstructed using the base mesh frame and displacement information.
[0363] The decoding process performed by the decoding device 200 will be described in detail below.
[0364] Figure 33 This is a flowchart illustrating the processing of the decoding device 200. Figure 34 This is a conceptual illustration of the decoding of a 3D mesh. (See reference...) Figure 33 and Figure 34 Explain the processing of the decoding device 200.
[0365] In step S201, the decoding device 200 decodes the base grid frame and attributes from the bitstream (equivalent to a compressed bitstream). An example of a decoded base grid frame (equivalent to a decoded base grid frame) is shown as decoded base grid frame 2301 (see reference). Figure 34 ).
[0366] In step S202, the decoding device 200 generates sub-segmented vertices by sub-segmenting the base mesh frame decoded in step S201. An example of a base mesh frame including sub-segmented vertices is represented as base mesh frame 2302 (see reference). Figure 34 ).
[0367] In step S203, the decoding device 200 decodes the displacement information from the bitstream (equivalent to a compressed bitstream). An example of the decoded displacement information is represented as displacement information 2303 (refer to...). Figure 34 The displacement information 2303 is in vector form; in other words, it is represented as a displacement vector.
[0368] In step S204, the decoding device 200 reconstructs the shape of the mesh frame by moving the vertices of the base mesh frame, including the sub-segmented vertices, to new positions using displacement information, and then restores the mesh frame by applying attribute information. One example of an attribute is texture. An example of the reconstructed mesh frame is represented as mesh frame 2304 (see reference). Figure 34 ).
[0369] Figure 35 This is a block diagram illustrating a structural example of the decoding device according to this embodiment.
[0370] Figure 35 This is an example of a block diagram representing a typical intra-frame decoding.
[0371] Figure 35 The decoding device shown includes a demultiplexer 1231, a switch 1232, a static grid decoder 1233, a grid buffer 1234, a motion decoder 1235, a basic grid reconstructor 1236, an inverse quantizer 1237, a video decoder 1238, an image unpacker 1239, an inverse quantizer 1240, an inverse wavelet transform 1241, a reconstructor 1242, a video decoder 1243, and a color converter 1244.
[0372] Demultiplexer 1231 obtains the compressed bitstream and separates the compressed data associated with the base grid, the image containing displacement data (also known as the displacement bitstream), and the image containing attribute data (also known as the attribute bitstream). The compressed data associated with the base grid is passed to switch 1232. Switch 1232 determines whether to perform intra-frame decoding or inter-frame decoding based on the parameters within the bitstream.
[0373] When intra-frame decoding is selected, the bitstream is passed to a static mesh decoder 1233 that generates the quantized base mesh. The static mesh decoder 1233 is, for example, a decoder that uses an edgebreaker algorithm in the decoding of 3D mesh data. The static mesh decoder 1233 generates the quantized base mesh based on the bitstream. The quantized base mesh generated by the static mesh decoder 1233 is stored in a mesh buffer 1234 for reference when inter-frame decoding is selected.
[0374] When inter-frame decoding is selected, switch 1232 passes compressed data associated with the base mesh to motion decoder 1235. Motion decoder 1235 receives the previously decoded quantized base mesh and decodes motion data representing the difference in vertex coordinates between the quantized base mesh stored in mesh buffer 1234 and the current quantized base mesh. Base mesh reconstructor 1236 reconstructs the current quantized base mesh using the motion data and the quantized base mesh stored in mesh buffer 1234. The quantized base mesh obtained from inter-frame or intra-frame decoding is passed to inverse quantizer 1237 to obtain the decoded base mesh.
[0375] The image containing displacement data is passed to the video decoder 1238 because its bitstream contains displacement data in an image format with two chromaticity information and one luminance information. The video decoder 1238 decodes the data using a video frame decompression method. In another example, an arithmetic decoder is used to decode the displacement data. This decompressed data is passed to the image unpacker 1239, which extracts wavelet coefficients associated with each vertex from the decompressed data in image form. The inverse quantizer 1240 inverse-quantizes the quantized wavelet coefficients using the three components associated with each vertex. The inverse wavelet transform 1241 performs an inverse transform on the result, ultimately obtaining the decoded displacement data. The decoded displacement data and the decoded base mesh are passed to the reconstructor 1242. The reconstructor 1242 subdivides the edges of the decoded base mesh and uses the decoded displacement data to displace the vertices, obtaining the decoded mesh.
[0376] To obtain the decoded attribute bitstream, the image containing the attribute data is passed to another video decoder 1243. For color space and color format conversion, the decoded attribute bitstream is further processed in the color converter 1244 to obtain the decoded attribute map.
[0377] Figure 36 This is a block diagram illustrating a structural example of the decoding device according to this embodiment.
[0378] Figure 36 This represents an example of a reconstructor that obtains a decoded 3D mesh 1256 from the decoded base mesh 1251 and the decoded displacement data 1254.
[0379] The decoded base grid 1251 is passed to the sub-segmenter 1252.
[0380] Sub-segmenter 1252 subdivides any two connected vertices of the overall 3D mesh by appending a new vertex between them. To generate a predefined number of vertices, this process can be repeated multiple times, including vertices created through previous sub-segmentation steps. Iterations of each sub-segmentation of the overall 3D mesh generate new levels of detail (LoD). The sub-segmented mesh 1253 and the decoded displacement data 1254 are passed to translator 1255. Transducer 1255 moves each vertex to a new position according to the corresponding displacement data, thereby generating the decoded 3D mesh 1256.
[0381] The following describes sub-segmentation. Sub-segmentation is performed by a sub-segmenter (specifically, sub-segmenter 1206 or sub-segmenter 2204).
[0382] Figure 37 This is an illustrative diagram representing an example of sub-segmentation.
[0383] Figure 37 The basic mesh shown in (a) contains vertices A, B and C, as well as connection information representing their connectivity.
[0384] exist Figure 37 (b) shows the mesh generated by the first sub-segmentation, in other words, the mesh after the first sub-segmentation. The sub-segmenter generates vertices D, E, and F, as well as connection information representing their connectivity, in the first sub-segmentation. The mesh generated by the sub-segmenter is also referred to as LoD1 or the first LoD.
[0385] Vertex D of the mesh after the first sub-segmentation is generated by sub-segmenting vertices A and B. Similarly, vertex E is generated by sub-segmenting vertices B and C. Vertex F is generated by sub-segmenting vertices A and C.
[0386] Furthermore, as an example, vertex D can be the midpoint of line segment AB (in other words, edge AB) connecting vertices A and B, which will form the basis of this generation. Similarly, vertex E can be the midpoint of line segment AC. Vertex F can be the midpoint of line segment BC.
[0387] exist Figure 37 Figure (c) shows the mesh generated by the second sub-segmentation, in other words, the mesh after the second sub-segmentation. In the second sub-segmentation, the sub-segmenter generates vertices G, H, I, J, K, L, M, N, and O, as well as connection information representing their connectivity. The mesh generated by the sub-segmenter is also referred to as LoD2 or the second LoD.
[0388] Vertex G of the mesh after the second sub-segmentation is generated by sub-segmenting vertices A and D. Similarly, vertex H is generated by sub-segmenting vertices A and E. Vertex I is generated by sub-segmenting vertices B and D. Vertex J is generated by sub-segmenting vertices D and F. Vertex K is generated by sub-segmenting vertices E and F. Vertex L is generated by sub-segmenting vertices C and E. Vertex M is generated by sub-segmenting vertices B and F. Vertex N is generated by sub-segmenting vertices C and F. Vertex O is generated by sub-segmenting vertices D and E.
[0389] Furthermore, as an example, vertex G can be the midpoint of line segment AD (in other words, edge AD) connecting vertices A and D, which will serve as the basis for this generation. Similarly, vertex H can be the midpoint of line segment AE. Vertex I can be the midpoint of line segment BD. Vertex J can be the midpoint of line segment DF. Vertex K can be the midpoint of line segment EF. Vertex L can be the midpoint of line segment CE. Vertex M can be the midpoint of line segment BF. Vertex N can be the midpoint of line segment CF. Vertex O can be the midpoint of line segment DE.
[0390] The following is for reference Figure 38 and Figure 39 The displacement of the vertices is described. The displacement of the vertices is performed by the reconstructor 2209.
[0391] Figure 38 This is an explanatory diagram showing an example of the displacement of vertices after sub-segmentation and displacement. Figure 39 This is an illustrative diagram representing an example of the vertices of the original mesh.
[0392] Figure 38 The basic mesh shown in (a) includes vertices A, B, C and Z, as well as connection information representing their connectivity.
[0393] exist Figure 38 (b) shows the mesh generated by the first sub-segmentation, in other words, the mesh after the first sub-segmentation (i.e., the first LoD). The sub-segmenter generates vertices S, T, U, X, or Y, and connection information representing their connectivity, in the first sub-segmentation. Regarding vertices S, T, U, X, or Y, and... Figure 37 The vertices D, E, and F shown in (b) are the same.
[0394] exist Figure 38 (c) shows the mesh generated by the second sub-segmentation; in other words, it shows the mesh after the second sub-segmentation (i.e., the second LoD). The sub-segmenter generates vertices D, E, F, G, and H in the second sub-segmentation, along with connection information representing their connectivity. Regarding vertices D, E, F, G, and H, and... Figure 37 The vertices G, H, I, J, K, L, M, N, or O shown in (c) are the same.
[0395] Figure 38 (d) represents the mesh containing the vertices after the subdivision and displacement. Figure 38 The vertices A, B, C, D, E, F, G, H, S, T, U, X, Y, and Z shown in (d) are located using displacement information from Figure 38 The position of the vertex shown in (c) after displacement.
[0396] Figure 39The original grid shown is an example of the grid input to the encoding device 100, i.e., the grid before encoding.
[0397] Figure 38 The grid shown has the same Figure 39 The shape shown is close to that of the original mesh. The displacement information is generated by the displacement vector calculator 1207 of the encoding device 100 as information representing the displacement from the vertices of the base mesh to the vertices of the original mesh. Therefore, by using the displacement information generated in this way, the mesh is reconstructed to generate a mesh with a shape close to that of the original mesh.
[0398] Decoding device 200 can output Figure 38 The grid shown in (d) is a grid.
[0399] Next, refer to Figure 40 and Figure 41 This describes the division of a grid into subgrids.
[0400] The mesh is divided into multiple smaller parts, each of which can be encoded. During mesh division, the vertices of the mesh are segmented in a way that allows the coordinates and connectivity of the vertices within each segment to be encoded independently.
[0401] Figure 40 This is an illustrative diagram representing an example of a grid. Figure 41 This is an illustrative diagram showing an example of dividing a grid into subgrids.
[0402] Figure 40 The grid shown is the original grid, sometimes referred to as the full grid in contrast to the subgrid.
[0403] Figure 41 Indicates will Figure 40 The diagram shows the case where the entire mesh is divided into two sub-mesh segments. Regarding vertices A, B, and C of the entire mesh (refer to...). Figure 40 Vertex A is copied to vertices A1 and A2, vertex B is copied to vertices B1 and B2, and vertex C is copied to vertices C1 and C2, thus creating two sub-meshes (i.e., sub-mesh 1 and sub-mesh 2) from the full mesh. Sub-mesh 1 and sub-mesh 2 are meshes that can be decoded independently.
[0404] The following is for reference Figure 42 , Figure 43 and Figure 44 This indicates that displacement information is packaged into the image frame.
[0405] Figure 42 , Figure 43 and Figure 44 This is an illustrative diagram showing an example of packing displacement information into an image frame. Additionally, an image frame can also be referred to as a video frame.
[0406] Vertex displacement data can be encoded into image frame data, for example, by mapping to each component of a YUV format image frame (i.e., each of the Y component (Y plane), U component (U plane), and V component (V plane). This will be explained below as an example. Alternatively, vertex displacement data can also be encoded into image frame data by mapping to each component of an RGB format image frame (each of the R component, G component, and B component).
[0407] The decoding device 200 can use an image encoding module to extract displacement data. The displacement data can be the X, Y, or Z components in a global coordinate system (e.g., a Cartesian coordinate system), or the shape of normal, tangent, or two tangent components in a local coordinate system. Methods for mapping displacement data to image frames include the following.
[0408] For example, in method 1, displacement data is configured in the image frame according to the scan order. An example of the packing of displacement data in this case is as follows: Figure 42 As shown, displacement data is directly mapped to image frames according to a predefined scanning order.
[0409] Additionally, since the height and width of an image frame are fixed, sometimes the displacement data cannot be fully contained within the frame. In such cases, the remaining portion of the image frame is filled with padding data (also known as padded data) (see [reference]). Figure 42 ).
[0410] For example, in method 2, the displacement data is separated into multiple LoDs and mapped to the Y, U, and V components of the image frame. An example of this type of displacement data packing is shown below. Figure 43 As shown. Here, the displacement data of the next LoD image frame begins immediately after the end of the displacement data of the previous LoD. Similar to method 1, if the displacement data does not fit perfectly into the image frame, the end of the image frame is padded (see [reference]). Figure 43 ).
[0411] For example, in method 3, the displacement data corresponding to the LoD is mapped to the Y, U, and V components of the image frame in a manner different from that in method 2. An example of the packing of displacement data in this case is as follows... Figure 44 As shown. Therefore, each LoD can be decoded independently. In method 3, the displacement data of each LoD is padded in the middle, and CTU alignment is performed together with the padding at the end of the video frame (see [reference]). Figure 44 ).
[0412] <Other examples> The above describes the encoding and decoding devices according to the embodiments, but the methods of encoding and decoding devices are not limited to the embodiments described above. Modifications that can be conceived by those skilled in the art can be implemented in the embodiments, and the multiple constituent elements in the embodiments can be arbitrarily combined.
[0413] For example, the processes performed by specific components in the implementation can be performed by other components instead of specific components. Furthermore, the order of multiple processes can be changed, or multiple processes can be executed in parallel.
[0414] Furthermore, as described above, at least a portion of the various structures disclosed herein can be implemented as an integrated circuit. At least a portion of the various processes disclosed herein can be used as encoding or decoding methods. A program for causing a computer to execute the encoding or decoding method can also be used. Alternatively, a non-transitory, computer-readable recording medium on which the program is recorded can also be used. Additionally, a bitstream for causing a decoding device to perform decoding processing can also be used.
[0415] Furthermore, at least a portion of the various structures and processes disclosed herein can also be used as a transmitting device, a receiving device, a transmitting method, and a receiving method. A program for causing a computer to execute the transmitting or receiving method can also be used. Additionally, a non-transitory, computer-readable recording medium containing the program can also be used.
[0416] Furthermore, in the above embodiments, each component can be constructed by dedicated hardware or implemented by executing software programs suitable for each component. Each component can also be implemented by a program execution unit such as a CPU or processor reading and executing software programs recorded on recording media such as hard disks or semiconductor memories. Here, the software for the encoding device, etc., implementing the above embodiments is the following program.
[0417] That is, the above software is a program that enables a computer to perform the following encoding method: obtaining a first frame as the object of encoding, encoding the first data containing one or more levels in the first frame by referring to the second data containing one or more levels in the second frame, and when encoding the first data, determining one of a plurality of processes for each of the one or more levels of the first data using values representing that level and values related to the one or more levels of the second data, and performing the determined one process on the first data of that level.
[0418] Furthermore, the software for implementing the decoding device, etc., described above is the following program.
[0419] That is, the above software is a program that enables a computer to perform the following decoding method: obtaining a first frame as the object of decoding, decoding the first data containing one or more layers in the first frame by referring to the second data containing one or more layers in the second frame, and when decoding the first data, determining one of a plurality of processes for each of the one or more layers of the first data using values representing that layer and values related to the one or more layers of the second data, and performing the determined one process on the first data of that layer.
[0420] Next, an example of attribute information in grid data will be explained. Here, "device" refers to an encoding or decoding device. Figures 45-47 This is a diagram illustrating an example of attribute information in grid data used to explain an implementation method.
[0421] like Figure 45 As shown, sometimes multiple 3D points G (vertexes) within a grid are projected onto a plane X, and the 2D coordinates on the projected plane X are set as the attribute information of these 3D points. For example, when the 3D point (x, y, z) is projected onto plane X and its projected 2D coordinates are (u, v), the 2D coordinates (u, v) are set as the attribute information of the 3D point (x, y, z). Figure 45 The example shown illustrates how a set 1501 of vertex coordinates of a three-dimensional mesh represented by mesh data is transformed into a set 1502 of two-dimensional coordinates on plane X by projecting it onto plane X.
[0422] In addition, such as Figure 46 As shown, sometimes multiple 3D points within a grid are projected onto a plane X. The plane information to which the 2D coordinates on the projected plane X belong is set as the attribute information of the 3D points. For example, when the projected 2D coordinates (u, v) of a 3D point (x, y, z) onto plane X belong to plane information face0, plane information face0 is set as the attribute information of the 3D point (x, y, z). In this case, by setting (u, v) = (face0, 0), the 3D point (x, y, z) is associated with plane information face0 as uv coordinates. Using this method, the relationship between the 3D point (x, y, z) and plane information face0 can be defined using the same structure as uv coordinates.
[0423] Furthermore, when projecting connected 3D points onto a 2D image, attribute information specific to those 3D points can be attached by setting the same planar information. For example, this can be done as follows.
[0424] exist Figure 46In this case, for a 3D point G and 3D points connected to G, face0 is set as attribute information. Similarly, for a 3D point G' and 3D points connected to G', face1 is set as attribute information. According to this method, the decoding device can obtain the set plane information by decoding the attribute information of the 3D points. Furthermore, using this plane information, for example, associated projection information can be obtained.
[0425] Furthermore, such as Figure 47 As shown, three-dimensional points within the mesh data, or the normal vectors corresponding to the plane to which those three-dimensional points belong, can also be set as attribute information. For example, the device can set the normal vector (nx, ny, nz) of a three-dimensional point (x, y, z) as attribute information. Additionally, it is also possible to set the three-dimensional point G and the normal vector (nx, ny, nz) corresponding to the plane to which the three-dimensional point G belongs as attribute information for G.
[0426] Furthermore, the attribute information in mesh data is not limited to this. Attribute information can include, for example, any information such as the color information of a 3D point, reflectivity, or the group ID to which the 3D point belongs. In addition, attribute information can also include 2D information such as UV coordinates, 1D information such as face0 (plane information), 3D information such as normal vectors, and other information with various dimensions.
[0427] Furthermore, it is possible to assign multiple attribute information to a 3D point. For example, the device can assign multiple attribute information such as UV coordinates and normal vectors to a 3D point (x, y, z). In addition, there may be 3D points without assigned attribute information. In this way, by assigning various attribute information to 3D points, more expressive mesh data can be generated.
[0428] This embodiment illustrates an example of a method for encoding or decoding attribute information using a device. The example shown illustrates the case of encoding or decoding one attribute piece of information assigned to a 3D point, but is not limited to this. For example, when two or more attribute pieces of information are assigned to a 3D point, this embodiment can also be used to encode or decode each attribute piece of information individually. Therefore, even for a 3D point assigned multiple attribute pieces of information, this attribute information can be encoded or decoded efficiently.
[0429] In this embodiment, an example of using UV coordinates as attribute information is shown, but it can also be applied to other attribute information. Figures 48-53 This is a diagram illustrating a specific example of an attribute information encoding method.
[0430] First, such as Figure 48As shown, the device selects a first initial 3D point v0 and encodes its uv coordinates without prediction. For example, the device can also directly encode the value of (u0, v0) in the following cases. Furthermore, the selection order of the 3D points for encoding attribute information is arbitrary; for example, the device can select them in the same order as when encoding the position information of the 3D points. Thus, the device can apply predictive coding while reducing the processing workload of selecting 3D points.
[0431] Next, as Figure 49 As shown, the device selects a 3D point v1. When 3D point v1 is connected to an already encoded 3D point v (i.e., has an edge), the device can use the uv coordinates of 3D point v0 as the predicted value to perform differential encoding on the uv coordinates of 3D point v1. For example, in the following case, the device can encode (u1-u0, v1-v0). Thus, the encoding amount can be reduced when the uv coordinate values of 3D point v0 and 3D point v1 are close. Furthermore, this method of predictive encoding using the uv coordinates of a connected 3D point is called "coarse prediction."
[0432] Next, as Figure 50 As shown, the device selects a 3D point v2. When 3D point v2 is connected to the already encoded 3D points v0 and v1 (i.e., when 3D point v2 has edges with 3D point v1 and v0), the device can generate a predicted value using the uv coordinates of 3D points v0 and v1, and perform differential encoding on the uv coordinates of 3D point v2. For example, in the following case, the device can encode (u2-predx, v2-predy). Thus, by generating predicted values with high accuracy using 3D points v0 and v1, the amount of encoding can be reduced. Furthermore, this method of predictive encoding using the uv coordinates of two or more connected 3D points is called "fine prediction."
[0433] Next, as Figure 51 As shown, the device selects a 3D point v3. When the 3D point v3 is connected to the encoded 3D points v1 and v2, that is, when the 3D point v3 has edges with the 3D point v1 and v2, the device can use the uv coordinates of the 3D points v1 and v2 to generate a prediction value based on fine prediction, and perform differential encoding on the uv coordinates of the 3D point v3.
[0434] Next, as Figure 52As shown, the device selects a 3D point v4. When 3D point v4 is connected to the already encoded 3D points v0, v1, and v3 (i.e., 3D point v4 has edges with 3D point v3, v4 with 3D point v1, and v4 with 3D point v3), the device can use the UV coordinates of 3D points v0, v1, and v3 to generate a prediction value based on fine prediction, and perform differential encoding on the UV coordinates of 3D point v4. In this case, for example, the device can also use the UV coordinates of 3D points v0 and v1 to calculate fine prediction value 1, use the UV coordinates of 3D points v1 and v3 to calculate fine prediction value 2, and use the average of fine prediction value 1 and fine prediction value 2 as the predicted value of the UV coordinates of 3D point v4. In cases where more than two fine prediction values can be calculated, using their average as the predicted value improves prediction accuracy and reduces the amount of encoding.
[0435] Next, as Figure 53 As shown, the device selects a 3D point v5. When 3D point v5 has no connectivity with any previously encoded 3D point, the device can select 3D point v5 as the second initial 3D point and, similar to the initial processing, directly encode the value of (u5, v5) without prediction. Likewise, the device reduces the amount of encoding for uv coordinates by applying predictive encoding through various processes.
[0436] Next, an example of a prediction pattern used to encode attribute information is given.
[0437] First, let's explain the no-prediction mode. The no-prediction mode is an example of the first prediction mode. The no-prediction mode is applied when the 3D point of the encoded object is an initial 3D point. Here, an initial 3D point refers to a 3D point of the processed object that has already been encoded and has no other encoded 3D points connected to it.
[0438] The device can directly encode attribute information values without a prediction mode. For example, with uv coordinates (100, 60), the device can encode the value of u (100) and v (60) respectively. It is also conceivable that the device performs fixed-length encoding with n bits (n is an integer value greater than 0). For example, with uv coordinates (100, 60), the device can perform fixed-length encoding of each component with 7 bits.
[0439] When the device uses fixed-length encoding of n bits, it can also append the value n to the header of the bitstream. Thus, by decoding the value n in the header, the decoding device can correctly decode the attribute information of the initial 3D points encoded with the fixed-length n bits.
[0440] The device can also change the bit length used for fixed-length encoding for each component of the attribute information. For example, with uv coordinates of (100, 60), the device applies 7-bit fixed-length encoding to u and 6-bit fixed-length encoding to v, thereby reducing the amount of encoding. In this case, the device can also append the bit length used in the fixed-length encoding of each component to the header of the bitstream. Thus, by decoding the bit length of each component in the header, the decoding device can correctly decode the attribute information of the initial 3D point encoded by changing the bit length for each component.
[0441] The device can set the bit length of each component to 0 bits. A bit length of 0 bits can also indicate that the component is not included in the bitstream. When the bit length appended to the header is 0 bits, the decoding device can determine that the component is not included in the bitstream and infer its value as 0. For example, if the attribute information contains a component whose value is always 0, the device sets the bit length of the fixed-length encoding for that component to 0 bits, and does not encode the value 0, thereby reducing the amount of encoding. Furthermore, when the bit length appended to the header is 0 bits, the decoding device can determine that the component is not included in the bitstream and infer its value as 0.
[0442] For example, such as using Figure 46 As explained, the device can also project a 3D point G(x, y, z) onto the xz plane, and use the plane face0 to which the 2D coordinates (u, v) on the projected xy plane belong as attribute information for G. In this case, the device can establish an association between G and face0 as uv coordinates by setting (u, v) = (face0, 0). In this case, since v is always 0, the device sets the bit length of the fixed-length encoding of v to 0 bits, and does not include the value 0 of v in the bit stream, thereby reducing the amount of encoding.
[0443] The device can also encode the attribute information of the initial 3D points using variable-length coding based on unary coding or exponential golomb coding. Furthermore, the device can reduce the coding complexity by assigning context to each binarized bit and applying arithmetic coding.
[0444] Furthermore, it is not necessary to apply a prediction-free mode to the initial 3D points; the device can also apply other prediction methods. For example, the device can calculate a representative value for the initial 3D points and use this representative value as a predicted value to predictively encode the attribute information of the initial 3D points. The representative value could be, for example, the average value of the initial 3D points. In this case, by attaching the representative value to the header, the decoding device can decode the header to obtain the predicted value of the initial 3D points, enabling correct decoding. Moreover, as the predicted value for the attribute information of the initial 3D points, the device can also use the attribute information of the initial 3D points that was encoded or decoded immediately before the initial 3D points in the encoding or decoding sequence. Thus, the device can reduce the amount of coding by predictively encoding the attribute information of the initial 3D points.
[0445] Next, the coarse prediction mode will be explained. The coarse prediction mode is an example of the second prediction mode. The coarse prediction mode is applied when there is a 3D point among the encoded 3D points that is connected to the 3D point of the encoded object.
[0446] When applying coarse prediction, the device can use the attribute information of an encoded and connected 3D point as the prediction value to encode the prediction residual. Furthermore, the tendency of the prediction residual may differ from that of the fine prediction; therefore, the device can also use different encoding or decoding methods for the prediction residual in coarse and fine prediction.
[0447] For example, when the device binarizes the prediction residuals using unary coding or exponential Golomb coding, assigns context to each bit, and applies arithmetic coding, it can use different contexts for coarse and fine prediction. Thus, even when the prediction residuals tend to have different values in coarse and fine prediction, the coding complexity can be reduced. Furthermore, the device can further reduce the coding complexity by using different binarization methods for coarse and fine prediction.
[0448] Next, the fine prediction mode will be explained. The fine prediction mode is an example of the third prediction mode. The fine prediction mode is applied when there are two or more 3D points in the encoded 3D points that are connected to the 3D points of the encoded object.
[0449] In the application of fine prediction, the device can use the attribute information of two or more coded and connected 3D points to calculate the predicted value and encode the prediction residual. Furthermore, when a 3D point with encoded object attribute information belongs to the vertices of two or more triangles formed by the encoded 3D points (e.g., ... Figure 52 (3D point v4), the device can sometimes use more than two triangles to generate more than two fine predictions. In this case, the device can also use the average of more than two fine predictions as the prediction (average fine prediction).
[0450] For example, if the device can generate *count* detailed prediction values, it can also calculate the sum of *count* detailed prediction values, divide this sum by *count* to calculate the average detailed prediction value, and use this average detailed prediction value as the prediction value for the encoded object attribute information. Thus, the device can improve the accuracy of predictions based on detailed predictions and reduce the amount of coding.
[0451] Additionally, the device can restrict the value of count to powers of 2. For example, by restricting the count used for averaging the fine forecast to powers of 2 such as 1, 2, 4, or 8, the device can use right shift instead of division when performing average arithmetic, thus reducing the computational load. For instance, a 1-bit right shift is applied when count=2, and a 2-bit right shift is applied when count=4, thereby generating the average fine forecast value without division.
[0452] Alternatively, in this case, averaging fine predictions using counts that are not powers of 2 can be prohibited, and the device can omit this calculation. This reduces the processing load for calculating the average fine prediction value. Furthermore, the device can limit the number of counts used to calculate the average fine prediction. For example, the device can set a `fineMaxcount` representing the maximum value of the count, limiting the number of fine predictions used to calculate the average fine prediction value to below `fineMaxcount`. This further reduces the processing load for calculating the average fine prediction.
[0453] The device can also append fineMaxcount to the header of the bitstream. Thus, by decoding the fineMaxcount in the header, the decoding device can calculate the average fine prediction value using the same maximum value of count as the encoding device, and can correctly decode the bitstream.
[0454] Furthermore, by setting fineMaxcount=1, the device can limit the number of fine predictions used in the calculation of the average fine prediction value to one, effectively eliminating the need for averaging calculations. This reduces the processing load of averaging calculations and prevents the generation of values below the decimal point due to averaging calculations.
[0455] Furthermore, by setting fineMaxcount=0, the device can prevent fine prediction. In this case, the prediction mode is (1) no prediction mode or (2) coarse prediction mode. For example, when the attribute information of all subsequent 3D points of the initial 3D point is the same, the device applies no prediction mode to the initial 3D point and coarse prediction to the subsequent 3D points, thereby ensuring that the prediction residual is always 0, and sometimes fine prediction is not required. In such cases, by setting fineMaxcount=0, the device can prevent fine prediction from being applied, thereby reducing the amount of processing involved in encoding or decoding.
[0456] In addition, the device sets fineMaxcount=0 when there is additional information required for fine prediction, so that fine prediction is not applied, thereby reducing additional information and reducing the amount of coding.
[0457] Furthermore, the predictive coding described above can also be applied to predictive decoding. That is, by replacing "coding" with "decoding" above, the description of predictive coding can be used as a description of predictive decoding.
[0458] Figure 54 This is an example of a flowchart illustrating the encoding of attribute information of three-dimensional points in an implementation method.
[0459] The encoding device acquires multiple three-dimensional points (vertices) of the three-dimensional mesh (S1501).
[0460] The encoding device determines whether there are any 3D points whose attribute information has not been encoded (S1502). If there are any 3D points whose attribute information has not been encoded ("Yes" in S1502), the encoding device proceeds to step S1503; otherwise ("No" in S1502), the process ends.
[0461] The encoding device selects the three-dimensional point of the object to be encoded from the unencoded three-dimensional points among a plurality of three-dimensional points (S1503).
[0462] The encoding device determines whether there are two or more encoded 3D points that are connected to the 3D points of the encoding object (S1504). If there are two or more encoded 3D points that are connected to the 3D points of the encoding object ("Yes" in S1504), the encoding device proceeds to step S1505; otherwise ("No" in S1504), it proceeds to step S1508.
[0463] The encoding device uses attribute information from two or more three-dimensional points to calculate the predicted value (S1505).
[0464] The encoding device determines whether count is greater than 0 (S1506). If count is greater than 0 ("Yes" in S1506), the encoding device proceeds to step S1507; otherwise ("No" in S1506), it proceeds to step S1508.
[0465] The encoding device encodes the attribute information of the three-dimensional points of the object using the calculated predicted values (S1507). That is, the encoding device calculates the residual between the attribute information of the three-dimensional points of the object and the predicted values calculated using the attribute information of two or more three-dimensional points, and encodes the calculated residual. Step S1507 is the predictive encoding of the third prediction mode.
[0466] The encoding device determines whether there exists an encoded 3D point that is connected to the 3D point of the encoding object (S1508). If there is an encoded 3D point that is connected to the 3D point of the encoding object ("Yes" in S1508), the encoding device proceeds to step S1509; otherwise ("No" in S1508), it proceeds to step S1510.
[0467] The encoding device uses the attribute information of a three-dimensional point as a prediction value to encode the attribute information of the three-dimensional point of the object being encoded (S1509). That is, the encoding device calculates the residual between the attribute information of the three-dimensional point of the object being encoded and the attribute information of a three-dimensional point, and encodes the calculated residual. Step S1509 is the predictive encoding of the second prediction mode.
[0468] The encoding device encodes the attribute information of the three-dimensional points of the object in a prediction-free mode (S1510). That is, the encoding device does not use predicted values to encode the attribute information of the three-dimensional points of the object. Step S1510 is the predictive encoding of the first prediction mode.
[0469] Figure 55 This is a flowchart illustrating an example of the calculation process for the predicted value of the third prediction mode in the implementation method.
[0470] The encoding device sets the count to 0 (S1511).
[0471] The encoding device determines whether count is less than fineMaxcount (S1512). If count is less than fineMaxcount ("Yes" in S1512), the encoding device proceeds to step S1513; otherwise ("No" in S1512), it proceeds to step S1515. Additionally, the encoding device can disable fine prediction by setting fineMaxcount to 0. Furthermore, the encoding device can also append the set value of fineMaxcount to the bitstream.
[0472] Furthermore, the decoder can also decode attribute information without applying fine prediction if the value of fineMaxcount decoded from the bitstream is 0.
[0473] The encoding device calculates the count-th fine prediction value (S1513).
[0474] The encoding device increments count by 1, updates count (S1514), and returns to step S1512.
[0475] The encoding device determines whether count is greater than 0 (S1515). If count is greater than 0 ("Yes" in S1515), the encoding device proceeds to step S1516; otherwise ("No" in S1515), it proceeds to step S1517.
[0476] The encoding device calculates the average fine prediction value (S1516). That is, the encoding device calculates the average of more than one fine prediction value as the average fine prediction value. Alternatively, if there is only one fine prediction value, the encoding device may also calculate one fine prediction value as the average fine prediction value. In addition, more than one fine prediction value refers to the count of fine prediction values obtained in steps S1511 to S1514.
[0477] The encoding device outputs count (S1517).
[0478] Figure 56 This is an example of a flowchart illustrating the decoding of attribute information of three-dimensional points in an implementation method.
[0479] The decoding device acquires multiple three-dimensional points (vertices) of the three-dimensional mesh (S1521).
[0480] The decoding device determines whether there are any 3D points whose attribute information has not been decoded (S1522). If there are any 3D points whose attribute information has not been decoded ("Yes" in S1522), the decoding device proceeds to step S1523; otherwise ("No" in S1522), the processing ends.
[0481] The decoding device selects the three-dimensional point of the object to be decoded from the undecoded three-dimensional points among a plurality of three-dimensional points (S1523).
[0482] The decoding device determines whether there are two or more decoded 3D points that are connected to the 3D points of the decoding object (S1524). If there are two or more decoded 3D points that are connected to the 3D points of the decoding object ("Yes" in S1524), the decoding device proceeds to step S1525; otherwise ("No" in S1524), it proceeds to step S1528.
[0483] The decoding device uses attribute information from two or more three-dimensional points to calculate the predicted value (S1525).
[0484] The decoding device determines whether count is greater than 0 (S1526). If count is greater than 0 ("Yes" in S1526), the decoding device proceeds to step S1527; otherwise ("No" in S1526), it proceeds to step S1528.
[0485] The decoding device uses the calculated predicted values to decode the attribute information of the three-dimensional points of the object to be decoded (S1527). That is, the decoding device calculates the residual between the attribute information of the three-dimensional points of the object to be decoded and the predicted values calculated using the attribute information of two or more three-dimensional points, and decodes the calculated residual. Step S1527 is the prediction decoding of the third prediction mode.
[0486] The decoding device determines whether there exists a decoded 3D point that is connected to the 3D point of the decoding object (S1528). If there is a decoded 3D point that is connected to the 3D point of the decoding object ("Yes" in S1528), the decoding device proceeds to step S1529; otherwise ("No" in S1528), it proceeds to step S1530.
[0487] The decoding device uses the attribute information of a three-dimensional point as the prediction value to decode the attribute information of the three-dimensional point of the decoding object (S1529). That is, the decoding device calculates the residual between the attribute information of the three-dimensional point of the decoding object and the attribute information of a three-dimensional point, and decodes the calculated residual. Step S1529 is the prediction decoding of the second prediction mode.
[0488] The decoding device decodes the attribute information of the 3D points of the decoding object in a prediction-free mode (S1530). That is, the decoding device does not use predicted values to decode the attribute information of the 3D points of the decoding object. Step S1530 is predictive decoding in the first prediction mode.
[0489] Next, we will explain mesh_attribute_coding, which is the unit for encoding the attribute information of three-dimensional points. Figure 57 This is a diagram illustrating an example of the syntax for mesh_attribute_coding in an implementation. Figure 58 This is a diagram illustrating an example of the syntax of the mesh_attribute_header in the implementation method.
[0490] `mesh_attribute_coding` represents the unit used to encode the attribute information of 3D points, and `mesh_attribute_header` represents the header information related to the attribute information. Furthermore, `mesh_attribute_data` represents the encoded data related to the attribute information.
[0491] AttributeBitDepthMinus1 represents the bit precision of the attribute information encoded into the bitstream. The bit precision of the attribute information is calculated by adding 1 to AttributeBitDepthMinus1.
[0492] NumComponent represents the number of components in an attribute. For example, if the attribute is uv coordinates, NumComponent is set to 2; if the attribute is a normal vector, NumComponent is set to 3.
[0493] AttributeStartDeltaBitDepth[j] represents the bit precision of the initial three-dimensional point of the attribute information encoded into the bitstream. The bit precision of the j-th component of the initial three-dimensional point, AttributeStartBitDepth[j], is calculated as follows.
[0494] AttributeStartBitDepth[j]=AttributeBitDepthMinus1+1-AttributeStartDeltaBitDepth[j] For example, when AttributeBitDepthMinus1 is 9, AttributeStartDeltaBitDepth[0] is 0, and AttributeStartDeltaBitDepth[1] is 2, AttributeStartBitDepth[0]=9+1-0=10 AttributeStartBitDepth[1]=9+1-2=8 It can show that the bit precision of the 0th component of the initial three-dimensional point is 10 bits and the bit precision of the 1st component is 8 bits.
[0495] In this way, by setting AttributeStartDeltaBitDepth for each component of the initial 3D point, different bit precision can be set for each component. Furthermore, by setting the bit precision of the initial 3D point differently from the bit precision of the attribute information encoded into the bitstream (AttributeBitDepthMinus1), for example, if the range of possible values for each component of the initial 3D point is smaller than the range of possible values for each component of the attribute information encoded into the bitstream, the value of AttributeStartBitDepth may be smaller than the value obtained by adding 1 to AttributeBitDepthMinus1. In such cases, by using the value of AttributeStartBitDepth instead of the value obtained by adding 1 to AttributeBitDepthMinus1 to encode the values of each component of the initial 3D point (i.e., AttributeStart), as described later, the amount of encoding can be reduced.
[0496] Furthermore, in this embodiment, the difference between the bit precision of the attribute information encoded into the bitstream (AttributeBitDepthMinus1 plus 1) and the bit precision of the j-th component of the initial 3D point can also be encoded as AttributeStartDeltaBitDepth. Thus, when the bit precision of the attribute information is the same as the bit precision of the initial 3D point, the difference value added to the bitstream becomes 0, thereby reducing the amount of encoding.
[0497] Furthermore, the value of AttributeStartBitDepth[j] can be set to 0 by setting AttributeStartBitDepth[j] to the same value as adding 1 to AttributeBitDepthMinus1. In this case, for the j-th component where AttributeStartBitDepth[j] is 0, the value 0 can be omitted from the bitstream. The decoder can infer the value 0 for the j-th component where AttributeStartBitDepth[j] is 0. Thus, by not appending the component where AttributeStartBitDepth[j] is 0 to the bitstream, the amount of encoding can be reduced.
[0498] This embodiment shows an example of appending AttributeStartDeltaBitDepth[j] as the difference value to the bitstream, but it is not limited to this. For example, AttributeStartBitDepth[j] can also be directly encoded into the bitstream. This method can reduce the amount of processing required to calculate the difference value.
[0499] Furthermore, the bit length of AttributeStartDeltaBitDepth[j] can be calculated based on the value of AttributeBitDepthMinus1. Specifically, the bit length of AttributeStartDeltaBitDepth[j] can be calculated as ceil(log2(AttributeBitDepthMinus1+2)). Here, ceil(x) refers to the function that outputs the integer value of x rounded up. In this way, by determining the bit length of AttributeStartDeltaBitDepth[j] based on the value of AttributeBitDepthMinus1, the amount of encoding can be reduced.
[0500] AttributePredMode indicates the method used to encode attribute information. For example, it can also represent... Figure 54 The examples shown include prediction methods such as methodA, which includes fine prediction, coarse prediction, and no prediction mode. However, the prediction method is not limited to methodA. For example, it could be a prediction method that combines two of the three prediction modes from methodA, or a prediction method that uses only one prediction mode. Specifically, consider prediction methods such as methodB, which includes coarse prediction and no prediction mode. By setting methodB as the prediction method, predicted values can be generated without division or arithmetic operations requiring decimal precision, thus reducing processing overhead and enabling lossless encoding.
[0501] fineMaxcount represents the maximum number of fine predictions used when calculating the average fine prediction value. When fine prediction is included in AttributePredMode, such as when AttributePredMode is methodA, fineMaxcount can also be appended to the bitstream.
[0502] Furthermore, by setting fineMaxcount to 1, the number of fine forecasts used in the calculation of the average fine forecast value can be limited to one, effectively eliminating the need for averaging. This reduces the processing load of averaging. Additionally, it prevents the generation of values below the decimal point due to averaging calculations.
[0503] Alternatively, setting fineMaxcount to 0 can disable fine prediction. In this case, coarse prediction or no prediction mode is used as the prediction mode. For example, if the attribute information of all subsequent 3D points are identical, applying no prediction mode to the initial 3D point and coarse prediction to subsequent 3D points can result in a prediction residual of 0, thus eliminating the need for fine prediction. In this case, setting fineMaxcount to 0 disables fine prediction, thereby reducing the processing workload involved in encoding or decoding.
[0504] Furthermore, in cases where additional information is required for fine prediction, setting fineMaxcount to 0 prevents fine prediction from being applied, thereby reducing additional information and the amount of coding.
[0505] AttributeBitDepthMinus1, AttributePredMode, and fineMaxcount can also be binarized using exponential Golomb coding or unary coding, applying variable-length encoding. Furthermore, allocating context to each bit and updating the probability of occurrence based on the frequency of 0 and 1 occurrences during encoding improves encoding efficiency. Alternatively, this information can be encoded with a fixed length, thereby reducing processing overhead.
[0506] Additionally, fineMaxcount can be set to different values per sequence unit, frame unit, basemesh unit, submesh unit, or 3D point. This allows for reduction in processing power by controlling the number of fine predictions used for averaging in appropriate units.
[0507] Figure 59 This is a diagram illustrating an example of the syntax of mesh_attribute_data in an implementation.
[0508] `AttributeStartCount` indicates the number of initial 3D points. `AttributeStart[c][j]` indicates the value of the j-th component of the attribute information related to the c-th initial 3D point. For example, when the attribute information is uv coordinates, `AttributeStart[c][0]` represents the value of u of the c-th initial 3D point, and `AttributeStart[c][1]` represents the value of v of the c-th initial 3D point. Alternatively, `AttributeStart[c][j]` can also be used... Figure 58The bit length calculated as AttributeStartBitDepth is used for fixed-length encoding as described in the text. For example, when AttributeStartBitDepth[0] is 10 and AttributeStartBitDepth[1] is 8, it is possible to encode AttributeStart[c][0], which represents the 0th component of the c-th initial 3D point, with 10-bit precision, and AttributeStart[c][1], which represents the 1st component, with 8-bit precision. By using 10 bits to encode the 0th component with a fixed length and 8 bits to encode the 1st component with a fixed length, the amount of encoding can be reduced.
[0509] Furthermore, if the value of AttributeStartBitDepth[j] is 0, AttributeStart[c][j] may not be encoded into the bitstream. The decoder, by confirming that the value of AttributeStartBitDepth[j] is 0, can infer that the value of AttributeStart[c][j] is 0. Thus, by not including components with a value of 0 in the bitstream, the amount of encoding can be reduced.
[0510] Furthermore, AttributeStart can also apply variable-length encodings such as unary encoding or exponential Golomb encoding. Additionally, the encoding size can be reduced by assigning context to each bit after binarization and applying arithmetic encoding.
[0511] Furthermore, when AttributeStartCount is 0, encoding AttributeStart is unnecessary. In this case, the decoder can confirm that AttributeStartCount is 0 and infer the value of AttributeStart as 0. This method reduces the amount of encoding required.
[0512] Furthermore, ensuring that at least one initial 3D point exists, AttributeStartCountMinus1 can be appended to the bitstream instead of AttributeStartCount. In this case, the decoder can calculate AttributeStartCount by adding 1 to AttributeStartCountMinus1. This further reduces the amount of coding required.
[0513] AttributeFineResiCount represents the number of prediction residuals for attribute information after prediction encoding through fine prediction. Specifically, for optimal efficiency, we assume AttributeFineResiCount is set to 0 when fineMaxcount is 0.
[0514] AttributeFineResiSize indicates the amount of encoding for AttributeFineResi. For example, it can represent the amount of encoding when AttributeFineResi is binarized and arithmetic-encoded. Thus, by informing the decoder of the amount of encoding using AttributeFineResiSize before AttributeFineResi, the decoder can pre-read the bits related to AttributeFineResi from the bitstream. Specifically, while the decoder is performing arithmetic decoding related to AttributeFineResi, the subsequent decoding of AttributeCoarseResi can be performed in parallel, thereby reducing decoding time.
[0515] AttributeFineResi[c][j] represents the prediction residual information related to the j-th component of the attribute information after prediction encoding via the c-th fine prediction. AttributeFineResi[c][j] can be binarized using exponential Golomb coding or unary coding, enabling variable-length encoding. Furthermore, by allocating context to each bit and updating the occurrence probability based on the frequency of 0 and 1 occurrences during encoding, encoding efficiency can be improved.
[0516] Furthermore, the context used in the arithmetic encoding of each bit can be shared with the context used in AttributeCoarseResi (described later). This method reduces the number of contexts. Alternatively, the context used in the arithmetic encoding of each bit can be set to a different context than that used in AttributeCoarseResi (described later). By using different contexts, coding efficiency can be improved even when the tendency to predict residuals differs in fine and coarse prediction. Additionally, by using different contexts, AttributeFineResi and AttributeCoarseResi can be decoded in parallel, thus further improving the efficiency of the decoding process.
[0517] AttributeCoarseResiCount represents the number of prediction residuals for attribute information after coarse prediction encoding. AttributeCoarseResiCount plays an important role in the encoding of AttributeCoarseResi, which will be discussed later.
[0518] AttributeCoarseResiSize indicates the amount of encoding for AttributeCoarseResi. For example, it can represent the amount of encoding when AttributeCoarseResi is binarized and arithmetic-encoded. Thus, by informing the decoder of AttributeCoarseResiSize before encoding AttributeCoarseResi, the decoder can pre-read the bits associated with AttributeCoarseResi from the bitstream. Specifically, during the decoder's execution of arithmetic decoding associated with AttributeCoarseResi, the decoding of subsequent bits in the bitstream can be performed in parallel, thereby reducing decoding time.
[0519] AttributeCoarseResi[c][j] represents the prediction residual related to the j-th component of the attribute information after prediction encoding via the c-th coarse prediction. AttributeCoarseResi[c][j] can be binarized using exponential Golomb coding or unary coding, enabling variable-length encoding. Furthermore, by allocating context to each bit and updating the occurrence probability based on the frequency of 0 and 1 occurrences during encoding, encoding efficiency can be improved.
[0520] Furthermore, the context used in the arithmetic encoding of each bit can be shared with the context used in AttributeFineResi described above. This method reduces the number of contexts. Alternatively, the context used in the arithmetic encoding of each bit can be set to a different context than that used in AttributeFineResi described above. By using different contexts in this way, encoding efficiency can be improved even when the tendency to predict residuals differs between fine and coarse prediction.
[0521] Furthermore, AttributeStartCount, AttributeFineResiCount, AttributeFineResiSize, AttributeFineResi, AttributeCoarseResiCount, AttributeCoarseResiSize, and AttributeCoarseResi can also be binarized using exponential Golomb coding or unary coding for variable-length encoding. In addition, by allocating context to each bit and updating the probability of occurrence based on the frequency of 0 and 1 occurrences while encoding, encoding efficiency can be improved. Furthermore, this information can also be encoded with a fixed length, reducing processing load.
[0522] In this description of the embodiment, examples related to the encoding or decoding of attribute information of three-dimensional points are mainly shown, but it is not necessarily limited to this. For example, this embodiment can also be applied to the encoding or decoding of position information of three-dimensional points. Syntactic examples for the case of applying to position information of three-dimensional points will be described later.
[0523] In this embodiment, for example, such as Figure 46 As shown, the following example illustrates how, when a device projects a 3D point G(x, y, z) onto the xz plane, and sets the plane face0, to which the 2D coordinates (u, v) on the projected xy plane belong, as attribute information for G, it is set as (u, v) = (face0, 0). This establishes an association between G and face0 as uv coordinates, but it is not necessarily limited to this. For example, when face0 is encoded or decoded as uv coordinates, it is uv coordinates as attribute information. However, it is also possible to encode only u in the uv coordinates by setting the attribute information to NumComponent=1. Thus, face0 is encoded as u, while v is not encoded, thereby reducing the amount of encoding.
[0524] Alternatively, connected 3D points can be projected onto a 2D image, with the same planar information appended as attribute information for each 3D point, and this attribute information can be encoded. Thus, by decoding the attribute information of the 3D points, the decoder can obtain the planar information associated with each 3D point, and can use this value to obtain, for example, associated projection information. The method of generating the UV coordinates of 3D points using this projection information on the decoder side can also be called "ortho atlas".
[0525] When using Ortho Atlas to encode planar information as UV coordinates, the attribute information of connected 3D points, for example... Figure 60All of them shown are the same value (face0, 0). Figure 60 This is a diagram illustrating an example of attribute information for a variation of the implementation method.
[0526] In this case, the device encodes the initial 3D points in a no-prediction mode, and by applying coarse encoding to subsequent 3D points, the prediction residual can be reduced to 0. Therefore, when using ortho atlas, by setting fineMaxcount=0 as described in this embodiment, fine prediction is not applied, thus reducing the processing load involved in encoding or decoding. Furthermore, when additional information required for fine prediction exists, setting fineMaxcount to 0 to prevent fine prediction from being applied reduces the amount of additional information and the amount of encoding.
[0527] Furthermore, in uv coordinates, v is always 0. Therefore, when using ortho atlas, by using AttributeBitDepthMinus1 and AttributeStartDeltaBitDepth as described in this embodiment, the AttributeStartBitDepth for the v component of the initial 3D point is set to 0, and the bit length of the fixed-length encoding of the v component is set to 0 bits, thereby not encoding the value 0 of v, thus reducing the amount of encoding.
[0528] Furthermore, when using ortho atlases, by applying coarse and fine predictions, the prediction residual can be reduced to zero. Therefore, the coding complexity can be reduced by not encoding at least one of AttributeFineResiCount, AttributeFineResiSize, AttributeFineResi, AttributeCoarseResiCount, AttributeCoarseResiSize, and AttributeCoarseResi, which are information related to the prediction residual. The unencoded information can be inferred as a value of 0 at the decoder side.
[0529] Furthermore, when using ortho atlas without applying fine predictions, the amount of coding can also be reduced by not encoding the additional information associated with fine predictions.
[0530] Figure 61 This is a diagram illustrating an example of the syntax of mesh_attribute_data, representing a variation of the implementation method.
[0531] Alternatively, if AttributeFineResiCount or AttributeCoarseResiCount is not encoded, the number of 3D points connected to the initial 3D points, OtherAttributeCount, can be appended to the bitstream according to the number of initial 3D points. Figure 60 An example of its syntax is shown in the figure. In this case, the decoder decodes the uv coordinates of the initial 3D point c and copies the uv coordinates to the uv coordinates of OtherAttributeCount[c] 3D points that are connected to the initial 3D point c, thereby enabling the correct decoding of the uv coordinates.
[0532] Furthermore, OtherAttributeCount can also be binarized using exponential Golomb coding, unary coding, etc., for variable-length encoding. By assigning context to each bit and updating the probability of occurrence based on the frequency of 0 and 1 occurrences during encoding, encoding efficiency can be improved. Alternatively, it can be encoded with a fixed length. In this case, the processing load can be reduced.
[0533] Furthermore, an AttributePredMode for orthoatlas can be defined. When the encoder encodes attribute information using the method described above, it appends information indicating that AttributePredMode is the prediction method used for orthoatlas to the bitstream. Thus, by decoding AttributePredMode, the decoder can determine whether the bitstream was encoded using the method described above. If the determination result is "yes," the decoder can decode the attribute information using the method described above.
[0534] This embodiment illustrates a method for encoding or decoding attribute information corresponding to three-dimensional points, such as UV coordinates, but is not limited thereto. For example, the encoding or decoding method described in this embodiment can also be applied to attribute information corresponding to the faces of a polygon composed of three-dimensional points.
[0535] Specifically, the face information corresponding to the face of a polygon, i.e., face_id, can be encoded or decoded as attribute information with NumComponent=1. Therefore, this implementation method can reduce the amount of encoding required for attribute information corresponding to 3D points or for attribute information corresponding to polygons composed of 3D points.
[0536] Figure 62 This is a diagram illustrating an example of the geometry of an implementation method. Figure 63 This is an example of a texture map representing an implementation method.
[0537] Texture mapping is a general technique used to add visual appearance by projecting an image onto the surfaces of a 3D model. This technique can alter visual details without directly modifying the model itself, thus improving performance when rendering is required. Information stored in a 2D texture image can also include color, smoothness, and transparency. The relationship between the 3D mesh and the 2D texture image is established using UV mapping. UV mapping is created by deforming and unfolding the 3D mesh onto a 2D plane. Then, by overlaying the texture image onto the UV mapping, attribute values from the image are projected onto the 3D model during rendering. Therefore, encoding and decoding techniques capable of efficiently storing and streaming UV mapping are needed.
[0538] Different prediction schemes can be used to encode texture coordinates in UV maps. These schemes include those based on parametric techniques used when projecting a 3D mesh onto a 2D map, or those based on the relationship between the mesh's geometry and texture coordinates. In isoangular parametric cases, the edge lengths of triangles maintain approximately the same ratio in both 3D space and UV coordinates. Therefore, texture coordinates on the UV map can be predicted using ratios derived from the corresponding edges in 3D space.
[0539] like Figure 62 As shown, texture coordinates A2, B2, and C2 correspond to A1, B1, and C1 on the geometry, respectively. The ratio between edges A1B1 and A2B2 in 3D space is calculated as (A1C1) / (C1B1) = R. Finally, the coordinates of A2 in UV space are predicted as A2C2 = R × C2B2 and A2 = C2 + A2C2.
[0540] like Figure 63 As shown, the device can calculate the predicted uv coordinates A2 for a 3D point A1 projected onto a 2D image (texture map). The device can also calculate this predicted value using encoded or decoded C2 and B2, and the position information of the 3D points A1, B1, and C1 corresponding to A2, B2, and C2.
[0541] More specifically, the device can assume that triangle A1B1C1 and triangle A2B2C2 are similar, and calculate A2C2 by multiplying the already determined value of C2B2 by the similarity ratio R, which is calculated as (A1C1) / (C1B1) = R. Furthermore, the device can predict the position of A2 using C2 + A2C2. Thus, when triangles formed by corresponding 3D points are similar in both the geometry and texture maps, the device can use the 3D point information in the encoded or decoded geometry and the UV coordinates in the encoded or decoded texture to accurately predict the attribute information of the 3D points of the encoded or decoded object, thereby reducing the amount of encoding.
[0542] Alternatively, a method for calculating predicted values of the attribute information of the three-dimensional points of an encoded or decoded object using such a device, which uses the three-dimensional point information in the encoded or decoded geometry and the uv coordinates in the encoded or decoded texture map, can be used as fine prediction in this embodiment.
[0543] Next, we will explain mesh_position_coding, which is a unit for encoding the position information of three-dimensional points. Figure 64 This is a diagram illustrating an example of the syntax for mesh_position_coding in an implementation. Figure 65 This is a diagram illustrating an example of the syntax of mesh_position_header in the implementation.
[0544] `mesh_position_coding` is a unit used to encode the position information of 3D points, performing processing related to the encoding and decoding of position information. `mesh_position_header` represents the header information related to position information, and `mesh_position_data` represents the encoded data related to position information.
[0545] PositionBitDepthMinus1 represents the bit precision of the position information encoded into the bitstream. The bit precision of the position information can be calculated by adding 1 to PositionBitDepthMinus1.
[0546] NumComponent represents the number of components in the location information. For example, when the location information is xyz coordinates, the value of NumComponent can be set to 3.
[0547] PositionStartDeltaBitDepth[j] represents the bit precision of the initial three-dimensional point whose position information is encoded into the bitstream. The bit precision PositionStartBitDepth[j] of the j-th component of the initial three-dimensional point can be calculated as follows.
[0548] PositionStartBitDepth[j] can be calculated by adding 1 to PositionBitDepthMinus1 and subtracting PositionStartDeltaBitDepth[j]. For example, when PositionBitDepthMinus1 is 9, PositionStartDeltaBitDepth[0] is 0, PositionStartDeltaBitDepth[1] is 2, and PositionStartDeltaBitDepth[2] is 4, the value of PositionStartBitDepth[0] is 9+1-0=10, the value of PositionStartBitDepth[1] is 9+1-2=8, and the value of PositionStartBitDepth[2] is 9+1-4=6. Thus, it can be shown that the 0th component of the initial 3D point has 10-bit precision, the 1st component has 8-bit precision, and the 2nd component has 6-bit precision.
[0549] In this way, by setting PositionStartDeltaBitDepth for each component of the initial 3D point, different bit precision can be set for each component. Furthermore, by enabling the initial 3D point's bit precision to be set differently from the bit precision of the position information encoded into the bitstream (i.e., PositionBitDepthMinus1), for example, if the range of possible values for each component of the initial 3D point is smaller than the range of possible values for each component of the position information encoded into the bitstream, the value of PositionStartBitDepth may be smaller than the value obtained by adding 1 to PositionBitDepthMinus1.
[0550] In this case, by using the value of PositionStartBitDepth instead of adding 1 to PositionBitDepthMinus1 to encode the values of each component of the initial 3D point (described later), i.e., PositionStart, the amount of coding can be reduced. Furthermore, this setting avoids unnecessary specification of bit precision, thereby improving coding efficiency.
[0551] In this embodiment, the difference between the bit precision of the position information encoded into the bitstream (i.e., PositionBitDepthMinus1 plus 1) and the bit precision of the j-th component of the initial 3D point can be encoded as PositionStartDeltaBitDepth. By using this method, when the value of PositionBitDepthMinus1 (the bit precision of the position information) plus 1 is the same as the bit precision of the j-th component of the initial 3D point, the difference value appended to the bitstream becomes 0, thus suppressing the amount of encoding.
[0552] Furthermore, by setting PositionStartDeltaBitDepth[j] to the same value as adding 1 to PositionBitDepthMinus1, the value of PositionStartBitDepth[j] for the j-th component can be made 0. In this case, for the j-th component where PositionStartBitDepth[j] is 0, the value 0 can also be omitted from the bitstream. For the j-th component where PositionStartBitDepth[j] is 0, the decoder can infer that it is a value 0. Thus, by not appending the component where PositionStartBitDepth[j] is 0 to the bitstream, the amount of encoding can be reduced.
[0553] This embodiment illustrates a method of appending PositionStartDeltaBitDepth[j], which is the difference value, to the bitstream, but it is not limited to this method. For example, it is also possible to directly encode PositionStartBitDepth[j] into the bitstream. This method can reduce the amount of processing required to compute the difference value.
[0554] Furthermore, the bit length of PositionStartDeltaBitDepth[j] can be determined based on the value of PositionBitDepthMinus1. Specifically, the bit length of PositionStartDeltaBitDepth[j] can be calculated as ceil(log2(PositionBitDepthMinus1+2)). Here, ceil(x) is a function that outputs the integer value of x rounded up. In this way, by determining the bit length of PositionStartDeltaBitDepth[j] based on the value of PositionBitDepthMinus1, the amount of coding can be reduced efficiently.
[0555] PositionPredMode represents information about the method used to encode position information. For example, method A could be used as a prediction method that includes fine prediction, coarse prediction, and no prediction modes as shown in the encoding flowchart. The prediction method is not limited to method A; it could also be a prediction method that combines two of the three prediction modes, or a prediction method consisting of only one prediction mode. As a specific example, consider method B, which combines coarse prediction and no prediction modes. By using method B, predicted values can be generated without applying division or arithmetic operations requiring decimal precision, thus reducing processing overhead and achieving lossless encoding. Furthermore, in the case of position information, fine prediction could include methods such as parallelogram prediction.
[0556] `fineMaxcount` represents the maximum number of fine predictions used when calculating the average fine prediction value. When `PositionPredMode` includes fine predictions, such as when `PositionPredMode` is set to `methodA`, `fineMaxcount` can be appended to the bitstream. By setting `fineMaxcount` to 1, the number of fine predictions used in calculating the average fine prediction value can be limited to one, effectively eliminating the need for averaging. This reduces the processing load of averaging and prevents values with decimal precision from being generated during averaging.
[0557] Furthermore, setting fineMaxcount to 0 disables fine prediction. In this case, coarse prediction or no prediction mode can be used as the prediction mode. For example, if the positional information of all subsequent 3D points are identical, applying no prediction mode to the initial 3D point and coarse prediction to subsequent 3D points ensures that the prediction residual is always 0. In this case, setting fineMaxcount to 0 disables fine prediction, thereby reducing the processing required for encoding or decoding.
[0558] Furthermore, when additional information is required for fine prediction, by setting fineMaxcount to 0, fine prediction is not applied and the additional information is not encoded, thereby reducing the amount of coding.
[0559] Figure 66 This is a diagram illustrating an example of the syntax of mesh_position_data in the implementation method.
[0560] PositionStartCount represents the number of initial 3D points. PositionStart[c][j] represents the value of the j-th component of the position information related to the c-th initial 3D point. For example, when the position information is xyz coordinates, PositionStart[c][0] represents the x value of the c-th initial 3D point, PositionStart[c][1] represents the y value of the c-th initial 3D point, and PositionStart[c][2] represents the z value of the c-th initial 3D point.
[0561] Alternatively, PositionStart[c][j] can also be encoded with a fixed length using the bit length calculated using PositionStartBitDepth as described above. For example, when PositionStartBitDepth[0] is 10, PositionStartBitDepth[1] is 8, and PositionStartBitDepth[2] is 6, PositionStart[c][0], representing the 0th component of the c-th initial 3D point, represents 10-bit precision; PositionStart[c][1], representing the 1st component, represents 8-bit precision; and PositionStart[c][2], representing the 2nd component, represents 6-bit precision. The 0th component is encoded with 10 bits, the 1st component with 8 bits, and the 2nd component with 6 bits, respectively. By setting an appropriate bit length for each component of the initial 3D point and using it for encoding, the amount of encoding can be reduced.
[0562] Furthermore, when PositionStartBitDepth[j] is 0, PositionStart[c][j] may not be encoded into the bitstream, and the decoder may also infer the value of PositionStart[c][j] as 0 when PositionStartBitDepth[j] is 0. According to this method, by not appending the j-th component where PositionStartBitDepth[j] is 0 to the bitstream, the amount of encoding can be reduced.
[0563] Alternatively, PositionStart can use variable-length encoding such as unary coding or exponential Golomb coding. Furthermore, context can be assigned to each bit after binarization, and arithmetic coding can be applied. This reduces the amount of coding required.
[0564] Furthermore, when PositionStartCount is 0, PositionStart can be omitted from encoding, and the decoder can also infer the value of PositionStart as 0 when PositionStartCount is 0. This reduces the amount of encoding required.
[0565] Furthermore, if there is at least one initial 3D point, PositionStartCountMinus1 can be appended to the bitstream instead of PositionStartCount. In this case, the decoder can also calculate PositionStartCount by adding 1 to PositionStartCountMinus1.
[0566] PositionFineResiCount represents the number of prediction residuals for the position information after prediction encoding using fine prediction. PositionFineResiSize represents the amount of coding for PositionFineResi, for example, the amount of coding when PositionFineResi is binarized and arithmetic-coded.
[0567] In this way, by informing the decoder of the coding amount using PositionFineResiSize before PositionFineResi, the decoder can pre-read the bits related to PositionFineResi from the bitstream. For example, it is possible to perform arithmetic decoding related to PositionFineResi while simultaneously performing the subsequent decoding processing of PositionCoarseResi, thereby reducing decoding time.
[0568] PositionFineResi[c][j] represents the prediction residual of the j-th component of the position information after prediction encoding using the c-th fine prediction. PositionFineResi[c][j] can also be binarized using exponential Golomb coding or unary coding for variable-length encoding. Furthermore, encoding can be performed by assigning context to each bit and updating the probability of occurrence based on the frequency of 0 and 1 occurrences. This improves encoding efficiency.
[0569] Furthermore, the context used in the arithmetic encoding of each bit can be shared with the context used in PositionCoarseResi (described later). This method reduces the number of contexts. Additionally, the context used in the arithmetic encoding of each bit can be different from the context used in PositionCoarseResi (described later). This improves encoding efficiency when the tendency to predict residuals differs between fine and coarse predictions. Moreover, by using different contexts, PositionFineResi and PositionCoarseResi can be decoded in parallel.
[0570] PositionCoarseResiCount represents the number of prediction residuals for the position information after coarse prediction encoding. PositionCoarseResiSize represents the amount of coding for PositionCoarseResi, for example, the amount of coding when PositionCoarseResi is binarized and arithmetic encoded.
[0571] In this way, by informing the decoder of the coding amount using PositionCoarseResiSize before PositionCoarseResi, the decoder can pre-read the bits related to PositionCoarseResi from the bitstream. For example, it is possible to perform arithmetic decoding related to PositionCoarseResi while simultaneously performing decoding processing of the subsequent bitstream in parallel, thereby reducing decoding time.
[0572] PositionCoarseResi[c][j] represents the prediction residual of the j-th component of the position information after prediction encoding using the c-th coarse prediction. PositionCoarseResi[c][j] can also be binarized using exponential Golomb coding or unary coding for variable-length encoding. Furthermore, context can be assigned to each bit, and the probability of occurrence can be updated based on the frequency of 0 and 1 occurrences during encoding. This improves encoding efficiency.
[0573] Furthermore, the context used in the arithmetic encoding of each bit can be shared with the context used in PositionFineResi described above. This method reduces the number of contexts. Additionally, the context used for the arithmetic encoding of each bit can be different from the context used in PositionFineResi described above. This improves encoding efficiency when the tendency to predict residuals differs between fine and coarse prediction.
[0574] Furthermore, by using different contexts, it is also possible to decode PositionFineResi and PositionCoarseResi in parallel.
[0575] Additionally, PositionStartCount, PositionFineResiCount, PositionFineResiSize, PositionFineResi, PositionCoarseResiCount, PositionCoarseResiSize, and PositionCoarseResi can also be binarized using exponential Golomb coding or unary coding for variable-length encoding. Furthermore, context can be assigned to each bit, updating the probability of occurrence based on the frequency of 0 and 1 occurrences while encoding. This improves encoding efficiency. Alternatively, this information can be encoded with a fixed length, reducing processing overhead.
[0576] (Structure example) Figure 67 This is a diagram illustrating an example of the structure of the encoding device in the embodiment. Figure 68 This is a flowchart illustrating an example of an encoding method using an encoding device in an embodiment.
[0577] The encoding device 1530 includes a circuit 1531 and a memory 1532 connected to the circuit 1531. The encoding device 1530 can also perform operations via... Figure 54 and Figure 55 The processing of the description.
[0578] Circuit 1531 performs the following actions.
[0579] Circuit 1531 acquires multiple vertices contained in the 3D mesh (S1541). Circuit 1531 encodes the attribute information of one vertex among the multiple vertices using one of at least three prediction modes (S1542). The three prediction modes include a first prediction mode that does not perform prediction, a second prediction mode that performs prediction encoding based on one processed vertex, and a third prediction mode that performs prediction encoding based on two or more processed vertices. When the number of components of the attribute information is set to 1, in the encoding (S1542), the attribute information is encoded using a prediction mode different from the third prediction mode.
[0580] Therefore, by not applying the third prediction mode (predictive coding based on more than two processed vertices) when the number of components is set to 1, the computational load can be reduced. Furthermore, by eliminating redundant prediction calculations, the efficiency of the coding process can be improved.
[0581] For example, in the case where a vertex has connectivity with other vertices among multiple vertices, the number of components of the attribute information is set to 1.
[0582] Therefore, by ensuring connectivity when the number of components is 1, an efficient encoding method based on connectivity can be applied. This improves the compression efficiency of encoded data.
[0583] For example, attribute information is specific attribute information.
[0584] Therefore, by encoding specific attribute information, it is easy to clarify the encoded object and optimize it to improve encoding efficiency.
[0585] For example, specific attribute information is planar information representing the region to which a vertex belongs when a vertex is projected onto a two-dimensional plane.
[0586] Therefore, encoding based on projection onto a two-dimensional plane can efficiently represent three-dimensional information. Furthermore, encoding planar information as attribute information simplifies the data and improves encoding efficiency.
[0587] For example, the attribute information of one vertex is the same as the attribute information of other vertices.
[0588] Therefore, when multiple connected vertices share the same attribute information, redundant data can be omitted, thus reducing the amount of encoding.
[0589] For example, the prediction residual is 0 when a vertex is encoded using the second prediction mode.
[0590] Therefore, the prediction residual is 0 when the predictions based on the second prediction mode are completely consistent, which can further improve coding efficiency.
[0591] Figure 69 This is a diagram illustrating an example of the structure of the decoding device in an embodiment. Figure 70 This is a flowchart illustrating an example of a decoding method using a decoding apparatus according to an embodiment.
[0592] The decoding device 1540 includes a circuit 1541 and a memory 1542 connected to the circuit 1541. The decoding device 1540 can also perform operations in... Figure 56 The processing is explained in the text.
[0593] Circuit 1541 performs the following actions.
[0594] Circuit 1541 acquires multiple vertices contained in the 3D mesh (S1551). Circuit 1541 decodes the attribute information of one vertex among the multiple vertices using one of at least three prediction modes (S1552). The three prediction modes include a first prediction mode that does not perform prediction, a second prediction mode that performs prediction decoding based on one processed vertex, and a third prediction mode that performs prediction decoding based on two or more processed vertices. When the number of components of the attribute information is set to 1, in decoding (S1552), the attribute information is decoded using a prediction mode different from the third prediction mode.
[0595] Therefore, by not using the third prediction mode when the number of components is 1 during decoding, the efficiency of the decoding process can be improved. Furthermore, by omitting redundant processing, the computational load of the decoding process can be reduced.
[0596] For example, in the case where a vertex has connectivity with other vertices among multiple vertices, the number of components of the attribute information is set to 1.
[0597] Therefore, by determining the number of components to be 1 when connectivity is present, efficient connectivity-based decoding can be performed, reducing the processing load.
[0598] For example, attribute information is specific attribute information.
[0599] Therefore, by performing decoding processing on specific attribute information, the decoding object can be clearly identified, thus improving decoding efficiency.
[0600] For example, specific attribute information is planar information representing the region to which a vertex belongs when a vertex is projected onto a two-dimensional plane.
[0601] Therefore, by performing decoding processing based on two-dimensional planar information, it is possible to achieve efficient decoding of attribute information for three-dimensional meshes.
[0602] For example, the attribute information of one vertex is the same as the attribute information of other vertices.
[0603] Therefore, when using common attribute information, redundant decoding of attribute information for multiple vertices can be omitted, thus improving decoding efficiency.
[0604] For example, the prediction residual is 0 when decoding a vertex using the second prediction mode.
[0605] Therefore, it can reduce the load of decoding processing and improve the efficiency of data decoding.
[0606] Furthermore, the encoding and decoding apparatus of this disclosure can also be implemented by combining at least a portion of other methods. Moreover, the structure or processing of the encoding and decoding apparatus of this disclosure can also be implemented by combining a portion of the processing shown in any of the flowcharts according to this method, a portion of the structure of any apparatus, a portion of the syntax, etc., with other methods.
[0607] Furthermore, the processing of the decoding apparatus of this disclosure can also be performed in the encoding apparatus in the same way.
[0608] In addition, circuit 1531 of encoding device 1530 can also perform the following operation. This operation example is the second example.
[0609] Circuit 1531 obtains the multiple vertices contained in the three-dimensional mesh (S1541). Circuit 1531 performs arithmetic encoding on one of the multiple vertices using a first prediction mode (a second prediction mode in the above embodiment) or a second prediction mode (a third prediction mode in the above embodiment) (S1542). The first context used in the arithmetic encoding of the first prediction mode and the second context used in the arithmetic encoding of the second prediction mode are common.
[0610] Therefore, by using a common context under different prediction modes (first prediction mode and second prediction mode), efficient context management is achieved. This reduces the processing load involved in context initialization or updating, thereby improving the efficiency of encoding processing. Furthermore, by sharing the context, if the same data is favored in arithmetic encoding, encoding efficiency may be improved even with different parameters. By sharing the context, the use of redundant context memory can be reduced, further improving encoding efficiency.
[0611] For example, the first prediction mode is a prediction mode that performs predictive coding based on a single processed vertex. The second prediction mode is a prediction mode that performs predictive coding based on two or more processed vertices.
[0612] Therefore, by using different prediction modes, such as a first prediction mode (prediction based on one processed vertex) and a second prediction mode (prediction based on two or more processed vertices), it is possible to improve prediction accuracy according to the situation. In particular, prediction based on two or more processed vertices enables more accurate encoding. Furthermore, by sharing the context for the residuals of attribute information generated through different predictions, the coding efficiency of arithmetic coding can be improved while reducing the number of contexts.
[0613] For example, the arithmetic code of a vertex includes the arithmetic code of the vertex's position information.
[0614] Therefore, by encoding location information, the structure or vertex configuration of a 3D mesh can be efficiently encoded. The efficiency of arithmetic coding can be further improved by applying context sharing to the location information. In particular, efficient encoding is possible by sharing the context of the residuals of location information generated under different prediction modes.
[0615] For example, the arithmetic code of a vertex includes the arithmetic code of the vertex's attribute information.
[0616] Therefore, by including the encoding of attribute information (e.g., color, normal vectors, etc.), the rendering or display quality of 3D meshes can be improved. Encoding efficiency can be improved by also applying a common context to the attribute information. Furthermore, encoding efficiency can be further improved by sharing the context for the residuals of attribute information generated from different predictions.
[0617] For example, multiple vertices are contained in a group based on a 3D mesh. That is, multiple vertices are contained in a group of multiple vertices that make up the same 3D mesh, or a group of multiple vertices that make up the same submesh.
[0618] Therefore, by using a shared context when encoding vertices contained in the same group, the prediction accuracy of associated vertices can be improved. Furthermore, by sharing the context at the grid and subgrid levels, the number of contexts can be reduced, further improving encoding efficiency.
[0619] In addition, circuit 1541 of decoding device 1540 can also perform the following operation. This operation example is the second example.
[0620] Circuit 1541 obtains the multiple vertices contained in the three-dimensional mesh (S1551). Circuit 1541 performs arithmetic decoding on one of the multiple vertices using a first prediction mode (a second prediction mode in the above embodiment) or a second prediction mode (a third prediction mode in the above embodiment) (S1552). The first context used in the arithmetic decoding of the first prediction mode and the second context used in the arithmetic decoding of the second prediction mode are common.
[0621] Therefore, by using a common context across different prediction modes, decoding efficiency can be improved. By sharing the context, memory usage and processing load during arithmetic decoding can be reduced. Furthermore, by using a common context, decoding efficiency can be improved even when different parameters tend to be similar.
[0622] For example, the first prediction mode is a prediction mode that performs prediction decoding based on a single processed vertex. The second prediction mode is a prediction mode that performs prediction decoding based on two or more processed vertices.
[0623] Therefore, by using a first prediction mode (prediction decoding based on a single processed vertex) and a second prediction mode (prediction decoding based on two or more processed vertices), efficient decoding corresponding to the prediction method used during encoding can be achieved. Furthermore, by corresponding to multiple prediction methods, the accuracy of the decoding process can be improved. By sharing the context, the computational load and memory consumption in the decoding process can be reduced.
[0624] For example, the arithmetic decoding of a vertex includes the arithmetic decoding of the vertex's position information.
[0625] Therefore, corresponding to the case where location information is encoded during encoding, location information can be efficiently recovered during decoding. Decoding efficiency can be improved by applying a common context to the location information. Furthermore, the processing load can be reduced by using a common context in different prediction modes of the location information.
[0626] For example, the arithmetic decoding of a vertex includes the arithmetic decoding of the vertex's attribute information.
[0627] Therefore, corresponding to the case where attribute information is encoded during encoding, attribute information can be efficiently recovered during decoding. Decoding efficiency can be improved by applying a common context to the attribute information. Furthermore, efficient decoding can be achieved by commonalizing the context of the residuals of attribute information generated under different prediction modes.
[0628] For example, multiple vertices are contained in a group based on a 3D mesh. That is, multiple vertices are contained in a group of multiple vertices that make up the same 3D mesh, or a group of multiple vertices that make up the same submesh.
[0629] Therefore, by using a shared context on a grid or subgrid basis, the processing efficiency during decoding can be improved. This is especially true when processing multiple vertices as a group, as it reduces redundant context and enables efficient decoding.
[0630] Figure 71 This is a flowchart illustrating another example of the calculation process for the predicted value of the third prediction mode in the implementation method. That is, it is... Figure 55 A variation of the flowchart.
[0631] The encoding device sets the count to 0 (S1561).
[0632] The encoding device determines whether count is less than fineMaxcount (S1562). If count is less than fineMaxcount ("Yes" in S1562), the encoding device proceeds to step S1563; otherwise ("No" in S1562), it proceeds to step S1565. Furthermore, the encoding device can disable fine prediction by setting fineMaxcount to 0. Additionally, the encoding device can append the set value of fineMaxcount to the bitstream. Furthermore, if the value of fineMaxcount decoded from the bitstream is 0, the decoding device can also decode the attribute information without applying fine prediction.
[0633] The encoding device calculates the count-th fine prediction value with decimal precision (S1563).
[0634] The encoding device increments count by 1, updates count (S1564), and returns to step S1562.
[0635] The encoding device determines whether count is greater than 0 (S1565). If count is greater than 0 ("Yes" in S1565), the encoding device proceeds to step S1566; otherwise ("No" in S1565), it proceeds to step S1568.
[0636] The encoding device calculates the average fine prediction value with decimal precision (S1566). That is, the encoding device calculates the average of one or more fine prediction values as the average fine prediction value. Furthermore, the one or more fine prediction values are the count of fine prediction values obtained in steps S1561 to S1564. Alternatively, if there is only one fine prediction value, a single fine prediction value can be calculated as the average fine prediction value. In addition, in S1566, it is acceptable to integrate two or more fine prediction values to calculate a single fine prediction value; it is not limited to calculating the average fine prediction value.
[0637] The encoding device adjusts the number of bits to reduce the number of bits in the fractional part of the average fine prediction value (integrated value) (S1567). For example, the encoding device adjusts the number of bits in the fractional part to 0. That is, the encoding device performs integer precision (integerization) on the average fine prediction value (integrated value). Additionally, the encoding device may perform rounding operations such as rounding when adjusting the number of bits.
[0638] The encoding device outputs count (S1568).
[0639] Next, an example of using information from three-dimensional space to derive detailed prediction values in the UV coordinates of a two-dimensional plane will be explained.
[0640] Figure 72This is a diagram illustrating an example of a vector in three-dimensional space when a detailed prediction is made in the implementation method. Figure 73 This is a diagram illustrating an example of a vector on a two-dimensional plane in the case of detailed prediction of an implementation method.
[0641] like Figure 72 As shown, consider two encoded 3D points G(next) and G(prev) connected to the 3D point G(curr). Additionally, as... Figure 73 As shown, the points projected onto the UV coordinates from these 3D points G(curr), G(next), and G(prev) are denoted as UV(curr), UV(next), and UV(prev), respectively. The following describes an example of calculating the predicted value (predUV) when performing fine prediction on the vector of UV(curr).
[0642] Here, the three-dimensional vectors corresponding to the three-dimensional points G(curr), G(next), and G(prev) are denoted as gCurr, gNext, and gPrev, respectively, and the two-dimensional vectors corresponding to the two-dimensional points UV(curr), UV(next), and UV(prev) are denoted as uvCurr, uvNext, and uvPrev, respectively. Furthermore, the vector from gNext to gPrev is denoted as gNgP, the vector from gNext to gCurr is denoted as gNgC, and the vector from UV(next) to UV(prev) is denoted as uvNuvP. Moreover, the inner product of gNgP and gNgC is denoted as dot(gNgP, gNgC), and the inner product of gNgP vectors is denoted as dot(gNgP, gNgP).
[0643] The projections of the 3D point G(curr) and the 2D point UV(curr) onto their respective opposite sides are designated as 3D point G(proj) and 2D point UV(proj). Vectors gProj and uvProj are calculated by assuming that the triangle formed by the 3D points G(curr), G(next), and G(prev) is similar to the triangle formed by the 2D points UV(curr), UV(next), and UV(prev). Specifically, vectors gProj and uvProj are represented as follows.
[0644] uvProj = uvNext + uvNuvP × (dot(gNgP, gNgC) / dot(gNgP, gNgP)) gProj = gNext + gNgP × (dot(gNgP, gNgC) / dot(gNgP, gNgP)) Furthermore, the inner product of the vectors from the projection point G(proj) to G(curr) is dot(gCurr - gProj, gCurr - gProj). Here, the vector orthogonal to uvNuvP is denoted as orthogonal(uvNuvP).
[0645] At this point, the distance between projection points G(proj) and G(curr) corresponds to the distance between projection points UV(proj) and UV(curr) under the assumption that their respective triangles are similar. Therefore, the vector uvProjuvCurr from projection point UV(proj) to UV(curr) can be represented as follows.
[0646] uvProjuvCurr = orthogonal(uvNuvP) sqrt(dot(gCurr – gProj, gCurr -gProj) / dot(gNgP, gNgP)) Where sqrt is the square root.
[0647] Using this result, the prediction vector predUV can be calculated as follows.
[0648] predUV = uvProj + uvProjuvCur Alternatively, the prediction vector can be used as predUV1, as shown below.
[0649] predUV1 = uvProj - uvProjuvCur In this way, by considering the prediction vector in the opposite direction to predUV, prediction accuracy can be improved and the coding amount can be reduced.
[0650] The above provides a specific example of fine prediction calculation. However, in cases where the range of vector components is extremely wide, setting a large number of bits in the fractional part to ensure computational precision may lead to overflow. On the other hand, reducing the number of bits in the fractional part to avoid overflow, or performing integer-precision calculations, may degrade computational precision.
[0651] To solve this problem, the number of bits in the fractional part can be adjusted based on the size of the dot product of vectors gNgP and gNgC, or the size of the absolute value of the dot product.
[0652] The device can also determine whether the absolute value of dot(gNgP, gNgC) exceeds the threshold TH and perform calculations corresponding to the determination result. Figure 74AThis is a diagram illustrating an example of calculating the predicted value when the absolute value of dot(gNgP, gNgC) in the implementation exceeds the threshold TH. Figure 74B This is a graph showing an example of calculating the predicted value when the absolute value of dot(gNgP, gNgC) in the implementation method is below the threshold TH.
[0653] As a specific example, when the absolute value of dot(gNgP, gNgC) exceeds the threshold TH = (1 << (9) In the case of 2))-1, such as Figure 74A As shown, the device can also perform integer operations without retaining the fractional part.
[0654] On the other hand, the magnitude of the absolute value of dot(gNgP, gNgC) is TH = (1 << (9 2)) Below -1, such as Figure 74B As shown, the device can also perform calculations in a manner that ensures a specified number of bits (e.g., 9 bits) for the fractional part. Therefore, when the vector is large, the calculation precision can be reduced. Conversely, when the vector is small, the calculation precision can be increased. Thus, by adjusting the number of bits in the fractional part, the device can expand the range of possible vector values.
[0655] The above describes a method for switching the number of bits in the fractional part to two stages. However, the device can also determine the number of bits in the fractional part based on the number of bits corresponding to the absolute value of dot(gNgP, gNgC). For example, the device can also determine the number of bits in the fractional part using a base-2 logarithm based on the absolute value of dot(gNgP, gNgC). This allows the device to adjust the number of bits in the fractional part more accurately. Furthermore, the number of bits in the fractional part can be set to a single fixed value. This reduces the processing load.
[0656] Furthermore, the device can also determine the number of bits in the fractional part based on values other than the absolute value of dot(gNgP, gNgC). For example, a method based on the magnitude of the vector itself can also be used. Thus, the device can reduce the processing load.
[0657] In the example above, the fractional part is set to 9 bits, but it is not limited to this setting. For example, the device can also set the number of bits in the fractional part based on the number of bits in the hardware registers and arithmetic unit. Alternatively, the number of bits can be set based on the performance of the software and system. Furthermore, the optimal number of bits can be set by considering the trade-off between precision and processing power. In this way, by setting an appropriate number of bits according to the execution environment, the device can improve computational precision and reduce the amount of coding.
[0658] Furthermore, the device can also append a threshold TH to the header of the bitstream. Thus, by encoding by appending the threshold TH used on the encoding device side to the header, the decoding device can obtain the threshold TH from the header and appropriately decode the bitstream using the same number of bits as the fractional part used in the encoding device.
[0659] Figure 75 This is a diagram illustrating another example of the calculation of predicted values in the implementation method.
[0660] The device can also adjust the number of bits in the fractional part to 8 bits after calculating the predicted value, for example, when calculating the average of the fine predicted values, so as... Figure 75 The device ensures precision in division operations. This adjustment can also be performed regardless of the absolute value of dot(gNgP, gNgC). Furthermore, the device can either transform the average of the fine predicted values to integer precision or ensure the number of bits in the fractional part remains consistent. In this case, the device can also perform rounding operations such as rounding. Rounding operations can be performed in the same direction for positive and negative numbers, or in different directions. Alternatively, depending on the situation, the device may choose not to perform rounding operations.
[0661] Figure 76 This is a flowchart illustrating an example of the calculation process for the predicted value in an implementation method.
[0662] The apparatus calculates the inner product dot(gNgP, gNgC) of vectors gNgP and gNgC (S1571).
[0663] The device determines whether the absolute value of the inner product dot(gNgP, gNgC) exceeds the threshold TH (S1572). If the absolute value of the inner product dot(gNgP, gNgC) exceeds the threshold TH ("Yes" in S1572), the device proceeds to step S1573; otherwise ("No" in S1572), it proceeds to step S1575.
[0664] The device calculates the predicted value (S1573). A specific example of the calculation of the predicted value is provided. Figure 72 and Figure 73 As explained in the text.
[0665] The device adjusts the fractional part of the predicted value to 8 bits (S1574). That is, the device converts the fractional part of the predicted value to 8 bits.
[0666] The device adjusts the fractional part of the predicted value to 9 bits (S1575). That is, the device converts the fractional part of the predicted value to 9 bits.
[0667] The device calculates the predicted value (S1576). A specific example of the calculation of the predicted value is provided. Figure 72 and Figure 73 As explained in the text.
[0668] The device adjusts the fractional part of the predicted value to 8 bits (S1577). That is, the device converts the fractional part of the predicted value to 8 bits.
[0669] When step S1574 or S1577 is executed, the calculation and processing of the predicted value ends.
[0670] Next, a specific example of calculating the predicted value is shown.
[0671] The device can set the inner product of vectors gNgP and gNgC as dot(gNgP, gNgC), and set their respective x, y, and z components as .x, .y, and .z, respectively, as follows.
[0672] dot(gNgP, gNgC) = gNgP.x gNgC.x + gNgP.y gNgC.y + gNgP.z gNgC.z Similarly, the device can set the inner product of vectors gNgP to dot(gNgP, gNgP) and perform the calculation as follows.
[0673] dot(gNgP, gNgP) = gNgP.x gNgP.x + gNgP.y gNgP.y + gNgP.z gNgP.z Next, the device sets the absolute value of the inner product dot(gNgP, gNgC) to |dot(gNgP, gNgC)|, and determines the number of bits shiftBits representing the fractional part of the bit shift as follows.
[0674] The device can use the bit shift amount shiftBits to calculate the vector up to the projection point as follows.
[0675] uvProj = (uvNext< <shiftBits)+(uvNuvP (dot(gNgP, gNgC)< <shiftBits)) / dot(gNgP, gNgP) gProj = (gNext< <shiftBits)+(gNgP (dot(gNgP, gNgC)< <shiftBits)) / dot(gNgP, gNgP) Then, the device sets the inner product of the vectors from the projection point G(proj) to the point G(curr) as dot(gCurr - gProj, gCurr - gProj), and calculates it as follows.
[0676] dot(gCurr – gProj, gCurr – gProj) = (gCurr.x< <shiftBits) - gProj.x) (gCurr.x< <shiftBits) - gProj.x) + (gCurr.y< <shiftBits) - gProj.y) (gCurr.y< <shiftBits) - gProj.y) + (gCurr.z< <shiftBits) - gProj.z) (gCurr.z< <shiftBits) - gProj.z) In addition, the device sets the vector orthogonal to the two-dimensional vector uvNuvP as orthogonal(uvNuvP), and calculates the vector uvProjuvCurr as follows.
[0677] uvProjuvCurr =((orthogonal(uvNuvP)< <shiftBits) sqrt(dot(gCurr - gProj, gCurr -gProj))) / sqrt(dot(gNgP,gNgP)<<(shiftBits 2)) Here, sqrt represents the square root. Alternatively, the device can also use an approximate integer value of the square root for sqrt. Thus, it can perform operations using integers.
[0678] The device can calculate the candidate predUV0 or predUV1 of the prediction vector as follows.
[0679] predUV0 = uvProj + uvProjuvCurr Furthermore, when uvProjuvCurr is oriented in the opposite direction, the following calculation can be performed.
[0680] predUV1 = uvProj - uvProjuvCurr If the device wants to set the number of bits in the fractional part to 8, the following processing can also be performed.
[0681] Through the above processing, the device can change the number of bits in the fractional part from shiftBits to 8 bits. Furthermore, this embodiment shows an example of changing to 8 bits, but it is not limited to this; the device can also change the number of bits in the fractional part to any number of n bits (n being an integer greater than or equal to 0).
[0682] Alternatively, the following processing can be implemented when the device changes the number of bits in the fractional part from shiftBits to N bits.
[0683] The device can change the number of bits in the fractional part from shiftBits to any N bits through this process.
[0684] Choose either predUV0 or predUV1 as a candidate prediction vector, and set the prediction vector predUV as follows.
[0685] (1) When predUV0 is selected: predUV = predUV0 (2) When predUV1 is selected: predUV = predUV1 Furthermore, when calculating predUV based on predUV0 or predUV1, for example, considering that the fractional part is 8 bits, certain offset values can be added or subtracted. An example is shown below.
[0686] predUV = (offset0<<8) + predUV0 predUV = (offset1<<8)-predUV1 This process adjusts the prediction vector that uses the offset.
[0687] Alternatively, several prediction vectors can be added together to calculate the average. For example, the prediction vector obtained by accumulating and adding count prediction vectors can be set as predUVcount, and then averaged by rounding.
[0688] In this process, the direction of rounding addition is adjusted by the sign of predUVcount. This improves both the accuracy of the predicted vector and the efficiency of the encoding.
[0689] Furthermore, in the example above, the direction of rounding addition is adjusted by the sign of predUVcount, but it is also possible to use a method of rounding in the same direction, or to use a method of not performing rounding addition.
[0690] In addition, the above calculation formula "predUV=(predUVcount / count+rounding)>>8" represents the following separate processing.
[0691] predUV.x = (predUVcount.x / count + rounding.x)>>8 predUV.y = (predUVcount.y / count + rounding.y)>>8 Therefore, each component should be processed appropriately.
[0692] Next, an example of using information from three-dimensional space to derive detailed prediction values in the UV coordinates will be explained.
[0693] Figure 77 This is a diagram illustrating an example of a vector in three-dimensional space in the case of making detailed predictions for a modified embodiment. Figure 78 This is a diagram illustrating an example of a vector on a two-dimensional plane in the case of detailed prediction of a modified embodiment.
[0694] like Figure 77 As shown, the two encoded 3D points connected to the 3D point G (Curr) are designated as G (Next) and G (Prev). Additionally, as... Figure 78 As shown, the points projected from these 3D points G(curr), G(next), and G(prev) onto the UV coordinates are designated as UV(Next) and UV(Prev), respectively. Additionally, the point projected from G(Curr) onto the UV coordinates is designated as UV(Curr). Here, an example of fine prediction of the vector for UV(Curr) is shown (derived from predUV).
[0695] Here, the three-dimensional vectors for the known three-dimensional points G(Curr), G(Next), and G(Prev) are respectively designated as three-dimensional vectors gCurr, gNext, and gPrev. Similarly, the two-dimensional vectors for the known two-dimensional points UV(Next) and UV(Prev) are designated as two-dimensional vectors uvNext and uvPrev. Furthermore, the vector from three-dimensional vector gNext to three-dimensional vector gPrev is designated as three-dimensional vector gNgP, the vector from three-dimensional vector gNext to three-dimensional vector gCurr is designated as three-dimensional vector gNgC, and the vector from two-dimensional vector uvNext to two-dimensional vector uvPrev is designated as two-dimensional vector uvNuvP.
[0696] Let the inner product of the three-dimensional vectors gNgP and gNgC be dot(gNgP, gNgC), and let the inner product of the three-dimensional vectors gNgP be dot(gNgP, gNgP). Here, let the projection points from the three-dimensional point G (Curr) and the two-dimensional point UV (Curr) to their respective opposite edges be the three-dimensional point G (Proj) and the two-dimensional point UV (Proj).
[0697] Assuming that the triangle formed by three-dimensional points G(Curr), G(Next), and G(Prev) is similar to the triangle formed by two-dimensional points UV(Curr), UV(Next), and UV(Prev), the vector gProj for three-dimensional point G(Proj) and the vector uvProj for two-dimensional point UV(Proj) can be represented by the inner product as follows.
[0698] uvProj = uvNext + uvNuvP (dot(gNgP,gNgC) / dot(gNgP,gNgP)) gProj = gNext + gNgP (dot(gNgP,gNgC) / dot(gNgP,gNgP)) Furthermore, the inner product of the vectors from the projection point 3D point G(Proj) to the 3D point G(Curr) is set as dot(gCurr - gProj, gCurr - gProj). If the vector orthogonal to the 2D vector uvNuvP is set as orthogonal(uvNuvP), similarly using the similarity of triangles, the vector from the projection point 2D point UV(Proj) to the 2D point UV(Curr) can be represented as follows.
[0699] uvProjuvCurr = orthogonal(uvNuvP) sqrt(dot(gCurr – gProj, gCurr -gProj) / dot(gNgP, gNgP)) Here, sqrt represents the square root.
[0700] Therefore, the two-dimensional vector predUV can be calculated, for example, as follows.
[0701] predUV = uvProj + uvProjuvCur Additionally, based on the direction of the two-dimensional vector uvProjuvCurr, the prediction vector can be set as predUV0 or its reverse predUV1, and the calculation can be performed as follows.
[0702] predUV0 = uvProj + uvProjuvCur predUV1 = uvProj - uvProjuvCur In this way, by considering predUV as a bidirectional prediction vector, prediction accuracy can be improved and the amount of coding can be reduced.
[0703] Figure 79 This is a diagram illustrating another example of the calculation of predicted values in the implementation method.
[0704] In this embodiment, a method for ensuring the fixed number of bits N of the fractional part in order to guarantee the precision of integer operations during the calculation of fine predictions in three-dimensional space will be described. For example, as Figure 79 As shown, the fractional part can also be set to N bits. N bits can be set to, for example, N=1, N=2, N=3, ..., N=5, N=6, N=7, N=8. Alternatively, it can be set to N=16 or N=0 (treated as an integer without a fractional part).
[0705] The number of bits N in the fractional part can also be changed depending on the specific processing requirements. For example, when calculating the square root, since the number of bits in the fractional part is halved, by doubling the number of bits in the fractional part to 2N bits before the operation, the fractional part of the result can be maintained at N bits.
[0706] Furthermore, to prevent overflow, the number of bits N in the fractional part can be shifted to the right to decrease it. Conversely, to prevent underflow, the number of bits N in the fractional part can be shifted to the left to increase it beforehand. Moreover, to improve the accuracy of operations such as integer division, the fractional part can be shifted to the left to increase the number of bits before the operation, and then shifted to the right to decrease the number of bits after the operation. In these cases, rounding can suppress calculation errors. This allows for handling a wide range of input values.
[0707] The number of bits N in the fractional part is not limited to the value set above. For example, it can be determined based on the number of bits in the hardware registers and arithmetic unit. Alternatively, it can be set based on the performance of the software and system, or the optimal number of bits can be determined by considering the trade-off with computational precision. In this way, by setting an appropriate number of bits according to the execution environment, computational precision can be improved and the amount of coding can be reduced.
[0708] Furthermore, the number of bits N in the fractional part can also be appended to the header of the bitstream. According to this method, by including the number of bits N used on the encoding device side in the header for encoding, the decoding device can obtain the number of bits N from the header. Thus, the decoding device can perform decoding processing using the same number of bits as the number of bits in the fractional part used in the encoding device, and can appropriately decode the bitstream.
[0709] Furthermore, the number of bits N in the fractional part can be switched by using different values depending on the standard profile or level. For example, in profile A, which targets high image quality, a larger value (e.g., N=16) can be used to improve computational precision, while in profile B, which targets low processing power, a smaller value (e.g., N=5) can be used to suppress the required number of bits. In this way, by specifying the number of bits N in the fractional part according to the standard profile or level, the decoding device can appropriately switch the number of bits N based on the profile or level information contained in the stream, and correctly decode the bitstream.
[0710] Furthermore, the number of bits N in the fractional part can be determined based on the input value and conditions. For example, in the calculation of the count-th fine prediction value, the number of bits N in the fractional part can be determined based on the number of bits in the absolute value of dot(gNgP, gNgC). More specifically, the number of bits N in the fractional part can be determined using the number of bits in dot(gNgP, gNgC) and the base-2 logarithm of the absolute value. This ensures an appropriate number of bits N in the fractional part that corresponds to the range of input values, thus improving computational accuracy.
[0711] Figure 80 This is a flowchart illustrating another example of the calculation process for the predicted value in the implementation method.
[0712] The device converts the fractional part of the predicted value to N bits (S1581). That is, the device calculates the predicted value with fractional precision.
[0713] The device calculates the inner product value and inner product ratio of the vectors (S1582).
[0714] The device will divide the fractional part into 2 N-bitization (S1583).
[0715] The device calculates the square root (S1584). Thus, the decimal part can also be 2 in step S1583 before calculating the square root. N-bitization.
[0716] The device performs N-bit conversion on the fractional part (S1585). Step S1585 can also be incorporated into step S1584. That is, the device can also perform N-bit conversion on the fractional part when calculating the square root.
[0717] The device calculates the predicted value (vector) (S1586).
[0718] As shown in the flowchart, since the number of bits in the fractional part is also halved when calculating the square root, by pre-setting the number of bits N in the fractional part to 2, the same precision as the number of bits N in the fractional part on the output side can be maintained.
[0719] In addition, as input to the square root calculation process, the number of bits can be kept to N, or the fractional part can be converted to N bits after the output of the square root calculation process.
[0720] Next, a specific example of calculating the predicted value is shown.
[0721] First, let the inner product of vectors gNgP and gNgC be dot(gNgP, gNgC), and let the notations representing their respective x, y, and z components be .x, .y, and .z, respectively. The following calculations can then be performed.
[0722] dot(gNgP, gNgC) = gNgP.x gNgC.x + gNgP.y gNgC.y + gNgP.z gNgC.z Similarly, the device can set the inner product of vectors gNgP to dot(gNgP, gNgP) and perform the calculation as follows.
[0723] dot(gNgP, gNgP) = gNgP.x gNgP.x + gNgP.y gNgP.y + gNgP.z gNgP.z Here, we'll use N=8 bits for the fractional part and shiftBits=N=8 for illustration. Alternatively, shiftBits could be set to 5, 6, 7, etc.
[0724] Next, the method for finding the vector to the projection point will be explained. Here, the " / " operator refers to integer division with the decimal point rounded down. The vector to the projection point is calculated using the following formula.
[0725] uvProj = (uvNext< <shiftBits)+ uvNuvP ((dot(gNgP, gNgC)< <shiftBits) / dot(gNgP, gNgP)) gProj = (gNext< <shiftBits)+ gNgP ((dot(gNgP, gNgC)< <shiftBits) / dot(gNgP,gNgP)) Next, the inner product of the vectors from the projection point G(Proj) to G(Curr) is set as dot(gCurr - gProj, gCurr - gProj), and the calculation is performed as follows.
[0726] dot(gCurr – gProj, gCurr – gProj) = (gCurr.x< <shiftBits) - gProj.x) (gCurr.x< <shiftBits) - gProj.x) + (gCurr.y< <shiftBits) - gProj.y) (gCurr.y< <shiftBits) - gProj.y) + (gCurr.z< <shiftBits) - gProj.z) (gCurr.z< <shiftBits) - gProj.z) Furthermore, the vector orthogonal to the two-dimensional vector uvNuvP can be set as orthogonal (uvNuvP), and the vector uvProjuvCurr can be represented as follows.
[0727] uvProjuvCurr = orthogonal(uvNuvP) sqrt(dot(gCurr – gProj, gCurr -gProj) / dot(gNgP, gNgP)) Here, sqrt() represents the square root processing.
[0728] Additionally, at this point, since the fractional part of dot(gCurr - gProj, gCurr - gProj) has 2N bits (2... Therefore, the input to sqrt() is also 2N bits (2^N bits for the fractional part). shiftBits).
[0729] Therefore, the candidate vector predUV0 or predUV1 is calculated as follows.
[0730] predUV0 = uvProj + uvProjuvCurr predUV1=uvProj - uvProjuvCurr (the case where uvProjuvCurr is facing the opposite direction) Either predUV0 or predUV1, obtained in this way, can be used as the prediction vector.
[0731] Choose either the prediction vector candidate predUV0 or predUV1, and set the prediction vector predUV as follows.
[0732] (1) When predUV0 is selected: predUV=predUV0 (2) When predUV1 is selected: predUV=predUV1 Furthermore, when calculating predUV based on predUV0 or predUV1, for example, considering that the decimal part is shiftBits, some offset values can be added or subtracted. An example is shown below.
[0733] predUV = (offset0 < <shiftBits)+ predUV0 predUV = (offset1 < <shiftBits)- predUV1 Alternatively, several prediction vectors can be added together to calculate the average. For example, the prediction vector obtained by accumulating and adding count prediction vectors can be set as predUVcount, and then averaged by rounding.
[0734] By setting different directions for rounding based on the sign of predUVcount, rounding errors can be reduced. Alternatively, rounding can be performed in the same direction, or rounding addition can be omitted altogether.
[0735] In addition, the above formula predUV=(predUVcount / count+rounding)>>shiftBits means that each x component and y component is processed separately.
[0736] predUV.x = (predUVcount.x / count + rounding.x)>>shiftBits predUV.y = (predUVcount.y / count + rounding.y)>>shiftBits Next, a specific example of square root calculation (sqrt) is shown. Let the input value for the square root calculation be 'a', as shown below, representing the square root of 'a' as 'root'.
[0737] [Mathematical Expression 1] The square root calculation can also be performed using Newton's method based on the following recursive formula.
[0738] [Mathematical Expression 2] After convergence, it becomes root=x n .
[0739] More specifically, the calculation is as follows.
[0740] [Mathematical Expression 3] Furthermore, the device can switch the number of iterations k in the recursive formula to use different values depending on the standard level or grade. For example, in the case of level A, which aims for high image quality, the device can use a larger value for the number of iterations k in the recursive formula to improve computational accuracy. Specifically, consider using a larger value such as k=5.
[0741] On the other hand, in the case of level B, where the goal is to reduce processing volume, the device can use a smaller value as the number of iterations k in the recursive formula in order to suppress processing volume. For example, consider using a smaller value such as k=2.
[0742] In this way, the device specifies the value of the iteration number k of the recursive formula according to the grade or level of the standard, and the decoding device appropriately switches the iteration number of the recursive formula based on the grade or level information contained in the stream, thereby correctly decoding the bit stream.
[0743] Additionally, in a certain level C, the device can also set the number of iterations k in the recursive formula to 1. In this case, the device can perform one iteration by right shifting the bits, thus eliminating the need for division and allowing for a level that reduces the processing load.
[0744] Alternatively, the device can append the recursive iteration count k to the header of the bitstream. Thus, the device (encoding unit side) can append the used iteration count k to the header for encoding. Then, the decoding unit obtains the iteration count k from the header and performs decoding using the same value as the iteration count used in the encoding unit, thereby enabling proper decoding of the bitstream.
[0745] In the processing of mathematical expression 3, "<<" means left shift, ">>" means right shift, " / " means discarding integer division below the decimal point, "++" means increment, and ">>=" means substituting the shifted value.
[0746] The initial value x0 of the recursive formula, as an approximation of the true value (i.e., the square root), can be calculated using the bit count value bits(a) of the input value a. Specifically, it can also be set as x0 = 2. ((bits(a)>>1)) For example, with an input value a = 8, the bit count becomes bits(a) = 4, and the initial value x0 can be set to x0 = 4. Conversely, with an input value a = 40, the bit count becomes bits(a) = 6, and the initial value x0 can be set to x0 = 8.
[0747] In this way, the initial value used as an approximation can be easily calculated without performing complex operations involving lookup tables and division. Furthermore, by using the approximation of the square root of the true value as the initial value, the number of iterations required until the recursion converges can be reduced, and a highly accurate square root can be obtained. Moreover, by reducing the number of iterations, the square root can be calculated at a high speed.
[0748] For example, by using an approximation of the square root of the true value as the initial value, a high-precision square root can be obtained even if the number of iterations of the recursive formula is set to 2, thus enabling low-processing quantization.
[0749] Additionally, the initial value x0 of the recursive formula can be set to a power of 2, for example, x0 = 2. ((bits(a)>>1)) This allows for right-shifting to perform division in the initial iteration of the recursive process. This reduces the number of computationally expensive division operations.
[0750] For example, if the number of iterations of the recursive formula is set to 2, the division in the first iteration can be performed by setting the initial value of the recursive formula to a power of 2 and then performing the division in the second iteration. This reduces the number of divisions by one.
[0751] Additionally, since the input value a has 2 With an N-bit fractional part, sufficient precision can be achieved in the division during the second iteration of the recursive process by directly performing integer division. Therefore, there is no need to separately ensure the fractional part, thus reducing the processing load.
[0752] Furthermore, if a high-precision square root is not required, the initial iteration based on the recursive formula can be terminated, and the value representing the square root can be set as root=x1. On the other hand, if a higher-precision square root is required, k iterations can be performed, implementing (k-1) iterations based on the recursive formula from the second to the last, with the value representing the square root set as root=x1. n .
[0753] In this case, the subsequent iterations based on the recursive formula can also be repeated as follows.
[0754] [Mathematical Expression 4] Alternatively, the initial iteration process based on recursion, which uses shifting, can be omitted and included in subsequent iterations, with all iterations performed in the same way.
[0755] Next, another specific example of square root calculation (sqrt) is shown. Let the input value for the square root calculation be 'a', as shown below, representing the square root of 'a'.
[0756] [Mathematical Expression 5] The square root calculation can also be performed using Newton's method, which is based on a recursive formula, to find the reciprocal of the square root.
[0757] [Mathematical Expression 6] After convergence, it becomes root = 1 / x n .
[0758] More specifically, the calculation is performed as follows.
[0759] [Mathematical Expression 7] In the above processing, "<<" means left shift, ">>" means right shift, "++" means increment, and ">>=" means substitute the shifted value.
[0760] The initial value x0 of the recursive formula, used as an approximation of the true value (the reciprocal of the square root), can also be calculated using the bit count value bits(a) of the input value a. In this case, x0 can be calculated as x0 = 2. -((bits(a)>>1)) .
[0761] Therefore, the device can easily calculate approximate values without performing complex operations including lookup tables and division. Furthermore, by using an approximation of the reciprocal of the square root of the true value as the initial value, the device can calculate the square root with high accuracy while minimizing the number of iterations required until the recursive formula converges. Moreover, by minimizing the number of iterations, the device can calculate the square root at high speed.
[0762] For example, by using an approximation of the reciprocal of the square root of the truth as the initial value, the device can find the square root with high precision even when the number of iterations of the recursive formula is set to 2, and can achieve low-processing quantization.
[0763] Furthermore, since x0 is a power of 2, the device can perform multiplication by right shifting during the initial iteration of the recursive process. This avoids multiplying by a value with a fractional part, easily preventing overflow due to insufficient bits. Additionally, the device's implementation is simplified because it can be achieved through shift operations.
[0764] Next, another specific example of square root calculation (sqrt) will be explained in more detail, including its effects. Figures 81-85 This is a diagram used to illustrate another specific example of the square root calculation process (sqrt) of the implementation method.
[0765] like Figure 81 As shown, assuming the integer part of the input 'a' for square root calculation is 32 bits and the fractional part is N=8, that is, 8 If 2 = 16 bits, then the maximum total is 48 bits.
[0766] Furthermore, the recurrence relation in the first iteration is shown in mathematical formula 6. If the initial value x0 = 2... -((bits(a)>>1)) This requires multiplying the input a (which has more bits) with x0 (which contains more fractional parts). Furthermore, since x0 needs to be multiplied three times, a considerable number of bits are required.
[0767] Here, a fractional part (N×2 + (bits(a)>>1)) bits is added to the input 'a', which is (16 + (bits(a)>>1)) bits when N=8. At this point, the (N×2) part is a fixed fractional part independent of the number of bits in the input 'a', and the (bits(a)>>1) part is a variable fractional part dependent on the number of bits in the input 'a'. Figure 82 ).
[0768] As mentioned above, simply adding the fractional part would require more than 80 bits, which could lead to overflow in systems with 64-bit registers or arithmetic units. Therefore, taking advantage of the case where x0 is a negative power of 2, the addition (left shift) of the variable fractional part and the multiplication with x0 are treated as a right shift based on (bits(a) >> 1) bits. Here, by implementing the addition of the variable fractional part as a left shift operation and treating the reduction of the number of bits accompanying the multiplication with x0 as a right shift operation, the left shift and right shift are canceled out. In this way, by simultaneously performing the process of adding the variable fractional part by left shift and the right shift process accompanying the multiplication with x0, the increase in the number of bits can be prevented. Therefore, the device can implement x0 as follows: adding a fixed fractional part (bits(a) >> 1) to input a by left shift operation Figure 83 This process avoids overflow in 64-bit systems while adding the necessary and sufficient fractional part for square root calculation.
[0769] Next, in the recursive formula of the first iteration, the device can perform the multiplication by x0 by right shifting (bits(a) >> 1). This is represented as Figure 84 and Figure 85 As shown.
[0770] Through this process, even after the first iteration is completed, the precision of x1 can be maintained at 16 bits.
[0771] Next, a more detailed explanation will be given regarding the second iteration of the square root calculation (sqrt), including its effects. Figures 86-93 This is a diagram used to illustrate another specific example of the square root calculation process (sqrt) of the implementation method.
[0772] First, if the operation a·x1 is performed, the result is as follows: Figure 86 This is shown as 64 bits. However, if a·x1 is further implemented later... 2 Operations involving this may overflow. Therefore, as... Figure 87 The diagram shows the process of discarding (N×2=16) bits via a right shift operation, followed by the execution of a·x1. 2 Operations ( Figure 88 This prevents overflow.
[0773] Next, if a·x1 is implemented 2 The operation then becomes 64 bits again. However, if x1×(3-a·x1) is further performed afterwards... 2 Operations involving ) may overflow. Therefore, as Figure 89 As shown, after discarding (bits(a)>>1) bits through a right shift operation, x1×(3-a·x1) is performed. 2 Operations of () Figure 90 Furthermore, since N×2=16 bits are discarded first through a right shift operation, only (bits(a)>>1) bits are discarded through the right shift operation in this process. Thus, overflow can be prevented and accurate calculations can be performed without excessively reducing the number of bits.
[0774] Furthermore, if x1×(3-a·x1) is implemented... 2 The operation results in a 56-bit value. Finally, to calculate the square root *root* from the reciprocal of the square root multiplied by 2, multiplying it by the input *a* (48 bits) could lead to overflow. Therefore, as... Figure 91 As shown, a right shift operation is needed to discard (16 + (bits(a) >> 1)) bits. Additionally, in this process, when calculating (x1 / 2) × (3 - a·x1)... 2 In the case of ), (16 + (bits(a)>>1) + 1) bits are discarded by right shifting.
[0775] If the second iteration ends and the reciprocal x2 of the convergent square root is found, then to calculate the square root root, as follows: Figure 92 As shown, the reciprocal x2 is multiplied by the input a.
[0776] Finally, as Figure 93 As shown, by right-shifting to discard the fractional part (16 + (bits(a) >> 1)) bits added in the square root processing, the final square root output root can be obtained.
[0777] By performing this process, square roots can be calculated without using computationally expensive division. Furthermore, even with input values of a wide range (a), square roots can be calculated without overflow within a register and arithmetic unit with a finite number of bits.
[0778] Furthermore, in this description, N=8 is shown as an example of adding 16 bits to the fixed fractional part, but N=7, N=6, N=5, etc. are also possible. The fixed fractional part can be 14 bits, 12 bits, 10 bits, etc., or other bit numbers. Additionally, for systems with registers or arithmetic units exceeding 64 bits, such as 128 bits, N=9 (18 bits for the fixed fractional part) or higher is also possible. The variable fractional part is not limited to (bits(a) >> 1) bits and can also be other bit numbers.
[0779] Furthermore, in cases where the number of bits in the input 'a' is small, and where repeated multiplication will not overflow, a right shift operation can be omitted from each multiplication, and the shift operation can be performed only at the end of the processing, thereby reducing the number of shift operations. In this case, the fractional part added during the left shift can be reduced or omitted entirely.
[0780] Furthermore, the lower-order bits of input a, the fractional part of the original input a, can also be used as part or the whole of the fixed fractional part as shown in other specific examples of square root calculation (sqrt). Thus, even registers and arithmetic units with a small number of bits can perform the calculation, reducing the number of shift operations.
[0781] Furthermore, if a high-precision square root cannot be obtained, the process can end after the initial iteration based on the recursive formula, and the square root can be set as root = (a·x1) >> (16 + (bits(a) >> 1)). Thus, the square root can be obtained without additional multiplication, reducing the processing workload.
[0782] Conversely, to obtain a more precise square root, it can be set to perform k iterations, implementing (k-1) iterations based on the recursive formula after the second iteration, with the final square root set as root = (a·xn) >> (16 + (bits(a) >> 1)). In this case, a more precise square root can be obtained by repeating the second iteration based on the recursive formula in other specific examples of square root calculation (sqrt) as follows.
[0783] [Mathematical Expression 8] Furthermore, the number of iterations k in the recursive formula can be switched to different values depending on the standard level or grade. For example, in the case of level A, which aims for high image quality, a larger value can be used as the number of iterations in the recursive formula to improve computational accuracy. Specifically, consider using a larger value such as k=5.
[0784] On the other hand, in the case of tier B, where low processing power is the goal, the device can also use a smaller value as the number of iterations in the recursive formula to suppress processing power. For example, consider using a small value such as k=2. In this way, by specifying the number of iterations in the recursive formula according to the standard tier or level, the decoding device can select an appropriate number of iterations based on the tier or level information contained in the bitstream, thereby correctly decoding the bitstream.
[0785] Furthermore, in a certain level C, the number of iterations k=1 can also be set. In this case, one iteration can be performed by right shifting, reducing the processing load without multiplication.
[0786] Furthermore, the iteration count k of the recursive formula can also be appended to the header of the bitstream. Thus, by appending the iteration count k used on the encoding side to the header for encoding, the decoding device can obtain the iteration count k from the header. And, the decoding device can appropriately decode the bitstream by performing decoding processing using the same value of the iteration count used in the encoding device.
[0787] Furthermore, the specific examples of square root calculation processing (sqrt) described in this embodiment, as well as other specific examples, are not only applicable to the calculation of fine prediction values, but also to other processes that require square root calculation. In this case, as shown in these square root calculation examples, by setting the initial value of the recursive formula to a power of 2, division or multiplication in the initial iteration of the recursive formula can be implemented through shift operations, thereby reducing the processing load.
[0788] Figure 94 This is a diagram illustrating another example of the structure of the encoding device in the embodiment. Figure 95 This is a flowchart illustrating another example of an encoding method using an encoding device in an implementation scheme.
[0789] Encoding device 1550 includes circuitry 1551 and memory 1552 connected to circuitry 1551. Encoding device 1550 can also perform operations in... Figures 71-93 The processing is explained in the text.
[0790] Circuit 1551 performs the following actions.
[0791] Circuit 1551 obtains multiple vertices contained in the three-dimensional mesh (S1561). Circuit 1551 performs predictive coding on one of the multiple vertices (S1562). In predictive coding, a first predicted value of a vertex is calculated with fractional precision, the number of bits of the fractional part of the first predicted value is reduced to calculate a second predicted value, and the second predicted value is used to perform predictive coding on a vertex.
[0792] Therefore, by reducing the number of bits in the fractional part of the predicted value calculated with fractional precision, the amount of data can be reduced while controlling the precision of the operation. By reducing the number of bits, the amount of processing required for encoding is reduced, thus improving processing efficiency.
[0793] For example, in the calculation of the first predicted value, two or more third predicted values are calculated with decimal precision based on each of the two or more vertices that have been processed, and the first predicted value is calculated by integrating the two or more third predicted values.
[0794] Therefore, by integrating multiple predicted values, a highly accurate predicted value can be obtained. This reduces prediction error and improves coding efficiency. Furthermore, integrating multiple predicted values allows for greater flexibility in selecting appropriate predicted values.
[0795] For example, in the calculation of the first predicted value, two or more third predicted values are integrated with decimal precision.
[0796] Therefore, by performing integration processing with decimal precision followed by integerization, encoding processing can be performed while maintaining the precision of the integration process. By performing high-precision integration followed by integerization, the increase in prediction error can be suppressed.
[0797] For example, two or more third predicted values and the first predicted value have a specified number of fractional parts. In the calculation of the second predicted value, the second predicted value is calculated by reducing the specified number of fractional parts of the first predicted value.
[0798] Therefore, by reducing the fractional part by a specified number of bits, integerization can be performed easily. Reducing the number of bits reduces the amount of encoding and improves the efficiency of computation.
[0799] For example, the first predicted value is the average of two or more third predicted values.
[0800] Therefore, by averaging multiple predictions, the overall prediction error can be reduced. Through averaging, noise components and prediction errors can potentially be canceled out, thus improving coding accuracy.
[0801] For example, in predictive coding, a specified number of bits are not reduced before the calculation of the second predicted value.
[0802] Therefore, by not reducing the number of bits before integerization, the accuracy of the predicted value can be maintained. Since this does not cause a decrease in precision during computation, high-precision encoding can be achieved.
[0803] For example, in predictive coding of a vertex that uses a second predicted value, the second predicted value is rounded to calculate a fourth predicted value, and the fourth predicted value is used to predictively code the vertex.
[0804] Therefore, by performing rounding, errors below the decimal point can be suppressed. This improves the precision of the encoded data and minimizes errors when reducing the number of bits.
[0805] For example, in rounding, the second predicted value is rounded in different directions depending on whether it is positive or negative.
[0806] Therefore, by switching the rounding direction based on whether the predicted value is positive or negative, coding accuracy can be improved. In particular, optimal rounding processing corresponding to the magnitude and sign of the predicted value can be achieved.
[0807] Figure 96 This is a diagram illustrating another example of the structure of the decoding device in the embodiment. Figure 97 This is a flowchart illustrating another example of a decoding method using a decoding apparatus according to an embodiment.
[0808] The decoding device 1560 includes circuitry 1561 and a memory 1562 connected to circuitry 1561. The decoding device 1560 can also perform operations in... Figures 71-93 The processing is explained in the text.
[0809] Circuit 1561 performs the following actions.
[0810] Circuit 1561 obtains multiple vertices contained in the three-dimensional mesh (S1571). Circuit 1561 performs prediction decoding on one of the multiple vertices (S1572). In the prediction decoding, a first prediction value of a vertex is calculated with fractional precision, the number of bits of the fractional part of the first prediction value is reduced to calculate a second prediction value, and the second prediction value is used to perform prediction decoding on a vertex.
[0811] Therefore, by reducing the number of bits in the fractional part of the predicted value calculated with fractional precision, the amount of data can be reduced while controlling the precision of the calculation. By reducing the number of bits, the amount of processing required for decoding is reduced, thus improving processing efficiency.
[0812] For example, in the calculation of the first predicted value, two or more third predicted values are calculated with decimal precision based on each of the two or more vertices that have been processed, and the first predicted value is calculated by integrating the two or more third predicted values.
[0813] Therefore, by integrating multiple predicted values, a highly accurate predicted value can be obtained. This reduces decoding errors and improves decoding accuracy.
[0814] For example, in the calculation of the first predicted value, two or more third predicted values are integrated with decimal precision.
[0815] Therefore, by integrating fractional precision, higher-precision predicted values can be calculated. Subsequent integerization enables efficient decoding.
[0816] For example, two or more third predicted values and the first predicted value have a specified number of fractional parts. In the calculation of the second predicted value, the second predicted value is calculated by reducing the specified number of fractional parts of the first predicted value.
[0817] Therefore, by reducing the number of bits in the fractional part and converting it to integers, the burden of decoding can be reduced, and high-precision predictions can be utilized.
[0818] For example, the first predicted value is the average of two or more third predicted values.
[0819] Therefore, by averaging multiple predicted values, prediction errors can be reduced. Averaging can suppress deviations in predicted values, achieving high-precision decoding.
[0820] For example, in predictive decoding, the prescribed number of bits is not reduced before the calculation of the second predicted value.
[0821] Therefore, no reduction in precision due to bit reduction occurs during the decoding process before integerization, thus maintaining high-precision decoding.
[0822] For example, in the prediction decoding of a vertex that uses the second prediction value, the second prediction value is rounded to calculate the fourth prediction value, and the fourth prediction value is used to perform prediction decoding on the vertex.
[0823] Therefore, by applying rounding, the error during bit reduction can be minimized. This results in highly accurate predictions during decoding.
[0824] For example, in rounding, the second predicted value is rounded in different directions depending on whether it is positive or negative.
[0825] Therefore, by adjusting the rounding direction based on whether the predicted value is positive or negative, the accuracy of the predicted value can be improved. Furthermore, it can suppress the generation of rounding errors.
[0826] Furthermore, not all of the constituent elements described herein are always necessary; it may be necessary to include only a portion of the constituent elements of the encoding and decoding apparatus of this disclosure.
[0827] Industrial applicability This disclosure is useful, for example, for encoding devices, decoding devices, transmitting devices, and receiving devices related to three-dimensional meshes, and can be applied to computer graphics systems and three-dimensional data display systems.
[0828] Explanation of reference numerals in the attached figures 100 encoding device Vertex Information Encoder (101, 121, 144) 102, 145 Connect information encoder Attribute Information Encoder 103, 122 104, 204, 1103 Preprocessors Post-processors 105, 205, and 2106 110 Three-dimensional data encoding system 111, 211 controllers 112, 212 Input / Output Processors 113 Three-dimensional data encoder 114 System Multiplexer 115 3D Data Generator 123 Metadata Encoder 124 Multiplexer 131 Vertex Image Generator 132 Attribute Image Generator 133 Metadata Generator 134 Image Encoder 141 Two-dimensional data encoder 142 Grid Data Encoder 143 Texture Encoder 148. Description of the encoder 151 and 251 circuits 152, 252 memory 200 Decoding Device Vertex information decoder 201, 221, 244 202, 245 Connection Information Decoder 203, 222 Attribute Information Decoder 210 Three-Dimensional Data Decoding System 213 3D Data Decoder 214 System Demultiplexer 215, 247 prompts 216 User Interface 223 Metadata Decoder 224 Demultiplexer 231 Vertex Information Generator 232 Attribute Information Generator 234 Image Decoder 241 Two-dimensional data decoder 242 Grid Data Decoder 243 Texture Decoder 246 Mesh Reconstructor 248 Describe the decoder 300 Network 310 External Connector 511 Volume Capturer 512 Projector 513 Basic Mesh Encoder 514 Displacement Encoder 515 Attribute Encoder 516 Other types of encoders 613 Basic Mesh Decoder 614 Displacement Decoder 615 Attribute Decoder 616 Other Types of Decoders 617 3D Reconstructor 1101 Input Grid 1102 Attribute Diagram 1104 Basic Mesh 1105 Displacement Data 1106 Compressor 1107 bitstream 1108 Metadata 1206 sub-segmenters 1207 Displacement Vector Calculator 1231 Demultiplexer 1232 switch 1233 Static Mesh Decoder 1234 Grid Buffer 1235 Motion Decoder 1236 Basic Mesh Reconstructor 1237 Inverse Quantizer 1238, 1243 video decoders 1239 Image Unpacker 1240 Inverse Quantizer 1241 Inverse Wavelet Transform 1242 Reconstructor 1244 Color Converter 1251 Decoded base mesh 1252 sub-segmenters 1253 Sub-divided Mesh 1254 Decoded displacement data 1255 displacement device 1256 Decoded 3D Mesh 1301 grid frames 1302 Basic Grid Frames 1303 Displacement Information 1304 bit stream A set of 1501 vertex coordinates 1502 A set of two-dimensional coordinates 1530 and 1550 encoding devices Circuits 1531, 1541, and 1551 1532, 1542, 1552 memory 1540 and 1560 decoding devices 2101 bitstream 2102 decompressor 2103 Basic Mesh 2104 Displacement Data 2105 Metadata 2106 Post-processor 2107 Output Mesh 2108 Attribute Diagram 2204 Sub-segmenter 2209 Reconfigurator 2301 Basic Grid Frame 2302 Basic Grid Frame 2303 Displacement Information 2304 grid frames
Claims
1. An encoding method, executed by an encoding device, wherein, Obtain multiple vertices contained in a 3D mesh. Perform predictive encoding on one of the plurality of vertices. In the predictive coding, The first predicted value of the vertex is calculated with decimal precision. The second predicted value is calculated by reducing the number of bits in the fractional part of the first predicted value. The second predicted value is used to predict and encode the vertex.
2. The encoding method according to claim 1, wherein, In the calculation of the first predicted value Calculate two or more third predicted values with decimal precision, based on each of the two or more vertices that have been processed. The first prediction value is calculated by integrating two or more third prediction values.
3. The encoding method according to claim 2, wherein, In the calculation of the first predicted value, the two or more third predicted values are integrated with decimal precision.
4. The encoding method according to claim 2, wherein, The two or more third predicted values and the first predicted value have a specified number of fractional parts. In the calculation of the second predicted value, the second predicted value is calculated by cutting off the predetermined number of bits from the fractional part of the first predicted value.
5. The encoding method according to any one of claims 2 to 4, wherein, The first predicted value is the average of the two or more third predicted values.
6. The encoding method according to claim 4, wherein, In the predictive encoding, the specified number of bits is not reduced before calculating the second predicted value.
7. The encoding method according to any one of claims 2 to 4, wherein, In the prediction encoding of the vertex that uses the second predicted value, The second predicted value is rounded to calculate the fourth predicted value. The fourth prediction value is used to predict and encode the vertex.
8. The encoding method according to claim 7, wherein, In the rounding process, the second predicted value is rounded in different directions depending on whether the second predicted value is positive or negative.
9. A decoding method, executed by a decoding device, wherein, Obtain multiple vertices contained in a 3D mesh. Perform prediction decoding on one of the multiple vertices. In the prediction decoding, The first predicted value of the vertex is calculated with decimal precision. The second predicted value is calculated by reducing the number of bits in the fractional part of the first predicted value. The second predicted value is used to predict and decode the vertex.
10. The decoding method according to claim 9, wherein, In the calculation of the first predicted value Calculate two or more third predicted values with decimal precision, based on each of the two or more vertices that have been processed. The first prediction value is calculated by integrating two or more third prediction values.
11. The decoding method according to claim 10, wherein, In the calculation of the first predicted value, the two or more third predicted values are integrated with decimal precision.
12. The decoding method according to claim 10, wherein, The two or more third predicted values and the first predicted value have a specified number of fractional parts. In the calculation of the second predicted value, the second predicted value is calculated by cutting off the predetermined number of bits from the fractional part of the first predicted value.
13. The decoding method according to any one of claims 10 to 12, wherein, The first predicted value is the average of the two or more third predicted values.
14. The decoding method according to claim 12, wherein, In the prediction decoding, the specified number of bits is not reduced before calculating the second predicted value.
15. The decoding method according to any one of claims 10 to 12, wherein, In the prediction decoding of the vertex using the second predicted value, The second predicted value is rounded to calculate the fourth predicted value. The fourth prediction value is used to predict and decode the vertex.
16. The decoding method according to claim 15, wherein, In the rounding process, the second predicted value is rounded in different directions depending on whether the second predicted value is positive or negative.
17. An encoding device, wherein, have: Circuits; and The memory is connected to the circuit. The circuit is in operation. Obtain multiple vertices contained in a 3D mesh. Perform predictive encoding on one of the plurality of vertices. In the predictive coding, The first predicted value of the vertex is calculated with decimal precision. The second predicted value is calculated by reducing the number of bits in the fractional part of the first predicted value. The second predicted value is used to predict and encode the vertex.
18. A decoding device, wherein, have: Circuits; and The memory is connected to the circuit. The circuit is in operation. Obtain multiple vertices contained in a 3D mesh. Perform prediction decoding on one of the multiple vertices. In the prediction decoding, The first predicted value of the vertex is calculated with decimal precision. The second predicted value is calculated by reducing the number of bits in the fractional part of the first predicted value. The second predicted value is used to predict and decode the vertex.
Citation Information
Patent Citations
Progressive three-dimensional mesh information coding / decoding method, and apparatus therefor
JP2006187015A
