3D data decoding device and 3D data encoding device

Abstract

To improve encoding efficiency in mesh displacement to provide encoding and decoding 3D data with high quality, in encoding and decoding 3D data using an arithmetic coding system.SOLUTION: In a 3D data decoding device that decodes mesh data or point group data, a mesh displacement decoding unit comprises: a header decoding unit that decodes header information from encoding data; an arithmetic decoding unit that arithmetically decodes mesh displacement from the encoding data; a context selection unit that selects a context in the arithmetic decoding unit; and a context initialization unit that sets an initial value of the context. The header decoding unit decodes, from the encoding data, offset that indicates the position of the encoding data.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] An embodiment of the present invention relates to a 3D data encoding device and a 3D data decoding device. [Background technology]

[0002] In order to efficiently transmit or record 3D data, there are 3D data encoding devices that convert the 3D data into a 2D image, encode it using a video encoding method, and generate encoded data, and 3D data decoding devices that decode the encoded data into a 2D image and reconstruct the 3D data.

[0003] Specific examples of 3D data encoding methods include MPEG-I's ISO / IEC 23090-5 V3C (Volumetric Video-based Coding) and V-PCC (Video-based Point Cloud Compression). V3C can encode and decode point clouds consisting of point positions and attribute information. Furthermore, it is also used for encoding and decoding multi-viewpoint images and mesh images by ISO / IEC 23090-12 (MPEG Immersive Video, MIV) and ISO / IEC 23090-29 (Video-based Dynamic Mesh Coding, V-DMC), which is currently being standardized. The latest draft document of the V-DMC method is disclosed in Non-Patent Document 1.

[0004] In these 3D data coding methods, the geometry and attributes that make up the 3D data are encoded and decoded as images using video coding methods such as H.265 / HEVC (High Efficiency Video Coding) and H.266 / VVC (Versatile Video Coding).

[0005] In the case of a point cloud, the geometry image is the depth to the projection plane, and the attribute image is the projection of the attributes onto the projection plane.

[0006] 3D data (mesh) such as that in Non-Patent Document 1 consists of a base mesh, mesh displacement, and texture mapping image. A vertex coding method such as Draco can be used to code the base mesh. The mesh displacement can be coded by coding the mesh displacement image, which is a two-dimensional representation of the mesh displacement, with a video codec, or by directly coding it using arithmetic coding. The texture mapping image is coded as an attribute image with a video codec. The video codec can be the above-mentioned HEVC or VVC. [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] WD 4.0 of V-DMC (MDS23075_WG07_N00680_clean), ISO / IEC JTC 1 / SC 29 / WG 7 N00680, July 2023 [Non-Patent Document 2] On Displacement Sub-Bitstream (m64392-v3), ISO / IEC JTC 1 / SC 29 / WG 7 m64392, July 2023 Summary of the Invention [Problem to be solved by the invention]

[0008] In the 3D data coding method in Non-Patent Document 1, mesh displacements (mesh displacement sequence, mesh displacement image) constituting 3D data (mesh) can be coded and decoded using an arithmetic coding method. In Non-Patent Document 2, mesh displacements are arithmetically coded so as to be initialized on a frame-by-frame basis, but even if initialization is performed, there is a problem that parallel decoding is not possible if the starting point is unknown.

[0009] An object of the present invention is to improve the coding efficiency of mesh displacement in the coding and decoding of 3D data using an arithmetic coding method, and to code and decode 3D data with high efficiency. [Means for solving the problem]

[0010] In order to solve the above problems, a 3D data decoding device according to one embodiment of the present invention is a 3D data decoding device that decodes mesh data or point cloud data, and includes a header decoding unit that decodes header information from the encoded data, an arithmetic decoding unit that arithmetically decodes mesh displacement from the encoded data, a context selection unit that selects a context in the arithmetic decoding unit, and a context initialization unit that sets an initial value of the context, and is characterized in that an offset indicating the position of the encoded data is decoded from the encoded data in the header decoding unit.

[0011] In order to solve the above problems, a 3D data encoding device according to one embodiment of the present invention is a 3D data encoding device that encodes mesh data or point cloud data, and includes a header encoding unit that encodes header information, an arithmetic encoding unit that arithmetically encodes mesh displacement, a context selection unit that selects a context in the arithmetic encoding, and a context initialization unit that sets an initial value of the context, and is characterized in that the header encoding unit encodes an offset that indicates the position of the encoded data. Effect of the Invention

[0012] According to one aspect of the present invention, the encoding efficiency of mesh displacement is improved, and 3D data can be encoded and decoded with high quality. [Brief description of the drawings]

[0013] [Figure 1] 1 is a schematic diagram showing the configuration of a 3D data transmission system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram showing a hierarchical structure of data in an encoded stream. [Diagram 3] FIG. 2 is a functional block diagram showing a schematic configuration of a 3D data decoding device 31. [Figure 4] FIG. 2 is a functional block diagram showing a configuration of a base mesh decoding unit 303. [Diagram 5] FIG. 2 is a functional block diagram showing the configuration of a mesh displacement decoding unit 305. [Figure 6] FIG. 2 is a functional block diagram showing the configuration of a mesh reconstruction unit 307. [Figure 7] 13 is an example of a syntax for transmitting coordinate transformation parameters and context initialization parameters at the sequence level (DSPS). [Figure 8] 13 is an example of a syntax for transmitting coordinate transformation parameters and context initialization parameters at the picture / frame level (DFPS). [Figure 9] FIG. 13 is a diagram for explaining the operation of the mesh reconstructing unit 307. [Figure 10] 1 is a functional block diagram showing a schematic configuration of a 3D data encoding device 11. FIG. [Figure 11] FIG. 2 is a functional block diagram showing a configuration of a base mesh encoding unit 103. [Figure 12] FIG. 2 is a functional block diagram showing a configuration of a mesh displacement encoding unit 107. [Figure 13] FIG. 2 is a functional block diagram showing the configuration of a mesh separation unit 115. [Figure 14] 11 is a diagram for explaining the operation of the mesh separating unit 115. FIG. [Figure 15] 1 is an example of a syntax for dimensionally parallel mesh displacement data. [Figure 16] 13 is an example of the syntax of a dimensionally parallel mesh displacement header (offset information). [Figure 17] 13 is an example of the syntax for mesh displacement data for LoD parallelism. [Figure 18] This is an example of the mesh displacement header (offset information) syntax for LoD parallelism. [Figure 19] 13 is an example of the syntax for mesh displacement data for dimensional LoD parallelism. [Figure 20] 13 is an example of the syntax of mesh displacement header (offset information) for 3-dimensional LoD parallelism. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0015] FIG. 1 is a schematic diagram showing the configuration of a 3D data transmission system 1 according to this embodiment.

[0016] The 3D data transmission system 1 is a system that transmits an encoded stream obtained by encoding 3D data to be encoded, decodes the transmitted encoded stream, and displays the 3D data. The 3D data transmission system 1 includes a 3D data encoding device 11, a network 21, a 3D data decoding device 31, and a 3D data display device 41.

[0017] 3D data T is input to the 3D data encoding device 11.

[0018] The network 21 transmits the encoded stream Te generated by the 3D data encoding device 11 to the 3D data decoding device 31. The network 21 is the Internet, a wide area network (WAN), a local area network (LAN), or a combination of these. The network 21 is not necessarily limited to a bidirectional communication network, and may be a unidirectional communication network that transmits broadcast waves such as terrestrial digital broadcasting and satellite broadcasting. Furthermore, the network 21 may be replaced by a storage medium on which the encoded stream Te is recorded, such as a DVD (Digital Versatile Disc: registered trademark) or a BD (Blu-ray Disc: registered trademark).

[0019] The 3D data decoding device 31 decodes each of the encoded streams Te transmitted by the network 21, and generates one or more decoded 3D data Td.

[0020] The 3D data display device 41 displays all or part of one or more pieces of decoded 3D data Td generated by the 3D data decoding device 31. The 3D data display device 41 includes a display device such as a liquid crystal display or an organic EL (Electro-luminescence) display. Display forms include stationary, mobile, HMD, and the like. Furthermore, when the 3D data decoding device 31 has high processing power, it displays high quality images, and when it has only lower processing power, it displays images that do not require high processing power or display power.

[0021] <operator> The operators used in this specification are listed below.

[0022] >> is a right bit shift, << is a left bit shift, & is a bitwise AND, | is a bitwise OR, |= is the OR assignment operator, and || indicates logical OR.

[0023] x?y:z is a ternary operator that takes y if x is true (non-zero) and z if x is false (zero). Let y..z denote the set of integers from y to z.

[0024] <Structure of the coding stream Te> Before going into a detailed description of the 3D data encoding device 11 and the 3D data decoding device 31 according to this embodiment, the data structure of the encoded stream Te generated by the 3D data encoding device 11 and decoded by the 3D data decoding device 31 will be described.

[0025] 2 is a diagram showing a hierarchical structure of data in an encoded stream Te. The encoded stream Te has the data structure of either a V3C sample stream or a V3C unit stream. The V3C sample stream includes a sample stream header and a V3C unit. The V3C unit stream includes a V3C unit.

[0026] A V3C unit contains a V3C unit header and a V3C unit payload. This is the Unit Type, which is an ID indicating the type, and takes on values ​​indicated by labels such as V3C_VPS, V3C_AD, V3C_AVD, V3C_GVD, and V3C_OVD.

[0027] If the Unit Type is V3C_VPS (Video Parameter Set), the V3C unit includes a V3C parameter set.

[0028] When the Unit Type is V3C_AD (Atlas Data), the V3C unit includes a VPS ID, an atlasID, a sample stream nal header, and multiple NAL units. The atlasID is an ID (Idenfication) and takes an integer value of 0 or more.

[0029] If Unit Type is V3C_AVD (Attribute Video Data), V3C The unit includes the VPS ID, atlasID, attrIdx, which is the ID of the attribute image, partIdx, which is the partition ID, mapIdx, which is the map ID, auxFlag, which indicates whether the data is auxiliary data, and the video stream. The video stream is data encoded by HEVC, VVC, etc. In V-DMC, the attribute data corresponds to the texture image.

[0030] When the Unit Type is V3C_GVD (Geometry Video Data), the V3C unit includes the VPS ID, atlasID, mapIdx, auxFlag, and video stream. Geometry data corresponds to mesh displacement in V-DMC.

[0031] If the Unit Type is V3C_OVD (Occupancy Video Data), the V3C unit includes the VPS ID, atlasID, and video stream.

[0032] If the Unit Type is V3C_MD (Mesh data), the V3C unit includes a VPS ID, an atlas ID, and multiple NAL units. In V-DMC, it corresponds to a base mesh.

[0033] A NAL unit contains a NALUnitType, a layerID, a TemporalID, and a RBSP (Raw byte sequence payload).

[0034] NAL units are identified by NALUnitType and include DSPS (Displacement Sequence Parameter Set), DFPS (Displacement Frame Parameter Set), Displ Layer (Displacement layer), SEI (Supplemental Enhancement Information), etc.

[0035] The Displ Layer includes a displ header and a displ data unit, and the displ header includes information such as boundary positions.

[0036] The SEI includes payloadType, which indicates the type of SEI, payloadSize, which indicates the size (number of bytes) of the SEI, and sei_payload, which is the data of the SEI.

[0037] (Configuration of 3D data decoding device according to the first embodiment) 3 is a functional block diagram showing a schematic configuration of a 3D data decoding device 31 according to the first embodiment. The 3D data decoding device 31 is composed of a demultiplexing unit 301, an atlas information decoding unit 302, a base mesh decoding unit 303, a mesh displacement decoding unit 305, a mesh reconstruction unit 307, an attribute decoding unit 306, and a color space conversion unit 308. The 3D data decoding device 31 inputs encoded data of 3D data, and outputs atlas information, meshes, and attribute images.

[0038] The demultiplexing unit 301 inputs encoded data multiplexed in a byte stream format, ISOBMFF (ISO Base Media File Format), or the like, and demultiplexes the data to generate an atlas information encoded stream (Atlas Data stream of V3C_AD, NAL unit), a base mesh encoded stream (Mesh Data stream of V3C_MD, NAL unit), a mesh displacement encoded stream (video stream of V3C_GVD, NAL unit), and a mesh information encoded stream (video stream of V3C_GVD, NAL unit). m) and output the attribute video stream (V3C_AVD video stream).

[0039] The atlas information decoding unit 302 receives the atlas information encoded stream output from the demultiplexing unit 301 and decodes the atlas information.

[0040] 3 decodes, from the encoded data, coordinate system transformation information displacementCoordinateSystem (asps_vdmc_ext_displacement_coordinate_system, afps_vdmc_ext_displacement_coordinate_system) indicating the coordinate system. Note that a separate gating flag may be provided, and each piece of coordinate system transformation information may be decoded only when the gating flag is 1. The gating flag may be, for example, afps_vdmc_ext_displacement_coordinate_system_enable_flag.

[0041] The base mesh decoder 303 decodes the base mesh coded stream that has been coded using vertex coding (a 3D data compression coding method, for example, Draco), and outputs a base mesh. The base mesh will be described later.

[0042] The mesh displacement decoding unit 305 decodes the mesh displacement coded stream and outputs the mesh displacement.

[0043] A mesh reconstructing unit 307 receives the base mesh and the mesh displacement, and reconstructs the mesh in the 3D space.

[0044] The attribute decoding unit 306 decodes the attribute video stream encoded by VVC, HEVC, or the like, and outputs an attribute image. The attribute image may be a texture image expanded on the UV axis (a texture mapping image converted by the UV atlas method) in YCbCr format. The type of codec used for encoding is indicated by ptl_profile_codec_group_idc obtained by decoding the V3C parameter set of the encoded data. It may also be indicated by the Four CC code indicated by ai_geometry_codec_id[atlasID] of the V3C parameter set. ai_geometry_codec_id[atlasID] indicates an index in the atlas ID corresponding to the codec ID of the decoder used to decode the attribute video stream.

[0045] A color space conversion unit 308 converts the color space of the attribute image from the YCbCr format to the RGB format. Note that an attribute video stream coded in the RGB format may be decoded and color space conversion may be omitted.

[0046] (Decoding the base mesh) FIG. 4 is a functional block diagram showing the configuration of the base mesh decoding unit 303. The base mesh decoding unit 303 is composed of a mesh decoding unit 3031, a motion information decoding unit 3032, a mesh motion compensation unit 3033, a reference mesh memory 3034, a switch 3035, and a switch 3036. The base mesh decoding unit 303 may be configured to include a base mesh inverse quantization unit (not shown) before the output of the base mesh. When the base mesh to be decoded is coded (intra-coded) without referring to other base meshes (for example, base meshes that have already been coded and decoded), the switches 3035 and 3036 are connected to the side that does not perform motion compensation. On the other hand, when the base mesh to be decoded is coded (inter-coded) with reference to other base meshes, the switches 3035 and 3036 are connected to the side that performs motion compensation. When motion compensation is performed, the target vertex coordinates are derived by referring to the vertex coordinates and motion information that have already been decoded.

[0047] The mesh decoding unit 3031 decodes the intra-coded base mesh coded stream and outputs the base mesh. As the coding method, Draco, Edge Breaker, etc. are used.

[0048] The motion information decoding unit 3032 decodes the inter-coded base mesh coded stream and outputs motion information for each vertex of a reference mesh, which will be described later. Entropy coding, such as arithmetic coding, is used as the coding method.

[0049] The mesh motion compensation unit 3033 performs motion compensation on each vertex of the reference mesh input from the reference mesh memory 3034 based on the motion information, and outputs a motion-compensated mesh.

[0050] The reference mesh memory 3034 is a memory that holds decoded meshes for reference in subsequent decoding processes.

[0051] (Mesh displacement decoding) 5 is a functional block diagram showing the configuration of the mesh displacement decoding unit 305. The mesh displacement decoding unit 305 is made up of a header decoding unit 3058, a CABAC decoding unit (arithmetic decoding unit 3051, multi-value conversion unit 3052, context selection unit 3056, and context initialization unit 3057), an inverse quantization unit 3053, an inverse transformation unit 3054, and a coordinate system transformation unit 3055.

[0052] (Context-Adaptive Variable Length Coding) The header decoder 3058 uses a decoding method that uses contexts, called Context-based Adaptive Variable Length Coding (CAVLC). In CAVLC, the surrounding conditions of the coding target, called context, are adaptively used based on statistics of the most recently coded transform coefficients to select an appropriate variable length code table from among multiple code tables.

[0053] (Context-Adaptive Binary Arithmetic Coding) The arithmetic decoding unit 3051, the multi-value conversion unit 3052, the context selection unit 3056, and the context initialization unit 3057 use a decoding method using a context called Context-Adaptive Binary Arithmetic Coding (CABAC). In CABAC, a binary string consisting of 0 and 1 is encoded and decoded bit by bit using a state variable called a context (CABAC state). All CABAC states are initialized at the beginning of a segment (subset). The CABAC decoding unit decodes each bit of a binary string (Bin String) corresponding to a syntax element. When a context is used, a context index ctxInc is derived for each bit of the syntax element, the bit is decoded using the context, and the CABAC state of the context is updated. Bits that do not use a context are decoded with equal probability (EP, bypass), and updating of the index ctxIdx that specifies the context and the identified context is omitted. A context is a variable (memory area) for holding the probability (state) of CABAC, and is identified by the value of ctxIdx (0, 1, 2, ...). When 0 and 1 are always equally probable, in other words, when they are 0.5, 0.5, this is called EP (Equal Probability) or bypass. In this case, no context is used because there is no need to hold a state for a specific syntax element. It is also possible to use a static context where the probability is fixed at 0.5 and does not need to be updated. In this sense, it may be called static rather than bypass. An integer value such as 128 may be used to indicate a probability of 0.5.

[0054] The process of decoding one bit without using a context (bypassing) may use the following pseudo code.

[0055] rangeTimesProb = IvlRange >> 1 binVal = ( rangeTimesProb <= ( IvlCode - IvlLow ) ) if (binVal == 0) IvlRange = rangeTimesProb else { IvlLow += rangeTimesProb IvlRange -= rangeTimesProb } The process of decoding one bit using a context may use the following pseudo code: Here, prob0 is a variable indicating the probability of the context.

[0056] rangeTimesProb = IvlRange * prob0 >> 16 binVal = ( rangeTimesProb <= ( IvlCode - IvlLow ) ) if (binVal == 0) IvlRange = rangeTimesProb else { IvlLow += rangeTimesProb IvlRange -= rangeTimesProb } The CABAC decoder may decode serially. It may decode in 3D parallel for each dimension using offset information. Dim#1 and Dim#2 may be decoded serially and Dim#0 may be decoded in parallel. It may also decode in parallel for each LoD. It may decode LoD#0, LoD#1, ..., LoD#(b-1) in serial with an arbitrary value b, and decode LoD#b, ..., LoD#(displ_lod_count-1) in parallel. It may also decode in parallel for each dimension and LoD, or decode LoD#0, Lod#1, ..., LoD#(b-1) of Dim#1 and LoD#0, Lod#1, ..., LoD#(b-1) of Dim#2 in serial and decode LoD#0, Lod#1, ..., LoD#(b-1) of Dim#0 in parallel. Dim indicates the dimension of the mesh displacement. LoD may be defined as the level of detail of the mesh displacement.

[0057] (Coordinate System) The following two types of coordinate systems are used for mesh displacement (3D vector).

[0058] Cartesian coordinate system (canonical): A rectangular coordinate system that is commonly defined throughout the entire 3D space. (X,Y,Z) coordinate system. A rectangular coordinate system whose direction does not change at the same time (within the same frame, within the same tile).

[0059] Local coordinate system (local): A Cartesian coordinate system defined for each region or vertex in 3D space. A Cartesian coordinate system whose direction can change at the same time (within the same frame, within the same tile). normal A coordinate system with axes (D), tangent (U), and bi-tangent (V). That is, it is an orthogonal coordinate system consisting of the first axis (D) indicated by the normal vector n_vec at a vertex (or the face that contains the vertex), and the second axis (U) and third axis (V) indicated by two tangent vectors t_vec and b_vec that are perpendicular to the normal vector n_vec. n_vec, t_vec, and b_vec are three-dimensional vectors. The (D, U, V) coordinate system may also be called the (n, t, b) coordinate system.

[0060] (Decoding and deriving sequence-level control parameters) Here, the control parameters used in the mesh displacement decoding unit 305 will be described.

[0061] Figure 7 shows an example of the syntax for transmitting coordinate system transformation parameters in sequence-level DSPS. DSPS (Displacement Sequence Parameter Set) is one of the mesh information NAL units and contains syntax elements that apply to mesh information coding streams. The semantics of each field are as follows:

[0062] dsps_sequence_parameter_set_id: indicates the identifier of the mesh displacement sequence parameter set for reference by other syntax elements.

[0063] dsps_single_dimension_flag: A flag indicating whether the mesh displacement is one-dimensional. If the value is true, it indicates that the mesh displacement is one-dimensional. If the value is false, it indicates that the mesh displacement is three-dimensional.

[0064] dsps_lod_count: Indicates the number of Level of Detail (LoD) of mesh displacement. Note that the number of LoDs minus 1 (dsps_lod_count_minus1) may be encoded and decoded. In that case, use dsps_lod_count = dsps_lod_count_minus1 + 1.

[0065] dsps_entry_point_offsets_present_flag: A flag indicating whether to signal the entry point offset. If the value is true, it indicates that the entry point offset may be signaled. If the value is false, it indicates that the entry point offset is not signaled.

[0066] (Decoding and derivation of picture / frame level control parameters) Figure 8 shows an example of the syntax for transmitting coordinate system transformation parameters at the picture / frame level in DFPS. DFPS (Displacement Frame Parameter Set) is one of the NAL units of mesh information and contains syntax elements that are applied to the mesh information coding stream. The semantics of each field are as follows:

[0067] dfps_displ_sequence_parameter_set_id: indicates the value of dsps_sequence_parameter_set_id for the active mesh displacement sequence parameter set.

[0068] dfps_output_flag_present_flag: A flag indicating whether the syntax element is present in the associated mesh displacement header.

[0069] (Mesh displacement syntax structure) Figures 15, 17, and 19 are examples of the syntax structure of mesh displacement data. The semantics are as follows. Mesh displacement is a sequence of values ​​(coefficients) of the position pos and k components (k components), and is indicated by the array Qdisp[pos][k]. Displacement is a three-dimensional signal in the Cartesian coordinate system (xyz) or the local coordinate system (ntb), and each component of the three-dimensional displacement is called a component. The displacement Qdisp here is also called a coefficient because it is a value after transformation by discrete wavelet transformation, lifting transformation, DCT transformation, etc. The component variable k, which indicates the type of coordinate system dimension, takes the values ​​0, 1, and 2. The variable name is not limited to k and can be other variable names (e.g. dim). The order of the index of QDisp can be reversed, that is, Qdisp[k][pos] can be used instead of Qdisp[pos][k]. Here, the variable b is used to indicate LoD. In the figure, "displ" means "displacement".

[0070] diu_vertex_count_lod[b]: indicates the number of coordinates (displacements) included in decomposition (LoD) level b. The number of blocks b, vertexCount, may be derived as vertexCount[b] = diu_vertex_count_lod[b].

[0071] diu_last_sig_coeff[k]: an index indicating the last position of a non-zero mesh displacement coefficient in the kth component.

[0072] diu_coded_block_flag[k][b]: indicates whether the block with index b in the kth component contains a non-zero mesh displacement coefficient: 1 if so, 0 if not.

[0073] diu_coded_subblock_flag[k][b][s]: indicates whether the subblock with index s of the block with index b in the kth component contains a non-zero mesh displacement coefficient: 1 if so, 0 if not.

[0074] diu_coeff_abs_level_gt0[k][b][s][v]: Absolute value of non-zero mesh displacement coefficient of vertex of index v of subblock of index s of block of index b in kth component Indicates whether the value is greater than 0. 1 if it is greater, 0 if it is not.

[0075] diu_coeff_abs_level_gt1[k][b][s][v]: Indicates whether the absolute value of the non-zero mesh displacement coefficient of the vertex with index v of the subblock with index s of the block with index b in the kth component is greater than 1. If it is greater, it is set to 1, otherwise it is set to 0. If this syntax does not exist, it is assumed to be 0.

[0076] diu_coeff_abs_level_gt2[k][b][s][v]: Indicates whether the absolute value of the non-zero mesh displacement coefficient of the vertex with index v of the subblock with index s of the block with index b in the kth component is greater than 2. If it is greater, it is set to 1, otherwise it is set to 0. If this syntax does not exist, it is assumed to be 0.

[0077] diu_coeff_abs_level_gt3[k][b][s][v]: Indicates whether the absolute value of the non-zero mesh displacement coefficient of the vertex with index v of the subblock with index s of the block with index b in the kth component is greater than 3. If it is greater, it is set to 1, otherwise it is set to 0. If this syntax does not exist, it is assumed to be 0.

[0078] diu_coeff_sign[k][b][s][v]: indicates whether the non-zero mesh displacement coefficient of the vertex with index v of the subblock with index s of the block with index b in the kth component is positive or not. If it is positive, it is set to 1, otherwise (if it is negative), it is set to 0. If this syntax is not present, it is assumed to be 1.

[0079] diu_coeff_abs_level_rem[k][b][s][v]: The absolute value of the non-zero mesh displacement coefficient of the vertex with index v of the subblock with index s of the block with index b in the kth component minus 4. If this syntax is not present, it is assumed to be 0.

[0080] The mesh displacement decoder 305 decodes diu_last_sig_coeff for each component of the mesh displacement.

[0081] The mesh displacement decoder 305 decodes diu_coded_block_flag for each level of detail (LoD) of the mesh displacement. If diu_coded_block_flag[k][b] is 1, it decodes the following syntax elements in component #k and LoD#b.

[0082] The mesh displacement decoding unit 305 decodes diu_coded_subblock_flag for each block of the mesh displacement, and derives the start position of the subblock s from diu_coded_subblock_flag[k][b][s].

[0083] The mesh displacement decoding unit 305 decodes diu_coeff_abs_level_gt0 for each sub-block of the mesh displacement, and if diu_coeff_abs_level_gt0 is a predetermined value (for example, other than 0), decodes the subsequent diu_coeff_sign and diu_coeff_abs_level_gt1.

[0084] When diu_coeff_abs_level_gt1 is a predetermined value (for example, other than 0), the mesh displacement decoding unit 305 decodes the subsequent diu_coeff_abs_level_gt2.

[0085] When diu_coeff_abs_level_gt2 is a predetermined value (for example, other than 0), the mesh displacement decoding unit 305 decodes the subsequent diu_coeff_abs_level_gt3.

[0086] If diu_coeff_abs_level_gt3 is a predetermined value (for example, other than 0), the mesh displacement decoding unit 305 decodes the subsequent diu_coeff_abs_level_rem.

[0087] (Dimension-Parallel Displacement Data Structure) Fig. 15 shows an example of the syntax of the dimension parallel mesh displacement data of this embodiment. Fig. 16 shows an example of the syntax of the dimension parallel mesh displacement header (offset information, length information).

[0088] In the 3D data coding method in Non-Patent Document 1, the displacement data is coded and decoded using an arithmetic coding method, but there is a problem that the initialization and starting point of the arithmetic coding are not defined. In other words, unless initialization is performed at a meaningful appropriate division or unit, and the starting point (offset) on the bit stream indicating the division is coded and decoded, parallel decoding is not possible even if some initialization is performed.

[0089] As shown in the top part of the figure, in this example we have the following bitstream structure: / / Displacement of Dim#0 Dim#0 Displacement of LoD#0 (k=0, b=0) Dim#0 LoD#1 displacement (k=0, b=1) Dim#0 Displacement of LoD#2 (k=0, b=2) .. Dim#0 LoD#(dsps_lod_count-1) displacement (k=0, b= dsps_lod_count-1) byte_alignment() / / Displacement of Dim#1 Dim#1 LoD#0 displacement (k=1, b=0) Dim#1 Displacement of LoD#1 (k=1, b=1) Dim#1 LoD#2 displacement (k=1, b=2) .. Dim#1 LoD#(dsps_lod_count-1) displacement (k=1, b= dsps_lod_count-1) byte_alignment() / / Displacement of Dim#2 Dim#2 LoD#0 displacement (k=2, b=0) Dim#2 Displacement of LoD#1 (k=2, b=1) Dim#2 Displacement of LoD#2 (k=2, b=2) .. Dim#2 LoD#(dsps_lod_count-1) displacement (k=2, b= dsps_lod_count-1) byte_alignment() That is, byte alignment (byte_alignment) is inserted into the displacement data in units of Dim (segments (subsets) are divided in units of Dim). Also, one bit with a fixed value (e.g., 1), such as end_of_subset_one_bit, may be inserted before the byte alignment. After the byte alignment, the arithmetic code context is initialized before decoding the syntax element at the beginning of the next Dim.

[0090] In this embodiment, the mesh displacement decoding unit 305 performs byte alignment in units of dimensions, initializes the CABAC context, and performs encoding and decoding of the displacement data. Specifically, the mesh displacement decoding unit 305 has a configuration of an outer loop that loops a number of times depending on dsps_single_dimension_flag in the loop variable k, and an inner loop that loops a number of times from 0 to dsps_lod_count in the loop variable b. Note that k loops once when dsps_single_dimension_flag is 1, and three times when dsps_single_dimension_flag is other than 1. In this configuration, the displacements of dimension#k and LoD#b are decoded, and the byte-aligned encoded data is decoded every time LoD is finished (b=dsps_lod_count-1). That is, the byte-aligned encoded data is decoded every time dimension#k is decoded. Here, byte alignment refers to a process of aligning the current position of the bit stream to a byte-aligned position, that is, the current position to a position that is an integer multiple of 8 bits from the first bit of the bit stream. Byte alignment may use the following syntax:

[0091] byte_alignment() { alignment_bit_equal_to_one / * equal to 1 * / while( !byte_aligned() ) alignment_bit_equal_to_zero / * equal to 0 * / } alignment_bit_equal_to_one is a bit that always indicates 1. alignment_bit_equal_to_zero is a bit that always indicates 0, and this 0 bit is repeatedly encoded and decoded until it reaches a byte-aligned position. byte_aligned() returns a value of True if the current position of the bitstream is byte-aligned, that is, if the current position is an integer multiple of 8 bits from the first bit of the bitstream.

[0092] Additionally, byte alignment may use the following syntax:

[0093] byte_alignment() { while( !length_aligned() ) alignment_bit_equal_to_zero / * equal to 0 * / } The above byte alignment has the effect of reducing the overhead of the alignment code amount because alignment_bit_equal_to_one is omitted.

[0094] In addition, when dsps_entry_point_offsets_present_flag is true, the mesh displacement decoding unit 305 decodes the number of EntryPoint offsets that depend on the number of dimensions from the encoded data of the displacement header. Specifically, when dsps_single_dimension_flag is 1, 0 EntryPoint offsets are decoded, and otherwise, 2 EntryPoint offsets are decoded.

[0095] dh_entry_point_offset_minus1[i]+1 specifies the i-th EntryPoint offset in bytes and may be represented as df_entry_offset_len_minus1+1 bits. The mesh displacement header is followed by mesh displacement data consisting of NumEntryPoints+1 segments (subsets), with segment (subset) index values ​​ranging from 0 to NumEntryPoints. The first byte of the mesh displacement data is considered to be 0. Segment (subset) 0 consists of the 0th byte through dh_entry_point_offset_minus1[0]th byte of the coded slice data, and segment (subset) s consists of the firstByte[s]th byte through lastByte[s]th byte of the coded mesh displacement data, with s ranging from 1 to NumEntryPoints-1, where the firstByte[s]th byte and lastByte[s]th byte may be derived as follows:

[0096] firstByte[s]=Σ(dh_entry_point_offset_minus1[n-1]+1) In the above equation, Σ calculates the sum of dh_entry_point_offset_minus1[n-1]+1 from n=1 to s.

[0097] lastByte[s]=firstByte[s]+dh_entry_point_offset_minus1[s] A final segment (subset) may consist of the remaining bytes of the encoded mesh displacement data.

[0098] According to the above configuration, the coded data is initialized and aligned at the dimensional unit delimitation. Therefore, by identifying the starting point from the offset (entry point) of the number of decoded dimensions minus 1, it is possible to decode the displacement of each dimension in parallel from the dimensional unit delimitation.

[0099] A similar configuration can also be used for scalability. In other words, since the dimensions are byte-aligned on a dimensional basis as an outer loop, it is easy to discard data after the required dimension in a Media-Aware Network Element (MANE). By discarding data after the required dimension, only the data of the required dimension needs to be transmitted, reducing the transmission size and the amount of decoding processing.

[0100] (LoD parallel displacement data structure) Figure 17 shows an example of the syntax of mesh displacement data for LoD parallelism. Figure 18 shows an example of the syntax of mesh displacement header (offset information, length information) for LoD parallelism.

[0101] As shown in the top part of the figure, in this example we have the following bitstream structure: / / Displacement of LoD#0 Dim#0 Displacement of LoD#0 (k=0, b=0) Dim#1 LoD#0 displacement (k=1, b=0) Dim#2 LoD#0 displacement (k=2, b=0) byte_alignment() / / Displacement of LoD#1 Dim#0 LoD#1 displacement (k=0, b=1) Dim#1 Displacement of LoD#1 (k=1, b=1) Dim#2 Displacement of LoD#1 (k=2, b=1) byte_alignment() / / Displacement of LoD#2 Dim#0 Displacement of LoD#2 (k=0, b=2) Dim#1 LoD#2 displacement (k=1, b=2) Dim#2 Displacement of LoD#2 (k=2, b=2) byte_alignment() ... Dim#0 LoD#(dsps_lod_count-1) displacement (k=0, b= dsps_lod_count-1) Dim#1 LoD#(dsps_lod_count-1) displacement (k=1, b= dsps_lod_count-1) Dim#2 LoD#(dsps_lod_count-1) displacement (k=2, b= dsps_lod_count-1) byte_alignment() That is, byte alignment is inserted into the displacement data in units of LoD (segments (subsets) are divided in units of LoD). Also, one bit with a fixed value (e.g., 1), such as end_of_subset_one_bit, may be inserted before the byte alignment. After the byte alignment, the arithmetic code context is initialized before decoding the syntax element at the top of the next LoD.

[0102] In this embodiment, the mesh displacement decoding unit 305 performs byte alignment in LoD units, initializes the CABAC context, and encodes and decodes the displacement data. Specifically, the mesh displacement decoding unit 305 has a configuration of an outer loop that loops from 0 to dsps_lod_count times in the loop variable b, and an inner loop that loops from 0 to a number of times that depends on dsps_single_dimension_flag in the loop variable k. In this configuration, the displacements of LoD#b and dimension#k are decoded, and the byte-aligned encoded data is decoded every time the dimension ends (k=2). In other words, the byte-aligned encoded data is decoded every time LoD#b is decoded. Note that the loop variable k has a configuration of an inner loop that loops once when dsps_single_dimension_flag is 1, and three times otherwise.

[0103] Furthermore, when dsps_entry_point_offsets_present_flag is true, the mesh displacement decoding unit 305 decodes an EntryPoint offset of dsps_lod_count-1 from the encoded data of the displacement header.

[0104] According to the above configuration, since the encoded data is initialized and aligned at the LoD unit boundaries, the start point can be identified from the offset (entry point) of the decoded LoD-1, which has the effect of making it possible to decode the displacement of each LoD in parallel at the LoD unit boundaries.

[0105] A similar configuration can also be used for scalability. In other words, since the LoD is byte-aligned on a dimensional basis as an outer loop, it is easy to discard data after the required LoD in a Media-Aware Network Element (MANE). By discarding data after the required LoD, only the data of the required LoD needs to be transmitted, reducing the transmission size and the amount of decoding processing.

[0106] (2-dimensional LoD parallel displacement data structure) Figure 19 shows an example of the syntax of mesh displacement data for 3-dimensional LoD parallelism. Figure 20 shows an example of the syntax of mesh displacement header (offset information, length information) for 3-dimensional LoD parallelism.

[0107] As shown in the top part of the figure, in this example we have the following bitstream structure: / / Displacement of Dim#0 Dim#0 Displacement of LoD#0 (k=0, b=0) byte_alignment() Dim#0 LoD#1 displacement (k=0, b=1) byte_alignment() Dim#0 Displacement of LoD#2 (k=0, b=2) byte_alignment() .. Dim#0 LoD#dsps_lod_count-1 displacement (k=0, b= dsps_lod_count-1) byte_alignment() / / Displacement of Dim#1 Dim#1 LoD#0 displacement (k=1, b=0) byte_alignment() Dim#1 Displacement of LoD#1 (k=1, b=1) byte_alignment() Dim#1 LoD#2 displacement (k=1, b=2) byte_alignment() .. Dim#1 LoD#(dsps_lod_count-1) displacement (k=1, b= dsps_lod_count-1) byte_alignment() / / Displacement of Dim#2 Dim#2 LoD#0 displacement (k=2, b=0) byte_alignment() Dim#2 Displacement of LoD#1 (k=2, b=1) byte_alignment() Dim#2 Displacement of LoD#2 (k=2, b=2) byte_alignment() .. Dim#2 LoD#(dsps_lod_count-1) displacement (k=2, b= dsps_lod_count-1) byte_alignment() That is, byte alignment is inserted into the displacement data in units of Dim (segments (subsets) are divided in units of Dim). Also, one bit of a fixed value (e.g., 1), such as end_of_subset_one_bit, may be inserted before the byte alignment. After the byte alignment, the context of the arithmetic code is initialized before the decoding of the syntax element at the beginning of the next Dim.

[0108] In this embodiment, the mesh displacement decoding unit 305 performs byte alignment for each dimension, initializes the CABAC context of arithmetic coding, and encodes and decodes the displacement data. Specifically, the mesh displacement decoding unit 305 decodes the encoded data that has been byte aligned every time the displacement of dimension#k, LoD#b is decoded in a configuration of an outer loop that loops from 0 to dsps_single_dimension_flag?1:3 times in the loop variable k, and an inner loop that loops from 0 to dsps_lod_count times in the loop variable b.

[0109] Furthermore, when dsps_entry_point_offsets_present_flag is true, the mesh displacement decoding unit 305 decodes the EntryPoint offset of dsps_lod_count x number of dimensions-1 from the encoded data of the displacement header.

[0110] According to the above configuration, since the encoded data is initialized and aligned at the divisions of dimensions and LoD units, by identifying the starting point from an offset (entry point) of the number of decoded dimensions x the number of LoDs - 1, it is possible to decode the displacements of each dimension and LoD in parallel from the divisions of these dimensions and LoD units.

[0111] A similar configuration can also be used for scalability. In other words, because byte alignment is performed on a dimension and LoD basis, a Media-Aware Network Element (MANE) can easily discard data after the required dimension or LoD. By discarding data after the required dimension or LoD, only the data of the required dimension or LoD needs to be transmitted, reducing the transmission size and the amount of decoding processing.

[0112] (LoD dimensional parallel displacement data structure) The above describes an example of adding byte alignment to displacement data in units of Dim (dividing segments (subsets) in units of Dim), but it is also possible to add byte alignment to displacement data in units of LoD (dividing segments (subsets) in units of LoD).

[0113] In this example, the bitstream structure is as follows: / / Displacement of LoD#0 Dim#0 Displacement of LoD#0 (k=0, b=0) byte_alignment() Dim#1 LoD#0 displacement (k=1, b=0) byte_alignment() Dim#2 LoD#0 displacement (k=2, b=0) byte_alignment() / / Displacement of LoD#1 Dim#0 LoD#1 displacement (k=0, b=1) byte_alignment() Dim#1 Displacement of LoD#1 (k=1, b=1) byte_alignment() Dim#2 Displacement of LoD#1 (k=2, b=1) byte_alignment() .. / / Displacement of LoD#(dsps_lod_count-1) Dim#0 LoD#(dsps_lod_count-1) displacement (k=1, b= dsps_lod_count-1) byte_alignment() Dim#1 LoD#(dsps_lod_count-1) displacement (k=1, b= dsps_lod_count-1) byte_alignment() Dim#2 LoD#(dsps_lod_count-1) displacement (k=1, b= dsps_lod_count-1) byte_alignment() That is, byte alignment is inserted into the displacement data in units of LoD (segments (subsets) are divided in units of LoD). Also, one bit with a fixed value (e.g., 1), such as end_of_subset_one_bit, may be inserted before the byte alignment. After the byte alignment, the context of the arithmetic code is initialized before the decoding of the syntax element at the top of the next LoD.

[0114] In this embodiment, the mesh displacement decoding unit 305 performs byte alignment in LoD units, initializes the CABAC context of arithmetic coding, and encodes and decodes the displacement data. Specifically, the mesh displacement decoding unit 305 decodes the encoded data that has been byte aligned every time the displacement of dimension#k and LoD#b is decoded in a configuration of an outer loop that loops from 0 to dsps_lod_count times in loop variable b, and an inner loop that loops from 0 to dsps_single_dimension_flag?1:3 times in loop variable k.

[0115] Furthermore, when dsps_entry_point_offsets_present_flag is true, the mesh displacement decoding unit 305 decodes the EntryPoint offset of dsps_lod_count x number of dimensions-1 from the encoded data of the displacement header.

[0116] According to the above configuration, since the encoded data is initialized and aligned at the divisions of dimensions and LoD units, by identifying the starting point from an offset (entry point) of the number of decoded dimensions x the number of LoDs - 1, it is possible to decode the displacements of each dimension and LoD in parallel from the divisions of these dimensions and LoD units.

[0117] A similar configuration can also be used for scalability. In other words, because byte alignment is performed on a dimension and LoD basis, a Media-Aware Network Element (MANE) can easily discard data after the required dimension or LoD. By discarding data after the required dimension or LoD, only the data of the required dimension or LoD needs to be transmitted, reducing the transmission size and the amount of decoding processing.

[0118] Although an example in which offset information and length information are included in the mesh displacement header has been described above, offset information and length information may be included in the SEI for each NAL unit of the mesh data stream.

[0119] (Mesh displacement derivation process) The mesh displacement decoding unit 305 decodes the syntax elements diu_last_sig_coeff, diu_coded_block_flag, diu_coded_subblock_flag, diu_coeff_abs_level_gt0, diu_coeff_abs_level_gt1, diu_coeff_abs_level_gt2, diu_coeff_abs_level_gt3, diu_coeff_abs_level_rem, and diu_coeff_sign, and derives the mesh displacement Qdisp.

[0120] Here, the mesh displacement decoding unit 305 decodes diu_last_sig_coeff on a component basis or LoD basis (block basis). It decodes diu_coded_block_flag on a component basis or LoD basis, and decodes diu_coded_subblock_flag on a subblock basis of the subBlockSize size. If diu_coded_subblock_flag is a predetermined value, it decodes the mesh displacement coefficients in the subblock.

[0121] (Operation of Mesh Displacement Decoding Unit) The arithmetic decoding unit 3051 decodes the mesh displacement coded stream that has been arithmetically coded according to a value (context) indicating a random variable, and outputs a binary signal. The binary signal may be an alpha code or a k-th order Exp-Golomb code. Golomb coding is composed of a prefix and a suffix code. The prefix is ​​a value that increases exponentially, and the suffix is ​​the remainder.

[0122] The multi-value unit 3052 decodes the binary signal into the quantized mesh displacement Qdisp, which is a multi-value signal.

[0123] The context selection unit 3056 (context memory) has a memory for holding contexts, derives a context used for arithmetic decoding of mesh displacements according to the state, and updates the value as necessary. In arithmetic decoding of each coefficient of mesh displacements, the following different context arrays may be used according to the frame type ft (e.g., 0: intraframe, 1: interframe), the mesh division level lod (level of detail), and the component dim of the mesh displacement vector. The context includes at least a variable indicating the occurrence probability of a binary signal.

[0124] ctxLastSigCoeff[numFT][numDim] ctxCodedBlock[numFT][numLoD][numDim] ctxCodedSubBlock[numFT][numLoD][numDim] ctxCoeffGtN[numFT][numLoD][MAX_GTN+1][numDim] ctxCoeffRemPrefix[numFT][numLoD][numDim] Note that a static context with a fixed probability that is not updated is denoted as ctxStatic. The syntax component indicated by ctxStatic may be decoded without using the context.

[0125] Here, numFT is the number of frame types, and numFT=2.

[0126] numLoD is the maximum number of levels of detail for mesh division, and the value of the syntax element dsps_lod_count decoded from the bitstream may be used, or numLoD=4 may be used.

[0127] numDim is the number of dimensions of the mesh displacement vector. The value of the syntax element dsps_single_dimension_flag decoded from the bitstream may be used, or numDim=3 may be used. numDim = dsps_single_dimension_flag ? 1 : 3 The maximum value of the threshold for the coefficients, MAX_GTN, is 3.

[0128] ctxLastSigCoeff[numFT][numDim] is a context array used for decoding the syntax element diu_last_sig_coeff. The arithmetic decoding unit 3051 decodes diu_last_sig_coeff in the frame type ft and the dimension dim of the mesh displacement vector, using the value of ctxLastSigCoeff[ft][dim].

[0129] ctxCodedBlock[numFT][numLoD][numDim] is a context array used for decoding the syntax element diu_coded_block_flag. The arithmetic decoding unit 3051 decodes diu_coded_block_flag in the frame type ft, the detail level lod, and the dimension dim of the mesh displacement vector, using the values ​​of ctxCodedBlock[ft][lod][dim].

[0130] ctxCodedSubBlock[numFT][numLoD][numDim] is a context array used for decoding the syntax element diu_coded_subblock_flag. The arithmetic decoding unit 3051 uses the values ​​of ctxCodedSubBlock[ft][lod][dim] to decode diu_coded_subblock_flag in the frame type ft, the level of detail lod, and the dimension dim of the mesh displacement vector.

[0131] ctxCoeffGtN[numFT][numLoD][MAX_GTN+1][numDim] is a context array used for decoding the syntax element diu_coeff_abs_level_gtN (N is replaced with 0, 1, 2, MAX_GTN). The arithmetic decoding unit 3051 uses the values ​​of ctxCoeffGtN[ft][lod][N][dim] to decode diu_coeff_abs_level_gtN in the frame type ft, the level of detail lod, and the dimension dim of the mesh displacement vector.

[0132] The arithmetic decoding unit 3051 uses a bypass to decode diu_coeff_sign for the frame type ft, the level of detail lod, and the dimension dim of the mesh displacement vector.

[0133] ctxCoeffRemPrefix[numFT][numLoD][numDim] is a context array used for decoding the syntax element diu_coeff_abs_level_rem. The arithmetic decoding unit 3051 uses the value of ctxCoeffRemPrefix[ft][lod][dim] to decode diu_coeff_abs_level_rem in the frame type ft, the detail level lod, and the dimension dim of the mesh displacement vector.

[0134] The context initialization unit 3057 initializes the context (probability of occurrence of a binary signal).

[0135] In a configuration example in which segments (subsets) are generated on a dimension basis as in Fig. 15, the context initialization unit 3057 may initialize the context of the syntax element related to the displacement for each dimension. The syntax related to the displacement may include diu_last_sig_coeff, diu_coded_block_flag, diu_coded_subblock_flag, diu_coeff_abs_level_gt0[k][b][s][v], diu_coeff_abs_level_gt1, diu_coeff_abs_level_gt2, diu_coeff_abs_level_gt3, diu_coeff_sign, and diu_coeff_abs_level_rem. All contexts may be initialized regardless of the context of the syntax related to the displacement.

[0136] As another configuration, in a configuration example in which segments (subsets) are generated in LoD units as in FIG. 17, the context initialization unit 3057 may initialize the context of a syntax element related to displacement for each level of detail LoD.

[0137] As another configuration, in a configuration example in which segments (subsets) are generated in dimension and LoD units as in FIG. 19, the context initialization unit 3057 may initialize the context of a syntax element related to displacement for each dimension and LoD.

[0138] Also, the context may be initialized for each frame, or for each set of one or more frames (Group of Frames, GoF). When the context is initialized for each dimension, LoD, or frame, there is no context dependency between dimensions, LoDs, or frames. Therefore, random access to any dimension, LoD, or frame can be easily performed.

[0139] The inverse quantization unit 3053 performs inverse quantization based on the quantization scale value iscale, and derives the mesh displacement Tdisp after transformation (for example, wavelet transformation). Tdisp may be in a Cartesian coordinate system or a local coordinate system. iscale is a value derived from the quantization parameter of each component of the mesh displacement image.

[0140] Tdisp[0][] = (Qdisp[0][] * iscale[0] + iscaleOffset) >> iscaleShift Tdisp[1][] = (Qdisp[1][] * iscale[1] + iscaleOffset) >> iscaleShift Tdisp[2][] = (Qdisp[2][] * iscale[2] + iscaleOffset) >> iscaleShift Here, iscaleOffset = 1<<(iscaleShift-1). iscaleShift may be a predetermined constant, or may be a value encoded at the sequence level, picture / frame level, tile / patch level, etc., and decoded from the encoded data.

[0141] The inverse transform unit 3054 performs an inverse transform g (eg, an inverse wavelet transform) to derive the mesh displacement d.

[0142] d[0][] = g(Tdisp[0][]) d[1][] = g(Tdisp[1][]) d[2][] = g(Tdisp[2][]) The coordinate system conversion unit 3055 converts the mesh displacement (coordinate system of the mesh displacement) into the Cartesian coordinate system based on the value of the coordinate system conversion information displacementCoordinateSystem. Specifically, when displacementCoordinateSystem==1, the displacement in the local coordinate system is converted into the displacement in the Cartesian coordinate system. Here, d is a three-dimensional vector indicating the mesh displacement before the coordinate system conversion. disp is a three-dimensional vector indicating the mesh displacement after the coordinate system conversion, and is a Cartesian coordinate system. n_vec, t_vec, and b_vec are three-dimensional vectors (in the Cartesian coordinate system) corresponding to each axis of the local coordinate system of the target area or target vertex.

[0143] if (displacementCoordinateSystem == 0) { } else if (displacementCoordinateSystem == 1){ disp = d[0] * n_vec + d[1] * t_vec + d[2] * b_vec } The derivation method shown above for vector multiplication can be expressed individually as scalars as follows.

[0144] if (displacementCoordinateSystem == 0) { for (i = 0; i < 3; i++) {disp[i] = d[i]} } else if (displacementCoordinateSystem == 1){ for (i = 0; i < 3; i++) {disp[i] = d[0] * n_vec[i] + d[1] * t_vec[i] + d[2 ] * b_vec[i]} } Note that a configuration may be adopted in which disp=d is used, the same variable name is assigned before and after the transformation, and the value of d is updated by the coordinate transformation.

[0145] Alternatively, the following configuration may be used.

[0146] if (displacementCoordinateSystem == 0) { disp = d } else if (displacementCoordinateSystem == 1){ disp = d[0] * n_vec + d[1] * t_vec + d[2] * b_vec } else if (displacementCoordinateSystem == 2){ disp = d[0] * n_vec2 + d[1] * t_vec2 + d[2] * b_vec2 } Here, n_vec2, t_vec2, and b_vec2 are three-dimensional vectors (in the Cartesian coordinate system) corresponding to each axis of the local coordinate system of the adjacent region.

[0147] The following configuration may also be used.

[0148] if (displacementCoordinateSystem == 0) { disp = d } else if (displacementCoordinateSystem == 1){ disp = d[0] * n_vec3 + d[1] * t_vec3 + d[2] * b_vec3 } Here, n_vec3, t_vec3, and b_vec3 are three-dimensional vectors (in Cartesian coordinate system) corresponding to each axis of the local coordinate system of the target region with reduced fluctuation. For example, the vector of the coordinate system used for decoding is derived from the previous coordinate system and the current coordinate system as follows.

[0149] n_vec3 = (w*n_vec3 + (WT-w)*n_vec)>>wShift t_vec3 = (w * t_vec3 + (WT - w) * t_vec) >> wShift b_vec3 = (w * b_vec3 + (WT - w) * b_vec) >> wShift Here, for example, wShift = 2, 3, 4, WT = 1 << wShift, and w = 1..WT - 1. For example, when w = 3 and wShift = 3, it is derived by the following formula.

[0150] n_vec3 = (3 * n_vec3 + 5 * n_vec) >> 3 t_vec3 = (3 * t_vec3 + 5 * t_vec) >> 3 b_vec3 = (3 * b_vec3 + 5 * b_vec) >> 3 Also, it may be configured to be selectable according to the value of the displacementCoordinateSystem of the coordinate system conversion information decoded from the encoded data as follows.

[0151] if (displacementCoordinateSystem == 0) { disp = d } else if (displacementCoordinateSystem == 1) { disp = d[0] * n_vec + d[1] * t_vec + d[2] * b_vec } else if (displacementCoordinateSystem == 6) { disp = d[0] * n_vec3 + d[1] * t_vec3 + d[2] * b_vec3 } (Bypass Configuration) In the above, an example using the context is described, but some syntax elements may be decoded by bypassing without using the context. The configuration of bypass encoding / decoding has the effect of reducing the memory and processing amount of the context.

[0152] For example, block-based syntax elements diu_last_sig_coeff and diu_coded_block_flag may be bypass-decoded without using a context. That is, they may be statically decoded using ctxStatic instead of ctxLastSigCoeff[numFT][numDim] and ctxCodedBlock[numFT][numLoD][numDim]. Alternatively, diu_last_sig_coeff and diu_coded_block_flag may be decoded without using a context. By bypass-encoding and decoding these syntax elements, it is possible to reduce the memory and processing amount of the context while maintaining the coding efficiency.

[0153] Also, the syntax element diu_coeff_abs_level_rem may be decoded by bypass decoding. In other words, instead of ctxCoeffRemPrefix[ft][b][k], ctxStatic (static context) may be used for decoding. Alternatively, decoding may be performed without using a context. This has the effect of further reducing the update process and memory required for the context. The syntax elements diu_coeff_abs_level_gt0, diu_coeff_abs_level_gt1, diu_coeff_abs_level_gt2, and diu_coeff_abs_level_gt3 are not decoded by bypass decoding, but are decoded by using a context.

[0154] (Mesh reconstruction) 6 is a functional block diagram showing the configuration of the mesh reconstruction unit 307. The mesh reconstruction unit 307 is made up of a mesh division unit 3071 and a mesh deformation unit 3072.

[0155] The mesh dividing unit 3071 divides the base mesh output from the base mesh decoding unit 303 to generate divided meshes.

[0156] Fig. 9(a) shows a part (triangle) of the base mesh, and the triangle is composed of vertices v1, v2, and v3. v1, v2, and v3 are three-dimensional vectors. The mesh division unit 3071 generates and outputs a divided mesh by adding new vertices v12, v13, and v23 to the middle of each side of the triangle (Fig. 9(b)).

[0157] v12 = (v1 + v2) / 2 v13 = (v1 + v3) / 2 v23 = (v2 + v3) / 2 The following is also acceptable.

[0158] v12 = (v1 + v2 + 1) >> 1 v13 = (v1 + v3 + 1) >> 1 v23 = (v2 + v3 + 1) >> 1 The mesh deformation unit 3072 receives the division mesh and the mesh displacement, and outputs the mesh displacement d12, A deformed mesh is generated and output by adding d13 and d23 (FIG. 9(c)). The mesh displacement is the output of the mesh displacement decoding unit 305 (coordinate system conversion unit 3055). d12, d13, and d23 are mesh displacements corresponding to the vertices v12, v13, and v23 added by the mesh division unit 3071.

[0159] v12' = v12 + d12 v13' = v13 + d13 v23' = v23 + d23 In addition, d12 = disp[0][], d23 = disp[1][], and d23 = disp[3][] may also be used.

[0160] (Configuration of 3D data encoding device according to the first embodiment) 10 is a functional block diagram showing a schematic configuration of a 3D data encoding device 11 according to the first embodiment. The 3D data encoding device 11 includes an atlas information encoding unit 101, a base mesh encoding unit 103, a base mesh decoding unit 104, a mesh displacement updating unit 106, a mesh displacement encoding unit 107, a mesh displacement decoding unit 108, a mesh reconstruction unit 109, an attribute updating unit 110, a padding unit 111, a color space conversion unit 112, an attribute encoding unit 113, a multiplexing unit 114, and a mesh separation unit 115. The 3D data encoding device 11 inputs atlas information, a base mesh, a mesh displacement, a mesh, and an attribute image as 3D data, and outputs encoded data.

[0161] The atlas information encoding unit 101 encodes the atlas information and outputs an atlas information encoded stream.

[0162] The base mesh encoding unit 103 encodes the base mesh and outputs a base mesh encoded stream. Draco or the like is used as the encoding method.

[0163] The base mesh decoding unit 104 is similar to the base mesh decoding unit 303, and therefore a description thereof will be omitted.

[0164] The mesh displacement update unit 106 adjusts the mesh displacement based on the (original) base mesh and the decoded base mesh, and outputs an updated mesh displacement.

[0165] The mesh displacement encoding unit 107 encodes the updated mesh displacement and outputs a mesh displacement encoded stream.

[0166] The mesh displacement decoding unit 108 is similar to the mesh displacement decoding unit 305, and therefore a description thereof will be omitted.

[0167] The mesh reconstructing unit 109 is similar to the mesh reconstructing unit 307, and therefore a description thereof will be omitted.

[0168] The attribute update unit 110 inputs the (original) mesh, the reconstructed mesh output from the mesh reconstruction unit 109 (mesh deformation unit 3072), and an attribute image, updates the attribute image to match the position (coordinates) of the reconstructed mesh, and outputs the updated attribute image.

[0169] The padding unit 111 receives the attribute image and performs padding processing on areas where pixel values ​​are empty.

[0170] A color space conversion unit 112 performs color space conversion from the RGB format to the YCbCr format.

[0171] The attribute encoding unit 113 encodes the attribute image in the YCbCr format output from the color space conversion unit 112, and outputs an attribute video stream. As an encoding method, VVC, HEVC, or the like is used.

[0172] The multiplexing unit 114 multiplexes the atlas information coded stream, the base mesh coded stream, the mesh displacement coded stream, and the attribute video stream, and outputs the multiplexed data as coded data. As a multiplexing method, a byte stream format, ISOBMFF, or the like is used.

[0173] (Mesh separation unit operation) The mesh separator 115 generates a base mesh and a mesh displacement from the mesh.

[0174] 13 is a functional block diagram showing the configuration of the mesh separation unit 115. The mesh separation unit 115 is made up of a mesh thinning unit 1151, a mesh division unit 1152, and a mesh displacement derivation unit 1153.

[0175] The mesh thinning unit 1151 generates a base mesh by thinning out some of the vertices from the mesh.

[0176] FIG. 14(a) shows a part of a mesh, which is made up of vertices v1, v2, v3, v4, v5, 14B, the mesh is composed of vertices v1, v2, v3, v4, v5, and v6. Each of v1, v2, v3, v4, v5, and v6 is a three-dimensional vector. The mesh thinning unit 1151 generates and outputs a base mesh by thinning out the vertices v4, v5, and v6 (FIG. 14B).

[0177] The mesh dividing unit 1152, like the mesh dividing unit 3071, divides the base mesh to generate divided meshes (FIG. 14(c)).

[0178] v4' = (v1 + v2) / 2 v5' = (v1 + v3) / 2 v6' = (v2 + v3) / 2 The mesh displacement derivation unit derives and outputs the displacements d4, d5, d6 of vertices v4, v5, v6 relative to vertices v4', v5', v6' as mesh displacements based on the mesh and the divided meshes (FIG. 14(d)).

[0179] d4 = v4 - v4' d5 = v5 - v5' d6 = v6 - v6' (Base mesh encoding) FIG. 11 is a functional block diagram showing the configuration of the base mesh encoding unit 103. The base mesh encoding unit 103 is composed of a mesh encoding unit 1031, a mesh decoding unit 1032, a motion information encoding unit 1033, a motion information decoding unit 1034, a mesh motion compensation unit 1035, a reference mesh memory 1036, a switch 1037, and a switch 1038. The base mesh encoding unit 103 may be configured to include a base mesh quantization unit (not shown) after the input of the base mesh. When the base mesh is encoded without referring to other base meshes (for example, an already encoded base mesh) (intra-encoding), the switch 1037 and the switch 3038 are connected to the side that does not perform motion compensation. When the base mesh is encoded with reference to other base meshes (inter-encoding), on the other hand, they are connected to the side that performs motion compensation.

[0180] The mesh encoding unit 1031 has an intra-encoding function, intra-encodes the base mesh, and outputs a base mesh encoded stream. Draco or the like is used as the encoding method.

[0181] The mesh decoding unit 1032 is similar to the mesh decoding unit 3031, and therefore a description thereof will be omitted.

[0182] The motion information coding unit 1033 has an inter-coding function, inter-codes the base mesh, and outputs a base mesh coded stream. The coding method used is entropy coding such as arithmetic coding.

[0183] The motion information decoding unit 1034 is similar to the motion information decoding unit 3032, and therefore a description thereof will be omitted.

[0184] The mesh motion compensation unit 1035 is similar to the mesh motion compensation unit 3033, and therefore a description thereof will be omitted.

[0185] The reference mesh memory 1036 is similar to the reference mesh memory 3034, and therefore a description thereof will be omitted.

[0186] (Mesh displacement encoding) 12 is a functional block diagram showing the configuration of the mesh displacement encoding unit 107. The mesh displacement encoding unit 107 is made up of a coordinate system conversion unit 1071, a conversion unit 1072, a quantization unit 1073, a binarization unit 1074, an arithmetic encoding unit 1075, a context selection unit 1076, a context initialization unit 1077, and a header encoding unit 1078.

[0187] The coordinate system conversion unit 1071 converts the coordinate system of the mesh displacement from a Cartesian coordinate system to a coordinate system that encodes the displacement (for example, a local coordinate system) based on the value of the coordinate system conversion information displacementCoordinateSystem. Here, disp is a three-dimensional vector indicating the mesh displacement before the coordinate system conversion, d is a three-dimensional vector indicating the mesh displacement after the coordinate system conversion, and n_vec, t_vec, and b_vec are three-dimensional vectors (of the Cartesian coordinate system) indicating each axis of the local coordinate system.

[0188] if (displacementCoordinateSystem == 0) { d = disp } else if (displacementCoordinateSystem == 1){ d = (disp * n_vec, disp * t_vec, disp * b_vec) } The mesh displacement encoding unit 107 may update the value of displacementCoordinateSystem at the sequence level, or at the picture / frame level. The initial value is 0, which indicates a Cartesian coordinate system.

[0189] When updating the displacementCoordinateSystem at the sequence level, use the syntax of the configuration in Figure 7. Set asps_vdmc_ext_displacement_coordinate_system to 0 for a Cartesian coordinate system and 1 for a local coordinate system.

[0190] When changing the displacementCoordinateSystem at the picture / frame level, use the syntax in the configuration of Figure 8. For afps_vdmc_ext_displacement_coordinate_system_enable_flag, set 1 if you want to update the coordinate system, or 0 if you do not want to update the coordinate system. For afps_vdmc_ext_displacement_coordinate_system, set 0 if you want to use a Cartesian coordinate system, or 1 if you want to use a local coordinate system.

[0191] A transform unit 1072 performs a transform f (eg, wavelet transform) and derives a mesh displacement Tdisp after the transform.

[0192] Tdisp[0][] = f(d[0][]) Tdisp[1][] = f(d[1][]) Tdisp[2][] = f(d[2][]) The quantization unit 1073 performs quantization based on a quantization scale value "scale" derived from the quantization parameter of each component of the mesh displacement, and derives a mesh displacement Qdisp after quantization.

[0193] Qdisp[0][] = Tdisp[0][] / scale[0] Qdisp[1][] = Tdisp[1][] / scale[1] Qdisp[2][] = Tdisp[2][] / scale[2] Alternatively, the scale value may be approximated by a power of 2 and Qdisp may be derived using the following formula:

[0194] scale[i] = 1 << scale2[i] Qdisp[0][] = Tdisp[0][] >> scale2[0] Qdisp[1][] = Tdisp[1][] >> scale2[1] Qdisp[2][] = Tdisp[2][] >> scale2[2] The binarization unit 1074 encodes the quantized mesh displacement Qdisp, which is a multi-value signal, into a binary signal. The binary signal may be a k-th order exponential-Golomb code.

[0195] The arithmetic coding unit 1075 arithmetically codes the binary signal.

[0196] The context selection unit 1076 is similar to the context selection unit 3056, and therefore a description thereof will be omitted.

[0197] The context initialization unit 1077 is similar to the context initialization unit 3057, and therefore a description thereof will be omitted.

[0198] The header encoding unit 1078 encodes the header information of the arithmetic encoding unit 1075 and the like, and outputs a mesh displacement encoded stream.

[0199] Although an example using a context will be described here, some syntax elements may be bypass coded without using a context. The bypass coding configuration has the effect of reducing the memory and processing amount of the context.

[0200] For example, the syntax elements diu_last_sig_coeff, diu_coded_block_flag, and diu_coeff_abs_level_rem may be bypass coded without using a context. By bypass coding these syntax elements, it is possible to reduce the memory and processing amount of the context while maintaining the coding efficiency.

[0201] The mesh displacement encoding unit 107 encodes the mesh displacement Qdisp by the following process.

[0202] for (k = 0; k < numDim; k++) { / / dimension (component) loop / / encode diu_last_sig_coeff encodeExpGolomb(diu_last_sig_coeff[k], ctxLastSigCoeff[ft][k]) if (!lastSig) continue dispOffset = 0 for (b = 0; b <numLoD; b++) { / / level of detail loop, block loop / / encode diu_coded_block_flag encode(diu_coded_block_flag[k][b], ctxCodedBlock[ft][b][k]) numBlocks = dispCount[b] / subBlockSize + 1 for (s = 0; s < numBlocks; s++) { / / subblock loop / / encode of diu_coded_subblock_flag encode(diu_coded_subblock_flag[k][b][s], ctxCodedSubBlock[ft][b][k]) for (v = 0; v < subBlockSize; v++) { / / coefficient loop in subblock / / encode diu_coeff_abs_level_gt0 d = Qdisp[dispOffset + s * subBlockSize + v][k] encode(d != 0, ctxCoeffGtN[ft][b][0][k]) if (!d) continue / / encode diu_coeff_sign encode(d < 0, ctxSign[ft][b][k]) d = abs(d) - 1 / / encode diu_coeff_abs_level_gt1 encode(d != 0, ctxCoeffGtN[ft][b][1][k]) if (!d) continue d = abs(d) - 1 / / encode diu_coeff_abs_level_gt2 encode(d != 0, ctxCoeffGtN[ft][b][2][k]) if (!d) continue d = abs(d) - 1 / / encode diu_coeff_abs_level_gt3 encode(d != 0, ctxCoeffGtN[ft][b][3][k]) if (!d) continue / / encode diu_coeff_abs_level_rem encodeExpGolomb(--d, ctxCoeffRemPrefix[ft][b][k]) } } dispOffset += dispCount[b] } } Here, encode() and encodeExpGolomb() are functions that take a value and corresponding context as arguments and arithmetically encode a 1-bit value and a k-th order Golomb code binary string, respectively. dispCount[b] is the number of mesh displacements at detail level b. lastSig is a flag indicating whether the current coefficient is the last nonzero coefficient in the subblock in scan order. lastSig=0 indicates that the current coefficient is not the last nonzero coefficient in the subblock in scan order. lastSig=1 indicates that the current coefficient is the last nonzero coefficient in the subblock in scan order.

[0203] Although one embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design changes, etc. are possible within the scope that does not deviate from the gist of the present invention.

[0204] [Application example] The above-mentioned 3D data encoding device 11 and 3D data decoding device 31 can be mounted on various devices that transmit, receive, record, and play back 3D data for use. Note that the 3D data may be natural 3D data captured by a camera or the like, or artificial 3D data (including CG and GUI) generated by a computer or the like.

[0205] The present invention is not limited to the above-described embodiment, and various modifications are possible within the scope of the claims. In other words, the technical scope of the present invention also includes embodiments obtained by combining technical means that are appropriately modified within the scope of the claims. [Industrial Applicability]

[0206] The embodiments of the present invention can be suitably applied to a 3D data decoding device that decodes encoded data in which 3D data is encoded, and a 3D data encoding device that generates encoded data in which 3D data is encoded. Also, the embodiments of the present invention can be suitably applied to the data structure of encoded data generated by the 3D data encoding device and referenced by the 3D data decoding device. [Explanation of symbols]

[0207] 11 3D data encoding device 101 Atlas Information Encoding Unit 103 Base mesh coding unit 1031 Mesh coding unit 1032 Mesh Decoding Unit 1033 Motion information encoding unit 1034 Motion information decoding unit 1035 Mesh motion compensation unit 1036 Reference Mesh Memory 1037 Switch 1038 Switch 104 Base mesh decoding unit 106 Mesh displacement update section 107 Mesh Displacement Encoding Unit 1071 Coordinate system conversion section 1072 Converter 1073 Quantization section 1074 Binarization section 1075 Arithmetic encoder 1076 Context Selection 1077 Context Initialization Section 1078 Header Encoding Part 108 Mesh Displacement Decoding Unit 109 Mesh reconstruction unit 110 Attribute Update Section 111 Padding section 112 Color space conversion section 113 Attribute Encoding Unit 114 Multiplexer 115 Mesh separation section 1151 Mesh thinning section 1152 Mesh division section 1153 Mesh displacement derivation part 21 Network 31 3D data decoding device 301 Demultiplexer 302 Atlas Information Decoding Unit 303 Base Mesh Decoding Unit 3031 Mesh Decoding Unit 3032 Motion information decoding unit 3033 Mesh Motion Compensation Unit 3034 Reference Mesh Memory 3035 Switch 3036 Switch 305 Mesh Displacement Decoding Unit 3051 Arithmetic Decoding Unit 3052 Multilevel conversion section 3053 Inverse quantization section 3054 Reverse conversion unit 3055 Coordinate system conversion section 3056 Context Selection Section 3057 Context Initialization Unit 3058 Header Decoding Unit 307 Mesh reconstruction unit 306 Attribute Decoder 3071 Mesh division section 3072 Mesh transformation part 308 Color Space Conversion Unit 41 3D data display device

Claims

1. A 3D data decoding device that decodes mesh data or point cloud data comprises a header decoding unit that decodes header information from encoded data, an arithmetic decoding unit that arithmetically decodes mesh displacement from the encoded data, a context selection unit that selects a context in the arithmetic decoding unit, and a context initialization unit that sets an initial value of the context, wherein the header decoding unit decodes an offset indicating the position of the encoded data from the encoded data.

2. The 3D data decoding device according to claim 1, characterized in that the offset is information indicating the position of data for each dimension of mesh displacement contained in the encoded data.

3. The 3D data decoding device according to claim 1, characterized in that the offset is information indicating the position of data of each level of detail of mesh displacement contained in the encoded data.

4. A 3D data encoding device that encodes mesh data or point cloud data, comprising: a header encoding unit that encodes header information; an arithmetic encoding unit that arithmetically encodes mesh displacement; a context selection unit that selects a context in the arithmetic encoding; and a context initialization unit that sets an initial value of the context, wherein the header encoding unit encodes an offset that indicates a position of the encoded data.

5. The 3D data encoding device according to claim 4, characterized in that the offset is information indicating the position of data for each dimension of mesh displacement contained in the encoded data.

6. The 3D data encoding device according to claim 4, characterized in that the offset is information indicating a position of data of each level of detail of mesh displacement contained in the encoded data.

Images