3D data decoding device and 3D data encoding device

Abstract

To encode and decode mesh displacement images as 4:2:0 format images, reduce distortion caused by encoding, and encode and decode 3D data with high quality in encoding and decoding 3D data using a video encoding method.SOLUTION: A 3D data decoding device includes a video decoding unit that decodes a mesh displacement image decoded from a geometry video stream in which the Unit Type of the encoded data is V3C_GVD, and a displacement unmapping unit that derives mesh displacement for each position pos and component compIdx from the mesh displacement image, and when the geometry image is 420, the displacement unmapping unit derives the mesh displacement by deriving the Y coordinate of the geometry image from the product of a variable ranging from 0 to -1 indicating the number of dimensions of the geometry and the height.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] An embodiment of the present invention relates to a 3D data encoding device and a 3D data decoding device. [Background technology]

[0002] In order to efficiently transmit or record 3D data, there are 3D data encoding devices that convert the 3D data into a 2D image, encode it using a video encoding method, and generate encoded data, and 3D data decoding devices that decode the encoded data into a 2D image and reconstruct the 3D data.

[0003] Specific examples of 3D data encoding methods include MPEG-I's ISO / IEC 23090-5 V3C (Volumetric Video-based Coding) and V-PCC (Video-based Point Cloud Compression). V3C can encode and decode point clouds consisting of point positions and attribute information. Furthermore, it is also used for encoding and decoding multi-viewpoint images and mesh images by ISO / IEC 23090-12 (MPEG Immersive Video, MIV) and ISO / IEC 23090-29 (Video-based Dynamic Mesh Coding, V-DMC), which is currently being standardized. The latest draft document of the V-DMC method is disclosed in Non-Patent Document 1.

[0004] In these 3D data coding methods, the geometry and attributes that make up the 3D data are encoded and decoded as images using video coding methods such as H.265 / HEVC (High Efficiency Video Coding) and H.266 / VVC (Versatile Video Coding).

[0005] In the case of a point cloud, the geometry image is the depth to the projection plane, and the attribute image is the image of the attributes projected onto the projection plane.

[0006] 3D data (mesh) as described in Non-Patent Document 1 is composed of a base mesh, mesh displacement, mesh displacement array, and texture mapping image. A vertex coding method such as Draco can be used for the base mesh, a geometry image is a mesh displacement image obtained by two-dimensionalizing the mesh displacement, and an attribute image is a texture mapping image. These are coded and decoded using a video coding method such as HEVC or VVC as described above. [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] WD 3.0 of V-DMC (MDS22775_WG07_N00611_d9), ISO / IEC JTC 1 / SC 29 / WG 7 N0611, April 2023 Summary of the Invention [Problem to be solved by the invention]

[0008] The method of encoding and decoding a mesh displaced image as an image in 4:2:0 format in Non-Patent Document 1 has problems such as insufficient scalability function according to the capabilities of the decoding device and insufficient error resilience function.

[0009] The present invention aims to encode and decode mesh displacement images as images in YCbCr4:2:0 format in encoding and decoding 3D data using a video encoding method, thereby reducing distortion caused by encoding and enabling high-quality encoding and decoding of 3D data. [Means for solving the problem]

[0010] In order to solve the above problem, a 3D data decoding device according to one aspect of the present invention is provided. a displacement uncapping unit that derives mesh displacement QDisp[compIdx][pos] for each position pos and component compIdx from the mesh displacement image gFrame[compIdx][y][x] having the x, y position and component compIdx, wherein when the geometry image is 420, the displacement uncapping unit derives the mesh displacement by deriving the Y coordinate of the geometry image from the product of a variable compIdx, which ranges from 0 to a value -1 indicating the number of dimensions DisplacementDim of the geometry, and a height dispCompHeight. Effect of the Invention

[0011] According to one aspect of the present invention, distortion caused by encoding of mesh displacement images can be reduced, and 3D data can be encoded and decoded with high quality. [Brief description of the drawings]

[0012] [Figure 1] 1 is a schematic diagram showing the configuration of a 3D data transmission system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram showing a hierarchical structure of data in an encoded stream. [Diagram 3] FIG. 2 is a functional block diagram showing a schematic configuration of a 3D data decoding device 31. [Figure 4] FIG. 2 is a functional block diagram showing a configuration of a base mesh decoding unit 303. [Diagram 5] FIG. 2 is a functional block diagram showing the configuration of a mesh displacement decoding unit 305. [Figure 6] FIG. 2 is a functional block diagram showing the configuration of a mesh reconstruction unit 307. [Figure 7] 13 is an example of a syntax for transmitting coordinate transformation parameters and displacement mapping parameters at the sequence level (ASPS). [Figure 8]13 is an example of a syntax for transmitting coordinate transformation parameters and displacement mapping parameters at the picture / frame level (AFPS). [Figure 9] In this example, the mesh displacement image gFrame is divided into segments (slices) equal to the number of dimensions of the geometry. [Figure 10] This is an example where the mesh displacement image gFrame is divided into segments (slices) based on the geometry dimensionality and LOD. [Figure 11] FIG. 13 is a diagram for explaining the operation of the mesh reconstructing unit 307. [Figure 12] 1 is a functional block diagram showing a schematic configuration of a 3D data encoding device 11. FIG. [Figure 13] FIG. 2 is a functional block diagram showing a configuration of a base mesh encoding unit 103. [Figure 14] FIG. 2 is a functional block diagram showing a configuration of a mesh displacement encoding unit 107. [Figure 15] FIG. 2 is a functional block diagram showing the configuration of a mesh separation unit 115. [Figure 16] 11 is a diagram for explaining the operation of the mesh separating unit 115. FIG. [Figure 17] 13 is an example of a syntax for transmitting displacement slice partitioning parameters for each level of detail at the sequence level (ASPS). [Figure 18] 13 is an example of a syntax for transmitting displacement slice partitioning parameters for each level of detail at the picture / frame level (AFPS). [Figure 19] 13 is an example of the syntax of a displacement slice division parameter. [Figure 20] 13 is an example of the syntax of a displacement slice division parameter. [Figure 21] 13 is an example of the syntax of a displacement slice division parameter. [Figure 22] 13 is an example of the syntax of a displacement slice division parameter. [Figure 23] 13 is an example of the syntax of a displacement slice division parameter. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0014] FIG. 1 is a schematic diagram showing the configuration of a 3D data transmission system 1 according to this embodiment.

[0015] The 3D data transmission system 1 transmits an encoded stream obtained by encoding 3D data to be encoded. The 3D data transmission system 1 includes a 3D data encoding device 11, a network 21, a 3D data decoding device 31, and a 3D data display device 41.

[0016] 3D data T is input to the 3D data encoding device 11.

[0017] The network 21 transmits the encoded stream Te generated by the 3D data encoding device 11 to the 3D data decoding device 31. The network 21 is the Internet, a wide area network (WAN), a local area network (LAN), or a combination of these. The network 21 is not necessarily limited to a bidirectional communication network, and may be a unidirectional communication network that transmits broadcast waves such as terrestrial digital broadcasting and satellite broadcasting. Furthermore, the network 21 may be replaced by a storage medium on which the encoded stream Te is recorded, such as a DVD (Digital Versatile Disc: registered trademark) or a BD (Blu-ray Disc: registered trademark).

[0018] The 3D data decoding device 31 decodes each of the encoded streams Te transmitted by the network 21, and generates one or more decoded 3D data Td.

[0019] The 3D data display device 41 displays all or part of one or more pieces of decoded 3D data Td generated by the 3D data decoding device 31. The 3D data display device 41 includes a display device such as a liquid crystal display or an organic EL (Electro-luminescence) display. Display forms include stationary, mobile, HMD, and the like. Furthermore, when the 3D data decoding device 31 has high processing power, it displays high quality images, and when it has only lower processing power, it displays images that do not require high processing power or display power.

[0020] <operator> The operators used in this specification are listed below.

[0021] >> is a right bit shift, << is a left bit shift, & is a bitwise AND, | is a bitwise OR, |= is the OR assignment operator, and || indicates logical OR.

[0022] x?y:z is a ternary operator that takes y if x is true (non-zero) and z if x is false (zero). Let y..z denote the set of integers from y to z.

[0023] <Structure of the coding stream Te> Before going into a detailed description of the 3D data encoding device 11 and the 3D data decoding device 31 according to this embodiment, the data structure of the encoded stream Te generated by the 3D data encoding device 11 and decoded by the 3D data decoding device 31 will be described.

[0024] 2 is a diagram showing a hierarchical structure of data in the coded stream Te. The coded stream Te has a data structure of either a V3C sample stream or a V3C unit stream. The V3C sample stream includes a sample stream header and a V3C unit. The V3C unit stream includes a V3C unit.

[0025] A V3C unit includes a V3C unit header and a V3C unit payload. The V3C unit header begins with a Unit Type, which is an ID that indicates the type of V3C unit, and takes a value indicated by a label such as V3C_VPS, V3C_AD, V3C_AVD, V3C_GVD, or V3C_OVD.

[0026] If the Unit Type is V3C_VPS (Video Parameter Set), the V3C unit includes a V3C parameter set.

[0027] When the Unit Type is V3C_AD (Atlas Data), the V3C unit includes a VPS ID, an atlas ID, a sample stream nal header, and multiple NAL units. The ID (Idenfication) is an integer value equal to or greater than 0.

[0028] A NAL unit is a set of NALUnitType, layerID, TemporalID, and RBSP (Raw byte sequence payload).

[0029] NAL units are identified by NALUnitType, and are classified as ASPS (Atlas Sequence Parameter Set), AAPS (Atlas Adaptation Parameter Set), ATL (Atlas Tile layer), SEI (Supplemental Enhancement Information), etc.

[0030] The ATL includes an ATL header and an ATL data unit, and the ATL data unit includes information such as the position and size of a patch, such as patch information data.

[0031] The SEI includes payloadType, which indicates the type of SEI, payloadSize, which indicates the size (number of bytes) of the SEI, and sei_payload, which is the data of the SEI.

[0032] If Unit Type is V3C_AVD (Attribute Video Data), V3C The unit includes the VPS ID, atlasID, attrIdx, which is the ID of the attribute image, partIdx, which is the partition ID, mapIdx, which is the map ID, auxFlag, which indicates whether it is auxiliary data or not, and the video stream. The video stream indicates data encoded in HEVC, VVC, etc. In V-DMC, it corresponds to a texture image.

[0033] When the Unit Type is V3C_GVD (Geometry Video Data), the V3C unit includes VPS ID, atlasID, mapIdx, auxFlag, and video stream. In V-DMC, it corresponds to mesh displacement.

[0034] If the Unit Type is V3C_OVD (Occumancy Video Data), the V3C unit includes the VPS ID, atlasID, and video stream.

[0035] If the Unit Type is V3C_MD (Mesh data), the V3C unit contains the VPS ID, atlasID, and mesh_payload. In V-DMC, it corresponds to the base mesh.

[0036] (Configuration of 3D data decoding device according to the first embodiment) 3 is a functional block diagram showing a schematic configuration of a 3D data decoding device 31 according to the first embodiment. The 3D data decoding device 31 is composed of a demultiplexing unit 301, an atlas information decoding unit 302, a base mesh decoding unit 303, a mesh displacement decoding unit 305, a mesh reconstruction unit 307, an attribute decoding unit 306, and a color space conversion unit 308. The 3D data decoding device 31 inputs encoded data of 3D data, and outputs atlas information, meshes, and attribute images.

[0037] The demultiplexing unit 301 inputs encoded data multiplexed in a byte stream format, ISOBMFF (ISO Base Media File Format), etc., demultiplexes it, and outputs an atlas information encoded stream (V3C_AD Atlas Data stream, NAL unit), a base mesh encoded stream (V3C_MD mesh_payload), a mesh displacement encoded stream (V3C_GVD video stream), and an attribute video stream (V3C_AVD video stream).

[0038] The atlas information decoding unit 302 receives the atlas information encoded stream output from the demultiplexing unit 301 and decodes the atlas information.

[0039] The base mesh decoder 303 performs vertex coding (3D data compression coding method, for example, Draco). The encoded base mesh encoded stream is decoded to output a base mesh, which will be described later.

[0040] The mesh displacement decoding unit 305 decodes a geometry video stream (mesh displacement coded stream) coded by VVC, HEVC, or the like, and outputs mesh displacement. The type of codec used for coding is indicated by ptl_profile_codec_group_idc obtained by decoding the V3C parameter set of the coded data. It may also be indicated by a Four CC code (four-character code, 4CC code) indicated by gi_geometry_codec_id[atlasID] of the V3C parameter set. gi_geometry_codec_id[atlasID] indicates an index corresponding to the codec ID of the decoder used to decode the geometry video stream in the atlas ID. A set indicating the correspondence between the codec ID (ccm_codec_id) and its 4CC code (ccm_codec_4cc[ccm_codec_id]) may be transmitted in a separate codec mapping SEI (component_codec_mapping SEI). The codec may decode the mesh displacement in segments (slices) obtained by further dividing a frame. HEVC and VVC can divide a frame into slices. Slices are coded in units of Coded Tree Units (CTUs). Note that subpictures or tiles may be used as segments instead of slices. These subpictures, tiles, and slices can be decoded independently, so it is possible to decode only a part of a frame without decoding the entire frame. When using subpictures or tiles, slices are replaced with subpictures or tiles.

[0041] A mesh reconstructing unit 307 receives the base mesh and the mesh displacement, and reconstructs the mesh in the 3D space.

[0042] The attribute decoding unit 306 decodes the attribute video stream encoded by VVC, HEVC, or the like, and outputs an attribute image in YCbCr format. The attribute image may be a texture image expanded on the UV axis (a texture mapping image converted by the UV atlas method). The type of codec used for encoding is indicated by ptl_profile_codec_group_idc obtained by decoding the V3C parameter set of the encoded data. It may also be indicated by the Four CC code indicated by ai_geometry_codec_id[atlasID] of the V3C parameter set. ai_geometry_codec_id[atlasID] indicates an index in the atlas ID corresponding to the codec ID of the decoder used to decode the attribute video stream.

[0043] A color space conversion unit 308 converts the color space of the attribute image from the YCbCr format to the RGB format. Note that an attribute video stream coded in the RGB format may be decoded and color space conversion may be omitted.

[0044] (Decoding the base mesh) FIG. 4 is a functional block diagram showing the configuration of the base mesh decoding unit 303. The base mesh decoding unit 303 is composed of a mesh decoding unit 3031, a motion information decoding unit 3032, a mesh motion compensation unit 3033, a reference mesh memory 3034, a switch 3035, and a switch 3036. The base mesh decoding unit 303 may be configured to include a base mesh inverse quantization unit (not shown) before the output of the base mesh. When the base mesh to be decoded is coded (intra-coded) without referring to other base meshes (for example, base meshes that have already been coded and decoded), the switches 3035 and 3036 are connected to the side that does not perform motion compensation. On the other hand, when the base mesh to be decoded is coded (inter-coded) with reference to other base meshes, the switches 3035 and 3036 are connected to the side that performs motion compensation. When motion compensation is performed, the target vertex coordinates are derived by referring to the vertex coordinates and motion information that have already been decoded.

[0045] The mesh decoding unit 3031 decodes the intra-coded base mesh coded stream and outputs the base mesh. The coding method used is Draco, Edge Breaker, etc. There are.

[0046] The motion information decoding unit 3032 decodes the inter-coded base mesh coded stream and outputs motion information for each vertex of a reference mesh, which will be described later. Entropy coding, such as arithmetic coding, is used as the coding method.

[0047] The mesh motion compensation unit 3033 performs motion compensation on each vertex of the reference mesh input from the reference mesh memory 3034 based on the motion information, and outputs a motion-compensated mesh.

[0048] The reference mesh memory 3034 is a memory that holds decoded meshes for reference in subsequent decoding processes.

[0049] (Mesh displacement decoding) FIG. 5 is a functional block diagram showing the configuration of the mesh displacement decoding unit 305. The mesh displacement decoding unit 305 is composed of a displacement unmapper 3052 (image unpacker, displacement decoder), an inverse quantizer 3053, an inverse transformer 3054, and a coordinate system converter 3055. The displacement unmapper may also be called a displacement mapper. As shown in the figure, the mesh displacement decoding unit 305 may further include a video decoder 3051, or the mesh displacement decoding unit 305 may not include the video decoder 3051 and may use the video decoder 31 to decode the displacement image (displacement sequence). The mesh displacement decoding unit 305 may also not include the inverse quantizer 3053 and may be configured so that image quality control is performed only by the video decoder 31.

[0050] The atlas information decoding unit 302 decodes, from the encoded data, coordinate system transformation information displacementCoordinateSystem (asps_vdmc_ext_displacement_coordinate_system, afps_vdmc_ext_displacement_coordinate_system) indicating a coordinate system. In addition, slice division information of mesh displacement (displacement slice division parameter, displacement segment parameter, displacement slice division flag, displacement segment flag) may be decoded. The slice division information may be displacementSliceFlag indicating whether to divide into segments, or may be a slice division type, displacementSliceType (asps_vdmc_ext_displacement_slice_type, afps_vdmc_ext_displacement_slice_type), indicating how to divide the segments. In addition, the slice division information may include a component height dispCompHeight. The slice division information may also include a syntax element dispPos[lodIdx] indicating the start position of the mesh displacement for each LOD, or dispCount[lodIdx], the number of mesh displacements for each LOD. It may also include an index dispCountIdx[lodIdx] indicating the number of mesh displacements. The slice division information may also include a size ctuSize of the block for aligning the slices, or an index ctuSizeIdx indicating ctuSize. The slice division type is a parameter indicating the type of slice division. The component height is a parameter indicating the height of the image corresponding to each component (e.g., n, t, b) of the mesh displacement 3D vector. The components n (normal), t (tangent), and b (bitangent) are also called components.

[0051] The slice division information may be a syntax element displacementSliceFlag (displacementSliceEnabledFlag, displacementSliceUsedFlag) indicating that the displacement is divided into slices on a component basis. Here, the displacement being divided into slices on a component basis means that, when encoding the displacement using AVC, HEVC, VVC, etc., at least the first component (normal) of normal, tangent, and bitangent is encoded in a slice different from the second component (tangent) and the third component (bitangent). For example, displacementSliceFlag==1 may be a case where the screen is divided into two, slice 0 is normal, and slice 1 is tangent and bitangent. DisplacementSliceFlag==0 is a case where the screen is not divided into slices, or where it is not explicitly stated that the screen is divided into slices. Also, for example, displacementSliceFlag==1 may be a case where the screen is divided into three, slice 0 is normal, slice 1 is tangent, and slice 2 is bitangent. It may be the case where slice 2 is bitangent. It may also be defined that the slice is divided at the boundary of the displacement component. Also, as described later, asps_vdmc_ext_displacement_slice_type may be used to further indicate the case where the screen is divided into two slices, or the case where the screen is divided into three slices. In this way, by indicating in advance that the screen is divided into slices with an explicit syntax element, the 3D data decoding device can realize scalable decoding according to the decoding capability and power consumption by decoding only specific slices. Note that although slice division is mentioned here, when supporting segments (decoding units) other than slices such as HEVC and VVC, slices may be read as tiles or subpictures. The same applies below.

[0052] The slice division information may further be a flag indicating that a plurality of frames consecutive in the time direction are divided into rectangles of the same size and that inter-screen prediction is performed only within the corresponding rectangular region. For example, tile division using the Motion Constrained Tile Set (MCTS) of HEVC may be used. Alternatively, division that restricts prediction other than from subpictures consecutive in the time direction of VVC may be used. The subpicture that can be referred to is a subpicture at a collocated position with a restriction on temporal prediction. Generally, slice division has independence in which prediction and filtering are not performed with slices other than the slice in question within a screen, but the subpicture is characterized by being independent and not having a dependency not only in the spatial direction but also in the temporal direction. The information on the HEVC tiles or VVC subpictures is called spatiotemporally independent slice division information.

[0053] Note that a gating flag may be provided separately, and each piece of coordinate system transformation information may be decoded only when the gating flag is 1. The gating flag may be, for example, afps_vdmc_ext_overriden_flag. In addition, a gating flag may be provided for slice division information, and the slice division information may be decoded only when the gating flag is 1. The gating flag may be, for example, afps_vdmc_ext_displacement_slice_alignment_flag.

[0054] (Coordinate System) The following two types of coordinate systems are used for mesh displacement (3D vector). Cartesian coordinate system (canonical): A rectangular coordinate system that is commonly defined throughout the entire 3D space. (X,Y,Z) coordinate system. A rectangular coordinate system whose direction does not change at the same time (within the same frame, within the same tile). Local coordinate system (local): A Cartesian coordinate system defined for each region or vertex in 3D space. A Cartesian coordinate system whose direction can change at the same time (within the same frame, the same tile). Normal (D), tangent (U), bi-tangent (V) coordinate system. In other words, it is a Cartesian coordinate system consisting of the first axis (D) indicated by the normal vector n_vec at a certain vertex (a face containing a certain vertex), and the second axis (U) and third axis (V) indicated by two tangent vectors t_vec and b_vec that are perpendicular to the normal vector n_vec. n_vec, t_vec, and b_vec are three-dimensional vectors. The (D, U, V) coordinate system may also be called the (n, t, b) coordinate system.

[0055] (Decoding and deriving sequence-level control parameters) Here, the control parameters used in the mesh displacement decoding unit 305 will be described.

[0056] FIG. 7 shows an example of syntax for transmitting coordinate system transformation parameters and displacement slice division parameters in sequence-level ASPS. ASPS (Atlas Sequence Parameter Set, or Atlas sequence mesh information) is one of the NAL units of atlas information and contains syntax elements that apply to the coded atlas sequence. In ASPS, coordinate system transformation parameters and displacement slice division parameters are transmitted by the asps_vdmc_extension() syntax. The semantics of each field are as follows:

[0057] asps_vdmc_ext_displacement_coordinate_system: Coordinate system transformation information indicating the coordinate system of the mesh displacement. If the value is equal to a certain first value (e.g. 0), it indicates a Cartesian coordinate system. If the value is equal to another second value (e.g. 1), it indicates a local coordinate system.

[0058] asps_vdmc_ext_displacement_slice_type: Indicates the slice division type of each component of the mesh displacement. The meaning of the value is as follows. 0:Reservation. 1: Assign the first component of the mesh displacement (e.g., normal, n) to the first slice of the geometry video stream, the second component (e.g., tangent, t) to the second slice of the geometry video stream, and the third component of the geometry video stream (e.g., bi-tangent, b) to the third slice. 2: Assign the first component (eg, n) of the mesh displacement to a first slice, and the second and third components (eg, t, b) to a second slice. 3: Reservation.

[0059] As mentioned above, the following may also be true: 0: Specifies that the mesh displacement is not sliced ​​or is not sliced. 1: Assign the first component of the mesh displacement (e.g., normal) to the first slice of the geometry video stream, the second component (e.g., tangent) to the second slice of the geometry video stream, and the third component (e.g., bitangent) to the third slice of the geometry video stream. As mentioned above, the following may also be true: 0: Specifies that the mesh displacement is not sliced ​​or is not sliced. 1: Assign the first component (e.g., normal) of the mesh displacement to the first slice, and the second and third components (e.g., tangent, bitangent) to the second slice.

[0060] asps_vdmc_ext_displacement_component_height: indicates the image height corresponding to each component of the mesh displacement (e.g. Y, U, V).

[0061] (Decoding and derivation of picture / frame level control parameters) Figure 8 shows an example of syntax for transmitting coordinate system transformation parameters and displacement slice partitioning parameters in AFPS at the picture / frame level. AFPS (Atlas Frame Parameter Set or Atlas frame mesh information) is one of the NAL units of atlas information and contains syntax elements that apply to coded atlas frames. In AFPS, coordinate system transformation parameters and displacement slice partitioning parameters are transmitted using the afps_vdmc_extension() syntax. The semantics of each field are as follows:

[0062] afps_vdmc_ext_overriden_flag: A flag indicating whether to update the coordinate system of the mesh displacement. If this flag is true, the coordinate system of the mesh displacement is updated based on the value of afps_vdmc_ext_displacement_coordinate_system described below. If this flag is false, the coordinate system of the mesh displacement is not updated.

[0063] afps_vdmc_ext_displacement_coordinate_system: Coordinate system transformation information indicating the coordinate system of the mesh displacement. If the value is equal to the first value (e.g. 0), it indicates a Cartesian coordinate system. If the value is equal to the second value (e.g. 1), it indicates a local coordinate system. If the syntax element is not present, the value is inferred as the value decoded by ASPS and the default coordinate system is the coordinate system indicated by ASPS.

[0064] afps_vdmc_ext_displacement_slice_alignment_update_flag: A flag indicating whether to update slice division information for mesh displacement. If this flag is true (e.g., 1), slice division information for mesh displacement is updated based on the values ​​of asps_vdmc_ext_displacement_slice_type and asps_vdmc_ext_displacement_component_height described below. If this flag is false (e.g., 0), slice division information for mesh displacement is not updated.

[0065] afps_vdmc_ext_displacement_slice_type: indicates the type of slice division for each component of the mesh displacement. The meaning of the values ​​is as described in the semantics of asps_vdmc_ext_displacement_slice_type. If afps_vdmc_ext_displacement_slice_type is not present, afps_vdmc_ext_displacement_slice_type is set equal to asps_vdmc_ext_displacement_slice_type.

[0066] afps_vdmc_ext_displacement_component_height: indicates the image height corresponding to each component of the mesh displacement 3D vector (e.g. n, t, b). If afps_vdmc_ext_displacement_component_height does not appear, afps_vdmc_ext_displacement_component_height is set equal to asps_vdmc_ext_displacement_component_height.

[0067] The mesh displacement decoding unit 305 may derive the coordinate system transformation information displacementCoordinateSystem as follows:

[0068] displacementCoordinateSystem = afps_vdmc_ext_displacement_coordinate_system Note that if afps_vdmc_ext_displacement_coordinate_system is not present then afps_vdmc_ext_displacement_coordinate_system is set equal to asps_vdmc_ext_displacement_coordinate_system.

[0069] (Derivation of displacement slice division parameters) The mesh displacement decoding unit 305 derives the displacement slice division parameters displacementSliceType and dispCompHeight as follows.

[0070] displacementSliceType = afps_vdmc_ext_displacement_slice_type dispCompHeight = afps_vdmc_ext_displacement_component_height That is, if a syntax element of a displacement slice division parameter appears in the AFPS, the value of the corresponding syntax element in the AFPS is used, and if not, the value of the corresponding syntax element in the ASPS is used.

[0071] Note that a slice division flag, displacementSliceFlag, indicating whether or not a segment is used, may be decoded from the Atlas frame mesh information, instead of displacementSliceType.

[0072] <Example of displacement slice division parameter syntax> Figure 19 shows an example of the syntax of the displacement slice division parameter. Here, variable names that are coded and decoded in ASPS (variables beginning with the prefix asps_) are used for explanation, but variable names that are coded and decoded in AFPS units (variables beginning with the prefix afps_) may also be used.

[0073] asps_vdmc_ext_subdivision_method (afps_vdmc_ext_subdivision_method) indicates the method of dividing the displacement. If the value is 0, no division of the displacement is performed. If the value is 1, division of the displacement is performed. If afps_vdmc_ext_subdivision_method does not appear, it may be set as follows.

[0074] afps_vdmc_ext_subdivision_method = asps_vdmc_ext_subdivision_method asps_vdmc_ext_subdivision_iteration_count (afps_vdmc_ext_subdivision_iteration_count) indicates the number of divisions of the displacement. It is decoded when asps_vdmc_ext_subdivision_method is other than 0. The number of divisions of the displacement corresponds to the number of LODs, lodCount, and when asps_vdmc_ext_subdivision_iteration_count=0, 1, 2, the number of LODs is 1, 2, 3, respectively.

[0075] asps_vdmc_ext_displacement_coordinate_system indicates the coordinate system as already explained.

[0076] asps_vdmc_ext_packing_method indicates the packing method used when packing displacements into a rectangular image. If asps_vdmc_ext_packing_method=0, the displacement value is set as a 2D pixel value by scanning from the front. If asps_vdmc_ext_packing_method=1, the displacement value is set as a 2D pixel value by scanning from the back. A value of 0 may be called forward, and a value of 1 may be called reverse. Alternatively, a value of 0 may be called ascending order, and a value of 1 may be called descending order. If afps_vdmc_ext_packing_method is not present, it may be set as follows:

[0077] afps_vdmc_ext_subdivision_method= asps_vdmc_ext_packing_method asps_vdmc_ext_displacement_video_block_size_idc (afps_vdmc_ext_displacement_video_block_size_idc) indicates the size of the basic block (CTU size) when encoding the displacement with a video codec. If afps_vdmc_ext_video_block_size_idc does not appear, it may be set as follows:

[0078] afps_vdmc_ext_video_block_size_idc = asps_vdmc_ext_video_block_size_idc Also, asps_vdmc_ext_displacement_video_block_size_idc indicates the size of the basic block when rearranging the displacement in units of blocks in an image. Block sizes of 16, 32, 64, 128, and 256 may be assigned to asps_vdmc_ext_displacement_video_block_size_idc=0, 1, 2, 3, 4, and 5, respectively.

[0079] blockSize = ctuSize = 16< <asps_vdmc_ext_displacement_video_block_size_idc Alternatively, block sizes of 32, 64, 128, 256, and 512 may be assigned.

[0080] blockSize = ctuSize = 32< <asps_vdmc_ext_displacement_video_block_size_idc asps_vdmc_ext_1d_displacement_flag is a flag indicating whether or not to use one-dimensional displacement. If the value is 1, one-dimensional displacement is used and only normal is included. In this case, the number of displacement components DisplacementDim is 1. If the value is 0, three-dimensional displacement is used and the number of displacement components DisplacementDim is 3.

[0081] The asps_vdmc_ext_displacement_video_segment_enabled_flag is slice division information, as already described, and is a flag indicating whether the displacement is divided into slices in component units (whether the displacement is divided into slices at the boundary of the components). Note that the asps_vdmc_ext_displacement_video_segment_enabled_flag may indicate a subpicture with a restriction on the temporal prediction of HEVC, which is an independent segment in the spatio-temporal direction, or a subpicture with a restriction on the temporal prediction of VVC. A subpicture with a restriction on the temporal prediction of VVC is a subpicture with sps_subpic_treated_as_pic_flag=1. It may further indicate that the loop filter between slices (segments) is prohibited (sps_loop_filter_across_subpic_enabled_flag==0). The syntax name may be asps_vdmc_ext_displacement_video_independent_segment_enabled_flag, etc., to clarify that the slices (segments) are independent. In addition, it may be asps_vdmc_ext_displacement_video_scalable_enabled_flag, which indicates that divisional decoding is possible.

[0082] For spatiotemporal independent segments, sps_subpic_treated_as_pic_flag=1 and sps_loop_filter_across_subpic_enabled_flag==0. Note that sps_subpic_treated_as_pic_flag[i] = 1 indicates that the i-th subpicture of each CLVS coded picture is treated as an image in a decoding process other than the loop filter. 0 indicates that the i-th subpicture of each CLVS coded picture is not treated as an image in a decoding process other than the loop filter. sps_loop_filter_across_subpic_enabled_flag[i] = 1 indicates that a loop filter may be applied between the i-th subpictures of each CLVS coded picture. 0 indicates that a loop filter is not applied between the i-th subpictures of each CLVS coded picture. In addition, when sps_subpic_treated_as_pic_flag=1, there is a restriction that the screen width, screen height, and CTU size must be equal between the target picture and the reference picture that is an active entry in reference picture list 0 (RefPicList[0]) or reference picture list 1 (RefPicList[1]). Also, in temporal prediction, when sps_subpic_treated_as_pic_flag=1, the right range and bottom range of the reference area are derived based on the range of the subpicture as well as the width and height of the picture according to the following formula. In other words, the x coordinate of the reference position is clipped to be equal to or less than rightBoundaryPos, and the y coordinate of the reference position is clipped to be equal to or less than botBoundaryPos. rightBoundaryPos = sps_subpic_treated_as_pic_flag? SubpicRightBoundaryPos : pps_pic_width_in_luma_samples - 1 botBoundaryPos = sps_subpic_treated_as_pic_flag ? SubpicBotBoundaryPos : pps_pic_height_in_luma_samples - 1 Note that SubpicRightBoundaryPos indicates the right edge of the target subpicture, SubpicBotBoundaryPos indicates the bottom edge of the target subpicture, and pps_pic_width_in_luma_samples and pps_pic_height_in_luma_samples indicate the width and height of the picture.

[0083] 20 is an example of the syntax of the displacement slice division parameter. Here, when asps_vdmc_ext_1d_displacement_flag is a predetermined value (a value other than one dimension, a value indicating three dimensions), slice division information (for example, asps_vdmc_ext_displacement_video_segment_enabled_flag and asps_vdmc_ext_displacement_slice_type) is decoded from the encoded data.

[0084] 21 is an example of the syntax of the displacement slice division parameter. In this example, when asps_vdmc_ext_1d_displacement_flag is a predetermined value (a value other than one dimension, a value indicating three dimensions) or the number of divisions of the LOD is 1 or more (asps_vdmc_ext_subdivision_iteration_count>0), slice division information (for example, asps_vdmc_ext_displacement_video_segment_enabled_flag and asps_vdmc_ext_displacement_slice_type) is decoded from the encoded data.

[0085] FIG. 22 is an example of the syntax of the displacement slice division parameter. Here, when asps_vdmc_ext_subdivision_method is a predetermined value indicating division, the syntax elements related to the displacement feature and the encoding method are decoded. The syntax elements related to the displacement feature and the encoding method are, for example, asps_vdmc_ext_subdivision_iteration_count, asps_vdmc_ext_displacement_coordinate_system, asps_vdmc_ext_1d_displacement_flag, asps_vdmc_ext_displacement_video_block_size_idc, asps_vdmc_ext_packing_method, and asps_vdmc_ext_displacement_video_segment_enabled_flag. This provides an effect of reducing the amount of code because unnecessary syntax elements are not transmitted when the displacement is not divided into slices.

[0086] FIG. 23 is an example of the syntax of the displacement slice division parameter. Here, a syntax element asps_vdmc_ext_displacement_video_flag is decoded. asps_vdmc_ext_displacement_video_flag is a syntax element indicating whether or not the displacement is to be transmitted by a video codec. When asps_vdmc_ext_displacement_video_flag indicates that the displacement is to be transmitted by a video codec, parameters used when packing (placing) the displacement in an image are decoded. The parameters related to the packing of the displacement are, for example, the block size asps_vdmc_ext_di splacement_video_block_size_idc, packing method asps_vdmc_ext_packing_method, and slice division information asps_vdmc_ext_displacement_video_segment_enabled_flag. This has the effect of reducing the amount of code because unnecessary syntax elements are not transmitted when the displacement is transmitted by arithmetic coding (when a video codec is not used). Note that instead of asps_vdmc_ext_displacement_video_flag, a flag asps_vdmc_ext_displacement_arithmetric_coding_flag indicating whether the displacement is transmitted by arithmetic coding may be used. In this case, if (!asps_vdmc_ext_displacement_arithmetric_coding_flag) is used instead of the determination of if (asps_vdmc_ext_displacement_video_flag). In other words, when asps_vdmc_ext_displacement_arithmetric_coding_flag does not indicate the use of arithmetic coding, parameters related to the packing of the displacement are encoded and decoded.

[0087] (Operation of Mesh Displacement Decoding Unit) The video decoding unit 3051 decodes a geometry video stream (V3C_GVD video stream) encoded by VVC, HEVC, or the like, and outputs a decoded image (mesh displacement image, mesh displacement array) in which the (quantized) mesh displacement is used as a pixel value. The color components of the geometry are represented by DecGeoChromaFormat. The image may be in YCbCr4:2:0 format. The mesh displacement image may be a converted mesh displacement image. It may also be a residual of the mesh displacement image.

[0088] The displacement unmapping unit 3052 generates a mesh displacement from a mesh displacement image. Specifically, a mesh displacement Qdisp[pos][compIdx] of a one-dimensional signal in component compIdx units (=cIdx) is derived from gFrame[compIdx][y][x] of a two-dimensional mesh displacement image according to the correspondence of coordinate positions. Note that gFrame may be an image array DecGeoFrames[mapIdx][frameIdx] or GeoFramesNF[mapIdx][compTimeIdx] decoded from a geometry video stream (video stream of V3C_GVD). Here, the correspondence of coordinate positions may be a Z-order scan in units of blocks. NF stands for nominal format, which is an image after adjusting image size, color sampling, etc. frameIdx and compTimeIdx are composition time indexes. In addition, the name of the array of the mesh displaced image gFrame[compIdx][y][x] may be quantized displaced wavelet coefficients dispQuantCoeffFrame, etc., and the order of the array indices is not limited to gFrame[compIdx][y][x] but may be dispQuantCoeffFrame[x][y][compIdx] (same below).

[0089] The displacement unmapper 3052 derives DisplacementDim according to the value of a flag asps_vdmc_ext_1d_displacement_flag indicating whether or not to use the one-dimensional displacement decoded from the encoded data.

[0090] DisplacementDim = (asps_vdmc_ext_1d_displacement_flag) ? 1 : 3 Here, asps_vdmc_ext_1d_displacement_flag=1 indicates that only one dimension of the three-dimensional displacement is transmitted. It indicates that the normal component, or x-component (first component), of the displacement is present in the (compressed) geometry image. If the 1D flag is true, the displacement unmapper 3052 infers that the remaining two components are 0. asps_vdmc_ext_1d_displacement_flag=0 indicates that all three components of the displacement are present in the (compressed) geometry image.

[0091] The displacement unmapping unit 3052 derives the number of blocks, blockCount, from the number of mesh displacements (number of points), verCoordCount, and derives the displacement height, dispCompHeight, from blockCount. A block is a coefficient block of displacement, and blockSize is a variable indicating the size of the coefficient block of displacement. Width and height are variables indicating the width and height of the mesh displacement image. pixelsPerBlock = blockSize * blockSize widthInBlocks = width / blockSize shift = (1 << bitDepth) >> 1 blockCount = (verCoordCount + pixelsPerBlock - 1) / pixelsPerBlock heightInBlocks = (blockCount + widthInBlocks - 1) / widthInBlocks In one configuration, the displacement unmapper 3052 may use dispCompHeight, which is decoded from a syntax element (eg, asps_vdmc_ext_displacement_component_height) as described above.

[0092] In another configuration, the displacement unmapper 3052 may derive dispCompHeight using ⅓ of the height of the mesh divided image if the mesh displacement is sliced ​​(displacementSliceType is a predefined value, displacementSliceFlag is true).

[0093] dispCompHeight = height / 3 In another configuration, when the mesh displacement is divided into slices (when displacementSliceType is a predetermined value, when displacementSliceFlag is true), the displacement unmapper 3052 may derive the image height dispCompHeight corresponding to each component of the mesh displacement three-dimensional vector by aligning to a predetermined size according to the size of the Coded Tree Unit (CTU Size, ctuSize, videoBlockSize) of the codec used to encode the displacement. Note that it is not limited to the size of the CTU, and may be a predetermined block size of the mesh image. In this case, it is called videoBlockSize instead of ctuSize. dispCompHeight = heightInBlock * blockSize dispCompHeight = (dispCompHeight + ctuSize - 1) / ctuSize * ctuSize dispCompHeight = (dispCompHeight + ctuSize - 1) & ~(ctuSize - 1) Here, ~ is the bitwise negation operator, which inverts each bit. In other words, it may be derived to be a constant multiple of a predetermined value ctuSize. Alternatively, blockSize = ctuSize may be used. Also, blockSize = ctuSize may be used regardless of the value of displacementSliceFlag. Here, the displacement unmapping unit 3052 may derive ctuSize from the parameters SPS (Sequence Parameter Set) of the geometry video stream of the codec indicated by gi_geometry_codec_id[DecAtlasID] of V3C.

[0094] The displacement unmapping unit 3052 may decode ctuSize from the encoded data of the NAL unit of the atlas, for example, from the syntax elements of ASPS. Also, the value of ctuSizeIdx (videoBlockSizeIdx) may be decoded, and ctuSize may be derived from 16<<ctuSizeIdx or 32<<ctuSizeIdx, 64<<ctuSizeIdx.

[0095] For example, the displacement unmapping unit 3052 may use 64 if gi_geometry_codec_id[DecAtlasID] is HEVC and 128 if it is VVC as follows.

[0096] ctuSize = ptl_profile_codec_group_idc == 3 (VVC)? 128 : 64 Here, the value of ptl_profile_codec_group_idc indicates 0: AVC Progressive High, 1: HEVC Main 10, 2: HEVC Main 444, 3: VVC Main 10.

[0097] ctuSize = gi_geometry_codec_id[DecAtlasID] has a 4CC code indicating HEVC? 64 : 128 Here, the 4CC code strings indicating HEVC and VVC are "hev1" and "vvi1" respectively.

[0098] Alternatively, the displacement unmapping unit 3052 may fixedly use 128, which is the larger of the maximum CTU size of HEVC, which is 64, and the maximum CTU size of VVC, which is 128.

[0099] Figure 9 shows an example where the mesh displacement image gFrame is divided into segments (slices) equal to the number of dimensions of the geometry. Here, it is divided into three slices, and the values ​​of the first, second, and third components of QDisp are placed in the first component of gFrame. Here, W=width, H=height ght.

[0100] In one configuration, the displacement unmapping unit 3052 derives a (quantized) mesh displacement array Qdisp from a mesh displacement image gFrame of 3 x height x width. In the following configuration, a 3D data decoding device includes a video decoding unit that decodes a mesh displacement image decoded from a geometry video stream whose encoded data has a Unit Type of V3C_GVD, and a displacement unmapping unit that derives a mesh displacement QDisp[compIdx][pos] for each position pos and component compIdx from the mesh displacement image gFrame[compIdx][y][x] having the component compIdx at the x, y position, and when the geometry image is 420, the displacement unmapping unit derives the mesh displacement by deriving the Y coordinate of the geometry image from the product of a variable compIdx whose value is from 0 to -1 indicating the number of dimensions DisplacementDim of the geometry and the height dispCompHeight.

[0101] Here, in the case of a 420 image (DecGeoChromaFormat==1), Qdisp may be derived from gFrame[0][y][x], which is the first component (luminance (Y) image component) of gFrame, according to dispCompHeight. if (!asps_vdmc_ext_1d_displacement_flag) last = dispCompHeight * width - 1 else last = (width * height) - 1 for (v=0; v <verCoordCount; v++) { / / basic process Frame2QDisp v0 = asps_vdmc_ext_packing_method ? last - v : v blockIndex = v0 / pixelsPerBlock indexWithinBlock = v0 % pixelsPerBlock x0 = (blockIndex % widthInBlocks) * blockSize y0 = (blockIndex / widthInBlocks) * blockSize (x, y) = computeMorton2D(indexWithinBlock) x1 = x0 + x y1 = y0 + y for (compIdx=0; compIdx <DisplacementDim; compIdx++) { if (DecGeoChromaFormat==4:2:0) { Qdisp[v0][compIdx] = gFrame[0][compIdx * dispCompHeight + y1][x1] - shift } else { Qdisp[v0][compIdx] = gFrame[compIdx][y1][x1] - shift } } } Here, asps_vdmc_ext_packing_method=0 indicates that the displacement component samples are packed in ascending order, asps_vdmc_ext_packing_method=1 indicates that the displacement component samples are packed in descending order, and computeMorton2D is a function for implementing Z-order scan and is defined as follows: x = extracOddBits(x) { x = x & 0x55555555 x = (x | (x >> 1)) & 0x33333333 x = (x | (x >> 2)) & 0x0F0F0F0F x = (x | (x >> 4)) & 0x00FF00FF x = (x | (x >> 8)) & 0x0000FFFF } (x, y) = computeMorton2D(i) { x = extracOddBits(i>>1) y = extracOddBits(i) } In the following, loop processing, variable derivation processing, and branching will be omitted and only the part where DecGeoChromaFormat==4:2:0 will be shown, but in the case of branching by DecGeoChromaFormat or 4:4:4, the above processing can be applied as is. for(v=0; v <verCoordCount; v++) { .. for(compIdx=0; compIdx <DisplacementDim; compIdx++) { Qdisp[v0][compIdx] = gFrame[0][compIdx * dispCompHeight + y1][x1] - shift } } In one arrangement, the mesh displacement array Qdisp may be derived based on the value of the displacement slice division parameter dispCompHeight, for example by the displacement unmapper 3052 decoding the height value dispCompHeight from the encoded data.

[0102] In one configuration, the mesh displacement array Qdisp may be derived based on the value of the displacement slice division flag displacementSliceFlag. For example, the displacement unmapper 3052 may further derive the value of the height dispCompHeight in response to a syntax element indicating whether or not slicing is used.

[0103] Furthermore, the displacement unmapper 3052 may derive the value of the height dispCompHeight to be a constant multiple of a predetermined value, as already described.

[0104] Furthermore, the displacement unmapper 3052 may derive the value of the height dispCompHeight according to the height of the CTU in the geometry video stream.

[0105] The mesh displacement decoding unit 305 may be configured to decode only the first slice that contains the first component of the mesh displacement when the displacementSliceType is 1 or 2, or when specified by external means. In this case, Qdisp may be derived by the following process. DisplacementDim = 1 for (v=0; v <verCoordCount; v++) { .. for (compIdx=0; compIdx <DisplacementDim; compIdx++) { Qdisp[v0][compIdx] = gFrame[0][compIdx * dispCompHeight + y1][x1] - shift } } or Qdisp[v0][compIdx] = gFrame[0][y1][x1] - shift According to the above, it is possible to decode only the necessary part according to the communication path and the capacity of the decoding side, thereby achieving an effect of realizing scalability.

[0106] The mesh displacement decoding unit 305 may be configured to decode only the first slice containing the first component of the mesh displacement and the second slice containing the second component when the displacementSliceType is 1 or specified by an external means. In this case, Qdisp may be derived by the following process. DisplacementDim = 2 for (v=0; v <verCoordCount; v++) { .. for (compIdx=0; compIdx <DisplacementDim; compIdx++) { Qdisp[v0][compIdx] = gFrame[0][compIdx * dispCompHeight + y1][x1] - shift } } or for (v=0; v <verCoordCount; v++) { .. Qdisp[v0][0] = gFrame[0][y1][x1] - shift Qdisp[v0][1] = gFrame[0][dispCompHeight + y1][x1] - shift } The mesh displacement decoding unit 305 may be configured to decode only the first slice containing the first component of the mesh displacement and the third slice containing the third component when the displacementSliceType is 1 or specified by an external means. In this case, Qdisp may be derived by the following process. for (v=0; v <verCoordCount; v++) { … Qdisp[v0][0] = gFrame[0][y1][x1] - shift Qdisp[v0][2] = gFrame[0][2*dispCompHeight + y1][x1] - shift } As described above, the processing can be simplified by decoding only some components of the mesh displacement. Also, a scalability function can be realized by decoding slices including the second and third components of the mesh displacement as necessary (for example, decoding only the first component during high-speed playback and decoding all components during normal playback). Also, even if an error is mixed into the encoded data, error resistance can be enhanced by decoding only error-free slices (components).

[0107] Constraints may be placed on the bitstream to ensure that it is coded using segments.

[0108] For example, when the coordinate system transformation information displacementCoordinateSystem is a predetermined value, a conformance stream may be required to have a geometry video stream divided into slices. In particular, a conformance stream may be required to have a slice division in units of mesh displacement components.

[0109] This allows the 3D data decoding device to decode a geometry video stream encoded using segments, on a component compIdx basis of a mesh image, when a syntax element indicating whether to encode on a segment basis indicates that segments are to be used.

[0110] For example, when asps_vdmc_ext_1d_displacement_flag is a predetermined value (for example, 1), a conformance stream may be required to have the geometry video stream divided into slices. In particular, a conformance stream may be required to have the geometry video stream divided into slices in units of mesh displacement components.

[0111] According to this, when one-dimensional encoding is performed, the 3D data decoding device always decodes a geometry video stream that has been encoded using segments in units of mesh image components compIdx. (decoding and derivation at each level of detail) Also, a configuration may be adopted in which individual displacement slice division parameters are decoded and derived for each level of detail (levelOfDetails, LOD) of the mesh. In Fig. 17 and Fig. 18, when slice_type is 1 or 2, the same displacement division parameters are used regardless of the level of detail of the mesh, as in Fig. 7 and Fig. 8, and when slice_type is 3, different displacement division parameters are used for each level of detail of the mesh. For example, slice_type=3 is a type that assigns different slices to each level of detail of mesh displacement.

[0112] 17 is an example of syntax for transmitting the displacement slice division parameter asps_vdmc_ext_displacement_component_height_lod[i] for each level of detail at the sequence level (ASPS). asps_vdmc_ext_displacement_component_height_lod[i] indicates the height of the image corresponding to each component of the mesh displacement.

[0113] 18 shows an example of the syntax for transmitting the displacement slice division parameter afps_vdmc_ext_displacement_component_height_lod[i] for each level of detail at the picture / frame level (AFPS). afps_vdmc_ext_displacement_component_height_lod[i] indicates the height of the slice in the image corresponding to each component of the mesh displacement.

[0114] Syntax elements in Figures 17 and 18 that are not explained have the same definitions as those in Figures 7 and 8. The displacement unmapper 3052 may derive Qdisp by the following process. lodCount = asps_vdmc_ext_subdivision_iteration_count + 1 offsetHeight = 0 for (compIdx=0; compIdx<DisplacementDim; compIdx++) { for (lodIdx=0; lodIdx<lodCount; lodIdx++) { if (!asps_vdmc_ext_1d_displacement_flag) last = (offsetHeight + dispCompHeight[lodIdx]) * width - 1 else last = (width * height) - 1 vStart = lodIdx ? verCoordCount[lodIdx - 1] : 0 vEnd = vStart + verCoordCount[lodIdx] for (v=vStart; v<vEnd; v++) { v0 = asps_vdmc_ext_packing_method ? last - v : v blockIndex = v0 / pixelsPerBlock indexWithinBlock = v0 % pixelsPerBlock x0 = (blockIndex % widthInBlocks) * blockSize y0 = (blockIndex / widthInBlocks) * blockSize (x, y) = computeMorton2D(indexWithinBlock) x1 = x0 + x y1 = y0 + y if (DecGeoChromaFormat == 4:2:0) { Qdisp[v0][compIdx] = gFrame[0][offsetHeight + compIdx * dispCompHeight[lodIdx] + y1][x1] - shift } else { Qdisp[v0][compIdx] = gFrame[compIdx][offsetHeight + y1][x1] - shift } } offsetHeight += dispCompHeight[lodIdx] } } Here, width and height are the width and height of the mesh displacement image gFrame[compIdx][y][x] (quantized displacement wavelet coefficient dispQuantCoeffFrame) (the same applies hereinafter). Note that dispCompHeight = height / 3 may be used. In particular, in the case of a spatio-temporally independent slice (e.g., a sub-picture of VVC), since the same screen height is used across multiple frames, it is appropriate to use dispCompHeight = height / 3.

[0115] When using slices that are continuous in CTU units instead of rectangular slices (CTU line units), the starting position dispCompPos[lodIdx] of the slice decoded from the encoded data may be used instead of the height compIdx * dispCompHeight[lodIdx] of the slice.

[0116] The displacement demapping unit 3052 may derive Qdisp by the following process.

[0117] <Example 1 of LOD unit processing> lodCount = asps_vdmc_ext_subdivision_iteration_count + 1 offsetHeight = 0 for (lodIdx = 0; lodIdx<lodCount; lodIdx++) { if (!asps_vdmc_ext_1d_displacement_flag) last = (offsetHeight + dispCompHeight[lodIdx]) * width - 1 else last = (width * height) - 1 vStart = lodIdx ? verCoordCount[lodIdx - 1] : 0 vEnd = vStart + verCoordCount[lodIdx] for (v=vStart; v<vEnd; v++) { v0 = asps_vdmc_ext_packing_method ? last - v : v blockIndex = v0 / pixelsPerBlock indexWithinBlock = v0 % pixelsPerBlock x0 = ( blockIndex % widthInBlocks ) * blockSize y0 = ( blockIndex / widthInBlocks ) * blockSize (x, y) = computeMorton2D(indexWithinBlock) x1 = x0 + x y1 = y0 + y for (compIdx=0; compIdx<DisplacementDim; compIdx++) { if (DecGeoChromaFormat == 4:2:0) { Qdisp[v0][compIdx] = gFrame[0][offsetHeight + compIdx * dispCompHeight[lodIdx] + y1][x1] - shift} else { Qdisp[v0][compIdx] = gFrame[compIdx][offsetHeight + y1][x1] - shift } } } offsetHeight += dispCompHeight[lodIdx] } Here, asps_vdmc_ext_subdivision_iteration_count and asps_vdmc_ext_displacement_component_height_lod[i] may be replaced with afps_vdmc_ext_subdivision_iteration_count and afps_vdmc_ext_displacement_component_height_lod[i] respectively.

[0118] When displacementSliceType is 3 (the type that assigns different slices for each detailed level of mesh displacement), the mesh displacement decoding unit 305 may be configured to decode the displacement slice division parameters only for slices including components below a predetermined detailed level of mesh displacement (for example, only levelOfDetails = 0, only levelOfDetails = 0 and 1, etc.). Note that dispCompHeight = height / 3 may be used. In particular, in the case of spatio-temporally independent slices (e.g., sub-pictures of VVC), since the same screen height is used across multiple frames, it is appropriate to use dispCompHeight = height / 3.

[0119] <Another configuration for slice division in LOD units> FIG. 10 shows an example in which the mesh displacement image gFrame is divided into segments (slices) of the number of dimensions of the geometry. Here, it is divided into three slices, and the values of the first, second, and third components of QDisp are arranged in the first component of gFrame. Here, W = width and H = height.

[0120] The displacement unmapping unit 305 may derive Qdisp from the mesh transformed image divided into slices in units of LOD by the following process: Here, levelOfDetailAttributeCounts[i] is a variable indicating the start position pos of the mesh displacement of the LOD indicated by index i+1. When the displacement slice division parameter decoded from the encoded data indicates LOD packing, the displacement unmapper 305 aligns the LOD size numBlocksInLod to a constant multiple of the block size pixelsPerBlock. The block size may be used to loop through blocks in units of LOD to derive Qdisp. The displacement slice division parameter indicates LOD packing when, for example, displacementSliceType has a predetermined value, and is indicated by lodBlockPacking=1. lodBlockPacking=0 indicates no LOD packing. lodBlockPacking=displacementSliceFlag may also be used. The formula for deriving the number of blocks in LOD units aligned by a constant multiple is, for example, given below.

[0121] numBlocksInLod[0] = ( levelOfDetailAttributeCounts[0] + pixelsPerBlock - 1 ) / pixelsPerBlock In addition, when the mesh displacement image (geometry video stream) indicates that it is divided into slices, pixelsPerBlock may be derived from ctuSize (blockSize = ctuSize). In addition, ctuSize may be included in the displacement slice division parameters and decoded, or may be determined based on the type of codec. In addition, a constant such as 128 may be used for ctuSize. In addition, when the displacement slice division parameters indicate LOD packing, pixelsPerBlock may be derived from ctuSize.

[0122] In another configuration, the number of LOD unit blocks aligned by a constant multiple may be directly transmitted as the displacement slice division parameter. In this case, it may be derived as follows.

[0123] numBlocksInLod[i] = dispCount[i], i = 0..lodCount Or it may be derived as follows.

[0124] numBlocksInLod[i] = 32 << dispCountIdx[i], i = 0..lodCount Here, 32 may also be 64, 128, or ctuSize.

[0125] <Example 2 of LOD unit processing> lodCount = asps_vdmc_ext_subdivision_iteration_count + 1 pixelsPerBlock = blockSize * blockSize widthInBlocks = width / blockSize shift = (1 << bitDepth) >> 1 if( lodBlockPacking ) { blockSize = ctuSize pixelsPerBlock = blockSize * blockSize blocksInVideo = ( width * height - 1 ) / pixelsPerBlock numBlocksInLod[0] = ( levelOfDetailAttributeCounts[0] + pixelsPerBlock - 1 ) / pixelsPerBlock for( i = 1; i < lodCount; i++ ) { numBlocksInLod[i] = ( levelOfDetailAttributeCounts[i] - levelOfDetailAttributeCounts[i - 1] + pixelsPerBlock - 1 ) / pixelsPerBlock } for( lodIdx = 0; lodIdx < lodCount; lodIdx++ ) { if ( lodIdx == 0 ) { vStart = 0 vEnd = levelOfDetailAttributeCounts[ lodIdx ] } else { vStart = vStart + numBlocksInLod[ lodIdx - 1] vEnd = vStart + numBlocksInLod[ lodIdx ] } for( v = vStart; v < vEnd; v++ ) { blockIndex = v / pixelsPerBlock indexWithinBlock = v % pixelsPerBlock if( asps_vdmc_ext_packing_method ){ blockIndex = blocksInVideo - blockIndex indexWithinBlock = pixelsPerBlock - 1 - indexWithinBlock } x0 = (blockIndex % widthInBlocks) * blockSize y0 = (blockIndex / widthInBlocks) * blockSize ( x, y ) = computeMorton2D(indexWithinBlock) x = x0 + x y = y0 + y for (compIdx = 0; compIdx < DisplacementDim; compIdx++) { if (DecGeoChromaFormat == 4:2:0) { Qdisp[v][compIdx] = gFrame[compIdx][y + compIdx * dispCompHeight][x] - shift } else { QDisp[v][compIdx] = gFrame[compIdx][y][x] - shift } } } } } else { / / same as basic process Frame2QDisp (description omitted) } Note that dispCompHeight = height / 3 may be used. Especially, in the case of spatio-temporal independent slices (e.g. sub-pictures of VVC), since the same picture height is used across multiple frames, it is appropriate to use dispCompHeight = height / 3.

[0126] <Example 3 of LOD unit processing> Also, the displacement unmapper 305 may use ctuSize decoded from the encoded data to derive the block size blockSize, which is the scanning unit of the displacement of the mesh displacement image, as follows. This allows the encoder to freely set the block size used in the geometry video stream of the mesh displacement image, and to match the scanning unit of the mesh displacement with the block size of the video stream. Also, in a loop in LOD units from 0 to lodCount, the displacement unmapper 305 derives the start and length of the block, vStart and vEnd, outside the loop, and updates vStart to be equal to the previous vEnd just before the end of the loop, and adds a predetermined block size to vEnd, vEnd = startBlock + numBlock. It may also be derived using cksInLod[lodIdx+1]. lodCount = asps_vdmc_ext_subdivision_iteration_count + 1 blockSize = ctuSize or if (lodBlockPacking) blockSize = ctuSize pixelsPerBlock = blockSize * blockSize widthInBlocks = width / blockSize shift = (1 << bitDepth) >> 1 if( lodBlockPacking ) { blocksInVideo = ( width * height + pixelsPerBlock - 1 ) / pixelsPerBlock numBlocksInLod[0] = ( levelOfDetailAttributeCounts[0] + pixelsPerBlock - 1 ) / pixelsPerBlock for( i = 1; i < lodCount; i++ ) { numBlocksInLod[i] = ( levelOfDetailAttributeCounts[i] - levelOfDetailAttributeCounts[i - 1] + pixelsPerBlock - 1 ) / pixelsPerBlock } startBlock = 0 for( lodIdx = 0; lodIdx < lodCount; lodIdx++ ) { vStart = 0 vEnd = levelOfDetailAttributeCounts[ lodIdx ] for( v = vStart; v < vEnd; v++ ) { blockIndex = v / pixelsPerBlock indexWithinBlock = v % pixelsPerBlock if( asps_vdmc_ext_packing_method ){ blockIndex = blocksInVideo - blockIndex indexWithinBlock = pixelsPerBlock - 1 - indexWithinBlock } x0 = (blockIndex % widthInBlocks) * blockSize y0 = (blockIndex / widthInBlocks) * blockSize ( x, y ) = computeMorton2D(indexWithinBlock) x = x0 + x y = y0 + y for( compIdx = 0; compIdx < DisplacementDim; compIdx ++ ) { if (DecGeoChromaFormat == 4:2:0) { Qdisp[v][compIdx] = gFrame[0][y+compIdx*dispCompHeight][x] - shift } else { QDisp[v][compIdx] = gFrame[compIdx][y][x] - shift } } } vStart = vEnd vEnd = vStart + numBlocksInLod[lodIdx+1] } } else { / / same as basic process Frame2QDisp (description omitted) } Note that it is also possible to always perform the process with lodBlockPacking = 1 without using the flag of lodBlockPacking (the same applies hereinafter). Also, in the above, dispCompHeight may be set to height / 3. In particular, in the case of spatio-temporally independent slices (e.g., sub-pictures of VVC), since the same screen height is used across multiple frames, it is appropriate to use dispCompHeight = height / 3.

[0127] <Example 4 of LOD unit processing> Furthermore, as follows, the displacement unmapping unit 305 may calculate the start and end positions of LOD units and their block units in advance, with the number of LODs in the case of processing in LOD units (lodBlockPacking) being lodCount and other LODs being 0, and derive QDisp through loops in LOD units and point units. lodCount = asps_vdmc_ext_subdivision_iteration_count + 1 blockSize = ctuSize / / or if (lodBlockPacking) blockSize = ctuSize pixelsPerBlock = blockSize * blockSize widthInBlocks = width / blockSize shift = (1 << bitDepth) >> 1 if( lodBlockPacking ) { blocksInVideo = ( width * height - 1 ) / pixelsPerBlock numBlocksInLod[0] = ( levelOfDetailAttributeCount + pixelsPerBlock - 1 ) / pixelsPerBlock for( i = 1; i < lodCount; i++ ) { numBlocksInLod[i] = ( levelOfDetailAttributeCounts[i] - levelOfDetailAttributeCounts[i - 1] + pixelsPerBlock - 1 ) / pixelsPerBlock } startBlock[0] = 0 extraPoints = 0 for( lodIdx = 1; lodIdx < lodCount; lodIdx++ ) { numPointsInLod = levelOfDetailAttributeCounts[lodIdx] - levelOfDetailAttributeCounts [lodIdx-1] numBlockInLod[lodIdx] = (numPointsInLod + pixelsPerBlock - 1) / pixelsPerBlock vStart[lodIdx] = levelOfDetailAttributeCounts[lodIdx-1] vEnd[lodIdx] = levelOfDetailAttributeCounts[lodIdx] extraPoints += (numBlockInLod[i] * pixelsPerBlock) - numPointsInLod startBlock[lodIdx] = numBlockInLod[lodIdx-1] + startBlock[lodIdx-1] } } else { lodCount = 0 vStart[0] = vStart vEnd[0] = verCoordCount } blockCount = (verCoordCount + extraPoints + pixelsPerBlock - 1) / pixelsPerBlockblockHeight = (blockCount + blockSize - 1) / blockSize dispCompHeight = blockHeight * blockSize / / in the case that blockSize == ctuSize beforehand, dispCompHeight can be an integer multiple of blockSize and it can be aligned. for( lodIdx = 0; lodIdx < lodCount; lodIdx++ ) { for( v = vStart[lodIdx]; v < vEnd[lodIdx]; v++ ) { blockIndex = (v - vStart[lodIdx]) / pixelsPerBlock + startBlock[lodIdx] indexWithinBlock = (v - vStart[lodIdx]) % pixelsPerBlock if( asps_vdmc_ext_packing_method ){ blockIndex = blocksInVideo - blockIndex indexWithinBlock = pixelsPerBlock - 1 - indexWithinBlock } x0 = (blockIndex % widthInBlocks) * blockSize y0 = (blockIndex / widthInBlocks) * blockSize ( x, y ) = computeMorton2D(indexWithinBlock) x = x0 + x y = y0 + y for( compIdx = 0; compIdx < DisplacementDim; compIdx ++ ) { if (DecGeoChromaFormat == 4:2:0) { Qdisp[v][compIdx] = gFrame[0][y+compIdx*dispCompHeight][x] - shift } else { QDisp[v][compIdx] = gFrame[compIdx][y][x] - shift } } } } According to the above, the loop is performed in units of lodCount regardless of whether processing is performed in units of LOD or not, so the processing becomes simple. Also, in the above, dispCompHeight = height / 3 may be used. In particular, in the case of spatiotemporal independent slices (e.g. VVC subpictures), it is appropriate to use dispCompHeight = height / 3 because the same screen height is used across multiple frames.

[0128] For example, when displacementSliceType is a predetermined value (when displacementSliceFlag is true), the request for a standard-compliant stream (conformance stream) may be that the mesh displacement image (geometry video stream) is divided into slices. In particular, the request for a conformance stream may be that the mesh displacement image (geometry video stream) is divided into slices in units of LOD. Also, when lodBlockPacking is a predetermined value (for example, 1), the request for a conformance stream may be that the mesh displacement image (geometry video stream) is divided into slices in units of LOD.

[0129] As a result, the 3D data decoding device may decode a geometry video stream encoded using slices (segments) in units of LOD of mesh images when a syntax element indicating whether to use slice (segment) unit encoding indicates that slices (segments) are to be used.

[0130] For example, when asps_vdmc_ext_1d_displacement_flag is a predetermined value (for example, 1), a conformance stream may be required to have the geometry video stream divided into slices. In particular, a conformance stream may be required to have the geometry video stream divided into slices in units of mesh displacement LOD.

[0131] According to this, when the mesh images are one-dimensionally encoded, the 3D data decoding device always decodes the geometry video stream that has been encoded using segments in units of LOD.

[0132] asps_vdmc_ext_subdivision_iteration_count and afps_vdmc_ext_subdivision_iteration_count are parameters that indicate the number of mesh division iterations notified at the ASPS level and AFPS level, and are parameters used to derive displacement slice division parameters for each level of detail.

[0133] The inverse quantization unit 3053 performs inverse quantization based on the quantization scale value isescale, and performs a transformation (e.g. , wavelet transform) is derived. Tdisp may be in a Cartesian coordinate system or a local coordinate system. iscale is a value derived from the quantization parameters of each component of the mesh displacement image. Tdisp[0][pos] = (Qdisp[0][pos] * iscale[0] + iscaleOffset) >>iscaleShift Tdisp[1][pos] = (Qdisp[1][pos] * iscale[1] + iscaleOffset) >>iscaleShift Tdisp[2][pos] = (Qdisp[2][pos] * iscale[2] + iscaleOffset) >>iscaleShift Here, iscaleOffset = 1<<(iscaleShift-1). iscaleShift may be a predetermined constant, or may be a value encoded at the sequence level, picture / frame level, tile / patch level, etc., and decoded from the encoded data.

[0134] The inverse transform unit 3054 performs an inverse transform g (eg, an inverse wavelet transform) to derive the mesh displacement d. d[0][pos] = g(Tdisp[0][pos]) d[1][pos] = g(Tdisp[1][pos]) d[2][pos] = g(Tdisp[2][pos]) The coordinate system conversion unit 3055 converts the mesh displacement (coordinate system of the mesh displacement) into the Cartesian coordinate system based on the value of the coordinate system conversion information displacementCoordinateSystem. Specifically, when displacementCoordinateSystem = 1, the displacement in the local coordinate system is converted into the displacement in the Cartesian coordinate system. Here, d is a three-dimensional vector indicating the mesh displacement before the coordinate system conversion. disp is a three-dimensional vector indicating the mesh displacement after the coordinate system conversion, and is a Cartesian coordinate system. n_vec, t_vec, and b_vec are three-dimensional vectors (in the Cartesian coordinate system) corresponding to each axis of the local coordinate system of the target area or target vertex. if (displacementCoordinateSystem == 0) { disp = d } else if (displacementCoordinateSystem == 1){ disp = d[0] * n_vec + d[1] * t_vec + d[2] * b_vec } The derivation method shown in the vector multiplication above can be expressed individually as scalars as follows. if (displacementCoordinateSystem == 0) { for (i = 0; i < 3; i++) { disp[i] = d[i] } } else if (displacementCoordinateSystem == 1){ for (i = 0; i < 3; i++) { disp[i] = d[0] * n_vec[i] + d[1] * t_vec[i] + d[2] * b_vec[i] }} Note that a configuration may be adopted in which disp=d is used, the same variable name is assigned before and after the transformation, and the value of d is updated by the coordinate transformation.

[0135] Alternatively, the following configuration may be used. if (displacementCoordinateSystem == 0) { disp = d } else if (displacementCoordinateSystem == 1){ disp = d[0] * n_vec + d[1] * t_vec + d[2] * b_vec } else if (displacementCoordinateSystem == 2){ disp = d[0] * n_vec2 + d[1] * t_vec2 + d[2] * b_vec2 } Here, n_vec2, t_vec2, and b_vec2 are three-dimensional vectors (in the Cartesian coordinate system) corresponding to each axis of the local coordinate system of the adjacent region.

[0136] The following configuration may also be used. if (displacementCoordinateSystem == 0) { disp = d } else if (displacementCoordinateSystem == 1){ disp = d[0] * n_vec3 + d[1] * t_vec3 + d[2] * b_vec3 } Here, n_vec3, t_vec3, and b_vec3 are three-dimensional vectors (in Cartesian coordinate system) corresponding to each axis of the local coordinate system of the target region with reduced fluctuation. For example, the vector of the coordinate system used for decoding is derived from the previous coordinate system and the current coordinate system as follows.

[0137] n_vec3 = (w*n_vec3 + (WT-w)*n_vec)>>wShift t_vec3 = (w*t_vec3 + (WT-w)*t_vec)>>wShift b_vec3 = (w*b_vec3 + (WT-w)*b_vec)>>wShift For example, wShift=2,3,4, WT=1< <wShift、w=1..WT-1。 For example, when w=3 and wShift=3, the following may be used.

[0138] n_vec3 = (3*n_vec3 + 5*n_vec)>>3 t_vec3 = (3*t_vec3 + 5*t_vec)>>3 b_vec3 = (3*b_vec3 + 5*b_vec)>>3 Alternatively, the selection may be made according to the value of the coordinate system transformation information displacementCoordinateSystem decoded from the encoded data as follows. if (displacementCoordinateSystem == 0) { disp = d } else if (displacementCoordinateSystem == 1){ disp = d[0] * n_vec + d[1] * t_vec + d[2] * b_vec } else if (displacementCoordinateSystem == 6){ disp = d[0] * n_vec3 + d[1] * t_vec3 + d[2] * b_vec3 } (Mesh reconstruction) 6 is a functional block diagram showing the configuration of the mesh reconstruction unit 307. The mesh reconstruction unit 307 is made up of a mesh division unit 3071 and a mesh deformation unit 3072.

[0139] The mesh dividing unit 3071 divides the base mesh output from the base mesh decoding unit 303 to generate divided meshes.

[0140] Fig. 11(a) shows a part (triangle) of the base mesh, and the triangle is composed of vertices v1, v2, and v3. v1, v2, and v3 are three-dimensional vectors. The mesh division unit 3071 generates and outputs a divided mesh by adding new vertices v12, v13, and v23 to the middle of each side of the triangle (Fig. 11(b)). v12 = (v1 + v2) / 2 v13 = (v1 + v3) / 2 v23 = (v2 + v3) / 2 The following is also acceptable. v12 = (v1 + v2 + 1) >>1 v13 = (v1 + v3 + 1) >>1 v23 = (v2 + v3 + 1) >>1 The mesh deformation unit 3072 receives the division mesh and the mesh displacement, and outputs the mesh displacement d12, The mesh displacement is output by the mesh displacement decoding unit 305 (coordinate system conversion unit 3055). d12, d13, and d23 are the mesh These are mesh displacements corresponding to the vertices v12, v13, and v23 added by the mesh division unit 3071. v12' = v12 + d12 v13' = v13 + d13 v23' = v23 + d23 In addition, d12 = disp[0][], d23 = disp[1][], and d23 = disp[3][] may also be used.

[0141] (Configuration of 3D data encoding device according to the first embodiment) 12 is a functional block diagram showing a schematic configuration of a 3D data encoding device 11 according to the first embodiment. The 3D data encoding device 11 includes an atlas information encoding unit 101, a base mesh encoding unit 103, a base mesh decoding unit 104, a mesh displacement updating unit 106, a mesh displacement encoding unit 107, a mesh displacement decoding unit 108, a mesh reconstruction unit 109, an attribute updating unit 110, a padding unit 111, a color space conversion unit 112, an attribute encoding unit 113, a multiplexing unit 114, and a mesh separation unit 115. The 3D data encoding device 11 inputs atlas information, a base mesh, a mesh displacement, a mesh, and an attribute image as 3D data, and outputs encoded data.

[0142] The atlas information encoding unit 101 encodes the atlas information and outputs an atlas information encoded stream.

[0143] The base mesh encoding unit 103 encodes the base mesh and outputs a base mesh encoded stream. Draco or the like is used as the encoding method.

[0144] The base mesh decoding unit 104 is similar to the base mesh decoding unit 303, and therefore a description thereof will be omitted.

[0145] The mesh displacement update unit 106 adjusts the mesh displacement based on the (original) base mesh and the decoded base mesh, and outputs an updated mesh displacement.

[0146] The mesh displacement encoding unit 107 encodes the updated mesh displacement and outputs a mesh displacement encoded stream. The encoding method used is VVC, HEVC, or the like.

[0147] The mesh displacement decoding unit 108 is similar to the mesh displacement decoding unit 305, and therefore a description thereof will be omitted.

[0148] The mesh reconstructing unit 109 is similar to the mesh reconstructing unit 307, and therefore a description thereof will be omitted.

[0149] The attribute update unit 110 inputs the (original) mesh, the reconstructed mesh output from the mesh reconstruction unit 109 (mesh deformation unit 3072), and an attribute image, updates the attribute image to match the position (coordinates) of the reconstructed mesh, and outputs the updated attribute image.

[0150] The padding unit 111 receives the attribute image and performs padding processing on areas where pixel values ​​are empty.

[0151] A color space conversion unit 112 performs color space conversion from the RGB format to the YCbCr format.

[0152] The attribute encoding unit 113 encodes the attribute image in the YCbCr format output from the color space conversion unit 112, and outputs an attribute video stream. As an encoding method, VVC, HEVC, or the like is used.

[0153] The multiplexing unit 114 multiplexes the atlas information coded stream, the base mesh coded stream, the mesh displacement coded stream, and the attribute video stream, and outputs the multiplexed stream as coded data. As a multiplexing method, a byte stream format, ISOBMFF, etc. is used.

[0154] (Mesh separation unit operation) The mesh separator 115 generates a base mesh and a mesh displacement from the mesh.

[0155] 15 is a functional block diagram showing the configuration of the mesh separation unit 115. The mesh separation unit 115 is made up of a mesh thinning unit 1151, a mesh division unit 1152, and a mesh displacement derivation unit 1153.

[0156] The mesh thinning unit 1151 generates a base mesh by thinning out some of the vertices from the mesh.

[0157] FIG. 16(a) shows a part of a mesh, which is made up of vertices v1, v2, v3, v4, v5, 16B, the mesh is composed of vertices v1, v2, v3, v4, v5, and v6. Each of v1, v2, v3, v4, v5, and v6 is a three-dimensional vector. The mesh thinning unit 1151 generates and outputs a base mesh by thinning out the vertices v4, v5, and v6 (FIG. 16B).

[0158] The mesh dividing unit 1152, like the mesh dividing unit 3071, divides the base mesh to generate divided meshes (FIG. 16(c)). v4' = (v1 + v2) / 2 v5' = (v1 + v3) / 2 v6' = (v2 + v3) / 2 The mesh displacement derivation unit derives and outputs the displacements d4, d5, d6 of vertices v4, v5, v6 relative to vertices v4', v5', v6' as mesh displacements based on the mesh and divided meshes (FIG. 16(d)). d4 = v4 - v4' d5 = v5 - v5' d6 = v6 - v6' (Base mesh encoding) FIG. 13 is a functional block diagram showing the configuration of the base mesh encoding unit 103. The base mesh encoding unit 103 is composed of a mesh encoding unit 1031, a mesh decoding unit 1032, a motion information encoding unit 1033, a motion information decoding unit 1034, a mesh motion compensation unit 1035, a reference mesh memory 1036, a switch 1037, and a switch 1038. The base mesh encoding unit 103 may be configured to include a base mesh quantization unit (not shown) after the input of the base mesh. When the base mesh is encoded without referring to other base meshes (for example, an already encoded base mesh) (intra-encoding), the switch 1037 and the switch 3038 are connected to the side that does not perform motion compensation. When the base mesh is encoded with reference to other base meshes (inter-encoding), instead, the switch 1037 and the switch 3038 are connected to the side that performs motion compensation.

[0159] The mesh encoding unit 1031 has an intra-encoding function, intra-encodes the base mesh, and outputs a base mesh encoded stream. Draco or the like is used as the encoding method.

[0160] The mesh decoding unit 1032 is similar to the mesh decoding unit 3031, and therefore a description thereof will be omitted.

[0161] The motion information coding unit 1033 has an inter-coding function, inter-codes the base mesh, and outputs a base mesh coded stream. The coding method used is entropy coding such as arithmetic coding.

[0162] The motion information decoding unit 1034 is similar to the motion information decoding unit 3032, and therefore a description thereof will be omitted.

[0163] The mesh motion compensation unit 1035 is similar to the mesh motion compensation unit 3033, and therefore a description thereof will be omitted.

[0164] The reference mesh memory 1036 is similar to the reference mesh memory 3034, and therefore a description thereof will be omitted.

[0165] (Mesh displacement encoding) 14 is a functional block diagram showing the configuration of the mesh displacement encoding unit 107. The mesh displacement encoding unit 107 is composed of a coordinate system conversion unit 1071, a conversion unit 1072, a quantization unit 1073, and a displacement mapping unit 1074 (image packing unit, displacement encoding unit). As shown in the figure, the mesh displacement encoding unit 107 may further include a video encoding unit 1075. Alternatively, the video encoding unit 1075 may not be included in the mesh displacement encoding unit 107, and an external image encoding device may be used to encode the displacement image.

[0166] The coordinate system conversion unit 1071 converts the coordinate system of the mesh displacement from a Cartesian coordinate system to a coordinate system that encodes the displacement (for example, a local coordinate system) based on the value of the coordinate conversion information displacementCoordinateSystem. Here, disp is a three-dimensional vector indicating the mesh displacement before the coordinate system conversion, d is a three-dimensional vector indicating the mesh displacement after the coordinate system conversion, and n_vec, t_vec, and b_vec are three-dimensional vectors (of the Cartesian coordinate system) indicating each axis of the local coordinate system. if (displacementCoordinateSystem == 0) { d = disp } else if (displacementCoordinateSystem == 1){ d = (disp * n_vec, disp * t_vec, disp * b_vec) } The mesh displacement encoding unit 107 may update the value of the displacementCoordinateSystem at the picture / frame level.

[0167] When coding the displacementCoordinateSystem at the sequence level, the syntax of the configuration in Figure 7 is used. For asps_vdmc_ext_displacement_coordinate_system, 0 is set for the Cartesian coordinate system, and 1 is set for the local coordinate system.

[0168] When changing the displacementCoordinateSystem at the picture / frame level, use the syntax in the configuration of Figure 8. For afps_vdmc_ext_overriden_flag, set 1 if you want to update the coordinate system, or 0 if you do not want to update the coordinate system. For afps_vdmc_ext_displacement_coordinate_system, set 0 if you want a Cartesian coordinate system, or 1 if you want a local coordinate system.

[0169] A transform unit 1072 performs a transform f (eg, a wavelet transform) to derive the transformed mesh displacement Tdisp for pos=0..NumDisp-1, where NumDisp is the number of mesh vertices. Tdisp[0][pos] = f(d[0][pos]) Tdisp[1][pos] = f(d[1][pos]) Tdisp[2][pos] = f(d[2][pos]) The quantization unit 1073 performs quantization based on a quantization scale value "scale" derived from the quantization parameter of each component of the mesh displacement, and derives a mesh displacement Qdisp after quantization. Qdisp[0][pos] = Tdisp[0][pos] / scale[0] Qdisp[1][pos] = Tdisp[1][pos] / scale[1] Qdisp[2][pos] = Tdisp[2][pos] / scale[2] Alternatively, the scale value may be approximated by a power of 2 and Qdisp may be derived using the following formula: scale[i] = 1 << scale2[i] Qdisp[0][pos] = Tdisp[0][pos] >> scale2[0] Qdisp[1][pos] = Tdisp[1][pos] >> scale2[1] Qdisp[2][pos] = Tdisp[2][pos] >> scale2[2] A displacement mapping unit 1074 generates an image gFrame from the quantized mesh displacements Qdisp based on the value of the displacement mapping parameter displacementChromaLocationType.

[0170] The displacement mapper 1074 packs the first element of the (quantized) mesh displacement array, Qdisp[0], into the luma (Y) image component. Do the following for x=0..W-1, y=0..H-1.

[0171] For image width W and height H, (y=0..H-1, x=0..W-1), apply the following, where yc=y / 2 and xc=x / 2. H = dispCompHeight shift = (1 << bitDepth) >> 1 gFrame[0][ y][x] = Qdisp[0][pos] + shift gFrame[0][ H+y][x] = Qdisp[1][pos] + shift gFrame[0][2*H+y][x] = Qdisp[2][pos] + shift pos++ Do the following for xc=0..W / 2-1, yc=0..H / 2-1. gFrame[1][ yc][xc] = shift gFrame[1][H / 2+yc][xc] = shift gFrame[1][ H+yc][xc] = shift gFrame[2][ yc][xc] = shift gFrame[2][H / 2+yc][xc] = shift gFrame[2][ H+yc][xc] = shift Alternatively, you may do the following: for (compIdx = 0; compIdx < 3; compIdx++) { gFrame[0][compIdx*H+y][x] = Qdisp[compIdx][pos] + shift if (compIdx != 0) { gFrame[compIdx][compIdx*H / 2+yc][xc] = shift } } pos++ Alternatively, you may do the following: asps_vdmc_ext_subdivision_iteration_count = lodCount - 1 H = 0 for (i=0; i <lodCount; i++) { H[i] = dispCount[i] / W + 1 asps_vdmc_ext_displacement_component_height_lod[i] = H[i] H += H[i] } shift = (1 << bitDepth) >> 1 y0 = 0 pos0 = 0 for (i=0; i <lodCount; i++) { pos = 0 for (y=0; y <H[i]; y++) { for (x=0; x <W; x++) { if (pos == dispCount[i]) continue gFrame[0][ y0+y][x] = Qdisp[0][pos0+pos] + shift gFrame[0][ H+y0+y][x] = Qdisp[1][pos0+pos] + shift gFrame[0][2*H+y0+y][x] = Qdisp[2][pos0+pos] + shift pos++ } } y0 += H[i] pos0 += dispCount[i] } Alternatively, you may do the following: asps_vdmc_ext_subdivision_iteration_count = lodCount - 1 for (i=0; i <lodCount; i++) { H[i] = (dispCount[i] * 3) / W + 1 asps_vdmc_ext_displacement_component_height_lod[i] = H[i] } shift = (1 << bitDepth) >> 1 y0 = 0 pos0 = 0 for (i=0; i <lodCount; i++) { pos = 0 dim = 0 for (y=0; y <H[i]; y++) { for (x=0; x <W; x++) { if (pos == dispCount[i]) { pos = 0 dim++ } if (dim == 3) continue gFrame[0][y0+y][x] = Qdisp[dim][pos0+pos] + shift pos++ } } y0 += H[i] pos0 += dispCount[i] } The processing may be switched according to DecGeoChromaFormat. That is, when DecGeoChromaFormat=1 (4:2:0), the above processing is performed, and when DecGeoChromaFormat=3 (4:4:4), the following processing is performed. gFrame[0][y][x] = Qdisp[0][pos] gFrame[1][y][x] = Qdisp[1][pos] gFrame[2][y][x] = Qdisp[2][pos] pos=pos+1 The mesh displacement encoding unit 107 may update the values ​​of displacementSliceType and dispCompHeight at the picture / frame level.

[0172] The meaning of the displacementSliceType and dispCompHeight values ​​is as explained in asps_vdmc_ext_displacement_slice_type and asps_vdmc_ext_displacement_component_height.

[0173] When updating displacementSliceType at the picture / frame level, use the syntax of the configuration in Figure 8. Set the value of displacementSliceType to afps_vdmc_ext_displacement_slice_type. Set the value of displacementSliceType to afps_vdmc_ext_displacement_component_height.

[0174] The video encoding unit 1075 encodes the image in YCbCr4:2:0 format including the (quantized) mesh displacement image, and outputs a mesh displacement encoded stream. The encoding method used is VVC, HEVC, or the like.

[0175] The video encoding unit 1075 may divide the mesh displaced image into slices for each dispCompHeight and encode the slices. Also, the dispCompHeight may be aligned to a predetermined size according to the CTU size.

[0176] The video encoding unit 1075 may encode the mesh displacement by assigning a first component (e.g., D) to a first slice, a second component (e.g., U) to a second slice, and a third component (e.g., V) to a third slice (displacementSliceType=1).

[0177] Furthermore, the video encoding unit 1075 may encode the mesh displacement by allocating the first component to the first slice and the second and third components to the second slice (displacementSliceType=2).

[0178] As described above, by assigning different slices to each component of the mesh displacement, the decoding device can decode only some components, so that the processing can be simplified. Also, the decoding device can decode slices including the second and third components of the mesh displacement as necessary, so that the scalability function can be realized. Also, even if an error is mixed into the encoded data, the decoding device can decode only slices (components) without errors, so that the error resistance can be enhanced.

[0179] Although one embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design changes, etc. are possible within the scope that does not deviate from the gist of the present invention.

[0180] [Application example] The above-mentioned 3D data encoding device 11 and 3D data decoding device 31 can be mounted on various devices that transmit, receive, record, and play back 3D data for use. Note that the 3D data may be natural 3D data captured by a camera or the like, or artificial 3D data (including CG and GUI) generated by a computer or the like.

[0181] The present invention is not limited to the above-described embodiment, and various modifications are possible within the scope of the claims. In other words, the technical scope of the present invention also includes embodiments obtained by combining technical means that are appropriately modified within the scope of the claims. [Industrial Applicability]

[0182] The embodiments of the present invention can be suitably applied to a 3D data decoding device that decodes encoded data in which 3D data is encoded, and a 3D data encoding device that generates encoded data in which 3D data is encoded. Also, the embodiments of the present invention can be suitably applied to the data structure of encoded data generated by the 3D data encoding device and referenced by the 3D data decoding device. [Explanation of symbols]

[0183] 11 3D data encoding device 101 Atlas Information Encoding Unit 103 Base mesh coding unit 1031 Mesh coding unit 1032 Mesh Decoding Unit 1033 Motion information encoding unit 1034 Motion information decoding unit 1035 Mesh motion compensation unit 1036 Reference Mesh Memory 1037 Switch 1038 Switch 104 Base mesh decoding unit 106 Mesh displacement update section 107 Mesh Displacement Encoding Unit 1071 Coordinate system conversion section 1072 Converter 1073 Quantization section 1074 Displacement Mapping Section 1075 Video Encoding Unit 108 Mesh Displacement Decoding Unit 109 Mesh reconstruction unit 110 Attribute Update Section 111 Padding section 112 Color space conversion section 113 Attribute Encoding Unit 114 Multiplexer 115 Mesh separation section 1151 Mesh thinning section 1152 Mesh division section 1153 Mesh displacement derivation part 21 Network 31 3D data decoding device 301 Demultiplexer 302 Atlas Information Decoding Unit 303 Base Mesh Decoding Unit 3031 Mesh Decoding Unit 3032 Motion information decoding unit 3033 Mesh Motion Compensation Unit 3034 Reference Mesh Memory 3035 Switch 3036 Switch 305 Mesh Displacement Decoding Unit 3051 Video Decoding Unit 3052 Displacement Unmapping Unit 3053 Inverse quantization section 3054 Reverse conversion unit 3055 Coordinate system conversion section 307 Mesh reconstruction unit 306 Attribute Decoder 3071 Mesh division section 3072 Mesh transformation part 308 Color Space Conversion Unit 41 3D data display device

Claims

1. 1. A 3D data decoding device for decoding encoded data, comprising: a video decoder that decodes a mesh displacement image decoded from a geometry video stream in which the Unit Type of the encoded data is V3C_GVD; a displacement unmapping unit that derives mesh displacement QDisp[compIdx][pos] for each position pos and component compIdx from the mesh displacement image gFrame[compIdx][y][x] having the x, y position and component compIdx, The displacement unmapping unit derives the mesh displacement by deriving the Y coordinate of the mesh displacement image from the product of the variable compIdx, which ranges from 0 to a value -1 indicating the number of dimensions of the geometry, DisplacementDim, and the height dispCompHeight, when the geometry image is 420.

2. 2. The 3D data decoding device according to claim 1, wherein the displacement unmapper decodes a syntax element indicating whether or not a slice is used from the encoded data.

3. The 3D data decoding device according to claim 1, characterized in that the displacement unmapper derives the value of the height dispCompHeight according to a syntax element indicating whether or not a slice is used.

4. The 3D data decoding device according to claim 1, characterized in that the displacement umapping unit derives the value of the height dispCompHeight to be a constant multiple of a predetermined value.

5. The 3D data decoding device according to claim 4, characterized in that the displacement unmapping unit derives the value of the height dispCompHeight according to the height of a CTU of a geometry video stream.

6. The 3D data decoding device according to claim 1, characterized in that, when a syntax element indicating whether to use segment-based encoding such as slices indicates that segments are to be used, the 3D data decoding device decodes a geometry video stream encoded using segments in units of mesh image components compIdx.

7. The 3D data decoding device according to claim 1, characterized in that, when one-dimensionally encoded, the 3D data decoding device always decodes a geometry video stream encoded using segments in units of mesh image components compIdx.

8. The 3D data decoding device described in claim 1, characterized in that when the 3D data decoding device is one-dimensionally encoded and encoded in 420, it always decodes a geometry video stream encoded using segments in units of mesh image components compIdx.

9. In a 3D data encoding device that decodes encoded data, a video encoding unit that encodes a mesh displacement image to encode a geometry video stream in which the Unit Type of the encoded data is V3C_GVD; a displacement mapping unit for deriving a mesh displacement image gFrame[compIdx][y][x] having an x, y position and a component compIdx from the mesh displacement QDisp[compIdx][pos]; The displacement mapping unit derives the geometry image by deriving the Y coordinate of the geometry image from the product of the variable compIdx, which ranges from 0 to a value -1 indicating the number of dimensions of the geometry, DisplacementDim, and the height dispCompHeight, when the geometry image is 420.

Images