V3C syntax extension for mesh compression

Abstract

In a current implementation of V-DMC, (u, v) coordinates are generated using Microsoft UVAtlas, and are carried in a base mesh bitstream together with 3D positions and topologies. The high-level syntactic structures described herein support projection-based atlas map generation, as well as means of deriving (u, v) coordinates on the decoder side using V3C syntactic structure extensions. In comparison to previous implementations, and in order to preserve the current V3C geometry bitstream concept, a separate sub-bitstream, referred to herein as a vertex attribute sub-bitstream, is used to carry displacement information.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority under 35 USC §119(e) to U.S. Provisional Patent Application Serial No. 63 / 380,087, filed on October 19, 2022, entitled “V3C SYNTAX EXTENSION FOR MESH COMPRESSION,” which is incorporated herein by reference in its entirety for all purposes. Technical Field

[0003] The present invention relates to three-dimensional graphics. More particularly, the present invention relates to mesh compression. Background Art

[0004] A novel method for compressing volumetric content (such as point clouds) based on projection from 3D to 2D is being standardized recently. The method, also called V3C (Visual Volumetric Video-based Compression), maps the 3D volume data into several 2D tiles, which are then further arranged into an atlas image, which is then encoded with a video encoder. The atlas image corresponds to the geometry of the points, the corresponding texture, and an occupancy map, which indicates which locations will be considered for point cloud reconstruction.

[0005] In 2017, MPEG issued a call for proposals (CfP) for point cloud compression. After evaluating several proposals, MPEG is currently considering two different techniques for point cloud compression: 3D native coding techniques (based on octrees and similar coding methods), or 3D to 2D projection and traditional video coding afterwards. In the case of dynamic 3D scenes, MPEG is using test model software (TMC2) based on tile surface modeling, tile projection from 3D to 2D images, and encoding of 2D images with video encoders such as HEVC. This approach has been shown to be more efficient than native 3D coding and can achieve competitive bit rates with acceptable quality.

[0006] Due to the success of projection-based methods (also called video-based methods or V-PCC) for encoding 3D point clouds, it is expected that the standard will include additional 3D data, such as 3D meshes, in the future. However, the current version of the standard is only suitable for the transmission of non-connected sets of points, so there is no mechanism to transmit the connectivity of the points required in 3D mesh compression.

[0007] Methods have been proposed to extend the functionality of V-PCC to meshes as well. One possible approach is to encode the vertices using V-PCC and then encode the connectivity using a mesh compression method (such as TFAN or Edgebreaker). The limitation of this method is that the original mesh must be dense, so that the point cloud generated from the vertices is not sparse and can be efficiently encoded after projection. In addition, the order of vertices after connectivity encoding and different methods for reorganizing mesh connectivity are proposed. An alternative way to encode sparse meshes is to encode the vertex positions in 3D using RAW (unprocessed) tile data. Since RAW tiles encode (x, y, z) directly, in this method, all vertices are encoded as RAW data, while connectivity is encoded using a similar mesh compression method as mentioned above. It should be noted that in the RAW tiles, the vertices can be sent in any preferred order, so the order generated from the connectivity encoding can be used. The method can encode sparse point clouds, but RAW tiles are not efficient for encoding 3D data, and additional data such as properties of triangular surfaces may be missing from this method.

[0008] UVAtlas from Microsoft is a state-of-the-art automatic texture map generation, but it takes a lot of time and is only optimized for local frames. V-PCC generates patches using orthographic projection, but only for point clouds, so it does not solve patch generation for meshes. Summary of the invention

[0009] In the current implementation of V-DMC, (u,v) coordinates are generated using Microsoft UVAtlas and carried in the base mesh sub-bitstream along with the 3D position and topology. The high-level syntax structure described herein supports projection-based atlas mapping generation, as well as a means of deriving (u,v) coordinates on the decoder side using the V3C syntax structure extension. Compared to previous implementations, and in order to preserve the current V3C geometry bitstream concept, a separate sub-bitstream referred to herein as the vertex attribute sub-bitstream is used to carry the displacement information.

[0010] In one aspect, a method programmed in a non-transitory memory of a device includes implementing an atlas mapping process, implementing a connectivity process to partition a triangle to generate vertex information, and implementing a vertex position process to adjust vertex positions in the vertex information. The method also includes implementing a displacement sub-bitstream to include displacement information, wherein the displacement information is utilized by the vertex position process. The method also includes reconstructing a mesh based on the displaced vertices and connectivity information. The atlas mapping process includes receiving tile identification information and mapping function parameters, and generating (u,v) coordinates based on the tile identification and mapping function parameters. The method also includes utilizing one or more flags to bypass one or more V-DMC methods and using a texture feature map video as a V3C feature component video. Implementing the atlas mapping process occurs before implementing the connectivity process and the vertex position process. Implementing the atlas mapping process occurs after implementing the connectivity process and the vertex position process.

[0011] In another aspect, an apparatus includes a non-transitory memory for storing an application, the application being for implementing an atlas mapping process, performing connectivity processing to partition triangles to generate vertex information, and performing vertex position processing to adjust vertex positions in the vertex information; and a processor coupled to the memory, the processor being configured to process the application. The application is further configured to implement a displacement sub-bitstream to include displacement information, wherein the displacement information is utilized by the vertex position processing. The application is further configured to reconstruct a mesh based on the displaced vertices and connectivity information. The atlas mapping process includes receiving tile identification information and mapping function parameters, and generating (u,v) coordinates based on the tile identification and mapping function parameters. The application is further configured to utilize one or more flags to bypass one or more V-DMC methods, and use a texture property map video as a V3C property component video. Implementing the atlas mapping process occurs before implementing the connectivity processing and the vertex position processing. Implementing the atlas mapping process occurs after implementing the connectivity processing and the vertex position processing.

[0012] In another aspect, a system includes: an encoder configured to encode a 3D mesh to generate tile identification information and mapping function parameters; and a decoder configured to: implement atlas mapping processing on the tile identification information and mapping function parameters, implement connectivity processing to partition triangles to generate vertex information, and implement vertex position processing to adjust vertex positions in the vertex information. The decoder is also used to: implement a displacement sub-bitstream to include displacement information, wherein the displacement information is utilized by the vertex position processing. The decoder is also used to: reconstruct the mesh based on the displaced vertices and connectivity information. The atlas mapping processing includes: receiving the tile identification information and mapping function parameters, and generating (u, v) coordinates based on the tile identification and mapping function parameters. The decoder is also used to: utilize one or more flags to bypass one or more V-DMC methods, and use the texture characteristic map video as the V3C characteristic component video. Implementing the atlas mapping processing occurs before implementing the connectivity processing and the vertex position processing. Implementing the atlas mapping processing occurs after implementing the connectivity processing and the vertex position processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 A diagram showing a decoding implementation according to some embodiments.

[0014] Figure 2 A diagram showing a decoding implementation according to some embodiments.

[0015] Figure 3 A diagram showing a modified V-DMC standard according to some embodiments.

[0016] Figure 4 A block diagram of an exemplary computing device configured to implement a mesh compression method according to some embodiments is shown. DETAILED DESCRIPTION

[0017] A mesh consists of a collection of polygons that generally describe the surface of a volume. An efficient way to describe the surface properties of a mesh (e.g., its color characteristics) is to generate a texture atlas that maps the properties of the 3D surface to a 2D surface. The result of the mapping function is stored in (u, v) coordinates and added to the mesh data, which is then further encoded using a mesh compression method. However, the presence of (u, v) coordinates may significantly increase the size of the compressed mesh.

[0018] In the latest international point cloud compression standard V-PCC, orthographic projection is being used to generate depth map images for point clouds. The parameters of the projection are then encoded in a metadata bitstream called an atlas bitstream, so that the decoder only needs to receive these parameters and apply a mapping function to each (u,v) coordinate of the depth map to subsequently reconstruct the 3D information.

[0019] The present invention describes extending the current V3C standard to include syntax elements that allow transmission of parameters for generating mapping coordinates for a grid using orthographic projection. The new syntax elements define a displacement video sub-bitstream and retain the original definition of the geometric image already defined in the V3C standard. The displacement of a point (e.g., the movement of a point in 3D space) is not necessarily in the direction of projection, and a new bitstream (replacing the geometry bitstream) can be used to indicate the displacement. Also described is a method for defining the relationship between tiles and faces to be encoded with a general grid encoder and indicating appropriate signaling.

[0020] The new VDMC standard for compression of dynamic meshes will be a future extension of the V3C standard, which is the current international standard for video-based volumetric coding. The new syntax disclosed herein addresses some key new technologies for efficient coding, such as the use of texture mapping and displacement images, and also integrates this new technology while retaining the existing V3C syntax. The disclosed syntax allows the derivation of atlas mapping coordinates on the decoder side and better integrates with the current V3C syntax elements currently used for point cloud coding.

[0021] New syntax elements allow passing orthoAtlas parameters to the encoder: syntax for texture map generation using orthographic projection.

[0022] Flexibility in signaling projection-based atlas parameterizations (e.g., "orthoAtlas") includes description and use of V3C syntax structure and extensions to allow (u,v) coordinates to be derived on the decoder side, rather than carried in the base grid sub-bitstream as is currently practiced, and the V3C tile header syntax remains unchanged by assuming a one-to-one correspondence between a single tile and a single sub-grid, making the tile header syntax structure identical to the V3C tile header syntax. Additionally, signaling of the association between tiles and sub-grids is implemented as part of the afps extension. Use of new tile types is similar to those introduced previously (e.g., I_MESH), but with minimal modifications to the original V3C tile data type information.

[0023] Compared to the previous approach of using the geometry sub-bitstream to carry the displacement information, the geometry and displacement sub-bitstreams are separated and clearly distinguished. Instead, the displacement sub-bitstream is made a separate sub-bitstream. In this way, the geometry sub-bitstream is semantically consistent with the current V3C geometry concept and has the potential for possible enhancements to the surface reconstruction process.

[0024] A generic module for "connectivity" processing is introduced, one example of which is the "tessellation" method. A number of "flags" can provide options to use / bypass specific V-DMC methods. Texture property map video can be treated as V3C property component video. V3C atlas tile segmentation information and tile information data can be used instead of grid-specific tile segmentation as previously described.

[0025] The association between patchld and facegroupld is signaled in the mesh bitstream as a (patchld, facegroupld) pair, allowing a more generic and not necessarily Draco-specific signaling approach.

[0026] Figure 1 A diagram showing a decoding implementation according to some embodiments is shown. A V3C sub-bitstream is received. For example, a base grid sub-bitstream 100, an atlas sub-bitstream 102, a displacement sub-bitstream 104, a characteristic sub-bitstream 106, and a geometry sub-bitstream 108 are received. The sub-bitstreams are decoded by a grid codec 110, a first video codec 112, a second video codec 114, or a third video codec 116. For example, the base grid sub-bitstream 100 is decoded by the grid codec 110; the displacement sub-bitstream 104 is decoded by the first video codec 112; and the characteristic sub-bitstream 106 is decoded by the second video codec 114. The displacement sub-bitstream 104 is decoded by the video codec 112 and the displacement decoder 122.

[0027] The trellis codec 110 decodes the base trellis sub-bitstream 100 and outputs a decoded base trellis 120 including a tile ID.

[0028] The atlas mapping process 130 is an additional implementation. The atlas mapping process 130 receives (patchID, 0) information based on the decoded base grid. The atlas mapping process 130 also receives parameters (eg, C, P) from the atlas sub-bitstream 102 in the form of tiles to regenerate u,v coordinates.

[0029] The generated u,v coordinates are used in connectivity processing 132 and vertex processing 134. Surface processing 140 is also implemented to generate bumps or other features on the surface. Based on the processing, the object / mesh is reconstructed 136. In addition, a texture map 138 is reconstructed based on the decoded characteristic sub-bitstream 106.

[0030] Figure 2 A diagram showing a decoding implementation according to some embodiments. The implementation is similar to Figure 1; however, in this implementation, the atlas mapping process 130 is performed after the vertex position process 134. The (u,v) coordinate generation is performed in the fragment shading. Since the mesh has been subdivided, more (u,v) coordinates will be generated.

[0031] Figure 3 A diagram of a modified V-DMC standard according to some embodiments is shown. Instead of receiving (u,v) coordinates, simplified information ((x,y,z) and (v1,v2,v3)) is received and an atlas mapping process 130 generates (u,v) coordinates. The simplified information and (u,v) coordinates enter a connectivity process 132 which partitions the triangles to generate vertex information. A vertex position process 134 adjusts the vertex positions in the vertex information. For example, gray dots 300 represent original vertex positions and black dots 302 represent adjusted vertex positions. Subsequently, the displaced vertices with connectivity are used to reconstruct a mesh 304. Texture transfer 306 also occurs.

[0032] V-DMC High-Level Syntax and Semantics

[0033] V3C Unit

[0034]

[0035] vuh_vertex_property_index represents the index of the vertex attribute data carried in the vertex attribute data unit. The value of vuh_vertex_property_index shall be in the range of 0 to (vps_vdmc_vertex_property_video_count[vuh_atlas_id]-1), including the boundary.

[0036] When present, vuh_reserved_zero_10bits shall be equal to 0 in bitstreams conforming to this version of this document. Other values ​​of vuh_reserved_zero_10bits are reserved for future use by ISO / IEC. A decoder shall ignore the value of vuh_reserved_zero_10bits.

[0037]

[0038] V3C parameter set extension

[0039] As a basis, the previously introduced extended syntactic structure is used, but some syntactic elements are added and removed as indicated below.

[0040]

[0041]

[0042] vps_dmc_mesh_data_substream_codec_id represents the identifier of the codec used to compress the base mesh data. The codec may be identified through the profile and component codec mapping SEI message or through another implementation manner.

[0043] vps_vdmc_mesh_data_facegroup_id_property_present_flag equal to 1 indicates that one of the property types present in the base mesh data stream is face group ID.

[0044] vps_vdmc_mesh_property_count represents the number of properties (e.g., textures, normals, other) in the base mesh sub-bitstream. vps_ext_mesh_property shall be in the range of 0 to 127, inclusive.

[0045] vps_vdmc_mesh_property_type[j][i] represents the type of the i-th property present in the mesh from the atlas with atlasID equal to j.

[0046]

[0047] vps_vmc_ext_displacement_video_present_flag[j] is equal to 1, indicating that the type of at least one of the vertex attribute video data is DISPLACEMENT.

[0048] vps_vmc_ext_vertex_property_video_count[j] represents the number of vertex attributes associated with the atlas with atlas ID j. vps_vmc_ext_vertex_property_video_count[j] shall be in the range of 0 to 127, inclusive.

[0049] vps_ext_vertex_property_type[j][i] represents the vertex attribute type of the vertex attribute video data unit with index i for the atlas with atlas ID j. The following table describes the list of supported vertex attributes and their relationship with vps_ext_vertex_property_type[j][i].

[0050]

[0051] di_displacement_codec_id[j][i] represents the identifier of the codec used to compress the vertex attribute video data unit with index i for the atlas with atlas ID j. di_displacement_codec_id[j][i] shall be in the range of 0 to 255, inclusive.

[0052] di_displacement_prevent_video_conversion_flag[j][i] prevents conversion of the output of the vertex attribute video data unit substream decoder with index i from atlas with atlas ID j. When the flag is true, the output is used as is without any conversion processing from Appendix B in [5]. When the flag is true, the size of the displacement video will be the same as the nominal video size indicated in the bitstream.

[0053] di_displacement_2d_bit_depth_minus1[j][1] plus 1 represents the nominal 2D bit depth to which the vertex attribute video data unit with index i for the atlas with atlas ID j will be converted. di_displacement_2d_bit_depth_minus1[j] shall be in the range of 0 to 31, inclusive.

[0054] di_displacement_MSB_align_flag[j][i] indicates how the decoded displacement video samples associated with the vertex attribute video data unit with index i for atlas ID j are converted to samples at the nominal displacement bit depth.

[0055] di_displacement_frame_width[j][i] represents the frame width of the displacement video for the vertex attribute video data unit with index i and atlas ID atlasID.

[0056] di_displacement_frame_height[j][i] represents the frame height of the displacement video for the vertex attribute video data unit with index i and atlas ID atlasID.

[0057] Atlas sequence parameter set extension

[0058]

[0059] asps_vmc_ext_atlas_mapping_present_flag equal to 1 indicates that the atlas_map_processing_information() syntax structure is present. asps_vmc_ext_atlas_mapping_present_flag equal to 0 indicates that the atlas_map_processing_information() syntax structure is not present.

[0060] asps_vmc_ext_connectivity_present_flag equal to 1 indicates that the connectivity_processing_information() syntax structure is present. asps_vmc_ext_connectivity_present_flag equal to 0 indicates that the connectivity_processing_information() syntax structure is not present.

[0061] asps_vmc_ext_displacement_present_flag equal to 1 indicates that the displacement_processing_information() syntax structure is present. asps_vmc_ext_displacement_present_flag equal to 0 indicates that the displacement_processing_information() syntax structure is not present.

[0062] asps_vcm_ext_log2_displacement_packing_block_size[j][i] indicates that the value of the variable DisplacementPackingBlockSize used for the horizontal and vertical placement of the transformed displacement vector within the vertex attribute video data unit with index i for atlas ID j is as follows:

[0063] DsplacementPackingBlockSize=2 di_log2_displacement_packing_block_size[j][i] The value of di_log2_displacement_packing_block_size[j][i] shall be in the range of 0 to 7, inclusive.

[0064] asps_vmc_ext_num_displacement_video indicates the number of displacement videos. If vps_vmc_ext_displacement_video_present_flag is equal to 1, the default value of asps_vmc_ext_num_displacement_video is 1.

[0065] asps_vmc_ext_property_index_to_displacement_index[i] is a structure that maps a vertex attribute video data index to a displacement with index i.

[0066] asps_vmc_ext_patch_mapping_method indicates how to map the face group ID of the sub-mesh to the patch.

[0067] When asps_vmc_ext_patch_mapping_method is equal to 0, all triangles in the corresponding submesh are associated with the current tile. In this case, only one tile is associated with the submesh.

[0068] When asps_vmc_ext_patch_mapping_method is equal to 1, the association of the surface group ID and the tile ID is explicitly signaled in the base mesh data unit.

[0069] When asps_vmc_ext_patch_mapping_method is equal to 2, the association of the plane group ID and the tile ID is explicitly signaled in the mesh patch data unit.

[0070] In other cases, the triangle faces in the corresponding sub-mesh are divided into face groups using the method indicated by asps_vmc_ext_patch_mapping_method.

[0071]

[0072] aspc_vmc_ext_patch_mesh_data_enable_flag equal to 1 indicates that the mesh information (i.e., number of triangles, number of vertices, 3D border) of the tile with index i in the frame with index j will be indicated in the bitstream for each tile data unit. If aspc_vmc_ext_patch_mesh_data_enable_flag is equal to 0, the mesh information will be obtained by processing the information in the basic mesh data unit.

[0073]

[0074] ampi_projection_enabled_flag indicates that the 2D position to which the feature is projected is explicitly signaled in the grid patch data unit. Therefore, the projection id and pointing index discussed previously can also be used.

[0075] ampi_atlas_mapping_method indicates the type of atlas mapping method used. When the signaled ampi_atlas_mapping_method is equal to 0, it indicates that the 2d (u, v) positions of the orthographic projection of the 3d vertices are derived at the decoder side instead of being signaled in the mesh bitstream.

[0076] ampi_atlas_mapping_method The name of the atlas mapping method 0 orthoAtlas

[0077] ampi_ortho_atlas_gutter: Guard space between patches during patch packing process when using "orthoAtlas".

[0078] ampi_ortho_atlas_lod_width_scale: Width scaling factor used to pack tiles in all frames in a sequence. If not present, 1.0 should be used.

[0079] ampi_ortho_atlas_height_scale: Height scaling factor used for packing tiles in all frames in the sequence. If not present, 1.0 should be used.

[0080]

[0081] cpi_connectivity_method indicates the type of connectivity method used.

[0082] cpi_connectivity_method The name of the connectivity method

[0083] 0 SUBDIVISION

[0084] cpi_subdivision_method indicates the subdivision method.

[0085] cpi_subdivision_method The name of the subdivision method

[0086] 0 MIDPOINT

[0087] cpi_subdivision_iteration_count indicates the number of times subdivision needs to be recursively applied.

[0088] cpi_tjunction_removing_method represents the method used to remove T-junctions generated by different subdivision methods or by different subdivision iterations of two triangles that share an edge.

[0089]

[0090] dppi_displacement_coordinate_system[ltpIndex] equal to 0 indicates that the global coordinate system will be used. dppi_displacement_coordinate_system[ltpIndex] equal to 0 indicates that the transformation to the local coordinate system will be used.

[0091] dp4pi_transform_index[lptIndex][i] indicates the transform applied to the displacement video with index i. The transform index may indicate that no transform is applied. When the transform is "LINEAR_LIFTING", the necessary parameters are signaled as "vmc_lifting_transform_parameters".

[0092] vppi_displacement_transform_index The name of the transformation method

[0093] 0 NONE

[0094] 1 LINEAR_LIFTING

[0095] dppi_extension_transform_parameters_present_flag[lptIndex][i] equal to 1 indicates that there is a vmc_lifting_transform_parameters() syntax structure for the displacement video with index i. dppi_extension_transform_parameters_present_flag equal to 0 indicates that there is no vmc_lifting_transform_parameters() syntax structure.

[0096]

[0097] ltpIndex represents the system level, where level 0 is for the entire sequence, level 1 is for frames, and level 2 is for tiles.

[0098] Atlas Frame Parameter Set Extension

[0099]

[0100]

[0101]

[0102] afps_num_submesh indicates the number of submeshes in the grid. In the current high-level syntax, it is assumed to be equal to afti_num_tiles_in_atlas_frame_minus1+1.

[0103] afps_submesh_id[i] represents the ID associated with the submesh with index 1.

[0104] afps_vmc_ext_single_submesh_in_frame_flag indicates that there is only one submesh for a mesh frame.

[0105] afps_vmc_ext_overriden_flag When true, the subdivision method, displacement coordinate system, transform indices, transform parameters and characteristic transform parameters override those signaled in asps_vmc_extension().

[0106] afps_vmc_ext_single_attribute_tile_in_frame_flag indicates that there is only one tile for each attribute signaled in the video stream.

[0107] It should be noted that in the case of mapped features, the previously specified feature tile information can use atlas_frame_tile_information.

[0108]

[0109]

[0110] Patch Data Unit

[0111]

[0112]

[0113]

[0114] mdu_mesh_present_flag[tileID][p] equal to 1 indicates that there are mesh-related syntax elements in tile p in the current atlas tile (tile ID equal to tileID). mdu_mesh_present_flag equal to 0 indicates that there are no mesh-related syntax elements.

[0115] mdu_vertex_count_minus1[tileID][p] plus 1 indicates the number of vertices in the tile with index p of the current atlas tile (tile ID equals tileID). If not present, this value will be determined by processing the base mesh data unit.

[0116] mdu_triangle_count_minus1[tileID][p] plus 1 indicates the number of triangles in the tile with index p of the current atlas tile (tile ID equals tileID). If not present, this value will be determined by processing the base mesh data unit.

[0117] mdu_3d_bounding_box_min[tileID][p][i] indicates the minimum value of the bounding box of the tile with index p of the current atlas tile (tile ID equals tileID) along the i-th axis. The value of mdu_3d_bounding_box_min[tileID][p][i] will be between 0 and 2. asps_geometry_3d_bit_depth_minus1+1 -1, inclusive. The number of bits used to represent mdu_3d_bounding_box_min[tileID][p] is asps_geometry_3d_bit_depth_minus1+1. If not present, the value is determined by processing the underlying mesh data unit.

[0118] mdu_3d_bounding_box_max[tileID][p][i] indicates the maximum value of the bounding box of the tile with index p of the current atlas tile (tile ID equals tileID) along the i-th axis. mdu_3d_bounding_box_min[tileID][p][i] will have a value between 0 and 2. asps_geometry_3d_bit_depth_minus1+1 -1, inclusive. The number of bits used to represent mdu_3d_bounding_box_max[tileID][p] is asps_geometry_3d_bit_depth_minus1 + 1. If not present, the value is determined by processing the underlying mesh data unit.

[0119] mdu_2d_pos_x[tileID][p] indicates the x coordinate of the upper left corner of the tile bounding box size of tile p in the current atlas tile (tile ID equals tileID) expressed as a multiple of PatchPackingBlockSize.

[0120] mdu_2d_pos_y[tileID][p] indicates the y coordinate of the upper left corner of the tile border size of tile p in the current atlas tile (tile ID equals tileID) expressed as a multiple of PatchPackingBlockSize.

[0121] mdu_2d_size_x_minus1[tileID][p] plus 1 indicates the width value of the tile with index p in the current atlas tile (tile ID equals tileID).

[0122] mdu_2d_size_y_minus1[tileID][p] plus 1 indicates the height value of the tile with index p in the current atlas tile (tile ID equals tileID).

[0123] mdu_projection_id[tileID][p] indicates the projection mode and the index of the normal of the projection plane for the tile with index p of the current atlas tile (tile ID equals tileID). The value of mdu_projection_id[tileID][p] shall be in the range of 0 to asps_max_number_projections_minus1, inclusive.

[0124] The number of bits used to represent mdu_projection_id[tileID][p] is Ceil(Log2(asps_max_number_projections_minus1+1)).

[0125] mdu_orientation_index[tileID][p] indicates the tile orientation index of the tile with index p of the current atlas tile (tile ID equal to tileID) used to determine the matrices P, R and O as shown in equation (1) and Table 1, respectively, for reconstruction.

[0126] mdu_lod_enabled_flag[tileID][p] equal to 1 indicates that LOD parameters exist for the current tile p of the current atlas tile (tile ID equals tileID). If mdu_lod_enabled_flag[tileID][p] is equal to 0, then LOD parameters do not exist for the current tile. If mdu_lod_enabled_flag[tileID][p] does not exist, its value is inferred to be equal to 0.

[0127] mdu_lod_scale_x_minus1[tileID][p] indicates the LOD scaling factor to be applied to the local x coordinate of the point in the tile with index p of the current atlas tile (tile ID equal to tileID) before being added to the tile coordinate TilePatch3dOffsetU[tileID][p]. If mdu_lod_scale_x_minus1[tileID][p] is not present, its value is inferred to be equal to 0.

[0128] mdu_lod_scale_y_idc[tileID][p] indicates the LOD scaling factor to be applied to the local y coordinate of the point in the tile with index p of the current atlas tile (tile ID equal to tileID) before being added to the tile coordinate TilePatch3dOffsetV[tileID][p]. If mdu_lod_scale_y_idc[tileID][p] is not present, its value is inferred to be equal to 0.

[0129] mesh_inter_data_unit(tileID,patchIdx), mesh_merge_data_unit(tileID,patchIdx) and mesh_skip_data_unit(tileID,patchIdx) can be easily defined. But the group should further discuss the use of mesh_raw_data_unit(tileID,patchIdx). It should be noted that if tileId=submeshID is used (with the same independent decoding function), there is no need to send different IDs to associate tiles to sub-meshes, and tileID can be used for this purpose.

[0130] reconstruction

[0131] For generating the connectivity associated with a patch projected with an orthoAtlas, the following applies:

[0132] Assign the variables pIdx, oIdx, lodX, and lodY as follows:

[0133] occRes = PatchPackingBlockSize

[0134] U0 = TilePatch2dPosX[tileID][p] / occRes

[0135] V0 = TilePatch2dPosY[tileID][p] / occRes

[0136] widthOccCC = TilePatch2dSizeX[tileID][p] / occRes

[0137] heightOccCC = TilePatch2dSizeY[tileID][p] / occRes

[0138] lodX = TilePatchLoDScaleX[tileID][p] * ampi_ortho_atlas_width_scale

[0139] lodY = TilePatchLoDScaleY[tileID][p] * ampi_ortho_atlas_height_scale

[0140] pID = TilePatchProjectionID[tileID][p]

[0141] oIdx = TilePatchOrientationIndex[tileID][p]

[0142] width = asps_frame_width

[0143] height = asps_frame_height

[0144] Assign the variable BB as follows min (x), Bbmin (y), BB min (z), BB max (x), BB max (y), BB max (y):

[0145] BB min BB(x) = TilePatch3dBoundingBoxMin[tileID][p][0]

[0146] BB min(y)=TilePatch3dBoundingBoxMin[tileID][p][1]

[0147] BB min (z)=TilePatch3dBoundingBoxMin[tileID][p][2]

[0148] BB max (x)=TilePatch3dBoundingBoxMin[tileID][p][0]

[0149] BB max (y)=TilePatch3dBoundingBoxMin[tileID][p][1]

[0150] BB max (z)=TilePatch3dBoundingBoxN-in[tileID][p][2]

[0151] Then, for each sample with coordinates (x, y, z) belonging to the reconstructed vertex position, we transform (x, y, z) to a local patch coordinate pair (u * , v * ) to implement the transformation to the 2D u,v coordinate system:

[0152]

[0153] in:

[0154]

[0155] Then, the final (u,v) coordinates are obtained using the following formula:

[0156]

[0157] in:

[0158]

[0159]

[0160] Table 1

[0161] Figure 4A block diagram of an exemplary computing device configured to implement a grid compression method according to some embodiments is shown. Computing device 400 can be used to obtain, store, calculate, process, convey and / or display information such as images and videos including 3D content. Computing device 400 can implement any encoding / decoding aspects. In general, a hardware structure suitable for implementing computing device 400 includes a network interface 402, a memory 404, a processor 406, (multiple) I / O devices 408, a bus 410 and a storage device 412. As long as a suitable processor with sufficient speed is selected, the selection of the processor is not critical. Memory 404 can be any conventional computer memory known in the art. Storage device 412 can include a hard disk drive, CDROM, CDRW, DVD, DVDRW, high-definition disk / drive, ultra-HD drive, flash memory card or any other storage device. Computing device 400 can include one or more network interfaces 402. Examples of network interfaces include network cards connected to Ethernet or other types of LANs. I / O device(s) 408 can include one or more of the following: keyboard, mouse, monitor, screen, printer, modem, touch screen, button interface, and other devices. Mesh compression application(s) 430 used to implement the mesh compression implementation may be stored in storage device 412 and memory 404 and processed as applications are typically processed. Figure 4 In some embodiments, mesh compression hardware 420 is included. Although Figure 4 The computing device 400 in the embodiment includes an application 430 and hardware 420 for a mesh compression implementation, but the mesh compression method can be implemented on the computing device in hardware, firmware, software, or any combination thereof. For example, in some embodiments, the mesh compression application 430 is programmed in a memory and executed using a processor. In another example, in some embodiments, the mesh compression hardware 420 is a programmed hardware logic including gates specifically designed to implement the mesh compression method.

[0162] In some embodiments, (multiple) grid compression applications 430 include several applications and / or modules. In some embodiments, each module also includes one or more sub-modules. In some embodiments, fewer or additional modules can be included.

[0163] Examples of suitable computing devices include a personal computer, a laptop computer, a computer workstation, a server, a mainframe computer, a handheld computer, a personal digital assistant, a cellular / mobile phone, a smart appliance, a game console, a digital camera, a digital camcorder, a camera phone, a smart phone, a portable music player, a tablet computer, a mobile device, a video player, a video disc writer / player (e.g., a DVD writer / player, a high definition disc writer / player, an ultra high definition disc writer / player), a television, a home entertainment system, an augmented reality device, a virtual reality device, smart jewelry (e.g., a smart watch), a vehicle (e.g., an autonomous vehicle), or any other suitable computing device.

[0164] In order to utilize the mesh compression method, a device acquires or receives 3D content (eg, point cloud content). The mesh compression method can be implemented with the assistance of a user, or automatically implemented without user involvement.

[0165] In operation, the mesh compression method provides flexibility for signal projection based atlas parameterization methods (such as "orthoAtlas"), including:

[0166] Description and use of V3C syntax structures and extensions to allow (u,v) coordinates to be derived at the decoder side, rather than carried in the base grid sub-bitstream as is currently practiced;

[0167] The V3C tile header syntax remains unchanged by the one-to-one correspondence between a single tile and a single subgrid. This change leaves the V3C tile header syntax structure unchanged compared to previous implementations. The signaling of the association between tiles and subgrids is implemented as part of the afps extension.

[0168] Use V3C atlas tile segmentation information instead of its extension;

[0169] The usage of the new tile types is similar to those introduced previously (eg I_MESH), but with minimal modifications to the original V3C tile data type syntax.

[0170] Compared to the previous approach of using a geometry sub-bitstream to carry displacement information, the mesh compression method separates and makes a clear distinction between the geometry and displacement sub-bitstreams. As described herein, the displacement is carried in the vertex attribute sub-bitstream. The introduction of such a separate sub-bitstream leads to various future vertex processing operations (such as mesh tracking and vertex correspondence). In addition, with this change, the geometry sub-bitstream is semantically consistent with the current V3C geometry concept, providing us with the potential for possible enhancements to the surface reconstruction process.

[0171] Mesh compression methods introduce a common module for "connectivity" processing, one example of which is the "subdivision" method.

[0172] Instead of using Darco specific "properties" items, as is done for example in the "ply" file format, use generic "attributes" to name all data associated with a mesh / submesh.

[0173] A number of "flags" are used to provide options to use / bypass specific V-DMC methods, and also to treat texture feature map video as V3C feature component video.

[0174] The mesh compression method implements the signaling of associations between patchId and facegroupId as (patchId, facegroupId) pairs in the mesh bitstream, allowing for a more generic rather than Draco-specific signaling method.

[0175] Some embodiments of V3C syntax extensions for mesh compression

[0176] 1. A method of programming in a non-transitory memory of a device, comprising:

[0177] Implement atlas mapping processing;

[0178] Performing connectivity processing to partition the triangles to generate vertex information; and

[0179] Vertex position processing is performed to adjust the vertex positions in the vertex information.

[0180] 2. The method of clause 1, further comprising: implementing a displacement sub-bitstream to include displacement information, wherein the displacement information is utilized by vertex position processing.

[0181] 3. The method of item 2 further includes: reconstructing the mesh based on the displaced vertices and connectivity information.

[0182] 4. The method of clause 1, wherein the atlas mapping process comprises:

[0183] receiving patch identification information and mapping function parameters; and

[0184] Generate (u,v) coordinates based on the patch identifier and mapping function parameters.

[0185] 5. The method of clause 1, further comprising: utilizing one or more flags to bypass one or more V-DMC methods, and using the texture characteristic map video as the V3C characteristic component video.

[0186] 6. The method of clause 1, wherein implementing atlas mapping processing occurs before implementing connectivity processing and vertex position processing.

[0187] 7. The method of clause 1, wherein implementing atlas mapping processing occurs after implementing connectivity processing and vertex position processing.

[0188] 8. A device comprising:

[0189] A non-transitory memory for storing applications for:

[0190] Implement atlas mapping processing;

[0191] Performing connectivity processing to partition the triangles to generate vertex information; and

[0192] performing vertex position processing to adjust the vertex positions in the vertex information; and

[0193] A processor is coupled to the memory, the processor being configured to process the application.

[0194] 9. The apparatus of clause 8, wherein the application is further used to: implement a displacement sub-bitstream to include displacement information, wherein the displacement information is utilized by vertex position processing.

[0195] 10. The apparatus of clause 9, wherein the application is further used to: reconstruct a mesh based on the displaced vertices and connectivity information.

[0196] 11. The device of clause 8, wherein the atlas mapping process comprises:

[0197] receiving patch identification information and mapping function parameters; and

[0198] Generate (u,v) coordinates based on the patch identifier and mapping function parameters.

[0199] 12. The apparatus of clause 8, wherein the application is further configured to: utilize one or more flags to bypass one or more V-DMC methods and use the texture property map video as the V3C property component video.

[0200] 13. The apparatus of clause 8, wherein implementing the atlas mapping process occurs before implementing the connectivity process and the vertex position process.

[0201] 14. The apparatus of clause 8, wherein implementing the atlas mapping process occurs after implementing the connectivity process and the vertex position process.

[0202] 15. A system comprising:

[0203] an encoder configured to encode the 3D mesh to generate tile identification information and mapping function parameters; and

[0204] The decoder is configured as:

[0205] Implementing atlas mapping processing on the patch identification information and mapping function parameters;

[0206] Performing connectivity processing to partition the triangles to generate vertex information; and

[0207] Vertex position processing is performed to adjust the vertex positions in the vertex information.

[0208] 16. The system of clause 15, wherein the decoder is further configured to: implement a displacement sub-bitstream to include displacement information, wherein the displacement information is utilized by vertex position processing.

[0209] 17. The system of clause 16, wherein the decoder is further configured to: reconstruct the mesh based on the displaced vertices and connectivity information.

[0210] 18. The system of clause 15, wherein the atlas mapping process comprises:

[0211] receiving patch identification information and mapping function parameters; and

[0212] Generate (u,v) coordinates based on the patch identifier and mapping function parameters.

[0213] 19. The system of clause 15, wherein the decoder is further configured to: utilize one or more flags to bypass one or more V-DMC methods and use the texture property map video as the V3C property component video.

[0214] 20. The system of clause 15, wherein implementing the atlas mapping process occurs before implementing the connectivity process and the vertex position process.

[0215] 21. The system of clause 15, wherein implementing the atlas mapping process occurs after implementing the connectivity process and the vertex position process.

[0216] The present invention has been described above by way of specific embodiments, in which details are incorporated to facilitate understanding of the construction and operating principle of the present invention. The specific embodiments and details thereof mentioned herein are not intended to limit the scope of the appended claims. Those skilled in the art will readily appreciate that other various modifications may be made in the embodiments selected for illustration without departing from the spirit and scope of the present invention as defined by the claims.

Claims

1. A method of programming in a non-transitory memory of a device, comprising: Implement atlas mapping processing; Implementing connectivity processing to partition triangles to generate vertex information; as well as Vertex position processing is performed to adjust the vertex positions in the vertex information.

2. The method according to claim 1, further comprising: The displacement sub-bitstream is implemented to include displacement information, wherein the displacement information is utilized by vertex position processing.

3. The method according to claim 2, further comprising: Reconstruct the mesh based on the displaced vertices and connectivity information.

4. The method according to claim 1, wherein: Atlas mapping processing includes: Receiving patch identification information and mapping function parameters; and Generate (u,v) coordinates based on the patch identifier and mapping function parameters.

5. The method according to claim 1, further comprising: One or more flags are utilized to bypass one or more V-DMC methods and use the texture property map video as the V3C property component video.

6. The method according to claim 1, wherein: The atlas mapping process occurs before the connectivity process and vertex position process.

7. The method according to claim 1, wherein: Implementing the atlas mapping process occurs after implementing the connectivity process and vertex position process.

8. An apparatus comprising: A non-transitory memory for storing applications for: Implement atlas mapping processing; Implementing connectivity processing to partition triangles to generate vertex information; as well as Implementing vertex position processing to adjust vertex positions in vertex information; as well as A processor is coupled to the memory, the processor being configured to process the application.

9. The device according to claim 8, wherein: The application is further configured to implement a displacement sub-bitstream to include displacement information, wherein the displacement information is utilized by vertex position processing.

10. The device according to claim 9, wherein: The application is further used to reconstruct a mesh based on the displaced vertices and connectivity information.

11. The device according to claim 8, wherein: Atlas mapping processing includes: Receiving patch identification information and mapping function parameters; and Generate (u,v) coordinates based on the patch identifier and mapping function parameters.

12. The device according to claim 8, wherein: The application is also for utilizing one or more flags to bypass one or more V-DMC methods and using the texture property map video as the V3C property component video.

13. The device according to claim 8, wherein: The atlas mapping process occurs before the connectivity process and vertex position process.

14. The device according to claim 8, wherein: Implementing the atlas mapping process occurs after implementing the connectivity process and vertex position process.

15. A system comprising: An encoder configured to encode the 3D mesh to generate patch identification information and mapping function parameters; as well as The decoder is configured as: Implementing atlas mapping processing on the patch identification information and mapping function parameters; Implementing connectivity processing to partition triangles to generate vertex information; as well as Vertex position processing is performed to adjust the vertex positions in the vertex information.

16. The system of claim 15, wherein: The decoder is further configured to implement a displacement sub-bitstream to include displacement information, wherein the displacement information is utilized by vertex position processing.

17. The system of claim 16, wherein: The decoder is further configured to reconstruct a mesh based on the displaced vertices and connectivity information.

18. The system of claim 15, wherein: Atlas mapping processing includes: receiving patch identification information and mapping function parameters; and Generate (u,v) coordinates based on the patch identifier and mapping function parameters.

19. The system of claim 15, wherein: The decoder is further configured to utilize one or more flags to bypass one or more V-DMC methods and use the texture property map video as the V3C property component video.

20. The system of claim 15, wherein: The atlas mapping process occurs before the connectivity process and vertex position process.

21. The system of claim 15, wherein: Implementing the atlas mapping process occurs after implementing the connectivity process and vertex position process.

Images