Method, apparatus and system for video-based mesh compression

Abstract

This paper describes a method for compressing 3D mesh data using a video representation of the projection of the mesh surface data and the connectivity data. The method utilizes 3D surface patches to represent a collection of connected triangles on the mesh surface. The projected surface data is stored in patches (mesh patches) encoded in atlas data. The connectivity of the mesh, i.e., the vertices and triangles of the surface patches, is encoded using video-based compression techniques. The data is encapsulated in a new video component called vertex video data, and the disclosed structure allows for progressive mesh encoding and decoding by separating the collection of vertices in each layer and creating levels of detail for the mesh connectivity. This approach extends the functionality of the V3C (Volumetric Video) standard currently used for encoding and decoding of point clouds and multi-view and depth content.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATION(S)

[0002] This application claims priority under 35 U.S.C. §119(e) to U.S. Provisional Patent Application Serial No. 63 / 088,705, filed on October 7, 2020, entitled “VIDEO BASEDMESH COMPRESSION,” and U.S. Provisional Patent Application Serial No. 63 / 087,958, filed on October 6, 2020, entitled “VIDEO BASED MESH COMPRESSION,” which are incorporated herein by reference in their entirety for all purposes. Technical Field

[0003] The present invention relates to three-dimensional graphics, and more particularly to the coding and decoding of three-dimensional graphics. Background Art

[0004] Recently, a novel method for compressing volumetric content (such as point clouds) based on projection from 3D to 2D is being standardized. This method, also known as V3C (Visual Volumetric Video-Based Compression), maps 3D volumetric data into several 2D patches, which are then further arranged into atlas images, which are then encoded using a video encoder. The atlas images correspond to the point geometry, the corresponding texture, and an occupancy map that indicates which locations to consider for point cloud reconstruction.

[0005] In 2017, MPEG issued a call for proposals (CfP) on point cloud compression. After evaluating several proposals, MPEG is currently considering two different point cloud compression techniques: 3D native codec technology (based on octrees and similar codecs), or 3D to 2D projection followed by traditional video codecs. In the case of dynamic 3D scenes, MPEG uses test model software (TMC2) based on patch surface modeling, projects patches from 3D to 2D images, and encodes and decodes the 2D images using a video codec such as HEVC. This approach has been shown to be more efficient than native 3D codecs and can achieve competitive bit rates with acceptable quality.

[0006] Due to the success of projection-based methods (also known as video-based methods or V-PCC) for encoding and decoding 3D point clouds, the standard is expected to include more 3D data, such as 3D meshes, in future versions. However, the current version of the standard is only suitable for transmitting disconnected sets of points and therefore has no mechanism for transmitting the connectivity of the points, as it is required for 3D mesh compression.

[0007] Methods have been proposed to extend the functionality of V-PCC to meshes. One possible approach is to encode the vertices using V-PCC and then encode the connectivity using a mesh compression method such as TFAN or Edgebreaker. A limitation of this approach is that the original mesh must be dense, so the point cloud generated from the vertices is not sparse and can be efficiently encoded after projection. Moreover, the order of the vertices affects the encoding and decoding of the connectivity, and different methods have been proposed to reorganize the mesh connectivity. Another way to encode sparse meshes is to encode the vertex positions in 3D using RAW patch data. Since RAW patches directly encode (x,y,z), in this approach all vertices are encoded as RAW data, while the connectivity is encoded using a similar mesh compression method as mentioned previously. Note that in RAW patches, the vertices can be sent in any preferred order, so the order generated from the connectivity encoding can be used. This method can encode sparse point clouds, but RAW patches are not efficient for encoding 3D data, and this method will lose more data, such as the attributes of triangular faces. Summary of the Invention

[0008] This paper describes a method for compressing 3D mesh data using a video representation of the projection of the mesh surface data and the connectivity data. The method utilizes 3D surface patches to represent a collection of connected triangles on the mesh surface. The projected surface data is stored in patches (mesh patches) encoded in atlas data. The connectivity of the mesh (i.e., the vertices and triangles of the surface patches) is encoded using video-based compression techniques. The data is encapsulated in a new video component called vertex video data, and the disclosed structure allows for progressive mesh encoding by separating sets of vertices in layers and creating levels of detail for mesh connectivity. This approach extends the functionality of the V3C (Volumetric Video Based) standard currently used for encoding point clouds and multi-view and depth content.

[0009] In one aspect, a method includes performing mesh voxelization on an input mesh, implementing patch generation to partition the mesh into patches comprising a rasterized mesh surface and vertex positions and connectivity information, generating a visual volumetric video-based compression (V3C) image from the rasterized mesh surface, implementing video-based mesh compression using the vertex positions and connectivity information, and generating a V3C bitstream based on the V3C image and the video-based mesh compression. The vertex positions and connectivity information comprise triangle information for the surface patches. Data from implementing video-based mesh compression using vertex positions and connectivity information is encapsulated in a vertex video component structure. The vertex video component structure implements progressive mesh coding by separating sets of vertices in a layer and generating a level of detail for mesh connectivity. When only one layer is implemented, the video data is embedded in an occupancy map. The connectivity information is generated using a surface reconstruction algorithm including Poisson surface reconstruction or spherical pivoting. Generating the V3C image from the rasterized mesh surface includes combining untracked and tracked mesh information. The method also includes implementing an edge-collapse filter in the two-dimensionally projected patch domain. The method also includes implementing patch-based surface subdivision of the connectivity information.

[0010] In another aspect, an apparatus includes a non-transitory memory for storing an application for performing mesh voxelization on an input mesh, implementing patch generation to segment the mesh into patches comprising a rasterized mesh surface and vertex positions and connectivity information, generating a visual volumetric video-based compression (V3C) image from the rasterized mesh surface, implementing video-based mesh compression using the vertex positions and connectivity information, and generating a V3C bitstream based on the V3C image and the video-based mesh compression; the processor being configured to process the application. The vertex positions and connectivity information comprise triangle information for the surface patches. Data from implementing video-based mesh compression using the vertex positions and connectivity information is encapsulated in a vertex video component structure. The vertex video component structure implements progressive mesh coding by separating sets of vertices in a layer and generating a level of detail for mesh connectivity. When only one layer is implemented, the video data is embedded in an occupancy map. The connectivity information is generated using a surface reconstruction algorithm including Poisson surface reconstruction or spherical pivoting. Generating the V3C image from the rasterized mesh surface includes combining untracked and tracked mesh information. The application is further configured to implement an edge folding filter in a two-dimensional projected patch domain. The application is further configured to implement patch-based surface subdivision of connectivity information.

[0011] In another aspect, a system includes one or more cameras for acquiring three-dimensional content, an encoder for encoding the three-dimensional content, mesh voxelization on an input mesh, patch generation for partitioning the mesh into patches comprising a rasterized mesh surface and vertex positions and connectivity information, generating a visual volumetric video-based compression (V3C) image from the rasterized mesh surface, performing video-based mesh compression using the vertex positions and connectivity information, and generating a V3C bitstream based on the V3C image and the video-based mesh compression. The vertex positions and connectivity information comprise triangle information for the surface patches. Data from the video-based mesh compression using the vertex positions and connectivity information is encapsulated in a vertex video component structure. The vertex video component structure implements progressive mesh encoding and decoding by separating sets of vertices in a layer and generating a level of detail for mesh connectivity. When only one layer is implemented, the video data is embedded in an occupancy map. The connectivity information is generated using a surface reconstruction algorithm including Poisson surface reconstruction or spherical pivoting. Generating the V3C image from the rasterized mesh surface includes combining untracked and tracked mesh information. The encoder is configured to implement an edge folding filter in a two-dimensional projected patch domain. The encoder is configured to implement patch-based surface subdivision of connectivity information. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 A flowchart illustrating a method for implementing V3C trellis encoding and decoding according to some embodiments is shown.

[0013] Figure 2 Diagram illustrating mesh voxelization according to some embodiments.

[0014] Figure 3 A diagram illustrating patch generation according to some embodiments.

[0015] Figure 4 Diagram illustrating patch rasterization according to some embodiments.

[0016] Figure 5 Diagram illustrating V3C image generation according to some embodiments.

[0017] Figure 6 An image illustrating vertex video data according to some embodiments is illustrated.

[0018] Figure 7 An image illustrating exemplary level-of-detail generation according to some embodiments is illustrated.

[0019] Figure 8 A diagram illustrating a grid in accordance with some embodiments is illustrated.

[0020] Figure 9 A diagram illustrating mesh reconstruction according to some embodiments.

[0021] Figure 10 A diagram illustrating high-level syntax and images that allows sending a mix of point clouds and mesh patches, according to some embodiments.

[0022] Figure 11A -B illustrates a diagram of combined untracked and tracked mesh information according to some embodiments.

[0023] Figure 12 An exemplary image illustrating patch-based edge collapsing in accordance with some embodiments.

[0024] Figure 13 An exemplary image illustrating patch-based cluster extraction in accordance with some embodiments.

[0025] Figure 14 An exemplary image illustrating patch-based surface tessellation in accordance with some embodiments is illustrated.

[0026] Figure 15 Illustrated are exemplary images of patch-based surface reconstructions in accordance with some embodiments.

[0027] Figure 16 Diagram illustrating triangle edge detection in accordance with some embodiments.

[0028] Figure 17 A diagram illustrating the edges of a segmentation of a triangle based on color separation in accordance with some embodiments.

[0029] Figure 18 A diagram illustrating the partitioned edges of a combined triangle in accordance with some embodiments.

[0030] Figure 19-23 A diagram illustrating resized and rescaled edges in accordance with some embodiments.

[0031] Figure 24 Illustrated is a block diagram of an exemplary computing device configured to implement a video-based mesh compression method in accordance with some embodiments.

[0032] Figure 25 A diagram illustrating a system configured to implement video-based mesh compression in accordance with some embodiments is illustrated. DETAILED DESCRIPTION

[0033] This paper describes a method for compressing 3D mesh data using a video representation of the projection of the mesh surface data and the connectivity data. The method utilizes 3D surface patches to represent a collection of connected triangles on the mesh surface. The projected surface data is stored in patches (mesh patches) encoded in atlas data. The connectivity of the mesh (i.e., the vertices and triangles of the surface patches) is encoded using video-based compression techniques. The data is encapsulated in a new video component called vertex video data, and the disclosed structure allows for progressive mesh encoding by separating sets of vertices in layers and creating levels of detail for mesh connectivity. This approach extends the functionality of the V3C (Volumetric Video) standard currently used for codecs of point clouds and multi-view and depth content.

[0034] In the encoding and decoding of 3D point clouds using video encoders, the projection from 3D to 2D is important for generating the video that will represent the point cloud. The most efficient way to generate the video is to use 3D patches, which segment the surface of the object and use orthogonal projection to generate segmented depth images, which are bundled together and used as input to the video encoder. In addition, points not captured by the projection step can also be encoded directly in the video signal. In current point cloud standards, it is not possible to encode 3D meshes because there is no defined method to encode the connectivity of the mesh. In addition, if the vertex data is sparse, the performance of the standard is poor because it cannot exploit the correlation between vertices.

[0035] This paper describes a method for encoding and decoding meshes using the V3C standard for encoding and decoding volumetric data. A method for segmenting the mesh surface and proposing joint surface sampling and 2D patch generation is described. For each patch, the local connectivity and the positions of the vertices projected onto the 2D patch are encoded. A method for signaling the connectivity and vertex positions is described, enabling reconstruction of the original input mesh. Furthermore, a method for mapping vertices and connectivity into video frames and encoding the mesh connectivity data into a video sequence, referred to as vertex video data, using video codec tools is described.

[0036] Figure 1 A flow chart of a method for implementing V3C mesh encoding and decoding according to some embodiments is illustrated. In step 100, an input mesh is received or acquired. For example, the input mesh is downloaded (e.g., from a network device) or acquired / captured by a device (e.g., a camera or an autonomous vehicle).

[0037] In step 102, mesh voxelization is performed. A mesh can have floating point vertex positions, so these positions are converted to integer space. V-PCC and V3C assume voxelized point clouds.

[0038] In step 104, mesh generation (or creation) is performed. Mesh generation includes: normal calculation, adjacency calculation, initial segmentation; refinement, mesh projection, and mesh rasterization. Normal calculation is the calculation of the normal for each triangle (e.g., the cross product of the triangle's edges). Adjacency calculation involves calculating the adjacency relationship of each triangle (e.g., which triangles in the mesh are adjacent to or touching the current triangle or other triangles). Initial segmentation includes classifying normals according to orientation. For example, a triangle's normal can point up, down, left, right, front, or back, and can be classified based on direction / orientation. In some embodiments, triangles are color-coded based on the orientation of their normals (e.g., all triangles with normals pointing upward are colored green). Refinement involves locating outliers (e.g., a single red triangle surrounded by blue triangles) and smoothing outliers (e.g., changing a single red triangle to match its blue neighbors). Refinement is performed by analyzing neighbors and smoothing orientation (e.g., adjusting the orientation of normals). Once a smooth surface exists, mesh projection is performed, which involves projecting a mesh for a specific triangle classification (e.g., based on orientation). Vertices and connectivity are shown on the patch through projection. For example, the body and face in this example are separate projections because there are triangles of different classifications separating the two. V3C and V-PCC do not understand this aspect; rather, V3C and V-PCC understand points, so the projections are rasterized (e.g., sampling points on the surface include distances to the points to generate the geometric image and surface properties). The rasterized mesh surface is very similar to a V3C image.

[0039] The patch generation results in a rasterized mesh surface with vertex positions and connectivity. In step 106, the rasterized mesh surface is used for V3C image generation / creation. In step 108, the vertex positions and connectivity are used for mesh codec (e.g., video-based mesh compression). In step 110, a V3C bitstream is generated from the generated V3C image and the base mesh codec. In some embodiments, the mesh codec does not involve any additional codecs, and the vertex positions and connectivity go directly into the V3C bitstream.

[0040] The V3C bitstream enables point cloud reconstruction in step 112 and / or mesh construction in step 114. The ability to extract point clouds and / or meshes from the V3C bitstream provides significant flexibility. In some embodiments, fewer or additional steps may be implemented. In some embodiments, the order of the steps may be modified.

[0041] The methods described herein are related to U.S. patent application serial No. 17 / 161,300, filed on January 28, 2021, entitled “PROJECTION-BASED MESHCOMPRESSION,” which is incorporated herein by reference in its entirety for all purposes.

[0042] To address voxelization, grid scale and offset information is sent in the Atlas Adaptation Parameter Set (AAPS). Available camera parameters can be used. Alternatively, new syntax elements are introduced for voxelization (in the case of using only scale and offset). The following is an example syntax:

[0043]

[0044] aaps_voxelization_parameters_present_flag equal to 1 specifies that the voxelization parameters should be present in the current atlas adaptation parameter set.

[0045] aaps_voxelization_parameters_present_flag equal to 0 specifies that voxelization parameters for the current adaptation parameter set shall not be present.

[0046] vp_scale_enabled_flag equal to 1 indicates that scaling parameters for the current voxelization are present. vp_scale_enabled_flag equal to 0 indicates that scaling parameters for the current voxelization are not present. When vp_scale_enabled_flag is not present, it shall be inferred to be equal to 0.

[0047] vp_offset_enabled_flag equal to 1 indicates that the offset parameters for the current voxelization are present. vp_offset_enabled_flag equal to 0 indicates that the offset parameters for the current voxelization are not present. When vp_offset_enabled_flag is not present, it shall be inferred to be equal to 0.

[0048] vp_scale specifies the scaling value used for the current voxelization. Scale is 2 -16 The value of vp_scale should be between 1 and 2. 32 -1, inclusive. When vp_scale is not present, it should be inferred to be equal to 2. 16 The value of Scale is calculated as follows:

[0049] Scale = vp_scale ÷ 2 16

[0050] vp_offset_on_axis[d] is 2 -16 The increment of vp_offset_on_axis[d] indicates the offset value along the d axis used for the current voxelization. The value of vp_offset_on_axis[d] should be between -231 to 2 31- 1, inclusive, where d is in the range of 0 to 2, inclusive. The values of d are equal to 0, 1, and 2, corresponding to the X, Y, and Z axes, respectively. When vp_offset_on_axis[d] is not present, it shall be inferred to be equal to 0.

[0051] Offset[d]=vp_offset_on_axis[d]÷2 16

[0052] This pass specifies the inverse voxelization process to get from voxelized decoded vertex values to floating point reconstructed values. The following apply:

[0053] for(n=0;n <VertexCnt;n++)

[0054] for(k=0;k<3;k++)

[0055] vertexReconstructed[n][k]=Scale*(decodedVertex[n][k])+Offset[k]

[0056] Figure 2 A diagram illustrating mesh voxelization according to some embodiments is shown. Each frame has a different bounding box. A bounding box is obtained for each frame (e.g., frame 200 at t=1, t=16, and t=32). A sequence bounding box 202 is then computed from the many bounding boxes, sequenceBB=(minPoint, maxPoint). Sequence bounding box 202 contains all vertices, regardless of the frame. A scale is computed that fits within the range defined by bitdepth, maxRange=max(maxPoint16[0..2]-minPoint16[0..2]), scale=(2 bitdepth -1) / maxRange->scale16. The result is scaled and shifted by the minimum value, voxelizedPoint=floor((originalPoint-minPoint16) / scale16). The scale and shift amounts can be user-defined or computer-generated based on a learning algorithm (e.g., by analyzing the bounding box and automatically calculating the scale and shift amounts). These values are stored in AAPS with shift=minPoint16 and scale=scale16.

[0057] Input parameters (modelScale):

[0058] (-1): Automatically calculate the zoom for each frame

[0059] (0): Automatically calculate sequence scaling

[0060] (>1): User-defined scaling.

[0061] In some embodiments, mesh voxelization involves converting floating point values of the positions of points of the input mesh into integers. The precision of the integers can be set by the user or automatically. In some embodiments, mesh voxelization includes shifting values so that there are no negative numbers.

[0062] For example, the original mesh is below the axis, resulting in negative numbers. Through mesh voxelization, the mesh is shifted and / or scaled to avoid negative and non-integer values. In one embodiment, the lowest vertex value below zero is found, and then these values can be shifted so that the lowest vertex value is above zero. In some embodiments, the range of values fits within a specified bit range (e.g., by scaling), such as 11 bits.

[0063] The voxelized grid 210 is the original grid after scaling and shifting. For example, the voxelized grid 210 is the original grid after it has been grown and shifted so that it is only positive values (which is better for encoding in some cases).

[0064] Voxelization causes triangles to become degenerate (vertices occupying the same position), but the encoding and decoding process removes degenerate vertices and increases the number of vertices due to mesh segmentation (for example, a remove duplicate vertex filter can be used to reduce the number of vertices). In this example, the original mesh has 20,692 vertices; 39,455 faces; and 20,692 points in the voxelized vertices, while the reconstructed mesh has 27,942 vertices; 39,240 faces and the reconstructed point cloud has 1,938,384 points.

[0065] Figure 3

[0014] A diagram illustrating patch generation according to some embodiments is shown. As described above, patch generation involves normal calculation, adjacency calculation, initial segmentation (or normal classification), and segmentation refinement (or class refinement). Calculating the normal for each triangle involves a cross product between the triangle edges. The adjacency calculation determines whether the triangles share vertices, and if so, the triangles are neighbors. Initial segmentation and segmentation refinement are performed identically to V-PCC by analyzing the orientation of the normals, classifying the orientation of the normals (e.g., up, down, left, right, forward, backward), and determining whether the orientation of the normal is classified differently from neighboring normals that are all classified the same (e.g., the first patch is classified as up, while most or all neighboring patches are classified as forward), and then changing the classification of the patch's normal to match the orientation of the neighbors (e.g., changing the classification of the first patch to forward).

[0066] As described above, patch generation is implemented to divide a mesh into patches. Patch generation also generates 1) a rasterized mesh surface and 2) vertex position and connectivity information. The rasterized mesh surface is a collection of points generated from a V3C or V-PCC image and encoded as a V3C or V-PCC image. Vertex position and connectivity information are received for basic mesh encoding and decoding.

[0067] The patch generation described herein is similar to that in V-PCC. However, instead of calculating the normal for each point, the normal is calculated for each triangle. The normal for each triangle is calculated using the cross product between the edges to determine the normal vector. The triangles are then classified based on the normal. For example, the normals are divided into n (e.g., 6) categories, such as front, back, top, bottom, left, and right. The normals are indicated in different colors to illustrate the initial segmentation. Figure 3 Different colors in grayscale, such as black and light gray, are shown, with different colors indicating different normals. Although it may be difficult to see, the top surface (e.g., the top of the person's head, the top of the ball, and the top of the sneaker) is one color (e.g., green), the first side of the person / ball is dark and represents another color (e.g., red), the bottom of the ball is another color (e.g., purple), and the front of the person and ball, which are primarily light gray, represent another color (e.g., cyan).

[0068] By multiplying the product of the normals by the direction, the principal direction can be found. The smoothing / thinning process can be implemented by looking at the neighboring triangles. For example, if a higher than a threshold number of neighboring triangles are all blue, then this triangle is also classified as blue, even if there are anomalies that initially indicate the triangle is red.

[0069] Connected components of the triangles are generated to identify which triangles have the same color (eg, triangles with the same category share at least one vertex).

[0070] Connectivity information describes how points are connected in 3D. These connections together generate triangles (more specifically, 3 different connections that share 3 points), thereby generating a surface (described by a collection of triangles). Although triangles are described here, other geometric shapes are also allowed (for example, rectangles).

[0071] By identifying triangles with different colors, we can use color to encode connectivity. Each triangle identified by three connections is encoded with a unique color.

[0072] Figure 4A diagram illustrating patch rasterization, one of the components of the patch generation process, in accordance with some embodiments. Patch generation also includes generating connected components of triangles (triangles of the same category share at least one vertex). If the bounding box of the connected components is smaller than a predefined area, the triangle is moved to a separate list for independent triangle encoding and decoding. These unprojected triangles are not rasterized, but are instead encoded as vertices with associated color vertices. Otherwise, each triangle is projected into the patch. If the position of the projection of the vertex is already occupied, the triangle is encoded into another patch and enters the lost triangle list to be processed again later. Alternatively, the graph can be used to identify overlapping vertices and still be able to represent triangles with overlapping vertices. The triangles are rasterized to generate points for point cloud representation.

[0073] The original voxelized vertices 400 are shown. The rasterized surface points 402 (added to the point cloud representation) follow the structure of the mesh, so the point cloud geometry can be as coarse as the underlying mesh. However, the geometry can be improved by sending additional positions for each rasterized pixel.

[0074] Figure 5 Figure 1 illustrates V3C image generation according to some embodiments. When projecting a mesh, an occupancy map, a geometry map, and a texture map are generated. For V3C image generation, the occupancy map and geometry map are generated in the same manner as before. The attribute map (texture) is generated from the uncompressed geometry.

[0075] When generating a patch, information is added to indicate where the patch is located on the 2D image. It also indicates where the vertices are and how they are connected. The following syntax performs these tasks:

[0076]

[0077]

[0078]

[0079]

[0080] mpdu_binary_object_present_flag[tileID][p] equal to 1 specifies that the syntax elements mpdu_mesh_binary_object_size_bytes[tileID][p] and mpdu_mesh_binary_object[tileID][p][i] are present for the current patch p of the current atlas tile with tileID equal to tileID. If mpdu_binary_object_present_flag[tileID][p] is equal to 0, then the syntax elements mpdu_mesh_binary_object_size_bytes[tileID][p] and mpdu_mesh_binary_object[tileID][p][i] are not present for the current patch. If mpdu_binary_object_present_flag[tileID][p] is not present, then its value shall be inferred to be 0.

[0081] mpdu_mesh_binary_object_size_bytes[tileID][p] specifies the number of bytes used to represent the mesh information in binary form.

[0082] mpdu_mesh_binary_object[tileID][p][i] specifies the i bytes of the binary representation of the mesh for the p-th patch.

[0083] mpdu_vertex_count_minus3[tileID][p] plus 3 specifies the number of vertices present in the patch.

[0084] mpdu_face_count[tileID][p] specifies the number of triangles present in the patch. When not present, the value of mpdu_face_count[tileID][p] should be zero.

[0085] mpdu_face_vertex[tileID][p][i][k] specifies the kth value of the vertex index of the i-th triangle or quadrilateral used for the current patch p of the current atlas tile, where tile ID is equal to tileID. The value of mpdu_face_vertex[tileID][p][i][k] shall be in the range of 0 to mpdu_vert_count_minus3[tileID][p]+2.

[0086] mpdu_vertex_pos_x[tileID][p][i] specifies the x-coordinate value of the i-th vertex of the current patch p for the current atlas tile, where tile ID is equal to tileID.

[0087] The value of mpdu_vertex_pos_x[p][i] shall be in the range of 0 to mpdu_2d_size_x_minus1[p], inclusive.

[0088] mpdu_vertex_pos_y[tileID][p][i] specifies the y-coordinate value of the i-th vertex of the current patch p for the current atlas tile, where tile ID is equal to tileID.

[0089] The value of mpdu_vertex_pos_y[tileID][p][i] shall be in the range of 0 to mpdu_2d_size_y_minus1[tileID][p], inclusive.

[0090] Some elements of mesh patch data are controlled by parameters defined in an Atlas Sequence Parameter Set (ASPS). A new extension to the ASPS for meshes is now available.

[0091]

[0092]

[0093] asps_mesh_extension_present_flag equal to 1 specifies that the asps_mesh_extension() syntax structure is present in the atlas_sequence_parameter_set_rbsp syntax structure.

[0094] asps_mesh_extension_present_flag equal to 0 specifies that this syntax structure is not present. When not present, the value of asps_mesh_extension_present_flag is inferred to be equal to 0.

[0095] asps_extension_6bits equal to 0 specifies that no asps_extension_data_flag syntax element is present in the ASPS RBSP syntax structure. When present, asps_extension_6bits shall be equal to 0 in bitstreams conforming to this version of this document. Values of asps_extension_6bits not equal to 0 are reserved for future use by ISO / IEC. Decoders shall allow values of asps_extension_6bits not equal to 0 and shall ignore all asps_extension_data_flag syntax elements in ASPS NAL units. When not present, the value of asps_extension_6bits is inferred to be equal to 0.

[0096]

[0097] asps_mesh_binary_coding_enabled_flag equal to 1 indicates that vertex and connectivity information associated with the patch is present in binary format.

[0098] asps_mesh_binary_coding_enabled_flag equal to 0 indicates that the mesh vertex and connectivity data is not present in binary format. When not present, asps_mesh_binary_coding_enabled_flag is inferred to be 0.

[0099] asps_mesh_binary_codec_id indicates the identifier of the codec used to compress vertex and connectivity information for the patch. asps_mesh_binary_codec_id should be in the range of 0 to 255, inclusive. This codec can be identified by the profile defined in Appendix A or by means outside of this document.

[0100] asps_mesh_quad_face_flag equal to 1 indicates that quadrilaterals are used for polygonal representation. asps_mesh_quad_face_flag equal to 0 indicates that triangles are used for polygonal representation of the mesh. When not present, the value of asps_mesh_quad_flag is inferred to be equal to 0.

[0101] asps_mesh_vertices_in_vertex_map_flag equal to 1 indicates that vertex information is present in the vertex video data. asps_mesh_vertices_in_vertex_flag equal to 0 indicates that vertex information is present in the patch data. When not present, the value of asps_mesh_vertices_in_vertex_map_flag is inferred to be equal to 0.

[0102] The syntax allows four different types of vertex / connectivity encoding and decoding:

[0103] 1. Send vertex and connectivity information directly in the patch.

[0104] 2. Send vertex information in the vertex graph and face connectivity in the patch.

[0105] 3. Send vertex information in a vertex graph and infer facet connectivity on the decoder side (e.g., using ball pivoting or Poisson reconstruction).

[0106] 4. Use an external mesh encoder (e.g., SC3DM, Draco, etc.) to encode and decode vertex and connectivity information.

[0107] Unlike regular 3D grids, 2D grids are encoded.

[0108] The new V3C video data unit carries information about vertex positions. The data unit can contain a binary value indicating the position of a projected vertex, or it can contain multi-level information for connectivity reconstruction. The data unit allows constructing levels of detail by encoding and decoding vertices in multiple layers. The data unit uses the VPS extension to define additional parameters.

[0109]

[0110]

[0111]

[0112] vuh_lod_index, when present, indicates the lod index of the current vertex stream.

[0113] When not present, the lod index of the current vertex sub-bitstream is derived based on the type of sub-bitstream and the operations described in Subclause XX for the vertex video sub-bitstream respectively. The value of vuh_lod_index, when present, shall be in the range of 0 to vms_lod_count_minus1[vuh_atlas_id], inclusive.

[0114] vuh_reserved_zero_13bits, when present, shall be equal to 0 in bitstreams conforming to this version of this document. Other values for vuh_reserved_zero_13bits are reserved for future use by ISO / IEC. A decoder shall ignore the value of vuh_reserved_zero_13bits.

[0115]

[0116]

[0117] vme_lod_count_minus1[k] plus 1 indicates the number of lods used to encode the vertex data of the atlas with atlas ID k. vme_lod_count_minus1[j] should be in the range of 0 to 15, inclusive.

[0118] vme_embed_vertex_in_occupancy_flag[k] equal to 1 specifies that the vertex information is derived from the occupancy map specified in clause XX for the atlas with atlas ID k.

[0119] vme_embed_vertex_in_occupancy_flag[k] equal to 0 specifies that vertex information is not derived from occupancy video. When vme_embed_vertex_in_occupancy_flag[k] is not present, it is inferred to be equal to 0.

[0120] vme_multiple_lod_streams_present_flag[k] equal to 0 indicates that all lods for the atlas with atlas ID k are placed in a single vertex video stream. vme_multiple_lod_streams_present_flag[k] equal to 1 indicates that all lods for the atlas with atlas ID k are placed in separate video streams. When vme_multiple_lod_streams_present_flag[k] is not present, its value shall be inferred to be equal to 0.

[0121] vme_lod_absolute_coding_enabled_flag[j][i] equal to 1 indicates that lod with index i for atlas with atlas ID k is coded without any form of picture prediction. vme_lod_absolute_coding_enabled_flag[k][i] equal to 0 indicates that lod with index i for atlas with atlas ID k is first predicted from another earlier coded picture before coding. If vme_lod_absolute_coding_enabled_flag[j][i] is not present, its value shall be inferred to be equal to 1.

[0122] When vps_map_absolute_coding_enabled_flag[j][i] is equal to 0, vme_lod_predictor_index_diff[k][i] is used to calculate the predictor for lod with index i of atlas with atlas ID k. More specifically, the map predictor index for lod i, LodPredictorIndex[i], shall be calculated as:

[0123] LodPredictorIndex[i]=(i-1)-vme_lod_predictor_index_diff[j][i] The value of vme_lod_predictor_index_diff[j][i] shall be in the range of 0 to i–1, inclusive. When vme_lod_predictor_index_diff[j][i] is not present, its value shall be inferred to be equal to 0.

[0124] vme_vegrtex_video_present_flag[k] is equal to 0 to indicate that the atlas with ID k has no vertex data. vms_vertex_video_present_flag[k] is equal to 1 to indicate that the atlas with ID k has vertex data. When vms_vertex_video_present_flag[j] is not present, it is inferred to be equal to 0.

[0125]

[0126] vi_vertex_codec_id[j] indicates the identifier of the codec used to compress vertex information for the atlas with atlas ID j.

[0127] vi_vertex_codec_id[j] shall be in the range of 0 to 255, inclusive. This codec may be identified by the profile, component codec map SEI message, or by means outside this document.

[0128] vi_lossy_vertex_compression_threshold[j] indicates the threshold to be used for deriving binary vertices from the decoded vertex video of the atlas with atlas ID j. vi_lossy_vertex_compression_threshold[j] should be in the range of 0 to 255, inclusive.

[0129] vi_vertex_2d_bit_depth_minus1[j] plus 1 indicates the nominal 2D bit depth to which the vertex video for the atlas with atlas ID j should be converted. vi_vertex_2d_bit_depth_minus1[j] should be in the range of 0 to 31, inclusive. vi_vertex_MSB_align_flag[j] indicates how the decoded vertex video samples associated with the atlas with atlas ID j should be converted to samples of the nominal vertex bit depth.

[0130] Figure 6 An image of vertex video data according to some embodiments is illustrated. Vertex video data uses multiple layers to indicate the level of detail, which is useful for progressive mesh encoding and decoding. If only one layer is used, the video data can be embedded in the occupancy map to avoid generating multiple decoding instances. Connectivity information can be generated using surface reconstruction algorithms (e.g., Poisson surface reconstruction and ball pivoting).

[0131] Vertex video data is displayed as dots / points in the image. These dots indicate the location of the vertices in the projected image. Vertex video data can be sent directly in a separate video, as shown in image 600, or it can be embedded in the occupancy image 602. Diagram 604 shows an example of a ball pivoting.

[0132] Figure 7Images illustrate exemplary level-of-detail generation according to some embodiments. Vertices can be combined in layers to generate a multi-level representation of a mesh. Specifying level-of-detail generation. The original point cloud is shown in image 700, which has 762 vertices and 1,245 faces. Instead of sending all the data, only 10% of the vertices / connectivity is sent. This is a 10% cluster decimation, as shown in image 702, which sends 58 vertices and 82 faces. Image 704 shows a 5% cluster decimation, which sends 213 vertices and 512 faces. Image 706 shows a 2.5% cluster decimation, which sends 605 vertices and 978 faces. By separating the layers, it is possible to gradually send multiple layers to improve quality (e.g., a first layer of 10% decimation, a second layer of 5%, and a third layer of 2.5%). These layers can be combined to obtain the original number of vertices and faces of the mesh (or close to it).

[0133] Figure 8 A diagram illustrating a mesh according to some embodiments. In some embodiments, SC3DM (MPEG) can be used to encode mesh information for each patch. SC3DM can be used to encode connectivity and (u,v) information. In some embodiments, Draco can be used to encode mesh information for each patch. Draco can be used to encode connectivity and (u,v) information.

[0134] Figure 9 A diagram illustrating mesh reconstruction according to some embodiments. Connectivity uses new vertex numbering, but patches can be added together. Due to compression, vertices at seams may not match. Mesh smoothing or zippering algorithms can be used to address this issue.

[0135] Figure 10 A diagram illustrating high-level syntax and images that allow for sending a mix of point cloud and mesh patches, according to some embodiments. Since meshes are described at the patch level, it is possible to mix and match patches used for an object. For example, point cloud-only patches could be used for a head or hair, while mesh patches could be used for flat areas such as the body.

[0136] Tracked mesh patch data units can use patches to indicate that connectivity has not changed from one patch to another. This is particularly useful for the tracked mesh case, as only delta positions are sent. In the case of tracked meshes, global motion occurs which can be captured by bounding box positions and rotations (newly introduced syntax elements using quaternions) and surface motion captured by vertex motion. If the reference patch is using V3C_VVD data, then vertex motion can be sent explicitly in the patch information, or derived from the video data. The number of bits to send delta vertex information can be sent in the Atlas Frame Parameter Set (AFPS). Alternatively, motion information can also be sent as a homography transform.

[0137]

[0138]

[0139]

[0140] tmpdu_vertices_changed_position_flag specifies whether the vertices have changed their positions. tmpdu_vertex_delta_pos_x[p][i] specifies the difference in the x-coordinate value of the i-th vertex of patch p from the matching patch indicated by tmpdu_ref_index[p]. The value of tmpdu_vertex_pos_x[p][i] should be in the range of 0 to pow2(afps_num_bits_delta_x)-1, inclusive.

[0141] tmpdu_vertex_delta_pos_y[p][i] specifies the difference in the y-coordinate value of the i-th vertex of patch p from the matching patch indicated by tmpdu_ref_index[p]. The value of tmpdu_vertex_pos_x[p][i] should be in the range of 0 to pow2(afps_num_bits_delta_y)-1, inclusive.

[0142] tmpdu_rotation_present_flag specifies whether the rotation value is present.

[0143] tmpdu_3d_rotation_qx specifies the x component qX of the geometric rotation of the current patch using quaternion representation. The value of tmpdu_3d_rotation_qx should be between -2 15 to 2 15The value of qX is in the range of -1, inclusive. When tmpdu_3d_rotation_qx is not present, its value shall be inferred to be equal to 0. The value of qX is calculated as follows:

[0144] qX=tmpdu_3d_rotation_qx÷2 15

[0145] tmpdu_3d_rotation_qy specifies the y component qY of the geometric rotation of the current patch using quaternion representation. The value of tmpdu_3d_rotation_qy should be between -2 15 to 2 15 The value of qY is in the range of -1, inclusive. When tmpdu_3d_rotation_qy is not present, its value shall be inferred to be equal to 0. The value of qY is calculated as follows:

[0146] qY=tmpdu_3d_rotation_qy÷2 15

[0147] tmpdu_3d_rotation_qz specifies the z component qZ of the geometric rotation of the current patch using quaternion representation. The value of tmpdu_3d_rotation_qz should be between -2 15 to 2 15 The value of qZ is in the range of -1, inclusive. When tmpdu_3d_rotation_qz is not present, its value shall be inferred to be equal to 0. The value of qZ is calculated as follows:

[0148] qZ=tmpdu_3d_rotation_qz÷2 15

[0149] The fourth component qW of the geometric rotation of the current point cloud image represented by the quaternion is calculated as follows:

[0150] qW=Sqrt(1-(qX 2 +qY 2 +qZ 2 ))

[0151] The unit quaternion can be represented as a rotation matrix R as follows:

[0152]

[0153] Figure 11A-B illustrates a diagram of combined untracked and tracked mesh information according to some embodiments. To avoid tracking problems, some algorithms split the mesh into a tracked part and an untracked part. The tracked part is consistent in time and can be represented by the proposed tracked_mesh_patch_data_unit(), while the untracked part is new in each frame and can be represented by mesh_patch_data_unit(). Since the notation allows blending point clouds into geometry, the surface representation can also be improved (e.g., keeping the original mesh and inserting a point cloud on top of the mesh to hide defects).

[0154] Figure 12 An example image illustrating patch-based edge collapsing according to some embodiments. An edge collapsing filter can be applied to patch data to reduce the number of triangles that are encoded. Geometry and texture information can be kept intact to improve rendering. Mesh simplification can be reversed by using refined geometry data. Meshlab has the option to apply edge collapsing even with boundary preservation. However, the algorithm works in 3D space and it does not use the projected characteristics of the mesh. The new idea is to do an "edge collapsing filter in the 2D projected patch domain", i.e., apply edge collapsing to the patch data taking into account the 2D characteristics of the edges.

[0155] Image 1200 shows the original dense mesh with 5,685 vertices and 8,437 faces. Image 1202 shows the edge-collapsed (boundary-preserving) patch with 2,987 vertices and 3,041 faces. Image 1204 shows the edge-collapsed patch with 1,373 vertices and 1,285 faces. Image 1206 shows the edge-collapsed complete mesh with 224 vertices and 333 faces.

[0156] Figure 13 An exemplary image of patch-based cluster extraction according to some embodiments is illustrated. Meshlab can choose to extract based on a 3D mesh (cluster extraction). Since patches are projected data in 2D, extraction can be performed in 2D space instead. In addition, the number of extracted vertices is retained to reconstruct the surface (using fine geometry data and surface subdivision). This information can be sent in the occupancy map.

[0157] Image 1300 shows the original dense mesh with 5,685 vertices and 8,437 faces. Image 1302 shows a cluster extraction with 3,321 vertices and 4,538 faces (1%). Image 1304 shows a cluster extraction with 730 vertices and 870 faces (2.5%). Image 1306 shows a cluster extraction with 216 vertices and 228 faces (5%). Image 1308 shows a cluster extraction with 90 vertices and 104 faces (10%).

[0158] Figure 14 An exemplary image of a patch-based surface subdivision according to some embodiments is illustrated. Meshlab has several filters to generate finer meshes, but they assume some heuristics (e.g., dividing triangles at midpoints). Better results can be achieved if geometric information is used to guide where triangles should be divided. For example, a high-resolution mesh can be generated from a low-resolution mesh. The upsampling of mesh information is guided by the geometry.

[0159] Image 1400 shows a mesh with 224 vertices and 333 faces. Image 1402 shows a subdivision surface with 784 vertices and 1,332 face midpoints (1 iteration). Image 1404 shows a subdivision surface with 2,892 vertices and 5,308 face midpoints (2 iterations). Image 1406 shows a subdivision surface with 8,564 vertices and 16,300 face midpoints (3 iterations).

[0160] Figure 15 An exemplary image of patch-based surface reconstruction according to some embodiments is illustrated. Meshlab has filters to reconstruct mesh surfaces from point clouds (screen Poisson and ball pivoting). The algorithm can be used to reconstruct mesh connectivity on the decoder side at the patch level (vertex lists are signaled, e.g., available via occupancy maps). In some embodiments, connectivity information is not sent and can be regenerated using Poisson or ball pivoting.

[0161] Image 1500 shows a dense vertex cloud with 5,685 vertices. Image 1502 shows a Poisson surface reconstruction with 13,104 vertices and 26,033 faces. Image 1504 shows a ball pivot with 5,685 vertices and 10,459 faces.

[0162] In some embodiments, the positions of the vertices are obtained from an occupancy map. Color information embedded in the occupancy map can be used. On the encoder side, the face regions associated with each triplet of vertex sets are painted with a fixed color. The paint color is different for each face and is chosen to facilitate color segmentation. An m-ary occupancy map is used. On the decoder side, the occupancy map is decoded. Face information is derived based on the segmented colors.

[0163] In some embodiments, vertex positions are derived from occupancy. A new attribute is assigned to carry face information. On the encoder side, an attribute rectangle is generated with dimensions (width x height) equal to the number of faces. This attribute has three dimensions, each of which carries the index of one of the triplet's vertices. On the decoder side, the attribute video is decoded. Face information is derived from the decoded attribute video.

[0164] In some embodiments, the positions of vertices are obtained from the occupancy map using Delaunay triangulation. At the decoder side, the occupancy map video is decoded. The vertices obtained from the decoded occupancy map are triangulated. The triangulated points are used to obtain surface information.

[0165] Figure 16 16. A diagram illustrating triangle edge detection according to some embodiments. An original image 1600 has blue, red, and yellow triangles. A segmented blue triangle image 1602 shows blue triangles, a segmented red triangle image 1604 shows red triangles, and a segmented yellow triangle image 1606 shows yellow triangles.

[0166] Triangles can be grouped based on color. Triangles can be split based on intersecting edges, which indicates where the triangles are, where the edges of the triangles are, and even where the vertices are.

[0167] Figure 17 1 illustrates a diagram of segmented edges of triangles based on color separation according to some embodiments. Original image 1600 has blue, red, and yellow triangles. Segmented blue-edge image 1700 shows blue triangle edges, segmented red-edge image 1702 shows red triangle edges, and segmented yellow-edge image 1704 shows yellow triangle edges.

[0168] Figure 18 18. Illustrated is a diagram of segmented edges of combined triangles according to some embodiments. The original image 1600 has blue, red, and yellow triangles. The combined edges are shown in the triangle edge image 1800.

[0169] Figure 19-23 A diagram illustrating resizing and rescaling edges according to some embodiments. Images, triangles, and / or edges can be resized (e.g., shrunk), then resized triangles can be detected, rescaled edges can be determined, and resized edges can be determined / generated.

[0170] As described herein, color can be used to encode triangle positions and triangle connectivity by performing segmentation, finding edges, triangles, and vertices, and determining which positions are connected.

[0171] Figure 24A block diagram of an exemplary computing device configured to implement a video-based mesh compression method according to some embodiments is illustrated. Computing device 2400 can be used to acquire, store, calculate, process, transmit, and / or display information, such as images and videos including 3D content. Computing device 2400 can implement any encoding / decoding aspects. Generally speaking, a hardware structure suitable for implementing computing device 2400 includes a network interface 2402, memory 2404, processor 2406, (one or more) I / O devices 2408, bus 2410, and storage device 2412. The choice of processor is not important, as long as a suitable processor with sufficient speed is selected. Memory 2404 can be any conventional computer memory known in the art. Storage device 2412 can include a hard drive, CDROM, CDRW, DVD, DVDRW, high-definition disk / drive, ultra-high-definition drive, flash memory card, or any other storage device. Computing device 2400 can include one or more network interfaces 2402. Examples of network interfaces include network cards connected to Ethernet or other types of LANs. The I / O device(s) 2408 can include one or more of the following: a keyboard, a mouse, a monitor, a screen, a printer, a modem, a touch screen, a button interface, and other devices. The video-based mesh compression application(s) 2430 used to implement the video-based mesh compression implementation are likely stored in the storage device 2412 and the memory 2404 and processed as applications are typically processed. Figure 24 More or fewer of the components shown in can be included in the computing device 2400. In some embodiments, video-based mesh compression hardware 2420 is included. Figure 24 The computing device 2400 in FIG. 2 includes an application 2430 and hardware 2420 for implementing a video-based mesh compression method, but the video-based mesh compression method can be implemented on the computing device in hardware, firmware, software, or any combination thereof. For example, in some embodiments, the video-based mesh compression application 2430 is programmed in a memory and executed using a processor. In another example, in some embodiments, the video-based mesh compression hardware 2420 is programmed hardware logic including gates specifically designed to implement the video-based mesh compression method.

[0172] In some embodiments, (one or more) video-based mesh compression applications 2430 include several applications and / or modules. In some embodiments, a module also includes one or more submodules. In some embodiments, fewer or additional modules can be included.

[0173] Examples of suitable computing devices include personal computers, laptops, computer workstations, servers, mainframe computers, handheld computers, personal digital assistants, cellular / mobile phones, smart appliances, game consoles, digital cameras, digital video cameras, camera phones, smartphones, portable music players, tablet computers, mobile devices, video players, video disc recorders / players (e.g., DVD recorders / players, high-definition disc recorders / players, ultra-high-definition disc recorders / players), televisions, home entertainment systems, augmented reality devices, virtual reality devices, smart jewelry (e.g., smart watches), vehicles (e.g., self-driving vehicles), or any other suitable computing device.

[0174] Figure 25 Illustrated is a diagram of a system configured to implement video-based grid compression according to some embodiments. Encoder 2500 is configured to implement a coding process. As described herein, any coding, such as video-based grid compression, can be implemented. Grid and other information can be directly transmitted to decoder 2504 or transmitted via network 2502. The network can be any type of network, such as a local area network (LAN), the Internet, a wireless network, a wired network, a cellular network and / or any other network or a combination of networks. Decoder 2504 decodes the encoded content.

[0175] To utilize the video-based mesh compression method, a device acquires or receives 3D content (eg, point cloud content). The video-based mesh compression method can be implemented with user assistance or automatically without user involvement.

[0176] In operation, the video-based mesh compression approach enables more efficient and accurate encoding of 3D content compared to previous implementations.

[0177] Some embodiments of video-based mesh compression

[0178] 1. A method comprising:

[0179] Perform mesh voxelization on the input mesh;

[0180] Implementing patch generation, which segments a mesh into patches including a rasterized mesh surface and vertex position and connectivity information;

[0181] Generate visual volumetric video-based compression (V3C) images from rasterized mesh surfaces;

[0182] Video-based mesh compression using vertex position and connectivity information; and

[0183] Generate V3C bitstream based on V3C image and video-based mesh compression.

[0184] 2. The method of clause 1, wherein the vertex position and connectivity information includes triangle information for the surface patch.

[0185] 3. The method of clause 1, wherein data from implementing video-based mesh compression using vertex position and connectivity information is encapsulated in a vertex video component structure.

[0186] 4. The method of clause 3, wherein the vertex video component structure implements progressive mesh coding by separating sets of vertices in layers and generating levels of detail for mesh connectivity.

[0187] 5. The method of clause 1, wherein when only one layer is implemented, the video data is embedded in the occupancy map.

[0188] 6. The method of clause 1, wherein the connectivity information is generated using a surface reconstruction algorithm comprising Poisson surface reconstruction or spherical pivoting.

[0189] 7. The method of clause 1, wherein generating a V3C image from the rasterized mesh surface comprises combining untracked and tracked mesh information.

[0190] 8. The method of clause 1 further comprising implementing an edge folding filter in the two-dimensional projected patch domain.

[0191] 9. The method of clause 1, further comprising implementing patch-based surface subdivision of connectivity information.

[0192] 10. A device comprising:

[0193] Non-transitory memory for storing applications for:

[0194] Perform mesh voxelization on the input mesh;

[0195] Implementing patch generation, which segments a mesh into patches including a rasterized mesh surface and vertex position and connectivity information;

[0196] Generating Visual Volumetric Video Compression (V3C) from Rasterized Mesh Surfaces

[0197] image;

[0198] Video-based mesh compression using vertex position and connectivity information; and

[0199] generating a V3C bitstream based on the V3C image and video-based mesh compression; and

[0200] A processor coupled to the memory, the processor configured to process the application.

[0201] 11. The apparatus of clause 10, wherein the vertex position and connectivity information comprises triangle information for a surface patch.

[0202] 12. The apparatus of clause 10, wherein data from implementing video-based mesh compression using vertex position and connectivity information is encapsulated in a vertex video component structure.

[0203] 13. The apparatus of clause 12, wherein the vertex video component structure implements progressive mesh coding by separating sets of vertices in layers and generating levels of detail for mesh connectivity.

[0204] 14. The apparatus of clause 10, wherein when only one layer is implemented, the video data is embedded in the occupancy map.

[0205] 15. The apparatus of clause 10, wherein the connectivity information is generated using a surface reconstruction algorithm comprising Poisson surface reconstruction or spherical pivoting.

[0206] 16. The apparatus of clause 10, wherein generating a V3C image from the rasterized mesh surface comprises combining untracked and tracked mesh information.

[0207] 17. The apparatus of clause 10, wherein the application is further configured to implement an edge folding filter in a two-dimensional projected patch domain.

[0208] 18. The apparatus of clause 10, wherein the application is further configured to implement patch-based surface subdivision of connectivity information.

[0209] 19. A system comprising:

[0210] One or more cameras for acquiring three-dimensional content;

[0211] Encoder for encoding 3D content:

[0212] Perform mesh voxelization on the input mesh;

[0213] Implementing patch generation, which segments a mesh into patches including a rasterized mesh surface and vertex position and connectivity information;

[0214] Generating Visual Volumetric Video Compression (V3C) from Rasterized Mesh Surfaces

[0215] image;

[0216] Video-based mesh compression using vertex position and connectivity information; and

[0217] Generate V3C bitstream based on V3C image and video-based mesh compression.

[0218] 20. The system of clause 19, wherein the vertex position and connectivity information comprises triangle information for a surface patch.

[0219] 21. The system of clause 19, wherein data from implementing video-based mesh compression using vertex position and connectivity information is encapsulated in a vertex video component structure.

[0220] 22. The system of clause 21, wherein the vertex video component structure implements progressive mesh coding by separating sets of vertices in layers and generating levels of detail for mesh connectivity.

[0221] 23. The system of clause 19, wherein when only one layer is implemented, the video data is embedded in the occupancy map.

[0222] 24. The system of clause 19, wherein the connectivity information is generated using a surface reconstruction algorithm comprising Poisson surface reconstruction or spherical pivoting.

[0223] 25. The system of clause 19, wherein generating a V3C image from the rasterized mesh surface comprises combining untracked and tracked mesh information.

[0224] 26. The system of clause 19, wherein the encoder is configured to implement the edge folding filter in a two-dimensional projected patch domain.

[0225] 27. The system of clause 19, wherein the encoder is configured to implement patch-based surface subdivision of connectivity information.

[0226] The present invention has been described with reference to specific embodiments, which incorporate details that facilitate an understanding of the principles of construction and operation of the invention. Reference herein to specific embodiments and details thereof is not intended to limit the scope of the appended claims. It will be apparent to one skilled in the art that various other modifications may be made in the embodiments chosen for illustration without departing from the spirit and scope of the invention as defined in the claims.

Claims

1. A method for video-based mesh compression, comprising: Perform mesh voxelization on the input mesh; Implementing patch generation, which segments a mesh into patches including a rasterized mesh surface and vertex position and connectivity information; generating a compressed V3C image based on the visual volume video from the rasterized mesh surface, wherein generating the V3C image from the rasterized mesh surface comprises combining untracked and tracked mesh information; Video-based mesh compression using vertex position and connectivity information; as well as Generate V3C bitstream based on V3C image and video-based mesh compression. 2 . The method of claim 1 , wherein the vertex position and connectivity information comprises triangle information of a surface patch.

3. The method of claim 1, wherein data from implementing video-based mesh compression using vertex position and connectivity information is encapsulated in a vertex video component structure.

4. The method of claim 3, wherein the vertex video component structure implements progressive mesh coding by separating sets of vertices in layers and generating levels of detail for mesh connectivity. The method of claim 1 , wherein when only one layer is implemented, the video data is embedded in the occupancy map. The method of claim 1 , wherein the connectivity information is generated using a surface reconstruction algorithm including Poisson surface reconstruction or spherical pivoting.

7. The method of claim 1 further comprising implementing an edge folding filter in the two-dimensional projected patch domain.

8. The method of claim 1, further comprising implementing patch-based surface subdivision of connectivity information.

9. An apparatus for video-based mesh compression, comprising: Non-transitory memory for storing applications for: Perform mesh voxelization on the input mesh; Implementing patch generation, which segments a mesh into patches including a rasterized mesh surface and vertex position and connectivity information; generating a compressed V3C image based on the visual volume video from the rasterized mesh surface, wherein generating the V3C image from the rasterized mesh surface comprises combining untracked and tracked mesh information; Video-based mesh compression using vertex position and connectivity information; as well as Generate V3C bitstream based on V3C image and video-based mesh compression; and a processor coupled to the memory, the processor configured to process the application.

10. The apparatus of claim 9, wherein the vertex position and connectivity information comprises triangle information of a surface patch.

11. The apparatus of claim 9, wherein data from implementing video-based mesh compression using vertex positions and connectivity information is encapsulated in a vertex video component structure.

12. The apparatus of claim 11, wherein the vertex video component structure implements progressive mesh coding by separating sets of vertices in layers and generating levels of detail for mesh connectivity.

13. The apparatus of claim 9, wherein when only one layer is implemented, the video data is embedded in the occupancy map.

14. The apparatus of claim 9, wherein the connectivity information is generated using a surface reconstruction algorithm including Poisson surface reconstruction or spherical pivoting.

15. The apparatus of claim 9, wherein the application is further configured to implement an edge folding filter in a two-dimensional projected patch domain.

16. The apparatus of claim 9, wherein the application is further configured to implement patch-based surface subdivision of connectivity information.

17. A system for video-based mesh compression, comprising: One or more cameras for acquiring three-dimensional content; Encoder for encoding 3D content: Perform mesh voxelization on the input mesh; Implementing patch generation, which segments a mesh into patches including a rasterized mesh surface and vertex position and connectivity information; generating a compressed V3C image based on the visual volumetric video from the rasterized mesh surface, wherein generating the V3C image from the rasterized mesh surface comprises combining untracked and tracked mesh information; Video-based mesh compression using vertex position and connectivity information; as well as Generate V3C bitstream based on V3C image and video-based mesh compression.

18. The system of claim 17, wherein the vertex position and connectivity information includes triangle information for a surface patch.

19. The system of claim 17, wherein data from implementing video-based mesh compression using vertex position and connectivity information is encapsulated in a vertex video component structure.

20. The system of claim 19, wherein the vertex video component structure implements progressive mesh coding by separating sets of vertices in layers and generating levels of detail for mesh connectivity.

21. The system of claim 17, wherein when only one layer is implemented, the video data is embedded in the occupancy map.

22. The system of claim 17, wherein the connectivity information is generated using a surface reconstruction algorithm comprising Poisson surface reconstruction or spherical pivoting.

23. The system of claim 17, wherein the encoder is configured to implement the edge folding filter in a two-dimensional projected patch domain.

24. The system of claim 17, wherein the encoder is configured to implement patch-based surface subdivision of connectivity information.

Images