Mesh Patch Syntax

Abstract

New syntax elements are used to extend patch types, and the syntax will be added to the V3C standard. The new syntax defines patches that encode a mesh by projecting connected triangles onto a 2D surface, patches that encode triangles or triangle strips without projection, or patches that are tracked over time and encoded by projecting connected triangles onto a 2D surface. Furthermore, the syntax allows mesh-specific information to be coded in different ways. For example, the syntax allows three different coding methods for vertex positions: explicit (added directly to the atlas stream), embedded in the video data (occupancy map data), or coded using an external mesh encoder.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority under 35 U.S.C. § 119 to U.S. Provisional Patent Application No. 63 / 269,912, entitled "MESH PATCH SYNTAX," filed March 25, 2022, which is incorporated herein by reference in its entirety for all purposes.

[0002] The present invention relates to three-dimensional graphics, and more particularly to coding of three-dimensional graphics. [Background technology]

[0003] In recent years, a new method for compressing volumetric content such as point clouds is becoming standardized, based on a 3D to 2D projection. Also known as V3C (visual volumetric video-based compression), this method maps the 3D volume data into a number of 2D patches, which are then placed into an atlas image, which is then encoded by a video encoder. The atlas image corresponds to the geometry of the points, their respective textures, and an occupancy map that indicates which locations should be considered for point cloud reconstruction.

[0004] In 2017, MPEG issued a Call for Proposals (CfP) for the compression of point clouds. After evaluation of several proposals, MPEG is currently considering two different techniques for point cloud compression: 3D native coding techniques (based on octrees and similar coding methods) or 3D to 2D projection followed by traditional video coding. For dynamic 3D scenes, MPEG is using Test Model Software (TMC2), based on patch surface modeling, projection of the patches from 3D to a 2D image, and coding of the 2D image by a video encoder such as HEVC. This method has proven to be more efficient than native 3D coding and can achieve competitive bitrates with acceptable quality.

[0005] Due to the success of coding 3D point clouds with projection-based methods (also known as video-based methods, or V-PCC), future versions of this standard are expected to include additional 3D data, such as 3D meshes. However, the current version of this standard is only suitable for transmitting a set of unconnected points, and therefore has no mechanism for transmitting point connectivity as required for 3D mesh compression.

[0006] Methods have also been proposed to extend the functionality of V-PCC to meshes. One possible method is to use V-PCC to encode the vertices and then use a mesh compression method such as TFAN or Edgebreaker to encode the connectivity. A limitation of this method is that the original mesh must be dense so that the point cloud generated from the vertices is not sparse and can be efficiently encoded after projection. Furthermore, the order of the vertices affects the coding of connectivity, so different methods have been proposed to reorganize the connectivity of the mesh. An alternative method for encoding sparse meshes is to use the raw patch data to encode the 3D vertex positions. Since raw patches directly encode (x,y,z), in this method all vertices are encoded as raw data, while the connectivity is encoded by a similar mesh compression method, as mentioned before. It is noted that in raw patches, the vertices can be sent in any preferred order, so the order generated from the connectivity encoding can be used. While this method can encode sparse point clouds, raw patches are not efficient for encoding 3D data and further data, such as triangular face attributes, may be missing from this method. Summary of the Invention [Problem to be solved by the invention]

[0007] New syntax elements are used to extend patch types, and the syntax will be added to the V3C standard. The new syntax defines patches that encode a mesh by projecting connected triangles onto a 2D surface, patches that encode triangles or triangle strips without projection, or patches that are tracked over time and encoded by projecting connected triangles onto a 2D surface. Furthermore, the syntax allows mesh-specific information to be coded in different ways. For example, the syntax allows three different coding methods for vertex positions: explicit (added directly to the atlas stream), embedded in the video data (occupancy map data), or coded using an external mesh encoder. [Means for solving the problem]

[0008] In one aspect, a method programmed into a non-transitory memory of a device includes encoding a mesh using patches and performing an encoding implementation of vertex positions selected from adding vertex position information directly to an atlas stream, embedding the vertex position information in video data, or using an external mesh coder. Encoding the mesh using patches includes projecting connected triangles onto a two-dimensional surface. Encoding the mesh using patches includes using patches that encode triangles or triangle strips without projection. Encoding the mesh using patches includes using patches that are tracked over time and encoded by projecting connected triangles onto a two-dimensional surface. The video data includes occupancy map data. Performing the encoding implementation of vertex positions includes delta information for geometry correction. The method further includes encoding patch connectivity using a binary or explicit implementation, encoding patch vertices using a binary, occupancy map, or explicit implementation, and encoding patch mapping coordinates using a binary, explicit, or implicit implementation.

[0009] In another aspect, an apparatus includes a non-transitory memory for storing an application, the application for encoding a mesh using patches and performing a vertex position encoding implementation selected from adding vertex position information directly to an atlas stream, embedding the vertex position information in video data, or using an external mesh coder; and a processor coupled to the memory and configured to process the application. Encoding the mesh using patches includes projecting connected triangles onto a two-dimensional surface. Encoding the mesh using patches includes using patches that encode triangles or triangle strips without projection. Encoding the mesh using patches includes using patches that are tracked over time and encoded by projecting connected triangles onto a two-dimensional surface. The video data includes occupancy map data. Performing the vertex position encoding implementation includes delta information for geometry correction. The apparatus further includes encoding patch connectivity using a binary implementation or an explicit implementation, encoding patch vertices using a binary implementation, an occupancy map implementation, or an explicit implementation, and encoding patch mapping coordinates using a binary implementation, an explicit implementation, or an implicit implementation.

[0010] In another aspect, a system includes one or more cameras for acquiring three-dimensional content, and an encoder configured to perform a vertex position encoding implementation selected from encoding a mesh using patches and adding vertex position information directly to an atlas stream, embedding the vertex position information in video data, or using an external mesh coder. Encoding the mesh using patches includes projecting connected triangles onto a two-dimensional surface. Encoding the mesh using patches includes using patches that encode triangles or triangle strips without projection. Encoding the mesh using patches includes using patches that are tracked over time and encoded by projecting connected triangles onto a two-dimensional surface. The video data includes occupancy map data. Performing the vertex position encoding implementation includes delta information for geometry correction. The system further includes encoding patch connectivity using a binary implementation or an explicit implementation, encoding patch vertices using a binary implementation, an occupancy map implementation, or an explicit implementation, and encoding patch mapping coordinates using a binary implementation, an explicit implementation, or an implicit implementation. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram of a mesh patch data syntax according to some embodiments. [Diagram 2] FIG. 2 is a diagram of a triangle primitive in accordance with some embodiments. [Diagram 3] FIG. 1 is a diagram of color expansion according to some embodiments. [Figure 4] FIG. 1 is a diagram of a tracked mesh patch data syntax according to some embodiments. [Diagram 5] FIG. 1 illustrates a flowchart of a method for patch mesh coding according to some embodiments. [Figure 6] FIG. 1 is a block diagram of an example computing device configured to implement a patch mesh coding method according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] New syntax elements are used to extend patch types, and the syntax will be added to the V3C standard. The new syntax defines patches that encode a mesh by projecting connected triangles onto a 2D surface, patches that encode triangles or triangle strips without projection, or patches that are tracked over time and encoded by projecting connected triangles onto a 2D surface. Furthermore, the syntax allows mesh-specific information to be coded in different ways. For example, the syntax allows three different coding methods for vertex positions: explicit (added directly to the atlas stream), embedded in the video data (occupancy map data), or coded using an external mesh encoder.

[0013] An alternative way to represent mesh information using existing patch syntax elements is to use mesh extensions, similar to what was done in MIV. This would also have minimal impact on the V3C specification. The text shown here indicates what is being added to the specification. For example, a mesh patch type could be an extension of the patch data unit, as shown here. TIFF2025510244000002.tif234170

[0014] The triangle patch can be an extension of the raw patch data as shown below. TIFF2025510244000003.tif133169

[0015] Finally, the tracked mesh patch type can be an extension of the Interpatch data, as shown below. TIFF2025510244000004.tif167170

[0016] The extension contains the specific mesh information associated with each patch. The syntax allows the following encoding options: There are two types of connectivity coding: explicit coding (each triangle is represented by three or four vertex indices in the patch data), or binary coding (an external encoder is used). There are three types of vertex encoding: explicit coding (two coordinates indicate a position within the patch, from which a third coordinate is derived), embedded in the occupancy map (carries a symbol that identifies the vertex's position within the patch), or binary coding (uses an external encoder). There are three types of UV coordinate encoding: explicit encoding (sending the coordinates within the patch data), implicit encoding (assuming they have the same value as the patch position of the corresponding vertex), or binary (using an external mesh encoder).

[0017] Figure 1 shows a diagram of a mesh patch data syntax according to some embodiments. Patch connectivity can be encoded using a binary implementation or an explicit implementation. Patch vertices can be encoded using a binary implementation, an occupancy map implementation or an explicit implementation. Patch mapping coordinates can be encoded using a binary implementation, an explicit implementation or an implicit implementation.

[0018] The new syntax elements and semantics are described below. TIFF2025510244000005.tif211170 TIFF2025510244000006.tif78170 If mpdu_binary_object_present_flag[tileID][p] is equal to 1, it specifies that the syntax elements mpdu_mesh_binary_object_size_bytes[tileID][p] and mpdu_mesh_binary_object[tileID][p][i] are present for the patch of index p of the current atlas tile whose tile ID is equal to tileID. If mpdu_binary_object_present_flag[tileID][p] is equal to 0, the syntax elements mpdu_mesh_binary_object_size_bytes[tileID][p] and mpdu_mesh_binary_object[tileID][p][i] are not present for the current patch. If mpdu_binary_object_present_flag[tileID][p] is not present, its value is inferred to be equal to 0. mpdu_mesh_binary_object_size_bytes[tileID][p] specifies the number of bytes used to represent, in binary form, the mesh information for the patch at index p in the current atlas tile whose tile ID is equal to tileID. mpdu_mesh_binary_object[ tileID ][ p ][ i ] specifies i bytes of the binary representation of the mesh of the patch with index p in the current atlas tile whose tile ID is equal to tileID. mpdu_vertex_count_minus3[ tileID ][ p ] + 3 specifies the number of vertices present in the patch with index p of the current atlas tile whose tile ID is equal to tileID. mpdu_face_count[tileID][p] specifies the number of triangles present in the patch with index p of the current atlas tile whose tile ID is equal to tileID. If none exist, the value of mpdu_face_count[tileID][p] is set to 0. mpdu_face_vertex[tileID][p][i][k] specifies the kth value of the vertex index of the i-th triangle or quad of the current patch of index p of the current atlas tile whose tile ID is equal to tileID. The value of mpdu_face_vertex[tileID][p][i][k] shall be in the range from 0 to mpdu_vert_count_minus3[tileID][p]+2, inclusive. mpdu_vertex_pos_x[tileID][p][i] specifies the value of the x-coordinate of the i-th vertex of the current patch of index p in the current atlas tile whose tile ID is equal to tileID. The value of mpdu_vertex_pos_x[p][i] must be in the range from 0 to mpdu_2d_size_x_minus1[tileID][p], inclusive. mpdu_vertex_pos_y[tileID][p][i] specifies the value of the y coordinate of the i-th vertex of the current patch p of the current atlas tile whose tile ID is equal to tileID. The value of mpdu_vertex_pos_y[tileID][p][i] must be in the range from 0 to mpdu_2d_size_y_minus1[tileID][p], inclusive. mpdu_vertex_pos_delta_z[tileID][p][i] specifies the difference value between the value derived from the geometry video and the z coordinate of the i-th vertex of the current patch p in the current atlas tile with tile ID equal to tileID. mpdu_vertex_u_coord[tileID][p][i] specifies the value of the u coordinate of the mapping of the i-th vertex of the current patch of index p of the current atlas tile whose tile ID is equal to tileID. The value of mpdu_vertex_u_coord[p][i] must be between 0 and 2. asps_mesh_coordinates_bit_depth_minus1 + 1 It is assumed to be in the range up to -1 (inclusive). mpdu_vertex_pos_y[tileID][p][i] specifies the value of the y coordinate of the i-th vertex of the current patch p in the current atlas tile whose tile ID is equal to tileID. The value of mpdu_vertex_pos_y[tileID][p][i] must be between 0 and 2. asps_mesh_coordinates_bit_depth_minus1 + 1 It is assumed to be in the range up to -1 (inclusive).

[0019] Figure 2 shows a diagram of triangle primitives according to some embodiments. The triangles can be arranged as individual triangles 200, triangle strips 202, or triangle fans 204. Other triangle arrangements are possible.

[0020] 3 shows a diagram of color expansion according to some embodiments. Color expansion has an x, y, z packing implementation and a line packing implementation. TIFF2025510244000007.tif39170 tpdu_vertices_minus3[tileID][p]+3 specifies the number of vertices present in the triangle coding patch of index p of the current atlas tile whose tile ID is equal to tileID. The value of tpdu_vertices_minus3[tileID][p] must be in the range from 0 to ((tpdu_2d_size_x_minus1[tileID][p] + 1) * (tpdu_2d_size_y_minus1[tileID][p] + 1)) / 3 - 3, inclusive. tpdu_primitive_idc[tileID][p] points to a geometry primitive that defines how to obtain triangles from vertices present in the coding patch of index p of the current atlas tile whose tile ID is equal to tileID. If tpdu_primitive_idc[tileID][p] is not present, its value is inferred to be equal to 0. If tpdu_color_expansion_flag[tileID][p] is equal to 1, it specifies that the vertex coordinates are packed line-interleaved and the color values ​​are expanded for the current patch p of the current atlas tile whose tile ID is equal to tileID. If tpdu_color_expansion_flag[tileID][p] is equal to 0, the vertex coordinates are packed sequentially and no colors are expanded for the current patch. If tpdu_color_expansion_flag[tileID][p] is not present, its value is inferred to be equal to 0.

[0021] FIG. 4 illustrates a diagram of tracked mesh patch data syntax according to some embodiments. TIFF2025510244000008.tif185170 TIFF2025510244000009.tif202170 TIFF2025510244000010.tif168170 tmpdu_rotation_present_flag[t][p] equal to 1 indicates the presence of rotation parameters for the patch with index p and the tile with tile ID t. tmpdu_rotation_present_flag[t][p] equal to 0 indicates the absence of rotation parameters for the patch with index p and the tile with tile ID t. If tmpdu_rotation_present_flag[t][p] is not present, it is inferred to be equal to 0. tmpdu_3d_rotation_qx[ t ][ p ] specifies the x-component qX of the geometry rotation of the patch with index p and the tile with tile ID t using quaternion representation. The value of tmpdu_3d_rotation_qx[ t ][ p ] is -2 14 From 2 14It is assumed to be in the range up to -1 (inclusive). If tmpdu_3d_rotation_qx[ t ][ p ] is not present, its value is inferred to be equal to 0. The value of qX is calculated as follows: qX=tmpdu_3d_rotation_qx,2 14 tmpdu_3d_rotation_qy[ t ][ p ] specifies the y component qY of the geometric rotation of the patch with index p and the tile with tile ID t using quaternion representation. The value of tmpdu_3d_rotation_qy[ t ][ p ] is -2 14 From 2 14 It is assumed to be in the range up to -1 (inclusive). If tmpdu_3d_rotation_qy[ t ][ p ] is not present, its value is inferred to be equal to 0. The value of qY is calculated as follows: qY=tmpdu_3d_rotation_qy,2 14 tmpdu_3d_rotation_qz[t][p] specifies the z-component qZ of the geometric rotation of the patch with index p and the tile with tile ID t using quaternion representation. The value of tmpdu_3d_rotation_qz[t][p] is -2 14 From 2 14 It is assumed to be in the range up to -1, inclusive. If tmpdu_3d_rotation_qz[ t ][ p ] is not present, it is inferred to be equal to 0. The value of qZ is calculated as follows: qZ=tmpdu_3d_rotation_qz,2 14 The fourth component of the geometric rotation, qW, of the patch with index p and tile with tile ID t using the quaternion representation is calculated as follows: qW = Sqrt(1-(qX 2 +qY 2 +qZ 2 ))

[0022] The unit quaternion can be expressed as a rotation matrix R as follows: JPEG2025510244000011.jpg20157tmpdu_vertices_changed_position_flag[ t ][ p ] equal to 1 indicates the presence of vertex displacement for the patch with index p and the tile with tile ID t. tmpdu_vertices_changed_position_flag[ t ][ p ] equal to 0 indicates the absence of vertex displacement for the patch with index p and the tile with tile ID t. If tmpdu_vertices_changed_position_flag[ t ][ p ] is not present, it is inferred to be equal to 0. tmpdu_vertex_delta_pos_x[t][p][i] specifies the difference between the x-coordinate of the i-th vertex of the patch with index p and the tile with tile ID t and the matching patch indicated by tmpdu_ref_index[t][p]. The value of tmpdu_vertex_pos_x[t][p][i] is in the range from 0 to 2afps_num_bits_vertex_delta_x - 1, inclusive. tmpdu_vertex_delta_pos_y[t][p][i] specifies the difference between the y coordinate value of the i-th vertex of the patch with index p and the tile with tile ID t and the matching patch indicated by tmpdu_ref_index[t][p]. The value of tmpdu_vertex_pos_y[t][p][i] is in the range from 0 to 2afps_num_bits_vertex_delta_y - 1, inclusive. tmpdu_vertex_pos_delta_z[tileID][p][i] specifies the difference value between the value derived from the geometry video and the z coordinate of the i-th vertex of the current patch p in the current atlas tile with tile ID equal to tileID. If tmpdu_binary_object_present_flag[tileID][p] is equal to 1, it specifies that the syntax elements tmpdu_mesh_binary_object_size_bytes[tileID][p] and tmpdu_mesh_binary_object[tileID][p][i] are present for the patch of index p of the current atlas tile whose tile ID is equal to tileID. If tmpdu_binary_object_present_flag[tileID][p] is equal to 0, the syntax elements tmpdu_mesh_binary_object_size_bytes[tileID][p] and tmpdu_mesh_binary_object[tileID][p][i] are not present for the current patch. If tmpdu_binary_object_present_flag[tileID][p] is not present, its value is inferred to be equal to 0. tmpdu_mesh_binary_object_size_bytes[ tileID ][ p ] specifies the number of bytes used to represent, in binary form, the mesh information of the patch at index p in the current atlas tile whose tile ID is equal to tileID. tmpdu_mesh_binary_object[ tileID ][ p ][ i ] specifies i bytes of the binary representation of the mesh of the patch with index p in the current atlas tile whose tile ID is equal to tileID. tmpdu_vertex_count_minus3[ tileID ][ p ] + 3 specifies the number of vertices present in the patch with index p of the current atlas tile whose tile ID is equal to tileID. tmpdu_face_count[tileID][p] specifies the number of triangles present in the patch with index p of the current atlas tile whose tile ID is equal to tileID. If none exist, the value of tmpdu_face_count[tileID][p] is set to 0. tmpdu_face_vertex[tileID][p][i][k] specifies the kth value of the vertex index of the ith triangle or quad of the current patch of index p of the current atlas tile with tile ID equal to tileID. The value of tmpdu_face_vertex[tileID][p][i][k] is bounded to be in the range from 0 to tmpdu_vert_count_minus3[tileID][p]+2, inclusive. tmpdu_vertex_pos_x[tileID][p][i] specifies the value of the x-coordinate of the i-th vertex of the current patch of index p of the current atlas tile whose tile ID is equal to tileID. The value of tmpdu_vertex_pos_x[p][i] must be in the range from 0 to tmpdu_2d_size_x_minus1[tileID][p], inclusive. tmpdu_vertex_pos_y[tileID][p][i] specifies the value of the y coordinate of the i-th vertex of the current patch p of the current atlas tile with tile ID equal to tileID. The value of tmpdu_vertex_pos_y[tileID][p][i] must be in the range from 0 to tmpdu_2d_size_y_minus1[tileID][p], inclusive. tmpdu_vertex_u_coord[tileID][p][i] specifies the value of the u coordinate of the mapping of the i-th vertex of the current patch of index p of the current atlas tile whose tile ID is equal to tileID. The value of tmpdu_vertex_u_coord[p][i] ranges from 0 to 2. asps_mesh_coordinates_bit_depth_minus1 + 1 It is assumed to be in the range up to -1 (inclusive). tmpdu_vertex_v_coord[tileID][p][i] specifies the value of the y-coordinate of the i-th vertex of the current patch p in the current atlas tile whose tile ID is equal to tileID. The value of tmpdu_vertex_v_coord[tileID][p][i] must be between 0 and 2. asps_mesh_coordinates_bit_depth_minus1 + 1 It is assumed to be in the range up to -1 (inclusive).

[0023] In addition to the above syntax elements, the following mesh extensions are proposed by adding and modifying the current atlas sequence parameter set: TIFF2025510244000012.tif203170 TIFF2025510244000013.tif195170 TIFF2025510244000014.tif98170

[0024] Also, the following semantic changes: When asps_extension_present_flag is equal to 1, it specifies that the syntax elements asps_vpcc_extension_present_flag, asps_miv_extension_present_flag, asps_mesh_extension_present_flag, and asps_extension_5bits are present in the atlas_sequence_parameter_set_rbsp( ) syntax structure. When asps_extension_present_flag is equal to 0, it indicates that the syntax elements asps_vpcc_extension_present_flag, asps_miv_extension_present_flag, asps_mesh_extension_present_flag, and asps_extension_5bits are not present. When asps_mesh_extension_present_flag is equal to 1, it specifies that the asps_mesh_extension() syntax structure is present in the atlas_sequence_parameter_set_rbsp() syntax structure. When asps_mesh_extension_present_flag is equal to 0, it specifies that this syntax structure is not present. If not present, the value of asps_mesh_extension_present_flag is inferred to be equal to 0. asps_extension_5bits equal to 0 indicates the absence of the asps_extension_data_flag syntax element in the ASPS RBSP syntax structure. If present, asps_extension_5bits shall be equal to 0 in bitstreams conforming to this version of this document. The value of asps_extension_5bits other than 0 is reserved for future use by ISO / IEC. Decoders shall allow the value of asps_extension_5bits to be different from 0 and shall ignore all asps_extension_data_flag syntax elements in ASPS NAL units. If not present, the value of asps_extension_5bits shall be inferred to be equal to 0.

[0025] The ASPS Mesh Extension is used to identify the mesh codec used for the binary coding of connectivity, the use of quads, the presence of vertex data in an occupancy map, and the explicit coding of mapping coordinates, as shown in the syntax and semantics below. TIFF2025510244000015.tif98170 asps_mesh_binary_coding_enabled_flag equal to 1 indicates that the vertex and connectivity information associated with the patch is present in binary format. asps_mesh_binary_coding_enabled_flag equal to 0 specifies that mesh vertex and connectivity data is not present in binary format. If not present, asps_mesh_binary_coding_enabled_flag is inferred to be 0. asps_mesh_binary_codec_id indicates the identifier of the codec used to compress the vertex and connectivity information of a patch. asps_mesh_binary_codec_id shall be in the range of 0 to 255, inclusive. TIFF2025510244000016.tif39170 If asps_mesh_quad_face_flag is equal to 1, it indicates that quads are used for the polygon representation. If asps_mesh_quad_face_flag is equal to 0, it indicates that triangles are used for the polygon representation of the mesh. If not present, the value of asps_mesh_quad_flag is inferred to be equal to 0. asps_mesh_vertices_in_occupancy_video_data_flag equal to 1 indicates the presence of vertex information in the occupancy video data. asps_mesh_vertices_in_occupancy_video_data_flag equal to 0 indicates the presence of vertex information in the patch data. If not present, the value of asps_mesh_vertices_in_occupancy_video_data_flag is inferred to be equal to 0. asps_mesh_mapping_coordinates_present_flag equal to 1 indicates that there is mapping information associated with the patch vertices. Asps_mesh_mapping_coordinates_present_flag equal to 0 specifies that there is no mapping information associated with the patch vertices, and they are assumed to be equal to the vertex atlas positions. If not present, asps_mesh_mapping_coordinates_present_flag is inferred to be 0. asps_mesh_coordinates_bit_depth_minus1+1 indicates the bit depth of the mapping information associated with the patch vertices. asps_mesh_coordinates_bit_depth_minus1 shall be in the range 0 to 31, inclusive. asps_mesh_vertices_delta_z_present_flag equal to 1 indicates that there is a difference between the z coordinate reconstructed from the geometry image and the actual value. asps_mesh_vertices_delta_z_present_flag equal to 0 specifies that the difference value is not present and is assumed to be equal to 0. If not present, asps_mesh_vertices_delta_z_present_flag is inferred to be 0.

[0026] The transformation of vertex positions from atlas coordinates to 3D coordinates is modified as follows:

[0027] The inputs to the process are: Variable pIdx (patch index), The variable depthValue (depth of the point), Variable x (x atlas coordinate), Variable y (y atlas coordinate).

[0028] The output of the process is a 2D array pos3D of size 3 that specifies the 3D coordinates of the point.

[0029] The following applies: The variable oIdx is set to AtlasPatchOrientationIndex[ pIdx ]. The variables posX, posY, sizeX, sizeY, lodX, lodY are assigned as follows: posX = AtlasPatch2dPosX[ pIdx ] posY = AtlasPatch2dPosY[ pIdx ] deltaZ = AtlasPatch2dPosDeltaZ[ pIdx ] sizeX = AtlasPatch2dSizeX[ pIdx ] sizeY = AtlasPatch2dSizeY[ pIdx ] lodX = AtlasPatchLoDScaleX[ pIdx ] lodY = AtlasPatchLoDScaleY[ pIdx ]

[0030] An atlas coordinate (x,y) is transformed into a local patch coordinate pair (u,v) as follows: JPEG2025510244000017.jpg9156 where Ro and Rs are specified (e.g., in a table). A local patch coordinate pair (u,v) is transformed into 3D coordinates as follows: pos3D[ AtlasPatchAxisU[ pIdx ] ] = AtlasPatch3dOffsetU[ pIdx ] + u pos3D[ AtlasPatchAxisV[ pIdx ] ] = AtlasPatch3dOffsetV[ pIdx ] + v tempD = ( 1 - 2* AtlasPatchProjectionFlag[ pIdx ] ) * (depthValue + deltaZ) pos3D[AtlasPatchAxisD[pIdx]] = Max(0, AtlasPatch3dOffsetD[ pIdx ] + tempD) Some V-MESH decoder implementations may choose to clip the reconstructed 3D coordinates to the range 0 to (1 << (asps_geometry_3d_bit_depth_minus1 + 1)) - 1, inclusive. Other decoder implementations may apply the clipping operation after the 45 degree transform or after some post-reconstruction process.

[0031] As described in U.S. Patent Application Serial No. Attorney Docket No. Sony-75300, which is incorporated herein by reference in its entirety for all purposes, there are three ways to encode vertex mapping information: implicit, explicit, and binary. In an implicit implementation, the projection is the same as the mapping when projecting onto a 2D surface. For example, the location that will be hit when projecting onto the projection surface is the UV coordinates. In an explicit implementation, projection is performed but different coordinates are sent to the texture. In a binary implementation, the explicit information is encoded in an external encoder (e.g., Draco or AFX).

[0032] If binary coding is implemented, an external mesh encoder can be used to encode the patch mesh information. U and V are added to the ply, and vertex mapping information is encoded with the ply. In some embodiments, delta information of the z coordinate is added. The delta information can be used for geometry correction.

[0033] Further details regarding mesh compression can be found in U.S. Patent Application No. 17 / 322,662, filed May 17, 2021, entitled "VIDEO BASED MESH COMPRESSION," U.S. Provisional Patent Application No. 63 / 088,705, filed October 7, 2020, entitled "VIDEO BASED MESH COMPRESSION," and U.S. Provisional Patent Application No. 63 / 087,958, filed October 6, 2020, entitled "VIDEO BASED MESH COMPRESSION," all of which are incorporated by reference in their entirety for all purposes. The connectivity of a 3D mesh or a 2D patch mesh can be encoded using temporal correlations using an occupancy map and using video-based mesh compression.

[0034] FIG. 5 shows a flow chart of a method for patch mesh coding according to some embodiments. In step 500, a mesh is coded using patches. Coding a mesh using patches includes projecting connected triangles onto a two-dimensional surface, using patches that code triangles or triangle strips without projection, or using patches that are tracked over time and coded by projecting connected triangles onto a two-dimensional surface. In step 502, a vertex position coding implementation is performed that is selected from adding vertex position information directly to the atlas stream, embedding vertex position information into the video data, or using an external mesh coder. Delta information can be sent for geometry correction. In some embodiments, the order of steps is changed. In some embodiments, fewer or additional steps are implemented. For example, coding patch connectivity using a binary implementation or an explicit implementation, coding patch vertices using a binary implementation, an occupancy map implementation or an explicit implementation, and coding patch mapping coordinates using a binary implementation, an explicit implementation or an implicit implementation.

[0035] FIG. 6 illustrates a block diagram of an exemplary computing device configured to implement a patch mesh coding method according to some embodiments. The computing device 600 can be used to acquire, store, compute, process, communicate, and / or display information, such as images and videos, including 3D content. The computing device 600 can implement any of the encoding / decoding aspects. In general, a hardware structure suitable for implementing the computing device 600 includes a network interface 602, a memory 604, a processor 606, an I / O device(s) 608, a bus 610, and a storage device 612. The selection of the processor is not critical as long as a suitable processor of sufficient speed is selected. The memory 604 can be any conventional computer memory known in the art. The storage device 612 can include a hard drive, CDROM, CDRW, DVD, DVDRW, high definition disk / drive, ultra high definition drive, flash memory card, or any other storage device. The computing device 600 can include one or more network interfaces 602. Examples of network interfaces include a network card connected to an Ethernet or other type of LAN. The I / O device(s) 608 may include one or more of a keyboard, mouse, monitor, screen, printer, modem, touch screen, button interface, and other devices. The storage device 612 and memory 604 likely store the patch mesh coding application(s) 630 used to execute the implementation of the patch mesh coding and process as an application would normally be processed. The computing device 600 may also include more or fewer components than those shown in FIG. 6. In some embodiments, a patch mesh coding hardware 620 is included. Although the computing device 600 of FIG. 6 includes an application 630 and hardware 620 for the implementation of patch mesh coding, the patch mesh coding method may also be implemented on the computing device in hardware, firmware, software, or any combination thereof.For example, in some embodiments, the patch mesh coding application 630 is programmed into memory and executed using a processor. As another example, in some embodiments, the patch mesh coding hardware 620 is programmed hardware logic that includes gates specifically designed to implement the patch mesh coding method.

[0036] In some embodiments, the patch mesh coding application(s) 630 include multiple applications and / or modules. In some embodiments, a module also includes one or more sub-modules. In some embodiments, fewer or additional modules may be included.

[0037] Examples of suitable computing devices include a personal computer, a laptop computer, a computer workstation, a server, a mainframe computer, a handheld computer, a personal digital assistant, a cellular / mobile phone, a smart appliance, a game console, a digital camera, a digital camcorder, a camera phone, a smartphone, a portable music player, a tablet computer, a mobile device, a video player, a video disc writer / player (e.g., DVD writer / player, high definition disc writer / player, ultra high definition disc writer / player), a television, a home entertainment system, an augmented reality device, a virtual reality device, smart jewelry (e.g., a smart watch), a vehicle (e.g., a self-driving vehicle), or any other suitable computing device.

[0038] To utilize the patch mesh coding method, a device acquires or receives 3D content (e.g., point cloud content). The patch mesh coding method can be implemented with user assistance or automatically without user involvement.

[0039] In operation, the patch mesh coding method enables more efficient and accurate 3D content encoding compared to conventional implementations.

[0040] Some embodiments of the mesh patch syntax 1. A method programmed into a non-transitory memory of a device, comprising: encoding a mesh using the patches; performing a vertex position coding implementation selected from adding vertex position information directly to the atlas stream, embedding said vertex position information in the video data, or using an external mesh coder; The method includes:

[0041] 2. The method of claim 1, wherein the step of encoding the mesh using patches includes projecting connected triangles onto a two-dimensional surface.

[0042] 3. The method of claim 1, wherein the step of encoding the mesh using patches includes using patches to encode triangles or triangle strips without projection.

[0043] 4. The method of claim 1, wherein the step of encoding the mesh using patches includes using patches that are tracked over time and encoded by projecting connected triangles onto a two-dimensional surface.

[0044] 5. The method of claim 1, wherein the video data includes occupancy map data.

[0045] 6. The method according to claim 1, wherein the step of performing the encoded implementation of vertex positions includes delta information for geometry correction.

[0046] 7. Encoding patch connectivity using a binary or explicit implementation; encoding the patch vertices using a binary implementation, an occupancy map implementation or an explicit implementation; encoding patch mapping coordinates using a binary, explicit or implicit implementation; 2. The method of claim 1, further comprising:

[0047] 8. An apparatus comprising: A non-transitory memory for storing an application, the application comprising: encoding a mesh using patches; performing a vertex position encoding implementation selected from adding vertex position information directly to the atlas stream, embedding said vertex position information in the video data, or using an external mesh coder; a non-transient memory for performing a processor coupled to the memory and configured to process the application; An apparatus comprising:

[0048] 9. The apparatus of claim 8, wherein encoding the mesh using patches includes projecting connected triangles onto a two-dimensional surface.

[0049] 10. The apparatus of claim 8, wherein encoding the mesh using patches includes using patches to encode triangles or triangle strips without projection.

[0050] 11. The apparatus of claim 8, wherein encoding the mesh using patches includes using patches that are tracked over time and encoded by projecting connected triangles onto a two-dimensional surface.

[0051] 12. The apparatus of claim 8, wherein the video data includes occupancy map data.

[0052] 13. The apparatus of claim 8, wherein performing the encoded implementation of vertex positions includes delta information for geometry correction.

[0053] 14. Encoding patch connectivity using a binary or explicit implementation; encoding patch vertices using a binary implementation, an occupancy map implementation, or an explicit implementation; encoding patch mapping coordinates using a binary implementation, an explicit implementation, or an implicit implementation; 9. The apparatus of claim 8, further comprising:

[0054] 15. A system comprising: one or more cameras for acquiring three-dimensional content; Encode the mesh using patches, performing a vertex position encoding implementation selected from adding vertex position information directly to the atlas stream, embedding said vertex position information in the video data, or using an external mesh coder; an encoder configured to A system including:

[0055] 16. The system of claim 15, wherein encoding the mesh using patches includes projecting connected triangles onto a two-dimensional surface.

[0056] 17. The system of claim 15, wherein encoding the mesh using patches includes using patches to encode triangles or triangle strips without projection.

[0057] 18. The system of claim 15, wherein encoding the mesh using patches includes using patches that are tracked over time and encoded by projecting connected triangles onto a two-dimensional surface.

[0058] 19. The system of claim 15, wherein the video data includes occupancy map data.

[0059] 20. The system of claim 15, wherein performing the encoded implementation of vertex positions includes delta information for geometry correction.

[0060] 21. Encoding patch connectivity using a binary or explicit implementation; encoding patch vertices using a binary implementation, an occupancy map implementation, or an explicit implementation; encoding patch mapping coordinates using a binary implementation, an explicit implementation, or an implicit implementation; 16. The system of claim 15, further comprising:

[0061] The present invention has been described with respect to specific embodiments containing details so as to facilitate an understanding of the principles of construction and operation of the invention. Reference herein to such specific embodiments and their details is not intended to limit the scope of the claims appended hereto. It will be readily apparent to those skilled in the art that various other modifications can be made in the embodiments selected for illustration without departing from the spirit and scope of the invention as defined by the claims. [Explanation of symbols]

[0062] 200 triangles 202 Triangle Strip 204 Triangular Fan Encode the mesh using 500 patches 502 Execute vertex position encoding implementation 600 Computer equipment 602 Network Interface 604 Memory 606 Processor 608 I / O device 610 Bus 612 Storage device 620 Patch Mesh Coding Hardware 630 Patch Mesh Coding Application

Claims

1. A method programmed into a non-transitory memory of a device, comprising: encoding a mesh using the patches; performing a vertex position coding implementation selected from adding vertex position information directly to the atlas stream, embedding said vertex position information in the video data, or using an external mesh coder; The method according to claim 1, further comprising:

2. 2. The method of claim 1, wherein encoding the mesh with patches comprises projecting connected triangles onto a two-dimensional surface.

3. 2. The method of claim 1, wherein encoding the mesh using patches comprises using patches to encode triangles or triangle strips without projection.

4. 2. The method of claim 1, wherein encoding the mesh using patches includes using patches that are tracked over time and encoded by projecting connected triangles onto a two-dimensional surface.

5. 2. The method of claim 1, wherein the video data includes occupancy map data.

6. The method of claim 1 , wherein performing the encoded implementation of vertex positions includes delta information for geometry correction.

7. encoding patch connectivity using a binary or explicit implementation; encoding the patch vertices using a binary implementation, an occupancy map implementation or an explicit implementation; encoding patch mapping coordinates using a binary, explicit or implicit implementation; The method of claim 1 further comprising:

8. An apparatus comprising: A non-transitory memory for storing an application, the application comprising: encoding a mesh using patches; performing a vertex position encoding implementation selected from adding vertex position information directly to the atlas stream, embedding said vertex position information in the video data, or using an external mesh coder; a non-transient memory for performing a processor coupled to the memory and configured to process the application; An apparatus comprising:

9. The apparatus of claim 8 , wherein encoding the mesh using patches comprises projecting connected triangles onto a two-dimensional surface.

10. The apparatus of claim 8 , wherein encoding the mesh using patches comprises using patches to encode triangles or triangle strips without projection.

11. 10. The apparatus of claim 8, wherein encoding the mesh using patches includes using patches that are tracked over time and encoded by projecting connected triangles onto a two-dimensional surface.

12. 9. The apparatus of claim 8, wherein the video data includes occupancy map data.

13. The apparatus of claim 8 , wherein performing the encoded implementation of vertex positions includes delta information for geometry correction.

14. Encoding patch connectivity using a binary or explicit implementation; encoding patch vertices using a binary implementation, an occupancy map implementation, or an explicit implementation; encoding patch mapping coordinates using a binary implementation, an explicit implementation, or an implicit implementation; The apparatus of claim 8 further comprising:

15. 1. A system comprising: one or more cameras for acquiring three-dimensional content; Encode the mesh using patches, performing a vertex position encoding implementation selected from adding vertex position information directly to the atlas stream, embedding said vertex position information in the video data, or using an external mesh coder; an encoder configured to A system comprising:

16. 16. The system of claim 15, wherein encoding the mesh using patches comprises projecting connected triangles onto a two-dimensional surface.

17. 16. The system of claim 15, wherein encoding the mesh using patches comprises using patches to encode triangles or triangle strips without projection.

18. 16. The system of claim 15, wherein encoding the mesh using patches includes using patches that are tracked over time and encoded by projecting connected triangles onto a two-dimensional surface.

19. 16. The system of claim 15, wherein the video data includes occupancy map data.

20. 16. The system of claim 15, wherein performing the encoded implementation of vertex positions includes delta information for geometry correction.

21. Encoding patch connectivity using a binary or explicit implementation; encoding patch vertices using a binary implementation, an occupancy map implementation, or an explicit implementation; encoding patch mapping coordinates using a binary implementation, an explicit implementation, or an implicit implementation; The system of claim 15 further comprising:

Images