Mesh patch syntax

Abstract

New syntax elements are used to extend patch types, and the syntax will be added to the V3C standard. The new syntax defines patches that encode a mesh by projecting connected triangles onto a 2D surface, patches that encode triangles or triangle strips without projection, or patches that are tracked over time and encoded by projecting connected triangles onto a 2D surface. Furthermore, the syntax allows mesh-specific information to be coded in different ways. For example, the syntax allows three different coding methods for vertex positions: explicit (added directly to the atlas stream), embedded in the video data (occupancy map data), or coded using an external mesh encoder.

Description

[Technical Field]

[0001] [Cross-reference of related applications] This application claims priority under § 119 of U.S. Patent Act Provisional Patent Application No. 63 / 269,912, entitled “Mesh Patch Syntax,” filed on 25 March 2022, which is incorporated herein by reference in its entirety for all purposes.

[0002] This invention relates to three-dimensional graphics. More specifically, this invention relates to the coding of three-dimensional graphics. [Background technology]

[0003] In recent years, a new method for compressing volumetric content, such as point clouds, based on 3D-to-2D projection, is becoming standardized. Also known as V3C (Visual Volume Video-Based Compression), this method maps 3D volumetric data into several 2D patches, then places these patches into an atlas image, which is then encoded with a video encoder. The atlas image corresponds to the geometry of the points, their respective textures, and an occupancy map indicating which locations should be considered for point cloud reconstruction.

[0004] In 2017, MPEG issued a Call for Proposals (CfP) for point cloud compression. After evaluating several proposals, MPEG is currently considering two different techniques for point cloud compression: 3D native coding techniques (based on octree and similar coding methods), or conventional video coding after 3D-to-2D projection. For dynamic 3D scenes, MPEG uses Test Model Software (TMC2) based on patch surface modeling, projection of patches from 3D to 2D images, and coding of the 2D images by a video encoder such as HEVC. This method has proven to be more efficient than native 3D coding and can achieve competitive bitrates with acceptable quality.

[0005] Since coding 3D point clouds using projection-based methods (also known as video-based methods or V-PCC) has been successful, future versions of this standard are expected to include further 3D data, such as 3D meshes. However, the current version of this standard is only suitable for transmitting sets of unconnected points and lacks a mechanism for transmitting point connectivity, as required for 3D mesh compression.

[0006] Methods have also been proposed to extend the functionality of V-PCC to meshes. One possible method is to encode vertices using V-PCC and then encode connectivity using a mesh compression method such as TFAN or Edgebreaker. A limitation of this method is that the original mesh must be dense so that the point cloud generated from the vertices is not sparse and can be efficiently encoded after projection. Furthermore, since the order of vertices affects the coding of connectivity, different methods have been proposed for reorganizing the connectivity of the mesh. An alternative method for encoding sparse meshes is to encode the positions of 3D vertices using RAW patch data. Since RAW patches directly encode (x,y,z), in this method all vertices are encoded as RAW data, while connectivity is encoded by a similar mesh compression method as described above. Note that in RAW patches, vertices can be sent in any preferred order, so the order generated from connectivity encoding can be used. While this method can encode sparse point clouds, RAW patches are not efficient for encoding 3D data, and further data such as triangular face attributes may be missing from this method. [Overview of the project] [Problems that the invention aims to solve]

[0007] The new syntax elements are used to extend patch types, and the syntax will be added to the V3C standard. The new syntax defines patches that encode a mesh by projecting connected triangles onto a 2D surface, patches that encode triangles or strips of triangles without projection, or patches that are tracked over time and encoded by projecting connected triangles onto a 2D surface. Furthermore, this syntax allows mesh-specific information to be coded in different ways. For example, this syntax allows three different encoding methods for vertex positions: explicit (added directly to the atlas stream), embedded in video data (occupied map data), or encoded using an external mesh encoder. [Means for solving the problem]

[0008] In one embodiment, a method programmed into the device's non-temporary memory includes the steps of: encoding a mesh using patches; and performing an encoding implementation of vertex locations, selected from directly adding vertex location information to an atlas stream, embedding the vertex location information into video data, or using an external mesh coder. The step of encoding the mesh using patches includes projecting connected triangles onto a two-dimensional surface. The step of encoding the mesh using patches includes using patches that encode triangles or strips of triangles without projection. The step of encoding the mesh using patches includes using patches that are tracked over time and encoded by projecting connected triangles onto a two-dimensional surface. The video data includes occupancy map data. The step of performing the encoding implementation of vertex locations includes delta information for geometry correction. The method further includes the steps of: encoding patch connectivity using a binary implementation or an explicit implementation; encoding patch vertices using a binary implementation, an occupancy map implementation or an explicit implementation; and encoding patch mapping coordinates using a binary implementation, an explicit implementation or an implicit implementation.

[0009] In another embodiment, the device includes non-temporary memory for storing an application, the application being for performing: encoding a mesh using patches; performing an encoding implementation of vertex locations, selected from: directly adding vertex location information to an atlas stream; embedding the vertex location information in video data; or using an external mesh coder; and a processor coupled to the memory and configured to process the application. Encoding the mesh using patches includes projecting connected triangles onto a two-dimensional surface. Encoding the mesh using patches includes using patches that encode triangles or strips of triangles without projection. Encoding the mesh using patches includes using patches that are tracked over time and encoded by projecting connected triangles onto a two-dimensional surface. The video data includes occupancy map data. Performing the encoding implementation of vertex locations includes delta information for geometry correction. The device further includes encoding patch connectivity using a binary or explicit implementation; encoding patch vertices using a binary, occupancy map, or explicit implementation; and encoding patch mapping coordinates using a binary, explicit, or implicit implementation.

[0010] In another embodiment, the system includes one or more cameras for acquiring 3D content, and an encoder configured to perform a vertex location encoding implementation, selected from encoding a mesh using patches and adding vertex location information directly to an atlas stream, embedding the vertex location information into video data, or using an external mesh coder. Encoding the mesh using patches includes projecting connected triangles onto a 2D surface. Encoding the mesh using patches includes using patches that encode triangles or strips of triangles without projection. Encoding the mesh using patches includes using patches that are tracked over time and encoded by projecting connected triangles onto a 2D surface. The video data includes occupancy map data. Performing the vertex location encoding implementation includes delta information for geometry correction. The system further includes encoding patch connectivity using a binary or explicit implementation, encoding patch vertices using a binary, occupancy map, or explicit implementation, and encoding patch mapping coordinates using a binary, explicit, or implicit implementation. [Brief explanation of the drawing]

[0011] [Figure 1] This is a diagram of mesh patch data syntax according to several embodiments. [Figure 2] This is a diagram of a triangular primitive in several embodiments. [Figure 3] This is a diagram illustrating color extension according to several embodiments. [Figure 4] This is a diagram of the traced mesh patch data syntax in several embodiments. [Figure 5] This figure shows flowcharts of patch mesh coding methods according to several embodiments. [Figure 6] This is a block diagram of an exemplary computer device configured to implement a patch mesh coding method according to several embodiments.

Best Mode for Carrying Out the Invention

[0012] The new syntax elements are used to extend the patch type, and the syntax is added to the V3C standard. The new syntax defines patches that encode a mesh by projecting connected triangles onto a 2D surface, patches that encode triangles or triangle strips without projection, or patches that are tracked over time and encoded by projecting connected triangles onto a 2D surface. Further, in this syntax, mesh-specific information can be coded in different ways. For example, this syntax enables three different coding methods for vertex positions, which can be coded explicitly (added directly to the atlas stream), embedded in video data (occupation map data), or coded using an external mesh encoder.

[0013] An alternative way to represent mesh information using existing patch syntax elements is to use mesh extensions, as was done in MIV. This also minimizes the impact on the V3C specification. The text shown in this document indicates what is added to the specification. For example, the mesh patch type can be an extension of the patch data unit, as shown in this document. TIFF0007881740000001.tif234170

[0014] The triangle patch can be an extension of the raw patch data, as shown below. TIFF0007881740000002.tif133169

[0015] Finally, the tracked mesh patch type can be an extension of the inter-patch data, as shown below. TIFF0007881740000003.tif167170

[0016] The extensions include specific mesh information associated with each patch. The following coding options are possible in the syntax. Two types of connectivity coding: explicit coding (each triangle is represented by three or four vertex indices in the patch data), or binary coding (an external encoder is used). There are three types of vertex coding: explicit coding (two coordinates indicate the position within the patch, from which a third coordinate is derived), embedded in an occupancy map (carrying symbols that identify the position of vertices within the patch), or binary coding (using an external encoder). Three types of UV coordinate coding: explicit coding (sending coordinates within the patch data), implicit coding (assuming the corresponding vertex has the same value as the patch position), or binary (using an external mesh encoder).

[0017] Figure 1 shows diagrams of mesh patch data syntax according to several embodiments. Patch connectivity can be encoded using a binary implementation or an explicit implementation. Patch vertices can be encoded using a binary implementation, an occupancy map implementation, or an explicit implementation. Patch mapping coordinates can be encoded using a binary implementation, an explicit implementation, or an implicit implementation.

[0018] The new syntactic elements and semantics are explained below. TIFF0007881740000004.tif211170 TIFF0007881740000005.tif78170 If mpdu_binary_object_present_flag[tileID][p] is equal to 1, it specifies that the syntax elements mpdu_mesh_binary_object_size_bytes[tileID][p] and mpdu_mesh_binary_object[tileID][p][i] exist for the patch at index p of the current atlas style where the tile ID is equal to tileID. If mpdu_binary_object_present_flag[tileID][p] is equal to 0, the syntax elements mpdu_mesh_binary_object_size_bytes[tileID][p] and mpdu_mesh_binary_object[tileID][p][i] do not exist for the current patch. If mpdu_binary_object_present_flag[tileID][p] does not exist, its value is presumed to be equal to 0. mpdu_mesh_binary_object_size_bytes[tileID][p] specifies the number of bytes used to represent the mesh information of the patch at index p of the current Atlas style, where the tile ID is equal to tileID, in binary format. mpdu_mesh_binary_object[tileID][p][i] specifies i bytes of the binary representation of the mesh of the patch at index p of the current Atlas style, where the tile ID is equal to tileID. mpdu_vertex_count_minus3[tileID][p]+3 specifies the number of vertices present in the patch at index p of the current atlas style, where the tile ID is equal to tileID. mpdu_face_count[tileID][p] specifies the number of triangles present in the patch at index p of the current Atlas style whose tile ID is equal to tileID. If none exist, the value of mpdu_face_count[tileID][p] is set to 0. mpdu_face_vertex[tileID][p][i][k] specifies the k-th value of the vertex index of the i-th triangle or quadrilateral in the current patch at index p of the current atlas style where the tile ID is equal to tileID. The value of mpdu_face_vertex[tileID][p][i][k] is within the range of 0 to mpdu_vert_count_minus3[tileID][p]+2 (including the values ​​at both ends). mpdu_vertex_pos_x[tileID][p][i] specifies the x-coordinate of the i-th vertex of the current patch at index p of the current atlas style, where the tile ID is equal to tileID. The value of mpdu_vertex_pos_x[p][i] is within the range of 0 to mpdu_2d_size_x_minus1[tileID][p] (including the values ​​at both ends). mpdu_vertex_pos_y[tileID][p][i] specifies the y-coordinate of the i-th vertex of the current patch p of the current atlas style whose tile ID is equal to tileID. The value of mpdu_vertex_pos_y[tileID][p][i] is within the range of 0 to mpdu_2d_size_y_minus1[tileID][p] (including the values ​​at both ends). mpdu_vertex_pos_delta_z[tileID][p][i] specifies the difference between the value derived from the geometry video and the z-coordinate of the i-th vertex of the current patch p of the current atlas style where the tile ID is equal to tileID. mpdu_vertex_u_coord[tileID][p][i] specifies the u-coordinate value of the mapping for the i-th vertex of the current patch at index p of the current atlas style, where the tile ID is equal to tileID. The value of mpdu_vertex_u_coord[p][i] is between 0 and 2 asps_mesh_coordinates_bit_depth_minus1 + 1 The range is considered to be up to -1 (including the values ​​at both ends). mpdu_vertex_pos_y[tileID][p][i] specifies the y-coordinate value of the i-th vertex of the current patch p of the current atlas style whose tile ID is equal to tileID. The value of mpdu_vertex_pos_y[tileID][p][i] is between 0 and 2 asps_mesh_coordinates_bit_depth_minus1 + 1 The range is considered to be up to -1 (including the values ​​at both ends).

[0019] Figure 2 shows diagrams of triangular primitives according to several embodiments. The triangles can be configured as individual triangles 200, triangular strips 202, or triangular fans 204. Other triangular configurations are also possible.

[0020] Figure 3 shows diagrams of color extension according to several embodiments. Color extension includes x, y, and z packing implementations and line packing implementations. TIFF0007881740000006.tif39170 tpdu_vertices_minus3[tileID][p]+3 specifies the number of vertices present in the triangle coding patch at index p of the current Atlas style, where the tile ID is equal to tileID. The value of tpdu_vertices_minus3[tileID][p] is within the range of ( (tpdu_2d_size_x_minus1[tileID][p] + 1) * (tpdu_2d_size_y_minus1[tileID][p] + 1) ) / 3 - 3 (including the values ​​at both ends). tpdu_primitive_idc[tileID][p] indicates a geometry primitive that defines how to obtain triangles from vertices present in the coded patch at index p of the current Atlas style, where the tile ID is equal to tileID. If tpdu_primitive_idc[tileID][p] does not exist, its value is presumed to be equal to 0. If tpdu_color_expansion_flag[tileID][p] is equal to 1, it specifies that the vertex coordinates are packed line-interleaved and the color value is expanded for the current patch p of the current atlas style where the tile ID is equal to tileID. If tpdu_color_expansion_flag[tileID][p] is equal to 0, the vertex coordinates are packed sequentially and the color is not expanded for the current patch. If tpdu_color_expansion_flag[tileID][p] does not exist, its value is presumed to be equal to 0.

[0021] Figure 4 shows diagrams of the traced mesh patch data syntax in several embodiments. TIFF0007881740000007.tif185170 TIFF0007881740000008.tif202170 TIFF0007881740000009.tif168170 If tmpdu_rotation_present_flag[ t ][ p ] is equal to 1, it indicates that the rotation parameter exists for the patch at index p and the tile with tile ID t. If tmpdu_rotation_present_flag[ t ][ p ] is equal to 0, it indicates that the rotation parameter does not exist for the patch at index p and the tile with tile ID t. If tmpdu_rotation_present_flag[ t ][ p ] does not exist, it is presumed to be equal to 0. tmpdu_3d_rotation_qx[ t ][ p ] specifies the x component qX of the geometric rotation of the patch at index p and the tile with tile ID t, using a quaternion representation. The value of tmpdu_3d_rotation_qx[ t ][ p ] is -2 14 From 2 14It is within the range from -1 (including the values at both ends). If tmpdu_3d_rotation_qx[t][p] does not exist, its value is assumed to be 0. The value of qX is calculated as follows. qX = tmpdu_3d_rotation_qx,2 14 tmpdu_3d_rotation_qy[t][p] specifies the y-component qY of the geometric rotation of the patch with index p and the tile with tile ID t using quaternion representation. The value of tmpdu_3d_rotation_qy[t][p] is from -2 14 to 2 14 within the range (including the values at both ends). If tmpdu_3d_rotation_qy[t][p] does not exist, its value is assumed to be 0. The value of qY is calculated as follows. qY = tmpdu_3d_rotation_qy,2 14 tmpdu_3d_rotation_qz[t][p] specifies the z-component qZ of the geometric rotation of the patch with index p and the tile with tile ID t using quaternion representation. The value of tmpdu_3d_rotation_qz[t][p] is from -2 14 to 2 14 within the range (including the values at both ends). If tmpdu_3d_rotation_qz[t][p] does not exist, its value is assumed to be 0. The value of qZ is calculated as follows. qZ = tmpdu_3d_rotation_qz,2 14 The fourth component qW of the geometric rotation of the patch with index p and the tile with tile ID t using quaternion representation is calculated as follows. qW = Sqrt(1 - (qX 2 + qY 2 + qZ 2 ))

[0022] The unit quaternion can be represented as a rotation matrix R as follows. JPEG0007881740000010.jpg20157 If tmpdu_vertices_changed_position_flag[ t ][ p ] is equal to 1, it indicates that vertex displacements exist for the patch at index p and the tile with tile ID t. If tmpdu_vertices_changed_position_flag[ t ][ p ] is equal to 0, it indicates that vertex displacements do not exist for the patch at index p and the tile with tile ID t. If tmpdu_vertices_changed_position_flag[ t ][ p ] does not exist, it is presumed to be equal to 0. tmpdu_vertex_delta_pos_x[ t ][ p ][ i ] specifies the difference between the x-coordinate value of the i-th vertex of the patch with index p and tile ID t, and the matching patch indicated by tmpdu_ref_index[ t ][ p ]. The value of tmpdu_vertex_pos_x[ t ][ p ][ i ] is within the range of 0 to 2afps_num_bits_vertex_delta_x - 1 (including the values ​​at both ends). tmpdu_vertex_delta_pos_y[t][p][i] specifies the difference between the y-coordinate of the i-th vertex of the patch with index p and tile ID t, and the matching patch indicated by tmpdu_ref_index[t][p]. The value of tmpdu_vertex_pos_y[t][p][i] is within the range of 0 to 2afps_num_bits_vertex_delta_y - 1 (including the values ​​at both ends). tmpdu_vertex_pos_delta_z[tileID][p][i] specifies the difference between the value derived from the geometry video and the z-coordinate of the i-th vertex of the current patch p of the current atlas style where the tile ID is equal to tileID. If tmpdu_binary_object_present_flag[tileID][p] is equal to 1, it specifies that the syntax elements tmpdu_mesh_binary_object_size_bytes[tileID][p] and tmpdu_mesh_binary_object[tileID][p][i] exist for the patch at index p of the current atlas style where the tile ID is equal to tileID. If tmpdu_binary_object_present_flag[tileID][p] is equal to 0, the syntax elements tmpdu_mesh_binary_object_size_bytes[tileID][p] and tmpdu_mesh_binary_object[tileID][p][i] do not exist for the current patch. If tmpdu_binary_object_present_flag[tileID][p] does not exist, its value is presumed to be equal to 0. tmpdu_mesh_binary_object_size_bytes[tileID][p] specifies the number of bytes used to represent the mesh information of the patch at index p of the current Atlas style, where the tile ID is equal to tileID, in binary format. tmpdu_mesh_binary_object[tileID][p][i] specifies i bytes of the binary representation of the mesh of the patch at index p of the current Atlas style, where the tile ID is equal to tileID. tmpdu_vertex_count_minus3[tileID][p]+3 specifies the number of vertices present in the patch at index p of the current Atlas style, where the tile ID is equal to tileID. tmpdu_face_count[tileID][p] specifies the number of triangles present in the patch at index p of the current Atlas style whose tile ID is equal to tileID. If none exist, the value of tmpdu_face_count[tileID][p] is set to 0. tmpdu_face_vertex[tileID][p][i][k] specifies the k-th value of the vertex index of the i-th triangle or quadrilateral in the current patch at index p of the current atlas style, where the tile ID is equal to tileID. The value of tmpdu_face_vertex[tileID][p][i][k] is within the range of 0 to tmpdu_vert_count_minus3[tileID][p]+2 (including the values ​​at both ends). tmpdu_vertex_pos_x[tileID][p][i] specifies the x-coordinate of the i-th vertex of the current patch at index p of the current atlas style, where the tile ID is equal to tileID. The value of tmpdu_vertex_pos_x[p][i] is within the range of 0 to tmpdu_2d_size_x_minus1[tileID][p] (including the values ​​at both ends). tmpdu_vertex_pos_y[tileID][p][i] specifies the y-coordinate value of the i-th vertex of the current patch p of the current atlas style whose tile ID is equal to tileID. The value of tmpdu_vertex_pos_y[tileID][p][i] is within the range of 0 to tmpdu_2d_size_y_minus1[tileID][p] (including the values ​​at both ends). tmpdu_vertex_u_coord[tileID][p][i] specifies the u-coordinate value of the mapping for the i-th vertex of the current patch at index p of the current atlas style, where the tile ID is equal to tileID. The value of tmpdu_vertex_u_coord[p][i] is between 0 and 2 asps_mesh_coordinates_bit_depth_minus1 + 1 The range is considered to be up to -1 (including the values ​​at both ends). tmpdu_vertex_v_coord[tileID][p][i] specifies the y-coordinate value of the i-th vertex of the current patch p of the current atlas style whose tile ID is equal to tileID. The value of tmpdu_vertex_v_coord[tileID][p][i] is between 0 and 2 asps_mesh_coordinates_bit_depth_minus1 + 1 The range is considered to be up to -1 (including the values ​​at both ends).

[0023] In addition to the syntactic elements mentioned above, the following mesh extensions are proposed by adding and modifying the current atlas sequence parameter set. TIFF0007881740000011.tif203170 TIFF0007881740000012.tif195170 TIFF0007881740000013.tif98170

[0024] Also, the following semantic changes: If asps_extension_present_flag is equal to 1, it specifies that the syntax elements asps_vpcc_extension_present_flag, asps_miv_extension_present_flag, asps_mesh_extension_present_flag, and asps_extension_5bits exist within the atlas_sequence_parameter_set_rbsp() syntax structure. If asps_extension_present_flag is equal to 0, it indicates that the syntax elements asps_vpcc_extension_present_flag, asps_miv_extension_present_flag, asps_mesh_extension_present_flag, and asps_extension_5bits do not exist. If asps_mesh_extension_present_flag is equal to 1, it specifies that the asps_mesh_extension() syntax structure exists within the atlas_sequence_parameter_set_rbsp() syntax structure. If asps_mesh_extension_present_flag is equal to 0, it specifies that this syntax structure does not exist. If it does not exist, the value of asps_mesh_extension_present_flag is inferred to be equal to 0. If asps_extension_5bits is equal to 0, it indicates that the asps_extension_data_flag syntax element does not exist within the ASPS RBSP syntax structure. If it exists, asps_extension_5bits is treated as equal to 0 in bitstreams conforming to this version of this document. If the value of asps_extension_5bits is not 0, it is reserved for future use by ISO / IEC. The decoder shall allow the value of asps_extension_5bits to be non-equal to 0 and ignore all asps_extension_data_flag syntax elements within the ASPS NAL unit. If it does not exist, the value of asps_extension_5bits is presumed to be equal to 0.

[0025] The ASPS mesh extension is used to identify mesh codecs used for connectivity binary coding, the use of quadrilaterals, the presence of vertex data within the occupied map, and explicit coding of mapping coordinates, as shown in the following syntax and semantics. TIFF0007881740000014.tif98170 If asps_mesh_binary_coding_enabled_flag is equal to 1, it indicates that vertex and connectivity information associated with the patch exists in binary format. If asps_mesh_binary_coding_enabled_flag is equal to 0, it specifies that the mesh vertex and connectivity data does not exist in binary format. If they do not exist, asps_mesh_binary_coding_enabled_flag is presumed to be 0. The `asps_mesh_binary_codec_id` indicates the identifier of the codec used to compress the patch's vertex and connectivity information. The `asps_mesh_binary_codec_id` is within the range of 0 to 255 (including values ​​at both ends). TIFF0007881740000015.tif39170 If asps_mesh_quad_face_flag is equal to 1, it indicates that quadrilaterals will be used for the polygon representation. If asps_mesh_quad_face_flag is equal to 0, it indicates that triangles will be used for the polygon representation of the mesh. If it is not present, the value of asps_mesh_quad_flag is presumed to be equal to 0. If asps_mesh_vertices_in_occupancy_video_data_flag is equal to 1, it indicates that vertex information exists in the occupied video data. If asps_mesh_vertices_in_occupancy_video_data_flag is equal to 0, it indicates that vertex information exists in the patch data. If it does not exist, the value of asps_mesh_vertices_in_occupancy_video_data_flag is presumed to be equal to 0. If asps_mesh_mapping_coordinates_present_flag is equal to 1, it indicates that mapping information exists associated with the patch's vertices. If asps_mesh_mapping_coordinates_present_flag is equal to 0, it specifies that no mapping information exists associated with the patch's vertices and that it is assumed to be equal to the vertex atlas position. If none exists, asps_mesh_mapping_coordinates_present_flag is inferred to be 0. asps_mesh_coordinates_bit_depth_minus1+1 indicates the bit depth of the mapping information associated with the vertices of the patch. asps_mesh_coordinates_bit_depth_minus1 is considered to be in the range of 0 to 31 (including values ​​at both ends). If asps_mesh_vertices_delta_z_present_flag is equal to 1, it indicates that there is a difference between the z coordinate reconstructed from the geometry image and the actual value. If asps_mesh_vertices_delta_z_present_flag is equal to 0, it specifies that there is no difference value and it is assumed to be equal to 0. If it does not exist, asps_mesh_vertices_delta_z_present_flag is inferred to be 0.

[0026] The conversion of vertex positions from atlas coordinates to 3D coordinates is changed as follows:

[0027] The inputs to the process are as follows: Variable pIdx (patch index), Variable depthValue (point depth), Variable x (x atlas coordinate), The variable y (y-atlas coordinate).

[0028] The process output is a 2D array pos3D of size 3, which specifies the 3D coordinates of the points.

[0029] The following applies: The variable oIdx is set to AtlasPatchOrientationIndex[pIdx]. The variables posX, posY, sizeX, sizeY, lodX, and lodY are assigned as follows: posX = AtlasPatch2dPosX[ pIdx ] posY = AtlasPatch2dPosY[ pIdx ] deltaZ = AtlasPatch2dPosDeltaZ[ pIdx ] sizeX = AtlasPatch2dSizeX[ pIdx ] sizeY = AtlasPatch2dSizeY[ pIdx ] lodX = AtlasPatchLoDScaleX[ pIdx ] lodY = AtlasPatchLoDScaleY[ pIdx ]

[0030] The atlas coordinates (x,y) are converted to local patch coordinate pairs (u,v) as follows: JPEG0007881740000016.jpg9156 Here, Ro and Rs are specified (for example, in the table). The local patch coordinate pair (u,v) is converted to 3D coordinates as follows: pos3D[ AtlasPatchAxisU[ pIdx ] ] = AtlasPatch3dOffsetU[ pIdx ] + u pos3D[ AtlasPatchAxisV[ pIdx ] ] = AtlasPatch3dOffsetV[ pIdx ] + v tempD = ( 1 - 2* AtlasPatchProjectionFlag[ pIdx ] ) * (depthValue + deltaZ) pos3D[AtlasPatchAxisD[pIdx]] = Max(0, AtlasPatch3dOffsetD[ pIdx ] + tempD) Some V-MESH decoder implementations can choose to clip the reconstructed 3D coordinates to the range of 0 to (1 << (asps_geometry_3d_bit_depth_minus1 + 1)) - 1 (including the values ​​at both ends). Other decoder implementations can apply the clipping operation after the 45-degree transformation or after the reconstruction process.

[0031] As described in U.S. Patent Application Serial Number Agent Reference Number Sony-75300 (this application is incorporated herein by reference in its entirety for all purposes), there are three methods for encoding vertex mapping information: implicit, explicit, and binary. In an implicit implementation, when projecting onto a 2D surface, the projection is the same as the mapping. For example, the location hit when projecting onto a projection surface is the UV coordinates. In an explicit implementation, even though projection is performed, different coordinates are sent to the texture. In a binary implementation, explicit information is encoded by an external encoder (e.g., Draco or AFX).

[0032] If binary coding is implemented, an external mesh encoder can be used to encode the patch mesh information. U and V are added to the ply, and vertex mapping information is encoded along with the ply. In some embodiments, delta information for the z coordinate is added. The delta information can be used for geometry correction.

[0033] Further details regarding mesh compression can be found in U.S. Patent Application No. 17 / 322,662, “Video-Based Mesh Compression,” filed 17 May 2021; U.S. Provisional Patent Application No. 63 / 088,705, “Video-Based Mesh Compression,” filed 7 October 2020; and U.S. Provisional Patent Application No. 63 / 087,958, “Video-Based Mesh Compression,” filed 6 October 2020, all of which are incorporated herein by reference in their entirety for all purposes. Connectivity of a 3D mesh or 2D patch mesh can be encoded using temporal correlation with the use of occupancy maps and video-based mesh compression.

[0034] Figure 5 shows flowcharts of patch mesh coding methods according to several embodiments. In step 500, the mesh is coded using patches. Coded using patches includes projecting connected triangles onto a two-dimensional surface, using patches that code triangles or strips of triangles without projection, or using patches that are tracked over time and coded by projecting connected triangles onto a two-dimensional surface. In step 502, an implementation of coding vertex locations is performed, selected from adding vertex location information directly to the atlas stream, embedding vertex location information in video data, or using an external mesh coder. Delta information can be transmitted for geometry correction. In some embodiments, the order of the steps is changed. In some embodiments, fewer or additional steps are implemented. For example, coding patch connectivity using a binary or explicit implementation, coding patch vertices using a binary, occupy-map, or explicit implementation, and coding patch mapping coordinates using a binary, explicit, or implicit implementation.

[0035] Figure 6 shows a block diagram of an exemplary computer device configured to implement a patch mesh coding method according to several embodiments. The computer device 600 can be used to acquire, store, compute, process, communicate, and / or display information such as images and videos, including 3D content. The computer device 600 can implement any of the encoding / decoding modes. Generally, a suitable hardware structure for implementing the computer device 600 includes a network interface 602, memory 604, a processor 606, (one or multiple) I / O devices 608, a bus 610, and a storage device 612. The choice of processor is not critical as long as a suitable processor of sufficient speed is selected. The memory 604 can be any conventional computer memory known in the art. The storage device 612 can include a hard drive, CD-ROM, CDRW, DVD, DVDRW, high-definition disk / drive, ultra-high-definition drive, flash memory card, or any other storage device. The computer device 600 can include one or more network interfaces 602. Examples of network interfaces include a network card connected to Ethernet or other types of LANs. The (single or duplicate) I / O devices 608 may include one or more of the following: a keyboard, mouse, monitor, screen, printer, modem, touchscreen, button interface, and other devices. The storage devices 612 and memory 604 store the (single or duplicate) patch mesh coding application 630 used to perform the patch mesh coding implementation and are likely to be processed as the application normally would. The computer device 600 may also include more or fewer components than those shown in Figure 6. In some embodiments, patch mesh coding hardware 620 is included. The computer device 600 in Figure 6 includes the application 630 and hardware 620 for the implementation of patch mesh coding, but the patch mesh coding method may also be implemented on the computer device in hardware, firmware, software, or a combination of these.For example, in some embodiments, the patch mesh coding application 630 is programmed in memory and executed using a processor. In another example, in some embodiments, the patch mesh coding hardware 620 is programmed hardware logic including gates specifically designed to implement the patch mesh coding method.

[0036] In some embodiments, the (single or multiple) patch mesh coding application 630 includes multiple applications and / or modules. In some embodiments, a module also includes one or more submodules. In some embodiments, fewer modules or additional modules may be included.

[0037] Examples of suitable computer devices include personal computers, laptop computers, computer workstations, servers, mainframe computers, handheld computers, personal digital assistants (PDAs), cellular / mobile phones, smart home appliances, game consoles, digital cameras, digital camcorders, camera phones, smartphones, portable music players, tablet computers, mobile devices, video players, video disc writers / players (e.g., DVD writers / players, high-definition disc writers / players, ultra-high-definition disc writers / players), televisions, home entertainment systems, augmented reality devices, virtual reality devices, smart jewelry (e.g., smartwatches), vehicles (e.g., autonomous vehicles), or any other suitable computer device.

[0038] To utilize the patch mesh coding method, the device acquires or receives 3D content (e.g., point cloud content). The patch mesh coding method can be implemented with or without user assistance.

[0039] During operation, the patch mesh coding method enables more efficient and accurate 3D content encoding compared to conventional implementations.

[0040] Some embodiments of mesh patch syntax 1. A method programmed into the non-temporary memory of a device, The steps involve encoding the mesh using patches, The steps include performing a vertex position encoding implementation selected from directly adding vertex position information to an atlas stream, embedding the vertex position information into video data, or using an external mesh coder, A method that includes this.

[0041] 2. The method according to paragraph 1, wherein the step of encoding the mesh using patches includes projecting connected triangles onto a two-dimensional surface.

[0042] 3. The method according to paragraph 1, wherein the step of encoding the mesh using patches includes using patches that encode triangles or triangular strips without projection.

[0043] 4. The method according to paragraph 1, wherein the step of encoding the mesh using patches is to use patches that are tracked over time and encoded by projecting connected triangles onto a two-dimensional surface.

[0044] 5. The method according to paragraph 1, wherein the video data includes the occupied map data.

[0045] 6. The method according to paragraph 1, wherein the step of performing the encoding implementation of vertex positions includes delta information for geometry correction.

[0046] 7. The step of encoding patch connectivity using a binary implementation or an explicit implementation, The steps include encoding patch vertices using a binary implementation, an occupy map implementation, or an explicit implementation, The steps include encoding the patch mapping coordinates using a binary implementation, explicit implementation, or implicit implementation, The method described in paragraph 1, further including the method described in paragraph 1.

[0047] 8. A device, Non-temporary memory for storing applications, wherein the applications are Encoding the mesh using patches, This involves performing a vertex position encoding implementation selected from directly adding vertex position information to the atlas stream, embedding the vertex position information into video data, or using an external mesh coder. Non-temporary memory is used to perform this task, A processor coupled to the memory and configured to process the application, A device that includes this.

[0048] 9. The apparatus described in paragraph 8, wherein encoding the mesh using patches includes projecting connected triangles onto a two-dimensional surface.

[0049] 10. The apparatus described in paragraph 8, which involves encoding the mesh using patches, including using patches that encode triangles or triangular strips without projection.

[0050] 11. The apparatus described in paragraph 8, wherein encoding the mesh using patches is tracked over time and involves using patches that are encoded by projecting connected triangles onto a two-dimensional surface.

[0051] 12. The apparatus described in paragraph 8, which includes the video data and the occupied map data.

[0052] 13. The apparatus described in Section 8, which performs the encoding implementation of vertex positions, including delta information for geometry correction.

[0053] 14. Encoding patch connectivity using a binary implementation or explicit implementation, Encoding patch vertices using a binary implementation, an occupy map implementation, or an explicit implementation, Encoding patch mapping coordinates using a binary implementation, explicit implementation, or implicit implementation, The apparatus described in paragraph 8, further including the following.

[0054] 15. A system, One or more cameras for acquiring 3D content, Use patches to encode the mesh, The system performs a vertex position encoding implementation selected from the following: directly adding vertex position information to the atlas stream, embedding the vertex position information into video data, or using an external mesh coder. An encoder configured as follows, A system that includes this.

[0055] 16. The system described in Section 15, wherein encoding the mesh using patches includes projecting connected triangles onto a two-dimensional surface.

[0056] 17. Encoding the mesh using patches, the system described in paragraph 15, which includes using patches that encode triangles or triangular strips without projection.

[0057] 18. The system described in paragraph 15, wherein encoding the mesh using patches is tracked over time and involves using patches that are encoded by projecting connected triangles onto a two-dimensional surface.

[0058] 19. The system described in paragraph 15, in which the video data includes the occupied map data.

[0059] 20. The system described in Section 15, which performs the encoding implementation of vertex positions, including delta information for geometry correction.

[0060] 21. Encoding patch connectivity using a binary implementation or explicit implementation, Encoding patch vertices using a binary implementation, an occupy map implementation, or an explicit implementation, Encoding patch mapping coordinates using a binary implementation, explicit implementation, or implicit implementation, The system described in paragraph 15, further including the following.

[0061] The present invention has been described in relation to specific embodiments, including details, to facilitate understanding of its structure and operating principles. Such references to specific embodiments and their details herein are not intended to limit the claims appended herein. Those skilled in the art will readily see that various other modifications can be made to the embodiments selected for illustrative purposes without departing from the spirit and scope of the invention as defined by the claims. [Explanation of Symbols]

[0062] 200 triangles 202 Triangular Strips 204 Triangular Fan Encode the mesh using 500 patches. 502 Implementing the encoding of vertex positions 600 Computer devices 602 Network Interface 604 memory 606 Processor 608 I / O devices 610 Bus 612 Storage device 620 Patch Mesh Coding Hardware 630 Patch Mesh Coding Application

Claims

1. A method programmed into the device's non-temporary memory, The steps involve encoding the mesh using patches, The steps include performing a vertex position encoding implementation selected from directly adding vertex position information to an atlas stream, embedding the vertex position information into video data, or using an external mesh coder, Includes, The step of performing the coding implementation of vertex positions is characterized by including delta information for geometry correction.

2. The method according to claim 1, characterized in that the step of encoding the mesh using patches includes projecting connected triangles onto a two-dimensional surface.

3. The method according to claim 1, characterized in that the step of encoding the mesh using patches includes using patches that encode triangles or triangular strips without projection.

4. The method according to claim 1, characterized in that the step of encoding the mesh using patches includes using patches that are tracked over time and encoded by projecting connected triangles onto a two-dimensional surface.

5. The method according to claim 1, characterized in that the video data includes occupancy map data.

6. The steps include: encoding patch connectivity using a binary implementation or an explicit implementation; The steps include encoding patch vertices using a binary implementation, an occupy map implementation, or an explicit implementation, The steps include encoding the patch mapping coordinates using a binary implementation, explicit implementation, or implicit implementation, The method according to claim 1, further comprising:

7. It is a device, Non-temporary memory for storing applications, wherein the applications are Encoding the mesh using patches, This involves performing a vertex position encoding implementation selected from directly adding vertex position information to the atlas stream, embedding the vertex position information into video data, or using an external mesh coder. Non-temporary memory is used to perform this task, A processor coupled to the memory and configured to process the application, Includes, The apparatus is characterized in that the encoding implementation of vertex positions includes delta information for geometry correction.

8. The apparatus according to claim 7, characterized in that encoding the mesh using patches includes projecting connected triangles onto a two-dimensional surface.

9. The apparatus according to claim 7, characterized in that encoding the mesh using patches includes using patches that encode triangles or triangular strips without projection.

10. The apparatus according to claim 7, characterized in that encoding the mesh using patches is tracked over time and the patches are encoded by projecting connected triangles onto a two-dimensional surface.

11. The apparatus according to claim 7, characterized in that the video data includes occupancy map data.

12. Encoding patch connectivity using a binary or explicit implementation, Encoding patch vertices using a binary implementation, an occupy map implementation, or an explicit implementation, Encoding patch mapping coordinates using a binary implementation, explicit implementation, or implicit implementation, The apparatus according to claim 7, further comprising the following:

13. It is a system, One or more cameras for acquiring 3D content, Use patches to encode the mesh, The system performs a vertex position encoding implementation selected from the following: directly adding vertex position information to the atlas stream, embedding the vertex position information into video data, or using an external mesh coder. An encoder configured as follows, Includes, The system is characterized in that the coding implementation of vertex positions includes delta information for geometry correction.

14. The system according to claim 13, characterized in that encoding the mesh using patches includes projecting connected triangles onto a two-dimensional surface.

15. The system according to claim 13, characterized in that encoding the mesh using patches includes using patches that encode triangles or triangular strips without projection.

16. The system according to claim 13, characterized in that encoding the mesh using patches is tracked over time and the patches are encoded by projecting connected triangles onto a two-dimensional surface.

17. The system according to claim 13, characterized in that the video data includes occupancy map data.

18. Encoding patch connectivity using a binary or explicit implementation, Encoding patch vertices using a binary implementation, an occupy map implementation, or an explicit implementation, Encoding patch mapping coordinates using a binary implementation, explicit implementation, or implicit implementation, The system according to claim 13, further comprising the following:

Images