Patch Generation for Dynamic Mesh Coding

Abstract

The architecture includes a new block that transforms the connectivity of the mesh to enable lossy compression. Additionally, the method generates surface patches from the mesh and generates the patches for efficient encoding using the V3C standard while preserving the global aspect of the object. Triangles are joined together into connected components and a conditional projection onto the surface of each triangle is performed to enforce connectivity at the projected surface. A new depth filtering algorithm can be used to improve compression performance. Here we describe the new blocks that are added for connectivity compression and the techniques that are introduced to ensure 2D connectivity to enable transmission of UV texture maps and to filter depth values ​​to avoid high frequency edges in the depth image.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority under 35 U.S.C. § 119 to U.S. Provisional Patent Application No. 63 / 269,910, entitled “PATCH GENERATION FOR DYNAMIC MESH CODING,” filed March 25, 2022, which is incorporated herein by reference in its entirety for all purposes.

[0002] The present invention relates to three-dimensional graphics, and more particularly to coding of three-dimensional graphics. [Background technology]

[0003] In recent years, a new method for compressing volumetric content such as point clouds is becoming standardized, based on a 3D to 2D projection. Also known as V3C (visual volumetric video-based compression), this method maps the 3D volume data into a number of 2D patches, which are then placed into an atlas image, which is then encoded by a video encoder. The atlas image corresponds to the geometry of the points, their respective textures, and an occupancy map that indicates which locations should be considered for point cloud reconstruction.

[0004] In 2017, MPEG issued a Call for Proposals (CfP) for the compression of point clouds. After evaluation of several proposals, MPEG is currently considering two different techniques for point cloud compression: 3D native coding techniques (based on octrees and similar coding methods) or 3D to 2D projection followed by traditional video coding. For dynamic 3D scenes, MPEG is using Test Model Software (TMC2), based on patch surface modeling, projection of the patches from 3D to a 2D image, and coding of the 2D image by a video encoder such as HEVC. This method has proven to be more efficient than native 3D coding and can achieve competitive bitrates with acceptable quality.

[0005] Due to the success of coding 3D point clouds with projection-based methods (also known as video-based methods, or V-PCC), future versions of this standard are expected to include additional 3D data, such as 3D meshes. However, the current version of this standard is only suitable for transmitting a set of unconnected points, and therefore has no mechanism for transmitting point connectivity as required for 3D mesh compression.

[0006] Methods have also been proposed to extend the functionality of V-PCC to meshes. One possible method is to use V-PCC to encode the vertices and then use a mesh compression method such as TFAN or Edgebreaker to encode the connectivity. A limitation of this method is that the original mesh needs to be dense so that the point cloud generated from the vertices is not sparse and can be efficiently encoded after projection. Furthermore, the order of the vertices affects the coding of connectivity, so different methods have been proposed to reorganize the connectivity of the mesh. An alternative method for encoding sparse meshes is to use the raw patch data to encode the 3D vertex positions. Since raw patches directly encode (x,y,z), in this method all vertices are encoded as raw data, while the connectivity is encoded by a similar mesh compression method, as mentioned before. It is noted that in raw patches, the vertices can be sent in any preferred order, so the order generated from the connectivity encoding can be used. While this method can encode sparse point clouds, raw patches are not efficient for encoding 3D data and further data, such as triangular face attributes, may be missing from this method. Summary of the Invention [Problem to be solved by the invention]

[0007] The architecture includes a new block that transforms the connectivity of the mesh to enable lossy compression. Additionally, the method generates surface patches from the mesh and generates the patches for efficient encoding using the V3C standard while preserving the global aspect of the object. Triangles are joined together into connected components and a conditional projection onto the surface of each triangle is performed to enforce connectivity at the projected surface. A new depth filtering algorithm can be used to improve compression performance. This document describes new blocks that are added for connectivity compression and techniques that are introduced to ensure 2D connectivity to enable transmission of UV texture maps and to filter depth values ​​to avoid high frequency edges in the depth image. [Means for solving the problem]

[0008] In one aspect, a method of patch generation includes the steps of: computing face normals of a triangle; computing face adjacencies of the triangle; computing face neighbors of the triangle; implementing an initial segmentation including classifying the normals of the triangle; performing refinement including identifying and smoothing outliers in the classified normals; and performing patch segmentation of the triangle. The steps of computing face adjacencies of the triangle and computing face neighbors of the triangle include generating a list of adjacent triangles and neighboring triangles. Generating the list of adjacent triangles and neighboring triangles includes taking into account texture map boundaries. Taking into account texture map boundaries includes excluding triangles on the other side of the boundary of the current triangle. The step of performing patch segmentation includes removing degenerates, checking for vertex occlusion, checking half edges, checking for surface occlusion, depth filtering, analyzing depth range and connectivity, and rasterizing triangles. Removing degenerate triangles includes removing triangles that are lines or points by determining that the area between their points is less than a threshold; checking vertex occlusion includes determining whether a vertex of the triangle is occluded by an already projected triangle; checking half-edges includes determining whether the triangle shares an edge that is already shared by two projected triangles; checking surface occlusion includes determining whether an area between points of the triangle is occluded or may occlude the already projected triangle; depth filtering includes determining whether a depth difference between a neighboring point and a point of the triangle is less than a depth threshold; analyzing depth range and connectivity includes determining whether the triangle borders at least one edge or vertex of the already projected triangle; and projecting and rasterizing the triangle.When the criteria are not met, the triangle is added to a missing triangles list.

[0009] In another aspect, an apparatus includes a non-transitory memory for storing an application, the application for: calculating a surface normal of a triangle; calculating a surface adjacency of the triangle; calculating a surface neighborhood of the triangle; implementing an initial segmentation including classifying the normal of the triangle; performing refinement including identifying and smoothing outliers in the classified normal; and performing a patch segmentation of the triangle; and a processor coupled to the memory and configured to process the application. The calculating the surface adjacency of the triangle and the calculating the surface neighborhood of the triangle include generating a list of adjacent triangles and neighboring triangles. The generating the list of adjacent triangles and neighboring triangles includes taking into account texture map boundaries. Taking into account texture map boundaries includes excluding triangles on the other side of a boundary of a current triangle. Performing patch segmentation includes removing degenerates, checking vertex occlusions, checking half-edges, checking surface occlusions, depth filtering, depth range and connectivity analysis, and rasterizing triangles. Removing degenerate triangles includes removing triangles that are lines or points by determining if the area between points is less than a threshold, checking vertex occlusions includes determining if the vertices of the triangle are occluded by an already projected triangle, checking half-edges includes determining if the triangle shares an edge that is already shared by two projected triangles, checking surface occlusions includes determining if the area between the points of the triangle is occluded or may occlude the already projected triangle, depth filtering determines if the depth difference between a neighboring point and a point of the triangle is less than a depth threshold, depth range and connectivity analysis includes determining if the triangle is bordered by at least one edge or vertex of the already projected triangle, and projecting and rasterizing the triangle.When the criteria are not met, the triangle is added to a missing triangle list.

[0010] In another aspect, a method for patch segmentation includes removing degenerate information, checking triangle vertex occlusion, checking the triangle half-edges, checking the triangle surface occlusion, depth filtering the triangle, performing depth range and connectivity analysis of the triangle, and projecting and rasterizing the triangle onto a canvas. Removing degenerate triangles includes removing triangles that are lines or points by determining that the area between points is less than a threshold. Checking vertex occlusion includes determining whether the triangle vertices are occluded by an already projected triangle. Checking half-edges includes determining whether the triangle shares an edge that is already shared by two projected triangles. Checking surface occlusion includes determining whether the area between the triangle points is occluded or may occlude the already projected triangle. Depth filtering determines whether the depth difference between a neighboring point and the triangle point is less than a depth threshold. The depth range and connectivity analysis includes determining whether the triangle touches at least one edge or vertex of the already projected triangle. When the criteria are not met, the triangle is added to a missing triangle list. [Brief description of the drawings]

[0011] [Figure 1] 1 is a flowchart of a method for implementing dynamic mesh coding according to some embodiments. [Diagram 2] 1 is a flowchart of a method for implementing patch generation according to some embodiments. [Diagram 3] FIG. 13 is a diagram illustrating calculating normals according to some embodiments. [Figure 4]FIG. 1 is an illustration of face adjacency and face neighborhood analysis according to some embodiments. [Diagram 5] FIG. 1 is an illustration of patch generation according to some embodiments. [Figure 6] 1A-1C are flowcharts and diagrams of patch segmentation using conditional rasterization according to some embodiments. [Figure 7] FIG. 1 is an illustration of depth filtering according to some embodiments. [Figure 8] 1 is a block diagram of an example computing device configured to implement a patch generation method for dynamic mesh coding according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] The architecture includes a new block that transforms the connectivity of the mesh to enable lossy compression. Additionally, the method generates surface patches from the mesh and generates the patches for efficient encoding using the V3C standard while preserving the global aspect of the object. Triangles are joined together into connected components and a conditional projection onto the surface of each triangle is performed to enforce connectivity at the projected surface. A new depth filtering algorithm can be used to improve compression performance. This document describes new blocks that are added for connectivity compression and techniques that are introduced to ensure 2D connectivity to enable transmission of UV texture maps and to filter depth values ​​to avoid high frequency edges in the depth image. The architecture described herein is related to U.S. Patent Application No. 17 / 322,662, entitled "VIDEO BASED MESH COMPRESSION," filed May 17, 2021, U.S. Provisional Patent Application No. 63 / 088,705, entitled "VIDEO BASED MESH COMPRESSION," filed October 7, 2020, and U.S. Provisional Patent Application No. 63 / 087,958, entitled "VIDEO BASED MESH COMPRESSION," filed October 6, 2020, all of which are incorporated by reference in their entirety for all purposes. Additionally, the methods described herein are related to U.S. Patent Application No. 17 / 161,300, entitled "PROJECTION-BASED MESH COMPRESSION," filed January 28, 2021, which is incorporated by reference in its entirety for all purposes.

[0013] 1 illustrates a flowchart of a method for implementing dynamic mesh coding according to some embodiments. An input mesh 100 is received or acquired. For example, the input mesh is downloaded (e.g., from a network device) or acquired / captured by a device (e.g., a camera or an autonomous vehicle).

[0014] Mesh voxelization 102 is performed. Since meshes can have vertex positions in floating point numbers, these positions are transformed to integer space. V-PCC and V3C assume a voxelized point cloud.

[0015] Next, 3D mesh simplification 104 is performed. 3D mesh simplification 104 is performed before patch generation. 3D mesh simplification 104 reduces the number of triangles to project and generates patches that contain low-resolution information at the boundaries, allowing high-resolution information to be recovered from the geometry image, but the boundaries are still low-resolution.

[0016] In step 106, patch generation (or creation) is implemented. Patch generation 106 includes normal calculation 150, adjacency calculation 152, neighborhood calculation 154, initial segmentation 156, refinement 158, and patch segmentation 160. Normal calculation 150 calculates the normal of each triangle (e.g., the cross product of the triangle's edges). Adjacency calculation 152 calculates the adjacency of each triangle (e.g., which triangles in the mesh are neighbors or touching the current triangle or other triangles). Neighborhood calculation 154 includes calculating neighboring triangles. Initial segmentation 156 includes classifying normals according to their orientation. For example, triangle normals can point up, down, left, right, front, or back and can be classified based on the direction / orientation. In some embodiments, triangles are colored based on the orientation of their normals (e.g., all triangles with normals pointing up are colored green). Refinement 158 ​​includes identifying outliers (e.g., a single red triangle surrounded by blue triangles) and smoothing the outliers (e.g., modifying the single red triangle to match its neighboring blue triangles). Refinement 158 ​​is performed by analyzing the neighborhood and smoothing the orientation (e.g., adjusting the orientation of normals). Once a smooth surface is achieved, patch segmentation 160 is performed to segment the patch for specific classifications of triangles (e.g., based on orientation). Segmentation can include patch projection. In a segmentation, vertices and connectivity are displayed on the patch. For example, the body and face in this example are separate segmentations because there are different classifications of triangles separating them. In some embodiments, the segmentation is rasterized (e.g., sampling points on the surface, including the distance of the points, to generate a geometry image and attributes of the surface). The rasterized mesh surface is very similar to the V3C image.

[0017] The result of patch generation is a segmented patch or a rasterized mesh surface, along with vertex positions and connectivity, which are utilized in patch packing, step 108. Patch packing 108 may include packing the patches into an atlas or a 2D canvas.

[0018] After patch packing 108, the process continues with texture image generation 110, geometry image generation 112, and 2D mesh simplification 114. 2D mesh simplification 114 is performed after patch generation. 2D mesh simplification 114 reduces only triangles within the patch (boundary triangles are not reduced), generating a patch that contains high-resolution information at the boundary, and the high-resolution information can be recovered from the geometry image including the boundary regions stored in the patch.

[0019] Image padding 116 occurs from texture image generation 110, and the padded image is then sent to video codec 122 for encoding. The texture image stores texture information. The image padding 116 provides spacing between patches. The video codec 122 can perform any suitable encoding scheme.

[0020] Image padding 118 occurs from geometry image generation 112, and the padded image is then sent to video codec 124 for encoding. The geometry image stores the geometry information. The image padding 118 provides spacing between patches. The video codec 124 can perform any suitable encoding scheme.

[0021] Connectivity encoding 120 occurs from the 2D mesh simplification 114, after which the connectivity information is sent to mesh zippering 126 and atlas codec 128 for encoding. Mesh zippering 126 also receives information from video codec 124. Connectivity encoding 120 involves encoding connectivity information of points. Mesh zippering 126 is an implementation for removing gaps between triangles or patches.

[0022] The multiplexer 130 outputs a V3C bitstream. The V3C bitstream allows reconstruction of the point cloud and / or construction of a mesh. The point cloud and / or mesh can be extracted from the V3C bitstream, which allows for great flexibility. In some embodiments, fewer or additional steps are implemented. In some embodiments, the order of the steps is changed.

[0023] FIG. 2 shows a flow chart of a method for implementing patch generation according to some embodiments. Patch segmentation produces connected components 200. Connected components 200 are neighboring and / or adjacent triangles that have the same classification (e.g., the same color in FIG. 5). For each triangle in the connected component, conditional rasterization 202 is performed. That is, a triangle is projected into a patch only if it meets specified criteria. This ensures that the triangle is projected consistently. If a triangle does not meet the specified criteria, it is considered a missing triangle, the missing triangle is put back into a list, and the analysis process with the specified criteria is repeated with different neighboring or adjacent triangles.

[0024] Rasterization criteria (e.g., whether a triangle is projected) include degeneracy removal 204, vertex occlusion check 206, half-edge check 208, surface occlusion check 210, depth filtering 212, depth range and connectivity 214, and triangle rasterization 216.

[0025] Removing degenerate triangles 204 involves removing triangles that are lines (e.g., when two points overlap and there is a line rather than a triangle). Lines or points are removed and only triangles that have at least a certain area (e.g., above a threshold) are kept.

[0026] Checking vertex occlusion 206 involves determining whether a triangle vertex is occluded: if a triangle vertex is occluded (or has the potential to occlude) by an already projected triangle, the projection is not allowed and the triangle is removed from the list and moved to a missing triangle list.

[0027] Check Halfedge 208 determines if the triangle to be projected shares an edge that is already shared by two projected triangles: if so, the projection is not allowed and the triangle is removed from the list and moved to the missing triangle list.

[0028] In the surface occlusion check 210, if a triangle vertex is not occluded, but the area between the points is (or could occlude) an already projected triangle (e.g., a vertex of an already projected triangle), the projection is not allowed and the triangle is removed from the list and moved to the missing triangle list.

[0029] Depth filtering 212 verifies that the depth difference of neighboring points in the triangle is less than a threshold (e.g., the current triangle point is compared to the neighboring triangle point). If the depth difference is greater than the threshold, the triangle is not projected.

[0030] Depth range and connectivity 214 involves determining whether the triangle to be projected touches at least one of the already projected triangles (e.g., a shared edge or vertex). If there is no connectivity between the triangle to be projected and the already projected triangles, the projection is not allowed and the triangle is removed from the list and moved to a missing triangle list. Depth range is also checked.

[0031] If there is connectivity and an acceptable depth range (and all other criteria are met), the triangle is allowed to be projected (e.g., onto the canvas) and the triangle is rasterized (216) (e.g., to generate points on the surface and generate geometry and texture images to encode the connectivity on the patch).

[0032] 3 shows a diagram of calculating normals according to some embodiments. In some embodiments, only surface normals are calculated. A normal 300 is the cross product of two vectors (e.g., AB and AC). Image 302 shows an example image including surface normals.

[0033] FIG. 4 shows an illustration of face adjacency and face neighborhood analysis according to some embodiments. Triangle 400 has many adjacent and neighboring triangles. An adjacent triangle is a triangle that shares at least one vertex with the specified triangle. For example, triangles 1 through 15 {T1, ..., T 15} all share a vertex with triangle 400. Neighboring triangles share edges with the triangle. For example, triangles 4, 10, 14 {T4, T 10 , T 14} share an edge with triangle 400. A list of adjacent and neighboring triangles can be kept for each triangle. A flag can also be included to keep bounds.

[0034] Three-dimensional (3D) triangles are mapped onto two-dimensional (2D) surfaces. To do so, the surface of the mesh is separated (e.g., into different groups or sections). For example, texture map boundary 402 separates one portion or group 404 on the texture map from a second portion or group 406 on the texture map. In other words, even if two triangles are next to each other in 3D space (e.g., they are neighboring triangles), they may be separated in 2D space (e.g., they are not neighboring triangles) because they are in different groups in 2D space. To maintain the texture map boundary, the texture map boundary 402 is taken into account when generating the two lists of adjacent and neighboring triangles. Thus, taking into account the texture map boundary 402 (e.g., excluding triangles that are not on the same side of the boundary as the target triangle), for triangle 400, the list of neighboring triangles is {T6, ..., T 14} and the list of neighboring triangles is {T 10 , T 14}. Thus, when a patch is projected, the boundary is preserved and the texture scene of the region is saved so that it can be used on the decoder side.

[0035] FIG. 5 shows a diagram of patch generation according to some embodiments. As described, patch generation involves normal calculation, adjacency calculation, initial segmentation (or normal categorization), and segmentation refinement (or category refinement). Normal calculation for each triangle involves a cross product between the edges of the triangle. Adjacency calculation determines whether the triangles share a vertex, and if so, determines that the triangles are neighboring triangles. Initial segmentation and segmentation refinement are performed similarly to V-PCC by analyzing the normal orientation, classifying the normal orientation (e.g., up, down, left, right, front, back), determining whether the normal orientation is classified differently compared to neighboring normals that are all similarly classified (e.g., a first patch is classified as up-facing while most or all neighboring patches are classified as front-facing), and then changing the classification of the patch's normal to match the neighboring normal orientation (e.g., changing the classification of the first patch to front-facing).

[0036] As described, a patch generation is implemented that segments the mesh into patches. The patch generation also generates 1) a rasterized mesh surface, and 2) vertex positions and connectivity information. The rasterized mesh surface is a set of points that undergoes V3C image or V-PCC image generation and is encoded as a V3C image or V-PCC image. The vertex positions and connectivity information are received for base mesh coding.

[0037] The patch generation described herein is similar to that in V-PCC. However, instead of calculating normals per point, we calculate normals per triangle. We calculate normals per triangle using the cross product between edges to determine normal vectors. We then categorize the triangles according to their normals. For example, we divide the normals into n (e.g., 6) categories, such as front, back, top, bottom, left and right. The normals are shown in different colors to indicate the initial segmentation. In FIG. 5, different colors, such as black and light gray, are shown in grayscale to indicate different normals. It may be difficult to see, but the top surface (e.g., the top of the person's head, the top of the ball and the top of the sneakers) is one color (e.g., green), the first side of the person / ball is very dark and represents another color (e.g., red), the bottom of the ball is another color (e.g., purple), and the front of the person and ball, which is mostly light gray, represents another color (e.g., cyan).

[0038] The main direction can be found by multiplying the product of the normals by the direction. A smoothing / refinement process can be implemented by looking at neighboring triangles. For example, a triangle that has more than a threshold number of neighbors that are all blue will be classified as blue even if there is an anomaly that would initially be shown as red.

[0039] Generate connected components of triangles to identify which triangles have the same color (eg, triangles of the same category that share at least one vertex).

[0040] Connectivity information describes how points are connected in 3D. These connections (more specifically, three different connections sharing three points) combine to generate triangles, which result in a surface (represented by a group of triangles). Triangles are described herein, but other geometric shapes (e.g., rectangles) are also possible.

[0041] Color is used to encode connectivity by distinguishing triangles of different colors: each triangle that is distinguished by three connections is coded with a unique color.

[0042] FIG. 6 shows a flowchart and diagram of patch segmentation using conditional rasterization according to some embodiments. Patch segmentation produces connected components 200. Connected components 200 are neighboring and / or adjacent triangles with the same classification (e.g., the same color in FIG. 5). For each triangle in the connected component, conditional rasterization 202 is performed. That is, a triangle is projected onto a patch only if it meets a specified criterion. This ensures that the triangle is projected consistently. If a triangle does not meet the specified criterion, it is considered a missing triangle, and the missing triangle is put back into a list and retried with different neighboring or adjacent triangles, and the analysis process with the specified criterion is repeated.

[0043] Rasterization criteria (e.g., whether a triangle is projected) include degeneracy removal 204, vertex occlusion check 206, half-edge check 208, surface occlusion check 210, depth filtering 212, depth range and connectivity 214, and triangle rasterization 216.

[0044] Removing degenerate triangles 204 involves removing triangles that are lines (e.g., when two points overlap and there is a line rather than a triangle). Lines or points are removed and only triangles that have at least a certain area (e.g., above a threshold) are kept.

[0045] Checking vertex occlusion 206 involves determining whether a triangle vertex is occluded: if a triangle vertex is occluded (or has the potential to occlude) by an already projected triangle, the projection is not allowed and the triangle is removed from the list and moved to a missing triangle list.

[0046] Check Halfedge 208 determines if the triangle to be projected shares an edge that is already shared by two projected triangles: if so, the projection is not allowed and the triangle is removed from the list and moved to the missing triangle list.

[0047] In the surface occlusion check 210, if a triangle vertex is not occluded, but the area between the points is (or could occlude) an already projected triangle (e.g., a vertex of an already projected triangle), the projection is not allowed and the triangle is removed from the list and moved to the missing triangle list.

[0048] Depth filtering 212 verifies that the depth difference of neighboring points in the triangle is less than a threshold (e.g., the current triangle point is compared to the neighboring triangle point). If the depth difference is greater than the threshold, the triangle is not projected.

[0049] Depth range and connectivity 214 involves determining whether the triangle to be projected touches at least one of the already projected triangles (e.g., a shared edge or vertex). If there is no connectivity between the triangle to be projected and the already projected triangles, the projection is not allowed and the triangle is removed from the list and moved to a missing triangle list. Depth range is also checked.

[0050] If there is connectivity and an acceptable depth range (and all other criteria are met), the triangle is allowed to be projected (e.g., onto the canvas) and the triangle is rasterized (216) (e.g., to generate points on the surface and generate geometry and texture images to encode the connectivity on the patch).

[0051] FIG. 7 shows an illustration of depth filtering according to some embodiments. Depth filtering verifies that the depth difference of neighboring points of a triangle is less than a threshold. If the depth difference is greater than a threshold, the triangle is not projected. For example, the depth of the point and the depth of the neighboring points are analyzed. However, only neighboring points of other triangles are analyzed. In the illustration, the point labeled T is not analyzed because it is from a triangle. Triangle A (labeled T A ) and triangle B (labeled T B ) are analyzed. In other words, to compare the depth of point T in the center of the grid, the depth of point T is compared to the depth of point T in the bottom left square of the grid. A The depth of the grid, T in the middle left square A The depth of the grid, T A Depth of the grid, T in the top center square B and the depth of T in the top right square of the grid. B The depth of the compared point is compared to the depth of any other point (e.g., their difference is calculated). If the difference between the depth of the compared point and the depth of any other point is greater than a threshold, triangle 700 is not projected. By performing depth filtering, large gradients are avoided. Although a 3x3 filter is shown, larger filters can be implemented. In some embodiments, the threshold is adaptive, i.e., the threshold can be changed depending on circumstances such as the information received or the particular implementation.

[0052] FIG. 8 illustrates a block diagram of an exemplary computing device configured to implement a patch generation method for dynamic mesh coding according to some embodiments. The computing device 800 can be used to acquire, store, compute, process, communicate and / or display information such as images and videos including 3D content. The computing device 800 can implement any of the encoding / decoding aspects. In general, a hardware structure suitable for implementing the computing device 800 includes a network interface 802, a memory 804, a processor 806, an I / O device(s) 808, a bus 810 and a storage device 812. The selection of the processor is not critical as long as a suitable processor of sufficient speed is selected. The memory 804 can be any conventional computer memory known in the art. The storage device 812 can include a hard drive, a CDROM, a CDRW, a DVD, a DVDRW, a high definition disk / drive, an ultra high definition drive, a flash memory card, or any other storage device. The computing device 800 can include one or more network interfaces 802. Examples of network interfaces include a network card connected to an Ethernet or other type of LAN. The I / O device(s) 808 may include one or more of a keyboard, mouse, monitor, screen, printer, modem, touch screen, button interface, and other devices. The storage device 812 and memory 804 likely store a patch generation for dynamic mesh coding application(s) 830 used to execute the implementation of patch generation for dynamic mesh coding and process as an application would normally be processed. The computing device 800 may also include more or fewer components than those shown in FIG. 8. In some embodiments, patch generation for dynamic mesh coding hardware 820 is included.Although the computing device 800 of Figure 8 includes application 830 and hardware 820 for implementing the patch generation for dynamic mesh coding, the patch generation for dynamic mesh coding method may also be implemented on the computing device in hardware, firmware, software, or any combination thereof. For example, in some embodiments, the patch generation for dynamic mesh coding application 830 is programmed into memory and executed using a processor. As another example, in some embodiments, the patch generation for dynamic mesh coding hardware 820 is programmed hardware logic that includes gates specifically designed to implement the patch generation for dynamic mesh coding method.

[0053] In some embodiments, the patch generation application for dynamic mesh coding 830 includes multiple applications and / or modules. In some embodiments, a module also includes one or more sub-modules. In some embodiments, fewer or additional modules may be included.

[0054] Examples of suitable computing devices include a personal computer, a laptop computer, a computer workstation, a server, a mainframe computer, a handheld computer, a personal digital assistant, a cellular / mobile phone, a smart appliance, a game console, a digital camera, a digital camcorder, a camera phone, a smart phone, a portable music player, a tablet computer, a mobile device, a video player, a video disc writer / player (e.g., DVD writer / player, high definition disc writer / player, ultra high definition disc writer / player), a television, a home entertainment system, an augmented reality device, a virtual reality device, smart jewelry (e.g., a smart watch), a vehicle (e.g., a self-driving vehicle), or any other suitable computing device.

[0055] To utilize the patch generation method for dynamic mesh coding, a device acquires or receives 3D content (e.g., point cloud content). The patch generation method for dynamic mesh coding can be implemented with user assistance or automatically without user involvement.

[0056] In operation, the patch generation method for dynamic mesh coding enables more efficient and accurate 3D content encoding compared to conventional implementations.

[0057] Some embodiments of patch generation for dynamic mesh coding 1. A method of patch generation, comprising: calculating surface normals of the triangles; calculating face adjacencies of said triangles; calculating a face neighborhood of said triangle; implementing an initial segmentation comprising classifying the normals of the triangles; performing a refinement, which includes identifying and smoothing outliers in the classified normals; performing a patch segmentation of said triangles; The method includes:

[0058] 2. The method of claim 1, wherein the steps of calculating face adjacencies of the triangle and calculating face neighborhoods of the triangle include generating a list of adjacent triangles and neighboring triangles.

[0059] 3. The method of claim 2, wherein generating the list of adjacent triangles and neighboring triangles includes taking into account texture map boundaries.

[0060] 4. The method of claim 3, wherein taking into account the texture map boundary includes excluding triangles on the opposite side of the boundary of the current triangle.

[0061] 5. The method according to claim 1, wherein the step of performing patch segmentation includes removing degeneracy, checking vertex occlusion, checking half-edges, checking surface occlusion, depth filtering, depth range and connectivity analysis, and rasterizing triangles.

[0062] 6. Removing degenerate triangles includes removing triangles that are lines or points by determining that the area between the points is less than a threshold; checking for vertex occlusion includes determining whether a vertex of the triangle is occluded by an already projected triangle; checking the half-edges includes determining whether the triangle shares an edge that is already shared by two projected triangles; checking for surface occlusion includes determining whether areas between points of the triangle are occluded or may occlude the already projected triangle; Depth filtering determines whether the depth difference between neighboring points and the triangle point is less than a depth threshold; analyzing the depth range and connectivity includes determining whether the triangle borders at least one edge or vertex of the already projected triangle; projecting and rasterizing said triangles; 5. The method according to claim 5.

[0063] 7. The method of claim 5, wherein when the criteria are not met, the triangle is added to a missing triangle list.

[0064] 8. An apparatus comprising: A non-transitory memory for storing an application, the application comprising: Calculating the surface normals of the triangle; calculating face adjacencies of said triangles; calculating a face neighborhood of said triangle; implementing an initial segmentation comprising classifying the normals of the triangles; performing a refinement that includes identifying and smoothing outliers in the classified normals; performing a patch segmentation of said triangles; a non-transient memory for performing a processor coupled to the memory and configured to process the application; An apparatus comprising:

[0065] 9. The apparatus of claim 8, wherein calculating the face adjacencies of the triangle and calculating the face neighborhoods of the triangle include generating a list of adjacent triangles and neighboring triangles.

[0066] 10. The apparatus of claim 9, wherein generating the list of adjacent triangles and neighboring triangles includes taking into account texture map boundaries.

[0067] 11. The apparatus of claim 10, wherein taking into account the texture map boundary includes excluding triangles on the other side of the boundary of the current triangle.

[0068] 12. The apparatus of claim 8, wherein performing patch segmentation includes removing degeneracy, checking vertex occlusion, checking half-edges, checking surface occlusion, depth filtering, depth range and connectivity analysis, and rasterizing triangles.

[0069] 13. Removing degenerate triangles includes removing triangles that are lines or points by determining that the area between the points is less than a threshold; checking for vertex occlusion includes determining whether a vertex of the triangle is occluded by an already projected triangle; checking the half-edges includes determining whether the triangle shares an edge that is already shared by two projected triangles; checking for surface occlusion includes determining whether areas between points of the triangle are occluded or may occlude the already projected triangle; Depth filtering determines whether the depth difference between neighboring points and the triangle point is less than a depth threshold; analyzing the depth range and connectivity includes determining whether the triangle borders at least one edge or vertex of the already projected triangle; projecting and rasterizing said triangles; 13. The apparatus according to claim 12.

[0070] 14. The apparatus of claim 12, wherein when the criteria are not met, the triangle is added to a missing triangle list.

[0071] 15. A method of patch segmentation comprising: removing degenerate information; checking for occlusion of triangle vertices; checking the half-edges of the triangle; checking for occlusion of the surface of said triangle; depth filtering of the triangles; performing a depth range and connectivity analysis of said triangles; projecting and rasterizing the triangle onto a canvas; The method includes:

[0072] 16. The method of clause 15, wherein removing degenerate triangles includes removing triangles that are lines or points by determining that the area between the points is less than a threshold.

[0073] 17. The method of claim 15, wherein checking for vertex occlusion includes determining whether a vertex of the triangle is occluded by an already projected triangle.

[0074] 18. The method of claim 15, wherein checking the half-edge includes determining whether the triangle shares an edge that is already shared by two projected triangles.

[0075] 19. The method of claim 15, wherein checking for surface occlusion includes determining whether areas between points of the triangle are occluded or have the potential to occlude the already projected triangle.

[0076] 20. The method of claim 15, wherein depth filtering determines whether the depth difference between neighboring points and the triangle point is less than a depth threshold.

[0077] 21. The method of claim 15, wherein the depth range and connectivity analysis includes determining whether the triangle borders at least one edge or vertex of the already projected triangle.

[0078] 22. The method of claim 15, wherein when the criteria are not met, the triangle is added to a missing triangle list.

[0079] The present invention has been described with respect to specific embodiments containing details so as to facilitate an understanding of the principles of construction and operation of the invention. Reference herein to such specific embodiments and their details is not intended to limit the scope of the claims appended hereto. It will be readily apparent to those skilled in the art that various other modifications can be made in the embodiments selected for illustration without departing from the spirit and scope of the invention as defined by the claims. [Explanation of symbols]

[0080] 100 input meshes 102 Mesh Voxelization 104 3D Mesh Simplification 106 Patch Generation 108 Patch Packing 110 Texture Image Generation 112 Geometry Image Generation 114 Simplifying 2D Meshes 116 Image Padding 118 Image Padding 120 Connectivity Coding 122 Video Codec 124 Video Codec 126 Mesh Zipper Ring 128 Atlas Codec 130 Multiplexer 150 Normal calculation 152 Adjacent Calculation 154 Neighborhood Calculation 156 Initial Segmentation 158 Refinement 160 Patch Segmentation 200 Connected Components 202 Conditional Rasterization 204 Elimination of Degeneracy 206 Check for vertex occlusion 208 Checking for half edges 210 Check for surface occlusion 212 Depth Filtering 214 Depth Range and Connectivity 216 Rasterization of Triangles 300 Normal 302 images 400 triangles 700 triangle 800 Computer Equipment 802 Network Interface 804 Memory 806 Processor 808 I / O device 810 Bus 812 Storage device 820 Patch Generation Hardware for Dynamic Mesh Coding 830 Patch Generation Application for Dynamic Mesh Coding

Claims

1. 1. A method of patch generation, comprising: calculating surface normals of the triangles; calculating face adjacency of the triangles; computing face neighbors of the triangle; implementing an initial segmentation comprising classifying the normals of the triangles; performing a refinement, which includes identifying and smoothing outliers in the classified normals; performing a patch segmentation of said triangles; The method according to claim 1, further comprising:

2. 2. The method of claim 1, wherein the steps of computing face adjacencies of a triangle and computing face neighborhoods of a triangle include generating lists of adjacent triangles and neighboring triangles.

3. 3. The method of claim 2, wherein generating the list of adjacent and neighboring triangles includes taking into account texture map boundaries.

4. 4. The method of claim 3, wherein taking into account texture map boundaries includes excluding triangles on the other side of a boundary of a current triangle.

5. 2. The method of claim 1, wherein the step of performing patch segmentation includes removing degenerates, checking vertex occlusion, checking half-edges, checking surface occlusion, depth filtering, depth range and connectivity analysis, and rasterizing triangles.

6. Removing degenerate triangles includes removing triangles that are lines or points by determining that the area between the points is less than a threshold; checking for vertex occlusion includes determining whether a vertex of the triangle is occluded by an already projected triangle; checking the half-edges includes determining whether the triangle shares an edge that is already shared by two projected triangles; checking for surface occlusion includes determining whether areas between points of the triangle are occluded or may occlude the already projected triangle; Depth filtering determines whether the depth difference between neighboring points and the triangle point is less than a depth threshold; analyzing the depth range and connectivity includes determining whether the triangle borders at least one edge or vertex of the already projected triangle; projecting and rasterizing said triangles; 6. The method according to claim 5 .

7. 6. The method of claim 5, wherein when the criteria are not met, the triangle is added to a missing triangles list.

8. An apparatus comprising: A non-transitory memory for storing an application, the application comprising: Calculating the surface normals of the triangle; calculating face adjacencies of said triangles; calculating a face neighborhood of said triangle; implementing an initial segmentation comprising classifying the normals of the triangles; performing a refinement that includes identifying and smoothing outliers in the classified normals; performing a patch segmentation of said triangles; a non-transient memory for performing a processor coupled to the memory and configured to process the application; An apparatus comprising:

9. 9. The apparatus of claim 8, wherein calculating the face adjacencies of the triangles and calculating the face neighborhoods of the triangles comprises generating lists of adjacent triangles and neighboring triangles.

10. The apparatus of claim 9 , wherein generating the list of adjacent and neighboring triangles includes taking into account texture map boundaries.

11. The apparatus of claim 10 , wherein taking into account the texture map boundary includes excluding triangles on opposite sides of a boundary of a current triangle.

12. 9. The apparatus of claim 8, wherein performing patch segmentation includes removing degeneracy, checking for vertex occlusion, checking for half-edges, checking for surface occlusion, depth filtering, depth range and connectivity analysis, and rasterizing triangles.

13. Removing degenerate triangles includes removing triangles that are lines or points by determining that the area between the points is less than a threshold; checking for vertex occlusion includes determining whether a vertex of the triangle is occluded by an already projected triangle; checking the half-edges includes determining whether the triangle shares an edge that is already shared by two projected triangles; checking for surface occlusion includes determining whether areas between points of the triangle are occluded or may occlude the already projected triangle; Depth filtering determines whether the depth difference between neighboring points and the triangle point is less than a depth threshold; analyzing the depth range and connectivity includes determining whether the triangle borders at least one edge or vertex of the already projected triangle; projecting and rasterizing said triangles; 13. Apparatus according to claim 12, characterized in that

14. 13. The apparatus of claim 12, wherein when the criteria are not met, the triangle is added to a missing triangle list.

15. 1. A method of patch segmentation comprising: removing degenerate information; checking for occlusion of triangle vertices; checking the half-edges of the triangle; checking for occlusion of the surface of said triangle; depth filtering of the triangles; performing a depth range and connectivity analysis of said triangles; projecting and rasterizing the triangle onto a canvas; The method according to claim 1, further comprising:

16. 16. The method of claim 15, wherein removing degenerate triangles includes removing triangles that are lines or points by determining that the area between points is less than a threshold.

17. 16. The method of claim 15, wherein checking for vertex occlusion comprises determining whether a vertex of the triangle is occluded by an already projected triangle.

18. 16. The method of claim 15, wherein checking half-edges includes determining whether the triangle shares an edge that is already shared by two projected triangles.

19. 16. The method of claim 15, wherein checking for surface occlusion includes determining whether areas between points of the triangle are occluded or have the potential to occlude the already projected triangle.

20. 16. The method of claim 15, wherein depth filtering determines whether the depth difference between neighboring points and the triangle point is less than a depth threshold.

21. 16. The method of claim 15, wherein analyzing depth range and connectivity includes determining whether the triangle borders at least one edge or vertex of the already projected triangle.

22. 16. The method of claim 15, wherein when the criteria are not met, the triangle is added to a missing triangle list.

Images