Simplification of mesh patches

Abstract

Here, we describe a method for simplifying patch connectivity data. Patches are generated considering high-resolution mesh information. The connectivity data is simplified at patch level, but the geometry image is preserved. The connectivity simplification only simplifies triangles within the patch. If the boundaries are preserved, reconstruction in 3D will not introduce artifacts. The high-resolution geometry image can be used to undo the simplification and improve the connectivity on the decoder side. Three embodiments of patch mesh simplification are described: quadratic error edge collapse, boundary distance edge collapse, and boundary triangles only.

Description

Technical Field

[0001] 〔Cross - Reference to Related Applications〕 This application claims priority under 35 U.S.C. § 119 to U.S. Provisional Patent Application No. 63 / 269,909, entitled "MESH PATCH SIMPLIFICATION", filed on March 25, 2022, which is hereby incorporated by reference in its entirety for all purposes.

[0002] The present invention relates to 3D graphics. More specifically, the present invention relates to the coding of 3D graphics.

Background Art

[0003] In recent years, new methods for compressing volumetric content such as point clouds based on projections from 3D to 2D are being standardized. This method, also known as V3C (Visual Volume Video - based Compression), maps 3D volume data onto several 2D patches, further arranges the patches in an atlas image, and then encodes it with a video encoder. The atlas image corresponds to the geometry of the points, their respective textures, and an occupancy map indicating which positions should be considered for point cloud reconstruction.

[0004] In 2017, MPEG issued a Call for Proposals (CfP) for point cloud compression. After evaluating several proposals, MPEG is currently considering two different techniques for point cloud compression: 3D native coding techniques (based on octree and similar coding methods), or conventional video coding after 3D-to-2D projection. For dynamic 3D scenes, MPEG uses Test Model Software (TMC2) based on patch surface modeling, projection of patches from 3D to 2D images, and coding of the 2D images by a video encoder such as HEVC. This method has proven to be more efficient than native 3D coding and can achieve competitive bitrates with acceptable quality.

[0005] Since coding 3D point clouds using projection-based methods (also known as video-based methods or V-PCC) has been successful, future versions of this standard are expected to include further 3D data, such as 3D meshes. However, the current version of this standard is only suitable for transmitting sets of unconnected points and lacks a mechanism for transmitting point connections, as required for 3D mesh compression.

[0006] Methods have also been proposed to extend the functionality of V-PCC to meshes. One possible method is to encode vertices using V-PCC and then encode connections using a mesh compression method such as TFAN or Edgebreaker. A limitation of this method is that the original mesh must be dense enough so that the point cloud generated from the vertices is not sparse and can be efficiently encoded after projection. Furthermore, since the order of vertices affects the coding of connections, different methods have been proposed for reorganizing the connections of the mesh. An alternative method for encoding sparse meshes is to encode the vertex positions in 3D using RAW patch data. In this method, all vertices are encoded as RAW data, as RAW patches directly encode (x,y,z), while connections are encoded by a similar mesh compression method as described above. Note that in RAW patches, vertices can be sent in any preferred order, so the order generated from connection encoding can be used. While this method can encode sparse point clouds, RAW patches are not efficient for encoding 3D data, and further data such as triangular face attributes may be missing from this method. [Overview of the project] [Problems that the invention aims to solve]

[0007] This specification describes methods for simplifying patch connectivity data. First, patches are generated considering high-resolution mesh information. Connectivity data is simplified at the patch level, but the geometric image is preserved. Connectivity simplification can only simplify triangles within a patch. When boundaries are preserved, 3D reconstruction does not introduce artifacts such as gaps between patches due to boundary vertex mismatches. High-resolution geometric images can be used to reverse the simplification and improve decoder-side connectivity. Three embodiments of patch mesh simplification are described: quadric error edge collapse, border distance edge collapse, and boundary triangle only. The first two methods perform edge collapse but with different criteria. The quadric error metric uses a 3D metric to check for potential distortions that may occur when moving vertices, enhancing the consistency of triangle normals and boundary preservation in 2D projection space. In the case of boundary distance, the criterion for edge collapse is the distance to the boundary, and the vertices furthest from the boundary are collapsed first. Finally, the method for sending only boundary triangles involves removing all triangles that are not connected to the boundary and sending only triangles that have at least one edge belonging to the patch boundary. [Means for solving the problem]

[0008] In one embodiment, a method programmed into the device's non-temporary memory includes the steps of generating a patch and performing a simplification of the two-dimensional patch mesh to generate a new face list and a new vertex list. Performing the simplification of the two-dimensional patch mesh includes an implementation of a quadratic error edge collapse. The implementation of the quadratic error edge collapse includes a half-edge collapse based on a quadratic error metric. Performing the simplification of the two-dimensional patch mesh includes an implementation of a boundary distance edge collapse. The implementation of the boundary distance edge collapse includes ordering a list of vertices according to the minimum distance to the boundary of the patch and collapsing the vertices having the furthest adjacent vertices from the boundary. Performing the simplification of the two-dimensional patch mesh includes an implementation of boundary triangles only. The implementation of boundary triangles only includes rotating the triangles so that the first edge of the triangles lies on the boundary. The method further includes, in the decoder, implementing a remeshing that includes a marching cube using the boundary triangles and the decoded geometry video.

[0009] In another embodiment, the device includes non-temporary memory for storing an application, the application being for generating a patch and performing a simplification of a two-dimensional patch mesh to generate a new face list and a new vertex list; and a processor coupled to the memory and configured to process the application. Performing the simplification of the two-dimensional patch mesh includes an implementation of a second-order error edge collapse. The implementation of the second-order error edge collapse includes a half-edge collapse based on a second-order error metric. Performing the simplification of the two-dimensional patch mesh includes an implementation of a boundary distance edge collapse. The implementation of the boundary distance edge collapse includes ordering a list of vertices according to the minimum distance to the boundary of the patch and collapsing the vertices having the furthest adjacent vertices from the boundary. Performing the simplification of the two-dimensional patch mesh includes an implementation of boundary triangles only. The implementation of boundary triangles only includes rotating the triangles so that the first edge of the triangles lies on the boundary. The application is further configured in the decoder to implement remeshing, including marching cubes using boundary triangles and the decoded geometry video.

[0010] In another embodiment, the system includes an encoder configured to generate patches and perform a simplification of the 2D patch mesh to generate a new list of faces and a new list of vertices, and a decoder configured to implement a remeshing that includes a marching cube using boundary triangles and a decoded geometry video. Performing the simplification of the 2D patch mesh includes an implementation of a second-order error edge collapse. The implementation of the second-order error edge collapse includes a half-edge collapse based on a second-order error metric. Performing the simplification of the 2D patch mesh includes an implementation of a boundary distance edge collapse. The implementation of the boundary distance edge collapse includes ordering a list of vertices according to the minimum distance to the boundary of the patch and collapsing the vertices having the furthest adjacent vertices from the boundary. Performing the simplification of the 2D patch mesh includes an implementation of boundary triangles only. The implementation of boundary triangles only includes rotating the triangles so that the first edge of the triangle is on the boundary. [Brief explanation of the drawing]

[0011] [Figure 1] This is a flowchart of a method for generating a new face list and a new vertex list according to several embodiments. [Figure 2] This is a diagram of quadric error collapse in several embodiments. [Figure 3] This figure shows an image of a mesh in which only boundary triangles have been transmitted according to several embodiments. [Figure 4] This is a diagram of boundary distance edge collapse in several embodiments. [Figure 5] This is a block diagram of an exemplary computer device configured to implement a mesh patch simplification method according to several embodiments. [Modes for carrying out the invention]

[0012] This specification describes methods for simplifying patch connectivity data. First, patches are generated considering high-resolution mesh information. Connectivity data is simplified at the patch level, but the geometric image is preserved. Connectivity simplification can only simplify triangles within a patch. When boundaries are preserved, 3D reconstruction does not introduce artifacts such as gaps between patches due to mismatched boundary vertices. High-resolution geometric images can be used to reverse the simplification and improve connectivity on the decoder side. Three embodiments of patch mesh simplification are described: quadratic error edge collapse, boundary distance edge collapse, and boundary triangle only. The first two methods perform edge collapse but with different criteria. The quadratic error metric uses a 3D metric to check for distortions that may occur when moving vertices, enhancing the consistency of triangle normals and boundary preservation in 2D projection space. In the case of boundary distance, the criterion for edge collapse is the distance to the boundary, and the vertices furthest from the boundary are collapsed first. Finally, the method for sending only boundary triangles involves removing all triangles that are not connected to the boundary and sending only triangles that have at least one edge belonging to the patch boundary.

[0013] Figure 1 shows flowcharts of methods for generating a new face list and a new vertex list according to several embodiments. In step 100, the input mesh is received or acquired. For example, the input mesh is downloaded (e.g., from a network device) or acquired / captured by a device (e.g., a camera or an autonomous vehicle). In step 102, the input mesh is segmented into patches. Any patch segmentation implementation can be performed. Patch segmentation generates a face list (e.g., connectivity) and a vertex list in step 104. Depth generation is performed in step 106. In step 108, a high-resolution depth map is generated based on the depth generation. In step 110, the patch obj can be used to regenerate the object using the face list and vertex list.

[0014] In step 112, patch generation (or creation) is implemented. In some embodiments, patch generation includes normal calculation, adjacency calculation, initial segmentation, refinement, patch projection, and patch rasterization. Normal calculation calculates the normal of each triangle (e.g., the cross product of the triangle's edges). Adjacency calculation calculates the adjacency relationships of each triangle (e.g., which triangles in the mesh are adjacent to or touching the current triangle or other triangles). Initial segmentation includes classifying the normals according to their orientation. For example, the normals of a triangle can point up, down, left, right, forward, or backward and can be classified based on their direction / orientation. In some embodiments, triangles are color-coded based on the orientation of their normals (e.g., all triangles with upward-pointing normals are green). Refinement identifies outliers (e.g., a single red triangle surrounded by blue triangles) and smooths them out (e.g., modifying a single red triangle to match its adjacent blue triangles). Refinement is performed by analyzing adjacent triangles and smoothing their orientations (e.g., adjusting the orientation of the normals). Once a smooth surface is obtained, patch projection is performed, projecting patches of a specific triangle classification (e.g., based on orientation). In this projection, vertices and connections are shown on the patches. For example, the body and face in this example are separate projections because there are triangles of different classifications separating them. However, V3C and V-PCC do not understand this and instead understand points, so the projection is rasterized (e.g., points on the surface are sampled, including the distance between points, to generate a geometric image and surface attributes). The rasterized mesh surface is very similar to the V3C image. Steps 100-112 can be performed in any standard implementation.

[0015] In step 114, simplification of the patch mesh (2D) is performed on the generated patch. By simplifying the patch mesh (2D), the amount of data to be transmitted (e.g., encoded) is reduced. However, the data transmitted is sufficient to restore the mesh. As described herein, there are three different implementations of the simplification of the patch mesh (2D), namely quadratic error edge collapse, boundary distance edge collapse, and only boundary triangles. In step 116, based on the simplification of the patch mesh (2D), a new face list and a new vertex list are generated and transmitted / encoded. In some embodiments, fewer steps or additional steps are implemented. For example, a decoder can be implemented to decode the patch mesh. In some embodiments, the order of the steps is changed.

[0016] FIG. 2 shows a diagram of quadratic error collapse according to some embodiments. The original patch mesh 200 has 334 faces. Using quadratic error calculation, a quadratic error metric can be determined.

[0017] Vertex v i (QM vi )'s quadratic error matrix is obtained as follows. JPEG0007862594000001.jpg1336In the above formula, N i is the set of triangles containing vertex i. The quadratic error matrix of the triangle with vertices (v A , v B , v C ) is calculated as follows. JPEG0007862594000002.jpg8035

[0018] Next, the cost of moving vertex v i to position v j is given by the following formula. Cost = TIFF0007862594000003.tif973

[0019] Based on the quadratic error metric, half-edge collapse can be implemented. As shown in the figure, the original connection 202 can collapse into a connection 204 that contains fewer points. The simplified patch 206 has 166 faces as compared to the 334 faces of the original patch mesh 200. The simplified patch 206 retains the vertices around the patch (e.g., the vertices of the outer edges), but the internal triangles have a different configuration from the original patch mesh. The complete original mesh 210 is also shown in comparison to the complete simplified mesh 212. By having fewer faces (and due to having fewer vertices), the data to be encoded is much less.

[0020] Figure 3 shows an image of a mesh where only boundary triangles have been transmitted according to some embodiments. Only the triangles on the boundary are retained. To know which triangles are on the boundary, the triangles are rotated so that the first edge of the triangle becomes an edge on the boundary. If there are multiple edges on the boundary, multiple triangles are transmitted. If there is no edge on the boundary of a triangle, that triangle is not transmitted. On the decoder side, remeshing is implemented including using marching cubes with the boundary triangles and the decoded geometry video. The geometry video indicates where the surface should be, and then the marching cubes are executed on the surface to regenerate the triangles. When executing the marching cubes, the process starts from the boundary triangles that have already been transmitted.

[0021] Figure 4 shows a diagram of boundary distance edge collapse according to some embodiments. Edge collapse can be based on the distance to the boundary. The list of vertices can be ordered according to the minimum distance to the boundary of the patch. The vertex with the adjacent vertex farthest from the boundary is collapsed. This implementation is more efficiently encoded, but the peak signal-to-noise ratio (PSNR) is lower. The implementation of vertex splitting can be used to improve the PSNR.

[0022] Vertices on the boundary can be properly decoded because the boundary can be restored using zippering. Regarding video compression, points in the center of a patch may exhibit fewer artifacts than points on the boundary, because central points are typically located in smoother regions.

[0023] As shown in the figure, the initial patch mesh 400 has 334 faces, while the patch mesh using boundary distance edge collapse 402 has 166 faces, which is the same as or similar to the number of faces (e.g., 166) of the second-order error implementation 404.

[0024] Figure 5 shows a block diagram of an exemplary computer device configured to implement a mesh patch simplification method according to several embodiments. The computer device 500 can be used to acquire, store, compute, process, communicate, and / or display information such as images and videos, including 3D content. The computer device 500 can implement any form of encoding / decoding. Generally, a suitable hardware structure for implementing the computer device 500 includes a network interface 502, memory 504, a processor 506, (one or more) I / O devices 508, a bus 510, and a storage device 512. The choice of processor is not critical as long as a suitable processor of sufficient speed is selected. The memory 504 can be any conventional computer memory known in the art. The storage device 512 can include a hard drive, CD-ROM, CDRW, DVD, DVDRW, high-definition disk / drive, ultra-high-definition drive, flash memory card, or any other storage device. The computer device 500 can include one or more network interfaces 502. Examples of network interfaces include a network card connected to Ethernet or other types of LANs. The (single or duplicate) I / O device 508 may include one or more of the following: a keyboard, mouse, monitor, screen, printer, modem, touch screen, button interface, and other devices. The storage device 512 and memory 504 store the (single or duplicate) mesh patch simplification application 530 used to perform the implementation of mesh patch simplification, and are likely to be processed as the application would normally be processed. The computer device 500 may also include more or fewer components than those shown in Figure 5. In some embodiments, mesh patch simplification hardware 520 is included. The computer device 500 in Figure 5 includes the application 530 and hardware 520 for the implementation of mesh patch simplification, but the mesh patch simplification method may also be implemented on the computer device in hardware, firmware, software, or a combination of these.For example, in some embodiments, the mesh patch simplification application 530 is programmed in memory and executed using a processor. In another example, in some embodiments, the mesh patch simplification hardware 520 is programmed hardware logic including gates specifically designed to implement the mesh patch simplification method.

[0025] In some embodiments, the (single or multiple) mesh patch simplification application 530 includes multiple applications and / or modules. In some embodiments, a module also includes one or more submodules. In some embodiments, fewer or more modules may be included.

[0026] Examples of suitable computer devices include personal computers, laptop computers, computer workstations, servers, mainframe computers, handheld computers, personal digital assistants (PDAs), cellular / mobile phones, smart home appliances, game consoles, digital cameras, digital camcorders, camera phones, smartphones, portable music players, tablet computers, mobile devices, video players, video disc writers / players (e.g., DVD writers / players, high-definition disc writers / players, ultra-high-definition disc writers / players), televisions, home entertainment systems, augmented reality devices, virtual reality devices, smart jewelry (e.g., smartwatches), vehicles (e.g., autonomous vehicles), or any other suitable computer device.

[0027] To utilize the mesh patch simplification method, the device acquires or receives 3D content (e.g., point cloud content). The mesh patch simplification method can be implemented with or without user assistance.

[0028] During operation, the mesh patch simplification method enables more efficient and accurate 3D content encoding compared to conventional implementations.

[0029] Several embodiments of mesh patch simplification 1. A method programmed into the non-temporary memory of a device, The steps to generate the patch, The steps include: performing simplification of a 2D patch mesh to generate a new face list and a new vertex list; A method that includes this.

[0030] 2. The method described in paragraph 1, which includes implementing a second-order error edge collapse, which involves performing the simplification of the two-dimensional patch mesh.

[0031] 3. The implementation of the second-order error edge collapse is the method described in Section 2, which includes a half-edge collapse based on a second-order error metric.

[0032] 4. Performing the simplification of the two-dimensional patch mesh is the method described in paragraph 1, including the implementation of boundary distance edge collapse.

[0033] 5. The method according to paragraph 4, wherein the implementation of the boundary distance edge collapse includes ordering a list of vertices according to the minimum distance to the boundary of the patch, and collapsing the vertex having the furthest adjacent vertex from the boundary.

[0034] 6. The method described in paragraph 1, which includes implementing only boundary triangles, to simplify the two-dimensional patch mesh.

[0035] 7. The method of paragraph 6, wherein the implementation of the boundary triangle only includes rotating the triangle so that the first edge of the triangle is on the boundary.

[0036] 8. The method of Section 6, further comprising implementing remeshing in the decoder, including a marching cube using boundary triangles and the decoded geometry video.

[0037] 9. A device, Non-temporary memory for storing applications, wherein the applications are Generating a patch, Simplifying the 2D patch mesh generates a new face list and a new vertex list, Non-temporary memory is used to perform this task, A processor coupled to the memory and configured to process the application, A device that includes this.

[0038] 10. The apparatus described in Section 9, which includes the implementation of a second-order error edge collapse, for performing the simplification of the two-dimensional patch mesh.

[0039] 11. The implementation of the second-order error edge collapse is the apparatus described in paragraph 10, which includes a half-edge collapse based on a second-order error metric.

[0040] 12. The apparatus described in Section 9, which includes the implementation of boundary distance edge collapse, is used to perform the simplification of the two-dimensional patch mesh.

[0041] 13. The apparatus according to paragraph 12, wherein the implementation of the boundary distance edge collapse includes ordering a list of vertices according to the minimum distance to the boundary of the patch, and collapsing the vertex having the furthest adjacent vertex from the boundary.

[0042] 14. The device described in Section 9, which performs the simplification of the two-dimensional patch mesh, including the implementation of boundary triangles only.

[0043] 15. The apparatus according to paragraph 14, wherein the implementation of the boundary triangle alone includes rotating the triangle so that the first edge of the triangle lies on the boundary.

[0044] 16. The apparatus according to Section 14, wherein the application is further configured in the decoder to implement remeshing including a marching cube using boundary triangles and a decoded geometry video.

[0045] 17. It is a system, Generate a patch, Perform simplification of the 2D patch mesh to generate a new face list and a new vertex list. An encoder configured as follows, Implement remeshing, including marching cubes, using boundary triangles and decoded geometry video. A decoder configured as follows, A system that includes this.

[0046] 18. The system described in Section 17, which includes the implementation of a second-order error edge collapse, is used to perform the simplification of the two-dimensional patch mesh.

[0047] 19. The implementation of the second-order error edge collapse is the system described in Section 18, which includes a half-edge collapse based on a second-order error metric.

[0048] 20. Performing the simplification of the two-dimensional patch mesh is the system described in Section 17, including the implementation of boundary distance edge collapse.

[0049] 21. The implementation of the boundary distance edge collapse is the system described in paragraph 20, which includes ordering a list of vertices according to the minimum distance to the boundary of the patch, and collapsing the vertex having the furthest adjacent vertex from the boundary.

[0050] 22. The simplification of the two-dimensional patch mesh is performed by the system described in Section 17, which includes the implementation of boundary triangles only.

[0051] 23. The implementation of the boundary triangle only includes rotating the triangle so that the first edge of the triangle is on the boundary, as described in paragraph 22.

[0052] The present invention has been described in relation to specific embodiments, including details, to facilitate understanding of its structure and operating principles. Such references to specific embodiments and their details herein are not intended to limit the claims appended herein. It will be readily apparent to those skilled in the art that various other modifications can be made to the embodiments selected for illustrative purposes without departing from the spirit and scope of the invention as defined by the claims. [Explanation of symbols]

[0053] 100 input mesh 102 Patch Segmentation 104 Face List and Vertex List 106 Depth generation 108 high-resolution depth maps 110 patch obj 112 Patch Generation 114 Simplification of Patch Mesh (2D) 116 New face list and new vertex list 200 original patch meshes 202 Original connection 204 Connection 206 Simplified Patch 210 Complete original mesh 212 Complete Simplified Mesh 400 initial patch mesh 402 Boundary distance edge collapse 404 Second-order error implementation 500 computer devices 502 Network Interface 504 memory 506 Processors 508 I / O devices 510 Bus 512 Storage device 520 Mesh Patch Simplified Hardware 530 Mesh Patch Simplification Application

Claims

1. A method programmed into the device's non-temporary memory, The steps to generate the patch, The steps include: performing simplification of a 2D patch mesh to generate a new face list and a new vertex list; Includes, Performing the aforementioned simplification of the two-dimensional patch mesh includes the implementation of a boundary distance edge collapse, The implementation of the boundary distance edge collapse is characterized by a method comprising ordering a list of vertices according to the minimum distance to the boundary of the patch, and collapsing the vertex having the furthest adjacent vertex from the boundary.

2. The method according to claim 1, characterized in that the simplification of the two-dimensional patch mesh includes the implementation of quadric error edge collapse.

3. The method according to claim 2, characterized in that the implementation of the second-order error edge collapse includes a half-edge collapse based on a second-order error metric.

4. The method according to claim 1, characterized in that the simplification of the two-dimensional patch mesh includes the implementation of only boundary triangles.

5. The method according to claim 4, characterized in that the implementation of the boundary triangle alone includes rotating the triangle so that the first edge of the triangle lies on the boundary.

6. The method according to claim 4, further comprising implementing remeshing in the decoder, which includes a marching cube using boundary triangles and a decoded geometry video.

7. It is a device, Non-temporary memory for storing applications, wherein the applications are Generating a patch, Simplifying the 2D patch mesh generates a new face list and a new vertex list, Non-temporary memory is used to perform this task, A processor coupled to the memory and configured to process the application, Includes, Performing the aforementioned simplification of the two-dimensional patch mesh includes the implementation of boundary distance edge collapse, The implementation of the boundary distance edge collapse is characterized by a device that includes ordering a list of vertices according to the minimum distance to the boundary of the patch, and collapsing the vertex having the furthest adjacent vertex from the boundary.

8. The apparatus according to claim 7, characterized in that the simplification of the two-dimensional patch mesh includes the implementation of a second-order error edge collapse.

9. The apparatus according to claim 8, characterized in that the implementation of the second-order error edge collapse includes a half-edge collapse based on a second-order error metric.

10. The apparatus according to claim 7, characterized in that the simplification of the two-dimensional patch mesh includes the implementation of only boundary triangles.

11. The apparatus according to claim 10, characterized in that the implementation of the boundary triangle alone includes rotating the triangle so that the first edge of the triangle lies on the boundary.

12. The apparatus according to claim 10, wherein the application is further configured in the decoder to implement remeshing including a marching cube using boundary triangles and a decoded geometry video.

13. It is a system, Generate a patch, Perform simplification of the 2D patch mesh to generate a new face list and a new vertex list. An encoder configured as follows, Implement remeshing, including marching cubes, using boundary triangles and decoded geometry video. A decoder configured as follows, Includes, Performing the aforementioned simplification of the two-dimensional patch mesh includes the implementation of boundary distance edge collapse, The implementation of the boundary distance edge collapse is characterized by a system that includes ordering a list of vertices according to the minimum distance to the boundary of the patch, and collapsing the vertex having the furthest adjacent vertex from the boundary.

14. The system according to claim 13, characterized in that the simplification of the two-dimensional patch mesh includes the implementation of a second-order error edge collapse.

15. The system according to claim 14, characterized in that the implementation of the second-order error edge collapse includes a half-edge collapse based on a second-order error metric.

16. The system according to claim 13, characterized in that the simplification of the two-dimensional patch mesh includes the implementation of only boundary triangles.

17. The system according to claim 16, characterized in that the implementation of the boundary triangle alone includes rotating the triangle so that the first edge of the triangle lies on the boundary.

Images