Simplifying mesh patches

Abstract

Here, we describe a method for simplifying patch connectivity data. Patches are generated considering high-resolution mesh information. The connectivity data is simplified at patch level, but the geometry image is preserved. The connectivity simplification only simplifies triangles within the patch. If the boundaries are preserved, reconstruction in 3D will not introduce artifacts. The high-resolution geometry image can be used to undo the simplification and improve the connectivity on the decoder side. Three embodiments of patch mesh simplification are described: quadratic error edge collapse, boundary distance edge collapse, and boundary triangles only.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority under 35 U.S.C. § 119 to U.S. Provisional Patent Application No. 63 / 269,909, entitled “MESH PATCH SIMPLIFICATION,” filed March 25, 2022, which is hereby incorporated by reference in its entirety for all purposes.

[0002] The present invention relates to three-dimensional graphics, and more particularly to coding of three-dimensional graphics. [Background technology]

[0003] In recent years, a new method for compressing volumetric content such as point clouds is becoming standardized, based on a 3D to 2D projection. Also known as V3C (visual volumetric video-based compression), this method maps the 3D volume data into a number of 2D patches, which are then placed into an atlas image, which is then encoded by a video encoder. The atlas image corresponds to the geometry of the points, their respective textures, and an occupancy map that indicates which locations should be considered for point cloud reconstruction.

[0004] In 2017, MPEG issued a Call for Proposals (CfP) for the compression of point clouds. After evaluation of several proposals, MPEG is currently considering two different techniques for point cloud compression: 3D native coding techniques (based on octrees and similar coding methods) or 3D to 2D projection followed by traditional video coding. For dynamic 3D scenes, MPEG is using Test Model Software (TMC2), based on patch surface modeling, projection of the patches from 3D to a 2D image, and coding of the 2D image by a video encoder such as HEVC. This method has proven to be more efficient than native 3D coding and can achieve competitive bitrates with acceptable quality.

[0005] Due to the success of coding 3D point clouds with projection-based methods (also known as video-based methods, or V-PCC), it is expected that future versions of this standard will include additional 3D data, such as 3D meshes. However, the current version of this standard is only suitable for transmitting a set of unconnected points, and therefore has no mechanism for transmitting point connectivity as required for 3D mesh compression.

[0006] Methods have also been proposed to extend the functionality of V-PCC to meshes. One possible method is to use V-PCC to code the vertices and then use a mesh compression method such as TFAN or Edgebreaker to code the connections. A limitation of this method is that the original mesh needs to be dense so that the point cloud generated from the vertices is not sparse and can be efficiently coded after projection. Furthermore, the order of the vertices affects the coding of the connections, so different methods have been proposed to reorganize the connections of the mesh. An alternative method for coding sparse meshes is to use the raw patch data to code the vertex positions in 3D. Since raw patches directly code (x,y,z), in this method all vertices are coded as raw data, while the connections are coded by a similar mesh compression method, as mentioned before. It is noted that in raw patches, the vertices can be sent in any preferred order, so the order generated from the connectivity coding can be used. While this method can code sparse point clouds, raw patches are not efficient for coding 3D data and further data, such as triangular face attributes, may be missing from this method. Summary of the Invention [Problem to be solved by the invention]

[0007] This paper describes a method for simplifying the connectivity data of a patch. First, a patch is generated considering high-resolution mesh information. The connectivity data is simplified at the patch level, but the geometry image is preserved. The connectivity simplification can simplify only the triangles in the patch. If the boundaries are preserved, the reconstruction in 3D will not introduce artifacts such as gaps between patches due to mismatched boundary vertices. The high-resolution geometry image can be used to undo the simplification and improve the connectivity on the decoder side. Three embodiments of patch mesh simplification are described: quadric error edge collapse, border distance edge collapse, and border triangles only. The first two methods perform edge collapse, but with different criteria. The quadratic error metric uses a 3D metric to check for possible distortions when moving vertices, and enforces the consistency of triangle normals and the preservation of boundaries in the 2D projection space. For the border distance, the criterion for edge collapse is the distance to the boundary, and the vertices farthest from the boundary are collapsed first. Finally, the method of sending only boundary triangles removes all triangles that are not connected to the boundary and sends only triangles that have at least one edge that belongs to the boundary of the patch. [Means for solving the problem]

[0008] In one aspect, a method programmed into a non-transitory memory of a device includes generating a patch and performing a simplification of the two-dimensional patch mesh to generate a new face list and a new vertex list. Performing the simplification of the two-dimensional patch mesh includes implementing a quadratic error edge collapse. The quadratic error edge collapse implementation includes a half-edge collapse based on a quadratic error metric. Performing the simplification of the two-dimensional patch mesh includes implementing a boundary distance edge collapse. The boundary distance edge collapse implementation includes ordering a list of vertices according to a minimum distance to a boundary of the patch and collapsing a vertex having a neighbor vertex farthest from the boundary. Performing the simplification of the two-dimensional patch mesh includes implementing a boundary triangles only implementation. The boundary triangles only implementation includes rotating a triangle such that a first edge of the triangle is at a boundary. The method further includes implementing, at a decoder, a remeshing including marching cubes using the boundary triangles and the decoded geometry video.

[0009] In another aspect, an apparatus includes a non-transitory memory for storing an application, the application for generating a patch and performing a simplification of a two-dimensional patch mesh to generate a new face list and a new vertex list, and a processor coupled to the memory and configured to process the application. Performing the simplification of the two-dimensional patch mesh includes implementing a quadratic error edge collapse. The quadratic error edge collapse implementation includes a half-edge collapse based on a quadratic error metric. Performing the simplification of the two-dimensional patch mesh includes implementing a boundary distance edge collapse. The boundary distance edge collapse implementation includes ordering a list of vertices according to a minimum distance to a boundary of the patch and collapsing a vertex having a neighbor vertex farthest from the boundary. Performing the simplification of the two-dimensional patch mesh includes implementing boundary triangles only. The boundary triangles only implementation includes rotating a triangle such that a first edge of the triangle is at a boundary. The application is further configured to implement, at the decoder, a remeshing including marching cubes using the bounding triangles and the decoded geometry video.

[0010] In another aspect, a system includes an encoder configured to generate a patch and perform a simplification of a two-dimensional patch mesh to generate a new face list and a new vertex list, and a decoder configured to implement a remeshing including marching cubes using boundary triangles and a decoded geometry video. Performing the simplification of the two-dimensional patch mesh includes implementing a quadratic error edge collapse. The quadratic error edge collapse implementation includes a half-edge collapse based on a quadratic error metric. Performing the simplification of the two-dimensional patch mesh includes implementing a boundary distance edge collapse. The boundary distance edge collapse implementation includes ordering a list of vertices according to a minimum distance to a boundary of the patch and collapsing a vertex having a neighbor vertex farthest from the boundary. Performing the simplification of the two-dimensional patch mesh includes implementing a boundary triangle only. The boundary triangle only implementation includes rotating a triangle such that a first edge of the triangle is at a boundary. [Brief description of the drawings]

[0011] [Figure 1] 4 is a flowchart of a method for generating a new face list and a new vertex list according to some embodiments. [Diagram 2] FIG. 1 is a diagram of quadric error collapse in accordance with some embodiments. [Diagram 3] FIG. 1 illustrates an image of a mesh with only the boundary triangles transmitted in accordance with some embodiments. [Figure 4] FIG. 13 is an illustration of boundary distance edge collapse in accordance with some embodiments. [Diagram 5] FIG. 1 is a block diagram of an example computing device configured to implement a mesh patch simplification method according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] This paper describes a method for simplifying connectivity data of a patch. First, a patch is generated considering high-resolution mesh information. The connectivity data is simplified at the patch level, but the geometry image is preserved. The connectivity simplification can simplify only triangles in the patch. If the boundaries are preserved, the reconstruction in 3D will not introduce artifacts such as gaps between patches due to mismatched boundary vertices. The high-resolution geometry image can be used to undo the simplification and improve the connectivity on the decoder side. Three embodiments of patch mesh simplification are described: quadratic error edge collapse, boundary distance edge collapse, and boundary triangles only. The first two methods perform edge collapse, but with different criteria. The quadratic error metric uses a 3D metric to check for possible distortions when moving vertices, and enforces the consistency of triangle normals and the preservation of boundaries in the 2D projection space. For boundary distance, the criterion for edge collapse is the distance to the boundary, and the vertices farthest from the boundary are collapsed first. Finally, the method of sending only boundary triangles removes all triangles that are not connected to the boundary and sends only triangles that have at least one edge that belongs to the boundary of the patch.

[0013] FIG. 1 shows a flow chart of a method for generating a new face list and a new vertex list according to some embodiments. In step 100, an input mesh is received or acquired. For example, the input mesh is downloaded (e.g., from a network device) or acquired / captured by a device (e.g., a camera or an autonomous vehicle). In step 102, the input mesh is segmented into patches. Any implementation of patch segmentation can be performed. The patch segmentation generates a face list (e.g., connectivity) and a vertex list in step 104. In step 106, depth generation is performed. In step 108, a high-resolution depth map is generated based on the depth generation. In step 110, a patch obj can regenerate the object using the face list and vertex list.

[0014] In step 112, patch generation (or creation) is implemented. In some embodiments, patch generation includes normal calculation, adjacency calculation, initial segmentation, refinement, patch projection, and patch rasterization. Normal calculation calculates the normal of each triangle (e.g., the cross product of the triangle edges). Adjacency calculation calculates the adjacency of each triangle (e.g., which triangles in the mesh are adjacent or touching the current triangle or other triangles). Initial segmentation includes classifying normals according to their orientation. For example, a triangle's normal can point up, down, left, right, front, or back, and can be classified based on the direction / orientation. In some embodiments, triangles are colored based on the orientation of their normals (e.g., all triangles with normals pointing up are green). Refinement identifies outliers (e.g., a single red triangle surrounded by blue triangles) and smoothes the outliers (e.g., changes the single red triangle to match the neighboring blue triangles). Refinement is performed by analyzing neighboring triangles and smoothing the orientation (e.g., adjusting the orientation of normals). Once a smooth surface is obtained, patch projection is performed, which projects patches of a particular triangle classification (e.g., based on the orientation). In this projection, the vertices and connections are shown on the patch. For example, the body and face in this example are separate projections since there are triangles of different classifications that separate them. However, V3C and V-PCC do not understand this, but instead understand points, so the projection is rasterized (e.g., sampling points on the surface, including the distance of the points, to generate a geometry image and attributes of the surface). The rasterized mesh surface is very similar to the V3C image. Steps 100-112 can be performed with any standard implementation.

[0015] In step 114, patch mesh (2D) simplification is performed on the generated patch. The patch mesh (2D) simplification reduces the amount of data transmitted (e.g., encoded). However, the transmitted data is sufficient to recover the mesh. As described herein, there are three different implementations of patch mesh (2D) simplification: quadratic error edge collapse, boundary distance edge collapse, and boundary triangles only. In step 116, a new face list and a new vertex list are generated and transmitted / encoded based on the patch mesh (2D) simplification. In some embodiments, fewer or additional steps are implemented. For example, a decoder can be implemented to decode the patch mesh. In some embodiments, the order of the steps is changed.

[0016] An illustration of quadratic error collapse according to some embodiments is shown in Figure 2. The original patch mesh 200 has 334 faces. A quadratic error calculation can be used to determine a quadratic error metric.

[0017] Vertex v i (QM vi ) the quadratic error matrix is ​​obtained as follows: JPEG2025510193000002.jpg1336In the above formula, N i is the set of triangles that contains vertex i. A ,v B ,v C ) is calculated as follows: JPEG2025510193000003.jpg8035

[0018] Next, vertex v i Position v j The cost of moving to is given by: Cost = TIFF2025510193000004.tif973

[0019] Based on a quadratic error metric, half-edge collapse can be implemented. As shown, original connection 202 can be collapsed into connection 204, which contains fewer points. Simplified patch 206 has 166 faces compared to 334 faces of original patch mesh 200. Simplified patch 206 retains the vertices on the perimeter of the patch (e.g., the vertices on the outer edges), but the interior triangles are configured differently than the original patch mesh. The full original mesh 210 is also shown compared to the full simplified mesh 212. By having fewer faces (and due to fewer vertices), there is much less data to encode.

[0020] FIG. 3 shows an image of a mesh where only boundary triangles have been transmitted according to some embodiments. Only the triangles that are on the boundary are kept. To know which triangles are on the boundary, the triangles are rotated so that the first edge of the triangle is the edge that is on the boundary. If there are multiple edges on the boundary, multiple triangles are transmitted. If a triangle does not have an edge on its boundary, it is not transmitted. On the decoder side, remeshing is implemented which includes using marching cubes using the boundary triangles and the decoded geometry video. The geometry video indicates where the surface should be, and then marching cubes is performed on the surface to regenerate the triangles. When performing marching cubes, the process starts with the boundary triangles already transmitted.

[0021] FIG. 4 shows an illustration of boundary distance edge collapse according to some embodiments. Edge collapse can be based on distance to the boundary. The list of vertices can be ordered according to the minimum distance to the boundary of the patch. The vertex with the neighbor furthest from the boundary is collapsed. This implementation is more efficiently coded, but has a lower peak signal-to-noise ratio (PSNR). A vertex split implementation can be used to improve the PSNR.

[0022] Vertices on the boundary are decoded properly because the boundary can be restored using zippering. With respect to video compression, points in the center of a patch are likely to have fewer artifacts than points on the boundary, since central points are usually in smoother regions.

[0023] As shown, the initial patch mesh 400 has 334 faces, while the patch mesh using boundary distance edge collapse 402 has 166 faces, which is the same or a similar number of faces (e.g., 166) as the quadratic error implementation 404.

[0024] FIG. 5 illustrates a block diagram of an exemplary computing device configured to implement a mesh patch simplification method according to some embodiments. The computing device 500 can be used to acquire, store, compute, process, communicate, and / or display information, such as images and videos, including 3D content. The computing device 500 can implement any of the encoding / decoding aspects. In general, a hardware structure suitable for implementing the computing device 500 includes a network interface 502, a memory 504, a processor 506, an I / O device(s) 508, a bus 510, and a storage device 512. The selection of the processor is not critical as long as a suitable processor of sufficient speed is selected. The memory 504 can be any conventional computer memory known in the art. The storage device 512 can include a hard drive, CDROM, CDRW, DVD, DVDRW, high definition disk / drive, ultra high definition drive, flash memory card, or any other storage device. The computing device 500 can include one or more network interfaces 502. Examples of network interfaces include a network card connected to an Ethernet or other type of LAN. The I / O device(s) 508 may include one or more of a keyboard, mouse, monitor, screen, printer, modem, touch screen, button interface, and other devices. The storage device 512 and memory 504 likely store mesh patch simplification application(s) 530 used to execute the implementation of mesh patch simplification and process as an application would normally be processed. The computing device 500 may also include more or fewer components than those shown in FIG. 5. In some embodiments, mesh patch simplification hardware 520 is included. Although the computing device 500 of FIG. 5 includes application 530 and hardware 520 for the implementation of mesh patch simplification, the mesh patch simplification method may also be implemented on the computing device in hardware, firmware, software, or any combination thereof.For example, in some embodiments, the mesh patch simplification application 530 is programmed into memory and executed using a processor. As another example, in some embodiments, the mesh patch simplification hardware 520 is programmed hardware logic that includes gates specifically designed to implement the mesh patch simplification method.

[0025] In some embodiments, the mesh patch simplification application(s) 530 include multiple applications and / or modules. In some embodiments, a module also includes one or more sub-modules. In some embodiments, fewer or additional modules may be included.

[0026] Examples of suitable computing devices include a personal computer, a laptop computer, a computer workstation, a server, a mainframe computer, a handheld computer, a personal digital assistant, a cellular / mobile phone, a smart appliance, a game console, a digital camera, a digital camcorder, a camera phone, a smart phone, a portable music player, a tablet computer, a mobile device, a video player, a video disc writer / player (e.g., DVD writer / player, high definition disc writer / player, ultra high definition disc writer / player), a television, a home entertainment system, an augmented reality device, a virtual reality device, smart jewelry (e.g., a smart watch), a vehicle (e.g., a self-driving vehicle), or any other suitable computing device.

[0027] To utilize the mesh patch simplification method, a device acquires or receives 3D content (e.g., point cloud content). The mesh patch simplification method can be implemented with user assistance or automatically without user involvement.

[0028] In operation, the mesh patch simplification method enables more efficient and accurate 3D content encoding compared to conventional implementations.

[0029] Some embodiments of mesh patch simplification 1. A method programmed into a non-transitory memory of a device, comprising: generating a patch; performing a simplification of the 2D patch mesh to generate a new face list and a new vertex list; The method includes:

[0030] 2. The method of claim 1, wherein performing the simplification of the two-dimensional patch mesh includes implementing second order error edge collapse.

[0031] 3. The method of claim 2, wherein the implementation of the quadratic error edge collapse includes a half-edge collapse based on a quadratic error metric.

[0032] 4. The method of claim 1, wherein performing simplification of the two-dimensional patch mesh includes implementing boundary distance edge collapse.

[0033] 5. The method of claim 4, wherein implementing the boundary distance edge collapse includes ordering a list of vertices according to their minimum distance to a boundary of the patch, and collapsing the vertex that has a neighbor vertex that is farthest from the boundary.

[0034] 6. The method of claim 1, wherein performing simplification of the two-dimensional patch mesh includes implementing only boundary triangles.

[0035] 7. The method of claim 6, wherein populating only the boundary triangles includes rotating a triangle so that a first edge of the triangle is on the boundary.

[0036] 8. The method of claim 6, further comprising implementing, at the decoder, remeshing including marching cubes using the bounding triangles and the decoded geometry video.

[0037] 9. An apparatus comprising: A non-transitory memory for storing an application, the application comprising: Generating a patch; performing a simplification of the two-dimensional patch mesh to generate a new face list and a new vertex list; a non-transient memory for performing a processor coupled to the memory and configured to process the application; An apparatus comprising:

[0038] 10. The apparatus of claim 9, wherein performing simplification of the two-dimensional patch mesh includes implementing second order error edge collapse.

[0039] 11. The apparatus of claim 10, wherein the implementation of the second order error edge collapse includes a half edge collapse based on a second order error metric.

[0040] 12. The apparatus of claim 9, wherein performing simplification of the two-dimensional patch mesh includes implementing boundary distance edge collapse.

[0041] 13. The apparatus of claim 12, wherein implementing the boundary distance edge collapse includes ordering a list of vertices according to a minimum distance to a boundary of the patch, and collapsing a vertex that has a neighbor vertex that is farthest from the boundary.

[0042] 14. The apparatus of claim 9, wherein performing simplification of the two-dimensional patch mesh includes implementing only boundary triangles.

[0043] 15. The apparatus of claim 14, wherein populating only the boundary triangles includes rotating a triangle so that a first edge of the triangle is on the boundary.

[0044] 16. The apparatus of claim 14, wherein the application is further configured to implement, at the decoder, remeshing including marching cubes using the bounding triangles and the decoded geometry video.

[0045] 17. A system comprising: Generate the patch, Perform a simplification of the 2D patch mesh to generate a new face list and a new vertex list; an encoder configured to Implement remeshing including marching cubes using bounding triangles and decoded geometry video; a decoder configured to A system including:

[0046] 18. The system of claim 17, wherein performing simplification of the two-dimensional patch mesh includes implementing second order error edge collapse.

[0047] 19. The system of claim 18, wherein the implementation of the quadratic error edge collapse includes a half-edge collapse based on a quadratic error metric.

[0048] 20. The system of claim 17, wherein performing simplification of the two-dimensional patch mesh includes implementing boundary distance edge collapse.

[0049] 21. The system of claim 20, wherein implementing the boundary distance edge collapse includes ordering a list of vertices according to a minimum distance to a boundary of the patch, and collapsing a vertex that has a neighbor vertex that is farthest from the boundary.

[0050] 22. The system of claim 17, wherein performing simplification of the two-dimensional patch mesh includes implementing only boundary triangles.

[0051] 23. The system of claim 22, wherein populating only the boundary triangles includes rotating a triangle so that a first edge of the triangle is on the boundary.

[0052] The present invention has been described with respect to specific embodiments containing details so as to facilitate an understanding of the principles of construction and operation of the invention. Reference herein to such specific embodiments and their details is not intended to limit the scope of the claims appended hereto. It will be readily apparent to those skilled in the art that various other modifications can be made in the embodiments selected for illustration without departing from the spirit and scope of the invention as defined by the claims. [Explanation of symbols]

[0053] 100 input meshes 102 Patch Segmentation 104 Face List and Vertex List 106 Depth generation 108 High Resolution Depth Maps 110 patch obj 112 Patch Generation 114 Simplifying patch meshes (2D) 116 New Face List and New Vertex List 200 original patch meshes 202 original connections 204 Connected 206 Simplified Patch 210 complete original mesh 212 Complete Simplified Mesh 400 initial patch meshes 402 Boundary Distance Edge Collapse 404 Second-order error implementation 500 Computer equipment 502 Network Interface 504 Memory 506 Processor 508 I / O device 510 Bus 512 Storage device 520 Mesh Patch Simplification Hardware 530 Mesh Patch Simplification Application

Claims

1. A method programmed into a non-transitory memory of a device, comprising: generating a patch; performing a simplification of the two-dimensional patch mesh to generate a new face list and a new vertex list; The method according to claim 1, further comprising:

2. The method of claim 1 , wherein performing a simplification of the two-dimensional patch mesh comprises a quadric error edge collapse implementation.

3. 3. The method of claim 2, wherein the implementation of the second-order error edge collapse comprises a half-edge collapse based on a second-order error metric.

4. The method of claim 1 , wherein performing a simplification of the two-dimensional patch mesh comprises a border distance edge collapse implementation.

5. 5. The method of claim 4, wherein implementing the boundary distance edge collapse comprises ordering a list of vertices according to a minimum distance to a boundary of the patch, and collapsing a vertex that has a neighbor farthest from the boundary.

6. The method of claim 1 , wherein performing a simplification of the two-dimensional patch mesh includes implementing only boundary triangles.

7. 7. The method of claim 6, wherein populating only the boundary triangles includes rotating a triangle so that a first edge of the triangle is on the boundary.

8. 7. The method of claim 6, further comprising: implementing, at the decoder, remeshing including marching cubes using bounding triangles and decoded geometry video.

9. An apparatus comprising: A non-transitory memory for storing an application, the application comprising: Generating a patch; performing a simplification of the two-dimensional patch mesh to generate a new face list and a new vertex list; a non-transient memory for performing a processor coupled to the memory and configured to process the application; An apparatus comprising:

10. The apparatus of claim 9 , wherein performing the simplification of the two-dimensional patch mesh comprises implementing a quadratic error edge collapse.

11. The apparatus of claim 10 , wherein the implementation of second order error edge collapse comprises a half edge collapse based on a second order error metric.

12. The apparatus of claim 9 , wherein performing the simplification of the two-dimensional patch mesh includes implementing a boundary distance edge collapse.

13. 13. The apparatus of claim 12, wherein implementing the boundary distance edge collapse comprises ordering a list of vertices according to a minimum distance to a boundary of the patch and collapsing a vertex having a neighbor vertex that is furthest from the boundary.

14. The apparatus of claim 9 , wherein performing a simplification of the two-dimensional patch mesh includes implementing only boundary triangles.

15. 15. The apparatus of claim 14, wherein populating only the boundary triangles includes rotating a triangle so that a first edge of the triangle is on the boundary.

16. 15. The apparatus of claim 14, wherein the application is further configured to implement, at a decoder, remeshing including marching cubes using bounding triangles and decoded geometry video.

17. 1. A system comprising: Generate the patch, performing a simplification of the 2D patch mesh to generate a new face list and a new vertex list; an encoder configured to Implement remeshing including marching cubes using bounding triangles and decoded geometry video; a decoder configured to A system comprising:

18. 20. The system of claim 17, wherein performing the simplification of the two-dimensional patch mesh includes implementing a quadratic error edge collapse.

19. 20. The system of claim 18, wherein the implementation of second order error edge collapse includes a half edge collapse based on a second order error metric.

20. The system of claim 17 , wherein performing the simplification of the two-dimensional patch mesh includes implementing a boundary distance edge collapse.

21. 21. The system of claim 20, wherein implementing the boundary distance edge collapse includes ordering a list of vertices according to a minimum distance to a boundary of the patch and collapsing a vertex that has a neighbor vertex that is furthest from the boundary.

22. The system of claim 17 , wherein performing a simplification of the two-dimensional patch mesh includes implementing only boundary triangles.

23. 23. The system of claim 22, wherein populating only the boundary triangles includes rotating a triangle so that a first edge of the triangle is on the boundary.

Images