Projection-based mesh compression

Abstract

This paper describes a method for compressing both untracked and tracked meshes using a projection-based approach, leveraging tools and syntax already developed for projection-based point cloud compression. Similar to the V‑PCC approach, the mesh is partitioned into surface surfels, with the difference that the partitioning respects the connectivity of the mesh. Each surface surfel (or 3D surfel) is then projected onto a 2D surfel, resulting in a triangular surface sampling similar to common rasterization methods used in computer graphics in the case of a mesh. For each surfel, the positions of the projected vertices are saved in a list along with the connectivity of these vertices. The sampled surface is similar to a point cloud and is encoded using the same methods used for point cloud compression. Additionally, a list of vertices and connectivity is encoded per surfel and the data is sent along with the encoded point cloud data.

Description

[0001] Citation of Related Applications

[0002] This application claims priority under 35 U.S.C. §119(e) to U.S. Provisional Patent Application Serial No. 62 / 991,128, filed on March 18, 2020, entitled "PROJECTION-BASED MESH COMPRESSION," which is incorporated herein by reference in its entirety. Technical Field

[0003] The present invention relates to three-dimensional graphics. More particularly, the present invention relates to encoding of three-dimensional graphics. Background Art

[0004] Recently, a new method for compressing point clouds based on projection from 3D to 2D is being standardized. This method, also known as V-PCC (video-based point cloud compression), maps the 3D point cloud data into several 2D bins, which are then further arranged into atlas images, which are then encoded using a video encoder. The atlas images correspond to the point geometry, the corresponding texture, and an occupancy map, which indicates which of the positions should be considered for point cloud reconstruction.

[0005] In 2017, MPEG issued a Call for Proposals (CfP) for point cloud compression. After evaluating several proposals, MPEG is currently considering two different techniques for point cloud compression: 3D native coding techniques (based on octrees and similar coding methods), or 3D to 2D projection followed by traditional video coding. In the case of dynamic 3D scenes, MPEG is using a test model software (TMC2) based on surface modeling of surfaces, projection of surfaces from 3D images to 2D images, and encoding of 2D images with video codecs such as HEVC. This approach has been shown to be more efficient than native 3D coding and can achieve competitive bit rates at acceptable quality.

[0006] Due to the success of projection-based methods (also known as video-based methods or V-PCC) for encoding 3D point clouds, it is expected that the standard will include more 3D data, such as 3D meshes, in future versions. However, the current version of the standard is only suitable for the transmission of a set of disconnected points, and thus there is no mechanism to transmit the connectivity of the points, which is required for 3D mesh compression.

[0007] Methods have been proposed to extend the functionality of V-PCC to meshes. One possible approach is to encode vertices using V-PCC and then encode connectivity using a mesh compression method such as TFAN or Edgebreaker. A limitation of this approach is that the original mesh must be dense so that the point cloud generated from the vertices is not sparse and can be efficiently encoded after projection. Furthermore, the order of the vertices affects the encoding of connectivity, leading to different methods for reorganizing mesh connectivity. An alternative approach to encoding sparse meshes is to use RAW surface data to encode vertex positions in 3D. Since RAW surfaces directly encode (x, y, z), in this approach, all vertices are encoded as RAW data, while connectivity is encoded using a similar mesh compression method as described above. In RAW surfaces, vertices can be sent in any preferred order, allowing the order derived from the connectivity encoding to be used. This approach can encode sparse point clouds; however, RAW surfaces are not very efficient for encoding 3D data, and additional data, such as triangular face attributes, may be lost with this approach. Summary of the Invention

[0008] This paper describes a method for compressing untracked and tracked meshes using a projection-based approach and leveraging tools and syntax that have been generated for projection-based point cloud compression. Similar to the V-PCC method, the mesh is partitioned into surface panels, with the difference that these partitions follow the connectivity of the mesh. Each surface panel (or 3D panel) is then projected onto a 2D panel, resulting in a triangular surface sampling similar to the common rasterization methods used in computer graphics in the case of a mesh. For each panel, the positions of the projected vertices are saved in a list along with the connectivity of these vertices. The sampled surface is similar to a point cloud and is encoded using the same method used for point cloud compression. In addition, the list of vertices and connectivity is encoded per panel and the data is sent along with the encoded point cloud data.

[0009] In one aspect, a method for programming in a non-transitory memory of a device. The method includes voxelizing an input mesh, implementing facet generation, wherein the facet generation partitions the mesh into facets comprising a rasterized mesh surface and vertex positions and connectivity information, generating a video-based point cloud compression (V-PCC) image from the rasterized mesh surface, implementing a base mesh encoding using the vertex positions and connectivity information, and generating a V-PCC bitstream based on the V-PCC image and the base mesh encoding, wherein the V-PCC bitstream includes information for point cloud reconstruction and mesh reconstruction. The mesh voxelization includes shifting and / or scaling mesh values ​​to avoid negative and non-integer values. The mesh voxelization includes finding the lowest vertex value below zero and shifting the mesh values ​​so that the lowest vertex value is above zero. Implementing facet generation includes calculating a normal for each triangle, wherein calculating the normal for the triangle includes using a cross product between edges, classifying the triangles according to their normals, and implementing refinement by analyzing neighboring triangles. The method also includes displaying content that includes, in part, a point cloud and in part, a mesh. Incremental position information for tracking vertices is included in the bitstream. The method also includes implementing zippering, which includes classifying vertices as boundary vertices or non-boundary vertices, determining neighbors of boundary vertices, and merging neighboring boundary vertices.

[0010] In another aspect, a device includes a non-transitory memory for storing an application and a processor coupled to the memory, the application for: mesh voxelizing an input mesh to implement facet generation, the facet generation partitioning the mesh into facets comprising a rasterized mesh surface and vertex positions and connectivity information, generating a video-based point cloud compression (V-PCC) image from the rasterized mesh surface, implementing a base mesh encoding using the vertex positions and connectivity information, and generating a V-PCC bitstream based on the V-PCC image and the base mesh encoding, wherein the V-PCC bitstream comprises information for point cloud reconstruction and mesh reconstruction, the processor being configured to process the application. The mesh voxelization includes shifting and / or scaling mesh values ​​to avoid negative and non-integer values. The mesh voxelization includes finding the lowest vertex value below zero and shifting the mesh values ​​so that the lowest vertex value is above zero. Implementing facet generation includes calculating a normal for each triangle, wherein calculating the normal for the triangle includes using a cross product between edges, classifying the triangles according to their normals, and implementing refinement by analyzing neighboring triangles. In the device, the application is further configured to display content that partially comprises a point cloud and partially comprises a mesh. Incremental position information of tracked vertices is included in the bitstream. In the device, the application is further configured to implement stitching, which includes classifying vertices as boundary vertices or non-boundary vertices, determining neighbors of boundary vertices, and merging neighboring boundary vertices.

[0011] In another aspect, a system includes one or more cameras for acquiring three-dimensional content, and an encoder for encoding the three-dimensional content by performing mesh voxelization on an input mesh of the three-dimensional content, performing surfel generation, wherein the surfel generation partitions the mesh into surfels comprising a rasterized mesh surface and vertex positions and connectivity information, generates a video-based point cloud compression (V-PCC) image from the rasterized mesh surface, performs a base mesh encoding using the vertex positions and connectivity information, and generates a V-PCC bitstream based on the V-PCC image and the base mesh encoding, wherein the V-PCC bitstream comprises information for point cloud reconstruction and mesh reconstruction. The mesh voxelization comprises shifting and / or scaling mesh values ​​to avoid negative and non-integer values. The mesh voxelization comprises finding the lowest vertex value below zero and shifting the mesh values ​​so that the lowest vertex value is above zero. Performing surfel generation comprises calculating a normal for each triangle, wherein calculating the normal for the triangle comprises using a cross product between edges, classifying the triangles according to their normals, and performing refinement by analyzing neighboring triangles. The encoder is further configured to display content that includes, in part, a point cloud and in part a mesh. Incremental position information for tracked vertices is included in the bitstream. The encoder is further configured to implement stitching, which includes classifying vertices as boundary vertices or non-boundary vertices, determining neighbors of boundary vertices, and merging neighboring boundary vertices. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 A flow chart illustrating a method of implementing projection-based mesh compression in accordance with some embodiments.

[0013] Figure 2 Diagram illustrating mesh voxelization in accordance with some embodiments.

[0014] Figure 3 A diagram illustrating bin generation in accordance with some embodiments.

[0015] Figure 4 A diagram illustrating bin generation involving bin projection and bin rasterization in accordance with some embodiments.

[0016] Figure 5 Illustrating the data structure used for base trellis coding / signaling in accordance with some embodiments.

[0017] Figure 6 Illustrated is an example of combining point cloud and mesh information in accordance with some embodiments.

[0018] Figure 7A and 7B Illustrated is an example of combining untracked and tracked mesh information in accordance with some embodiments.

[0019] Figure 8Diagram illustrating V-PCC image generation in accordance with some embodiments.

[0020] Figure 9 A diagram illustrating mesh reconstruction in accordance with some embodiments.

[0021] Figure 10 A diagram illustrating a stitching algorithm in accordance with some embodiments.

[0022] Figure 11 A diagram illustrating the classification of vertices as boundary or non-boundary, in accordance with some embodiments.

[0023] Figure 12 Diagram illustrating alternative connectivity encoding in accordance with some embodiments.

[0024] Figure 13 Illustrated is a block diagram of an exemplary computing device configured to implement a projection-based mesh compression method in accordance with some embodiments. DETAILED DESCRIPTION

[0025] This paper describes a method for compressing both untracked and tracked meshes using a projection-based approach and leveraging tools and syntax already developed for projection-based point cloud compression. An untracked mesh is one in which each frame has a different mesh and different connectivity. A tracked mesh has a first frame with the mesh, which is then tracked, and the structure of the mesh remains unchanged while its position changes over time. Similar to the V-PCC method, the mesh is partitioned into surface bins, with the difference being that this partitioning respects the connectivity of the mesh. Each surface bin (or 3D bin) is then projected onto a 2D bin, resulting in triangular surface sampling similar to common rasterization methods used in computer graphics in the case of a mesh. For each bin, the positions of the projected vertices are stored in a list along with the connectivity of these vertices. The sampled surface is similar to a point cloud and is encoded using the same methods used for point cloud compression. In addition, a list of vertices and connectivity is encoded per bin and the data is sent along with the encoded point cloud data.

[0026] Two new syntax elements are introduced to describe mesh information: the meshsurface metadata element and the trackingmeshsurface metadata element. The meshsurface metadata element can indicate vertex positions and their respective connectivity. Alternatively, it allows the positions of vertices to be sent via the occupancy map instead of via the surface metadata. For the case of tracking meshes, syntax elements are introduced that can convey two typical types of motion in the case of tracking meshes: global motion and surface motion. The trackingmeshsurface metadata element syntax element represents global motion through translation and rotation of the surface elements, while surface motion (such as the motion of fabric folds) is represented by vertex displacement.

[0027] The additional connectivity data can be interpreted as the base mesh generated for each surfel, giving the decoder the flexibility to use or not use this additional data. The syntax also allows the encoder to mix content and have point clouds and meshes coexist (e.g., you might want to use point clouds for finer structures like hair and meshes for flat surfaces), or even have point clouds coexist with tracked and untracked meshes (e.g., parts of a surface that are difficult to track can be sent as untracked parts, or even tracked parts that are too deformed can be hidden by the point cloud layer). This data can be used to improve rendering and for point filtering algorithms. Furthermore, the mesh is encoded using the same principles of projection-based compression, which leads to better integration with current V-PCC methods for projection-based coding of point clouds.

[0028] Figure 1 A flowchart illustrating a method for implementing projection-based mesh compression according to some embodiments is shown. At step 100, an input mesh is received or acquired. For example, the input mesh is downloaded (e.g., from a network device) or acquired / captured by a device (e.g., a camera or an autonomous vehicle).

[0029] At step 102, the mesh is voxelized. The mesh can have floating point vertex positions, so these positions are converted to integer space. V-PCC assumes a voxelized point cloud.

[0030] At step 104, facet generation (or creation) is implemented. Facet generation includes: normal calculation, adjacency calculation, initial segmentation; refinement, facet projection, and facet rasterization. Normal calculation is the calculation of the normal of each triangle (e.g., the cross product of the edges of the triangle). Adjacency calculation involves calculating the adjacency of each triangle (e.g., which triangles in the mesh are adjacent to or touching the current triangle or other triangles). Initial segmentation includes classifying normals by orientation. For example, the normal of a triangle can point up, down, left, right, front, or back, and can be classified based on direction / orientation. In some embodiments, triangles are color-coded based on the orientation of their normals (e.g., all triangles with normals pointing up are colored green). Refinement involves locating outliers (e.g., a single red triangle surrounded by blue triangles) and eliminating outliers (e.g., changing a single red triangle to match its blue neighbor). Refinement is performed by analyzing the neighbors and smoothing the orientation (e.g., adjusting the orientation of the normal). Once a smooth surface exists, a facet projection is performed, which involves projecting facets onto a specific classification (e.g., based on orientation) of triangles. Through the projection, vertices and connectivity are displayed on the facets. For example, the body and face in this example are separate projections because there are triangles of different classifications separating the two. However, V-PCC does not understand this; instead, V-PCC understands points, so the projections are rasterized (e.g., sampling points on the surface, including the distances of the points, to generate a geometric image and properties of the surface). The rasterized mesh surface is very similar to a V-PCC image.

[0031] The result of bin generation is a rasterized mesh surface along with vertex positions and connectivity. In step 106, the rasterized mesh surface is used in V-PCC image generation / creation. In step 108, the vertex positions and connectivity are used for base mesh encoding. In step 110, a V-PCC bitstream is generated from the generated V-PCC image and the base mesh encoding. In some embodiments, base mesh encoding does not involve any additional coding; vertex positions and connectivity are directly transferred to the V-PCC bitstream.

[0032] The V-PCC bitstream enables point cloud reconstruction at step 112 and / or mesh construction at step 114. The point cloud and / or mesh can be extracted from the V-PCC bitstream, which provides significant flexibility. In some embodiments, fewer or additional steps can be implemented. In some embodiments, the order of the steps is changed.

[0033] Figure 216 and 17. The following diagram illustrates a mesh voxelization in accordance with some embodiments. Each frame has a different bounding box. A bounding box is obtained for each frame (e.g., frame 200 at t=1, t=16, and t=32). A sequence bounding box 202 is then calculated from the many bounding boxes, SequenceBB=(minPoint, maxPoint). Sequence bounding box 202 contains all vertices regardless of the frame. A scale is calculated that fits within the range defined by bitdepth, maxRange=max(maxPoint[0..2]-minPoint[0..2]), scale=(2 bitdepth -1) / maxRange. The result is scaled and shifted by the minimum value, voxelizedpoint=floor(scale*originalPoint+minPoint). The scale and shift amount can be user-defined or computer-generated based on a learning algorithm (for example, by analyzing the bounding box and automatically calculating the scale and shift amount).

[0034] In some embodiments, mesh voxelization involves converting floating point values ​​of the positions of points of the input mesh into integers. The precision of the integers can be set by the user or automatically. In some embodiments, mesh voxelization includes shifting values ​​so that there are no negative numbers.

[0035] For example, the original mesh is below the axis, resulting in negative values. By voxelizing the mesh, the mesh is shifted and / or scaled to avoid negative and non-integer values. In one implementation, the lowest vertex value below zero is found, and these values ​​can then be shifted so that the lowest vertex value is above zero. In some embodiments, the range of values ​​fits within a specified bit range (e.g., by scaling), such as 11 bits.

[0036] The voxelized grid 210 is the original grid after scaling and shifting. For example, the voxelized grid 210 is the original grid after it has been grown and shifted so that it is only positive values, which is better for encoding in some cases.

[0037] Figure 3The diagram illustrates the generation of facets according to some embodiments. As described above, facet generation involves normal calculation, adjacency calculation, initial segmentation, and segmentation refinement. Calculating the normal for each triangle involves the cross product between the edges of the triangle. Normals can be represented as arrows or spikes, as shown in image 300. The adjacency calculation determines whether the triangles share vertices; if so, the triangles are neighbors. As shown, all gray triangles 302 are neighbors of the white triangle 304 in the middle. Initial segmentation and segmentation refinement are performed identically to V-PCC by analyzing the orientation of the normals, classifying the orientation of the normals (e.g., upward, downward, left, right, forward, backward), and determining whether the orientation of the normal is different from the classification of all adjacent normals that are classified the same (e.g., the first facet is classified as upward 306, while most or all adjacent facests are classified as forward), and then changing the classification of the facet's normal to match the orientation of the neighbor (e.g., changing the first facet classification to forward 308).

[0038] As described above, bin generation is performed to segment the mesh into bins. Bin generation also generates 1) a rasterized mesh surface and 2) vertex position and connectivity information. The rasterized mesh surface is a set of points that undergoes V-PCC image generation and is encoded as a V-PCC image. The vertex position and connectivity information are received for base mesh encoding.

[0039] The facet generation described in this article is similar to the facet generation in V-PCC. However, instead of calculating the normal for each point, the normal is calculated for each triangle. The normal for each triangle is calculated by determining the normal vector using the cross product between the edges. The triangles are then classified according to their normals. For example, the normals are divided into n (e.g., 6) categories, such as front, back, top, bottom, left, and right. The normals are indicated with different colors to represent the initial segmentation. Figure 3 Different colors in the grayscale, such as black and light gray, are shown as different colors indicating different normals. Although it may be difficult to see, the top surface (e.g., the top of the person's head, the top of the ball, and the top of the sneaker) is one color (e.g., green), the first side of the person / ball is very dark and represents another color (e.g., red), the bottom of the ball is another color (e.g., purple), and the front of the person and ball, which are mostly light gray, represents another color (e.g., cyan).

[0040] By multiplying the product of the normals by the direction, the main direction can be found. Smoothing / thinning can be done by looking at the neighboring triangles. For example, if all neighboring triangles above a threshold are blue, then the triangle is also classified as blue, even if there are anomalies that originally indicated the triangle was red.

[0041] Generate connected components of the triangles to identify which of the triangles have the same color (e.g., triangles of the same category that share at least one vertex).

[0042] Connectivity information describes how these points are connected in 3D. These connections together produce triangles (more specifically, 3 different connections that share 3 points), which in turn produce a surface (described by a collection of triangles). Although triangles are described in this article, other geometric shapes are also allowed (for example, rectangles).

[0043] Color can be used to encode connectivity by identifying triangles with different colors. Each triangle identified by three connections is coded with a unique color.

[0044] Figure 4 A diagram illustrating bin generation involving bin projection and bin rasterization in accordance with some embodiments. Bin projection includes components that generate triangles. All triangles that are connected by at least one vertex (e.g., adjacent) and belong to the same category are grouped together into a bin.

[0045] Surf projection also involves projecting each triangle onto a surfel on the canvas. If the projected position of a vertex is already occupied, the triangle is encoded in another surfel and thus placed in the missing triangle list so that it can be processed again later.

[0046] If a triangle produces a bounding box that results in depth values ​​exceeding the maximum allowed depth, these values ​​are not projected and also go into the missing triangle list.

[0047] Surfing involves rasterizing the projected triangles to generate points for the point cloud representation (geometry and attributes). Surfing and surfing are looped until all triangles are projected.

[0048] After all triangles have been projected, group the triangles in the missing triangle list together as a surfel and try again to project them into the new surfel.

[0049] By projecting the mesh onto a 2D surface, the area covered by the triangle's projection is also determined by the set of pixels. If the grouped pixels are coded with different colors, the triangles can be identified by their different colors in the image. Once the triangle is known, connectivity can be obtained by simply identifying the three connections that form the triangle.

[0050] Each triangle is projected onto a surfel. If the projected location of a vertex is already occupied, the triangle is encoded in another surfel, so that the triangle goes into a list of missing triangles to be processed again later. Alternatively, a map can be used to identify overlapping vertices and still represent triangles with overlapping vertices. In another alternative, the points can be separated into independent layers (e.g., one set of points in one layer and another set of points in another layer).

[0051] Image 400 shows a blank surfel with triangles of the same classification connected by at least one vertex. Image 402 shows a surfel with triangles showing vertices and connectivity. Image 404 shows a surfel with attribute information.

[0052] Figure 5 The diagram illustrates the data structure used for base mesh encoding / signaling according to some embodiments. Regarding base mesh signaling, packets and atlas streams are used to associate metadata with video sequences. Metadata is added by adding the positions of vertices in 2D space.

[0053] The data structure includes a header, a block group layer, and face information, such as the number of faces, the number of vertices, a list from 0 to the number of faces with face information, and a list from 0 to the number of vertices with vertex information. The vertex information is the position of the vertex in 2D space. Using these lists, a mesh can be reconstructed from the point cloud. By transmitting the vertex list and connectivity information, the transmission can be performed without any compression. In some embodiments, in order to take advantage of compression, the face connectivity can be encoded using TFAN or Edgebreaker. In some embodiments, parallelogram prediction of the vertices can be used. In some embodiments, DPCM encoding is implemented. In some embodiments, the vertex list is encoded using an occupancy map (for example, by including a flag to indicate whether the point exists, and using a flag to indicate whether the point is a vertex).

[0054] The list of points in a surfel is the vertices of a triangle, and the connectivity of the mesh is the same even after projection.

[0055] Connectivity is encoded (e.g., based on a color code). In some embodiments, a list of integer values ​​is encoded. Differential pulse code modulation (DPCM) of the list can be used. In some embodiments, the list can be refined, or smart grid coding (e.g., Edgebreaker) can be implemented. In some embodiments, more complex approaches are also possible (e.g., using Edgebreaker or TFAN, both of which are encoding algorithms).

[0056] Additional information can be sent per surfel. Within each surfel information, a list of connected components (e.g., vertices) and the position of the vertices in 2D space is sent. A more efficient representation can use a DPCM scheme for faces and vertices, as described in this paper.

[0057] Tracking mesh face metadata units can use facets to indicate that connectivity has not changed from one frame to the next. This is particularly useful for tracking meshes, as only delta positions are sent. In the case of tracking meshes, global motion can be captured by bounding box positions and rotations (newly introduced syntax elements using quaternions) and surface motion captured by vertex motion. The number of bits required to send delta vertex information can be sent in AFPS. Alternatively, motion information can be sent as a homography transform.

[0058] Figure 6 The diagram illustrates an example of combining point cloud and mesh information according to some embodiments. As described above, point cloud and mesh information can be combined. For example, pure point cloud bins can be used for the head, while point cloud and mesh bins can be used for the body.

[0059] Figure 7A and 7B This diagram illustrates an example of combining untracked and tracked mesh information in accordance with some embodiments. To avoid tracking issues, some implementations split the mesh into a tracked portion and an untracked portion. The tracked portion is temporally consistent and can be represented by a tracked_mesh_patch_data_unit(), while the untracked portion is new every frame and can be represented by a mesh_patch_data_unit(). Because point clouds can be blended into geometry as described herein, surface representation can be improved, for example, by retaining the original mesh and interpolating a point cloud on top of it to hide defects.

[0060] Figure 8

[0066] A diagram illustrating V-PCC image generation according to some embodiments. Occupancy maps and geometry are generated as usual for V-PCC. Attribute maps are generated from uncompressed geometry. Attribute transfer can be performed using a grid.

[0061] Figure 9 A diagram illustrating mesh reconstruction according to some embodiments is shown. Once the facets are generated / received, connectivity uses new vertex numbering, but the facets can be added together. Each facet is similar to a miniature mesh, so all facets are connected together. Although there may be vertex transformations, the facets are connected together to form a mesh. However, since some vertices do not match due to compression, further steps should be taken. A stitching algorithm can be implemented to address the problem of misaligned vertices.

[0062] Figure 10 A diagram illustrating a stitching algorithm according to some embodiments. The stitching algorithm includes classifying vertices into boundary vertices and non-boundary vertices. This classification can be performed in any manner, such as determining whether any other vertex is connected to the vertex in a specified direction. The nearest neighbors of the boundary are then determined, and these vertices adjacent to the boundary are merged (e.g., co-located). Furthermore, in V-PCC, two of the three coordinates are losslessly encoded, so from x, y, and z, it is known that two of the coordinates are accurate, which can help identify close points. Similarly, two points are considered to match if lines generated by changing only the normal coordinates intersect each other. An adaptive approach can be implemented: given a reference point at the boundary of a bin, these points are allowed to vary incrementally until they encounter at least one potential candidate point. The new distance is used to modify candidate points from other bins, and in the case of multiple matching candidate points, the reference point is modified to the value that produces the largest number of matches. Using the new modified reference point, the nearest boundary pixel from a neighboring bin is searched. If a match is found, the boundary vertex value is modified to the reference point, and the boundary vertex is removed from the list. Otherwise, the list of non-boundary vertices is searched for a match. If a match is found, the non-boundary vertex value is modified to the reference point.

[0063] Figure 11 A diagram illustrating the classification of vertices as boundaries or non-boundaries according to some embodiments. A vertex is a boundary if it is connected to an open area (e.g., a black area). In another example, a zero value for neighboring vertices in the occupied map indicates that the vertex is a boundary vertex.

[0064] Figure 12 A diagram illustrating an alternative connectivity coding according to some embodiments. Specific values ​​can be used to indicate the presence of a vertex and the presence of an edge. For example, the image represents an occupancy map where one of the surfels contains a point indicating a vertex and a black line indicating an edge.

[0065] Figure 13A block diagram illustrates an exemplary computing device configured to implement a projection-based mesh compression method in accordance with some embodiments. Computing device 1300 can be used to acquire, store, calculate, process, transmit, and / or display information such as images and videos including 3D content. Computing device 1300 can implement any aspect of projection-based mesh compression. Generally, a hardware structure suitable for implementing computing device 1300 includes a network interface 1302, memory 1304, a processor 1306, one or more I / O devices 1308, a bus 1310, and a storage device 1312. The choice of processor is not critical, as long as an appropriate processor with sufficient speed is selected. Memory 1304 can be any conventional computer memory known in the art. Storage device 1312 can include a hard drive, CDROM, CDRW, DVD, DVDRW, high-definition optical disc / drive, ultra-high-definition drive, flash memory card, or any other storage device. Computing device 1300 can include one or more network interfaces 1302. Examples of network interfaces include network cards connected to an Ethernet or other type of LAN. I / O devices 1308 can include one or more of the following: keyboard, mouse, monitor, screen, printer, modem, touch screen, button interface, and other devices. One or more projection-based mesh compression applications 1330 for implementing the projection-based mesh compression method may be stored in storage device 1312 and memory 1304 and processed in the same manner as applications are typically processed. The computing device 1300 can include Figure 13 In some embodiments, projection-based mesh compression hardware 1320 is included. Although Figure 13 The computing device 1300 in FIG. 1 includes an application 1330 and hardware 1320 for a projection-based mesh compression method, although the projection-based mesh compression method can be implemented on the computing device using hardware, firmware, software, or any combination thereof. For example, in some embodiments, the projection-based mesh compression application 1330 is programmed in memory and executed using a processor. In another example, in some embodiments, the projection-based mesh compression hardware 1320 is programmed hardware logic that includes logic gates specifically designed to implement the projection-based mesh compression method.

[0066] In some embodiments, the projection-based mesh compression application 1330 includes several applications and / or modules. In some embodiments, a module further includes one or more submodules. In some embodiments, fewer or additional modules can be included.

[0067] Examples of suitable computing devices include personal computers, laptops, computer workstations, servers, mainframe computers, handheld computers, personal digital assistants, cellular / mobile phones, smart appliances, game consoles, digital cameras, digital video cameras, camera phones, smartphones, portable music players, tablet computers, mobile devices, video players, video disc recorders / players (e.g., DVD recorders / players, high-definition disc recorders / players, ultra-high-definition disc recorders / players), televisions, home entertainment systems, augmented reality devices, virtual reality devices, smart jewelry (e.g., smart watches), vehicles (e.g., self-driving vehicles), or any other suitable computing device.

[0068] To utilize the projection-based mesh compression method, a device acquires or receives 3D content and processes and / or transmits the content in an optimized manner to enable proper and efficient display of the 3D content. The projection-based mesh compression method can be implemented with user assistance or automatically without user involvement.

[0069] In operation, the projection-based mesh compression method is able to achieve more efficient and accurate projection-based mesh compression compared to previous implementations.

[0070] In an exemplary implementation, the projection-based mesh compression described in this article was implemented on top of TMC2v8.0 using only one frame and one map. Information from this implementation includes:

[0071] Bitstream statistics:

[0072] Header: 16B 128b

[0073] vpcc unit size [VPCC_VPS]: 31B 248b

[0074] vpcc unit size [VPCC_AD]: 190396B 1523168B

[0075] vpcc unit size [VPCC_OVD]: 29591B 236728b (Ocm video = 29583B)

[0076] vpcc unit size [VPCC_GVD]: 59108B 472864B (Geo video = 59100B)

[0077] vpcc unit size [VPCC_AVD]: 172441B 1379528B (Tex video = 172433B)

[0078] Total metadata: 220050B 1760400b

[0079] Total geometry: 59100B 472800B

[0080] Total textures: 172433B 1379464b

[0081] Total: 451583B 1379464B

[0082] Total bitstream size 451599B

[0083] Some embodiments of projection-based mesh compression

[0084] 1. A method of programming a non-transitory memory of a device, comprising:

[0085] Voxelize the input mesh;

[0086] Implementing facet generation, which segments the mesh into facets including the rasterized mesh surface and vertex position and connectivity information;

[0087] Generate video-based point cloud compression (V-PCC) images from rasterized mesh surfaces;

[0088] Encoding the underlying mesh using vertex positions and connectivity information; and

[0089] A V-PCC bitstream is generated based on the V-PCC image and the basic grid coding, wherein the V-PCC bitstream includes information for point cloud reconstruction and grid reconstruction.

[0090] 2. The method of clause 1, wherein mesh voxelization comprises shifting and / or scaling mesh values ​​to avoid negative and non-integer values.

[0091] 3. The method of clause 2, wherein mesh voxelization comprises finding the lowest vertex value below zero and shifting the mesh values ​​so that the lowest vertex value is above zero.

[0092] 4. The method of clause 1, wherein implementing the bin generation comprises:

[0093] Calculate the normal of each triangle, where calculating the normal of a triangle involves using the cross product between the edges,

[0094] Sort triangles by their normals, and

[0095] Tessellation is performed by analyzing neighboring triangles.

[0096] 5. The method of clause 1, further comprising displaying content comprising partly a point cloud and partly a mesh.

[0097] 6. The method of clause 1, wherein incremental position information of tracked vertices is included in the bitstream.

[0098] 7. The method of clause 1, further comprising performing stitching, wherein the stitching comprises:

[0099] Classify vertices as boundary vertices or non-boundary vertices;

[0100] Determine the neighbors of the boundary vertices; and

[0101] Merge adjacent boundary vertices.

[0102] 8. A device comprising:

[0103] Non-transitory memory for storing applications used to:

[0104] Voxelize the input mesh;

[0105] Implementing facet generation, which segments the mesh into facets including the rasterized mesh surface and vertex position and connectivity information;

[0106] Generate video-based point cloud compression (V-PCC) images from rasterized mesh surfaces;

[0107] Encoding the underlying mesh using vertex positions and connectivity information; and

[0108] Generate a V-PCC bitstream based on the V-PCC image and the base grid coding, wherein the V-PCC bitstream includes information for point cloud reconstruction and grid reconstruction; and

[0109] A processor is coupled to the memory, the processor being configured to process the application.

[0110] 9. The apparatus of clause 8, wherein mesh voxelization comprises shifting and / or scaling mesh values ​​to avoid negative and non-integer values.

[0111] 10. The apparatus of clause 9, wherein mesh voxelization comprises finding the lowest vertex value below zero and shifting mesh values ​​so that the lowest vertex value is above zero.

[0112] 11. The apparatus of clause 8, wherein implementing bin generation comprises:

[0113] Calculate the normal of each triangle, where calculating the normal of a triangle involves using the cross product between the edges,

[0114] Sort triangles by their normals, and

[0115] Tessellation is performed by analyzing neighboring triangles.

[0116] 12. The apparatus of clause 8, wherein the application is further configured to display content comprising in part a point cloud and in part a mesh.

[0117] 13. The apparatus of clause 8, wherein incremental position information of tracked vertices is included in the bitstream.

[0118] 14. The apparatus of clause 8, wherein the application is further configured to implement stitching, the stitching comprising:

[0119] Classify vertices as boundary vertices or non-boundary vertices;

[0120] Determine the neighbors of the boundary vertices; and

[0121] Merge adjacent boundary vertices.

[0122] 15. A system comprising:

[0123] One or more cameras for acquiring three-dimensional content; and

[0124] An encoder for encoding 3D content by:

[0125] voxelize the input mesh of the 3D content;

[0126] Implementing facet generation, which segments the mesh into facets including the rasterized mesh surface and vertex position and connectivity information;

[0127] Generate video-based point cloud compression (V-PCC) images from rasterized mesh surfaces;

[0128] Encoding the underlying mesh using vertex positions and connectivity information; and

[0129] A V-PCC bitstream is generated based on the V-PCC image and the basic grid coding, wherein the V-PCC bitstream includes information for point cloud reconstruction and grid reconstruction.

[0130] 16. The system of clause 15, wherein mesh voxelization comprises shifting and / or scaling mesh values ​​to avoid negative and non-integer values.

[0131] 17. The system of clause 16, wherein mesh voxelization comprises finding the lowest vertex value below zero and shifting mesh values ​​so that the lowest vertex value is above zero.

[0132] 18. The system of clause 15, wherein implementing bin generation comprises:

[0133] Calculate the normal of each triangle, where calculating the normal of a triangle involves using the cross product between the edges,

[0134] Sort triangles by their normals, and

[0135] Tessellation is performed by analyzing neighboring triangles.

[0136] 19. The system of clause 15, further comprising displaying content comprising partly a point cloud and partly a mesh.

[0137] 20. The system of clause 15, wherein incremental position information for tracking vertices is included in the bitstream.

[0138] 21. The system of clause 15, further comprising implementing suturing, wherein the suturing comprises:

[0139] Classify vertices as boundary vertices or non-boundary vertices;

[0140] Determine the neighbors of the boundary vertices; and

[0141] Merge adjacent boundary vertices.

[0142] The present invention has been described by way of specific embodiments, which contain details that facilitate an understanding of the principles of construction and operation of the invention. Such reference herein to specific embodiments and details thereof is not intended to limit the scope of the appended claims. It will be apparent to those skilled in the art that various other modifications may be made in the embodiments chosen for illustration without departing from the spirit and scope of the invention as defined by the claims.

Claims

1. A method for projection-based mesh compression programmed in a non-transitory memory of a device, comprising: voxelizing an input mesh to generate a voxelized mesh, including generating a plurality of bounding boxes based on triangles of the input mesh, wherein when a bounding box includes a depth value above a maximum allowed depth, the corresponding triangle is added to a missing triangle list; Implementing facet generation, which segments the voxelized mesh into facets that include the rasterized mesh surface and vertex position and connectivity information; Generate video-based point cloud compression (V-PCC) images from rasterized mesh surfaces; Utilize vertex position and connectivity information to realize basic mesh encoding; and A V-PCC bitstream is generated based on the V-PCC image and the basic grid coding, wherein the V-PCC bitstream includes information for point cloud reconstruction and grid reconstruction. 2 . The method of claim 1 , wherein mesh voxelization comprises shifting and / or scaling mesh values ​​to avoid negative and non-integer values.

3. The method of claim 2, wherein mesh voxelization comprises finding the lowest vertex value below zero and shifting mesh values ​​so that the lowest vertex value is above zero.

4. The method according to claim 1, wherein implementing the facet generation comprises: Calculate the normal of each triangle, where calculating the normal of a triangle involves using the cross product between the edges, Sort triangles by their normals, and Tessellation is performed by analyzing neighboring triangles. The method of claim 1 , further comprising displaying content comprising partly a point cloud and partly a mesh. The method of claim 1 , wherein incremental position information of tracked vertices is included in the bitstream.

7. The method of claim 1 , further comprising performing suturing, wherein the suturing comprises: Classify vertices as boundary vertices or non-boundary vertices; Determine the nearest neighbors of the boundary vertices; as well as Merge adjacent boundary vertices.

8. An apparatus for projection-based mesh compression, comprising: Non-transitory memory for storing applications used to: voxelizing an input mesh to generate a voxelized mesh, including generating a plurality of bounding boxes based on triangles of the input mesh, wherein when a bounding box includes a depth value above a maximum allowed depth, the corresponding triangle is added to a missing triangle list; Implementing facet generation, which segments the voxelized mesh into facets that include the rasterized mesh surface and vertex position and connectivity information; Generate video-based point cloud compression (V-PCC) images from rasterized mesh surfaces; Utilize vertex position and connectivity information to realize basic mesh encoding; and Generate a V-PCC bitstream based on the V-PCC image and the base grid coding, wherein the V-PCC bitstream includes information for point cloud reconstruction and grid reconstruction; as well as A processor is coupled to the memory, the processor being configured to process the application.

9. The apparatus of claim 8, wherein mesh voxelization comprises shifting and / or scaling mesh values ​​to avoid negative and non-integer values.

10. The apparatus of claim 9, wherein mesh voxelization comprises finding the lowest vertex value below zero and shifting mesh values ​​so that the lowest vertex value is above zero.

11. The apparatus of claim 8, wherein implementing bin generation comprises: Calculate the normal of each triangle, where calculating the normal of a triangle involves using the cross product between the edges, Sort triangles by their normals, and Tessellation is performed by analyzing neighboring triangles.

12. The device of claim 8, wherein the application is further configured to display content comprising partly a point cloud and partly a mesh.

13. The apparatus of claim 8, wherein incremental position information of tracked vertices is included in the bitstream.

14. The device of claim 8, wherein the application is further configured to implement stitching, the stitching comprising: Classify vertices as boundary vertices or non-boundary vertices; Determine the nearest neighbors of the boundary vertices; as well as Merge adjacent boundary vertices.

15. A system for projection-based mesh compression, comprising: one or more cameras for acquiring three-dimensional content; and An encoder for encoding 3D content by: voxelizing an input mesh of three-dimensional content to generate a voxelized mesh, including generating a plurality of bounding boxes based on triangles of the input mesh, wherein when a bounding box includes a depth value above a maximum allowed depth, the corresponding triangle is added to a missing triangle list; Implementing facet generation, which segments the voxelized mesh into facets that include the rasterized mesh surface and vertex position and connectivity information; Generate video-based point cloud compression (V-PCC) images from rasterized mesh surfaces; Utilize vertex position and connectivity information to realize basic mesh encoding; and A V-PCC bitstream is generated based on the V-PCC image and the basic grid coding, wherein the V-PCC bitstream includes information for point cloud reconstruction and grid reconstruction.

16. The system of claim 15, wherein mesh voxelization comprises shifting and / or scaling mesh values ​​to avoid negative and non-integer values.

17. The system of claim 16, wherein mesh voxelization comprises finding the lowest vertex value below zero and shifting mesh values ​​so that the lowest vertex value is above zero.

18. The system of claim 15, wherein implementing bin generation comprises: Calculate the normal of each triangle, where calculating the normal of a triangle involves using the cross product between the edges, Sort triangles by their normals, and Tessellation is performed by analyzing neighboring triangles.

19. The system of claim 15, wherein the encoder is further configured to display content that partially comprises a point cloud and partially comprises a mesh.

20. The system of claim 15, wherein incremental position information of tracked vertices is included in the bitstream.

21. The system of claim 15, wherein the encoder is further configured to implement stitching, the stitching comprising: Classify vertices as boundary vertices or non-boundary vertices; Determine the nearest neighbors of the boundary vertices; as well as Merge adjacent boundary vertices.

Images