Dense grid compression
By dividing the mesh into surface patches and encoding them using the V3C standard, the problem of low efficiency in mesh connectivity encoding in existing technologies is solved, achieving efficient compression of sparse meshes and triangular faces while preserving important attribute information.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SONY GROUP CORP
- Filing Date
- 2021-09-28
- Publication Date
- 2026-07-21
AI Technical Summary
Existing 3D mesh compression methods cannot effectively encode mesh connectivity, especially for sparse meshes and data with triangular facets, resulting in low encoding efficiency and loss of important information.
A projection-based approach is adopted to divide the mesh into surface patches. The 3D surface patches represent the set of connected triangles on the mesh surface, and the groups of vertices represent the triangles not captured by the surface projection. The mesh is encoded using the V3C standard, and additional connectivity data is used to improve the rendering and point filtering algorithms.
It achieves efficient encoding of sparse meshes and triangles, improves encoding efficiency, preserves the attribute information of triangles, and better integrates with point cloud compression methods.
Smart Images
Figure CN115443485B_ABST
Abstract
Description
[0001] Cross-references to (one or more) related applications
[0002] This application claims priority to U.S. Provisional Patent Application Serial No. 63 / 086,142 entitled “DENSE MESHCOMPRESSION”, filed October 1, 2020, under 35 USC §119(e), which is incorporated herein by reference in its entirety for all purposes. Technical Field
[0003] This invention relates to three-dimensional graphics. More specifically, this invention relates to the encoding of three-dimensional graphics. Background Technology
[0004] Recently, a novel method for compressing volumetric content (such as point clouds) based on 3D-to-2D projection is being standardized. Also known as V3C (Visual Volumetric Video Compression), this method maps 3D volumetric data into several 2D patches, then arranges these patches into an atlas image, which is subsequently encoded using a video encoder. The atlas image corresponds to an occupancy map that considers the geometry of the points, their corresponding textures, and indicators of which locations are used for point cloud reconstruction.
[0005] In 2017, MPEG issued a Call for Proposals (CfP) for point cloud compression. After evaluating several proposals, MPEG is currently considering two different point cloud compression techniques: native 3D coding (based on octrees and similar coding methods), or 3D-to-2D projection, followed by traditional video coding. In the case of dynamic 3D scenes, MPEG uses Test Model Software (TMC2) based on patch surface modeling to project patches from 3D to a 2D image, and then encodes the 2D image using a video encoder such as HEVC. This approach has proven to be more efficient than native 3D coding and can achieve competitive bit rates with acceptable quality.
[0006] Due to the success of encoding 3D point clouds using projection-based methods (also known as video-based methods or V-PCC), the standard is expected to include further 3D data, such as 3D meshes, in future versions. However, the current version of the standard is only suitable for transmitting disconnected sets of points, and therefore lacks a mechanism to transmit point connectivity, as is required for 3D mesh compression.
[0007] Methods for extending the functionality of V-PCC to meshes have been proposed. One possible approach is to encode vertices using V-PCC and then encode connectivity using mesh compression methods such as TFAN or Edgebreaker. The limitation of this approach is that the original mesh must be dense so that the point cloud generated from the vertices is not sparse and can be efficiently encoded after projection. Furthermore, the order of the vertices affects the encoding of connectivity, and different methods have been proposed to reorganize mesh connectivity. An alternative to encoding sparse meshes is to encode vertex positions in 3D using RAW patch data. Since RAW patches directly encode (x, y, z), in this approach, all vertices are encoded as RAW data, and connectivity is encoded using a similar mesh compression method, as mentioned earlier. Note that in RAW patches, vertices can be sent in any preferred order, thus allowing the use of the order generated from connectivity encoding. This method can encode sparse point clouds; however, RAW patches are not efficient at encoding 3D data, and this method loses further data, such as the properties of triangular faces. Summary of the Invention
[0008] This paper describes a method for compressing meshes using a projection-based approach, leveraging and extending tools and syntax already developed for projection-based volumetric content compression. Similar to the V3C method, the mesh is divided into surface patches, but these partitions follow the mesh's connectivity. Dense mesh compression utilizes 3D surface patches to represent the set of connected triangles on the mesh surface and uses groups of vertices to represent triangles not captured by surface projection. Each surface patch (or 3D patch) is then projected onto a 2D patch, thus, in the case of the mesh, triangle surface sampling is analogous to common rasterization methods used in computer graphics. For each patch, the positions of the projected vertices and the connectivity of those vertices are maintained in a list. The sampled surfaces are analogous to point clouds and encoded using the same methods used for point cloud compression. Furthermore, the list of vertices and connectivity for each patch is encoded, and the data is sent along with the encoded point cloud data.
[0009] In one aspect, a method includes performing mesh voxelization on an input mesh, implementing patch generation that divides the mesh into patches comprising surface meshes and unprojected triangles, performing mesh encoding on the surface meshes, performing triangle encoding on the unprojected triangles, and generating a V3C bitstream based on the mesh encoding and triangle encoding. Mesh encoding includes determining and storing the positions of the projected vertices and the connectivity of the projected vertices. Triangle encoding includes encoding triangles not captured during surface projection. Triangle encoding includes encoding triangles using only vertex position and color vertex information. Triangle encoding includes determining that if the bounding box of a connected component is smaller than a predefined region, then unprojected triangles are moved to a separate list for independent triangle encoding, and the unprojected triangles are not rasterized but encoded as vertices with an associated color for each vertex; otherwise, each triangle is projected onto a patch, and if the projection position of a vertex is already occupied, then the triangle is encoded in another patch, thus placing the triangle in a list of missing triangles for later processing. Triangle encoding uses triangle patch data units. The triangular patch data unit stores triangles as separate triangles, triangular stripes, or triangular sectors. Triangle encoding also includes a color extension implementation, which writes the color of the triangle as c0 c1 c2; c m c m c m c0 c1 c2, where c m It is the color value of the centroid of the triangle. Triangle encoding also includes color compression.
[0010] In another aspect, an apparatus includes a non-transitory memory for storing an application for: performing mesh voxelization on an input mesh; performing patch generation, which divides the mesh into patches comprising surface meshes and unprojected triangles; performing mesh encoding on the surface meshes; performing triangle encoding on the unprojected triangles; and generating a V3C bitstream based on the mesh encoding and triangle encoding; and a processor coupled to the memory, configured to process the application. Mesh encoding includes determining and storing the positions of projected vertices and the connectivity of the projected vertices. Triangle encoding includes encoding triangles not captured during surface projection. Triangle encoding includes encoding triangles using only vertex position and color vertex information. Triangle encoding includes determining that if the bounding box of a connected component is smaller than a predefined region, then unprojected triangles are moved to a separate list for independent triangle encoding, and the unprojected triangles are not rasterized but encoded as vertices with an associated color for each vertex; otherwise, each triangle is projected onto a patch, and if the projection position of a vertex is already occupied, then the triangle is encoded in another patch, thus placing the triangle in a list of missing triangles for later processing. Triangle encoding uses triangular patch data units. These units store triangles as separate triangles, triangular stripes, or triangular sectors. Triangle encoding also includes a color extension implementation that writes the color of a triangle as c0 c1 c2; c m c m c m ;c0c1c2, where c m It is the color value of the centroid of the triangle. Triangle encoding also includes color compression.
[0011] On the other hand, the system includes one or more cameras for acquiring 3D content, and an encoder for encoding the 3D content: performing mesh voxelization on the input mesh of the 3D content, implementing patch generation that divides the mesh into patches including surface meshes and unprojected triangles, performing mesh encoding on the surface meshes, performing triangle encoding on the unprojected triangles, and generating a V3C bitstream based on the mesh encoding and triangle encoding. Mesh encoding includes determining and storing the positions of the projected vertices and the connectivity of the projected vertices. Triangle encoding includes encoding triangles not captured during surface projection. Triangle encoding includes encoding triangles using only vertex position and color vertex information. Triangle encoding includes determining that if the bounding box of a connected component is smaller than a predefined region, then unprojected triangles are moved to a separate list for independent triangle encoding, and the unprojected triangles are not rasterized but encoded as vertices with an associated color for each vertex; otherwise, each triangle is projected onto a patch, and if the projection position of a vertex is already occupied, then the triangle is encoded in another patch, and thus the triangle is placed in a list of missing triangles for later processing. Triangle encoding uses triangular patch data units. These units store triangles as separate triangles, triangular stripes, or triangular sectors. Triangle encoding also includes a color extension implementation that writes the color of a triangle as c0 c1 c2; c m c m c m c0 c1 c2, where c m It is the color value of the centroid of the triangle. Triangle encoding also includes color compression. Attached Figure Description
[0012] Figure 1 The diagram illustrates a flowchart of a dense grid coding method according to some embodiments.
[0013] Figure 2 The diagram illustrates an exemplary dense grid projection problem according to some embodiments.
[0014] Figure 3 The illustration shows a patch generated according to some embodiments.
[0015] Figure 4 The illustration shows a patch rasterization diagram according to some embodiments.
[0016] Figure 5 The diagram illustrates a triangular grouping according to some embodiments.
[0017] Figure 6 The diagram illustrates the encoding of triangles in a video image according to some embodiments.
[0018] Figure 7 The diagram illustrates a set of triangles for a patch according to some embodiments.
[0019] Figure 8 The illustration shows an example of encoding a dense grid using triangular patches without color expansion, according to some embodiments.
[0020] Figure 9 The illustration shows an example of encoding a dense grid using triangular patches with color extensions, according to some embodiments.
[0021] Figure 10 The illustration shows an image of a triangular patch implementation applied to a sparse grid according to some embodiments.
[0022] Figure 11 The illustration shows a block diagram of an exemplary computing device configured to implement a dense grid compression method according to some embodiments.
[0023] Figure 12 The illustration shows a schematic diagram of a system configured to implement dense grid coding according to some embodiments. Detailed Implementation
[0024] This paper describes a method for compressing meshes using a projection-based approach, leveraging and extending existing tools and syntax for volumetric content compression based on projection. Similar to the V3C method, the mesh is divided into surface patches, except that these partitions follow the mesh's connectivity. Each surface patch (or 3D patch) is then projected onto a 2D patch, thus, in the case of the mesh, triangular surface sampling is analogous to common rasterization methods used in computer graphics. For each patch, the positions of the projected vertices and the connectivity of those vertices are maintained in a list. The sampled surfaces are analogous to point clouds and encoded using the same methods used for point cloud compression. Furthermore, the list of vertices and connectivity for each patch is encoded, and the data is sent along with the encoded point cloud data.
[0025] The additional connectivity data is interpreted as a base mesh generated for each patch, allowing the decoder the flexibility to use or not use this additional data. This data can be used to improve rendering and in point filtering algorithms. Moreover, the mesh is encoded using the same principles as projection-based compression, leading to better integration with current V-PCC methods.
[0026] This paper also describes a method for encoding triangles that were not captured during surface projection, which is common for very dense meshes. The small triangles are then collected and encoded using only their vertex positions and color vertices.
[0027] State-of-the-art point cloud compression techniques use video encoders to represent point clouds as 3D patches and encode 2D images formed by projections of geometry and attributes onto a 2D canvas. However, such methods are primarily applicable to 3D point clouds and cannot be applied to the compression of 3D meshes because there is no suitable way to compress mesh connectivity. Furthermore, when encoding with V-PCC, sparsely sparse meshes perform poorly due to data sparsity and poor color representation.
[0028] Methods have been proposed to extend the functionality of V-PCC to meshes. One possible approach is to encode vertices using V-PCC and then encode connectivity using mesh compression methods such as TFAN or Edgebreaker. The limitation of this approach is that the original mesh should be dense so that the point cloud generated from the vertices is not sparse and can be efficiently encoded after projection. Furthermore, the order of the vertices affects the encoding of connectivity, and different methods have been proposed to reorganize mesh connectivity.
[0029] An alternative to encoding sparse meshes is to encode vertex positions in 3D using RAW patch data. Since RAW patches directly encode (x, y, z), all vertices are encoded as RAW data in this approach, while connectivity is encoded using a similar mesh compression method, as mentioned earlier. Vertices can be sent in any preferred order within the RAW patches, thus allowing the use of an order generated from connectivity encoding. This method can encode sparse point clouds; however, RAW patches are not efficient for encoding 3D data. Furthermore, this method loses further data, such as properties of triangle faces.
[0030] This paper describes a novel method for compressing 3D mesh data using projections of mesh surface data and isolated triangle encoding. Dense mesh compression utilizes 3D surface patches to represent sets of connected triangles on the mesh surface and groups of vertices to represent triangles not captured by surface projection. Both datasets are stored in patches (mesh patches and triangle patches) encoded in atlas data. This type of approach extends the functionality of the current V3C (Volume-Based Video) standard used for encoding point clouds and multi-view depth content.
[0031] In 3D point cloud encoding using a video encoder, 3D-to-2D projection is used to generate video representing the point cloud. The most efficient way to generate such videos is using 3D patches, which segment the surface of an object and use orthographic projection to generate segmented depth images, which are bundled together and used as input to the video encoder. Furthermore, points not captured through the projection step can also be directly encoded into the video signal. Current point cloud standards cannot encode 3D meshes because there is no defined method for encoding mesh connectivity. Moreover, the standard performs poorly if vertex data is sparse because it cannot utilize the correlation between vertices. Embodiments of the present invention include a method for encoding meshes to encode volumetric data using the V3C standard. Such methods segment the mesh surface and perform articulated surface sampling and 2D patch generation. The positions of vertices projected onto the 2D patches, local connectivity, and each patch are then encoded.
[0032] Methods for signaling connectivity and vertex positions are also disclosed, enabling the reconstruction of the original input mesh. Furthermore, methods for encoding triangles not captured during the surface projection stage are disclosed. The vertices of unprojected triangles are encoded as points, and the point coordinates are directly added to the video data. Further metadata provides an efficient way to reconstruct a list of triangles and recover the color data of the triangle vertices from the encoded vertex data encoded in the video sequence. Dense mesh compression can be applied to dense time-varying meshes whose mesh properties (such as texture) are described by texture coordinates or using the color of each vertex.
[0033] Figure 1 The diagram illustrates a flowchart of a method for dense mesh encoding according to some embodiments. In step 100, an input (dense) mesh is received or acquired. For example, the dense mesh is downloaded (e.g., from a network device) or acquired / captured by a device (e.g., a camera or autonomous vehicle).
[0034] In step 102, mesh voxelization is performed. Mesh can have floating-point vertex positions, so these positions are converted to integer space. V-PCC and V3C assume voxelized point clouds.
[0035] In step 104, patch generation (or creation) is performed. Patch generation can include: normal calculation, adjacency calculation, initial segmentation; thinning, patch projection, and patch rasterization. Normal calculation is calculating the normal of each triangle (e.g., the cross product of the triangle's edges). Adjacency calculation involves calculating the adjacency relationship of each triangle (e.g., which triangles in the mesh are adjacent to or contact the current triangle or other triangles). Initial segmentation involves classifying the normals according to their orientation. For example, the normals of a triangle can point up, down, left, right, front, or back, and can be classified based on direction / orientation. In some embodiments, triangles are color-coded based on the orientation of their normals (e.g., all triangles with normals pointing up are painted green). Thinning involves locating outliers (e.g., a single red triangle surrounded by blue triangles) and smoothing outliers (e.g., changing a single red triangle to match its blue neighbor). Thinning is performed by analyzing neighbors and smoothing orientations (e.g., adjusting the orientation of the normals). Once a smooth surface is found, patch projection is performed, which involves projecting patches for a specific classification of triangles (e.g., based on orientation). Through projection, vertices and connectivity are shown on the patches. For example, in this example, the body and face are separate projections because triangles with different classifications separate them. Although V3C and V-PCC do not understand this; more precisely, V3C and V-PCC understand points, so the projection is rasterized (e.g., sampling points on the surface include distances between points to generate a geometric image and surface properties). The rasterized mesh surface is very similar to the V3C image. Patch generation is able to produce a surface mesh 120 and unprojected triangles 130. Unprojected triangles 130 are shown as missing spaces in the surface mesh 120. Unprojected triangles 130 can appear for a variety of reasons, such as occlusion.
[0036] In step 106, mesh coding is performed. Mesh coding includes coding the surface mesh and can be the coding described herein or in U.S. Patent Application Serial 17 / 161,300 entitled “PROJECTION-BASED MESH COMPRESSION”, filed January 28, 2021, which is incorporated herein by reference in its entirety for all purposes.
[0037] In step 108, triangle encoding is performed. Triangle encoding includes encoding unprojected triangles. If the bounding box of a connected component is smaller than a predefined area, the triangle is moved to a separate list for independent triangle encoding. Unprojected triangles are not rasterized but are encoded as vertices with an associated color for each vertex. Otherwise, each triangle is projected onto a patch. If the projection position of a vertex is already occupied, the triangle is encoded into another patch, thus placing it in the missing triangle list for later processing.
[0038] In step 110, a V3C bitstream is generated based on the lattice encoding and triangle encoding. In some embodiments, fewer or additional steps are implemented. In some embodiments, the order of the steps is modified.
[0039] Figure 2 The diagram illustrates an exemplary dense mesh projection problem according to some embodiments. Not all triangles have the same classification.
[0040] Figure 3 The illustration shows a patch generation diagram according to some embodiments. Patch generation involves determining the adjacent neighborhood of each triangle, calculating the normal of each triangle, classifying the triangles based on the normal, and refining the categories based on adjacent values.
[0041] Figure 4 The diagram illustrates patch rasterization according to some embodiments, one of the components of the patch generation process. Patch generation also includes generating connected components of triangles (triangles of the same category share at least one vertex). If the bounding box of a connected component is smaller than a predefined region, the triangle is moved to a separate list for independent triangle encoding. These unprojected triangles are not rasterized but are encoded as vertices with associated colors. Otherwise, each triangle is projected onto a patch. If the projection position for a vertex is already occupied, the triangle is encoded into another patch and added to a list of missing triangles for later processing. Alternatively, a map can be used to identify overlapping vertices and still represent triangles with overlapping vertices. The triangles are rasterized to generate points for point cloud representation.
[0042] The original voxelized vertex 400 is shown. The rasterized surface points 402 (added to the point cloud representation) follow the structure of the mesh, so the point cloud geometry can be as coarse as the underlying mesh. However, the geometry can be improved by sending additional positions for each rasterized pixel.
[0043] Mesh patch data units (MPDUs) can be used to store a list of points in a patch and the connectivity of the mesh, where the points are the vertices of triangles, and this connectivity remains the same even after projection. Connectivity can be encoded using mesh patch data units. It can be encoded as a list of integer values, or using DPCM within the list. More complex methods are also possible, such as edgebreakers or TFAN. The (u,v) coordinates of the vertices are encoded instead of (x,y,z) coordinates. (u,v) positions can be encoded, or the DPCM method is possible. The order can be determined by connectivity. Parallelogram prediction (e.g., Draco) can be used. Vertex positions can be sent via an occupancy map.
[0044] The following is an example MPDU:
[0045]
[0046]
[0047] Alternative encoding methods include encoding patch connectivity using TFAN or Edgebreaker, using parallelogram prediction for vertices, using DPCM encoding, or using occupancy maps for vertex locations.
[0048] Triangle information is stored in Triangle Patch Data Units (TPDUs). TPDUs store a list of points in a patch that are the vertices of triangles, which are formed according to primitive indicators. Primitive indicators include: (0) Separated triangles: (0 1 2), (3 4 5), (6 7 8), where every three vertices define a triangle; (1) Triangle stripes: (0 1 2), (2 1 3), (2 3 4), where each new vertex generates a new triangle with the previous two vertices, as some vertices are shared; and (2) Triangle sectors: (0 1 2), (0 2 3), (0 3 4), where every two new vertices generate a new triangle centered on the first vertex.
[0049] In other words, there are many vertices, and the connectivity of the vertices is generated / determined. There are many ways to generate / determine connectivity: separate triangles (every three vertices are encoded), triangle strips (if triangles share vertices, then they can be encoded as strips), or triangle sectors (if triangles share a single vertex, then they can be encoded as sectors).
[0050] Figure 5 The diagram illustrates a grouping of triangles according to some embodiments. Triangular patch data includes a list of points within the patch, which are the vertices of the triangles. There can be separate triangles 500 (individual, unconnected triangles), triangle strips 502 (multiple, connected triangles), or triangle sectors 504 (multiple, connected triangles connected at a single vertex).
[0051] It is possible to pack triangular patch data using any method, such as component packing (similar to packing RAW patches) or row interleaving in which a row is assigned to each component.
[0052] Color expansion changes the packing of (x,y,z) coordinates to row packing. The class also indicates whether colors will be expanded, which is useful for textured meshes due to UV interpolation artifacts. If expansion is not used, colors are written sequentially (cl c2c3). Otherwise, they are written as: [c0 c1 c2; c...]. mc m c m [c0 c1 c2], where c m This is the color value at the centroid of the triangle, which is equivalent to the average color of the three vertices. Through color expansion, the UV coordinates should point to the center pixel.
[0053] Figure 6 The diagram illustrates the encoding of triangles in a video image according to some embodiments. The position of the triangle is encoded in the video image (e.g., x, y, z coordinates in the video image). There are many ways to pack the coordinates. In one example 600, all x coordinates are packed first, then y coordinates, and then z coordinates. In another example 602, a row interleaving implementation is performed and then used to implement color expansion.
[0054] Figure 7 The diagram illustrates a set of triangles for a patch according to some embodiments. A method for generating triangular patches from broken (or connected) triangles is described. The method also guarantees that the bit depth used to represent the vertices is within a certain value (e.g., dividing 10 bits of data into 8-bit partitions). The bounding boxes of the triangles are calculated, and if the size of the bounding box is greater than a predefined bit depth, the triangle is removed and not encoded. When the list of triangles is not empty, a selected bounding box with a predefined size (2^numBits-1) is generated, centered on the center of the bounding box of the first triangle. All triangles are iterated through, and for each triangle, it is checked whether the selected bounding box contains the bounding box of the triangle. If it contains the bounding box of the triangle, the triangle is added to the triangular patch and removed from the list of triangles. For the encoding of the triangles, instead of sending 3D data, only the bounding box points, reference points, and increments are sent.
[0055] like Figure 7 As shown, a bounding box is calculated for each of the three separate triangles, and then the selected bounding box is compared with the triangle bounding box. Since two triangle bounding boxes are inside the selected bounding box, they are in the same triangle patch (e.g., triangle patch #1), while the third triangle, which is not inside the selected bounding box, is in the second separate triangle patch (e.g., triangle patch #2).
[0056] The following is an example TPDU syntax:
[0057]
[0058] Figure 8The illustration shows an example of encoding a dense mesh using triangular patches without color expansion, according to some embodiments. If the triangles are placed adjacent to each other, there are many artifacts when reconstructing the triangles. This is because the colors used for reconstruction are interpolated using neighboring points, which contaminates the shading. One way to avoid contamination is to use vertex colors instead of UV textures, but this also has drawbacks.
[0059] Figure 9 The illustration depicts an example of encoding a dense mesh using triangular patches with color expansion, according to some embodiments. In addition to the vertex colors, the triangle colors are included. The intermediate colors of the triangles are also included. This results in much more accurate color (e.g., fewer artifacts, if any) and is similar to using vertex colors. This implementation significantly increases data usage. Color compression can be used to reduce data usage.
[0060] To improve color compression, separate video streams (using auxiliary video with RAW patches) can be used for triangular patches. Triangles within a patch can be sorted according to the color of their centroids. This clusters triangles with similar color values and creates images that are easier to encode. Instead of using the (x,y,z) coordinates of the vertices, if the vertices have already been encoded through the grid patches, the patch index and vertex index packed in the (x,y,z) data can be sent. This also eliminates the need to remove duplicate vertices, as long as the vertices have already been encoded.
[0061] Figure 10 Images illustrating triangular patches applied to sparse meshes according to some embodiments are shown. Triangular patches can be applied to sparse meshes, but color is affected because the entire surface will be represented by interpolation of three colors. However, color extension techniques can be used to improve texture quality. Image 1000 shows encoding with vertex colors. Image 1002 shows encoding without color extension, which has many artifacts. Image 1004 shows encoding with color extension, which has better quality than the other two.
[0062] Figure 11The illustration shows a block diagram of an exemplary computing device configured to implement a dense mesh compression method according to some embodiments. The computing device 1100 can be used to acquire, store, compute, process, transmit, and / or display information, such as images and videos including 3D content. The computing device 1100 can implement any encoding / decoding aspects. Generally, a suitable hardware architecture for implementing the computing device 1100 includes a network interface 1102, memory 1104, a processor 1106, one or more I / O devices 1108, a bus 1110, and a storage device 1112. The choice of processor is not critical, as long as a suitable processor with sufficient speed is selected. Memory 1104 can be any conventional computer memory known in the art. Storage device 1112 can include a hard disk drive, CD-ROM, CDRW, DVD, DVDRW, high-definition disc / drive, ultra-high-definition drive, flash memory card, or any other storage device. The computing device 1100 can include one or more network interfaces 1102. Examples of network interfaces include network interface cards (NICs) connected to Ethernet or other types of LANs. One or more I / O devices 1108 may include one or more of the following: keyboard, mouse, monitor, screen, printer, modem, touchscreen, button interface, and other devices. One or more dense mesh compression applications 1130 for implementing dense mesh compression implementations may be stored in storage device 1112 and memory 1104 and processed when the application is normally processed. Figure 11 More or fewer components as shown can be included in computing device 1100. In some embodiments, dense mesh compression hardware 1120 is included. Although Figure 11 The computing device 1100 includes an application 1130 and hardware 1120 for implementing dense grid compression, but the dense grid compression method can be implemented on the computing device in hardware, firmware, software, or any combination thereof. For example, in some embodiments, the dense grid compression application 1130 is programmed in memory and executed using a processor. In another example, in some embodiments, the dense grid compression hardware 1120 is programmed hardware logic that includes gates specifically designed to implement the dense grid compression method.
[0063] In some embodiments, one or more dense mesh compression applications 1130 include several applications and / or modules. In some embodiments, a module further includes one or more sub-modules. In some embodiments, fewer or additional modules may be included.
[0064] Examples of suitable computing devices include personal computers, laptop computers, computer workstations, servers, mainframe computers, handheld computers, personal digital assistants, cellular / mobile phones, smart appliances, game consoles, digital cameras, digital camcorders, camera phones, smartphones, portable music players, tablet computers, mobile devices, video players, video disc burners / players (e.g., DVD burners / players, HD disc burners / players, UHD disc burners / players), televisions, home entertainment systems, augmented reality devices, virtual reality devices, smart jewelry (e.g., smartwatches), vehicles (e.g., self-driving vehicles), or any other suitable computing device.
[0065] Figure 12 The illustration shows a schematic diagram of a system configured to implement dense mesh encoding according to some embodiments. Encoder 1200 is configured to implement the encoding process. As described herein, any encoding, such as dense mesh compression, can be implemented. The mesh and other information can be transmitted directly to decoder 1204 or via network 1202. The network can be any type of network, such as a local area network (LAN), the Internet, a wireless network, a wired network, a cellular network, and / or any other network or combination of networks. Decoder 1204 decodes the encoded content.
[0066] To utilize the dense mesh compression method, the device acquires or receives 3D content (e.g., point cloud content). The dense mesh compression method can be implemented with user assistance or automatically without user intervention.
[0067] In practice, the dense grid compression method enables more efficient and accurate 3D content encoding compared to previous implementations.
[0068] Some examples of dense mesh compression
[0069] 1. A method comprising:
[0070] Perform mesh voxelization on the input mesh;
[0071] Perform patch generation, which divides the mesh into patches that include surface meshes and unprojected triangles;
[0072] Perform mesh encoding on the surface mesh;
[0073] Perform triangle encoding on unprojected triangles; and
[0074] V3C bitstream is generated based on grid coding and triangle coding.
[0075] 2. The method of Clause 1, wherein mesh encoding includes determining and storing the positions of the projected vertices and the connectivity of the projected vertices.
[0076] 3. The method of Clause 1, wherein triangle encoding includes encoding triangles not captured during surface projection.
[0077] 4. The method of Clause 3, wherein triangle encoding includes encoding triangles using only vertex position and color vertex information.
[0078] 5. The method of Clause 1, wherein triangle encoding comprises determining that if the bounding box of a connected component is smaller than a predefined region, then unprojected triangles are moved to a separate list for independent triangle encoding, and the unprojected triangles are not rasterized but are encoded as vertices with an associated color for each vertex; otherwise, each triangle is projected onto a patch, and if the projection position of a vertex is already occupied, then the triangle is encoded in another patch, and thus the triangle is placed in a list of missing triangles for later processing.
[0079] 6. The method of Clause 1, wherein triangle encoding uses triangular patch data units.
[0080] 7. The method of Clause 6, wherein the triangular patch data unit stores the triangle as separate triangles, triangular stripes, or triangular sectors.
[0081] 8. The method of Clause 1, wherein the triangle encoding further includes a color extension implementation, wherein the color extension implementation writes the color of the triangle as c0 c1 c2; m c m c m c0 c1 c2, where c m It is the color value of the centroid of the triangle.
[0082] 9. The method of Clause 8, wherein triangle encoding also includes color compression.
[0083] 10. An apparatus comprising:
[0084] Non-transitory memory for storing applications, which are used for:
[0085] Perform mesh voxelization on the input mesh;
[0086] Perform patch generation, which divides the mesh into patches that include surface meshes and unprojected triangles;
[0087] Perform mesh encoding on the surface mesh;
[0088] Perform triangle encoding on unprojected triangles; and
[0089] V3C bitstream generation based on grid coding and triangle coding; and
[0090] A processor coupled to the memory, the processor being configured to process the application.
[0091] 11. The apparatus of Clause 10, wherein mesh encoding includes determining and storing the positions of the projected vertices and the connectivity of the projected vertices.
[0092] 12. The apparatus of Clause 10, wherein triangle encoding includes encoding triangles not captured during surface projection.
[0093] 13. The apparatus of Clause 12, wherein triangle encoding includes encoding triangles using only vertex position and color vertex information.
[0094] 14. The apparatus of Item 10, wherein triangle encoding comprises determining that if the bounding box of a connected component is smaller than a predefined region, then unprojected triangles are moved to a separate list for independent triangle encoding, and the unprojected triangles are not rasterized but are encoded as vertices with an associated color for each vertex; otherwise, each triangle is projected onto a patch, and if the projection position of a vertex is already occupied, then the triangle is encoded in another patch, and thus the triangle is placed in a list of missing triangles for later processing.
[0095] 15. The apparatus of Clause 10, wherein triangle encoding uses triangle patch data units.
[0096] 16. The apparatus of Clause 15, wherein the triangular patch data unit stores triangles as separate triangles, triangular stripes, or triangular sectors.
[0097] 17. The apparatus of Clause 10, wherein the triangle encoding further includes a color extension implementation, wherein the color extension implementation writes the color of the triangle as c0 c1 c2; m c m c m c0 c1 c2, where c m It is the color value of the centroid of the triangle.
[0098] 18. The apparatus of Clause 17, wherein the triangle encoding further includes color compression.
[0099] 19. A system comprising:
[0100] One or more cameras are used to acquire 3D content;
[0101] An encoder is used to encode 3D content.
[0102] Perform mesh voxelization on the input mesh of 3D content;
[0103] Perform patch generation, which divides the mesh into patches that include surface meshes and unprojected triangles;
[0104] Perform mesh encoding on the surface mesh;
[0105] Perform triangle encoding on unprojected triangles; and
[0106] V3C bitstream is generated based on grid coding and triangle coding.
[0107] 20. A system of Clause 19, wherein mesh encoding includes determining and storing the positions of projected vertices and the connectivity of projected vertices.
[0108] 21. The system of Clause 19, wherein triangle encoding includes encoding triangles not captured during surface projection.
[0109] 22. The system of Clause 21, wherein triangle encoding includes encoding triangles using only vertex position and color vertex information.
[0110] 23. The system of Clause 19, wherein triangle encoding includes determining that if the bounding box of a connected component is smaller than a predefined region, then unprojected triangles are moved to a separate list for independent triangle encoding, and the unprojected triangles are not rasterized, but are encoded as vertices with an associated color for each vertex; otherwise, each triangle is projected onto a patch, and if the projection position of a vertex is already occupied, then the triangle is encoded in another patch, and thus the triangle is placed in a list of missing triangles for later processing.
[0111] 24. The system of Clause 19, wherein triangle encoding uses triangular patch data units.
[0112] 25. The system of Clause 24, wherein the triangular patch data unit stores triangles as separate triangles, triangular stripes, or triangular sectors.
[0113] 26. The system of Clause 19, wherein the triangle encoding further includes a color extension implementation, wherein the color extension implementation writes the color of the triangle as c0 c1 c2; m c m c m c0 c1 c2, where c m It is the color value of the centroid of the triangle.
[0114] 27. The system of Clause 26, wherein triangle encoding also includes color compression.
[0115] The present invention has been described with reference to specific embodiments, which incorporate details that facilitate understanding of the construction and operating principles of the invention. Such reference to specific embodiments and their details herein is not intended to limit the scope of the appended claims. It will be apparent to those skilled in the art that various other modifications may be made to the embodiments chosen for illustration without departing from the spirit and scope of the invention as defined by the claims.
Claims
1. A method comprising: Perform mesh voxelization on the input mesh; Perform patch generation, which divides the mesh into patches that include surface meshes and unprojected triangles; Perform mesh encoding on the surface mesh; Perform triangle encoding on unprojected triangles; as well as V3C bitstream is generated based on lattice coding and triangle coding. The triangle encoding process involves determining that if the bounding box of a connected component is smaller than a predefined region, then unprojected triangles are moved to a separate list for independent triangle encoding, and the unprojected triangles are not rasterized, but are encoded as vertices with an associated color for each vertex. Otherwise, each triangle is projected onto a patch, and if the projection position of a vertex is already occupied, the triangle is encoded in another patch, and thus the triangle is placed in the list of missing triangles for later processing.
2. The method of claim 1, wherein mesh encoding includes determining and storing the positions of the projected vertices and the connectivity of the projected vertices.
3. The method of claim 1, wherein triangle encoding includes encoding triangles that were not captured during surface projection.
4. The method of claim 3, wherein triangle encoding includes encoding the triangle using only vertex position and color vertex information.
5. The method of claim 1, wherein the triangle encoding uses triangular patch data units.
6. The method of claim 5, wherein the triangular patch data unit stores the triangles as separate triangles, triangular strips, or triangular sectors.
7. The method of claim 1, wherein the triangle encoding further includes a color extension implementation, wherein the color extension implementation writes the color of the triangle as c0 c1 c2; c m c m c m c0 c1 c2, where c0, c1, and c2 are the color values of the three vertices of the triangle, and c is the color value of the vertex. m It is the color value of the centroid of the triangle, and c m It equals the average of c0, c1, and c2.
8. The method of claim 7, wherein the triangle encoding further includes color compression.
9. An apparatus comprising: Non-transitory memory for storing applications, which are used for: Perform mesh voxelization on the input mesh; Perform patch generation, which divides the mesh into patches that include surface meshes and unprojected triangles; Perform mesh encoding on the surface mesh; Perform triangle encoding on unprojected triangles; as well as V3C bitstream is generated based on grid coding and triangle coding; as well as A processor coupled to the memory, the processor being configured to process the application. The triangle encoding process involves determining that if the bounding box of a connected component is smaller than a predefined region, then unprojected triangles are moved to a separate list for independent triangle encoding, and the unprojected triangles are not rasterized, but are encoded as vertices with an associated color for each vertex. Otherwise, each triangle is projected onto a patch, and if the projection position of a vertex is already occupied, the triangle is encoded in another patch, and thus the triangle is placed in the list of missing triangles for later processing.
10. The apparatus of claim 9, wherein mesh encoding includes determining and storing the positions of the projected vertices and the connectivity of the projected vertices.
11. The apparatus of claim 9, wherein triangle encoding includes encoding triangles not captured during surface projection.
12. The apparatus of claim 11, wherein triangle encoding includes encoding the triangle using only vertex position and color vertex information.
13. The apparatus of claim 9, wherein the triangle encoding uses triangle patch data units.
14. The apparatus of claim 13, wherein the triangular patch data unit stores triangles as separate triangles, triangular strips, or triangular sectors.
15. The apparatus of claim 9, wherein the triangle encoding further includes a color extension embodiment, wherein the color extension embodiment writes the color of the triangle as c0 c1 c2; c m c m c m c0 c1 c2, where c0, c1, and c2 are the color values of the three vertices of the triangle, and c is the color value of the vertex. m It is the color value of the centroid of the triangle, and c m It equals the average of c0, c1, and c2.
16. The apparatus of claim 15, wherein the triangle encoding further includes color compression.
17. A system comprising: One or more cameras are used to acquire 3D content; An encoder is used to encode 3D content. Perform mesh voxelization on the input mesh of 3D content; Perform patch generation, which divides the mesh into patches that include surface meshes and unprojected triangles; Perform mesh encoding on the surface mesh; Perform triangle encoding on unprojected triangles; as well as V3C bitstream is generated based on lattice coding and triangle coding. The triangle encoding process involves determining that if the bounding box of a connected component is smaller than a predefined region, then unprojected triangles are moved to a separate list for independent triangle encoding, and the unprojected triangles are not rasterized, but are encoded as vertices with an associated color for each vertex. Otherwise, each triangle is projected onto a patch, and if the projection position of a vertex is already occupied, the triangle is encoded in another patch, and thus the triangle is placed in the list of missing triangles for later processing.
18. The system of claim 17, wherein mesh encoding includes determining and storing the positions of the projected vertices and the connectivity of the projected vertices.
19. The system of claim 17, wherein triangle encoding includes encoding triangles not captured during surface projection.
20. The system of claim 19, wherein triangle encoding includes encoding triangles using only vertex position and color vertex information.
21. The system of claim 17, wherein the triangle encoding uses triangle patch data units.
22. The system of claim 21, wherein the triangular patch data unit stores the triangles as separate triangles, triangular strips, or triangular sectors.
23. The system of claim 17, wherein the triangle encoding further includes a color extension implementation, wherein the color extension implementation writes the color of the triangle as c0 c1 c2; c m c m c m c0 c1 c2, where c0, c1, and c2 are the color values of the three vertices of the triangle, and c is the color value of the vertex. m It is the color value of the centroid of the triangle, and c m It equals the average of c0, c1, and c2.
24. The system of claim 23, wherein the triangle encoding further includes color compression.