Chart-Based Mesh Compression
The method addresses the inefficiencies in existing mesh compression standards by reconstructing two-dimensional meshes from three-dimensional meshes using adaptive sampling modes, enabling efficient compression and transmission of dynamic meshes for real-time applications.
Patent Information
- Application Number
- JP2024527694
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-03-28
- Filing Date
- 2023-04-06
- Publication Date
- 2024-11-21
- Estimated Expiration
- 2043-04-06
AI Technical Summary
Existing mesh compression standards do not effectively handle dynamic meshes with time-varying connectivity and attribute maps, especially under real-time constraints, leading to inefficient data storage and transmission.
A method for generating and reconstructing two-dimensional meshes from three-dimensional meshes using sampling and non-sampling modes, involving boundary vertex reconstruction and internal vertex determination based on a reconstruction mode, to efficiently compress and decompress dynamic meshes.
Enables efficient compression and transmission of dynamic meshes with time-varying connectivity and attribute maps, suitable for real-time applications such as AR and VR, by reducing data volume while maintaining high visual quality.
Smart Images

Figure 2024543440000021 
Figure 2024543440000022 
Figure 2024543440000023
Abstract
Description
[Technical field]
[0001] This application claims priority to U.S. Provisional Application No. 63 / 331,711, filed April 15, 2022, and U.S. Patent Application No. 18 / 127,432, filed March 28, 2023, the disclosures of which are incorporated by reference in their entireties herein.
[0002] FIELD OF THE DISCLOSURE This disclosure relates to a set of advanced video coding techniques, and more particularly, to video-based mesh compression, including coding methods for chart-based mesh compression. [Background technology]
[0003] Advanced three-dimensional (3D) representations of the world are enabling more immersive forms of interaction and communication. To achieve realism in the 3D representations, 3D models are becoming increasingly sophisticated, and a significant amount of data is associated with the creation and consumption of these 3D models. 3D meshes are widely used for 3D modeling of immersive content.
[0004] A 3D mesh may be composed of multiple polygons that describe the surface of a volumetric object. Dynamic mesh sequences may require large amounts of data since the mesh sequence may have a significant amount of information that changes over time. Therefore, efficient compression techniques are required to store and transmit such content.
[0005] Mesh compression standards IC, MESHGRID, and FAMC were previously developed to deal with dynamic meshes with constant connectivity and time-varying geometry and vertex attributes, but these standards do not consider time-varying attribute maps and connectivity information.
[0006] Additionally, it is also a challenge for volumetric acquisition techniques to generate dynamic meshes with consistent connectivity, especially under real-time constraints. This type of dynamic mesh content is not supported by existing standards. Summary of the Invention [Means for solving the problem]
[0007] According to one or more embodiments, a method executed by at least one processor of a decoder includes receiving a coded video bitstream including one or more two-dimensional (2D) meshes corresponding to a three-dimensional (3D) mesh. The method includes reconstructing one or more boundary vertices of each of the one or more 2D meshes. The method includes determining whether a reconstruction mode for reconstructing one or more interior vertices of each of the one or more 2D meshes is one of (i) a sampling mode and (ii) a non-sampling mode. The method includes reconstructing the one or more interior vertices of each of the one or more 2D meshes based on the determined reconstruction mode. The method includes reconstructing the 3D mesh based on the reconstructed one or more boundary vertices of each of the one or more 2D meshes and the reconstructed one or more interior vertices of each of the one or more 2D meshes.
[0008] According to one or more embodiments, a decoder comprises at least one memory configured to store program code and at least one processor configured to read the program code and to operate as directed by the program code. The program code includes a receiving code configured to cause the at least one processor to receive a coded video bitstream including one or more two-dimensional (2D) meshes corresponding to a three-dimensional (3D) mesh. The program code includes a first reconstruction code configured to cause the at least one processor to reconstruct one or more boundary vertices of each of the one or more 2D meshes. The program code includes a first decision code configured to cause the at least one processor to determine whether a reconstruction mode for reconstructing one or more interior vertices of each of the one or more 2D meshes is one of: (i) a sampling mode; and (ii) a non-sampling mode. The program code includes a second reconstruction code configured to cause the at least one processor to reconstruct the one or more interior vertices of each of the one or more 2D meshes based on the determined reconstruction mode. The program code includes third reconstruction code configured to cause the at least one processor to reconstruct the 3D mesh based on the one or more reconstructed boundary vertices of each of the one or more 2D meshes and the one or more reconstructed interior vertices of each of the one or more 2D meshes.
[0009] According to one or more embodiments, a non-transitory computer-readable medium having stored thereon instructions that, when executed by a processor in a decoder, cause the processor to perform the following steps: receive a coded video bitstream including one or more two-dimensional (2D) meshes corresponding to a three-dimensional (3D) mesh; reconstructing one or more boundary vertices of each of the one or more 2D meshes; determining whether a reconstruction mode for reconstructing one or more interior vertices of each of the one or more 2D meshes is one of: (i) a sampling mode; (ii) a non-sampling mode; reconstructing the one or more interior vertices of each of the one or more 2D meshes based on the determined reconstruction mode; and reconstructing the 3D mesh based on the reconstructed one or more boundary vertices of each of the one or more 2D meshes and the reconstructed one or more interior vertices of each of the one or more 2D meshes. [Brief description of the drawings]
[0010] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings.
[0011] [Figure 1] FIG. 1 is a schematic block diagram of a communication system according to an embodiment of the present disclosure. [Diagram 2] FIG. 1 is a schematic block diagram of a streaming system according to an embodiment of the present disclosure. [Diagram 3] FIG. 2 is a schematic block diagram of a video encoder and decoder according to an embodiment of the present disclosure. [Figure 4] 1 illustrates an example of UV parameterized mapping from a 3D mesh segment to a 2D chart, according to an embodiment of the present disclosure. [Diagram 5] Shows an example of different UV parameterization where a 3D mesh segment is mapped onto multiple separate charts. [Figure 6]1 illustrates an exemplary 2D UV atlas with multiple charts according to an embodiment of the present disclosure. [Figure 7] 1 illustrates an example of boundary vertices in a 2D patch, according to an embodiment of the present disclosure. [Figure 8] 1 illustrates an example geometry encoding pipeline according to an embodiment of the present disclosure. [Figure 9(A)-(B)] 1 illustrates an example of chart boundary detection according to an embodiment of the present disclosure. [Figure 10(A)-(B)] 1A-1C illustrate example geometry maps before and after filling unoccupied pixels, respectively, according to an embodiment of the present disclosure. [Figure 11] 1 illustrates an example geometry decoding pipeline according to an embodiment of the present disclosure. [Figure 12] Here is an example of finding interior sampling points inside a polygon defined by a boundary. [Figure 13] 1 illustrates an exemplary flowchart for encoding a video bitstream according to an embodiment of the present disclosure. [Figure 14] 1 illustrates an exemplary flowchart for decoding a coded video bitstream according to an embodiment of the present disclosure. [Figure 15] FIG. 1 illustrates a computer system suitable for implementing embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0012] The following detailed description of the exemplary embodiments refers to the accompanying drawings, in which the same reference numbers in different drawings can identify the same or similar elements, and in which:
[0013] The foregoing disclosure provides illustrations and descriptions, but is not intended to be exhaustive or to limit the implementation to the precise form disclosed. Modifications and variations are possible in light of the above disclosure or may be acquired from practice of the implementations. Furthermore, one or more features or components of one embodiment may be incorporated into or combined with another embodiment (or one or more features of another embodiment). In addition, in the flowcharts and descriptions of operations provided below, it is understood that one or more operations may be omitted, one or more operations may be added, one or more operations may be performed (at least partially) simultaneously, and one or more operations may be reordered.
[0014] It will be apparent that the systems and / or methods described herein may be implemented in different forms of hardware, firmware, or a combination of hardware and software. The actual specific control hardware or software code used to implement these systems and / or methods is not intended to limit the implementation. Thus, the operations and behavior of the systems and / or methods are described herein without reference to any specific software code. It will be understood that software and hardware may be designed to implement the systems and / or methods based on the description herein.
[0015] Although particular combinations of features are recited in the claims and / or disclosed herein, these combinations are not intended to limit the disclosure of possible implementations. Indeed, many of these features can be combined in ways not specifically recited in the claims and / or disclosed herein. Although each dependent claim listed below may depend directly on only one claim, the disclosure of possible implementations includes each dependent claim in combination with all other claims in the claim set.
[0016] No element, act, or instruction used herein is construed as critical or essential unless expressly described as such. Also, as used herein, the articles "a" and "an" are intended to include one or more items and may be used interchangeably with "one or more." Where only one item is intended, the term "one" or similar language is used. Also, as used herein, the terms "have," "having," "include," "including," and the like are intended to be open-ended terms. Furthermore, the phrase "based on" is intended to mean "based at least in part on," unless otherwise noted. Furthermore, phrases such as "at least one of [A] and [B]" or "at least one of [A] or [B]" shall be understood to include only A, only B, or both A and B.
[0017] Throughout this specification, references to "one embodiment," "an embodiment," or similar language mean that a particular feature, structure, or characteristic described in connection with the illustrated embodiment is included in at least one embodiment of the solution. Thus, throughout this specification, "in one embodiment," "in an embodiment," and similar language may, but do not necessarily, all refer to the same embodiment.
[0018] Furthermore, the described features, advantages, and characteristics of the present disclosure may be combined in any suitable manner in one or more embodiments. In light of the description herein, one skilled in the art will recognize that the present disclosure may be practiced without one or more specific features or advantages of a particular embodiment. In other instances, a particular embodiment may recognize additional features and advantages that may not be present in all embodiments of the present disclosure.
[0019] SUMMARY OF THE DISCLOSURE Embodiments of the present disclosure relate to compressing meshes. A mesh may be composed of a number of polygons that describe the surface of a volumetric object. Each polygon may be defined by its vertices in 3D space and information on how the vertices are connected, called connectivity information. Optionally, vertex attributes such as color, normal, etc. may be associated with the mesh vertices. Attributes may also be associated with the surface of the mesh by utilizing mapping information that parameterizes the mesh in a 2D attribute map. Such mapping may be defined using a set of parametric coordinates, called UV coordinates or texture coordinates, associated with the mesh vertices. The 2D attribute map may be used to store high-resolution attribute information such as texture, normal, displacement, etc. The high-resolution attribute information may be used for various purposes such as texture mapping and shading.
[0020] As mentioned above, a 3D mesh or dynamic mesh may require a large amount of data since it may consist of a significant amount of information that changes over time. Existing standards do not consider time-varying attribute maps and connectivity information. Existing standards also do not support volumetric acquisition techniques that generate dynamic meshes with constant connectivity, especially under real-time conditions.
[0021] Therefore, new mesh compression standards are needed to directly handle dynamic meshes that have time-varying connectivity information and may optionally have time-varying attribute maps. Embodiments of the present disclosure enable efficient compression techniques for storing and transmitting such dynamic meshes. Embodiments of the present disclosure enable lossy and / or lossless compression for a variety of applications such as real-time communication, storage, free viewpoint video, AR and VR, etc.
[0022] According to one or more embodiments of the present disclosure, a method, system, and non-transitory storage medium for dynamic mesh compression are provided. The embodiments of the present disclosure can also be applied to static meshes, where only one frame of the mesh or the mesh content does not change over time.
[0023] One or more embodiments of the present disclosure for implementing the encoding and decoding structures of the present disclosure will be described with reference to FIGS.
[0024] FIG. 1 shows a schematic block diagram of a communication system 100 according to an embodiment of the present disclosure. The system 100 may include at least two terminals 110, 120 interconnected via a network 150. In the case of unidirectional transmission of data, the first terminal 110 may code video data, which may include mesh data, at a local location for transmission to the other terminal 120 via the network 150. The second terminal 120 may receive the coded video data of the other terminal from the network 150, decode the coded data, and display the restored video data. The unidirectional data transmission may be common in media serving applications, etc.
[0025] 1 shows a second pair of terminals 130, 140 provided to support two-way transmission of coded video that may occur, for example, during a video conference. For two-way transmission of data, each terminal 130, 140 can code video data captured at a local location for transmission to the other terminal over network 150. Each terminal 130, 140 can also receive coded video data transmitted by the other terminal, can decode the coded data, and can display the recovered video data on a local display device.
[0026] In FIG. 1, terminals 110-140 may be, for example, servers, personal computers, smartphones, and / or any other type of terminal. For example, terminals (110-140) may be laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. Network 150 represents any number of networks that convey coded video data between terminals 110-140, including, for example, wired and / or wireless communication networks. Communications network 150 may exchange data over circuit-switched and / or packet-switched channels. Exemplary networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of this discussion, the architecture and topology of network 150 may not be important to the operation of the present disclosure unless otherwise described herein below.
[0027] 2 shows an arrangement of video encoders and decoders in a streaming environment as an example of an application of the disclosed subject matter. The disclosed subject matter can be used with other video-enabled applications including, for example, video conferencing, digital TV, and storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.
[0028] 2, the streaming system 200 may include a capture subsystem 213 that includes a video source 201 and an encoder 203. The streaming system 200 may further include at least one streaming server 205 and / or at least one streaming client 206.
[0029] The video source 201 may create a stream 202 that includes, for example, a 3D mesh and metadata associated with the 3D mesh. The 3D mesh may be composed of a number of polygons that describe the surface of a volumetric object. For example, the 3D mesh may include a number of vertices in a 3D space, where each vertex is associated with a 3D coordinate (e.g., x, y, z). The video source 201 may include, for example, a 3D sensor (e.g., a depth sensor) or a 3D imaging technique (e.g., a digital camera) and a computing device configured to generate the 3D mesh using data received from the 3D sensor or the 3D imaging technique. The sample stream 202, which may have a high amount of data compared to an encoded video bitstream, may be processed by an encoder 203 coupled to the video source 201. The encoder 203 may include hardware, software, or a combination thereof to enable or realize aspects of the disclosed subject matter, as described in more detail below. The encoder 203 may also generate an encoded video bitstream 204. The encoded video bitstream 204, which may have a lower amount of data compared to the uncompressed stream 202, may be stored on the streaming server 205 for future use. One or more streaming clients 206 may access the streaming server 205 to retrieve a video bitstream 209, which may be a copy of the encoded video bitstream 204.
[0030] The streaming client 206 may include a video decoder 210 and a display 212. The video decoder 210 may, for example, decode a video bitstream 209, which is an inbound copy of the encoded video bitstream 204, and create an outbound video sample stream 211 that may be rendered on a display 212 or another rendering device (not shown). In some streaming systems, the video bitstreams 204, 209 may be encoded according to a particular video coding / compression standard.
[0031] FIG. 3 is an example diagram of a framework 300 for dynamic mesh compression and mesh reconstruction using an encoder and a decoder.
[0032] 3, framework 300 may include an encoder 301 and a decoder 351. Encoder 301 may include one or more input meshes 305, one or more meshes with UV atlas 310, occupancy map 315, geometry map 320, attribute map 325, and metadata 330. Decoder 351 may include a decoded occupancy map 335, a decoded geometry map 340, a decoded attribute map 345, decoded metadata 350, and a reconstructed mesh 360.
[0033] According to one or more embodiments of the present disclosure, the input mesh 305 may include one or more frames, each of which may be pre-processed by a series of operations and used to generate a mesh with a UV atlas 310. By way of example, the pre-processing operations may include, but are not limited to, tracking, parameterization, remeshing, voxelization, etc. In some embodiments, the pre-processing operations may only be performed on the encoder side and not on the decoder side.
[0034] The mesh with UV atlas 310 may be a 2D mesh. The 2D mesh may be a chart of vertices, each associated with a coordinate in 2D space (e.g., 2D coordinates). Each vertex in the 2D mesh may be associated with a corresponding vertex in a 3D mesh, where the vertices in the 3D mesh are associated with coordinates in 3D space. The compressed 2D mesh may be a version of the 2D mesh with reduced information compared to the uncompressed 2D mesh. For example, the 2D mesh may be sampled at a sampling rate that includes the points at which the compressed 2D mesh was sampled. The 2D mesh with UV atlas may be a mesh where each vertex of the mesh may be associated with a UV coordinate on the 2D atlas. The mesh with UV atlas 310 may be processed and converted into multiple maps based on sampling. As an example, the UV atlas 310 may be processed and converted into an occupancy map, a geometry map, and an attribute map based on sampling of the 2D mesh with UV atlas. The generated occupancy map 335, geometry map 340, and attribute map 345 may be encoded using an appropriate codec (e.g., HVEC, VVC, AV1, AVS3, etc.) and transmitted to a decoder. In some embodiments, metadata (e.g., connectivity information, etc.) may also be transmitted to the decoder.
[0035] In some embodiments, the decoder side can reconstruct the mesh from the decoded 2D map. It can also apply post-processing and filtering to the reconstructed mesh. In some examples, metadata may be signaled to the decoder side for the purpose of 3D mesh reconstruction. If the boundary vertices of each patch are signaled, the occupancy map can be inferred from the decoder side.
[0036] According to an aspect, the decoder 351 may receive the encoded occupancy map, geometry map, and attribute map from the encoder. The decoder 315 may decode the occupancy map, geometry map, and attribute map using suitable techniques and methods in addition to the embodiments described herein. In some embodiments, the decoder 351 may generate the decoded occupancy map 335, the decoded geometry map 340, the decoded attribute map 345, and the decoded metadata 350. The input mesh 305 may be reconstructed into the reconstructed mesh 360 based on the decoded occupancy map 335, the decoded geometry map 340, the decoded attribute map 345, and the decoded metadata 350 using one or more reconstruction filters and techniques. In some embodiments, the metadata 330 may be sent directly to the decoder 351, and the decoder 351 may generate the reconstructed mesh 360 using the metadata based on the decoded occupancy map 335, the decoded geometry map 340, and the decoded attribute map 345. Post-filtering techniques, including but not limited to remeshing, parameterization, tracking, voxelization, etc., may also be applied to the reconstructed mesh 360.
[0037] According to some embodiments, a 3D mesh can be partitioned into a number of segments (or patches / charts). Each segment can consist of a set of connected vertices associated with geometry, attributes, and connectivity information. As shown in FIG. 4, the UV parameterization process maps a mesh segment 400 to a 2D chart (402, 404) in a 2D UV atlas. Each vertex in a mesh segment can be assigned a 2D UV coordinate in the 2D UV atlas. The vertices in the 2D chart (e.g., a 2D mesh) can form connected components as their 3D counterparts. The geometry, attributes, and connectivity information of each vertex can be inherited from their 3D counterparts as well.
[0038] According to some embodiments, a 3D mesh segment can also be mapped to multiple separate 2D charts. When a 3D mesh segment is mapped to separate 2D charts, a vertex in the 3D mesh segment may correspond to multiple vertices in the 2D UV atlas. As shown in FIG. 5, a 3D mesh segment 500 that may correspond to the 3D mesh segment 400 may be mapped to two 2D charts (502A, 502B) in the 2D UV atlas instead of a single chart. As shown in FIG. 5, the 3D vertices v1 and v4 have two 2D corresponding vertices v1' and v4', respectively.
[0039] FIG. 6 shows an example of a general 2D UV atlas 600 for a 3D mesh that includes multiple charts, where each chart may include multiple (e.g., three or more) vertices associated with their 3D geometry, attributes, and connectivity information.
[0040] Boundary vertices may be defined in 2D UV space. As shown in Figure 7, a filled vertex is a boundary vertex because it is on the boundary edge of a connected component (patch / chart). A boundary edge may be determined by checking if the edge appears in only one triangle. Geometry information (e.g., 3D xyz coordinates) and 2D UV coordinates may be signaled in the bitstream.
[0041] Dynamic mesh sequences can require large amounts of data since they can consist of a significant amount of information that changes over time, and therefore efficient compression techniques are needed to store and transmit such content.
[0042] The embodiments of the present disclosure may be used individually or combined in any order. Furthermore, each of the method (or embodiment), the encoder, and the decoder may be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors execute a program stored on a non-transitory computer-readable medium.
[0043] 8 illustrates an example geometry encoding pipeline 800 according to one or more embodiments. The encoding pipeline 800 may be performed by the encoder 301. The geometry encoding pipeline may include partitioning (802) a mesh into connected components (e.g., charts or 2D meshes). After partitioning, chart boundary extraction (804) and chart classification (806) may occur.
[0044] In one or more examples, each chart can be coded by either a raw chart mode or a sampling mode based on the chart statistics. In the raw chart mode, the interior vertices can be directly coded (808) by prediction. In the sampling mode, the chart is adaptively sampled (810) in 2D UV space and packed (812) into a 2D geometry image atlas, which is then coded (818). Adaptive sampling (810) may be applied to allow for different charts with different sampling rates. The geometry image is then coded (818) by a video codec. Boundary vertices are extracted and coded (816) by prediction, quantization, and entropy coding. In one or more examples, the texture map may be updated by a recoloring algorithm to transfer colors from the original texture map to a new texture map based on the reconstructed geometry.
[0045] According to one or more embodiments, for each chart, boundary vertices can be detected. A boundary vertex may be a vertex that is on a boundary edge, where a boundary edge is an edge associated with a single triangle. As shown in FIG. 9(A) and FIG. 9(B), there are two kinds of boundaries (e.g., exterior and interior boundaries). FIG. 9(A) and FIG. 9(B) show an example of chart boundary detection. The bold vertices and edges may be on the exterior boundary, the dashed vertices and edges are on the interior boundary, and the other vertices and edges do not belong to any boundary. In FIG. 9(A), there are no holes with one exterior boundary, and in FIG. 9(B), there is one hole with one exterior boundary and one interior boundary.
[0046] For each chart, there is one exterior boundary loop (e.g., the solid boundary in FIGS. 9(A) and 9(B)). The exterior boundary loop may be defined by vertices B0, B1, B2, B3, B4, B5, B6, and B7. These vertices may be referred to as boundary vertices. As shown in FIGS. 9(A) and 9(B), the edges defined by the boundary vertices appear in a single triangle. For example, the edge defined between vertices B0 and B1 appears in a single triangle. In one or more examples, the exterior boundary loop may define an exterior polygon of the chart, and all other vertices are inside this polygon. In one or more examples, there may be an interior boundary loop inside the exterior boundary loop (e.g., the dashed line in FIG. 9(B)). An interior boundary loop may be a loop defined by vertices that do not define a triangle. For example, the configuration of FIG. 9(A) does not include any interior boundary loops, while the configuration of FIG. 9(B) includes at least one interior boundary loop (e.g., the dashed line). As shown in Figure 9(B), the dashed lines do not define triangles. An interior bounded loop may correspond to a hole in the chart. In one or more examples, a vertex that does not belong to any exterior or interior bounded loop may be identified as an interior vertex.
[0047] In some embodiments, the encoder reparameterization algorithm can eliminate holes inside the chart. By eliminating the holes on the encoder side, inner boundary loops can be eliminated from the chart (e.g., 2D mesh).
[0048] According to one or more embodiments, each chart can be coded by different methods based on the characteristics of the chart and / or one or more application scenarios. In some embodiments, two coding methods can be used. The first method can be a sampling-based method, where the chart can be sampled at a predefined sampling rate (e.g., step size) and the generated geometry and attribute maps can be coded by a video codec. The second method can be a raw chart coding mode, where the interior vertices and connectivity between vertices can be directly coded without sampling.
[0049] According to one or more embodiments, a control flag can be signaled first in the high level syntax to indicate whether adaptive sampling is applied or not. If the flag is true, adaptive sampling can be enabled so that different sampling rates can be applied to each chart. If the flag is false, adaptive sampling can be disabled so that the same sampling rate can be applied to all charts.
[0050] In one or more examples, a base sampling rate (BS) may be signaled in the high level syntax whether adaptive sampling is enabled or not. The base sampling rate may be used in one or more of the following aspects: If adaptive sampling is enabled, the base sampling rate may be used as a predictor. Each chart may signal a sampling rate by a difference from the base sampling rate (e.g., S-BS). If adaptive sampling is disabled, the base sampling rate may be used as a constant sampling rate for all charts. The base sampling rate may be used as a quantization step in coding of border UV coordinates in charts, including raw charts, allowing for more efficient coding of border information.
[0051] A UV coordinate offset from the original sampling position may be signaled on a per-chart basis. u ,offset v ) can be used to find the sampled vertex in the packed geometry image. This allows for different alignment of the texture and geometry images.
[0052] According to one or more embodiments, the geometry map may be encoded by one or more video codecs. To improve coding efficiency of the geometry map, unoccupied pixels may be filled with neighboring occupied pixels in the geometry map, as shown in FIG. 10(A) and FIG. 10(B). FIG. 10(A) shows an example geometry map before filling unoccupied pixels, and FIG. 10(B) shows an example geometry map after filling unoccupied pixels. In one or more examples, occupancy information is not encoded in the bitstream, but can instead be inferred from the bound vertices. At the decoder side, the occupancy map can be subtracted from the UV coordinates of the bound vertices, and the occupied pixel values in the geometry map can be used for mesh reconstruction.
[0053] Border information, including UV and XYZ coordinates of border vertices, may be predicted, quantized, and entropy coded in the bitstream. The quantization step size may be set at the encoder side for quality vs. bitrate tradeoff. Border vertices with different UV coordinates may have the same XYZ coordinates since they are from the same 3D point. To ensure that the mesh reconstructed by the decoder is accurate across the 3D border edges, a 3D to 2D mapping function is maintained; otherwise, the 3D reconstruction may have holes and cracks that can significantly degrade the visual quality of the reconstructed 3D mesh. Therefore, a UV to XYZ (e.g., called UV2XYZ) index array may be signaled to indicate the mapping function of the border vertices. The UV2XYZ index array may be a 1D array of indices that maps each 2D UV vertex to a 3D XYZ vertex.
[0054] In one or more examples, for each chart, the number of boundary loops in the chart may be signaled, and for each boundary loop, the number of vertices in the boundary loop may be signaled. The outer boundary loops may be signaled before other inner boundary loops. The following information may be signaled to convey the boundary information: (i) the 2D UV coordinates of each chart, (ii) the UV2XYZ index array of each chart, and (iii) the unique 3D XYZ coordinates. Duplicate 3D coordinates may be removed, but the correspondence may be kept in the UV2XYZ index array. The unique boundary XYZ coordinates may be coded once and shared by all charts. Since the encoder side can guarantee that each chart can have only one outer boundary loop, the syntax table may be further simplified by skipping signaling the number of boundary loops of each chart. The UV2XYZ coding maintains the uniqueness of boundary xyz vertices when they have multiple correspondences in different charts, ensuring that the reconstructed geometry is accurate across the entire boundary without holes or cracks.
[0055] According to one or more embodiments, the boundary XYZ vertex information can be predicted from either the reconstructed geometry image (e.g., in the case of a sampled chart) or from previously reconstructed boundary XYZ vertices (e.g., in the case of a raw chart). The prediction residual for the boundary XYZ vertices can be derived by subtracting the original geometry from the predicted values as follows:
number
[0056] JPEG2024543440000002.jpg19170
[0057] The prediction residual may be quantized and entropy coded. For example, the residual may be quantized with a scalar value as follows:
number
[0058] Here, QS xyz is the quantization step size, which can be signaled in the high-level syntax.
[0059] The reconstructed boundary XYZ vertices may be derived as follows.
number
[0060] A prediction for the boundary XYZ vertices may be determined as follows: For boundary XYZ vertices that have a correspondence in at least one sampled chart, a prediction may be derived from the reconstructed geometry image as follows.
number
[0061] Here, Set(i) is the set of all indices of bounding UV vertices that correspond to the same bounding XYZ vertex and index i.
number
[0062] JPEG2024543440000007.jpg17170
number
[0063] For bounds XYZ vertices whose all corresponding UV coordinates belong to the raw chart, a prediction may be derived from the previously reconstructed bounds XYZ vertices as follows:
number
[0064] According to some embodiments, the raw chart coding mode directly encodes interior vertices without sampling them. In one or more examples, the interior vertices of the raw chart may be treated as boundary vertices and may be concatenated with the boundary vertices, where the same coding methods as described above may be applied. In one or more examples, the interior vertices of the raw chart may be separated from the boundary vertices. For example, for each raw chart, the boundary vertices may be coded first without the interior vertices, and then the interior vertices may be coded. The interior vertices may be subjected to the same prediction, quantization, and entropy coding scheme as the boundary vertices. In one or more examples, only the UV and XYZ coordinates need to be coded, and no UV2XYZ index array is required for the interior vertices. Both the number of boundary vertices and the number of interior vertices may be signaled for each raw chart.
[0065] FIG. 11 illustrates an exemplary geometry decoding pipeline 1100 according to one or more embodiments. The decoding pipeline 1100 may be performed by the decoder 351. Boundary vertices may be decoded (1102) and reconstructed (1106) from the boundary information substream. A geometry image may be decoded (1104) from the video substream. For each chart, the reconstruction of interior vertices differs in terms of coding mode (e.g., raw chart mode, sampling mode). If raw chart mode is used, the interior vertices are decoded (1108) directly from the boundary information substream. If a sampling mode is used in which the interior vertices are sampled vertices in the polygon defined by the boundary vertices, the sampled vertices are reconstructed (1110). After reconstruction, triangulation (1112) may be applied to each chart to derive connectivity information between the reconstructed boundary vertices and the interior vertices, and a reconstructed mesh (1114) may be obtained. In one or more examples, a mesh filtering process (1116) may be applied to further smooth the geometry, such as on boundaries, to improve reconstruction quality.
[0066] In one or more examples, to find the correct correspondence, the decoded UV2XYZ index array may be used to reconstruct the bounds XYZ vertices. For example, for each chart, the decoded bounds UV coordinates may be derived by:
number
[0067] JPEG2024543440000011.jpg13170
[0068] In one or more examples, i=0,1,...,N j -1, where N j is the number of border vertices of the jth chart. The decoded border XYZ coordinates of a chart can be derived by:
number
[0069] JPEG2024543440000013.jpg16170
[0070] According to some embodiments, for charts coded by a sampling-based method, interior vertices can be reconstructed by the decoded bounds vertices and geometry images. For example, first, the reconstructed bounds uv coordinates may define the polygon of the chart, and then all sampling points inside the bounding box of the chart are checked to see if they are inside the polygon. If they are inside the polygon, the points are identified as interior sampling points. FIG. 12 shows an example of a bounds polygon (e.g., a connected point), where the interior sampling points are inside the bounds polygon. In one or more examples, a "cross count" algorithm can be used to check if a point is inside a polygon. Other suitable algorithms known to those skilled in the art may be used to check if a point is inside a polygon. For example, a polygon defined by the reconstructed bounds uv coordinates is shown as follows:
number
[0071] The interior sampling UV coordinates may be expressed as follows:
number
[0072] where i=0,1,…,K j -1, and K j is the number of interior vertices in the jth chart.
[0073] JPEG2024543440000016.jpg24170
number
[0074] JPEG2024543440000018.jpg21170
[0075] JPEG2024543440000019.jpg44170
[0076] JPEG2024543440000020.jpg69170
[0077] Figure 13 shows an example process 1300 performed by an encoder, such as encoder 301, for compressing a 3D mesh and generating a coded video bitstream. The process may begin with operation S1302, as shown in Figure 5, and further in accordance with operation 802 (Figure 8), converting the 3D mesh into one or more 2D meshes. The process proceeds to operation S1304, where boundary extraction may be performed to identify boundary vertices in the one or more 2D meshes. For example, the boundary extraction may be performed in accordance with operation 804 (Figure 8).
[0078] The process proceeds to operation S1306, which performs chart classification, e.g., according to operation 806. The process proceeds to operation S1308, which performs coding of interior vertices. For example, interior vertices of one or more 2D meshes may be encoded according to a raw chart mode (operation 808) or a sampling mode (operations 810, 812, 818). The process proceeds to operation S1310, which performs coding of boundary vertices, e.g., according to operation 816. The process proceeds to operation S1312, which generates a coded video bitstream including the encoded mesh information and any other signaling information, e.g., coding mode of the interior vertices.
[0079] 14 shows an example process 1400 performed by a decoder, such as decoder 351, for decoding a video bitstream and reconstructing a 3D mesh. The process may begin with operation S1402, in which a coded video bitstream is received. The coded video bitstream may correspond to the bitstream generated in operation S1312 (FIG. 13). The coded video bitstream may include one or more 2D meshes that correspond to the 3D mesh.
[0080] The process proceeds to operation S1404, where one or more boundary vertices in each 2D mesh are reconstructed. The reconstruction of the boundary vertices may be performed according to operation 1106 (FIG. 11). The process proceeds to operation S1406, where a reconstruction mode for reconstructing interior vertices of the one or more 2D meshes is determined. For example, based on a flag included in the coded video bitstream, it may be determined whether to use a raw chart mode or a sampling mode for decoding the interior vertices.
[0081] The process proceeds to operation S1408, which reconstructs interior vertices of the one or more 2D meshes based on the determined reconstruction mode. The process proceeds to operation S1410, which reconstructs a 3D mesh based on the reconstructed boundary vertices of the one or more 2D meshes and the interior vertices of the one or more 2D meshes. The reconstruction performed in operation S1410 may include triangulation operation 1112, mesh reconstruction operation 1114, and mesh filtering operation 1116 of FIG.
[0082] The techniques described above may be implemented as computer software using computer readable instructions and physically stored on one or more computer readable media. For example, Figure 15 illustrates a computer system 1500 suitable for implementing certain embodiments of the present disclosure.
[0083] The computer software may be coded using any suitable machine code or computer language that may be assembled, compiled, linked, or similar mechanisms to create code containing instructions that may be executed by a computer central processing unit (CPU), graphics processing unit (GPU), etc., directly, or via interpretation, microcode execution, etc.
[0084] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, Internet of Things devices, and the like.
[0085] 15 for computer system 1500 are exemplary in nature and are not intended to suggest any limitations on the scope of use or functionality of the computer software implementing the embodiments of the present disclosure. The arrangement of components should not be interpreted as having any dependency or requirement regarding any one or combination of components illustrated in the non-limiting embodiment of computer system 1500.
[0086] The computer system 1500 may include certain human interface input devices. Such human interface input devices may respond to input by one or more human users, for example, via tactile input (e.g., keystrokes, swipes, data glove movements, etc.), audio input (e.g., voice, clapping, etc.), visual input (e.g., gestures, etc.), and olfactory input (not shown). Human interface devices may also be used to capture certain media that are not necessarily directly associated with conscious human input, such as audio (e.g., voice, music, ambient sounds, etc.), images (e.g., scanned images, photographic images obtained from a still camera, etc.), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic vision, etc.).
[0087] The input human interface devices may include one or more of a keyboard 1501, a mouse 1502, a trackpad 1503, a touch screen 1510, a data glove, a joystick 1505, a microphone 1506, a scanner 1507, and a camera 1508 (only one of each is shown).
[0088] The computer system 1500 may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the human user's senses, for example, through haptic output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touch screen 1510, data gloves, or joystick 1505, although there may also be haptic feedback devices that do not function as input devices). For example, such devices may be audio output devices (e.g., speakers 1509, headphones (not shown), etc.), visual output devices (e.g., screens 1510 including CRT screens, LCD screens, plasma screens, OLED screens, etc., each with or without touch screen input capabilities, each with or without haptic feedback capabilities, some of which may output two-dimensional visual output or output in more than three dimensions, such as through means of stereographic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).
[0089] The computer system 1500 may also include human accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW 1520 with media 1521 such as CDs / DVDs, thumb drives 1522, removable hard drives or solid state drives 1523, legacy magnetic media such as tapes or floppy disks (not shown), specialized ROM / ASIC / PLD based devices (not shown) such as security dongles, etc.
[0090] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other transitory signals.
[0091] The computer system 1500 may also include interfaces to one or more communication networks. The networks may be, for example, wireless, wired, optical. The networks may further be local, wide area, metropolitan, vehicular and industrial, real-time, delay tolerant, etc. Examples of networks include local area networks such as Ethernet, cellular networks including WLAN, GSM, 3G, 4G, 5G, LTE, etc., TV wired or wireless wide area digital networks including cable TV, satellite TV, terrestrial broadcast TV, vehicular and industrial including CANBus, etc. A particular network typically requires an external network interface adapter connected to a particular general-purpose data port or peripheral bus 1549 (e.g., a USB port on computer system 1500; others are commonly integrated into the core of computer system 1500 by connection to a system bus as described below (e.g., an Ethernet interface to a PC computer system, or a cellular network interface to a smartphone computer system). Using any of these networks, computer system 1500 can communicate with other entities. Such communications can be unidirectional, receive only (e.g., broadcast TV), unidirectional transmit only (e.g., CANBus to a particular CANBus device), or bidirectional, e.g., to other computer systems using local or wide area digital networks. Such communications can include communications to a cloud computing environment 1555. As discussed above, particular protocols and protocol stacks can be used with each of those networks and network interfaces.
[0092] The aforementioned human interface devices, human accessible storage devices, and network interface 1554 can be connected to the core 1540 of the computer system 1500 .
[0093] The core 1540 may include one or more central processing units (CPUs) 1541, graphics processing units (GPUs) 1542, specialized programmable processing units in the form of field programmable gate areas (FPGAs) 1543, hardware accelerators for specific tasks 1544, etc. These devices may be connected via a system bus 1548 along with a read only memory (ROM) 1545, a random access memory 1546, and internal mass storage 1547, e.g. an internal hard disk, SSD, etc., that is not accessible to the user. In some computer systems, the system bus 1548 is accessible in the form of one or more physical plugs, allowing expansion with additional CPUs, GPUs, etc. Peripheral devices may be directly connected to the core's system bus 1548 or may be connected via a peripheral bus 1549. Peripheral bus architectures include PCI, USB, etc. A graphics adapter 1550 may be included in the core 1540.
[0094] The CPU 1541, GPU 1542, FPGA 1543, and accelerator 1544 may combine to execute certain instructions that may constitute the aforementioned computer code. That computer code may be stored in ROM 1545 or RAM 1546. Transient data may also be stored in RAM 1546, while permanent data may be stored in, for example, internal mass storage 1547. The use of cache memory, which may be closely associated with one or more of the CPU 1541, GPU 1542, mass storage 1547, ROM 1545, RAM 1546, etc., may enable fast storage and retrieval from any memory device.
[0095] The computer-readable medium may have computer code thereon for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the kind well known and available to those skilled in the art of computer software.
[0096] By way of example, and not by way of limitation, a computer system having the architecture 1500, and in particular the core 1540, may provide functionality as a result of the processor (including CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media may be media associated with user-accessible mass storage as described above, as well as specific storage of the core 1540 of a non-transitory nature, such as the core internal mass storage 1547 or ROM 1545. Software implementing various embodiments of the present disclosure may be stored in such devices and executed by the core 1540. The computer-readable media may include one or more memory devices or chips, depending on the particular needs. The software may enable the core 1540, and in particular the processor therein (including CPU, GPU, FPGA, etc.), to perform certain operations, or certain portions of certain operations, described herein, including the definition of data structures stored in RAM 1546 and the modification of such data structures by operations defined by the software. Additionally, or alternatively, a computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator 1544) that may work in place of, or in conjunction with, software to perform certain processes, or certain portions of certain processes, described herein. References to software may include logic, and vice versa, where appropriate. Where appropriate, references to computer-readable media may encompass circuitry (e.g., integrated circuits (ICs), etc.) that stores software for execution, circuitry that embodies logic for execution, or both. The present disclosure encompasses any suitable combination of hardware and software.
[0097] While this disclosure has described several non-limiting embodiments, there are modifications, permutations, and various substitute equivalents that are within the scope of this disclosure. Thus, it will be appreciated that those skilled in the art will be able to invent numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within its spirit and scope.
[0098] The above disclosure also encompasses the embodiments listed below.
[0099] (1) A method executed by at least one processor of a decoder, comprising: receiving a coded video bitstream including one or more two-dimensional (2D) meshes corresponding to a three-dimensional (3D) mesh; reconstructing one or more boundary vertices of each of the one or more 2D meshes; determining whether a reconstruction mode for reconstructing one or more interior vertices of each of the one or more 2D meshes is one of: (i) a sampling mode; and (ii) a non-sampling mode; reconstructing the one or more interior vertices of each of the one or more 2D meshes based on the determined reconstruction mode; and reconstructing the 3D mesh based on the reconstructed one or more boundary vertices of each of the one or more 2D meshes and the reconstructed one or more interior vertices of each of the one or more 2D meshes.
[0100] (2) The method of any one of (1) to (10), wherein the determination of the reconstruction mode is based on a flag included in the coded video bitstream.
[0101] (3) The method of any one of (1) to (2), further comprising the step of decoding one or more geometry maps included in the coded video bitstream.
[0102] (4) The method according to feature (3), wherein the step of reconstructing the one or more interior vertices of each of the one or more 2D meshes based on determining that the reconstruction mode is the sampling mode includes the steps of: determining a boundary polygon of each of the one or more 2D meshes based on corresponding reconstructed boundary vertices of each of the one or more 2D meshes; and identifying each vertex included in the boundary polygon of each of the one or more 2D meshes as an interior vertex.
[0103] (5) The method according to feature (4), wherein the step of reconstructing the 3D mesh includes a step of deriving interior vertices in the 3D mesh based on corresponding positions in the one or more decoded geometry maps, a sampling rate, and offsets in the one or more decoded geometry maps.
[0104] (6) The method of any one of features (1) to (5), wherein based on determining that the reconstruction mode is the non-sampling mode, the one or more interior vertices of each of the one or more 2D meshes are reconstructed based on the one or more reconstructed boundary vertices of each of the one or more 2D meshes.
[0105] (7) The method according to feature (6), wherein the step of reconstructing the 3D mesh includes the step of reconstructing interior vertices in the 3D mesh by concatenating reconstructed interior vertices of the one or more 2D meshes with corresponding reconstructed boundary vertices in the one or more 2D meshes.
[0106] (8) The method according to any one of features (1) to (7), wherein the step of reconstructing the 3D mesh includes a step of reconstructing boundary vertices in the 3D mesh based on a 2D to 3D index array that maps one or more corresponding reconstructed boundary vertices of each of the one or more 2D meshes to the boundary vertices in the 3D mesh.
[0107] (9) A method for implementing a video bitstream comprising: at least one memory configured to store program code; and at least one processor configured to read the program code and to operate as directed by the program code, the program code comprising: a receiving code configured to cause the at least one processor to receive a coded video bitstream including one or more two-dimensional (2D) meshes corresponding to a three-dimensional (3D) mesh; a first reconstruction code configured to cause the at least one processor to reconstruct one or more boundary vertices of each of the one or more 2D meshes; and a second reconstruction code configured to cause the at least one processor to reconstruct one or more boundary vertices of each of the one or more 2D meshes. a first decision code configured to cause the at least one processor to determine whether a reconstruction mode for reconstructing interior vertices of each of the one or more 2D meshes is one of (i) a sampling mode and (ii) a non-sampling mode; a second reconstruction code configured to cause the at least one processor to reconstruct the one or more interior vertices of each of the one or more 2D meshes based on the determined reconstruction mode; and a third reconstruction code configured to cause the at least one processor to reconstruct the 3D mesh based on the reconstructed one or more boundary vertices of each of the one or more 2D meshes and the reconstructed one or more interior vertices of each of the one or more 2D meshes.
[0108] (10) The decoder of feature (9), wherein the determination of the reconstruction mode is based on a flag included in the coded video bitstream.
[0109] (11) The decoder of feature (9) or (10), wherein the program code further includes decoding code configured to cause the at least one processor to decode one or more geometry maps included in the coded video bitstream.
[0110] (12) The decoder of feature (11), wherein based on determining that the reconstruction mode is the sampling mode, the second reconstruction code further includes: second determination code configured to cause the at least one processor to determine a boundary polygon of each of the one or more 2D meshes based on corresponding reconstructed boundary vertices of each of the one or more 2D meshes; and identification code configured to cause the at least one processor to identify each vertex included in the boundary polygon of each of the one or more 2D meshes as an interior vertex.
[0111] (13) The decoder of feature (12), wherein the third reconstruction code further includes derivation code configured to cause the at least one processor to derive interior vertices in the 3D mesh based on corresponding positions in the one or more decoded geometry maps, a sampling rate, and offsets in the one or more decoded geometry maps.
[0112] (14) The decoder of any one of features (9) to (13), wherein based on determining that the reconstruction mode is the non-sampling mode, the one or more interior vertices of each of the one or more 2D meshes are reconstructed based on the one or more reconstructed boundary vertices of each of the one or more 2D meshes.
[0113] 15. The decoder of claim 14, wherein the third reconstruction code further includes fourth reconstruction code configured to cause the at least one processor to reconstruct interior vertices in the 3D mesh by concatenating reconstructed interior vertices of the one or more 2D meshes with corresponding reconstructed boundary vertices in the one or more 2D meshes.
[0114] (16) The decoder of any one of features (9) to (15), wherein the third reconstruction code further includes fourth reconstruction code configured to cause the at least one processor to reconstruct boundary vertices in the 3D mesh based on a 2D-to-3D index array that maps one or more corresponding reconstructed boundary vertices of each of the one or more 2D meshes to the boundary vertices in the 3D mesh.
[0115] (17) A non-transitory computer-readable medium having stored thereon instructions that, when executed by at least one processor in a decoder, cause the at least one processor to perform the following steps: receive a coded video bitstream including one or more two-dimensional (2D) meshes corresponding to a three-dimensional (3D) mesh; reconstructing one or more boundary vertices of each of the one or more 2D meshes; determining whether a reconstruction mode for reconstructing one or more interior vertices of each of the one or more 2D meshes is one of: (i) a sampling mode; and (ii) a non-sampling mode; reconstructing the one or more interior vertices of each of the one or more 2D meshes based on the determined reconstruction mode; and reconstructing the 3D mesh based on the reconstructed one or more boundary vertices of each of the one or more 2D meshes and the reconstructed one or more interior vertices of each of the one or more 2D meshes.
[0116] (18) The non-transitory computer-readable medium of (17), wherein the determination of the reconstruction mode is based on a flag included in the coded video bitstream.
[0117] (19) The non-transitory computer-readable medium of feature (17) or (18), wherein the instructions further cause the at least one processor to perform a step of decoding one or more geometry maps included in the coded video bitstream.
[0118] (20) The non-transitory computer-readable medium of feature (19), wherein the step of reconstructing the one or more interior vertices of each of the one or more 2D meshes based on determining that the reconstruction mode is the sampling mode further includes the steps of: determining a boundary polygon of each of the one or more 2D meshes based on corresponding reconstructed boundary vertices of each of the one or more 2D meshes; and identifying each vertex included in the boundary polygon of each of the one or more 2D meshes as an interior vertex.
Claims
1. A method executed by at least one processor of a decoder, comprising: receiving a coded video bitstream including one or more two-dimensional (2D) meshes corresponding to a three-dimensional (3D) mesh; reconstructing one or more boundary vertices of each of said one or more 2D meshes; determining whether a reconstruction mode for reconstructing one or more interior vertices of each of the one or more 2D meshes is one of: (i) a sampling mode; and (ii) a non-sampling mode; reconstructing the one or more interior vertices of each of the one or more 2D meshes based on the determined reconstruction mode; reconstructing the 3D mesh based on the one or more reconstructed boundary vertices of each of the one or more 2D meshes and the one or more reconstructed interior vertices of each of the one or more 2D meshes; A method comprising:
2. The method of claim 1 , wherein the determination of the reconstruction mode is based on a flag included in the coded video bitstream.
3. The method of claim 1 , further comprising the step of decoding one or more geometry maps included in the coded video bitstream.
4. based on determining that the reconstruction mode is the sampling mode, Reconstructing the one or more interior vertices of each of the one or more 2D meshes comprises: determining a boundary polygon for each of the one or more 2D meshes based on corresponding reconstructed boundary vertices for each of the one or more 2D meshes; and identifying each vertex contained in the boundary polygon of each of the one or more 2D meshes as an interior vertex.
5. The step of reconstructing the 3D mesh comprises:
5. The method of claim 4, comprising deriving interior vertices in the 3D mesh based on corresponding positions in the one or more decoded geometry maps, a sampling rate, and offsets in the one or more decoded geometry maps.
6. 2. The method of claim 1 , wherein based on determining that the reconstruction mode is the non-sampling mode, the one or more interior vertices of each of the one or more 2D meshes are reconstructed based on the reconstructed one or more boundary vertices of each of the one or more 2D meshes.
7. The step of reconstructing the 3D mesh comprises:
7. The method of claim 6, comprising reconstructing interior vertices in the 3D mesh by concatenating reconstructed interior vertices of the one or more 2D meshes with corresponding reconstructed boundary vertices in the one or more 2D meshes.
8. The step of reconstructing the 3D mesh comprises:
2. The method of claim 1 , comprising reconstructing boundary vertices in the 3D mesh based on a 2D to 3D index array that maps one or more corresponding reconstructed boundary vertices of each of the one or more 2D meshes to the boundary vertices in the 3D mesh.
9. at least one memory configured to store program code; at least one processor configured to read the program code and to act as directed by the program code; The program code comprises: receiving code configured to cause the at least one processor to receive a coded video bitstream including one or more two-dimensional (2D) meshes corresponding to a three-dimensional (3D) mesh; a first reconstruction code configured to cause the at least one processor to reconstruct one or more boundary vertices of each of the one or more 2D meshes; a first decision code configured to cause the at least one processor to determine whether a reconstruction mode for reconstructing one or more interior vertices of each of the one or more 2D meshes is one of: (i) a sampling mode; and (ii) a non-sampling mode; second reconstruction code configured to cause the at least one processor to reconstruct the one or more interior vertices of each of the one or more 2D meshes based on the determined reconstruction mode; and a third reconstruction code configured to cause the at least one processor to reconstruct the 3D mesh based on the one or more reconstructed boundary vertices of each of the one or more 2D meshes and the one or more reconstructed interior vertices of each of the one or more 2D meshes; A decoder including:
10. The decoder of claim 9 , wherein the reconstruction mode decision is based on a flag included in the coded video bitstream.
11. The program code comprises:
10. The decoder of claim 9, further comprising: decoding code configured to cause the at least one processor to decode one or more geometry maps included in the coded video bitstream.
12. based on determining that the reconstruction mode is the sampling mode, The second reconstruction code comprises: a second determination code configured to cause the at least one processor to determine a boundary polygon of each of the one or more 2D meshes based on corresponding reconstructed boundary vertices of each of the one or more 2D meshes; 12. The decoder of claim 11, further comprising: an identification code configured to cause the at least one processor to identify each vertex included in the bounding polygon of each of the one or more 2D meshes as an interior vertex.
13. The third reconstruction code comprises:
13. The decoder of claim 12, further comprising: derivation code configured to cause the at least one processor to derive interior vertices in the 3D mesh based on corresponding positions in the decoded one or more geometry maps, a sampling rate, and offsets in the decoded one or more geometry maps.
14. 10. The decoder of claim 9, wherein based on determining that the reconstruction mode is the non-sampling mode, the one or more interior vertices of each of one or more 2D meshes are reconstructed based on the reconstructed one or more boundary vertices of each of the one or more 2D meshes.
15. The third reconstruction code comprises:
15. The decoder of claim 14, further comprising fourth reconstruction code configured to cause the at least one processor to reconstruct interior vertices in the 3D mesh by concatenating reconstructed interior vertices of the one or more 2D meshes with corresponding reconstructed boundary vertices in the one or more 2D meshes.
16. The third reconstruction code comprises:
10. The decoder of claim 9, further comprising: fourth reconstruction code configured to cause the at least one processor to reconstruct bound vertices in the 3D mesh based on a 2D-to-3D index array that maps one or more corresponding reconstructed bound vertices of each of the one or more 2D meshes to the bound vertices in the 3D mesh.
17. When executed by at least one processor in a decoder, the at least one processor receiving a coded video bitstream including one or more two-dimensional (2D) meshes corresponding to a three-dimensional (3D) mesh; reconstructing one or more boundary vertices of each of said one or more 2D meshes; determining whether a reconstruction mode for reconstructing one or more interior vertices of each of the one or more 2D meshes is one of: (i) a sampling mode; and (ii) a non-sampling mode; reconstructing the one or more interior vertices of each of the one or more 2D meshes based on the determined reconstruction mode; reconstructing the 3D mesh based on the one or more reconstructed boundary vertices of each of the one or more 2D meshes and the one or more reconstructed interior vertices of each of the one or more 2D meshes; A non-transitory computer-readable medium having stored thereon instructions for causing a computer to execute a program.
18. 20. The non-transitory computer-readable medium of claim 17, wherein the determination of the reconstruction mode is based on a flag included in the coded video bitstream.
19. The instructions may further include:
20. The non-transitory computer-readable medium of claim 17, further comprising: causing the at least one processor to perform the step of decoding one or more geometry maps included in the coded video bitstream.
20. based on determining that the reconstruction mode is the sampling mode, Reconstructing the one or more interior vertices of each of the one or more 2D meshes comprises: determining a boundary polygon for each of the one or more 2D meshes based on corresponding reconstructed boundary vertices for each of the one or more 2D meshes; 20. The non-transitory computer-readable medium of claim 19, further comprising: identifying each vertex included in the boundary polygon of each of the one or more 2D meshes as an interior vertex.
Citation Information
Patent Citations
Adaptive selection of occupancy map precision
US20200221125A1
Video-Based Point Cloud Compression with Predicted Patches
US20210217203A1
Information processing device and method
WO2023281929A1
