Methods for mesh decompression, apparatus including processing circuitry, and related programs and storage media - Patents.com

The patch zipping technique addresses inefficiencies in 3D mesh compression by preserving boundary information, improving the quality and reducing data needs for dynamic meshes.

JP7581518B2Active Publication Date: 2024-11-12TENCENT AMERICA LLC
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
JP2023532133
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2022-08-23
Filing Date
2022-08-24
Publication Date
2024-11-12
Estimated Expiration
2042-08-24

AI Technical Summary

Technical Problem

Existing 3D mesh compression techniques suffer from inefficiencies and quality loss due to irreversible compression methods, particularly in dynamic meshes with time-varying connectivity and attributes, leading to gaps and distortions in reconstructed meshes.

Method used

A patch zipping technique is employed to preserve boundary information of UV patches, ensuring accurate reconstruction by encoding patch boundary information separately, which can be used in conjunction with existing video and point cloud compression codecs.

Benefits of technology

This approach enhances the quality of reconstructed meshes by eliminating gaps and maintaining fidelity, while reducing data requirements for efficient storage and transmission of dynamic 3D models.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007581518000001
    Figure 0007581518000001
  • Figure 0007581518000002
    Figure 0007581518000002
  • Figure 0007581518000003
    Figure 0007581518000003
Patent Text Reader

Abstract

In some examples, an apparatus for mesh coding includes processing circuitry. The processing circuitry receives a bitstream carrying encoded information of a mesh divided into patches. The bitstream includes a first portion and a second portion, where the first portion includes patch information and the second portion includes patch boundary information indicating that at least a first edge of a first patch and a second edge of a second patch are an edge-mate pair. The processing circuitry decodes the first portion to obtain the patch information and decodes the second portion to obtain the patch boundary information. The processing circuitry generates a reconstructed mesh based on the patch information and the patch boundary information, where the first edge and the second edge are mapped to the same edge in the reconstructed mesh to connect the first patch with the second patch.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical Field]

[0001] (Reference to Related Application) This application claims the benefit of priority to U.S. Provisional Application No. 63 / 237,097, entitled "Patch Zippering for Mesh Compression," filed August 25, 2021, which claims the benefit of priority to U.S. Provisional Application No. 17 / 893,892, entitled "PATCH ZIPPERING FOR MESH COMPRESSION," filed August 23, 2022. The disclosures of the prior applications are incorporated herein by reference in their entireties.

[0002] (Technical field) This disclosure describes embodiments generally related to mesh coding. [Background technology]

[0003] The background statement provided herein is intended to generally present the context for the disclosure. Work of the currently named inventors, to the extent that their work is described in this Background section, and aspects of the statement that may not otherwise qualify as prior art at the time of filing, are not admitted expressly or impliedly as prior art to the present disclosure.

[0004] Various techniques have been developed to capture and represent the world, such as objects in the world, environments in the world, and the like, in three-dimensional (3D) space. 3D representations of the world can enable more immersive forms of interaction and communication. In some examples, point clouds and meshes can be used as 3D representations of the world. Summary of the Invention

[0005] Aspects of the present disclosure provide methods and apparatus for mesh coding (e.g., compression and decompression). In some examples, the apparatus for mesh coding includes processing circuitry. The processing circuitry receives a bitstream carrying encoded information of a mesh divided into patches. The bitstream includes a first portion and a second portion, where the first portion includes patch information and the second portion includes patch boundary information indicating that at least a first edge of a first patch and a second edge of a second patch are an edge-mate pair. The processing circuitry decodes the first portion to obtain the patch information and decodes the second portion to obtain the patch boundary information. The processing circuitry generates a reconstructed mesh based on the patch information and the patch boundary information, where the first edge and the second edge are mapped to the same edge in the reconstructed mesh to connect (zip) the first patch and the second patch to each other.

[0006] In some examples, the patch information decoded from the first portion has different patch boundary information than the patch boundary information decoded from the second portion.

[0007] In some examples, the processing circuitry decodes a boundary table from the second portion, the boundary table including a list of boundary vertices for the patch. The processing circuitry determines a first list of boundary vertices for the first patch and a second list of boundary vertices for the second patch according to the boundary table. Further, the processing circuitry determines a first boundary edge for the first patch according to the first list of boundary vertices and a second boundary edge for the second patch according to the second list of boundary vertices. The first edge is one of the first boundary edges of the first patch, and the second edge is one of the second boundary edges of the second patch.

[0008] In some examples, the processing circuitry decodes a first array from the second portion. The first array carries UV coordinates for mapping bounding vertices of the patch to the UV atlas. The processing circuitry determines, according to the first array, first bounding UV coordinates of a first list of bounding vertices for mapping the first patch to a first UV patch in the UV atlas. The first UV patch corresponds to the first patch. The processing circuitry determines, according to the first array, second bounding UV coordinates of a second list of bounding vertices for mapping a second patch to a second UV patch in the UV atlas. The second UV patch corresponds to the second patch.

[0009] To generate the reconstructed mesh, in some examples, the processing circuitry determines a first UV coordinate of a first vertex inside the first UV patch and a second UV coordinate of a second vertex inside the second UV patch. In one example, the first vertex and the second vertex are in a regular grid of the UV atlas. Further, in some examples, the processing circuitry determines first connectivity information for the first UV patch based on the first UV coordinate of the first vertex and the first boundary UV coordinate of the first list of boundary vertices, and determines second connectivity information for the second UV patch based on the second UV coordinate of the second vertex and the second boundary UV coordinate of the second list of boundary vertices. In some examples, the processing circuitry determines the use of a particular connectivity determination technique based on a signal in the bitstream.

[0010] In some examples, the processing circuitry decodes a second array from the second portion, the second array having mesh vertex indices for the mesh corresponding to boundary vertices of the patch. The processing circuitry is configured to determine a first mesh vertex index for a first vertex of a first edge according to the second array, and determine a second mesh vertex index for a second vertex of a second edge according to the second array. The first mesh vertex index for the first vertex of the first edge matches the second mesh vertex index for the second vertex of the second edge.

[0011] In some examples, the processing circuitry decodes a third array from the second portion, the third array having three-dimensional coordinates corresponding to mesh vertex indices for boundary vertices of the patches within the mesh.

[0012] In some examples, the processing circuitry determines a first 3D coordinate for a first vertex of a first edge according to the third array, and determines a second 3D coordinate for a second vertex of a second edge according to the third array, wherein the first 3D coordinate for the first vertex of the first edge matches the second 3D coordinate for the second vertex of the second edge.

[0013] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by a computer, cause the computer to perform any one or combination of methods for mesh coding.

[0014] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings. [Brief explanation of the drawings]

[0015] [Figure 1] 1 illustrates a block diagram of a communication system in accordance with some examples.

[0016] [Figure 2] 1 illustrates a block diagram of a streaming system in some examples.

[0017] [Figure 3] 1 shows a block diagram of an encoder for encoding point cloud frames in some examples.

[0018] [Figure 4] 1 illustrates a block diagram of a decoder for decoding a compressed bitstream corresponding to a point cloud frame in some examples.

[0019] [Figure 5] 1 shows a block diagram of a video decoder in some examples.

[0020] [Figure 6] 1 shows a block diagram of a video encoder in some examples.

[0021] [Figure 7] 1 shows a block diagram of an encoder for encoding point cloud frames in some examples.

[0022] [Figure 8] 1 shows a block diagram of a decoder for decoding a compressed bitstream corresponding to a point cloud frame.

[0023] [Figure 9] 1 shows diagrams of reconstructed meshes in some examples.

[0024] [Figure 10] 1 shows a block diagram of a mesh coding (compression and decompression) framework in some examples.

[0025] [Figure 11]1 shows a flowchart outlining an example process in some examples.

[0026] [Figure 12] 1 shows a flowchart outlining an example process in some examples.

[0027] [Figure 13] 1 illustrates a schematic diagram of a computer system in some examples. DETAILED DESCRIPTION OF THE INVENTION

[0028] Aspects of the present disclosure provide techniques in the field of three-dimensional (3D) media processing.

[0029] Technological developments in three-dimensional (3D) media processing, such as advances in 3D capture, 3D modeling, and 3D rendering, and the like, are facilitating the ubiquitous presence of 3D media content across several platforms and devices. For example, a baby's first steps can be captured on one continent, and media technologies can enable grandparents on another continent to view (and possibly interact with) the baby and enjoy an immersive experience with the baby. According to one aspect of the present disclosure, to enhance the immersive experience, 3D models are becoming increasingly sophisticated, and the creation and consumption of 3D models occupies a significant amount of data resources, such as data storage and data transmission resources.

[0030] According to some aspects of the present disclosure, point clouds and meshes can be used as 3D models to represent immersive content.

[0031] A point cloud may generally refer to a set of points in 3D space, where each point is associated with attributes such as color, material properties, texture information, intensity attributes, reflectance attributes, motion-related attributes, modality attributes, and various other attributes. Point clouds can be used to reconstruct objects or scenes as a composition of such points.

[0032] An object's mesh (also called a mesh model) can include polygons that describe the object's surface. Each polygon can be defined by its vertices in 3D space and information about how the vertices connect to form polygons. The information about how the vertices connect to form polygons is called connectivity information. In some examples, a mesh can also include attributes, such as color, normals, and the like, associated with the vertices.

[0033] According to some aspects of the present disclosure, some coding tools for point cloud compression (PCC) can be used for mesh compression. For example, a mesh can be remeshed to generate a new mesh from which the connectivity of the new mesh can be inferred. The vertices of the new mesh and the attributes associated with the vertices of the new mesh can be considered as points in the point cloud and can be compressed using a PCC codec.

[0034] A point cloud can be used to reconstruct an object or scene as a composition of such points. The points can be captured using multiple cameras, depth sensors, or lidar in a variety of settings and may consist of thousands to billions of points to realistically represent the reconstructed scene or object. A patch may generally refer to a contiguous subset of the surface described by the point cloud. In one example, a patch includes points with surface normal vectors that deviate from each other by less than a threshold amount.

[0035] PCC can be performed according to various schemes, such as a geometry-based scheme called G-PCC, a video coding-based scheme called V-PCC, and equivalent schemes. According to some aspects of the present disclosure, G-PCC is a purely geometry-based approach that directly encodes 3D geometry and does not share much with video coding, while V-PCC is heavily based on video coding. For example, V-PCC can map points of a 3D cloud to pixels of a 2D grid (image). The V-PCC scheme can utilize a general-purpose video codec for point cloud compression. The PCC codec (encoder / decoder) in this disclosure can be a G-PCC codec (encoder / decoder) or a V-PCC codec.

[0036] According to one aspect of the present disclosure, the V-PCC scheme can compress the geometry, occupancy, and texture of a point cloud as three separate video sequences using an existing video codec. The extra metadata required to interpret the three video sequences is compressed separately. A small portion of the overall bitstream is the metadata, which, in one example, can be efficiently encoded / decoded using a software implementation. The majority of the information is handled by the video codec.

[0037] FIG. 1 illustrates a block diagram of a communication system 100 in some examples. The communication system 100 may include, for example, multiple terminal devices that can communicate with each other via a network 150. For example, the communication system 100 may include a pair of terminal devices 110 and 120 interconnected via the network 150. In the example of FIG. 1, the first pair of terminal devices 110 and 120 may perform unidirectional transmission of point cloud data. For example, the terminal device 110 may compress a point cloud (e.g., points representing structures) captured by a sensor 105 connected to the terminal device 110. The compressed point cloud may be transmitted to another terminal device 120 via the network 150, for example, in the form of a bit stream. The terminal device 120 may receive the compressed point cloud from the network 150, decompress the bit stream to reconstruct the point cloud, and appropriately display the reconstructed point cloud. One-way data transmission may be common in media service applications and the like.

[0038] In the example of FIG. 1 , terminal devices 110 and 120 may be illustrated as a server and a personal computer, but the principles of the present disclosure are not so limited. Embodiments of the present disclosure find application in laptop computers, tablet computers, smartphones, gaming consoles, media players, and / or dedicated three-dimensional (3D) appliances. Network 150 represents any number of networks that transmit compressed point clouds between terminal devices 110 and 120. Network 150 may include, for example, wireline and / or wireless communication networks. Network 150 may exchange data over circuit-switched and / or packet-switched channels. Exemplary networks include telecommunications networks, local area networks, wide area networks, the Internet, and the like.

[0039] 2 shows a block diagram of a streaming system 200 in some examples. The streaming system 200 is a point cloud utilization application. The disclosed subject matter may be equally applicable to other point cloud-enabled applications, such as 3D telepresence applications, virtual reality applications, and the like.

[0040] The streaming system 200 may include a capture subsystem 213. The capture subsystem 213 may include a point cloud source 201, such as a light detection and ranging (LIDAR) system, a 3D camera, a 3D scanner, a graphics generation component that generates an uncompressed point cloud in software, and the like, for example, to generate the uncompressed point cloud 202. In one example, the point cloud 202 includes points captured by a 3D camera. The point cloud 202 is depicted with bold lines to emphasize its high data volume when compared to a compressed point cloud 204 (a compressed point cloud bitstream). The compressed point cloud 204 can be generated by an electronic device 220 including an encoder 203 coupled to the point cloud source 201. The encoder 203 may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The compressed point cloud (204) (or compressed point cloud bitstream (204)), represented as a thin line to emphasize its lower data volume compared to the point cloud stream (202), can be stored on a streaming server (205) for future use. One or more streaming client subsystems, such as client subsystems (206) and (208) of FIG. 2, can access the streaming server (205) to retrieve copies (207) and (209) of the compressed point cloud (204). The client subsystem (206) can include a decoder (210), for example, within the electronic device (230). The decoder (210) decodes the incoming copy (207) of the compressed point cloud and creates an outgoing stream of a reconstructed point cloud (211), which can be rendered on a rendering device (212).

[0041] It should be noted that the electronic devices 220 and 230 may include other components (not shown). For example, the electronic device 220 may include a decoder (not shown), and the electronic device 230 may include an encoder (not shown).

[0042] In some streaming systems, the compressed point clouds 204, 207, and 209 (e.g., compressed point cloud bitstreams) may be compressed according to a particular standard. In some examples, a video coding standard is used in compressing the point cloud. Examples of such standards include High Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), and the like.

[0043] 3 shows a block diagram of a V-PCC encoder (300) for encoding point cloud frames, according to some embodiments. In some embodiments, the V-PCC encoder (300) can be used in the communication system (100) and the streaming system (200). For example, the encoder (203) can be configured and operate in a manner similar to the V-PCC encoder (300).

[0044] The V-PCC encoder (300) receives point cloud frames as uncompressed input and generates a bitstream corresponding to the compressed point cloud frames. In some embodiments, the V-PCC encoder (300) may receive point cloud frames from a point cloud source, such as the point cloud source (201) and the like.

[0045] In the example of Figure 3, the V-PCC encoder (300) includes a patch generation module (306), a patch packing module (308), a geometry image generation module (310), a texture image generation module (312), a patch information module (304), an occupancy map module (314), a smoothing module (336), image padding modules (316) and (318), a group dilation module (320), video compression modules (322), (323), and (332), an auxiliary patch information compression module (338), an entropy compression module (334), and a multiplexer (324).

[0046] According to one aspect of the present disclosure, the V-PCC encoder (300) converts three-dimensional point cloud frames into an image-based representation along with some metadata (e.g., occupancy map and patch information) used to convert the compressed point cloud back to a decompressed point cloud. In some examples, the V-PCC encoder (300) converts the 3D point cloud frames into a geometry image, a texture image, and an occupancy map, and then can encode the geometry image, texture image, and occupancy map into a bitstream using video coding techniques. In general, a geometry image is a 2D image with pixels filled with geometry values ​​associated with points projected onto the pixel, and the pixels filled with geometry values ​​can be referred to as geometry samples. A texture image is a 2D image with pixels filled with texture values ​​associated with points projected onto the pixel, and the pixels filled with texture values ​​can be referred to as texture samples. An occupancy map is a 2D image with pixels filled with values ​​indicating whether or not a patch is occupied.

[0047] The patch generation module (306) segments the point cloud into a set of patches that may or may not overlap, such that each patch may be described by a depth field for a plane in 2D space (e.g., a patch is defined as a contiguous subset of the surface described by the point cloud). In some embodiments, the patch generation module (306) aims to decompose the point cloud into a minimum number of patches with smooth boundaries while also minimizing reconstruction error.

[0048] In some examples, the patch information module (304) can collect patch information indicating the size and shape of the patch. In some examples, the patch information can be packed into an image frame and then encoded by the auxiliary patch information compression module (338) to generate compressed auxiliary patch information.

[0049] In some examples, the patch packing module (308) is configured to map the extracted patches onto a two-dimensional (2D) grid while minimizing unused space and ensuring that every M×M (e.g., 16×16) block of the grid is associated with a unique patch. Efficient patch packing can directly impact compression efficiency by either minimizing unused space or ensuring temporal consistency.

[0050] The geometry image generation module (310) can generate a 2D geometry image associated with the geometry of the point cloud at a given patch location. The texture image generation module (312) can generate a 2D texture image associated with the texture of the point cloud at a given patch location. The geometry image generation module (310) and the texture image generation module (312) use the 3D-to-2D mapping calculated during the packing process to store the geometry and texture of the point cloud as images. To better handle the case where multiple points project onto the same sample, each patch is projected onto two images, called layers. In one example, the geometry image is represented by a W x H monochrome frame in YUV420-8 bit format. To generate the texture image, the texture generation procedure uses the reconstructed / smoothed geometry to calculate the color associated with the resampled point.

[0051] The occupancy map module (314) can generate an occupancy map that describes padding information for each unit. For example, the occupancy image includes a binary map that indicates for each cell of a grid whether the cell belongs to empty space or to the point cloud. In one example, the occupancy map uses binary information that describes for each pixel whether the pixel is padded. In another example, the occupancy map uses binary information that describes for blocks of pixels whether the block of pixels is padded.

[0052] The occupancy map generated by the occupancy map module (314) can be compressed using lossless or lossy coding. When lossless coding is used, the occupancy map is compressed using an entropy compression module (334). When lossy coding is used, the occupancy map is compressed using a video compression module (332).

[0053] Note that the patch packing module (308) may leave some empty space between the 2D patches that are packed into the image frame. The image padding modules (316) and (318) may fill the empty space (called padding) to generate an image frame that may be suitable for 2D video and image codecs. Image padding is also called background filling, which may fill unused space with redundant information. In some instances, good background filling minimizes bitrate increase while not introducing significant coding distortion around patch boundaries.

[0054] The video compression modules (322), (323), and (332) can encode 2D images, such as padded geometry images, padded texture images, and occupancy maps, based on an appropriate video coding standard, such as HEVC, VVC, and the like. In one example, the video compression modules (322), (323), and (332) are separate components that operate separately. Note that in another example, the video compression modules (322), (323), and (332) can be implemented as a single component.

[0055] In some examples, the smoothing module (336) is configured to generate a smoothed image of the reconstructed geometry image. The smoothed image can be provided to the texture image generation (312). The texture image generation (312) may then adjust the generation of the texture image based on the reconstructed geometry image. For example, when the patch shape (e.g., geometry) is slightly distorted during encoding and decoding, the distortion may be taken into account when generating the texture image to correct for the distortion in the patch shape.

[0056] In some implementations, the group expansion (320) is configured to pad pixels around object boundaries with redundant low-frequency content to improve the visual quality of the reconstructed point cloud as well as the coding gain.

[0057] A multiplexer (324) can multiplex the compressed geometry images, compressed texture images, compressed occupancy maps, and compressed auxiliary patch information into a compressed bitstream.

[0058] 4 shows a block diagram of a V-PCC decoder (400) for decoding a compressed bitstream corresponding to a point cloud frame in some examples. In some examples, the V-PCC decoder (400) can be used in the communication system (100) and the streaming system (200). For example, the decoder (210) can be configured to operate in a manner similar to the V-PCC decoder (400). The V-PCC decoder (400) receives the compressed bitstream and generates a reconstructed point cloud based on the compressed bitstream.

[0059] In the example of Figure 4, the V-PCC decoder (400) includes a demultiplexer (432), video decompression modules (434) and (436), an occupancy map decompression module (438), an auxiliary patch information decompression module (442), a geometry reconstruction module (444), a smoothing module (446), a texture reconstruction module (448), and a color smoothing module (452).

[0060] The demultiplexer (432) can receive the compressed bitstream and separate the compressed bitstream into compressed texture images, compressed geometry images, compressed occupancy maps, and compressed auxiliary patch information.

[0061] The video compression modules (434) and (436) can decode images compressed according to an appropriate standard (e.g., HEVC, VVC, etc.) and output decompressed images. For example, the video decompression module (434) decodes compressed texture images and outputs decompressed texture images, and the video compression module (436) decodes compressed geometry images and outputs decompressed geometry images.

[0062] The occupancy map decompression module (438) can decode the compressed occupancy map according to an appropriate standard (e.g., HEVC, VVC, etc.) and output a decompressed occupancy map.

[0063] The auxiliary patch information decompression module (442) can decode the compressed auxiliary patch information according to an appropriate standard (e.g., HEVC, VVC, etc.) and output the decompressed auxiliary patch information.

[0064] A geometry reconstruction module (444) can receive the decompressed geometry image and generate a reconstructed point cloud geometry based on the decompressed occupancy map and the decompressed auxiliary patch information.

[0065] The smoothing module (446) can smooth inconsistencies at the edges of the patches. The smoothing procedure aims to mitigate potential discontinuities that may occur at patch boundaries due to compression artifacts. In some embodiments, a smoothing filter may be applied to pixels located at patch boundaries to mitigate distortions that may be caused by compression / decompression.

[0066] The texture reconstruction module (448) can determine texture information for points in the point cloud based on the decompressed texture image and the smoothed geometry.

[0067] The color smoothing module (452) can smooth out coloring discrepancies. Non-adjacent patches in 3D space are often packed next to each other in 2D video. In some instances, pixel values ​​from non-adjacent patches can be confused by block-based video codecs. The purpose of color smoothing is to reduce visible artifacts that appear at patch boundaries.

[0068] 5 shows a block diagram of a video decoder (510) in some examples. The video decoder (510) can be used in the V-PCC decoder (400). For example, the video decompression modules (434) and (436) and the occupancy map decompression module (438) can be configured similarly to the video decoder (510).

[0069] The video decoder (510) may include a parser (520) that reconstructs symbols (521) from a compressed image (compressed image), such as a coded video sequence. Categories of these symbols contain information used to manage the operation of the video decoder (510). The parser (520) may parse / entropy decode a received coded video sequence. The coding of the coded video sequence may follow a video coding technique or standard and may follow various principles, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (520) may extract a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to the group. Subgroups can include groups of pictures (GOPs), pictures, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), etc. The parser (520) can also extract from the coded video sequence information such as transform coefficients, quantizer parameter values, motion vectors, etc.

[0070] The parser (520) may perform entropy decoding / parsing operations on the video sequence received from the buffer memory to create symbols (521).

[0071] The symbol reconstruction (521) can involve multiple different units, depending on the type of coded video picture or portion thereof (such as inter-picture and intra-picture, inter-block and intra-block) and other factors. Which units are involved and how they are involved can be controlled by subgroup control information parsed from the coded video sequence by the parser (520). The flow of such subgroup control information between the parser (520) and the following units is not depicted for clarity.

[0072] Beyond the functional blocks already described, the video decoder (510) can be conceptually subdivided into a number of functional units, as described below. In a practical implementation operating under commercial constraints, many of these units will interact closely with each other and may be, at least in part, integrated with each other. However, for purposes of describing the disclosed subject matter, the following conceptual subdivision into functional units is appropriate.

[0073] The first unit is a scalar / inverse transform unit (551), which receives quantized transform coefficients as symbols (521) from the parser (520), as well as control information including which transform to use, block size, quantization coefficients, quantization scaling matrices, etc. The scalar / inverse transform unit (551) can output blocks containing sample values ​​that can be input to an aggregator (555).

[0074] In some cases, the output samples of the scaler / inverse transform unit (551) may relate to intra-coded blocks, i.e., blocks that do not use prediction information from a previously reconstructed picture but can use prediction information from a previously reconstructed portion of the current picture. Such prediction information may be provided by an intra picture prediction unit (552). In some cases, the intra picture prediction unit (552) generates blocks of the same size and shape as the block under reconstruction using surrounding reconstructed information fetched from a current picture buffer (558). The current picture buffer (558), for example, buffers a partially reconstructed current picture and / or a fully reconstructed current image. The aggregator (555) may, in some cases, add the prediction information generated by the intra prediction unit (552) to the output sample information as provided by the scaler / inverse transform unit (551) on a sample-by-sample basis.

[0075] In other cases, the output samples of the scalar / inverse transform unit (551) may relate to an inter-coded, potentially motion-compensated block. In such cases, the motion-compensated prediction unit (553) may access the reference picture memory (557) to fetch samples used for prediction. After motion-compensating the fetched samples according to the symbols (521) related to the block, these samples may be added by the aggregator (555) to the output of the scalar / inverse transform unit (551) (in this case, referred to as residual samples or residual signals) to generate output sample information. The addresses in the reference picture memory (557) from which the motion-compensated prediction unit (553) fetches prediction samples may be controlled by a motion vector, which is available to the motion-compensated prediction unit (553), for example, in the form of a symbol (521) that may have X, Y, and reference picture components. Motion compensation may also include interpolation of sample values ​​as fetched from a reference picture memory (557), a motion vector prediction mechanism, etc. when sub-sample exact motion vectors are used.

[0076] The output samples of the aggregator (555) can undergo various loop filtering techniques in a loop filter unit (556). Video compression techniques can include in-loop filter techniques that are controlled by parameters contained in the coded video sequence (also called the coded video bitstream) and made available to the loop filter unit (556) as symbols (521) from the parser (520), but can also be responsive to meta-information obtained during decoding of a coded picture or previous portion (in decoding order) of the coded video sequence, or to previously reconstructed and loop-filtered sample values.

[0077] The output of the loop filter unit (556) can be a sample stream that can be output to a render device and stored in a reference picture memory (557) for use in future inter-picture prediction.

[0078] Once a particular coded picture is fully reconstructed, it can be used as a reference picture for future prediction. For example, once the coded picture corresponding to the current picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) can become part of the reference picture memory (557), and a new current picture buffer can be reallocated before beginning reconstruction of a subsequent coded picture.

[0079] The video decoder (510) may perform decoding operations according to a predetermined video compression technique in a standard, such as ITU-T Rec. H.265. A coded video sequence may conform to the syntax specified by the video compression technique or standard being used, in the sense that the coded video sequence conforms to both the syntax of the video compression technique or standard and the profile as documented in the video compression technique or standard. Specifically, a profile may select specific tools from all tools available in the video compression technique or standard as the only tools available for that profile. A similar requirement for conformance may be that the complexity of the coded video sequence be within the boundaries defined by the level of the video compression technique or standard. In some cases, the level constrains the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The constraints set by the level may, in some cases, be further constrained through a Hypothetical Reference Decoder (HRD) specification and metadata for HRD buffer management signaled in the coded video sequence.

[0080] 6 shows a block diagram of a video encoder (603) according to one embodiment of the present disclosure. The video encoder (603) can be used in the V-PCC encoder (300) to compress point clouds. In one example, the video compression modules (322) and (323) and the video compression module (332) are configured similarly to the encoder (603).

[0081] The video encoder (603) may receive images such as padded geometry images, padded texture images, and the like, and generate compressed images.

[0082] According to one embodiment, the video encoder (603) may code and compress pictures of a source video sequence (images) into a coded video sequence (compressed images) in real time or under any other time constraints required by the application. Enforcing an appropriate coding rate is one function of the controller (650). In some embodiments, the controller (650) controls and is operatively coupled to other functional units as described below. For clarity, coupling is not depicted. Parameters set by the controller (650) may include rate control-related parameters (picture skip, quantization, lambda value for rate-distortion optimization techniques, ...), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) can be configured with other appropriate functions related to the video encoder (603) optimized for a particular system design.

[0083] In some embodiments, the video encoder (603) is configured to operate in a coding loop. As an overly simplified description, in one example, the coding loop can include a source coder (630) (e.g., responsible for creating symbols, such as a symbol stream, based on an input picture to be coded and a reference picture) and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to create sample data in a manner similar to that of the (remote) decoder (because in the video compression techniques considered in the disclosed subject matter, any compression between the symbols and the coded video bitstream is lossless). The reconstructed sample stream (sample data) is input to a reference picture memory (634). Because decoding of the symbol stream yields bit-exact results regardless of the location (local or remote) of the decoder, the contents of the reference picture memory (634) are also bit-exact between the local and remote encoders. In other words, the predictor of the encoder "sees" the exact same sample values ​​as the reference picture samples that the decoder would "see" when using the prediction during decoding. The basic principle of reference picture synchronism (and the resulting drift when synchronism cannot be maintained, e.g., due to channel errors) is also used in several related technologies.

[0084] The operation of the "local" decoder (633) can be the same as a "remote" decoder, such as the video decoder (510) already described in detail in conjunction with Figure 5. However, briefly referring also to Figure 5, because symbols are available and the encoding / decoding of the symbols into a coded video sequence by the entropy coder (645) and parser (520) can be lossless, the entropy decoding portion of the video decoder (510), including the parser (520), may not be fully implemented in the local decoder (633).

[0085] In operation, in some examples, the source coder (630) may perform motion-compensated predictive coding, which predictively codes an input picture with reference to one or more previously coded pictures from a video sequence designated as “reference pictures.” In this manner, the coding engine (632) codes differences between pixel blocks of the input picture and pixel blocks of reference pictures that may be selected as predictive references for the input picture.

[0086] The local video decoder (633) may decode coded video data of pictures that may be designated as reference pictures based on symbols produced by the source coder (630). The operation of the coding engine (632) may advantageously be a lossy process. When the coded video data may be decoded by a video decoder (not shown in FIG. 6), the reconstructed video sequence may typically be a replica of the source video sequence with some errors. The local video decoder (633) may replicate the decoding process that may be performed on reference pictures by the video decoder and store the reconstructed reference pictures in a reference picture cache (634). In this way, the video encoder (603) may locally store copies of reconstructed reference pictures with common content as reconstructed reference pictures to be retrieved by a far-end video decoder (without transmission errors).

[0087] The predictor (635) may perform the prediction search for the coding engine (632). That is, for a new picture to be coded, the predictor (635) may search the reference picture memory (634) for sample data (as candidate reference pixel blocks) or specific metadata such as reference picture motion vectors, block shapes, etc., that may serve as suitable prediction references for the new picture. The predictor (635) may operate on a sample block-by-pixel block basis to find suitable prediction references. In some cases, as determined by the search results obtained by the predictor (635), the input picture may have prediction references drawn from multiple reference pictures stored in the reference picture memory (634).

[0088] The controller (650) may manage the coding operations of the source coder (630), including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0089] The output of all the aforementioned functional units may undergo entropy coding in an entropy coder (645), which converts the symbols produced by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc.

[0090] A controller (650) may manage the operation of the video encoder (603). During coding, the controller (650) may assign a particular coded picture type to each coded image, which may affect the coding technique that may be applied to the respective picture. For example, a picture may often be assigned as one of the following picture types.

[0091] An intra picture (I-picture) may be coded and decoded without using any other picture in a sequence as a source of prediction. Some video codecs allow different types of intra pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art are aware of these variations of I-pictures and their respective uses and configurations.

[0092] A predicted picture (P picture) may be coded and decoded using intra- or inter-prediction, which uses at most one motion vector and reference index to predict the sample values ​​of each block.

[0093] Bidirectionally predicted pictures (B pictures) may be coded and decoded using intra- or inter-prediction, which uses up to two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multi-predicted pictures may use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0094] A source picture is typically spatially subdivided into multiple sample blocks (e.g., 4x4, 8x8, 4x8, or 16x16 sample blocks) and may be decoded on a block-by-block basis. Blocks may be predictively coded with reference to other (already coded) blocks as determined by the coding assignment applied to the block's respective picture. For example, blocks of an I-picture may be non-predictively coded, or they may be predictively coded with reference to already coded blocks of the same picture (spatial prediction or intra-prediction). Pixel blocks of a P-picture may be predictively coded via spatial prediction or via temporal prediction with reference to one previously coded reference picture. Blocks of a B-picture may be predictively coded via spatial prediction or via temporal prediction with reference to one or two previously coded reference pictures.

[0095] The video encoder (603) may perform coding operations according to a predetermined video coding technique or standard, such as ITU-T Rec. H.265. In doing so, the video encoder (603) may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the coded video data may conform to a syntax specified by the video coding technique or standard being used.

[0096] Video may be in the form of multiple source pictures (images) in a temporal sequence. Intra-picture prediction (often abbreviated as intra-prediction) exploits spatial correlation within a given picture, while inter-picture prediction exploits correlation (temporal or otherwise) between pictures. In one example, a particular picture under encoding / decoding, called the current picture, is divided into blocks. When a block in the current picture is similar to a reference block in a previously coded and still buffered reference picture in the video, the block in the current picture can be coded by a vector called a motion vector. A motion vector points to a reference block in a reference picture and can have three dimensions that identify the reference picture if multiple reference pictures are used.

[0097] In some embodiments, a bi-prediction technique can be used in inter-picture prediction. According to the bi-prediction technique, two reference pictures, such as a first reference picture and a second reference picture, are used, both of which precede a current picture in a video in decoding order (but may be past and future, respectively, in display order). A block in the current picture can be coded by a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. A block can be predicted by a combination of the first reference block and the second reference block.

[0098] Furthermore, merge mode techniques can be used in inter-picture prediction to improve coding efficiency.

[0099] According to some embodiments of the present disclosure, prediction, such as inter-picture prediction and intra-picture prediction, is performed in units of blocks. For example, according to the HEVC standard, pictures in a sequence of video pictures are divided into coding tree units (CTUs) for compression, and the CTUs within a picture have the same size, such as 64x64 pixels, 32x32 pixels, or 16x16 pixels. Generally, a CTU includes three coding tree blocks (CTBs): one luma (luminance) CTB and two chroma (color) CTBs. Each CTU can be recursively divided into one or more coding units (CUs). For example, a 64x64 pixel CTU can be divided into one CU of 64x64 pixels, four CUs of 32x32 pixels, or 16 CUs of 16x16 pixels. In one example, each CU is analyzed to determine a prediction type for the CU, such as an inter-prediction type or an intra-prediction type. A CU is divided into one or more prediction units (PUs) depending on temporal or spatial predictability. Generally, each PU includes one luma prediction block (PB) and two chroma PBs. In one embodiment, prediction operations in coding (encoding / decoding) are performed in units of prediction blocks. Using a luma prediction block as an example of a prediction block, the prediction block includes a matrix of values ​​(e.g., luma values) for pixels, such as 8x8 pixels, 16x16 pixels, 8x16 pixels, 16x8 pixels, etc.

[0100] 7 shows a block diagram of a G-PCC encoder (700) in some examples. The G-PCC encoder (700) can be configured to receive point cloud data, compress the point cloud data, and generate a bitstream carrying the compressed point cloud data. In one embodiment, the G-PCC encoder (700) can include a position quantization module (710), a duplicate point removal module (712), an octree encoding module (730), an attribute transfer module (720), a level of detail (LOD) generation module (740), an attribute prediction module (750), a residual quantization module (760), an arithmetic coding module (770), an inverse residual quantization module (780), an addition module (781), and a memory (790) for storing reconstructed attribute values.

[0101] As shown, an input point cloud (701) may be received by a G-PCC encoder (700). The positions (e.g., 3D coordinates) of the point cloud (701) are provided to a quantization module (710). The quantization module (710) is configured to quantize the coordinates to generate quantized positions. The duplicate point removal module (712) is configured to receive the quantized positions and perform a filtering process to identify and remove duplicate points. The octree encoding module (730) receives the filtered positions from the duplicate point removal module (712) and is configured to perform an octree-based coding process to generate a sequence of occupancy codes that describe a 3D grid of voxels. The occupancy codes are provided to an arithmetic coding module (770).

[0102] The attribute transfer module (720) is configured to receive attributes of the input point cloud and perform an attribute transfer process to determine an attribute value for each voxel when multiple attribute values ​​are associated with each voxel. The attribute transfer process can be performed on the reordered points output from the octree encoding module (730). The attributes after the transfer operation are provided to the attribute prediction module (750). The LOD generation module (740) is configured to operate on the reordered points output from the octree encoding module (730) and reorganize the points into different LODs. The LOD information is provided to the attribute prediction module (750).

[0103] The attribute prediction module (750) processes the points according to the LOD-based order indicated by the LOD information from the LOD generation module (740). The attribute prediction module (750) generates an attribute prediction for the current point based on the reconstructed attributes of the set of neighboring points of the current point stored in the memory (790). A prediction residual can subsequently be obtained based on the original attribute values ​​received from the attribute transfer module (720) and the locally generated attribute prediction. When a candidate index is used in each attribute prediction process, an index corresponding to the selected prediction candidate may be provided to the arithmetic coding module (770).

[0104] The residual quantization module (760) is configured to receive and quantize the prediction residual from the attribute prediction module (750) to generate a quantized residual, which is provided to the arithmetic coding module (770).

[0105] The inverse residual quantization module (780) is configured to receive the quantized residual from the residual quantization module (760) and generate a reconstructed prediction residual by performing the inverse of the quantization operation performed by the residual quantization module (760). The summation module (781) is configured to receive the reconstructed prediction residual from the inverse residual quantization module (780) and receive respective attribute predictions from the attribute prediction module (750). Reconstructed attribute values ​​are generated by combining the reconstructed prediction residual and the attribute predictions and stored in the memory (790).

[0106] The arithmetic coding module (770) is configured to receive the occupied codes, candidate indices (if used), quantized residuals (if generated), and other information and perform entropy encoding to further compress the received values ​​or information. As a result, a compressed bitstream (702) carrying the compressed information can be generated. The bitstream (702) may be transmitted or otherwise provided to a decoder that decodes the compressed bitstream, or may be stored on a storage device.

[0107] 8 shows a block diagram of a G-PCC decoder (800) according to one embodiment. The G-PCC decoder (800) can be configured to receive a compressed bitstream and perform point cloud data decompression to decompress the bitstream and generate decoded point cloud data. In one embodiment, the G-PCC decoder (800) can include an arithmetic decoding module (810), an inverse residual quantization module (820), an octree decoding module (830), an LOD generation module (840), an attribute prediction module (850), and a memory (860) for storing reconstructed attribute values.

[0108] As shown, a compressed bitstream (801) can be received by an arithmetic decoding module (810). The arithmetic decoding module (810) is configured to decode the compressed bitstream (801) to obtain quantized residuals and occupied codes (if generated) of the point cloud. The octree decoding module (830) is configured to determine reconstructed positions of points within the point cloud according to the occupied codes. The LOD generation module (840) is configured to reorganize the points into different LODs based on the reconstructed positions and determine an LOD-based order. The inverse residual quantization module (820) is configured to generate reconstructed residuals based on the quantized residuals received from the arithmetic decoding module (810).

[0109] The attribute prediction module (850) is configured to perform an attribute prediction process to determine attribute predictions for points according to an LOD-based order. For example, the attribute predictions for a current point can be determined based on reconstructed attribute values ​​of neighboring points of the current point stored in memory (860). In some examples, the attribute predictions can be combined with respective reconstructed residuals to generate reconstructed attributes for the current point.

[0110] The sequence of reconstructed attributes generated from the attribute prediction module (850), along with the reconstructed positions generated from the octree decoding module (830), corresponds to the decoded point cloud (802) output from the G-PCC decoder (800), in one example. Additionally, the reconstructed attributes are also stored in memory (860) and can be subsequently used to derive attribute predictions for subsequent points.

[0111] In various embodiments, the encoder (300), decoder (400), encoder (700), and / or decoder (800) can be implemented in hardware, software, or a combination thereof. For example, the encoder (300), decoder (400), encoder (700), and / or decoder (800) can be implemented in processing circuitry such as one or more integrated circuits (ICs) that operate with or without software, such as application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), and the like. In another example, the encoder (300), decoder (400), encoder (700), and / or decoder (800) can be implemented as software or firmware that includes instructions stored on a non-volatile (or non-transitory) computer-readable storage medium. The instructions, when executed by processing circuitry, such as one or more processors, cause the processing circuitry to perform the functions of the encoder (300), the decoder (400), the encoder (700), and / or the decoder (800).

[0112] It should be noted that the attribute prediction modules (750) and (850) configured to implement the attribute prediction techniques disclosed herein may be included in other decoders or encoders that may have a similar or different structure to those shown in Figures 7 and 8. Additionally, the encoder (700) and decoder (800) may be included in the same device or separate devices in various examples.

[0113] According to some aspects of the present disclosure, mesh compression may use a coding tool different from a PCC coding tool, or may use a PCC coding tool such as the above-mentioned PCC (e.g., G-PCC, V-PCC) encoder, the above-mentioned PCC (e.g., G-PCC, V-PCC) decoder, and the like.

[0114] An object's mesh (also called a mesh model) can include polygons that describe the object's surface. Each polygon can be defined by the polygon's vertices in 3D space and information about how the vertices connect to form the polygon. The information about how the vertices are connected is called connectivity information. In some examples, a mesh can also include attributes, such as color, normals, and the like, associated with the vertices. Attributes can be associated with the mesh's surface by utilizing mapping information that parameterizes the mesh in a 2D attribute map. The mapping information is typically described by a set of parametric coordinates, called UV coordinates or texture coordinates, that are associated with the mesh's vertices. 2D attribute maps (called texture maps in some examples) are used to store high-resolution attribute information, such as texture, normals, geometry, and the like. Such information can be used for various purposes, such as texture mapping and shading.

[0115] In some embodiments, a mesh may include components called geometry information, connectivity information, mapping information, vertex attributes, and attribute maps. In some examples, the geometry information is described by a set of 3D positions associated with the vertices of the mesh. In one example, (x, y, z) coordinates may be used to describe the 3D positions of the vertices. In some examples, the connectivity information includes a set of vertex indices that describe how the vertices are connected to create the 3D surface. In some examples, the mapping information describes how the mesh surface maps to a planar 2D region. In one example, the mapping information is described by a set of UV parametric / texture coordinates (u, v) associated with the mesh vertices along with the connectivity information. In some examples, the vertex attributes include scalar or vector attribute values ​​associated with the mesh vertices. In some examples, the attribute map includes attributes associated with the mesh surface and stored as a 2D image / video. In one example, the mapping information defines the mapping between a video (e.g., a 2D image / video) and the mesh surface.

[0116] According to aspects of the present disclosure, several techniques, called UV mapping or mesh parameterization, are used to map the surface of a mesh in a 3D domain to a 2D domain. In some examples, the mesh is divided into patches (also called segments in some examples) in the 3D domain. The patches are then each parameterized into a 2D shape. The 2D shapes can be packed (e.g., oriented and arranged) into maps, also called atlases in some examples. In some examples, the maps can be further processed using 2D image or video processing techniques.

[0117] In one example, a UV mapping technique generates a UV atlas (also called a UV map) and one or more texture atlases (also called texture maps) in 2D that correspond to patches of a 3D mesh. The UV atlas contains assignments of 3D vertices of the 3D mesh to 2D points in a 2D domain (e.g., a rectangle). The UV atlas is a mapping between coordinates of the 3D surface and coordinates of the 2D domain. In one example, a point in the UV atlas at 2D coordinate (u,v) has a value formed by the coordinates (x,y,z) of the vertex in the 3D domain. In one example, a texture atlas contains color information for the 3D mesh. For example, a point in the texture atlas at 2D coordinate (u,v) has a color equal to the color of the vertex at (x,y,z) in the 3D domain.

[0118] According to one aspect of the present disclosure, a dynamic mesh is a mesh in which at least one of its components (geometry information, connectivity information, mapping information, vertex attributes, and attribute maps) changes over time. A dynamic mesh can be described by a sequence of meshes (also called mesh frames). Dynamic meshes can require large amounts of data because they can contain significant amounts of information that change over time. Mesh compression techniques can enable efficient storage and transmission of media content in mesh representations.

[0119] In some examples, a dynamic mesh can have constant connectivity information, time-varying geometry, and time-varying vertex attributes. In some examples, a dynamic mesh can have time-varying connectivity information. In one example, a digital content creation tool typically generates a dynamic mesh with time-varying attribute maps and time-varying connectivity information. In some examples, a volumetric acquisition technique is used to generate a dynamic mesh. The volumetric acquisition technique can generate a dynamic mesh with time-varying connectivity information, especially under real-time constraints.

[0120] According to some aspects of the present disclosure, some mesh compression techniques are lossy, and the quality of the reconstructed mesh may suffer from gaps. In some examples, a remeshing technique such as a UV atlas sampling technique is used in mesh compression. The UV atlas sampling technique remeshes a 3D mesh model through regular grid points sampled onto a UV atlas. The original vertices in the UV atlas are replaced by the regular grid points. Connectivity information can be inferred from the regular grid points and does not need to be encoded separately. The sampling points in the UV atlas sampling technique do not coincide with the mesh vertices on the boundaries of the UV patches, and therefore the reconstructed mesh may have gaps.

[0121] According to one aspect of the present disclosure, a UV map (also called a UV atlas) is generated by dividing a mesh into patches along cutting paths consisting of specific edges and vertices of the mesh and parameterizing the patches into UV patches in the UV map. The edges and vertices in the cutting paths are mapped to boundary edges and boundary vertices of the UV patches in the UV map. For example, an edge in the cutting path is split into two boundary edges of two different UV patches in the UV map, and a vertex in the cutting path is split into n (n≧2) boundary vertices of different UV patches in the UV map. Boundary edges resulting from splitting the same edge are called edge mates. Boundary vertices resulting from splitting the same vertex are called vertex mates. Thus, the boundary of a UV patch consists of edge mates and vertex mates.

[0122] In some instances, when estimating parameterization using UV atlas sampling technique on a UV map, sampling points do not match at boundary vertices and boundary edges of UV patches in the UV map, in which case, at the decoder side, the reconstructed boundary of the patch may not satisfy the matching requirements for edge mates and vertex mates, and thus gaps may exist between the patches.

[0123] 9 shows a diagram of a reconstructed mesh (900) processed based on UV atlas sampling. The reconstructed mesh (900) contains gaps (901) between patches.

[0124] Some aspects of the present disclosure provide a patch zipping technique (also called a patch connection technique) for mesh compression. The patch zipping technique can preserve boundary information of UV patches (e.g., original UV patches before remeshing), such as boundary edges and boundary vertices of the UV patches. Using the boundary information of the UV patches, in one example, a waterproof mesh can be reconstructed by a UV atlas sampling technique.

[0125] The UV atlas sampling technique is used as an example of a lossy compression technique that can achieve gap-filled gaps between patches using the patch zipping technique, but it should be noted that other remeshing or compression techniques, regardless of parameterization, sampling, or projection approach, can be used with the patch zipping technique to achieve the gap-filling effect.

[0126] 10 shows a diagram of a framework (1000) for mesh compression according to some embodiments of the present disclosure. The framework (1000) includes a mesh encoder (1010) and a mesh decoder (1050). The mesh encoder (1010) encodes an input mesh (1005) (or a mesh frame, in the case of a dynamic mesh) into a bitstream (1045), and the mesh decoder (1050) decodes the bitstream (1045) to generate a reconstructed mesh (1095) (or a mesh frame, in the case of a dynamic mesh).

[0127] The mesh encoder (1010) can be any suitable device, such as a computer, a server computer, a desktop computer, a laptop computer, a tablet computer, a smartphone, a gaming device, an AR device, a VR device, and the like. The mesh decoder (1050) can be any suitable device, such as a computer, a client computer, a desktop computer, a laptop computer, a tablet computer, a smartphone, a gaming device, an AR device, a VR device, and the like. The bitstream (1045) can be transmitted from the mesh encoder (1010) to the mesh decoder (1050) via any suitable communications network (not shown).

[0128] In the example of Figure 10, the mesh encoder (1010) includes a lossy mesh encoder (1020) and a coupled boundary encoder (1040). The lossy mesh encoder (1020) uses techniques that can cause information loss, which can lead to gaps in the reconstructed mesh. The boundary encoder (1040) is configured to preserve boundary information of patches in the input mesh (1005) and encode the boundary information into a bitstream (1045).

[0129] In some examples, the lossy mesh encoder (1020) uses a UV atlas sampling technique, which may cause information loss and result in gaps in the reconstructed mesh. In the example of Figure 10, the lossy mesh encoder (1020) includes a pre-processing module (1021), a UV atlas sampling module (1025), and an encoder (1030), which are coupled together as shown in Figure 10.

[0130] The pre-processing module (1021) divides the input mesh (1005) to generate patches and parameterizes the patches to generate maps (1022) in the 2D domain, such as a UV map and one or more texture maps. The UV map contains UV patches that correspond to the patches of the input mesh (1005).

[0131] The UV atlas sampling module (1025) performs re-meshing by generating a new mesh (1027) by sampling the UV map on regular grid points. In one example, the new mesh (1027) is in the form of a new UV map and a new texture map whose vertices are mapped to the regular grid points. In some examples, the new mesh (1027) is a reconstructed mesh that is reconstructed from the new UV map and the new texture map. Note that connectivity information of the new mesh (1027) can be inferred according to the regular grid points.

[0132] The encoder (1030) can encode the new mesh (1027). In one example, the encoder (1030) includes a PCC encoder, such as a G-PCC encoder, a V-PCC encoder, or the like. The vertices of the new mesh (1027) can form a point cloud, and the encoder (1030) can encode the point cloud using a PCC encoder. In another example, the encoder (1030) includes a video encoder that can encode the new UV map and the new texture map according to an appropriate video coding standard. The encoder (1030) can be a lossy encoder or a lossless encoder.

[0133] 10, the boundary encoder (1040) can extract patch boundary information from the map (1022) and encode the patch boundary information into a bitstream (1045). The operation of the boundary encoder (1040) is described further in this disclosure.

[0134] In the example of Figure 10, the bitstream (1045) is provided to a mesh decoder (1050). The mesh decoder (1050) includes a decoder (1060), a mesh reconstruction module (1080), and a boundary decoder (1090). In one example, the decoder (1060) corresponds to the encoder (1030) and can decode portions of the bitstream (1045) encoded by the encoder (1030) to generate decoded information (1065).

[0135] In one example, the encoder (1030) is a PCC encoder, in which case the decoder (1060) is a PCC decoder. The decoder (1060) can decode portions of the bitstream (1045) encoded by the encoder (1030) to generate a point cloud having points that are vertices of a mesh. For example, the decoded information (1065) includes a point cloud having points that are vertices of a mesh.

[0136] In another example, the encoder (1030) includes one or more video encoders, and the decoder (1060) includes one or more video decoders. The decoder (1060) can decode portions of the bitstream (1045) encoded by the encoder (1030) to generate maps, such as a UV map, one or more texture maps, and the like. For example, the decoded information (1065) includes a UV map and one or more texture maps decoded from the bitstream (1045).

[0137] In the example of Figure 10, the boundary decoder (1090) corresponds to the boundary encoder (1040) and can decode the portion of the bitstream (1045) encoded by the boundary encoder (1040) to generate decoded patch boundary information (1091).

[0138] In the example of Figure 10, the decoded information (1065) and the decoded patch boundary information (1091) are provided to a mesh reconstruction module (1080), which generates a reconstructed mesh (1095) based on the decoded information (1065) and the decoded patch boundary information (1091). The operation of the boundary decoder (1090) and the mesh reconstruction (1080) is described further in this disclosure.

[0139] It should be noted that components within the mesh encoder (1010), such as the pre-processing module (1021), the UV atlas sampling module (1025), the encoder (1030), and the boundary encoder (1040), may each be implemented by a variety of techniques. In one example, the components are implemented by integrated circuit configurations. In another example, the components are implemented using software that may be executed on one or more processors.

[0140] It should be noted that components within the mesh decoder (1050), such as the decoder (1060), the mesh reconstruction module (1080), and the boundary decoder (1090), may each be implemented by a variety of techniques. In one example, the components are implemented by integrated circuit configurations. In another example, the components are implemented using software that can be executed by one or more processors.

[0141] According to one aspect of the present disclosure, the boundary encoder (1040) can determine UV patches from the UV map, determine boundaries of the UV patches, store the boundaries of the UV patches as patch boundary data in an appropriate data structure, and encode the patch boundary data into a bitstream (1045).

[0142] To determine UV patches from the UV map, in some examples, the boundary encoder (1040) uses a triangle traversal algorithm, such as Edgebreaker or a valence-based algorithm, to find sets of connected triangles in the UV map. Sets of connected triangles correspond to UV patches. The boundary encoder (1040) can also receive information (e.g., partition information, parameterization information, and the like) from the pre-processing module (1021) and determines UV patches from the information received from the pre-processing module (1021).

[0143] To determine the boundary of the UV patch, in some examples, the boundary encoder (1040) determines boundary vertices and boundary edges. Any suitable technique can be used to detect boundary vertices and boundary edges. In one example, when an edge in a UV patch is not shared by two adjacent triangles of the UV patch, for example, when the edge belongs to only one triangle of the UV patch, the edge is a boundary edge of the UV patch, and the vertices of the boundary edge are boundary vertices.

[0144] In another example, the boundary vertices of a UV patch are separated from the mesh vertices. Thus, vertex mates can be found from a UV patch based on the (x,y,z) of the vertices. For example, when multiple vertices in different UV patches have the same (x,y,z) values, the multiple vertices are vertex mates and also boundary vertices of the UV patch. Furthermore, the edge between two boundary vertices is a boundary edge.

[0145] In another example, the partition information is received from the pre-processing module (1021). For example, the partition information includes information on cutting paths that divide the input mesh (1005) into patches. The partition information indicates boundary edges and boundary vertices.

[0146] The patch boundary information can be stored in an appropriate data structure. In one example, the boundary vertices of a UV patch are stored in a specific order, such as clockwise order, counterclockwise order, and the like. In some examples, a hash table (e.g., called boundary_table) is constructed to store the boundary vertices for the UV patches, where the hash table (e.g., boundary_table) includes multiple table entries, each corresponding to a UV patch, and each table entry is a list of the boundary vertices of the corresponding UV patch. For example, an entry corresponding to a UV patch can be accessed according to the label of the UV patch, where the entry stores a list of the boundary vertex indexes of the UV patch in a fixed order, such as counterclockwise order. In one example, a hash function is applied to the label for the i-th UV patch to obtain a value i, and the table entry boundary_table[i] is accessed. The table entry boundary_table[i] stores a list of boundary vertex indices, such that boundary_table[i]=[idx_1,idx_2,idx_3,...,idx_m], where m indicates the number of boundary vertices of the ith UV patch, and the indices of the boundary vertices of the ith UV patch are idx_1,idx_2,idx_3,...,idx_m. Thus, the boundary edges are (idx_1,idx_2),(idx_2,idx_3),...,(idx_m,idx_1).

[0147] Note that in one example, the indices in the boundary table are boundary vertex indices that are defined only for boundary vertices in the UV atlas, excluding vertices inside the UV patch.

[0148] Further, detailed information about the boundary vertices of the UV patch is stored. For example, for each boundary vertex of the UV patch, the UV coordinates of the boundary vertex in the UV atlas, the corresponding mesh vertex index in, for example, the input mesh (1005), and the 3D coordinates in, for example, the input mesh (1005) are stored. In some examples, for each boundary vertex, six integers (e.g., u coordinate in the UV atlas, v coordinate in the UV atlas, mesh vertex index in the input mesh (1005), x coordinate in the input mesh (1005), y coordinate in the input mesh (1005), and z coordinate in the input mesh (1005)) are associated with the boundary vertex and stored. In one example, three arrays referred to as boundary_uv, boundary_idx, and boundary_xyz are used to store detailed information about the boundary vertices. Each of the three arrays contains an array entry for each boundary vertex. The array entries for the boundary vertices can be accessed by the boundary vertex index. The array entries in the array boundary_uv for the boundary vertices store the UV coordinates of the boundary vertices in the UV atlas, the array entries in the array boundary_idx for the boundary vertices store the mesh vertex index (e.g., of the input mesh (1005)) of the boundary vertices in the UV atlas, and the array entries in the array boundary_xyz for the boundary vertices store the 3D source coordinates of the corresponding mesh vertices in the input mesh (1005).

[0149] Thus, in one example, vertex mates can have the same value for mesh vertex index and the same value for 3D coordinate.

[0150] In another example, for a mesh sequence, detailed information of boundary vertices is stored using two three-channel images: for example, the UV coordinates of the boundary vertices and the corresponding mesh vertex indices are stored as three channels in a first three-channel image, and the 3D coordinates of the boundary vertices are stored as three channels in a second three-channel image.

[0151] Note that the mesh vertices corresponding to the boundary vertices of a UV patch are on the cut path, so the corresponding mesh vertex indices described in this publication are defined with respect to those cut path vertices.

[0152] According to one aspect of the present disclosure, due to the boundary information, the occupancy map may not necessarily be coded into the bitstream (1045). For example, points inside a UV patch and points outside a UV patch can be determined based on the boundary information. More specifically, points inside a polygon defined by the boundary vertices of a UV patch can be inferred as occupied points and belong to the UV patch. If a point on the UV atlas does not belong to any UV patch, the point can be inferred to be unoccupied. In some examples, the encoder (1030) includes a video encoder for encoding the geometry image, but does not include a video encoder for the occupancy map.

[0153] In one example, the boundary encoder (1040) encodes boundary_table and boundary_idx into the bitstream (1045) using a lossless (lossless) coding technique, and encodes boundary_xyz and boundary_uv into the bitstream (1045) using a lossy (lossless) coding technique. In another example, the boundary encoder (1040) encodes boundary_table, boundary_uv, and boundary_idx into the bitstream (1045) using a lossless coding technique, and encodes boundary_xyz into the bitstream (1045) using a lossy or lossless coding technique.

[0154] In one example, the boundary information of each mesh frame is coded independently. In some examples, the boundary information of the mesh frames can be coded by searching for correlation between frames. For example, inter-frame prediction and intra-frame related context modeling in entropy coding can be applied when coding the boundary information.

[0155] According to one aspect of the present disclosure, the boundary decoder (1090) is configured to decode boundary information, such as boundary_table, boundary_uv, boundary_idx, and boundary_xyz, from the bitstream (1045). In one example, the mesh reconstruction module (1080) can generate a reconstructed mesh (1095) for each patch. For each UV patch, the mesh reconstruction module (1080) applies a hash function to the UV patch's label to determine a table entry index for a table entry in the boundary_table. The mesh reconstruction module (1080) then accesses the table entry to obtain a list of boundary vertex indices of the boundary vertices for the UV patch and determines the boundary edges of the UV patch based on the boundary vertices.

[0156] In some examples, based on the boundary vertex indices of the boundary vertices of the UV patch from the boundary_table, the array boundary_uv is accessed to obtain the UV coordinates of the boundary vertices of the UV patch. Further, in some examples, when a UV atlas sampling technique is used, sampling points on a regular grid inside the UV patch can be determined, and the UV coordinates of the sampling points can be determined.

[0157] In some examples, the obtained UV coordinates and boundary edges of the UV patch can be used to infer connectivity information and generate the UV patch faces. In one example, a constrained Delaunay triangulation technique is used to generate remeshed triangular faces of the UV patch based on the obtained UV coordinates of points inside the UV patch and the boundary edges of the UV patch. It should be noted that other suitable triangulation or re-meshing techniques may be used to generate the UV patch faces. In some examples, the bitstream includes a signal indicating the particular triangulation or re-meshing technique to use.

[0158] According to one aspect of the present disclosure, the 3D coordinates of vertices inside a UV patch can be obtained from a geometry picture, for example, according to a UV map. For a boundary vertex on a UV patch boundary, in one example, the index of the boundary vertex of the UV patch can be determined from the boundary_table, and then the corresponding mesh vertex index can be determined according to the array boundary_idx. In one example, the array boundary_xyz stores 3D coordinates corresponding to the mesh vertex index, and then the 3D coordinates of the boundary vertex can be determined based on the mesh vertex index for the boundary vertex. In another example, the array boundary_xyz stores 3D coordinates corresponding to the index of the boundary vertex in the UV patch, and then the 3D coordinates of the boundary vertex can be obtained from the array boundary_xyz using the index of the boundary vertex from the boundary_table.

[0159] 11 shows a flowchart outlining a process 1100 according to an embodiment of the disclosure. The process 1100 can be used during an encoding process for a mesh. In various embodiments, the process 1100 is performed by processing circuitry. In some embodiments, the process 1100 is implemented in software instructions, such that the processing circuitry performs the process 1100 when it executes the software instructions. The process begins at S1101 and proceeds to S1110.

[0160] In step S1110, patch boundary information for a plurality of patches divided from the mesh is determined.

[0161] At (S1120), patch information for a plurality of patches is encoded into a first portion of a bitstream.

[0162] At (S1130), patch boundary information for the plurality of patches is encoded into a second portion of the bitstream, where the patch boundary information indicates that at least a first edge of a first patch and a second edge of a second patch are a pair of edge mates.

[0163] According to one aspect of the present disclosure, the patch information in the first portion has different patch boundary information than the patch boundary information in the second portion, e.g., the patch information is encoded by a technique having lossy compression properties, including, for example, UV atlas sampling techniques and the like.

[0164] In some embodiments, UV patches in the UV atlas are encoded in a first part of the bitstream, and boundary information for the UV patches is encoded in a second part of the bitstream. The UV patches in the UV atlas correspond to multiple patches in the mesh.

[0165] In some embodiments, to determine the patch boundary information, a UV patch is determined from a UV atlas, and a boundary of the UV patch is determined. In some examples, to determine the UV patch, a first set of connected triangles in the UV atlas is detected, and the first set of connected triangles forms a first UV patch. To determine the boundary of the UV patch, in one example, a set of edges of the first UV patch is detected, and each of the set of edges is not shared by two adjacent triangles of the first UV patch. In another example, a set of vertices of the first UV patch is detected, and each of the set of vertices has a vertex mate in at least another UV patch.

[0166] In some examples, to encode the UV patches, the UV atlas is sampled on a regular grid to generate a new mesh, and the information for the new mesh is encoded in the first part of the bitstream.

[0167] In some examples, a boundary table is encoded in the second portion to encode boundary information for the UV patches. The boundary table (e.g., boundary_table) includes a list of boundary vertices for the multiple UV patches. In some examples, a first array is encoded in the second portion, where the first array (e.g., boundary_uv) carries UV coordinates for mapping boundary vertices of the multiple patches to a UV atlas. Additionally, in some examples, a second array (e.g., boundary_idx) is encoded in the second portion. The second array carries mesh vertex indices for the mesh corresponding to the boundary vertices of the multiple UV patches. In some examples, a third array (e.g., boundary_xyz) is encoded in the second portion, where the third array carries three-dimensional coordinates corresponding to the mesh vertex indices for the boundary vertices of the multiple UV patches. In some other examples, a third array (e.g., boundary_xyz) is encoded in the second portion, where the third array carries three-dimensional coordinates corresponding to the boundary vertex indices for the boundary vertices of the multiple UV patches.

[0168] According to one aspect of the present disclosure, the occupancy map can be determined based on boundary information encoded in the second portion of the bitstream, and thus the first portion, in one example, does not need to include an occupancy map.

[0169] The process then proceeds to (S1199) and ends.

[0170] The process 1100 may be adapted as appropriate. Steps of the process 1100 may be modified and / or omitted. Additional steps may be added. Any suitable order of implementation may be used.

[0171] 12 shows a flowchart outlining a process (1200) according to one embodiment of the present disclosure. The process (1200) can be used during a decoding process for a mesh. In various embodiments, the process (1200) is performed by processing circuitry. In some embodiments, the process (1200) is implemented in software instructions, such that the processing circuitry performs the process (1200) when it executes the software instructions. The process starts at (S1201) and proceeds to (S1210).

[0172] At (S1210), a bitstream carrying encoded information of a mesh is received. The mesh is divided into patches, and the bitstream includes a first portion and a second portion. The first portion includes patch information, and the second portion includes patch boundary information indicating that at least a first edge of the first patch and a second edge of the second patch are an edge-mate pair.

[0173] In (S1220), the first part is decoded to obtain patch information.

[0174] At (S1230), the second part is decoded to obtain patch boundary information.

[0175] At (S1240), a reconstructed mesh is generated based on the patch information and the patch boundary information. The first edge and the second edge are mapped to the same edge in the reconstructed mesh to zip (connect) the first patch and the second patch together.

[0176] In some examples, the patch information decoded from the first portion has different patch boundary information than the patch boundary information decoded from the second portion, e.g., the patch information is encoded by a technique having lossy compression properties, including, for example, UV atlas sampling techniques and the like.

[0177] In some embodiments, to decode the second portion, a boundary table (e.g., boundary_table) is decoded from the second portion. The boundary table includes a list of boundary vertices of UV patches in the UV atlas. In some examples, a first list of boundary vertices for a first patch is determined according to the boundary table, and a second list of boundary vertices for a second patch is determined according to the boundary table. Furthermore, a first boundary edge for the first patch is determined according to the first list of boundary vertices, where the first edge is one of the first boundary edges of the first patch. A second boundary edge of the second patch is determined according to the second list of boundary vertices, where the second edge is one of the second boundary edges of the second patch.

[0178] To decode the second portion, in some examples, a first array (e.g., boundary_uv) is decoded from the second portion. The first array has UV coordinates for mapping boundary vertices of the patch to the UV atlas. According to the first array, first boundary UV coordinates of a first list of boundary vertices are determined for mapping the first patch to a first UV patch in the UV atlas, where the first UV patch corresponds to the first patch. According to the first array, second boundary UV coordinates of a second list of boundary vertices are determined for mapping the second patch to a second UV patch in the UV atlas, where the second UV patch corresponds to the second patch.

[0179] In some examples, to generate a reconstructed mesh, a first UV coordinate of a first vertex inside a first UV patch is determined, and a second UV coordinate of a second vertex inside a second UV patch is determined. For example, the first vertex and the second vertex are in a regular grid of a UV atlas. In some examples, first connectivity information for the first UV patch is determined based on the first UV coordinate of the first vertex inside the first UV patch and the first boundary UV coordinate of the first list of boundary vertices, and second connectivity information for the second UV patch is determined based on the second UV coordinate of the second vertex inside the second UV patch and the second boundary UV coordinate of the second list of boundary vertices. The first connectivity information and the second connectivity information may be determined by any suitable technique. In one example, the use of a particular connectivity determination technique is determined based on a signal in the bitstream.

[0180] In some examples, a second array (e.g., boundary_idx) is decoded from the second portion, and the second array has mesh vertex indices of the mesh corresponding to the boundary vertices of the patch. In one example, a first mesh vertex index for a first vertex of a first edge is determined, and a second mesh vertex index for a second vertex of a second edge is determined according to the second array. The first mesh vertex index for the first vertex of the first edge matches the second mesh vertex index for the second vertex of the second edge. Thus, the first edge and the second edge are edge mates. Furthermore, in one example, a third array (e.g., boundary_xyz) is decoded from the second portion. The third array has three-dimensional coordinates corresponding to the mesh vertex indices.

[0181] In some other examples, a third array (e.g., boundary_xyz) is decoded from the second portion. The third array has three-dimensional (3D) coordinates corresponding to boundary vertex indices of the UV patch. In one example, a first 3D coordinate for a first vertex of a first edge is determined, and a second 3D coordinate for a second vertex of a second edge is determined according to the third array. The first 3D coordinate for the first vertex of the first edge matches the second 3D coordinate for the second vertex of the second edge, and thus the first edge and the second edge are edge mates.

[0182] Next, the process proceeds to (S1299) and ends.

[0183] The process 1200 may be adapted as appropriate. Steps of the process 1200 may be modified and / or omitted. Additional steps may be added. Any suitable order of implementation may be used.

[0184] The techniques disclosed in this disclosure can be used separately or in combination in any order. Furthermore, each technique (e.g., method, embodiment), encoder, and decoder may be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In some examples, the one or more processors execute a program stored on a non-transitory computer-readable medium.

[0185] The techniques described above can be implemented as computer software using computer-readable instructions and can be physically stored on one or more computer-readable media. For example, Figure 13 illustrates a computer system (1300) suitable for implementing certain embodiments of the disclosed subject matter.

[0186] Computer software can be coded using any suitable machine code or computer language that can be subject to assembly, compilation, linking, or similar mechanisms to produce code containing instructions that can be executed by one or more computer central processing units (CPUs), graphics processing units (GPUs), and the like, either directly or through interpretation, microcode execution, and the like.

[0187] The instructions may be executed by various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, and the like.

[0188] 13 for computer system 1300 are exemplary in nature and are not intended to suggest any limitation on the scope of use or functionality of the computer software implementing embodiments of the present disclosure. The arrangement of components should not be interpreted as having any dependency or requirement regarding any one or combination of components illustrated in the exemplary embodiment of computer system 1300.

[0189] The computer system (1300) may include certain human interface input devices. Such human interface input devices may respond to input by one or more human users, for example, through tactile input (such as keystrokes, swipes, or data glove movements), audio input (such as voice or claps), visual input (such as gestures), or olfactory input (not depicted). Human interface devices may also be used to capture certain media not necessarily directly associated with conscious human input, such as audio (such as speech, music, or ambient sounds), images (such as scanned images or photographic images obtained from a still image camera), or video (such as two-dimensional video, three-dimensional video, including stereoscopic video).

[0190] The input human interface devices may include one or more (only one of each is depicted) of a keyboard (1301), a mouse (1302), a trackpad (1303), a touchscreen (1310), a data glove (not shown), a joystick (1305), a microphone (1306), a scanner (1307), and a camera (1308).

[0191] The computer system (1300) may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the human user's senses, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touchscreen (1310), data gloves (not shown), or joystick (1305), although some haptic feedback devices may not function as input devices), audio output devices (such as speakers (1309) or headphones (not depicted)), visual output devices (such as screens (1310) including CRT screens, LCD screens, plasma screens, and OLED screens, each with or without touchscreen input capability and each with or without haptic feedback capability, some of which may output two-dimensional visual output or output in greater than three dimensions through means such as stereoscopic video output), virtual reality glasses (not depicted), holographic displays and smoke tanks (not depicted), and printers (not depicted).

[0192] The computer system (1300) may also include human-accessible storage devices and their associated media such as optical media (1321) including CD / DVD ROM / RW (1320) including CD / DVD or similar media, thumb drives (1322), removable hard drives or solid state drives (1323), legacy magnetic media such as tape and floppy disks (not depicted), specialized ROM / ASIC / PLD-based devices such as security dongles (not depicted), and the like.

[0193] Those skilled in the art will similarly understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other transitory signals.

[0194] The computer system 1300 may also include an interface 1354 to one or more communication networks 1355. Networks can be, for example, wireless, wired, or optical. Networks can also be local, wide-area, metropolitan, vehicular, and industrial, real-time, delay-tolerant, and the like. Examples of networks include local area networks such as Ethernet; cellular networks including WLAN, GSM, 3G, 4G, 5G, LTE, and the like; television wired or wireless wide-area digital networks including cable TV, satellite TV, and terrestrial broadcast TV; and vehicular and industrial networks including CAN Bus. Particular networks typically require an external network interface adapter attached to a particular general-purpose data port (e.g., a USB port on the computer system 1300) or peripheral bus 1349. Others are typically integrated into the core of the computer system 1300 by attachment to a system bus as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (1300) can communicate with other entities. Such communication can be unidirectional, receive-only (e.g., television broadcast), unidirectional transmit-only (e.g., CANbus to a specific CANbus device), or bidirectional, for example, to other computer systems using local or wide-area digital networks. Specific protocols and protocol stacks can be used with each of these networks and network interfaces, as described above.

[0195] The aforementioned human interface devices, human-accessible storage devices, and network interfaces may be attached to the core (1340) of the computer system (1300).

[0196] The core (1340) may include one or more central processing units (CPUs) (1341), graphics processing units (GPUs) (1342), specialized programmable processing units (1343) in the form of field programmable gate arrays (FPGAs), task-specific hardware accelerators (1344), graphics adapters (1350), etc. These devices may be connected through a system bus (1348), along with read-only memory (ROM) (1345), random access memory (1346), internal mass storage such as a non-user-accessible internal hard drive, SSD, and the like (1347). In some computer systems, the system bus (1348) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, and the like. Peripheral devices may be attached directly to the core's system bus (1348) or through a peripheral bus (1349). In one example, the screen 1310 can be connected to a graphics adapter 1350. Architectures for peripheral buses include PCI, USB, and the like.

[0197] The CPU (1341), GPU (1342), FPGA (1343), and accelerator (1344) can execute specific instructions that, in combination, can constitute the aforementioned computer code. That computer code can be stored in ROM (1345) or RAM (1346). Transient data can also be stored in RAM (1346), while permanent data can be stored, for example, in internal mass storage (1347). Rapid storage and retrieval from any memory device can be enabled through the use of cache memory, which can be closely associated with one or more of the CPU (1341), GPU (1342), mass storage (1347), ROM (1345), RAM (1346), and the like.

[0198] The computer-readable medium may have computer code thereon for performing various computer-implemented operations. The media and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the kind well known and available to those skilled in the computer software arts.

[0199] By way of example, and not limitation, the architecture (1300), and in particular a computer system having a core (1340), can provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, and the like) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be media associated with user-accessible mass storage devices, as described above, as well as specific storage devices of the core (1340) that are non-transitory in nature, such as the core's internal mass storage device (1347) or ROM (1345). Software implementing various embodiments of the present disclosure can be stored in such devices and executed by the core (1340). The computer-readable medium can include one or more memory devices or chips according to particular needs. The software can cause the core (1340), and in particular the processor therein (including a CPU, GPU, FPGA, and the like), to perform particular processes or particular portions of particular processes described herein, including defining data structures stored in RAM (1346) and modifying such data structures according to software-defined processes. Additionally or alternatively, the computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator (1344)), which may operate in place of or in conjunction with software to perform particular processes or portions of particular processes described herein. References to software may include logic, where appropriate, and vice versa. References to computer-readable media may include, where appropriate, circuitry storing software (such as an integrated circuit (IC)) for execution, circuitry embodying logic for execution, or both. The present disclosure includes any appropriate combination of hardware and software.

[0200] While this disclosure has described several exemplary embodiments, there are alterations, substitutions, and various substitute equivalents that fall within the scope of this disclosure. Thus, it will be appreciated that those skilled in the art may devise numerous systems and methods that, although not explicitly shown or described in this disclosure, embody the principles of the present disclosure and are therefore within its spirit and scope.

Claims

1. 1. A method for mesh defrosting, comprising: receiving a bitstream carrying encoded information of a mesh divided into patches, the bitstream including a first portion and a second portion, the first portion including patch information and the second portion including patch boundary information indicating that at least a first edge of a first patch and a second edge of a second patch are an edge-mate pair; Decoding the first portion to obtain the patch information; decoding the second portion to obtain the patch boundary information; generating a reconstructed mesh based on the patch information and the patch boundary information; the first edge and the second edge are mapped to the same edge in the reconstructed mesh to connect the first patch with the second patch; Decoding the second portion comprises: decoding a boundary table from the second portion, the boundary table including a list of boundary vertices associated with the patch; determining a first list of boundary vertices associated with the first patch and a second list of boundary vertices associated with the second patch according to the boundary table; determining a first boundary edge for the first patch according to the first list of boundary vertices, the first edge being a boundary edge of the first patch; determining a second boundary edge of the second patch according to the second list of boundary vertices, wherein the second edge is a boundary edge of the second patch. method.

2. The method of claim 1 , wherein the patch information decoded from the first portion has different patch border information than the patch border information decoded from the second portion.

3. Decoding the second portion comprises: decoding a first array from the second portion, the first array having UV coordinates for mapping the bounding vertices of the patch to a UV atlas; determining first border UV coordinates of the first list of border vertices for mapping the first patch to a first UV patch in the UV atlas according to the first array; determining second border UV coordinates of the second list of border vertices to map the first patch to a second UV patch in the UV atlas according to the first array. The method of claim 1.

4. Generating the reconstructed mesh includes: determining a first UV coordinate of a first vertex inside the first UV patch; determining a second UV coordinate of a second vertex inside the second UV patch. The method according to claim 3.

5. The method of claim 4 , wherein the first vertex and the second vertex lie on a regular grid of the UV atlas.

6. determining first connectivity information for the first UV patch based on the first UV coordinate of the first vertex and the first border UV coordinate of the first list of border vertices; determining second connectivity information for the second UV patch based on the second UV coordinates of the second vertex and the second border UV coordinates of the second list of border vertices. The method according to claim 4.

7. The method of claim 6 , further comprising determining the use of a particular connectivity determination technique based on signals in the bitstream.

8. decoding a second array from the second portion, the second array having mesh vertex indices of the mesh that correspond to the boundary vertices of the patch; determining a first mesh vertex index for a first vertex of the first edge according to the second array; determining a second mesh vertex index for a second vertex of the second edge according to the second array; the first mesh vertex index for a first vertex of the first edge matches the second mesh vertex index for a second vertex of the second edge; The method according to claim 4.

9. 9. The method of claim 8, further comprising decoding a third array from the second portion, the third array having three-dimensional coordinates corresponding to the mesh vertex indices for the boundary vertices of the patch.

10. decoding a third array from the second portion, the third array having three-dimensional (3D) coordinates of the boundary vertices of the patch within the mesh; determining a first 3D coordinate for a first vertex of the first edge according to the third array; determining a second 3D coordinate of a second vertex of the second edge according to the third array; the first 3D coordinate for the first vertex of the first edge coincides with the second 3D coordinate of the second vertex of the second edge; The method according to claim 4.

11. 1. An apparatus including processing circuitry, The processing circuitry includes: configured to receive a bitstream carrying encoded information of a mesh divided into patches, the bitstream comprising a first portion and a second portion, the first portion comprising patch information and the second portion comprising patch boundary information indicating that at least a first edge of a first patch and a second edge of a second patch are an edge-mate pair; configured to decode the first portion to obtain the patch information; configured to decode the second portion to obtain the patch boundary information; configured to generate a reconstructed mesh based on the patch information and the patch boundary information, the first edge and the second edge being mapped to a same edge in the reconstructed mesh to connect the first patch with the second patch; The processing circuitry includes: configured to decode a boundary table from the second portion, the boundary table including a list of boundary vertices of the patch; configured to determine a first list of boundary vertices for the first patch and a second list of boundary vertices for the second patch according to the boundary table; configured to determine a first boundary edge for the first patch according to the first list of boundary vertices, the first edge being a boundary edge of the first patch; configured to determine a second boundary edge of the second patch according to the second list of boundary vertices, the second edge being a boundary edge of the second patch. Device.

12. The apparatus of claim 11 , wherein the patch information decoded from the first portion has different patch border information than the patch border information decoded from the second portion.

13. The processing circuitry includes: configured to decode a first array from the second portion, the first array having UV coordinates for mapping the bounding vertices of the patch to a UV atlas; configured to determine first border UV coordinates of the first list of border vertices to map the first patch to a first UV patch in the UV atlas according to the first array, the first UV patch corresponding to the first patch; determining second boundary UV coordinates of the second list of boundary vertices to map the second patch to a second UV patch in the UV atlas according to the first array, the second UV patch corresponding to the second patch; 12. The apparatus of claim 11.

14. The processing circuitry includes: configured to determine a first UV coordinate of a first vertex inside the first UV patch; determining second UV coordinates of a second vertex inside the second UV patch; 14. The apparatus of claim 13.

15. The apparatus of claim 14 , wherein the first vertex and the second vertex lie on a regular grid of the UV atlas.

16. The processing circuitry includes: configured to determine first connectivity information for the first UV patch based on the first UV coordinate of the first vertex and the first border UV coordinate of the first list of border vertices; determining second connectivity information for the second UV patch based on the second UV coordinates of the second vertex and the second border UV coordinates of the second list of border vertices.

15. The apparatus of claim 14.

17. The processing circuitry includes: configured to decode a second array from the second portion, the second array having mesh vertex indices of the mesh corresponding to the boundary vertices of the patch; configured to determine a first mesh vertex index for a first vertex of the first edge according to the second array; configured to determine a second mesh vertex index for a second vertex of the second edge according to the second array; the first mesh vertex index for a first vertex of the first edge matches the second mesh vertex index for a second vertex of the second edge; 15. The apparatus of claim 14.

18. The processing circuitry includes: configured to decode a third array from the second portion, the third array having three-dimensional (3D) coordinates of the boundary vertices of the patch within the mesh; configured to determine a first 3D coordinate for a first vertex of the first edge according to the third array; configured to determine a second 3D coordinate for a second vertex of the second edge according to the third array; the first 3D coordinate for the first vertex of the first edge coincides with the second 3D coordinate of the second vertex of the second edge; 12. The apparatus of claim 11.

19. A computer program comprising instructions which, when executed by at least one processor, cause a computer to perform the method of any one of claims 1-10.

20. A non-transitory computer readable medium storing instructions which, when executed by a computer, cause the computer to perform the method of any one of claims 1-10.

Citation Information

Patent Citations

  • Three-Dimensional Mesh Compression Using a Video Encoder

    US20210090301A1

  • Encoding images of 3-D objects with improved rendering time and transmission processes

    US6438266B1

  • Information processing device and method

    WO2022269944A1