Coding Connectivity in Vertex Reordering Methods for Mesh Compression

Vertex reordering methods for mesh compression address the challenge of efficient data handling in 3D media processing by explicitly encoding connectivity information, improving data compression and transmission of 3D models.

JP7586590B2Active Publication Date: 2024-11-19TENCENT AMERICA LLC
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
JP2023528042
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2022-09-07
Filing Date
2022-09-09
Publication Date
2024-11-19
Estimated Expiration
2042-09-09

AI Technical Summary

Technical Problem

Existing 3D media processing technologies face challenges in efficiently compressing and transmitting large amounts of data required for immersive 3D models, such as point clouds and meshes, due to resource constraints in data storage and transmission.

Method used

Implementing vertex reordering methods for mesh compression that involve encoding connectivity information explicitly in bitstreams, using image or video codecs to reorder and encode vertex attributes, and employing connectivity estimation rules to reconstruct meshes efficiently.

Benefits of technology

Enhances the compression efficiency of 3D meshes by reducing data size while maintaining quality, allowing for effective transmission and rendering of immersive 3D content across various devices.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007586590000001
    Figure 0007586590000001
  • Figure 0007586590000002
    Figure 0007586590000002
  • Figure 0007586590000003
    Figure 0007586590000003
Patent Text Reader

Abstract

Aspects of the present disclosure provide methods and apparatus for mesh coding (e.g., compression and decompression). In some examples, the apparatus for mesh coding includes a processing circuit. The processing circuit decodes at least a first array and a second array from a bitstream carrying a three-dimensional (3D) mesh frame. The first array includes connectivity values ​​for each of the vertices in the 3D mesh frame according to a vertex traversal order that matches the second array including the coordinates of the vertices. The processing circuit determines edges connecting the vertices based on at least the connectivity values ​​for each of the vertices and the coordinates of the vertices, and reconstructs the 3D mesh frame based on at least the coordinates of the vertices and the edges.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001]

[0001] Reference This application claims priority to U.S. Patent Application No. 17 / 939,706, entitled "Coding Connectivity in a Vertex Reordering Method for Mesh Compression," filed September 7, 2022, which claims priority to U.S. Provisional Patent Application No. 63 / 247,694, entitled "Coding Connectivity in a Vertex Reordering Method for Mesh Compression," filed September 23, 2021. The disclosures of the prior applications are incorporated herein by reference in their entireties.

[0002]

[0002] TECHNICAL FIELD This disclosure describes embodiments generally relating to mesh coding. [Background technology]

[0003] The background statement provided herein is intended to generally present the context of the present disclosure. Work under the name of the current inventor is not admitted, expressly or impliedly, as prior art to the present disclosure to the extent that such work is described in this background section or in a descriptive manner that would not otherwise qualify as prior art at the time of filing.

[0004]

[0004] Various techniques have been developed to capture and represent the world in three-dimensional (3D) space, such as objects in the world and environments around the world. 3D representations of the world can enable more immersive forms of interaction and communication. In some examples, point clouds and meshes can be used as the 3D representation of the world. Summary of the Invention

[0005]

[0005] Aspects of the present disclosure provide methods and apparatus for mesh coding (e.g., compression and decompression). In some examples, the apparatus for mesh coding includes a processing circuit. The processing circuit decodes at least a first array and a second array from a bitstream carrying a three-dimensional (3D) mesh frame. The first array includes connectivity values ​​of each of the vertices in the 3D mesh frame according to a vertex traversal order, and the second array includes coordinates of the vertices according to the vertex traversal order. The processing circuit determines edges connecting the vertices based at least on the connectivity values ​​of each of the vertices and the coordinates of the vertices, and reconstructs the 3D mesh frame based at least on the coordinates of the vertices and the edges.

[0006] In some examples, at least the first array and the second array are two-dimensional (2D) arrays. The processing circuitry decodes at least the first array and the second array using an image decoder and / or a video decoder.

[0007]

[0007] In some examples, the connectivity values ​​of a vertex include at least one of a connectivity pattern index of the vertex and a valence value of the vertex.

[0008] In some examples, the connectivity value of a vertex is a connectivity difference of the vertex relative to the estimated connectivity value of the vertex. In some examples, the processing circuit generates the estimated connectivity value of the vertex from the coordinates of the vertex according to a connectivity estimation rule, and generates a restored connectivity value of the vertex by combining the connectivity difference of the vertex with the estimated connectivity value. The processing circuit then determines the edges connecting the vertices based on the restored connectivity value of the vertex.

[0009]

[0009] The coordinates of the vertices may include at least one of 3D coordinates and / or uv coordinates.

[0010]

[0010] In some examples, the processing circuit receives a signal carried in a bitstream, the signal indicating a conjunctive estimation rule selected from among a plurality of conjunctive estimation rules.

[0011]

[0011] Aspects of the present disclosure also provide a non-transitory computer-readable storage medium storing instructions that, when executed by a computer, cause the computer to perform any one or combination of methods for mesh coding. [Brief description of the drawings]

[0012] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. [Figure 1]

[0013] FIG. 1 illustrates a block diagram of a communication system in some examples. [Diagram 2]

[0014] FIG. 2 illustrates a block diagram of a streaming system in some examples. [Diagram 3]

[0015] FIG. 3 illustrates a block diagram of an encoder for encoding a point cloud frame in some examples. [Figure 4]

[0016] FIG. 4 illustrates a block diagram of a decoder for decoding a compressed bitstream corresponding to a point cloud frame in some examples. [Diagram 5]

[0017] FIG. 5 illustrates a block diagram of a video decoder in some examples. [Figure 6]

[0018] FIG. 6 illustrates a block diagram of a video encoder in some examples. [Figure 7]

[0019] FIG. 7 illustrates a block diagram of an encoder for encoding a point cloud frame in some examples. [Figure 8]

[0020] FIG. 8 illustrates a block diagram of a decoder for decoding a compressed bitstream carrying a point cloud frame in some examples. [Figure 9]

[0021] FIG. 9 shows a diagram illustrating the mapping of a mesh to an atlas in some examples. [Figure 10A]

[0022] FIG. 10A shows an example of using vertex reordering for mesh compression. [Figure 10B] FIG. 10B shows an example of using vertex reordering for mesh compression. [Figure 10C] FIG. 10C illustrates an example of using vertex reordering for mesh compression. [Figure 10D] FIG. 10D shows an example of using vertex reordering for mesh compression. [Figure 10E] FIG. 10E illustrates an example of using vertex reordering for mesh compression. [Figure 11]

[0023] FIG. 11 illustrates a diagram of a mesh compression framework according to some embodiments of the present disclosure. [Figure 12A]

[0024] FIG. 12A illustrates an example of explicitly coding connectivity attributes of a mesh frame according to some embodiments of the present disclosure. [Figure 12B]

[0024] FIG. 12B illustrates an example of explicitly coding connectivity attributes of a mesh frame according to some embodiments of the present disclosure. [Figure 12C]

[0024] FIG. 12C illustrates an example of explicitly coding connectivity attributes of a mesh frame according to some embodiments of the present disclosure. [Figure 12D]

[0024] FIG. 12D illustrates an example of explicitly coding connectivity attributes of a mesh frame according to some embodiments of the present disclosure. [Figure 13]

[0025] FIG. 13 illustrates a diagram of a framework for mesh compression according to some embodiments of the present disclosure. [Figure 14A]

[0026] FIG. 14A illustrates an example of coding a mesh frame connectivity differential according to some embodiments of the present disclosure. [Figure 14B]

[0026] FIG. 14B illustrates an example of coding a mesh frame connectivity differential according to some embodiments of the present disclosure. [Figure 14C]

[0026] FIG. 14C illustrates an example of coding mesh frame connectivity differentials according to some embodiments of the present disclosure. [Figure 14D]

[0026] FIG. 14D illustrates an example of coding a mesh frame connectivity difference according to some embodiments of the present disclosure. [Figure 14E]

[0026] FIG. 14E illustrates an example of coding a mesh frame connectivity differential according to some embodiments of the present disclosure. [Figure 14F]

[0026] FIG. 14F illustrates an example of coding mesh frame connectivity differentials according to some embodiments of the present disclosure. [Figure 15]

[0027] FIG. 15 shows a flow chart outlining an example process in some examples. [Figure 16]

[0028] FIG. 16 shows a flow chart outlining an example process in some examples. [Figure 17]

[0029] FIG. 17 is a schematic diagram of a computer system in some examples. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013]

[0030] Aspects of the present disclosure provide techniques in the field of three-dimensional (3D) media processing.

[0014]

[0031] Technological developments in three-dimensional (3D) media processing, such as advances in 3D capture, 3D modeling, and 3D rendering, are facilitating the ubiquitous presence of 3D media content across several platforms and devices. For example, a baby's first steps can be captured on one continent, and media technologies can enable grandparents to view (and possibly interact with) the baby on another continent and enjoy an immersive experience with the baby. According to one aspect of the present disclosure, to improve the immersive experience, 3D models are becoming more and more sophisticated, and the creation and consumption of 3D models will occupy a significant amount of data resources, such as data storage and data transmission resources.

[0015]

[0032] According to some aspects of the present disclosure, point clouds and meshes may be used as 3D models to represent immersive content.

[0016]

[0033] A point cloud may generally refer to a collection of points in 3D space, each point having associated attributes, such as color, material properties, texture information, intensity attributes, reflectance attributes, motion-related attributes, modality attributes, and various other attributes. A point cloud can be used to reconstruct an object or scene as a composition of such points.

[0017]

[0034] A mesh of a body or object (also called a mesh model) may contain polygons that describe the surface of the object. Each polygon may be defined by its vertices in 3D space and information about how the vertices are connected to the polygon. The information about how the vertices are connected is called connectivity information. In some cases, a mesh may also include attributes associated with the vertices, such as color, normal, etc.

[0018]

[0035] According to some aspects of the present disclosure, any coding tool for point cloud compression (PCC) can be used for mesh compression. For example, a mesh can be remeshed to generate a new mesh, and the connectivity information of the new mesh can be estimated. The vertices of the new mesh and the attributes associated with the vertices of the new mesh can be considered as points in the point cloud and compressed using the PCC codec.

[0019]

[0036] A point cloud can be used to reconstruct an object or scene as a composition of points. The points can be captured using multiple cameras, depth sensors, or Lidar in a variety of setups and can consist of thousands to billions of points, realistically representing the scene or object being reconstructed. A patch can generally refer to a contiguous subset of the surface described by the point cloud. In one example, a patch includes points that have surface normal vectors that deviate from each other by less than a threshold amount.

[0020]

[0037] PCC can be implemented according to various schemes, such as a geometry-based scheme called G-PCC, a video coding-based scheme called V-PCC, etc. According to some aspects of the present disclosure, G-PCC is a purely geometry-based approach that directly encodes 3D geometry and does not share much with video coding, while V-PCC is heavily based on video coding. For example, V-PCC can map points of a 3D cloud to pixels of a 2D grid (image). The V-PCC scheme can utilize a general-purpose video codec for point cloud compression. The PCC codec (encoder / decoder) in the present disclosure can be a G-PCC codec (encoder / decoder) or a V-PCC codec.

[0021]

[0038] According to one aspect of the present disclosure, the V-PCC scheme can compress the geometry, occupancy, and texture of the point cloud into three separate video sequences using an existing video codec. Additional metadata required to interpret the three video sequences is compressed separately. A small portion of the overall bitstream is the metadata, which in one example can be efficiently encoded / decoded using a software implementation. The bulk of the information is handled by the video codec.

[0022]

[0039] FIG. 1 illustrates a block diagram of a communication system (100) in some examples. The communication system (100) includes a plurality of terminal devices capable of communicating with each other, for example, via a network (150). For example, the communication system (100) includes a pair of terminal devices (110) and (120) interconnected via the network (150). In the example of FIG. 1, the first pair of terminal devices (110) and (120) may perform unidirectional transmission of point cloud data. For example, the terminal device (110) may compress a point cloud (e.g., points representing structures) captured by a sensor (105) connected to the terminal device (110). The compressed point cloud may be transmitted to the other terminal device (120) via the network (150), for example, in the form of a bit stream. The terminal device (120) may receive the compressed point cloud from the network (150), decompress the bit stream to reconstruct the point cloud, and appropriately display the reconstructed point cloud. Unidirectional data transmission may be common, such as in media serving applications.

[0023]

[0040] In the example of FIG. 1, terminal devices (110) and (120) may be shown as a server and a personal computer, but the principles of the present disclosure need not be so limited. Embodiments of the present disclosure find application with laptop computers, tablet computers, smartphones, gaming consoles, media players, and / or dedicated three-dimensional (3D) devices. Network (150) represents any number of networks that transmit compressed point clouds between terminal devices (110) and (120). Network (150) may include, for example, wired (wired) and / or wireless communication networks. Network (150) may exchange data over circuit-switched and / or packet-switched channels. Exemplary networks include telecommunications networks, local area networks, wide area networks, the Internet, and the like.

[0024]

[0041] 2 illustrates a block diagram of a streaming system 200 in some examples. The streaming system 200 is a point cloud utilization application. The disclosed subject matter may be equally applicable to other point cloud-enabled applications, such as 3D telepresence applications, virtual reality applications, etc.

[0025]

[0042] The streaming system (200) may include a capture subsystem (213). The capture subsystem (213) may include a point cloud source (201), such as a light detection and ranging (LIDAR) system, a 3D camera, a 3D scanner, a graphics generation component, which may generate an uncompressed point cloud, such as software, and others, such as the uncompressed point cloud (202). In one example, the point cloud (202) includes points captured by a 3D camera. The point cloud (202) is depicted as a thick line to emphasize the large amount of data when compared to the compressed point cloud (204) (compressed point cloud bitstream). The compressed point cloud (204) may be generated by an electronic device (220) including an encoder (203) coupled to the point cloud source (201). The encoder (203) may include hardware, software, or a combination thereof, and may enable or achieve aspects of the disclosed subject matter, as described in more detail below. The compressed point cloud (204) (or a bitstream of the compressed point cloud (204)), depicted as a thin line to emphasize its smaller amount of data when compared to the stream of point cloud (202), can be stored on a streaming server (205) for future use. One or more streaming client subsystems, such as the client subsystems (206) and (208) of FIG. 2, can access the streaming server (205) to retrieve copies (207) and (209) of the compressed point cloud (204). The client subsystem (206) can include a decoder (210), for example in an electronic device (230). The decoder (210) decodes the incoming copy of the compressed point cloud (207) and generates an outgoing stream of a reconstructed point cloud (211) that can be rendered on a rendering device (212).

[0026]

[0043] It should be noted that electronic devices 220 and 230 may include other components (not shown), for example, electronic device 220 may include a decoder (not shown) and electronic device 230 may also include an encoder (not shown).

[0027]

[0044] In some streaming systems, the compressed point clouds (204), (207), and (209) (e.g., compressed point cloud bitstreams) may be compressed according to a particular standard. In some cases, a video coding standard is used to compress the point clouds. Examples of these standards include High Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), etc.

[0028]

[0045] 3 shows a block diagram of a V-PCC encoder (300) for encoding a point cloud frame according to some embodiments. In some embodiments, the V-PCC encoder (300) can be used in the communication system (100) and the streaming system (200). For example, the encoder (203) can be configured and operated similarly to the V-PCC encoder (300).

[0029]

[0046] The V-PCC encoder (300) receives a point cloud frame as an uncompressed input and generates a bitstream corresponding to the compressed point cloud frame. In some embodiments, the V-PCC encoder (300) may receive the point cloud frame from a point cloud source, such as the point cloud source (201).

[0030]

[0047] In the example of FIG. 3, the V-PCC encoder (300) includes a patch generation module (306), a patch packing module (308), a geometric image generation module (310), a texture image generation module (312), a patch information module (304), an occupancy map module (314), a smoothing module (336), image padding modules (316) and (318), a group expansion module (320), video compression modules (322), (323), and (332), an auxiliary patch information compression module (338), an entropy compression module (334), and a multiplexer (324).

[0031]

[0048] According to one aspect of the disclosure, the V-PCC encoder (300) converts the 3D point cloud frame into an image-based representation along with some metadata (e.g., occupancy map and patch information), which is used to convert the compressed point cloud back into a decompressed point cloud. In some examples, the V-PCC encoder (300) converts the 3D point cloud frame into a geometry image, a texture image, and an occupancy map, and then can encode the geometry image, texture image, and occupancy map into a bitstream using video coding techniques. In general, a geometry image is a 2D image of pixels filled with geometry values ​​associated with points projected onto the pixels, and the pixels filled with the geometry values ​​can be referred to as geometry samples. A texture image is a 2D image of pixels filled with texture values ​​associated with points projected onto the pixels, and the pixels filled with texture values ​​can be referred to as texture samples. An occupancy map is a 2D image of pixels filled with values ​​indicating whether they are occupied or unoccupied by a patch.

[0032]

[0049] The patch generation module (306) can segment the point cloud into a set of patches that may or may not overlap (e.g., a patch is defined as a contiguous subset of the surface described by the point cloud), so that each patch can be described by a depth field with respect to a plane in 2D space. In some embodiments, the patch generation module (306) decomposes the point cloud into a minimum number of patches with smooth boundaries, while also aiming to minimize the reconstruction error.

[0033]

[0050] In some examples, the patch information module (304) may collect patch information indicating the size and shape of the patch. In some examples, the patch information may be packed into an image frame and then encoded by the auxiliary patch information compression module (338) to generate compressed auxiliary patch information.

[0034]

[0051] In some examples, a patch packing module (308) is configured to map the extracted patches onto a two-dimensional (2D) grid while minimizing unused space and ensuring that every M×M (e.g., 16×16) block of the grid is associated with a unique patch. Efficient patch packing can directly impact compression efficiency by minimizing unused space or ensuring temporal consistency.

[0035]

[0052] The geometric image generation module (310) can generate a 2D geometric image associated with the geometry of the point cloud at a given patch location. The texture image generation module (312) can generate a 2D texture image associated with the texture of the point cloud at a given patch location. The geometric image generation module (310) and the texture image generation module (312) utilize the 3D-to-2D mapping calculated during the packing process to store the geometry and texture of the point cloud as images. To better handle the case where multiple points are projected onto the same sample, each patch is projected onto two images, referred to as layers. In one example, the geometric image is represented by a WxH monochrome frame in YUV420-8 bit format. To generate the texture image, the texture generation procedure utilizes the reconstructed / smoothed geometry to calculate the colors that will be associated with the resampled points.

[0036]

[0053] The occupancy map module (314) can generate an occupancy map that describes the padding information at each unit. For example, the occupancy image includes a binary map that indicates for each cell of the grid whether the cell belongs to empty space or to the point cloud. In one example, the occupancy map uses binary information that describes for each pixel whether the pixel is padded or not. In another example, the occupancy map uses binary information that describes for each block of pixels whether the block of pixels is padded or not.

[0037]

[0054] The occupancy map generated by the occupancy map module (314) can be compressed using lossless or non-lossy coding. If lossless coding is used, the entropy compression module (334) is used to compress the occupancy map. If non-lossless coding is used, the video compression module (332) is used to compress the occupancy map.

[0038]

[0055] Note that the patch packing module (308) may leave some empty space between the 2D patches packaged into an image frame. The image padding modules (316) and (318) can fill the empty space (called padding) to generate an image frame suitable for 2D video and image codecs. Image padding is also called background filling, which can fill the unused space with redundant information. In some cases, a good background fill increases the bit rate only minimally and does not introduce significant coding artifacts around the patch boundaries.

[0039]

[0056] The video compression modules (322), (323), and (332) can encode 2D images, such as the padded geometric images, the padded texture images, and the occupancy maps, according to an appropriate video coding standard, such as HEVC, VVC, etc. In one example, the video compression modules (322), (323), and (332) are individual components operating separately. It should be noted that in another example, the video compression modules (322), (323), and (332) can be implemented as a single component.

[0040]

[0057] In some examples, the smoothing module (336) is configured to generate a smoothed image of the reconstructed geometric image. The smoothed image can be provided to the texture image generation module (312). The texture image generation module (312) can then adjust the generation of the texture image based on the reconstructed geometric image. For example, if a patch shape (e.g., a geometric shape) is slightly distorted during encoding and decoding, the distortion can be taken into account in generating the texture image to correct the distortion of the patch shape.

[0041]

[0058] In some embodiments, the group expansion (320) is configured to pad pixels around object boundaries with redundant low-frequency content to improve coding gain as well as the visual quality of the reconstructed point cloud.

[0042]

[0059] A multiplexer (324) can multiplex the compressed geometry images, the compressed texture images, the compressed occupancy maps, and the compressed auxiliary patch information into a compressed bitstream.

[0043]

[0060] 4 shows a block diagram of a V-PCC decoder (400) for decoding a compressed bitstream corresponding to a point cloud frame in some examples. In some examples, the V-PCC decoder (400) can be used in the communication system (100) and the streaming system (200). For example, the decoder (210) can be configured to operate in a manner similar to the V-PCC decoder (400). The V-PCC decoder (400) receives a compressed bitstream and generates a reconstructed point cloud based on the compressed bitstream.

[0044]

[0061] In the example of FIG. 4, the V-PCC decoder (400) includes a demultiplexer (432), video decompression modules (434) and (436), an occupancy map decompression module (438), an auxiliary patch information decompression module (442), a geometric reconstruction module (444), a smoothing module (446), a texture reconstruction module (448), and a color smoothing module (452).

[0045]

[0062] The demultiplexer (432) can receive the compressed bitstream and separate it into compressed texture images, compressed geometric images, compressed occupancy maps, and compressed auxiliary patch information.

[0046]

[0063] The video decompression modules 434 and 436 can decode the compressed images according to an appropriate standard (e.g., HEVC, VVC, etc.) and output decompressed images. For example, the video decompression module 434 can decode a compressed texture image and output a decompressed texture image; the video decompression module 436 can decode a compressed geometry image and output a decompressed geometry image.

[0047]

[0064] The occupancy map decompression module (438) can decode the compressed occupancy map according to an appropriate standard (eg, HEVC, VVC, etc.) and output a decompressed occupancy map.

[0048]

[0065] The auxiliary patch information decompression module (442) can decode the compressed auxiliary patch information according to an appropriate standard (eg, HEVC, VVC, etc.) and output decompressed auxiliary patch information.

[0049]

[0066] A geometric reconstruction module (444) can receive the decompressed geometric image and generate a reconstructed point cloud geometry based on the decompressed occupancy map and the decompressed auxiliary patch information.

[0050]

[0067] The smoothing module (446) can smooth incongruences at the edges of the patches. The smoothing procedure aims to mitigate potential discontinuities that may occur at the patch boundaries due to compression artifacts. In some embodiments, a smoothing filter can be applied to pixels located at the patch boundaries to mitigate distortions that may be caused by compression / decompression.

[0051]

[0068] The texture reconstruction module (448) can determine texture information for the points in the point cloud based on the decompressed texture image and the smoothed geometry.

[0052]

[0069] The color smoothing module (452) can smooth color incongruence. Non-adjacent patches in 3D space are often packed adjacent to each other in 2D video. In some cases, pixel values ​​from non-adjacent patches can be confused by block-based video codecs. The purpose of color smoothing is to reduce visible artifacts that appear at patch boundaries.

[0053]

[0070] 5 shows a block diagram of a video decoder (510) in some examples. The video decoder (510) can be used in the V-PCC decoder (400). For example, the video decompression modules (434) and (436) and the occupancy map decompression module (438) can be configured similarly to the video decoder (510).

[0054]

[0071] The video decoder (510) may include a parser (520) for reconstructing symbols (521) from a compressed image, such as a coded video sequence. These symbol categories include information used to manage the operation of the video decoder (510). The parser (520) may parse / entropy decode a received coded video sequence. The coding of the coded video sequence may follow a video coding technique or standard and may follow a variety of principles, including variable length coding, Huffman coding, arithmetic coding with or without context effects, etc. The parser (520) may extract a set of subgroup parameters for at least one subgroup of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to the group. The subgroup may include a group of pictures (GOP), a picture, a tile, a slice, a macroblock, a coding unit (CU), a block, a transform unit (TU), a prediction unit (PU), etc. The parser (520) may also extract from the coded video sequence information such as transform coefficients, quantization parameter values, motion vectors, etc.

[0055]

[0072] The parser (520) is capable of performing an entropy decoding / parsing process on the video sequence received from the buffer memory to generate symbols (521).

[0056]

[0073] The reconstruction of the symbols (521) may include different units depending on the type of coded video picture or part thereof (e.g., inter and intra pictures, inter and intra blocks, etc.) and other factors. Which units are included and how can be controlled by subgroup control information parsed by the parser (520) from the coded video sequence. The flow of such subgroup control information between the parser (520) and the following units is not depicted for clarity.

[0057]

[0074] Beyond the functional blocks already mentioned, the video decoder (510) may be further conceptually subdivided into a number of functional units as described below. In a practical implementation operating under commercial constraints, many of these units may interact closely with each other and may be at least partially integrated with each other. However, for purposes of describing the disclosed subject matter, the following conceptual subdivision into functional units is adequate.

[0058]

[0075] The first unit is a scalar / inverse transform unit (551), which receives quantized transform coefficients and control information (including the transform to use, block size, quantization factor, quantization scaling matrix, etc.) as symbols (521) from the parser (520). The scalar / inverse transform unit (551) can output blocks containing sample values ​​that can be input to an aggregator (555).

[0059]

[0076] In some cases, the output samples of the scalar / inverse transform (551) may relate to intra-coded blocks: i.e., blocks that do not use prediction information from a previously reconstructed picture, but can use prediction information from a previously reconstructed part of the current picture. Such prediction information may be provided by an intra picture prediction unit (552). In some cases, the intra picture prediction unit (552) generates blocks of the same size and shape of the block being reconstructed using already reconstructed surrounding information retrieved from a current picture buffer (558). The current picture buffer (558) buffers, for example, a partially reconstructed current picture and / or a fully reconstructed current picture. The aggregator (555) adds, possibly on a sample-by-sample basis, the prediction information generated by the intra prediction unit (552) to the output sample information as provided by the scalar / inverse transform unit (551).

[0060]

[0077] In other cases, the output samples of the scalar / inverse transform unit (551) may relate to a block that may be inter-coded and motion compensated. In such a case, the motion compensated prediction unit (553) may access the reference picture memory (557) to retrieve samples used for prediction. After motion compensating the retrieved samples according to the symbols (521) associated with the block, these samples may be added by the aggregator (555) to the output of the scalar / inverse transform unit (551) (in this case called residual samples or residual signals) to generate output sample information. The addresses in the reference picture memory (557) from which the motion compensated prediction unit (553) fetches the prediction samples may be controlled by the motion vectors available to the motion compensated prediction unit (553), for example in the form of symbols (521), which may have X, Y, and reference picture components. Motion compensation may also include interpolation of sample values ​​taken from a reference picture memory (557), motion vector prediction mechanisms, etc., where sub-sample accurate motion vectors are used.

[0061]

[0078] The output samples of the aggregator (555) may be subjected to various loop filtering techniques in a loop filter unit (556). Video compression techniques may include in-loop filter techniques, controlled by parameters contained in the coded video sequence (also called coded video bitstream) and made available to the loop filter unit (556) as symbols (521) from the parser (520), which may be responsive to meta-information obtained during decoding of a previous part of the coded picture or coded video sequence (in decoding order), as well as to previously reconstructed loop filtered sample values.

[0062]

[0079] The output of the loop filter unit (556) may be a sample stream that can be output to a rendering device as well as stored in a reference picture memory (557) for use in future inter-picture prediction.

[0063]

[0080] Once a given coded picture has been fully reconstructed, it can be used as a reference picture for future predictions. For example, once a coded picture corresponding to a current picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) can become part of the reference picture memory (557) and a new current picture buffer can be reallocated before starting the reconstruction of a future coded picture.

[0064]

[0081] The video decoder (510) may perform decoding operations according to a given video compression technique in a standard such as ITU-T Rec. H.265. The coded video sequence may comply with the syntax specified by the video compression technique or standard used in the sense that the coded video sequence conforms to both the syntax of the video compression technique or standard and the profile as documented in the video compression technique or standard. In particular, a profile may select a particular tool from all tools available in the video compression technique or standard as the only tool available under that profile. Also required for compliance is that the complexity of the coded video sequence is within a range defined by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, the maximum frame rate, the maximum reconstruction sample rate (e.g., measured in megasamples per second), the maximum reference picture size, etc. The limits set by the levels may in some cases be further constrained by Hypothetical Reference Decoder (HRD) specifications and metadata for HRD buffer management signaled in the coded video sequence.

[0065]

[0082] 6 shows a block diagram of a video encoder (603) according to one embodiment of the present disclosure. The video encoder (603) can be used in the V-PCC encoder (300) to compress point clouds. In one example, the video compression modules (322) and (323) and the video compression module (332) are configured similarly to the encoder (603).

[0066]

[0083] A video encoder (603) can receive images, such as padded geometric images, padded texture images, etc., and generate compressed images.

[0067]

[0084] According to one embodiment, the video encoder (603) is capable of coding and compressing pictures of a source video sequence into a coded video sequence (compressed pictures) in real time or under some other time constraint required by an application. Imposing an appropriate coding rate is one function of the controller (650). In some embodiments, the controller (650) controls and is operatively coupled to other functional units as described below, the couplings of which are not depicted for clarity. Parameters set by the controller (650) may include rate control related parameters (picture skip, quantizer, lambda value for rate distortion optimization techniques, ...), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) may be configured to have other appropriate functions associated with the video encoder (603) optimized for a particular system design.

[0068]

[0085] In some embodiments, the video encoder (603) is configured to operate in a coding loop. As a simplified explanation, in one example, the coding loop may include a source coder (630) (e.g., responsible for generating symbols such as a symbol stream based on an input picture to be coded and reference pictures) and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to generate sample data in a similar manner to that generated by the (remote) decoder (since any compression between the symbols and the coded video bitstream is lossless in the video compression techniques contemplated by the disclosed subject matter). The reconstructed sample stream (sample data) is input to a reference picture memory (634). Since the decoding of the symbol stream produces bit-exact results independent of the location of the decoder (local or remote), the contents in the reference picture memory (634) are also bit-exact between the local and remote encoders. In other words, the predictor of the encoder "sees" exactly the same sample values ​​as the reference picture samples that the decoder would "see" if it were to use prediction during decoding. This basic principle of reference picture synchronization (and the resulting drift if synchronicity cannot be maintained, e.g. due to channel errors) is used in several related techniques as well.

[0069]

[0086] The operation of the "local" decoder (633) may be the same as that of a "remote" decoder, such as the video decoder (510) already described in detail above in connection with Figure 5. However, with brief reference to Figure 5 as well, because symbols are available and the encoding / decoding of symbols into a coded video sequence by the entropy coder (645) and parser (520) may be lossless, the entropy decoding portion of the video decoder (510), including the parser (520), may not be fully implemented in the local decoder (633).

[0070]

[0087] During operation, the source coder (630) may, in some examples, perform motion-compensated predictive coding, which predictively codes an input picture with reference to one or more previously coded pictures from a video sequence, designated as “reference pictures.” In this manner, the coding engine (632) codes differences between pixel blocks of the input picture and pixel blocks of the reference pictures that may be selected as predictive references for the input picture.

[0071]

[0088] The local video decoder (633) can decode the coded video data of pictures that can be designated as reference pictures based on the symbols generated by the source coder (630). The operation of the coding engine (632) can advantageously be a non-lossless process. When the coded video data is decoded in a video decoder (not shown in FIG. 6), the reconstructed video sequence can typically be a replica of the source video sequence with some errors. The local video decoder (633) can repeat the decoding process that can be performed by the video decoder on the reference pictures, causing the reconstructed reference pictures to be stored in the reference picture cache (634). In this way, the video encoder (603) can locally store copies of reconstructed reference pictures that have common content as reconstructed reference pictures obtained by the far-end video decoder (assuming there are no transmission errors).

[0072]

[0089] The predictor (635) can perform a prediction search for the coding engine (632). That is, for a new picture to be coded, the predictor (635) can search the reference picture memory (634) for sample data (as candidate reference pixel blocks) or predefined metadata (reference picture motion vectors, block shapes, etc.), which may serve as suitable prediction references for the new picture. The predictor (635) can operate on a sample block-pixel block basis to find suitable prediction references. In some cases, as determined by the search results obtained by the predictor (635), the input picture may have prediction references drawn from multiple reference pictures stored in the reference picture memory (634).

[0073]

[0090] The controller (650) may manage the coding operations of the source coder (630), including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0074]

[0091] All outputs of the aforementioned functional units can be subjected to entropy coding in an entropy coder (645), which converts the symbols produced by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc.

[0075]

[0092] The controller (650) can manage the operation of the video encoder (603). During coding, the controller (650) can assign a particular coded picture type to each coded picture, which can affect the coding technique that can be applied to each picture. For example, pictures may often be designated as one of the following picture types:

[0093] An Intra picture (I-picture) is one that can be coded and decoded without using any other picture in a sequence as a source of prediction. Some video codecs allow different types of Intra pictures, including, for example, Independent Decoder Refresh ("IDR") pictures. Those skilled in the art are aware of these variations of I-pictures and their respective uses and characteristics.

[0076]

[0094] A predictive picture (P-picture) is one that can be coded and decoded using intra- or inter-prediction, which uses at most one motion vector and reference index to predict the sample values ​​of each block.

[0077]

[0095] Bidirectionally predicted pictures (B-pictures) are those that can be coded and decoded using intra- or inter-prediction, which uses at most two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multiple predicted pictures can use more than two reference pictures and associated metadata for the reconstruction of a block.

[0078]

[0096] A source picture is usually spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples) and can be coded block by block. Blocks can be predictively coded with reference to other (already coded) blocks as determined by the coding assignment applied to the respective picture. For example, blocks of I pictures can be non-predictively coded or they can be predictively coded with reference to already coded blocks of the same picture (spatial or intra prediction). Pixel blocks of P pictures can be predictively coded with spatial or temporal prediction with reference to one previously coded reference picture. Blocks of B pictures can be predictively coded with spatial or temporal prediction with reference to one or two previously coded reference pictures.

[0079]

[0097] The video encoder (603) may perform coding operations according to a given video coding technique or standard, such as ITU-T Rec. H.265, in which the video encoder (603) may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancies in the input video sequence. The coded video data may thus conform to a syntax specified by the video coding technique or standard being used.

[0080]

[0098] Video may be in the form of multiple source pictures (images) in a time sequence. Intra-picture prediction (often abbreviated as intra prediction) exploits spatial correlation in a given picture, while inter-picture prediction exploits correlation (temporal or other) between pictures. In one example, a particular picture under encoding / decoding, referred to as the current picture, is divided into blocks. If a block in the current picture is similar to a reference block in a reference picture that was previously coded in the video and is still buffered, the block in the current picture can be coded by a vector called a motion vector. A motion vector points to a reference block in the reference picture and can have a third dimension that identifies the reference picture if multiple reference pictures are used.

[0081]

[0099] In some embodiments, bi-prediction techniques may be used for inter-picture prediction. According to bi-prediction techniques, two reference pictures are used, such as a first reference picture and a second reference picture, both preceding in decoding order (but potentially past and future, respectively, in display order) a current picture. A block in the current picture may be coded with a first motion vector that points to a first reference block in the first reference picture and a second motion vector that points to a second reference block in the second reference picture. A block may be predicted by a combination of the first and second reference blocks.

[0082]

[0100] Furthermore, to improve coding efficiency, it is possible to use merge mode techniques for inter-picture prediction.

[0083]

[0101] According to some embodiments of the present disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed on a block-by-block basis. For example, according to the HEVC standard, pictures in a sequence of video pictures are partitioned into coding tree units (CTUs) for compression, and the CTUs in a picture have the same size, such as 64x64 pixels, 32x32 pixels, or 16x16 pixels. In general, a CTU includes three coding tree blocks (CTBs), one luma CTB and two chroma CTBs. Each CTU can be recursively quadtree partitioned into one or more coding units (CUs). For example, a CTU of 64x64 pixels can be partitioned into one CU of 64x64 pixels, four CUs of 32x32 pixels, or 16 CUs of 16x16 pixels. In one example, each CU is analyzed to determine a prediction type of the CU, such as an inter prediction type or an intra prediction type. A CU is divided into one or more Prediction Units (PUs) depending on temporal and / or spatial predictability. In general, each PU includes a luma prediction block (PB) and two chroma PBs. In one embodiment, prediction operations in coding (encoding / decoding) are performed in units of prediction blocks. Taking a luma prediction block as an example of a prediction block, the prediction block includes a matrix of values ​​(e.g., luma values) for pixels, such as 8x8 pixels, 16x16 pixels, 8x16 pixels, 16x8 pixels, etc.

[0084]

[0102] FIG. 7 illustrates a block diagram of a G-PCC encoder (700) in some examples. The G-PCC encoder (700) may be configured to receive point cloud data and compress the point cloud data to generate a bitstream carrying the compressed point cloud data. In one embodiment, the G-PCC encoder (700) may include a position quantization module (710), a duplicate point removal module (712), an octree encoding module (730), an attribute transfer module (720), a level of detail (LOD) generation module (740), an attribute prediction module (750), a residual quantization module (760), an arithmetic coding module (770), an inverse residual quantization module (780), an append module (781), and a memory (790) for storing reconstructed attribute values.

[0085]

[0103] As shown, an input point cloud (701) may be received at a G-PCC encoder (700). Locations (e.g., 3D coordinates) of the point cloud (701) are provided to a quantization module (710), which is configured to quantize the coordinates to generate quantized locations. A duplicate point removal module (712) is configured to receive the quantized locations and perform a filtering process to identify and remove duplicate points. An octree encoding module (730) is configured to receive the filtered locations from the duplicate point removal module (712) and perform an octree-based encoding process to generate a sequence of occupancy codes that describe the voxels of a 3D grid. The occupancy codes are provided to an arithmetic coding module (770).

[0086]

[0104] The attribute transfer module (720) is configured to receive attributes of the input point cloud and is configured to perform an attribute transfer process to determine an attribute value for each voxel when multiple attribute values ​​are associated with each voxel. The attribute transfer process can be performed on the reordered points output from the octree encoding module (730). The transferred attributes are provided to an attribute prediction module (750). The LOD generation module (740) is configured to operate on the reordered points output from the octree encoding module (730) and reorganize the points into different LODs. The LOD information is provided to the attribute prediction module (750).

[0087]

[0105] The attribute prediction module (750) processes the points according to the LOD-based order indicated by the LOD information from the LOD generation module (740). The attribute prediction module (750) generates an attribute prediction for the current point based on the reconstructed attributes for a set of points in the vicinity of the current point stored in the memory (790). A prediction residual can then be obtained based on the original attribute values ​​received from the attribute transfer module (720) and the locally generated attribute prediction. If a candidate index is used in each attribute prediction process, an index corresponding to the selected prediction candidate may be provided to the arithmetic coding module (770).

[0088]

[0106] The residual quantization module (760) is configured to receive the prediction residual from the attribute prediction module (750) and perform quantization to generate a quantized residual, which is provided to the arithmetic coding module (770).

[0089]

[0107] The inverse residual quantization module (780) is configured to receive the quantized residual from the residual quantization module (760) and generate a reconstructed prediction residual by performing the inverse of the quantization operation performed by the residual quantization module (760). The summation module (781) is configured to receive the reconstructed prediction residual from the inverse residual quantization module (780) and the respective attribute predictions from the attribute prediction module (750). By combining the reconstructed prediction residual and the attribute predictions, reconstructed attribute values ​​are generated and stored in the memory (790).

[0090]

[0108] The arithmetic coding module (770) is configured to receive the occupation codes, the candidate indexes (if used), the quantized residuals (if generated), and other information, and to perform entropy coding to further compress the received values ​​or information, thereby generating a compressed bitstream (702) carrying the compressed information. The bitstream (702) may be transmitted or otherwise provided to a decoder that decodes the compressed bitstream, or may be stored in a storage device.

[0091]

[0109] 8 shows a block diagram of a G-PCC decoder (800) according to one embodiment. The G-PCC decoder (800) may be configured to receive a compressed bitstream and perform point cloud data decompression to decompress the bitstream and generate decoded point cloud data. In an embodiment, the G-PCC decoder (800) may include an arithmetic decoding module (810), an inverse residual quantization module (820), an octree decoding module (830), an LOD generation module (840), an attribute prediction module (850), and a memory (860) for storing reconstructed attribute values.

[0092]

[0110] As shown, a compressed bitstream (801) may be received at an arithmetic decoding module (810). The arithmetic decoding module (810) is configured to decode the compressed bitstream (801) to obtain quantized residuals (if generated) and occupation codes of the point cloud. The octree decoding module (830) is configured to determine reconstructed positions of the points in the point cloud according to the occupation codes. The LOD generation module (840) is configured to reorganize the points into different LODs and determine an order based on the LODs based on the reconstructed positions. The inverse residual quantization module (820) is configured to generate reconstructed residuals based on the quantized residuals received from the arithmetic decoding module (810).

[0093]

[0111] The attribute prediction module (850) is configured to perform an attribute prediction process to determine attribute predictions for the points according to an order based on the LOD. For example, an attribute prediction for a current point can be determined based on reconstructed attribute values ​​of points in the vicinity of the current point stored in the memory (860). In some examples, the attribute predictions can be combined with the respective reconstructed residuals to generate reconstructed attributes for the current point.

[0094]

[0112] In one example, the sequence of reconstructed attributes generated from the attribute prediction module (850) along with the reconstructed positions generated from the octree decoding module (830) correspond to the decoded point cloud (802) output from the G-PCC decoder (800). Additionally, the reconstructed attributes are also stored in memory (860) and can be subsequently used to derive attribute predictions for future points.

[0095]

[0113] In various embodiments, the encoder (300), the decoder (400), the encoder (700), and / or the decoder (800) may be implemented in hardware, software, or a combination thereof. For example, the encoder (300), the decoder (400), the encoder (700), and / or the decoder (800) may be implemented by a processing circuit, such as one or more integrated circuits (ICs), operating with or without software, such as application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), etc. In another example, the encoder (300), the decoder (400), the encoder (700), and / or the decoder (800) may be implemented as software or firmware that includes instructions stored on a non-volatile (or non-transitory) computer-readable storage medium. The instructions, when executed by a processing circuit, such as one or more processors, cause the processing circuit to perform the functions of the encoder (300), the decoder (400), the encoder (700), and / or the decoder (800).

[0096]

[0114] It should be noted that the attribute prediction modules (750) and (850) configured to implement the attribute prediction techniques disclosed herein may be included in other decoders or encoders that may have similar or different structures to those shown in Figures 7 and 8. Additionally, the encoder (700) and decoder (800) may be included in the same device or in separate devices in various examples.

[0097]

[0115] According to some aspects of the present disclosure, mesh compression may use a coding tool different from the PCC coding tool, or may use a PCC coding tool such as a PCC (e.g., G-PCC, V-PCC) encoder, a PCC (e.g., G-PCC, V-PCC) decoder, etc.

[0098]

[0116] An object mesh (also called a mesh model or mesh frame) may include polygons that describe the surface of an object. Each polygon may be defined by the polygon's vertices in 3D space and the edges that connect the vertices to the polygon. The information of how the vertices are connected is called the connectivity information. In some examples, an object mesh is formed by connected triangles that describe the object's surface. Two triangles that share an edge may be referred to as two connected triangles. In some other examples, an object mesh is formed by connected quadrilaterals. Two quadrilaterals that share an edge may be referred to as two connected quadrilaterals. Note that a mesh may be formed by other polygons as appropriate.

[0099]

[0117] In some examples, meshes may also include attributes such as color, normals, etc., associated with vertices. Attributes may be associated with the surface of a mesh by utilizing mapping information that parameterizes the mesh with a 2D attribute map. The mapping information is described by a set of parametric coordinates, usually called UV coordinates or texture coordinates, associated with the mesh vertices. 2D attribute maps (sometimes referred to as texture maps) are used to store high-resolution attribute information such as texture, normals, displacements, etc. Such information may be used for various purposes such as texture mapping and shading. In some embodiments, meshes may include geometry information, connectivity information, mapping information, vertex attributes, and components (or components) referred to as attribute maps.

[0100]

[0118] In some embodiments, a mesh may include components called geometric information, connectivity information, mapping information, vertex attributes, and attribute maps. In some examples, the geometric information is described by a set of 3D positions associated with the vertices of the mesh. In one example, (x,y,z) coordinates may be used to describe the 3D positions of the vertices, also referred to as 3D coordinates. In some examples, the connectivity information includes a set of vertex indices that describe how to connect the vertices to create a 3D surface. In some examples, the mapping information describes how to map the mesh surface to a planar 2D region. In one example, the mapping information is described by a set of UV parametric / texture coordinates (u,v) associated with the mesh vertices along with the connectivity information. In some examples, the vertex attributes include scalar or vector attribute values ​​associated with the mesh vertices. In some examples, the attribute maps include attributes associated with the mesh surface that are stored as a 2D image / video. In one example, the mapping between a video (e.g., a 2D image / video) and the mesh surface is defined by the mapping information.

[0101]

[0119] According to one aspect of the disclosure, a technique called UV mapping or mesh parameterization is used to map the surface of a mesh in a 3D domain to a 2D domain. In some examples, the mesh is partitioned into patches in the 3D domain. A patch is a contiguous subset of the mesh with a boundary formed by boundary edges. A boundary edge of a patch is an edge that belongs to only one polygon of the patch and is not shared by two adjacent polygons of the patch. In some examples, the vertices of the boundary edge of a patch are called boundary vertices of the patch, and the non-boundary vertices of the patch are called interior vertices of the patch.

[0120] In some examples, a mesh of an object is formed by connected triangles, and the mesh can be partitioned into patches, where each patch is a subset of the connected triangles. A boundary edge of a patch is an edge that belongs to only one triangle of the patch and is not shared by adjacent triangles of the patch. In some examples, the vertices of a boundary edge of a patch are called boundary vertices of the patch, and the non-boundary vertices of the patch are called interior vertices of the patch.

[0102]

[0121] According to one aspect of the disclosure, in some examples, the patches are each parameterized into a 2D shape (also called a UV patch). The 2D shapes can be packed (e.g., oriented and arranged) into a map, also referred to as an atlas, in some examples. In some examples, the map can be further processed using 2D image or video processing techniques.

[0103]

[0122] In one example, a UV mapping technique generates a UV atlas (also called a UV map) and one or more texture atlases (also called texture maps) in 2D that correspond to patches of a 3D mesh. The UV atlas includes assigning 3D vertices of the 3D mesh to 2D points in the 2D domain (e.g., quads). The UV atlas is a mapping between the coordinates of the 3D surface and the coordinates of the 2D domain. In one example, a point in the UV atlas at 2D coordinates (u,v) has a value formed by the coordinates (x,y,z) of the vertex in the 3D domain. In one example, the texture atlas includes color information for the 3D mesh. For example, a point in the texture atlas at 2D coordinates (u,v) (which has a 3D value of (x,y,z) in the UV atlas) has a color that specifies the color attributes of the point at (x,y,z) in the 3D domain. In some cases, coordinates in the 3D domain (x,y,z) are referred to as 3D coordinates, or xyz coordinates, and 2D coordinates (u,v) are referred to as uv coordinates, or UV coordinates.

[0104]

[0123] According to some aspects of the present disclosure, mesh compression can be performed by representing the mesh using one or more 2D maps (also referred to in some instances as 2D atlases) and then encoding the 2D maps using an image or video codec. Various techniques can be used to generate the 2D maps.

[0105]

[0124] Figure 9 shows a diagram illustrating the mapping of a 3D mesh (910) to a 2D atlas (920) in some examples. In the example of Figure 9, the 3D mesh (910) includes four vertices 1-4 that form four patches AD. Each patch has a set of attribute information associated with the vertices. For example, Patch A is formed by vertices 1, 2, and 3 connected into a triangle; Patch B is formed by vertices 1, 3, and 4 connected to a triangle; Patch C is formed by vertices 1, 2, and 4 connected to a triangle; and Patch D is formed by vertices 2, 3, and 4 which are connected into a triangle. In some examples, vertices 1, 2, 3, and 4 may have respective attributes, and the triangle formed by vertices 1, 2, 3, and 4 may have respective attributes.

[0106]

[0125] In one example, the 3D patches A, B, C, and D are mapped to a 2D domain, such as a 2D atlas (920), also called a map (920). For example, Patch A is mapped to a 2D shape (also called a UV patch) A' in the map (920), Patch B is mapped to a 2D shape (also called a UV patch) B' in the map (920), Patch C is mapped to a 2D shape (also called a UV patch) C' in the map (920), Patch D is mapped to a 2D shape (also called a UV patch) D' in the map (920). In some examples, coordinates in the 3D domain are referred to as (x,y,z) coordinates, and coordinates in a 2D domain, such as the map (920), are referred to as UV coordinates. Vertices in a 3D mesh can have corresponding UV coordinates in the map (920).

[0107]

[0126] The map (920) may be a geometric map having geometric information, or it may be a texture map having color, normal, textile, or other attribute information, or it may be an occupancy map having occupancy information.

[0108]

[0127] It should be noted that although each patch is represented by a triangle in the example of Figure 9, a patch may include any suitable number of vertices that are connected to form a contiguous subset of the mesh. In some examples, the vertices of a patch are connected into triangles. It should be noted that the vertices within a patch may be connected using other suitable shapes.

[0109]

[0128] In one example, the geometric information of the vertices can be stored in a 2D geometric map, which stores the (x,y,z) coordinates of the sampling points in the corresponding points of the 2D geometric map. For example, a point in the 2D geometric map at a (u,v) location has three-component vector values ​​that correspond respectively to the x,y,z values ​​of the corresponding sampling point in the 3D mesh.

[0110]

[0129] According to one aspect of the present disclosure, areas within the map may not be fully occupied. For example, in FIG. 9, the areas outside the 2D shapes A', B', C', D' are undefined. Sample values ​​of the areas outside the 2D shapes A', B', C', D' after decoding can be discarded. In some cases, the occupancy map is used to store some additional information for each pixel, for example a binary value, identifying whether the pixel belongs to the patch or is undefined.

[0111]

[0130] According to one aspect of the present disclosure, a dynamic mesh is a mesh in which at least one of its components (such as geometry information, connectivity information, mapping information, vertex attributes, and attribute maps) changes over time. A dynamic mesh can be described by a sequence of meshes (also called mesh frames). Because dynamic meshes can contain a significant amount of information that changes over time, dynamic meshes can require large amounts of data. Mesh compression techniques can allow efficient storage and transmission of media content in mesh representations.

[0112]

[0131] In some examples, a dynamic mesh may have constant connectivity information, time-varying geometry, and time-varying vertex attributes. In some examples, a dynamic mesh can have time-varying connectivity information. In one example, digital content creation tools typically generate dynamic meshes with time-varying attribute maps and time-varying connectivity information. In some examples, volumetric acquisition techniques are used to generate dynamic meshes. Volumetric acquisition techniques can generate dynamic meshes with time-varying connectivity information, especially under real-time constraints.

[0113]

[0132] Several techniques are used for mesh compression. In some cases, UV atlas sampling and V-PCC can be used for mesh compression. For example, the UV atlas is sampled on a regular grid to generate a geometric image with regular grid samples. The connectivity of the regular grid samples can be estimated. The regular grid samples can be considered as points in a point cloud and therefore can be coded using a PCC codec, such as the V-PCC codec.

[0114]

[0133] In some other examples, vertex reordering techniques are used for mesh compression. The vertices of a mesh are reordered according to certain rules to increase the correlation of nearby vertices, and thus can increase the coding efficiency using predictive coding methods. In one example, the vertices are reordered to increase the correlation of adjacent vertices in a sequence. In another example, the vertices of a mesh are reordered and rearranged in a two-dimensional (2D) frame to increase the correlation of adjacent vertices in a local region of the 2D frame, and then the 2D frame can be more efficiently coded by an image / video codec, for example, by using intra prediction techniques. In another example, the vertices of different mesh frames are reordered and rearranged within a 2D frame to increase the correlation between the 2D frames, and then the 2D frame can be more efficiently coded by a video codec, for example, by using inter prediction techniques.

[0115]

[0134] Figures 10A-10E show an example of the use of vertex reordering for mesh compression. Figure 10A shows a diagram of an input mesh (1005) (e.g., an original mesh). The input mesh (1005) includes connected triangles that represent the surface of an object. Each triangle is defined by vertices and edges that connect the vertices to the triangle. The input mesh (1005) includes vertices 0 through 8 that are connected to triangles, as shown in Figure 10A.

[0116]

[0135] 10B shows a diagram of the vertex traversal order (1015) in some examples. The vertex traversal order (1015) is shown by an arrowed line that traverses vertex 0, vertex 1, vertex 2, vertex 3, vertex 4, vertex 5, vertex 6, vertex 7, and vertex 8 in order.

[0117]

[0136] Then, attributes of the vertices, such as the 3D (geometric) coordinates of the vertices, the UV coordinates of the vertices, and other attributes of the vertices, may be sorted into an array according to the vertex traversal order (1015). For example, the 3D coordinates of the vertices may be sorted into an array of 3D coordinates in the vertex traversal order (1015), and the UV coordinates of the vertices may be sorted into an array of UV coordinates in the vertex traversal order (1015). The array may be a 1D array or may be a 2D array.

[0118]

[0137] In some examples, the vertex attributes are reordered and reshaped according to the raster scan lines into a 2D array forming a 2D image. The 2D image can be coded by a predictive coding technique such as an image or video codec. In an example of a dynamic mesh including a sequence of mesh frames, the attributes of the dynamic mesh can be reordered to form a sequence of 2D images, which in one example can be coded by a video codec.

[0119]

[0138] FIG. 10C shows a diagram illustrating a raster scan line (1025) used to reshape vertex attributes into a 2D array (1035). The raster scan line (1025) is shown as an arrowed line. The 2D array (1035) may also be referred to as a 2D map or a 2D image. In one example, the attributes of a vertex are stored in an entry in the 2D array (1035) corresponding to that vertex, the entry being a pixel in the 2D image, and the attribute value may be considered as color information for the pixel. In one example, the 2D array (1035) may be a 3D coordinate map. In another example, the 2D array (1035) may be a UV coordinate map. The 2D array (1035) may be encoded into a bitstream to carry the mesh frame.

[0120]

[0139] In some cases, connectivity information (e.g., how vertices are connected to triangle edges) is not explicitly encoded in the bitstream carrying the mesh frame, so at the decoder side, the decoder can decode the vertex attributes (e.g., 3D coordinates, uv coordinates, etc.) from, for example, a 2D map and reconstruct the vertices.

[0121]

[0140] 10D shows a diagram illustrating reconstructed vertices 0'-8' in some examples. For example, the coordinates can be decoded from the bitstream, and then the reconstructed vertices 0'-8' can be generated according to the decoded coordinates.

[0122]

[0141] In some examples, the connectivity information is not explicitly encoded in the bitstream, and the edges for connecting the reconstructed vertices 0'-8' are inferred according to a connectivity infer rule. The connectivity infer rule can infer the connectivity from the decoded 3D coordinates (xyz coordinates) and / or UV coordinates (uv coordinates) at the decoder side. In one example, the connectivity infer rule is properly established at the decoder side. Once all the vertices are decoded, the adjacent vertices are connected according to the connectivity infer rule. The decoder does not need to decode the connectivity information from the bitstream.

[0123]

[0142] FIG. 10E shows a diagram illustrating the inferred connectivity information (edges) that can connect the reconstructed vertices 0'-8' into triangles to form the reconstructed mesh (1095).

[0124]

[0143] As shown by Figures 10A and 10E, the connectivity (edges) of vertices in the reconstructed mesh (1095) may differ from the original input mesh (1005). The difference in connectivity may cause the reconstructed mesh to lose subjective quality in some examples. In accordance with another aspect of the present disclosure, estimating connectivity at the decoder side may be time consuming and computationally expensive in terms of complexity.

[0125]

[0144] Aspects of the present disclosure provide techniques for explicitly coding connectivity information of a mesh frame in a bitstream carrying the mesh frame. In some examples, the connectivity information is provided in the form of polygon faces. Each polygon face is defined by a set of vertices that are connected to form edges of the polygon face. In one example, the connectivity information is provided in the form of triangles (also called triangle faces), where each triangle is defined by three vertices that are connected to form three edges of the triangle. According to one aspect of the present disclosure, the connectivity information of a mesh frame can be reorganized as connectivity attributes of each of the vertices.

[0126]

[0145] According to one aspect of the present disclosure, the connectivity attribute of a vertex may be any suitable metric that represents connectivity information at a vertex. In one example, the connectivity attribute of a vertex includes a valence value that is the number of edges at the vertex. For example, the valence value of vertex 3 in the input mesh (1005) is 3, the valence value of vertex 5 in the input mesh (1005) is 4, the valence value of vertex 8 in the input mesh (1005) is 6, and so on.

[0127]

[0146] In another example, an algorithm called an Edgebreaker algorithm can traverse the vertices and mark each vertex with a pattern index that identifies connectivity patterns, such as interior vertices, border vertices, and other patterns, for connectivity compression. Thus, the connectivity attributes of a vertex can include the pattern index marked by the Edgebreaker algorithm.

[0128]

[0147] In some cases, the connectivity attributes of a vertex can be reordered and reshaped as other attributes of the vertex and explicitly coded in the bitstream carrying the mesh.

[0129]

[0148] 11 shows a diagram of a framework (1100) for mesh compression in accordance with some embodiments of the present disclosure. The framework (1100) includes a mesh encoder (1110) and a mesh decoder (1150). The mesh encoder (1110) encodes an input mesh (1105) (or a mesh frame in the case of dynamic mesh processing) into a bitstream (1145), and the mesh decoder (1150) decodes the bitstream (1145) to generate a reconstructed mesh (1195) (or a reconstructed mesh frame in the case of dynamic mesh processing).

[0130]

[0149] The mesh encoder (1110) may be any suitable device, such as a computer, a server computer, a desktop computer, a laptop computer, a tablet computer, a smartphone, a gaming device, an AR device, a VR device, etc. The mesh decoder (1150) may be any suitable device, such as a computer, a server computer, a desktop computer, a laptop computer, a tablet computer, a smartphone, a gaming device, an AR device, a VR device, etc. The bitstream (1145) may be transmitted from the mesh encoder (1110) to the mesh decoder (1150) via any suitable communication network (not shown).

[0131]

[0150] In the example of Figure 11, the mesh encoder (1110) includes a vertex reordering module (1120), a 1D / 2D encoder (1130), and an auxiliary data encoder (1140) coupled together. The vertex reordering module (1120) receives an input mesh (1105) and performs reordering to generate reordered vertex mesh information (1125). In some examples, the input mesh (1105) includes 3D position information of the mesh's vertices in their original order, connectivity information in the form of polygonal faces, mapping information that parameterizes the mesh in 2D (e.g., a UV atlas), and other 2D attribute maps (e.g., a 2D color map). The vertex reordering module (1120) can determine a vertex traversal order for the vertices and reorder and / or rearrange the vertices according to the vertex traversal order to increase attribute correlation of adjacent vertices (e.g., attribute correlation of adjacent vertices in a sequence (e.g., a 1D array), attribute correlation of adjacent vertices within a local region in a 2D array, attribute correlation of adjacent frames).

[0132]

[0151] In one example, the vertex reordering module (1120) can rearrange the connectivity information of the mesh frame in the form of a connectivity attribute (also referred to as an original connectivity attribute) of each of the vertices. In one example, the connectivity attribute of each vertex can include a pattern index indicating a connectivity pattern marked by an edge breaker algorithm for connectivity compression. In another example, the connectivity attribute of each vertex can include a value value indicating the number of edges connected to the vertex.

[0133]

[0152] The vertex reordering module (1120) outputs mesh information of reordered vertices (1125). The mesh information of the reordered vertices (1125) includes vertex information for the vertices in the reordered sequence (e.g., 1D). For example, the vertex information for each vertex may include various attributes such as 3D spatial information within the mesh (e.g., xyz coordinates), mapping information to 2D (e.g., uv coordinates), color information (e.g., RGB values), and connectivity attributes.

[0134]

[0153] In another example, the reordered vertex mesh information (1125) includes vertex information for the vertices in the form of a 2D image, for example, the reordered vertices may be rearranged (e.g., based on a raster scan line, etc.) into 2D samples (pixels), and the reordered vertex mesh information may form one or more 2D images, such as a 2D image of the 3D coordinates of the vertices, a 2D image of the UV coordinates of the vertices, a 2D image of the vertex connection attributes, etc.

[0135]

[0154] The 1D / 2D encoder (1130) is configured to encode the reordered vertex mesh information (1125) into a bitstream (1145). If the reordered vertex mesh information (1125) includes vertex information for vertices in a reordered sequence (1D), the 1D / 2D encoder (1130) may encode the vertex information for the vertices in the reordered sequence using a 1D encoding technique. If the reordered vertex mesh information (1125) includes a 2D image, the 1D / 2D encoder (1130) may encode the 2D image using image encoding and / or video encoding techniques (e.g., using an image codec or a video codec).

[0136]

[0155] The vertex reordering module (1120) also generates auxiliary data (1127) including auxiliary information. The auxiliary data encoder (1140) receives the auxiliary data (1127) and encodes the auxiliary data (1127) into a bitstream (1145). For example, the vertex reordering module (1120) may reorder the vertices by patch. The vertex reordering module (1120) may provide a value indicative of the number of vertices in each patch in the auxiliary data (1127). Further, in one example, the vertex reordering module (1120) may reorder the boundary vertices before the non-boundary vertices for each patch. The vertex reordering module (1120) may provide a value indicative of the number of boundary vertices in each patch in the auxiliary data (1127). In another example, the vertex reordering module (1120) may traverse the vertices according to an edge breaker algorithm and provide a signal indicative of the edge breaker algorithm in the auxiliary data (1127).

[0137]

[0156] In the example of FIG. 11, the bitstream (1145) is provided to a mesh decoder (1150). The mesh decoder (1150) includes a 1D / 2D decoder (1160), an auxiliary data decoder (1170), and a mesh reconstruction module (1190) coupled together as shown in FIG. 11. In one example, the 1D / 2D decoder (1160) corresponds to the 1D / 2D encoder (1130) and is capable of decoding a portion of the bitstream (1145) encoded by the 1D / 2D encoder (1130) to generate decoded information (1165). In one example, the decoded information (1165) includes a decoded connectivity attribute map and other decoded attribute maps, such as a decoded 3D coordinate map, a decoded uv coordinate map, a color map, etc.

[0138]

[0157] In the example of Figure 11, the auxiliary data decoder (1170) corresponds to the auxiliary data encoder (1140) and is capable of decoding a portion of the bitstream (1145) encoded by the auxiliary data encoder (1140) to generate decoded auxiliary data (1175).

[0139]

[0158] In the example of Figure 11, the decoded information (1165) and the decoded auxiliary data (1175) are provided to a mesh reconstruction module (1190), which generates a reconstructed mesh (1195) based on the decoded information (1165) and the decoded auxiliary data (1175).

[0140]

[0159] It should be noted that each of the components in the mesh encoder (1110), such as the vertex reordering module (1120), the auxiliary data encoder (1140), and the 1D / 2D encoder (1130), may be implemented by a variety of technologies. In one example, the components are implemented by integrated circuits. In another example, the components are implemented using software capable of being executed by one or more processors.

[0141]

[0160] It should be noted that each of the components in the mesh decoder (1150), such as the 1D / 2D decoder (1160), the auxiliary data decoder (1170), and the mesh reconstruction module (1190), may be implemented by a variety of technologies. In one example, the components are implemented by integrated circuits. In another example, the components are implemented using software that may be executed by one or more processors.

[0142]

[0161] 12A-12D show examples of explicitly coding mesh frame connectivity attributes in a bitstream carrying the mesh frame, according to some embodiments of the present disclosure. FIG. 12A shows a diagram of an input mesh (1205). The input mesh (1205) includes connected triangles that represent the surface of an object. Each triangle is defined by vertices and edges that connect the vertices to the triangle. The input mesh (1205) includes vertices 0 through 8 that are connected to triangles, as shown in FIG. 12A.

[0143]

[0162] 12B shows a diagram of the vertex traversal order (1215), which is indicated by an arrowed line traversing vertex 0, vertex 1, vertex 2, vertex 3, vertex 4, vertex 5, vertex 6, vertex 7, and vertex 8, in that order.

[0144]

[0163] Then, attributes of the vertices, such as the 3D (geometric) coordinates of the vertices, the UV coordinates of the vertices, the connection attributes of the vertices, and other attributes of the vertices, may be sorted into arrays according to the vertex traversal order (1215). For example, the 3D coordinates of the vertices may be sorted into an array of 3D coordinates in the vertex traversal order (1215), the UV coordinates of the vertices may be sorted into an array of UV coordinates in the vertex traversal order (1215), and the connection attributes of the vertices may be sorted into an array of connection attributes in the vertex traversal order (1215). The arrays may be 1D arrays or may be 2D arrays.

[0145]

[0164] In some examples, the vertex attributes are reordered and reshaped according to the raster scan lines into a 2D array forming a 2D image. The 2D image can be coded by a predictive coding technique such as an image or video codec. In an example of a dynamic mesh including a sequence of mesh frames, the attributes of the dynamic mesh can be reordered to form a sequence of 2D images, which in one example can be coded by a video codec.

[0146]

[0165] FIG. 12C shows a diagram illustrating raster scans (1225) used to reshape vertex attributes into 2D arrays such as 2D array (1245) and 2D array (1235). Raster scans (1225) are shown as arrowed lines. The 2D arrays (1235) and (1245) may also be referred to as 2D maps or 2D images. In one example, a connection attribute for a vertex is stored in an entry in the 2D array (1245) corresponding to that vertex, the entry being a pixel in the 2D image, and the connection attribute is color information for the pixel. In one example, another attribute, such as a 3D coordinate, UV coordinate, for the vertex is stored in an entry in the 2D array (1235) corresponding to that vertex, the entry being a pixel in the 2D image, and the attribute is color information for the pixel. In one example, the 2D array (1235) may be a 3D coordinate map. In another example, the 2D array (1235) may be a UV coordinate map. The 2D array (1235) and the 2D array (1245) can be encoded into a bitstream to carry the mesh frame.

[0147]

[0166] Note that in the example of FIG. 12C, the connectivity information (e.g., how vertices are connected to triangle edges) is reorganized in the form of vertex connectivity attributes, and the vertex connectivity attributes are explicitly encoded in the bitstream for carrying the mesh frame. In some examples, the vertices of an original mesh, such as an input mesh (1205), are traversed, and each vertex is marked with either a pattern index or a value value, which can represent the connectivity information of the vertex. For example, an edge breaker algorithm can be used to determine a vertex traversal order, such as a vertex traversal order (1215), to traverse the vertices, and the edge breaker algorithm can mark each vertex with a pattern index. The vertex connectivity attributes are sorted according to the vertex traversal order, and then stored in a 2D map (M), such as a 2D array (1245). org The connectivity map may be reshaped into another 2D map, such as the 2D array (1235), where each value at a pixel location in the connectivity map (e.g., 2D array (1245)) is the corresponding pattern index or value value of the vertices that are sorted and reshaped at the pixel location. Other attributes of the vertices may be sorted according to the vertex traversal order and then reshaped into another 2D map, such as the 2D array (1235). The connectivity map may be coded by any predictive coding method, such as an image codec, a video codec, etc. In some examples, the connectivity map may be coded in a non-lossless mode. In some examples, the connectivity map may be coded in a lossless mode.

[0148]

[0167] According to one aspect of the present disclosure, at the decoder side, the decoder can decode the vertex connection attributes together with other vertex attributes directly from the bitstream carrying the mesh frame, and then the decoder can reconstruct the mesh frame according to the vertex connection attributes and other vertex attributes.

[0149]

[0168] FIG. 12D shows a diagram illustrating a reconstructed mesh frame (1295). In one example, the reconstructed vertices 0'-8' can be reconstructed, for example, from the decoded coordinates of the vertices from the bitstream. The reconstructed vertices 0'-8' can then be connected according to the decoded connectivity attributes of the vertices from the bitstream, such as the valence of the vertices or the pattern index of the vertices. In one example, an edge breaker algorithm can be used to generate the reconstructed mesh frame (1295) according to the pattern index of the vertices. In another example, a suitable algorithm is used to generate the reconstructed mesh frame (1295) according to the valence of the vertices. The reconstructed mesh frame (1295) can have the same connectivity information (same edges) as the original mesh frame (1205).

[0150]

[0169] In some embodiments, connectivity difference information of a mesh frame can be encoded in a bitstream carrying the mesh frame. The connectivity difference information is a difference between the original connectivity information of the input mesh frame and the estimated connectivity information by the connectivity estimation rule.

[0151]

[0170] 13 shows a diagram of a framework (1300) for mesh compression in accordance with some embodiments of the present disclosure. The framework (1300) includes a mesh encoder (1310) and a mesh decoder (1350). The mesh encoder (1310) encodes an input mesh (1305) (or a mesh frame in the case of dynamic mesh processing) into a bitstream (1345), and the mesh decoder (1350) decodes the bitstream (1345) to generate a reconstructed mesh (1395) (or a reconstructed mesh frame in the case of dynamic mesh processing).

[0152]

[0171] The mesh encoder (1310) may be any suitable device, such as a computer, a server computer, a desktop computer, a laptop computer, a tablet computer, a smartphone, a gaming device, an AR device, a VR device, etc. The mesh decoder (1350) may be any suitable device, such as a computer, a server computer, a desktop computer, a laptop computer, a tablet computer, a smartphone, a gaming device, an AR device, a VR device, etc. The bitstream (1345) may be transmitted from the mesh encoder (1310) to the mesh decoder (1350) via any suitable communications network (not shown).

[0153]

[0172] In the example of Figure 13, the mesh encoder (1310) includes a vertex reordering module (1320), a 1D / 2D encoder (1330), an auxiliary data encoder (1340), and a connectivity estimation module (1335) coupled together. The vertex reordering module (1320) receives an input mesh (1305) and performs reordering to generate reordered vertex mesh information (1125) including connectivity differences. In some examples, the input mesh (1305) includes 3D position information of the vertices of the mesh in their original order, connectivity information in the form of polygonal faces, mapping information that parameterizes the mesh in 2D (e.g., a UV atlas), and other 2D attribute maps (e.g., a 2D color map). The vertex reordering module (1320) can determine a vertex traversal order for the vertices and reorder and / or rearrange the vertices according to the vertex traversal order to increase attribute correlation of adjacent vertices (e.g., attribute correlation of adjacent vertices in a sequence (e.g., a 1D array), attribute correlation of adjacent vertices within a local region in a 2D array, attribute correlation of adjacent frames).

[0154]

[0173] In one example, the vertex reordering module (1320) can rearrange the connectivity information of the mesh frame in the form of a connectivity attribute (also referred to as an original connectivity attribute) of each of the vertices. In one example, the connectivity attribute of each vertex can include a pattern index indicating a connectivity pattern marked by an edge breaker algorithm for connectivity compression. In another example, the connectivity attribute of each vertex can include a value value indicating the number of edges connected to the vertex.

[0155]

[0174] The vertex reordering module (1320) outputs mesh information of reordered vertices (1325). The mesh information of the reordered vertices (1325) includes vertex information for the vertices in the reordered sequence (e.g., 1D). For example, the vertex information for each vertex may include various attributes, such as 3D spatial information within the mesh (e.g., xyz coordinates), mapping information to 2D (e.g., uv coordinates), color information (e.g., RGB values), and the difference in connection attributes (also referred to as the connection difference) between the estimated connection attributes provided by the connection estimation module (1335) and the original connection attributes.

[0156]

[0175] In another example, the reordered vertex mesh information (1325) includes vertex information for the vertices in the form of a 2D image. For example, the reordered vertices may be rearranged (e.g., based on a raster scan line, etc.) into 2D samples (pixels), and the reordered vertex mesh information may form one or more 2D images, such as a 2D image of the 3D coordinates of the vertices, a 2D image of the UV coordinates of the vertices, a 2D image of the vertex connection attribute differences, etc.

[0157]

[0176] The 1D / 2D encoder (1330) is configured to encode the reordered vertex mesh information (1325) into a bitstream (1345). If the reordered vertex mesh information (1325) includes vertex information for vertices in a reordered sequence (1D), the 1D / 2D encoder (1330) may encode the vertex information for the vertices in the reordered sequence using a 1D encoding technique. If the reordered vertex mesh information (1325) includes a 2D image, the 1D / 2D encoder (1130) may encode the 2D image using image encoding and / or video encoding techniques (e.g., using an image codec or a video codec).

[0158]

[0177] In some examples, the connectivity estimation module (1335) may generate estimated connectivity attributes (1336) of the vertices based on the coordinates of the vertices in the original mesh input (1305). In some examples, the connectivity estimation module (1335) may generate estimated connectivity attributes (1336) of the vertices from the encoded coordinates of the vertices. For example, the connectivity estimation module (1335) may receive encoded coordinates of the vertices (1331) from the 1D / 2D encoder (1330), decode the encoded coordinates of the vertices to generate decoded coordinates (e.g., decoded 3D coordinates, decoded UV coordinates) of the vertices, and generate estimated connectivity attributes (1336) of the vertices based on the connectivity estimation rules according to the decoded coordinates of the vertices. The connectivity estimation module (1335) may provide the estimated attributes (1336) to the vertex reordering module (1320). The vertex reordering module (1320) may determine connection differences for the vertices between the original connection attributes of the vertices and the estimated connection attributes of the vertices (1336). The vertex reordering module (1320) may reorder the connection differences for the vertices according to the vertex traversal order to form a 1D or 2D array of connection differences. The 1D or 2D array of connection differences is provided to a 1D / 2D encoder for encoding into a bitstream (1345).

[0159]

[0178] The vertex reordering module (1120) also generates auxiliary data (1127) including auxiliary information. The auxiliary data encoder (1340) receives the auxiliary data (1327) and encodes the auxiliary data (1327) into a bitstream (1345). For example, the vertex reordering module (1320) may reorder the vertices by patch. The vertex reordering module (1320) may provide a value indicative of the number of vertices in each patch in the auxiliary data (1327). Further, in one example, the vertex reordering module (1320) may reorder the boundary vertices before the non-boundary vertices for each patch. The vertex reordering module (1320) may provide a value indicative of the number of boundary vertices in each patch in the auxiliary data (1327). The vertex reordering module (1320) may traverse the vertices according to an edge breaker algorithm and may provide a signal indicative of the edge breaker algorithm in the auxiliary data (1327). In another example, multiple connectivity infer rules may be used to infer connectivity attributes, and a signal indicating a selection of a particular connectivity infer rule from the multiple connectivity infer rules may be included in the auxiliary data (1327).

[0160]

[0179] In the example of FIG. 13, the bitstream (1345) is provided to a mesh decoder (1350). The mesh decoder (1350) includes a 1D / 2D decoder (1360), an auxiliary data decoder (1370), a connectivity determination module (1380), and a mesh reconstruction module (1390) coupled together as shown in FIG. 13. In one example, the 1D / 2D decoder (1360) corresponds to the 1D / 2D encoder (1330) and is capable of decoding a portion of the bitstream (1345) encoded by the 1D / 2D encoder (1330) to generate decoded information (1365). The decoded information (1165) includes decoded coordinate information, such as a decoded 3D coordinate map, a decoded UV coordinate map, etc., decoded connectivity differences (e.g., a decoded connectivity difference map), and other decoded attribute information (e.g., a decoded color map, etc.).

[0161]

[0180] In the example of Figure 13, the auxiliary data decoder (1370) corresponds to the auxiliary data encoder (1340) and is capable of decoding a portion of the bitstream (1345) encoded by the auxiliary data encoder (1340) to generate decoded auxiliary data (1375).

[0162]

[0181] In the example of FIG. 13 , the connection determination module (1380) can receive the decoded information (1365), such as the decoded coordinates of the vertices (e.g., decoded 3D coordinates, decoded uv coordinates, etc.) and the decoded connection differences of the vertices, to generate the recovered connection attributes (1385). In one example, the connection determination module (1380) includes a connection estimation module (1381). The connection estimation module (1381) operates similarly to the connection estimation module (1335) and can generate the estimated connection attributes based on the connection estimation rules according to the decoded coordinates of the vertices. It should be noted that in one example, the same connection estimation rules are used in the connection estimation module (1335) and the connection estimation module (1381). The connection determination module (1380) combines the estimated connection attributes of the vertices output from the connection estimation module (1381) with the decoded connection differences of the vertices to generate the recovered connection attributes of the vertices (1385).

[0163]

[0182] In the example of Figure 13, the decoded information (1365), the decoded auxiliary data (1375), and the restored vertex connectivity information (1385) are provided to a mesh reconstruction module (1390), which generates a reconstructed mesh (1395) based on the decoded information (1365), the decoded auxiliary data (1375), and the restored vertex connectivity information (1385).

[0164]

[0183] It should be noted that each of the components in the mesh encoder (1310), such as the vertex reordering module (1320), the auxiliary data encoder (1340), the 1D / 2D encoder (1330), and the connectivity estimation module (1335), may be implemented by a variety of techniques. In one example, the components are implemented by integrated circuits. In another example, the components are implemented using software capable of being executed by one or more processors.

[0165]

[0184] It should be noted that each of the components in the mesh decoder (1350), such as the 1D / 2D decoder (1360), the auxiliary data decoder (1370), the connectivity determination module (1380), and the mesh reconstruction module (1390), may be implemented by a variety of techniques. In one example, the components are implemented by integrated circuits. In another example, the components are implemented using software that may be executed by one or more processors.

[0166]

[0185] Figures 14A-14F show examples of coding vertex connectivity differences in a mesh frame into a bitstream carrying the mesh frame, according to some embodiments. Figure 14A shows a diagram of an input mesh (1405). The input mesh (1405) includes connected triangles that represent the surface of an object. Each triangle is defined by vertices and edges that connect the vertices to the triangle. The input mesh (1405) includes vertices 0 through 8 that are connected to triangles, as shown in Figure 14A.

[0167]

[0186] 14B shows a diagram of some example vertex traversal order (1415), which is illustrated by an arrowed line that traverses vertex 0, vertex 1, vertex 2, vertex 3, vertex 4, vertex 5, vertex 6, vertex 7, and vertex 8, in that order.

[0168]

[0187] Then, attributes of the vertices, such as the 3D (geometric) coordinates of the vertices, the UV coordinates of the vertices, and other attributes of the vertices, may be sorted into an array according to the vertex traversal order (1415). For example, the 3D coordinates of the vertices may be sorted into an array of 3D coordinates in the vertex traversal order (1415), and the UV coordinates of the vertices may be sorted into an array of UV coordinates in the vertex traversal order (1415). The array may be a 1D array or may be a 2D array.

[0169]

[0188] In some examples, the connectivity information of the original mesh frame (1405) (e.g., how vertices are connected to triangle edges) is reorganized in the form of original connectivity attributes of the vertices. In some examples, the vertices of an original mesh, such as the input mesh (1405), are traversed and each vertex is marked with either a pattern index or a value value, which may represent a connectivity attribute of the vertex. For example, an edge breaker algorithm may be used to determine a vertex traversal order, such as a vertex traversal order (1415), to traverse the vertices, and the edge breaker algorithm may mark each vertex with a pattern index. The connectivity attributes of the vertices are reordered according to the vertex traversal order and then stored in a 2D map (M org The reshaped vertices can then be reshaped into a pixel-position (also referred to as the original connectivity map, denoted by x). Each value at a pixel location in the connectivity map is the corresponding pattern index or value value of the vertex that is reordered and reshaped at the pixel location.

[0170]

[0189] On the encoder side, in some examples, the encoder generates estimated connection attributes according to connection estimation rules.

[0171]

[0190] FIG. 14C shows inferred connections (1420) for vertices 1-8 according to a connectivity inference rule based on, for example, the position of the vertices. The inferred connections (1420) may be in the form of inferred connectivity attributes of the vertices. In some examples, the inferred connectivity attributes of the vertices are sorted according to the vertex traversal order and then sorted by M infer The resulting vertices can then be reshaped into a 2D map, also called the estimated connectivity map, denoted by . Each value at a pixel location in the estimated connectivity map is the corresponding estimated pattern index or estimated value value of the vertices that have been sorted and reshaped at the pixel location.

[0172]

[0191] According to one aspect of the present disclosure, the original connectivity map M org The estimated connectivity map M inferCalculate the difference between diff The connectivity difference map shown by, for example, M diff =M org -M infer It is possible to determine the connection difference map M diff Each value at a pixel location in is the difference value between the pixel value at the same pixel location in the original connectivity map and the estimated connectivity map.

[0173]

[0192] In some examples, the vertex attributes are reordered and reshaped according to the raster scan lines into a 2D array forming a 2D image, which can be coded by a predictive coding technique such as an image or video codec. In an example of a dynamic mesh including a sequence of mesh frames, the dynamic mesh attributes can be reordered to form a sequence of 2D images, which can be coded by a video codec in one example.

[0174]

[0193] FIG. 14D shows a diagram illustrating raster scans (1425) used to reshape vertex attributes into 2D arrays, such as 2D array (1444) and 2D array (1435). Raster scans (1425) are shown as lines with arrows. The 2D arrays (1435) and (1444) may also be referred to as 2D maps or 2D images. In one example, the 2D array (1444) is a connected difference map M diffThe connectivity difference for a vertex is stored in an entry in the 2D array (1444) corresponding to the vertex, the entry being a pixel in the 2D image, and the connectivity difference is color information for the pixel. In one example, another attribute, such as a 3D coordinate or a UV coordinate, for the vertex is stored in an entry in the 2D array (1435) corresponding to the vertex, the entry being a pixel in the 2D image, and the attribute is color information for the pixel. In one example, the 2D array (1435) can be a 3D coordinate map. In another example, the 2D array (1435) can be a UV coordinate map. The 2D array (1435) and the 2D array (1444) can be encoded into a bitstream to carry the mesh frame.

[0175]

[0194] 14E shows a diagram illustrating reconstructed vertices 0'-8' in some examples. For example, the coordinates can be decoded from the bitstream, and then the reconstructed vertices 0'-8' can be generated according to the decoded coordinates.

[0176]

[0195] In some examples, the edges for connecting the reconstructed vertices 0'-8' are estimated according to a connectivity estimation rule. The connectivity estimation rule can estimate connectivity from decoded 3D coordinates (xyz coordinates) and / or UV coordinates (uv coordinates) at the decoder side. According to one aspect of the present disclosure, the same connectivity estimation rule is used at the encoder side and the decoder side. In some examples, the connectivity estimation rule is used at the encoder side to generate estimated connectivity attributes (1420) of the vertices. At the decoder side, the edges for connecting the reconstructed vertices 0'-8' of FIG. 14E are estimated according to a connectivity estimation rule. infer The same connectivity estimation rules are used to generate estimated connectivity attributes of the vertices in the form of an estimated connectivity map denoted by M'. infer is the estimated connectivity map M' obtained in the description of FIG. 14C. infer It is possible to assume that the .DELTA..times ...

[0177]

[0196] The decoder also extracts the vertex connectivity difference from the bitstream, e.g., M' diff In one example, the decoded connectivity difference map M′ diff and the estimated connectivity map M' infer is the recovered connectivity map M that contains the recovered connectivity attributes of the vertices. recover can be added to obtain the recovered connectivity map M recover is the connectivity map M of the original input mesh (1405). org , which can be used to reconstruct the mesh.

[0178]

[0197] FIG. 14F shows a diagram illustrating the reconstructed mesh (1495). The restored mesh includes restored vertices 0'-8', which are added to the restored connectivity map M. recover In some cases, the reconstructed mesh (1495) may be a match to the original input mesh (1405) with the same connections.

[0179]

[0198] According to one aspect of the present disclosure, at the encoder side, the encoder can estimate more than one connection map using different connection estimation rules (also called different connection estimation methods). Then, one of the connection estimation rules with the best estimated connection map compared with the original connection map is selected by the encoder, and an index indicating the selection is signaled in the bitstream to inform the decoder which connection estimation rule is selected.

[0180]

[0199] FIG. 15 shows a flow chart outlining a process (1500) according to an embodiment of the present disclosure. The process (1500) can be used during an encoding process for a mesh. In various embodiments, the process (1500) is performed by a processing circuit. In some embodiments, the process (1500) is implemented in software instructions, such that the processing circuit executes the software instructions to perform the process (1500). The process begins at (S1501) and proceeds to (S1510).

[0181]

[0200] At (S1510), a vertex traversal order for the vertices of the 3D mesh frame is determined.

[0182]

[0201] At least a first array and a second array are formed according to a vertex traversal order, the first array including connectivity values ​​for each of the vertices in the 3D mesh frame according to the vertex traversal order, and the second array including attribute values ​​(e.g., 3D coordinates, UV coordinates, colors, normals, etc.) for each of the vertices according to the vertex traversal order.

[0183]

[0202] At least the first array and the second array are encoded into a bitstream carrying a 3D mesh frame at (S1530).

[0184]

[0203] In some examples, at least the first array and the second array are two-dimensional (2D) arrays (e.g., 2D maps, 2D images). At least the first array and the second array are encoded into a bitstream using an image encoder and / or a video encoder.

[0185]

[0204] In some embodiments, the connectivity value of a vertex includes at least one of a connectivity pattern index of the vertex and a valence value of the vertex.

[0186]

[0205] In some embodiments, the connectivity value of a vertex is a connectivity difference of the vertex relative to an estimated connectivity value of the vertex. In some examples, the estimated connectivity value of a vertex is generated from the coordinates of the vertex according to a connectivity estimation rule. The coordinates of the vertex may include at least one of: an original 3D coordinate of the vertex in the 3D mesh frame; an original UV coordinate of the vertex in the 3D mesh frame; a decoded 3D coordinate of the vertex decoded from the encoded bits corresponding to the original 3D coordinate; and a decoded UV coordinate of the vertex decoded from the encoded bits corresponding to the original UV coordinate. In some examples, the connectivity difference of a vertex is calculated based on an original connectivity value of the vertex of the 3D mesh frame and an estimated connectivity value.

[0187]

[0206] In some examples, at least a first estimated connectivity value of the vertices is generated from the coordinates of the vertices according to a first connectivity estimation rule, and a second estimated connectivity value of the vertices is generated from the coordinates of the vertices according to a second connectivity estimation rule. The connectivity estimation rule is selected from at least the first connectivity estimation rule and the second connectivity estimation rule. Then, a signal is included in the bitstream to indicate the connectivity estimation rule.

[0188]

[0207] Thereafter, the process proceeds to (S1599) and ends.

[0189]

[0208] The process 1500 may be adapted as appropriate. Steps of the process 1500 may be modified and / or omitted. Additional steps may be added. Any suitable order of implementation may be used.

[0190]

[0209] FIG. 16 shows a flow chart outlining a process (1600) according to an embodiment of the present disclosure. The process (1600) may be used during a decoding process for a mesh. In various embodiments, the process (1600) is performed by a processing circuit. In some embodiments, the process (1600) is implemented in software instructions, such that the processing circuit executes the software instructions to perform the process (1600). The process begins at (S1601) and proceeds to (S1610).

[0191]

[0210] At least a first array and a second array are decoded from a bitstream carrying a 3D mesh frame, the first array including connectivity values ​​for each of the vertices in the 3D mesh frame according to a vertex traversal order that matches the second array including coordinates (e.g., 3D coordinates, uv coordinates) of the vertices.

[0192]

[0211] At (S1620), edges connecting the vertices are determined based on at least the connectivity values ​​of each of the vertices. In some examples, the edges connecting the vertices are determined based on the connectivity values ​​of the vertices and the coordinates of the vertices.

[0193]

[0212] At (S1630), a 3D mesh frame is reconstructed based on at least the coordinates of the vertices and edges.

[0194]

[0213] In some embodiments, at least the first array and the second array are two-dimensional (2D) arrays. At least the first array and the second array are decoded using an image decoder and / or a video decoder.

[0195]

[0214] In some embodiments, the connectivity value of a vertex includes at least one of a connectivity pattern index of the vertex and a valence value of the vertex.

[0196]

[0215] In some embodiments, the connectivity value of a vertex is a connectivity difference of the vertex relative to the estimated connectivity value of the vertex. In some examples, the estimated connectivity value of a vertex is estimated based on the coordinates of the vertex according to a connectivity estimation rule. The coordinates of the vertex may include at least one of the 3D coordinates of the vertex and / or the uv coordinates of the vertex.

[0197]

[0216] In some cases, the recovered connectivity values ​​are generated by combining the connectivity difference of the vertices with the estimated connectivity values, and the edges connecting the vertices are determined based on the recovered connectivity values.

[0198]

[0217] In some examples, a signal indicative of a connectivity estimation rule is received from the bitstream, and the connectivity estimation rule can be used to generate an estimated connectivity value for the vertices.

[0199]

[0218] Thereafter, the process proceeds to (S1699) and ends.

[0200]

[0219] The process 1600 may be adapted as appropriate. Steps of the process 1600 may be modified and / or omitted. Additional steps may be added. Any suitable order of implementation may be used.

[0201]

[0220] The techniques disclosed in this disclosure may be used separately or in combination in any order. Furthermore, each of the techniques (e.g., methods, embodiments), encoders, and decoders may be implemented by processing circuitry (e.g., one or more processors, or one or more integrated circuits). In some examples, the one or more processors execute a program stored on a non-transitory computer-readable medium.

[0202]

[0221] The techniques described above may be implemented as computer software using computer readable instructions and may be physically stored on one or more computer readable media. For example, Figure 17 illustrates a computer system (1700) suitable for implementing certain embodiments of the disclosed subject matter.

[0203]

[0222] Computer software may be coded using any suitable machine code or computer language that may be subject to assembly, compilation, linking, or similar mechanisms to produce code that contains instructions that may be executed directly by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., or instructions that may be executed via interpretation, microcode execution, etc.

[0204]

[0223] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0205]

[0224] 17 for computer system (1700) are exemplary in nature and are not intended to suggest any limitations on the scope of use or functionality of the computer software implementing the embodiments of the present disclosure, nor should the arrangement of components be interpreted as having any dependency or requirement regarding any one or combination of components illustrated in the exemplary embodiment of computer system (1700).

[0206]

[0225] The computer system (1700) may include certain human interface input devices that may be responsive to input by one or more human users, for example, via tactile input (e.g., keystrokes, swipes, data glove movements), auditory input (e.g., voice, clapping), visual input (e.g., gestures), or olfactory input (not shown). Human interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images obtained from still image cameras), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic pictures).

[0207]

[0226] The input human interface devices may include one or more of (although only one of each is depicted) a keyboard (1701), a mouse (1702), a trackpad (1703), a touch screen (1710), a data glove (not shown), a joystick (1705), a microphone (1706), a scanner (1707), and a camera (1708).

[0208]

[0227] The computer system (1700) may also include certain human interface output devices that may stimulate one or more of the senses of a human user, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touch screen (1710), data gloves (not shown), joystick (1705), although there may be haptic feedback devices that do not serve as input devices), auditory output devices (e.g., speakers (1709), headphones (not shown)), visual output devices (e.g., screens (1710), including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capability, each with or without haptic feedback capability, some of which may be capable of outputting two-dimensional visual output, or three or more dimensional output by means such as stereoscopic output; virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0209]

[0228] The computer system (1700) may also include human accessible storage devices and their associated media, such as optical media (1721), including CD / DVD ROM / RW (1720), thumb drives (1722), removable hard drives or solid state drives (1723), legacy magnetic media such as tapes and floppy disks (not shown), and specialized ROM / ASIC / PLD based devices such as security dongles (not shown).

[0210]

[0229] Those skilled in the art will also understand that the term "computer-readable medium" as used in connection with the subject matter disclosed herein does not encompass transmission media, carrier waves, or other transitional signals.

[0211]

[0230] The computer system (1700) may also include interfaces to one or more communication networks (1755). The networks may be, for example, wireless, wired, or optical. The networks may further be local, wide area, metropolitan, vehicular, and industrial, real-time, delay tolerant, and the like. Examples of networks include Ethernet, wireless LANs, cellular networks (including GSM, 3G, 4G, 5G, LTE, and the like), TV wired or wireless wide area digital networks (including cable TV, satellite TV, and terrestrial TV), vehicular, and industrial, including CANBus, and the like. Certain networks typically require an external network interface adapter that is attached to a particular general-purpose data port or peripheral bus (1749) (e.g., a USB port on the computer system (1700)); others are typically integrated into the core of the computer system (1700) by attaching to a system bus as described below (e.g., an Ethernet interface is integrated in a PC computer system, a cellular network interface is integrated in a smartphone computer system). Using any of these networks, the computer system (1700) can communicate with other entities. Such communications can be one-way receive only (e.g., broadcast television), one-way transmit only (e.g., CANbus to certain CANbus devices), or bi-directional, such as to other computer systems using local or wide area digital networks. Specific protocols and protocol stacks can be used with each of these networks and network interfaces, as described above.

[0212]

[0231] The aforementioned human interface devices, human accessible storage devices, and network interfaces may be attached to a core (1740) of the computer system (1700).

[0213]

[0232] The core (1740) may include one or more central processing units (CPUs) (1741), graphics processing devices (GPUs) (1742), specialized programmable processing devices in the form of field programmable gate areas (FPGAs) (1743), hardware accelerators for specific tasks (1744), graphics adapters (1750), etc. These devices may be connected via a system bus (1748) along with read only memory (ROM) (1745), random access memory (1746), and internal mass storage devices (e.g., internal non-user accessible hard drives, SSDs, etc.) (1747). In some computer systems, the system bus (1748) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus (1748) or via a peripheral bus (1749). Peripheral bus architectures include PCI, USB, etc. In one example, a screen (1710) can be connected to a graphics adapter (1750). Peripheral bus architectures include PCI, USB, etc.

[0214]

[0233] The CPU (1741), GPU (1742), FPGA (1743), and accelerator (1744) may combine to execute certain instructions that may constitute the aforementioned computer code. The computer code may be stored in a ROM (1745) or a RAM (1746). Temporary data may be stored in the RAM (1746), while persistent data may be stored in, for example, an internal mass storage (1747). Rapid storage and retrieval from any memory device may be made possible by utilizing a cache memory, which may be closely associated with one or more of the CPU (1741), GPU (1742), mass storage (1747), ROM (1745), RAM (1746), etc.

[0215]

[0234] The computer-readable medium can have computer code thereon for performing various computer-implemented operations. The medium and computer code can be specially designed and constructed for the purposes of the present disclosure, or they can be of the kind well known and available to those having skill in the computer software arts.

[0216]

[0235] By way of example, and not by way of limitation, a computer system having the architecture (1700), and in particular the core (1740), as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.), can provide the functionality of executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be media associated with user-accessible mass storage as described above, as well as specific storage of the core (1740) that is non-transitory in nature, such as mass storage (1747) within the core or ROM (1745). Software implementing various embodiments of the present disclosure can be stored in such devices and executed by the core (1740). The computer-readable media can include one or more memory devices or chips, depending on the particular needs. The software can cause the core (1740) and in particular the processor therein (including a CPU, GPU, FPGA, etc.) to perform certain processes or certain portions of certain processes described herein, including defining data structures stored in RAM (1746) and modifying such data structures according to processes defined by the software. Additionally or alternatively, a computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator (1744)), which may execute in place of or in conjunction with software to perform particular processes or portions of particular processes described herein. References to software include logic, and vice versa, where appropriate. References to computer-readable media may include circuitry (such as an integrated circuit (IC)) that stores software for execution, circuitry embodying logic for execution, or both, as appropriate. The present disclosure encompasses any appropriate combination of hardware and software.

[0217]

[0236] While this disclosure describes several exemplary embodiments, there are modifications, substitutions, and various alternative equivalents that fall within the scope of this disclosure. It will thus be appreciated that those skilled in the art will be able to devise numerous systems and methods that, although not explicitly shown or described herein, embody the principles of this disclosure and are therefore within its spirit and scope.

[0218]

[0237] <Additional Notes> (Appendix 1) 13. A method of mesh decompression comprising: decoding at least a first array and a second array from a bitstream carrying a three-dimensional (3D) mesh frame, the first array including connectivity values ​​for each of the vertices in the 3D mesh frame according to a vertex traversal order, and the second array including coordinates for each of the vertices according to the vertex traversal order; determining edges connecting the vertices based at least on the connectivity values ​​of each of the vertices and the coordinates of each of the vertices; and reconstructing the 3D mesh frame based on at least the coordinates of the vertices and the edges; The method includes:

[0219] (Appendix 2) 2. The method of claim 1, wherein at least the first array and the second array are two-dimensional (2D) arrays.

[0220] (Appendix 3) 3. The method of claim 2, wherein the step of decoding at least the first array and the second array comprises: Utilizing an image decoder and / or a video decoder, decoding the at least first array and the second array.

[0221] (Appendix 4) 2. The method of claim 1, wherein the connectivity value of the vertex comprises at least one of a connectivity pattern index of the vertex and a valence value of the vertex.

[0222] (Appendix 5) 2. The method of claim 1, wherein the connectivity value of the vertex is a connectivity difference of the vertex relative to an estimated connectivity value of the vertex.

[0223] (Appendix 6) In the method according to claim 5, generating estimated connectivity values ​​for the vertices from the coordinates of the vertices according to connectivity estimation rules; The method further comprises:

[0224] (Appendix 7) 7. The method of claim 6, wherein the step of determining the edges further comprises: generating recovered connectivity values ​​by combining the vertex connectivity differences with the estimated connectivity values; and determining edges connecting said vertices based on said recovered connectivity values; A method comprising:

[0225] (Appendix 8) 7. The method of claim 6, wherein the coordinates of the vertices include at least one of 3D coordinates and / or uv coordinates.

[0226] (Appendix 9) The method according to claim 6, further comprising: receiving a signal carried in the bitstream, the signal indicating a selected conjunctive estimation rule from among a plurality of conjunctive estimation rules; The method includes:

[0227] (Appendix 10) 1. A mesh decompression apparatus comprising a processing circuit, the processing circuit comprising: decoding at least a first array and a second array from a bitstream carrying a three-dimensional (3D) mesh frame, the first array including connectivity values ​​for each of the vertices in the 3D mesh frame according to a vertex traversal order, and the second array including coordinates of the vertices according to the vertex traversal order; determining edges connecting the vertices based at least on the connectivity values ​​of each of the vertices and the coordinates of the vertices; and reconstructing the 3D mesh frame based on at least the coordinates of the vertices and the edges; The apparatus is configured to:

[0228] (Appendix 11) 11. The apparatus of claim 10, wherein at least the first array and the second array are two-dimensional (2D) arrays.

[0229] (Appendix 12) 12. The apparatus of claim 11, wherein the processing circuitry comprises: The apparatus is configured to decode the at least first array and the second array using an image decoder and / or a video decoder.

[0230] (Appendix 13) 11. The apparatus of claim 10, wherein the connectivity value of the vertex includes at least one of a connectivity pattern index of the vertex and a valence value of the vertex.

[0231] (Appendix 14) 11. The apparatus of claim 10, wherein the vertex connectivity value is a vertex connectivity difference relative to an estimated connectivity value of the vertex.

[0232] (Appendix 15) 15. The apparatus of claim 14, wherein the processing circuitry: The apparatus is configured to generate estimated connectivity values ​​for the vertices from coordinates of the vertices according to a connectivity estimation rule.

[0233] (Appendix 16) 16. The apparatus of claim 15, wherein the processing circuitry: generating recovered connectivity values ​​by combining the vertex connectivity differences with the estimated connectivity values; and determining edges connecting said vertices based on said recovered connectivity values; The apparatus is configured to:

[0234] (Appendix 17) 16. The apparatus of claim 15, wherein the coordinates of the vertices include at least one of 3D coordinates and / or uv coordinates.

[0235] (Appendix 18) 16. The apparatus of claim 15, wherein the processing circuitry: An apparatus configured to perform the steps of: receiving a signal carried in the bitstream, the signal indicating a selected conjunctive estimation rule from among a plurality of conjunctive estimation rules.

[0236] (Appendix 19) A non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to: decoding at least a first array and a second array from a bitstream carrying a three-dimensional (3D) mesh frame, the first array including connectivity values ​​for each of the vertices in the 3D mesh frame according to a vertex traversal order, and the second array including coordinates for each of the vertices; determining edges connecting the vertices based at least on the connectivity values ​​of each of the vertices and the coordinates of the vertices; and reconstructing the 3D mesh frame based on at least the coordinates of the vertices and the edges; A storage medium that executes the above.

[0237] (Appendix 20) 20. The non-transitory computer-readable storage medium of claim 19, wherein at least the first array and the second array are two-dimensional (2D) arrays, and the instructions further comprise: A storage medium for performing the step of decoding the at least first array and the second array using an image decoder and / or a video decoder.

Claims

1. 1. A mesh decompression method comprising: decoding at least a first array and a second array from a bitstream carrying a three-dimensional (3D) mesh frame, the first array including connectivity values ​​for each of the vertices in the 3D mesh frame according to a vertex traversal order, the second array including coordinates for each of the vertices according to the vertex traversal order, the vertex connectivity values ​​indicating vertex connectivity differentials relative to estimated connectivity values ​​for the vertices; determining edges connecting the vertices based at least on the connectivity values ​​of each of the vertices and the coordinates of each of the vertices; and reconstructing the 3D mesh frame based on at least the coordinates of the vertices and the edges; The method includes:

2. The method of claim 1 , wherein at least the first array and the second array are two-dimensional (2D) arrays.

3. 3. The method of claim 2, wherein the step of decoding at least the first array and the second array comprises: Utilizing an image decoder and / or a video decoder, decoding the at least first and second arrays.

4. The method of claim 1 , wherein the connectivity value of the vertex comprises at least one of a connectivity pattern index of the vertex and a valence value of the vertex.

5. 13. The method of claim 1 : generating estimated connectivity values ​​of the vertices from the coordinates of the vertices according to connectivity estimation rules; The method further comprises:

6. 6. The method of claim 5, wherein the step of determining the edges further comprises: generating recovered connectivity values ​​by combining the connectivity differences of the vertices with the estimated connectivity values; and determining edges connecting said vertices based on said recovered connectivity values; A method comprising:

7. The method of claim 5 , wherein the coordinates of the vertices include at least one of 3D coordinates and / or uv coordinates.

8. 6. The method of claim 5 further comprising: receiving a signal carried in the bitstream, the signal indicating a selected conjunctive estimation rule from among a plurality of conjunctive estimation rules; The method includes:

9. A mesh decompression apparatus comprising a processing circuit, the processing circuit configured to perform the method of any one of claims 1 to 8.

10. A computer program product configured to cause at least one processor to carry out a method according to any one of claims 1 to 8.

Citation Information

Patent Citations

  • System and method to obtain surface structures of multi-dimensional objects, and to represent those surface structures for animation, transmission and display

    US20060290695A1

  • Encoding and decoding of texture mapping data in textured 3D mesh models

    US20180253867A1

  • Mesh compression

    US20200286261A1