Vertex reordering for mesh compression

Vertex reordering techniques for mesh compression improve data efficiency, addressing inefficiencies in existing 3D mesh data transmission methods by optimizing coding with image/video codecs, thus reducing storage and transmission requirements for immersive experiences.

JP7593547B2Active Publication Date: 2024-12-03TENCENT AMERICA LLC
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
JP2023534146
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2022-09-06
Filing Date
2022-09-07
Publication Date
2024-12-03
Estimated Expiration
2042-09-07

AI Technical Summary

Technical Problem

Existing methods for compressing and transmitting 3D mesh data are inefficient, leading to high data storage and transmission requirements, which can hinder immersive experiences across various platforms and devices.

Method used

The implementation of vertex reordering techniques for mesh compression, which reorder vertices to increase correlation between adjacent vertices, allowing for more efficient coding using image/video codecs, thereby reducing data requirements.

Benefits of technology

This approach enhances the efficiency of mesh compression, allowing for more immersive 3D media experiences by minimizing data storage and transmission needs while maintaining high-quality rendering.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007593547000001
    Figure 0007593547000001
  • Figure 0007593547000002
    Figure 0007593547000002
  • Figure 0007593547000003
    Figure 0007593547000003
Patent Text Reader

Abstract

Aspects of the present disclosure provide methods and apparatus for mesh coding (e.g., compression and decompression). In some examples, the apparatus for mesh coding includes a processing circuit. The processing circuit decodes an array of attributes from a bitstream carrying a three-dimensional (3D) mesh frame including a plurality of patches. The array of attributes corresponds to vertices of the 3D mesh frame. The vertices of the 3D mesh frame are ordered into subsets that each belong to a plurality of patches. The processing circuit determines a first portion of the array of attributes that corresponds to a first subset of vertices of the 3D mesh frame. The first subset belongs to a first patch among the plurality of patches. The processing circuit determines first connectivity information for the first subset of vertices and reconstructs a first patch of the 3D mesh frame based on the first portion of the array of attributes and the first connectivity information.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] [Incorporated by reference] This application claims the benefit of priority to U.S. Patent Application No. 17 / 903,505, entitled "VERTEX REORDERING FOR MESH COMPRESSION," filed September 6, 2022, which claims the benefit of priority to U.S. Provisional Application No. 63 / 241,876, entitled "Vertex Reordering for Mesh Compression," filed September 8, 2021. The entire disclosure of the prior application is incorporated by reference.

[0002] [Technical field] This disclosure describes embodiments generally related to mesh coding. [Background technology]

[0003] The background discussion provided herein is intended to generally present the context of the present disclosure. Work of the inventors named in this application, to the extent that their work is described in this background section, and aspects of this description that may not otherwise qualify as prior art at the time of filing, are not admitted, either explicitly or implicitly, as prior art to the present disclosure.

[0004] Various techniques have been developed to capture and represent the world, including objects within the world, the environment within the world, and so on, in three-dimensional (3D) space. The 3D representation of the world allows for more immersive forms of interaction and communication. In some examples, point clouds and meshes can be used as the 3D representation of the world. Summary of the Invention

[0005] Aspects of the present disclosure provide methods and apparatus for mesh coding (e.g., compression and decompression). In some examples, an apparatus for mesh coding includes a processing circuit. The processing circuit decodes an array of attributes from a bitstream carrying a three-dimensional (3D) mesh frame including a plurality of patches. The array of attributes corresponds to vertices of the 3D mesh frame. The vertices of the 3D mesh frame are ordered into subsets that each belong to a plurality of patches. The processing circuit determines a first portion of the array of attributes that corresponds to a first subset of vertices of the 3D mesh frame. The first subset belongs to a first patch within the plurality of patches. The processing circuit determines first connectivity information for the first subset of vertices and reconstructs the first patch of the 3D mesh frame based on the first portion of the array of attributes and the first connectivity information.

[0006] In one example, the processing circuitry decodes a two-dimensional (2D) image corresponding to an array of three-dimensional coordinates of the vertices. In another example, the processing circuitry decodes a 2D image corresponding to an array of uv coordinates of the vertices. In another example, the processing circuitry decodes a first 2D image and a second 2D image corresponding to a first array of u coordinates of the vertices and a second array of v coordinates of the vertices, respectively.

[0007] In some examples, the processing circuitry decodes from the bitstream a first value indicating a number of vertices in a first subset of vertices of the 3D mesh frame and selects a first portion of the array of attributes based on the first value. Further, in one example, the processing circuitry decodes from the bitstream a second value indicating a number of bound vertices in the first subset and determines attributes of the bound vertices from the first portion of the array of attributes based on the second value.

[0008] In some examples, the processing circuitry selects from the first portion of the array of attributes an attribute of the boundary vertex that is ordered before attributes of other vertices in the first subset.

[0009] In some examples, the attributes include coordinates, and the processing circuitry decodes boundary coordinate residuals for the bound vertices and combines the coordinates of the bound vertices with the boundary coordinate residuals of the bound vertices to obtain recovered coordinates of the bound vertices. The coordinates of the bound vertices may be three-dimensional (3D) coordinates of the bound vertices and / or uv coordinates of the bound vertices, and the boundary coordinate residuals for the bound vertices may be boundary 3D coordinate residuals and / or boundary uv coordinate residuals.

[0010] In some examples, the processing circuit detects that the first vertex and the second vertex have the same coordinates and removes one of the first vertex and the second vertex from the reconstruction of the first patch.

[0011] In some examples, the processing circuit detects that a non-border vertex of the first patch is outside a boundary (border edge) formed by the boundary vertices of the first patch, and the processing circuit removes the non-border vertex and reconstructs the first patch of the 3D mesh frame without the non-border vertex.

[0012] In some examples, the processing circuit infers the first connectivity information for the first subset of vertices, hi some other examples, the processing circuit decodes the first connectivity information for the first subset of vertices from the bitstream.

[0013] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by a computer, cause the computer to perform any or combination of methods for mesh coding. [Brief description of the drawings]

[0014] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings.

[0015] [Figure 1] 1 illustrates a block diagram of a communication system in accordance with some examples.

[0016] [Diagram 2] 1 illustrates a block diagram of a streaming system in accordance with some examples.

[0017] [Diagram 3] 1 shows a block diagram of an encoder for encoding point cloud frames in some examples.

[0018] [Figure 4] 1 shows a block diagram of a decoder for decoding a compressed bitstream corresponding to a point cloud frame in some examples.

[0019] [Diagram 5] 1 illustrates a block diagram of a video decoder in some examples.

[0020] [Figure 6] 1 illustrates a block diagram of a video encoder in some examples.

[0021] [Figure 7] 1 shows a block diagram of an encoder for encoding point cloud frames in some examples.

[0022] [Figure 8] 1 illustrates a block diagram of a decoder for decoding a compressed bitstream carrying point cloud frames in some examples.

[0023] [Figure 9] 1 shows a diagram illustrating the mapping of meshes to an atlas in some examples.

[0024] [Figure 10] FIG. 1 shows a diagram illustrating two UV atlas frames in some examples.

[0025] [Figure 11]1 illustrates a diagram of a framework for mesh compression according to some embodiments of the present disclosure.

[0026] [Figure 12] FIG. 1 shows a diagram of a framework for mesh compression according to some embodiments of the present disclosure.

[0027] [Figure 13] 1 shows a flowchart outlining a process for reordering vertices in a mesh according to some embodiments of the present disclosure.

[0028] [Figure 14] 1 shows a flowchart outlining an example process in some examples.

[0029] [Figure 15] 1 shows a flowchart outlining an example process in some examples.

[0030] [Figure 16] FIG. 1 is a schematic diagram of a computer system according to some examples. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0031] Aspects of the present disclosure provide techniques in the field of three-dimensional (3D) media processing.

[0032] Technological developments in three-dimensional (3D) media processing, such as advances in 3D capture, 3D modeling, and 3D rendering, have facilitated the ubiquitous presence of 3D media content across several platforms and devices. In one example, a baby's first steps can be captured on one continent, and media technology allows grandparents to enjoy an immersive experience watching (and possibly interacting with) the baby on another continent. According to one aspect of the disclosure, to enhance the immersive experience, 3D models have become increasingly sophisticated, and the creation and consumption of 3D models occupies a significant amount of data resources, such as data storage, data transmission resources, etc.

[0033] According to some aspects of the present disclosure, point clouds and meshes can be used as 3D models to represent immersive contents.

[0034] A point cloud may generally refer to a set of points in 3D space with associated attributes such as color, material properties, texture information, intensity attributes, reflectance attributes, motion-related attributes, modality attributes, and various other attributes. A point cloud can be used to reconstruct an object or scene as a composition of such points.

[0035] An object's mesh (also called a mesh model) may contain polygons that describe the object's surface. Each polygon may be defined by the polygon's vertices in 3D space and information about how the vertices are connected to the polygon. The information about how the vertices are connected is called connectivity information. In some instances, the mesh may also contain attributes, such as color, normals, etc., associated with the vertices.

[0036] According to some aspects of the present disclosure, some coding tools for point cloud compression (PCC) can be used for mesh compression. For example, a mesh can be remeshed to generate a new mesh from which connectivity information of the new mesh can be inferred. The vertices of the new mesh and the attributes associated with the vertices of the new mesh can be viewed as points in a point cloud and compressed using a PCC codec.

[0037] A point cloud can be used to reconstruct an object or scene as a configuration of points. The points can be captured using multiple cameras, depth sensors, or Lidar in a variety of settings, and may consist of thousands to billions of points to realistically represent the reconstructed scene or object. A patch may generally refer to a contiguous subset of a surface described by a point cloud. In one example, a patch includes points with surface normal vectors that are offset from each other by less than a threshold amount.

[0038] PCC can be performed according to various schemes, such as a geometry-based scheme called G-PCC, a video coding-based scheme called V-PCC, etc. According to some aspects of the present disclosure, G-PCC is a purely geometry-based approach that directly encodes 3D geometry and does not have much in common with video coding, while V-PCC is heavily based on video coding. For example, V-PCC can map points of a 3D cloud to pixels of a 2D grid (image). The V-PCC scheme can utilize a general-purpose video codec for point cloud compression. The PCC codec (encoder / decoder) in the present disclosure can be a G-PCC codec (encoder / decoder) or a V-PCC codec.

[0039] According to one aspect of the present disclosure, the V-PCC scheme can use existing video codecs to compress the geometry, occupancy, and texture of the point cloud as three separate video sequences. Additional metadata required to interpret the three video sequences is compressed separately. A small portion of the overall bitstream is the metadata, which in one example can be efficiently encoded / decoded using a software implementation. The majority of the information is handled by the video codec.

[0040] FIG. 1 illustrates a block diagram of a communication system (100) in some examples. The communication system (100) includes a plurality of terminal devices that can communicate with each other, for example, via a network (150). For example, the communication system (100) includes a pair of terminal devices (110) and (120) that are connected to each other via the network (150). In the example of FIG. 1, a first pair of terminal devices (110) and (120) may perform unidirectional transmission of point cloud data. For example, the terminal device (110) may compress a point cloud (e.g., points representing structures) captured by a sensor (105) connected to the terminal device (110). The compressed point cloud can be transmitted to another terminal device (120) via the network (150), for example, in the form of a bit stream. The terminal device (120) may receive the compressed point cloud from the network (150), decompress the bit stream to reconstruct the point cloud, and appropriately display the reconstructed point cloud. One-way data transmission is common, for example, in media service applications.

[0041] In the example of FIG. 1, terminal devices (110) and (120) may be illustrated as a server and a personal computer, although the principles of the present disclosure are not so limited. Embodiments of the present disclosure find application in laptop computers, tablet computers, smart phones, gaming consoles, media players, and / or dedicated three-dimensional (3D) devices. Network (150) represents any number of networks that transmit compressed point clouds between terminal devices (110) and (120). Network (150) may include, for example, wireline and / or wireless communication networks. Network (150) may exchange data over circuit-switched and / or packet-switched channels. Exemplary networks include telecommunications networks, local area networks, wide area networks, the Internet, and the like.

[0042] 2 illustrates a block diagram of a streaming system 200 in some examples. The streaming system 200 is a point cloud utilization application. The disclosed subject matter can be similarly applicable to other point cloud-enabled applications, such as 3D telepresence applications, virtual reality applications, etc.

[0043] The streaming system (200) may include a capture subsystem (213). The capture subsystem (213) may include a point cloud source (201), such as a light detection and ranging (lidar) system, a three-dimensional camera, a three-dimensional scanner, a graphics generation component that generates the uncompressed point cloud in software, etc., that generates the uncompressed point cloud (202). In one example, the point cloud (202) includes points captured by a 3D camera. The point cloud (202) is shown in bold to emphasize the high amount of data compared to the compressed point cloud (204) (compressed point cloud bitstream). The compressed point cloud (204) may be generated by an electronic device (220) that includes an encoder (203) coupled to the point cloud source (201). The encoder (203) may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The compressed point cloud (204) (or bitstream of compressed point cloud (204)), depicted as thin lines to emphasize its low amount of data compared to the stream of point cloud (202), can be stored on a streaming server (205) for future use. One or more streaming client subsystems, such as client subsystems (206) and (208) of FIG. 2, can access the streaming server (205) to obtain copies (207) and (209) of the compressed point cloud (204). The client subsystem (206) can include a decoder (210), for example on an electronic device (230). The decoder (210) decodes the incoming copy (207) of the compressed point cloud and creates an outgoing stream of reconstructed point cloud (211) that can be rendered on a rendering device (212).

[0044] It is noted that electronic devices 220 and 230 may include other components (not shown). For example, electronic device 220 may include a decoder (not shown) and electronic device 230 may include an encoder (not shown).

[0045] In some streaming systems, the compressed point clouds 204, 207, and 209 (e.g., compressed point cloud bitstreams) may be compressed according to a particular standard. In some examples, a video coding standard is used to compress the point cloud. Examples of these standards include High Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), etc.

[0046] 3 shows a block diagram of a V-PCC encoder (300) for encoding point cloud frames, according to some embodiments. In some embodiments, the V-PCC encoder (300) can be used in the communication system (100) and the streaming system (200). For example, the encoder (203) can be configured and operate in a manner similar to the V-PCC encoder (300).

[0047] The V-PCC encoder (300) receives point cloud frames as uncompressed input and generates a bitstream that corresponds to the compressed point cloud frames. In some embodiments, the V-PCC encoder (300) may receive the point cloud frames from a point cloud source, such as the point cloud source (201).

[0048] In the example of FIG. 3, the V-PCC encoder (300) includes a patch generation module (306), a patch packing module (308), a geometry image generation module (310), a texture image generation module (312), a patch information module (304), an occupancy map module (314), a smoothing module (336), image padding modules (316) and (318), a group expansion module (320), video compression modules (322), (323), and (332), an auxiliary patch information compression module (338), an entropy compression module (334), and a multiplexer (324).

[0049] According to one aspect of the disclosure, the V-PCC encoder (300) converts the 3D point cloud frames into an image-based representation along with some metadata (e.g., occupancy map and patch information) that is used to convert the compressed point cloud back to a decompressed point cloud. In some examples, the V-PCC encoder (300) can use video coding techniques to convert the 3D point cloud frames into a geometry image, a texture image, and an occupancy map, and then encode the geometry image, texture image, and occupancy map into a bitstream. In general, a geometry image is a 2D image with pixels filled with geometry values ​​associated with points projected onto pixels, and pixels filled with geometry values ​​can be referred to as geometry samples. A texture image is a 2D image with pixels filled with texture values ​​associated with points projected onto pixels, and pixels filled with texture values ​​can be referred to as texture samples. An occupancy map is a 2D image with pixels filled with values ​​that indicate whether they are occupied or not occupied by a patch.

[0050] The patch generation module (306) segments the point cloud into a set of patches (e.g., a patch is defined as a contiguous subset of the surface described by the point cloud), which may or may not be overlapping, such that each patch can be described by a depth field relative to a plane in 2D space. In some embodiments, the patch generation module (306) aims to decompose the point cloud into a minimum number of patches with smooth boundaries while minimizing reconstruction error.

[0051] In some examples, the patch information module (304) can collect patch information indicating the size and shape of the patch, hi some examples, the patch information can be packed into an image frame and then encoded by the auxiliary patch information compression module (338) to generate compressed auxiliary patch information.

[0052] In some examples, the patch packing module (308) is configured to map the extracted patches onto a two-dimensional (2D) grid while minimizing unused space and ensuring that every M×M (e.g., 16×16) block of the grid is associated with a unique patch. Efficient patch packing can directly impact compression efficiency by minimizing unused space or ensuring temporal consistency.

[0053] The geometry image generation module (310) can generate a 2D geometry image associated with the geometry of the point cloud at a given patch location. The texture image generation module (312) can generate a 2D texture image associated with the texture of the point cloud at a given patch location. The geometry image generation module (310) and the texture image generation module (312) utilize the 3D to 2D mapping calculated during the packing process to store the geometry and texture of the point cloud as images. To better handle the case where multiple points are projected onto the same sample, each patch is projected onto two images called layers. In one example, the geometry image is represented as a WxH monochromatic frame in YUV 420-8 bit format. To generate the texture image, the texture generation procedure utilizes the reconstructed / smoothed geometry to calculate the color associated with the resampled points.

[0054] The occupancy map module (314) can generate an occupancy map that describes the padding information for each unit. For example, the occupancy image includes a binary map that indicates for each cell of the grid whether the cell belongs to empty space or to the point cloud. In one example, the occupancy map uses binary information that describes for each pixel whether the pixel is padded or not. In another example, the occupancy map uses binary information that describes for each block of pixels whether the block of pixels is padded or not.

[0055] The occupancy map generated by the occupancy map module (314) can be compressed using lossless or lossy coding. If lossless coding is used, the entropy compression module (334) is used to compress the occupancy map. If lossy coding is used, the video compression module (332) is used to compress the occupancy map.

[0056] It is noted that the patch packing module (308) may leave some empty space between the 2D patches packed into the image frame. The image padding modules (316) and (318) may fill the empty space (called padding) to generate an image frame that may be suitable for 2D video and image codecs. Image padding is also called background filling, which may fill unused space with redundant information. In some examples, good background filling introduces no significant coding distortion around patch boundaries while minimally increasing the bit rate.

[0057] The video compression modules (322), (323), (332) can encode 2D images, such as padded geometry images, padded texture images, occupancy maps, etc., based on an appropriate video coding standard, such as HEVC, VVC, etc. In one example, the video compression modules (322), (323), (332) are individual components operating independently. It is noted that the video compression modules (322), (323), (332) can be implemented as a single component in another example.

[0058] In some examples, a smoothing module (336) is configured to generate a smoothed image of the reconstructed geometry image. The smoothed image can be provided to a texture image generation (312). The texture image generation (312) can then adjust the generation of the texture image based on the reconstructed geometry image. For example, if a patch shape (e.g., geometry) is slightly distorted during encoding and decoding, the distortion can be taken into account when generating the texture image to correct for the distortion of the patch shape.

[0059] In some embodiments, the group expansion (320) is configured to pad pixels around object boundaries with redundant low-frequency content to improve coding gain as well as the visual quality of the reconstructed point cloud.

[0060] A multiplexer (324) can multiplex the compressed geometry images, the compressed texture images, the compressed occupancy maps, and the compressed auxiliary patch information into a compressed bitstream.

[0061] 4 illustrates a block diagram of a V-PCC decoder (400) for decoding a compressed bitstream corresponding to a point cloud frame in some examples. In some examples, the V-PCC decoder (400) can be used in the communication system (100) and the streaming system (200). For example, the decoder (210) can be configured to operate in a manner similar to the V-PCC decoder (400). The V-PCC decoder (400) receives a compressed bitstream and generates a reconstructed point cloud based on the compressed bitstream.

[0062] In the example of FIG. 4, the V-PCC decoder (400) includes a demultiplexer (432), video decompression modules (434) and (436), an occupancy map decompression module (438), an auxiliary patch information decompression module (442), a geometry reconstruction module (444), a smoothing module (446), a texture reconstruction module (448), and a color smoothing module (452).

[0063] A demultiplexer (432) can receive the compressed bitstream and separate it into compressed texture images, compressed geometry images, compressed occupancy maps, and compressed auxiliary patch information.

[0064] The video decompression modules (434) and (436) can decode images compressed according to an appropriate standard (e.g., HEVC, VVC, etc.) and output decompressed images. For example, the video decompression module (434) can decode compressed texture images and output decompressed texture images. The video decompression module (436) can decode compressed geometry images and output decompressed geometry images.

[0065] An occupancy map decompression module (438) can decode the compressed occupancy map according to an appropriate standard (e.g., HEVC, VVC, etc.) and output a decompressed occupancy map.

[0066] The auxiliary patch information decompression module (442) can decode the auxiliary patch information compressed according to an appropriate standard (e.g., HEVC, VVC, etc.) and output the decompressed auxiliary patch information.

[0067] A geometry reconstruction module (444) can receive the decompressed geometry image and generate a reconstructed point cloud geometry based on the decompressed occupancy map and the decompressed auxiliary patch information.

[0068] A smoothing module (446) can smooth incongruences at the edges of the patches. The smoothing procedure aims to mitigate potential discontinuities that may occur at the patch boundaries due to compression artifacts. In some embodiments, a smoothing filter can be applied to pixels located at the patch boundaries to mitigate distortions that may occur due to compression / decompression.

[0069] A texture reconstruction module (448) can determine texture information for points in the point cloud based on the decompressed texture image and the smoothed geometry.

[0070] The color smoothing module (452) can smooth out coloring discrepancies. Non-adjacent patches in 3D space are often packed adjacent to each other in 2D video. In some instances, pixel values ​​from non-adjacent patches may be confused by block-based video codecs. The purpose of color smoothing is to reduce visible artifacts that appear at patch boundaries.

[0071] 5 shows a block diagram of a video decoder (510) in some examples. The video decoder (510) can be used in the V-PCC decoder (400). For example, the video decompression modules (434) and (436) and the occupancy map decompression module (438) can be configured similarly to the video decoder (510).

[0072] The video decoder (510) may include a parser (520) that reconstructs symbols (521) from a compressed image, such as a coded video sequence. These symbol categories include information used to manage the operation of the video decoder (510). The parser (520) may parse / entropy decode a received coded video sequence. The coding of the coded video sequence may follow a video coding technique or standard and may follow various principles, such as variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (520) may extract from the coded video sequence at least one set of subgroup parameters for a subgroup of pixels in the video decoder based on at least one parameter corresponding to the group. The subgroup may include a Group of Pictures (GOP), a picture, a tile, a slice, a macroblock, Coding Units (CU), a block, Transform Units (TU), Prediction Units (PU), etc. The parser (520) may also extract from the coded video sequence information, such as transform coefficients, quantization parameter values, motion vectors, and the like.

[0073] The parser (520) may perform an entropy decoding / parsing operation on the video sequence received from the buffer memory to produce symbols (521).

[0074] The reconstruction of the symbols (521) can involve a number of different units, depending on the type of coded video picture or part thereof (inter and intra pictures, inter and intra blocks, etc.), and other factors. Which units are involved and how they are involved can be controlled by subgroup control information parsed from the coded video sequence by the parser (520). The flow of such subgroup control information between the parser (520) and the following units is not depicted for clarity.

[0075] Beyond the functional blocks already mentioned, the video decoder (510) may be conceptually subdivided into a number of functional units, as described below. In a practical implementation operating under commercial constraints, many of these units may interact closely with each other and may be at least partially integrated with each other. However, for purposes of describing the disclosed subject matter, the following conceptual subdivision into functional units is adequate:

[0076] The first unit is a scalar / inverse transform unit (551), which receives quantized transform coefficients as well as control information from the parser (520) including which transform to use, block size, quantization coefficients, quantization scaling matrices, etc. as symbol(s) (521). The scalar / inverse transform unit (551) can output blocks containing sample values ​​that can be input to an aggregator (555).

[0077] In some cases, the output samples of the scaler / inverse transform (551) may relate to intra-coded blocks; i.e., blocks that do not use prediction information from a previously reconstructed picture, but may use prediction information from a previously reconstructed portion of the current picture. Such prediction information may be provided by an intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) generates a block of the same size and shape as the block being reconstructed using surrounding already reconstructed information fetched from a current picture buffer (558). The current picture buffer (558) may, for example, buffer a partially reconstructed current picture and / or a fully reconstructed current picture. The aggregator (555) may add the prediction information generated by the intra-prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551) on a sample-by-sample basis.

[0078] In other cases, the output samples of the scalar / inverse transform unit (551) may relate to an inter-coded, potentially motion compensated block. In such cases, the motion compensated prediction unit (553) may access the reference picture memory (557) to fetch samples used for prediction. After motion compensating the fetched samples according to the symbols (521) associated with the block, these samples may be added by the aggregator (555) to the output of the scalar / inverse transform unit (551) (in this case referred to as residual samples or residual signals) to generate output sample information. The addresses in the reference picture memory (557) from which the motion compensated prediction unit (553) fetches the prediction samples may be controlled by a motion vector and may be available to the motion compensated prediction unit (553) in the form of symbols (521), which may have, for example, X, Y, and reference picture components. Motion compensation may also include interpolation of sample values ​​fetched from the reference picture memory (557) when sub-sample accurate motion vectors are used, motion vector prediction mechanisms, etc.

[0079] The output samples of the aggregator (555) can be subject to various loop filtering techniques in the loop filter unit (556). The video compression techniques can include in-loop filter techniques that are controlled by parameters contained in the coded video sequence (also called the coded video bitstream) and made available to the loop filter unit (556) as symbols (521) from the parser (520), but can also be responsive to meta-information obtained during decoding of a coded picture or previous (in decoding order) parts of the coded video sequence, or to previously reconstructed loop filtered sample values.

[0080] The output of the loop filter unit (556) can be a sample stream that can be output to a render device as well as stored in a reference picture memory (557) for use in future inter-picture prediction.

[0081] Once a particular coded picture is fully reconstructed, it can be used as a reference picture for future prediction. For example, once a coded picture corresponding to a current picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) can be made part of the reference picture memory (557), and a new current picture buffer can be reallocated before starting reconstruction of the next coded picture.

[0082] The video decoder (510) may perform decoding operations according to a given video compression technique, in a standard such as ITU-T Recommendation H.265. The coded video sequence may comply with the syntax specified by the video compression technique or standard being used, in the sense that the coded video sequence complies with both the syntax of the video compression technique or standard and the profile described in the video compression technique or standard. In particular, the profile may select a particular tool as the only tool available in the profile from all tools available in the video compression technique or standard. Also, compliance may require that the complexity of the coded video sequence is within a range defined by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples / second, etc.), maximum reference picture size, etc. The limits set by the level may be further limited by a hypothetical reference decoder (HRD) specification and metadata for HRD buffer management, which may be signaled in the coded video sequence.

[0083] 6 shows a block diagram of a video encoder (603) according to one embodiment of the present disclosure. The video encoder (603) can be used in the V-PCC encoder (300) to compress a point cloud. In one example, the video compression modules (322) and (323) and the video compression module (332) are configured similarly to the encoder (603).

[0084] The video encoder (603) may receive images, such as padded geometry images, padded texture images, etc., and generate compressed images.

[0085] According to one embodiment, the video encoder (603) may code and compress pictures of a source video sequence (images) into a coded video sequence (compressed images) in real-time or under other time constraints required by the application. Enforcing an appropriate coding rate is one of the functions of the controller (650). In some embodiments, the controller (650) controls and is operatively coupled to other functional units as described below. Coupling is not depicted for clarity. Parameters set by the controller (650) may include rate control related parameters (picture skip, quantization, lambda value for rate distortion optimization techniques, ...), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) may be configured with other appropriate functions associated with the video encoder (603) optimized for a particular system design.

[0086] In some embodiments, the video encoder (603) is configured to operate in a coding loop. As an oversimplified explanation, in one example, the coding loop can include a source coder (630) (e.g., responsible for creating symbols, such as a symbol stream, based on an input picture to be coded and reference picture(s)) and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to create sample data in a similar manner to that created by the (remote) coder (since in the video compression techniques contemplated in the disclosed subject matter, the compression between the symbols and the coded video bitstream is lossless). The reconstructed sample stream (sample data) is input to a reference picture memory (634). Since the decoding of the symbol stream produces bit-accurate results that are independent of the location of the decoder (local or remote), the content in the reference picture memory (634) is also bit-accurate between the local and remote encoders. In other words, the prediction part of the encoder "sees" the exact same sample values ​​as the decoder would "see" if it were to use prediction during decoding. This basic principle of reference picture synchrony (if synchrony cannot be maintained due to channel errors, etc., drift will result) is also used in several related technologies.

[0087] The operation of the "local" decoder (633) can be the same as a "remote" decoder, such as the video decoder (510) already described in detail in conjunction with Figure 5. However, with brief reference also to Figure 5, because symbols are available and the encoding / decoding of symbols into a coded video sequence by the entropy coder (645) and parser (520) can be lossless, the entropy decoding portion of the video decoder (510), including the parser (520), may not be fully implemented in the local decoder (633).

[0088] During operation, in some examples, the source coder (630) may perform motion-compensated predictive coding to predictively code an input picture with reference to one or more previously coded pictures from a video sequence designated as “reference pictures.” In this manner, the coding engine (632) codes differences between pixel blocks of the input picture and pixel blocks of reference picture(s) that may be selected as predictive reference(s) to the input picture.

[0089] The local video decoder (633) may decode the coded video data of pictures that may be designated as reference pictures based on the symbols created by the source coder (630). The operation of the coding engine (632) may advantageously be a lossy process. If the coded video data may be decoded in a video decoder (not shown in FIG. 6), the reconstructed video sequence may be a replica of the source video sequence, usually with some errors. The local video decoder (633) replicates the decoding process that may be performed on the reference pictures by the video decoder and may cause the reconstructed reference pictures to be stored in a reference picture cache (634). In this way, the video encoder (603) may locally store copies of reconstructed reference pictures with common content as reconstructed reference pictures to be retrieved by the far-end video decoder (free of transmission errors).

[0090] The predictor (635) may perform the prediction search of the coding engine (632). That is, for a new picture to be coded, the predictor (635) may search the reference picture memory (634) for sample data (as candidate reference pixel blocks) or specific metadata, such as reference picture motion vectors, block shapes, etc., that may serve as suitable prediction references for the new picture. The predictor (635) may operate on a sample block / pixel block basis to find a suitable prediction reference. In some cases, as determined by the search results obtained by the predictor (635), the input picture may have prediction references drawn from multiple reference pictures stored in the reference picture memory (634).

[0091] The controller (650) may manage the coding operations of the source coder (630), including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0092] The output of all the aforementioned functional units may undergo entropy coding in an entropy coder (645), which converts the symbols produced by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc.

[0093] The controller (650) may manage the operation of the video encoder (603). During coding, the controller (650) may assign a particular coded picture type to each coded picture, which may affect the coding technique that may be applied to the respective picture. For example, pictures may often be assigned as one of the following picture types:

[0094] An intra picture (I-picture) may be one that can be coded and decoded without using other pictures in a sequence as a source of prediction. Some video codecs allow different kinds of intra pictures, including, for example, Independent Decoder Refresh ("IDR") pictures. Those skilled in the art are aware of these variants of I-pictures and their respective uses and characteristics.

[0095] A predictive picture (P picture) may be one that can be coded and decoded using intra prediction or inter prediction, using at most one motion vector and reference index to predict the sample values ​​of each block.

[0096] A bidirectionally predicted picture (B-picture) may be one that can be coded and decoded using intra- or inter-prediction, using at most two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, a multi-predicted picture may use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0097] A source picture is generally spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples, respectively) and may be coded block by block. Blocks may be predictively coded with reference to other (already coded) blocks as determined by the coding assignment applied to the respective picture of the block. For example, blocks of I pictures may be non-predictively coded or they may be predictively coded with reference to already coded blocks of the same picture (spatial or intra prediction). Pixel blocks of P pictures may be predictively coded by spatial prediction with reference to one previously coded reference picture or by temporal prediction. Blocks of B pictures may be predictively coded by spatial prediction with reference to one or two previously coded reference pictures or by temporal prediction.

[0098] The video encoder (603) may perform coding operations according to a given video coding technique or standard, such as ITU-T Recommendation H.265. In its operations, the video encoder (603) may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the coded video data may conform to a syntax defined by the video coding technique or standard being used.

[0099] Video may be in the form of multiple source pictures (images) in a temporal sequence. Intra-picture prediction (often abbreviated as intra-prediction) exploits spatial correlation in a given picture, while inter-picture prediction exploits correlation (temporal or other) between pictures. In one example, a particular picture being encoded / decoded, called the current picture, is partitioned into blocks. If a block in the current picture is similar to a reference block in a previously coded and still buffered reference picture in the video, then that block in the current picture can be coded by a vector called a motion vector. A motion vector points to a reference block in a reference picture, and may have a third dimension that identifies the reference picture if multiple reference pictures are used.

[0100] In some embodiments, bi-prediction techniques can be used in inter-picture prediction. According to bi-prediction techniques, two reference pictures are used, for example a first reference picture and a second reference picture that both precede in decoding order (but may be in the past and future, respectively, in display order) the current picture in the video. A block in the current picture can be coded by a first motion vector that points to a first reference block in the first reference picture and a second motion vector that points to a second reference block in the second reference picture. A block can be predicted by a combination of the first reference block and the second reference block.

[0101] Furthermore, merge mode techniques can be used in inter-picture prediction to improve coding efficiency.

[0102] According to some embodiments of the present disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed in units of blocks. For example, according to the HEVC standard, a picture in a sequence of video pictures is partitioned into coding tree units (CTUs) for compression, and the CTUs in a picture have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. In general, a CTU includes three coding tree blocks (CTBs), which are one luma CTB and two chroma CTBs. Each CTU can be recursively quadtree split into one or more coding units (CUs). For example, a CTU of 64×64 pixels can be split into one CU of 64×64 pixels, or four CUs of 32×32 pixels, or 16 CUs of 16×16 pixels. In one example, each CU is analyzed to determine a prediction type for the CU, such as an inter prediction type or an intra prediction type. A CU is divided into one or more Prediction Units (PUs) according to temporal and / or spatial predictability. Generally, each PU includes one luma prediction block (PB) and two chroma PBs. In one embodiment, prediction operations in coding (encoding / decoding) are performed in units of prediction blocks. Using a luma prediction block as an example of a prediction block, the prediction block includes a matrix of pixel values ​​(e.g., luma values), such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0103] 7 illustrates a block diagram of a G-PCC encoder (700) in some examples. The G-PCC encoder (700) can be configured to receive point cloud data and compress the point cloud data to generate a bitstream carrying the compressed point cloud data. In one embodiment, the G-PCC encoder (700) can include a position quantization module (710), a duplicated points removal module (712), an octree encoding module (730), an attribute transfer module (720), a level of detail (LOD) generation module (740), an attribute prediction module (750), a residual quantization module (760), an arithmetic coding module (770), an inverse residual quantization module (780), an addition module (781), and a memory (790) for storing reconstructed attribute values.

[0104] As shown, an input point cloud (701) may be received at a G-PCC encoder (700). Positions (e.g., 3D coordinates) of the point cloud (701) are provided to a quantization module (710). The quantization module (710) is configured to quantize the coordinates to generate quantized positions. A remove duplicate points module (712) is configured to receive the quantized positions and perform a filter process to identify and remove duplicate points. An octree encoding module (730) receives the filtered positions from the remove duplicate points module (712) and is configured to perform an octree-based encoding process to generate a sequence of occupancy codes that describe a 3D grid of voxels. The occupancy codes are provided to an arithmetic coding module (770).

[0105] The attribute transfer module (720) is configured to receive attributes of the input point cloud and perform an attribute transfer process to determine an attribute value for each voxel when multiple attribute values ​​are associated with each voxel. The attribute transfer process can be performed on the reordered points output from the octree encoding module (730). The attributes after the transfer operation are provided to the attribute prediction module (750). The LOD generation module (740) is configured to operate on the reordered points output from the octree encoding module (730) and re-organize the points into different LODs. The LOD information is provided to the attribute prediction module (750).

[0106] The attribute prediction module (750) processes the points according to the LOD-based order indicated by the LOD information from the LOD generation module (740). The attribute prediction module (750) generates an attribute prediction for the current point based on the reconstructed attributes of a set of neighboring points of the current point stored in the memory (790). A prediction residual can be subsequently obtained based on the original attribute values ​​received from the attribute transfer module (720) and the locally generated attribute prediction. If a candidate index is used in each attribute prediction process, an index corresponding to a selected prediction candidate can be provided to the arithmetic coding module (770).

[0107] The residual quantization module (760) is configured to receive the prediction residual from the attribute prediction module (750) and perform quantization to generate a quantized residual. The quantized residual is provided to the arithmetic coding module (770).

[0108] The inverse residual quantization module (780) is configured to receive the quantized residual from the residual quantization module (760) and generate a reconstructed prediction residual by performing an inverse of the quantization operation performed by the residual quantization module (760). The summation module (781) is configured to receive the reconstructed prediction residual from the inverse residual quantization module (780) and the respective attribute predictions from the attribute prediction module (750). By combining the reconstructed prediction residual and the attribute predictions, reconstructed attribute values ​​are generated and stored in the memory (790).

[0109] The arithmetic coding module (770) is configured to receive the occupation codes, the candidate indexes (if used), the quantized residuals (if generated), and other information, and to perform entropy encoding to further compress the received values ​​or information. As a result, a compressed bitstream (702) carrying the compressed information can be generated. The bitstream (702) can be transmitted or otherwise provided to a decoder that decodes the compressed bitstream, or can be stored in a storage device.

[0110] 8 shows a block diagram of a G-PCC decoder (800) according to one embodiment. The G-PCC decoder (800) may be configured to receive a compressed bitstream and perform point cloud data decompression to decompress the bitstream to generate decoded point cloud data. In one embodiment, the G-PCC decoder (800) may include an arithmetic decoding module (810), an inverse residual quantization module (820), an octree decoding module (830), an LOD generation module (840), an attribute prediction module (850), and a memory (860) for storing reconstructed attribute values.

[0111] As shown, a compressed bitstream (801) may be received at an arithmetic decoding module (810). The arithmetic decoding module (810) is configured to decode the compressed bitstream to obtain quantized residuals (if generated) and occupied codes of the point cloud. The octree decoding module (830) is configured to determine reconstructed positions of points in the point cloud according to the occupied codes. The LOD generation module (840) is configured to reorganize the points into different LODs based on the reconstructed positions and determine an LOD-based order. The inverse residual quantization module (820) is configured to generate reconstructed residuals based on the quantized residuals received from the arithmetic decoding module (810).

[0112] The attribute prediction module (850) is configured to perform an attribute prediction process to determine attribute predictions for the points according to the LOD-based order. For example, an attribute prediction for a current point can be determined based on reconstructed attribute values ​​of neighboring points of the current point stored in the memory (860). In some examples, the attribute predictions can be combined with the respective reconstructed residuals to generate reconstructed attributes for the current point.

[0113] The sequence of reconstructed attributes generated from the attribute prediction module (850) along with the reconstructed positions generated from the octree decoding module (830) corresponds, in one example, to the decoded point cloud (802) output from the G-PCC decoder (800). In addition, the reconstructed attributes may also be stored in memory (860) and subsequently used to derive attribute predictions for subsequent points.

[0114] In various embodiments, the encoder (300), the decoder (400), the encoder (700), and / or the decoder (800) may be implemented in hardware, software, or a combination thereof. For example, the encoder (300), the decoder (400), the encoder (700), and / or the decoder (800) may be implemented in a processing circuit, such as one or more integrated circuits (ICs) that operate with or without software, such as application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), and the like. In other examples, the encoder (300), the decoder (400), the encoder (700), and / or the decoder (800) may be implemented as software or firmware that includes instructions stored on a non-volatile (or non-transitory) computer-readable storage medium. The instructions, when executed by a processing circuit, such as one or more processors, cause the processing circuit to perform the functions of the encoder (300), the decoder (400), the encoder (700), and / or the decoder (800).

[0115] It is noted that the attribute prediction modules (750) and (850) configured to implement the attribute prediction techniques disclosed herein can be included in other decoders or encoders that may have similar or different structures to those shown in Figures 7 and 8. Additionally, the encoder (700) and decoder (800) can be included in the same device or separate devices in various examples.

[0116] According to some aspects of the present disclosure, mesh compression may use a coding tool different from a PCC coding tool, or may use a PCC coding tool such as a PCC (e.g., G-PCC, V-PCC) encoder, a PCC (e.g., G-PCC, V-PCC) decoder, etc.

[0117] An object's mesh (also called a mesh model or mesh frame) may contain polygons that describe the object's surface. Each polygon may be defined by the polygon's vertices in 3D space and information about how the vertices are connected. The information about how the vertices are connected is called connectivity information. In some examples, a mesh may also contain attributes, such as color, normals, etc., associated with the vertices. The attributes may be associated with the mesh's surface by utilizing mapping information that parameterizes the mesh with a 2D attribute map. The mapping information is typically described by a set of parametric coordinates, called UV coordinates or texture coordinates, associated with the mesh vertices. The 2D attribute map (called a texture map in some examples) is used to store high-resolution attribute information, such as texture, normals, and displacements. Such information may be used for various purposes, such as texture mapping and shading.

[0118] In some embodiments, a mesh can include components called geometry information, connectivity information, mapping information, vertex attributes, and attribute maps. In some examples, the geometry information is described by a set of 3D positions associated with the vertices of the mesh. In one example, (x, y, z) coordinates can be used to describe the 3D positions of the vertices, also referred to as 3D coordinates. In some examples, the connectivity information includes a set of vertex indices that describe how to connect the vertices to create a 3D surface. In some examples, the mapping information describes how to map the mesh surface to a planar 2D region. In an example, the mapping information is described by a set of UV parametric / texture coordinates (u, v) associated with the mesh vertices along with the connectivity information. In some examples, the vertex attributes include scalar or vector attribute values ​​associated with the mesh vertices. In some examples, the attribute maps include attributes associated with the mesh surface and stored as a 2D image / video. In one example, the mapping between a video (e.g., a 2D image / video) and the mesh surface is defined by the mapping information.

[0119] According to one aspect of the disclosure, several techniques, called UV mapping or mesh parameterization, are used to map the surface of a mesh in a 3D domain to a 2D domain. In some examples, the mesh is partitioned into patches in the 3D domain. A patch is a contiguous subset of the mesh with a boundary formed by boundary edges. A boundary edge of a patch is an edge that belongs to only one polygon of the patch and is not shared by two adjacent polygons in the patch. The vertices of the boundary edges in a patch are referred to as boundary vertices of the patch, and the non-boundary vertices in a patch may be referred to as interior vertices of the patch in some examples.

[0120] In some instances, the mesh of an object is formed by connected triangles, and the mesh can be partitioned into patches, where each patch is a subset of the connected triangles. A boundary edge of a patch is an edge that belongs to only one triangle in the patch and is not shared by adjacent triangles in the patch. The vertices of a boundary edge in a patch are called boundary vertices of the patch, and the non-boundary vertices in a patch may be called interior vertices of the patch in some instances.

[0121] According to one aspect of the present disclosure, in some examples, the patches are each parameterized into a 2D shape (also referred to as a UV patch). The 2D shapes can be packed (e.g., oriented and positioned) into a map, also referred to as an atlas, in some examples. In some examples, the map can be further processed using 2D image or video processing techniques.

[0122] In one example, the UV mapping technique generates a 2D UV atlas (also called a UV map) and one or more texture atlases (also called texture maps) that correspond to patches of the 3D mesh. The UV atlas includes an assignment of 3D vertices of the 3D mesh to 2D points in a 2D domain (e.g., a rectangle). The UV atlas is a mapping from the coordinates of the 3D surface to the coordinates of the 2D domain. In one example, a point in the UV atlas at 2D coordinate (u,v) has a value formed by the coordinates (x,y,z) of the vertex in the 3D domain. In one example, the texture atlas includes color information for the 3D mesh. For example, a point in the texture atlas at 2D coordinate (u,v) (which has a 3D value of (x,y,z) in the UV atlas) has a color that specifies the color attributes of the point at (x,y,z) in the 3D domain. In some examples, the coordinates (x,y,z) of the 3D domain are referred to as 3D coordinates, or xyz coordinates, and the 2D coordinates (u,v) are referred to as uv coordinates.

[0123] According to some aspects of the present disclosure, mesh compression can be performed by representing the mesh using one or more 2D maps (also referred to in some examples as 2D atlases) and then encoding the 2D maps using an image or video codec.

[0124] FIG. 9 illustrates a diagram showing the mapping of a 3D mesh (910) to a 2D atlas (920) in some examples. In the example of FIG. 9, the 3D mesh (910) includes four vertices 1-4 that form four patches A-D. Each patch has a set of vertices and associated attribute information. For example, patch A is formed by vertices 1, 2, and 3 that are connected to a triangle; patch B is formed by vertices 1, 3, and 4 that are connected to a triangle; patch C is formed by vertices 1, 2, and 4 that are connected to a triangle; patch D is formed by vertices 2, 3, and 4 that are connected to a triangle. In some examples, vertices 1, 2, 3, and 4 can have respective attributes, and the triangle formed by vertices 1, 2, 3, and 4 can have respective attributes.

[0125] In one example, 3D patches A, B, C, and D are mapped to a 2D domain, such as a 2D atlas (920), also referred to as a map (920). For example, patch A is mapped to a 2D shape (also referred to as a UV patch) A' in the map (920), patch B is mapped to a 2D shape (also referred to as a UV patch) B' in the map (920), patch C is mapped to a 2D shape (also referred to as a UV patch) C' in the map (920), and patch D is mapped to a 2D shape (also referred to as a UV patch) D' in the map (920). In some examples, coordinates in the 3D domain are referred to as (x,y,z) coordinates, and coordinates in a 2D domain, such as the map (920), are referred to as UV coordinates. Vertices in a 3D mesh can have corresponding UV coordinates in the map (920).

[0126] The map (920) can be a geometry map containing geometry information, or it can be a texture map with color, textile, or other attribute information, or it can be an occupancy map with occupancy information.

[0127] In the example of Figure 9, each patch is represented by a triangle, however, it is noted that a patch may include any suitable number of vertices that are connected to form a contiguous subset of the mesh. In some examples, the vertices in a patch are connected into triangles. It is noted that the vertices in a patch may be connected using other suitable shapes.

[0128] In one example, the geometry information of the vertices can be stored in a 2D geometry map. For example, the geometry map stores the (x,y,z) coordinates of the sampling points at the corresponding points in the 2D geometry map. For example, the geometry map point at the (u,v) location is a three-component vector value that corresponds respectively to the x, y, and z values ​​of the corresponding sampling point in the 3D mesh.

[0129] According to one aspect of the present disclosure, the regions in the map may not be fully occupied. For example, in FIG. 9, the regions outside the 2D shapes A', B', C', and D' are undefined. Sample values ​​of the regions outside the 2D shapes A', B', C', and D' after decoding can be discarded. In some cases, the occupancy map is used to store some additional information for each pixel, such as storing a binary value to identify whether the pixel belongs to a patch or is undefined.

[0130] According to one aspect of the present disclosure, a dynamic mesh is a mesh in which at least one of the components (geometry information, connectivity information, mapping information, vertex attributes, attribute maps) changes over time. A dynamic mesh can be described by a sequence of meshes (also called mesh frames). Because dynamic meshes may contain a large amount of information that changes over time, dynamic meshes may require a large amount of data. Mesh compression techniques enable efficient storage and transmission of media content in mesh representations.

[0131] In some examples, the dynamic mesh can have constant connectivity information, time-varying geometry, and time-varying vertex attributes. In some examples, the dynamic mesh can have time-varying connectivity information. In one example, digital content creation tools typically generate dynamic meshes with time-varying attribute maps and time-varying connectivity information. In some examples, volume acquisition techniques are used to generate the dynamic mesh. Volume acquisition techniques can generate dynamic meshes with time-varying connectivity information, especially under real-time constraints.

[0132] Several techniques are used for mesh compression. In some examples, UV atlas sampling and V-PCC can be used for mesh compression. For example, the UV atlas is sampled in a regular grid to generate a geometry image with the regular grid samples. The connectivity of the regular grid samples can be inferred. The regular grid samples can be considered as points in the point cloud and thus can be coded using a PCC codec, such as the V-PCC codec. UV atlas sampling and V-PCC based methods are based on sampling of the original mesh vertices, and the number of regular grid samples generated by sampling is usually much larger than the number of original vertices. Also, when a constant sampling step is used, regions of dense vertices may be undersampled while regions of sparse vertices may be oversampled. Furthermore, in UV atlas sampling methods, the generated geometry images may not be consistent across frames because patches in the UV atlas may be positioned differently in different frames.

[0133] Figure 10 shows two UV atlas frames (1010) and (1020) in some examples. In one example, the UV atlas frames (1010) and (1020) are consecutive frames. As shown in Figure 10, patches are inconsistently assigned between different UV atlas frames. For example, patch (1011) in UV atlas frame (1010) and patch (1021) in UV atlas frame (1020) may correspond to the same part of an object at different times, but are assigned to different regions of UV atlas frame (1010) and UV atlas frame (1020) and have different orientations.

[0134] UV atlas sampling based methods can suffer from issues such as undersampling, oversampling, more vertices to compress, inconsistent sampling data between frames, etc. These issues can negatively impact the compression ratio.

[0135] Aspects of the present disclosure provide a vertex reordering technique for mesh compression. The vertices of a mesh are reordered according to a certain rule to increase the correlation of adjacent vertices, and thus can increase the coding efficiency using predictive coding methods. In one example, the vertices are reordered to increase the correlation of adjacent vertices in a sequence. In another example, the vertices of a mesh are reordered and rearranged within a two-dimensional (2D) frame to increase the correlation of adjacent vertices in a local region of the 2D frame, and thus the 2D frame can be more efficiently coded by an image / video codec, such as using an intra-prediction technique. In another example, the vertices of different mesh frames are reordered and rearranged within a 2D frame to increase the correlation between the 2D frames, and thus the 2D frame can be more efficiently coded by a video codec, such as using an inter-prediction technique.

[0136] 11 shows a diagram of a framework (1100) for mesh compression in accordance with some embodiments of the present disclosure. The framework (1100) includes a mesh encoder (1110) and a mesh decoder (1150). The mesh encoder (1110) encodes an input mesh (1105) (or a mesh frame in the case of dynamic mesh processing) into a bitstream (1145), and the mesh decoder (1150) decodes the bitstream (1145) to generate a reconstructed mesh (1195) (or a reconstructed mesh frame in the case of dynamic mesh processing).

[0137] The mesh encoder (1110) can be any suitable device, such as a computer, a server computer, a desktop computer, a laptop computer, a tablet computer, a smartphone, a gaming device, an AR device, a VR device, etc. The mesh decoder (1150) can be any suitable device, such as a computer, a client computer, a desktop computer, a laptop computer, a tablet computer, a smartphone, a gaming device, an AR device, a VR device, etc. The bitstream (1145) can be transmitted from the mesh encoder (1110) to the mesh decoder (1150) over any suitable communication network (not shown).

[0138] In the example of FIG. 11, the mesh encoder (1110) includes a vertex reordering module (1120), a 1D / 2D encoder (1130), and an auxiliary data encoder (1140) coupled together. The vertex reordering module (1120) receives an input mesh (1105) and performs reordering to generate reordered vertex mesh information (1125). In some examples, the input mesh (1105) includes 3D position information of the vertices of the mesh in the original order, vertex connectivity information, mapping information that parameterizes the mesh in 2D (e.g., a UV atlas), and other 2D attribute maps (e.g., a 2D color map). The vertex reordering module (1120) can reorder and / or rearrange the vertices to increase attribute correlation of adjacent vertices (e.g., attribute correlation of adjacent vertices in a sequence (e.g., a 1D array), attribute correlation of adjacent vertices in a region of a 2D array, attribute correlation of adjacent frames).

[0139] The vertex reordering module (1120) outputs reordered vertex mesh information (1125). In some examples, the reordered vertex mesh information (1125) does not include connectivity information. In one example, the reordered vertex mesh information (1125) includes vertex information for the vertices of the reordered sequence. For example, the vertex information for each vertex can include various attributes, such as 3D spatial information within the mesh (e.g., xyz coordinates), mapping information to 2D (e.g., uv coordinates), color information (e.g., RGB values), etc.

[0140] In another example, the reordered vertex mesh information (1125) includes vertex information for the vertices in the form of a 2D image, e.g., the reordered vertices are rearranged into samples (pixels) in 2D (e.g., based on raster scanlines, etc.) and the reordered vertex mesh information can form one or more 2D images.

[0141] The 1D / 2D encoder (1130) is configured to encode the reordered vertex mesh information (1125) into a bitstream (1145). If the reordered vertex mesh information (1125) includes vertex information for a reordered sequence of vertices (1D), the 1D / 2D encoder (1130) can encode the vertex information for the reordered sequence of vertices using a 1D encoding technique. If the reordered vertex mesh information (1125) includes a 2D image, the 1D / 2D encoder (1130) can encode the 2D image using an image encoding and / or video encoding technique (e.g., using an image codec or a video codec).

[0142] The vertex reordering module (1120) also generates auxiliary data (1127) including the assistance information. The auxiliary data encoder (1140) receives the auxiliary data (1127) and encodes the auxiliary data (1127) into a bitstream (1145). For example, the vertex reordering module (1120) may reorder the vertex patches on a patch-by-patch basis. The vertex reordering module (1120) may provide a value indicative of the number of vertices in each patch in the auxiliary data (1127). Additionally, in one example, the vertex reordering module (1120) may reorder, for each patch, boundary vertices before non-boundary vertices. The vertex reordering module (1120) may provide a value indicative of the number of boundary vertices in each patch in the auxiliary data (1127).

[0143] In some examples, the auxiliary data encoder (1140) can receive the reconstruction information (1131) from the 1D / 2D encoder (1130). In one example, lossy compression is used in the 1D / 2D encoder (1130). The 1D / 2D encoder (1130) can provide reconstructed vertex information, such as reconstructed boundary coordinates, in the reconstruction information (1131). In one example, the auxiliary data encoder (1140) can determine a boundary coordinate residual based on the original boundary coordinates and the reconstructed boundary coordinates and encode the boundary coordinate residual into a bitstream (1145).

[0144] The operation of the vertex reordering module (1120), the 1D / 2D encoder (1130), and the auxiliary data encoder (1140) are described further in this disclosure.

[0145] In the example of Figure 11, the bitstream (1145) is provided to a mesh decoder (1150). The mesh decoder (1150) includes a 1D / 2D decoder (1160), a connectivity determination module (1180), an auxiliary data decoder (1170), and a mesh reconstruction module (1190) coupled together as shown in Figure 11. In one example, the 1D / 2D decoder (1160) corresponds to the 1D / 2D encoder (1130) and can decode a portion of the bitstream (1145) encoded by the 1D / 2D encoder (1130) to generate decoded information (1165).

[0146] In the example of Figure 11, the auxiliary data decoder (1170) corresponds to the auxiliary data encoder (1140) and can decode a portion of the bitstream (1145) encoded by the auxiliary data encoder (1140) to generate decoded auxiliary data (1175).

[0147] In the example of FIG. 11, a connectivity determination module (1180) can determine (infer) connectivity information (1185) from the decoded information (1165).

[0148] In the example of Figure 11, the decoded information (1165), the decoded auxiliary data (1175), and the connectivity information (1185) are provided to a mesh reconstruction module (1190), which generates a reconstructed mesh (1195) based on the decoded information (1165), the decoded auxiliary data (1175), and the connectivity information (1185). The operation of the 1D / 2D decoder (1160), the auxiliary data decoder (1170), and the connectivity determination module (1180) are described further in this disclosure.

[0149] It is noted that the components of the mesh encoder (1110), such as the vertex reordering module (1120), the auxiliary data encoder (1140), and the 1D / 2D encoder (1130), can each be implemented by a variety of technologies. In one example, the components are implemented by integrated circuits. In another example, the components are implemented using software that can run on one or more processors.

[0150] It is noted that the components of the mesh decoder (1150), such as the 1D / 2D decoder (1160), the auxiliary data decoder (1170), the connectivity determination module (1180), and the mesh reconstruction module (1190), can each be implemented in a variety of ways. In one example, the components are implemented by integrated circuits. In another example, the components are implemented using software that can be executed on one or more processors.

[0151] 12 shows a diagram of a framework (1200) for mesh compression according to some embodiments of the present disclosure. The framework (1200) includes a mesh encoder (1210) and a mesh decoder (1250). The mesh encoder (1210) encodes an input mesh (1205) (or a mesh frame in the case of dynamic mesh processing) into a bitstream (1245), and the mesh decoder (1250) decodes the bitstream (1245) to generate a reconstructed mesh (1295) (or a reconstructed mesh frame in the case of dynamic mesh processing).

[0152] The mesh encoder (1210) can be any suitable device, such as a computer, a server computer, a desktop computer, a laptop computer, a tablet computer, a smartphone, a gaming device, an AR device, a VR device, etc. The mesh decoder (1250) can be any suitable device, such as a computer, a client computer, a desktop computer, a laptop computer, a tablet computer, a smartphone, a gaming device, an AR device, a VR device, etc. The bitstream (1245) can be transmitted from the mesh encoder (1210) to the mesh decoder (1250) over any suitable communication network (not shown).

[0153] In the example of FIG. 12, the mesh encoder (1210) includes a vertex reordering module (1220), a 1D / 2D encoder (1230), an auxiliary data encoder (1240), and a connectivity encoder (1235) coupled together. The vertex reordering module (1220) operates similarly to the vertex reordering module (1120). The vertex reordering module (1220) receives an input mesh (1205) and performs reordering to generate reordered vertex mesh information (1225). In some examples, the input mesh (1205) includes 3D position information of the vertices of the mesh in the original order, vertex connectivity information, mapping information that parameterizes the mesh in 2D (e.g., a UV atlas), and 2D attribute maps. The vertex reordering module (1220) can reorder and / or rearrange the vertices to increase the correlation (e.g., sequential neighbor correlation, spatial neighbor correlation, temporal neighbor correlation) of adjacent vertices.

[0154] The vertex reordering module (1220) outputs reordered vertex mesh information (1225). In some examples, the reordered vertex mesh information (1225) does not include connectivity information. In one example, the reordered vertex mesh information (1225) includes vertex information for the vertices of the reordered sequence. For example, the vertex information for each vertex can include various attributes, such as 3D spatial information within the mesh (e.g., xyz coordinates), mapping information to 2D (e.g., uv coordinates), color information (e.g., RGB values), etc.

[0155] In another example, the reordered vertices mesh information (1225) includes vertex information for vertices in the form of 2D images, e.g., the reordered vertices are rearranged into 2D samples (pixels), and the reordered vertices mesh information can form one or more 2D images.

[0156] The 1D / 2D encoder (1230) is configured to encode the reordered vertex mesh information (1225) into a bitstream (1245). If the reordered vertex mesh information (1225) includes vertex information for a reordered sequence of vertices (1D), the 1D / 2D encoder (1230) can encode the vertex information for the reordered sequence of vertices using a 1D encoding technique. If the reordered vertex mesh information (1225) includes a 2D image, the 1D / 2D encoder (1230) can encode the 2D image using an image encoding and / or video encoding technique (e.g., using an image codec and / or a video codec).

[0157] Further, in the example of Figure 12, the connectivity encoder (1235) can encode the connectivity information of the input mesh (1205) into the bitstream (1245). The connectivity encoder (1235) can use any suitable technique for encoding the connectivity information.

[0158] The vertex reordering module (1220) also generates auxiliary data (1227) including auxiliary information. The auxiliary data encoder (1240) receives the auxiliary data (1227) and encodes the auxiliary data (1227) into a bitstream (1245). For example, the vertex reordering module (1220) may reorder the vertex patches on a patch-by-patch basis. The vertex reordering module (1220) may provide a value indicative of the number of vertices in each patch in the auxiliary data (1227). Additionally, in one example, the vertex reordering module (1220) may reorder, for each patch, boundary vertices before non-boundary vertices. The vertex reordering module (1220) may provide a value indicative of the number of boundary vertices in each patch in the auxiliary data (1227).

[0159] In some examples, the auxiliary data encoder (1240) can receive the reconstructed information (1231) from the 1D / 2D encoder (1230). In one example, lossy compression is used in the 1D / 2D encoder (1230). The 1D / 2D encoder (1230) can provide reconstructed vertex information, such as reconstructed boundary coordinates, in the reconstructed information (1231). In one example, the auxiliary data encoder (1240) can determine a boundary coordinate residual based on the original and reconstructed boundary coordinates and encode the boundary coordinate residual into the bitstream (1245).

[0160] The operations of the vertex reordering module (1220), the auxiliary data encoder (1240), and the 1D / 2D encoder (1230) are similar to the operations of the vertex reordering module (1120), the auxiliary data encoder (1140), and the 1D / 2D encoder (1130), and are described further in this disclosure.

[0161] In the example of Figure 12, the bitstream (1245) is provided to a mesh decoder (1250). The mesh decoder (1250) includes a 1D / 2D decoder (1260), an auxiliary data decoder (1270), a connectivity decoder (1280), and a mesh reconstruction module (1290), coupled together as shown in Figure 12. In one example, the 1D / 2D decoder (1260) corresponds to the 1D / 2D encoder (1230) and can decode a portion of the bitstream (1245) encoded by the 1D / 2D encoder (1230) to generate decoded information (1265).

[0162] In the example of Figure 12, the auxiliary data decoder (1270) corresponds to the auxiliary data encoder (1240) and can decode a portion of the bitstream (1245) encoded by the auxiliary data encoder (1240) to generate decoded auxiliary data (1275).

[0163] In the example of Figure 12, the connectivity decoder (1280) corresponds to the connectivity encoder (1235) and can decode a portion of the bitstream (1245) encoded by the connectivity encoder (1235) to generate a decoded connectivity (1285).

[0164] In the example of Figure 12, the decoded information (1265), the decoded auxiliary data (1275), and the decoded connectivity (1285) are provided to a mesh reconstruction module (1290). The mesh reconstruction module (1290) generates a reconstructed mesh (1295) based on the decoded information (1265), the decoded auxiliary data (1275), and the decoded connectivity (1285). The operation of the 1D / 2D decoder (1260) and the connectivity decoder (1280) are described further in this disclosure.

[0165] It is noted that components in the mesh encoder (1210), such as the vertex reordering module (1220), the auxiliary data encoder (1240), the 1D / 2D encoder (1230), and the connectivity encoder (1235), can each be implemented in a variety of techniques. In one example, the components are implemented by integrated circuits. In another example, the components are implemented using software that can be executed on one or more processors.

[0166] It is noted that components in the mesh decoder (1250), such as the 1D / 2D decoder (1260), the auxiliary data decoder (1270), the connectivity decoder (1280), and the mesh reconstruction module (1290), can each be implemented in a variety of techniques. In one example, the components are implemented by integrated circuits. In another example, the components are implemented using software that can be executed on one or more processors.

[0167] According to some aspects of the present disclosure, the vertices of a mesh may be reordered on a patch-by-patch basis based on spatial properties of the vertices within the mesh.

[0168] FIG. 13 shows a flowchart outlining a process (1300) for reordering vertices in a mesh (mesh frame for dynamic mesh processing) according to some embodiments of the present disclosure. In some examples, the process (1300) is performed by a vertex reordering module, such as the vertex reordering module (1120), the vertex reordering module (1220), etc. In various implementations, the process (1300) is performed by a processing circuit. In some embodiments, the process (1300) is implemented with software instructions, such that the processing circuit performs the process (1300) when the processing circuit executes the software instructions. The process (1300) starts at (S1301) and proceeds to (S1310).

[0169] At (S1310), a starting vertex in a mesh is determined, and a current patch containing the starting vertex is determined. In some examples, the mesh includes multiple vertices, and the vertices can be connected by edges. In some examples, the mesh includes triangles formed by edges. Two triangles are connected if they share an edge. The mesh is partitioned into patches in the 3D domain. A patch is a contiguous subset of the mesh with connected triangles. In one example, if an edge in a patch is not shared by two adjacent triangles of the patch, e.g., if the edge belongs to only one triangle of the patch, then the edge is a boundary edge of the patch, and the vertices of the boundary edge are boundary vertices. The vertices in a patch can be classified as boundary vertices and non-boundary vertices.

[0170] In some embodiments, vertex attributes with specific spatial properties are used to select a starting vertex such that vertex traversal begins at approximately the same location for each frame in a group of mesh frames in a dynamic mesh. In one example, a group of mesh frames is a similar concept to a 2D group of pictures (GOP).

[0171] In some examples, the starting vertex is selected based on its 3D coordinates. In one example, the vertex with the largest vertical coordinate (e.g., the largest z coordinate) is selected as the starting vertex, so the starting vertex is the tallest vertex. It is noted that other attributes such as Gaussian curvature, local vertex density, color information, etc. can be used to select the starting vertex.

[0172] After the starting vertex is selected, the corresponding patch containing the starting vertex is determined as the first patch to be processed. Thus, the first patch is the current patch for processing, and the starting vertex is also the starting vertex of the current patch.

[0173] At (S1320), the vertices of the current patch are traversed starting from the start vertex of the current patch. The vertices of the current patch can be traversed using various techniques. In one example, a breadth-first search can be performed for the traversal. In another example, a depth-first search can be performed for the traversal. In another example, the boundary vertices of the current patch are traversed first and the interior vertices (also called non-boundary vertices) of the current patch are traversed later. Thus, the boundary vertices are better grouped together. In some examples, lossy compression is used to encode the vertex attributes, but additional processing can be applied to the boundary vertices so that their coordinates can be compressed without information loss (e.g., lossless compression).

[0174] In step S1330, if there are any unvisited patches (for which vertex traversal has not been performed), the process proceeds to step S1340. Otherwise, all patches are processed (traversed) and the process proceeds to step S1399, where it ends.

[0175] In (S1340), a patch for the next traversal after the current patch (e.g., the old current patch whose vertices have been traversed) is determined, and the determined next patch is set to be the current patch for processing (e.g., the new current patch for further traversal processing). For example, after traversing all the vertices of the first patch, the patch for traversal after the first patch is determined. Various techniques can be used to determine the next patch. In one example, the next patch is selected as the patch that includes the highest vertex among the unvisited (untraversed) vertices. For example, if the first patch is selected as the patch that includes the highest vertex, the next patch of the first patch can be determined as the patch that includes the vertex with the largest vertical coordinate among the unvisited vertices.

[0176] In some examples, the patch that is closest (e.g., has the shortest distance) to the current patch (e.g., the old current patch whose vertices were traversed) is selected as the next patch for traversal. The distance between the current patch (e.g., the old current patch whose vertices were traversed) and another patch can be defined by various techniques. In one example, for an unvisited patch, a 3D Euclidean distance between each vertex of the unvisited vertices and the last visited vertex of the current patch (e.g., the old current patch whose vertices were traversed) is calculated. Then, among the 3D Euclidean distances, the smallest 3D Euclidean distance is determined as the distance between the current patch (the old current patch) and the unvisited patch.

[0177] In another example, the average 3D coordinates (average x-coordinate, average y-coordinate, average z-coordinate) of all vertices in an unvisited patch are calculated. Then, the 3D Euclidean distance between the point of the average 3D coordinate of the unvisited patch and the last visited vertex in the current patch (e.g., the old current patch) is calculated, defined as the distance between the current patch (old current patch) and the unvisited patch.

[0178] In another example, a first average 3D coordinate (average x-coordinate, average y-coordinate, average z-coordinate) of all vertices in a current patch (e.g., old current patch) is calculated, and a second average 3D coordinate (average x-coordinate, average y-coordinate, average z-coordinate) of all vertices in an unvisited patch is calculated. Then, the distance between a first point with the first average 3D coordinate and a second point with the second average 3D coordinate is calculated and defined as the distance between the current patch and the unvisited patch.

[0179] At (S1350), a starting vertex of the current patch (e.g., the new current patch) is determined to traverse. Various techniques can be used to select a starting vertex to traverse in the current patch (e.g., the new current patch). In one example, the highest vertex (e.g., with the highest z coordinate) in the current patch (e.g., the new current patch) is selected as the starting vertex. In another example, the vertex closest to the last visited vertex (in the old current patch) in the current patch (e.g., the new current patch) is selected as the starting vertex in the current patch. In another example, the highest boundary vertex in the current patch (e.g., the new current patch) is selected as the starting vertex, such that the boundary vertices in the current patch (e.g., the new current patch) will be traversed before the interior vertices (e.g., non-boundary vertices) in the current patch (e.g., the new current patch). In another example, the boundary vertex that is the closest boundary vertex (e.g., in the new current patch) to the last visited vertex (in the old current patch) is selected to be the starting vertex, such that the boundary vertices in the current patch (e.g., the new current patch) are traversed before the interior vertices (e.g., non-boundary vertices) in the current patch (e.g., the new current patch). Then, the process (S1300) returns to (S1320).

[0180] According to one aspect of the disclosure, after the vertices in a mesh are traversed, e.g., according to process (1300), attribute information of the vertices, such as xyz coordinates (also called 3D coordinates), uv coordinates, RGB color, normals, etc., of the vertices in the mesh are recorded according to the traversal order, e.g., in the sorted mesh information (1125) or the sorted mesh information (1225). In some examples, after each patch is visited, the number of vertices in the patch and the number of boundary vertices in the patch are recorded, e.g., in the auxiliary data (1127), auxiliary data (1227), etc.

[0181] In some examples, the reordered mesh information (1125) or the reordered mesh information (1225) includes a vertex attribute array, such as an array of xyz coordinates, an array of uv coordinates, or an array of other attributes (e.g., normals, colors, etc.). In one example, the vertex attribute array is a 1D array of vertices in traverse order. The 1D array can be encoded by the 1D / 2D encoder (1130) or the 1D / 2D encoder (1230).

[0182] In another example, the vertices in the traverse order can be rearranged into a 2D format, such as according to a raster scan line pattern. The attributes of the vertices can then be rearranged into a 2D image according to the rearranged 2D format. In a multiple mesh frame example (e.g., dynamic mesh compression), after a mesh frame or a group of mesh frames in a sequence is visited, the number of vertices in the frame (in the sequence or in the group) can be determined. A maximum number of vertices is then determined from the number of vertices in the mesh frames. Based on the maximum number of vertices, a width (denoted by m) and a height (denoted by n) of the 2D image can be determined. In one example, the width is the number of pixels in the horizontal direction and the height is the number of pixels in the vertical direction, and the product of the width and height is equal to or greater than the maximum number of vertices for the group of mesh frames. For each mesh frame, if the number of vertices in the mesh frame is less than the product of the width (m) and the height (n), padding vertices (having a padding attribute value) can be added accordingly.

[0183] According to one aspect of the present disclosure, attributes such as xyz coordinates, uv coordinates, normals, and vertex colors of each mesh frame are reshaped into an m×n 2D image. Thus, the attributes of the mesh frames such as xyz coordinates, uv coordinates, etc. are stored in a 2D image of the same size, and the 2D image can be encoded using a 2D video codec. For example, the xyz coordinates of the vertices in the mesh frame can be reshaped into an m×n×3 2D image (3 channels). The uv coordinates of the vertices in the mesh frame can be reshaped into an m×n×2 2D image (2 channels).

[0184] In one example, the xyz coordinates of the vertices in the mesh frame can be stored as color components of a 2D image, such as using an RGB 4:4:4 color format. In another example, the uv coordinates of the vertices in the mesh frame can be split and stored in two groups of a one-channel 2D image, such as using a 4:0:0 color format. For example, the u coordinates of the vertices in the mesh frame are stored in a first group of a one-channel 2D image, and the v coordinates of the vertices in the mesh frame are stored in a second group of a one-channel 2D image.

[0185] In another example, a transformation function (e.g., a two-tap Haar transform) can be applied to the 2D images with uv coordinates to decorrelate the two channels. The two-channel decorrelated 2D images can then be split into two groups of one-channel 2D images. The two groups of 2D images can be encoded separately.

[0186] In another example, the uv coordinates of vertices in a mesh frame can be stored as color components (three channels) of a 2D image, such as using the RGB 4:4:4 format. For example, the u coordinate can be stored as the first channel, the v coordinate as the second channel, and the third channel can be filled with any number and discarded at the decoder side.

[0187] In some examples, lossy compression is used in an encoder, such as the 1D / 2D encoder (1130) and the 1D / 2D encoder (1230), to encode vertex attributes of the mesh. To preserve more information of the boundary vertices of the mesh (e.g., to avoid gaps between patches), the reconstructed coordinates (e.g., xyz coordinates, uv coordinates) of the boundary vertices (e.g., in the reconstructed information (1131), the reconstructed information (1231)) can be determined from the encoded vertex attributes, and differences between the original vertex coordinates and the reconstructed coordinates of the boundary vertices can be calculated. The differences between the original vertex coordinates and the reconstructed coordinates of the boundary vertices are denoted as boundary coordinate residuals. The boundary coordinate residuals can be coded into a bitstream carrying the mesh, such as the bitstream (1145), the bitstream (1245), for example, by the auxiliary data encoder (1140), the auxiliary data encoder (1240), etc.

[0188] According to some aspects of the present disclosure, a bitstream carrying a 3D mesh (e.g., bitstream (1145), bitstream (1245)) includes vertex attributes such as xyz coordinates, uv coordinates, etc. in the form of a 1D array or a 2D image, and includes auxiliary data to aid in the accurate decoding and reconstruction of the 3D mesh.

[0189] In some examples, for each mesh frame, attributes such as xyz coordinates, uv coordinates, etc. can be decoded from the bitstream by a 2D video codec or from other compressed formats such as using a 1D / 2D decoder (1160), a 1D / 2D decoder (1260), etc. For example, each pixel of a 2D frame output from a 2D video decoder represents vertex attributes such as xyz coordinates, uv coordinates, RGB color, or normal. In one example, a 4:4:4 picture format can be used to represent the xyz coordinates of all vertices in the mesh frame. Two 1-channel 4:0:0 format pictures can be used to represent the uv coordinates of all vertices in the mesh frame.

[0190] In some examples, the auxiliary data is decoded from the bitstream by the auxiliary data decoder (1170), the auxiliary data decoder (1270), etc. In one example, the auxiliary data includes the number of vertices and the number of bound vertices in each patch. In another example, the auxiliary data includes the number of vertices and the number of bound vertices in each patch, and includes bound coordinate residuals.

[0191] In some examples, the decoded vertices of a mesh frame can be grouped by patch based on the number of vertices in each patch and the number of boundary vertices. In general, the coded vertices in a picture are arranged by patch or in any order that the decoder can recover. Mesh reconstruction can then be performed by patch for each mesh frame.

[0192] According to one aspect of the present disclosure, for a patch, the boundary vertices in the patch can be identified according to the number of vertices in the patch and the number of boundary vertices, and the xyz and uv coordinates of the vertices in the patch can be determined from the decoded information, e.g., from the output from the 1D / 2D decoder (1160), the 1D / 2D decoder (1260), etc. In one example, both the encoder side and the decoder side agree that, for each patch, all boundary vertices are placed before all other interior patch vertices (e.g., non-boundary vertices). Thus, the boundary vertices of each patch can be identified with all the decoded vertices of the patch.

[0193] In some examples using lossy compression, the boundary coordinate residuals are decoded (e.g., by the auxiliary data decoder (1170), by the auxiliary data decoder (1270)) and added to the decoded boundary coordinates (being the difference between the original coordinates and the decoded coordinates) to obtain the original xyz and uv coordinates of the boundary vertices, by the mesh reconstruction module (1190), the mesh reconstruction module (1290), etc.

[0194] In some examples, in the case of lossy compression, duplicate vertices with the same xyz or uv coordinates may be present in the decoded information, and the duplicate vertices are removed. Additionally, in some examples, some decoded vertices that correspond to interior vertices of a patch (original interior vertices in the original mesh) may be outside the patch (e.g., outside the boundary formed by the boundary edges of the patch) due to lossy compression, and the decoded vertices that are outside the patch can be removed.

[0195] In some examples, to obtain connectivity, the uv coordinates of all vertices of a patch and the boundary vertices of the patch are used to perform triangulation such as constrained Delaunay triangulation to obtain (infer) the connectivity of the patch. In some other examples, the connectivity information of each patch can be explicitly signaled in the bitstream so that the decoder side can directly reconstruct the topology of the mesh without a triangulation algorithm.

[0196] FIG. 14 shows a flow chart outlining a process (1400) according to an embodiment of the disclosure. The process (1400) may be used during mesh encoding. In various embodiments, the process (1400) is performed by a processing circuit. In some embodiments, the process (1400) is implemented with software instructions, and thus, the processing circuit performs the process (1400) when the processing circuit executes the software instructions. The process starts at (S1401) and proceeds to (S1410).

[0197] At (S1410), a traversal order of the vertices of the 3D mesh frame is determined. The 3D mesh frame is partitioned into patches, and the traversal order reorders the vertex patches by patch, e.g., the vertices are reordered into subsets that each belong to a patch.

[0198] At (S1420), an array of vertex attributes is formed according to the traversal order of the vertices of the 3D mesh frame.

[0199] At (S1430), the array of attributes is encoded into a bitstream carrying the 3D mesh frame.

[0200] To determine the traversal order, in some embodiments, a starting vertex is determined from the vertices of the 3D mesh frame and a first patch of the first subset of vertices is determined. The starting vertex is a vertex in the first subset of vertices. In some examples, the starting vertex is determined according to a spatial property of the 3D mesh frame that is maintained over the sequence of 3D mesh frames. Thus, the starting vertex for each 3D mesh frame in the sequence of 3D mesh frames is at approximately the same location for each 3D mesh frame. In one example, the starting vertex has a maximum coordinate value. In another example, the starting vertex has a minimum coordinate value. In another example, the starting vertex has a maximum Gaussian curvature. In another example, the starting vertex has a minimum Gaussian curvature. In another example, the starting vertex has a maximum local vertex density. In another example, the starting vertex has a minimum local vertex density. In another example, the starting vertex has a maximum value for color information. In another example, the starting vertex has a minimum value for color information.

[0201] In some examples, from a starting vertex for the 3D mesh frame, a first subset of vertices are traversed before traversing the first exterior vertices, hi some examples, boundary vertices of the first patch are traversed before traversing the interior (non-boundary) vertices of the first patch.

[0202] In some examples, after traversing the last vertex of the previous patch (also called the old current patch), the next patch to traverse (also called the new current) is determined from the untraversed patch according to the spatial relationship of the previous patch to the untraversed patch. Then, the start vertex of the next patch is determined.

[0203] In some examples, the next patch is the closest patch to the previous patch among the untraversed patches. In one example, to calculate the distance between the untraversed patch and the previous patch, a 3D Euclidean distance between the last vertex traversed in the previous patch and each vertex of the untraversed patch is calculated, and the minimum distance of the 3D Euclidean distances is determined to be the distance between the untraversed patch and the previous patch. In another example, to calculate the distance between the untraversed patch and the previous patch, an average 3D coordinate of a plurality of vertices of the untraversed patch is calculated, and a distance between the untraversed patch and the previous patch is determined based on a distance between a point having the average 3D coordinate and the last vertex traversed in the previous patch. In another example, to calculate the distance between the untraversed patch and the previous patch, a first average 3D coordinate of a first plurality of vertices of the previous patch is calculated, and a second average 3D coordinate of a plurality of vertices of the untraversed patch is calculated, and a distance between the untraversed patch and the previous patch is calculated based on a distance between a first point having the first average 3D coordinate and a second point having the second average 3D coordinate.

[0204] To determine the start vertex of the next patch, in one example, the vertex of the next patch with the maximum coordinate value is selected as the start vertex in the next patch. In another example, the vertex in the next patch with the minimum coordinate value is selected as the start vertex in the next patch. In another example, the vertex in the next patch that is closest to the last traversed vertex of the previous patch is selected as the start vertex in the next patch. In another example, the boundary vertex in the next patch with the maximum coordinate value is selected as the start vertex in the next patch. In another example, the boundary vertex in the next patch with the minimum coordinate value is selected as the start vertex in the next patch. In another example, the boundary vertex in the next patch that is closest to the last vertex traversed in the previous patch is selected as the start vertex in the next patch.

[0205] According to one aspect of the present disclosure, to form an array of attributes, in some examples, after vertex traversal of a sequence of mesh frames, a maximum number of vertices in the mesh frames is determined. Based on the maximum number of vertices, a 2D image size can be determined. The attributes of the vertices in each of the mesh frames can then be reshaped into a 2D array of the 2D image size.

[0206] To encode the array of attributes, in one example, a 2D image corresponding to an array of 3D coordinates of the vertices is encoded into the bitstream. In another example, a 2D image corresponding to an array of uv coordinates of the vertices is encoded into the bitstream. In another example, a first 2D image and a second 2D image corresponding to a first array of u coordinates of the vertices and a second array of v coordinates of the vertices, respectively, are encoded into the bitstream. In some examples, a 2D image corresponding to an array of other attributes of the vertices (e.g., color information, normals, etc.) is encoded into the bitstream.

[0207] In some examples, for each patch, a first value is encoded into the bitstream that indicates the number of vertices in the patch. Additionally, in some examples, for each patch, a second value is encoded into the bitstream that indicates the number of boundary vertices in the patch.

[0208] In some examples, attributes of boundary vertices in a patch are encoded into the bitstream before attributes of other vertices (eg, non-boundary vertices) in the patch.

[0209] In some examples, lossy compression is used. At the encoder side, for a patch, reconstructed attributes of the bound vertices of the patch are determined based on the encoded attributes of the vertices. Then, boundary coordinate residuals for the bound vertices of the patch are determined based on the reconstructed coordinates of the bound vertices and the original coordinates of the bound vertices. The boundary coordinate residuals are encoded into the bitstream. The coordinates of the bound vertices can be the 3D coordinates of the bound vertices and / or the uv coordinates of the bound vertices.

[0210] In some examples, connectivity information for vertices within each patch is encoded into the bitstream.

[0211] Thereafter, the process proceeds to (S1499) and ends.

[0212] The process 1400 may be adapted as appropriate. Step(s) of the process 1400 may be modified and / or omitted. Additional step(s) may be added. Any suitable order of implementation may be used.

[0213] FIG. 15 shows a flow chart outlining a process (1500) according to one embodiment of the present disclosure. The process (1500) may be used during a mesh decoding process. In various embodiments, the process (1500) is performed by a processing circuit. In some embodiments, the process (1500) is implemented with software instructions, and thus, when the processing circuit executes the software instructions, the processing circuit performs the process (1500). The process starts at (S1501) and proceeds to (S1510).

[0214] At (S1510), an array of attributes is decoded from a bitstream carrying a 3D mesh frame including a plurality of patches. The array of attributes corresponds to the vertices of the 3D mesh frame ordered by patch. For example, the vertices are ordered into subsets that each belong to a plurality of patches.

[0215] At (S1520), a first portion of the array of attributes corresponding to a first subset of vertices of the 3D mesh frame is determined, the first subset belonging to a first patch in the plurality of patches.

[0216] At (S1530), first connectivity information for a first subset of vertices is determined.

[0217] At (S1540), a first patch of the 3D mesh frame is reconstructed based on the first portion of the array of attributes and the first connectivity information.

[0218] To decode the array of attributes, in one example, a 2D image corresponding to the array of 3D coordinates of the vertices is decoded, in another example, a 2D image corresponding to the array of uv coordinates of the vertices is decoded, or in another example, a first 2D image and a second 2D image corresponding to a first array of u coordinates of the vertices and a second array of v coordinates of the vertices, respectively, are decoded.

[0219] To determine the first portion of the array of attributes, in one example, a first value indicating a number of vertices in a first subset of vertices of the 3D mesh frame is decoded from the bitstream. The first portion of the array of attributes is selected based on the first value. In some examples, a second value indicating a number of boundary vertices in the first subset is decoded from the bitstream. Then, attributes of the boundary vertices are determined from the first portion of the array of attributes based on the second value.

[0220] In some examples, attributes of boundary vertices of the first patch are ordered before attributes of other vertices in the first subset for the first patch.

[0221] In some examples, lossy compression is used to code vertex attributes, such as the coordinates of the vertices. In one example, boundary coordinate residuals for the boundary vertices are decoded from the bitstream. The decoded coordinates of the boundary vertices are combined with the boundary coordinate residuals of the boundary vertices to obtain the recovered coordinates of the boundary vertices. The recovered coordinates of the boundary vertices may be equal to the original coordinates of the boundary vertices on the encoder side. The coordinates of the boundary vertices may be the 3D coordinates of the boundary vertices or may be the uv coordinates of the boundary vertices.

[0222] In some examples, due to lossy compression, there may be duplicated vertices (having the same 3D coordinates or the same uv coordinates) according to the decoded attributes. In one example, a first vertex and a second vertex are detected to have the same coordinates, and one of the first vertex and the second vertex is removed from the reconstruction of the 3D mesh frame.

[0223] In some examples, due to lossy compression, non-boundary vertices of a patch may be outside the boundary of the patch. In one example, non-boundary vertices of a patch that are outside the boundary formed by the boundary vertices of the patch are removed from the reconstruction of the 3D mesh frame.

[0224] In some examples, first connectivity information for a first subset of vertices of the first patch is inferred based on the first subset of vertices of the 3D mesh frame.

[0225] In some examples, first connectivity information for a first subset of vertices is decoded from the bitstream.

[0226] Thereafter, the process proceeds to (S1599) and ends.

[0227] The process (1500) may be adapted as appropriate. Step(s) of the process (1500) may be modified and / or omitted. Additional step(s) may be added. Any suitable order of implementation may be used.

[0228] The techniques disclosed in this disclosure may be used separately or in combination in any order. Furthermore, each technique (e.g., method, embodiment), encoder, and decoder may be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In some examples, the one or more processors execute a program stored on a non-transitory computer-readable medium.

[0229] The techniques described above can be implemented as computer software using computer readable instructions and physically stored on one or more computer readable media. For example, Figure 16 illustrates a computer system (1600) suitable for implementing certain embodiments of the disclosed subject matter.

[0230] The computer software can be coded using any suitable machine code or computer language that can be subject to mechanisms such as assembly, compilation, linking, etc. to generate code including instructions that can be executed by one or more central processing units (CPUs), graphics processing units (GPUs), etc., directly, or through interpretation, microcode execution, etc.

[0231] The instructions may be executed in various types of computers or components thereof including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, and the like.

[0232] 16 with respect to computer system (1600) are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing embodiments of the present disclosure. The arrangement of components should not be interpreted as having any dependency or requirement regarding any one or combination of components illustrated in the exemplary embodiment of computer system (1600).

[0233] The computer system (1600) may include certain human interface input devices. Such human interface input devices may be responsive to input by one or more users through, for example, tactile input (e.g., keyboard, swipes, dataglove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), olfactory input (not shown). Human interface devices may also be used to capture certain media that are not necessarily directly related to conscious input by a person, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images obtained from a still camera), video (e.g., two-dimensional video, three-dimensional video including stereoscopic video), and the like.

[0234] The input human interface devices may include one or more of a keyboard (1601), a mouse (1602), a trackpad (1603), a touch screen (1610), a tag globe (not shown), a joystick (1605), a microphone (1606), a scanner (1607), and a camera (1608) (only one of each is depicted).

[0235] The computer system (1600) may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the user's senses, for example, through haptic output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touch screen (1610), tag gloves (not shown), or joystick (1605), although there may be haptic feedback devices that do not function as input devices), audio output devices (e.g., speakers (1609), headphones (not shown)), visual output devices (e.g., screens (1610) including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capability, each with or without haptic feedback capability, some of which may provide two-dimensional visual output or output in more than three dimensions, such as by means of stereoscopic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0236] The computer system (1600) may also include human accessible storage devices and their associated media, such as CD / DVD ROM / RW (1620) along with CD / DVD or similar media (1621), thumb drives (1622), removable hard disks or solid state drives (1623), legacy magnetic media such as tapes and floppy disks (not shown), dedicated ROM / ASIC / PLD based devices such as security dongles (not shown), and the like.

[0237] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other transitory signals.

[0238] The computer system (1600) may also include an interface (1654) to one or more communication networks (1655). The networks may be, for example, wireless, wireline, optical. The networks may further be local, wide area, metropolitan, vehicular and industrial, real-time, delay tolerant, etc. Examples of networks include local area networks such as Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., TV wireline or wireless wide area digital networks including cable TV, satellite TV, and terrestrial broadcast TV, vehicular and industrial networks including CAN bus, etc. Certain networks generally require an external network interface adapter attached to a particular general purpose data port or peripheral bus (1649) (e.g., a USB port of the computer system (1600)). Others are generally built into the core of the computer system (1600) by attachment to a system bus as described below (e.g., an Ethernet network to a PC computer system, or a cellular network interface to a smartphone computer system). Using any of these networks, computer system (1600) can communicate with other entities. Such communications can be one-way receive-only (e.g., broadcast TV), one-way transmit-only (e.g., a CAN bus to a specific CAN bus device), or bidirectional to other computer systems, for example, using local or wide area digital networks. Specific protocols or protocol stacks can be used in each of the networks and network interfaces as described above.

[0239] The aforementioned human interface devices, human accessible storage devices, and network interfaces may be attached to a core (1640) of the computer system (1600).

[0240] The core (1640) may include one or more central processing units (CPUs) (1641), graphics processing units (GPUs) (1642), dedicated programmable processing units in the form of field programmable gate areas (FPGAs) (1643), hardware accelerators (1644) for specific tasks, graphics adapters (1650), etc. These devices may be connected through a system bus (1648), along with read only memory (ROM) (1645), random access memory (RAM) (1646), internal mass storage devices such as internal non-user accessible hard drives, SSDs, etc. (1647). In some computer systems, the system bus (1648) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached to the core's system bus (1648) directly or through a peripheral bus (1649). Architectures for peripheral buses include PCI, USB, etc.

[0241] The CPU (1641), GPU (1642), FPGA (1643), and accelerator (1644) can execute certain instructions that, in combination, may constitute the above computer code. The computer code may be stored in a ROM (1645) or a RAM (1646). Temporary data may also be stored in the RAM (1646), while persistent data may be stored, for example, in an internal mass storage device (1647). Rapid storage and retrieval from any of the memory devices may be enabled through the use of a cache memory, which may be closely associated with one or more of the CPU (1641), GPU (1642), mass storage device (1647), ROM (1645), RAM (1646), etc.

[0242] The computer-readable medium can bear computer code for performing various computer-implemented operations. The medium and computer code can be those specially designed and constructed for the purposes of the present disclosure, or they can be of the kind well known and available to those of ordinary skill in the computer software arts.

[0243] By way of example, and not by way of limitation, a computer system having the architecture (1600), and in particular the core (1640), can provide functionality as a result of a processor(s) (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be media associated with the user-accessible mass storage devices introduced above, in addition to specific storage of the core (1640) that is non-transitory in nature, such as the core's internal mass storage (1647) or ROM (1645). Software implementing various embodiments of the present disclosure can be stored in such devices and executed by the core (1640). The computer-readable media can include one or more memory devices or chips, depending on the particular needs. The software can cause the core (1640) and in particular the processors therein (including CPUs, GPUs, FPGAs, etc.) to perform certain processes or certain portions of certain processes described herein, including defining data structures stored in RAM (1646) and modifying such data structures according to the software-defined processes. Additionally or alternatively, the computer system may provide functionality as a result of hardwired or otherwise embodied logic in circuitry (e.g., accelerator (1644)) that may operate in place of or in conjunction with software to perform particular processes or particular portions of particular processes described herein. References to software may encompass logic, and vice versa, where appropriate. References to computer-readable media may encompass circuitry (e.g., integrated circuits (ICs)) that store software for execution, circuitry that embodies logic for execution, or both, where appropriate. The present disclosure encompasses any appropriate combination of hardware and software.

[0244] While this disclosure has described several non-limiting exemplary embodiments, there are modifications, permutations, and various substitute equivalents that are within the scope of this disclosure. It is thus recognized that those skilled in the art will be able to devise numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the disclosure and are therefore within its spirit and scope.

Claims

1. 1. A method for mesh decompression comprising: decoding an array of attributes from a bitstream carrying a three-dimensional (3D) mesh frame including a plurality of patches, the array of attributes corresponding to vertices of the 3D mesh frame, the vertices of the 3D mesh frame being ordered into subsets each belonging to a respective one of the plurality of patches; determining a first portion of the array of attributes corresponding to a first subset of the vertices of the 3D mesh frame, the first subset belonging to a first patch; determining first connectivity information for the first subset of vertices; reconstructing the first patch of the 3D mesh frame based on the first portion of the array of attributes and the first connectivity information; The step of decoding the array of attributes comprises: decoding a two-dimensional (2D) image corresponding to an array of three-channel elements indicating the three-dimensional coordinates of the vertices; decoding a 2D image corresponding to an array of two-channel elements indicating the uv coordinates of the vertices; or decoding a first 2D image and a second 2D image corresponding to the first array of u coordinates of the vertices and the second array of v coordinates of the vertices, respectively. method.

2. The step of determining the first portion of the array of attributes comprises: decoding, from the bitstream, a first value indicative of a number of vertices in the first subset of vertices of the 3D mesh frame; selecting the first portion of the array of attributes based on the first value; The method of claim 1.

3. decoding, from the bitstream, a second value indicative of a number of boundary vertices in the first subset; determining an attribute of the bounding vertex from the first portion of the array of attributes based on the second value; The method of claim 2.

4. selecting, from the first portion of the array of attributes, the attribute of the boundary vertex that is ordered before the attributes of other vertices in the first subset. The method according to claim 3.

5. The attributes include coordinates, and the method further comprises: decoding boundary coordinate residuals for the boundary vertices; combining the coordinates of the bounding vertices with the boundary coordinate residuals of the bounding vertices to obtain recovered coordinates of the bounding vertices. The method according to claim 3.

6. the coordinates of the bounds vertices include at least one of three-dimensional (3D) coordinates of the bounds vertices and / or uv coordinates of the bounds vertices, and the boundary coordinate residuals for the bounds vertices include at least one of boundary 3D coordinate residuals and / or boundary uv coordinate residuals. The method according to claim 5.

7. detecting that the first vertex and the second vertex have the same coordinates; and deleting one of the first vertex and the second vertex from the reconstructing step. The method according to claim 5.

8. detecting non-boundary vertices of the first patch that are outside a boundary formed by the boundary vertices of the first patch; reconstructing the first patch of the 3D mesh frame without the non-boundary vertices. The method according to claim 5.

9. The step of determining the first connectivity information for the first subset of vertices further comprises: inferring the first connectivity information for the first subset of vertices. The method of claim 1.

10. The step of determining the first connectivity information for the first subset of vertices further comprises: decoding the first connectivity information for the first subset of the vertices from the bitstream; The method of claim 1.

11. An apparatus for mesh decompression, comprising a processing circuit configured to carry out the method according to any one of claims 1 to 10. Device.

12. A computer program product which, when executed by a computer, causes at least one processor of said computer to carry out the method according to any one of claims 1 to 10.

Citation Information

Patent Citations

  • Three-Dimensional Mesh Compression Using a Video Encoder

    US20210090301A1

  • Point cloud data transmission device, point cloud data transmission method, point cloud data reception device, and point cloud data reception method

    US20210209806A1

  • 2D mesh geometry and motion vector compression

    US6047088A

  • Mesh compression via point cloud representation

    WO2021116838A1

  • An apparatus, a method and a computer program for volumetric video

    WO2021136876A1