METHOD, APPARATUS AND PROGRAM FOR MESH EXPANSION
Adaptive sampling of 2D maps associated with 3D meshes, combined with video encoding, addresses the high data requirements of 3D mesh encoding, enhancing storage and transmission efficiency.
Patent Information
- Application Number
- JP2023531691
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2022-09-06
- Filing Date
- 2022-09-07
- Publication Date
- 2024-11-13
- Estimated Expiration
- 2042-09-07
AI Technical Summary
Existing technologies face challenges in efficiently encoding and decoding three-dimensional (3D) meshes due to the large data requirements for immersive experiences, leading to high storage and transmission costs.
The use of adaptive sampling techniques for 2D maps associated with 3D meshes, where different sampling rates are applied to various portions of the maps based on information density, combined with video encoding and decoding methods to compress and decompress 3D mesh frames.
This approach reduces the bandwidth required for representing mesh information while maintaining quality by adaptively sampling 2D maps, thus optimizing storage and transmission efficiency.
Smart Images

Figure 0007583171000001 
Figure 0007583171000002 
Figure 0007583171000003
Abstract
Description
[Technical field]
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to U.S. Provisional Application No. 63 / 241,478, entitled "2D Atlas Adaptive Sampling in 3D Mesh Compression," filed September 7, 2021, which claims the benefit of priority to U.S. Provisional Application No. 17 / 903,634, entitled "2D Atlas Adaptive Sampling in 3D Mesh Compression," filed September 6, 2022. The disclosures of the prior applications are incorporated herein by reference in their entireties.
[0002] This disclosure generally describes embodiments related to mesh coding. [Background technology]
[0003] The discussion of the background art provided herein is intended to provide a general context for the present disclosure. The work of the inventors named herein is not admitted, expressly or impliedly, as prior art to the present disclosure to the extent that their work is described in this Background section, and aspects of the description that may not otherwise qualify as prior art at the time of filing.
[0004] Various techniques have been developed to capture and represent the world, including the world's objects, the world's environment, and so on, in three-dimensional (3D) space. The 3D representation of the world may enable more immersive interaction, and more immersive communication. In some examples, point clouds and meshes can be used as the 3D representation of the world. Summary of the Invention [Means for solving the problem]
[0005] Aspects of the present disclosure provide a method and apparatus for mesh coding (e.g., compression and decompression). In some examples, the apparatus for mesh coding includes a processing circuit. The processing circuit decodes a two-dimensional multiple initial maps from a bitstream carrying a three-dimensional (3D) mesh frame. The processing circuit determines at least two sampling rates associated with different portions of the multiple initial maps, and reconstructs from the multiple initial maps based on the at least two sampling rates associated with the different portions of the multiple initial maps to obtain multiple reconstructed maps. A first portion of the multiple initial maps is reconstructed based on a first sampling rate of the at least two sampling rates associated with the first portion, and a second portion of the multiple initial maps is reconstructed based on a second sampling rate of the at least two sampling rates associated with the second portion. The processing circuit reconstructs the 3D mesh frame based on the multiple reconstructed maps.
[0006] In some examples, the processing circuit determines a first sampling rate associated with a first initial map of a first map type and determines a second sampling rate associated with a second initial map of a second map type.
[0007] In some examples, the processing circuitry determines a first sampling rate associated with a first sub-region in the first initial map and determines a second sampling rate associated with a second sub-region in the first initial map, where the first sub-region or the second sub-region can include at least one of a plurality of pixel rows, a plurality of pixel columns, a sampled coding tree unit (CTU) line, a sampled slice, a sampled tile, a sampled tile group, and / or a sampled CTU.
[0008] In some examples, the processing circuit determines a first sampling rate associated with a first patch in the first initial map and determines a second sampling rate associated with a second patch in the first initial map.
[0009] In some examples, the processing circuit decodes from the bitstream a first flag associated with a first portion of the plurality of initial maps. The first flag indicates whether a first sampling rate associated with the first portion differs from a default sampling rate. When the first flag indicates that the first sampling rate differs from the default sampling rate, in one example, the processing circuit decodes the first sampling rate from the bitstream. In another example, the processing circuit decodes a difference between the first sampling rate and the default sampling rate from the bitstream.
[0010] In some examples, the processing circuitry decodes from the bitstream an index that indicates a selection from a set of pre-defined sampling rates as the sampling rate associated with the portions of the multiple initial maps.
[0011] In some examples, the bitstream carries a plurality of 3D mesh frames. The processing circuit restores a first map associated with the first 3D mesh frame based on a first sampling rate associated with a first patch of the first 3D mesh frame, and determines a corresponding sampling rate associated with a second patch in the second 3D mesh frame according to the first sampling rate. The second patch in the second 3D mesh frame is a patch corresponding to the first patch of the first 3D mesh frame. In one example, the first 3D mesh frame is a key frame of a group of pictures (GOP), and the corresponding sampling rate is equal to the first sampling rate.
[0012] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by a computer, cause the computer to perform any one or combination of methods for mesh encoding.
[0013] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. [Brief description of the drawings]
[0014] [Figure 1] 1 illustrates a block diagram of a communication system in accordance with some examples. [Diagram 2] 1 illustrates a block diagram of a streaming system in accordance with some examples. [Diagram 3] 1 illustrates a block diagram of an encoder for encoding a point cloud frame in some examples. [Figure 4] 1 illustrates a block diagram of a decoder for decoding a compressed bitstream corresponding to a point cloud frame in some examples. [Diagram 5] 1 illustrates a block diagram of a video decoder in some examples. [Figure 6] 1 illustrates a block diagram of a video encoder in some examples. [Figure 7] 1 illustrates a block diagram of an encoder for encoding a point cloud frame in some examples. [Figure 8] 1 illustrates a block diagram of a decoder for decoding a compressed bitstream carrying a point cloud frame in some examples. [Figure 9] 1 shows a diagram illustrating mapping of a mesh to an atlas in some examples. [Figure 10] 1 shows a diagram illustrating downsampling in some examples. [Figure 11] 1 shows a diagram of a framework for mesh compression according to some embodiments of the present disclosure. [Figure 12] 1 shows a diagram of adaptive sampling in some examples. [Figure 13] 1 shows a diagram of adaptive sampling in some examples. [Figure 14] 1 shows a flowchart outlining an example process in some examples. [Figure 15]1 shows a flowchart outlining an example process in some examples. [Figure 16] FIG. 1 is a schematic diagram of a computer system according to some examples. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0015] Aspects of the present disclosure provide techniques in the field of three-dimensional (3D) media processing.
[0016] Technological developments in 3D media processing, such as advances in three-dimensional (3D) capture, 3D modeling, and 3D rendering, are facilitating the ubiquitous presence of 3D media content across several platforms and devices. In one example, a baby's first steps may be captured on one continent, and media technology may enable grandparents to view (and possibly interact with) and enjoy an immersive experience with the baby on another continent. According to one aspect of the present disclosure, to improve the immersive experience, 3D models have become increasingly sophisticated, and the creation and consumption of 3D models occupies a significant amount of data resources, such as data storage, data transmission resources, etc.
[0017] According to some aspects of the present disclosure, point clouds and meshes can be used as 3D models to represent immersive content.
[0018] A point cloud may generally refer to a set of points in 3D space, each of which has associated attributes such as, for example, color, material properties, texture information, intensity attributes, reflectance attributes, motion-related attributes, modality attributes, and various other attributes. A point cloud may be used to reconstruct an object or scene as a configuration of such points.
[0019] An object's mesh (also called a mesh model) may include polygons that describe the object's surface. Each polygon may be defined by the polygon's vertices in 3D space and information about how the vertices are connected to the polygon. The information about how the vertices are connected is called connectivity information. In some examples, the mesh may also include attributes, such as color, normals, etc., associated with the vertices.
[0020] According to some aspects of the present disclosure, some coding tools for point cloud compression (PCC) may be used for mesh compression. For example, a mesh may be remeshed to generate a new mesh from which connectivity information of the new mesh can be inferred. The vertices of the new mesh and the attributes associated with the vertices of the new mesh may be considered as points in a point cloud and may be compressed using a PCC codec.
[0021] A point cloud can be used to reconstruct an object or scene as a configuration of points. The points can be captured using multiple cameras, depth sensors, or Lidar in a variety of settings and can consist of thousands up to billions of points to realistically represent the reconstructed scene or object. A patch may generally refer to a contiguous subset of the surface described by the point cloud. In one example, a patch includes points that have surface normal vectors that are offset from each other by less than a threshold amount.
[0022] PCC can be implemented according to various schemes, such as a geometry-based scheme called G-PCC, a video coding-based scheme called V-PCC, etc. According to some aspects of the present disclosure, G-PCC is a purely geometry-based approach that directly encodes 3D geometry and has little in common with video coding, while V-PCC is heavily based on video coding. For example, V-PCC can map points of a 3D cloud to pixels of a 2D grid (image). The V-PCC scheme can utilize a general-purpose video codec for point cloud compression. The PCC codec (encoder / decoder) in the present disclosure can be a G-PCC codec (encoder / decoder) or a V-PCC codec.
[0023] According to one aspect of the present disclosure, the V-PCC scheme can compress the geometry, occupancy, and texture of the point cloud as three separate video sequences using an existing video codec. Additional metadata required to interpret the three video sequences is compressed separately. A small portion of the overall bitstream is the metadata, which in one example can be efficiently encoded / decoded using a software implementation. The majority of the information is handled by the video codec.
[0024] FIG. 1 shows a block diagram of a communication system (100) in some examples. The communication system (100) includes a plurality of terminal devices capable of communicating with each other, for example, via a network (150). For example, the communication system (100) includes a pair of terminal devices (110) and (120) interconnected via the network (150). In the example of FIG. 1, a first pair of terminal devices (110) and (120) can perform unidirectional transmission of point cloud data. For example, the terminal device (110) can compress a point cloud (e.g., points representing structures) captured by a sensor (105) connected to the terminal device (110). The compressed point cloud can be transmitted to another terminal device (120) via the network (150), for example, in the form of a bit stream. The terminal device (120) can receive the compressed point cloud from the network (150), decompress the bit stream to reconstruct the point cloud, and appropriately display the reconstructed point cloud. One-way data transmission may be common, such as in media serving applications.
[0025] In the example of FIG. 1, terminal devices (110) and (120) may be shown as a server and a personal computer, although the principles of the present disclosure may not be so limited. It is understood that embodiments of the present disclosure find application with laptop computers, tablet computers, smartphones, gaming consoles, media players, and / or dedicated three-dimensional (3D) equipment. Network (150) represents any number of networks that transmit compressed point clouds between terminal device (110) and terminal device (120). Network (150) may include, for example, a wired communication (cable) network and / or a wireless communication network. Network (150) may exchange data over circuit-switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks, wide area networks, the Internet, and the like.
[0026] 2 illustrates a block diagram of a streaming system 200 in some examples. The streaming system 200 is a point cloud application. The disclosed subject matter may be equally applicable to other point cloud-enabled applications, such as 3D telepresence applications, virtual reality applications, and the like.
[0027] The streaming system (200) can include a capture subsystem (213). The capture subsystem (213) can include a point cloud source (201), such as a light detection and ranging (LIDAR) system, a 3D camera, a 3D scanner, a graphics generation component that generates an uncompressed point cloud, such as software that generates an uncompressed point cloud (202). In one example, the point cloud (202) includes points captured by a 3D camera. The point cloud (202) is illustrated as a bold line to emphasize the high amount of data compared to a compressed point cloud (204) (compressed point cloud bitstream). The compressed point cloud (204) can be generated by an electronic device (220) that includes an encoder (203) connected to the point cloud source (201). The encoder (203) can include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The compressed point cloud (204) (or a bitstream of the compressed point cloud (204)), shown as a thin line to emphasize the lower amount of data compared to the stream of point cloud (202), may be stored on a streaming server (205) for future use. One or more streaming client subsystems, such as the client subsystems (206) and (208) of FIG. 2, may access the streaming server (205) to retrieve copies (207) and (209) of the compressed point cloud (204). The client subsystem (206) may include a decoder (210), for example in the electronic device (230). The decoder (210) decodes the incoming copy of the compressed point cloud (207) and generates an outgoing stream of a reconstructed point cloud (211) that can be rendered on a rendering device (212).
[0028] It should be noted that the electronic devices (220) and (230) may include other components (not shown). For example, the electronic device (220) may include a decoder (not shown), and the electronic device (230) may also include an encoder (not shown).
[0029] In some streaming systems, the compressed point clouds (204), (207), and (209) (e.g., compressed point cloud bitstreams) may be compressed according to a particular standard. In some examples, a video coding standard is used to compress the point clouds. Such standards include, for example, High Efficiency Video Coding (HEVC) (HEVC) and Versatile Video Coding (VVC) (VVC).
[0030] 3 shows a block diagram of a V-PCC encoder (300) for encoding a point cloud frame according to some embodiments. In some embodiments, the V-PCC encoder (300) may be used in the communication system (100) and the streaming system (200). For example, the encoder (203) may be configured and operate similarly to the V-PCC encoder (300).
[0031] The V-PCC encoder (300) receives a point cloud frame as an uncompressed input and generates a bitstream corresponding to the compressed point cloud frame. In some embodiments, the V-PCC encoder (300) can receive the point cloud frame from a point cloud source, such as the point cloud source (201).
[0032] In the example of FIG. 3, the V-PCC encoder (300) includes a patch generation module (306), a patch packing module (308), a geometry image generation module (310), a texture image generation module (312), a patch information module (304), an occupancy map module (314), a smoothing module (336), image padding modules (316) and (318), a group expansion module (320), video compression modules (322), (323), and (332), an auxiliary patch information compression module (338), an entropy compression module (334), and a multiplexer (324).
[0033] According to one aspect of the disclosure, the V-PCC encoder (300) converts the 3D point cloud frame into an image-based representation along with some metadata (e.g., occupancy map and patch information) used to convert the compressed point cloud into an uncompressed point cloud. In some examples, the V-PCC encoder (300) converts the 3D point cloud frame into a geometry image, a texture image, and an occupancy map, and then can encode the geometry image, texture image, and occupancy map into a bitstream using video coding techniques. In general, a geometry image is a 2D image with pixels filled with geometry values associated with points projected onto the pixels, and the pixels filled with geometry values can be referred to as geometry samples. A texture image is a 2D image with pixels filled with texture values associated with points projected onto the pixels, and the pixels filled with texture values can be referred to as texture samples. An occupancy map is a 2D image with pixels filled with values that indicate whether they are occupied or unoccupied by a patch.
[0034] The patch generation module (306) segments the point cloud into a set of patches (e.g., a patch is defined as a contiguous subset of the surface described by the point cloud), which may or may not overlap, such that each patch can be described by a depth field for a plane in 2D space. In some embodiments, the patch generation module (306) aims to decompose the point cloud into a minimum number of patches with smooth boundaries while minimizing the reconstruction error.
[0035] In some examples, the patch information module (304) may collect patch information indicating the size and shape of the patch, and in some examples, the patch information may be packed into an image frame and then encoded by the auxiliary patch information compression module (338) to generate compressed auxiliary patch information.
[0036] In some examples, the patch packing module (308) is configured to map the extracted patches onto a two-dimensional (2D) grid while minimizing unused space and ensuring that every M×M (e.g., 16×16) block of the grid is associated with a unique patch. Efficient patch packing can directly impact compression efficiency by either minimizing unused space or ensuring temporal consistency.
[0037] The geometry image generation module (310) can generate a 2D geometry image associated with the geometry of the point cloud at a given patch location. The texture image generation module (312) can generate a 2D texture image associated with the texture of the point cloud at a given patch location. The geometry image generation module (310) and the texture image generation module (312) use the 3D to 2D mapping calculated during the packing process to store the geometry and texture of the point cloud as images. To better handle the case where multiple points are projected onto the same sample, each patch is projected onto two images, called layers. In one example, the geometry image is represented by a WxH monochromatic frame in YUV420-8 bit format. To generate the texture image, the texture generation procedure uses the reconstructed / smoothed geometry to calculate the color associated with the resampled points.
[0038] The occupancy map module (314) can generate an occupancy map that describes the padding information for each unit. For example, the occupancy image includes a binary map that indicates for each cell of the grid whether the cell belongs to free space or to the point cloud. In one example, the occupancy map uses binary information that describes, for each pixel, whether the pixel is padded or not. In another example, the occupancy map uses binary information that describes, for each block of pixels, whether the block of pixels is padded or not.
[0039] The occupancy map generated by the occupancy map module (314) can be compressed using lossless or lossy encoding. When lossless encoding is used, the entropy compression module (334) is used to compress the occupancy map. When lossy encoding is used, the video compression module (332) is used to compress the occupancy map.
[0040] It should be noted that the patch packing module (308) may leave some empty space between the 2D patches packed within the image frame. The image padding modules (316) and (318) may fill the empty space (called padding) to generate an image frame that may be suitable for 2D video and image codecs. Image padding is also called background filling, which may fill unused space with redundant information. In some instances, good background filling increases the bit rate minimally but does not introduce significant coding distortion around the patch boundaries.
[0041] The video compression modules (322), (323), and (332) can encode 2D images, such as padded geometry images, padded texture images, and occupancy maps, based on an appropriate video coding standard, such as HEVC, VVC, etc. In one example, the video compression modules (322), (323), and (332) are individual components that operate separately. It should be noted that in another example, the video compression modules (322), (323), and (332) can be implemented as a single component.
[0042] In some examples, the smoothing module (336) is configured to generate a smoothed image of the reconstructed geometry image. The smoothed image may be provided to the texture image generation (312). The texture image generation (312) may then adjust the generation of the texture image based on the reconstructed geometry image. For example, if there is some distortion in the patch shape (e.g., geometry) during encoding and decoding, the distortion may be taken into account when generating the texture image to correct the distortion of the patch shape.
[0043] In some embodiments, the group expansion (320) is configured to pad pixels around object boundaries with redundant low-frequency content to improve the coding gain as well as the visual quality of the reconstructed point cloud.
[0044] A multiplexer (324) may multiplex the compressed geometry images, the compressed texture images, the compressed occupancy maps, and the compressed auxiliary patch information into a compressed bitstream.
[0045] 4 shows a block diagram of a V-PCC decoder (400) for decoding a compressed bitstream corresponding to a point cloud frame in some examples. In some examples, the V-PCC decoder (400) can be used in the communication system (100) and the streaming system (200). For example, the decoder (210) can be configured to operate similarly to the V-PCC decoder (400). The V-PCC decoder (400) receives a compressed bitstream and generates a reconstructed point cloud based on the compressed bitstream.
[0046] In the example of FIG. 4, the V-PCC decoder (400) includes a demultiplexer (432), video unfolding modules (434) and (436), an occupancy map unfolding module (438), an auxiliary patch information unfolding module (442), a geometry reconstruction module (444), a smoothing module (446), a texture reconstruction module (448), and a color smoothing module (452).
[0047] A demultiplexer (432) can receive the compressed bitstream and separate it into compressed texture images, compressed geometry images, compressed occupancy maps, and compressed auxiliary patch information.
[0048] The video decompression modules (434) and (436) can decode compressed images according to an appropriate standard (e.g., HEVC, VVC, etc.) and output decompressed images. For example, the video decompression module (434) decodes compressed texture images and outputs decompressed texture images. The video decompression module (436) decodes compressed geometry images and outputs decompressed geometry images.
[0049] An occupancy map decompression module (438) can decode the compressed occupancy map according to an appropriate standard (e.g., HEVC, VVC, etc.) and output a decompressed occupancy map.
[0050] The auxiliary patch information extraction module (442) can decode the auxiliary patch information compressed according to an appropriate standard (e.g., HEVC, VVC, etc.) and output the extracted auxiliary patch information.
[0051] A geometry reconstruction module (444) can receive the unfolded geometry image and generate a reconstructed point cloud geometry based on the unfolded occupancy map and the unfolded auxiliary patch information.
[0052] A smoothing module (446) can smooth discrepancies at the edges of the patches. The smoothing procedure aims to mitigate potential discontinuities that may occur at patch boundaries due to compression artifacts. In some embodiments, a smoothing filter can be applied to pixels located on the patch boundaries to mitigate distortions that may occur due to compression / expansion.
[0053] A texture reconstruction module (448) can determine texture information for points in the point cloud based on the unfolded texture image and the smoothed geometry.
[0054] The color smoothing module (452) can smooth color inconsistencies. Non-adjacent patches in 3D space are often packed adjacent to each other in 2D video. In some instances, pixel values from non-adjacent patches may be mixed by block-based video codecs. The purpose of color smoothing is to reduce visible artifacts that appear at patch boundaries.
[0055] 5 shows a block diagram of a video decoder (510) in some examples. The video decoder (510) may be used in the V-PCC decoder (400). For example, the video unfolding modules (434) and (436) and the occupancy map unfolding module (438) may be configured similarly to the video decoder (510).
[0056] The video decoder (510) may include a parser (520) for reconstructing symbols (521) from a compressed image, such as a coded video sequence. These symbol categories include information used to manage the operation of the video decoder (510). The parser (520) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may follow a video coding technique or standard and may follow a variety of principles including variable length coding, Huffman coding, arithmetic coding, with or without context sensitivity, etc. The parser (520) may extract a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to the group. The subgroups may include a group of pictures (GOP), a picture, a tile, a slice, a macroblock, a coding unit (CU), a block, a transform unit (TU), a prediction unit (PU), etc. The parser (520) may also extract from the coded video sequence information such as transform coefficients, quantization parameter values, motion vectors, etc.
[0057] The parser (520) can perform entropy decoding / parsing operations on the video sequence received from the buffer memory to produce symbols (521).
[0058] The reconstruction of the symbols (521) may involve a number of different units, depending on the type of coded video picture or part thereof (inter and intra pictures, inter and intra blocks, etc.), and other factors. Which units are involved and how can be controlled by subgroup control information parsed from the coded video sequence by the parser (520). The flow of such subgroup control information between the parser (520) and the following units is not shown for clarity.
[0059] In addition to the functional blocks already mentioned, the video decoder (510) may be conceptually subdivided into several functional units, as described below. In an actual implementation operating under commercial constraints, many of these units may interact closely with each other and may be at least partially integrated with each other. However, for purposes of describing the disclosed subject matter, the following conceptual subdivision into functional units is appropriate:
[0060] The first unit is a scalar / inverse transform unit (551). The scalar / inverse transform unit (551) receives quantized transform coefficients as well as control information including which transform to use, block size, quantization coefficients, quantization scaling matrices, etc. as symbol(s) (521) from the parser (520). The scalar / inverse transform unit (551) may output a block comprising sample values that may be input to an aggregator (555).
[0061] In some cases, the output samples of the scaler / inverse transform (551) may relate to intra-coded blocks, i.e., blocks that do not use prediction information from a previously reconstructed image, but can use prediction information from a previously reconstructed portion of the current image. Such prediction information may be provided by an intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) generates a block of the same size and shape as the block being reconstructed using surrounding already reconstructed information fetched from a current picture buffer (558). The current picture buffer (558) buffers, for example, a partially reconstructed current picture and / or a fully reconstructed current picture. The aggregator (555) adds, possibly on a sample-by-sample basis, the prediction information generated by the intra-prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551).
[0062] In other cases, the output samples of the scalar / inverse transform unit (551) may relate to an inter-coded, potentially motion-compensated block. In such cases, the motion compensated prediction unit (553) may access the reference picture memory (557) to fetch samples used for prediction. After motion compensating the fetched samples according to the symbols (521) associated with the block, these samples may be added to the output of the scalar / inverse transform unit (551) by the aggregator (555) to generate output sample information (in this case referred to as residual samples or residual signals). The addresses in the reference picture memory (557) from which the motion compensated prediction unit (553) fetches the prediction samples may be controlled by motion vectors available to the motion compensated prediction unit (553) in the form of symbols (521), which may have, for example, X, Y, and reference picture components. Motion compensation may also include interpolation of sample values fetched from the reference picture memory (557) when sub-sample accurate motion vectors are used, motion vector prediction mechanisms, and the like.
[0063] The output samples of the aggregator (555) may be subjected to various loop filtering techniques in a loop filter unit (556). Video compression techniques may include in-loop filter techniques that are controlled by parameters contained in the coded video sequence (also called the coded video bitstream) and made available to the loop filter unit (556) as symbols (521) from the parser (520), but may also be responsive to previously reconstructed and loop filtered sample values as well as to meta-information obtained during decoding of a coded picture or previous portion (in decoding order) of the coded video sequence.
[0064] The output of the loop filter unit (556) may be a sample stream that can be output to a rendering device and also stored in a reference picture memory (557) for use in future inter-picture prediction.
[0065] Once a particular coded picture has been fully reconstructed, it may be used as a reference picture for future predictions. For example, once a coded picture corresponding to a current picture has been fully reconstructed and the coded picture has been identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) may become part of the reference picture memory (557), and a fresh current picture buffer may be reallocated before beginning reconstruction of the next coded picture.
[0066] The video decoder (510) may perform decoding operations according to a given video compression technique in a standard such as ITU-T Rec. H.265. The coded video sequence may conform to the syntax specified by the video compression technique or standard being used, in the sense that the coded video sequence adheres to both the syntax of the video compression technique or standard and the profile documented in the video compression technique. Specifically, the profile may select some tools from all tools available in the video compression technique or standard as the only tools available for use under that profile. Also, what is required for compliance may be that the complexity of the coded video sequence is within a range defined by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstructed sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the level may be further limited in some cases by the specification of a Hypothetical Reference Decoder (HRD) and metadata of HRD buffer management signaled in the coded video sequence.
[0067] 6 shows a block diagram of a video encoder (603) according to one embodiment of the present disclosure. The video encoder (603) may be used in the V-PCC encoder (300) to compress the point cloud. In one example, the video compression modules (322) and (323) and the video compression module (332) are configured similarly to the encoder (603).
[0068] The video encoder (603) can receive images, such as padded geometry images, padded texture images, etc., and generate compressed images.
[0069] According to one embodiment, the video encoder (603) can encode and compress images of a source video sequence into encoded video sequences (compressed images) in real-time or under any other time constraint required by the application. Enforcing an appropriate encoding rate is one function of the controller (650). In some embodiments, the controller (650) controls and is functionally coupled to other functional units described below. For clarity, coupling is not depicted. Parameters set by the controller (650) can include rate control related parameters (picture skip, quantizer, lambda value for rate distortion optimization techniques, ...), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) can be configured to have other appropriate functions for the video encoder (603) optimized for a particular system design.
[0070] In some embodiments, the video encoder (603) is configured to operate in a coding loop. As an oversimplified explanation, in one example, the coding loop can include a source coder (630) (e.g., responsible for creating symbols, such as a symbol stream, based on an input picture to be coded and a reference picture), as well as a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to create sample data in a manner similar to that which the (remote) decoder also creates (since any compression between the symbols and the coded video bitstream is lossless in the video compression techniques contemplated in the disclosed subject matter). The reconstructed sample stream (sample data) is input to a reference picture memory (634). Since the decoding of the symbol stream leads to a bit-exact result regardless of the location of the decoder (local or remote), the content in the reference picture memory (634) is also bit-exact between the local and remote encoders. In other words, the prediction part of the encoder "sees" exactly the same sample values as the decoder "sees" when using the prediction during decoding. This basic principle of reference picture synchrony (if synchrony cannot be maintained due to, for example, channel error, the resulting drift) is also used in some related technologies.
[0071] The operation of the "local" decoder (633) may be the same as the operation of a "remote" decoder, such as the video decoder (510), already described in detail above in conjunction with Figure 5. However, with brief reference also to Figure 5, because symbols are available and the encoding / decoding of symbols into an encoded video sequence by the entropy coder (645) and parser (520) may be lossless, the entropy decoding portion of the video decoder (510), including the parser (520), may not be fully implemented in the local decoder (633).
[0072] In operation, in some examples, the source coder (630) may perform motion-compensated predictive coding, which predictively codes an input picture with reference to one or more previously coded pictures from a video sequence designated as “reference pictures.” In this manner, the coding engine (632) codes differences between pixel blocks of the input picture and pixel blocks of reference pictures that may be selected as predictive references for the input picture.
[0073] The local video decoder (633) can decode coded video data of pictures that may be designated as reference pictures based on symbols created by the source coder (630). The operation of the coding engine (632) may advantageously be a lossy process. When the coded video data may be decoded in a video decoder (not shown in FIG. 6), the reconstructed video sequence may be a replica of the source video sequence, usually with some errors. The local video decoder (633) can replicate the decoding process that may be performed by the video decoder on the reference pictures, such that the reconstructed reference pictures are stored in a reference picture cache (634). In this way, the video encoder (603) can locally store copies of reconstructed reference pictures that have common content as reconstructed reference pictures obtained by a far-end video decoder (without transmission errors).
[0074] The predictor (635) may perform a prediction search for the coding engine (632). That is, for a new picture to be coded, the predictor (635) may search the reference picture memory (634) for sample data (as candidate reference pixel blocks) or specific metadata such as reference picture motion vectors, block shapes, etc., that may serve as suitable prediction references for the new picture. The predictor (635) may operate on sample blocks on a pixel block by pixel block basis to find suitable prediction references. In some cases, as determined by the search results obtained by the predictor (635), the input picture may have prediction references drawn from multiple reference pictures stored in the reference picture memory (634).
[0075] The controller (650) can manage the encoding operations of the source coder (630), including, for example, setting the parameters and subgroup parameters used to encode the video data.
[0076] The output of all the aforementioned functional units may undergo entropy coding in an entropy coder (645), which converts the symbols produced by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc.
[0077] The controller (650) may manage the operation of the video encoder (603). During coding, the controller (650) may assign a particular coded picture type to each coded picture, which may affect the coding techniques that may be applied to the respective picture. For example, pictures may often be assigned as one of the following picture types:
[0078] An intra picture (I-picture) may be a picture that can be coded and decoded without using any other picture in a sequence as a prediction source. Some video codecs allow various types of intra pictures, including, for example, independent decoder refresh ("IDR") pictures. Those skilled in the art are aware of these variations of I-pictures and their respective uses and characteristics.
[0079] A predictive picture (P picture) may be a picture that can be coded and decoded using intra- or inter-prediction, which predicts the sample values of each block using at most one motion vector and reference index.
[0080] A bidirectionally predicted picture (B-picture) may be a picture that can be coded and decoded using intra- or inter-prediction, which uses up to two motion vectors and reference indexes to predict the sample values of each block. Similarly, a multi-predicted picture may use more than two reference pictures and associated metadata for the reconstruction of a single block.
[0081] A source picture may generally be spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and coded block by block. Blocks may be predictively coded with reference to other (already coded) blocks as determined by a coding assignment applied to the respective picture of the block. For example, blocks of an I picture may be non-predictively coded, or they may be predictively coded with reference to already coded blocks of the same picture (spatial or intra prediction). Pixel blocks of a P picture may be predictively coded via spatial prediction or via temporal prediction with reference to one previously coded reference picture. Blocks of a B picture may be predictively coded by spatial prediction or via temporal prediction with reference to one or two previously coded reference pictures.
[0082] The video encoder (603) may perform coding operations in accordance with a given video coding technique or standard, such as ITU-T Rec. H.265. In its operations, the video encoder (603) may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the coded video data may conform to a syntax specified by the video coding technique or standard being used.
[0083] Video may be in the form of multiple source pictures (images) in a time sequence. Intra-picture prediction (often abbreviated as intra-prediction) exploits spatial correlation in a given picture, while inter-picture prediction exploits correlation (temporal or other) between pictures. In one example, a particular picture being coded / decoded, called the current picture, is divided into blocks. When a block in the current picture is similar to a reference block in a reference picture that was previously coded and is still buffered in the video, the block in the current picture can be coded by a vector called a motion vector. The motion vector points to a reference block in a reference picture, and may have a third dimension that identifies the reference picture if multiple reference pictures are used.
[0084] In some embodiments, bi-prediction techniques can be used for inter-picture prediction. According to bi-prediction techniques, two reference pictures, such as a first reference picture and a second reference picture, both of which are earlier in decoding order than the current picture in the video (but may be past and future in display order, respectively), are used. A block in the current picture can be coded by a first motion vector that points to a first reference block in the first reference picture and a second motion vector that points to a second reference block in the second reference picture. A block can be predicted by a combination of the first reference block and the second reference block.
[0085] Furthermore, to improve coding efficiency, merge mode techniques can be used for inter-picture prediction.
[0086] According to some embodiments of the present disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed on a block-by-block basis. For example, according to the HEVC standard, pictures in a sequence of video pictures are divided into coding tree units (CTUs) for compression, and the CTUs in a picture have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. Generally, a CTU includes three coding tree blocks (CTBs), one luma CTB and two chroma CTBs. Each CTU can be recursively quadtree partitioned into one or more coding units (CUs). For example, a 64×64 pixel CTU can be partitioned into one 64×64 pixel CU, or four 32×32 pixel CUs, or sixteen 16×16 pixel CUs. In one example, each CU is analyzed to determine the prediction type of the CU, such as an inter prediction type or an intra prediction type. The CU is partitioned into one or more prediction units (PUs) depending on the temporal and / or spatial predictability. Generally, each PU includes one luma prediction block (PB) and two chroma PBs. In one embodiment, prediction operations in coding (encoding / decoding) are performed in units of prediction blocks. Taking luma prediction blocks as an example of prediction blocks, the prediction blocks include a matrix of pixel values (e.g., luma values) of 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.
[0087] 7 shows a block diagram of a G-PCC encoder (700) in some examples. The G-PCC encoder (700) can be configured to receive point cloud data and compress the point cloud data to generate a bitstream carrying the compressed point cloud data. In one embodiment, the G-PCC encoder (700) can include a position quantization module (710), a duplicate point removal module (712), an octree encoding module (730), an attribute transfer module (720), a level of detail (LOD) generation module (740), an attribute prediction module (750), a residual quantization module (760), an arithmetic coding module (770), an inverse residual quantization module (780), an addition module (781), and a memory (790) for storing reconstructed attribute values.
[0088] As shown, an input point cloud (701) may be received at a G-PCC encoder (700). Locations (e.g., 3D coordinates) of the point cloud (701) are provided to a quantization module (710). The quantization module (710) is configured to quantize the coordinates to generate quantized locations. A duplicate point removal module (712) is configured to receive the quantized locations and perform a filtering process to identify and remove duplicate points. An octree encoding module (730) is configured to receive the filtered locations from the duplicate point removal module (712) and perform an octree-based encoding process to generate a sequence of occupancy codes that describe a 3D grid of voxels. The occupancy codes are provided to an arithmetic coding module (770).
[0089] The attribute transfer module (720) is configured to receive attributes of the input point cloud and perform an attribute transfer process to determine an attribute value for each voxel when multiple attribute values are associated with each voxel. The attribute transfer process can be performed on the reordered points output from the octree encoding module (730). The attributes after the transfer operation are provided to the attribute prediction module (750). The LOD generation module (740) is configured to operate on the reordered points output from the octree encoding module (730) and reorganize the points into different LODs. The LOD information is provided to the attribute prediction module (750).
[0090] The attribute prediction module (750) processes the points in a LOD-based order indicated by the LOD information from the LOD generation module (740). The attribute prediction module (750) generates an attribute prediction for the current point based on the reconstructed attributes of a set of neighboring points of the current point stored in the memory (790). A prediction residual may then be obtained based on the original attribute values received from the attribute transfer module (720) and the locally generated attribute predictions. When a candidate index is used in each attribute prediction process, an index corresponding to the selected prediction candidate may be provided to the arithmetic coding module (770).
[0091] The residual quantization module (760) is configured to receive the prediction residual from the attribute prediction module (750) and perform quantization to generate a quantized residual. The quantized residual is provided to the arithmetic coding module (770).
[0092] The inverse residual quantization module (780) is configured to receive the quantized residual from the residual quantization module (760) and generate a reconstructed prediction residual by performing the inverse of the quantization operation performed in the residual quantization module (760). The summation module (781) is configured to receive the reconstructed prediction residual from the inverse residual quantization module (780) and the respective attribute predictions from the attribute prediction module (750). By combining the reconstructed prediction residual and the attribute predictions, a reconstructed attribute value is generated and stored in the memory (790).
[0093] The arithmetic coding module (770) is configured to receive the occupation codes, the candidate indexes (if used), the quantized residuals (if generated), and other information, and perform entropy coding to further compress the received values or information. This may generate a compressed bitstream (702) that conveys the compressed information. The bitstream (702) may be transmitted or provided to a decoder that decodes the compressed bitstream, or may be stored in a storage device.
[0094] 8 shows a block diagram of a G-PCC decoder (800) according to one embodiment. The G-PCC decoder (800) may be configured to receive a compressed bitstream and perform point cloud data decompression to decompress the bitstream and generate decoded point cloud data. In one embodiment, the G-PCC decoder (800) may include an arithmetic decoding module (810), an inverse residual quantization module (820), an octree decoding module (830), an LOD generation module (840), an attribute prediction module (850), and a memory (860) for storing reconstructed attribute values.
[0095] As shown, a compressed bitstream (801) may be received at an arithmetic decoding module (810). The arithmetic decoding module (810) is configured to decode the compressed bitstream (801) to obtain quantized residuals (if generated) and occupancy codes of the point cloud. The octree decoding module (830) is configured to determine reconstructed positions of the points of the point cloud according to the occupancy codes. The LOD generation module (840) is configured to reorganize the points into different LODs based on the reconstructed positions and determine an LOD-based order. The inverse residual quantization module (820) is configured to generate reconstructed residuals based on the quantized residuals received from the arithmetic decoding module (810).
[0096] The attribute prediction module (850) is configured to perform an attribute prediction process to determine attribute predictions for the points in a LOD-based order. For example, the attribute predictions for the current point may be determined based on reconstructed attribute values of neighboring points of the current point stored in the memory (860). In some examples, the attribute predictions may be combined with respective reconstructed residuals to generate reconstructed attributes for the current point.
[0097] The sequence of reconstructed attributes generated from the attribute prediction module (850), along with the reconstructed positions generated from the octree decoding module (830), in one example correspond to the decoded point cloud (802) output from the G-PCC decoder (800). In addition, the reconstructed attributes are also stored in memory (860) and may subsequently be used to derive attribute predictions for subsequent points.
[0098] In various embodiments, the encoder (300), the decoder (400), the encoder (700), and / or the decoder (800) may be implemented in hardware, software, or a combination thereof. For example, the encoder (300), the decoder (400), the encoder (700), and / or the decoder (800) may be implemented using a processing circuit, such as one or more integrated circuits (ICs) that operate with or without software, such as application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), and the like. In another example, the encoder (300), the decoder (400), the encoder (700), and / or the decoder (800) may be implemented as software or firmware that includes instructions stored in a non-volatile (or non-transitory) computer-readable storage medium. The instructions, when executed by a processing circuit, such as one or more processors, cause the processing circuit to perform the functions of the encoder (300), the decoder (400), the encoder (700), and / or the decoder (800).
[0099] It should be noted that the attribute prediction modules (750) and (850) configured to implement the attribute prediction techniques disclosed herein may be included in other decoders or encoders that may have similar or different structures to those shown in Figures 7 and 8. In addition, the encoder (700) and decoder (800) may be included in the same device, or in various examples, in separate devices.
[0100] According to some aspects of the present disclosure, mesh compression may use a coding tool different from a PCC coding tool, or may use a PCC coding tool such as a PCC (e.g., G-PCC, V-PCC) encoder, a PCC (e.g., G-PCC, V-PCC) decoder, or the like.
[0101] An object's mesh (also called a mesh model or mesh frame) may contain polygons that describe the surface of the object. Each polygon may be defined by the polygon's vertices in 3D space and information about how the vertices are connected to the polygon. The information about how the vertices are connected is called connectivity information. In some examples, a mesh may also contain attributes, such as color, normals, etc., associated with the vertices. Attributes may be associated with the mesh's surface by utilizing mapping information that parameterizes the mesh with a 2D attribute map. The mapping information is typically described by a set of parametric coordinates, called UV coordinates or texture coordinates, associated with the mesh vertices. The 2D attribute map (called a texture map in some examples) is used to store high-resolution attribute information, such as texture, normals, and displacements. Such information may be used for various purposes, such as texture mapping and shading.
[0102] In some embodiments, a mesh can include components called geometry information, connectivity information, mapping information, vertex attributes, and attribute maps. In some examples, the geometry information is described by a set of 3D positions associated with the vertices of the mesh. In one example, (x,y,z) coordinates can be used to describe the 3D positions of the vertices. In some examples, the connectivity information includes a set of vertex indices that describe how to connect the vertices to create a 3D surface. In some examples, the mapping information describes how to map the mesh surface to a planar 2D region. In one example, the mapping information is described by a set of UV parametric / texture coordinates (u,v) associated with the mesh vertices along with the connectivity information. In some examples, the vertex attributes include scalar or vector attribute values associated with the mesh vertices. In some examples, the attribute maps include attributes associated with the mesh surface and stored as a 2D image / video. In one example, the mapping between a visual (e.g., a 2D image / video) and the mesh surface is defined by the mapping information.
[0103] According to one aspect of the disclosure, a number of techniques, called UV mapping or mesh parameterization, are used to map the surface of a mesh in a 3D domain to a 2D domain. In some examples, the mesh is divided into patches in the 3D domain. A patch is a contiguous subset of the mesh. Then, in some examples, the patches are each parameterized into a 2D shape (also called a UV patch). The 2D shapes can be packed (e.g., oriented and positioned) into a map, also called an atlas in some examples. In some examples, the map can be further processed using 2D image or video processing techniques.
[0104] In one example, the UV mapping technique generates a 2D UV atlas (also called a UV map) and one or more texture atlases (also called texture maps) that correspond to patches of a 3D mesh. The UV atlas includes an assignment of 3D vertices of the 3D mesh to 2D points in a 2D domain (e.g., a rectangle). The UV atlas is a mapping between the coordinates of the 3D surface and the coordinates of the 2D domain. In one example, a point in the UV atlas at 2D coordinates (u,v) has a value formed by the coordinates (x,y,z) of the vertex in the 3D domain. In one example, the texture atlas includes color information for the 3D mesh. For example, a point in the texture atlas at 2D coordinates (u,v) (which has a 3D value of (x,y,z) in the UV atlas) has a color that specifies the color attributes of the point at (x,y,z) in the 3D domain.
[0105] According to one aspect of the present disclosure, a dynamic mesh is a mesh in which at least one of its components (geometry information, connectivity information, mapping information, vertex attributes, attribute maps) changes over time. A dynamic mesh can be described by a sequence of meshes (also called mesh frames). Because dynamic meshes can contain a significant amount of information that changes over time, dynamic meshes may require large amounts of data. Mesh compression techniques can enable efficient storage and transmission of media content in mesh representations.
[0106] In some examples, the dynamic mesh can have constant connectivity information, time-varying geometry, and time-varying vertex attributes. In some examples, the dynamic mesh can have time-varying connectivity information. In one example, digital content creation tools typically generate dynamic meshes with time-varying attribute maps and time-varying connectivity information. In some examples, volumetric acquisition techniques are used to generate the dynamic mesh. Volumetric acquisition techniques can generate dynamic meshes with time-varying connectivity information, especially under real-time constraints.
[0107] According to some aspects of the present disclosure, mesh compression can be performed by representing the mesh using one or more 2D maps (also referred to as 2D atlases in some examples) and then encoding the 2D maps using an image or video codec.
[0108] FIG. 9 shows a diagram illustrating the mapping of a 3D mesh (910) to a 2D atlas (920) in some examples. In the example of FIG. 9, the 3D mesh (910) includes four vertices 1-4 that form four patches A-D. Each patch has a set of vertices and associated attribute information. For example, patch A is formed by vertices 1, 2, and 3 connected to a triangle. Patch B is formed by vertices 1, 3, and 4 connected to a triangle. Patch C is formed by vertices 1, 2, and 4 connected to a triangle. Patch D is formed by vertices 2, 3, and 4 connected to a triangle. In some examples, vertices 1, 2, 3, and 4 can have respective attributes, and the triangle formed by vertices 1, 2, 3, and 4 can have respective attributes.
[0109] In one example, 3D patches A, B, C, and D are mapped to a 2D domain, such as a 2D atlas (920), also referred to as a map (920). For example, patch A is mapped to a 2D shape (also referred to as a UV patch) A' in the map (920), patch B is mapped to a 2D shape (also referred to as a UV patch) B' in the map (920), patch C is mapped to a 2D shape (also referred to as a UV patch) C' in the map (920), and patch D is mapped to a 2D shape (also referred to as a UV patch) D' in the map (920). In some examples, coordinates in the 3D domain are referred to as (x, y, z) coordinates, and coordinates in a 2D domain, such as the map (920), are referred to as UV coordinates. Vertices in a 3D mesh can have corresponding UV coordinates in the map (920).
[0110] The map (920) may be a geometry map having geometry information, a texture map having color, textile, or other attribute information, or an occupancy map having occupancy information.
[0111] In one example, the geometry information of the vertices can be stored in a 2D geometry map. For example, the geometry map stores the (x, y, z) coordinates of the sampling points at the corresponding pixels. A pixel in the geometry map at a (u, v) location has a three-component vector value that corresponds, respectively, to the x, y, and z values of the corresponding sampling point in the 3D mesh.
[0112] According to one aspect of the present disclosure, the regions in the map may not be fully occupied. For example, in FIG. 9, the regions outside the 2D shapes A', B', C', and D' are undefined. Sample values of the regions outside the 2D shapes A', B', C', and D' after decoding can be discarded. In some cases, the occupancy map is used to store some additional information for each pixel, such as storing a binary value to identify whether the pixel belongs to a patch or is undefined.
[0113] According to some aspects of the present disclosure, in order to efficiently compress 3D mesh information, 2D maps, such as geometry maps, texture maps, occupancy maps, etc., may be downsampled before being encoded.
[0114] Figure 10 illustrates downsampling in some examples. In Figure 10, a map (1020) is downsampled by a factor of 2 in both the horizontal and vertical directions to generate a downsampled map (1030). The width (e.g., number of horizontal pixels) of the downsampled map (1030) is 1 / 2 the width (e.g., number of horizontal pixels) of the map (1020), and the height (e.g., number of vertical pixels) of the downsampled map (1030) is 1 / 2 the height (e.g., number of vertical pixels) of the map (1020).
[0115] In FIG. 10, the map (1020) includes 2D shapes (also called UV patches) A', B', C', and D', and the downsampled map (1030) includes sampled 2D shapes A'', B'', C'', and D'', which correspond to the 2D shapes A', B', C', and D', respectively. The downsampled map (1030) is, in some examples, encoded by an image or video encoder on the mesh encoder side.
[0116] In some examples, at the mesh decoder side, the downsampled map is decoded. After decoding the downsampled map, the downsampled map is restored to the original resolution (e.g., the original number of pixels in the vertical direction and the original number of pixels in the horizontal direction) to reconstruct the 3D mesh.
[0117] In general, dynamic mesh sequences require a large amount of data because they may consist of a significant amount of information that changes over time. Sampling steps applied to 2D maps (e.g., geometry maps, attribute maps) can help reduce the bandwidth required to represent the mesh information. However, the sampling steps may also remove important information, such as some important geometric shapes of the 3D mesh during downsampling.
[0118] Aspects of the present disclosure provide adaptive sampling techniques for processing 2D atlases (also called 2D maps) without losing significant information. Adaptive sampling techniques can be used for static mesh (one mesh frame or mesh content does not change over time) compression, and dynamic mesh compression. Various adaptive sampling techniques can be applied individually or in any form of combination. In the following description, the adaptive sampling method is applied to a 2D atlas (e.g., a 2D map), which can be a geometry map or an attribute (texture) map, or both.
[0119] 11 shows a diagram of a framework (1100) for mesh compression according to some embodiments of the present disclosure. The framework (1100) includes a mesh encoder (1110) and a mesh decoder (1150). The mesh encoder (1110) encodes an input mesh (1105) (or a mesh frame in the case of dynamic mesh processing) into a bitstream (1145), and the mesh decoder (1150) decodes the bitstream (1145) to generate a reconstructed mesh (1195) (or a reconstructed mesh frame in the case of dynamic mesh processing).
[0120] The mesh encoder (1110) can be any suitable device, such as a computer, a server computer, a desktop computer, a laptop computer, a tablet computer, a smartphone, a gaming device, an AR device, a VR device, etc. The mesh decoder (1150) can be any suitable device, such as a computer, a client computer, a desktop computer, a laptop computer, a tablet computer, a smartphone, a gaming device, an AR device, a VR device, etc. The bitstream (1145) can be transmitted from the mesh encoder (1110) to the mesh decoder (1150) over any suitable communications network (not shown).
[0121] In the example of FIG. 11, the mesh encoder (1110) includes an adaptive sampling module (1120), a video encoder (1130), and an auxiliary data encoder (1140) coupled together. The video encoder (1130) is configured to encode image or video data, such as a 2D map within a representation of a 3D mesh. The adaptive sampling module (1120) receives an input mesh (1105) and performs adaptive sampling to generate an adaptively sampled map (1125). In some examples, the input mesh (1105) includes a 2D map to represent the 3D mesh. The adaptive sampling module (1120) can detect information density within the map and determine different sampling rates for sampling the map based on the information density. The 2D map can then be sampled according to the different sampling rates to generate an adaptively sampled map (1125).
[0122] The video encoder (1130) can encode the adaptively sampled map (1125) into a bitstream (1145) using image coding and / or video coding techniques.
[0123] The adaptive sampling module (1120) also generates auxiliary data (1127) that indicates the assistance information used for adaptive sampling. The auxiliary data encoder (1140) receives the auxiliary data (1127) and encodes the auxiliary data (1127) into a bitstream (1145).
[0124] The operation of the adaptive sampling module (1120) and the auxiliary data encoder (1140) are described further in this disclosure.
[0125] In the example of Figure 11, the bitstream (1145) is provided to a mesh decoder (1150). The mesh decoder (1150) includes a video decoder (1160), an auxiliary data decoder (1170), a reconstruction module (1180), and a mesh reconstruction module (1190) coupled together as shown in Figure 11. In one example, the video decoder (1160) corresponds to the video encoder (1130) and can decode portions of the bitstream (1145) encoded by the video encoder (1130) to generate decoded information (1165) (e.g., an initial decoding map).
[0126] In the example of Figure 11, the auxiliary data decoder (1170) corresponds to the auxiliary data encoder (1140) and is capable of decoding portions of the bitstream (1145) encoded by the auxiliary data encoder (1140) to generate decoded auxiliary data (1175).
[0127] In the example of Figure 11, the decoded information (1165) and the decoded auxiliary data (1175) are provided to a reconstruction module (1180). The reconstruction module (1180) can form a reconstructed map (1185) of the original resolution (e.g., the same number of pixels vertically and horizontally) as a map within the input mesh (1105). The mesh reconstruction module (1190) generates a reconstructed mesh (1195) based on the reconstructed map (1185). The operation of the auxiliary data decoder (1170) and the reconstruction module (1180) are described further in this disclosure.
[0128] It should be noted that components within the mesh encoder (1110), such as the segmentation module (1120), the video encoder (1130), and the auxiliary data encoder (1140), may each be implemented by a variety of techniques. In one example, the components are implemented by integrated circuits. In another example, the components are implemented using software that may be executed by one or more processors.
[0129] It should be noted that components within the mesh decoder (1150), such as the video decoder (1160), the auxiliary data decoder (1170), the reconstruction module (1180), and the mesh reconstruction module (1190), may each be implemented by a variety of technologies. In one example, the components are implemented by integrated circuits. In another example, the components are implemented using software that may be executed by one or more processors.
[0130] In some embodiments, sampling adaptation can be based on map type. In some examples, the adaptive sampling module (1120) can apply different sampling rates to different types of maps. For example, different sampling rates can be applied to geometry maps and attribute maps. In one example, a mesh is a model of an object with a regular shape and rich texture. For example, the object is rectangular but has rich colors. Thus, the information density of the geometry map is relatively low. In one example, the adaptive sampling module (1120) applies a first sampling rate of 2:1 (both vertically and horizontally) on the geometry map and a second sampling rate of 1:1 (both vertically and horizontally) on the texture map.
[0131] In some examples, a sampling rate of A:B in a direction indicates that B samples are generated from A pixels in the original map in that direction. For example, a sampling rate of 2:1 in the horizontal direction indicates that one sample is generated for every two pixels in the original map in the horizontal direction. A sampling rate of 2:1 in the vertical direction indicates that one sample is generated for every two pixels in the original map in the vertical direction. In some examples, a sampling step is used to indicate the sampling rate. The sampling step in a direction indicates the number of pixels as a step to generate a sample. For example, a sampling step of 2 in the horizontal direction indicates that a sample is generated for every two pixels in the horizontal direction. A sampling step of 2 in the vertical direction indicates that a sample is generated for every two pixels in the vertical direction.
[0132] In some embodiments, the sampling adaptation is based on sub-regions within the map. Different sampling rates can be applied to different parts of the map. In some examples, some rows of pixels have less information preserved and a larger sampling step can be applied along these rows, resulting in a smaller number of sample rows being encoded. In some examples, some columns of pixels have less information preserved and a larger sampling step can be applied along these columns, resulting in a smaller number of sample columns being encoded. In other regions, a smaller sampling step is applied to minimize information loss after sampling.
[0133] FIG. 12 shows an illustration of adaptive sampling in some examples. The map (1220) is divided into several block rows, each of which contains a fixed number of sample (pixel) rows. Different sampling rates are applied to the block rows in the vertical direction to generate the adaptively sampled map (1230). For example, each block row is a CTU row (also called a CTU line) and contains 64 rows of samples (also called pixels). In the example of FIG. 12, a first sampling rate of 2:1 is applied vertically to block rows 0 and 6 in the map (1220), resulting in 32 rows of samples for each of block rows 0 and 6 in the sampled adaptively sampled map (1230). A second sampling rate of 1:1 is applied vertically to block rows 1 to 5 in the map (1220), resulting in 64 rows of samples for each of block rows 1 to 5 in the adaptively sampled map (1230).
[0134] In some examples, the adaptively sampled map (1230) is encoded by an image or video encoder, such as the video encoder (1130). On the decoder side, in one example, the adaptively sampled map (1230) is decoded. After decoding, the top 32 rows of samples are restored (upsampled) to an original resolution, such as 64 rows of samples. The bottom 32 rows of samples are restored (upsampled), for example, by the restoration module (1180), to an original resolution, such as 64 rows of samples.
[0135] In some other examples, the map to be coded within the 2D representation of the 3D mesh can be divided into several sub-regions. Such divisions within the map (e.g., pictures) can include, for example, slices, tiles, tile groups, coding tree units, etc. In some examples, different sampling rates can be applied to different sub-regions. In one example, different sampling rates associated with different sub-regions can be signaled in a bitstream carrying the 3D mesh. At the decoder side, after decoding of the adaptively sampled map, each sub-region is restored to its original resolution according to the sampling rate associated with the sub-region, for example by a restoration module (1180).
[0136] In some instances, the process of restoring the adaptively sampled map to its original resolution is referred to as an inverse sampling process that produces a restored map. After restoration from the inverse sampling process, the output of the restored map in the form of a 2D atlas can be used for 3D mesh reconstruction.
[0137] Although the example of FIG. 12 shows adaptive sampling for different vertical block rows, similar adaptive sampling may be applied to different horizontal columns as well, or may be applied in both the vertical and horizontal directions.
[0138] In some embodiments, the sampling adaptation is patch-based. In some examples, different patches in a map can have different sampling rates.
[0139] FIG. 13 shows an illustration of adaptive sampling in some examples. A map (1320), such as a high-resolution 2D atlas, includes multiple 2D shapes, also called UV patches, that correspond to patches in a 3D mesh, such as a first 2D shape A′ and a second 2D shape B′. In the example of FIG. 13, a first sampling rate of 2:1 is applied to the first 2D shape A′ in both vertical and horizontal directions to generate a first sampled 2D shape A″. A second sampling rate of 1:1 is applied to the second 2D shape B′ in both vertical and horizontal directions to generate a second sampled 2D shape B″. The first sampled 2D shape A″ and the second sampled 2D shape B″ are placed into a new map, called an adaptively sampled map (1330).
[0140] In the example of FIG. 13, the first sampled 2D shape A″ is smaller than the first 2D shape A′, and the second sampled 2D shape B″ is the same size as the second 2D shape B′. The adaptively sampled map (1330) is encoded by an image or video encoder, such as a video encoder (1130), into a bitstream carrying the 3D mesh. In some examples, the sampling rate associated with the sampled 2D shapes is encoded into the bitstream carrying the 3D mesh, for example by an auxiliary data encoder (1140).
[0141] In some examples, on the decoder side, an image / video decoder, such as a video decoder (1160), decodes an initial map, such as an adaptively sampled map (1330), from a bitstream. Furthermore, a sampling rate associated with the sampled 2D shape is decoded from the bitstream, for example by an auxiliary data decoder (1170). According to the sampling rate associated with the sampled 2D shape, the sampled 2D shape in the adaptively sampled map (1330) is restored to its original size (e.g., the same number of pixels in the vertical and horizontal directions), for example by a restoration module (1180), to generate a restored map. The restored map is then used for 3D mesh reconstruction.
[0142] According to one aspect of the present disclosure, adaptive sampling information, such as sampling rates of different map types, sampling rates of different sub-regions, sampling rates of different patches, etc., is known at the mesh encoder side and the mesh decoder side. In some examples, the adaptive sampling information is appropriately encoded into the bitstream carrying the 3D mesh. Thus, the mesh decoder and the mesh encoder can operate based on the same adaptive sampling information. The mesh decoder can restore the map to the correct size.
[0143] In some examples, a default sampling rate is assumed by both the mesh encoder and the mesh decoder. In some other examples, the default sampling rate is signaled at the frame level or sequence level. In one example, for a map type, a flag is used to indicate if the sampling rate of the map type differs from the default sampling rate. If the sampling rate of the map type differs from the default sampling rate, the new sampling rate (or the difference from the default sampling rate) is signaled.
[0144] In another example, for a subregion of a map (block row, block column, CTU line, slice, tile, tile group, CTU, etc.), a flag is used to indicate whether the sampling rate of the subregion differs from the default sampling rate. If the sampling rate of the subregion differs from the default sampling rate, the new sampling rate (or the difference from the default sampling rate) is signaled.
[0145] In another example, for a patch in a map (e.g. a UV patch, a 2D shape), a flag is used to indicate if the sampling rate of the patch differs from the default sampling rate. If the sampling rate of the patch differs from the default sampling rate, the new sampling rate (or difference relative to the default sampling rate) is signaled.
[0146] In some examples, the set of sampling rates used in a frame or sequence is signaled in the bitstream or agreed upon by both the mesh encoder and mesh decoder. Examples of such sampling rates include 2:1, 4:1, 8:1, etc. If a particular sampling rate is selected for a map type, subregion, or patch, the index of the sampling rate in the set can be used to indicate the selection instead of signaling the sampling rate itself.
[0147] In some examples, patches in different mesh frames may not be in the same location. Over time, a patch in a mesh frame has a corresponding patch in another mesh frame (e.g., a pair of patches covering the same or similar portion of the mesh). A sampling rate applied to one patch in a previous mesh frame (e.g., applied to a corresponding UV patch in a map for representing the previous mesh frame and applied to a corresponding 2D shape in a map for representing the previous mesh frame) may be applied to a corresponding patch in a current mesh frame (e.g., applied to a corresponding UV patch in a map for representing the current mesh frame and applied to a corresponding 2D shape in a map for representing the current mesh frame). In some examples, a sequence of mesh frames may be a group of pictures (GOP). In one example, a sampling rate is signaled for a patch or sub-region or CTU line in a key mesh frame (e.g., the first frame) of the GOP. For patches or regions or lines in other mesh frames in the GOP, their sampling rates may be referenced to their counterparts in the key mesh frame.
[0148] FIG. 14 shows a flow chart outlining a process (1400) according to one embodiment of the present disclosure. The process (1400) can be used during the encoding process of a mesh. In various embodiments, the process (1400) is performed by a processing circuit. In some embodiments, the process (1400) is implemented in software instructions, and thus the processing circuit performs the process (1400) when the processing circuit executes the software instructions. The process starts at (S1401) and proceeds to (S1410).
[0149] At least two sampling rates associated with different portions of the 2D plurality of maps are determined (S1410). The plurality of maps are representations of a three-dimensional (3D) mesh frame. In one example, the adaptive sampling module (1120) determines at least two sampling rates associated with different portions of the 2D plurality of maps.
[0150] At (S1420), the plurality of maps are sampled based on at least two sampling rates to generate a plurality of sampled maps. In one example, the adaptive sampling module (1120) applies at least two sampling rates to different portions of the plurality of maps to generate a plurality of sampled maps. For example, a first portion of the plurality of maps is downsampled based on a first sampling rate and a second portion of the plurality of maps is downsampled based on a second sampling rate.
[0151] At (S1430), the multiple sampled maps are encoded into a bitstream. In one example, the video encoder (1130) encodes the multiple sampled maps into a bitstream (1145).
[0152] To determine the at least two sampling rates, in some examples, a first sampling rate associated with a first map of a first map type is determined, and a second sampling rate associated with a second map of a second map type is determined. The multiple maps include a first map and a second map. For example, the first map is a geometry map and the second map is a texture map.
[0153] To determine the at least two sampling rates, in some examples, a first sampling rate associated with a first sub-region in the first map is determined, and a second sampling rate associated with a second sub-region in the first map is determined. In one example, each of the first sub-region and the second sub-region includes a plurality of pixel rows (e.g., block rows). In another example, each of the first sub-region and the second sub-region includes a plurality of pixel columns. In another example, each of the first sub-region and the second sub-region includes a coding tree unit (CTU) line. In another example, each of the first sub-region and the second sub-region includes a slice. In another example, each of the first sub-region and the second sub-region includes a tile. In another example, each of the first sub-region and the second sub-region includes a tile group. In another example, each of the first sub-region and the second sub-region includes a CTU.
[0154] To determine the at least two sampling rates, in some examples, a first sampling rate associated with a first patch (e.g., a first UV patch, a first 2D shape) in a first map is determined, and a second sampling rate associated with a second patch (e.g., a second UV patch, a second 2D shape) in the first map is determined.
[0155] In some examples, the first flag is determined based on a comparison of a first sampling rate associated with a first portion of the plurality of sampled maps to a default sampling rate. The first flag associated with the first portion of the plurality of sampled maps is encoded into the bitstream, for example, by the auxiliary data encoder (1140). In one example, the first sampling rate is encoded into the bitstream, for example, by the auxiliary data encoder (1140), in response to the first sampling rate differing from the default sampling rate. In another example, a difference between the first sampling rate and the default sampling rate is encoded into the bitstream, for example, by the auxiliary data encoder (1140), in response to the first sampling rate differing from the default sampling rate.
[0156] In some examples, the first sampling rate is selected from a set of predefined sampling rates for a first portion of the plurality of maps. An index is then encoded into the bitstream, for example by the auxiliary data encoder (1140). The index indicates the selection of the first sampling rate from the set of predefined sampling rates.
[0157] In some examples, the bitstream carries a plurality of 3D mesh frames. In one example, a first sampling rate associated with a first patch of a first 3D mesh frame is applied during encoding of the first 3D mesh frame. Then, a second sampling rate associated with a second patch in a second 3D mesh frame is determined according to the first sampling rate. The second patch in the second 3D mesh frame is a patch corresponding to the first patch of the first 3D mesh frame. In one example, the first 3D mesh frame is a key mesh frame of a group of pictures (GOP), and the second sampling rate is equal to the first sampling rate.
[0158] Next, the process proceeds to (S1499) and ends.
[0159] The process (1400) may be adapted as appropriate. Step(s) of the process (1400) may be modified and / or omitted. Additional step(s) may be added. Any suitable order of performance may be used.
[0160] FIG. 15 shows a flow chart outlining a process (1500) according to an embodiment of the present disclosure. The process (1500) may be used during mesh decoding. In various embodiments, the process (1500) is performed by a processing circuit. In some embodiments, the process (1500) is implemented in software instructions, and thus the processing circuit performs the process (1500) when the processing circuit executes the software instructions. The process starts at (S1501) and proceeds to (S1510).
[0161] At (S1510), the 2D initial maps are decoded from a bitstream carrying the 3D mesh frame. In one example, the video decoder (1160) decodes the 2D initial maps from the bitstream (1145).
[0162] At least two sampling rates associated with different portions of the multiple initial maps are determined (S1520). In one example, the auxiliary data decoder (1170) determines at least two sampling rates associated with different portions of the multiple initial maps.
[0163] At (S1530), a plurality of restoration maps are obtained from the plurality of initial maps based on at least two sampling rates associated with different portions of the plurality of initial maps. In one example, the restoration module (1180) generates a plurality of restoration maps from the plurality of initial maps based on at least two sampling rates associated with different portions of the plurality of initial maps. For example, a first portion of the plurality of initial maps is restored (e.g., upsampled) based on a first sampling rate of the at least two sampling rates associated with the first portion, and a second portion of the plurality of initial maps is restored (e.g., upsampled) based on a second sampling rate of the at least two sampling rates associated with the second portion.
[0164] At (S1540), a 3D mesh frame is reconstructed based on the multiple restored maps. In one example, the mesh reconstruction module (1190) reconstructs the 3D mesh frame.
[0165] To determine the at least two sampling rates, in some examples, a first sampling rate associated with a first initial map of a first map type is determined, and a second sampling rate associated with a second initial map of a second map type is determined.
[0166] To determine the at least two sampling rates, in some examples, a first sampling rate associated with a first sub-region in the first initial map is determined. A second sampling rate associated with a second sub-region in the first initial map is determined. In one example, each of the first sub-region and the second sub-region includes a plurality of pixel columns (e.g., sampled block rows). In another example, each of the first sub-region and the second sub-region includes a plurality of pixel columns. In another example, each of the first sub-region and the second sub-region corresponds to a sampled coding tree unit (CTU) line. In another example, each of the first sub-region and the second sub-region includes a sampled slice. In another example, each of the first sub-region and the second sub-region includes a sampled tile. In another example, each of the first sub-region and the second sub-region includes a sampled tile group. In another example, each of the first sub-region and the second sub-region includes a sampled CTU.
[0167] To determine the at least two sampling rates, in some examples, a first sampling rate associated with a first patch (e.g., a first UV patch, a first 2D shape) in the first initial map is determined, and a second sampling rate associated with a second patch (e.g., a second UV patch, a second 2D shape) in the first initial map is determined.
[0168] In some examples, a first flag associated with a first portion of the multiple initial maps is decoded from the bitstream. The first flag indicates whether a first sampling rate associated with the first portion is different from a default sampling rate. In one example, the first flag indicates that the first sampling rate is different from the default sampling rate, and then the first sampling rate is decoded from the bitstream. In another example, the first flag indicates that the first sampling rate is different from the default sampling rate, and then the difference between the first sampling rate and the default sampling rate is decoded from the bitstream.
[0169] In some examples, an index is decoded from the bitstream, the index indicating a selection from a set of pre-defined sampling rates as the sampling rate associated with portions of the multiple initial maps.
[0170] In some examples, the bitstream carries a plurality of 3D mesh frames. A first map associated with a first 3D mesh frame is restored based on a first sampling rate associated with a first patch of the first 3D mesh frame. Then, a corresponding sampling rate associated with a second patch in a second 3D mesh frame is determined according to the first sampling rate. The second patch in the second 3D mesh frame is a patch corresponding to the first patch of the first 3D mesh frame. In one example, the first 3D mesh frame is a key mesh frame of a group of pictures (GOP), and the corresponding sampling rate is equal to the first sampling rate.
[0171] Next, the process proceeds to (S1599) and ends.
[0172] The process (1500) may be adapted as appropriate. Step(s) of the process (1500) may be modified and / or omitted. Additional step(s) may be added. Any suitable order of performance may be used.
[0173] The techniques disclosed in this disclosure may be used separately or combined in any order. Furthermore, each of the techniques (e.g., methods, embodiments), encoders, and decoders may be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In some examples, the one or more processors execute a program stored in a non-transitory computer-readable medium.
[0174] The techniques described above may be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, Figure 16 illustrates a computer system (1600) suitable for implementing certain embodiments of the disclosed subject matter.
[0175] The computer software may be coded using any suitable machine code or computer language that may undergo assembly, compilation, linking, or similar mechanisms to produce code containing instructions that may be executed, such as by one or more computer central processing units (CPUs) and graphics processing units (GPUs), either directly, or through interpretation and execution of microcode.
[0176] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, and the like.
[0177] 16 with respect to computer system (1600) are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing the embodiments of the present disclosure, nor should the arrangement of components be interpreted as having any dependency or requirement regarding any one or combination of components illustrated in the exemplary embodiment of computer system (1600).
[0178] The computer system (1600) may include certain human interface input devices. Such human interface input devices may be responsive to input by one or more human users, for example, via tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), olfactory input (not shown). Human interface devices may also be used to capture certain media not necessarily directly associated with conscious human input, such as audio (speech, music, ambient sounds, etc.), images (scanned images, photographic images obtained from still image cameras, etc.), and video (two-dimensional video, three-dimensional video including stereoscopic video, etc.).
[0179] The input human interface devices may include one or more of a keyboard (1601) (only one of each is depicted), a mouse (1602), a trackpad (1603), a touch screen (1610), a data glove (not shown), a joystick (1605), a microphone (1606), a scanner (1607), and a camera (1608).
[0180] The computer system (1600) may also include certain human interface output devices that may stimulate one or more of the senses of a human user, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touchscreen (1610), data gloves (not shown), or joystick (1605), although there may also be haptic feedback devices that do not function as input devices), audio output devices (such as speakers (1609), headphones (not depicted)), visual output devices (such as screens (1610), including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touchscreen input capability, each with or without haptic feedback capability, some capable of outputting two-dimensional visual output, or output in more than three dimensions through means such as stereoscopic output, i.e., virtual reality glasses (not depicted), holographic displays, and smoke tanks (not depicted)), and printers (not depicted).
[0181] The computer system (1600) may also include human-accessible storage devices and their associated media, such as optical media (1621), including CD / DVD ROM / RW (1620) with CD / DVD or similar media, thumb drives (1622), removable hard drives or solid state drives (1623), legacy magnetic media such as tapes and floppy disks (not depicted), and dedicated ROM / ASIC / PLD based devices such as security dongles (not depicted).
[0182] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the subject matter of this disclosure does not encompass transmission media, carrier waves, or other transitory signals.
[0183] The computer system (1600) may also include an interface (1654) to one or more communications networks (1655). The networks may be, for example, wireless, wired, optical. The networks may further be local, wide area, metropolitan, vehicular and industrial, real-time, delay tolerant, etc. Examples of networks include local area networks such as Ethernet, wireless LAN, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., television wired or wireless wide area digital networks including cable television, satellite television, and terrestrial broadcast television, vehicular and industrial including CANBus, and many others. Certain networks generally require an external network interface adapter attached to a particular general-purpose data port or peripheral bus (1649) (e.g., a USB port on the computer system (1600)), while others are generally integrated into the core of the computer system (1600) by attachment to a system bus as described below (e.g., an Ethernet interface to a PC computer system, or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (1600) may communicate with other entities. Such communications may be unidirectional and receive only (e.g., broadcast TV), unidirectional and transmit only (e.g., CANbus to certain CANbus devices), or bidirectional, for example, to other computer systems using local or wide area digital networks. Specific protocols and protocol stacks may be used with each of these networks and network interfaces, as described above.
[0184] The aforementioned human interface devices, human accessible storage devices, and network interfaces may be attached to a core (1640) of the computer system (1600).
[0185] The core (1640) may include one or more central processing units (CPUs) (1641), graphics processing units (GPUs) (1642), dedicated programmable processing units in the form of field programmable gate arrays (FPGAs) (1643), hardware accelerators for specific tasks (1644), and graphics adapters (1650). These devices may be connected via a system bus (1648), along with read-only memory (ROM) (1645), random access memory (1646), and internal mass storage devices (1647) such as internal hard drives, SSDs, etc. that are not user accessible. In some computer systems, the system bus (1648) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus (1648) or via a peripheral bus (1649). In one example, a screen 1610 may be connected to a graphics adapter 1650. Peripheral bus architectures include PCI, USB, etc.
[0186] The CPU (1641), GPU (1642), FPGA (1643), and accelerator (1644) can execute certain instructions that can combine to constitute the aforementioned computer code. The computer code can be stored in ROM (1645) or RAM (1646). Transient data can also be stored in RAM (1646), while persistent data can be stored, for example, in internal mass storage (1647). Rapid storage and retrieval to any of the memory devices can be enabled through the use of cache memory, which can be closely associated with one or more CPUs (1641), GPUs (1642), mass storage (1647), ROM (1645), RAM (1646), etc.
[0187] The computer-readable medium may bear computer code for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the type well known and available to those skilled in the computer software arts.
[0188] By way of example and not of limitation, the computer system (1600) having the architecture, and specifically the core (1640), may provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media may be user-accessible mass storage devices as described above, as well as media associated with the core's (1640) specific storage device of a persistent nature, such as the core's internal mass storage device (1647) or ROM (1645). Software implementing various embodiments of the present disclosure may be stored in such devices and executed by the core (1640). The computer-readable media may include one or more memory devices or chips, depending on the particular needs. The software may cause the core (1640), and specifically the processor therein (including a CPU, GPU, FPGA, etc.) to perform a particular process or a particular portion of a particular process, as described herein. The process may include defining data structures stored in RAM (1646) and modifying such data structures according to the process defined by the software. Additionally, or alternatively, the computer system may provide functionality as a result of logic circuitry, hardwired or otherwise embodied in circuitry (e.g., accelerator (1644)), which may operate in place of or in conjunction with software to perform particular operations or portions of particular operations described herein. References to software may encompass logic circuitry, and vice versa, where appropriate. Where appropriate, references to computer-readable media may encompass circuitry (such as integrated circuits (ICs)) that store software for execution, circuitry that embodies logic for execution, or both. The present disclosure encompasses any suitable combination of hardware and software.
[0189] While this disclosure has described several exemplary embodiments, there are modifications, substitutions, and various substitute equivalents that are within the scope of this disclosure. It will thus be appreciated that those skilled in the art will be able to devise numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within its spirit and scope. [Explanation of symbols]
[0190] 100 Communication Systems 105 Sensors 110 Terminal Devices 120 Terminal Devices 150 Network 200 Streaming System 201 Point cloud source 202 point cloud 203 Encoder 204 Compressed Point Cloud 205 Streaming Server 206 Client Subsystem 207 Copy 208 Client Subsystem 209 Copy 210 Decoder 211 point cloud 212 Rendering Devices 213 Capture Subsystem 220 Electronic Devices 230 Electronic Devices 300 V-Point Cloud Compression (PCC) Encoder 304 Patch Information Module 306 Patch Generation Module 308 Patch Packing Module 310 Geometry Image Generation Module 312 Texture Image Generation Module 314 Occupancy Map Module 316 Image Padding Module 318 Image Padding Module 320 Group Expansion Module 322 Video Compression Module 323 Video Compression Module 324 Multiplexer 332 Video Compression Module 334 Entropy Compression Module 336 Smoothing Module 338 Auxiliary patch information compression module 400V-PCC Decoder 432 Demultiplexer 434 Video Extraction Module 436 Video Extraction Module 438 Occupancy Map Expansion Module 442 Auxiliary patch information deployment module 444 Geometry Reconstruction Module 446 Smoothing Module 448 Texture Reconstruction Module 452 Color Smoothing Module 510 Video Decoder 520 Parser 521 Symbols 551 Scaler / Descaler Unit 552 Intra-picture prediction unit 553 Motion Compensation Prediction Unit 555 Aggregator 556 Loop Filter Unit 557 Reference Picture Memory 558 Current Picture Buffer 603 Video Encoder 630 Source Coder 632 Coding Engine 633 Decoder 634 Reference Picture Memory, Reference Picture Cache 635 Predictor 645 Entropy Coder 650 Controller 700G-PCC Encoder 701 point cloud 702 Bitstream 710 Position Quantization Module 712 Duplicate point removal module 720 Attribute Transfer Module 730 Octree Encoding Module 740 Level of Detail (LOD) Generation Module 750 Attribute Prediction Module 760 Residual Quantization Module 770 Arithmetic Coding Module 780 Inverse Residual Quantization Module 781 Addition Module 790 Memory 800 G-PCC Decoder 801 Bitstream 802 Decoded point cloud 810 Arithmetic Decoding Module 820 Inverse Residual Quantization Module 830 Octree Decoding Module 840 LOD Generation Module 850 Attribute Prediction Module 860 Memory 910 Three-dimensional (3D) Mesh 920 2D Atlas, Maps 1020 Map 1030 Downsampled Map 1100 Framework 1105 Input Mesh 1110 Mesh Encoder 1120 Adaptive Sampling Module 1125 Adaptively Sampled Maps 1127 Auxiliary Data 1130 Video Encoder 1140 Auxiliary Data Encoder 1145 Bitstream 1150 Mesh Decoder 1160 Video Decoder 1165 Decoded Information 1170 Auxiliary Data Decoder 1175 Decoded auxiliary data 1180 Restoration Module 1185 Reconstructed Maps 1190 Mesh Reconstruction Module 1195 Reconstructed Meshes 1220 Map 1230 Adaptively Sampled Maps 1320 Map 1330 Adaptively Sampled Maps 1400 Processing 1500 Processing 1600 Computer Systems 1601 Keyboard 1602 Mouse 1603 Trackpad 1605 Joystick 1606 Mike 1607 Scanner 1608 Camera 1609 Speaker 1610 Touch Screen 1620 CD / DVD ROM / RW 1621 Optical Media 1622 Thumb Drive 1623 Solid State Drive 1640 cores 1641 Central Processing Unit (CPU) 1642 Graphics Processing Unit (GPU) 1643 Field Programmable Gate Array (FPGA) 1644 Hardware Accelerator 1645 Read-Only Memory (ROM) 1646 Random Access Memory 1647 Internal mass storage 1648 System Bus 1649 Peripheral Bus 1650 Graphics Adapter 1654 Interface 1655 Communication Network
Claims
1. A method for decoding a plurality of two-dimensional initial maps from a bitstream carrying a plurality of three-dimensional (3D) mesh frames; determining at least two sampling rates associated with different portions of the plurality of initial maps; reconstructing from the initial maps based on the at least two sampling rates associated with the different portions of the initial maps to obtain a plurality of reconstructed maps, wherein a first portion of the initial maps is reconstructed based on a first sampling rate of the at least two sampling rates associated with the first portion, and a second portion of the initial maps is reconstructed based on a second sampling rate of the at least two sampling rates associated with the second portion; reconstructing the 3D mesh frame based on the plurality of reconstructed maps; and recovering a first map associated with a first patch of a first 3D mesh frame based on the first sampling rate associated with the first patch of the first 3D mesh frame; determining a corresponding sampling rate associated with a second patch in a second 3D mesh frame according to the first sampling rate, wherein the second patch in the second 3D mesh frame is a patch that corresponds to the first patch of the first 3D mesh frame.
2. The step of determining the at least two sampling rates comprises: determining the first sampling rate associated with a first initial map of a first map type; determining the second sampling rate associated with a second initial map of a second map type; The method of claim 1 further comprising:
3. The step of determining the at least two sampling rates comprises: determining the first sampling rate associated with a first sub-region in a first initial map; determining the second sampling rate associated with a second sub-region in the first initial map; The method of claim 1 further comprising:
4. The first sub-region comprises: Multiple pixel rows, A plurality of pixel columns, Sampled coding tree unit (CTU) lines, Sampled slices, Sampled tiles The tiles sampled, and / or Sampled CTU, The method of claim 3 , comprising at least one of:
5. The step of determining the at least two sampling rates comprises: determining the first sampling rate associated with a first patch in a first initial map; determining the second sampling rate associated with a second patch in the first initial map; The method of claim 1 further comprising:
6. The step of determining the at least two sampling rates comprises: decoding, from the bitstream, a first flag associated with the first portion of the plurality of initial maps, the first flag indicating whether the first sampling rate associated with the first portion is different from a default sampling rate; The method of claim 1 further comprising:
7. the first flag indicating that the first sampling rate is different from the default sampling rate, the method further comprising: Decoding the first sampling rate from the bitstream; and / or decoding a difference between the first sampling rate and the default sampling rate from the bitstream; The method of claim 6 , further comprising at least one of:
8. The step of determining the at least two sampling rates comprises: decoding from the bitstream an index indicating a selection from a set of predefined sampling rates as a sampling rate associated with portions of the plurality of initial maps; The method of claim 1 further comprising:
9. The method of claim 1 , wherein the first 3D mesh frame is a key frame of a group of pictures (GOP), and the corresponding sampling rate is equal to the first sampling rate.
10. 10. Apparatus comprising processing circuitry configured to perform the method of any one of claims 1 to 9.
11. A program comprising instructions which, when executed by a computer, cause the computer to carry out the method according to any one of claims 1 to 9.
Citation Information
Patent Citations
Volumetric video encoding / decoding method, apparatus, and streams
JP2021502033A
Video based point cloud compression-patch alignment and size determination in bounding box
US20200314435A1
Mesh compression via point cloud representation
WO2021116838A1
