Methods, apparatuses, and storage media for mesh compression and decompression

Abstract

Aspects of the disclosure provide methods, apparatuses, and storage media for mesh compression and decompression. In some examples, an apparatus for mesh coding includes processing circuitry. The processing circuitry decodes, from a bitstream carrying a three-dimensional (3D) mesh frame, a plurality of initial maps that are two-dimensional. The processing circuitry determines at least two sampling rates associated with different portions of the plurality of initial maps, and recovers a plurality of recovered maps from the plurality of initial maps based on the at least two sampling rates associated with the different portions of the plurality of initial maps. A first portion of the plurality of initial maps is recovered based on a first sampling rate, and a second portion of the plurality of initial maps is recovered based on a second sampling rate. The processing circuitry reconstructs the 3D mesh frame based on the plurality of recovered maps.

Description

[0001] Related applications

[0002] This application claims priority to U.S. Patent Application No. 17 / 903,634, filed September 6, 2022, entitled "2D Atlas Adaptive Sampling in 3D Mesh Compression," which claims priority to U.S. Provisional Application No. 63 / 241,478, filed September 7, 2021, entitled "2D Atlas Adaptive Sampling in 3D Mesh Compression." The entire disclosure of the prior applications is incorporated herein by reference. Technical Field

[0003] This disclosure describes embodiments that generally involve grid coding. Background Technology

[0004] The background description provided herein is intended to provide a general overview of the context of this disclosure. To the extent described in this background section, the work of the currently attributed inventors and aspects of the description that may not conform to the prior art at the time of filing are neither explicitly nor implicitly considered to be prior art of this disclosure.

[0005] Various technologies have been developed to capture and represent the world, such as objects and environments in three-dimensional (3D) space. A 3D representation of the world enables more immersive forms of interaction and communication. In some examples, point clouds and meshes can be used as 3D representations of the world. Summary of the Invention

[0006] This disclosure provides methods, apparatus, and storage media for mesh encoding (e.g., compression and decompression). In some examples, the apparatus for mesh decompression includes processing circuitry. The processing circuitry decodes multiple two-dimensional (2D) initial images from a bitstream carrying a three-dimensional (3D) mesh frame. The processing circuitry determines at least two sampling rates associated with different portions of the multiple initial images and recovers multiple restored images from the multiple initial images based on the at least two sampling rates associated with the different portions of the multiple initial images. A first portion of the multiple initial images is recovered based on a first sampling rate associated with a first portion of the multiple initial images, and a second portion of the multiple initial images is recovered based on a second sampling rate associated with a second portion of the multiple initial images. The processing circuitry reconstructs a 3D mesh frame based on the multiple restored images.

[0007] In some examples, the processing circuitry determines a first sampling rate associated with a first initial graph of a first graph type; and determines a second sampling rate associated with a second initial graph of a second graph type.

[0008] In some examples, the processing circuitry determines a first sampling rate associated with a first sub-region in the first initial image; and determines a second sampling rate associated with a second sub-region in the first initial image. The first or second sub-region may include at least one of the following: multiple rows of pixels; multiple columns of pixels; sampled coding tree unit (CTU) lines; sampled slices; sampled tiles; sampled groups of tiles; and / or sampled CTUs.

[0009] In some examples, the processing circuitry determines a first sampling rate associated with a first patch in the first initial diagram; and determines a second sampling rate associated with a second patch in the first initial diagram.

[0010] In some examples, the processing circuitry decodes a first flag associated with a first portion of a plurality of initial graphs from the bitstream. The first flag indicates whether a first sampling rate associated with the first portion differs from a default sampling rate. When the first flag indicates that the first sampling rate differs from the default sampling rate, in one example, the processing circuitry decodes the first sampling rate from the bitstream; in another example, the processing circuitry decodes the difference between the first sampling rate and the default sampling rate from the bitstream.

[0011] In some examples, the processing circuitry decodes an index from the bitstream that indicates a sampling rate selected from a set of predefined sampling rates as part of multiple initial graphs.

[0012] In some examples, the bitstream carries multiple 3D mesh frames. The processing circuitry reconstructs a first image associated with the first 3D mesh frame based on a first sampling rate associated with a first patch in the first 3D mesh frame; and determines a sampling rate associated with a second patch in the second 3D mesh frame based on the first sampling rate. The second patch in the second 3D mesh frame is the patch corresponding to the first patch in the first 3D mesh frame. In one example, the first 3D mesh frame is a keyframe of a Group of Pictures (GOP), and its corresponding sampling rate is equal to the first sampling rate.

[0013] This disclosure also provides a non-transitory computer-readable storage medium for storing instructions that, when executed by a computer, cause the computer to perform any one or a combination of methods for grid decompression.

[0014] In some examples, the apparatus for mesh compression includes processing circuitry. The processing circuitry determines at least two sampling rates associated with different portions of a plurality of two-dimensional (2D) graphs, wherein the plurality of graphs are represented in a three-dimensional (3D) mesh frame; samples the plurality of graphs based on the at least two sampling rates to generate a plurality of sampled graphs; and encodes the plurality of sampled graphs into a bitstream.

[0015] This disclosure also provides a non-transitory computer-readable storage medium for storing instructions that, when executed by a computer, cause the computer to perform any one or a combination of methods for grid compression. Attached Figure Description

[0016] Further features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings, wherein:

[0017] Figure 1 Block diagrams of communication systems in some examples are shown.

[0018] Figure 2 Block diagrams of some example flow systems are shown.

[0019] Figure 3 Block diagrams of encoders for encoding point cloud frames are shown in some examples.

[0020] Figure 4 Block diagrams of decoders for decoding compressed bitstreams corresponding to point cloud frames are shown in some examples.

[0021] Figure 5 Block diagrams of video decoders in some examples are shown.

[0022] Figure 6 Block diagrams of video encoders are shown in some examples.

[0023] Figure 7 Block diagrams of encoders for encoding point cloud frames are shown in some examples.

[0024] Figure 8 Block diagrams of decoders for decoding compressed bitstreams carrying point cloud frames are shown in some examples.

[0025] Figure 9 The diagram shows some examples of grid-to-map mappings.

[0026] Figure 10 The following are some examples of downsampling plots.

[0027] Figure 11 A framework diagram of mesh compression according to some embodiments of the present disclosure is shown.

[0028] Figure 12 Adaptive sampling plots are shown in some examples.

[0029] Figure 13 Adaptive sampling plots are shown in some examples.

[0030] Figure 14 A flowchart outlining some of the processing examples is shown.

[0031] Figure 15 A flowchart outlining some of the processing examples is shown.

[0032] Figure 16 These are schematic diagrams of computer systems in some examples. Detailed Implementation

[0033] This disclosure provides techniques in the field of three-dimensional (3D) media processing.

[0034] Technological advancements in 3D media processing (e.g., progress in 3D capture, 3D modeling, and 3D rendering) have facilitated the widespread availability of 3D media content across multiple platforms and devices. In one example, while a baby's first steps can be captured on one continent, media technology could allow grandparents on another continent to watch (and potentially interact with) and enjoy an immersive experience with the baby. According to one aspect of this disclosure, to enhance the immersive experience, 3D models are becoming increasingly complex, and the creation and consumption of 3D models consume significant data resources, such as data storage and data transmission resources.

[0035] According to some aspects of this disclosure, point clouds and meshes can be used as 3D models to represent immersive content.

[0036] A point cloud typically refers to a set of points in 3D space, each point possessing associated attributes such as color, material properties, texture information, intensity, reflectivity, motion-related properties, modal properties, and various other attributes. Point clouds can be used to reconstruct objects or scenes as combinations of these points.

[0037] An object's mesh (also called a mesh model) can include polygons describing the object's surface. Each polygon can be defined by its vertices in 3D space and information about how these vertices connect to form the polygon. Information about how the vertices are connected is called connectivity information. In some examples, the mesh may also include properties associated with the vertices, such as color, normals, etc.

[0038] According to some aspects of this disclosure, some encoding tools for point cloud compression (PCC) can be used for mesh compression. For example, a mesh can be remeshed to generate a new mesh, and the connectivity information of the new mesh can be inferred. The vertices of the new mesh and the properties associated with the vertices of the new mesh can be considered as points in the point cloud and can be compressed using a PCC codec.

[0039] Point clouds can be used to reconstruct objects or scenes as combinations of points. These points can be captured using multiple cameras, depth sensors, or LiDAR in various settings and can consist of thousands to billions of points to realistically represent the reconstructed scene or object. Patches can typically refer to a continuous subset of a surface described by a point cloud. In one example, a patch consists of points whose surface normals are offset from each other by less than a threshold amount.

[0040] PCC can be performed using various schemes, such as geometry-based schemes (referred to as G-PCC) and video coding-based schemes (referred to as V-PCC). According to some aspects of this disclosure, G-PCC directly encodes 3D geometry and is a purely geometry-based method with little sharing with video coding; while V-PCC is primarily based on video coding, for example, mapping points in a 3D point cloud to pixels in a 2D raster (image). The V-PCC scheme can utilize general-purpose video codecs for point cloud compression. The PCC codec (encoder / decoder) in this disclosure can be a G-PCC codec (encoder / decoder) or a V-PCC codec (encoder / decoder).

[0041] According to one aspect of this disclosure, the V-PCC scheme can use existing video codecs to compress the geometry, occupancy, and texture of a point cloud into three separate video sequences. The additional metadata required to interpret the three video sequences is compressed separately. The metadata occupies only a small portion of the entire bitstream, and in one example, software implementations can be used to efficiently encode / decode the metadata. Most of the information is handled by the video codec.

[0042] Figure 1 Block diagrams of a communication system 100 as shown in some examples are illustrated. The communication system 100 includes multiple terminal devices capable of communicating with each other via, for example, a network 150. For example, the communication system 100 includes a pair of terminal devices 110 and 120 interconnected via network 150. Figure 1In the example, a one-way transmission of point cloud data can be performed between the first pair of terminal devices 110 and 120. For example, terminal device 110 can compress a point cloud (e.g., points representing structures) captured by sensor 105 connected to terminal device 110. The compressed point cloud can be transmitted, for example, as a bitstream to another terminal device 120 via network 150. Terminal device 120 can receive the compressed point cloud from network 150, decompress the bitstream to reconstruct the point cloud, and display the reconstructed point cloud appropriately. One-way data transmission is common in media service applications, etc.

[0043] exist Figure 1 In the examples, terminal devices 110 and 120 may be illustrated as servers and personal computers, but the principles of this disclosure are not limited thereto. Embodiments of this disclosure are applicable to laptop computers, tablet computers, smartphones, gaming terminals, media players, and / or dedicated three-dimensional (3D) devices. Network 150 represents any number of networks transmitting compressed point clouds between terminal devices 110 and 120. Network 150 may include, for example, wired communication networks and / or wireless communication networks. Network 150 may exchange data in circuit-switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks (LANs), wide area networks (WANs), the Internet, etc.

[0044] Figure 2 Block diagrams of streaming system 200 are shown in some examples. Streaming system 200 is an application that uses point clouds. The disclosed subject matter can also be applied to other applications that utilize point clouds, such as 3D telepresence applications, virtual reality applications, etc.

[0045] Streaming system 200 may include a capture subsystem 213. Capture subsystem 213 may include a point cloud source 201 that generates, for example, an uncompressed point cloud 202, such as a Light Detection and Ranging (LIDAR) system, a 3D camera, a 3D scanner, a graphics generation component that generates the uncompressed point cloud in software, etc. In one example, point cloud 202 comprises points captured by a 3D camera. Point cloud 202 is depicted as thick lines to emphasize its higher data volume compared to a compressed point cloud 204 (a bitstream of the compressed point cloud). Compressed point cloud 204 may be generated by electronic device 220, which includes an encoder 203 coupled to point cloud source 201. Encoder 203 may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The compressed point cloud 204 (or the bitstream 204 of the compressed point cloud) is depicted as thin lines to emphasize its lower data volume compared to the point cloud stream 202, and can be stored on the streaming server 205 for future use. One or more streaming client subsystems (e.g., Figure 2Client subsystems 206 and 208 can access streaming server 205 to retrieve copies 207 and 209 of compressed point cloud 204. Client subsystem 206 may include, for example, decoder 210 in electronic device 230. Decoder 210 decodes input copy 207 of compressed point cloud and creates output stream 211 of reconstructed point cloud that can be rendered on rendering device 212.

[0046] It should be noted that electronic devices 220 and 230 may include other components (not shown). For example, electronic device 220 may include a decoder (not shown), and electronic device 230 may also include an encoder (not shown).

[0047] In some streaming systems, compressed point clouds (204, 207, and 209) (e.g., a bitstream of a compressed point cloud) can be compressed according to specific standards. In some examples, video coding standards are used in the compression of point clouds. Examples of these standards include High Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), and others.

[0048] Figure 3 A block diagram of a V-PCC encoder 300 for encoding point cloud frames is shown according to some embodiments. In some embodiments, the V-PCC encoder 300 can be used in communication system 100 and streaming system 200. For example, encoder 203 can be configured and operated in a similar manner to V-PCC encoder 300.

[0049] V-PCC encoder 300 receives point cloud frames as uncompressed input and generates a bitstream corresponding to compressed point cloud frames. In some embodiments, V-PCC encoder 300 may receive point cloud frames from a point cloud source (e.g., point cloud source 201, etc.).

[0050] exist Figure 3 In the example, the V-PCC encoder 300 includes a patch generation module 306, a patch packing module 308, a geometry image generation module 310, a texture image generation module 312, a patch information module 304, a occupancy map module 314, a smoothing module 336, image filling modules 316 and 318, a group dilation module 320, video compression modules 322, 323 and 332, an auxiliary patch information compression module 338, an entropy compression module 334, and a multiplexer 324.

[0051] According to one aspect of this disclosure, a V-PCC encoder 300 converts a 3D point cloud frame into an image-based representation and some metadata (e.g., occupancy map and patch information) for converting the compressed point cloud back to a decompressed point cloud. In some examples, the V-PCC encoder 300 may convert the 3D point cloud frame into a geometry image, a texture image, and an occupancy map, and then encode the geometry image, texture image, and occupancy map into a bitstream using video coding techniques. Typically, a geometry image is a 2D image in which pixels are filled with geometric values ​​associated with points projected onto the pixel; pixels filled with geometric values ​​may be referred to as geometry samples. A texture image is a 2D image in which pixels are filled with texture values ​​associated with points projected onto the pixel; pixels filled with texture values ​​may be referred to as texture samples. An occupancy map is a 2D image in which pixels are filled with values ​​indicating whether they are occupied or not by a patch.

[0052] The patch generation module 306 divides the point cloud into a set of patches (e.g., patches are defined as a continuous subset of surfaces described by the point cloud), which may overlap or not, such that each patch can be described by a depth field relative to a plane in 2D space. In some embodiments, the patch generation module 306 aims to decompose the point cloud into a minimum number of patches with smooth boundaries while minimizing reconstruction errors.

[0053] In some examples, the patch information module 304 can collect patch information indicating the size and shape of the patch. In some examples, the patch information can be packaged into an image frame and then encoded by the auxiliary patch information compression module 338 to generate compressed auxiliary patch information.

[0054] In some examples, the patch packing module 308 is configured to map extracted patches onto a two-dimensional (2D) grid while minimizing unused space and ensuring that each M×M (e.g., 16×16) block of the grid is associated with a unique patch. Efficient patch packing can directly impact compression efficiency by minimizing unused space or ensuring time consistency.

[0055] The geometry image generation module 310 generates a 2D geometry image associated with the geometry of the point cloud at a given patch location. The texture image generation module 312 generates a 2D texture image associated with the texture of the point cloud at a given patch location. Both the geometry image generation module 310 and the texture image generation module 312 utilize a 3D-to-2D mapping computed during the packing process to store the geometry and texture of the point cloud as images. To better handle cases where multiple points are projected onto the same sample, each patch is projected onto two images, called layers. In one example, the geometry image is represented by a WxH monochrome frame in YUV420-8-bit format. To generate the texture image, the texture generation process utilizes the reconstructed / smoothed geometry to compute the color associated with the resampled points.

[0056] Occupancy map module 314 can generate an occupancy map describing the filling information for each cell. For example, the occupancy image includes a binary map that indicates whether each cell of the raster belongs to empty space or to a point cloud. In one example, the occupancy map uses binary information describing whether each pixel is filled. In another example, the occupancy map uses binary information describing whether each pixel block is filled.

[0057] The occupancy map generated by the occupancy map module 314 can be compressed using either lossless or lossy encoding. When lossless encoding is used, the entropy compression module 334 is used to compress the occupancy map. When lossy encoding is used, the video compression module 332 is used to compress the occupancy map.

[0058] It is important to note that the patch packing module 308 may leave some blank space between the 2D patches packed in the image frame. Image padding modules 316 and 318 may fill this blank space (called padding) to generate image frames suitable for 2D video and image codecs. Image padding, also known as background padding, fills unused space with redundant information. In some examples, good background padding increases the bit rate minimally without introducing significant coding distortion around patch boundaries.

[0059] Video compression modules 322, 323, and 332 can encode 2D images (e.g., filled geometric images, filled texture images, and occupancy maps) based on a suitable video coding standard (e.g., HEVC, VVC, etc.). In one example, video compression modules 322, 323, and 332 are independent components that operate separately. It should be noted that in another example, video compression modules 322, 323, and 332 can be implemented as a single component.

[0060] In some examples, the smoothing module 336 is configured to generate a smoothed image of the reconstructed geometry. This smoothed image can be provided to the texture image generation module 312. The texture image generation module 312 can then adjust the generation of the texture image based on the reconstructed geometry. For example, when the patch shape (e.g., geometry) is slightly distorted during encoding and decoding, this distortion can be taken into account when generating the texture image to correct for the patch shape distortion.

[0061] In some embodiments, the group dilation module 320 is configured to fill pixels around the object boundary with redundant low-frequency content in order to improve coding gain and visual quality of the reconstructed point cloud.

[0062] Multiplexer 324 can multiplex compressed geometric images, compressed texture images, compressed occupancy maps, and compressed auxiliary patch information into a compressed bitstream.

[0063] Figure 4 Block diagrams of a V-PCC decoder 400 for decoding compressed bitstreams corresponding to point cloud frames are shown in some examples. In some examples, the V-PCC decoder 400 can be used in communication system 100 and streaming system 200. For example, decoder 210 can be configured to operate in a similar manner to V-PCC decoder 400. V-PCC decoder 400 receives a compressed bitstream and generates a reconstructed point cloud based on the compressed bitstream.

[0064] exist Figure 4 In the example, the V-PCC decoder 400 includes a demultiplexer 432, video decompression modules 434 and 436, occupancy map decompression module 438, auxiliary patch information decompression module 442, geometry reconstruction module 444, smoothing module 446, texture reconstruction module 448, and color smoothing module 452.

[0065] The demultiplexer 432 can receive a compressed bitstream and separate it into a compressed texture image, a compressed geometric image, a compressed occupancy map, and compressed auxiliary patch information.

[0066] Video decompression modules 434 and 436 can decode compressed images and output decompressed images according to appropriate standards (e.g., HEVC, VVC, etc.). For example, video decompression module 434 decodes compressed texture images and outputs decompressed texture images; and video decompression module 436 decodes compressed geometric images and outputs decompressed geometric images.

[0067] Occupancy map decompression module 438 can decode the compressed occupancy map according to a suitable standard (e.g., HEVC, VVC, etc.) and output the decompressed occupancy map.

[0068] The auxiliary patch information decompression module 442 can decode the compressed auxiliary patch information according to a suitable standard (e.g., HEVC, VVC, etc.) and output the decompressed auxiliary patch information.

[0069] The geometric reconstruction module 444 can receive the decompressed geometric image and generate the reconstructed point cloud geometry based on the decompressed occupancy map and the decompressed auxiliary patch information.

[0070] The smoothing module 446 smooths out inconsistencies at patch edges. The smoothing process is designed to mitigate potential discontinuities that may occur at patch boundaries due to compression distortion. In some embodiments, a smoothing filter may be applied to pixels located at patch boundaries to mitigate distortion that may be caused by compression / decompression.

[0071] The texture reconstruction module 448 can determine the texture information of points in the point cloud based on the decompressed texture image and smoothed geometry.

[0072] Color smoothing module 452 smooths out inconsistencies in shading. In 2D video, non-adjacent tiles in 3D space are often packed close together. In some examples, pixel values ​​from non-adjacent tiles may be mixed by block-based video codecs. The goal of color smoothing is to reduce visible artifacts appearing at tile boundaries.

[0073] Figure 5 Block diagrams of video decoder 510 in some examples are shown. Video decoder 510 can be used in V-PCC decoder 400. For example, video decompression modules 434 and 436, and occupation graph decompression module 438 can be similarly configured as video decoder 510.

[0074] Video decoder 510 may include parser 520 to reconstruct symbols 521 from compressed images (e.g., encoded video sequences). Categories of these symbols include information for managing the operation of video decoder 510. Parser 520 may perform parsing / entropy decoding on the received encoded video sequence. The encoding of the encoded video sequence may be based on video coding techniques or standards and may follow various principles, including variable-length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. Parser 520 may extract a set of subgroup parameters for at least one pixel subgroup in the video decoder from the encoded video sequence based on at least one parameter corresponding to the group. Subgroups may include Groups of Pictures (GOPs), pictures, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), etc. Parser 520 may also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, etc.

[0075] The parser 520 can perform entropy decoding / parsing operations on the video sequence received from the buffer memory in order to create symbol 521.

[0076] Depending on the type of encoded video picture or its components (e.g., inter- and intra-pictures, inter- and intra-blocks) and other factors, the reconstruction of symbol 521 can involve multiple distinct units. Which units are involved and how they are involved can be controlled by subgroup control information parsed from the encoded video sequence by parser 520. For clarity, the flow of this subgroup control information between parser 520 and the various units below is not described.

[0077] In addition to the functional blocks already mentioned, the video decoder 510 can be conceptually subdivided into several functional units as described below. In a practical implementation operating under commercial constraints, many of these units interact closely with each other and can be integrated at least partially with one another. However, for the sake of describing the disclosed subject matter, it is appropriate to conceptually subdivide it into the following functional units.

[0078] The first unit is the scaler / inverse transform unit 551. The scaler / inverse transform unit 551 receives the quantized transform coefficients and control information (including which transform to use, block size, quantization factor, quantization scaling matrix, etc.) as symbols 521 from the parser 520. The scaler / inverse transform unit 551 can output blocks containing sample values, which can be input into the aggregator 555.

[0079] In some cases, the output samples of the scaler / inverse transform unit 551 may belong to intra-coded blocks; that is, blocks that do not use prediction information from previously reconstructed images but can use prediction information from previously reconstructed portions of the current image. This prediction information can be provided by the intra-picture prediction unit 552. In some cases, the intra-picture prediction unit 552 uses surrounding reconstructed information obtained from the current picture buffer 558 to generate blocks of the same size and shape as the blocks in the reconstruction. The current picture buffer 558 buffers, for example, partially reconstructed current images and / or fully reconstructed current images. In some cases, the aggregator 555 adds the prediction information already generated by the intra-picture prediction unit 552 to the output sample information provided by the scaler / inverse transform unit 551 for each sample.

[0080] In other cases, the output samples of the scaler / inverse transform unit 551 may belong to inter-frame coded blocks and may be motion-compensated. In this case, the motion compensation prediction unit 553 can access the reference image memory 557 to obtain samples for prediction. After motion compensation of the extracted samples according to the symbols 521 associated with the block, these samples can be added by the aggregator 555 to the output of the scaler / inverse transform unit 551 (referred to as residual samples or residual signals in this case) to generate output sample information. The addresses in the reference image memory 557 from which the motion compensation prediction unit 553 obtains the predicted samples can be controlled by motion vectors, and the motion compensation prediction unit 553 can obtain these addresses in the form of symbols 521, which may have, for example, X, Y, and reference image components. Motion compensation may also include interpolation of sampled values ​​obtained from the reference image memory 557 when using subsampled precise motion vectors, and may also include motion vector prediction mechanisms, etc.

[0081] The output samples of aggregator 555 can be processed by various loop filtering techniques in loop filter unit 556. Video compression techniques may include loop filtering techniques controlled by parameters contained in the encoded video sequence (also known as the encoded video bitstream), which are provided to loop filter unit 556 as symbols 521 from parser 520, or in response to metadata obtained during decoding of a previous (in the order of decoding) portion of the encoded image or encoded video sequence, and in response to previously reconstructed and loop-filtered sample values.

[0082] The output of the loop filter unit 556 can be a sample stream, which can be output to a rendering device and stored in a reference image memory 557 for future inter-frame image prediction.

[0083] Once fully reconstructed, certain encoded images can be used as reference images for future predictions. For example, once the encoded image corresponding to the current image has been fully reconstructed and that encoded image has been identified as a reference image (e.g., by parser 520), the current image buffer 558 can become part of the reference image memory 557, and a new current image buffer can be reallocated before the reconstruction of the next encoded image begins.

[0084] The video decoder 510 can perform decoding operations according to predetermined video compression techniques in standards such as ITU-T Rec.H.265. The encoded video sequence can conform to the syntax specified by the video compression technique or standard used, in the sense that the encoded video sequence conforms to the syntax of the video compression technique or standard and the brief documented therein. Specifically, the brief may select certain tools from all available tools in the video compression technique or standard, representing all tools available under that brief. Standard conformance also requires that the complexity of the encoded video sequence be within the range defined by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sampling rate (e.g., measured in megasamples per second), maximum reference picture size, etc. In some cases, the limitations set by the level can be further restricted by the hypothetical reference decoder (HRD) specification and metadata managed by the HRD buffer of signaling in the encoded video sequence.

[0085] Figure 6 A block diagram of a video encoder 603 according to an embodiment of the present disclosure is shown. The video encoder 603 can be used in a V-PCC encoder 300 for compressing point clouds. In one example, video compression modules 322 and 323, as well as video compression module 332, can be configured in a manner similar to encoder 603.

[0086] The video encoder 603 can receive images, such as filled geometric images, filled texture images, etc., and generate compressed images.

[0087] According to one embodiment, the video encoder 603 can encode and compress images of a source video sequence (images) into an encoded video sequence (compressed images) in real time or under any other time constraints required by the application. Implementing an appropriate encoding rate is a function of the controller 650. In some embodiments, the controller 650 controls and is functionally coupled to other functional units as described below. For clarity, the above coupling is not specifically described. Parameters set by the controller 650 may include rate control related parameters (image skipping, quantizer, λ value of rate-distortion optimization techniques, etc.), image size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller 650 can be configured to have other suitable functions related to the video encoder 603 optimized for a particular system design.

[0088] In some embodiments, the video encoder 603 is configured to operate within an encoding loop. As a simplified description, in one example, the encoding loop may include a source encoder 630 (e.g., responsible for creating symbols, such as a symbol stream, based on the input image to be encoded and a reference image) and a (local) decoder 633 embedded in the video encoder 603. The decoder 633 reconstructs the symbols in a manner that creates sample data (since any compression between the symbols and the encoded video bitstream is lossless in the video compression techniques considered in the disclosed subject matter), and the (remote) decoder also creates sample data in a similar manner. The reconstructed sample stream (sample data) is input to a reference image memory 634. Because the decoding of the symbol stream results in a bit-precise result independent of the decoder location (local or remote), the contents of the reference image memory 634 are also bit-precise between the local encoder and the remote encoder. In other words, when prediction is used during decoding, the prediction portion of the encoder "sees" sample values ​​that are exactly the same as the sample values ​​"see" by the decoder, as reference image samples. The basic principle of reference image synchronization (and the resulting drift when synchronization cannot be maintained, for example, due to channel errors) is also used in some related technologies.

[0089] The operation of the "local" decoder 633 can be the same as that of a "remote" decoder such as the video decoder 510, as already described above. Figure 5 The operation of the "remote" decoder is described in detail. However, a brief reference is provided. Figure 5 Since symbols are available, and the encoding / decoding of symbols in the encoded video sequence by the entropy encoder 645 and the parser 520 can be lossless, the entropy decoding part of the video decoder 510, including the parser 520, can be partially implemented in the local decoder 633.

[0090] During operation, in some examples, the source encoder 630 may perform motion-compensated predictive coding, which predictively encodes the input image by referencing one or more previously encoded images from the video sequence designated as "reference images." In this way, the encoding engine 632 encodes the differences between pixel blocks of the input image and pixel blocks of the reference image, which can be selected as a predictive reference for the input image.

[0091] The local video decoder 633 can decode encoded video data of a picture that may be designated as a reference picture, based on symbols created by the source encoder 630. The operation of the encoding engine 632 can advantageously be a lossy process. When the encoded video data can be decoded by the video decoder (… Figure 6 When decoded at (not shown), the reconstructed video sequence can typically be a copy of the source video sequence with some errors. The local video decoder 633 replicates the decoding process performed on the reference image by the video decoder, and the reconstructed reference image can be stored in the reference image cache 634. In this way, the video encoder 603 can locally store copies of the reconstructed reference images that have the same content as the reconstructed reference images that will be obtained by the remote video decoder (without transmission errors).

[0092] Predictor 635 can perform a prediction search on encoding engine 632. That is, for a new image to be encoded, predictor 635 can search the reference image memory 634 for sample data (as candidate reference pixel blocks) or certain metadata (e.g., reference image motion vectors, block shapes, etc., which can be used as appropriate prediction references for the new image). Predictor 635 can operate on a sample block-by-pixel basis to find suitable prediction references. In some cases, as determined by the search results obtained by predictor 635, the input image may have prediction references extracted from multiple reference images stored in reference image memory 634.

[0093] The controller 650 can manage the encoding operations of the source encoder 630, including, for example, the settings of parameters and subgroup parameters for encoding video data.

[0094] The outputs of all the aforementioned functional units can undergo entropy encoding 645 in the entropy encoder. The entropy encoder 645 converts the symbols generated by the various functional units into an encoded video sequence by losslessly compressing the symbols according to techniques such as Huffman coding, variable-length coding, and arithmetic coding.

[0095] The controller 650 manages the operation of the video encoder 603. During encoding, the controller 650 can assign a specific encoded picture type to each encoded picture, which can affect the encoding technique used for the corresponding picture. For example, a picture can typically be designated as one of the following picture types:

[0096] An intra-frame picture (I-picture) can be a picture that is encoded and decoded without using any other pictures in the sequence as a prediction source. Some video codecs allow different types of intra-frame pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art are aware of all variations of I-pictures and their corresponding applications and characteristics.

[0097] A predicted image (P-image) can be an image that uses at most one motion vector and a reference index to predict the sample value of each block, and is encoded and decoded using intra-frame prediction or inter-frame prediction.

[0098] A bidirectional prediction image (B-image) can be an image that uses up to two motion vectors and a reference index to predict sample values ​​for each block, and is encoded and decoded using intra-frame prediction or inter-frame prediction. Similarly, a multi-prediction image can reconstruct a single block using two or more reference images and associated metadata.

[0099] Source images can typically be spatially subdivided into multiple sample blocks (e.g., each sample block includes 4×4, 8×8, 4×8, or 16×16 samples) and encoded on a block-by-block basis. Blocks can be predictively encoded by referencing other (already encoded) blocks, where these other (already encoded) blocks can be determined by the coding assignment applied to the corresponding images of the block. For example, blocks of image I can be nonpredictively encoded, or they can be predictively encoded (spatial prediction or intra-frame prediction) by referencing already encoded blocks of the same image. Pixel blocks of image P can be predictively encoded by referencing a previously encoded reference image, either spatially or temporally. Blocks of image B can be predictively encoded by referencing one or two previously encoded reference images, either spatially or temporally.

[0100] The video encoder 603 can perform encoding operations according to a predetermined video coding technique or standard (e.g., ITU-T REC.H.265). In its operation, the video encoder 603 can perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Therefore, the encoded video data can conform to the syntax specified by the video coding technique or standard being used.

[0101] Video can take the form of multiple source pictures (images) in a time series. Intra-frame picture prediction (often abbreviated as intra-prediction) utilizes spatial correlations within a given picture, while inter-frame picture prediction utilizes (temporal or other) correlations between pictures. In one example, a specific picture in the encoding / decoding process, referred to as the current picture, is segmented into blocks. When a block in the current picture resembles a reference block in a previously encoded and still buffered reference picture in the video, the block in the current picture can be encoded using a vector called a motion vector. In the case of using multiple reference pictures, the motion vector points to the reference block in the reference picture and can have a third dimension for identifying the reference picture.

[0102] In some embodiments, bidirectional prediction techniques can be used in inter-frame image prediction. According to bidirectional prediction, two reference images are used, for example, a first reference image and a second reference image, both of which precede the current image in the video in the decoding order (but can be displayed in the past and future, respectively). A block in the current image can be encoded by a first motion vector pointing to a first reference block in the first reference image and a second motion vector pointing to a second reference block in the second reference image. The block can be predicted using a combination of the first and second reference blocks.

[0103] In addition, merging mode techniques can be used in inter-frame image prediction to improve coding efficiency.

[0104] According to some embodiments of this disclosure, prediction is performed on a block-by-block basis, such as inter-frame picture prediction and intra-frame picture prediction. For example, according to the HEVC standard, pictures in a video picture sequence are segmented into Coding Tree Units (CTUs) for compression. The CTUs in the pictures have the same size, for example, 64×64 pixels, 32×32 pixels, or 16×16 pixels. Typically, a CTU comprises three Coding Tree Blocks (CTBs): one luma CTB and two chroma CTBs. Each CTU can be recursively quadtree-segmented into one or more Coding Units (CUs). For example, a 64×64 pixel CTU can be segmented into one 64×64 pixel CU, or four 32×32 pixel CUs, or sixteen 16×16 pixel CUs. In one example, each CU is analyzed to determine its prediction type, such as inter-frame prediction or intra-frame prediction. Based on temporal and / or spatial predictability, CUs are divided into one or more Prediction Units (PUs). Typically, each PU includes one luma prediction block (PB) and two chroma PBs. In embodiments, prediction operations in encoding / decoding are performed on a per-prediction-block basis. Using a luma prediction block as an example, a prediction block contains a matrix of pixel values ​​(e.g., luma values), such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0105] Figure 7 Block diagrams of a G-PCC encoder 700 in some examples are shown. The G-PCC encoder 700 can be configured to receive point cloud data and compress the point cloud data to generate a bitstream carrying the compressed point cloud data. In one embodiment, the G-PCC encoder 700 may include a position quantization module 710, a duplicate point removal module 712, an octree encoding module 730, an attribute transfer module 720, a Level of Detail (LOD) generation module 740, an attribute prediction module 750, a residual quantization module 760, an arithmetic encoding module 770, an inverse residual quantization module 780, an addition module 781, and a memory 790 for storing the reconstructed attribute values.

[0106] As shown in the figure, an input point cloud 701 can be received at the G-PCC encoder 700. The position (e.g., 3D coordinates) of the point cloud 701 is provided to the quantization module 710. The quantization module 710 is configured to quantize the coordinates to generate quantized positions. A duplicate point removal module 712 is configured to receive the quantized positions and perform filtering processing to identify and remove duplicate points. An octree encoding module 730 is configured to receive the filtered positions from the duplicate point removal module 712 and perform octree-based encoding processing to generate an octet code sequence for describing a 3D raster of voxels. The octet code is provided to the arithmetic encoding module 770.

[0107] The attribute transfer module 720 is configured to receive attributes of the input point cloud and perform attribute transfer processing to determine the attribute value of each voxel when multiple attribute values ​​are associated with corresponding voxels. Attribute transfer processing can be performed on the reordered points output from the octree encoding module 730. The attributes after the transfer operation are provided to the attribute prediction module 750. The LOD generation module 740 is configured to operate on the reordered points output from the octree encoding module 730 and reorganize these points into different LODs. The LOD information is provided to the attribute prediction module 750.

[0108] The attribute prediction module 750 processes points based on LOD-based sequential processing points indicated by LOD information from the LOD generation module 740. The attribute prediction module 750 generates an attribute prediction for the current point based on the reconstructed attributes of a set of neighboring points stored in the memory 790. The prediction residual can then be obtained based on the original attribute values ​​received from the attribute transfer module 720 and the locally generated attribute prediction. When candidate indices are used in the corresponding attribute prediction processing, the index corresponding to the selected prediction candidate can be provided to the arithmetic encoding module 770.

[0109] The residual quantization module 760 is configured to receive the prediction residuals from the attribute prediction module 750 and perform quantization to generate quantized residuals. The quantized residuals are then provided to the arithmetic coding module 770.

[0110] The inverse residual quantization module 780 is configured to receive the quantized residual from the residual quantization module 760 and generate the reconstructed prediction residual by performing the inverse operation of the quantization operation performed at the residual quantization module 760. The addition module 781 is configured to receive the reconstructed prediction residual from the inverse residual quantization module 780 and receive the corresponding attribute prediction from the attribute prediction module 750. By combining the reconstructed prediction residual and the attribute prediction, a reconstructed attribute value is generated and stored in memory 790.

[0111] Arithmetic coding module 770 is configured to receive occupancy codes, candidate indices (if used), quantization residuals (if generated), and other information, and perform entropy coding to further compress the received values ​​or information. As a result, a compressed bitstream 702 carrying the compressed information can be generated. Bitstream 702 can be transmitted or otherwise provided to a decoder that decodes the compressed bitstream, or it can be stored in a storage device.

[0112] Figure 8 A block diagram of a G-PCC decoder 800 according to one embodiment is shown. The G-PCC decoder 800 can be configured to receive a compressed bitstream and perform point cloud data decompression to decompress the bitstream to generate decoded point cloud data. In one embodiment, the G-PCC decoder 800 may include an arithmetic decoding module 810, an inverse residual quantization module 820, an octree decoding module 830, an LOD generation module 840, an attribute prediction module 850, and a memory 860 for storing reconstructed attribute values.

[0113] As shown in the figure, the compressed bitstream 801 can be received at the arithmetic decoding module 810. The arithmetic decoding module 810 is configured to decode the compressed bitstream 801 to obtain the quantization residual (if generated) and occupancy code of the point cloud. The octree decoding module 830 is configured to determine the reconstruction position of the points in the point cloud based on the occupancy code. The LOD generation module 840 is configured to reorganize the points into different LODs based on the reconstruction positions and determine the LOD-based order. The inverse residual quantization module 820 is configured to generate the reconstruction residual based on the quantization residual received from the arithmetic decoding module 810.

[0114] The attribute prediction module 850 is configured to perform attribute prediction processing to determine the attribute predictions of points based on LOD-based order. For example, the attribute prediction 860 of the current point can be determined based on the reconstructed attribute values ​​of the current point's neighboring points stored in memory. In some examples, the attribute predictions can be combined with the corresponding reconstruction residuals to generate the reconstructed attributes of the current point.

[0115] In one example, the reconstructed attribute sequence generated from the attribute prediction module 850 and the reconstructed positions generated from the octree decoding module 830 correspond to the decoded point cloud 802 output from the G-PCC decoder 800. Furthermore, the reconstructed attributes are also stored in memory 860 and can be subsequently used to derive attribute predictions for subsequent points.

[0116] In various embodiments, encoder 300, decoder 400, encoder 700, and / or decoder 800 can be implemented in hardware, software, or a combination thereof. For example, encoder 300, decoder 400, encoder 700, and / or decoder 800 can be implemented using processing circuitry (e.g., one or more integrated circuits (ICs)) that operate with or without software, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), etc. In another example, encoder 300, decoder 400, encoder 700, and / or decoder 800 can be implemented as software or firmware, comprising instructions stored in a non-volatile (or non-transitory) computer-readable storage medium. When executed by processing circuitry (e.g., one or more processors), these instructions cause the processing circuitry to perform the functions of encoder 300, decoder 400, encoder 700, and / or decoder 800.

[0117] It should be noted that the attribute prediction modules 750 and 850, configured to implement the attribute prediction techniques disclosed herein, may be included in other decoders or encoders, which may have the same characteristics as... Figure 7 and Figure 8 Similar or different structures are shown. Furthermore, in various examples, the encoder 700 and decoder 800 may be included in the same device or in separate devices.

[0118] According to some aspects of this disclosure, mesh compression can use encoding tools different from PCC encoding tools, or it can use PCC encoding tools, such as the PCC (e.g., G-PCC, V-PCC) encoders and decoders described above.

[0119] An object's mesh (also called a mesh model or mesh frame) can include polygons describing the object's surface. Each polygon can be defined by its vertices in 3D space and information about how those vertices connect to the polygon. This information about how vertices connect is called connectivity information. In some examples, the mesh can also include properties associated with the vertices, such as color, normals, etc. Properties can be associated with the mesh's surface by utilizing the mapping information of the mesh, which is parameterized using a 2D property graph. This mapping information is typically described by a set of parametric coordinates associated with the mesh vertices, called UV coordinates or texture coordinates. The 2D property graph (called a texture graph in some examples) is used to store high-resolution property information, such as texture, normals, displacements, etc. This information can be used for various purposes, such as texture mapping and shading.

[0120] In some embodiments, the mesh may include components referred to as geometric information, connectivity information, mapping information, vertex attributes, and a property graph. In some examples, geometric information is described by a set of 3D positions associated with mesh vertices. In one example, (x, y, z) coordinates may be used to describe the 3D position of a vertex. In some examples, connectivity information includes a set of vertex indices describing how vertices are connected to create a 3D surface. In some examples, mapping information describes how the mesh surface is mapped to a 2D region of a plane. In one example, mapping information is described by a set of UV parameter / texture coordinates (u, v) associated with mesh vertices, along with connectivity information. In some examples, vertex attributes include scalar or vector attribute values ​​associated with mesh vertices. In some examples, the property graph includes attributes associated with the mesh surface and is stored as a 2D image / video. In one example, the mapping between the video (e.g., a 2D image / video) and the mesh surface is defined by the mapping information.

[0121] According to one aspect of this disclosure, techniques known as UV mapping or mesh parameterization are used to map a mesh surface in a 3D domain to a 2D domain. In some examples, the mesh is segmented into patches in the 3D domain. A patch is a contiguous subset of the mesh. Then, in some examples, the patches are parameterized individually as 2D shapes (also known as UV patches). In some examples, the 2D shapes can be packed (e.g., oriented and placed) into a graph, also known as a map. In some examples, 2D image or video processing techniques can be used to further process the graph.

[0122] In one example, UV mapping generates a UV map (also called a UV map) and one or more texture maps (also called texture maps) in 2D corresponding to a 3D mesh patch. The UV map consists of assigning 3D vertices of the 3D mesh to 2D points within a 2D domain (e.g., a rectangle). The UV map is a mapping between coordinates on a 3D surface and coordinates in a 2D domain. In one example, a point at 2D coordinates (u, v) in the UV map has a value formed by the coordinates (x, y, z) of the vertices in the 3D domain. In one example, the texture map includes color information for the 3D mesh. For example, a point at 2D coordinates (u, v) in the texture map (having a 3D value of (x, y, z) in the UV map) has a color that specifies the color attribute of the point at (x, y, z) in the 3D domain.

[0123] According to one aspect of this disclosure, a dynamic mesh is a mesh in which at least one component (geometric information, connectivity information, mapping information, vertex attributes, and attribute graph) changes over time. A dynamic mesh can be described by a series of meshes (also called mesh frames). Dynamic meshes may require large amounts of data because they may include a large amount of information that changes over time. Mesh compression techniques can allow for the efficient storage and transmission of media content within a mesh representation.

[0124] In some examples, dynamic meshes can possess constant connectivity information, time-varying geometry, and time-varying vertex properties. In other examples, dynamic meshes can have time-varying connectivity information. In one example, digital content creation tools typically generate dynamic meshes with time-varying property graphs and time-varying connectivity information. In some examples, volumetric acquisition techniques are used to generate dynamic meshes. Volumetric acquisition techniques can generate dynamic meshes with time-varying connectivity information, especially under real-time constraints.

[0125] According to some aspects of this disclosure, mesh compression can be performed by representing a mesh using one or more 2D graphs (also referred to as 2D atlases in some examples) and then encoding the 2D graphs using an image or video codec.

[0126] Figure 9 A mapping from 3D mesh 910 to 2D map 920 is shown in some examples. Figure 9 In the example, the 3D mesh 910 includes four vertices 1-4 forming four patches AD. Each patch has a set of vertices and associated attribute information. For example, patch A is formed by vertices 1, 2, and 3 connected to form a triangle; patch B is formed by vertices 1, 3, and 4 connected to form a triangle; patch C is formed by vertices 1, 2, and 4 connected to form a triangle; and patch D is formed by vertices 2, 3, and 4 connected to form a triangle. In some examples, vertices 1, 2, 3, and 4 may have corresponding attributes, and the triangle formed by vertices 1, 2, 3, and 4 may have corresponding attributes.

[0127] In one example, patches A, B, C, and D in 3D are mapped to a 2D domain, such as 2D atlas 920, also known as Figure 920. For example, patch A is mapped to a 2D shape (also known as a UV patch) A' in Figure 920, patch B is mapped to a 2D shape (also known as a UV patch) B' in Figure 920, patch C is mapped to a 2D shape (also known as a UV patch) C' in Figure 920, and patch D is mapped to a 2D shape (also known as a UV patch) D' in Figure 920. In some examples, the coordinates in the 3D domain are referred to as (x, y, z) coordinates, and the coordinates in the 2D domain (e.g., Figure 920) are referred to as UV coordinates. Vertices in the 3D mesh can have corresponding UV coordinates in Figure 920.

[0128] Figure 920 can be a geometric graph with geometric information, or a texture graph with color, fabric or other attribute information, or an occupancy graph with occupancy information.

[0129] In one example, the geometric information of a vertex can be stored in a 2D geometry graph. For instance, the geometry graph stores the (x, y, z) coordinates of a sample point at the corresponding pixel. The pixel at position (u, v) in the geometry graph has a three-component vector value, corresponding to the x, y, and z values ​​of the corresponding sample point in the 3D mesh.

[0130] According to one aspect of this disclosure, the area in the drawing may not be fully occupied. For example, in Figure 9 In the occupancy map, the regions outside the 2D shapes A', B', C', and D' are undefined. After decoding, sample values ​​from these regions can be discarded. In some cases, the occupancy map is used to store additional information for each pixel, such as storing binary values ​​to identify whether a pixel belongs to a patch or is undefined.

[0131] According to some aspects of this disclosure, in order to effectively compress 3D mesh information, 2D images (e.g., geometric images, texture images, occupancy images, etc.) can be downsampled before encoding.

[0132] Figure 10 The following are examples of downsampling plots. Figure 10 In this example, Figure 1020 is downsampled by a factor of 2 in both the horizontal and vertical directions, and a downsampled figure 1030 is generated accordingly. The width (e.g., the number of pixels in the horizontal direction) of the downsampled figure 1030 is half the width (e.g., the number of pixels in the horizontal direction) of Figure 1020, and the height (e.g., the number of pixels in the vertical direction) of the downsampled figure 1030 is half the height (e.g., the number of pixels in the vertical direction) of Figure 1020.

[0133] exist Figure 10 In Figure 1020, 2D shapes (also known as UV patches) A', B', C', and D' are included, and downsampled Figure 1030 includes sampled 2D shapes A”, B”, C”, and D” corresponding to 2D shapes A', B', C', and D', respectively. In some examples, downsampled Figure 1030 is then encoded by an image or video encoder on the mesh encoder side.

[0134] In some examples, the downsampled image is decoded on the mesh decoder side. After decoding, the downsampled image is restored to its original resolution (e.g., the original number of pixels in the vertical direction and the original number of pixels in the horizontal direction) for reconstructing the 3D mesh.

[0135] Dynamic mesh sequences typically require large amounts of data because they may contain a significant amount of information that changes over time. Sampling steps applied to 2D graphs (e.g., geometry graphs, property graphs) can help reduce the bandwidth required to represent mesh information. However, sampling steps can also remove critical information during downsampling, such as certain key geometries of a 3D mesh.

[0136] This disclosure provides adaptive sampling techniques for processing 2D maps (also referred to as graphs in 2D) without losing much important information. Adaptive sampling techniques can be used for both static mesh compression (where a single mesh frame or mesh content does not change over time) and dynamic mesh compression. Various adaptive sampling techniques can be applied individually or in arbitrary combinations. In the following description, adaptive sampling methods are applied to 2D maps (e.g., 2D graphs), which can be geometric or attribute (texture) graphs, or both.

[0137] Figure 11 A schematic diagram of a frame 1100 for mesh compression according to some embodiments of the present disclosure is shown. The frame 1100 includes a mesh encoder 1110 and a mesh decoder 1150. The mesh encoder 1110 encodes an input mesh 1105 (a mesh frame in the case of dynamic mesh processing) into a bitstream 1145, and the mesh decoder 1150 decodes the bitstream 1145 to generate a reconstructed mesh 1195 (a reconstructed mesh frame in the case of dynamic mesh processing).

[0138] The mesh encoder 1110 can be any suitable device, such as a computer, server computer, desktop computer, laptop computer, tablet computer, smartphone, gaming device, AR device, VR device, etc. The mesh decoder 1150 can be any suitable device, such as a computer, client computer, desktop computer, laptop computer, tablet computer, smartphone, gaming device, AR device, VR device, etc. The bitstream 1145 can be transmitted from the mesh encoder 1110 to the mesh decoder 1150 via any suitable communication network (not shown).

[0139] exist Figure 11In the example, the mesh encoder 1110 includes an adaptive sampling module 1120, a video encoder 1130, and an auxiliary data encoder 1140 coupled together. The video encoder 1130 is configured to encode image or video data, such as a 2D graph in a 3D mesh representation. The adaptive sampling module 1120 receives an input mesh 1105 and performs adaptive sampling to generate an adaptively sampled graph 1125. In some examples, the input mesh 1105 includes a 2D graph representing a 3D mesh. The adaptive sampling module 1120 can detect the information density in the graph and determine different sampling rates for sampling the graph based on the information density. The 2D graph can then be sampled according to the different sampling rates to generate the adaptively sampled graph 1125.

[0140] The video encoder 1130 can use image encoding and / or video encoding techniques to encode the adaptive sampled graph 1125 into a bitstream 1145.

[0141] The adaptive sampling module 1120 also generates auxiliary data 1127, which indicates auxiliary information for adaptive sampling. The auxiliary data encoder 1140 receives the auxiliary data 1127 and encodes it into a bitstream 1145.

[0142] The operation of the adaptive sampling module 1120 and the auxiliary data encoder 1140 will be further described in this disclosure.

[0143] exist Figure 11 In the example, bitstream 1145 is provided to grid decoder 1150. Grid decoder 1150 includes, for example... Figure 11 The video decoder 1160, auxiliary data decoder 1170, recovery module 1180, and mesh reconstruction module 1190 are coupled together. In one example, the video decoder 1160 corresponds to the video encoder 1130 and can decode a portion of the bitstream 1145 encoded by the video encoder 1130 and generate decoded information 1165 (e.g., an initial decoded graph).

[0144] exist Figure 11 In the example, the auxiliary data decoder 1170 corresponds to the auxiliary data encoder 1140, which can decode a portion of the bitstream 1145 encoded by the auxiliary data encoder 1140 and generate decoded auxiliary data 1175.

[0145] exist Figure 11In the example, decoded information 1165 and decoded auxiliary data 1175 are provided to the recovery module 1180. The recovery module 1180 can form a recovery map 1185 of the original resolution (e.g., the same number of pixels in the vertical and horizontal directions) as a map in the input grid 1105. The grid reconstruction module 1190 generates a reconstructed grid 1195 based on the recovery map 1185. The operation of the auxiliary data decoder 1170 and the recovery module 1180 will be further described in this disclosure.

[0146] It should be noted that the components in the mesh encoder 1110 (e.g., segmentation module 1120, video encoder 1130, and auxiliary data encoder 1140) can be implemented using various technologies. In one example, the components are implemented using integrated circuits. In another example, the components are implemented using software that can be executed by one or more processors.

[0147] It should be noted that the components in the mesh decoder 1150 (e.g., video decoder 1160, auxiliary data decoder 1170, recovery module 1180, and mesh reconstruction module 1190) can be implemented using various techniques. In one example, the components are implemented using integrated circuits. In another example, the components are implemented using software that can be executed by one or more processors.

[0148] In some embodiments, adaptive sampling can be based on graph type. In some examples, the adaptive sampling module 1120 can apply different sampling rates to different types of graphs. For example, different sampling rates can be applied to geometry graphs and attribute graphs. In one example, a mesh is a model of an object with a regular shape and rich texture. For example, the object has a rectangular shape but rich colors. Therefore, the information density of the geometry graph is relatively low. In one example, the adaptive sampling module 1120 applies a first sampling rate of 2:1 on the geometry graph (in both the vertical and horizontal directions) and a second sampling rate of 1:1 on the texture graph (in both the vertical and horizontal directions).

[0149] In some examples, a sampling rate of A:B in one direction means generating B samples from A pixels in the original image in that direction. For example, a 2:1 sampling rate in the horizontal direction means generating one sample for every two pixels in the original image in the horizontal direction. A 2:1 sampling rate in the vertical direction means generating one sample for every two pixels in the original image in the vertical direction. In some examples, sampling steps are used to indicate the sampling rate. A sampling step in one direction means the number of pixels used as steps to generate a sample. For example, a 2 sampling step in the horizontal direction means generating one sample for every two pixels in the horizontal direction; a 2 sampling step in the vertical direction means generating one sample for every two pixels in the vertical direction.

[0150] In some embodiments, sampling adaptation is based on sub-regions in the graph. Different sampling rates can be applied to different parts of the graph. In some examples, some pixel rows have less information to retain, and a larger sampling step can then be applied along these rows, resulting in fewer sample rows being encoded. In some examples, some pixel columns have less information to retain, and a larger sampling step can then be applied along these columns, resulting in fewer sample columns being encoded. For other regions, a smaller sampling step is applied to minimize information loss after sampling.

[0151] Figure 12 The adaptive sampling map is shown in some examples. Figure 1220 is divided into several block rows, each block row comprising a fixed number of sample (pixel) rows. Different sampling rates are applied to the block rows in the vertical direction to generate the adaptive sampling map 1230. For example, each block row is a CTU row (also called a CTU line) and includes 64 rows of samples (also called pixels). Figure 12 In the example, for block rows 0 and 6 in Figure 1220, a first sampling rate of 2:1 is applied in the vertical direction, and after sampling, 32 rows of samples are generated for each of block rows 0 and 6 in adaptive sampling Figure 1230. For block rows 1 to 5 in Figure 1220, a second sampling rate of 1:1 is applied in the vertical direction, generating 64 rows of samples for each of block rows 1 to 5 in adaptive sampling Figure 1230.

[0152] In some examples, the adaptive sampled image 1230 is then encoded by an image or video encoder (e.g., video encoder 1130). On the decoder side, in one example, the adaptive sampled image 1230 is decoded. After decoding, the first 32 rows of samples are restored (upsampled) to the original resolution, e.g., 64 rows of samples; for example, the bottom 32 rows of samples are restored (upsampled) to the original resolution, e.g., 64 rows of samples, by the restoration module 1180.

[0153] In other examples, the image to be encoded in the 2D representation of the 3D mesh can be divided into multiple sub-regions. Examples of such division within an image (e.g., a picture) include slices, tiles, tile groups, coding tree units, etc. In some examples, different sampling rates can be applied to different sub-regions. In one example, different sampling rates associated with different sub-regions can be signaled in the bitstream carrying the 3D mesh. On the decoder side, after decoding the adaptively sampled image, for example, each sub-region is restored to its original resolution by the recovery module 1180 according to the sampling rate associated with the sub-region.

[0154] In some examples, the process of restoring the adaptively sampled map to its original resolution is referred to as inverse sampling, which generates the restored map. After restoration from the inverse sampling, the output of the restored map in 2D atlas form can be used for 3D mesh reconstruction.

[0155] Although Figure 12 The example in the text shows adaptive sampling of different block rows in the vertical direction, but similar adaptive sampling can be applied to different columns in the horizontal direction, or to both the vertical and horizontal directions.

[0156] In some embodiments, sampling adaptation is based on patches. In some examples, different patches in the graph may have different sampling rates.

[0157] Figure 13 Adaptive sampling maps are shown in some examples. Figure 1320 (e.g., a high-resolution 2D map) includes multiple 2D shapes, also referred to as UV patches corresponding to patches in a 3D mesh, e.g., first 2D shape A' and second 2D shape B'. Figure 13 In the example, a first sampling rate of 2:1 is applied to the first 2D shape A' in the vertical and horizontal directions to generate a first sampled 2D shape A”; and a second sampling rate of 1:1 is applied to the second 2D shape B' in the vertical and horizontal directions to generate a second sampled 2D shape B”. The first sampled 2D shape A” and the second sampled 2D shape B” are placed in a new graph called adaptive sampling graph 1330.

[0158] exist Figure 13 In the example, the first sampled 2D shape A” is smaller than the first 2D shape A', and the second sampled 2D shape B” is the same size as the second 2D shape B'. The adaptive sampling shape 1330 is encoded into a bitstream carrying a 3D mesh by an image or video encoder (e.g., video encoder 1130). In some examples, for example, the sampling rate associated with the sampled 2D shape is encoded into the bitstream carrying the 3D mesh by an auxiliary data encoder 1140.

[0159] In some examples, on the decoder side, an image / video decoder (e.g., video decoder 1160) decodes an initial map from the bitstream, such as an adaptive sampling map 1330. Additionally, for example, an auxiliary data decoder 1170 decodes the sampling rate associated with the sampled 2D shape from the bitstream. Based on the sampling rate associated with the sampled 2D shape, the 2D shape sampled in the adaptive sampling map 1330 is restored to its original size (e.g., the same number of pixels in the vertical and horizontal directions) by the restoration module 1180 to generate a restored map. The restored map is then used for 3D mesh reconstruction.

[0160] According to one aspect of this disclosure, adaptive sampling information (e.g., sampling rates for different map types, sampling rates for different sub-regions, sampling rates for different patches, etc.) is known on both the mesh encoder and mesh decoder sides. In some examples, the adaptive sampling information is appropriately encoded into the bitstream carrying the 3D mesh. Therefore, the mesh decoder and mesh encoder can operate based on the same adaptive sampling information. The mesh decoder can restore the map to the correct size.

[0161] In some examples, both the grid encoder and grid decoder use the default sampling rate. In other examples, the default sampling rate is signaled at the frame level or sequence level. In one example, for graph types, a flag is used to indicate whether the graph type's sampling rate differs from the default sampling rate. If the graph type's sampling rate differs from the default sampling rate, the new sampling rate (or the difference between the new and default sampling rates) is signaled.

[0162] In another example, for sub-regions of the graph (e.g., block rows, block columns, CTU lines, slices, tiles, tile groups, CTUs, etc.), a flag is used to indicate whether the sampling rate of the sub-region differs from the default sampling rate. If the sampling rate of the sub-region differs from the default sampling rate, the new sampling rate (or the difference between the new and default sampling rates) is signaled.

[0163] In another example, for patches in the graph (e.g., UV patches, 2D shapes), a flag is used to indicate whether the patch's sampling rate differs from the default sampling rate. If the patch's sampling rate differs from the default sampling rate, the new sampling rate (or the difference between the new and default sampling rates) is signaled.

[0164] In some examples, a set of sampling rates used in a frame or sequence is signaled in the bitstream or agreed upon by both the grid encoder and grid decoder. Examples of such sampling rates include 2:1, 4:1, 8:1, etc. When selecting a specific sampling rate for a map type, sub-region, or patch, the selection can be indicated using the index of the sampling rate in the group, rather than the signaling sampling rate itself.

[0165] In some examples, patches in different grid frames may not be in the same location. Over time, a patch in one grid frame may have a corresponding patch in another grid frame (e.g., a pair of patches covering the same or similar portions of the grid). The sampling rate applied to a patch in a previous grid frame (e.g., applied to a corresponding UV patch in a graph representing a previous grid frame, applied to a corresponding 2D shape in a graph representing a previous grid frame) can be applied to the corresponding patch in the current grid frame (e.g., applied to a corresponding UV patch in a graph representing a current grid frame, applied to a corresponding 2D shape in a graph representing a current grid frame). In some examples, the grid frame sequence may be a group of pictures (GOP). In one example, a signaling sampling rate is used for patches, sub-regions, or CTU lines in a key grid frame (e.g., the first frame) of a GOP. For patches, regions, or lines in other grid frames within a GOP, their sampling rates can be referenced to the corresponding sampling rate in the key grid frame.

[0166] Figure 14 A flowchart outlining a process 1400 according to an embodiment of the present disclosure is shown. This process 1400 can be used in the encoding process of a mesh. In various embodiments, process 1400 is executed by processing circuitry. In some embodiments, process 1400 is implemented in software instructions, so that when the processing circuitry executes the software instructions, the processing circuitry executes process 1400. The process begins at S1401 and proceeds to S1410.

[0167] In S1410, at least two sampling rates are determined to be associated with different portions of the multiple 2D graphs. The multiple graphs are represented in a three-dimensional (3D) mesh frame. In one example, the adaptive sampling module 1120 determines at least two sampling rates associated with different portions of the multiple 2D graphs.

[0168] In S1420, multiple graphs are sampled based on at least two sampling rates to generate multiple sampled graphs. In one example, the adaptive sampling module 1120 applies at least two sampling rates to different parts of the multiple graphs to generate multiple sampled graphs. For example, a first part of the multiple graphs is downsampled based on a first sampling rate, and a second part of the multiple graphs is downsampled based on a second sampling rate.

[0169] In S1430, multiple sampled images are encoded into a bitstream. In one example, video encoder 1130 encodes multiple sampled images into a bitstream 1145.

[0170] To determine at least two sampling rates, in some examples, a first sampling rate associated with a first graph of a first graph type is determined, and a second sampling rate associated with a second graph of a second graph type is determined. Multiple graphs include a first graph and a second graph. For example, the first graph is a geometry graph, and the second graph is a texture graph.

[0171] To determine at least two sampling rates, in some examples, a first sampling rate is determined associated with a first sub-region in the first figure, and a second sampling rate is determined associated with a second sub-region in the first figure. In one example, each of the first and second sub-regions includes multiple rows of pixels (e.g., block rows). In another example, each of the first and second sub-regions includes multiple columns of pixels. In another example, each of the first and second sub-regions includes a coding tree unit (CTU) line. In another example, each of the first and second sub-regions includes a slice. In another example, each of the first and second sub-regions includes a tile. In another example, each of the first and second sub-regions includes a group of tiles. In another example, each of the first and second sub-regions includes a CTU.

[0172] In order to determine at least two sampling rates, in some examples, a first sampling rate is determined to be associated with a first patch (e.g., a first UV patch, a first 2D shape) in the first figure, and a second sampling rate is determined to be associated with a second patch (e.g., a second UV patch, a second 2D shape) in the first figure.

[0173] In some examples, a first flag is determined based on a comparison between a first sampling rate associated with a first portion of multiple sampled maps and a default sampling rate. For example, the first flag associated with the first portion of the multiple sampled maps is encoded into bitstream 1140 by an auxiliary data encoder. In one example, in response to a first sampling rate that differs from a default sampling rate, for example, the first sampling rate is encoded into bitstream 1140 by an auxiliary data encoder. In another example, in response to a first sampling rate that differs from a default sampling rate, for example, the difference between the first sampling rate and the default sampling rate is encoded into bitstream 1140 by an auxiliary data encoder.

[0174] In some examples, a first sampling rate is selected for the first portion of multiple graphs from a set of predefined sampling rates. Then, for example, an index is encoded into the bitstream 1140 by an auxiliary data encoder. This index indicates the first sampling rate selected from the set of predefined sampling rates.

[0175] In some examples, the bitstream carries multiple 3D grid frames. In one example, during the encoding of the first 3D grid frame, a first sampling rate associated with a first patch of the first 3D grid frame is applied. Then, a second sampling rate associated with a second patch in a second 3D grid frame is determined based on the first sampling rate. The second patch in the second 3D grid frame is the patch corresponding to the first patch of the first 3D grid frame. In one example, the first 3D grid frame is the key grid frame of a group of pictures (GOP), and the second sampling rate is equal to the first sampling rate.

[0176] The process then proceeds to 1499 and terminates.

[0177] Process 1400 can be adjusted as appropriate. Steps in process 1400 can be modified and / or omitted. Additional steps can be added. Any suitable implementation order can be used.

[0178] Figure 15 A flowchart outlining a process 1500 according to an embodiment of the present disclosure is shown. Process 1500 can be used during the decoding process of a mesh. In various embodiments, process 1500 is executed by processing circuitry. In some embodiments, process 1500 is implemented in software instructions, so that when the processing circuitry executes the software instructions, the processing circuitry executes process 1500. The process begins at S1501 and proceeds to S1510.

[0179] In S1510, multiple initial images in 2D are decoded from a bitstream carrying 3D mesh frames. In one example, video decoder 1160 decodes multiple initial images in 2D from bitstream 1145.

[0180] In S1520, at least two sampling rates associated with different portions of the multiple initial maps are determined. In one example, the auxiliary data decoder 1170 determines at least two sampling rates associated with different portions of the multiple initial maps.

[0181] In S1530, multiple restored maps are obtained from the multiple initial maps based on at least two sampling rates associated with different portions of the multiple initial maps. In one example, restoration module 1180 generates multiple restored maps from the multiple initial maps based on at least two sampling rates associated with different portions of the multiple initial maps. For example, a first portion of the multiple initial maps is restored (e.g., upsampled) based on a first sampling rate of at least two sampling rates associated with a first portion, and a second portion of the multiple initial maps is restored (e.g., upsampled) based on a second sampling rate of at least two sampling rates associated with a second portion.

[0182] In S1540, the 3D mesh frame is reconstructed based on multiple recovery maps. In one example, the mesh reconstruction module 1190 reconstructs the 3D mesh frame.

[0183] In order to determine at least two sampling rates, in some examples, a first sampling rate associated with a first initial graph of a first graph type is determined, and a second sampling rate associated with a second initial graph of a second graph type is determined.

[0184] To determine at least two sampling rates, in some examples, a first sampling rate is determined associated with a first sub-region in the first initial image; and a second sampling rate is determined associated with a second sub-region in the first initial image. In one example, each of the first and second sub-regions includes multiple rows of pixels (e.g., sampled block rows). In another example, each of the first and second sub-regions includes multiple columns of pixels. In another example, each of the first and second sub-regions corresponds to a sampled coding tree unit (CTU) line. In another example, each of the first and second sub-regions includes a sampled slice. In another example, each of the first and second sub-regions includes a sampled tile. In another example, each of the first and second sub-regions includes a sampled group of tiles. In another example, each of the first and second sub-regions includes a sampled CTU.

[0185] In order to determine at least two sampling rates, in some examples, a first sampling rate is determined that is associated with a first patch (e.g., a first UV patch, a first 2D shape) in the first initial image; and a second sampling rate is determined that is associated with a second patch (e.g., a second UV patch, a second 2D shape) in the first initial image.

[0186] In some examples, a first flag associated with a first portion of multiple initial graphs is decoded from the bitstream. The first flag indicates whether a first sampling rate associated with the first portion differs from a default sampling rate. In one example, the first flag indicates that the first sampling rate differs from the default sampling rate, and then the first sampling rate is decoded from the bitstream. In another example, the first flag indicates that the first sampling rate differs from the default sampling rate, and then the difference between the first sampling rate and the default sampling rate is decoded from the bitstream.

[0187] In some examples, an index is decoded from the bitstream, indicating a sample rate selected from a set of predefined sample rates as part of multiple initial graphs.

[0188] In some examples, the bitstream carries multiple 3D grid frames. A first image associated with the first 3D grid frame is recovered based on a first sampling rate associated with a first patch in the first 3D grid frame. Then, a corresponding sampling rate associated with a second patch in a second 3D grid frame is determined based on the first sampling rate. The second patch in the second 3D grid frame is the patch corresponding to the first patch in the first 3D grid frame. In one example, the first 3D grid frame is the key grid frame of a group of pictures (GOP), and the corresponding sampling rate is equal to the first sampling rate.

[0189] Then, the process proceeds to S1599 and terminates.

[0190] Process 1500 can be adjusted as appropriate. Steps in Process 1500 can be modified and / or omitted. Additional steps can be added. Any suitable implementation order can be used.

[0191] The techniques disclosed in this disclosure can be used alone or in any combination in any order. Furthermore, each of the techniques (e.g., methods, embodiments), encoders, and decoders can be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In some examples, one or more processors execute a program stored on a non-transitory computer-readable medium.

[0192] The above-described technology can be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media. For example, Figure 16 A computer system 1600 suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0193] Computer software can be encoded using any suitable machine code or computer language, and can be assembled, compiled, linked, or similarly to create code containing instructions that can be executed directly by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., or executed through interpretation, microcode execution, etc.

[0194] The instructions can be executed on various types of computers or their components, including, for example, personal computers, tablets, servers, smartphones, gaming devices, and Internet of Things (IoT) devices.

[0195] Figure 16 The components of the computer system 1600 shown are exemplary in nature and do not impose any limitation on the scope or functionality of the computer software used to implement the embodiments of this disclosure. The configuration of the components should also not be construed as having any dependency or requirement on any component or combination of components shown in the exemplary embodiments of the computer system 1600.

[0196] Computer system 1600 may include certain human-machine interface input devices. Such human-machine interface input devices may respond to input from one or more human users via, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), and olfactory input (not shown). The human-machine interface device may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images obtained from still image cameras), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0197] The input human-machine interface device may include one or more of the following (only one of each is depicted): keyboard 1601, mouse 1602, trackpad 1603, screen / touchscreen 1610, data glove (not shown), joystick 1605, microphone 1606, scanner 1607, and camera 1608.

[0198] Computer system 1600 may also include certain human-machine interface output devices. Such human-machine interface output devices can stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell / taste. These human-machine interface output devices may include tactile output devices (e.g., tactile feedback via touchscreen 1610, data gloves (not shown), or joystick 1605, but may also include tactile feedback devices not used as input devices), audio output devices (e.g., speakers 1609, headphones (not shown)), visual output devices (e.g., screen 1610, including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touchscreen input capability, each with or without tactile feedback capability—some of which are capable of outputting two-dimensional or more than three-dimensional visual output in a manner such as stereoscopic output; virtual reality glasses (not shown), holographic displays, and smoke boxes (not shown)), and printers (not shown).

[0199] The computer system 1600 may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW 1620 having CD / DVD or similar media 1620, thumb drives 1622, removable hard disk drives or solid-state drives 1623, conventional magnetic media such as magnetic tapes and floppy disks (not shown), dedicated ROM / ASIC / PLD devices such as security dongles (not shown), etc.

[0200] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other transient signals.

[0201] Computer system 1600 may also include an interface 1654 to one or more communication networks 1655. The network may be, for example, wireless, wired, or optical. The network may also be local area, wide area, metropolitan, vehicular, and industrial, real-time, latency-tolerant, etc. Examples of networks include local area networks such as Ethernet and wireless LANs; cellular networks including GSM, 3G, 4G, 5G, LTE, etc.; cable or wireless wide area digital television networks including cable television, satellite television, and terrestrial broadcast television; and vehicular and industrial networks including CANBus. Some networks typically require external network interface adapters to connect to certain general-purpose data ports or peripheral buses 1649 (e.g., USB ports of computer system 1600); others are typically integrated into the core of computer system 1600 by connecting to system buses as described below (e.g., Ethernet interfaces in PC computer systems or cellular network interfaces in smartphone computer systems). Using any of these networks, computer system 1600 can communicate with other entities. This communication can be unidirectional, receiving-only (e.g., broadcast television), unidirectional, transmitting-only (e.g., to a CANbus device), or bidirectional, such as to other computer systems using local or wide area digital networks. As mentioned above, certain protocols and protocol stacks can be used on each of these networks and network interfaces.

[0202] The aforementioned human-machine interface devices, human-accessible storage devices, and network interfaces can be attached to the core 1640 of the computer system 1600.

[0203] Core 1640 may include one or more central processing units (CPUs) 1641, graphics processing units (GPUs) 1642, dedicated programmable processing units in the form of field-programmable gate areas (FPGAs) 1643, task-specific hardware accelerators 1644, graphics adapters 1650, etc. These devices, along with read-only memory (ROM) 1645, random access memory 1646, and internal mass storage 1647 such as internal non-user-accessible hard drives or SSDs, can be connected via system bus 1648. In some computer systems, system bus 1648 may be accessed as one or more physical connectors to allow for the expansion of additional CPUs, GPUs, etc. Peripheral devices may be connected directly or via peripheral bus 1649 to the core's system bus 1648. In one example, a screen 1610 may be connected to graphics adapter 1650. Peripheral bus architectures include PCI, USB, etc.

[0204] The CPU 1641, GPU 1642, FPGA 1643, and accelerator 1644 can execute certain instructions, which, when combined, constitute the aforementioned computer code. This computer code can be stored in ROM 1645 or RAM 1646. Transient data can be stored in RAM 1646, while permanent data can be stored, for example, in internal mass storage 1647. Fast storage and retrieval of any storage device can be achieved by using a cache memory, which can be closely associated with one or more CPUs 1641, GPUs 1642, mass storage 1647, ROM 1645, RAM 1646, etc.

[0205] Computer-readable media may contain computer code for performing operations of various computer implementations. The media and computer code may be specifically designed and constructed for the purposes of this disclosure, or may be of a type known and available to those skilled in the art of computer software.

[0206] By way of example and not limitation, a computer system having architecture 1600, particularly core 1640, can provide functionality as a result of a processor (including CPU, GPU, FPGA, accelerator, etc.) executing software contained in one or more tangible computer-readable media. Such computer-readable media can be media associated with user-accessible mass storage as described above, as well as some memory of the non-transitory core 1640, such as internal mass storage 1647 or ROM 1645. Software implementing various embodiments of this disclosure can be stored in such a device and executed by core 1640. Depending on specific needs, the computer-readable medium may include one or more storage devices or chips. The software can cause core 1640, and particularly the processors therein (including CPU, GPU, FPGA, etc.), to execute specific processes or specific portions of specific processes described herein, including defining data structures stored in RAM 1646 and modifying such data structures according to software-defined processes. Furthermore or alternatively, the computer system can provide functionality as a result of hard-wired or otherwise contained logic (e.g., accelerator 1644) that can replace or operate with software to execute specific processes or specific portions of specific processes described herein. Where appropriate, references to software may include logic, and vice versa. Where appropriate, references to computer-readable media may include circuitry storing software for execution (e.g., an integrated circuit (IC)), circuitry containing logic for execution, or both. This disclosure includes any suitable combination of hardware and software.

[0207] While several exemplary embodiments have been described in this disclosure, variations, substitutions, and various alternative equivalents fall within the scope of this disclosure. Therefore, it should be understood that those skilled in the art will be able to design numerous systems and methods that, although not expressly shown or described herein, embody the principles of this disclosure and are therefore within its spirit and scope.

Claims

1. A method for mesh decompression, comprising: Decode multiple initial 2D images from a bitstream carrying 3D mesh frames; At least two sampling rates are determined that are associated with different portions of the plurality of initial images, the at least two sampling rates being obtained based on the information density corresponding to the different portions of the plurality of initial images; Multiple restored maps are recovered from the multiple initial maps based on at least two sampling rates associated with different portions of the multiple initial maps, wherein the first portion is recovered based on a first sampling rate associated with a first portion of the multiple initial maps, and the second portion is recovered based on a second sampling rate associated with a second portion of the multiple initial maps; and The 3D mesh frame is reconstructed based on the plurality of recovery maps, wherein, when the bitstream carries a plurality of 3D mesh frames, a first recovery map associated with the first 3D mesh frame is recovered based on the first sampling rate associated with a first mesh frame patch of a first 3D mesh frame among the plurality of 3D mesh frames, and the first recovery map is used to reconstruct the first 3D mesh frame; and a corresponding sampling rate associated with a second mesh frame patch of a second 3D mesh frame among the plurality of 3D mesh frames is determined according to the first sampling rate, and a second recovery map associated with the second 3D mesh frame is recovered using the corresponding sampling rate, and the second recovery map is used to reconstruct the second 3D mesh frame, wherein the second mesh frame patch has the same or similar portions as the first mesh frame patch.

2. The method according to claim 1, wherein, Determining the at least two sampling rates also includes: Determine the first sampling rate associated with the first initial graph of the first graph type; and Determine the second sampling rate associated with the second initial graph of the second graph type.

3. The method according to claim 1, wherein, Determining the at least two sampling rates also includes: Determine the first sampling rate associated with the first sub-region in the first initial map; and Determine the second sampling rate associated with the second sub-region in the first initial map.

4. The method according to claim 3, wherein, The first sub-region includes at least one of the following: Multi-row pixels; Multi-column pixels; Sampling coding tree unit (CTU) line; A sampled slice; Sampling tiles; Sampling tile groups; and / or Sampling CTU.

5. The method according to claim 1, wherein, Determining the at least two sampling rates also includes: Determine the first sampling rate associated with the first patch in the first initial image; and Determine the second sampling rate associated with the second patch in the first initial image.

6. The method according to claim 1, wherein, Determining the at least two sampling rates also includes: Decode from the bitstream a first flag associated with the first portion of the plurality of initial graphs, wherein the first flag indicates whether the first sampling rate associated with the first portion is different from the default sampling rate.

7. The method according to claim 6, wherein, The first flag indicates that the first sampling rate is different from the default sampling rate, and the method further includes at least one of the following: Decode the first sampling rate from the bitstream; and / or Decode the difference between the first sampling rate and the default sampling rate from the bitstream.

8. The method according to claim 1, wherein, Determining the at least two sampling rates also includes: Decode an index from the bitstream, wherein the index indicates a sampling rate selected from a set of predefined sampling rates as part of the plurality of initial graphs.

9. The method according to claim 1, wherein, The first 3D mesh frame is a keyframe of the group of pictures (GOP), and the corresponding sampling rate is equal to the first sampling rate.

10. An apparatus for mesh decompression, comprising processing circuitry configured to: Decode multiple initial 2D images from a bitstream carrying 3D mesh frames; At least two sampling rates are determined that are associated with different portions of the plurality of initial images, the at least two sampling rates being obtained based on the information density corresponding to the different portions of the plurality of initial images; Multiple restored maps are recovered from the multiple initial maps based on at least two sampling rates associated with different portions of the multiple initial maps, wherein the first portion is recovered based on a first sampling rate associated with a first portion of the multiple initial maps, and the second portion is recovered based on a second sampling rate associated with a second portion of the multiple initial maps; and The 3D mesh frame is reconstructed based on the plurality of recovery maps, wherein, when the bitstream carries a plurality of 3D mesh frames, a first recovery map associated with the first 3D mesh frame is recovered based on the first sampling rate associated with a first mesh frame patch of a first 3D mesh frame among the plurality of 3D mesh frames, and the first recovery map is used to reconstruct the first 3D mesh frame; and a corresponding sampling rate associated with a second mesh frame patch of a second 3D mesh frame among the plurality of 3D mesh frames is determined according to the first sampling rate, and a second recovery map associated with the second 3D mesh frame is recovered using the corresponding sampling rate, and the second recovery map is used to reconstruct the second 3D mesh frame, wherein the second mesh frame patch has the same or similar portions as the first mesh frame patch.

11. The apparatus according to claim 10, wherein, The processing circuit is configured as follows: Determine the first sampling rate associated with the first initial graph of the first graph type; and Determine the second sampling rate associated with the second initial graph of the second graph type.

12. The apparatus according to claim 10, wherein, The processing circuit is configured as follows: Determine the first sampling rate associated with the first sub-region in the first initial map; and Determine the second sampling rate associated with the second sub-region in the first initial map.

13. The apparatus according to claim 12, wherein, The first sub-region includes at least one of the following: Multi-row pixels; Multi-column pixels; Sampling coding tree unit (CTU) line; A sampled slice; Sampling tiles; Sampling tile groups; and / or Sampling CTU.

14. The apparatus according to claim 10, wherein, The processing circuit is configured as follows: Determine the first sampling rate associated with the first patch in the first initial image; and Determine the second sampling rate associated with the second patch in the first initial image.

15. The apparatus according to claim 10, wherein, The processing circuit is configured as follows: Decode from the bitstream a first flag associated with the first portion of the plurality of initial graphs, wherein the first flag indicates whether the first sampling rate associated with the first portion is different from the default sampling rate.

16. The apparatus according to claim 15, wherein, The first flag indicates that the first sampling rate is different from the default sampling rate, and the processing circuit is configured to: Decode at least one of the following: Decode the first sampling rate from the bitstream; and / or Decode the difference between the first sampling rate and the default sampling rate from the bitstream.

17. The apparatus according to claim 10, wherein, The processing circuit is configured as follows: Decode an index from the bitstream, wherein the index indicates a sampling rate selected from a set of predefined sampling rates as part of the plurality of initial graphs.

18. The apparatus according to claim 10, wherein, The first 3D mesh frame is a keyframe of the group of pictures (GOP), and the corresponding sampling rate is equal to the first sampling rate.

Images