Method and apparatus for encoding a mesh
By determining the sampling rate according to characteristics in different regions of the grid frame and generating non-overlapping sampling areas, the problem of low grid data compression efficiency in the prior art is solved, and more efficient data transmission and storage is achieved. Applicable to both high resolution and dynamic mesh models.
Patent Information
- Application Number
- CN202280007284.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2022-09-14
- Filing Date
- 2022-09-16
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2042-09-16
AI Technical Summary
The prior art is inefficient in the compression and decompression of grid data in three-dimensional space, especially when dealing with high resolution and dynamic grid models, resulting in waste of data transmission and storage resources.
Adaptive sampling technology is adopted to achieve more efficient grid compression and decompression by determining the sampling rate according to characteristics in different regions of the grid frame, non-overlapping sampling areas are generated and encoded into the code stream.
Improves the compression efficiency of grid data, reduces data transmission and storage requirements, maintains image quality, and is suitable for static and dynamic grid models.
Smart Images

Figure CN116569217B_ABST
Abstract
Description
[0001] Incorporation by reference
[0002] This application claims the benefit of priority to U.S. Patent Application No. 17 / 944,994, filed on September 14, 2022, entitled “METHOD AND APPARATUS OF ADAPTIVE SAMPLING FOR MESH COMPRESSION BY ENCODERS,” which claims the benefit of priority to U.S. Provisional Application No. 63 / 252,063, filed on October 4, 2021, entitled “Method and Apparatus of Adaptive Sampling for Mesh Compression by Encoders.” The disclosures of these prior applications are incorporated herein by reference in their entirety. Technical Field
[0003] This disclosure describes embodiments generally related to trellis coding. Background Art
[0004] The background description provided herein is for the purpose of generally presenting the content of the present disclosure. The work of the presently named inventors is neither explicitly nor implicitly admitted as prior art to the present disclosure as long as it is within the scope of the work described in this background section and in aspects of the specification that may not be prior art at the time of filing.
[0005] Various technologies have been developed to capture and represent the world in three-dimensional (3D) space, such as objects in the world, environments in the world, etc. 3D representations of the world can enable more immersive forms of interaction and communication. In some examples, point clouds and meshes can be used as 3D representations of the world. Summary of the invention
[0006] Various aspects of the present disclosure provide methods and apparatus for mesh encoding (e.g., compression and decompression). In some examples, the apparatus for mesh encoding includes a processing circuit. The processing circuit receives a data structure of a mesh frame, the mesh frame having polygons representing the surface of an object. The data structure of the mesh frame includes a UV atlas, which associates vertices of the mesh frame with UV coordinates in the UV atlas. The processing circuit determines the respective sampling rates of the multiple regions of the mesh frame based on the respective characteristics of the multiple regions of the mesh frame, and applies the respective sampling rates of the multiple regions of the mesh frame on the UV atlas to determine multiple sampling positions on the UV atlas. The processing circuit generates one or more sampled two-dimensional (2D) graphs for the mesh frame based on the multiple sampling positions on the UV atlas, and encodes the one or more sampled 2D graphs into a bitstream.
[0007] To determine the sampling rates of each of the plurality of regions of the grid frame, in some examples, the processing circuit determines a first sampling rate for the first region based on a requirement to limit the number of first sampling positions in the first region of the grid frame. In one example, the processing circuit determines the first sampling rate to achieve an oversampling ratio (OR) that is closest to a target value. The OR is a ratio of the number of first sampling positions to the number of vertices initially located in the first region of the grid frame. In another example, the processing circuit adjusts the first sampling rate until the OR is less than a first threshold. In another example, the processing circuit adjusts the first sampling rate until the OR is less than the first threshold and greater than a second threshold.
[0008] In some embodiments, a sampling 2D map in the one or more sampling 2D maps includes a sampling region having a plurality of sampling points corresponding to a plurality of sampling positions on a UV atlas. The processing circuit determines that a sampling position is occupied in response to the sampling position being located in a polygon; and determines the attributes of the sampling points in the sampling 2D map corresponding to the sampling position based on the attributes of the vertices of the polygon.
[0009] To generate one or more sampling 2D maps, in some examples, the processing circuit forms multiple sampling regions corresponding to multiple regions of the grid based on the respective sampling rates of the multiple regions of the grid, and arranges the multiple sampling regions in a non-overlapping configuration to form a sampling map.
[0010] In order to arrange the plurality of sampling regions in a non-overlapping configuration, in some examples, after arranging the subset of the plurality of sampling regions in the non-overlapping configuration, for the current sampling region, the processing circuit determines an initial arrangement position according to the sampling rate of the current sampling region, and determines whether the current sampling region at the initial arrangement position overlaps with the subset of the plurality of sampling regions. In one example, in response to the current sampling region at the initial arrangement position overlapping with the subset of the plurality of sampling regions, the processing circuit determines an offset of the initial arrangement position, the offset allowing the current sampling region not to overlap with the subset of the plurality of sampling regions.
[0011] In some embodiments, the non-overlapping configuration includes a minimum distance requirement between the plurality of sampling regions.
[0012] In one example, the processing circuit directly encodes the value of the first sampling rate of the first area into the code stream. In another example, the processing circuit encodes a first index into the code stream, the first index indicating that the first sampling rate is selected from a set of sampling rates. In another example, the processing circuit encodes a syntax that instructs the predictor to predict the first sampling rate from a pre-established set of sampling rates. In another example, the processing circuit encodes a syntax that instructs the predictor to predict the first sampling rate based on a sampling rate previously used in an encoded area of a grid frame. In another example, the processing circuit encodes a syntax that instructs the predictor to predict the first sampling rate based on a sampling rate previously used in an encoded area of another grid frame encoded before the grid frame.
[0013] In some embodiments, the processing circuit makes a decision to signal a first sampling rate associated with the first region or predict the first sampling rate and encodes a first syntax value indicating the decision.
[0014] In some examples, in response to the decision to signal the first sampling rate, the processing circuit may directly encode the value of the first sampling rate into the bitstream, or may encode an index into the bitstream, the index indicating that the first sampling rate is selected from the set of sampling rates.
[0015] In some examples, in response to the decision to predict the first sampling rate, the processing circuit may encode a second syntax into the bitstream, the second syntax indicating the use of the predictor to predict the first sampling rate. In addition, in one example, the processing circuit encodes a prediction residual into the bitstream. The prediction residual is the difference between the first sampling rate and the sampling rate of the predictor.
[0016] In some examples, the processing circuit encodes the base sampling rate into the bitstream.
[0017] In some examples, the processing circuit encodes a control flag indicating that adaptive 2D atlas sampling can be enabled or disabled.
[0018] In some examples, the processing circuit determines at least a first UV offset associated with a first region of the grid frame. The first UV offset is applied to a first sampling region corresponding to the first region to avoid overlapping with other sampling regions. The processing circuit encodes syntax indicating the first UV offset into a bitstream. In one example, the processing circuit encodes syntax into the bitstream, the syntax having a value for the first UV offset. In another example, the processing circuit encodes syntax indicating that a predictor predicts the first UV offset based on a pre-established set of UV offsets. In another example, the processing circuit encodes syntax indicating that a predictor predicts the first UV offset based on a UV offset previously used for an encoded region of the grid frame. In another example, the processing circuit encodes syntax indicating that a predictor predicts the first UV offset based on a UV offset previously used for an encoded region in another grid frame encoded prior to the grid frame.
[0019] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions, which, when executed by a computer, cause the computer to perform any one or a combination of the methods for grid coding. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Further features, properties and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings, in which:
[0021] Figure 1 A block diagram of a communication system in some examples is shown.
[0022] Figure 2 A block diagram of a streaming system in some examples is shown.
[0023] Figure 3 A block diagram of an encoder for encoding a point cloud frame in some examples is shown.
[0024] Figure 4 A block diagram of a decoder for decoding a compressed codestream corresponding to a point cloud frame in some examples is shown.
[0025] Figure 5 A block diagram of a video decoder in some examples is shown.
[0026] Figure 6 A block diagram of a video encoder in some examples is shown.
[0027] Figure 7 A block diagram of an encoder for encoding a point cloud frame in some examples is shown.
[0028] Figure 8A block diagram of a decoder for decoding a compressed codestream carrying point cloud frames in some examples is shown.
[0029] Fig. 9 Schematic diagram showing the mapping of grids to atlases in some examples.
[0030] Fig.10 Schematic diagram showing downsampling in some examples.
[0031] Fig.11 A diagram showing a framework for mesh compression according to some embodiments of the present disclosure.
[0032] Fig.12 A diagram showing adaptive sampling in some examples.
[0033] Fig.13 A diagram showing adaptive sampling in some examples.
[0034] Fig.14 A flow chart outlining examples of the process in some examples is shown.
[0035] Fig.15 A flow chart outlining examples of the process in some examples is shown.
[0036] Fig.16 Schematic diagrams of computer systems in some examples are shown. DETAILED DESCRIPTION
[0037] Aspects of the present disclosure provide techniques in the field of 3D media processing.
[0038] Technological developments in 3D media processing, such as advances in 3D acquisition, 3D modeling, and 3D rendering, have facilitated the ubiquity of 3D media content on several platforms and devices. In one example, a baby's first steps can be captured on one continent, and media technology can enable grandparents to watch (and possibly interact with the baby) on another continent and enjoy an immersive experience with the baby. According to one aspect of the present disclosure, in order to improve the immersive experience, 3D models are becoming increasingly sophisticated, and the creation and consumption of 3D models consumes a large amount of data resources, such as data storage and data transmission resources.
[0039] According to some aspects of the present disclosure, point clouds and meshes may be used as 3D models representing immersive content.
[0040] Point clouds generally refer to a set of points in 3D space, each point having associated attributes such as color, material, texture information, intensity attributes, reflectivity attributes, motion-related attributes, modality attributes, and various other attributes. Point clouds can be used to reconstruct objects or scenes as a composite of these points.
[0041] The mesh of an object (also referred to as a mesh model) may include polygons that describe the surface of the object. Each polygon may be defined by the vertices of the polygon in 3D space and information about how the vertices are connected to form the polygon. The information about how the vertices are connected is called connectivity information. In some examples, the mesh may also include attributes associated with the vertices, such as color, normals, etc.
[0042] According to some aspects of the present disclosure, some coding tools for point cloud compression (PCC) can be used for mesh compression. For example, a mesh can be re-meshed to generate a new mesh, and the connectivity information of the new mesh can be inferred. The vertices of the new mesh and the attributes associated with the vertices of the new mesh can be considered as points in a point cloud and can be compressed using a PCC codec.
[0043] Point clouds can be used to reconstruct an object or scene as a composite of points. In various settings, the points can be collected using multiple cameras, depth sensors, or lidar, and can consist of thousands to billions of points in order to realistically represent the reconstructed scene or object. A patch can generally refer to a continuous subset of a surface described by a point cloud. In one example, a patch includes points whose surface normal vectors deviate from each other by less than a threshold amount.
[0044] PCC can be performed according to various schemes, such as a geometry-based scheme (referred to as G-PCC), a video coding-based scheme (referred to as V-PCC), etc. According to some aspects of the present disclosure, G-PCC directly encodes 3D geometry and is a purely geometry-based approach that does not share much with video coding, while V-PCC is largely based on video coding. For example, V-PCC can map points of a 3D cloud to pixels of a 2D grid (image). The V-PCC scheme can utilize a general video codec for point cloud compression. In the present disclosure, the PCC codec (encoder / decoder) can be a G-PCC codec (encoder / decoder) or a V-PCC codec.
[0045] According to one aspect of the present disclosure, the V-PCC scheme can compress the geometry, occupancy and texture of the point cloud into three separate video sequences using an existing video codec. The additional metadata required to interpret the three video sequences is compressed separately. A small portion of the entire bitstream is metadata, and in one example, software can be used to efficiently encode / decode the metadata. Most of the information is processed by the video codec.
[0046] Figure 1A block diagram of a communication system (100) in some examples is shown. The communication system (100) includes a plurality of terminal devices that can communicate with each other via, for example, a network (150). For example, the communication system (100) includes a pair of terminal devices (110) and (120) interconnected via the network (150). Figure 1 In the example of , a first terminal device pair (110) and (120) may perform a one-way transmission of point cloud data. For example, the terminal device (110) may compress a point cloud (e.g., points representing a structure), which is collected by a sensor (105) connected to the terminal device (110). The compressed point cloud may be transmitted to another terminal device (120) via a network (150), for example, in the form of a code stream. The terminal device (120) may receive the compressed point cloud from the network (150), decompress the code stream to reconstruct the point cloud, and appropriately display the reconstructed point cloud. One-way data transmission may be common in applications such as media services.
[0047] exist Figure 1 In the example of , the terminal device (110) and the terminal device (120) may be shown as a server and a personal computer, but the principles of the present disclosure may not be limited to this. Embodiments of the present disclosure may be applied to laptop computers, tablet computers, smart phones, game consoles, media players and / or dedicated three-dimensional (3D) devices. The network (150) represents any number of networks that transmit compressed point clouds between the terminal device (110) and the terminal device (120). The network (150) may include, for example, a wired (wired) and / or wireless communication network. The network (150) may exchange data in a circuit switching channel and / or a packet switching channel. Representative networks include telecommunication networks, local area networks, wide area networks, the Internet, etc.
[0048] Figure 2 A block diagram of a streaming system (200) in some examples is shown. The streaming system (200) is a point cloud usage application. The disclosed subject matter is equally applicable to other point cloud-enabled applications, such as 3D telepresence applications, virtual reality applications, and the like.
[0049] The streaming system (200) may include an acquisition subsystem (213). The acquisition subsystem (213) may include a point cloud source (201), such as a light detection and ranging (LIDAR) system, a 3D camera, a 3D scanner, a graphics generation component that generates an uncompressed point cloud in software, and similar components that generate, for example, an uncompressed point cloud (202). In one example, the point cloud (202) includes points acquired by a 3D camera. Compared to a compressed point cloud (204) (a code stream of the compressed point cloud), the point cloud (202) is depicted as a thick line to emphasize the high amount of data. The compressed point cloud (204) may be generated by an electronic device (220), which includes an encoder (203) coupled to the point cloud source (201). The encoder (203) may include hardware, software, or a combination of hardware and software to implement or implement various aspects of the disclosed subject matter as described in more detail below. The compressed point cloud (204) (or a stream of the compressed point cloud (204)), depicted as thin lines to emphasize the lower data volume compared to the point cloud stream (202), can be stored on the streaming server (205) for future use. One or more streaming client subsystems, such as Figure 2 A client subsystem (206) and a client subsystem (208) in a streaming server (205) may access a copy (207) and a copy (209) of a compressed point cloud (204). The client subsystem (206) may include, for example, a decoder (210) in an electronic device (230). The decoder (210) decodes the incoming copy (207) of the compressed point cloud and creates an output stream (211) of a reconstructed point cloud that can be rendered on a rendering device (212).
[0050] It should be noted that the electronic device (220) and the electronic device (230) may include other components (not shown). For example, the electronic device (220) may include a decoder (not shown), and the electronic device (230) may further include an encoder (not shown).
[0051] In some streaming systems, the compressed point clouds (204), (207), and (209) (e.g., code streams of compressed point clouds) may be compressed according to certain standards. In some examples, a video coding standard is used when compressing the point clouds. Examples of these standards include High Efficiency Video Coding (HEVC), Next Generation Video Coding (VVC), etc.
[0052] Figure 3A block diagram of a V-PCC encoder (300) for encoding a point cloud frame according to some embodiments is shown. In some embodiments, the V-PCC encoder (300) can be used in a communication system (100) and a streaming system (200). For example, the encoder (203) can be configured and operated in a manner similar to the V-PCC encoder (300).
[0053] The V-PCC encoder (300) receives a point cloud frame as an uncompressed input and generates a bitstream corresponding to the compressed point cloud frame. In some embodiments, the V-PCC encoder (300) may receive the point cloud frame from a point cloud source (e.g., point cloud source (201)).
[0054] exist Figure 3 In the example, the V-PCC encoder (300) includes a patch generation module (306), a patch packing module (308), a geometric image generation module (310), a texture image generation module (312), a patch information module (304), an occupancy map module (314), a smoothing module (336), image filling modules (316) and (318), a group expansion module (320), video compression modules (322), (323) and (332), an auxiliary patch information compression module (338), an entropy compression module (334) and a multiplexer (324).
[0055] According to one aspect of the present disclosure, a V-PCC encoder (300) converts a 3D point cloud frame into an image-based representation together with some metadata (e.g., occupancy map and patch information) that is used to convert the compressed point cloud back into a decompressed point cloud. In some examples, the V-PCC encoder (300) can convert the 3D point cloud frame into a geometric image, a texture image, and an occupancy map, and then encode the geometric image, texture image, and occupancy map into a bitstream using video coding techniques. Typically, a geometric image is a 2D image in which pixels are filled with geometric values associated with points projected to the pixels, and pixels filled with geometric values may be referred to as geometric samples. A texture image is a 2D image in which pixels are filled with texture values associated with points projected to the pixels, and pixels filled with texture values may be referred to as texture samples. An occupancy map is a 2D image in which pixels are filled with values indicating whether they are occupied or not occupied by patches.
[0056] The patch generation module (306) segments the point cloud into a set of patches (e.g., a patch is defined as a continuous subset of the surface described by the point cloud), which may or may not overlap, such that each patch can be described by a depth field for a plane in 2D space. In some embodiments, the patch generation module (306) aims to decompose the point cloud into a minimum number of patches with smooth boundaries while also minimizing reconstruction errors.
[0057] In some examples, the patch information module (304) may collect patch information indicating the size and shape of the patch. In some examples, the patch information may be packed into an image frame and then encoded by the auxiliary patch information compression module (338) to generate compressed auxiliary patch information.
[0058] In some examples, the patch packing module (308) is configured to map the extracted patches onto a two-dimensional (2D) grid while minimizing unused space and ensuring that each M×M (e.g., 16×16) block of the grid is associated with a unique patch. Effective patch packing can directly affect compression efficiency by minimizing unused space or ensuring temporal consistency.
[0059] The geometry image generation module (310) can generate a 2D geometry image associated with the geometry of the point cloud at a given patch location. The texture image generation module (312) can generate a 2D texture image associated with the texture of the point cloud at a given patch location. The geometry image generation module (310) and the texture image generation module (312) use the 3D to 2D mapping calculated in the packing process to store the geometry and texture of the point cloud as images. In order to better handle the situation where multiple points are projected onto the same sample, each patch is projected onto two images (called layers). In one example, the geometry image is represented by a W×H monochrome frame in a YUV 420-8 bit format. To generate the texture image, the texture generation process uses the reconstructed / smoothed geometry to calculate the color to be associated with the resampled points.
[0060] The occupancy map module (314) can generate an occupancy map that describes filling information at each cell. For example, the occupancy image includes a binary map that indicates, for each cell of the grid, whether the cell belongs to free space or to a point cloud. In one example, the occupancy map uses binary information for each pixel to describe whether the pixel is filled. In another example, the occupancy map uses binary information for each pixel block to describe whether the pixel block is filled.
[0061] The occupancy map generated by the occupancy map module (314) can be compressed using lossless coding or lossy coding. When lossless coding is used, the entropy compression module (334) is used to compress the occupancy map. When lossy coding is used, the video compression module (332) is used to compress the occupancy map.
[0062] It should be noted that the patch packing module (308) may leave some free space between the 2D patches packed into the image frame. The image padding modules (316) and (318) may fill the free space (referred to as padding) to generate an image frame that may be suitable for 2D video and image codecs. Image padding is also referred to as background padding, which may fill unused space with redundant information. In some examples, good background padding minimally increases the bit rate while not introducing significant coding distortion near patch boundaries.
[0063] The video compression modules (322), (323) and (332) can encode 2D images based on a suitable video coding standard (e.g., HEVC, VVC, etc.), such as filled geometry images, filled texture images and occupancy maps. In one example, the video compression modules (322), (323) and (332) are independent components that operate independently. It should be noted that in another example, the video compression modules (322), (323) and (332) can be implemented as a single component.
[0064] In some examples, the smoothing module (336) is configured to generate a smoothed image of the reconstructed geometric image. The smoothed image can be provided to the texture image generation (312). The texture image generation (312) can then adjust the generation of the texture image based on the reconstructed geometric image. For example, when the patch shape (e.g., the geometric shape) is slightly distorted during encoding and decoding, the distortion can be taken into account when generating the texture image to correct the distortion in the patch shape.
[0065] In some embodiments, the group expansion (320) is configured to fill pixels near object boundaries with redundant low-frequency content to improve coding gain and visual quality of the reconstructed point cloud.
[0066] The multiplexer (324) can multiplex the compressed geometry image, the compressed texture image, the compressed occupancy map, and the compressed auxiliary patch information into the compressed code stream.
[0067] Figure 4 A block diagram of a V-PCC decoder (400) for decoding a compressed code stream corresponding to a point cloud frame in some examples is shown. In some examples, the V-PCC decoder (400) can be used in a communication system (100) and a streaming system (200). For example, the decoder (210) can be configured to operate in a manner similar to the V-PCC decoder (400). The V-PCC decoder (400) receives a compressed code stream and generates a reconstructed point cloud based on the compressed code stream.
[0068] exist Figure 4In the example of , the V-PCC decoder (400) includes a demultiplexer (432), video decompression modules (434) and (436), an occupancy map decompression module (438), an auxiliary patch information decompression module (442), a geometry reconstruction module (444), a smoothing module (446), a texture reconstruction module (448), and a color smoothing module (452).
[0069] The demultiplexer (432) may receive the compressed code stream and separate the compressed code stream into a compressed texture image, a compressed geometry image, a compressed occupancy map, and compressed auxiliary patch information.
[0070] The video decompression modules (434) and (436) can decode the compressed images according to appropriate standards (e.g., HEVC, VVC, etc.) and output the decompressed images. For example, the video decompression module (434) decodes the compressed texture images and outputs the decompressed texture images; and the video decompression module (436) decodes the compressed geometric images and outputs the decompressed geometric images.
[0071] The occupancy map decompression module (438) may decode the compressed occupancy map according to a suitable standard (eg, HEVC, VVC, etc.) and output a decompressed occupancy map.
[0072] The auxiliary patch information decompression module (442) can decode the compressed auxiliary patch information according to a suitable standard (such as HEVC, VVC, etc.) and output the decompressed auxiliary patch information.
[0073] The geometry reconstruction module (444) may receive the decompressed geometry image and generate a reconstructed point cloud geometry based on the decompressed occupancy map and the decompressed auxiliary patch information.
[0074] The smoothing module (446) can smooth out inconsistencies at the edges of patches. The smoothing process is intended to mitigate potential discontinuities that may appear at patch boundaries due to compression artifacts. In some embodiments, a smoothing filter may be applied to pixels located on patch boundaries to mitigate distortion that may be caused by compression / decompression.
[0075] The texture reconstruction module (448) can determine texture information for points in the point cloud based on the decompressed texture image and the smoothed geometry.
[0076] The color smoothing module (452) can smooth out inconsistencies in coloring. Non-adjacent patches in 3D space are usually packed adjacent to each other in 2D video. In some examples, pixel values from non-adjacent patches can be blended by a block-based video codec. The goal of color smoothing is to reduce visible artifacts that appear at patch boundaries.
[0077] Figure 5A block diagram of a video decoder (510) in some examples is shown. The video decoder (510) may be used in the V-PCC decoder (400). For example, the video decompression modules (434) and (436), and the occupancy map decompression module (438) may be similarly configured for the video decoder (510).
[0078] The video decoder (510) may include a parser (520) to reconstruct symbols (521) from a compressed image (e.g., an encoded video sequence). The categories of these symbols include information for managing the operation of the video decoder (510). The parser (520) may parse / entropy decode the received encoded video sequence. The encoding of the encoded video sequence may be performed according to a video coding technique or standard, and may follow various principles, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (520) may extract a subgroup parameter set for at least one subgroup of the subgroups of pixels in the video decoder from the encoded video sequence based on at least one parameter corresponding to the group. The subgroup may include a group of pictures (Groups of Picture, GOP), a picture, a tile, a slice, a macroblock, a coding unit (CodingUnit, CU), a block, a transform unit (Transform Unit, TU), a prediction unit (Prediction Unit, PU), etc. The parser (520) may also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, etc.
[0079] The parser (520) may perform entropy decoding / parsing operations on the video sequence received from the buffer memory, thereby creating symbols (521).
[0080] Depending on the type of coded video picture or part of coded video picture (e.g., inter-picture and intra-picture, inter-block and intra-block) and other factors, the reconstruction of symbol (521) may involve multiple different units. Which units are involved and how they are involved may be controlled by subgroup control information parsed by parser (520) from the coded video sequence. For clarity, such subgroup control information flow between parser (520) and the multiple units below is not depicted.
[0081] In addition to the functional blocks already mentioned, the video decoder (510) can be conceptually subdivided into a number of functional units as described below. In actual implementations operating under commercial constraints, many of these units interact closely with each other and may be at least partially integrated with each other. However, for the purposes of describing the disclosed subject matter, the conceptual subdivision into the following multiple functional units is appropriate.
[0082] The first unit is a sealer / inverse transform unit (551). The sealer / inverse transform unit (551) receives quantized transform coefficients as symbols (521) from the parser (520) and control information including which transform to use, block size, quantization factor, quantization scaling matrix, etc. The sealer / inverse transform unit (551) may output a block including sample values, which may be input into an aggregator (555).
[0083] In some cases, the output samples of the scaler / inverse transform (551) may belong to an intra-coded block; that is, a block that does not use prediction information from a previously reconstructed picture, but may use prediction information from a previously reconstructed portion of the current picture. Such prediction information may be provided by an intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) uses surrounding reconstructed information extracted from a current picture buffer (558) to generate a block of the same size and shape as the block being reconstructed. For example, the current picture buffer (558) buffers a partially reconstructed current picture and / or a fully reconstructed current picture. In some cases, the aggregator (555) adds the prediction information generated by the intra-prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551) on a per-sample basis.
[0084] In other cases, the output samples of the scaler / inverse transform unit (551) may belong to a block that is inter-coded and potentially motion compensated. In this case, the motion compensated prediction unit (553) may access the reference picture memory (557) to extract samples for prediction. After the extracted samples are motion compensated according to the symbols (521) belonging to the block, these samples may be added to the output of the scaler / inverse transform unit (551) (in this case, referred to as residual samples or residual signals) by the aggregator (555) to generate output sample information. The extraction of prediction samples by the motion compensated prediction unit (553) from the address in the reference picture memory (557) may be controlled by a motion vector, which may be provided to the motion compensated prediction unit (553) in the form of symbols (521), which may have, for example, an X component, a Y component, and a reference picture component. Motion compensation may also include interpolation of sample values extracted from the reference picture memory (557) when using sub-sample accurate motion vectors, motion vector prediction mechanisms, etc.
[0085] The output samples of the aggregator (555) may be subjected to various loop filtering techniques in a loop filter unit (556). The video compression techniques may include in-loop filter techniques controlled by parameters included in the coded video sequence (also referred to as a coded video bitstream) and available to the loop filter unit (556) as symbols (521) from the parser (520), however, the video compression techniques may also be responsive to meta-information obtained during decoding of a previous (in decoding order) portion of a coded picture or coded video sequence, and to previously reconstructed and loop filtered sample values.
[0086] The output of the loop filter unit (556) may be a sample stream that may be output to a rendering device as well as stored in a reference picture memory (557) for future inter-picture prediction.
[0087] Once fully reconstructed, certain coded pictures may be used as reference pictures for future prediction. For example, once the coded picture corresponding to the current picture is fully reconstructed, and the coded picture is identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) may become part of the reference picture memory (557), and a new current picture buffer may be reallocated before starting to reconstruct a subsequent coded picture.
[0088] The video decoder (510) may perform decoding operations according to a predetermined video compression technique in a standard such as ITU-T H.265 Recommendation. The encoded video sequence may conform to the syntax specified by the video compression technology or standard used in the sense that the encoded video sequence follows the syntax of the video compression technology or standard and the profile recorded in the video compression technology or standard. Specifically, the profile may select certain tools from all the tools available in the video compression technology or standard as the only tools available for use under the profile. For compliance, the complexity of the encoded video sequence may also be required to be within the range defined by the hierarchy of the video compression technology or standard. In some cases, the hierarchy limits the maximum picture size, the maximum frame rate, the maximum reconstruction sampling rate (measured in, for example, mega samples per second), the maximum reference picture size, etc. In some cases, the limits set by the hierarchy may be further limited by the hypothetical reference decoder (HRD) specification and metadata of the HRD buffer management signaled in the encoded video sequence.
[0089] Figure 6A block diagram of a video encoder (603) according to one embodiment of the present disclosure is shown. The video encoder (603) may be used for a V-PCC encoder (300) that compresses a point cloud. In one example, the video compression modules (322) and (323) and the video compression module (332) are similarly configured to the encoder (603).
[0090] The video encoder (603) may receive an image (eg, a filled geometry image, a filled texture image, etc.) and generate a compressed image.
[0091] According to one embodiment, the video encoder (603) can encode and compress pictures of a source video sequence (multiple images) into an encoded video sequence (multiple compressed images) in real time or under any other time constraints required by the application. Implementing an appropriate encoding speed is a function of the controller (650). In some embodiments, the controller (650) controls other functional units as described below and is functionally coupled to the other functional units. For clarity, the coupling is not depicted in the figure. The parameters set by the controller (650) may include rate control related parameters (picture skipping, quantizer, lambda value of rate-distortion optimization techniques, etc.), picture size, picture group (GOP) layout, maximum motion vector search range, etc. The controller (650) can be configured to have other suitable functions that are related to the video encoder (603) optimized for a certain system design.
[0092] In some embodiments, the video encoder (603) is configured to operate in an encoding loop. As an oversimplified description, in one example, the encoding loop may include a source encoder (630) (e.g., responsible for creating symbols, such as a symbol stream, based on an input picture to be encoded and a reference picture) and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to create sample data in a manner similar to how the (remote) decoder may also create sample data (because in the video compression techniques considered in the disclosed subject matter, any compression between the symbols and the encoded video code stream is lossless). The reconstructed sample stream (sample data) is input to a reference picture memory (634). Since the decoding of the symbol stream produces bit-accurate results that are independent of the decoder location (local or remote), the contents of the reference picture memory (634) also correspond bit-accurately between the local encoder and the remote encoder. In other words, the reference picture samples "seen" by the prediction portion of the encoder are exactly the same as the sample values that the decoder will "see" when using the prediction during decoding. This basic principle of reference picture synchronization (and the drift that occurs when synchronization cannot be maintained, eg due to channel errors) is also used in some related techniques.
[0093] The operation of the "local" decoder (633) may be similar to that described above in conjunction with Figure 5 The "remote" decoder of the video decoder (510) described in detail is identical. However, additional brief reference is made to Figure 5 , since the symbols are available and the entropy encoder (645) and the parser (520) are capable of losslessly encoding / decoding the symbols into an encoded video sequence, the entropy decoding portion of the video decoder (510) including the parser (520) may not be fully implemented in the local decoder (633).
[0094] During operation, in some examples, the source encoder (630) may perform motion compensated predictive coding, which predictively encodes an input picture by reference to one or more previously encoded pictures from a video sequence designated as "reference pictures." In this manner, the encoding engine (632) encodes the differences between pixel blocks of the input picture and pixel blocks of a reference picture, which may be selected as a prediction reference for the input picture.
[0095] The local video decoder (633) may decode the encoded video data of the picture that may be designated as the reference picture based on the symbol created by the source encoder (630). The operation of the encoding engine (632) may advantageously be a lossy process. When the encoded video data may be decoded at the video decoder ( Figure 6 When the video encoder (603) is decoded at a remote video decoder (not shown), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. The local video decoder (633) replicates the decoding process that may be performed by the video decoder on the reference picture and may cause the reconstructed reference picture to be stored in the reference picture memory (634). In this way, the video encoder (603) may store a copy of the reconstructed reference picture locally that has common content (absent transmission errors) with the reconstructed reference picture to be obtained by the remote video decoder.
[0096] The predictor (635) may perform a prediction search for the encoding engine (632). That is, for a new picture to be encoded, the predictor (635) may search the reference picture memory (634) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference picture motion vectors, block shapes, etc., that may be used as appropriate prediction references for the new picture. The predictor (635) may operate on a pixel block by pixel block basis to find a suitable prediction reference. In some cases, as determined by the search results obtained by the predictor (635), the input picture may have prediction references taken from multiple reference pictures stored in the reference picture memory (634).
[0097] The controller (650) may manage encoding operations of the source encoder (630), including, for example, setting parameters and subgroup parameters for encoding video data.
[0098] The outputs of all the above functional units may be entropy encoded in an entropy encoder (645). The entropy encoder (645) performs lossless compression on the symbols generated by the various functional units according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc., thereby converting the symbols into a coded video sequence.
[0099] The controller (650) may manage the operation of the video encoder (603). During encoding, the controller (650) may assign a certain coded picture type to each coded picture, but this may affect the coding techniques that can be applied to the corresponding picture. For example, a picture may generally be assigned to any of the following picture types:
[0100] An intra picture (I picture) may be a picture that can be encoded and decoded without using any other picture in the sequence as a prediction source. Some video codecs allow for different types of intra pictures, including, for example, independent decoder refresh ("IDR") pictures. Those skilled in the art are aware of these variations of I pictures and their corresponding applications and features.
[0101] A predictive picture (P picture) may be a picture that can be encoded and decoded using intra prediction or inter prediction, which uses at most one motion vector and a reference index to predict sample values for each block.
[0102] Bidirectional predictive pictures (B pictures), which can be pictures that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses up to two motion vectors and reference indices to predict the sample values of each block. Similarly, multiple predictive pictures can use more than two reference pictures and associated metadata to reconstruct a single block.
[0103] The source picture may typically be spatially subdivided into blocks of samples (e.g., 4×4, 8×8, 4×8, or 16×16 samples per block), and coded block by block. These blocks may be predictively coded with reference to other (already coded) blocks, which are determined by the coding allocation applied to the corresponding picture of the block. For example, blocks of an I picture may be non-predictively coded, or blocks of an I picture may be predictively coded (spatial prediction or intra-frame prediction) with reference to already coded blocks of the same picture. Blocks of pixels of a P picture may be predictively coded by spatial prediction with reference to one previously coded reference picture or by temporal prediction. Blocks of a B picture may be predictively coded by spatial prediction with reference to one or two previously coded reference pictures or by temporal prediction.
[0104] The video encoder (603) may perform encoding operations according to a predetermined video encoding technique or standard, such as ITU-T H.265 Recommendation. In operation, the video encoder (603) may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancy in an input video sequence. Thus, the encoded video data may conform to the syntax specified by the video encoding technique or standard used.
[0105] The video may be in the form of multiple source pictures (images) in a temporal sequence. Intra-picture prediction (often shortened to intra-prediction) exploits spatial correlations in a given picture, while inter-picture prediction exploits (temporal or other) correlations between pictures. In one example, a particular picture being encoded / decoded is divided into blocks, and the particular picture being encoded / decoded is called the current picture. When a block in the current picture is similar to a reference block in a reference picture that was previously encoded in the video and is still buffered, the block in the current picture may be encoded by a vector called a motion vector. The motion vector points to a reference block in a reference picture, and in the case where multiple reference pictures are used, the motion vector may have a third dimension that identifies the reference picture.
[0106] In some embodiments, a bidirectional prediction technique may be used for inter-picture prediction. According to the bidirectional prediction technique, two reference pictures are used, for example, a first reference picture and a second reference picture that precede the current picture in the video in decoding order (but may be in the past and future, respectively, in display order). A block in the current picture may be encoded by a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. The block may be predicted by a combination of the first reference block and the second reference block.
[0107] In addition, merge mode technology can be used for inter-picture prediction to improve coding efficiency.
[0108] According to some embodiments of the present disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed in units of blocks. For example, according to the HEVC standard, pictures in a video picture sequence are divided into coding tree units (CTUs) for compression, and the CTUs in the pictures have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. Typically, a CTU includes three coding tree blocks (CTBs), which are a luminance CTB and two chrominance CTBs. Each CTU can be recursively divided into one or more coding units (CUs) using a quadtree. For example, a 64×64 pixel CTU can be divided into a 64×64 pixel CU, or 4 32×32 pixel CUs, or 16 16×16 pixel CUs. In one example, each CU is analyzed to determine a prediction type for the CU, such as an inter-prediction type or an intra-prediction type. The CU is divided into one or more prediction units (PUs) based on temporal and / or spatial predictability. Typically, each PU includes a luma prediction block (PB) and two chroma PBs. In one embodiment, the prediction operation in encoding (encoding / decoding) is performed in units of prediction blocks. Using a luma prediction block as an example of a prediction block, the prediction block includes a matrix of values (e.g., luma values) for pixels, such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.
[0109] Figure 7 A block diagram of a G-PCC decoder (700) in some examples is shown. The G-PCC decoder (700) can be configured to receive point cloud data and compress the point cloud data to generate a bitstream carrying the compressed point cloud data. In one embodiment, the G-PCC encoder (700) can include a position quantization module (710), a duplicate point removal module (712), an octree encoding module (730), an attribute transfer module (720), a level of detail (LOD) generation module (740), an attribute prediction module (750), a residual quantization module (760), an arithmetic coding module (770), an inverse residual quantization module (780), an addition module (781), and a memory (790) for storing reconstructed attribute values.
[0110] As shown, an input point cloud (701) may be received at a G-PCC encoder (700). Positions (e.g., 3D coordinates) of the point cloud (701) are provided to a quantization module (710). The quantization module (710) is configured to quantize the coordinates to generate quantized positions. A duplicate point removal module (712) is configured to receive the quantized positions and perform a filtering process to identify and remove duplicate points. An octree encoding module (730) is configured to receive the filtered positions from the duplicate point removal module (712) and perform an octree-based encoding process to generate an occupancy code sequence describing a 3D grid of voxels. The occupancy codes are provided to an arithmetic coding module (770).
[0111] The attribute transfer module (720) is configured to receive attributes of the input point cloud and, when multiple attribute values are associated with corresponding voxels, perform an attribute transfer process to determine the attribute value of each voxel. The attribute transfer process can be performed on the reordered points output from the octree encoding module (730). The attributes after the transfer operation are provided to the attribute prediction module (750). The LOD generation module (740) is configured to operate on the reordered points output from the octree encoding module (730) and reorganize the points into different LODs. The LOD information is provided to the attribute prediction module (750).
[0112] The attribute prediction module (750) processes the points according to an LOD-based order indicated by the LOD information from the LOD generation module (740). The attribute prediction module (750) generates an attribute prediction for the current point based on the reconstructed attributes of the neighboring point set of the current point stored in the memory (790). Subsequently, a prediction residual may be obtained based on the original attribute value received from the attribute transfer module (720) and the locally generated attribute prediction. When the candidate index is used in the corresponding attribute prediction process, the index corresponding to the selected prediction candidate may be provided to the arithmetic coding module (770).
[0113] The residual quantization module (760) is configured to receive the prediction residual from the attribute prediction module (750) and perform quantization to generate a quantized residual. The quantized residual is provided to the arithmetic coding module (770).
[0114] The inverse residual quantization module (780) is configured to receive the quantized residual from the residual quantization module (760) and generate a reconstructed prediction residual by performing an inverse operation of the quantization operation performed at the residual quantization module (760). The addition module (781) is configured to receive the reconstructed prediction residual from the inverse residual quantization module (780) and receive the corresponding attribute prediction from the attribute prediction module (750). By combining the reconstructed prediction residual and the attribute prediction, a reconstructed attribute value is generated and stored in the memory (790).
[0115] The arithmetic coding module (770) is configured to receive the occupancy code, the candidate index (if used), the quantized residual (if generated), and other information, and perform entropy coding to further compress the received values or information. As a result, a compressed code stream (702) carrying compressed information can be generated. The code stream (702) can be sent or otherwise provided to a decoder that decodes the compressed code stream, or the code stream (702) can be stored in a storage device.
[0116] Figure 8 A block diagram of a G-PCC decoder (800) according to one embodiment is shown. The G-PCC decoder (800) may be configured to receive a compressed bitstream and perform point cloud data decompression to decompress the bitstream and generate decoded point cloud data. In one embodiment, the G-PCC decoder (800) may include an arithmetic decoding module (810), an inverse residual quantization module (820), an octree decoding module (830), an LOD generation module (840), an attribute prediction module (850), and a memory (860) for storing reconstructed attribute values.
[0117] As shown, a compressed code stream (801) can be received at an arithmetic decoding module (810). The arithmetic decoding module (810) is configured to decode the compressed code stream (801) to obtain a quantized residual (if generated) and an occupancy code of the point cloud. The octree decoding module (830) is configured to determine the reconstructed position of the points in the point cloud based on the occupancy code. The LOD generation module (840) is configured to reorganize the points into different LODs based on the reconstructed position and determine an order based on the LOD. The inverse residual quantization module (820) is configured to generate a reconstructed residual based on the quantized residual received from the arithmetic decoding module (810).
[0118] The attribute prediction module (850) is configured to perform an attribute prediction process to determine an attribute prediction of a point according to an LOD-based order. For example, the attribute prediction of the current point may be determined based on the reconstructed attribute values of the neighboring points of the current point stored in the memory (860). In some examples, the attribute prediction may be combined with the corresponding reconstruction residual to generate a reconstructed attribute of the current point.
[0119] In one example, the reconstructed attribute sequence generated from the attribute prediction module (850) and the reconstructed positions generated from the octree decoding module (830) correspond to the decoded point cloud (802) output from the G-PCC decoder (800). In addition, the reconstructed attributes are also stored in the memory (860) and can be subsequently used to derive attribute predictions for subsequent points.
[0120] In various embodiments, the encoder (300), decoder (400), encoder (700) and / or decoder (800) may be implemented using hardware, software, or a combination of hardware and software. For example, the encoder (300), decoder (400), encoder (700) and / or decoder (800) may be implemented using a processing circuit, such as one or more integrated circuits (ICs) that operate with or without software, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc. In another example, the encoder (300), decoder (400), encoder (700) and / or decoder (800) may be implemented as software or firmware including instructions, wherein the instructions are stored in a non-volatile (or non-temporary) computer-readable storage medium. When the instructions are executed by a processing circuit such as one or more processors, the instructions cause the processing circuit to perform the functions of the encoder (300), decoder (400), encoder (700) and / or decoder (800).
[0121] It should be noted that the attribute prediction modules (750) and (850) configured to implement the attribute prediction techniques disclosed herein may be included in other decoders or encoders that may have the same Figure 7 and Figure 8 The structures shown may be similar or different structures. In addition, in various examples, the encoder (700) and decoder (800) may be included in the same device, or in different devices.
[0122] According to some aspects of the present disclosure, grid compression may use a coding tool different from the PCC coding tool, or may use a PCC coding tool, such as the above-mentioned PCC (e.g., G-PCC, V-PCC) encoder, the above-mentioned PCC (e.g., G-PCC, V-PCC) decoder, etc.
[0123] The mesh of an object (also referred to as a mesh model, mesh frame) may include polygons that describe the surface of the object. Each polygon may be defined by the vertices of the polygon in 3D space and the edges that connect the vertices into the polygon. Information about how the vertices are connected (e.g., information about the edges) is called connectivity information. In some examples, the mesh of an object is formed by connected triangles that describe the surface of the object. Two triangles that share an edge are referred to as two connected triangles. In some other examples, the mesh of an object is composed of connected quadrilaterals. Two quadrilaterals that share an edge may be referred to as two connected quadrilaterals. It should be noted that the mesh may be formed by other suitable polygons.
[0124] In some examples, the mesh may also include attributes associated with the vertices, such as color, normal, etc. The attributes may be associated with the surface of the mesh by utilizing mapping information that parameterizes the mesh with a 2D attribute map. The mapping information is typically described by a set of parameter coordinates (referred to as UV coordinates or texture coordinates) associated with the mesh vertices. The 2D attribute map (referred to as a texture map in some examples) is used to store high-resolution attribute information, such as texture, normal, displacement, etc. This information can be used for various purposes, such as texture mapping and shading.
[0125] In some embodiments, the mesh may include components called geometric information, connectivity information, mapping information, vertex attributes, and property graphs. In some examples, the geometric information is described by a set of 3D positions associated with the vertices of the mesh. In one example, (x, y, z) coordinates can be used to describe the 3D position of the vertex and are also referred to as 3D coordinates. In some examples, the connectivity information includes a set of vertex indices that describes how to connect vertices to create a 3D surface. In some examples, the mapping information describes how to map the mesh surface to a 2D area of a plane. In one example, the mapping information is described by a set of UV parameters / texture coordinates (u, v) associated with the mesh vertices and connectivity information. In some examples, vertex attributes include scalar or vector attribute values associated with the mesh vertices. In some examples, the property graph includes attributes associated with the mesh surface and stored as a 2D image / video. In one example, the mapping between a video (e.g., a 2D image / video) and a mesh surface is defined by the mapping information.
[0126] According to one aspect of the present disclosure, some techniques referred to as UV mapping or mesh parameterization are used to map the surface of a mesh in a 3D domain to a 2D domain. In some examples, the mesh is divided into multiple patches in the 3D domain. A patch is a continuous subset of the mesh whose boundaries are formed by boundary edges. A boundary edge of a patch is an edge that belongs to only one polygon of the patch and is not shared by two adjacent polygons in the patch. In some examples, the vertices of the boundary edges in the patch are called boundary vertices of the patch, and the non-border vertices in the patch can be called internal vertices of the patch.
[0127] In some examples, the mesh of the object is formed by connected triangles, and the mesh can be divided into a plurality of patches, each patch being a subset of the connected triangles. The border edges of a patch are edges that belong to only one triangle in the patch and are not shared by adjacent triangles in the patch. In some examples, the vertices of the border edges in the patch are referred to as the border vertices of the patch, and the non-border vertices in the patch can be referred to as the internal vertices of the patch.
[0128] According to one aspect of the present disclosure, in some examples, patches are parameterized accordingly as 2D shapes (also referred to as UV patches). The 2D shapes can be packed (e.g., oriented and placed) into a graph, which in some examples is also referred to as an atlas. In some examples, the graph can be further processed using 2D image or video processing techniques.
[0129] In one example, the UV mapping technique generates a UV atlas (also referred to as a UV map) and one or more texture atlases (also referred to as texture maps) corresponding to a patch of a 3D mesh in a 2D manner. The UV atlas includes assigning 3D vertices of a 3D mesh to 2D points in a 2D domain (e.g., a rectangle). The UV atlas is a mapping between the coordinates of a 3D surface and the coordinates of a 2D domain. In one example, the value of a point at a 2D coordinate (u, v) in the UV atlas is formed by the coordinates (x, y, z) of the vertex in the 3D domain. In one example, the texture atlas includes color information of the 3D mesh. For example, the color of a point at a 2D coordinate (u, v) in the texture atlas (which has a 3D value (x, y, z) in the UV atlas) specifies the color attribute of a point at (x, y, z) in the 3D domain. In some examples, the coordinates (x, y, z) in the 3D domain are referred to as 3D coordinates or xyz coordinates, and the 2D coordinates (u, v) are referred to as uv coordinates or UV coordinates.
[0130] According to some aspects of the present disclosure, mesh compression may be performed by representing the mesh using one or more 2D graphs (also referred to as 2D atlases in some examples) and then encoding the 2D graphs using an image or video codec. Different techniques may be used to generate the 2D graphs.
[0131] Fig. 9 Figures illustrating the mapping of a 3D grid (910) to a 2D atlas (920) in some examples are shown. Fig. 9 In the example of , the 3D mesh (910) includes four vertices 1 to 4 that form four patches A to D. Each patch has a set of vertices and associated attribute information. For example, patch A is formed by vertices 1, 2, and 3 connected into a triangle; patch B is formed by vertices 1, 3, and 4 connected into a triangle; patch C is formed by vertices 1, 2, and 4 connected into a triangle; and patch D is formed by vertices 2, 3, and 4 connected into a triangle. In some examples, vertices 1, 2, 3, and 4 may have respective attributes, and the triangles formed by vertices 1, 2, 3, and 4 may have respective attributes.
[0132] In one example, patches A, B, C, and D in 3D are mapped to a 2D domain, such as a 2D atlas (920), which is also referred to as a UV atlas (920) or a graph (920). For example, patch A is mapped to a 2D shape (also referred to as a UV patch) A' in the graph (920), patch B is mapped to a 2D shape (also referred to as a UV patch) B' in the graph (920), patch C is mapped to a 2D shape (also referred to as a UV patch) C' in the graph (920), and patch D is mapped to a 2D shape (also referred to as a UV patch) D' in the graph (920). In some examples, coordinates in a 3D domain are referred to as (x, y, z) coordinates, and coordinates in a 2D domain such as the graph (920) are referred to as UV coordinates. Vertices in a 3D mesh may have corresponding UV coordinates in the graph (920).
[0133] The map (920) may be a geometry map having geometry information, or may be a texture map having color, normal, fabric or other attribute information, or may be an occupancy map having occupancy information.
[0134] Although in Fig. 9 In the examples of each patch is represented by a triangle, but it should be noted that a patch may include any suitable number of vertices that are connected to form a continuous subset of the mesh. In some examples, the vertices in the patch are connected into triangles. It should be noted that other suitable shapes can be used to connect the vertices in the patch.
[0135] In one example, the geometric information of the vertices may be stored in a 2D geometric map. For example, the 2D geometric map stores the (x, y, z) coordinates of the sampling points at the corresponding points in the 2D geometric map. For example, the vector value of a point at the (u, v) position in the 2D geometric map has three components corresponding to the x, y, and z values of the corresponding sampling point in the 3D grid, respectively.
[0136] According to one aspect of the present disclosure, the area in the figure may not be fully occupied. Fig. 9 In the 2D shapes A', B', C' and D', the area outside is not defined. Sample values of the area outside the 2D shapes A', B', C' and D' can be discarded after decoding. In some cases, the occupancy map is used to store some additional information for each pixel, such as storing a binary value for identifying whether the pixel belongs to a patch or is not defined.
[0137] According to one aspect of the present disclosure, a dynamic mesh is a mesh whose at least one component (geometry information, connectivity information, mapping information, vertex attributes, and attribute graph) changes over time. A dynamic mesh can be described by a sequence of meshes (also referred to as mesh frames). A dynamic mesh may require a large amount of data because it may include a large amount of information that changes over time. Compression techniques for meshes may enable efficient storage and transmission of media content in a mesh representation.
[0138] In some examples, a dynamic mesh may have constant connectivity information, time-varying geometry, and time-varying vertex attributes. In some examples, a dynamic mesh may have time-varying connectivity information. In one example, a digital content creation tool typically generates a dynamic mesh with a time-varying property graph and time-varying connectivity information. In some examples, a volumetric acquisition technique is used to generate the dynamic mesh. The volumetric acquisition technique can generate a dynamic mesh with time-varying connectivity information, especially in the case of real-time constraints.
[0139] Some techniques are used for mesh compression. In some examples, UV atlas sampling and V-PCC can be used for mesh compression. For example, the UV atlas is sampled with a regular grid to generate a geometric image with regular grid samples. The connectivity of the regular grid samples can be inferred. The regular grid samples can be considered as points in a point cloud and can therefore be encoded using a PCC codec (e.g., a V-PCC codec).
[0140] According to one aspect of the present disclosure, in order to effectively compress 3D mesh information, 2D maps such as geometry maps, texture maps (also referred to as attribute maps in some examples), occupancy maps, etc. may be downsampled before encoding.
[0141] Fig.10 Figures illustrating downsampling in some examples are shown. Fig.10 In the embodiment of the present invention, the image (1020) is downsampled by a factor of 2 along the horizontal direction and the vertical direction, and a downsampled image (1030) is generated accordingly. The width (e.g., the number of pixels along the horizontal direction) of the downsampled image (1030) is 1 / 2 of the width (e.g., the number of pixels along the horizontal direction) of the image (1020), and the height (e.g., the number of pixels along the vertical direction) of the downsampled image (1030) is 1 / 2 of the height (e.g., the number of pixels along the vertical direction) of the image (1020).
[0142] exist Fig.10 , the map (1020) includes 2D shapes (also called UV patches) A', B', C' and D', and the downsampled map (1030) includes sampled 2D shapes A", B", C" and D" corresponding to the 2D shapes A', B', C' and D', respectively. In some examples, the downsampled map (1030) is subsequently encoded by an image or video encoder on the grid encoder side.
[0143] In some examples, at the mesh decoder side, the downsampled image is decoded. After decoding the downsampled image, the downsampled image is restored to the original resolution (eg, the original number of pixels along the vertical direction and the original number of pixels along the horizontal direction) to reconstruct the 3D mesh.
[0144] Typically, dynamic mesh sequences require a lot of data because they can include a lot of information that changes over time. The sampling step applied to 2D maps (e.g. UV atlas, property maps) can help reduce the bandwidth required to represent the mesh information. However, the sampling step can also remove critical information during downsampling, such as some critical geometry of a 3D mesh.
[0145] In some examples, adaptive sampling techniques can be used to process 2D atlases (also referred to as 2D-form graphs) without losing too much important information. Adaptive sampling techniques can be used for static mesh (a mesh frame or mesh content that does not change over time) compression and dynamic mesh compression. Various adaptive sampling techniques can be applied individually or in any combination. In the following description, the adaptive sampling method is applied to a 2D atlas (e.g., a 2D-form graph), which can be a geometric graph or an attribute (texture) graph, or a geometric graph and an attribute (texture) graph.
[0146] Fig.11 A diagram of a framework (1100) for mesh compression according to some embodiments of the present disclosure is shown. The framework (1100) includes a mesh encoder (1110) and a mesh decoder (1150). The mesh encoder (1110) receives an input mesh (1101) (a mesh frame in the case of dynamic mesh processing) and encodes the input mesh (1101) into a code stream (1145), while the mesh decoder (1150) decodes the code stream (1145) to generate a reconstructed mesh (1195) (a reconstructed mesh frame in the case of dynamic mesh processing).
[0147] The trellis encoder (1110) may be any suitable device, such as a computer, a server computer, a desktop computer, a laptop computer, a tablet computer, a smart phone, a gaming device, an AR device, a VR device, etc. The trellis decoder (1150) may be any suitable device, such as a computer, a client computer, a desktop computer, a laptop computer, a tablet computer, a smart phone, a gaming device, an AR device, a VR device, etc. The code stream (1145) may be transmitted from the trellis encoder (1110) to the trellis decoder (1150) via any suitable communication network (not shown).
[0148] exist Fig.11 In an example of , a grid encoder (1110) includes a pre-processing module (1111), an adaptive sampling module (1120), a video encoder (1130), and an auxiliary data encoder (1140) coupled together. The video encoder (1130) is configured to encode image or video data (e.g., a 2D image in a representation of a 3D grid).
[0149] exist Fig.11 In the example of , the pre-processing module (1111) is configured to perform appropriate operations on the input mesh (1101) to generate a mesh with a UV atlas (1105). For example, the pre-processing module (1111) may perform a series of operations including tracking, re-meshing, parameterization, and voxelization. Fig.11 In some examples, this series of operations may be only a process of the encoder, not a part of the decoding process. In some examples, the mesh with UV atlas (1105) includes 3D position information of vertices, a UV atlas that maps the 3D position information to 2D, and other 2D attribute maps (e.g., 2D color maps, etc.).
[0150] It should be noted that in some examples, the input mesh (1101) is in the form of a mesh with a UV atlas, and then the pre-processing module (1111) can convert the input mesh (1011) into a mesh with a UV atlas (1105).
[0151] The adaptive sampling module (1120) receives a mesh with a UV atlas (1105) and performs adaptive sampling to generate an adaptive sampling map (1125). In some examples, the adaptive sampling module (1120) can use various techniques to detect characteristics in the map or characteristics in different regions of the map (e.g., information density in the map), and determine different sampling rates for sampling the map or different regions of the map based on these characteristics. Then, the 2D map can be sampled according to different sampling rates to generate an adaptive sampling map (1125). The adaptive sampling map (1125) can include a geometry map (also referred to as a geometry image in some examples), an occupancy map, other attribute maps (e.g., a color map), etc.
[0152] The video encoder (1130) may encode the adaptive sampling map (1125) into a bitstream (1145) using image coding and / or video coding techniques.
[0153] The adaptive sampling module (1120) also generates auxiliary data (1127) indicating auxiliary information for adaptive sampling. The auxiliary data encoder (1140) receives the auxiliary data (1127) and encodes the auxiliary data (1127) into a code stream (1145).
[0154] The operation of the adaptive sampling module (1120) and the auxiliary data encoder (1140) will be further described in this disclosure.
[0155] exist Fig.11 In the example of FIG. 1 , the code stream ( 1145 ) is provided to a trellis decoder ( 1150 ). The trellis decoder ( 1150 ) includes: Fig.11The video decoder (1160), auxiliary data decoder (1170) and mesh reconstruction module (1190) coupled together are shown. In one example, the video decoder (1160) corresponds to the video encoder (1130), can decode a portion of the bitstream (1145) encoded by the video encoder (1130), and generate a decoded map (1165). In some examples, the decoded map (1165) includes a decoded UV map, one or more decoded attribute maps, etc. In some examples, the decoded map (1165) includes a decoded occupancy map (e.g., an initial decoded map).
[0156] exist Fig.11 In the example of , the auxiliary data decoder (1170) corresponds to the auxiliary data encoder (1140), can decode a portion of the code stream (1145) encoded by the auxiliary data encoder (1140), and generate decoded auxiliary data (1175).
[0157] exist Fig.11 In an example of the decoded graph (1165) and the decoded auxiliary data (1175) are provided to a mesh reconstruction module (1190). The mesh reconstruction module (1190) generates a reconstructed mesh (1195) based on the decoded graph (1165) and the decoded auxiliary data (1175). In some examples, the mesh reconstruction module (1190) can determine vertices and information of vertices in the reconstructed mesh (1195), such as corresponding 3D coordinates, UV coordinates, colors, etc. associated with the vertices. The operation of the auxiliary data decoder (1170) and the mesh reconstruction module (1190) will be further described in the present disclosure.
[0158] It should be noted that the components in the grid encoder (1110), such as the pre-processing module (1111), the adaptive sampling module (1120), the video encoder (1130), and the auxiliary data encoder (1140), can be implemented by various technologies. In one example, the components are implemented by integrated circuits. In another example, the components are implemented using software that can be executed by one or more processors.
[0159] It should be noted that the components in the grid decoder (1150), such as the video decoder (1160), the auxiliary data decoder (1170), and the grid reconstruction module (1190), can be implemented by various technologies. In one example, the components are implemented by integrated circuits. In another example, the components are implemented using software that can be executed by one or more processors.
[0160] In some embodiments, sampling adaptation may be based on the type of graph. In some examples, the adaptive sampling module (1120) may apply different sampling rates to different types of graphs. For example, different sampling rates may be applied to geometric graphs and attribute graphs. In one example, a grid is a model of an object with a regular shape and rich texture. For example, an object has a rectangular shape but rich colors. Therefore, the information density of a geometric graph is relatively low. In one example, the adaptive sampling module (1120) applies a first sampling rate of 2:1 on a geometric graph (along the vertical and horizontal directions) and applies a second sampling rate of 1:1 on a texture graph (along the vertical and horizontal directions).
[0161] In some examples, a sampling rate of A:B along a direction indicates that B samples are generated from A pixels in the original image along that direction. For example, a sampling rate of 2:1 along the horizontal direction indicates that 1 sample is generated for every two pixels in the original image along the horizontal direction. A sampling rate of 2:1 along the vertical direction indicates that 1 sample is generated for every two pixels in the original image along the vertical direction.
[0162] In some examples, the term sampling step is used. The sampling step along a direction indicates the number of pixels between two adjacent sampling positions along that direction. For example, a sampling step of 2 along the horizontal direction indicates two pixels between adjacent sampling positions in the horizontal direction; and a sampling step of 2 along the vertical direction indicates two pixels between adjacent sampling positions in the vertical direction. It should be noted that in the present disclosure, the sampling rate is equivalent to the sampling step. For example, a sampling rate of 2 (e.g., 2:1) is equivalent to two pixels between adjacent sampling positions.
[0163] In some embodiments, sampling is adaptively based on sub-regions in the graph. Different sampling rates can be applied to different parts of the graph. In some examples, some pixel rows have less information to retain, so a larger sampling rate can be applied along these rows, so that the number of sample rows to be encoded is smaller. In some examples, some pixel columns have less information to retain, so a larger sampling rate can be applied along these columns, so that the number of sample columns to be encoded is smaller. For other areas, a smaller sampling rate is applied to minimize the loss of information after sampling.
[0164] Fig.12 A diagram of adaptive sampling in some examples is shown. The diagram (1220) is divided into a number of block rows, each block row including a fixed number of rows of samples (pixels). Different sampling rates are applied to these block rows along the vertical direction to generate an adaptive sampling diagram (1230). For example, each block row is a CTU row (also called a CTU line) and includes 64 rows of samples (also called pixels). Fig.12In the example of FIG. 1 , for block row 0 and block row 6 in the image (1220), a first sampling rate of 2:1 is applied along the vertical direction, and after sampling, 32 lines of samples are generated for each of block row 0 and block row 6 in the adaptive sampling image (1230). For block row 1 to block row 5 in the image (1220), a second sampling rate of 1:1 is applied along the vertical direction, thereby generating 64 lines of samples for each of block row 1 to block row 5 in the adaptive sampling image (1230).
[0165] It should be noted that Fig.12 In , a 1:1 sampling ratio is applied in the horizontal direction.
[0166] In some examples, the adaptive sampling map (1230) is then encoded by an image or video encoder, such as a video encoder (1130). At the decoder side, in one example, the adaptive sampling map (1230) is decoded. After decoding, the top 32 rows of samples are restored (upsampled) to the original resolution, such as 64 rows of samples; and the bottom 32 rows of samples are restored (upsampled) to the original resolution, such as 64 rows of samples.
[0167] In some other examples, the to-be-encoded graph in the 2D representation of the 3D grid may be divided into a plurality of sub-regions. Examples of such divisions within a graph (e.g., a picture) include slices, tiles, tile groups, coding tree units, etc. In some examples, different sampling rates may be applied to different sub-regions. In one example, different sampling rates associated with different sub-regions may be signaled in a bitstream carrying the 3D grid. On the decoder side, after decoding the adaptively sampled graph, each sub-region is restored to its original resolution according to the sampling rate associated with the sub-region.
[0168] In some examples, the process of restoring the adaptively sampled map to the original resolution is referred to as an inverse sampling process to generate a restored map. After restoration from the inverse sampling process, the output of the restored map in the form of a 2D atlas can be used for 3D mesh reconstruction.
[0169] Although Fig.12 The example of shows adaptive sampling of different block rows along the vertical direction, but similar adaptive sampling may be applied to different columns in the horizontal direction, or may be applied in both the vertical and horizontal directions.
[0170] In some embodiments, sampling adaptation is based on patches. In some examples, different patches in the graph may have different sampling rates.
[0171] Fig.13A graph of adaptive sampling in some examples is shown. A graph (1320) (e.g., a 2D atlas with high resolution) includes a plurality of 2D shapes, also called UV patches, corresponding to patches in a 3D mesh, such as a first 2D shape A' and a second 2D shape B'. Fig.13 In the example, a first sampling rate of 2:1 is applied to a first 2D shape A' in vertical and horizontal directions to generate a first sampled 2D shape A", and a second sampling rate of 1:1 is applied to a second 2D shape B' in vertical and horizontal directions to generate a second sampled 2D shape B". The first sampled 2D shape A" and the second sampled 2D shape B" are placed in a new graph called an adaptive sampling graph (1330).
[0172] exist Fig.13 In an example, the first sampled 2D shape A" is smaller than the first 2D shape A', and the second sampled 2D shape B" is the same size as the second 2D shape B'. The adaptive sampling map (1330) is encoded into a bitstream carrying the 3D mesh by an image or video encoder, such as a video encoder (1130). In some examples, a sampling rate associated with the sampled 2D shapes is encoded into a bitstream carrying the 3D mesh, for example, by an auxiliary data encoder (1140).
[0173] In some examples, at the decoder side, an image / video decoder such as a video decoder (1160) decodes an initial map (e.g., an adaptive sampling map (1330)) from a bitstream. In addition, a sampling rate associated with the sampled 2D shapes is decoded from the bitstream, for example, by an auxiliary data decoder (1170). According to the sampling rate associated with the sampled 2D shapes, the sampled 2D shapes in the adaptive sampling map (1330) are restored to their original size (e.g., the same number of pixels in the vertical and horizontal directions) to generate a restored map. The restored map is then used for 3D mesh reconstruction.
[0174] According to one aspect of the present disclosure, adaptive sampling information (e.g., sampling rates of different image types, sampling rates of different sub-regions, sampling rates of different patches, etc.) is known on the grid encoder side and the grid decoder side. In some examples, the adaptive sampling information is appropriately encoded into the bitstream carrying the 3D grid. Therefore, the grid decoder and the grid encoder can operate based on the same adaptive sampling information. The grid decoder can restore the image to a suitable size.
[0175] According to one aspect of the present disclosure, a mesh encoder such as a mesh encoder (1110) may perform 2D atlas sampling (also referred to as UV atlas sampling). For example, an adaptive sampling module (1120) may receive a mesh having a UV atlas (1105). Each vertex of the mesh having the UV atlas (1105) has a corresponding point in the UV atlas, and the position of the corresponding point in the UV atlas is specified by a UV coordinate. In the UV atlas, the corresponding point may have a vector value, which includes the 3D coordinates (e.g., (x, y, z)) of the vertex in 3D space. In addition, the mesh having the UV atlas (1105) includes one or more attribute maps, which store attribute values associated with the vertex as attribute values of pixels at positions specified by the UV coordinates in the one or more attribute maps. For example, a color map may store the color of a vertex as the color of a pixel at a position specified by the UV coordinates in the color map.
[0176] The adaptive sampling module (1120) can apply an adaptive sampling technique to a mesh having a UV atlas (1105) and output an adaptive sampling map (1125) (also referred to in some examples as an adaptive sampling atlas), which can include, for example, a sampling geometry map (also referred to as a sampling UV map or a sampling UV atlas), one or more sampling attribute maps (also referred to in some examples as texture maps), etc. In some examples, the adaptive sampling map (1125) includes a sampling occupancy map.
[0177] According to an aspect of the present disclosure, the same sampling rate configuration can be applied to various graphs, such as geometry graphs, attribute graphs, occupancy graphs, etc., to generate adaptive sampling graphs. In some examples, the adaptive sampling module (1120) can generate an adaptive sampling graph (1125) by sampling on a UV atlas (e.g., based on a sampling position on a UV atlas). The adaptive sampling module (1120) can determine the sampling position on the UV atlas and then generate an adaptive sampling graph based on the sampling position in the UV atlas. For example, after determining the sampling position on the UV atlas, for example, based on the sampling rate, the adaptive sampling module (1120) determines the position of the sampling point in each sampling graph (1125) and then determines the value of the corresponding sampling point in the sampling graph (1125).
[0178] In one example, when the sampling position on the UV atlas is located within the polygon defined by the mesh vertices, the sampling position is occupied, so the sampling point corresponding to the sampling position in the sampling occupancy map is set to occupied (for example, the value of the sampling point in the sampling occupancy map is "1"). However, when the sampling position is not located within any polygon defined by the mesh vertices, the sampling position is unoccupied, so the sampling point corresponding to the sampling position in the sampling occupancy map is set to unoccupied (for example, the value of the sampling point in the sampling occupancy map is "0").
[0179] For each occupied sampling position on the UV atlas, the adaptive sampling module (1120) can determine the 3D (geometric) coordinates of the occupied sampling position and assign the determined 3D coordinates to the vector values of the corresponding sampling points in the sampling geometric map (also referred to as the sampling UV atlas). Similarly, the adaptive sampling module (1120) can determine the attribute values (e.g., color, normal, etc.) at the occupied sampling position and assign the determined attribute values to the attribute values of the corresponding sampling points in the sampling attribute map. In some examples, the adaptive sampling module (1120) can determine the 3D coordinates and attributes of the occupied sampling position by interpolating from the associated polygon vertices.
[0180] In one example, the mesh is formed by triangles. The sampling position is located within a triangle defined by three vertices of the mesh, so the sampling position is an occupied sampling position. The adaptive sampling module (1120) may determine the 3D (geometric) coordinates at the occupied sampling position based on, for example, a weighted average 3D coordinate of the 3D coordinates of the three vertices of the triangle. The adaptive sampling module (1120) may assign the weighted average 3D coordinate to the vector value of the corresponding sampling point in the sampling geometric map. Similarly, the adaptive sampling module (1120) may determine the attribute value at the occupied sampling position based on, for example, a weighted average attribute value (e.g., weighted average color, weighted average normal, etc.) of the attributes of the three vertices of the triangle. The adaptive sampling module (1120) may assign the weighted average attribute value to the attribute value of the corresponding sampling point in the sampling attribute map.
[0181] In some examples, the sampling rate (SR) can be consistent across the entire 2D atlas (e.g., geometry, property maps, etc.), but the u-axis and v-axis sampling rates can be different. Using different sampling rates on the u-axis and v-axis enables anisotropic remeshing. As shown in reference Fig.12 and Fig.13 As described, in some examples, a 2D atlas may be divided into a plurality of regions, such as slices or tiles or patches, and the plurality of regions may have respective sampling rates. For example, a grid is formed by connecting triangles, and the grid may be divided into a plurality of patches, each patch including a subset of the entire grid. For example, different sampling rates may be applied to corresponding patches by an adaptive sampling module (1120).
[0182] According to an aspect of the present disclosure, a grid encoder, such as the grid encoder (1110), may determine a suitable sampling rate for each region in a 2D atlas.
[0183] In some examples, the adaptive sampling module (1120) may determine a sampling rate distribution on a 2D atlas (e.g., a geometric map, an attribute map, etc.). For example, the adaptive sampling module (1120) may determine a specific sampling rate (SR) for the region based on the characteristics of the region. In one example, the specific sampling rate is determined based on the frequency spectrum of the region. For example, a texture-rich region (or the entire 2D atlas) may have a high spatial frequency component in the texture attribute value, and the adaptive sampling module (1120) may assign a sampling rate suitable for the high spatial frequency component (e.g., a lower sampling rate, a lower sampling step size) to the texture-rich region. In another example, a region with high activity (or the entire 2D atlas) may include a high spatial frequency component in the coordinates (e.g., 3D coordinates, UV coordinates), and the adaptive sampling module (1120) may assign a sampling rate suitable for the high activity (e.g., a lower sampling rate, a lower sampling step size) to the region. In another example, a smooth region (or the entire 2D atlas) may lack high spatial frequency components in the texture attribute values, and the adaptive sampling module (1120) may assign a sampling rate suitable for the smooth region (e.g., a higher sampling rate, a higher sampling step size). In another example, a region with low activity (or the entire 2D atlas) may lack high spatial frequency components in coordinates (e.g., 3D coordinates, UV coordinates), and the adaptive sampling module (1120) may assign a sampling rate suitable for the low activity in the region (e.g., a higher sampling rate, a lower sampling step size).
[0184] According to one aspect of the present disclosure, the sampling rate can be represented by an oversampling rate (OR) parameter. The OR parameter is defined as the ratio of the number of sampling points in a region to the number of original vertices in the region. When the OR parameter of a region is greater than 1, the OR parameter indicates that the region is oversampled compared to the number of original vertices; when the OR parameter of a region is less than 1, the OR parameter indicates that the region is undersampled compared to the number of original vertices. For example, a region of a mesh consists of 1000 vertices, and when a specific sampling rate (SR) is applied in the region, 3000 sampling points are obtained. Then, in the case of SR, the OR parameter of the region is equal to 3, that is, OR(SR)=3.
[0185] In some embodiments, the adaptive sampling module (1120) may use an algorithm to determine the final sampling rate of the region to achieve an OR parameter that is closest to the predetermined target OR parameter. For example, for a specific region i (e.g., i is a region index for identifying the specific region), a target OR is defined for the specific region i (defined by TOR i The adaptive sampling module (1120) can convert the final sampling rate (denoted by SR) of a specific area i into i indicates) is determined to produce the closest target OR (TOR i ) parameter OR parameter sampling rate, for example, as expressed by equation (1):
[0186] SR i =argmin SR |OR(SR)-TOR i | Equation (1)
[0187] In one example, the adaptive sampling module (1120) may try multiple sampling rates and select a sampling rate from the multiple sampling rates that will produce an OR parameter that is closest to the target OR parameter. In another example, the adaptive sampling module (1120) may use an algorithm such as a binary search algorithm to perform a search within a search range of sampling rates to determine a final sampling rate that will produce an OR parameter that is closest to the target OR parameter.
[0188] In some embodiments, the adaptive sampling module (1120) may use an algorithm to determine a final sampling rate for a specific region i (e.g., i is a region index for identifying a specific region) that can achieve a maximum OR parameter that is less than a predetermined threshold (represented by Th0). In some examples, the algorithm starts with a relatively small (e.g., 1:1) basic sampling rate (BSR) and uses an iterative process with an iteration cycle to determine the final sampling rate. In each iteration cycle in which the current BSR is tested, the OR parameter for the region is determined. When the OR parameter is less than the threshold Th0, the current BSR is the final sampling rate for the specific region; and when the OR parameter is greater than the threshold Th0, a new BSR is calculated based on the current BSR. For example, a scaling factor F0 (e.g., greater than 1) is applied to the current BSR to determine the new BSR. Then, the new BSR becomes the current BSR, and the iterative process enters the next iteration cycle. In some examples, the process of determining the final sampling rate for region (i) is determined as equation (2):
[0189]
[0190] Among them, SR i is the final sampling rate, Th0 is the threshold of the OR parameter, and F0>1 is the scaling factor for increasing the BSR. Therefore, when the OR parameter with BSR is less than the threshold Th0, the current region will only use BSR as the final sampling rate, otherwise the final sampling rate is changed by the scaling factor F0. This process can be iteratively performed until OR(SR i ) is less than the threshold Th0.
[0191] In some embodiments, the adaptive sampling module (1120) may use an algorithm to determine a final sampling rate (SR) for a specific region i (eg, i is a region index used to identify a specific region). i) that can implement an OR parameter within a specific range (e.g., less than a first predetermined threshold (denoted by Th0) and greater than a second predetermined threshold (denoted by Th1)). In some examples, the algorithm starts from an arbitrary basic sampling rate (BSR) and uses an iterative process with an iterative period to determine the final sampling rate. In each iterative period of testing the current BSR, the OR parameter of a specific region is determined. When the OR parameter is within the specific range (e.g., less than the first predetermined threshold Th0 and greater than the second predetermined threshold Th1), the current BSR is the final sampling rate of the specific region. However, when the OR parameter is greater than the first predetermined threshold Th0, a first scaling factor F0 (e.g., greater than 1) is applied to the current BSR to determine a new BSR; when the OR parameter is less than the second predetermined threshold Th1, a second scaling factor F1 (e.g., less than 1) is applied to the current BSR to determine a new BSR. Then, the new BSR becomes the current BSR, and the iterative process enters the next iterative period.
[0192] In some examples, the process of determining the final sampling step of region i is determined as equation (3):
[0193]
[0194] where BSR represents the basic sampling rate, Th0 represents the first predetermined threshold, Th1 represents the second predetermined threshold, F0>1 represents the first scaling factor for increasing the BSR, and 0<F1<1 represents the second scaling factor for decreasing the BSR.
[0195] It should be noted that for each region, the scaling factors F0 and F1 can be different. When the OR parameter with the BSR is within the specific range defined by the thresholds (e.g., Th0 and Th1), the current region can use the BSR as the final sampling rate. When the OR parameter with the BSR is equal to or greater than Th0, the final sampling rate is increased by the scaling factor F0. When the OR parameter with the BSR is equal to or less than Th1, the final sampling rate is decreased by the scaling factor F1. This process can be iteratively performed until OR(SR i ) is within the range defined by Th0 and Th1.
[0196] According to some aspects of the present disclosure, the adaptive sampling module (1120) can place regions with different sampling rates (or different sampling steps) in a single graph.
[0197] It should be noted that when applying the adaptive sampling rate, compared with the original UV atlas, or compared with the case of using a uniform sampling rate, the sizes of the sampling regions in the sampling graph can change at different ratios. The adaptive sampling module (1120) can place the sampling regions with different sampling rates in the sampling graph and keep the sampling regions non-overlapping with each other, so that each point in the sampling graph corresponds one-to-one with a specific region.
[0198] In some examples, the adaptive sampling module (1120) can determine a bounding box for each sampling area and then place the sampling area according to the bounding box. In one example, for a specific area in the original UV atlas, u min is the minimum u coordinate of all vertices in a particular region, v min is the minimum v coordinate of all vertices in the specific region. Based on the minimum u coordinate and the minimum v coordinate, the adaptive sampling module (1120) can determine a bounding box of the sampling region corresponding to the specific region. For example, the upper left corner of the bounding box of the sampling region can be placed at the position (u0, v0) in the sampling map, which can be obtained by and to calculate, where SR represents the sampling rate applied to a specific area when the same sampling rate is used along the u direction and the v direction, represents a ceiling function that determines the smallest integer greater than a value C.
[0199] In some embodiments, the adaptive sampling module (1120) may place the sampling regions one by one in the sampling map in a specific order. To place the current sampling region, the adaptive sampling module (1120) may first place the current sampling region according to the position (u0, v0) (e.g., the upper left corner of the bounding box of the current sampling region). When the adaptive sampling module (1120) detects that the current sampling region overlaps with the already placed sampling region, the adaptive sampling module (1120) may determine a new position to place the current sampling region to avoid overlapping with the previously placed sampling region.
[0200] In some examples, a specific search window and / or criteria may be defined, and the adaptive sampling module (1120) may find a new location for placing the current sampling area according to the specific search window and / or criteria (given by (u n , v n )). It should be noted that in some examples, the new position (u n , v n ) and the original position (u0, v0) (also called UV offset) can be sent from the encoder side to the decoder side with a signal for reconstruction.
[0201] In some embodiments, the sampling areas are not only placed so as not to overlap, but also have a certain amount of gap between the areas. For example, each sampling area may need to have a minimum distance of 10 pixels from other sampling areas. It should be noted that the spacing between sampling areas can be defined by various techniques. In some examples, the minimum distance between sampling areas can be defined by a minimum horizontal distance l0 and a minimum vertical distance l1.
[0202] Some aspects of the present disclosure also provide signaling techniques for adaptive sampling of grid compression.
[0203] In some embodiments, the sampling rates of different regions of the grid may be signaled in the codestream that carries the grid information. It should be noted that the sampling rates may be signaled at different levels in the codestream. In one example, the sampling rate may be signaled in a sequence header of the entire grid sequence that includes a grid frame sequence. In another example, the sampling rate may be signaled in a group header of a grid frame group (conceptually similar to a group of pictures (GOP)). In another example, the sampling rate may be signaled in a frame header of each grid frame. In another example, the sampling rates of slices in a grid frame are signaled in a slice header of a slice. In another example, the sampling rates of tiles in a grid frame are signaled in a tile header of a tile. In another example, the sampling rates of patches in a grid frame are signaled in a patch header of a patch.
[0204] Specifically, in some embodiments, a control flag may be signaled to indicate whether an adaptive sampling method is applied at different levels in a code stream. In one example, a control flag is signaled in a sequence header of an entire grid sequence. In another example, a control flag is signaled in a group header of a grid frame group. In another example, a control flag is signaled in a frame header of each grid frame. In another example, a control flag of a slice in a grid frame is signaled in a slice header of a slice. In another example, a control flag of a tile in a grid frame is signaled in a tile header of a tile. In another example, a control flag of a patch in a grid frame is signaled in a patch header of a patch. When a control flag at a certain level is true, adaptive sampling is enabled at that level, so that an adaptive sampling rate can be applied. When a control flag at a certain level is false, adaptive sampling is disabled, so that a uniform sampling rate is applied at that level.
[0205] In some examples, the control flag includes 1 bit and can be encoded by various techniques. In one example, the control flag is encoded by entropy coding with fixed probability or updated probability (such as arithmetic coding and Huffman coding). In another example, the control flag is encoded by a coding technique with lower complexity (called bypass coding).
[0206] In some examples, a base sampling rate can be signaled regardless of whether adaptive sampling is enabled. When adaptive sampling is enabled, the base sampling rate can be used as a predictor, and each region can signal a difference from the base sampling rate to indicate the actual sampling rate for that region. When adaptive sampling is disabled, the base sampling rate can be used as a uniform sampling rate throughout the content at the appropriate level.
[0207] The basic sampling rate may also be signaled at different levels in the codestream. In one example, the basic sampling rate may be signaled in a sequence header for the entire grid sequence. In another example, the basic sampling rate may be signaled in a group header for a grid frame group. In another example, the basic sampling rate may be signaled in a frame header for each grid frame. In another example, the basic sampling rate of a slice may be signaled in a slice header for a slice in a grid frame. In another example, the basic sampling rate of a tile in a grid frame may be signaled in a tile header for a tile. In another example, the basic sampling rate of a patch in a grid frame may be signaled in a patch header for a patch.
[0208] In some examples, the basic sampling rate can be binarized using a fixed-length or variable-length representation (e.g., a fixed k-bit representation and a k-ordered Exp-Golomb representation), and each bit can be encoded using entropy coding with fixed probability or updated probability (e.g., arithmetic coding and Huffman coding), or each bit can be encoded using bypass coding with lower complexity.
[0209] According to one aspect of the present disclosure, when adaptive sampling is enabled, the sampling rate of the regions in the grid frame is appropriately signaled. In some examples, the number of regions in the entire grid frame can be signaled or derived (e.g., as the number of CTU rows, the number of tiles, the number of patches, etc.).
[0210] According to one aspect of the present disclosure, the sampling rate of a region can be represented by a signal without prediction. In one example, the sampling rate of each region (or the entire 2D atlas) can be directly represented by a signal without prediction. In another example, the sampling rate of each region (or the entire 2D atlas) can be selected from a pre-established sampling rate set known to both the encoder and the decoder. The signaling of a specific sampling rate can be performed by using a signal to represent the index of a specific sampling rate in a pre-established rate set. For example, a pre-established sampling step set may include (every 2 pixels, every 4 pixels, every 8 pixels, etc.). Index 1 can be represented by a signal to indicate a sampling rate of every 2 pixels (e.g., 2:1); index 2 can be represented by a signal to indicate a sampling rate of every 4 pixels (e.g., 4:1); index 3 can be represented by a signal to indicate a sampling rate of every 8 pixels (e.g., 8:1).
[0211] According to another aspect of the present disclosure, the sampling rate of a region may be predicted. It should be noted that any suitable prediction technique may be used.
[0212] In one example, the sampling rate of each region of the grid frame (or the entire 2D atlas) can be predicted based on a pre-established rate set. In another example, the sampling rate of each region of the grid frame (or the entire 2D atlas) can be predicted based on sampling rates previously used in other encoded regions of the same frame. In another example, the sampling rate of each region of the grid frame (or the entire 2D atlas) can be predicted based on sampling rates previously used in other encoded grid frames.
[0213] According to another aspect of the present disclosure, the sampling rate of each region (or the entire 2D atlas) can be determined in a manner that allows prediction and direct signaling. In one example, the syntax can be constructed to indicate whether the sampling rate will be predicted or directly signaled. When the syntax indicates that the sampling rate is predicted, it is further signaled, such as by another syntax, which predictor is used to predict the sampling rate. When the syntax indicates that the sampling rate is directly signaled, the value of the sampling rate is signaled, such as by another syntax.
[0214] In some examples, when the sampling rate is directly signaled (by signaling the sampling rate or signaling an index to the sampling rate), the sampling rate or the index to the sampling rate may be binarized via a fixed-length or variable-length representation (e.g., a fixed k-bit representation and a k-order exponential Golomb representation), and each bit may be encoded via entropy coding with fixed probability or updated probability (e.g., arithmetic coding and Huffman coding), or each bit may be encoded via a bypass coding with lower complexity.
[0215] In some examples, when the sampling rate is signaled by prediction, the prediction residual can be signaled. The prediction residual can be binarized by a fixed length or variable length representation (e.g., a fixed k-bit representation and a k-order exponential Golomb representation), and each bit can be encoded by entropy coding with a fixed probability or an update probability (e.g., arithmetic coding and Huffman coding), or each bit can be encoded by bypass coding with a lower complexity. For example, the sign bit of the prediction residual can be encoded by bypass coding, and the absolute value of the prediction residual can be encoded by entropy coding with an update probability.
[0216] According to some aspects of the present disclosure, when adaptive sampling is enabled, an offset (also referred to as UV offset) of the UV coordinates of each region in the grid frame can be calculated in the code stream carrying the grid frame. u =u o -u n and offset v =v o -v nIn some examples, the number of regions in the entire grid frame can be signaled or derived (e.g., as the number of CTU rows, the number of tiles, the number of patches, etc.).
[0217] According to one aspect of the present disclosure, the UV offset of a region may be represented by a signal without prediction. In one example, the UV offset of each region may be represented by a signal directly without any prediction.
[0218] According to another aspect of the present disclosure, the UV offset of a region may be predicted. It should be noted that any suitable prediction technique may be used. In one example, the UV offset of each region may be predicted based on a pre-established set of UV offsets. In another example, the UV offset of each region may be predicted based on UV offsets previously used in other encoded regions of the same grid frame. In another example, the UV offset of each region may be predicted based on UV offsets previously used in other encoded grid frames.
[0219] According to another aspect of the present disclosure, the UV offset of each region may be signaled in a manner that allows prediction and direct signaling. In some examples, the syntax may be constructed to indicate whether the UV offset is to be predicted or directly signaled. When the syntax indicates that the UV offset is predicted, which predictor is used to predict the UV offset is further signaled by another syntax. When the syntax indicates that the UV offset is to be directly signaled, the value of the UV offset is signaled by another syntax.
[0220] In some examples, when the UV offset is directly represented by a signal, the value of the UV offset may be binarized by a fixed-length or variable-length representation (e.g., a fixed k-bit representation and a k-order exponential Golomb representation), and each bit may be encoded by entropy coding with a fixed probability or an updated probability (e.g., arithmetic coding and Huffman coding), or each bit may be encoded by a bypass coding with a lower complexity. In one example, the sign bit of the UV offset may be encoded by bypass coding, and the absolute value of the UV offset may be encoded by entropy coding with an updated probability.
[0221] In some examples, the sign bit of the UV offset can be inferred or predicted based on the value of the sampling rate. For example, when the sampling rate of the region is greater than the base sampling rate, the sign bit of the UV offset can be inferred or predicted to be positive; and when the sampling rate of the region is less than the base sampling rate, the sign bit of the UV offset can be inferred or predicted to be negative. If the sampling rate of the region is equal to the base sampling rate, the UV offset can be inferred or predicted to be zero.
[0222] In some examples, the UV offset may be predicted and the prediction residual may be represented by a signal. For example, the prediction residual may be binarized by a fixed length or variable length representation (e.g., a fixed k-bit representation and a k-order exponential Golomb representation), and each bit may be encoded by entropy coding (e.g., arithmetic coding and Huffman coding) with a fixed probability or an update probability, or each bit may be encoded by a bypass coding with a lower complexity. In one example, the sign bit of the prediction residual may be encoded by bypass coding, and the absolute value of the prediction residual may be encoded by entropy coding with an update probability.
[0223] Some aspects of the present disclosure also provide mesh reconstruction techniques for use at the decoder side.The mesh reconstruction techniques may be used in a mesh reconstruction module, such as the mesh reconstruction module (1190).
[0224] In some examples, the decoded map (1165) includes a decoded occupancy map, and the mesh reconstruction module (1190) can reconstruct the mesh frame based on the decoded map (1165) including the decoded occupancy map, the decoded geometry map, and one or more decoded property maps. It should be noted that in some examples, the decoded occupancy map corresponds to a sampled occupancy map, which can be the result of adaptive sampling using the same sampling rate as the decoded geometry map and the decoded property map, so that the decoded occupancy map includes a sampled area.
[0225] In some examples, the decoded map (1165) does not include an occupancy map, and the codestream (1145) includes information identifying boundary vertices of each region as auxiliary data encoded, for example, by an auxiliary data encoder (1140). The auxiliary data decoder (1170) may decode the information identifying the boundary vertices of the region. The mesh reconstruction module (1190) may infer regions for the inferred occupancy map based on the boundary vertices of the region. It should be noted that in some examples, the inferred occupancy map is not processed using adaptive sampling. The mesh reconstruction module (1190) may reconstruct a mesh frame based on the inferred occupancy map and the decoded map (1165) including the decoded geometry map and one or more decoded attribute maps.
[0226] According to one aspect of the present disclosure, the mesh reconstruction module (1190) can determine the UV coordinates of the vertices in the mesh frame based on the decoded map (1165) and the decoded auxiliary data (such as sampling rate, UV offset, information for identifying boundary vertices, etc.), and determine the 3D coordinates and attribute values of the vertices in the mesh frame.
[0227] In some embodiments, to determine the UV coordinates of the vertices in the region, the sampling rate (SR) of the region is determined based on syntax values decoded from the bitstream (1145). In some examples, the UV offset of the region is determined based on syntax values decoded from the bitstream (1145), such as (offsetu , offset v ).
[0228] In one example, the decoded map (1165) includes a decoded occupancy map. The mesh reconstruction module (1190) can determine the UV coordinates of the vertex corresponding to each occupied point in the sampled area of the decoded occupancy map. For example, for a vertex having coordinates (u i , v i ), the mesh reconstruction module (1190) may determine the UV coordinates (u ) of the vertices corresponding to the occupied points according to equations (4) and (5). i , v i ):
[0229] U i =(u i +offset u )×SR Equation (4)
[0230] V i =(v i +offset v )×SR Equation (5)
[0231] In another example, the decoded map (1165) does not include a decoded occupancy map. The mesh reconstruction module (1190) can determine the inferred region based on the boundary vertices of the inferred region in the inferred occupancy map. The UV coordinates of the vertices corresponding to the occupied points in the inferred region can be directly inferred by the positions of the occupied points in the region of the inferred occupancy map. For example, the occupied points in the inferred occupancy map have a UV coordinate (UV coordinate) of s , V s ) limited position, specific UV coordinates (U s , V s ) are the UV coordinates of the corresponding vertex in the mesh frame.
[0232] In some embodiments, for each occupied point on an occupancy map (e.g., a decoded occupancy map, an inferred occupancy map), a mesh reconstruction module (1190) may recover vertices on a mesh frame, and may determine corresponding geometry values (e.g., 3D coordinates) and attribute values based on corresponding positions in a decoded geometry map and a decoded attribute map.
[0233] In some embodiments, to derive the corresponding positions of vertices in the region in the geometry graph and the attribute graph, the sampling rate (SR) of the region is determined based on the syntax values decoded from the bitstream (1145). In some examples, the UV offset of the region is determined based on the syntax values decoded from the bitstream (1145), such as (offset u , offset v ).
[0234] In one example, the decoded map (1165) includes a decoded occupancy map. For a sample region in the decoded occupancy map having coordinates (u i , v i ) can be directly obtained from each occupied point of (u i , v i ) derives corresponding positions in the decoded geometry map and the decoded attribute map. For example, the sampling rate applied to the occupancy map to obtain the sampled occupancy map is consistent with the sampling rate applied to the geometry map (to obtain the sampled geometry map) and the sampling rate applied to the attribute map (to obtain the sampled attribute map). The decoded occupancy map corresponds to the sampled occupancy map, the decoded geometry map corresponds to the sampled geometry map, and the decoded attribute map corresponds to the sampled attribute map. Thus, occupied points in the decoded occupancy map may have corresponding points in the decoded geometry map and the decoded attribute map with the same coordinates.
[0235] In another example, the decoded map (1165) does not include a decoded occupancy map. The mesh reconstruction module (1190) can determine the inferred region based on the boundary vertices of the inferred region in the inferred occupancy map. i , V i ), the corresponding positions in the geometric graph and the attribute graph can be derived as Equation (6) and Equation (7):
[0236]
[0237] In some embodiments, the mesh reconstruction module (1190) may infer connectivity information between vertices by inferring from occupied positions. In some embodiments, connectivity information may be explicitly signaled in the bitstream (1145).
[0238] Fig.14 A flow chart outlining a process (1400) according to one embodiment of the present disclosure is shown. The process (1400) may be used in an encoding process of a grid. In various embodiments, the process (1400) is performed by a processing circuit. In some embodiments, the process (1400) is implemented in software instructions, so when the processing circuit executes the software instructions, the processing circuit performs the process (1400). The process starts at (S1401) and proceeds to (S1410).
[0239] At (S1410), a data structure of a mesh frame is received, the mesh frame having polygons representing a surface of an object. The data structure of the mesh frame includes a UV atlas, the UV atlas associating vertices of the mesh frame with UV coordinates in the UV atlas.
[0240] At (S1420), sampling rates of the plurality of regions of the grid frame are determined according to the characteristics of the plurality of regions of the grid frame.
[0241] At (S1430), the sampling rates of the respective regions of the grid frame are applied on the UV atlas to determine a plurality of sampling positions on the UV atlas.
[0242] At (S1440), one or more sampling two-dimensional (2D) maps are generated for the mesh frame according to the plurality of sampling positions on the UV atlas.
[0243] At (S1450), one or more sampled 2D pictures are encoded into a code stream.
[0244] To determine the sampling rates of each of the plurality of regions of the mesh frame, in some examples, a first sampling rate for the first region is determined based on a requirement to limit the number of first sampling positions in the first region of the mesh frame. In one example, the first sampling rate is determined to achieve an oversampling ratio (OR) that is closest to a target value. The OR is a ratio of the number of first sampling positions to the number of vertices initially located in the first region of the mesh frame.
[0245] In another example, the first sampling rate is initialized to be relatively small, and the first sampling rate is adjusted until the OR is less than a first threshold. The OR is a ratio of the number of first sampling positions to the number of vertices initially located in the first region of the mesh frame.
[0246] In another example, the first sampling rate is adjusted until an oversampling ratio (OR) is less than a first threshold and greater than a second threshold. The OR is a ratio of the number of first sampling positions to the number of vertices initially located in the first region of the mesh frame.
[0247] In some examples, a sampling 2D map in the one or more sampling 2D maps includes a sampling region having a plurality of sampling points corresponding to a plurality of sampling positions on the UV atlas. In one example, when a sampling position is located in a polygon, the sampling position is determined to be occupied. Then, based on the attributes of the vertices of the polygon, the attributes of the sampling points in the sampling 2D map corresponding to the sampling position are determined.
[0248] In some examples, to form one or more sampled 2D maps, a plurality of sampling regions corresponding to the plurality of regions of the grid may be determined based on the respective sampling rates of the plurality of regions of the grid. The plurality of sampling regions are then arranged in a non-overlapping configuration to form a sampled map. In one example, the plurality of sampling regions are arranged one after the other. In one example, a bounding box of a sampling region is determined and the sampling region is arranged to avoid one or more corners of the bounding box overlapping with other arranged sampling regions.
[0249] For example, in order to arrange multiple sampling areas in a non-overlapping configuration, after arranging a subset of the multiple sampling areas in a non-overlapping configuration, an initial arrangement position is determined for a current sampling area according to a sampling rate of the current sampling area, and then, it is determined whether the current sampling area at the initial placement position will overlap with the subset of the multiple sampling areas.
[0250] In some examples, in response to the current sampling region at the initial placement position overlapping with a subset of the plurality of sampling regions, an offset of the initial placement position is determined, the offset allowing the current sampling region to not overlap with the subset of the plurality of sampling regions.
[0251] In some examples, the non-overlapping configuration includes a minimum distance requirement between the plurality of sampling regions.
[0252] In some embodiments, the sampling rates respectively associated with multiple regions are encoded by various techniques. In one example, the value of the first sampling rate of the first region is directly encoded into the codestream. In another example, a first index is encoded into the codestream, and the first index indicates that the first sampling rate is selected from a sampling rate set. In another example, a syntax is encoded into the codestream, and the syntax indicates that a predictor predicts the first sampling rate from a pre-established sampling rate set. In another example, the syntax is encoded into the codestream, and the syntax indicates that the predictor predicts the first sampling rate based on the sampling rate previously used in the encoded region of the grid frame. In another example, the syntax is encoded into the codestream, and the syntax indicates that the predictor predicts the first sampling rate based on the sampling rate previously used in the encoded region of another grid frame encoded before the grid frame.
[0253] In some embodiments, the encoder side may make a decision to signal a first sampling rate associated with the first region or predict the first sampling rate, and encode a first syntax value indicating the decision. In one example, in response to the decision to signal the first sampling rate, the value of the first sampling rate is directly encoded into the bitstream. In another example, in response to the decision to signal the first sampling rate, an index is encoded into the bitstream, the index indicating that the first sampling rate is selected from the sampling rate set.
[0254] In some examples, in response to the decision to predict the first sampling rate, a second syntax is encoded into the bitstream. The second syntax indicates using a predictor to predict the first sampling rate. In addition, in one example, a prediction residual is encoded into the bitstream. The prediction residual is the difference between the first sampling rate and the sampling rate of the predictor.
[0255] In some examples, a base sampling rate is encoded into the bitstream. The base sampling rate can be encoded at any suitable level, whether or not adaptive sampling is used. In one example, the base sampling rate can be used as a predictor.
[0256] In some examples, a control flag is encoded that indicates when adaptive 2D atlas sampling is used on a grid frame.
[0257] In some examples, a first UV offset associated with a first region of a grid frame is determined. The first UV offset is applied to a first sampling region corresponding to the first region to avoid overlapping with other sampling regions. One or more syntaxes may be encoded into a bitstream, the one or more syntaxes indicating the first UV offset. In one example, the syntax is directly encoded into the bitstream, the syntax having a value of the first UV offset. In another example, the syntax is encoded into the bitstream, the syntax indicating a predictor to predict the first UV offset based on a pre-established set of UV offsets. In another example, the syntax is encoded into the bitstream, the syntax indicating a predictor to predict the first UV offset based on a UV offset previously used for an encoded region of the grid frame. In another example, the syntax is encoded into the bitstream, the syntax indicating a predictor to predict the first UV offset based on a UV offset previously used for an encoded region in another grid frame encoded prior to the grid frame.
[0258] In some examples, syntax indicating that the predictor predicts the first UV offset in the codestream is encoded into the codestream, and a prediction residual is encoded into the codestream, the prediction residual being the difference between the first UV offset and the UV offset of the predictor.
[0259] Then, the process proceeds to (S1499) and ends.
[0260] The process (1400) may be adapted as appropriate. Steps in the process (1400) may be modified and / or omitted. Additional steps may be added. Any suitable order of implementation may be used.
[0261] Fig.15 A flow chart outlining a process (1500) according to one embodiment of the present disclosure is shown. The process (1500) may be used in a decoding process of a trellis. In various embodiments, the process (1500) is performed by a processing circuit. In some embodiments, the process (1500) is implemented in software instructions, so when the processing circuit executes the software instructions, the processing circuit performs the process (1500). The process starts at (S1501) and proceeds to (S1510).
[0262] At (S1510), a plurality of 2D graphs are decoded from a code stream carrying a mesh frame. The mesh frame uses polygons to represent the surface of an object, and the plurality of 2D graphs at least include a decoded geometric graph and a decoded attribute graph, wherein adaptive 2D atlas sampling is applied to the decoded geometric graph and the decoded attribute graph.
[0263] At (S1520), at least a first sampling rate and a second sampling rate are determined according to syntax in the code stream. The first sampling rate is applied to a first region of the grid frame during adaptive 2D atlas sampling, and the second sampling rate is applied to a second region of the grid frame during adaptive 2D atlas sampling. The first sampling rate is different from the second sampling rate.
[0264] At (S1530), based on the plurality of graphs, at least first vertices of the mesh frame are reconstructed according to a first sampling rate, and second vertices of the mesh frame are reconstructed according to a second sampling rate.
[0265] In some examples, the plurality of graphs include a decoded occupancy map to which adaptive 2D atlas sampling is applied. To reconstruct at least a first vertex according to a first sampling rate, an initial UV coordinate of an occupied point in a first sampling region of the decoded occupancy map is determined, the first sampling region corresponding to a first region of the mesh frame, and the occupied point corresponding to the first vertex. Then, a restored UV coordinate of the first vertex is determined based on the initial UV coordinate and the first sampling rate. In one example, a first UV offset for the first sampling region is determined based on a syntax from a bitstream. The restored UV coordinate of the first vertex is determined based on the initial UV coordinate, the first sampling rate, and the first UV offset. Further, in one example, a restored 3D coordinate of the first vertex is determined based on a pixel in the decoded geometry map located at the initial UV coordinate, and a restored attribute value of the first vertex is determined based on a pixel in the decoded attribute map located at the initial UV coordinate.
[0266] In some embodiments, the plurality of graphs lack an occupancy map. To reconstruct at least the first vertex according to the first sampling rate, information indicating a first boundary vertex of the first region is decoded from the bitstream. A first occupied region corresponding to the first region in the occupancy map is inferred according to the first boundary vertex. UV coordinates of an occupied point in the first occupied region are obtained, and the occupied point may correspond to the first vertex. The UV coordinates are converted into sampled UV coordinates according to at least the first sampling rate. Based on the plurality of graphs, the first vertex is reconstructed according to the sampled UV coordinates.
[0267] In some examples, to reconstruct the first vertex, a restored 3D coordinate of the first vertex is determined based on the pixel at the sampled UV coordinate in the decoded geometry map, and a restored attribute value of the first vertex is determined based on the pixel at the sampled UV coordinate in the decoded attribute map.
[0268] In some examples, to convert the UV coordinates into sampled UV coordinates, a first UV offset associated with the first region is decoded from the bitstream, and the UV coordinates are converted into sampled UV coordinates according to the first sampling rate and the first UV offset.
[0269] In some embodiments, at least the first sampling rate and the second sampling rate may be determined by various techniques. In one example, the values of at least the first sampling rate and the second sampling rate are decoded directly from the code stream. In another example, at least a first index and a second index are decoded from the code stream, the first index indicating that the first sampling rate is selected from a sampling rate set, and the second index indicating that the second sampling rate is selected from a sampling rate set. In another example, the first sampling rate is predicted based on a pre-established rate set. In another example, the first sampling rate is predicted based on a sampling rate previously used in a decoded area of a grid frame. In another example, the first sampling rate is predicted based on a sampling rate previously used in a decoded area in another grid frame decoded before the grid frame.
[0270] In some embodiments, to determine the first sampling rate, a first syntax value is decoded from the codestream, the first syntax value indicating whether the first sampling rate is signaled or predicted. In one example, in response to the first syntax value indicating that the first sampling rate is signaled, the value of the first sampling rate is decoded directly from the codestream. In another example, in response to the first syntax value indicating that the first sampling rate is signaled, an index is decoded from the codestream. The index indicates that the first sampling rate is selected from a set of sampling rates.
[0271] In some examples, in response to the first syntax value indicating a predicted first sampling rate, a second syntax is decoded from the code stream, the second syntax indicating the use of a predictor to predict the first sampling rate. In addition, in one example, a prediction residual is determined based on the syntax value decoded from the code stream, and the first sampling rate is determined based on the predictor and the prediction residual.
[0272] In some examples, a base sampling rate is decoded from the bitstream, and at least the first sampling rate and the second sampling rate can be determined based on the base sampling rate. For example, the base sampling rate is used as a predictor.
[0273] In some examples, a control flag is decoded from a bitstream, the control flag indicating application of adaptive 2D atlas sampling on a grid frame. Then, the number of regions in the grid frame and the sampling rates for the regions are determined, for example, based on syntax in the bitstream.
[0274] In some examples, a first UV offset associated with the first region is determined based on the codestream, and a first vertex of the mesh frame is reconstructed based on the first sampling rate and the first UV offset based on the plurality of graphs. In one example, to determine the first UV offset associated with the first region, a value of the first UV offset is decoded directly from the codestream. In another example, the first UV offset is predicted based on a pre-established set of UV offsets. In another example, the first UV offset is predicted based on a UV offset previously used by a decoded region of the mesh frame. In another example, the first UV offset is predicted based on a UV offset previously used by a decoded region in another mesh frame decoded before the mesh frame.
[0275] In some embodiments, to determine a first UV offset associated with the first region, a first syntax value is decoded from the codestream, the first syntax value indicating whether the first UV offset is signaled or predicted from the codestream. In one example, in response to the first syntax value indicating that the first UV offset is signaled, the value of the first UV offset is decoded directly from the codestream. In one example, the sign of the first UV offset is inferred based on a comparison of the first sampling rate with the base sampling rate.
[0276] In some examples, in response to a first syntax value indicating a predicted first UV offset, a second syntax is decoded from a codestream, the second syntax indicating the use of a predictor to predict the first UV offset. In addition, a prediction residual is determined based on the syntax value decoded from the codestream, and the first UV offset is determined based on the predictor and the prediction residual.
[0277] The process then proceeds to (S1599) and ends.
[0278] The process (1500) may be adapted as appropriate. Steps in the process (1500) may be modified and / or omitted. Additional steps may be added. Any suitable order of implementation may be used.
[0279] The techniques disclosed in this disclosure may be used alone or in any order. In addition, each of the techniques (e.g., methods, embodiments), encoders, and decoders may be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-transitory computer-readable medium.
[0280] The above techniques may be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media. Fig.16 A computer system (1600) suitable for implementing certain embodiments of the disclosed subject matter is shown.
[0281] Computer software may be encoded using any suitable machine code or computer language that may be subjected to assembly, compilation, linking or similar mechanisms to create code comprising instructions that may be executed directly by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., or through interpretation, microcode execution, etc.
[0282] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablets, servers, smartphones, gaming devices, IoT devices, etc.
[0283] Fig.16The components of the computer system (1600) shown are exemplary in nature and are not intended to suggest any limitation on the scope of use or functionality of computer software implementing embodiments of the present disclosure. Neither should the configuration of components be interpreted as having any dependency or requirement relating to any one or combination of components shown in the exemplary embodiment of the computer system (1600).
[0284] The computer system (1600) may include certain human interface input devices. Such human interface input devices may be responsive to input from one or more human users through, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), olfactory input (not depicted). Human interface devices may also be used to capture certain media that are not necessarily directly related to human conscious input, such as audio (e.g., voice, music, ambient sounds), images (e.g., scanned images, captured images obtained from a still image camera), and videos (e.g., two-dimensional video, three-dimensional video including stereoscopic video).
[0285] The human-machine interface input device may include one or more of the following (only one of each is shown): keyboard (1601), mouse (1602), touchpad (1603), touch screen (1610), data gloves (not shown), joystick (1605), microphone (1606), scanner (1607), camera (1608).
[0286] The computer system (1600) may also include certain human-computer interface output devices. Such human-computer interface output devices may stimulate one or more human user senses through, for example, tactile output, sound, light, and smell / taste. Such human-computer interface output devices may include tactile output devices (e.g., tactile feedback of a touch screen (1610), a data glove (not shown), or a joystick (1605), but may also be a tactile feedback device that is not an input device), audio output devices (e.g., speakers (1609), headphones (not depicted)), visual output devices (e.g., screens (1610) including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capabilities, each with or without tactile feedback capabilities, some of which are capable of outputting two-dimensional visual outputs or outputs in more than three dimensions through devices such as stereoscopic image output, virtual reality glasses (not depicted), holographic displays and smoke boxes (not depicted), and printers (not depicted).
[0287] The computer system (1600) may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW (1620) with CD / DVD etc. media (1621), thumb drives (1622), removable hard drives or solid-state drives (1623), traditional magnetic media such as tapes and floppy disks (not depicted), dedicated ROM / ASIC / PLD based devices such as security dongles (not depicted), and the like.
[0288] Those skilled in the art should also understand that the term "computer-readable media" used in connection with the presently disclosed subject matter does not encompass transmission media, carrier waves, or other transitory signals.
[0289] The computer system (1600) may also include an interface (1654) to one or more communication networks (1655). The network may be, for example, a wireless network, a wired network, an optical network. The network may further be a local network, a wide area network, a metropolitan area network, a vehicle and industrial network, a real-time network, a delay-tolerant network, etc. Examples of networks include local area networks such as Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., television wired or wireless wide area digital networks including cable television, satellite television, and terrestrial broadcast television, vehicle and industrial networks including CANBus, etc. Some networks typically require an external network interface adapter attached to some common data port or peripheral bus (1649) (e.g., a USB port of the computer system (1600)); as described below, other network interfaces are typically integrated into the kernel of the computer system (1600) by attaching to the system bus (e.g., an Ethernet interface connected to a PC computer system or a cellular network interface connected to a smartphone computer system). The computer system (1600) can use any of these networks to communicate with other entities. Such communications may be one-way receive only (e.g., broadcast television), one-way send only (e.g., a CANBus connected to certain CANBus devices), or bidirectional, e.g., connecting to other computer systems using a LAN or WAN digital network. As described above, certain protocols and protocol stacks may be used on each of those networks and network interfaces.
[0290] The above-mentioned human interface device, human-accessible storage device, and network interface may be attached to the kernel (1640) of the computer system (1600).
[0291] The kernel (1640) may include one or more central processing units (CPUs) (1641), graphics processing units (GPUs) (1642), dedicated programmable processing units in the form of field programmable gate arrays (FPGAs) (1643), hardware accelerators (1644) for certain tasks, graphics adapters (1650), and the like. These devices, as well as read-only memory (ROM) (1645), random access memory (1646), internal mass storage (1647) such as internal non-user accessible hard drives, SSDs, and the like, may be connected via a system bus (1648). In some computer systems, the system bus (1648) may be accessed in the form of one or more physical plugs to enable expansion by additional CPUs, GPUs, and the like. Peripheral devices may be attached directly to the kernel's system bus (1648) or to the kernel's system bus (1648) via a peripheral bus (1649). In one example, a screen (1610) may be connected to a graphics adapter (1650). The architecture of the peripheral bus includes PCI, USB, and the like.
[0292] The CPU (1641), GPU (1642), FPGA (1643) and accelerator (1644) can execute certain instructions, which can be combined to form the above-mentioned computer code. The computer code can be stored in ROM (1645) or RAM (1646). Transitional data can also be stored in RAM (1646), while permanent data can be stored, for example, in internal mass storage (1647). Fast storage and retrieval to any storage device can be performed by using a cache, which can be closely associated with the following: one or more CPUs (1641), GPUs (1642), mass storage (1647), ROM (1645), RAM (1646), etc.
[0293] The computer readable medium may have thereon computer codes for performing various computer-implemented operations. The medium and computer codes may be those specially designed and constructed for the purposes of the present disclosure, or the medium and computer codes may be of a type well known and available to those skilled in the art of computer software.
[0294] As an example, and not a limitation, a computer system having an architecture (1600), particularly a kernel (1640), may provide functionality due to one or more processors (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software contained in one or more tangible computer-readable media. Such computer-readable media may be media associated with user-accessible mass storage as described above, as well as memories of certain non-temporary kernels (1640), such as kernel internal mass storage (1647) or ROM (1645). Software implementing various embodiments of the present disclosure may be stored in such devices and executed by the kernel (1640). Depending on specific needs, the computer-readable medium may include one or more storage devices or chips. The software may cause the kernel (1640), particularly the processors therein (including CPUs, GPUs, FPGAs, etc.), to perform specific processes or specific parts of specific processes described herein, including defining data structures stored in RAM (1646) and modifying such data structures according to processes defined by the software. Additionally or alternatively, the computer system may provide functionality due to logic hardwired or otherwise embodied in circuitry (e.g., accelerator (1644)) that may replace software or operate in conjunction with software to perform specific processes or specific portions of specific processes described herein. Where appropriate, references to portions of software may include logic and vice versa. Where appropriate, references to portions of computer-readable media may include circuitry (e.g., integrated circuits (ICs)) storing software for execution, circuitry embodying logic for execution, or both. The present disclosure includes any suitable combination of hardware and software.
[0295] Although the present disclosure has described multiple exemplary embodiments, there are modifications, permutations, and various replacement equivalents that fall within the scope of the present disclosure. Therefore, it should be understood that those skilled in the art will be able to design many systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and therefore fall within the spirit and scope of the present disclosure.
Claims
1. A method for encoding a grid, characterized in that include: Receiving a data structure of a mesh frame, the mesh frame having polygons representing a surface of an object, the data structure of the mesh frame comprising a UV atlas, the UV atlas comprising: UV coordinates and a mapping between vertices of the mesh frame and the UV coordinates, wherein the UV coordinates are coordinates in a two-dimensional domain; Determining sampling rates of the plurality of regions of the grid frame according to respective characteristics of the plurality of regions of the grid frame, wherein the characteristics include information density of the plurality of regions; Applying the respective sampling rates of the plurality of regions of the grid frame on the UV atlas to determine a plurality of sampling positions on the UV atlas; generating one or more sample two-dimensional (2D) maps for the grid frame according to the plurality of sample positions on the UV atlas; and encoding the one or more sampled 2D images into a bitstream; making a decision to signal a first sampling rate associated with a first region or predict the first sampling rate, encoding a first syntax value indicating the decision; In response to the decision to predict the first sampling rate, a second syntax is encoded into the codestream, the second syntax indicating the use of a predictor to predict the first sampling rate.
2. The method according to claim 1, characterized in that: Determining the sampling rates of the plurality of regions of the grid frame comprises: A first sampling rate of the first area of the grid frame is determined according to a requirement of limiting the number of first sampling positions in the first area of the grid frame.
3. The method according to claim 2, characterized in that Determining a first sampling rate of the first area of the grid frame includes: The first sampling rate is determined to achieve an oversampling ratio (OR) that is closest to a target value, wherein the OR is a ratio of the number of the first sampling positions to the number of vertices initially located in the first region of the mesh frame.
4. The method according to claim 2, characterized in that Determining a first sampling rate of the first area of the grid frame includes: The first sampling rate is adjusted until an oversampling ratio (OR) is less than a first threshold, wherein the OR is a ratio of the number of the first sampling positions to the number of vertices initially located in the first region of the mesh frame.
5. The method according to claim 2, characterized in that: Determining a first sampling rate of the first area of the grid frame includes: The first sampling rate is adjusted until an oversampling ratio (OR) is less than a first threshold and greater than a second threshold, wherein the OR is a ratio of the number of the first sampling positions to the number of vertices initially located in the first region of the mesh frame.
6. The method according to claim 1, characterized in that A sampling 2D map in the one or more sampling 2D maps includes a sampling area having a plurality of sampling points corresponding to the plurality of sampling positions on the UV atlas, and after determining the plurality of sampling positions on the UV atlas, the method further includes: In response to a sampling position being within the polygon, determining that the sampling position is occupied; and According to the attributes of the vertices of the polygon, the attributes of the sampling points in the sampling 2D graph corresponding to the sampling positions are determined.
7. The method according to claim 1, characterized in that The generating one or more sampled 2D graphs further comprises: forming a plurality of sampling regions corresponding to the plurality of regions of the grid frame based on respective sampling rates of the plurality of regions of the grid frame; and The plurality of sampling regions are arranged in a non-overlapping configuration to form a sampling map.
8. The method according to claim 7, characterized in that Arranging the plurality of sampling areas in a non-overlapping configuration further comprises: After arranging the subset of the plurality of sampling regions in the non-overlapping configuration, For a current sampling area, determining an initial arrangement position according to a sampling rate of the current sampling area; and It is determined whether the current sampling region at the initial arrangement position overlaps with a subset of the plurality of sampling regions.
9. The method according to claim 8, characterized in that Further including: In response to the current sampling region at the initial arrangement position overlapping with a subset of the plurality of sampling regions, an offset of the initial arrangement position is determined, the offset allowing the current sampling region to not overlap with the subset of the plurality of sampling regions.
10. The method according to claim 7, characterized in that The non-overlapping configuration includes a minimum distance requirement between the plurality of sampling regions.
11. The method according to any one of claims 1 to 10, characterized in that Further includes at least one of the following: directly encoding the value of the first sampling rate of the first region into the bitstream; Encoding a first index into the bitstream, the first index indicating that the first sampling rate is selected from a sampling rate set; encoding syntax that instructs a predictor to predict the first sampling rate from a pre-established set of sampling rates; encoding syntax that instructs a predictor to predict the first sampling rate based on sampling rates previously used by an encoded region of the grid frame; as well as Syntax is encoded that instructs a predictor to predict the first sampling rate based on a sampling rate previously used for an encoded region in another trellis frame that was encoded prior to the trellis frame.
12. The method according to any one of claims 1 to 10, characterized in that In response to the determining to signal the first sampling rate, the method includes at least one of: directly encoding the value of the first sampling rate into the bitstream; and An index is encoded into the code stream, the index indicating that the first sampling rate is selected from a sampling rate set.
13. The method according to any one of claims 1 to 10, characterized in that Further including: A prediction residual is encoded into the bitstream, wherein the prediction residual is the difference between the first sampling rate and the sampling rate of the predictor.
14. The method according to any one of claims 1 to 10, characterized in that Further including: A base sampling rate is encoded into the bitstream.
15. The method according to any one of claims 1 to 10, characterized in that Further including: Encodes a control flag indicating the enablement or disablement of adaptive 2D atlas sampling.
16. The method according to any one of claims 1 to 10, characterized in that Further including: determining at least a first UV offset associated with a first region of the grid frame, the first UV offset being applied to a first sampling region corresponding to the first region to avoid overlapping with other sampling regions; as well as A syntax indicating the first UV offset is encoded into the codestream.
17. The method according to claim 16, characterized in that Further including: Encode at least one of the following syntaxes: encoding a syntax into the bitstream, the syntax having a value of the first UV offset; encoding a syntax that instructs a predictor to predict the first UV offset based on a pre-established set of UV offsets; encoding syntax that instructs a predictor to predict the first UV offset based on a previously used UV offset for an already encoded region of the grid frame; as well as Syntax is encoded that instructs a predictor to predict the first UV offset based on a previously used UV offset for an encoded region in another grid frame that was encoded prior to the grid frame.
18. A device for encoding a grid, characterized in that: comprising a processing circuit configured to perform a method according to any one of claims 1 to 17.
