Improved attribute layer and signaling in point cloud coding

Abstract

To provide an encoder, a decoder, and a method implemented by them.SOLUTION: A method for encoding a point cloud coding (PCC) video sequence includes receiving a bitstream comprising a plurality of coded sequences of PCC frames. The plurality of coded sequences of PCC frames represent a plurality of PCC attributes including geometry, texture, and one or more of reflectance, transparency, and normal. Each coded PCC frame is represented by one or more PCC network abstraction layer (NAL) units. The method also includes: parsing the bitstream to obtain, for each PCC attribute, an indication of one of a plurality of video coder / decoders (codecs) used to code the corresponding PCC attribute; and decoding the bitstream based on the indicated video codecs for the PCC attributes.SELECTED DRAWING: Figure 11

Description

Technical Field

[0001] This patent application claims the benefit of U.S. Provisional Patent Application No. 62 / 731,693, titled "High-Level Syntax Designs for Point Cloud Coding," filed on September 14, 2018, by Ye-Kui Wang et al., which is incorporated herein by reference.

[0002] The present disclosure generally relates to video coding, and more particularly to coding video attributes for point cloud coding (PCC) video frames.

Background Art

[0003] The amount of video data required to represent relatively short videos is substantial and can pose difficulties when the data is streamed or communicated over a communication network with a limited bandwidth capacity in other ways. Thus, video data is generally compressed before being communicated over modern telecommunications networks. Also, the size of the video can be a problem when the video is stored on a storage device because memory resources may be limited. Video compression devices often use software and / or hardware at the source to encode the video data before transmission or storage, thereby reducing the amount of data required to represent the digital video image. The compressed data is then received at the destination by a video decompression device that decodes the video data. Improved compression and decompression techniques that improve the compression ratio without sacrificing much of the image quality are desirable because network resources are limited and the demand for higher video quality is constantly increasing.

Summary of the Invention

[0004] According to one embodiment, the disclosure includes a method implemented by a video decoder. The method includes a receiver receiving a bitstream containing a coded sequence of multiple point cloud coding (PCC) frames, wherein the coded sequence of multiple PCC frames represents multiple PCC attributes, including geometric shape, texture, and one or more of reflectivity, transparency, and normals, and each coded PCC frame is represented by one or more PCC network abstraction layer (NAL) units. The method further includes a processor parsing the bitstream to obtain instructions for each PCC attribute to one of multiple video codeder decoders (codecs) used to code the corresponding PCC attribute. The method further includes a processor decoding the bitstream based on the indicated video codecs for the PCC attributes. In some video coding systems, the entire sequence of PCC frames is coded using a single codec. A PCC frame may contain multiple PCC attributes. Some video codecs may be more efficient at coding some PCC attributes than others. This embodiment enables different video codecs to encode different PCC attributes for the same sequence of PCC frames. This embodiment also provides various syntactic elements to support coding flexibility when PCC frames in a sequence use multiple PCC attributes (e.g., three or more). By providing more attributes, the encoder can encode more complex PCC frames. Furthermore, the decoder can decode and display more complex PCC frames. Additionally, by enabling different codecs to employ different attributes, the coding process can be optimized based on codec selection. This can reduce processor resource usage in both the encoder and decoder.Furthermore, this can support increased compression and coding efficiency, which reduces memory and network resource usage while transmitting bitstreams between the encoder and decoder.

[0005] Optionally, in any of the embodiments described above, another embodiment of the embodiment provides that each sequence of PCC frames is associated with a sequence-level data unit containing sequence-level parameters, the sequence-level data unit containing a first syntactic element indicating that a first attribute was coded by a first video codec and a second attribute was coded by a second video codec.

[0006] Optionally, in any of the embodiments described above, another embodiment of the embodiment provides that the first syntactic element is an identified_codec_for_attribute element included in a group of frame headers in a bitstream.

[0007] Optionally, in any of the embodiments described above, another embodiment of the embodiment provides that the first attribute is organized into multiple streams, and the second syntactic element indicates stream membership to the data unit of the bitstream associated with the first attribute.

[0008] Optionally, in any of the embodiments described above, another embodiment of the embodiment provides that the first attribute is organized into multiple layers, and the third syntactic element indicates layer membership to the bitstream data unit associated with the first attribute.

[0009] Optionally, in any of the embodiments described above, another embodiment of the embodiment provides that the second syntactic element is a num_streams_for_attribute element included in a group of frame headers in the bitstream, and the third syntactic element is a num_layers_for_attribute element included in a group of frame headers in the bitstream.

[0010] Optionally, in any of the embodiments described above, another embodiment of the embodiment provides that the fourth syntactic element indicates that the first layer of a plurality of layers contains data associated with an irregular point cloud.

[0011] Optionally, in any of the embodiments described above, another embodiment of the embodiment provides that the fourth syntactic element is a regular_points_flag element included in a group of frame headers in the bitstream.

[0012] Optionally, in any of the embodiments described above, another embodiment of the embodiment provides that the bitstream is decoded into a decoded sequence of PCC frames, and further includes a processor transferring the decoded sequence of PCC frames toward a display for presentation.

[0013] According to one embodiment, the disclosure includes a method implemented in a video encoder. The method includes a processor encoding a plurality of PCC attributes of a sequence of PCC frames into a bitstream using a plurality of codeder decoders (codecs), wherein the plurality of PCC attributes include geometric shape, texture, and one or more of reflectance, transmittance, and normals, and each encoded PCC frame is represented by one or more PCC network abstraction layer (NAL) units. The method further includes the processor encoding, for each PCC attribute, an instruction for one of the video codecs used to encode the corresponding PCC attribute. The method further includes a transmitter transmitting the bitstream to the decoder. In some video coding systems, the entire sequence of PCC frames is encoded using a single codec. A PCC frame may contain multiple PCC attributes. Some video codecs may be more efficient at encoding some PCC attributes than others. This embodiment allows different video codecs to encode different PCC attributes for the same sequence of PCC frames. This embodiment also provides various syntactic elements to support coding flexibility when PCC frames in a sequence use multiple PCC attributes (e.g., three or more). By providing more attributes, the encoder can encode more complex PCC frames. Furthermore, the decoder can decode and display more complex PCC frames. In addition, by allowing different codecs to be employed for different attributes, the coding process can be optimized based on codec selection. This can reduce the use of processor resources in both the encoder and decoder. Furthermore, this can support increased compression and coding efficiency, which reduces memory and network resource usage while transmitting bitstreams between the encoder and decoder.

[0014] Optionally, in any of the embodiments described above, another embodiment of the embodiment provides that each sequence of PCC frames is associated with a sequence-level data unit containing sequence-level parameters, the sequence-level data unit containing a first syntactic element indicating that a first PCC attribute was encoded by a first video codec and a second PCC attribute was encoded by a second video codec.

[0015] Optionally, in any of the embodiments described above, another embodiment of the embodiment provides that the first syntactic element is an identified_codec_for_attribute element included in a group of frame headers in a bitstream.

[0016] Optionally, in any of the embodiments described above, another embodiment of the embodiment provides that the first attribute is organized into multiple streams, and the second syntactic element indicates stream membership to the data unit of the bitstream associated with the first attribute.

[0017] Optionally, in any of the embodiments described above, another embodiment of the embodiment provides that the first attribute is organized into multiple layers, and the third syntactic element indicates layer membership to the bitstream data unit associated with the first attribute.

[0018] Optionally, in any of the embodiments described above, another embodiment of the embodiment provides that the second syntactic element is a num_streams_for_attribute element included in a group of frame headers in the bitstream, and the third syntactic element is a num_layers_for_attribute element included in a group of frame headers in the bitstream.

[0019] Optionally, in any of the embodiments described above, another embodiment of the embodiment provides that the fourth syntactic element indicates that the first layer of a plurality of layers contains data associated with an irregular point cloud.

[0020] Optionally, in any of the embodiments described above, another embodiment of the embodiment provides that the fourth syntactic element is a regular_points_flag element included in a group of frame headers in the bitstream.

[0021] In one embodiment, the disclosure includes a processor, a receiver coupled to the processor, and a transmitter coupled to the processor, wherein the processor, receiver, and transmitter are configured to perform any of the methods described above.

[0022] In one embodiment, the Disclosure relates to a non-temporary computer-readable medium including a computer program product used by a video coding device, wherein the computer program product includes computer-executable instructions stored on the non-temporary computer-readable medium, such that when executed by a processor, the video coding device performs any of the methods described above.

[0023] In one embodiment, the disclosure comprises an encoder comprising a first attribute encoding means and a second attribute encoding means for encoding a plurality of PCC attributes of a sequence of PCC frames into a bitstream using a plurality of codeder decoders (codecs), wherein the plurality of PCC attributes include geometric shape, texture, and one or more of reflectance, transmittance, and normal, and each encoded PCC frame is represented by one or more PCC network abstraction layer (NAL) units. The encoder further comprises a syntactic encoding means for encoding, for each PCC attribute, an instruction for one of the video codecs used to encode the corresponding PCC attribute. The encoder further comprises a transmission means for transmitting the bitstream toward a decoder.

[0024] Optionally, in any of the embodiments described above, another embodiment of the embodiment provides that the encoder is further configured to perform any of the methods described above.

[0025] In one embodiment, the disclosure includes a decoder that includes receiving means for receiving a bitstream containing a coded sequence of a plurality of PCC frames, the coded sequence of a plurality of PCC frames representing a plurality of PCC attributes, each coded PCC frame being represented by one or more PCC network abstraction layer (NAL) units. The decoder further includes parsing means for parsing the bitstream to obtain, for each PCC attribute, an instruction for one of a plurality of video codeder decoders (codecs) used to code the corresponding PCC attribute. The decoder further includes decoding means for decoding the bitstream based on the indicated video codecs for the PCC attributes.

[0026] Optionally, in any of the foregoing aspects, another embodiment of the aspect is provided that is further configured to perform any of the methods of the foregoing aspects.

[0027] For clarity, any one of the foregoing embodiments may be combined with any one or more of the other foregoing embodiments to create new embodiments within the scope of the present disclosure.

[0028] These and other features will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings and the claims.

Brief Description of the Drawings

[0029] To more fully understand the present disclosure, reference is made to the following brief description in connection with the accompanying drawings and detailed description. Here, like reference numerals represent like parts.

[0030] [Figure 1] It is a flowchart of an exemplary method for coding a video signal. [Figure 2] It is a schematic diagram of an exemplary coding and decoding (codec) system for video coding. [Figure 3] It is a schematic diagram showing an exemplary video encoder. [Figure 4] It is a schematic diagram showing an exemplary video decoder. [Figure 5] It is an example of a point cloud medium that can be coded according to the PCC mechanism. [Figure 6] It is an example of data segmentation and packing for a point cloud medium frame. [Figure 7] It is a schematic diagram showing an exemplary PCC video stream having an extended attribute set. [Figure 8] It is a schematic diagram showing an exemplary mechanism for coding PCC attributes having multiple codecs. [Figure 9] It is a schematic diagram showing an example of an attribute layer. [Figure 10] This is a schematic diagram illustrating an example of an attribute stream. [Figure 11] This is a flowchart illustrating an exemplary method for encoding a PCC video sequence with multiple codecs. [Figure 12] This is a flowchart illustrating an exemplary method for decoding a PCC video sequence containing multiple codecs. [Figure 13] This is a schematic diagram of an exemplary video coding device. [Figure 14] This is a schematic diagram of an exemplary system for coding PCC video sequences with multiple codecs. [Figure 15] This is a flowchart illustrating another exemplary method for coding a PCC video sequence with multiple codecs. [Figure 16] This is a flowchart illustrating another exemplary method for decoding a PCC video sequence with multiple codecs. [Modes for carrying out the invention]

[0031] Firstly, while exemplary implementations of one or more embodiments are provided below, it should be understood that the disclosed systems and / or methods may be implemented using any number of techniques, whether currently known or existing. This disclosure should not be limited in any way to the exemplary embodiments, drawings and techniques described below, and includes the exemplary designs and embodiments illustrated and described herein, but may be modified within the scope of the appended claims, along with the entire scope of their equivalents.

[0032] Many video compression techniques can be used to reduce the size of video files with minimal data loss. For example, video compression techniques may include performing interstitial (e.g., intra-picture) and / or temporal (e.g., inter-picture) predictions to reduce or eliminate data redundancy in a video sequence. For block-based video coding, a video slice (e.g., a video picture or a portion of a video picture) may be divided into video blocks, which may also be called tree blocks, coding tree blocks (CTBs), coding tree units (CTUs), coding units (CUs), and / or coding nodes. In an intra-coded (I) slice of a picture, a video block is coded using spatial prediction with respect to a reference sample in an adjacent block within the same picture. In an inter-coded (P or B) slice of a picture, a video block may be coded by employing spatial prediction with respect to a reference sample in an adjacent block within the same picture, or by employing temporal prediction with respect to a reference sample in another reference picture. A picture is called a frame, and a reference picture is called a reference frame. Spatial or temporal prediction yields predicted blocks representing image blocks. Residual data represents the pixel difference between the original image blocks and the predicted blocks. Thus, intercoded blocks are encoded according to motion vectors pointing to the blocks of reference samples forming the predicted blocks and residual data indicating the difference between the coded blocks and the predicted blocks. Intracoded blocks are encoded according to the intracoding mode and residual data. For further compression, the residual data may be transformed from the pixel domain to the transformation domain. These yield residual transformation coefficients, which may be quantized. The quantized transformation coefficients may first be placed in a two-dimensional array. The quantized transformation coefficients may be scanned to generate a one-dimensional vector of transformation coefficients. Entropy coding may be applied to achieve greater compression. Such video compression techniques are discussed in more detail below.

[0033] To ensure that encoded video is correctly decoded, video is encoded and decoded according to the corresponding video coding standard. Video coding standards include International Telecommunication Union (ITU) Standardization Sector (ITU-T) H.261, International Organization for Standardization / International Electrotechnical Commission (ISO / IEC) Moving Picture Experts Group (MPEG)-1 Part 2, ITU-T H.262, or Advanced Video Coding (AVC), also known as ISO / IEC MPEG-2 Part 2, ITU-T H.263, ISO / IEC MPEG-4 Part 2, ITU-T H.264, or ISO / IEC MPEG-4 Part 10, and High-Efficiency Video Coding (HEVC), also known as ITU-T H.265 or MPEG-H Part 2. AVC includes extensions such as Scalable Video Coding (SVC), Multiview Video Coding (MVC), Multiview Video Coding Plus Depth (MVC+D), and Three-Dimensional (3D) AVC (3D-AVC). HEVC includes extensions such as Scalable HEVC (SHVC), Multiview HEVC (MV-HEVC), and 3D HEVC (3D-HEVC). The Joint Video Expert Team (JVET) of ITU-T and ISO / IEC has begun development of a video coding standard called Versatile Video Coding (VVC), which is included in the Working Draft (WD), including JVET-K1001-v4 and JVET-K1002-v1.

[0034] PCC is a mechanism for encoding video of 3D objects. A point cloud is a set of data points in 3D space. Such data points include parameters that determine, for example, their position and color in space. Point clouds may be used in a variety of applications, such as real-time 3D immersive telepresence, interactive parallax content virtual reality (VR) viewing, 3D free viewpoint sports relay broadcasting, geographic information systems, cultural heritage, autonomous navigation based on large 3D dynamic maps, and automotive applications. The ISO / IEC MPEG codec for PCC may operate on lossless and / or lossy compressed point clouds with substantial coding efficiency and robustness to network environments. Using this codec, it is possible to manipulate point clouds as computer data, store them on various storage media, transmit and receive them over networks, and distribute them on broadcast channels. PCC coding environments are classified into PCC Category 1, PCC Category 2, and PCC Category 3. This disclosure is directed towards PCC Category 2 related to MPEG output documents N17534 and N17533. The PCC Category 2 codec design aims to leverage other video codecs to compress the geometric shape and texture information of dynamic point clouds by compressing point cloud data as a set of different video sequences. For example, two video sequences, one representing the geometric shape information of point cloud data and the other representing the texture information, can be generated and compressed using one or more video codecs. Additional metadata supporting the interpretation of the video sequences (e.g., occupancy maps and auxiliary patch information) can also be generated and compressed separately.

[0035] A PCC system may support geometric shape PCC attributes containing positional data and texture PCC attributes containing color data. However, some video applications may include other types of data such as reflectance, transparency, and normal vectors. Some of these types of data may be encoded more efficiently using one codec than another. However, a PCC system may require that the entire PCC stream, and therefore all PCC attributes, be encoded by the same codec. Furthermore, PCC attributes may be divided into multiple layers. Such layers may then be combined and / or encoded into one or more PCC attribute streams. For example, the attribute layers may be encoded according to a temporarily interleaved coding scheme, where the first layer is encoded in a PCC access unit (AU) with an even number of picture output order values, and the second layer is encoded in a PCC AU with an odd number of image output order values. Since there may be 0 to 4 streams for each attribute and various combinations of such stream layers, proper identification of streams and layers can be a challenge. However, a PCC system may not be able to determine how many layers are coded or combined with a PCC attribute stream in a given PCC bitstream. Furthermore, a PCC system may not have a mechanism to indicate the combination of layers and / or the correspondence between such layers and the PCC attribute stream. Finally, patches are employed to encode PCC video data. For example, a three-dimensional (3D) PCC object may be represented as a set of two-dimensional (2D) patches. This allows PCC to work with video codecs designed to encode 2D video frames. However, some points within a point cloud may not be captured by patches in some cases. For example, isolated points in 3D space may be difficult to encode as part of a patch.In such cases, the only patch that makes sense is a 1x1 pixel patch containing a single point, which significantly increases signaling overhead for many such points. Alternatively, an irregular point cloud can be used, which is a special type of patch containing multiple isolated points. A different approach is used to signal attributes for irregular point cloud patches compared to other patch types. However, PCC systems may not be able to indicate that the PCC attribute layer carries an irregular point cloud / patch.

[0036] Disclosed herein are mechanisms for improving PCC by addressing the above-mentioned problems. In one embodiment, the PCC system may employ different codecs to encode different PCC attributes. Specifically, separate syntactic elements may be employed to identify the video codec for each attribute. In another embodiment, the PCC system explicitly signals the number of layers to be encoded and / or combined to represent each PCC attribute stream. Additionally, the PCC system may employ syntactic elements to signal the mode used to encode and / or combine the layers of PCC attributes in the PCC attribute stream. Furthermore, the PCC system may employ syntactic elements to specify the layer index of the layer associated with each data unit of the corresponding PCC attribute stream. In yet another embodiment, a flag may be employed for each PCC attribute layer to indicate whether the PCC attribute layer carries any irregular point cloud points. Such embodiments can be used individually or in combination. Furthermore, such embodiments allow the PCC system to employ more complex coding mechanisms in a manner that is recognizable by a decoder and therefore decodeable by a decoder. These and other examples are described in detail below.

[0037] Figure 1 is a flowchart of an exemplary operating method 100 for coding a video signal. Specifically, the video signal is encoded by an encoder. The encoding process compresses the video signal by employing various mechanisms to reduce the video file size. A smaller file size allows the compressed video file to be sent to the user while reducing the associated bandwidth overhead. The decoder then decodes the compressed video file to reconstruct the original video signal for display to the end user. The decoding process generally mirrors the encoding process to allow the decoder to consistently reconstruct the video signal.

[0038] In step 101, the video signal is input to the encoder. For example, the video signal may be an uncompressed video file stored in memory. As another example, the video file may be captured by a video capture device such as a video camera and encoded to support live streaming of the video. The video file may contain both audio and video components. The video component contains a series of image frames that give the impression of visual motion when viewed in sequence. Each frame contains pixels, which are expressed in terms of light and referred to herein as luminance components (or luminance samples), and color, which is referred to as saturation components (or color samples). In some examples, the frames may also contain depth values ​​to support three-dimensional viewing.

[0039] In step 103, the video is divided into blocks. The division involves further dividing the pixels within each frame into square and / or rectangular blocks for compression. For example, in High-Efficiency Video Coding (HEVC) (also known as H.265 and MPEG-H Part 2), a frame can first be divided into coding tree units (CTUs), which are blocks of a given size (e.g., 64 pixels × 64 pixels). A CTU contains both luminance and saturation samples. A coding tree can be employed to divide the CTUs into blocks, and then the blocks can be recursively further divided until a configuration supporting further coding is achieved. For example, the luminance component of a frame may be further divided until the individual blocks contain relatively uniform illumination values. Furthermore, the saturation component of a frame may be further divided until the individual blocks contain relatively uniform color values. Thus, the division mechanism varies depending on the content of the video frame.

[0040] In step 105, various compression mechanisms are employed to compress the image blocks divided in step 103. For example, interpretation and / or intrapretation may be employed. Interpretation is designed to take advantage of the fact that objects in a common scene tend to appear in consecutive frames. Therefore, a block representing an object in a reference frame does not need to be repeatedly depicted in adjacent frames. Specifically, an object such as a table can remain in the same position across multiple frames. Thus, a table is depicted once, and adjacent frames can refer to the reference frame. A pattern matching mechanism may be employed to match objects across multiple frames. Furthermore, moving objects may be represented across multiple frames, for example, due to the movement of the object or the movement of the camera. As a particular example, a video may show a car moving across the screen across multiple frames. To describe such movement, motion vectors may be employed. A motion vector is a two-dimensional vector that provides an offset from the coordinates of an object in the frame to the coordinates of an object in the reference frame. Thus, interpretation can encode an image block in the current frame as a set of motion vectors that indicate the offset from the corresponding block in the reference frame.

[0041] Intra-prediction encodes blocks within a common frame. Intra-prediction leverages the fact that luminance and saturation components tend to be concentrated within a frame. For example, a green patch in a part of a tree tends to be positioned adjacent to similar green patches. Intra-prediction employs multi-directional prediction modes (e.g., 33 in HEVC), planar mode, and direct current (DC) mode. Directional modes indicate that the current block is similar to / identical to samples of adjacent blocks in the corresponding direction. Planar mode indicates that a series of blocks along a row / column (e.g., a plane) can be interpolated based on adjacent blocks at the edges of the row. Planar mode effectively exhibits smooth light / color transitions across rows / columns by employing a relatively constant slope when changing values. DC mode is used for boundary smoothing, indicating that a block is similar to / identical to the mean associated with samples of all adjacent blocks associated with the angular direction of the directional prediction mode. Thus, intra-prediction blocks can represent image blocks as various relational prediction mode values ​​instead of actual values. Furthermore, the interpretation block can represent the image block as a motion vector value instead of the actual value. In either case, the prediction block may not accurately represent the image block in some cases. Any differences are stored in the residual block. The transformation may be applied to the residual block to further compress the file.

[0042] In step 107, various filtering techniques may be applied. In HEVC, filters are applied according to an in-loop filtering scheme. The block-based prediction discussed above may result in the generation of blocky images in the decoder. Furthermore, the block-based prediction scheme may encode blocks and then reconstruct the encoded blocks for later use as reference blocks. The in-loop filtering scheme iteratively applies noise suppression filters, deblocking filters, adaptive loop filters, and sample-adaptive offset (SAO) filters to blocks / frames. These filters mitigate such blocking artifacts so that the encoded file can be accurately reconstructed. In addition, these filters mitigate artifacts in the reconstructed reference blocks, and artifacts are less likely to generate additional artifacts in subsequent blocks encoded based on the reconstructed reference blocks.

[0043] Once the video signal has been split, compressed, and filtered, the resulting data is encoded into a bitstream in step 109. The bitstream contains the data discussed above and any signaling data desirable to support proper video signal reconstruction at the decoder. For example, such data may include partition data, prediction data, residual blocks, and various flags that provide coding instructions to the decoder. The bitstream may be stored in memory for transmission to the decoder upon request. The bitstream may also be broadcast and / or multicast to multiple decoders. The generation of the bitstream is an iterative process. Therefore, steps 101, 103, 105, 107, and 109 may occur sequentially and / or simultaneously across many frames and blocks. The order shown in Figure 1 is presented for clarity and ease of discussion and is not intended to limit the video coding process to a specific order.

[0044] The decoder receives the bitstream and begins the decoding process in step 111. Specifically, the decoder employs an entropy decoding scheme that converts the bitstream into corresponding syntax and video data. The decoder takes the syntax data from the bitstream and determines the partitions for the frames in step 111. The partitioning should match the result of the block partitioning in step 103. The entropy coding / decoding employed in step 111 is described here. The encoder makes many choices during the compression process, such as selecting a block partitioning scheme from several possible choices based on the spatial positioning of values ​​in the input image. Strict choice signaling may be employed by a number of bins. As used herein, a bin is a binary value (e.g., a bit value that may vary depending on the context) treated as a variable. Entropy coding allows the encoder to discard any option that is obviously not feasible in particular, leaving a set of acceptable options. Each acceptable option is assigned a codeword. The length of the codeword is based on the number of acceptable options (e.g., one bin for two options, two bins for three or four options), and the encoder then encodes the codeword for the selected options. This scheme reduces the size of the codeword because it is desirable that the codeword uniquely indicates a selection from a small subset of acceptable options, as opposed to a codeword uniquely indicating a selection from a potentially large set of all possible options. The decoder then decodes the selection by determining the set of acceptable options in a similar manner to the encoder. By determining the set of acceptable options, the decoder can read the codeword and determine the selection made by the encoder.

[0045] In step 113, the decoder performs block decoding. Specifically, the decoder employs an inverse transform to generate residual blocks. The decoder then employs the residual blocks and corresponding prediction blocks to reconstruct the image blocks according to the partitioning. The prediction blocks may include both intra-prediction blocks and inter-prediction blocks, as generated by the encoder in step 105. The reconstructed image blocks are then positioned within the frame of the reconstructed video signal according to the partitioning data determined in step 111. The syntax for step 113 may also be signaled in the bitstream via entropy coding as discussed above.

[0046] In step 115, filtering is performed on the frames of the reconstructed video signal in a manner similar to that in step 107 in the encoder. For example, noise suppression filters, deblocking filters, adaptive loop filters, and SAO filters may be applied to the frames to remove blocking artifacts. Once the frames are filtered, the video signal is output to a display in step 117 and can be viewed by the end user.

[0047] Figure 2 is a schematic diagram of an exemplary coding and decoding (codec) system 200 for video coding. Specifically, the codec system 200 provides functionality to support the implementation of operation method 100. The codec system 200 is generalized to show the components employed in both the encoder and the decoder. The codec system 200 receives and splits the video signal, as discussed with respect to steps 101 and 103 of operation method 100, thereby yielding the split video signal 201. Next, when acting as an encoder, the codec system 200 compresses the split video signal 201 into an encoded bitstream, as discussed with respect to steps 105, 107, and 109 of method 100. When acting as a decoder, the codec system 200 generates an output video signal from the bitstream, as discussed with respect to steps 111, 113, 115, and 117 of operation method 100. The codec system 200 includes a general coder control component 211, a transform scaling and quantization component 213, an intrapicture estimation component 215, an intrapicture prediction component 217, a motion compensation component 219, a motion estimation component 221, a scaling and inverse transform component 229, a filter-controlled analysis component 227, an in-loop filter component 225, a decoded picture buffer component 223, and a header formatting and context-adaptive binary arithmetic coding (CABAC) component 231. Such components are combined as shown. In Figure 2, black lines indicate the movement of data to be encoded / decoded, and dashed lines indicate the movement of control data that controls the operation of other components. All components of the codec system 200 may reside within the encoder. The decoder may include a subset of the components of the codec system 200. For example, the decoder may include the intrapicture prediction component 217, the motion compensation component 219, the scaling and inverse transform component 229, the in-loop filter component 225, and the decoded picture buffer component 223. Here, we will explain these components.

[0048] The split video signal 201 is a captured video sequence that has been divided into blocks of pixels by a coding tree. The coding tree employs various split modes to divide the blocks of pixels into smaller blocks of pixels. These blocks can then be further divided into even smaller blocks. Blocks are sometimes called nodes on the coding tree. A large parent node is split into smaller child nodes. The number of times a node is further divided is called the node / coding tree depth. In some cases, the divided blocks can be contained within a coding unit (CU). For example, a CU may be a sub-part of a CTU containing luminance blocks, red contrast (Cr) blocks, and blue contrast (Cb) blocks along with the corresponding CU syntax instructions. Split modes may include binary trees (BT), triple trees (TT), and quad trees (QT) that are employed to divide a node into two, three, or four child nodes of different shapes, depending on the split mode employed. The divided video signal 201 is transferred for compression to the general coder control component 211, the transformation scaling and quantization component 213, the intrapicture estimation component 215, the filter control analysis component 227, and the motion estimation component 221.

[0049] The general coder control component 211 is configured to make decisions related to coding the images of a video sequence into a bitstream, in accordance with application constraints. For example, the general coder control component 211 manages the optimization of bitrate / bitstream size versus reconstruction quality. Such decisions may be made based on memory / bandwidth availability and image resolution requirements. The general coder control component 211 also manages buffer utilization in relation to the transmission rate to mitigate buffer underrun and overrun issues. To manage these issues, the general coder control component 211 manages partitioning, prediction, and filtering by other components. For example, the general coder control component 211 can dynamically increase the complexity of compression to increase resolution and bandwidth usage, or decrease the complexity of compression to decrease resolution and bandwidth usage. Thus, the general coder control component 211 controls other components of the codec system 200 to balance bitrate concerns with video signal reconstruction quality. The general coder control component 211 generates control data that controls the operation of other components. The control data is also transferred to the header formatting and CABAC component 231, which are encoded in a bitstream into signal parameters for decoding by the decoder.

[0050] The divided video signal 201 is also sent to the motion estimation component 221 and the motion compensation component 219 for interpretation. A frame or slice of the divided video signal 201 may be divided into multiple video blocks. The motion estimation component 221 and the motion compensation component 219 perform interpretation coding of the received video blocks for one or more blocks in one or more reference frames in order to provide temporal prediction. The codec system 200 may perform multiple coding passes, for example, to select an appropriate coding mode for each block of video data.

[0051] The motion estimation component 221 and the motion compensation component 219 may be highly integrated, but are illustrated separately for conceptual purposes. The motion estimation performed by the motion estimation component 221 is a process that generates motion vectors, which estimate the motion of video blocks. The motion vectors may, for example, represent the displacement of a coded object relative to a predicted block. A predicted block is a block that is found to closely match the coded block with respect to pixel difference. Predicted blocks are sometimes also called reference blocks. Such pixel difference may be determined by the sum of absolute differences (SAD), sum of squared differences (SSD), or other difference metrics. HEVC employs several coded objects, including CTUs, coding tree blocks (CTBs), and CUs. For example, a CTU can be split into CTBs, and a CTB can be split into CBs to be included in a CU. A CU can be coded as a prediction unit (PU) containing prediction data and / or a transformation unit (TU) containing transformation residual data of the CU. The motion estimation component 221 uses rate distortion analysis as part of the rate distortion optimization process to generate motion vectors, PUs, and TUs. For example, the motion estimation component 221 may determine multiple reference blocks, multiple motion vectors, etc. for the current block / frame, or it may select reference blocks, motion vectors, etc. with the best rate distortion characteristics. The best rate distortion characteristics balance both the quality of video reconstruction (e.g., amount of data loss due to compression) and coding efficiency (e.g., size of the final encoding).

[0052] In some examples, the codec system 200 may calculate the values ​​of sub-integer pixel positions of the reference picture stored in the decoded picture buffer component 223. For example, the video codec system 200 can interpolate the values ​​of quarter-pixel, eighth-pixel, or other sub-pixel positions of the reference picture. Thus, the motion estimation component 221 may perform a motion search on all pixel positions and fractional pixel positions and output a motion vector with fractional pixel precision. The motion estimation component 221 calculates the motion vector for the PU of the video block in the intercoded slice by comparing the PU position with the predicted block position of the reference picture. The motion estimation component 221 outputs the calculated motion vector as motion data to the header formatting and CABAC component 231 for encoding and outputs motion to the motion compensation component 219.

[0053] Motion compensation performed by the motion compensation component 219 may involve fetching or generating a predicted block based on a motion vector determined by the motion estimation component 221. In some examples, the motion estimation component 221 and the motion compensation component 219 may also be functionally integrated. Upon receiving the motion vector for the PU of the current video block, the motion compensation component 219 may position the predicted block that the motion vector points to. A residual video block is then formed by subtracting the pixel values ​​of the predicted block from the pixel values ​​of the current video block to be coded, thereby forming a pixel difference value. Generally, the motion estimation component 221 performs motion estimation on the luminance component, and the motion compensation component 219 uses a motion vector calculated based on the luminance component for both the saturation and luminance components. The predicted and residual blocks are transferred to the scaling and quantization component 213 for transformation.

[0054] The divided video signal 201 is also sent to the intra-picture estimation component 215 and the intra-picture prediction component 217. Similar to the motion estimation component 221 and the motion compensation component 219, the intra-picture estimation component 215 and the intra-picture prediction component 217 may be highly integrated, but are illustrated separately for conceptual purposes. The intra-picture estimation component 215 and the intra-picture prediction component 217 intra-predict the current block for a block in the current frame, instead of inter-predicting performed by the inter-frame motion estimation component 221 and the motion compensation component 219, as described above. In particular, the intra-picture estimation component 215 determines the intra-prediction mode to use for encoding the current block. In some examples, the intra-picture estimation component 215 selects an appropriate intra-prediction mode from several tested intra-picture prediction modes to encode the current block. The selected intra-prediction mode is then forwarded to the header formatting and CABAC component 231 for encoding.

[0055] For example, the intrapicture estimation component 215 calculates rate distortion values ​​for various tested intraprediction modes using rate distortion analysis and selects the intraprediction mode with the best rate distortion characteristics among the tested modes. Rate distortion analysis generally determines the amount of distortion (or error) between the coded block and the original uncoded block coded to generate the coded block, and the bit rate (e.g., number of bits) used to generate the coded block. The intrapicture estimation component 215 calculates a ratio from the distortion and rate for various coded blocks to determine which intraprediction mode exhibits the best rate distortion value for the block. Additionally, the intrapicture estimation component 215 may be configured to encode depth blocks in the depth map using a depth modeling mode (DMM) based on rate distortion optimization (RDO).

[0056] The intra-picture prediction component 217 may generate residual blocks from prediction blocks based on a selected intra-prediction mode determined by the intra-picture estimation component 215 when performed in the encoder, or it may read residual blocks from the bitstream when performed in the decoder. The residual blocks contain the difference in values ​​between the prediction blocks and the original blocks, which are represented as a matrix. The residual blocks are then transferred to the transformation scaling and quantization component 213. The intra-picture estimation component 215 and the intra-picture prediction component 217 may operate for both luminance and saturation components.

[0057] The transform scaling and quantization component 213 is configured to further compress the residual block. The transform scaling and quantization component 213 applies a transform, such as a discrete cosine transform (DCT), discrete sine transform (DST), or a conceptually similar transform, to the residual block to generate a video block containing residual transform coefficient values. Wavelet transforms, integer transforms, subband transforms, or other types of transforms can also be used. The transform may convert the residual information from the pixel value domain to a transform domain, such as the frequency domain. The transform scaling and quantization component 213 is also configured to scale the transformed residual information, for example, based on frequency. Such scaling involves applying a scale factor to the residual information so that different frequency information is quantized at different granularities, which may affect the final visual quality of the reconstructed video. The transform scaling and quantization component 213 is also configured to quantize the transform coefficients to further reduce the bitrate. The quantization process may reduce the bit depth associated with some or all of the coefficients. The degree of quantization may be modified by adjusting the quantization parameters. In some examples, the transformation scaling and quantization component 213 may then perform a scan of a matrix containing the quantized transformation coefficients. The quantized transformation coefficients are then transferred to the header formatting and CABAC component 231 and encoded in the bitstream.

[0058] The scaling and inverse transform component 229 applies the inverse operation of the transform scaling and quantization component 213 to support motion estimation. The scaling and inverse transform component 229 applies inverse scaling, transform, and / or quantization to reconstruct residual blocks within the pixel domain, which can then be used as reference blocks, for example, as predicted blocks for another current block. The motion estimation component 221 and / or motion compensation component 219 may compute the reference blocks by adding the residual blocks to the corresponding predicted blocks for use in motion estimation of subsequent blocks / frames. Filters are applied to the reconstructed reference blocks to mitigate artifacts generated during scaling, quantization, and transform. Otherwise, such artifacts would cause inaccurate predictions (generating additional artifacts) when subsequent blocks are predicted.

[0059] The filter-controlled analysis component 227 and the in-loop filter component 225 apply filters to residual blocks and / or reconstructed image blocks. For example, the transformed residual blocks from the scaling and inverse transform component 229 may be combined with the corresponding predicted blocks from the intra-picture prediction component 217 and / or the motion compensation component 219 to reconstruct the original image block. The filter may then be applied to the reconstructed image block. In some examples, the filter may be applied to the residual block instead. Like the other components in Figure 2, the filter-controlled analysis component 227 and the in-loop filter component 225 are highly integrated and may be performed together, but are shown separately for conceptual purposes. The filters applied to the reconstructed reference block are applied to a specific spatial region and include several parameters to adjust how such filters are applied. The filter-controlled analysis component 227 analyzes the reconstructed reference block to determine where such filters should be applied and sets the corresponding parameters. Such data is transferred to the header formatting and CABAC component 231 as filter-controlled data for encoding. The in-loop filter component 225 applies such a filter based on filter control data. The filter may include a deblocking filter, a noise suppression filter, a SAO filter, and an adaptive loop filter. Such filters may be applied in the spatial / pixel domain (e.g., a reconstructed pixel block) or the frequency domain, as an example.

[0060] When operating as an encoder, the filtered and reconstructed image blocks, residual blocks, and / or predicted blocks are stored in the decoded picture buffer component 223 for use in motion estimation, as discussed later above. When operating as a decoder, the decoded picture buffer component 223 stores the reconstructed and filtered blocks as part of the output video signal and transmits them to the display. The decoded picture buffer component 223 may be any memory device capable of storing the predicted blocks, residual blocks, and / or reconstructed image blocks.

[0061] The header formatting and CABAC component 231 receives data from various components of the codec system 200 and encodes such data into coded bitstreams for transmission to the decoder. Specifically, the header formatting and CABAC component 231 generates various headers for encoding control data such as general control data and filter control data. Furthermore, prediction data, including intra-prediction and motion data, as well as residual data in the form of quantization conversion coefficient data, are all encoded into bitstreams. The final bitstream contains all the information desired by the decoder to reconstruct the original segmented video signal 201. Such information may also include an intra-prediction mode index table (also called a codeword mapping table), definitions of coding contexts for various blocks, indications of the most likely intra-prediction mode, and indications of partition information. Such data may be encoded by employing entropy coding. For example, the information may be encoded by using context-adaptive variable-length coding (CAVLC), CABAC, syntax-based context-adaptive binary arithmetic coding (SBAC), probabilistic interval-partitioned entropy coding (PIPE), or another entropy coding technique. According to entropy coding, the encoded bitstream may be sent to another device (e.g., a video decoder) or archived for later transmission or retrieval.

[0062] Figure 3 is a block diagram showing an exemplary video encoder 300. The video encoder 300 may be employed to perform the encoding function of the codec system 200 and / or to perform steps 101, 103, 105, 107, and / or 109 of the operation method 100. The encoder 300 splits the input video signal, resulting in a split video signal 301, which is substantially similar to the split video signal 201. The split video signal 301 is then compressed by the components of the encoder 300 and encoded into a bitstream.

[0063] Specifically, the divided video signal 301 is transferred to the intra-picture prediction component 317 for intra-prediction. The intra-picture prediction component 317 may be substantially the same as the intra-picture estimation component 215 and the intra-picture prediction component 217. The divided video signal 301 is also transferred to the motion compensation component 321 for inter-prediction based on the reference block in the decoded picture buffer component 323. The motion compensation component 321 may be substantially the same as the motion estimation component 221 and the motion compensation component 219. The prediction block and residual block from the intra-picture prediction component 317 and the motion compensation component 321 are transferred to the transformation and quantization component 313 for transformation and quantization of the residual block. The transformation and quantization component 313 may be substantially the same as the transformation scaling and quantization component 213. The transformed and quantized residual block and the corresponding prediction block (along with the associated control data) are transferred to the entropy coding component 331 for coding into a bitstream. The entropy coding component 331 may be substantially the same as the header formatting and CABAC component 231.

[0064] The transformed and quantized residual blocks and / or corresponding prediction blocks are also transferred from the transform and quantization component 313 to the inverse transform and quantization component 329 for reconstruction into reference blocks used by the motion compensation component 321. The inverse transform and quantization component 329 may be substantially the same as the scaling and inverse transform component 229. The in-loop filters in the in-loop filter component 325 are also applied to the residual blocks and / or reconstructed reference blocks, as example. The in-loop filter component 325 may be substantially the same as the filter-controlled analysis component 227 and the in-loop filter component 225. The in-loop filter component 325 may contain multiple filters, as discussed with respect to the in-loop filter component 225. The filtered blocks are then stored in the decoded picture buffer component 323 for use as reference blocks by the motion compensation component 321. The decoded picture buffer component 323 may be substantially the same as the decoded picture buffer component 223.

[0065] Figure 4 is a block diagram showing an exemplary video decoder 400. The video decoder 400 may be employed to perform the decoding function of the codec system 200 and / or to perform steps 111, 113, 115, and / or 117 of the operation method 100. The decoder 400 receives a bitstream from, for example, the encoder 300 and generates an output video signal reconstructed based on the bitstream for display to the end user.

[0066] The bitstream is received by the entropy decoding component 433. The entropy decoding component 433 is configured to implement an entropy decoding scheme such as CAVLC, CABAC, SBAC, PIPE coding, or other entropy coding techniques. For example, the entropy decoding component 433 may employ header information to provide context for interpreting additional data encoded as codewords in the bitstream. The decoded information includes any desired information for decoding the video signal, such as general control data, filter control data, partition information, motion data, prediction data, and quantization transformation coefficients from residual blocks. The quantized transformation coefficients are transferred to the inverse transform and quantization component 429 for reconstruction into residual blocks. The inverse transform and quantization component 429 may be similar to the inverse transform and quantization component 329.

[0067] The reconstructed residual blocks and / or predicted blocks are transferred to the intra-picture prediction component 417 for reconstruction into image blocks based on intra-predictive operation. The intra-picture prediction component 417 may be similar to the intra-picture estimation component 215 and the intra-picture prediction component 217. Specifically, the intra-picture prediction component 417 employs a prediction mode to position a reference block in the frame and applies the residual blocks to the result to reconstruct the intra-predicted image block. The reconstructed intra-predicted image block and / or residual block and the corresponding intra-predictive data are transferred to the decoded picture buffer component 423 via the in-loop filter component 425, which may be substantially similar to the decoded picture buffer component 223 and the in-loop filter component 225, respectively. The in-loop filter component 425 filters the reconstructed image block, residual block and / or predicted block, and such information is stored in the decoded picture buffer component 423. The reconstructed image blocks from the decoded picture buffer component 423 are transferred to the motion compensation component 421 for interpretation. The motion compensation component 421 may be substantially the same as the motion estimation component 221 and / or motion compensation component 219. Specifically, the motion compensation component 421 takes motion vectors from a reference block to generate a prediction block and applies a residual block to the result to reconstruct the image block. The resulting reconstructed block may be transferred to the decoded picture buffer component 423 via the in-loop filter component 425. The decoded picture buffer component 423 continues to store additional reconstructed image blocks that can be reconstructed into frames via partition information. Such frames may be placed in a sequence. This sequence is output to a display as a reconstructed output video signal.

[0068] Figure 5 shows an example of a point cloud medium 500 that can be coded according to the PCC mechanism. A point cloud is a set of data points in space. A point cloud may be generated by a 3D scanner that measures a number of points on the outer surface of surrounding objects. A point cloud may be described in terms of geometric attributes, texture attributes, reflectivity attributes, transparency attributes, normal attributes, etc. Each attribute can be coded by a codec such as a video codec system 200, an encoder 300, and / or a decoder 400 as part of method 100. Specifically, each attribute of a PCC frame can be coded separately by an encoder, decoded by a decoder, and then recombined to recreate a PCC frame.

[0069] The point cloud medium 500 includes three bounding boxes 502, 504, and 506. Each of the bounding boxes 502, 504, and 506 represents a portion or segment of a 3D image from the current frame. The bounding boxes 502, 504, and 506 contain 3D images of people, but other objects may be included in the bounding boxes in actual applications. Each bounding box 502, 504, and 506 includes x, y, and z axes indicating the number of pixels occupied by the 3D image in the x, y, and z directions, respectively. For example, the x and y axes might represent approximately 400 pixels (e.g., approximately 0 to 400 pixels), and the z axis might represent approximately 1000 pixels (e.g., approximately 0 to 1000 pixels).

[0070] Each of the bounding boxes 502, 504, and 506 contains one or more patches 508, which are represented by the cube or box in Figure 5. Each patch 508 contains a portion of the whole object within one of the bounding boxes 502, 504, or 506, and may be described or represented by patch size information. The patch information may include, for example, two-dimensional (2D) coordinates and / or three-dimensional (3D) coordinates describing the location of the patch 508 within the bounding box 502, 504, or 506. The patch information may also include other parameters. For example, the patch information may include parameters such as normalAxis, which are inherited from reference patch information to the current patch information. That is, one or more parameters from the patch information of the reference frame may be inherited to the patch information of the current frame. In addition, one or more metadata parts from the reference frame (e.g., patch rotation, scale parameters, material identifiers, etc.) may be inherited to the current frame. Patch 508 may be referred to herein interchangeably as a 3D patch or patch data unit. A list of patches 508 within each bounding box 502, 504, or 506 may be generated and stored in a patch buffer in descending order from the largest patch to the smallest patch. This patch can then be encoded by an encoder and / or decoded by a decoder.

[0071] Patch 508 can describe various attributes of the point cloud medium 500. Specifically, the position of each pixel on the x, y, and z axes is the geometric shape of that pixel. Patch 508, which contains the positions of all pixels in the current frame, can be coded to capture the geometric attributes of the point cloud medium 500 relative to the current frame. Furthermore, each pixel may contain color values ​​in the red, blue, and green (RGB) and / or luminance and saturation (YUV) spectrum. Patch 508, which contains the colors of all pixels in the current frame, can be coded to capture the texture attributes of the point cloud medium 500 relative to the current frame.

[0072] Additionally, each pixel may (or may not) have some reflectivity. Reflectivity is the amount of light (e.g., colored light) projected from a pixel to an adjacent pixel. Shiny objects have high reflectivity and therefore spread the light / color of their corresponding pixels to other nearby pixels. Matte objects, on the other hand, may have little or no reflectivity and do not affect the color / light level of adjacent pixels. Patch 508 containing the reflectivity of all pixels in the current frame can be coded to capture the reflectivity attribute of the point cloud medium 500 relative to the current frame. Some pixels may be partially or completely transparent (e.g., glass, transparent plastic, etc.). Transparency is the amount of light / color of adjacent pixels that can pass through the current pixel. Patch 508 containing the transparency levels of all pixels in the current frame can be coded to capture the transparency attribute of the point cloud medium 500 relative to the current frame. Furthermore, points in the point cloud medium may generate surfaces. Surfaces can be associated with normal vectors, which are vectors perpendicular to the surface. Normal vectors can be useful when describing the motion and / or interaction of objects. Therefore, in some cases, users may want to encode normal vectors for surfaces to support further functionality. Patch 508, which contains normal vectors for surfaces in the current frame, can be coded to capture the normal attribute of the point cloud medium 500 to the current frame.

[0073] Geometric shapes, textures, reflectivity, transparency, and normal attributes may, as an example, include data describing some or all data points within the point cloud medium 500. For example, reflectivity, transparency, and normal attributes are optional and therefore may occur individually or in combination in some examples of point cloud medium 500, even within the same bitstream, while not occurring in others. Thus, the number of patches 508, and even the number of attributes, may vary from frame to frame and from video to video, based on the filmed subject, video settings, etc.

[0074] Figure 6 shows an example of data segmentation and packing for a point cloud medium frame 600. Specifically, the example in Figure 6 shows a 2D representation of patch 508 of the point cloud medium 500. The point cloud medium frame 600 includes a bounding box 602 corresponding to the current frame from a video sequence. The bounding box 602 is 2D, in contrast to the 3D bounding boxes 502, 504, and 506 in Figure 5. As shown, the bounding box 602 contains a number of patches 604. Patches 604 may be referred to herein interchangeably as 2D patches or patch data units. In summary, patch 604 in Figure 6 is a representation of the image in bounding box 504 in Figure 5. Thus, the 3D image in bounding box 504 in Figure 5 is projected onto the bounding box 602 via patch 604. The portion of the bounding box 602 that does not contain one of the patches 604 is called a gap 606. Gap 606 may also be called a void, empty sample, etc.

[0075] With the above in mind, it should be noted that the video-based point cloud compression (PCC) codec solution is based on segmenting 3D point cloud data (e.g., patch 508 in Figure 5) into 2D projection patches (e.g., patch 604 in Figure 6). In fact, the coding method or process described above may be usefully implemented for various types of technologies, such as immersive 6 degrees of freedom (6 DoF), dynamic augmented reality / virtual reality (AR / VR) objects, cultural heritage, graphics information systems (GIS), computer-aided design (CAD), and autonomous navigation.

[0076] The position of each patch (e.g., one of the patches 604 in Figure 6) within a bounding box (e.g., bounding box 602) may be determined solely by the size of the patch. For example, the largest of the patches 604 in Figure 6 is initially projected onto the bounding box 602, starting from the top-left corner (0,0). After the largest patch 604 is projected onto the bounding box 602, the next largest patch 604 is projected (also known as filled) onto the bounding box 602, and so on, until the smallest patch 604 is projected onto the bounding box 602. Again, in this process, only the size of each patch 604 is considered. In some cases, a smaller patch 604 may occupy space between larger patches and will be closer to the top-left corner of the bounding box 602 than a larger patch 604. During encoding, this process may be repeated for each associated attribute until the patch for each attribute in the frame is encoded into one or more corresponding attribute streams. Next, groups of data units in attribute streams used to recreate a single frame can be stored in a bitstream within the PCC AU. In the decoder, these attribute streams are retrieved from the PCC AU, decoded, and generate a patch 604. Such patches 604 can then be combined to recreate the PCC medium. Thus, the point cloud medium frame 600 can be encoded by a codec such as a video codec system 200, an encoder 300, and / or a decoder 400 as part of a method 100 for compressing the point cloud medium 500 for transmission.

[0077] Figure 7 is a schematic diagram illustrating an exemplary PCC video stream 700 having an extended attribute set. For example, the PCC video stream 700 may be generated when point cloud medium frames 600 from point cloud medium 500 are encoded according to method 100, for example by employing a video codec system 200, an encoder 300, and / or a decoder 400.

[0078] The PCC video stream 700 contains a sequence of PCC AUs 710. Each PCC AU 710 contains enough data to reconstruct a single PCC frame. The data is placed within the PCC AUs 710 in NAL units 720. A NAL unit 720 is a packet-sized data container. For example, a single NAL unit 720 is generally sized to allow for simple network transmission. A NAL unit 720 may include a header indicating the type of NAL unit 720 and a payload containing the associated video data. The PCC video stream 700 is designed for extended attribute sets and therefore includes several attribute-specific NAL units 720.

[0079] The PCC video stream 700 includes a Group of Frames (GOF) header 721, an auxiliary information frame 722, an occupancy map frame 723, a geometric shape NAL unit 724, a texture NAL unit 725, a reflection NAL unit 726, a transparency NAL unit 727, and a normal NAL unit 728, each of which is a type of NAL unit 720. The GOF header 721 contains various syntactic elements describing the corresponding PCC AU 710, the frame associated with the corresponding PCC AU 710, and / or other NAL units 720 within the PCC AU 710. The PCC AU 710 may contain a single GOF header 721 or it may not contain a GOF header 721, as is the case for example. The auxiliary information frame 722 may contain frame-related metadata, such as information related to the patch used to encode the attributes. The occupancy map frame 723 may contain further frame-related metadata, such as an occupancy map showing the area of ​​an empty frame versus the area of ​​a frame occupied by data. The remaining NAL units 720 contain attribute data for the PCC AU 710. Specifically, the geometric shape NAL unit 724, texture NAL unit 725, reflection NAL unit 726, transparency NAL unit 727, and normal NAL unit 728 contain geometric attributes, texture attributes, reflection attributes, transparency attributes, and normal attributes, respectively.

[0080] As described above, attributes can be organized into streams. For example, there may be 0 to 4 streams for each attribute. Streams may contain logically separated portions of PCC video data. For example, attributes for different objects may be encoded in multiple attribute streams of the same type (e.g., a first geometric shape stream for the first 3D bounding box, a second attribute stream for the second 3D bounding box, etc.). In another example, attributes associated with different frames may be encoded in multiple attribute streams (e.g., a transparency attribute stream for even frames, a transparency attribute stream for odd frames, etc.). In yet another example, patches may be placed in layers to represent 3D objects. Thus, separate layers may be contained in separate streams (e.g., a first texture attribute stream for the top layer, a second texture attribute stream for the second layer, etc.). Regardless of the example, PCC AU710 may contain 0, 1 or more NAL units for the corresponding attributes.

[0081] This disclosure supports increased flexibility for coding various attributes (e.g., including geometric shape NAL unit 724, texture NAL unit 725, reflection NAL unit 726, transparency NAL unit 727, and / or normal NAL unit 728). In a first example, different codecs can be employed to code different PCC attributes. Specifically, a first codec can be employed to code the geometric shape of the PCC video into geometric shape NAL unit 724, and a second codec can be employed to code the reflection of the PCC video into reflection NAL unit 726. In another example, up to five codecs (e.g., one codec for each attribute) can be employed when coding the PCC video. The codecs used for the attributes can then be signaled as syntactic elements in the PCC video stream 700, e.g., the GOF header 721.

[0082] Furthermore, as described above, PCC attributes may employ many combinations of layers and / or streams. Therefore, to enable the decoder to determine the combination of layers and / or streams for each attribute when decoding, syntactic elements (e.g., in the GOF header 721) can be used to signal the combination of layers and / or streams used by the encoder when encoding each attribute. Additionally, syntactic elements (e.g., in the GOF header 721) can be used to signal the mode used to code and / or combine the layers of the PCC attributes in the PCC attribute stream. Moreover, syntactic elements (e.g., in the GOF header 721) can be used to specify the layer index of the layer associated with each NAL unit 720 corresponding to the PCC attribute stream. For example, the GOF header 721 can be used to signal the number of layers and streams associated with the geometric shape attribute, the way in which such layers and streams are arranged, and the layer index for each geometric NAL unit 724, so that the decoder can assign each geometric NAL unit 724 to the appropriate layer when decoding the PCC frame.

[0083] Finally, a flag (for example, within GOF header 721) can indicate whether any PCC attribute layer contains any irregular point cloud points. An irregular point cloud is a set of one or more data points that are discontinuous with their adjacent data points and therefore cannot be represented in a 2D patch like patch 604. Instead, such points are represented as part of an irregular point cloud patch that includes coordinates and / or transformation parameters associated with such an irregular point cloud point. Since irregular point clouds are represented using a different data structure than 2D patches, the flag allows the decoder to properly recognize the presence of an irregular point cloud and select the appropriate mechanism for decoding such data.

[0084] The following is an exemplary mechanism for implementing the above-described embodiment. Definition: A video NAL unit is a PCC NAL unit whose PccNalUnitType is equal to GMTRY_NALU, TEXTURE_NALU, REFLECT_NALU, TRANSP_NALU, or NORMAL_NALU.

[0085] Bitstream Format: This section specifies the relationship between NAL unit streams and byte streams, both of which are referred to as bitstreams. A bitstream can be one of two formats: NAL unit stream format or byte stream format. The NAL unit stream format is conceptually a more basic type and contains a set of syntactic structures called PCC NAL units. This sequence is ordered in decoding order. There are constraints imposed on the decoding order (and content) of PCC NAL units in a NAL unit stream. A byte stream format can be constructed from the NAL unit stream format by arranging the NAL units in decoding order and prefixing each NAL unit with a start code prefix and zero or more zero-value bytes to form a byte stream. The NAL unit stream format can be extracted from the byte stream format by searching for the position of a unique start code prefix pattern within the stream of bytes. The byte stream format is similar to the format used in HEVC and AVC.

[0086] The PCC NAL unit header syntax may be implemented as described in Table 1 below. [Table 1]

[0087] [Table 2]

[0088] The PCC profile and level syntax may be implemented as described in Table 3 below. [Table 3]

[0089] The PCC NAL unit header semantics may be implemented as follows: `forbidden_zero_bit` may be set to 0. `pcc_nal_unit_type_plus1-1` specifies the value of the variable `PccNalUnitType`, which specifies the type of RBSP data structure contained in the PCC NAL unit, as specified in Table 4 below. The variable `NalUnitType` is specified as follows: PccNalUnitType=pcc_nal_unit_type_plus1-1 (7-1) PCC NAL units with nal_unit_type in the range of UNSPEC25-UNSPEC30 for which no semantics are specified do not affect the decoding process specified herein. Note that PCC NAL units in the range of UNSPEC25-UNSPEC30 may be used as determined by the application. The decoding process for these values ​​of PccNalUnitType is not specified in this disclosure. Because different applications may use these PCC NAL unit types for different purposes, special care should be taken in the design of encoders that produce PCC NAL units with these PccNalUnitType values ​​and in the design of decoders that interpret the contents of PCC NAL units with these PccNalUnitType values. This disclosure does not define any control over these values. These PccNalUnitType values ​​may only be suitable for use in contexts where usage conflicts (e.g., different definitions of the content of a PCC NAL unit for the same PccNalUnitType value) are not critical and are not possible, such as being defined or managed in a control application or transport specification, or managed by controlling the environment in which the bitstream is distributed.

[0090] For purposes other than determining the amount of data in the bitstream's PCC AU, the decoder may ignore (remove from the bitstream and discard) the contents of all PCC NAL units that use the reserved value of PccNalUnitType. This requirement may allow for the future definition of compatible extensions to this disclosure. [Table 4]

[0091] The identified video codec (e.g., HEVC or AVC) is indicated to a group of frame header NAL units present in the first PCC AU of each Cloud Point Stream (CPS). `pcc_stream_id` specifies the PCC stream identifier (ID) of the PCC NAL unit. The value of `pcc_stream_id` is set to zero when `PccNalUnitType` is equal to `GOF_HEADER`, `AUX_INFO`, or `OCP_MAP`. In the definition of one or more sets of PCC profiles and levels, the value of `pcc_stream_id` may be constrained to less than 4.

[0092] The order of PCC NAL units and their relationship to PCC AUs is described below. A PCC AU contains one or more video AUs that carry data units for PCC attributes such as geometric shape, texture, reflection, transparency, or normals, one or more of the frame header NAL units, one auxiliary information frame NAL unit, one occupied map frame NAL unit, and data units for PCC attributes such as geometric shape, texture, reflection, transparency, or normals. video_au(i,j) indicates a video AU where the PCC attribute ID is equal to attribute_type[i] and the pcc_stream_id is equal to j. Video AUs present in a PCC AU may be ordered as follows: If attributes_first_ordering_flag is equal to 1, the following applies to any two video AUs present in a PCC AU, video_au(i1,j1) and video_au(i2,j2): If i1 is less than i2, video_au(i1,j1) precedes video_au(i2,j2), regardless of the values ​​of j1 and j2. Otherwise, if i1 is equal to i2 and j1 is greater than j2, video_au(i1,j1) will be followed by video_au(i2,j2).

[0093] Otherwise (for example, if attributes_first_ordering_flag is equal to 0), the following applies to two video AUs, video_au(i1,j1) and video_au(i2,j2), present in a PCC AU: If j1 is less than j2, video_au(i1,j1) precedes video_au(i2,j2), regardless of the values ​​of i1 and i2. Otherwise, if j1 is equal to j2 and i1 is greater than i2, video_au(i1,j1) follows video_au(i2,j2). The above order of video AUs results in the following: If attributes_first_ordering_flag is equal to 1, the order of video AUs within a PCC AU (in the order listed) is as follows, if present: Here, within a PCC AU, all PCC NAL units of each PCC attribute are contiguous in decoding order, without interleaving with PCC NAL units of other PCC attributes, if present. That is, video_au(0,0),video_au(0,1),...,video_au(0,num_streams_for_attribute[0]),video_au(1,0),video_au(1,1),...,video_au(1,num_streams_for_attribute[1]),video_au(num_attributes-1,0),video_au(num_attributes-1,1),...,video_au(num_attributes-1,num_streams_for_attribute[1]). Otherwise (attributes_first_ordering_flag is equal to 0), the order of video AUs, if present, is as follows within the PCC AU (in the order listed), and within the PCC AU, all PCC NAL units for each particular pcc_stream_id value are consecutive in decoding order without being interleaved with PCC NAL units for other pcc_stream_id values, if present.i.e. video_au(0,0),video_au(1,0),...,video_au(num_attributes-1,0),video_au(0,1),video_au(1,1),...,video_au(num_attributes-1,1),video_au( 0,num_streams_for_attribute[1]),video_au(1,num_streams_for_attribute[1]),...,video_au(num_attributes-1,num_streams_for_attribute[1]).

[0094] The association of NAL units with video AUs and the order of NAL units within a video AU are specified in the identified video codec specification, e.g., HEVC or AVC. The identified video codec is indicated to the frame header NAL unit present in the first PCC AU of each CPS.

[0095] The first PCC AU in each CPS starts with a group of frame header NAL units, and each group of frame header NAL units specifies the start of a new PCC AU.

[0096] Other PCC AUs begin with an Auxiliary Information Frame NAL unit. In other words, a new PCC AU begins if an Auxiliary Information Frame NAL unit is not preceded by a group of Frame Header NAL units.

[0097] The semantics of the frame header RBSP group are as follows: num_attributes specifies the maximum number of PCC attributes (geometric shape, texture, etc.) that can be carried in the CPS. Note that the value of num_attributes may be constrained to 5 or less when defining one or more sets of PCC profiles and levels. attributes_first_ordering_flag, when equal to 0, specifies that within a PCC AU, all PCC NAL units of each PCC attribute, if present, should be consecutive in decoding order without being interleaved with PCC NAL units of other PCC attributes. attributes_first_ordering_flag, when set to equal to 0, specifies that within a PCC AU, all PCC NAL units of each particular pcc_stream_id value, if present, should be consecutive in decoding order without being interleaved with PCC NAL units of other pcc_stream_id values. attribute_type[i] specifies the PCC attribute type of the i-th PCC attribute. Interpretations of different PCC attribute types are specified in Table 5 below. In the definition of one or more sets of PCC profiles and levels, the values ​​of attribute_type[0] and attribute_type[1] may be constrained to be equal to 0 and 1, respectively. [Table 5]

[0098] identified_codec_for_attribute[i] specifies the identified video codec used for coding the i-th PCC attribute, as shown in Table 6 below. [Table 6]

[0099] num_streams_for_attribute[i] specifies the maximum number of PCC streams for the i-th PCC attribute. Note that in the definition of one or more sets of PCC profiles and levels, the value of num_streams_for_attribute[i] may be constrained to 4 or less. num_layers_for_attribute[i] specifies the number of attribute layers for the i-th PCC attribute. Note that in the definition of one or more sets of PCC profiles and levels, the value of num_layer_for_attribute[i] may be constrained to 4 or less. max_attribute_layer_idx[i][j] specifies the maximum attribute layer index for the PCC stream, given that pcc_stream_id is equal to j for the i-th PCC attribute. The value of max_attribute_layer_idx[i][j] should be less than num_layer_for_attribute[i]. `attribution_layers_combination_mode[i][j]` specifies the attribute layer combination mode for attribute layers transported in a PCC stream, where `pcc_stream_id` is equal to `j` for the i-th PCC attribute. Different interpretations of `attribution_layers_combination_mode[i][j]` are specified in Table 7 below. [Table 7]

[0100] When attribution_layers_combination_mode[i][j] exists and is equal to 0, the variable attrLayerIdx[i][j], which indicates the attribute layer index for the attribute layer of the PCC stream with pcc_stream_id equal to j for the i-th PCC attribute, and the PCC NAL units of the attribute layer carried by the video AU with the picture order count value equal to PicOrderCntVal as specified in the identified video codec specification, are derived as follows:

number

[0101] `regular_points_flag[i][j]`, if equal to 1, specifies that the attribute layer carries the normal points of the point cloud signal with a layer index equal to j for the i-th PCC attribute. `regular_points_flag[i][j]`, if equal to 0, specifies that the attribute layer carries the irregular points of the point cloud signal with a layer index equal to j for the i-th PCC attribute. Note that in the definition of one or more sets of PCC profiles and levels, the value of `regular_points_flag[i][j]` may be constrained to zero. `frame_width` indicates the frame width in pixels for geometric and textured videos. The frame width should be a multiple of `occupancyResolution`. `frame_height` indicates the frame height in pixels for geometric and textured videos. The frame height should be a multiple of `occupancyResolution`. `occupancy_resolution` indicates the horizontal and vertical resolution in pixels at which patches are packed into geometric and textured videos. `occupancy_resolution` should be an even multiple of `occupancyPrecision`. `radius_to_smoothing` indicates the radius used to detect neighbors for smoothing. The value of `radius_to_smoothing` should be in the range of 0 to 255.

[0102] `neighbor_count_smooting` indicates the maximum number of neighbors used for smoothing. The value of `neighbor_count_smooting` should be in the range of 0 to 255. `radius2_boundary_detection` indicates the radius for boundary point detection. The value of `radius2_boundary_detection` should be in the range of 0 to 255. `threshold_smooting` indicates the smoothing threshold. The value of `threshold_smooting` should be in the range of 0 to 255. `lossless_geometry` indicates reversible geometric shape coding. A value of `lossless_geometry` equal to 1 indicates that point cloud geometric shape information is reversibly coded. A value of `lossless_geometry` equal to 0 indicates that point cloud geometric shape information is irreversibly coded. `lossless_texture` indicates reversible texture coding. A value of `lossless_texture` equal to 1 indicates that point cloud texture information is reversibly coded. A value of `lossless_texture` equal to 0 indicates that point cloud texture information is irreversibly coded. `lossless_geometry_444` indicates whether to use a 4:2:0 or 4:4:4 video format for the geometric shape frame. A value of 1 for `lossless_geometry_444` indicates that the geometric shape video will be coded in 4:4:4 format. A value of 0 for `lossless_geometry_444` indicates that the geometric shape video will be coded in 4:2:0 format.

[0103] `absolute_d1_coding` indicates how geometric shape layers other than the layer closest to the projection plane are coded. If `absolute_d1_coding` is equal to 1, it indicates that the actual geometric shape values ​​are coded for geometric shape layers other than the layer closest to the projection plane. If `absolute_d1_coding` is equal to 0, it indicates that geometric shape layers other than the layer closest to the projection plane are coded differentially. `bin_agithmetic_coding` indicates whether binary arithmetic coding is used. If the value of `bin_agithmetic_coding` is equal to 1, it indicates that binary arithmetic coding is used for all syntax elements. If the value of `bin_agithmetic_coding` is equal to 0, it indicates that non-binary arithmetic coding is used for some syntax elements. If `gof_header_extension_flag` is equal to 0, it specifies that the `gof_header_extension_data_flag` syntax element does not exist in the group of frame header RBSP syntax structures. If gof_header_extension_flag is equal to 1, it specifies that the gof_header_extension_data_flag syntactic element exists in a group of frame header RBSP syntactic structures. The decoder may ignore all data following the value 1 of gof_header_extension_flag in a group of frame header NAL units. gof_header_extension_data_flag may have any value, and the presence and value of the flag do not affect the decoder's compatibility. The decoder may ignore all gof_header_extension_data_flag syntactic elements.

[0104] The PCC profile and level semantics are as follows: pcc_profile_idc indicates the profile to which the CPS conforms. pcc_pl_reserved_zero_19bits is equal to 0 for a bitstream conforming to this version of the disclosure. Other values ​​for pcc_pl_reserved_zero_19bits are reserved for future use by ISO / IEC. The decoder may ignore the value of pcc_pl_reserved_zero_19bits. pcc_level_idc indicates the level to which the CPS conforms. When an HEVC bitstream for a PCC attribute type equal to attribute_type[i] extracted as specified by the subbitstream extraction process is decoded by a conforming HEVC decoder, in the active SPS, hevc_ptl_12bytes_attribute[i] may be equal to the 12-byte value from general_profile_idc to general_level_idc. When an AVC bitstream for a PCC attribute type equal to attribute_type[i] extracted as specified by the subbitstream extraction process is decoded by a conforming AVC decoder, in the active SPS, avc_pl_3ytes_attribute[i] may be equal to the 3-byte value from profile_idc to level_idc.

[0105] The sub-bitstream extraction process is as follows: The input to this process is a PCC bitstream (inBitstream), a target PCC attribute type (targetAttrType), and a target PCC stream ID value (targetStreamId). The output of this process is a sub-bitstream. Any output sub-bitstream that is the output of the process specified in this section, having a compatible PCC bitstream (inBitstream), a targetAttrType indicating any type of PCC attribute present in (inBitstream), and a targetStreamId less than or equal to the maximum PCC stream ID value of PCC streams present in (inBitstream) for attribute type (targetAttrType), may be a video bitstream that is compatible for each identified video codec specification for attribute type (targetAttrType), which may be a bitstream compatibility requirement for the input bitstream.

[0106] The output subbitstream is derived by the following ordered steps. Depending on the value of targetAttrType, the following applies: If targetAttrType is equal to ATTR_GEOMETRY, all PCC NAL units whose PccNalUnitType is not equal to GMTRY_NALU or whose pcc_stream_id is not equal to targetStreamId are removed. Otherwise, if targetAttrType is equal to ATTR_TEXTURE, all PCC NAL units whose PccNalUnitType is not equal to TEXTURE_NALU or whose pcc_stream_id is not equal to targetStreamId are removed. Otherwise, if targetAttrType is equal to ATTR_REFLECT, all PCC NAL units whose PccNalUnitType is not equal to REFLECT_NALU or whose pcc_stream_id is not equal to targetStreamId are removed. Otherwise, if targetAttrType is equal to ATTR_TRANSP, all PCC NAL units whose PccNalUnitType is not equal to TRANSP_NALU or whose pcc_stream_id is not equal to targetStreamId are deleted. Otherwise, if targetAttrType is equal to ATTR_NORMAL, all PCC NAL units whose PccNalUnitType is not equal to NORMAL_NALU or whose pcc_stream_id is not equal to targetStreamId are deleted. For each PCC NAL unit, the first byte may be deleted.

[0107] In an alternative embodiment of the first set of methods summarized above, the PCC NAL unit header is designed to use more bits for pcc_stream_id and to allow four or more streams for each attribute. In that case, one more type is added to the PCC NAL unit header.

[0108] Figure 8 is a schematic diagram showing an exemplary mechanism 800 for encoding PCC attributes 841 and 842 having multiple codecs 843 and 844. For example, mechanism 800 can be employed to encode and / or decode attributes of a PCC video stream 700. Thus, mechanism 800 can be employed to encode and / or decode point cloud medium frames 600 based on point cloud medium 500. In this way, mechanism 800 may be used when the encoder 300 generates a bitstream from the PCC sequence and the decoder 400 reconstructs the PCC sequence from the bitstream. Thus, mechanism 800 can be employed by the codec system 200 and may further be employed to support method 100.

[0109] Mechanism 800 can be applied to multiple PCC attributes 841 and 842. For example, PCC attributes 841 and 842 may be any two attributes selected from a group including geometric shape attributes, texture attributes, reflectivity attributes, transparency attributes, and normal attributes. As shown in Figure 8, mechanism 800 shows the encoding process from left to right and the decoding process from right to left. Codecs 843 and 844 may be any two codecs such as HEVC, AVC, VVC, or any version thereof. Certain codecs 843 and 844, or their versions, may be more efficient than others when encoding certain PCC attributes 841 and 842. In this example, codec 843 is used to encode attribute 841, and codec 844 is used to encode attribute 842, respectively. The results of such encoding are combined to produce a PCC video stream 845 containing both PCC attributes 841 and 842. In the decoder, codec 843 is used to decode attribute 841, and codec 844 is used to decode attribute 842. The decoded attributes 841 and 842 can then be combined again to produce the decoded PCC video stream 845.

[0110] The advantage of employing mechanism 800 is that the most efficient codecs 843 and 844 can be selected for the corresponding attributes 841 and 842. Mechanism 800 is not limited to two attributes 841 and 842, as well as two codecs 843 and 844. For example, each attribute (geometric shape, texture, reflectivity, transparency, and normal) can be encoded by a separate codec. To ensure that the appropriate codecs 843 and 844 can be selected to decode the corresponding attributes 841 and 842, the encoder may signal the codecs 843 and 844, as well as their corresponding attributes 841 and 842, respectively. For example, the encoder may include a syntax element in the GOF header to indicate the codec for the attribute correspondence. The decoder can then read the relevant syntax, select the correct codecs 843 and 844 for attributes 841 and 842, and decode the PCC video stream 845. As a specific example, the identified_codec_for_attribute syntax element may be used to indicate codecs 843 and 844 for attributes 841 and 842, respectively.

[0111] Figure 9 is a schematic diagram 900 showing an example of attribute layers 931, 932, 933, and 934. For example, attribute layers 931, 932, 933, and 934 can be employed to carry the attributes of the PCC video stream 700. Thus, layers 931, 932, 933, and 934 can be used when encoding and / or decoding point cloud medium frames 600 based on the point cloud medium 500. In this way, layers 931, 932, 933, and 934 may be used by the encoder 300 to generate a bitstream from the PCC sequence, and may be used by the decoder 400 when reconstructing the PCC sequence from the bitstream. Thus, layers 931, 932, 933, and 934 can be employed by the codec system 200, and may further be employed to support method 100. Additionally, attribute layers 931, 932, 933, and 934 may be used to carry one or more of the attributes 841 and 842.

[0112] Attribute layers 931, 932, 933, and 934 are groupings of attribute-related data that can be stored and / or modified independently of other groups of data related to the same attribute. In this way, each attribute layer 931, 932, 933, and 934 can be modified and / or represented without affecting the remaining attribute layers 931, 932, 933, and / or 934. In some examples, attribute layers 931, 932, 933, and / or 934 may be visually represented on top of each other, as shown in Figure 9. For example, a texture that covers an entire object (e.g., a more general one) may be stored in attribute layer 931, while a more detailed texture (e.g., a more specific one) may be contained in attribute layers 932, 933, and / or 934. In another example, attribute layers 931 and / or 932 may be applied to odd-numbered frames, and attribute layers 933 and / or 934 may be applied to even-numbered frames. This may allow some layers to be omitted in response to changes in frame rate. Each attribute may have 0 to 4 attribute layers 931, 932, 933, and / or 934. To signal the configuration to be adopted, the encoder may adopt a syntactic element such as num_layers_for_attribute[i] in sequence-level data such as a GOF header. The decoder can read the syntactic element and determine the number of attribute layers 931, 932, 933, and / or 934 to be adopted for each attribute. Additional syntactic elements, such as attribute_layers_combination_mode[i][j], attrLayerIdx[i][j], etc., may also be adopted to indicate the combination of attribute layers adopted in the PCC video stream and the index of each layer used by the corresponding attribute, respectively.

[0113] As yet another example, some attribute layers (e.g., attribute layers 931, 932, and 933) may carry data about regular patches, while other attribute layers (e.g., attribute layer 934) may carry data associated with irregular point cloud patches. This is useful because irregular point clouds may be described using different data than regular cloud patches. To signal that a particular layer is carrying data associated with irregular point clouds, the encoder can encode another syntactic element in the sequence-level data. As a specific example, the regular_points_flag in the GOF header can be used to indicate that an attribute layer is carrying at least one irregular point cloud point. The decoder can then read the syntactic element and decode the corresponding attribute layer accordingly.

[0114] Figure 10 is a schematic diagram 1000 showing examples of attribute streams 1031, 1032, 1033, and 1034. For example, attribute streams 1031, 1032, 1033, and 1034 can be employed to carry the attributes of the PCC video stream 700. Thus, attribute streams 1031, 1032, 1033, and 1034 can be employed when encoding and / or decoding point cloud medium frames 600 based on the point cloud medium 500. In this way, attribute streams 1031, 1032, 1033, and 1034 may be used by the encoder 300 to create a bitstream from the PCC sequence and by the decoder 400 when reconstructing the PCC sequence from the bitstream. Thus, attribute streams 1031, 1032, 1033, and 1034 can be employed by the codec system 200 and may further be employed to support method 100. Additionally, attribute streams 1031, 1032, 1033, and 1034 may be used to carry one or more of the attributes 841 and 842. Furthermore, attribute streams 1031, 1032, 1033, and 1034 can be used to carry attribute layers 931, 932, 933, and 934.

[0115] Attribute streams 1031, 1032, 1033, and 1034 are sequences of attribute data over time. Specifically, attribute streams 1031, 1032, 1033, and 1034 are substreams of the PCC video stream. Each attribute stream 1031, 1032, 1033, and 1034 carries a sequence of attribute-identified NAL units and therefore acts as a storage and / or transmission data structure. Each attribute stream 1031, 1032, 1033, and 1034 may carry one or more attribute layers 931, 932, 933, and 934 of the data. For example, attribute stream 1031 may carry attribute layers 931 and 932, while attribute stream 1032 carries attribute layers 931 and 932 (attribute streams 1033 and 1034 are omitted). In another example, each attribute stream 1031, 1032, 1033, and 1034 carries a single corresponding attribute layer 931, 932, 933, and 934. In yet another example, some attribute streams 1031, 1032, 1033, and 1034 carry multiple attribute layers 931, 932, 933, and 934, while other attribute streams 1031, 1032, 1033, and 1034 carry a single attribute layer 931, 932, 933, and 934, or omit them altogether. As can be seen, many combinations and permutations of attribute streams 1031, 1032, 1033, and 1034, as well as attribute layers 931, 932, 933, and 934, are possible. Therefore, the encoder may employ a syntactic element such as num_streams_for_attribute in sequence-level data, such as a GOF header, to indicate the number of attribute streams 1031, 1032, 1033, and 1034 used to encode each attribute. The decoder can then decode the attribute streams 1031, 1032, 1033, and 1034 using such information, for example, in combination with attribute layer information, to reconstruct the PCC sequence.

[0116] Figure 11 is a flowchart of an exemplary method 1100 for encoding a PCC video sequence having multiple codecs. For example, method 1100 can organize data into a bitstream according to mechanism 800, using attribute layers 931, 932, 933, and 934 and / or streams 1031, 1032, 1033, and / or 1034. Method 1100 may also specify a mechanism used to encode attributes in the GOF header. Furthermore, method 1100 may generate a PCC video stream 700 by encoding point cloud medium frames 600 based on point cloud medium 500. In addition, method 1100 may be employed by a codec system 200 and / or encoder 300 while performing the encoding steps of method 100.

[0117] Method 1100 may be initiated when the encoder receives a sequence of PCC frames containing a point cloud medium. The encoder may decide to encode such frames, for example, in response to the receipt of a user command. In Method 1100, the encoder may decide that a first attribute should be encoded by a first codec, while a second attribute should be encoded by a second codec. This decision may be made on the basis of predetermined conditions and / or user input, for example, when the first codec is more efficient for the first attribute and the second codec is more efficient for the second attribute. Thus, in step 1101, the encoder encodes the first attribute of the sequence of PCC frames into a bitstream with the first codec. Furthermore, in step 1103, the encoder encodes the second attribute of the sequence of PCC frames into a bitstream with a second codec different from the first codec.

[0118] In step 1105, the encoder encodes various syntactic elements into the bitstream along with the encoded video data. For example, syntactic elements may be encoded into sequence-level data units containing sequence-level parameters to indicate to the decoder the decisions made during encoding so that the PCC frame can be properly reconstructed. Specifically, the encoder encodes a sequence-level data unit to include a first syntactic element indicating that a first attribute was encoded by a first codec and a second attribute was encoded by a second codec. As a particular example, a PCC frame may contain multiple attributes, including the first and second attributes. The multiple attributes of the PCC frame may also include geometric shape, texture, and one or more of reflectivity, transparency, and normals. Furthermore, the first syntactic element may be the identified_codec_for_attribute element included in the GOF header in the bitstream.

[0119] In some examples, the first attribute may be organized into multiple streams. In such cases, a second syntactic element can be used to indicate stream membership for the data units of the bitstream associated with the first attribute. In some examples, the first attribute may also be organized into multiple layers. In such cases, a third syntactic element may indicate layer membership for the data units of the bitstream associated with the first attribute. In a specific example, the second syntactic element may be a num_streams_for_attribute element, and the third syntactic element may be a num_layers_for_attribute element, each of which may be included in a group of frame headers in the bitstream. In yet another example, a fourth syntactic element may be used to indicate that the first layer of multiple layers contains data associated with an irregular point cloud. In a specific example, the fourth syntactic element may be a regular_points_flag element included in a group of frame headers in the bitstream.

[0120] By including such information in the sequence-level data, the decoder may have sufficient information to decode the PCC video sequence. Thus, in step 1107, the encoder may transmit a bitstream based on a first attribute encoded by a first codec and a second attribute encoded by a second codec, as well as other attributes and / or syntactic elements described herein, in order to support the generation of a decoded sequence of PCC frames.

[0121] Figure 12 is a flowchart of an exemplary method 1200 for decoding a PCC video sequence with multiple codecs. For example, method 1200 can read data from the bitstream according to mechanism 800, using attribute layers 931, 932, 933, and 934, and / or streams 1031, 1032, 1033, and / or 1034. Alternatively, method 1200 may determine the mechanism used to encode the attributes by reading the GOF header. Furthermore, method 1200 may read the PCC video stream 700 to reconstruct the point cloud medium frame 600 and the point cloud medium 500. In addition, method 1200 may be employed by a codec system 200 and / or decoder 400 while performing the decoding steps of method 100.

[0122] Method 1200 may begin in step 1201 when the decoder receives a bitstream containing a series of PCC frames. The decoder can then parse the bitstream or a portion of it in step 1205. For example, the decoder can parse the bitstream to obtain a sequence-level data unit containing sequence-level parameters. The sequence-level data unit may contain various syntactic elements that describe the encoding process. Thus, the decoder can use the syntactic elements to parse the video data from the bitstream and determine an appropriate process for decoding the video data.

[0123] For example, a sequence-level data unit may include a first syntactic element indicating that a first attribute was coded by a first codec and a second attribute was coded by a second codec. In a specific example, a PCC frame may include multiple attributes, including the first and second attributes. The multiple attributes of a PCC frame may also include geometric shape, texture, and one or more of reflectivity, transparency, and normals. Additionally, the first syntactic element may be the identified_codec_for_attribute element included in the GOF header in the bitstream.

[0124] In some examples, the first attribute may be organized into multiple streams. In such cases, a second syntactic element may be used to indicate stream membership for the data units of the bitstream associated with the first attribute. In some examples, the first attribute may also be organized into multiple layers. In such cases, a third syntactic element may indicate layer membership for the data units of the bitstream associated with the first attribute. In a specific example, the second syntactic element may be a num_streams_for_attribute element, and the third syntactic element may be a num_layers_for_attribute element, each of which may be included in a group of frame headers in the bitstream. In yet another example, a fourth syntactic element may be used to indicate that the first layer of multiple layers contains data associated with an irregular point cloud. In a specific example, the fourth syntactic element may be a regular_points_flag element included in a group of frame headers in the bitstream.

[0125] Thus, in step 1207, the decoder may decode a first attribute using a first codec and a second attribute using a second codec to generate a decoded sequence of PCC frames. The decoder may also use other attributes and / or syntactic elements, such as those described herein, when determining an appropriate mechanism to employ when decoding various attributes of the PCC video sequence based on the codec.

[0126] Figure 13 is a schematic diagram of an exemplary video coding device 1300. The video coding device 1300 is suitable for carrying out the disclosed examples / embodiments as described herein. The video coding device 1300 includes a transceiver unit 1310, which includes a downstream port 1320, an upstream port 1350, and / or a transmitter and / or receiver for communicating data upstream and / or downstream over a network. The video coding device 1300 also includes a processor 1330, which includes a logic unit and / or a central processing unit (CPU), and memory 1332 for storing data. The video coding device 1300 may also include electrical, optical-electrical (OE) components, electrical-optical (EO) components, and / or wireless communication components coupled to the upstream port 1350 and / or downstream port 1320 for communicating data over an electrical, optical, or wireless communication network. The video coding device 1300 may also include input and / or output (I / O) devices 1360 for communicating data with a user. The I / O device 1360 may include output devices such as a display for displaying video data and a speaker for outputting audio data. The input / output device 1360 may also include input devices such as a keyboard, mouse, or trackball, and / or corresponding interfaces for interacting with such output devices.

[0127] The processor 1330 is implemented by hardware and software. The processor 1330 may be implemented as one or more CPU chips, cores (e.g., as a multi-core processor), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), and digital signal processors (DSPs). The processor 1330 communicates with downstream port 1320, Tx / Rx 1310, upstream port 1350, and memory 1332. The processor 1330 includes a coding module 1314. The coding module 1314 implements the embodiments disclosed above, for example, methods 100, 1100, 1200, 1500, and 1600, and mechanism 800, which may employ point cloud media 500, point cloud media frames 600, and / or PCC video streams 700 coded to layers 931-934 and / or streams 1031-1034. Furthermore, the coding module 1314 may implement any other methods / mechanisms described herein. Additionally, the coding module 1314 may implement a codec system 200, an encoder 300, and / or a decoder 400. For example, the coding module 1314 may employ an extended attribute set for PCCs having multiple streams and layers, and may signal the use of such an attribute set in sequence-level data to support decoding. Thus, the coding module 1314 provides additional functionality and / or flexibility to the video coding device 1300 when coding PCC video data. In this way, the coding module 1314 improves the functionality of the video coding device 1300 and addresses problems specific to video coding techniques. Furthermore, the coding module 1314 results in the video coding device 1300 being transformed into different states. Alternatively, the coding module 1314 can be implemented as instructions stored in memory 1332 and executed by processor 1330 (e.g., as a computer program product stored on a non-temporary medium).

[0128] Memory 1332 includes one or more memory types, such as disks, tape drives, solid-state drives, read-only memory (ROM), random-access memory (RAM), flash memory, ternary content-addressable memory (TCAM), and static random-access memory (SRAM). Memory 1332 may also be used as an overflow data storage device to store the program when it is selected for execution and to store instructions and data read during program execution.

[0129] Figure 14 is a schematic diagram of an exemplary system 1400 for coding a PCC video sequence having multiple codecs. System 1400 includes a video encoder 1402 which includes a first attribute encoding module 1401 for encoding a first attribute of a sequence of PCC frames into a bitstream using a first codec. Video encoder 1402 further includes a second attribute encoding module 1403 for encoding a second attribute of a sequence of PCC frames into a bitstream using a second codec different from the first codec. Video encoder 1402 further includes a syntax encoding module 1405 for encoding a sequence-level data unit containing sequence-level parameters into a bitstream, the sequence-level data unit containing a first syntax element indicating that the first attribute was encoded by the first codec and the second attribute was encoded by the second codec. The video encoder 1402 further includes a transmit module 1407 for transmitting a bitstream to support the generation of a decoded sequence of PCC frames based on a first attribute encoded by a first codec and a second attribute encoded by a second codec. The modules of the video encoder 1402 can also be employed to perform any of the steps / items described above with respect to methods 1100 and / or 1500.

[0130] System 1400 also includes a video decoder 1410, which includes a receiving module 1411 for receiving a bitstream containing a sequence of PCC frames. The video decoder 1410 further includes an parsing module 1413 for parsing the bitstream to obtain a sequence-level data unit containing sequence-level parameters, the sequence-level data unit containing a first syntactic element indicating that a first attribute of the PCC frame was coded by a first codec and a second attribute of the PCC frame was coded by a second codec. The video decoder 1410 further includes a decoding module 1415 for decoding the first attribute by the first codec and decoding the second attribute by the second codec to produce a decoded sequence of PCC frames. The modules of the video decoder 1410 can also be employed to perform any of the steps / items described above with respect to method 1200 and / or 1600.

[0131] Figure 15 is a flowchart of another exemplary method 1500 for encoding a PCC video sequence with multiple codecs. For example, method 1500 can organize data into a bitstream according to mechanism 800, using attribute layers 931, 932, 933, and 934 and / or streams 1031, 1032, 1033, and / or 1034. Method 1500 may also specify a mechanism used to encode attributes in the GOF header. Furthermore, method 1500 may generate a PCC video stream 700 by encoding point cloud medium frames 600 based on point cloud medium 500. In addition, method 1500 may be employed by a codec system 200 and / or encoder 300 while performing the encoding steps of method 100.

[0132] In step 1501, multiple PCC attributes are encoded into a bitstream as part of a sequence of PCC frames. The PCC attributes are encoded using multiple codecs. The PCC attributes include geometric shapes and textures. The PCC attributes also include one or more of the following: reflectivity, transparency, and normals. Each encoded PCC frame is represented by one or more PCC NAL units. In step 1503, an instruction is encoded for each PCC attribute. This instruction indicates the video codec used to encode the corresponding PCC attribute. In step 1505, the bitstream is sent toward the decoder.

[0133] Figure 16 is a flowchart of another exemplary method 1600 for decoding a PCC video sequence with multiple codecs. For example, method 1600 can read data from the bitstream according to mechanism 800, using attribute layers 931, 932, 933, and 934, and / or streams 1031, 1032, 1033, and / or 1034. Alternatively, method 1600 may determine the mechanism used to encode the attributes by reading the GOF header. Furthermore, method 1600 may read the PCC video stream 700 to reconstruct the point cloud medium frame 600 and the point cloud medium 500. In addition, method 1600 may be employed by a codec system 200 and / or decoder 400 while performing the decoding step of method 100.

[0134] In step 1601, a bitstream is received. The bitstream contains an encoded sequence of multiple PCC frames. The encoded sequence of PCC frames represents multiple PCC attributes. PCC attributes include geometric shapes and textures. PCC attributes also include one or more of reflectivity, transparency, and normals. Each encoded PCC frame is represented by one or more PCC NAL units. In step 1603, the bitstream is parsed for each PCC attribute to obtain instructions for the codec used to encode the corresponding PCC attribute. In step 1605, the bitstream is decoded based on the video codec indicated for the PCC attribute.

[0135] The first component is directly joined to the second component when there are no intermediary components other than lines, traces, or other media between the first and second components. The first component is indirectly joined to the second component when there are intermediary components other than lines, traces, or other media between the first and second components. The term "joined" and its variations include both direct and indirect joining. The use of the term "about" means a range including ±10% of the following number unless otherwise specified.

[0136] While several embodiments are provided in this disclosure, it will be understood that the disclosed systems and methods can be embodied in many other specific forms without departing from the spirit or scope of this disclosure. These examples are illustrative and not limiting, and their intent is not limited to the details given herein. For example, various elements or components can be combined or integrated into another system, or certain features can be omitted or not implemented.

[0137] In addition, the technologies, systems, subsystems, and methods described and illustrated individually or separately in various embodiments may be combined with or integrated with other systems, components, technologies, or methods without departing from the scope of this disclosure. Other examples of modifications, substitutions, and alterations are readily apparent to those skilled in the art and may be made without departing from the spirit and scope disclosed herein.

Claims

1. A method of decryption, Receiving a bitstream of a point cloud, wherein the bitstream includes at least two attributes for describing the point cloud, each of which is a texture, reflectivity, transparency, or normal, and the bitstream further includes instructional information, the instructional information indicating that a first attribute of the at least two attributes is coded by a first video codec among a plurality of video coders (codecs), and a second attribute of the at least two attributes is coded by a second video codec among the plurality of video codecs, The bitstream is analyzed to obtain the instruction information, A method comprising decoding the at least two attributes indicated by the instruction information using a video codec.

2. Decoding the at least two attributes indicated by the instruction information using a video codec is: The method according to claim 1, comprising decoding the first attribute with the first video codec and decoding the second attribute with the second video codec.

3. The method according to claim 1 or 2, wherein the instruction information includes a first correspondence between the first video codec and the first attribute, and a second correspondence between the second video codec and the second attribute.

4. The method according to any one of claims 1 to 3, wherein the instruction information includes at least two syntactic elements that are included in a group of frame headers in the bitstream.

5. The method according to claim 4, wherein the i-th syntactic element in the at least two syntactic elements specifies a video codec used to code the attribute having index i, where i ranges from 0 to the number of attributes in the at least two attributes.

6. The method according to any one of claims 1 to 5, wherein the plurality of video codecs include high-efficiency video coding (HEVC) or advanced video coding (AVC).

7. A method of encoding, Encoding at least two attributes for describing the point cloud into a bitstream of the point cloud, wherein each of the at least two attributes is a texture, reflectivity, transparency, or normal, The method of encoding instruction information into the bitstream, wherein the instruction information indicates that a first attribute of the at least two attributes is encoded by a first video codec among a plurality of video coders / decoders (codecs), and a second attribute of the at least two attributes is encoded by a second video codec among the plurality of video codecs. Methods that include...

8. The method according to claim 7, wherein the instruction information includes a first correspondence between the first video codec and the first attribute, and a second correspondence between the second video codec and the second attribute.

9. The method according to claim 7 or 8, wherein the instruction information includes at least two syntactic elements that are included in a group of frame headers in the bitstream.

10. The method according to claim 9, wherein the i-th syntactic element in the at least two syntactic elements specifies a video codec used to code the attribute having index i, where i ranges from 0 to the number of attributes in the at least two attributes.

11. The method according to any one of claims 7 to 10, wherein the plurality of video codecs include high-efficiency video coding (HEVC) or advanced video coding (AVC).

12. The method according to any one of claims 7 to 11, further comprising transmitting the bitstream toward a decoder or decoding device.

13. A decoding device comprising means for performing the method described in any one of claims 1 to 6.

14. An encoding device comprising means for performing the method described in any one of claims 7 to 12.

15. A computer-readable medium containing a computer-executable instruction, wherein the computer-executable instruction, when executed by a processor, causes a device to perform the method according to any one of claims 1 to 12.

16. A decoder including a processing circuit for performing the method according to any one of claims 1 to 6.

17. An encoder including a processing circuit for performing the method according to any one of claims 7 to 12.

18. A method for storing a bitstream of a point cloud, The bitstream is stored on a non-temporary computer-readable recording medium, The bitstream includes at least two attributes for describing the point cloud, each of which is a texture, reflectance, transparency, or normal, and the bitstream further includes instructional information. The instruction information indicates that the first attribute of the at least two attributes is encoded by a first video codec among the plurality of video coders / decoders (codecs), and that the second attribute of the at least two attributes is encoded by a second video codec among the plurality of video codecs. A method comprising decoding the at least two attributes contained in the bitstream using the video codec indicated by the instruction information.

19. The instruction information includes a first correspondence between the first video codec and the first attribute, and a second correspondence between the second video codec and the second attribute. The method according to claim 18, wherein the instruction information causes a computer device or processor to decode the first attribute using the first video codec and decode the second attribute using the second video codec.

20. The method according to claim 18 or 19, wherein the instruction information includes at least two syntactic elements that are included in a group of frame headers in the bitstream.

21. The method according to claim 20, wherein the i-th syntactic element in the at least two syntactic elements specifies a video codec used to code the attribute having index i, where i ranges from 0 to the number of attributes in the at least two attributes.

22. The method according to any one of claims 18 to 21, wherein the plurality of video codecs include high-efficiency video coding (HEVC) or advanced video coding (AVC).

Images