Coding and decoding of patch data unit for point group coding

Abstract

To provide a method of a point group coding (PCC) executed by a decoder.SOLUTION: A method comprises steps of: receiving patch information data coded; acquiring a patch having a patch type corresponded to the patch information data coded; determining whether or not a patch type for a patch is a final patch type; and terminating decoding processing corresponded to the patch information data decoded, when the patch type is the final patch type. An another method comprises steps of: determining whether or not the patch type for the patch is a skip patch type; decoding a reference patch index corresponded to the patch when the patch type is the skip patch type; deciding a reference index for the patch on the basis of the reference patch index decoded, when a reference frame index corresponded to the patch and the patch type are the skip patch type; and decoding a stereoscopic expression of the patch.SELECTED DRAWING: Figure 18

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This patent application claims the benefit of U.S. Provisional Patent Application No. 62 / 817,391, filed March 12, 2019, by Vladyslav Zakharchenko et al., entitled “Patch Data Unit Coding and Decoding for Point-Cloud Coding,” which is incorporated herein by reference.

[0002] The present disclosure relates generally to point cloud coding, and more particularly to a high-level syntax for point cloud coding. [Background technology]

[0003] Point clouds are being adopted in a wide variety of applications, including the entertainment industry, intelligent automobile navigation, geospatial surveying, three-dimensional (3D) modeling and visualization of real-world objects, etc. Given the non-uniform sampling geometry of point clouds, a compact representation for storage and transmission of such data is useful. Compared to other 3D representations, irregular point clouds are more general and applicable to a wider range of sensors and data acquisition strategies. For example, when performing 3D representations in virtual reality worlds or remote rendering in telepresence environments, rendering of virtual figures and real-time instructions are processed as dense point cloud datasets. Summary of the Invention [Means for solving the problem]

[0004] A first aspect relates to a method of Point Cloud Coding (PCC) implemented by a decoder, the method including the steps of receiving, by a receiver of the decoder, encoded patch information data, obtaining, by a processor of the decoder, a patch corresponding to the encoded patch information data, the patch having a patch type (patch_mode), determining, by the processor, whether the patch type for the patch is the last patch type, and terminating, by the processor, a recovery process corresponding to the encoded patch information data when the patch type is the last patch type.

[0005] Using this coding method allows for flexible ordering of patches within a patch frame data unit. That is, patches with different patch modes (e.g., inter, intra, PCM, etc.) may be included in a patch frame data unit in any order. In addition, additional patches may be added to a patch frame data unit. Added patches may be added either to the end of the patch frame data unit or at any random patch index position. The coding technique also allows for a skip patch data unit type (spdu) to be used. A skip patch data unit indicates that all parameters for the current patch may be inherited from the reference patch when the patch data unit elements for the current patch data unit and the reference patch data unit are identical or within an acceptable tolerance. The coding technique also allows for the syntax element indicating the number of matched patches to be removed from the signaling, since the signaling of the patch types described herein is sufficient to represent the entire list of patches in a patch frame data unit. Furthermore, the coding technique eliminates the need to signal the total number of patches in a patch frame data unit (e.g., the size of the patch frame data unit) and replaces that process with a special end patch type. Furthermore, the coding technique allows a pulse-coded modulation (PCM) patch, which is usually located at the end of a patch frame data unit, to be replaced with a PCM patch type indication. Multiple PCM patches are also allowed. Thus, the coder / decoder (a.k.a. "codec") in point cloud coding is improved over current codecs (e.g., the process of encoding and decoding patches in V-PCC). In practical terms, the improved point cloud coding process can improve coding efficiency, thereby providing users with a better user experience when point clouds are transmitted, received, and / or viewed.

[0006] A second aspect relates to a method of point cloud coding (PCC) implemented by a decoder, the method including the steps of receiving, by a receiver of the decoder, encoded patch information data, obtaining, by a processor of the decoder, a patch corresponding to the encoded patch information data, the patch having a patch type (patch_mode), determining, by the processor, whether the patch type for the patch is a skip patch type, decoding, by the processor, a reference patch index (spdu_patch_index) corresponding to the patch when the patch type is the skip patch type, determining, by the processor, a reference index (refIdx) for the patch based on a reference frame index ([refFrmIdx]) corresponding to the patch and the decoded reference patch index when the patch type is the skip patch type, and recovering, by the processor, a stereoscopic representation of the patch based on the determined reference index when the patch type is the skip patch type.

[0007] Using this coding method allows for flexible ordering of patches within a patch frame data unit. That is, patches with different patch modes (e.g., inter, intra, PCM, etc.) may be included in a patch frame data unit in any order. In addition, additional patches may be added to a patch frame data unit. Added patches may be added either to the end of the patch frame data unit or at any random patch index position. The coding technique also allows for a skip patch data unit type (spdu) to be used. A skip patch data unit indicates that all parameters for the current patch may be inherited from the reference patch when the patch data unit elements for the current patch data unit and the reference patch data unit are identical or within an acceptable tolerance. The coding technique also allows for the syntax element indicating the number of matched patches to be removed from the signaling, since the signaling of the patch types described herein is sufficient to represent the entire list of patches in a patch frame data unit. Furthermore, the coding technique eliminates the need to signal the total number of patches in a patch frame data unit (e.g., the size of the patch frame data unit) and replaces that process with a special end patch type. Furthermore, the coding technique allows a pulse-coded modulation (PCM) patch, which is usually located at the end of a patch frame data unit, to be replaced with a PCM patch type indication. Multiple PCM patches are also allowed. Thus, the coder / decoder (a.k.a. "codec") in point cloud coding is improved over current codecs (e.g., the process of encoding and decoding patches in V-PCC). In practical terms, the improved point cloud coding process can improve coding efficiency, thereby providing users with a better user experience when point clouds are transmitted, received, and / or viewed.

[0008] In a first implementation form of the method according to the second aspect itself, the method further comprises the steps of determining that a patch type for the patch is an intra-patch type; determining a reference index (refIdx) for the patch based on a frame index ([frmIdx]) corresponding to the patch and a decremented patch index; decoding the patch using a two-dimensional (2D) component corresponding to the patch and a three-dimensional (3D) component corresponding to the patch; and restoring a stereoscopic representation based on the decoded patch.

[0009] In a second implementation form of the method according to the second aspect itself or any of the aforementioned implementation forms of the second aspect, the method further comprises the steps of determining that a patch type for the patch is an inter-patch type, decoding a second reference patch index (dpdu_patch_index) corresponding to the patch, determining a reference index (refIdx) for the patch based on a reference frame index ([refFrmIdx]) corresponding to the patch and the decoded second reference patch index, decoding the patch using a two-dimensional (2D) component corresponding to the patch and a three-dimensional (3D) component corresponding to the patch, and restoring a stereoscopic representation based on the decoded patch.

[0010] In a third implementation of the method according to the second aspect itself or any of the aforementioned implementations of the second aspect, the method further comprises determining that a patch type for the patch is a pulse code modulation (PCM) patch type; determining a reference index (refIdx) for the patch based on a frame index ([frmIdx]) corresponding to the patch and a decremented patch index; decoding the patch using two-dimensional (2D) components for independent points corresponding to the patch and three-dimensional (3D) components for independent points corresponding to the patch; and restoring a stereoscopic representation based on the decoded patch.

[0011] In a fourth implementation form of the method according to the second aspect itself or any of the aforementioned implementation forms of the second aspect, the method further comprises a step of receiving an input corresponding to encoded patch information data, the input comprising one or more of patch_mode, a patch index, a reference index, a frame index, and a reference frame index.

[0012] In a fourth implementation form of the method according to the second aspect itself or any of the aforementioned implementation forms of the second aspect, the method further comprises a step of displaying an image generated based on the restored stereoscopic representation on a display of the electronic device.

[0013] A third aspect relates to a method of point cloud coding (PCC) implemented by a decoder, the method including the steps of receiving, by a receiver of the decoder, encoded patch information data, and obtaining, by a processor of the decoder, a patch corresponding to the encoded patch information data, the patch having a patch type (patch_mode), determining by the processor whether the patch type for the patch is a skip patch type, and when the patch type is a skip patch type, decoding by the processor a reference patch index (spdu_patch_index) corresponding to the patch, and determining by the processor a reference frame index ([refFrmIdx]) corresponding to the patch and whether the patch type is a skip patch type. determining, by the processor, a reference index (refIdx) for the patch based on the reference patch index decoded when the patch type is a skip patch type; restoring, by the processor, a stereoscopic representation of the patch based on the determined reference index when the patch type is a skip patch type; determining, by the processor, whether the more patch available flag has a first value or a second value; storing, by the processor, the restored stereoscopic representation in a memory of the decoder when the more patch available flag has the first value; and terminating, by the processor, the restoration process of the encoded patch information when the more patch available flag has the second value.

[0014] Using this coding method allows for flexible ordering of patches within a patch frame data unit. That is, patches with different patch modes (e.g., inter, intra, PCM, etc.) may be included in a patch frame data unit in any order. In addition, additional patches may be added to a patch frame data unit. Added patches may be added either to the end of the patch frame data unit or at any random patch index position. The coding technique also allows for a skip patch data unit type (spdu) to be used. A skip patch data unit indicates that all parameters for the current patch may be inherited from the reference patch when the patch data unit elements for the current patch data unit and the reference patch data unit are identical or within an acceptable tolerance. The coding technique also allows for the syntax element indicating the number of matched patches to be removed from the signaling, since the signaling of the patch types described herein is sufficient to represent the entire list of patches in a patch frame data unit. Furthermore, the coding technique eliminates the need to signal the total number of patches in a patch frame data unit (e.g., the size of the patch frame data unit) and replaces that process with a special end patch type. Furthermore, the coding technique allows a pulse-coded modulation (PCM) patch, which is usually located at the end of a patch frame data unit, to be replaced with a PCM patch type indication. Multiple PCM patches are also allowed. Thus, the coder / decoder (a.k.a. "codec") in point cloud coding is improved over current codecs (e.g., the process of encoding and decoding patches in V-PCC). In practical terms, the improved point cloud coding process can improve coding efficiency, thereby providing users with a better user experience when point clouds are transmitted, received, and / or viewed.

[0015] In a first implementation form of the method according to the third aspect itself, the method further comprises the steps of determining that a patch type for the patch is an intra-patch type; determining a reference index (refIdx) for the patch based on a frame index ([frmIdx]) corresponding to the patch and a decremented patch index; decoding the patch using a two-dimensional (2D) component corresponding to the patch and a three-dimensional (3D) component corresponding to the patch; and restoring a stereoscopic representation based on the decoded patch.

[0016] In a second implementation form of the method according to the third aspect itself, the method further comprises the steps of determining that a patch type for the patch is an inter-patch type; decoding a second reference patch index (dpdu_patch_index) corresponding to the patch; determining a reference index (refIdx) for the patch based on a reference frame index ([refFrmIdx]) corresponding to the patch and the decoded second reference patch index; decoding the patch using a two-dimensional (2D) component corresponding to the patch and a three-dimensional (3D) component corresponding to the patch; and restoring a stereoscopic representation based on the decoded patch.

[0017] In a third implementation of the method according to the third aspect itself, the method further comprises the steps of determining that a patch type for the patch is a pulse code modulation (PCM) patch type; determining a reference index (refIdx) for the patch based on a frame index ([frmIdx]) corresponding to the patch and a decremented patch index; decoding the patch using two-dimensional (2D) components for independent points corresponding to the patch and three-dimensional (3D) components for independent points corresponding to the patch; and restoring a stereoscopic representation based on the decoded patch.

[0018] In a fourth implementation form of the method according to the third aspect itself, the method further comprises a step of receiving an input corresponding to encoded patch information data, the input comprising one or more of patch_mode, a patch index, a reference index, a frame index, and a reference frame index.

[0019] In a fifth implementation form of the method according to the third aspect itself, the method further comprises a step of displaying an image generated based on the reconstructed stereoscopic representation on a display of the electronic device.

[0020] A fourth aspect relates to a method of point cloud coding (PCC) implemented by an encoder, the method including the steps of obtaining, by a receiver of the encoder, a patch frame data unit (pfdu) identifying a patch type (pdfu_patch_mode) for each of a plurality of patches, determining, by a processor of the encoder, whether a patch type for a patch from the plurality of patches is a last patch type, encoding, by the processor of the encoder, patch information data for the patch when the patch type is not the last patch type, the patch information data including the patch type for the patch, and encoding, by the processor of the encoder, patch information data for the patch when the patch type is set to the last patch type, the patch information data including the last patch type for the patch.

[0021] Using this coding method allows for flexible ordering of patches within a patch frame data unit. That is, patches with different patch modes (e.g., inter, intra, PCM, etc.) may be included in a patch frame data unit in any order. In addition, additional patches may be added to a patch frame data unit. Added patches may be added either to the end of the patch frame data unit or at any random patch index position. The coding technique also allows for a skip patch data unit type (spdu) to be used. A skip patch data unit indicates that all parameters for the current patch may be inherited from the reference patch when the patch data unit elements for the current patch data unit and the reference patch data unit are identical or within an acceptable tolerance. The coding technique also allows for the syntax element indicating the number of matched patches to be removed from the signaling, since the signaling of the patch types described herein is sufficient to represent the entire list of patches in a patch frame data unit. Furthermore, the coding technique eliminates the need to signal the total number of patches in a patch frame data unit (e.g., the size of the patch frame data unit) and replaces that process with a special end patch type. Furthermore, the coding technique allows a pulse-coded modulation (PCM) patch, which is usually located at the end of a patch frame data unit, to be replaced with a PCM patch type indication. Multiple PCM patches are also allowed. Thus, the coder / decoder (a.k.a. "codec") in point cloud coding is improved over current codecs (e.g., the process of encoding and decoding patches in V-PCC). In practical terms, the improved point cloud coding process can improve coding efficiency, thereby providing users with a better user experience when point clouds are transmitted, received, and / or viewed.

[0022] In a first implementation form of the method according to the fourth aspect itself, the patch type is one of a skip patch type, an inter patch type, an intra patch type, and a pulse code modulation (PCM) patch type.

[0023] In a second implementation form of the method according to the fourth aspect itself, the patch frame data unit includes a frame index (frmIdx), a two-dimensional (2D) component corresponding to the patch, and a three-dimensional (3D) component corresponding to the patch.

[0024] In a third implementation form of the method according to the fourth aspect itself, the patch information data includes a frame index, a two-dimensional (2D) component corresponding to the patch, and a three-dimensional (3D) component corresponding to the patch.

[0025] In a fourth implementation form of the method according to the fourth aspect itself, the method further comprises repeating the steps of determining whether the patch type is the last patch type and encoding patch information data for a subsequent patch from the plurality of patches until it is determined that one of the patches from the plurality of patches has the last patch type.

[0026] In a fifth implementation of the method according to the fourth aspect itself, the method further comprises performing byte alignment after all of the patch information data has been encoded.

[0027] In a sixth implementation form of the method according to the fourth aspect itself, the method further comprises a step of generating a compressed patch frame data unit after all of the patch information data has been encoded.

[0028] In a seventh implementation form of the method according to the fourth aspect itself, the method further comprises storing the compressed patch frame data unit in a memory of the encoder for transmission towards the decoder.

[0029] A fifth aspect relates to a method of point cloud coding (PCC) implemented by an encoder, the method including obtaining, by a receiver of the encoder, a patch frame data unit (PFDU) for each of a plurality of patches, adding, by a processor of the encoder, a last patch flag to each of the plurality of patches, determining, by the processor of the encoder, whether a patch type for a patch from the plurality of patches is a last patch type based on a value of the last patch flag, and encoding, by the processor of the encoder, patch information data for the patch when the patch type is not the last patch type, the patch information data including the patch type and the last patch flag for the patch.

[0030] Using this coding method allows for flexible ordering of patches within a patch frame data unit. That is, patches with different patch modes (e.g., inter, intra, PCM, etc.) may be included in a patch frame data unit in any order. In addition, additional patches may be added to a patch frame data unit. Added patches may be added either to the end of the patch frame data unit or at any random patch index position. The coding technique also allows for a skip patch data unit type (spdu) to be used. A skip patch data unit indicates that all parameters for the current patch may be inherited from the reference patch when the patch data unit elements for the current patch data unit and the reference patch data unit are identical or within an acceptable tolerance. The coding technique also allows for the syntax element indicating the number of matched patches to be removed from the signaling, since the signaling of the patch types described herein is sufficient to represent the entire list of patches in a patch frame data unit. Furthermore, the coding technique eliminates the need to signal the total number of patches in a patch frame data unit (e.g., the size of the patch frame data unit) and replaces that process with a special end patch type. Furthermore, the coding technique allows a pulse-coded modulation (PCM) patch, which is usually located at the end of a patch frame data unit, to be replaced with a PCM patch type indication. Multiple PCM patches are also allowed. Thus, the coder / decoder (a.k.a. "codec") in point cloud coding is improved over current codecs (e.g., the process of encoding and decoding patches in V-PCC). In practical terms, the improved point cloud coding process can improve coding efficiency, thereby providing users with a better user experience when point clouds are transmitted, received, and / or viewed.

[0031] In a first implementation form of the method according to the fifth aspect itself, the patch type is one of a skip patch type, an inter patch type, an intra patch type, and a pulse code modulation (PCM) patch type.

[0032] In a second implementation form of the method according to the fifth aspect itself, the patch frame data unit includes a frame index (frmIdx), a two-dimensional (2D) component corresponding to the patch, and a three-dimensional (3D) component corresponding to the patch.

[0033] In a third implementation form of the method according to the fifth aspect itself, the patch information data includes a frame index, a two-dimensional (2D) component corresponding to the patch, and a three-dimensional (3D) component corresponding to the patch.

[0034] In a fourth implementation form of the method according to the fifth aspect itself, the method further comprises a step of determining whether the patch type is the last patch type based on the value of the last patch flag, and repeating the step of encoding patch information data for a subsequent patch from the plurality of patches until one of the patches from the plurality of patches is determined to have the last patch type based on the value of the last patch flag.

[0035] In a fifth implementation form of the method according to the fifth aspect itself, the method further comprises a step of performing byte alignment after it is determined that one of the patches from the plurality of patches has a last patch type based on a value of the last patch flag.

[0036] In a sixth implementation form of the method according to the fifth aspect itself, the method further comprises a step of generating a compressed patch frame data unit after all of the patch information data has been encoded.

[0037] In a seventh implementation form of the method according to the fifth aspect itself, the method further comprises storing the compressed patch frame data unit in a memory of the encoder for transmission towards the decoder.

[0038] A sixth aspect relates to a decoding device including a receiver configured to receive encoded patch information; a memory coupled to the receiver, the memory storing instructions; and a processor coupled to the memory, the processor configured to execute instructions to cause the decoding device to obtain a patch corresponding to the encoded patch information data, the patch having a patch type (patch_mode), determine whether the patch type for the patch is the last patch type, and terminate a restoration process corresponding to the encoded patch information data when the patch type is the last patch type.

[0039] Using this coding method allows for flexible ordering of patches within a patch frame data unit. That is, patches with different patch modes (e.g., inter, intra, PCM, etc.) may be included in a patch frame data unit in any order. In addition, additional patches may be added to a patch frame data unit. Added patches may be added either to the end of the patch frame data unit or at any random patch index position. The coding technique also allows for a skip patch data unit type (spdu) to be used. A skip patch data unit indicates that all parameters for the current patch may be inherited from the reference patch when the patch data unit elements for the current patch data unit and the reference patch data unit are identical or within an acceptable tolerance. The coding technique also allows for the syntax element indicating the number of matched patches to be removed from the signaling, since the signaling of the patch types described herein is sufficient to represent the entire list of patches in a patch frame data unit. Furthermore, the coding technique eliminates the need to signal the total number of patches in a patch frame data unit (e.g., the size of the patch frame data unit) and replaces that process with a special end patch type. Furthermore, the coding technique allows a pulse-coded modulation (PCM) patch, which is usually located at the end of a patch frame data unit, to be replaced with a PCM patch type indication. Multiple PCM patches are also allowed. Thus, the coder / decoder (a.k.a. "codec") in point cloud coding is improved over current codecs (e.g., the process of encoding and decoding patches in V-PCC). In practical terms, the improved point cloud coding process can improve coding efficiency, thereby providing users with a better user experience when point clouds are transmitted, received, and / or viewed.

[0040] A seventh aspect relates to a decoding device including: a receiver configured to receive encoded patch information; a memory coupled to the receiver, the memory storing instructions; and a processor coupled to the memory, the processor configured to execute instructions to cause the decoding device to obtain a patch corresponding to the encoded patch information data, the patch having a patch type (patch_mode), determine whether the patch type for the patch is a skip patch type, decode a reference patch index (spdu_patch_index) corresponding to the patch when the patch type is the skip patch type, determine a reference index (refIdx) for the patch based on a reference frame index ([refFrmIdx]) corresponding to the patch and the reference patch index decoded when the patch type is the skip patch type, and restore a stereoscopic representation of the patch based on the reference index determined when the patch type is the skip patch type.

[0041] The use of this decoding device allows for flexible ordering of patches in the patch buffer. That is, patches with different patch modes (e.g., inter, intra, PCM, etc.) may be included in the patch buffer in any order. The coding technique also allows a skip patch data unit type (spdu) to be used. A skip patch data unit indicates that all parameters for the current patch may be inherited from a reference patch when the patch data unit elements for the current patch data unit and the reference patch data unit are identical or match exactly. The coding technique also allows syntax elements indicating a list of matched patches to be removed from the signaling, since the patch type signaling described herein is sufficient to represent the entire list of patches in the patch buffer. In addition, the coding technique eliminates the need to signal the total number of patches in the patch buffer (e.g., the size of the patch buffer) and replace that process with a special end patch type. Furthermore, the coding technique allows a pulse-coded modulation (PCM) patch, which is typically located at the end of the patch buffer, to be replaced with a PCM patch type indication. Thus, the coder / decoder (a.k.a. "codec") in video coding is improved over the current codec (e.g., the process of encoding and decoding a patch in a V-PCC). In practical terms, the improved video coding process can improve coding efficiency, thereby providing a better user experience when a video is transmitted, received, and / or viewed.

[0042] An eighth aspect includes a receiver configured to receive encoded patch information, a memory coupled to the receiver, the memory storing instructions, and a processor coupled to the memory, the processor obtaining a patch corresponding to the encoded patch information data, the patch having a patch type (patch_mode), determining whether the patch type for the patch is a skip patch type, and when the patch type is a skip patch type, decoding a reference patch index (spdu_patch_index) corresponding to the patch, decoding a reference frame index ([refFrmIdx]) corresponding to the patch, and determining whether the patch type is a skip patch type. and a processor configured to execute instructions to cause the decoding device to: determine a reference index (refIdx) for the patch based on the decoded reference patch index when the patch type is a skip patch type; restore a stereoscopic representation of the patch based on the determined reference index when the patch type is a skip patch type; determine whether a more patch available flag has a first value or a second value; store the restored stereoscopic representation in a memory when the more patch available flag has the first value; and terminate a restoration process of the encoded patch information when the more patch available flag has the second value.

[0043] Using this coding method allows for flexible ordering of patches within a patch frame data unit. That is, patches with different patch modes (e.g., inter, intra, PCM, etc.) may be included in a patch frame data unit in any order. In addition, additional patches may be added to a patch frame data unit. Added patches may be added either to the end of the patch frame data unit or at any random patch index position. The coding technique also allows for a skip patch data unit type (spdu) to be used. A skip patch data unit indicates that all parameters for the current patch may be inherited from the reference patch when the patch data unit elements for the current patch data unit and the reference patch data unit are identical or within an acceptable tolerance. The coding technique also allows for the syntax element indicating the number of matched patches to be removed from the signaling, since the signaling of the patch types described herein is sufficient to represent the entire list of patches in a patch frame data unit. Furthermore, the coding technique eliminates the need to signal the total number of patches in a patch frame data unit (e.g., the size of the patch frame data unit) and replaces that process with a special end patch type. Furthermore, the coding technique allows a pulse-coded modulation (PCM) patch, which is usually located at the end of a patch frame data unit, to be replaced with a PCM patch type indication. Multiple PCM patches are also allowed. Thus, the coder / decoder (a.k.a. "codec") in point cloud coding is improved over current codecs (e.g., the process of encoding and decoding patches in V-PCC). In practical terms, the improved point cloud coding process can improve coding efficiency, thereby providing users with a better user experience when point clouds are transmitted, received, and / or viewed.

[0044] In a first implementation form of the decoding device according to the sixth, seventh, and eighth aspects themselves, the decoding device further comprises a display configured to display an image generated based on the restored stereoscopic representation.

[0045] A ninth aspect relates to an encoding device comprising: a receiver configured to receive a three-dimensional (3D) image; a memory coupled to the receiver, the memory storing instructions; and a processor coupled to the memory, the processor configured to execute instructions that cause the encoding device to obtain a patch frame data unit (pfdu) identifying a patch type (pdfu_patch_mode) for each of a plurality of patches; determine whether a patch type for a patch from the plurality of patches is the last patch type; when the patch type is not the last patch type, encode patch information data for the patch, the patch information data including the patch type for the patch; and when the patch type is set to the last patch type, encode patch information data for the patch, the patch information data including the last patch type for the patch.

[0046] Using this coding method allows for flexible ordering of patches within a patch frame data unit. That is, patches with different patch modes (e.g., inter, intra, PCM, etc.) may be included in a patch frame data unit in any order. In addition, additional patches may be added to a patch frame data unit. Added patches may be added either to the end of the patch frame data unit or at any random patch index position. The coding technique also allows for a skip patch data unit type (spdu) to be used. A skip patch data unit indicates that all parameters for the current patch may be inherited from the reference patch when the patch data unit elements for the current patch data unit and the reference patch data unit are identical or within an acceptable tolerance. The coding technique also allows for the syntax element indicating the number of matched patches to be removed from the signaling, since the signaling of the patch types described herein is sufficient to represent the entire list of patches in a patch frame data unit. Furthermore, the coding technique eliminates the need to signal the total number of patches in a patch frame data unit (e.g., the size of the patch frame data unit) and replaces that process with a special end patch type. Furthermore, the coding technique allows a pulse-coded modulation (PCM) patch, which is usually located at the end of a patch frame data unit, to be replaced with a PCM patch type indication. Multiple PCM patches are also allowed. Thus, the coder / decoder (a.k.a. "codec") in point cloud coding is improved over current codecs (e.g., the process of encoding and decoding patches in V-PCC). In practical terms, the improved point cloud coding process can improve coding efficiency, thereby providing users with a better user experience when point clouds are transmitted, received, and / or viewed.

[0047] A tenth aspect relates to an encoding device comprising: a receiver configured to receive a three-dimensional (3D) image; a memory coupled to the receiver, the memory storing instructions; and a processor coupled to the memory, the processor configured to execute instructions that cause the encoding device to obtain a patch frame data unit (pfdu) for each of a plurality of patches; add a last patch flag to each of the plurality of patches; determine whether a patch type for a patch from the plurality of patches is the last patch type based on a value of the last patch flag; and when the patch type is not the last patch type, encode patch information data for the patch, wherein the patch information data includes the patch type for the patch and the last patch flag.

[0048] Using this coding method allows for flexible ordering of patches within a patch frame data unit. That is, patches with different patch modes (e.g., inter, intra, PCM, etc.) may be included in a patch frame data unit in any order. In addition, additional patches may be added to a patch frame data unit. Added patches may be added either to the end of the patch frame data unit or at any random patch index position. The coding technique also allows for a skip patch data unit type (spdu) to be used. A skip patch data unit indicates that all parameters for the current patch may be inherited from the reference patch when the patch data unit elements for the current patch data unit and the reference patch data unit are identical or within an acceptable tolerance. The coding technique also allows for the syntax element indicating the number of matched patches to be removed from the signaling, since the signaling of the patch types described herein is sufficient to represent the entire list of patches in a patch frame data unit. Furthermore, the coding technique eliminates the need to signal the total number of patches in a patch frame data unit (e.g., the size of the patch frame data unit) and replaces that process with a special end patch type. Furthermore, the coding technique allows a pulse-coded modulation (PCM) patch, which is usually located at the end of a patch frame data unit, to be replaced with a PCM patch type indication. Multiple PCM patches are also allowed. Thus, the coder / decoder (a.k.a. "codec") in point cloud coding is improved over current codecs (e.g., the process of encoding and decoding patches in V-PCC). In practical terms, the improved point cloud coding process can improve coding efficiency, thereby providing users with a better user experience when point clouds are transmitted, received, and / or viewed.

[0049] In a first implementation form of the encoding device according to the ninth and tenth aspects itself, the encoding device further comprises a transmitter coupled to the processor, the transmitter configured to transmit a bitstream having the encoded patch information data towards the decoder.

[0050] An eleventh aspect relates to a coding apparatus comprising: a receiver configured to receive stereoscopic pictures for encoding or to receive a bitstream for decoding; a transmitter coupled to the receiver, the transmitter configured to transmit the bitstream to a decoder or to transmit decoded stereoscopic images to a reconstruction device configured to reconstruct the decoded stereoscopic pictures; a memory coupled to at least one of the receiver or the transmitter, the memory configured to store instructions; and a processor coupled to the memory, the processor configured to execute the instructions stored in the memory to perform any of the methods disclosed herein.

[0051] Using this coding method allows for flexible ordering of patches within a patch frame data unit. That is, patches with different patch modes (e.g., inter, intra, PCM, etc.) may be included in a patch frame data unit in any order. In addition, additional patches may be added to a patch frame data unit. Added patches may be added either to the end of the patch frame data unit or at any random patch index position. The coding technique also allows for a skip patch data unit type (spdu) to be used. A skip patch data unit indicates that all parameters for the current patch may be inherited from the reference patch when the patch data unit elements for the current patch data unit and the reference patch data unit are identical or within an acceptable tolerance. The coding technique also allows for the syntax element indicating the number of matched patches to be removed from the signaling, since the signaling of the patch types described herein is sufficient to represent the entire list of patches in a patch frame data unit. Furthermore, the coding technique eliminates the need to signal the total number of patches in a patch frame data unit (e.g., the size of the patch frame data unit) and replaces that process with a special end patch type. Furthermore, the coding technique allows a pulse-coded modulation (PCM) patch, which is usually located at the end of a patch frame data unit, to be replaced with a PCM patch type indication. Multiple PCM patches are also allowed. Thus, the coder / decoder (a.k.a. "codec") in point cloud coding is improved over current codecs (e.g., the process of encoding and decoding patches in V-PCC). In practical terms, the improved point cloud coding process can improve coding efficiency, thereby providing users with a better user experience when point clouds are transmitted, received, and / or viewed.

[0052] In a first implementation form of the coding device according to the fifth aspect itself, the coding device further comprises a display configured to display an image based on the reconstructed patch.

[0053] A twelfth aspect relates to a system comprising an encoder and a decoder in communication with the encoder, the encoder or decoder comprising an encoding device, a decoding device, or a coding apparatus as described herein.

[0054] Using this coding method allows for flexible ordering of patches within a patch frame data unit. That is, patches with different patch modes (e.g., inter, intra, PCM, etc.) may be included in a patch frame data unit in any order. In addition, additional patches may be added to a patch frame data unit. Added patches may be added either to the end of the patch frame data unit or at any random patch index position. The coding technique also allows for a skip patch data unit type (spdu) to be used. A skip patch data unit indicates that all parameters for the current patch may be inherited from the reference patch when the patch data unit elements for the current patch data unit and the reference patch data unit are identical or within an acceptable tolerance. The coding technique also allows for the syntax element indicating the number of matched patches to be removed from the signaling, since the signaling of the patch types described herein is sufficient to represent the entire list of patches in a patch frame data unit. Furthermore, the coding technique eliminates the need to signal the total number of patches in a patch frame data unit (e.g., the size of the patch frame data unit) and replaces that process with a special end patch type. Furthermore, the coding technique allows a pulse-coded modulation (PCM) patch, which is usually located at the end of a patch frame data unit, to be replaced with a PCM patch type indication. Multiple PCM patches are also allowed. Thus, the coder / decoder (a.k.a. "codec") in point cloud coding is improved over current codecs (e.g., the process of encoding and decoding patches in V-PCC). In practical terms, the improved point cloud coding process can improve coding efficiency, thereby providing users with a better user experience when point clouds are transmitted, received, and / or viewed.

[0055] A thirteenth aspect relates to a means for coding comprising a receiving means configured to receive stereoscopic pictures for encoding or to receive a bitstream for decoding, reconstruction and projecting, a transmitting means coupled to the receiving means, the transmitting means configured to transmit the bitstream to a decoder or to transmit a decoded image to a display means, a storage means coupled to at least one of the receiving means or the transmitting means, the storage means configured to store instructions, and a processing means coupled to the storage means, the processing means configured to execute the instructions stored in the storage means to perform any of the methods disclosed herein.

[0056] Using this coding method allows for flexible ordering of patches within a patch frame data unit. That is, patches with different patch modes (e.g., inter, intra, PCM, etc.) may be included in a patch frame data unit in any order. In addition, additional patches may be added to a patch frame data unit. Added patches may be added either to the end of the patch frame data unit or at any random patch index position. The coding technique also allows for a skip patch data unit type (spdu) to be used. A skip patch data unit indicates that all parameters for the current patch may be inherited from the reference patch when the patch data unit elements for the current patch data unit and the reference patch data unit are identical or within an acceptable tolerance. The coding technique also allows for the syntax element indicating the number of matched patches to be removed from the signaling, since the signaling of the patch types described herein is sufficient to represent the entire list of patches in a patch frame data unit. Furthermore, the coding technique eliminates the need to signal the total number of patches in a patch frame data unit (e.g., the size of the patch frame data unit) and replaces that process with a special end patch type. Furthermore, the coding technique allows a pulse-coded modulation (PCM) patch, which is usually located at the end of a patch frame data unit, to be replaced with a PCM patch type indication. Multiple PCM patches are also allowed. Thus, the coder / decoder (a.k.a. "codec") in point cloud coding is improved over current codecs (e.g., the process of encoding and decoding patches in V-PCC). In practical terms, the improved point cloud coding process can improve coding efficiency, thereby providing users with a better user experience when point clouds are transmitted, received, and / or viewed.

[0057] For clarity, any one of the above-described embodiments may be combined with any one or more of the other above-described embodiments to create new embodiments within the scope of the present disclosure.

[0058] These and other features will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings and claims.

[0059] For a more complete understanding of the present disclosure, reference is now made to the following brief description in conjunction with the accompanying drawings and detailed description, in which like reference numerals represent like parts. [Brief description of the drawings]

[0060] [Figure 1] FIG. 1 is a block diagram illustrating an example coding system that can utilize context modeling techniques. [Diagram 2] FIG. 1 is a block diagram illustrating an example encoder in which context modeling techniques can be implemented. [Diagram 3] FIG. 2 is a block diagram illustrating an example decoder in which context modeling techniques can be implemented. [Figure 4] FIG. 1 is a diagram of a sequence of point cloud frames, each of which contains a three-dimensional (3D) cloud of points. [Diagram 5] FIG. 6 is an illustration of one of the 3D point clouds of FIG. 5 projected onto a bounding box to generate a two-dimensional (2D) projection. [Figure 6A] FIG. 6 is an occupancy map corresponding to a 2D projection from the bounding box of FIG. 5. [Figure 6B] FIG. 6 is a diagram of a geometric map corresponding to a 2D projection from the bounding box of FIG. 5. [Figure 6C] FIG. 6 is an illustration of an attribute map corresponding to a 2D projection from the bounding box of FIG. 5. [Figure 7] FIG. 1 is a diagram of a video-based point cloud coding (V-PCC) bitstream. [Figure 8] FIG. 1 is a diagram of a coding decomposition of a point cloud sequence for multiple frames. [Figure 9] FIG. 9 is a diagram of a single patch data unit extracted from one of the point cloud frames in the point cloud sequence of FIG. [Figure 10A] FIG. 10 is a diagram of an occupancy map corresponding to the single patch data unit of FIG. 9. [Figure 10B] FIG. 10 is a diagram of a geometric map corresponding to the single patch data unit of FIG. 9. [Figure 10C] FIG. 10 is a diagram of an attribute map corresponding to the single patch data unit of FIG. 9. [Figure 11] FIG. 13 is a diagram of 2D components for a patch data unit within a 2D bounding box. [Figure 12A] FIG. 12 is a schematic diagram of 3D components for the patch data unit of FIG. [Figure 12B] FIG. 12 is a data set diagram of 3D components for the patch data unit of FIG. 11. [Figure 13] FIG. 1 illustrates an embodiment of a patch frame data unit encoding process. [Figure 14] FIG. 13 illustrates an embodiment of a patch frame data unit encoding process using a last patch flag. [Figure 15] FIG. 13 is an embodiment of a patch information data frame with a description of the patch unit type. [Figure 16] FIG. 13 illustrates an embodiment of a patch frame data unit decoding process. [Figure 17] FIG. 13 illustrates an embodiment of a patch frame data unit decoding process using the last patch flag. [Figure 18] FIG. 1 illustrates an embodiment of a method of point cloud coding (PCC) performed by a decoder. [Figure 19] FIG. 1 illustrates an embodiment of a method of PCC implemented by a decoder. [Figure 20] FIG. 1 illustrates an embodiment of a method of PCC implemented by a decoder. [Figure 21] FIG. 1 illustrates one embodiment of a method of PCC implemented by an encoder. [Figure 22] FIG. 1 illustrates one embodiment of a method of PCC implemented by an encoder. [Figure 23] FIG. 2 is a schematic diagram of a coding device. [Figure 24] FIG. 2 is a schematic diagram of an embodiment of a means for coding. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0061] Although exemplary implementations of one or more embodiments are provided below, it should be understood at the outset that the systems and / or methods of the present disclosure may be implemented using any number of techniques, whether currently known or in existence. The present disclosure should not be limited in any respect to the exemplary implementations, drawings, and techniques illustrated below, including the exemplary designs and implementations illustrated and described herein, but may be modified within the full scope of the appended claims and their equivalents.

[0062] Throughout the specification and claims, the following terms have the meanings explicitly associated therewith, unless the context clearly dictates otherwise: A reference patch index (e.g., spdu_patch_index) indicates the difference between the index p of the current patch data unit [in the current patch frame data unit] and the index of the reference patch data unit [in the reference patch frame data unit]. A reference index (e.g., refIdx) indicates the actual index of the reference patch data unit in a previously decoded patch frame data unit [in the decoded patch frame data unit buffer]. A reference frame index (e.g., refFrmIdx) indicates the index of a previously decoded patch frame data unit [in the decoded patch frame data unit buffer].

[0063] Video coding standards include International Telecommunication Union Telecommunication Standardization Sector (ITU-T) H.261, International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC) Moving Picture Experts Group (MPEG)-1 Part 2, Advanced Video Coding (AVC), also known as ITU-T H.262 or ISO / IEC MPEG-2 Part 2, ITU-T H.263, ISO / IEC MPEG-4 Part 2, ITU-T H.264 or ISO / IEC MPEG-4 Part 10, and High Efficiency Video Coding (HEVC), also known as ITU-T H.265 or MPEG-H Part 2. AVC includes extensions such as Scalable Video Coding (SVC), Multiview Video Coding (MVC), Multiview Video Coding plus Depth (MVC+D), and 3D AVC (3D-AVC). HEVC includes extensions such as Scalable HEVC (SHVC), Multiview HEVC (MV-HEVC), and 3D HEVC (3D-HEVC).

[0064] A point cloud is a collection of data points in 3D space. Each data point includes parameters that determine the position (e.g., X, Y, Z), color (e.g., R, G, B or Y, U, V), and possibly other characteristics such as transparency, reflectance, acquisition time, etc. Typically, each point in the cloud has the same number of attributes attached to it. Point clouds may be used in a variety of applications such as real-time 3D immersive telepresence, content virtual reality (VR) viewing with two-way parallax, 3D free viewpoint sports replay broadcasting, geographic information systems, cultural heritage, autonomous navigation based on large-scale 3D dynamic maps, and automotive applications.

[0065] The ISO / IEC Moving Picture Experts Group (MPEG) started development of a new codec standard for point cloud coding for lossless and lossy compressed point cloud data with substantial coding efficiency and robustness to networked environments in 2016. The codec standard will enable point clouds to be manipulated as a form of computer data, stored on various storage media, transmitted and received over existing and future networks, and distributed over existing and future broadcast channels.

[0066] Recently, point cloud coding (PCC) work has been divided into three categories: PCC Category 1, PCC Category 2, and PCC Category 3, and two separate working drafts have been developed, one for PCC Category 2 (PCC Cat2) and one for PCC Categories 1 and 3 (PCC Cat13). MPEG Output Document N17534 contains the latest working draft (WD) for PCC Cat2, and MPEG Output Document N17533 contains the latest WD for PCC Cat13.

[0067] The main idea behind the design of the PCC Cat2 codec in PCC Cat2 WD is to leverage existing video codecs to compress the geometric and texture information of dynamic point clouds by compressing the point cloud data as a set of different video sequences. In detail, two video sequences, one representing the geometric information of the point cloud data and the other representing the texture information, are generated and compressed using a video codec. Additional metadata for interpreting the two video sequences, i.e. occupancy maps and auxiliary patch information, are also generated and compressed separately.

[0068] Unfortunately, the existing design of PCC has shortcomings. For example, data units related to one time instance, i.e., one access unit (AU), are not consecutive in decoding order. In PCC Cat2 WD, the texture, geometry, auxiliary information, occupancy map data units per AU are interleaved per group of frames, i.e., the geometry data for all frames in a group are together. The same is often true for texture data, etc. In PCC Cat13 WD, the geometry data units and general attributes per AU are interleaved at the level of the entire PCC bitstream (e.g., the same as PCC Cat2 WD, when there is only one group of frames with the same length as the entire PCC bitstream). Interleaving data units belonging to one AU essentially causes a huge end-to-end delay at least equal to the length of the group of frames in the presentation duration in the application system.

[0069] Another drawback concerns the bitstream format, which allows emulation of start code patterns such as 0x0003 and therefore does not work for transmission over MPEG-2 Transport Streams (TS), where start code emulation prevention is required. For PCC Cat2, when either HEVC or AVC is used for coding the geometry and texture components, currently only group_of_frames_geometry_video_payload() and group_of_frames_texture_video_payload() have start code emulation prevention in place. For PCC Cat13, start code emulation prevention is not in place anywhere in the bitstream.

[0070] In PCC Cat2 WD, some of the codec information for the geometry and texture bitstreams (e.g., which codec, profile, level, etc.) is deeply embedded in multiple instances of the structures group_of_frames_geometry_video_payload() and group_of_frames_texture_video_payload(). Additionally, some of the information such as profile and level that indicates the capabilities for decoding auxiliary information and occupancy map components, as well as point cloud reconstruction, is missing.

[0071] A high-level syntax design is provided that solves one or more of the aforementioned problems associated with point cloud coding. As described in more detail below, the present disclosure utilizes a type indicator in a data unit header (aka PCC Network Access Layer (NAL) header) to specify the type of content in the payload of a PCC NAL unit. In addition, the present disclosure utilizes a group of frame header NAL units to carry a group of frame header parameters. The group of frame header NAL units may also be used to signal the profile and level of each geometry or texture bitstream.

[0072] FIG. 1 is a block diagram illustrating an example coding system 10 that can utilize PCC video coding techniques. As shown in FIG. 1, coding system 10 includes a source device 12 that provides encoded video data that is subsequently decoded by a destination device 14. In particular, source device 12 can provide the video data to destination device 14 via a computer-readable medium 16. Source device 12 and destination device 14 may comprise any of a wide range of devices, including desktop computers, notebook (e.g., laptop) computers, tablet computers, set-top boxes, telephone handsets such as so-called "smart" phones, so-called "smart" pads, televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, and the like. In some cases, source device 12 and destination device 14 may be enabled for wireless communication.

[0073] The destination device 14 can receive the encoded video data to be decoded via a computer-readable medium 16. The computer-readable medium 16 may comprise any type of medium or device capable of moving encoded video data from the source device 12 to the destination device 14. In one example, the computer-readable medium 16 may comprise a communication medium that allows the source device 12 to transmit the encoded video data directly to the destination device 14 in real time. The encoded video data may be modulated according to a communication standard, such as a wireless communication protocol, and transmitted to the destination device 14. The communication medium may comprise any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. The communication medium may include routers, switches, base stations, or any other equipment that may be useful in facilitating communication from the source device 12 to the destination device 14.

[0074] In some examples, the encoded data may be output from output interface 24 to a storage device. Similarly, the encoded data may be accessed from a storage device by an input interface. The storage device may include any of a variety of distributed or locally accessed data storage media, such as a hard drive, a Blu-ray® disc, a digital video disc (DVD), a compact disc read only memory (CD-ROM), flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data. In a further example, the storage device may correspond to a file server or another intermediate storage device capable of storing the encoded video generated by source device 12. Destination device 14 may access the stored video data from the storage device via streaming or download. The file server may be any type of server capable of storing the encoded video data and transmitting the encoded video data to destination device 14. Exemplary file servers include a web server (e.g., for a website), a file transfer protocol (FTP) server, a network attached storage (NAS) device, or a local disk drive. Destination device 14 may access the encoded video data via any standard data connection, including an Internet connection. This may include wireless channels (e.g., a Wi-Fi connection), wired connections (e.g., digital subscriber line (DSL), cable modem, etc.), or a combination of both suitable for accessing the encoded video data stored on a file server. The transmission of the encoded video data from the storage device may be a streaming transmission, a download transmission, or a combination thereof.

[0075] The techniques of this disclosure are not necessarily limited to wireless applications or settings. The techniques may be applied to video coding to support any of a variety of multimedia applications, such as over-the-air television broadcasting, cable television transmission, satellite television transmission, Internet streaming video transmission such as Dynamic Adaptive Streaming over HTTP (DASH), digital video encoded on a data storage medium, decoding of digital video stored on a data storage medium, or other applications. In some examples, coding system 10 may be configured to support unidirectional or bidirectional video transmission to support applications such as video streaming, video playback, video broadcasting, and / or video telephony.

[0076] In the example of FIG. 1, source device 12 includes a video source 18 configured to provide stereoscopic images, a projection device 20, a video encoder 22, and an output interface 24. Destination device 14 includes an input interface 26, a video decoder 28, a restoration device 30, and a display device 32. According to this disclosure, video encoder 22 of source device 12 and / or video decoder 28 of destination device 14 may be configured to apply techniques for video coding. In other examples, the source device and destination device may include other components or configurations. For example, source device 12 may receive video data from an external video source, such as an external camera. Similarly, destination device 14 may interface with an external display device rather than including an integrated display device.

[0077] The illustrated coding system 10 of FIG. 1 is only one example. The techniques for video coding may be performed by any digital video encoding and / or decoding device. Although the techniques of this disclosure are generally performed by a coding device, the techniques may also be performed by a video encoder / decoder, usually referred to as a "codec." Moreover, the techniques of this disclosure may also be performed by a video preprocessor. The encoder and / or decoder may be a graphics processing unit (GPU) or similar device.

[0078] Source device 12 and destination device 14 are merely examples of coding devices, such that source device 12 generates coded video data for transmission to destination device 14. In some examples, source device 12 and destination device 14 may operate substantially symmetrically, such that source device 12 and destination device 14 each comprise video encoding and decoding components. Thus, coding system 10 may support unidirectional or bidirectional video transmission between video devices 12, 14, e.g., for video streaming, video playback, video broadcasting, or video telephony.

[0079] Video source 18 of source device 12 may include a video capture device such as a video camera, a video archive containing previously captured video, and / or a video feed interface for receiving stereoscopic images or video from a video content provider. As a further alternative, video source 18 can generate stereoscopic images and computer graphics-based data as source video or a combination of live video, archive video, and computer-generated video.

[0080] In some cases, when video source 18 is a video camera, source device 12 and destination device 14 may form a so-called camera phone or video phone. However, as noted above, the techniques described in this disclosure may be applicable to video coding in general, and may be applied to wireless and / or wired applications.

[0081] Projection device 20 is configured to project a stereoscopic image onto a plane (e.g., a bounding box), as described in more detail below, i.e., projection device 20 is configured to convert a three-dimensional (3D) image into one or more two-dimensional (2D) images.

[0082] In either case, the stereoscopic images, captured video, pre-captured video, or computer-generated video may be encoded by encoder 22. The encoded video information may then be output by output interface 24 onto computer-readable medium 16.

[0083] Computer-readable medium 16 may include a transitory medium, such as a wireless broadcast or wired network transmission, or a storage medium (i.e., a non-transitory storage medium), such as a hard disk, a flash drive, a compact disc, a digital video disc, a Blu-ray® disc, or other computer-readable medium. In some examples, a network server (not shown) may receive the encoded video data from source device 12 and provide the encoded video data to destination device 14, e.g., via a network transmission. Similarly, a computing device at a media production facility, such as a disc stamping facility, may receive the encoded video data from source device 12 and produce a disc including the encoded video data. Thus, computer-readable medium 16 may be understood to include one or more computer-readable media of various forms in various examples.

[0084] An input interface 26 of destination device 14 receives information from computer-readable medium 16. The information on computer-readable medium 16 may include syntax information defined by encoder 22, which is also used by decoder 28, including syntax elements that describe characteristics and / or processing of blocks and other coded units, such as groups of pictures (GOPs).

[0085] The reconstruction device 30 is configured to convert one or more planar images back into a stereoscopic image, as described in more detail below, i.e., the reconstruction device 30 is configured to convert one or more 2D images back into a 3D image.

[0086] Display device 32 displays the stereoscopic images or the decoded video data to a user and may comprise any of a variety of display devices, such as a cathode ray tube (CRT), a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or another type of display device.

[0087] Encoder 22 and decoder 28 may operate according to a video coding standard, such as the High Efficiency Video Coding (HEVC) standard currently under development, and may conform to the HEVC Test Model (HM). Alternatively, encoder 22 and decoder 28 may operate according to other proprietary or industry standards, such as the International Telecommunications Union Standardization Sector (ITU-T) H.264 standard, alternatively referred to as Moving Picture Experts Group (MPEG)-4, Part 10, Advanced Video Coding (AVC), H.265 / HEVC, or an extension of such a standard. However, the techniques of this disclosure are not limited to any particular coding standard. Other examples of video coding standards include MPEG-2 and ITU-T H.263. Although not shown in FIG. 1, in some aspects, encoder 22 and decoder 28 may be integrated with an audio encoder and audio decoder, respectively, and may include appropriate multiplexer-demultiplexer (MUX-DEMUX) units, or other hardware and software, to handle the encoding of both audio and video in a common data stream or separate data streams. If applicable, the MUX-DEMUX unit may conform to the ITU H.223 multiplexer protocol, or other protocols such as the User Datagram Protocol (UDP).

[0088] The encoder 22 and the decoder 28 may each be implemented as any of a variety of suitable encoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. When the techniques are implemented partially in software, the device may store instructions for the software on a suitable non-transitory computer-readable medium and execute the instructions in hardware using one or more processors to perform the techniques of this disclosure. Each of the encoder 22 and the decoder 28 may be included in one or more encoders or decoders, any of which may be integrated as part of a combined encoder / decoder (codec) in the respective device. The device including the encoder 22 and / or the decoder 28 may comprise an integrated circuit, a microprocessor, and / or a wireless communication device such as a mobile phone.

[0089] 2 is a block diagram illustrating an example of an encoder 22 that can implement video coding techniques. Encoder 22 can perform intra- and inter-coding of video blocks in a video slice. Intra-coding relies on spatial prediction to reduce or remove spatial redundancy in video within a given video frame or video picture. Inter-coding relies on temporal prediction to reduce or remove temporal redundancy in video within adjacent frames or pictures of a video sequence. Intra-mode (I-mode) can refer to any of several spatial-based coding modes. Inter-modes, such as unidirectional (aka uni-predictive) prediction (P-mode) or bi-predictive (aka bi-predictive) (B-mode), can refer to any of several temporal-based coding modes.

[0090] As shown in FIG. 2, the encoder 22 receives a current video block in a video frame to be encoded. In the example of FIG. 2, the encoder 22 includes a mode selection unit 40, a reference frame memory 64, an adder 50, a transform processing unit 52, a quantization unit 54, and an entropy coding unit 56. In turn, the mode selection unit 40 includes a motion compensation unit 44, a motion estimation unit 42, an intra prediction (aka intra-prediction) unit 46, and a division unit 48. For video block reconstruction, the encoder 22 also includes an inverse quantization unit 58, an inverse transform unit 60, and an adder 62. A deblocking filter (not shown in FIG. 2) may also be included to filter block boundaries to remove blockiness artifacts from the reconstructed video. If necessary, the deblocking filter typically filters the output of the adder 62. In addition to the deblocking filter, further filters (in-loop or post-loop) may also be used. The output of summer 50 can be filtered (as an in-loop filter) if desired, although such a filter is not shown for simplicity.

[0091] During the encoding process, encoder 22 receives a video frame or video slice to be coded. The frame or slice may be divided into multiple video blocks. Motion estimation unit 42 and motion compensation unit 44 perform inter-predictive coding of the received video block relative to one or more blocks in one or more reference frames to provide temporal prediction. Intra prediction unit 46 may alternatively perform intra-predictive coding of the received video block relative to one or more neighboring blocks in the same frame or slice as the block to be coded to provide spatial prediction. Encoder 22 may perform multiple coding passes to, for example, select an appropriate coding mode for each block of video data.

[0092] Moreover, the partition unit 48 may partition blocks of video data into sub-blocks based on an evaluation of a previous partitioning scheme in a previous coding pass. For example, the partition unit 48 may first partition a frame or slice into largest coding units (LCUs) and partition each of the LCUs into sub-coding units (sub-CUs) based on a rate-distortion analysis (e.g., rate-distortion optimization). The mode selection unit 40 may further generate a quadtree data structure indicating the partitioning of the LCUs into sub-CUs. A leaf node CU of the quadtree may include one or more prediction units (PUs) and one or more transform units (TUs).

[0093] This disclosure uses the term "block" to refer to either a CU, PU, ​​or TU in the context of HEVC, or similar data structures in the context of other standards (e.g., a macroblock and its subblocks in H.264 / AVC). A CU includes a coding node, a PU, and a TU associated with the coding node. The size of a CU corresponds to the size of the coding node and is square in shape. The size of a CU may range from 8x8 pixels up to the size of a treeblock of 64x64 pixels or more. Each CU may include one or more PUs and one or more TUs. Syntax data associated with a CU may, for example, describe the partitioning of the CU into one or more PUs. The partitioning mode may differ depending on whether the CU is coded in skip or direct mode, intra-prediction mode, or inter-prediction (a.k.a. inter-prediction) mode. A PU may be partitioned into non-square shapes. Syntax data associated with a CU may also describe the partitioning of a CU into one or more TUs, for example according to a quadtree. A TU may be square or non-square (e.g., rectangular) in shape.

[0094] Mode select unit 40 selects one of intra or inter coding modes, e.g., based on the error result, and provides the resulting intra- or inter-coded blocks to summer 50 to generate residual block data and to summer 62 to reconstruct the coded blocks for use as reference frames. Mode select unit 40 also provides syntactic information, such as motion vectors, intra-mode indicators, partition information, and other such syntactic information to entropy coding unit 56.

[0095] Motion estimation unit 42 and motion compensation unit 44 may be highly integrated, but are shown separately for conceptual purposes. Motion estimation performed by motion estimation unit 42 is a process of generating motion vectors that estimate motion for video blocks. A motion vector may indicate, for example, the displacement of a PU of a video block in a current video frame or picture relative to a predictive block in a reference frame (or other coding unit) relative to a current block being coded in a current frame (or other coding unit). A predictive block is a block that is found to closely match a block being coded in terms of pixel difference, which may be determined by sum of absolute differences (SAD), sum of squared differences (SSD), or other difference metric. In some examples, encoder 22 may calculate values ​​for fractional pixel locations of a reference picture stored in reference frame memory 64. For example, encoder 22 may interpolate values ​​for quarter pixel locations, eighth pixel locations, or other fractional pixel locations of a reference picture. Thus, motion estimation unit 42 may perform motion searches for full-pixel and fractional-pixel locations and output motion vectors with fractional-pixel predictions.

[0096] Motion estimation unit 42 calculates a motion vector for a PU of a video block in an inter-coded slice by comparing the position of the PU with the position of a predictive block of a reference picture. The reference picture may be selected from a first reference picture list (List 0) or a second reference picture list (List 1), each of which identifies one or more reference pictures stored in reference frame memory 64. Motion estimation unit 42 sends the calculated motion vector to entropy coding unit 56 and motion compensation unit 44.

[0097] The motion compensation performed by motion compensation unit 44 may include retrieving or generating a predictive block based on the motion vector determined by motion estimation unit 42. Again, in some examples, motion estimation unit 42 and motion compensation unit 44 may be functionally integrated. Upon receiving the motion vector for the PU of the current video block, motion compensation unit 44 may locate the predictive block to which the motion vector points in one of the reference picture lists. Adder 50 forms a residual video block by subtracting pixel values ​​of the predictive block from pixel values ​​of the current video block being coded and forming pixel difference values, as described below. In general, motion estimation unit 42 performs motion estimation on the luma component, and motion compensation unit 44 uses the motion vector calculated based on the luma component for both the chroma and luma components. Mode selection unit 40 also generates syntax elements associated with the video blocks and video slices for use by decoder 28 in decoding video blocks of the video slices.

[0098] Intra prediction unit 46 may intra predict the current block, instead of the inter prediction performed by motion estimation unit 42 and motion compensation unit 44, as described above. In particular, intra prediction unit 46 may determine an intra prediction mode to use to encode the current block. In some examples, intra prediction unit 46 may encode the current block using different intra prediction modes, e.g., during separate encoding passes, and intra prediction unit 46 (or, in some examples, mode selection unit 40) may select an appropriate intra prediction mode from the tested modes to use.

[0099] For example, intra prediction unit 46 may use a rate-distortion analysis to calculate rate-distortion values ​​for various tested intra prediction modes and select the intra prediction mode with the best rate-distortion characteristics among the tested modes. The rate-distortion analysis generally identifies the amount of distortion (or error) between a coded block and the original uncoded block that was coded to generate the coded block, as well as the bit rate (i.e., number of bits) used to generate the coded block. Intra prediction unit 46 may calculate ratios from the distortion and rate for the various coded blocks to determine which intra prediction mode represents the best rate-distortion value for the block.

[0100] In addition, intra prediction unit 46 may be configured to code the depth blocks of the depth map using a depth modeling mode (DMM). Mode selection unit 40 may determine whether an available DMM produces better coding results than the intra prediction mode and other DMM modes, for example, using rate-distortion optimization (RDO). Data for texture images corresponding to the depth map may be stored in reference frame memory 64. Motion estimation unit 42 and motion compensation unit 44 may also be configured to inter predict the depth blocks of the depth map.

[0101] After selecting an intra-prediction mode for the block (e.g., one of a conventional intra-prediction mode or a DMM mode), intra-prediction unit 46 may provide information indicating the selected intra-prediction mode for the block to entropy coding unit 56. Entropy coding unit 56 may encode the information indicating the selected intra-prediction mode. Encoder 22 may include transmitted bitstream configuration data, which may include multiple intra-prediction mode index tables and multiple modified intra-prediction mode index tables (also referred to as codeword mapping tables), definitions for encoding contexts for various blocks, and an indication of the most likely intra-prediction mode, intra-prediction mode index table, and modified intra-prediction mode index table to use for each of the contexts.

[0102] Encoder 22 forms a residual video block by subtracting the prediction data from mode select unit 40 from the original video block being coded. Summer 50 represents one or more components that perform this subtraction operation.

[0103] Transform processing unit 52 applies a transform, such as a discrete cosine transform (DCT) or a conceptually similar transform, to the residual block to generate a video block comprising residual transform coefficient values. Transform processing unit 52 may perform other transforms that are conceptually similar to the DCT. Wavelet transforms, integer transforms, subband transforms, or other types of transforms may also be used.

[0104] Transform processing unit 52 applies a transform to the residual block, producing a block of residual transform coefficients. The transform may convert the residual information from the pixel value domain to a transform domain, such as the frequency domain. Transform processing unit 52 may send the resulting transform coefficients to quantization unit 54. Quantization unit 54 quantizes the transform coefficients to further reduce the bit rate. The quantization process may reduce the bit depth associated with some or all of the coefficients. The degree of quantization may be modified by adjusting a quantization parameter. In some examples, quantization unit 54 may then perform a scan of a matrix including the quantized transform coefficients. Alternatively, entropy coding unit 56 may perform the scan.

[0105] Following quantization, entropy coding unit 56 entropy codes the quantized transform coefficients. For example, entropy coding unit 56 may perform context-adaptive variable length coding (CAVLC) or context-adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or another entropy coding technique. For context-based entropy coding, the context may be based on neighboring blocks. Following entropy coding by entropy coding unit 56, the encoded bitstream may be transmitted to another device (e.g., decoder 28) or archived for later transmission or retrieval.

[0106] Inverse quantization unit 58 and inverse transform unit 60 apply inverse quantization and inverse transformation, respectively, to reconstruct the residual block in the pixel domain, e.g., for later use as a reference block. Motion compensation unit 44 may calculate a reference block by adding the residual block to a predictive block of one of the frames in reference frame memory 64. Motion compensation unit 44 may also apply one or more interpolation filters to the reconstructed residual block to calculate fractional pixel values ​​for use in motion estimation. Adder 62 adds the reconstructed residual block to the motion compensated predictive block generated by motion compensation unit 44 to generate a reconstructed video block for storage in reference frame memory 64. The reconstructed video block may be used as a reference block by motion estimation unit 42 and motion compensation unit 44 to inter-code blocks in subsequent video frames.

[0107] FIG. 3 is a block diagram illustrating an example of a decoder 28 capable of implementing video coding techniques. In the example of FIG. 3, the decoder 28 includes an entropy decoding unit 70, a motion compensation unit 72, an intra prediction unit 74, an inverse quantization unit 76, an inverse transform unit 78, a reference frame memory 82, and an adder 80. The decoder 28 may, in some examples, perform a decoding path that is generally inverse to the encoding path described with respect to the encoder 22 (FIG. 2). The motion compensation unit 72 may generate prediction data based on the motion vectors received from the entropy decoding unit 70, and the intra prediction unit 74 may generate prediction data based on the intra prediction mode indicator received from the entropy decoding unit 70.

[0108] During the decoding process, decoder 28 receives from encoder 22 an encoded video bitstream representing video blocks of encoded video slices, and associated syntax elements. An entropy decoding unit 70 of decoder 28 entropy decodes the bitstream to generate quantized coefficients, motion vectors or intra-prediction mode indicators, and other syntax elements. Entropy decoding unit 70 forwards the motion vectors and other syntax elements to motion compensation unit 72. Decoder 28 may receive syntax elements at the video slice level and / or the video block level.

[0109] When a video slice is coded as an intra-coded (I) slice, intra prediction unit 74 may generate predictive data for video blocks of the current video slice based on intra prediction modes and data signaled from previously decoded blocks of the current frame or picture. When a video frame is coded as an inter-coded (e.g., B, P, or GPB) slice, motion compensation unit 72 generates predictive blocks for video blocks of the current video slice based on motion vectors and other syntax elements received from entropy decoding unit 70. The predictive blocks may be generated from one of the reference pictures in one of the reference picture lists. Decoder 28 may construct the reference frame lists, List 0 and List 1, using a default construction technique based on reference pictures stored in reference frame memory 82.

[0110] Motion compensation unit 72 parses the motion vectors and other syntax elements to determine prediction information for video blocks of the current video slice and uses the prediction information to generate predictive blocks for the current video block being decoded. For example, motion compensation unit 72 uses some of the received syntax elements to identify a prediction mode (e.g., intra prediction or inter prediction) used to code video blocks of the video slice, an inter prediction slice type (e.g., a B slice, a P slice, or a GPB slice), construction information for one or more of the reference picture lists for the slice, a motion vector for each inter coded video block of the slice, an inter prediction state for each inter coded video block of the slice, and other information for decoding video blocks in the current video slice.

[0111] Motion compensation unit 72 may also perform interpolation based on an interpolation filter. Motion compensation unit 72 may use an interpolation filter used by encoder 22 during encoding of the video block to calculate interpolated values ​​for fractional pixels of the reference block. In this case, motion compensation unit 72 may determine the interpolation filter used by encoder 22 from the received syntax element and generate the predictive block using the interpolation filter.

[0112] Data for texture images that correspond to the depth maps may be stored in reference frame memory 82. Motion compensation unit 72 may also be configured to inter-predict depth blocks of the depth maps.

[0113] Conventional coding techniques require that patches are ordered according to their patch mode in a patch frame data unit (aka patch buffer). That is, the representation of patch data unit types in auxiliary information patch data units in video-based point cloud coding (V-PCC) requires that all patch-related information is placed in a particular order in the patch frame data unit. First, matched patch data units are added to a patch data unit list, followed by unmatched patch data units. Matched patch data units may be coded using differential coding (inter prediction - DPDU), and unmatched patch data units are coded using absolute values ​​(intra prediction). That is, all inter-coded patches are enumerated in the patch frame data unit first, followed by all intra-coded patches, and so on. Thus, to add a new inter-coded patch or a new intra-coded patch to a patch frame data unit, the entire patch frame data unit must be reconstructed using the required order, resulting in coding inefficiencies.

[0114] Coding techniques are disclosed herein that allow for flexible ordering of patches within a patch frame data unit. That is, patches with different patch modes (e.g., inter, intra, PCM, etc.) may be included in a patch frame data unit in any order. The coding techniques also allow for a skip patch data unit type (spdu) to be used. A skip patch data unit indicates that all parameters for a current patch may be inherited from a reference patch when patch data unit elements for a current patch data unit and a reference patch data unit are identical or match exactly. The coding techniques also allow for syntax elements indicating the number of matched patches to be removed from the signaling, since the patch type signaling described herein is sufficient to represent the entire list of patches within a patch frame data unit. In addition, the coding techniques eliminate the need to signal the total number of patches within a patch frame data unit (e.g., the size of the patch frame data unit) and replace that process with a special end patch type. Furthermore, the coding techniques allow for a pulse-coded modulation (PCM) patch that is typically located at the end of a patch frame data unit to be replaced with a PCM patch type indication. Thus, the coder / decoder (a.k.a. "codec") in video coding is improved relative to the current codec (e.g., the process of encoding and decoding patches in V-PCC). In practical terms, the improved video coding process can improve coding efficiency, thereby providing a better user experience when point clouds are transmitted, received, and / or viewed.

[0115] FIG. 4 is a diagram of a sequence 400 of point cloud frames 402, 404, 406 (e.g., frame PCC_0, frame PCC_1, frame PCC_4), each of which includes a point cloud 408. The point cloud 408 is a volumetric representation of space on a regular 3D grid. That is, the point cloud 408 is three-dimensional (3D). As shown in FIG. 4, the point cloud 408 includes point cloud content 410 in a 3D space 412. The point cloud content 410 is represented by a set of points (e.g., voxels) in the 3D space 412. A voxel is a volume element that represents some numerical value, such as a color, of a point in a three-dimensional space, used to visualize and analyze three-dimensional data. Thus, a voxel can be considered the three-dimensional equivalent of a pixel in a 2D image.

[0116] Each voxel in the point cloud 408 of Figure 4 has coordinates (e.g., xyz coordinates) and one or more attributes (e.g., red / green / blue (RGB) color components, reflectance, etc.). Although the point cloud content 410 of Figure 4 depicts a person, the point cloud content 410 may be any other solid object or image in practical applications.

[0117] Figure 5 is a diagram of the point cloud 400 of Figure 4 projected onto a bounding box 500. As shown in Figure 5, the bounding box 500 includes patches 502 projected onto its two-dimensional (2D) surface or plane 504. The patches 502 are thus a 2D representation of a portion of a 3D image. The patches 502 collectively correspond to the point cloud content 410 of Figure 4. Data representation in video-based point cloud coding (V-PCC), sometimes referred to as point cloud compression, relies on this 3D to 2D transformation.

[0118] Data representation in the V-PCC is described as a set of planar 2D images (eg, patches 502) using, for example, occupancy map 610 of FIG. 6A, geometry map 612 of FIG. 6B, and attribute map 614 of FIG. 6C.

[0119] 6A is a diagram of an occupancy map 610 corresponding to a 2D projection (e.g., patch 502) from the bounding box 500 of FIG. 5. The occupancy map 610 is coded in a binary format. For example, a 0 represents that a portion of the bounding box 600 is not occupied by one of the patches 602. Those portions of the bounding box 600 represented by a 0 do not participate in the reconstruction of the volumetric representation (e.g., point cloud content 410). In contrast, a 1 represents that a portion of the bounding box 600 is occupied by one of the patches 602. Those portions of the bounding box 600 represented by a 1 do participate in the reconstruction of the volumetric representation (e.g., point cloud content 410).

[0120] Figure 6B is an illustration of a geometry map 612 that corresponds to a 2D projection (e.g., patch 502) from bounding box 500 of Figure 5. Geometry map 612 provides or depicts the contour or topography of each of patches 602. That is, geometry map 612 indicates the distance of each point in patch 602 away from the flat surface of bounding box 600 (e.g., plane 504).

[0121] 6C is an illustration of an attribute map 614 that corresponds to a 2D projection (e.g., patch 502) from bounding box 500 of FIG. 5. Attribute map 614 provides or depicts attributes of each point in patch 602 within bounding box 600. The attributes in attribute map 614 may be, for example, color components of the points. The color components may be based on an RGB color model, a YUV color model, or another known color model.

[0122] 7 is a diagram of a V-PCC bitstream 700. The V-PCC bitstream 700 may be utilized to carry encoded information necessary to reconstruct a point cloud (e.g., point cloud 408 of FIG. 4). As shown, the V-PCC bitstream 700 includes a number of V-PCC units 702. Each V-PCC unit 702 includes a V-PCC unit header 704 and a V-PCC unit payload 706. The V-PCC unit header 704 describes the V-PCC unit payload 706 carried by the V-PCC unit 702.

[0123] The V-PCC unit payload 706 includes a sequence parameter set 708, patch sequence data 710, occupancy video data 712, geometry video data 714, and attribute video data 716 (e.g., texture video data). The sequence parameter set 708 includes parameters and information corresponding to a sequence of point clouds (e.g., the sequence 400 of point cloud frames 402, 404, 406 each containing the point cloud 408 of FIG. 4). The occupancy video data 712, geometry video data 714, and attribute video data 716 carry occupancy, geometry, and attribute or texture data corresponding to a point cloud or a 3D image.

[0124] The patch sequence data 710 includes parameters and information corresponding to a sequence of patches (e.g., patch 502 of FIG. 5). The patch sequence data 710 includes a sequence parameter set 720, a frame parameter set 722, a geometry parameter set 724, an attribute parameter set 726, a geometry patch parameter set 728, an attribute patch parameter set 730, and a patch frame 732.

[0125] The sequence parameter set 720 includes parameters and information corresponding to a sequence of 2D patches (e.g., patch 502 in FIG. 5) that last for the entire duration of the point cloud sequence or until a new sequence parameter set is signaled in the bitstream. The frame parameter set 722 includes information corresponding to a sequence of 2D patches that last for the duration of a single frame of the point cloud sequence. The geometry parameter set 724 includes information corresponding to a restoration process of the geometry characteristics of the patch sequence data. The attribute parameter set 726 includes information corresponding to a restoration process of the attribute characteristics of the patch sequence data. The geometry patch parameter set 728 includes information corresponding to a decoding process of the geometry characteristics of the patch sequence data. The attributes 730 include information corresponding to a decoding process of the attribute characteristics of the patch sequence data.

[0126] A patch frame data unit 732 (referred to as a patch frame) contains a set of patch data units 734. The patch data units 734 collectively contain patch data (e.g., patch data F0, ..., Fk) for a patch (e.g., patch 502). Each of the patch data units 734 has a patch type, referred to herein as a patch mode. As described in more detail below, the patch mode (e.g., patch_mode) indicates how a particular patch should be treated during the coding process.

[0127] Figure 8 is a diagram of a coding decomposition of a point cloud sequence 800 for multiple frames 802. The frames 802 of Figure 8 are similar to the point cloud frames 402, 404, 406 of Figure 4. Thus, each of the frames 802 includes point cloud content 810 for a 3D point cloud. The point cloud content 810 in each frame 802 of Figure 8 corresponds to an occupancy map 812, a geometry map 814, and an attribute map 816 in a 2D projection that are used to reconstruct the point cloud content 810. As illustrated, the 2D projection may include more than one attribute map 816.

[0128] As shown, the point cloud sequence 800 also includes an auxiliary information patch frame data unit 850 (AUX_INF). In one embodiment, the auxiliary information patch frame data unit 850 includes both 2D and 3D bounding box information (e.g., patch 3D bounding box 1204 and patch 2D projected bounding box 1208 of FIG. 12). The collection of auxiliary information patch frame data units 850 corresponds to the patch sequence data 710 of FIG. 7. Each auxiliary information patch frame data unit 850 includes a sequence parameter set 720, a frame parameter set 722, a geometry parameter set 724, an attribute parameter set 726, a geometry patch parameter set 728, an attribute patch parameter set 730, and a patch frame 732. In one embodiment, the patch frame 732 is included in the auxiliary information patch frame data unit 850, and the remaining components in the patch sequence data 710 are optional. In one embodiment, the set of occupancy maps 812 corresponds to the occupancy video data 712, the set of geometry maps 814 corresponds to the geometry video data 714, and the set of attribute maps 816 corresponds to the attribute video data 716. Notably, no occupancy maps are present in this image. The auxiliary information patch data units 850 each correspond to the point cloud content 810 in one of the frames 802. The patch data units (e.g., 850) contain information used or needed to reconstruct the point cloud content 810 in the 3D point cloud based on the occupancy maps 812, the geometry maps 814, and the attribute maps 816 in a 2D projection.

[0129] Figure 9 is an illustration of a single patch data unit 900 extracted from one of the point cloud frames 802 in the point cloud sequence 800 of Figure 8. The single patch data unit 900 is similar to the patch data unit 734 in the patch frame 732 or the auxiliary information patch frame data unit 850 of Figure 7. The single patch data unit 900 is depicted relative to a surrounding 3D space 902.

[0130] 10A-10C are diagrams of an occupancy map 1000, a geometry map 1002, and an attribute map 1004 corresponding to a single patch data unit 900 of FIG. 9. The occupancy map 1000, the geometry map 1002, and the attribute map 1004 are similar to the occupancy map 610, the geometry map 612, and the attribute map 614 of FIG. 6A-6C. However, the occupancy map 1000, the geometry map 1002, and the attribute map 1004 of FIG. 10A-10C represent only a single patch data unit 900, rather than a collection of patches 502 from the entire bounding box 500. Thus, the occupancy map 1000, the geometry map 1002, and the attribute map 1004 of FIG. 10A-10C include only a single patch 1006 corresponding to the single patch data unit 900 of FIG. 9.

[0131] Figure 11 is an illustration of 2D components for a patch data unit 1152 within a 2D bounding box 1154. The patch data unit 1152 of Figure 11 is similar to the single patch data unit 900 of Figure 9. As shown in Figure 11, the 2D components represent the patch coordinates (e.g., [u0, v0]) and dimensions (e.g., Size_u0, Size_v0) for the patch data unit 1152. These 2D components are contained in a patch data unit (e.g., patch data unit 734 of Figure 7 or patch data unit 850 of Figure 8).

[0132] 12A-12B are a schematic diagram 1200 of 3D components and a dataset diagram 1252 for the patch data unit 1152 of FIG. 11. As shown, the schematic diagram 1200 includes a point cloud 3D bounding box 1202, a patch 3D bounding box 1204, a projection plane 1206, and a patch 2D projected bounding box 1208. Various coordinates (e.g., [0,0,0]) and dimensions (e.g., x, y, z, u, v, d1, u1, v1) and vectors (b, t, bt) are used to describe the process of projecting the contents of the point cloud 3D bounding box 1202 onto the patch 2D projected bounding box 1208. The dataset diagram 1252 depicts how a single patch data unit 1254 (similar to single patch data unit 900) can be projected onto the projection plane 1206 based on coordinates (e.g., [0,0,0], [u1,v1,d1]) and vectors (e.g., n,bt). These 3D components are also included in a patch data unit (e.g., patch data unit 734 in FIG. 7 or patch data unit 850 in FIG. 8).

[0133] The patch data unit (e.g., patch data unit 734 in FIG. 7 or patch data unit 850 in FIG. 8) contains information necessary for the 2D to 3D reconstruction process, including the following (see FIG. 11): 2d related components U0(Patch2dShiftU) V0(Patch2dShiftV) Size_u0(Patch2dSizeU) Size_v0(Patch2dSizeV) and the following (see Figures 12A-12B): 3D related ingredients: u1 (Patch3dShiftTangentAxis) - distance from the 3d patch bounding box to the point cloud bounding box along the tangent axis (t) v1 (Patch3dShiftBiTangentAxis) - Distance from the 3d patch bounding box to the point cloud bounding box along the bitangent axis (bt). d1 (Patch3dShiftNormalAxis) - distance from the 3d patch bounding box to the point cloud bounding box along the normal axis (n) n (PatchNormalAxis) - The axis perpendicular (normal) to the patch projection plane (any side of the bounding box can be made into the projection plane) t (Patch3dShiftTangentAxis) - the axis tangent to the patch surface and perpendicular to the normal axis bt (Patch3dShiftBiTangentAxis) - Axis perpendicular to the normal and tangent axes It is represented as follows.

[0134] As mentioned above, conventional coding techniques require that patches (patch data unit 734 in FIG. 7) be ordered according to their patch mode within the patch frame data unit. That is, the representation of patch data unit types within a patch frame data unit in the V-PCC requires that all patch-related information be placed in a specific order within the patch frame data unit. First, matched patch data units are added to the patch data frame, followed by non-matched patch data units. The matched patch data units may be coded using differential coding (inter prediction - dpdu), and non-matched patch data units are coded using absolute values ​​(intra prediction). That is, all inter-coded patches are enumerated first within the patch frame data unit, followed by all intra-coded patches, and so on. Thus, in order to add a new inter-coded patch or a new intra-coded patch to the patch frame data unit, the entire patch frame data unit must be reconstructed using the required order, resulting in coding inefficiencies. The method detailed below overcomes these coding inefficiencies.

[0135] Figure 13 is one embodiment of a patch frame data unit encoding process 1300. The patch frame data unit encoding process 1300 may be utilized to encode (aka compress) multiple patch data units, such as patch data unit 734 of Figure 7. The patch frame data unit encoding process 1300 may be implemented by an encoder, where a patch frame data unit corresponding to a 2D bounding box 1154 is a collection of patch data units corresponding to patch data unit 1152. In one embodiment, the input to the encoding process 1300 is a collection of patch data units corresponding to a patch frame data unit.

[0136] At block 1302, a patch frame data unit (pfdu) is obtained. The patch frame data unit may be received by a receiver of the encoder. The patch frame data unit consists of a collection of patch data units. The patch data unit identifies or includes a patch type (pdfu_patch_mode) for each of a plurality of patch data units (e.g., patch 502). In one embodiment, the patch type is one of skip, intra, inter, PCM, or last. The skip patch type indicates that the current patch is identical to the reference patch. Thus, parameters from the reference patch are used in coding the current patch. The intra patch type indicates that the current patch is coded using intra prediction (aka intra-coded). The inter patch type indicates that the current patch is coded using inter prediction (aka inter-coded). The PCM patch type indicates that the current patch represents scattered points independent of the 3D point cloud. The last patch type indicates that the current patch is the last patch to be encoded and therefore the compression process of the patch frame data unit should be finished or terminated.

[0137] At block 1304, the patch data unit index (p) is decremented. Decrementing the patch data unit index allows the patch data unit index to start at a negative value (e.g., -1). Thus, when the patch data unit index is incremented at block 1306, the patch data unit index is effectively initialized to start at zero. In one embodiment, process 1300 may also initially set the patch data unit index to 0 and then increment the patch data unit index after a loop or iterative portion of the process has been executed (as described below).

[0138] At block 1308, a determination is made whether the patch data unit type for the current patch data unit from the multiple patch data units in the patch frame data unit is the last patch type. When the current patch data unit is not the last patch, the process 1300 proceeds to block 1310. At block 1310, patch information data is encoded for the current patch. In one embodiment, the patch information data includes a frame index (frmIdx), a patch index (p), and a patch type (pdfu_patch_mode), as well as corresponding 2D and 3D information. After encoding at block 1310, the process 1300 loops back to block 1306, where the patch data unit index is incremented. By incrementing the patch index, a new current patch data unit is considered. These steps are repeated until it is determined at block 1308 that the current patch data unit is the last patch.

[0139] If it is determined that the current patch data unit is the last patch data unit, the process 1300 proceeds to block 1312. In block 1312, patch information data is encoded for the current patch data unit. In one embodiment, the patch information data includes a frame index (frmIdx), a patch index (p), and a patch type (pdfu_patch_mode), and it should be mentioned that the last patch data unit does not have 2D or 3D information associated with the patch data unit. Unlike block 1310, the patch type in block 1312 is encoded as the last patch type.

[0140] At block 1314, a byte alignment process is performed. In one embodiment, byte alignment organizes or places bytes of data in a desired orientation or size. A compressed patch frame data unit is then realized at block 1316. The compressed patch frame data unit may be incorporated into a bitstream for transmission to a decoder.

[0141] Figure 14 is one embodiment of a patch frame data unit encoding process 1400 that uses a last patch flag. The patch frame data unit encoding process 1400 may be utilized to encode (a.k.a. compress) multiple patch data units, such as patch data unit 734 of Figure 7. The patch frame data unit encoding process 1400 may be implemented by an encoder.

[0142] At block 1402, a patch frame data unit (pfdu) is obtained. The patch frame data unit may be received by a receiver of the encoder. The patch frame data unit includes multiple patch data units (e.g., patch 502).

[0143] At block 1404, the patch index (p) is decremented. Decrementing the patch index allows the patch index to start from a negative value (e.g., -1). At block 1406, a last patch flag is added to each of the multiple patches. In one embodiment, the last patch flag is specified as pfdu_is_last_patch_flag. The last patch flag may be set or assigned a value depending on the patch type of each patch. For example, the patch flag may indicate that the patch type for the patch is one of skip, intra, inter, PCM, or last.

[0144] At block 1408, the patch index is incremented to effectively initialize the patch index to start from zero. At block 1410, a determination is made whether a patch type for a current patch data unit from a plurality of patch data units corresponding to a patch frame data unit is the last patch type based on the value of the last patch flag. When the current patch data unit is not the last patch, the process 1400 proceeds to block 1412. At block 1412, patch information data is encoded for the current patch data unit. In one embodiment, the patch information data includes a frame index (frmIdx), a patch index (p), and a patch type (pdfu_patch_mode), as well as corresponding 2D and 3D information. At block 1412 of the process 1400, the patch type may be skip, inter, intra, or PCM, but is not last. At block 1414, the last patch flag is encoded.

[0145] After encoding at block 1414, process 1400 loops back to block 1408, where the patch index is incremented. By incrementing the patch index, the new current patch is considered. These steps are repeated until the current patch is determined to be the last patch at block 1410 based on the value of the last patch flag.

[0146] If the current patch is determined to be the last patch based on the last patch flag, the process 1400 proceeds to block 1416. At block 1416, a byte alignment process is performed. A compressed patch frame data unit is then realized at block 1418. The compressed patch frame data unit may be embedded in a bitstream for transmission to a decoder.

[0147] FIG. 15 is an embodiment of a patch information data frame 1500 with a description of patch unit types. The patch information data frame (patch frame data unit) 1500 is similar to the patch frame (patch frame data unit) 732 of FIG. 7. As shown, the patch information data frame 1500 includes a reference patch frame data unit 1502 (denoted by refFrmIdx), a current patch frame data unit (frmIdx) 1504, and a patch information unit 1506. The reference patch frame data unit 1502 includes a reference frame used in connection with, for example, inter prediction or skip prediction. The current patch frame data unit 1504 includes multiple patch data units being encoded or decoded. The patch information data unit 1506 includes a patch data unit of a patch having one of the available patch types (e.g., patch 502). In one embodiment, the patch data units may be intra patch type (pdu) 1508, PCM patch type (ppdu) 1510, inter patch type (dpdu) 1512, and skip patch type (spdu) 1514 patch data units. There may be any number of patches in the patch information unit 1506, and the patches may be in any order.

[0148] 16 is one embodiment of a patch frame data unit decoding process 1600. The patch frame data unit decoding process 1600 may be utilized to decode (a.k.a., decompress) a stereoscopic image that has been encoded by an encoder. The patch frame data unit decoding process 1600 may be implemented by the encoder (e.g., entropy decoding unit 70).

[0149] In block 1602, encoded patch information data is received and read. The encoded patch information data may be received by a decoder receiver as part of a bitstream. The encoded patch information data includes data and information necessary to decode a stereoscopic image encoded by an encoder. For example, the encoded patch information data is compressed and includes a number of patch data units (e.g., patches 502 or patch data units 850) necessary to reconstruct a stereoscopic image (e.g., a 3D point cloud) from the patches. As part of block 1602, when process 1600 starts, a patch (e.g., an initial patch data unit or a current patch data unit) may be obtained from a number of patches.

[0150] At block 1604, an input corresponding to encoded patch information data is received. In one embodiment, the input corresponds to one or more of the compressed patches. In one embodiment, the input comprises one or more of a patch_mode, a patch index, a reference index, a frame index, and a reference frame index corresponding to the patch.

[0151] At block 1606, a determination is made whether the patch type for the patch is a skip patch type. When the patch type for the patch is a skip patch type, process 1600 proceeds to block 1608. At block 1608, a reference patch index (spdu_patch_index) corresponding to the patch is decoded. That is, the value of the reference patch index is decoded. In one embodiment, spdu_patch_index is the index of the patch in the reference frame indicated by refFrmIdx.

[0152] At block 1610, a reference index (refIdx) for the patch is determined based on the reference frame index ([refFrmIdx]) corresponding to the patch and the reference patch index decoded when the patch type is a skip patch type. In one embodiment, the reference index, sometimes referred to as the reference patch index (RefPatchIdx), is determined based on the following: RefPatchIdx=predictorIdx+spdu_patch_index[frmIdx][p], where predictorIdx is the prediction index, spdu_patch_index is the skip type patch data unit index, frmIdx is the frame index, and p is the patch index. In one embodiment, a pointer to the correct patch data unit in the current patch frame data unit index (frmIdx) and patch index (p) points to the appropriate spot in the multi-dimensional array to find the actual reference index for the patch.

[0153] At block 1612, the patch is restored based on the determined reference index when the patch type is a skip patch type. In one embodiment, the patch is restored according to the following: Restoration: Patch2dShiftU[frmIdx][p]=Patch2dShiftU[refIdx][RefPatchIdx]*ops_occupancy_packing_block_size Patch2dShiftV[frmIdx][p]=Patch2dShiftV[refIdx][RefPatchIdx]*ops_occupancy_packing_block_size Patch2dSizeU[frmIdx][p]=Patch2dSizeU[refIdx][RefPatchIdx]*ops_occupancy_packing_block_size Patch2dSizeV[frmIdx][p]=Patch2dSizeV[refIdx][RefPatchIdx]*ops_occupancy_packing_block_size Patch3dShiftTangentAxis[frmIdx][p]=Patch3dShiftTangentAxis[refIdx][RefPatchIdx] Patch3dShiftBiTangentAxis[frmIdx][p]=Patch3dShiftBiTangentAxis[refIdx][RefPatchIdx] Patch3dShiftNormalAxis[frmIdx][p]=Patch3dShiftNormalAxis[refIdx][RefPatchIdx] PatchNormalAxis[frmIdx][p]=PatchNormalAxis[refIdx][RefPatchIdx] PatchTangentAxis[frmIdx][p]=PatchTangentAxis[refIdx][RefPatchIdx] PatchBiTangentAxis[frmIdx][p]=PatchBiTangentAxis[refIdx][RefPatchIdx] PatchOrientationSwapFlag[frmIdx][p]=PatchOrientationSwapFlag[refIdx][RefPatchIdx] PatchLod[frmIdx][p]=PatchLod[refIdx][RefPatchIdx] PatchProjectionMode[frmIdx][p]=dpdu_projection_mode[frmIdx][p]

[0154] In one embodiment, the patch is restored according to the following: Restoration: Patch2dShiftU[p]=pdu_2d_shift_u[refIdx] Patch2dShiftV[p]=pdu_2d_shift_v[refIdx] Patch2dSizeU[p]=Patch2dSizeU[refIdx] Patch2dSizeV[p]=Patch2dSizeV[refIdx] Patch3dShiftT[p]=Patch3dShiftT[refIdx] Patch3dShiftBT[p]=Patch3dShiftBT[refIdx] Patch3dShiftN[p]=Patch3dShiftN[refIdx] PatchNormalAxis[p]=PatchNormalAxis[refIdx] Orientation[p]=Orientation[refIdx] PatchLod[p]=PatchLod[refIdx]

[0155] At block 1650, the recovered patch information data is collected (eg, stored in memory) and process 1600 loops back to block 1602 so that a new patch can be decoded.

[0156] At block 1614, a determination is made whether the patch type for the patch data unit is an intra-patch type. When the patch type for the patch is an intra-patch type, process 1600 proceeds to block 1616. At block 1616, a reference patch data unit index (refIdx) for the patch data unit is determined based on the previous patch data unit (corresponding to the patch and decremented patch index) in the current frame index ([frmIdx]). In one embodiment, the current frame (e.g., patch frame data unit) is used for the reference.

[0157] At block 1618, the patch is decoded using a two-dimensional (2D) component corresponding to the patch and a three-dimensional (3D) component corresponding to the patch. In one embodiment, the patch is decoded according to the following: Decryption operation: u0(pdu_2d_shift_u) u1(pdu_2d_shift_v) d_size_u0(pdu_2d_delta_size_u) d_size_v0(pdu_2d_delta_size_v) u1(pdu_3d_shift_tangent_axis) v1(pdu_3d_shift_bitangent_axis) d1(pdu_3d_shift_normaI_axis) n(pdu_normI_axis) swap(pdu_orientation_swap_flag) L0D(pdu_Iod)

[0158] At block 1620, the patch is restored based on the decoded patch. In one embodiment, the patch is decoded according to: Restoration: Patch2dShiftU[p]=pdu_2d_shift_u[p] Patch2dShiftV[p]=pdu_2d_shift_v[p] Patch2dSizeU[p]=pdu_2d_delta_size_u[p]+ +Patch2dSizeU[refldx] Patch2dSizeV[p]=pdu_2d_deIta_size_v[p]+ +Patch2dsizeV[refldx] Patch3dShiftT[p]=pdu_3d_shift_tan[p] Patch3dShiftBT[p]=pdu_3d_shift_bitan[p] Patch3dShiftN[p]=pdu_shift_norm[p] PatchNormalAxis[p]=pdu_norm_axis[p] Orientation[p]=pdu_orientation_swap_flag[p] PatchLod[p]=pdu_Iod[p]

[0159] At block 1650, the recovered patch information data is collected (eg, stored in memory) and process 1600 loops back to block 1602 so that a new patch can be decoded.

[0160] At block 1622, a determination is made whether the patch type for the patch is an inter patch type. When the patch type for the patch is an inter patch type, the process 1600 proceeds to block 1624. At block 1624, a reference patch index (dpdu_patch_index) corresponding to the patch is decoded. That is, the value of the reference patch index is decoded. At block 1626, a reference index (refIdx) for the patch is determined based on the reference frame index ([refFrmIdx]) corresponding to the patch and the decoded reference patch index.

[0161] At block 1628, the patch is decoded using a two-dimensional (2D) component corresponding to the patch and a three-dimensional (3D) component corresponding to the patch. In one embodiment, the patch is decoded according to the following: Decryption operation: d_u0(pdu_2d_shift_u) d_u1(pdu_2d_shift_v) d_size_u0(pdu_2d_delta_size_u) d_size_v0(pdu_2d_delta_size_v) d_u1(pdu_3d_shift_tangent_axis) d_v1(pdu_3d_shift_bitangent_axis) d_d1(pdu_3d_shift-normal_axis)

[0162] At block 1630, the patch is reconstructed based on the decoded patch. In one embodiment, the patch is reconstructed according to the following: Restoration: Patch2dShiftU[p]=pdu_2d_shift_u[p]+ +Patch2dShiftU[refldx] Patch2dShiftVf[p]=pdu_2d_shift_v[p]+ +Patch2dshiftV[refldx] Patch2dsizeU[p]=pdu_2d_delta_size_u[p]+ +Patch2dSizeU[refldx] Patch2dSizeV[p]=pdu_2d_delta_size_v[p]+ +Patch2dSizeV[refldx] Patch3dShiftT[p]=pdu_3d_shift_tan[p]+ +Patch3dshiftT[refldx] Patch3dShiftBT[p]=pdu_3d_shift_bitan[p]+ +patch3dshiftBT[refldx] Patch3dShiftN[p]=pdu_shift_norm[p]+ +Patch3dShiftN[refldx] PatchNormaIAxis[p]=PatchnormaIAxis[refldx] Orientation[p]=Orientation[refldx] PatchLod[p] = PatchLod[refldx]

[0163] At block 1650, the recovered patch information data is collected (eg, stored in memory) and process 1600 loops back to block 1602 so that a new patch can be decoded.

[0164] At block 1632, a determination is made whether the patch type for the patch is a PCM patch (or raw) patch type. When the patch type for the patch is a PCM patch type, process 1600 proceeds to block 1634. At block 1634, a reference index (refIdx) for the patch is determined based on the frame index ([frmIdx]) corresponding to the patch and the decremented patch index.

[0165] At block 1636, the patch is decoded using a two-dimensional (2D) component corresponding to the patch and a three-dimensional (3D) component corresponding to the patch. In one embodiment, the 2D and 3D components comprise: Decryption operation: separate_video_flag(ppdu_patch…) u0(ppdu_2d_shift_u) u1(ppdu_2d_shift_y) d_size_u0(ppdu_2d_delta_size_u) d_sze_v0(ppdu_2d_delta_size_v) PCM points (ppdu_pcm_points)

[0166] At block 1638, the patch is reconstructed based on the decoded patch. In one embodiment, the patch is reconstructed according to the following: Restoration: Patch2dShiftU[p]=pdu_2d_shift_u[p] Patch2dShiftV[p]=pdu_2d_shifty[p] Patch2dSizeU[p]=pdu_2d_deltasize_u[p]+ +Patch2dSizeU[refIdx] Patch2dSizeV[p]=pdu_2d_deIta_sizev[p]+ +Patch2dSizeV[refldx] PatchPcmPoints[p]=ppdu_pcm_points[p]

[0167] At block 1650, the recovered patch information data is collected (eg, stored in memory) and process 1600 loops back to block 1602 so that a new patch can be decoded.

[0168] At block 1640, a determination is made whether the patch type for the patch is the last patch type. When the patch type for the patch is the last patch type, process 1600 proceeds to block 1642. When the patch type is the last patch type, at block 1642, the reconstruction process corresponding to the encoded patch information data is terminated. That is, the reconstruction process is terminated. In one embodiment, any data included in the patch having the last patch type may be decoded before the patch frame data unit decoding process 1600 is terminated.

[0169] 17 is one embodiment of a patch frame data unit decoding process 1700. The patch frame data unit decoding process 1700 may be utilized to decode (a.k.a., decompress) a stereoscopic image encoded by an encoder. The frame patch data unit decoding process 1700 may be implemented by an encoder (e.g., entropy decoding unit 70).

[0170] In block 1702, encoded patch information data is received and read. The encoded patch information data may be received by a decoder receiver as part of a bitstream. The encoded patch information data includes data and information necessary to decode a stereoscopic image encoded by an encoder. For example, the encoded patch information data is compressed and includes a number of patch data units (e.g., patches 502 or patch data units 850) necessary to reconstruct a stereoscopic image (e.g., a 3D point cloud) from the patches. As part of block 1702, when process 1700 starts, a patch (e.g., an initial patch or a current patch) may be obtained from a number of patches.

[0171] At block 1704, an input corresponding to encoded patch information data is received. In one embodiment, the input corresponds to one or more of the compressed patches. In one embodiment, the input comprises one or more of a patch_mode, a patch index, a reference index, a frame index, and a reference frame index corresponding to the patch.

[0172] At block 1706, a determination is made whether the patch type for the patch is a skip patch type. When the patch type for the patch is a skip patch type, process 1700 proceeds to block 1708. At block 1708, a reference patch index (spdu_patch_index) corresponding to the patch is decoded (or determined). That is, the value of the reference patch index is decoded or determined.

[0173] At block 1710, a reference index (refIdx) for the patch is determined based on a reference frame index ([refFrmIdx]) corresponding to the patch and a reference patch index decoded when the patch type is a skip patch type. In one embodiment, the reference index, sometimes referred to as a reference patch index (RefPatchIdx), is determined based on the following: RefPatchIdx=predictorIdx+spdu_patch_index[frmIdx][p], where predictorIdx is a prediction index, spdu_patch_index is a skip type patch data unit index, frmIdx is a frame index, and p is a patch index.

[0174] At block 1712, the patch is restored based on the determined reference index when the patch type is a skip patch type. In one embodiment, the patch is restored according to the following: Restoration: Patch2dShiftU[frmIdx][p]=Patch2dShiftU[refIdx][RefPatchIdx]*ops_occupancy_packing_block_size Patch2dShiftV[frmIdx][p]=Patch2dShiftV[refIdx][RefPatchIdx]*ops_occupancy_packing_block_size Patch2dSizeU[frmIdx][p]=Patch2dSizeU[refIdx][RefPatchIdx]*ops_occupancy_packing_block_size Patch2dSizeV[frmIdx][p]=Patch2dSizeV[refIdx][RefPatchIdx]*ops_occupancy_packing_block_size Patch3dShiftTangentAxis[frmIdx][p]=Patch3dShiftTangentAxis[refIdx][RefPatchIdx] Patch3dShiftBiTangentAxis[frmIdx][p]=Patch3dShiftBiTangentAxis[refIdx][RefPatchIdx] Patch3dShiftNormalAxis[frmIdx][p]=Patch3dShiftNormalAxis[refIdx][RefPatchIdx] PatchNormalAxis[frmIdx][p]=PatchNormalAxis[refIdx][RefPatchIdx] PatchTangentAxis[frmIdx][p]=PatchTangentAxis[refIdx][RefPatchIdx] PatchBiTangentAxis[frmIdx][p]=PatchBiTangentAxis[refIdx][RefPatchIdx] PatchOrientationSwapFlag[frmIdx][p]=PatchOrientationSwapFlag[refIdx][RefPatchIdx] PatchLod[frmIdx][p]=PatchLod[refIdx][RefPatchIdx] PatchProjectionMode[frmIdx][p]=dpdu_projection_mode[frmIdx][p]

[0175] In one embodiment, the patch is restored according to the following: Restoration: Patch2dShiftU[p]=pdu_2d_shift_u[refIdx] Patch2dShiftV[p]=pdu_2d_shift_v[refIdx] Patch2dSizeU[p]=Patch2dSizeU[refIdx] Patch2dSizeV[p]=Patch2dSizeV[refIdx] Patch3dShiftT[p]=Patch3dShiftT[refIdx] Patch3dShiftBT[p]=Patch3dShiftBT[refIdx] Patch3dShiftN[p]=Patch3dShiftN[refIdx] PatchNormalAxis[p]=PatchNormalAxis[refIdx] Orientation[p]=Orientation[refIdx] PatchLod[p]=PatchLod[refIdx]

[0176] At block 1760, the flag is decoded to determine whether there are more patches to consider. In one embodiment, the flag is designated as more_patches_available_flag. At block 1762, the value of the flag is determined. When the flag has a first value (e.g., 1), the process 1700 proceeds to block 1764. At block 1764, the restored patch information data is stored and the process loops back to block 1702 so that a new patch can be decoded. When the flag has a second value (e.g., 0), the process 1700 proceeds to block 1742. When the patch type is the last patch type, at block 1742, the restoration process corresponding to the encoded patch information data is terminated. That is, the restoration process is terminated.

[0177] At block 1714, a determination is made whether the patch type for the patch data unit is an intra-patch type. When the patch type for the patch is an intra-patch type, process 1700 proceeds to block 1716. At block 1716, a reference patch data unit index (refIdx) for the patch data unit is determined based on the previous patch data unit (corresponding to the patch and the decremented patch index) in the current frame index ([frmIdx]).

[0178] At block 1718, the patch is decoded using a two-dimensional (2D) component corresponding to the patch and a three-dimensional (3D) component corresponding to the patch. In one embodiment, the patch is decoded according to the following: Decryption operation: u0(pdu_2d_shift_u) u1(pdu_2d_shift_v) d_size_u0(pdu_2d_delta_size_u) d_size_v0(pdu_2d_delta_size_v) u1(pdu_3d_shift_tangent_axis) v1(pdu_3d_shift_bitangent_axis) d1(pdu_3d_shift_normaI_axis) n(pdu_normI_axis) swap(pdu_orientation_swap_flag) L0D(pdu_Iod)

[0179] At block 1720, the patch is restored based on the decoded patch. In one embodiment, the patch is decoded according to: Restoration: Patch2dShiftU[p]=pdu_2d_shift_u[p] Patch2dShiftV[p]=pdu_2d_shift_v[p] Patch2dSizeU[p]=pdu_2d_delta_size_u[p]+ +Patch2dSizeU[refldx] Patch2dSizeV[p]=pdu_2d_deIta_size_v[p]+ +Patch2dsizeV[refldx] Patch3dShiftT[p]=pdu_3d_shift_tan[p] Patch3dShiftBT[p]=pdu_3d_shift_bitan[p] Patch3dShiftN[p]=pdu_shift_norm[p] PatchNormalAxis[p]=pdu_norm_axis[p] Orientation[p]=pdu_orientation_swap_flag[p] PatchLod[p]=pdu_Iod[p]

[0180] At block 1760, the flag is decoded to determine whether there are more patches to consider. In one embodiment, the flag is designated as more_patches_available_flag. At block 1762, the value of the flag is determined. When the flag has a first value (e.g., 1), the process 1700 proceeds to block 1764. At block 1764, the restored patch information data is stored, and the process 1700 loops back to block 1702 so that a new patch can be decoded. When the flag has a second value (e.g., 0), the process 1700 proceeds to block 1742. When the patch type is the last patch type, at block 1742, the restoration process corresponding to the encoded patch information data is terminated. That is, the restoration process is terminated.

[0181] At block 1722, a determination is made whether the patch type for the patch is an inter patch type. When the patch type for the patch is an inter patch type, process 1700 proceeds to block 1724. At block 1724, a reference patch index (dpdu_patch_index) corresponding to the patch is decoded. That is, the value of the reference patch index is decoded. At block 1726, a reference index (refIdx) for the patch is determined based on the reference frame index ([refFrmIdx]) corresponding to the patch and the decoded reference patch index.

[0182] At block 1728, the patch is decoded using a two-dimensional (2D) component corresponding to the patch and a three-dimensional (3D) component corresponding to the patch. In one embodiment, the patch is decoded according to the following: Decryption operation: d_u0(pdu_2d_shift_u) d_u1(pdu_2d_shift_v) d_size_u0(pdu_2d_delta_size_u) d_size_v0(pdu_2d_delta_size_v) d_u1(pdu_3d_shift_tangent_axis) d_v1(pdu_3d_shift_bitangent_axis) d_d1(pdu_3d_shift-normal_axis)

[0183] At block 1730, the patch is reconstructed based on the decoded patch. In one embodiment, the patch is reconstructed according to the following: Restoration: Patch2dShiftU[p]=pdu_2d_shift_u[p]+ +Patch2dShiftU[refldx] Patch2dShiftVf[p]=pdu_2d_shift_v[p]+ +Patch2dshiftV[refldx] Patch2dsizeU[p]=pdu_2d_delta_size_u[p]+ +Patch2dSizeU[refldx] Patch2dSizeV[p]=pdu_2d_delta_size_v[p]+ +Patch2dSizeV[refldx] Patch3dShiftT[p]=pdu_3d_shift_tan[p]+ +Patch3dshiftT[refldx] Patch3dShiftBT[p]=pdu_3d_shift_bitan[p]+ +patch3dshiftBT[refldx] Patch3dShiftN[p]=pdu_shift_norm[p]+ +Patch3dShiftN[refldx] PatchNormaIAxis[p]=PatchnormaIAxis[refldx] Orientation[p]=Orientation[refldx] PatchLod[p] = PatchLod[refldx]

[0184] At block 1760, the flag is decoded to determine whether there are more patches to consider. In one embodiment, the flag is designated as more_patches_available_flag. At block 1762, the value of the flag is determined. When the flag has a first value (e.g., 1), the process 1700 proceeds to block 1764. At block 1764, the restored patch information data is stored, and the process 1700 loops back to block 1702 so that a new patch can be decoded. When the flag has a second value (e.g., 0), the process 1700 proceeds to block 1742. When the patch type is the last patch type, at block 1742, the restoration process corresponding to the encoded patch information data is terminated. That is, the restoration process is terminated.

[0185] At block 1732, a determination is made whether the patch type for the patch is a PCM patch (or raw) patch type. When the patch type for the patch is a PCM patch type, process 1700 proceeds to block 1734. At block 1734, a reference index (refIdx) for the patch is determined based on the frame index ([frmIdx]) corresponding to the patch and the decremented patch index.

[0186] At block 1736, the patch is decoded using a two-dimensional (2D) component corresponding to the patch and a three-dimensional (3D) component corresponding to the patch. In one embodiment, the 2D and 3D components comprise: Decryption operation: separate_video_flag(ppdu_patch…) u0(ppdu_2d_shift_u) u1(ppdu_2d_shift_y) d_size_u0(ppdu_2d_delta_size_u) d_sze_v0(ppdu_2d_delta_size_v) PCM points (ppdu_pcm_points)

[0187] At block 1738, the patch is reconstructed based on the decoded patch. In one embodiment, the patch is reconstructed according to the following: Restoration: Patch2dShiftU[p]=pdu_2d_shift_u[p] Patch2dShiftV[p]=pdu_2d_shifty[p] Patch2dSizeU[p]=pdu_2d_deltasize_u[p]+ +Patch2dSizeU[refIdx] Patch2dSizeV[p]=pdu_2d_deIta_sizev[p]+ +Patch2dSizeV[refldx] PatchPcmPoints[p]=ppdu_pcm_points[p]

[0188] At block 1760, the flag is decoded to determine whether there are more patches to consider. In one embodiment, the flag is designated as more_patches_available_flag. At block 1762, the value of the flag is determined. When the flag has a first value (e.g., 1), the process 1700 proceeds to block 1764. At block 1764, the restored patch information data is stored, and the process 1700 loops back to block 1702 so that a new patch can be decoded. When the flag has a second value (e.g., 0), the process 1700 proceeds to block 1742. When the patch type is the last patch type, at block 1742, the restoration process corresponding to the encoded patch information data is terminated. That is, the restoration process is terminated.

[0189] FIG. 18 is an embodiment of a method 1800 of PCC implemented by a decoder (e.g., entropy decoding unit 70). The method 1800 may be used to decode an encoded bitstream to reconstruct a stereoscopic image. At block 1802, a receiver of the decoder receives encoded patch information data. The encoded patch information may include information corresponding to a plurality of patches (e.g., patch 502). At block 1804, a processor of the decoder obtains a patch corresponding to the encoded patch information data. The patch has a patch type (patch_mode). The patch type may be, for example, skip, intra, inter, PCM, or last.

[0190] In block 1806, the decoder processor determines whether the patch type for the patch is the last patch type. In block 1808, the decoder processor ends the recovery process corresponding to the encoded patch information data when the patch type is the last patch type.

[0191] FIG. 19 is an embodiment of a method 1900 of PCC implemented by a decoder (e.g., entropy decoding unit 70). The method 1900 may be used to decode an encoded bitstream to reconstruct a stereoscopic image. At block 1902, a receiver of the decoder receives encoded patch information data. The encoded patch information may include information corresponding to a plurality of patches (e.g., patch 502). At block 1904, a processor of the decoder obtains a patch corresponding to the encoded patch information data. The patch has a patch type (patch_mode). The patch type may be, for example, skip, intra, inter, PCM, or last.

[0192] In block 1906, the decoder processor determines whether the patch type for the patch is a skip patch type. In block 1908, the decoder processor decodes a reference patch index (spdu_patch_index) corresponding to the patch when the patch type is a skip patch type. In block 1910, the decoder processor determines a reference index (refIdx) for the patch based on a reference frame index ([refFrmIdx]) corresponding to the patch and the reference patch index decoded when the patch type is a skip patch type. In block 1912, the decoder processor restores a stereoscopic representation of the patch based on the reference index determined when the patch type is a skip patch type. Once restored, the 3D image may be displayed on a display of an electronic device (e.g., a smartphone, a tablet, a laptop computer, etc.) for a user.

[0193] FIG. 20 is an embodiment of a method 2000 of PCC implemented by a decoder (e.g., entropy decoding unit 70). The method 2000 may be used to decode an encoded bitstream to reconstruct a stereoscopic image. In block 2002, a receiver of the decoder receives encoded patch information data. The encoded patch information may include information corresponding to a plurality of patches (e.g., patch 502). In block 2004, a processor of the decoder obtains a patch corresponding to the encoded patch information data. The patch has a patch type (patch_mode). The patch type may be, for example, skip, intra, inter, PCM, or last.

[0194] In block 2006, the decoder processor determines whether the patch type for the patch is a skip patch type. In block 2008, the decoder processor decodes a reference patch index (spdu_patch_index) corresponding to the patch when the patch type is a skip patch type. In block 2010, the decoder processor determines a reference index (refIdx) for the patch based on a reference frame index ([refFrmIdx]) corresponding to the patch and the reference patch index decoded when the patch type is a skip patch type. In block 2012, the decoder processor restores a stereoscopic representation of the patch based on the reference index determined when the patch type is a skip patch type.

[0195] In block 2014, the decoder processor determines whether the more patches available flag has a first value or a second value. In block 2016, when the more patches available flag has a first value, the restored stereoscopic representation is stored in the decoder memory. Once all of the patches are restored, the 3D image may be displayed on a display of an electronic device (e.g., a smartphone, a tablet, a laptop computer, etc.) for a user.

[0196] In block 2018, the decoder processor ends the process of recovering the encoded patch information data when the more patches available flag has the second value.

[0197] 21 is an embodiment of a method 2100 of PCC implemented by an encoder (e.g., entropy coding unit 56). Method 2100 may be performed to encode stereoscopic images into a bitstream for transmission to a decoder. At block 2102, a receiver of the encoder obtains a patch frame data unit (pfdu) that identifies a patch type (pdfu_patch_mode) for each of a plurality of patches. At block 2104, a processor of the encoder determines whether the patch type for a patch from the plurality of patches is the last patch type.

[0198] At block 2106, the encoder processor encodes patch information data for the patch when the patch type is not the last patch type, the patch information data including the patch type for the patch. At block 2108, the encoder processor encodes patch information data for the patch when the patch type is set to the last patch type, the patch information data including the last patch type for the patch.

[0199] In one embodiment, the patch information data and the final patch type are encoded in the bitstream transmitted to the decoder.

[0200] 22 is an embodiment of a method 2200 of PCC implemented by an encoder (e.g., entropy coding unit 56). Method 2200 may be performed to encode a stereoscopic image into a bitstream for transmission to a decoder. At block 2202, a receiver of the encoder obtains a patch frame data unit (PFDU) for each of a plurality of patches. At block 2204, a processor of the encoder adds a last patch flag to each of a plurality of patches.

[0201] In block 2206, the encoder processor determines whether the patch type for the patch from the multiple patches is the last patch type based on the value of the last patch flag. In block 2208, the encoder processor encodes patch information data for the patch when the patch type is not the last patch type, the patch information data including the patch type and the last patch flag for the patch.

[0202] In one embodiment, the patch information data, including the patch type and the last patch type, is encoded in the bitstream transmitted to the decoder.

[0203] 23 is a schematic diagram of a coding device 2300 (e.g., encoder 22, decoder 28, etc.) according to one embodiment of the present disclosure. The coding device 2300 is suitable for implementing the methods and processes disclosed herein. The coding device 2300 comprises an ingress port 2310 and a receiver unit (Rx) 2320 for receiving data, a processor, logic unit, or central processing unit (CPU) 2330 for processing data, a transmitter unit (Tx) 2340 and an egress port 2350 for transmitting data, and a memory 2360 for storing data. The coding device 2300 may also comprise optical-electrical (OE) and electrical-optical (EO) components coupled to the ingress port 2310, the receiver unit 2320, the transmitter unit 2340, and the egress port 2350 for ingress and egress of optical or electrical signals.

[0204] The processor 2330 is implemented by hardware and software. The processor 2330 may be implemented as one or more CPU chips, cores (e.g., as a multi-core processor), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), and digital signal processors (DSPs). The processor 2330 communicates with the ingress port 2310, the receiver unit 2320, the transmitter unit 2340, the egress port 2350, and the memory 2360. The processor 2330 comprises a coding module 2370. The coding module 2370 implements the above-disclosed embodiments. In one embodiment, the coding module 2370 is a reconstruction module configured to project a reconstructed stereoscopic image. Thus, the inclusion of the coding module 2370 provides substantial improvements in the functionality of the coding device 2300 and provides transitions of the coding device 2300 to different states. Alternatively, the coding module 2370 is implemented as instructions stored in the memory 2360 and executed by the processor 2330.

[0205] Coding device 2300 may also include input and / or output (I / O) devices 2380 for communicating data with a user. The I / O devices 2380 may include output devices such as a display for displaying video data, speakers for outputting audio data, etc. The I / O devices 2380 may also include input devices such as a keyboard, mouse, trackball, etc., and / or corresponding interfaces for interacting with such output devices.

[0206] The memory 2360 may comprise one or more disks, tape drives, and solid state drives, and may be used as an overflow data storage device to store programs when such programs are selected for execution, and to store instructions and data read during the execution of the programs. The memory 2360 may be volatile and non-volatile, and may be read only memory (ROM), random access memory (RAM), ternary content addressable memory (TCAM), and static random access memory (SRAM).

[0207] 24 is a schematic diagram of one embodiment of a means for coding 2400. In an embodiment, the means for coding 2400 is implemented in a coding device 2402 (e.g., the encoder 22 or the decoder 28). The coding device 2402 includes a means for receiving 2401. The means for receiving 2401 is configured to receive a picture to encode or to receive a bitstream to decode. The coding device 2402 includes a means for transmitting 2407 coupled to the means for receiving 2401. The means for transmitting 2407 is configured to transmit the bitstream to a decoder or to transmit a decoded image to a display means (e.g., one of the I / O devices 2380).

[0208] The coding device 2402 includes a storage means 2403. The storage means 2403 is coupled to at least one of the receiving means 2401 or the transmitting means 2407. The storage means 2403 is configured to store instructions. The coding device 2402 also includes a processing means 2405. The processing means 2405 is coupled to the storage means 2403. The processing means 2405 is configured to execute the instructions stored in the storage means 2403 to perform the methods disclosed herein.

[0209] In one embodiment, a syntax suitable for implementing the concepts disclosed herein is provided.

[0210] Patch Frame Data Unit (pfdu) In one embodiment, a special end patch data unit type is introduced to indicate the end of a patch frame data unit.

[0211] [Table 1]

[0212] In one embodiment, a special one-bit (1-bit) flag is added to each patch data unit to indicate the end of the patch frame data unit.

[0213] [Table 2]

[0214] In one embodiment, the patch information data is as follows:

[0215] [Table 3]

[0216] In one embodiment, the original delta_patch_data_unit syntax element is as follows:

[0217] [Table 4]

[0218] In one embodiment, the added skip_patch_data_unit syntax element is as follows:

[0219] [Table 5]

[0220] The skip patch data unit type has been introduced to reduce the amount of overhead in the coded bitstream representation.

[0221] The semantics of the newly added elements are as follows:

[0222] spdu_patch_index[frmIdx][p] plus p specifies the index PredIdx of the patch within the patch frame having index RefIdx corresponding to the first patch frame in the reference patch frame list.

[0223] The decoding process for a patch data unit coded in skip prediction mode is as follows: When pfdu_patch_mode[frmIdx][p] is equal to P_SKIP, the process for decoding skip coded patch data units is used, frmIdx and p are used as inputs to the process, and the outputs of the process (Patch2dShiftU, Patch2dShiftV, Patch2dSizeU, Patch2dSizeV, Patch3dShiftT, Patch3dShiftBT, Patch3dShiftN, PatchNormalAxis, Orientation, PatchLod) are used as outputs.

[0224] From the above, it should be appreciated that the patch orientation may be different relative to the default projection process. The patch orientation may be signaled in a simplified manner using a one-bit flag. A mechanism is introduced to switch between the default orientation and the preferred orientation.

[0225] Although several embodiments are provided in the present disclosure, it will be understood that the disclosed systems and methods may be embodied in many other specific forms without departing from the spirit or scope of the present disclosure. The examples of the present disclosure should be considered as illustrative rather than limiting, and the intention should not be limited to the details given herein. For example, various elements or components may be combined or integrated in another system, or certain features may be omitted or not implemented.

[0226] In addition, the techniques, systems, subsystems, and methods described and illustrated as separate or distinct in various embodiments may be combined or integrated with other systems, components, techniques, or methods without departing from the scope of the present disclosure. Other examples of modifications, substitutions, and alterations will be ascertainable by those skilled in the art and may be made without departing from the spirit and scope disclosed herein. [Explanation of symbols]

[0227] 10 Coding Systems 12 Source device, video device 14 Destination device, video device 16 Computer-Readable Media 18 Video Sources 20 Projection Devices 22 Video Encoder 24 Output Interface 26 Input Interface 28 Video Decoder 30 Restore Device 32 Display Devices 40 Mode Selection Unit 42 Motion Estimation Unit 44 Motion Compensation Unit 46 intra prediction units 48 Split Unit 50 Adder 52 Conversion Processing Unit 54 Quantization Units 56 Entropy Coding Units 58 Inverse Quantization Unit 60 Reverse conversion unit 62 Adder 64 Reference Frame Memory 70 Entropy Decoding Unit 72 Motion Compensation Unit 74 intra prediction units 76 Inverse Quantization Unit 78 Reverse conversion unit 80 Adder 82 Reference Frame Memory 400 point cloud sequences 402 Point Cloud Frame 404 Point Cloud Frame 406 Point Cloud Frame 408 point cloud 410 Point Cloud Content 412 3D space 500 bounding boxes 502 Patch 504 plane 600 bounding boxes 602 Patch 610 Occupied Map 612 Geometric Map 614 Attribute Map 700 V-PCC Bitstream 702 V-PCC unit 704 V-PCC unit header 706 V-PCC Unit Payload 708 Sequence Parameter Set 710 Patch sequence data 712 Exclusive Video Data 714 Geometric Video Data 716 Attribute Video Data 720 Sequence Parameter Set 722 Frame Parameter Set 724 Geometric Parameter Set 726 Attribute Parameter Set 728 Geometric Patch Parameter Set 730 Attribute Patch Parameter Set 732 Patch Frame Data Unit 734 Patch Data Unit 800 point cloud sequences 802 point cloud frame 810 Point Cloud Content 812 Occupied Map 814 Geometric Map 816 Attribute Map 850 Auxiliary Information Patch Frame Data Unit 900 patch data units 902 3D space 1000 Occupied Maps 1002 Geometric Map 1004 Attribute Map 1006 Patch 1152 Patch Data Unit 1154 2D bounding boxes 1200 Schematic 1202 Point Cloud 3D Bounding Box 1204 Patch 3D Bounding Box 1206 Projection plane 1208 Patch 2D Projected Bounding Box 1252 Dataset Diagram 1254 Patch Data Unit 1300 Patch Frame Data Unit Encoding Process 1400 Patch Frame Data Unit Encoding Process 1500 Patch Information Data Frame 1502 Reference patch frame data unit 1504 Current patch frame data unit 1506 Patch Information Data Unit 1508 Intra-patch type data unit (pdu) 1510 PCM Patch Type Data Unit (PPDU) 1512 Interpatch Type Data Unit (DPDU) 1514 Skip Patch Type Data Unit (SPDU) 1600 Patch Frame Data Unit Decoding Process 1700 Patch frame data unit decoding process 1800 PCC method implemented by decoder 1900 PCC method implemented by decoder 2000 PCC method implemented by the decoder 2100 PCC method implemented by encoder Method of PCC implemented by 2200 encoder 2300 coding device 2310 Inlet Port 2320 Receiver Unit (Rx) 2330 Processor 2340 Transmitter Unit (Tx) 2350 Exit Port 2360 Memory 2370 Coding Module 2380 Input and / or Output (I / O) Device 2400 Coding Tools 2401 Receiving means 2402 Video Coding Device 2403 Memory means 2405 Processing means 2407 Means of transmission

Claims

1. A method of point cloud coding (PCC) implemented by a decoder, comprising the steps of: receiving, by a receiver of the decoder, encoded patch information data; obtaining, by a processor of the decoder, a patch corresponding to the encoded patch information data, the patch having a patch type (patch_mode); determining, by the processor, whether the patch type for the patch is a last patch type; terminating, by the processor, a restoration process corresponding to the encoded patch information data when the patch type is the last patch type; A method comprising:

2. A method of point cloud coding (PCC) implemented by a decoder, comprising the steps of: receiving, by a receiver of the decoder, encoded patch information data; obtaining, by a processor of the decoder, a patch corresponding to the encoded patch information data, the patch having a patch type (patch_mode); determining, by the processor, whether the patch type for the patch is a skip patch type; decoding, by the processor, a reference patch index (spdu_patch_index) corresponding to the patch when the patch type is the skip patch type; determining, by the processor, a reference index (refIdx) for the patch based on a reference frame index ([refFrmIdx]) corresponding to the patch and the reference patch index decoded when the patch type is the skip patch type; recovering, by the processor, a stereoscopic representation of the patch based on the reference index determined when the patch type is the skip patch type; A method comprising:

3. determining that the patch type for the patch is an intra-patch type; determining the reference index (refIdx) for the patch based on a frame index ([frmIdx]) corresponding to the patch and a decremented patch index; decoding the patch using a two-dimensional (2D) component corresponding to the patch and a three-dimensional (3D) component corresponding to the patch; reconstructing the stereoscopic representation based on the decoded patches; The method of claim 2 , further comprising:

4. determining that the patch type for the patch is an inter-patch type; decoding a second reference patch index (dpdu_patch_index) corresponding to the patch; determining the reference index (refIdx) for the patch based on the reference frame index ([refFrmIdx]) corresponding to the patch and the decoded second reference patch index; decoding the patch using a two-dimensional (2D) component corresponding to the patch and a three-dimensional (3D) component corresponding to the patch; reconstructing the stereoscopic representation based on the decoded patches; The method of claim 2 or 3, further comprising:

5. determining that the patch type for the patch is a pulse code modulation (PCM) patch type; determining the reference index (refIdx) for the patch based on a frame index ([frmIdx]) corresponding to the patch and a decremented patch index; decoding the patch using two-dimensional (2D) components for the distinct points corresponding to the patch and three-dimensional (3D) components for the distinct points corresponding to the patch; reconstructing the stereoscopic representation based on the decoded patches; 5. The method of claim 2, further comprising:

6. 6. The method of claim 2, further comprising the step of receiving an input corresponding to the encoded patch information data, the input comprising one or more of the patch_mode, a patch index, the reference index, a frame index, and a reference frame index.

7. The method according to claim 2 , further comprising the step of displaying an image generated based on the reconstructed stereoscopic representation on a display of an electronic device.

8. A method of point cloud coding (PCC) implemented by a decoder, comprising the steps of: receiving, by a receiver of the decoder, encoded patch information data; obtaining, by a processor of the decoder, a patch corresponding to the encoded patch information data, the patch having a patch type (patch_mode); determining, by the processor, whether the patch type for the patch is a skip patch type; decoding, by the processor, a reference patch index (spdu_patch_index) corresponding to the patch when the patch type is the skip patch type; determining, by the processor, a reference index (refIdx) for the patch based on a reference frame index ([refFrmIdx]) corresponding to the patch and the reference patch index decoded when the patch type is the skip patch type; recovering, by the processor, a stereoscopic representation of the patch based on the reference index determined when the patch type is the skip patch type; determining, by the processor, whether a more patch available flag has a first value or a second value; storing the reconstructed stereoscopic representation in a memory of the decoder when the more patches available flag has the first value; terminating, by the processor, the process of recovering the encoded patch information when the more patches available flag has the second value; A method comprising:

9. determining that the patch type for the patch is an intra-patch type; determining the reference index (refIdx) for the patch based on a frame index ([frmIdx]) corresponding to the patch and a decremented patch index; decoding the patch using a two-dimensional (2D) component corresponding to the patch and a three-dimensional (3D) component corresponding to the patch; reconstructing the stereoscopic representation based on the decoded patches; The method of claim 8, further comprising:

10. determining that the patch type for the patch is an inter-patch type; decoding a second reference patch index (dpdu_patch_index) corresponding to the patch; determining the reference index (refIdx) for the patch based on the reference frame index ([refFrmIdx]) corresponding to the patch and the decoded second reference patch index; decoding the patch using a two-dimensional (2D) component corresponding to the patch and a three-dimensional (3D) component corresponding to the patch; reconstructing the stereoscopic representation based on the decoded patches; 10. The method of claim 8 or 9, further comprising:

11. determining that the patch type for the patch is a pulse code modulation (PCM) patch type; determining the reference index (refIdx) for the patch based on a frame index ([frmIdx]) corresponding to the patch and a decremented patch index; decoding the patch using two-dimensional (2D) components for the distinct points corresponding to the patch and three-dimensional (3D) components for the distinct points corresponding to the patch; reconstructing the stereoscopic representation based on the decoded patches; 11. The method of claim 8, further comprising:

12. 12. The method of claim 8, further comprising the step of receiving an input corresponding to the encoded patch information data, the input comprising one or more of the patch_mode, a patch index, the reference index, a frame index, and a reference frame index.

13. The method according to any one of claims 8 to 12, further comprising the step of displaying an image generated based on the reconstructed stereoscopic representation on a display of an electronic device.

14. 1. A method of point cloud coding (PCC) implemented by an encoder, comprising: obtaining, by a receiver of the encoder, a patch frame data unit (pfdu) identifying a patch type (pdfu_patch_mode) for each of a plurality of patches; determining, by a processor of the encoder, whether the patch type for a patch from the plurality of patches is a last patch type; encoding, by the processor of the encoder, patch information data for the patch when the patch type is not the last patch type, the patch information data including the patch type for the patch; encoding, by the processor of the encoder, the patch information data for the patch when the patch type is set to the last patch type, the patch information data including the last patch type for the patch; A method comprising:

15. 15. The method of claim 14, wherein the patch type is one of a skip patch type, an inter patch type, an intra patch type, and a pulse code modulation (PCM) patch type.

16. 16. The method of claim 14 or 15, wherein the patch frame data unit comprises a frame index (frmIdx), a two-dimensional (2D) component corresponding to the patch, and a three-dimensional (3D) component corresponding to the patch.

17. 17. The method of claim 14, wherein the patch information data comprises a frame index, a two-dimensional (2D) component corresponding to the patch, and a three-dimensional (3D) component corresponding to the patch.

18. 18. The method of claim 14, further comprising repeating the steps of determining whether the patch type is the last patch type and encoding the patch information data for a subsequent patch from the plurality of patches until one of the patches from the plurality of patches is determined to have the last patch type.

19. The method of claim 14 , further comprising the step of performing byte alignment after all of the patch information data has been encoded.

20. 20. The method of claim 19, further comprising generating a compressed patch frame data unit after all of the patch information data is encoded.

21. 21. The method of claim 20, further comprising storing the compressed patch frame data unit in a memory of the encoder for transmission to a decoder.

22. 1. A method of point cloud coding (PCC) implemented by an encoder, comprising: obtaining, by a receiver of the encoder, a patch frame data unit (PFDU) for each of a plurality of patches; adding, by a processor of the encoder, a last patch flag to each of the plurality of patches; determining, by the processor of the encoder, whether a patch type for a patch from the plurality of patches is a last patch type based on a value of the last patch flag; encoding, by the processor of the encoder, patch information data for the patch when the patch type is not the last patch type, the patch information data including the patch type and the last patch flag for the patch; A method comprising:

23. 23. The method of claim 22, wherein the patch type is one of a skip patch type, an inter patch type, an intra patch type, and a pulse code modulation (PCM) patch type.

24. 24. The method of claim 22 or 23, wherein the patch frame data unit comprises a frame index (frmIdx), a two-dimensional (2D) component corresponding to the patch, and a three-dimensional (3D) component corresponding to the patch.

25. 25. The method of claim 22, wherein the patch information data comprises a frame index, a two-dimensional (2D) component corresponding to the patch, and a three-dimensional (3D) component corresponding to the patch.

26. 26. The method of claim 22, further comprising repeating the steps of determining whether the patch type is the last patch type based on the value of the last patch flag, and encoding the patch information data for a subsequent patch from the plurality of patches until one of the patches from the plurality of patches is determined to have the last patch type based on the value of the last patch flag.

27. 23. The method of claim 22, further comprising: performing a byte alignment after determining that one of the patches from the plurality of patches has the last patch type based on the value of the last patch flag.

28. 28. The method of claim 27, further comprising generating a compressed patch frame data unit after all of the patch information data is encoded.

29. 30. The method of claim 28, further comprising storing the compressed patch frame data unit in a memory of the encoder for transmission to a decoder.

30. 1. A decoding device, comprising: a receiver configured to receive the encoded patch information; a memory coupled to the receiver, the memory storing instructions; and A processor coupled to the memory, the processor comprising: Obtain a patch corresponding to the encoded patch information data, the patch having a patch type (patch_mode); determining whether the patch type for the patch is a last patch type; When the patch type is the last patch type, the restoration process corresponding to the encoded patch information data is terminated. a processor configured to execute the instructions to cause the decoding device to A decoding device comprising:

31. 1. A decoding device, comprising: a receiver configured to receive the encoded patch information; a memory coupled to the receiver, the memory storing instructions; and A processor coupled to the memory, the processor comprising: Obtain a patch corresponding to the encoded patch information data, the patch having a patch type (patch_mode); determining whether the patch type for the patch is a skip patch type; When the patch type is the skip patch type, decoding a reference patch index (spdu_patch_index) corresponding to the patch; determining a reference index (refIdx) for the patch based on a reference frame index ([refFrmIdx]) corresponding to the patch and the reference patch index decoded when the patch type is the skip patch type; Reconstructing a stereoscopic representation of the patch based on the reference index determined when the patch type is the skip patch type. a processor configured to execute the instructions to cause the decoding device to A decoding device comprising:

32. 1. A decoding device, comprising: a receiver configured to receive the encoded patch information; a memory coupled to the receiver, the memory storing instructions; and A processor coupled to the memory, the processor comprising: Obtain a patch corresponding to the encoded patch information data, the patch having a patch type (patch_mode); determining whether the patch type for the patch is a skip patch type; When the patch type is the skip patch type, decoding a reference patch index (spdu_patch_index) corresponding to the patch; determining a reference index (refIdx) for the patch based on a reference frame index ([refFrmIdx]) corresponding to the patch and the reference patch index decoded when the patch type is the skip patch type; reconstructing a stereoscopic representation of the patch based on the reference index determined when the patch type is the skip patch type; determining whether the more patches available flag has a first value or a second value; storing the reconstructed stereoscopic representation in the memory when the more patches available flag has the first value; When the more patches available flag has the second value, the process of restoring the encoded patch information is terminated. a processor configured to execute the instructions to cause the decoding device to A decoding device comprising:

33. 33. A decoding device according to any one of claims 30 to 32, further comprising a display configured to display an image generated based on the reconstructed stereoscopic representation.

34. 1. A coding device, comprising: a receiver configured to receive a three dimensional (3D) image; a memory coupled to the receiver, the memory containing instructions; and A processor coupled to the memory, the processor comprising: obtaining a patch frame data unit (pfdu) identifying a patch type (pdfu_patch_mode) for each of a plurality of patches; determining whether the patch type for a patch from the plurality of patches is a last patch type; if the patch type is not the last patch type, encoding patch information data for the patch, the patch information data including the patch type for the patch; encoding the patch information data for the patch when the patch type is set to the last patch type, the patch information data including the last patch type for the patch; a processor configured to execute the instructions to cause the encoding device to: An encoding device comprising:

35. 1. A coding device, comprising: a receiver configured to receive a three dimensional (3D) image; a memory coupled to the receiver, the memory containing instructions; and A processor coupled to the memory, the processor comprising: Obtaining a patch frame data unit (PFDU) for each of a plurality of patches; adding a final patch flag to each of said plurality of patches; determining whether a patch type for a patch from the plurality of patches is a last patch type based on a value of the last patch flag; if the patch type is not the last patch type, encoding patch information data for the patch, the patch information data including the patch type and the last patch flag for the patch. a processor configured to execute the instructions to cause the encoding device to: An encoding device comprising:

36. 35. The encoding device of claim 34, further comprising a transmitter coupled to the processor, the transmitter configured to transmit a bitstream having the encoded patch information data toward the decoder.

37. a receiver configured to receive stereoscopic pictures for encoding or a bitstream for decoding; a transmitter coupled to the receiver, the transmitter configured to transmit the bitstream to a decoder or to transmit the decoded stereoscopic image to a reconstruction device configured to reconstruct the decoded stereoscopic picture; a memory coupled to at least one of the receiver or the transmitter, the memory configured to store instructions; and a processor coupled to the memory, the processor configured to execute the instructions stored in the memory to perform the method of any one of claims 1 to 28; A coding device comprising:

38. 38. The coding apparatus of claim 37, further comprising a display configured to display an image based on the reconstructed patches.

39. An encoder; a decoder in communication with the encoder; and wherein the encoder or decoder comprises an encoding device, a decoding device or a coding apparatus according to any one of claims 30 to 38. system.

40. Receiving means configured to receive stereoscopic pictures for encoding or to receive a bitstream for decoding, reconstruction and projection; a transmitting means coupled to said receiving means, said transmitting means being configured to transmit said bitstream to a decoder or to transmit a decoded image to a display means; a storage means coupled to at least one of said receiving means or said transmitting means, said storage means configured to store instructions; a processing means coupled to said storage means, said processing means being configured to execute the instructions stored in said storage means to perform the method of any one of claims 1 to 28; A means for coding comprising:

Images