Method and apparatus for storing and signaling subsample entry descriptions

Abstract

Methods, apparatus, and computer program products for signaling and storing compressed point clouds are described. A subsample entry associated with a subsample sequence within a sample sequence may indicate whether the subsample sequence is encapsulated in a track alone without other subsamples or additional header data. The subsample entry type may be indexed in a track-level subsample description box. Point cloud compression coded bitstream component types may be signaled by including corresponding point cloud unit header information in a codec-specific parameter-related field of a track-level subsample description box. The subsample information box may indicate a subsample entry index for a corresponding subsample. A flag in such an information box may indicate the presence of a subsample description entry index. The description index box may contain a subsample description entry index in the same container as the subsample information box. The track fragment header box may include a subsample description entry index for samples applied to a track fragment.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to and the benefit of U.S. Provisional Patent Application No. 62 / 909,259, filed on October 2, 2019, entitled “Method and Apparatus for Storage and Signaling of Sub-Sample Entry Descriptions in ISOBMFF”, the entire disclosure of which is hereby incorporated by reference in its entirety for all purposes. Technical Field

[0003] Example embodiments generally relate to volumetric video encoding and decoding. Background Art

[0004] Volumetric video data represents a three-dimensional scene or object and can be used as input for augmented reality (AR), virtual reality (VR), and mixed reality (MR) applications. Such data describes the geometry (shape, size, position in three-dimensional (3D) space) and corresponding attributes (e.g., color, opacity, reflectivity, etc.), as well as the temporal changes of the geometry and attributes at a given time instance (such as a frame in a two-dimensional (2D) video). Volumetric video is either generated from a 3D model (e.g., computer-generated imagery (CGI)) or captured from a real-world scene using various capture schemes (e.g., a combination of multiple cameras, laser scanning, video and dedicated depth sensors, etc.). An example representation format for such volumetric data is a point cloud. Temporal information about the scene can be contained in the form of individual capture instances, e.g., "frames" in a two-dimensional (2D) video, or other mechanisms, e.g., the position of an object as a function of time.

[0005] Point clouds are often very suitable for applications such as capturing real-world 3D scenes (where the topology does not have to be a 2D manifold). However, when the reconstructed 3D scene contains tens of millions or even hundreds of millions of points, the temporal compression performance of standard point clouds is poor. Identifying motion-compensated correspondences in 3D space is an ill-defined problem because the geometry and other corresponding attributes can change. For example, temporally consecutive frames do not have to have the same number of point clouds. Therefore, the compression efficiency of dynamic 3D scenes is very low. The number of points, their positions, and their attributes can change frame by frame. This representation requires a large amount of data, which can be expensive in terms of storage and transmission. Volumetric data compression methods based on 2D video (e.g., multi-view and depth) have better compression efficiency, but rarely cover the entire scene. These 2D-based methods only provide limited six-degree-of-freedom (6DOF) capabilities. Therefore, a method with good compression efficiency and high degrees of freedom is needed.

[0006] One approach that has both good compression efficiency and high degrees of freedom is to project a volume model represented by a compressed point cloud onto a 2D plane. This approach allows the use of 2D video coding tools with efficient temporal compression while maintaining a degree of freedom superior to purely 2D-based approaches. The compressed point cloud includes a coded video bitstream corresponding to the texture of the 3D object, a coded video bitstream corresponding to the geometry of the 3D object, and a coded metadata bitstream corresponding to all other auxiliary information necessary during the synthesis of the 3D object. However, there is no well-defined, efficient, and standardized storage and signaling mechanism for these streams. Summary of the invention

[0007] According to example embodiments, a method, apparatus, and computer program product are provided for signaling and storing compressed point clouds in video encoding. The method, apparatus, and computer program product can be used in conjunction with a variety of video formats.

[0008] In various embodiments, a subsample entry is associated with a subsample sequence within a sample sequence. The semantics of a subsample entry may mean that if a subsample sequence is encapsulated in its own track, then no other subsamples or other additional header data (such as a V-PCC unit header) are present in the track, and the sample sequence in the track will conform to the same sample entry as the subsample entry. The subsample entry type may be indexed in a track-level subsample description box, and the subsample information box may be extended to indicate a subsample entry index for each subsample. The subsample information box may be extended to include a version or flag that identifies the presence of a subsample description entry index. A subsample description index box may be formed to include a subsample description entry index in the same container as the subsample information box. A track fragment header box for a movie fragment track may be extended to include a set of subsample description entry indexes that apply to each sample of the track fragment. Alternatively, the sample entry at the track level may be extended to index the subsamples in the sample description box, and the subsample information box may reference the index of the sample entry at the track-level sample description box.

[0009] In some embodiments, a method may include: accessing a point cloud compression coded bitstream, the point cloud compression coded bitstream being configured to be stored as one or more media tracks; forming a track-level subsample description box, the track-level subsample description box including a plurality of subsample entry types for corresponding plurality of subsamples; and indicating one or more types of point cloud compression coded bitstream components by including corresponding point cloud unit header information in a codec-specific parameter-related field of the track-level subsample description box. In some embodiments, the method may further include: extending the subsample information box to indicate the subsample entry index by indicating the order of existence of the subsample entry in the relevant subsample description box, and further indicating its existence via a version value or by setting a flag in the subsample information box. In some embodiments, the at least one media track has a packaging type signaled via the subsample entry index. In some embodiments, storing the point cloud compression encoded bitstream includes: accessing an attribute information bitstream, a geometry information bitstream, an occupancy information bitstream, and a patch information bitstream; detecting that the attribute information bitstream, the geometry information bitstream, the occupancy information bitstream, and the patch information bitstream follow a compatible coding and timing structure; and constructing and storing a video media track based at least on the compatible coding and timing structure. In some embodiments, the point cloud compression encoded bitstream includes at least one attribute information bitstream, at least one geometry information bitstream, at least one occupancy information bitstream, and at least one patch information bitstream. In some embodiments, the method may further include: accessing the attribute information bitstream, the geometry information bitstream, the occupancy information bitstream, and the patch information bitstream; storing the attribute information bitstream in an attribute video media track; storing the geometry information bitstream in a geometry video media track; storing the occupancy information bitstream in an occupied video media track; storing the patch information bitstream in another coded media or metadata track; and constructing and storing metadata in the file, the metadata indicating a mode of one or more components of the attribute video media track, the geometry video media track, the occupied video media track, and the separate coded media or metadata track. In some embodiments, the metadata is interleaving indicator metadata indicating an interleaving mode of the one or more components of the attribute video media track, the geometry video media track, the occupied video media track, and the separate coded media or metadata track. In some embodiments, each of the one or more components for which the mode is indicated can be accessed using a subsample information box (SubSampleInformationBox), the SubSampleInformationBox defining order and configuration information for the one or more components, the SubSampleInformationBox being configured to indicate the mode.In some embodiments, storing includes storing different video-based point cloud compression structures in an International Organization for Standardization Base Media File Format (ISOBMFF).

[0010] In some embodiments, an apparatus is provided, comprising at least one processor and at least one memory including computer program code for one or more programs, the at least one memory and the computer program code being configured to, together with the at least one processor, cause the apparatus to at least: access a point cloud compression coded bitstream, the point cloud compression coded bitstream being configured to be stored as one or more media tracks; form a track-level subsample description box, the track-level subsample description box including a plurality of subsample entry types for corresponding plurality of subsamples; and indicate one or more types of point cloud compression coded bitstream components by including corresponding point cloud unit header information in a codec-specific parameter-related field of the track-level subsample description box. In some embodiments, the computer program code, at least one memory and / or one or more processors may be configured to further cause the apparatus to at least: extend the subsample information box to indicate the subsample entry index by indicating the order of existence of the subsample entry in the relevant subsample description box, and further indicating its existence via a version value or by setting a flag in the subsample information box. In some embodiments, the at least one media track has a packetization type signaled via the subsample entry index. In some embodiments, the at least one memory and the computer program code are further configured to, together with the at least one processor, cause the apparatus to at least: access an attribute information bitstream, a geometry information bitstream, an occupancy information bitstream, and a patch information bitstream; detect that the attribute information bitstream, the geometry information bitstream, the occupancy information bitstream, and the patch information bitstream follow a compatible coding and timing structure; and construct and store a video media track based at least on the compatible coding and timing structure. In some embodiments, the point cloud compression coded bitstream includes at least one attribute information bitstream, at least one geometry information bitstream, at least one occupancy information bitstream, and at least one patch information bitstream. In some embodiments, the at least one memory and the computer program code are further configured to, with the at least one processor, cause the apparatus to at least: access the attribute information bitstream, the geometry information bitstream, the occupancy information bitstream, and the patch information bitstream; store the attribute information bitstream in an attribute video media track; store the geometry information bitstream in a geometry video media track; store the occupancy information bitstream in an occupied video media track; store the patch information bitstream in another coded media or metadata track; and construct and store metadata in a file, the metadata indicating a pattern of one or more components of the attribute video media track, the geometry video media track, the occupied video media track, and a separate coded media or metadata track. In some embodiments, the metadata is interlacing indicator metadata indicating an interlacing pattern of the one or more components of the attribute video media track, the geometry video media track, the occupied video media track, and a separate coded media or metadata track.In some embodiments, each of the one or more components for which a mode is indicated is accessible using a SubSampleInformationBox, the SubSampleInformationBox defining order and configuration information for the one or more components, the SubSampleInformationBox being configured to indicate the mode. In some embodiments, storing includes storing different video-based point cloud compression structures in an International Organization for Standardization Base Media File Format (ISOBMFF).

[0011] In some embodiments, an apparatus is provided, comprising means for performing a method such as described herein, such as at least one processor and at least one memory including computer program code for one or more programs. In some embodiments, the apparatus may include means for accessing a point cloud compression coded bitstream, wherein the point cloud compression coded bitstream is configured to be stored as one or more media tracks; means for forming a track-level subsample description box, wherein the track-level subsample description box includes a plurality of subsample entry types for corresponding plurality of subsamples; and means for indicating one or more types of point cloud compression coded bitstream components by including corresponding point cloud unit header information in a codec-specific parameter-related field of the track-level subsample description box. In some embodiments, the apparatus may further include: means for extending the subsample information box to indicate the subsample entry index by indicating the order of existence of the subsample entry in the relevant subsample description box, and further indicating its existence via a version value or by setting a flag in the subsample information box. In some embodiments, the at least one media track has a packetization type signaled via the subsample entry index. In some embodiments, the apparatus may further include: a component for accessing an attribute information bitstream, a geometry information bitstream, an occupancy information bitstream, and a patch information bitstream; a component for detecting that the attribute information bitstream, the geometry information bitstream, the occupancy information bitstream, and the patch information bitstream follow a compatible coding and timing structure; and a component for constructing and storing a video media track based at least on the compatible coding and timing structure. In some embodiments, the point cloud compression coded bitstream includes at least one attribute information bitstream, at least one geometry information bitstream, at least one occupancy information bitstream, and at least one patch information bitstream. In some embodiments, the apparatus may further include: a component for accessing the attribute information bitstream, the geometry information bitstream, the occupancy information bitstream, and the patch information bitstream; a component for storing the attribute information bitstream in an attribute video media track; a component for storing the geometry information bitstream in a geometry video media track; a component for storing the occupancy information bitstream in an occupied video media track; a component for storing the patch information bitstream in another coded media or metadata track; and a component for constructing and storing metadata in a file, wherein the metadata indicates a pattern of one or more components of the attribute video media track, the geometry video media track, the occupied video media track, and a separate coded media or metadata track. In some embodiments, the metadata is interlacing indicator metadata indicating an interlacing pattern of the one or more components of the attribute video media track, the geometry video media track, the occupied video media track, and a separate coded media or metadata track.In some embodiments, each of the one or more components for which a mode is indicated is accessible using a SubSampleInformationBox, the SubSampleInformationBox defining order and configuration information for the one or more components, the SubSampleInformationBox being configured to indicate the mode. In some embodiments, storing includes storing different video-based point cloud compression structures in an International Organization for Standardization Base Media File Format (ISOBMFF).

[0012] In some embodiments, a computer program product is provided, comprising at least one non-transitory computer-readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions comprising program code instructions configured to perform the following operations when executed: access a point cloud compression coded bitstream, the point cloud compression coded bitstream being configured to be stored as one or more media tracks; form a track-level subsample description box, the track-level subsample description box comprising a plurality of subsample entry types for corresponding plurality of subsamples; and indicate one or more types of point cloud compression coded bitstream components by including corresponding point cloud unit header information in a codec-specific parameter-related field of the track-level subsample description box. In some embodiments, the program code instructions may be further configured to, when executed: extend the subsample information box to indicate the subsample entry index by indicating the order of existence of the subsample entry in the relevant subsample description box, and further indicating its existence via a version value or by setting a flag in the subsample information box. In some embodiments, the at least one media track has a packetization type signaled via the subsample entry index. In some embodiments, the computer executable program code instructions may further include program code instructions configured to perform the following operations when executed: accessing an attribute information bitstream, a geometry information bitstream, an occupancy information bitstream, and a patch information bitstream; detecting that the attribute information bitstream, the geometry information bitstream, the occupancy information bitstream, and the patch information bitstream follow a compatible coding and timing structure; and constructing and storing a video media track based at least on the compatible coding and timing structure. In some embodiments, the point cloud compression coded bitstream includes at least one attribute information bitstream, at least one geometry information bitstream, at least one occupancy information bitstream, and at least one patch information bitstream. In some embodiments, the computer executable program code instructions may further include program code instructions configured to perform the following operations when executed: access the attribute information bitstream, geometry information bitstream, occupancy information bitstream and patch information bitstream; store the attribute information bitstream in an attribute video media track; store the geometry information bitstream in a geometry video media track; store the occupancy information bitstream in an occupied video media track; store the patch information bitstream in another coded media or metadata track; and construct and store metadata in a file, the metadata indicating the mode of one or more components of the attribute video media track, geometry video media track, occupied video media track and separate coded media or metadata track. In some embodiments, the metadata is interlacing indicator metadata, which indicates the interlacing mode of the one or more components of the attribute video media track, geometry video media track, occupied video media track and separate coded media or metadata track.In some embodiments, each of the one or more components for which a mode is indicated is accessible using a SubSampleInformationBox, the SubSampleInformationBox defining order and configuration information for the one or more components, the SubSampleInformationBox being configured to indicate the mode. In some embodiments, storing includes storing different video-based point cloud compression structures in an International Organization for Standardization Base Media File Format (ISOBMFF).

[0013] In some embodiments, a method may include: accessing a point cloud compression encoded bitstream, the point cloud compression encoded bitstream being configured to be stored as one or more media tracks; defining a plurality of subsample description index boxes, the subsample description index boxes comprising a subsample description entry index, the subsample description index boxes being stored in the same container as a corresponding plurality of subsample information boxes associated with the plurality of subsample description index boxes; and adding a unique identifier to a corresponding subsample description index box of the plurality of subsample description index boxes and a corresponding subsample information box of the plurality of subsample information boxes, the unique identifier being operable to identify a first subsample description index box associated with a first subsample information box.

[0014] In some embodiments, a method may include: accessing a point cloud compression encoded bitstream, the point cloud compression encoded bitstream being configured to be stored as one or more media tracks associated with samples of a movie fragment; defining a plurality of subsample description index boxes, the subsample description index boxes comprising subsample description entry indexes, the subsample description index boxes being stored in the same container as a corresponding plurality of subsample information boxes associated with the plurality of subsample description index boxes; and extending a track fragment header box to include a set of the subsample description entry indexes applied to the samples of the movie fragment.

[0015] In some embodiments, a method may include: accessing a point cloud compression encoded bitstream, the point cloud compression encoded bitstream being configured to be stored as one or more media tracks associated with samples of a movie fragment; and indexing multiple sample entities within a sample description box of the one or more media tracks.

[0016] According to other embodiments, methods, apparatus, and / or computer program products are provided for decoding any point cloud video encoded according to any embodiment described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Having thus generally described certain example embodiments of the present disclosure, reference will now be made to the accompanying drawings, which are not necessarily drawn to scale, and in which:

[0018] Figure 1 is a block diagram of an apparatus that may be specifically configured according to an example embodiment of the present disclosure;

[0019] Figure 2A and 2B is an example embodiment according to the present disclosure showing that Figure 1 A graphical representation of a set of operations performed by a device;

[0020] Figure 3 shows an example of a video data storage structure for V-PCC according to an example embodiment of the present disclosure;

[0021] Figure 4 is an example embodiment according to the present disclosure showing that Figure 1 A flowchart of a set of operations performed by a device;

[0022] Figure 5 is an example embodiment according to the present disclosure showing that Figure 1 A flowchart of a set of operations performed by a device;

[0023] Figure 6 is an example embodiment according to the present disclosure showing that Figure 1 A flowchart of a set of operations performed by a device;

[0024] Figure 7 is an example embodiment according to the present disclosure showing that Figure 1 A flowchart of a set of operations performed by a device. DETAILED DESCRIPTION

[0025] Some embodiments will now be described more fully below with reference to the accompanying drawings, in which some but not all embodiments of the invention are shown. In fact, various embodiments of the invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; on the contrary, these embodiments are provided so that the present disclosure will satisfy applicable legal requirements. The same reference numerals always refer to the same elements. As used herein, the terms "data", "content", "information" and similar terms may be used interchangeably to refer to data that can be sent, received and / or stored in accordance with embodiments of the invention. Therefore, the use of any such terms should not be construed as limiting the spirit and scope of embodiments of the invention.

[0026] In addition, as used herein, the term "circuit" refers to: (a) a hardware circuit implementation only (e.g., an implementation of analog and / or digital circuits); (b) a combination of a circuit and a computer program product that includes software and / or firmware instructions stored on one or more computer-readable memories that work together to cause the device to perform one or more functions described herein; and (c) a circuit, such as a microprocessor or a portion of a microprocessor, that requires software or firmware to operate even if the software or firmware is not physically present. This definition of "circuit" applies to all uses of the term herein, including use in any claims. As another example, as used herein, the term "circuit" also covers an implementation that includes one or more processors and / or portions thereof and their accompanying software and / or firmware. As another example, the term "circuit" as used herein also includes, for example, a baseband integrated circuit or an application processor integrated circuit for a mobile phone, or similar integrated circuits in a server, cellular network device, other network device, and / or other computing device.

[0027] As defined herein, "computer-readable storage media" refers to non-transitory physical storage media (eg, volatile or non-volatile storage devices), which may be distinguished from "computer-readable transmission media," which refers to electromagnetic signals.

[0028] According to example embodiments, methods, devices, and computer program products for signaling and storing compressed point clouds in video coding are provided. The methods, devices, and computer program products can be used in conjunction with a variety of video formats, including the High Efficiency Video Coding standard (HEVC or H.265 / HEVC), the Advanced Video Coding standard (AVC or H.264 / AVC), the upcoming Versatile Video Coding standard (VVC or H.266 / VVC), and / or various video and multimedia file formats, including the International Organization for Standardization (ISO) Base Media File Format (ISO / IEC 14496-12, which can be abbreviated as ISOBMFF), the Moving Picture Experts Group (MPEG)-4 file format (ISO / IEC 14496-14, also known as the MP4 format), the file format for NAL (Network Abstraction Layer) unit structured video (ISO / IEC 14496-15), and the Third Generation Partnership Project (3GPP file format) (3GPP Technical Specification 26.244, also known as the 3GP format). ISOBMFF is the basis for deriving all of the above file formats. The example embodiments are described in conjunction with HEVC, however, the present disclosure is not limited to HEVC, but rather the description is given for one possible basis upon which the example embodiments of the present disclosure may be partially or fully implemented.

[0029] Some aspects of the present disclosure relate to container file formats, such as the International Organization for Standardization (ISO) Base Media File Format (ISO / IEC 14496-12, which may be abbreviated as ISOBMFF), the Moving Picture Experts Group (MPEG)-4 File Format (ISO / IEC 14496-14, also known as the MP4 format), the file format for NAL (Network Abstraction Layer) unit structured video (ISO / IEC 14496-15), and the Third Generation Partnership Project (3GPP File Format) (3GPP Technical Specification 26.244, also known as the 3GP format). Example embodiments are described in conjunction with ISOBMFF or its derivatives, however, the present disclosure is not limited to ISOBMFF, but rather is described for one possible basis on which example embodiments of the present disclosure may be partially or fully implemented.

[0030] The elementary unit for the output of an H.264 / Advanced Video Coding (AVC) or HEVC encoder and the input of an H.264 / AVC or HEVC decoder, respectively, is a Network Abstraction Layer (NAL) unit. NAL units may be encapsulated into packets or similar structures for transmission over a packet-oriented network or for storage in a structured file. In the ISO Base Media File Format (ISOBMFF), the NAL units of an access unit form samples, the size of which is provided in the file format metadata. ISOBMFF may, for example, encapsulate V-PCC data. For example, in some embodiments described herein, a data encapsulation and signaling mechanism based on ISOBMFF may be used to encapsulate V-PCC data of MPEG V-PCC encoded content.

[0031] A byte stream format has been specified in H.264 / AVC and HEVC for transmission or storage environments that do not provide a framing structure. The byte stream format separates NAL units from each other by appending a start code in front of each NAL unit. To avoid erroneous detection of NAL unit boundaries, the encoder runs a byte-oriented start code emulation prevention algorithm that adds emulation prevention bytes to the NAL unit payload if a start code appears. In order to enable direct gateway operation between packet-oriented and stream-oriented systems, start code emulation prevention can always be performed regardless of whether the byte stream format is in use. The NAL unit can be defined as a grammatical structure containing an indication of the type of data to be followed and bytes containing data in the form of a raw byte sequence payload (RBSP), where necessary, with emulation prevention bytes dispersed therein. The RBSP can be defined as a grammatical structure containing an integer number of bytes encapsulated in a NAL unit. The RBSP is either empty or in the form of a data bit string containing syntax elements, followed by an RBSP stop bit, and then zero or more subsequent bits equal to 0.

[0032] NAL units can be classified into video coding layer (VCL) NAL units and non-VCL NAL units. VCL NAL units are usually coded slice NAL units.

[0033] A non-VCL NAL unit may be, for example, one of the following types: a sequence parameter set, a picture parameter set, a supplemental enhancement information (SEI) NAL unit, an access unit delimiter, an end of sequence NAL unit, an end of bitstream NAL unit, or a padding data NAL unit. Parameter sets may be required to reconstruct a decoded picture, while many other non-VCL NAL units are not necessary for reconstructing decoded sample values.

[0034] The SEI NAL unit may contain one or more SEI messages that are not necessary for decoding the output image, but may assist related processes such as image output timing, rendering, error detection, error concealment, and resource reservation. Several SEI messages are specified in H.264 / AVC and HEVC, and user data SEI messages enable organizations and companies to specify SEI messages for their own use. H.264 / AVC and HEVC contain syntax and semantics for the specified SEI messages, but do not define the process for processing these messages in the receiver. Therefore, the encoder needs to follow the H.264 / AVC standard or the HEVC standard when creating SEI messages, and decoders that conform to the H.264 / AVC standard or the HEVC standard, respectively, are not required to process SEI messages to obtain output order conformance. One of the reasons for including the syntax and semantics of SEI messages in H.264 / AVC and HEVC is to allow different system specifications to interpret supplementary information in the same way, thereby achieving interoperability. It is expected that the system specification may require the use of specific SEI messages in both the encoding and decoding ends, and in addition, the procedure for handling specific SEI messages in the receiving end may be specified.

[0035] In HEVC, there are two types of SEI NAL units, namely, suffix SEI NAL units and prefix SEI NAL units, which have different nal_unit_type values ​​from each other. The SEI message contained in the suffix SEI NAL unit is associated with the VCL NAL unit that precedes the suffix SEI NAL unit in the decoding order. The SEI message contained in the prefix SEI NAL unit is associated with the VCL NAL unit that follows the prefix SEI NAL unit in the decoding order.

[0036] The video coding specification may contain a set of constraints for associating data units (e.g., NAL units in H.264 / AVC or HEVC) to access units. These constraints can be used to infer access unit boundaries from NAL unit sequences. For example, the following is specified in the HEVC standard:

[0037] - An access unit consists of a coded picture with nuh_layer_id equal to 0, zero or more video coding layer (VCL) NAL units with nuh_layer_id greater than 0, and zero or more non-VCL NAL units.

[0038] - Let firstBlPicNalUnit be the first VCL NAL unit of the coded picture with nuh_layer_id equal to 0. The first of any subsequent NAL units preceding firstBlPicNalUnit and following the last VCL NAL unit preceding firstBlPicNalUnit (if any) specifies the start of a new access unit:

[0039] an access unit delimiter NAL unit with nuh_layer_id equal to 0 (if present),

[0040] the video parameter set (VPS) NAL unit with nuh_layer_id equal to 0 (if present),

[0041] the sequence parameter set (SPS) NAL unit with nuh_layer_id equal to 0 (if present),

[0042] the picture parameter set (PPS) NAL unit with nuh_layer_id equal to 0 (if present),

[0043] o a prefixed Supplemental Enhancement Information (SEI) NAL unit with nuh_layer_id equal to 0 (if present),

[0044] onal_unit_type is in the range of RSV_NVCL41 ... RSV_NVCL44 (which indicates a reserved NAL unit) NAL unit with nuh_layer_id equal to 0 (if present),

[0045] onal_unit_type is in the range of UNSPEC48...UNSPEC55 (which means that the NAL unit is not specified) NAL unit with nuh_layer_id equal to 0 (if present).

[0046] The first NAL unit (if any) following the last VCL NAL unit preceding firstBlPicNalUnit and preceding firstBlPicNalUnit can only be one of the NAL units listed above.

[0047] - firstBlPicNalUnit starts a new access unit when there is no NAL unit (if any) preceding firstBlPicNalUnit and following the last VCL NAL preceding firstBlPicNalUnit.

[0048] Frame packing can be defined as comprising arranging more than one input image (which may be referred to as (input) constituent frames) into an output image. In general, frame packing is not limited to any particular type of constituent frame, or the constituent frames do not need to have a particular relationship to each other. In many cases, frame packing is used to arrange the constituent frames of a stereoscopic video clip into a single image sequence. The arranging may include placing the input images in spatially non-overlapping regions within the output image. For example, in a side-by-side arrangement, two input images are placed horizontally adjacent to each other within the output image. The arranging may also include dividing one or more input images into two or more constituent frame portions and placing these constituent frame portions in spatially non-overlapping regions within the output image. The output image or the sequence of frame-packed output images may be encoded into a bitstream, for example by a video encoder. The bitstream may be decoded, for example, by a video decoder. After decoding, a decoder or a post-processing operation may extract the decoded constituent frames from the decoded image, for example for display.

[0049] The basic building blocks in the ISO base media file format are called boxes. Each box has a header and a payload. The box header indicates the type of box and the size of the box in bytes. A box may contain other boxes, and the ISO file format specifies which box types are allowed in a particular type of box. Furthermore, the presence of some boxes in each file may be mandatory, while the presence of other boxes may be optional. Furthermore, for certain box types, more than one box may be allowed to exist in a file. Thus, the ISO base media file format may be considered to specify a hierarchy of boxes.

[0050] According to the ISO base media file format, a file includes media data and metadata encapsulated in boxes. Each box is identified by a four-character code (4CC) and starts with a header that informs about the type and size of the box.

[0051] Many files formatted according to the ISO base media file format begin with a file type box (also known as a FileTypeBox or ftyp box). The ftyp box contains information that marks the brand of the file. The ftyp box includes a primary type indication and a list of compatible types. The primary type identifies the most appropriate file format specification for parsing the file. The compatible type indicates which file format specifications and / or conformance points the file complies with. A file can comply with multiple specifications. All types indicating compatibility with these specifications should be listed so that readers who only know some compatible types can get an indication of the parsable file. The compatible type also allows a file parser for a specific file format specification to process files containing the same specific file format type in the ftyp box. A file player can check whether the ftyp box of a file includes the types it supports, and can only parse and play the file if any file format specification supported by the file player is listed in the compatible type.

[0052] In a file conforming to the ISO base media file format, media data may be provided in one or more instances of MediaDataBox ('mdat'), and MovieBox ('moov') may be used to include metadata for timed media. In some cases, both 'mdat' and 'moov' boxes may be required to be present in order for the file to be operational. A 'moov' box may include one or more tracks, and each track may reside in a corresponding TrackBox ('trak'). Each track is associated with a handler, identified by a four-character code that specifies the track type. Video, audio, and image sequence tracks may be collectively referred to as media tracks, and they contain elementary media streams. Other track types include hint tracks and timed metadata tracks.

[0053] A track includes samples, such as audio or video frames. For a video track, a media sample may correspond to a coded picture or an access unit. A media track refers to samples (which may also be referred to as media samples) formatted according to a media compression format (and its encapsulation of the ISO base media file format). A hint track refers to hint samples, which contain cookbook instructions for constructing packets for transmission over the indicated communication protocol. A timing metadata track may refer to samples that describe the referenced media and / or hint samples.

[0054] The 'trak' box includes a SampleTableBox (also called a sample table or sample table box) in its box hierarchy. The SampleTableBox contains a SampleDescriptionBox, which provides detailed information about the type of encoding used, as well as any initialization information required for the encoding. The SampleDescriptionBox contains an entry count and as many sample entries as indicated by the entry count. The format of the sample entry is track type specific, but is derived from a generic class (e.g., VisualSampleEntry, AudioSampleEntry). The type of sample entry form used to derive the track type specific sample entry format is determined by the media handler of the track.

[0055] TrackTypeBox can be contained within a TrackBox. The payload of a TrackTypeBox has the same syntax as the payload of a FileTypeBox. The contents of a TrackTypeBox instance can be applied the same way as the contents of a FileTypeBox would if all other tracks of the file were removed and only the track containing this box remained in the file.

[0056] For example, when recording content to an ISO file, movie fragments may be used, for example, to avoid data loss when the recording application crashes, runs out of memory space, or some other event occurs. Without movie fragments, data loss may occur because the file format may require that all metadata (e.g., movie boxes) be written to one contiguous area of ​​the file. In addition, when recording a file, there may not be enough memory space to buffer the movie boxes for the available storage size, and recalculating the contents of the movie boxes when the movie is closed may be too slow. In addition, movie fragments may enable the use of a conventional ISO file parser to record and play the file simultaneously. In addition, progressive download (e.g., when receiving and playing a file while using movie fragments) may require a shorter initial buffering duration and a smaller initial movie box than a file with the same media content but not constructed with movie fragments.

[0057] The movie fragment feature may enable metadata that may otherwise reside in the movie box to be divided into multiple pieces. Each piece may correspond to a track of a certain time period. In other words, the movie fragment feature may enable interleaving file metadata and media data. Thus, the size of the movie box may be limited and the above-mentioned use case is achieved.

[0058] In some examples, the media samples of a movie fragment may reside in an mdat box. However, for metadata of a movie fragment, a moof box may be provided. The moof box may include information of a certain duration of a certain playback time that has previously been in a moov box. The moov box itself may still represent a valid movie, but in addition, it may include an mvex box indicating that the movie fragment will follow in the same file. A movie fragment may extend a presentation that is temporally associated with a moov box.

[0059] Within a movie fragment there may be a set of track fragments, which may include anywhere from zero to multiple track fragments per track. A track fragment may in turn include anywhere from zero to multiple track runs, each track run file being a contiguous run sample of that track (and thus similar to chunks). In these structures, many fields are optional and may be defaulted. The metadata that may be included in a moof box may be limited to a subset of the metadata that may be included in a moov box, and in some cases may be encoded differently. Detailed information on the boxes that may be included in a moof box may be found from the ISOBMFF specification.

[0060] A self-contained movie fragment may be defined as consisting of a moof box and an mdat box that are contiguous in file order, wherein the mdat box contains a sample of a movie fragment (for which the moof box provides metadata) and does not contain a sample of any other movie fragment (i.e., any other moof box). A media fragment may include one or more self-contained movie fragments. A media fragment may be used for delivery, such as streaming, for example, using MPEG Dynamic Adaptive Streaming over Hypertext Transfer Protocol (HTTP) (MPEG-DASH).

[0061] A track reference mechanism can be used to associate tracks with each other. TrackReferenceBox includes boxes, each of which provides a reference from a containing track to a set of other tracks. These references are marked by the box type of the contained box (e.g., the four-character code of the box). The ISO base media file format contains three mechanisms for timed metadata that can be associated with a particular sample: sample groups, timed metadata tracks, and sample auxiliary information. Derived specifications can provide similar functionality through one or more of these three mechanisms.

[0062] A TrackGroupBox contained within a TrackBox enables indicating track groups, where each group shares certain characteristics or the tracks within the group have certain relationships. The box contains zero or more boxes, and the certain characteristics or relationships are indicated by the box type of the contained boxes. The contained boxes include identifiers, which can be used to identify tracks belonging to the same track group. Tracks that contain contained boxes of the same type in a TrackGroupBox and have the same identifier value within these contained boxes belong to the same track group. The syntax of the contained boxes can be defined by TrackGroupTypeBox as follows:

[0063] aligned(8)class TrackGroupTypeBox(unsigned int(32)track_group_type)extends FullBox(track_group_type,version=0,flags=0)

[0064] {

[0065] unsigned int(32)track_group_id;

[0066] / / the remaining data may be specified

[0067] / / for a particular track_group_type

[0068] }

[0069] The ISO Base Media File Format (ISOBMFF) contains three mechanisms for timed metadata that can be associated with a particular sample: sample groups, timed metadata tracks, and sample auxiliary information. Derived specifications may provide similar functionality through one or more of these three mechanisms.

[0070] Sample grouping in the ISO base media file format and its derivatives, such as the AVC file format and the Scalable Video Coding (SVC) file format, can be defined as assigning each sample in a track as a member of a sample group based on a grouping criterion. Sample groups in a sample grouping are not limited to consecutive samples, but can also contain non-adjacent samples. Since there can be more than one sample grouping for a sample in a track, each sample grouping can have a type field indicating the type of grouping. Sample grouping can be represented by two linked data structures: (1) SampleToGroupBox (sbgp box) represents the assignment of samples to sample groups; and (2) SampleGroupDescriptionBox (sgpd box) contains a sample group entry for each sample group, which describes the characteristics of the group. Based on different grouping criteria, there can be multiple instances of SampleToGroupBox and SampleGroupDescriptionBox. These instances can be distinguished by the type field indicating the type of grouping. SampleToGroupBox may include a grouping_type_parameter field, which can be used, for example, to indicate a subtype of the grouping.

[0071] Per-sample auxiliary information can be stored anywhere in the same file as the sample data itself. For self-contained media files, this is typically in a MediaDataBox or a box derived from the specification. It is stored in (a) chunks, where the number of samples per chunk and the number of chunks matches the chunking of the original sample data, or (b) a single chunk for all samples in a movie sample table (or movie fragment). Sample auxiliary information for all samples contained in a single chunk (or track run) is stored contiguously (similar to the sample data).

[0072] When present, sample auxiliary information is stored in the same file as the sample it is associated with, since they share the same data reference ('dref') structure. However, this data may be located anywhere in this file, using the auxiliary information offset ('saio') to indicate the location of this data.

[0073] Restricted video ('resv') sample entries and mechanisms have been specified for ISOBMFF to handle situations where the file author requires certain actions on the player or renderer after decoding a visual track. A player that does not recognize or cannot handle the required action will be stopped from decoding or rendering the restricted video track. The 'resv' sample entry mechanism applies to any type of video codec. The RestrictedSchemeInfoBox is present in the sample entry of a 'resv' track and includes an OriginalFormatBox, a SchemeTypeBox, and a SchemeInformationBox. Unless the 'resv' sample entry type is used, the original sample entry type will be contained in the OriginalFormatBox. The SchemeTypeBox provides an indication of what type of processing the player requires to process the video. The SchemeInformationBox includes further information on the required processing. The scheme type may place requirements on the contents of the SchemeInformationBox. For example, a stereoscopic video scheme indicated in the SchemeTypeBox indicates when the decoded frame contains representations of two spatially packed constituent frames forming a stereo pair (frame packing) or only one view of a stereo pair (left and right views in different tracks). A StereoVideoBox may be contained within a SchemeInformationBox to provide further information, for example, regarding which type of frame packing arrangement has been used (eg, side-by-side or top-bottom).

[0074] In some embodiments, the SchemeInformationBox may include a V3CUnitHeaderBox. In some embodiments, the SchemeInformationBox may have a box type of Multimap Video Box ('mmvi') with a quantity of 0 or 1. In some embodiments, 'mmvi' may be used to indicate that a decoded frame contains two or more temporally interlaced frames representing maps. In some embodiments, in the case of a single track encapsulation of V3C data, the 'mmvi' box may not be present. In some embodiments, when 'mmvi' or MultiMapVideoBox exists, it may indicate the use of a time-interlaced map packing arrangement. In some embodiments, the file parser may implicitly set the synthesis time of map_count_minus1+1 consecutive samples to be equal to the synthesis time of the first sample of the sample group in the interlaced map packing arrangement. In some embodiments, when a time-interleaved map packing arrangement is used, each sync sample indicated by a stream access point sample group and each SAP sample with SAP_type in the range of 1 to 3 (including endpoints) can represent an iterative arrangement of samples, represented by map 0 to map_count_minus1 in synthetic time order, until but not including the next sync sample indicated by the stream access point sample group or the next SAP sample with SAP_type in the range of 1 to 3 (including endpoints).

[0075] In some embodiments, the syntax for the 'mmvi' box may be, for example:

[0076]

[0077] In some embodiments, map_count_minus1 plus 1 may indicate the number of maps present in the track as consecutive samples.

[0078] Several types of Stream Access Points (SAPs) have been specified. SAP Type 1 corresponds to what is known in some coding schemes as a "closed group of pictures (GOP) random access point" (where all pictures in decoding order can be correctly decoded resulting in a continuous time sequence of correctly decoded pictures without gaps), and in addition, the first picture in the decoding order is also the first picture in the presentation order. SAP Type 2 corresponds to what is known in some coding schemes as a "closed GOP random access point" (where all pictures in decoding order can be correctly decoded resulting in a continuous time sequence of correctly decoded pictures without gaps), where the first picture in the decoding order may not be the first picture in the presentation order. SAP Type 3 corresponds to what is known in some coding schemes as an "open GOP random access point", where some pictures in decoding order may not be correctly decoded and have less presentation time than the intra-coded pictures associated with the SAP.

[0079] A stream access point (SAP) sample group as specified in ISOBMFF identifies a sample as being of the indicated SAP type.

[0080] Sync samples can be defined as samples corresponding to SAP Type 1 or 2. Sync samples can be considered as media samples that start a new independent sequence of samples; if decoding starts at a sync sample, then both that sync sample and subsequent samples in decoding order can be correctly decoded, and the resulting set of decoded samples forms a correct representation of the media starting with the decoded sample with the earliest composition time. Sync samples can be indicated with a SyncSampleBox (for those samples whose metadata is present in a TrackBox), or in sample flags indicated or implied for a track fragment run.

[0081] Files conforming to ISOBMFF may contain any non-timed object, referred to as an item, meta item, or metadata item, in a meta box (fourCC:'meta', also referred to as MetaBox). Although the name of the meta box refers to metadata, an item may contain metadata or media data in general. The meta box may reside at the top level of a file, within a movie box (fourCC:'moov'), and within a track box (fourCC:'trak'), but at most one meta box may appear at each file level, movie level, or track level. The meta box may need to contain an 'hdlr' box to indicate the structure or format of the contents of the 'meta' box. The meta box may list and characterize any number of items that can be referenced, and each item may be associated with a file name and may be uniquely identified with the file by an item identifier (item_id, its integer value). Metadata items may be stored, for example, in an 'idat' box, or an 'mdat' box of a meta box, or reside in a separate file. If the metadata is located outside the file, its location may be declared by a DataInformationBox(fourCC:'dinf'). In the specific case where metadata is formatted using Extensible Markup Language (XML) syntax and needs to be stored directly in a MetaBox, the metadata can be encapsulated into an XMLBox(fourCC:'xml') or a BinaryXMLBox(fourcc:'bxml'). Items can be stored as contiguous byte ranges, or can be stored in several extents, each of which is a contiguous byte range. In other words, items can be stored segmented into extents, for example to enable interleaving. An extent is a contiguous subset of the bytes of a resource, and a resource can be formed by concatenating multiple extents.

[0082] High Efficiency Image File Format (HEIF) is a standard developed by the Moving Picture Experts Group (MPEG) for storing images and image sequences. Among other things, the standard facilitates file encapsulation of data encoded according to the High Efficiency Video Coding (HEVC) standard. HEIF includes functionality built on top of the ISO Base Media File Format (ISOBMFF) used.

[0083] The ISOBMFF structure and features are largely used in the design of HEIF. The basic design of HEIF consists of storing still images as items and image sequences as tracks.

[0084] In the context of HEIF, the following boxes may be included within the root-level 'meta' box and may be used as described below. In HEIF, the handler value of the Handler box of the 'meta' box is 'pict'. Resources containing coded media data (whether within the same file, or in an external file identified by a uniform resource identifier) ​​are resolved via data information ('dinf') boxes, while item location ('iloc') boxes store the location and size of each item in the referenced file. The item reference ("iref") box uses typed references to record the relationship between items. If one item in a set of items is considered to be most important in some way compared to other items, that item is signaled via the primary item ('pitm') box. In addition to the boxes mentioned here, the 'meta' box can flexibly include other boxes required to describe the item.

[0085] Any number of image items may be included in the same file. Given a collection of images stored using the 'meta' box method, certain relationships between images may be eligible. Examples of such relationships include indicating a cover image for the collection, providing thumbnail images for some or all images in the collection, and associating some or all images in a collection with auxiliary images such as an alpha plane. A cover image in an image collection is indicated using the 'pitm' box. Thumbnails or auxiliary images are linked to the primary image item using an item reference of type 'thmb' or 'auxl', respectively.

[0086] The ItemPropertiesBox enables any item to be associated with an ordered set of item properties. Item properties are small data records. The ItemPropertiesBox consists of two parts: an ItemPropertyContainerBox, which contains an implicitly indexed list of item properties; and one or more ItemPropertyAssociationBoxes, which associate items with item properties. Item properties are formatted as boxes.

[0087] Descriptive item properties may be defined as item properties that describe rather than transform the associated item. Transformative item properties may be defined as item properties that transform a reconstructed representation of the content of an image item.

[0088] Entities can be defined as a collection of tracks or items. Entity groups are groupings of items and can also group tracks. Entity groups can be used instead of item references when there are no explicit dependencies or directed references between the grouped entities. Entities in an entity group share specific characteristics or have specific relationships, as indicated by the grouping type.

[0089] An entity group is a grouping of items and can also group tracks. The entities in an entity group share specific properties or have specific relationships, as indicated by the grouping type.

[0090] Entity groups are indicated in the GroupsListBox. Entity groups specified in the GroupsListBox of the file-level MetaBox refer to track- or file-level items. Entity groups specified in the GroupsListBox of the movie-level MetaBox refer to movie-level items. Entity groups specified in the GroupsListBox of the track-level MetaBox refer to track-level items for that track.

[0091] GroupsListBox contains EntityToGroupBox, each of which specifies an entity group. The syntax of EntityToGroupBox can be specified as follows:

[0092]

[0093] Wherein, when an item with item_ID equal to entity_id exists in the level (file, movie or track) containing the GroupsListBox, entity_id is resolved to the item, or, when a track with track_ID equal to entity_id exists and the GroupsListBox is contained in the file level, entity_id is resolved to the track.

[0094] The per-region packing information may be encoded as metadata in or along the video bitstream, for example, in the RegionWisePackingBox of OMAF and / or in the per-region packing SEI message of HEVC or H.264 / AVC. For example, the packing information may include a per-region mapping from a predefined or indicated source format to a packed frame format, for example, a mapping from a projected image to a packed image, as described above.

[0095] Packing by region can be, for example, packing by rectangular region. Next, example packing by rectangular region metadata is described in the context of associating a region of an omnidirectional projection image (e.g., an equirectangular or cubic projection) with a region of a packed image (e.g., subjected to encoding or decoding): for each region, the metadata defines a rectangle in the projected image, a corresponding rectangle in the packed image, and optional transformations of rotations of 90, 180, or 270 degrees and / or horizontal and / or vertical mirroring. A rectangle can be indicated, for example, by the position of the upper left corner and the lower right corner. Mapping can include resampling. Since the sizes of the corresponding rectangles in the projected and packed images can be different, this mechanism means resampling by region.

[0096] Multipurpose Internet Mail Extensions (MIME) is an extension of the email protocol that enables the sending and receiving of different kinds of data files on the Internet, such as video and audio, images, software, etc. An Internet media type is an identifier used on the Internet to indicate the type of data that a file contains. Such Internet media types may also be referred to as content types. There are several MIME type / subtype combinations that may contain different media formats. Content type information may be included in a MIME header by a sending entity at the start of a media transmission. Therefore, a receiving entity may need to examine the details of such media content to determine whether a particular element may be rendered given a set of available codecs. In particular, when an end system has limited resources, or a connection to an end system has limited bandwidth, it may be helpful to know whether content may be rendered simply from the content type.

[0097] Internet Engineering Task Force (IETF) Request for Comments (RFC) 6381 specifies two parameters, 'codecs' and 'profiles', which are used with various MIME types or type / subtype combinations to allow explicit specification of the codecs used by the media format contained therein, or the profile of the entire container format.

[0098] By marking the content with a specific codec in the codecs MIME parameter that is indicated for rendering the contained media, the receiving system can determine whether the end system supports the codec, and if not, can take appropriate action (such as rejecting the content, sending a notification, transcoding the content to a supported type, obtaining and installing the required codecs, further checking to determine whether it will adequately support a subset of the indicated codecs, etc.).

[0099] For file formats derived from ISOBMFF, the codec parameters specified in RFC 6381 are derived from the sample entry types of the tracks of the file described by the MIME type.

[0100] Similarly, the Profile parameter specified in RFC 6381 can provide a receiver with an overall indication of the specifications to which the content conforms. This is an indication of the compatibility of the container format and its contents with certain specifications. A receiver may be able to deduce the extent to which it can process and render the content by checking which of the declared profiles it supports and what they mean. For file formats derived from ISOBMFF, the Profile MIME parameter contains a list of four-character codes that are indicated as compatible types in the FileTypeBox of the file described by the MIME type.

[0101] Regardless of the file format of the video bitstream, the apparatus of the example embodiments may be provided by any of a variety of computing devices, including, for example, a video encoder, a video decoder, a computer workstation, a server, etc., or by any of a variety of mobile computing devices, such as a mobile terminal, for example, a smart phone, a tablet computer, a video game player, etc.

[0102] Regardless of the computing device embodying the apparatus, the apparatus 10 of the example embodiment includes the following: Figure 1 The processing circuitry 12, memory 14, communications interface 16, and optional user interface 18 are shown in, or are associated with, or otherwise in communication with, the foregoing.

[0103] The processing circuit 12 can communicate with the memory device 14 via a bus to transfer information between components of the apparatus 10. The memory device can be non-temporary and can include, for example, one or more volatile and / or non-volatile memories. In other words, for example, the memory device can be an electronic storage device (e.g., a computer-readable storage medium) that includes a gate configured to store data (e.g., bits) that can be retrieved by a machine (e.g., a computing device such as a processing circuit). The memory device can be configured to store information, data, content, applications, instructions, etc., so that the apparatus can perform various functions according to exemplary embodiments of the present disclosure. For example, the memory device can be configured to buffer input data for processing by the processing circuit. Additionally or alternatively, the memory device can be configured to store instructions executed by the processing circuit.

[0104] In some embodiments, the device 10 may be embodied in various computing devices as described above. However, in some embodiments, the device may be embodied as a chip or a chipset. In other words, the device may include one or more physical packages (e.g., chips) that include materials, components, and / or wires on a structural accessory (e.g., a substrate). The structural accessory may provide physical strength, size protection, and / or limitations on electrical interactions for the component circuits included thereon. Therefore, in some cases, the device may be configured to implement embodiments of the present disclosure on a single chip, or to implement embodiments of the present disclosure as a single "system on a chip." Thus, in some cases, a chip or chipset may constitute a component for performing one or more operations for providing the functionality described herein.

[0105] The processing circuit 12 can be embodied in a number of different ways. For example, the processing circuit can be implemented as one or more various hardware processing components, such as a coprocessor, a microprocessor, a controller, a digital signal processor (DSP), a processing unit with or without an accompanying DSP, or various other circuits, including integrated circuits, for example, ASICs (application-specific integrated circuits), FPGAs (field programmable gate arrays), microcontroller units (MCUs), hardware accelerators, special-purpose computer chips, etc. Thus, in some embodiments, the processing circuit may include one or more processing cores configured to execute independently. Multi-core processing circuits can implement multi-processing within a single physical package. Additionally or alternatively, the processing circuit may include one or more processors configured in cooperation via a bus to implement independent instruction execution, pipelining, and / or multi-threading.

[0106] In an exemplary embodiment, the processing circuit 12 may be configured to execute instructions stored in the memory device 14 or otherwise accessible to the processing circuit. Alternatively or additionally, the processing circuit may be configured to perform hard-coded functions. Thus, whether configured by hardware or software methods or by a combination thereof, when configured accordingly, the processing circuit may represent an entity (e.g., physically embodied in a circuit) capable of performing operations according to an embodiment of the present disclosure. Thus, for example, when the processing circuit is embodied as an ASIC, FPGA, etc., the processing circuit may be hardware specifically configured to perform the operations described herein. Alternatively, as another example, when the processing circuit is embodied as an executor of instructions, these instructions may specifically configure the processor to perform the algorithms and / or operations described herein when executing these instructions. However, in some cases, the processing circuit may be a processor of a specific device (e.g., an image or video processing system) that is configured to use embodiments of the present invention through instructions for performing the algorithms and / or operations described herein through further configuration of the processing circuit. The processing circuitry may include, among other things, a clock, an arithmetic logic unit (ALU), and logic gates configured to support operation of the processing circuitry.

[0107] The communication interface 16 can be any component configured to receive and / or send data (including video bitstreams), such as a device or circuit embodied in hardware or a combination of hardware and software. In this regard, the communication interface may, for example, include an antenna (or multiple antennas) and supporting hardware and / or software for supporting communication with a wireless communication network. Additionally or alternatively, the communication interface may include a circuit for interacting with the antenna to transmit signals via the antenna or process signals received via the antenna. In some environments, the communication interface may alternatively or also support wired communication. Thus, for example, the communication interface may include a communication modem and / or other hardware / software for supporting communication via a cable, a digital subscriber line (DSL), a universal serial bus (USB) or other mechanisms.

[0108] In some embodiments, such as when the device 10 is configured to encode a video bitstream, the device 10 may optionally include a user interface 18, which in turn may communicate with the processing circuit 12 to provide output to the user, such as by outputting the encoded video bitstream, and in some embodiments, receiving an indication of user input. As such, the user interface may include a display, and in some embodiments, may also include a keyboard, a mouse, a joystick, a touch screen, a touch area, a soft key, a microphone, a speaker, or other input / output mechanism. Alternatively or additionally, the processing circuit may include a user interface circuit configured to control at least some functions of one or more user interface units, such as a display, and in some embodiments, a speaker, a ringer, and / or a microphone, etc. The processing circuit and / or the user interface circuit including the processing circuit may be configured to control one or more functions of one or more user interface units by computer program instructions (e.g., software and / or firmware) stored on a memory accessible to the processing circuit (e.g., a memory device 14, etc.).

[0109] When describing HEVC and in certain example embodiments, common notation for arithmetic operators, logical operators, relational operators, bitwise operators, assignment operators, and range symbols may be used, for example, as specified in HEVC. In addition, common mathematical functions, for example, as specified in HEVC, may be used, and a common order or precedence and order of execution (left to right or right to left) of operators, for example, as specified in HEVC, may be used.

[0110] When describing an example embodiment using HEVC, only the relevant parts of the syntax structure are included in the description of syntax and semantics. However, other embodiments may be similarly implemented with different orders and names of syntax elements.

[0111] When describing the example embodiments, the term file is sometimes used as a synonym of a grammatical structure or an instance of a grammatical structure. In other contexts, the term file may be used to denote a computer file, ie, a resource forming an independent unit in storage.

[0112] When describing HEVC and in certain example embodiments, the grammatical structure may be specified as described herein. A group of statements enclosed in curly braces is a compound statement and is functionally treated as a single statement. The "while" structure specifies a test of whether a condition is true, and if true, specifies repeated evaluation of the statement (or compound statement) until the condition is no longer true. The "do...while" structure specifies that a statement is evaluated once, followed by a test of whether the condition is true, and if true, specifies repeated evaluation of the statement until the condition is no longer true. The "if...else" structure specifies a test of whether a condition is true, and if the condition is true, specifies an evaluation of the main statement, otherwise, specifies an evaluation of an alternative statement. If no alternative statement evaluation is required, the "else" portion of the structure and the associated alternative statements are omitted. The "for" structure specifies an evaluation of the initial statement, followed by a test of the condition, and if the condition is true, specifies repeated evaluation of the main statement, followed by subsequent statements until the condition is no longer true.

[0113] Volumetric video data represents a three-dimensional scene or object and can be used as input for AR, VR, and MR applications. This data describes the geometry (shape, size, and position in 3D space) and corresponding attributes (e.g., color, opacity, reflectivity, etc.), as well as any possible temporal changes in the geometry and attributes at a given time instance (such as a frame in a 2D video). Stereoscopic video is either generated from a 3D model (i.e., CGI) or captured from a real-world scene using a variety of capture solutions (e.g., a combination of multiple cameras, laser scanning, video, and dedicated depth sensors, etc.). In addition, a combination of CGI and real-world data is also possible. Typical representation formats for this volumetric data are triangular meshes, point clouds, or voxels. Temporal information about the scene can be included in the form of individual capture instances (i.e., "frames" in 2D video), or other mechanisms can be used, such as the position of an object as a function of time.

[0114] Point clouds are a form of volumetric content that is known to compress poorly. MPEG has been working on video-based point cloud compression, where point clouds are projected onto a 2D surface and compressed using traditional video coding tools. With associated metadata, 3D point clouds can be reconstructed from 2D videos.

[0115] The compressed point cloud includes one or more coded video bitstreams corresponding to the properties of the 3D object (such as spatial properties, texture, color, reflectivity, surface normals), one or more coded video bitstreams corresponding to the geometry of the 3D object, a coded video bitstream describing the 2D surface occupancy, and a coded metadata bitstream corresponding to all other auxiliary information necessary during the synthesis of the 3D object. However, there is no well-defined, efficient and standardized storage and signaling mechanism for these streams.

[0116] Because volumetric video represents a 3D scene (or object), such data can be viewed from any viewpoint. Therefore, volumetric video is an important format for any AR, VR or MR application, especially for providing 6DOF viewing capabilities.

[0117] The increase in computing resources and the advancement of 3D data acquisition equipment have enabled the reconstruction of highly detailed volumetric video representations of natural scenes. Infrared, laser, time of flight and structured light are all examples of devices that can be used to construct 3D video data. The representation of 3D data depends on how the 3D data is used. Dense voxel arrays have been used to represent volumetric medical data. In 3D graphics, polygonal meshes are widely used. On the other hand, point clouds are very suitable for capturing applications such as real-world 3D scenes where the topology is not necessarily a 2D manifold. Another method of representing 3D data is to encode the 3D data into a set of textures and depth maps, as in the case of multi-view plus depth. Closely related to the technology used in multi-view plus depth is the use of elevation maps and multi-level surface maps.

[0118] Reconstructing a 3D scene can contain tens or even hundreds of millions of points in a dense point cloud or voxel array. If such representations need to be stored or exchanged between computing entities, efficient compression becomes critical. Standard volumetric video representation formats (such as point clouds, meshes, and voxels) suffer from poor temporal compression performance. Identifying motion-compensated correspondences in 3D space is an ill-defined problem because both the geometry and the corresponding attributes can change. For example, temporally consecutive "frames" may not have the same number of meshes, points, or voxels. As a result, compression efficiency for dynamic 3D scenes is very low. 2D video-based methods for compressing volumetric data (e.g., multi-view and depth) have better compression efficiency, but rarely cover the entire scene. Therefore, they only provide limited 6DOF capabilities.

[0119] Due to the limitations of the above two methods, a third method has been developed. A 3D scene represented as a grid, point and / or voxel can be projected onto one or more geometric structures. These geometric structures are "unfolded" onto 2D planes (two planes per geometric structure: one for texture and one for depth), which are then encoded using standard 2D video compression techniques. The relevant projected geometry information is sent to the decoder along with the encoded video file. The decoder decodes the video and performs inverse projection to regenerate the 3D scene in any desired representation format (not necessarily an encoding format).

[0120] Projecting the volume model onto a 2D plane allows the use of 2D video coding tools with efficient time compression. Therefore, the coding efficiency is greatly improved. Using geometric projections instead of 2D video-based methods (e.g., multi-view and depth) provides better scene (and / or object) coverage. Therefore, 6DOF capabilities are improved. Using several geometric structures for individual objects can further improve the coverage of the scene. In addition, standard video coding hardware can be used for real-time compression / decompression of the projection plane. The complexity of the projection and back-projection steps is relatively low.

[0121] Figure 2A and 2B An overview of the compression and decompression process of a point cloud is provided, respectively. As shown in block 202, such as Figure 1 An apparatus such as the apparatus 10 of the invention includes components, such as processing circuit 12, for patch generation after receiving an input point cloud frame. The patch generation process aims to decompose the point cloud into a minimum number of patches with smooth boundaries while minimizing reconstruction errors. An example way of patch generation is specified in the MPEG document Point Cloud Coding Test Model Category 2 Version 0.0 (TMC2V0). The method is described as follows.

[0122] First, the normal at each point is estimated. Then an initial clustering of the point cloud is obtained by associating each point with one of the following six oriented planes, defined by its normal:

[0123] -(1.0,0.0,0.0),

[0124] -(0.0,1.0,0.0),

[0125] -(0.0,0.0,1.0),

[0126] -(-1.0,0.0,0.0),

[0127] -(0.0,-1.0,0.0), and

[0128] -(0.0,0.0,-1.0).

[0129] Each point is associated with the plane with the closest normal (e.g., maximizing the dot product of the point normal and the plane normal). In turn, the initial clusters are refined by iteratively updating the cluster index associated with each point based on the point's normal and the cluster index of the point's closest neighbors. The final step involves extracting patches by applying a connected component extraction process.

[0130] As shown in block 204, the apparatus includes means for packing, such as processing circuitry 12. The packing process aims to map the extracted patches onto a 2D grid while trying to minimize unused space and guaranteeing that each TxT (e.g., 16x16) block of the grid is associated with a unique patch. T is a user-defined parameter that is encoded in the bitstream and sent to the decoder. An example packing strategy is specified in TMC2v0.

[0131] The packing strategy specified in TMC2v0 is a simple one that iteratively tries to insert patches into a WxH grid. W and H are user-defined parameters that correspond to the resolution of the geometry / texture image to be encoded. The patch positions are determined by an exhaustive search performed in raster scan order. The first position that can guarantee a non-overlapping insertion of the patch is chosen, and the grid cells covered by the patch are marked as used. If there is no empty space in the current resolution image that can accommodate a patch, the height H of the grid is temporarily doubled and the search is repeated. At the end of this process, H is clipped to fit the used grid cells.

[0132] As shown in blocks 206A and 206B, the apparatus includes components for image generation, such as processing circuitry 12. The image generation process uses the 3D to 2D mapping computed during the packing process to store the geometry and texture of the point cloud as an image. To better handle the situation where multiple points are projected onto the same pixel, each patch is projected onto two images (referred to as layers). More precisely, let H(u,v) be a set of points of the current patch projected onto the same pixel (u,v). The first layer, also referred to as the near layer, stores the points with the lowest depth D0 in H(u,v). The second layer, referred to as the far layer, captures the points with the highest depth (within the interval [D0,D0+Δ]) in H(u,v), where Δ is a user-defined parameter describing the thickness of the surface. In some embodiments, the generated video may have a WxH YUV420-8bit geometry video and a WxH YUV420-8bit texture. YUV is a color encoding system in which Y represents the luminance component and UV represents the two chrominance components.

[0133] As shown in block 208, the apparatus includes components such as processing circuit 12 for image filling. The apparatus is configured to fill geometric and texture images. The filling process is intended to fill the empty spaces between patches to generate a piecewise smooth image suitable for video compression. An example filling method is provided in TMC2v0. The filling strategy in TMC2v0 is defined as follows:

[0134] • Each block of TxT (eg, 16x16) pixels is processed independently.

[0135] If a block is empty (ie, all its pixels belong to empty space), fill the block with pixels by copying the last row or column of the previous TxT block in raster order.

[0136] If the block is full (ie, no empty pixels), do nothing.

[0137] • If a block has both empty and filled pixels, the empty pixels are iteratively filled with the average of their non-empty neighbors.

[0138] As shown in block 210, the apparatus further includes components such as processing circuit 12 for video compression. The apparatus is configured to compress the filled texture image and the filled geometry image into a geometry video and a texture video. The generated images / layers are stored as video frames and compressed using a video codec. An example video codec used is a HEVC Test Model 16 (HM16) video codec configured according to the HM16 provided as a parameter. As shown in block 212, the apparatus includes components such as processing circuit 12 for assisting patch information compression. The following metadata may be encoded for each patch:

[0139] Index of the projection plane

[0140] ο Index 0 for the planes (1.0, 0.0, 0.0) and (-1.0, 0.0, 0.0)

[0141] ο Index 1 for the planes (0.0,1.0,0.0) and (0.0,-1.0,0.0)

[0142] ο Index 2 for the planes (0.0, 0.0, 1.0) and (0.0, 0.0, -1.0)

[0143] 2D bounding box (u0,v0,u1,v1)

[0144] The 3D position (x0, y0, z0) of the patch in terms of depth δ0, tangential offset s0 and bi-tangential offset r0. Depending on the chosen projection plane, (δ0, s0, r0) is calculated as follows:

[0145] ο Index 0, δ0 = x0, s0 = z0, r0 = y0

[0146] ο Index 1, δ0=y0, s0=z0, r0=x0

[0147] ο Index 2, δ0=z0, s0=x0, r0=y0

[0148] The mapping information providing for each TxT block its associated patch index may be encoded as follows:

[0149] For each TxT block, let L be an ordered list of indices of patches such that their 2D bounding boxes contain the block. The order in this list is the same as the order used to encode the 2D bounding boxes. L is called the candidate patch list.

[0150] • The empty space between patches is considered a patch and is assigned a special index of 0, which is added to the candidate patch list for all blocks.

[0151] Let I be the index of the patch to which the current TxT block belongs, and let J be the position of I in L. Instead of explicitly encoding the index I, its position J is arithmetically encoded, which leads to better compression efficiency.

[0152] As shown in block 214, the apparatus may further include means, such as processing circuitry 12, for compressing an occupancy map. As shown in block 216, the apparatus further includes means, such as processing circuitry 12, for multiplexing compressed geometry video, compressed texture video, compressed auxiliary patch information, and compressed occupancy map into a point cloud compression (PCC) coded bitstream. The compressed point cloud includes: a coded video bitstream corresponding to the texture of the 3D object; a coded video bitstream corresponding to the geometry of the 3D object; and a coded metadata bitstream corresponding to all other auxiliary information necessary during synthesis of the 3D object. These three information streams are multiplexed together and then used together to decode the 3D object point cloud data at a given time instance. Figure 3 Graphic examples of the information are provided.

[0153] The multiplexed representation is at the bitstream level and requires implementation of features such as: 1) random access in time and computation of exact byte positions of such random access; 2) compact representation of metadata when it is repeated for groups of frames; 3) file storage mechanisms and delivery via mechanisms such as ISOBMFF and MPEG-DASH; and 4) alternative representations of video and auxiliary data streams for delivery channel adaptation (quality, resolution, frame rate, bit rate, etc.).

[0154] In an embodiment related to ISOBMFF, a brand may be defined to indicate that a file contains a PCC encoded bitstream. Possible four-character codes for the type may be or include, but are not limited to, 'pccf'. The type may be contained in a FileTypeBox and / or may be used in signaling, such as in the value of an optional profile MIME parameter. The type may be present in a major, minor, or compatible type list. In an embodiment, if a track-level type storage mechanism exists, the type may be present at the track level, for example, in a TrackTypeBox. In an embodiment, if a group-level type storage mechanism exists, the type may be present at the group level, for example, in an EntityToGroupBox that groups texture, geometry, and PCC metadata tracks. A new media handler type for the PCC metadata track may be defined as 'pcch' and stored in the handler_type field of the HandlerBox('hdlr') in the MediaBox of the PCC metadata track. The structure as defined in ISOBMFF may be as follows:

[0155]

[0156] Among them, handler_type defines the media processor type of the PCC metadata track. This field may contain the 4-character code "pcch". In the above-mentioned syntax structure and other example syntax structures, syntax elements may be specified to have predefined values ​​that the file author must comply with. For example, in the above-mentioned syntax structure, the file writer is required to set the value of the syntax element pre_defined to 0. Similarly, the variables or input parameters of the function (such as the version in the above-mentioned syntax) may have predefined values. For example, in the above-mentioned syntax, the value of the version parameter must be equal to 0. However, the specification of such predefined values ​​may not be necessary for other embodiments, and some embodiments may be implemented similarly with other predefined values ​​or in the absence of syntax elements, variables or input parameters assigned to predefined values. In one embodiment, PCC global bitstream information (such as but not limited to global scaling, translation and rotation) may be stored in a separate box / complete box. This box may be called PCCInfoBox, with a 4-character code "pcci". Examples are provided as follows:

[0157]

[0158] global_scaling and global_translation are defined as independent values ​​that match each global axis x, y, and z. global_rotation is expressed as a quaternion (w, x, y, z). The benefit of using quaternions instead of Euler angles is that quaternions are more meaningful because they do not depend on the order of Euler rotations. In addition, quaternions are not susceptible to "gimbal locking". Using global_scaling, global_translation, and global_rotation, a transformation matrix can be generated to arbitrarily transform point cloud data in space. The benefit of providing these values ​​separately instead of the overall transformation matrix is ​​that it is easy for developers and certain components of the transformation can be ignored. number_of_layers indicates the total number of layers.

[0159] The syntax element PCCIInfoBox can be present in the Handler Box of the PCC metadata track or in the Media Header Box of the PCC metadata track. These boxes can be extended to new boxes, or extended with new complete box versions, or extended with flag definitions. To include PCCIInfoBox, an example implementation of this new data structure can be as follows:

[0160]

[0161] or

[0162]

[0163] In the first syntax structure above, the presence of PCCInfoBox is conditional on handler_type being "pcch". In the second syntax structure above, the presence of PCCInfoBox is conditional on the version field of the box header being equal to 1. In the third syntax structure above, the presence of PCCInfoBox is conditional on the flag field of the box header having a special bit equal to 1.

[0164] In the above syntax structure and other example syntax structures, "&" can be defined as a bitwise AND operation. When operating on (signed) integer arguments, the bitwise AND operation operates on the two's complement representation of the integer value. When operating on binary arguments containing fewer bits than another argument, the shorter argument is extended by adding more significant bits equal to 0.

[0165] In another embodiment, PCCInfoBox content can be stored in a PCC metadata sample entry. A new sample entry type "pcse" can be defined. An example can be as follows:

[0166]

[0167] In another embodiment, the PCC bitstream global information can be stored in the file / movie / track level Metabox using the ItemInfoBox. The item containing the PCC global bitstream information can use its own MIME type, such as but not limited to 'pcci' stored in the item_type field of the ItemInfoEntry() box. In addition, new entity groupings can be defined in the EntityToGroupBox() contained in the MetaBox. New entity groupings such as 'pcgr' can be defined. PCC metadata tracks, video and texture media tracks, and related metadata items can be grouped together within the defined 'pcgr' grouping. An example can be represented as follows: texture video track id=1; geometry video track id=2; PCC metadata track id=3; PCC bitstream global information is stored as a metadata item with item_id=101.

[0168] New entity groups can be defined using the following syntax:

[0169]

[0170] grouping_type identifies a new entity grouping type. A new grouping_type four-character code may be defined for entity groupings, such as but not limited to 'pcgr'. group_id represents the identifier of the group in numerical form. num_entities_in_group represents the number of entities in the group, which is set to 4 in this example (texture tracks, geometry video tracks, PCC metadata tracks, and PCC bitstream global information stored as metadata items). The listed entity_id is a list of entity_ids of the entities, which is [1,2,3,101] in this example.

[0171] In one embodiment, the order of these entities may be predefined, such as {texture, geometry, pcc metadata, pcc global information}. The order may be any defined order. In the absence of an order definition, the processor type and item type of the track may be used to identify the type of grouped entity.

[0172] In another embodiment, the EntitytoGroupBox box can be extended to store PCCInfoBox. The syntax can be as follows:

[0173]

[0174] In this case, there is no need to define the entry type 'pcci' in order to store the PCC global information.

[0175] In order to indicate the grouping relationship of PCC tracks, track grouping can be defined. Such track grouping information can be stored in each related media and timed metadata track or a subset of such tracks. Track grouping can be defined using the track grouping mechanism as follows:

[0176]

[0177] A new track_group_type may be defined for PCC track grouping, such as but not limited to 'pctg'. In another embodiment, PCC global information may be embedded in TrackGroupTypeBox as follows:

[0178]

[0179] Media and temporal metadata tracks with the same {track_group_type, track_group_id} pair may belong to the same PCC 3D representation.

[0180] In some embodiments, in addition to or in lieu of the track grouping mechanism and / or entity grouping mechanism described above, a track reference mechanism may be used to indicate PCC track relationships, as shown in block 59. As shown in block 59, the apparatus includes means, such as processing circuitry 12, for associating auxiliary timed metadata tracks with previously defined track groups using track references. A TrackReferenceBox having the following ISOBMFF definition syntax may be used as follows:

[0181]

[0182] The 4-character codes described herein are listed as examples. Any set of listed track references etc. may be present in a file storing a PCC track.

[0183] - PCC texture tracks can reference PCC metadata tracks with reference_type 'pcct'.

[0184] - A PCC geometry track can reference a PCC metadata track or a PCC texture track with reference_type 'pccg'.

[0185] - PCC metadata track can reference PCC texture and geometry tracks with reference_type 'pccm'. In this case, both texture and geometry track ids can be listed in TrackReferenceBox with type 'pccm' for PCC metadata track.

[0186] In some alternative embodiments, texture or geometry tracks may reference multiple metadata tracks if these tracks are represented by the same PCC metadata track but differ in certain characteristics such as bitrate, resolution, video quality, etc. Track alternatives may also be derived using the alternative track mechanism provided by ISOBMFF. For example, the same alternate_group value of TrackHeaderBox may be used to represent tracks of the same data with different characteristics.

[0187] In some alternative embodiments, the PCC metadata track may have two different track references pointing to the texture and geometry video tracks, such as TrackReferenceTypeBox having reference_type 'pcct' to point to the texture video track and TrackReferenceTypeBox having reference_type 'pccg' to point to the geometry video track.

[0188] In another embodiment, the video and geometry tracks may be packed into a single video frame. In this case, the packed video frame stored in the video track may have a TrackReferenceTypeBox with reference_type'pccp' to point to the PCC metadata track, or vice versa (e.g., the PCC metadata track contains a TrackReferenceTypeBox with reference_type'pccp' and points to the PCC packed video track).

[0189] In another embodiment, the texture and geometry tracks may be grouped using a track grouping type (but not limited to) 'pctg' with a specific track_group_id. The PCC's auxiliary metadata may be stored as a timed metadata track with a dedicated processor type (such as but not limited to 'pccm') and referenced by using the 'cdtg' track reference type to the track_group_id. This may indicate that the PCC auxiliary timed metadata describes both texture and geometry tracks.

[0190] In some embodiments, the PCC metadata track may include PCC auxiliary information samples in the following method. In one embodiment, the first sample of each PCC frame group (GOF) may include GOF header information and auxiliary metadata bitstreams corresponding to texture and geometry samples with the same synthesis time. In another embodiment, such samples may be marked as "sync" samples, for example by using a synchronization sample box (also known as SyncSampleBox). In this case, random access can be performed at each GOF boundary. All other PCC metadata samples may contain corresponding PCC auxiliary metadata for the texture of the geometry sample in their corresponding texture and geometry tracks. In another embodiment, each PCC metadata sample may have corresponding sample auxiliary information in a sample table box that can store a GOF header bitstream. The auxiliary sample information may be defined only for the synchronization sample of the PCC metadata track or for each sample of the PCC metadata track. In another embodiment, the GOF header information may be stored in a PCC metadata track sample entry. Furthermore, each GOF may reference a sample entry index containing relevant GOF header bitstream information. Possible data structures may be as follows:

[0191]

[0192] A GOFHeaderBox can have the following syntax:

[0193]

[0194] groupOfFramesSize indicates the number of frames in the current frame group. frameWidth indicates the width of the geometry or texture image. frameHeight indicates the height of the geometry / texture image. occupancyResolution indicates the occupancy map resolution. radiusToSmoothing indicates the radius of neighbors detected for smoothing. neighborCountSmoothing indicates the maximum number of neighbors used for smoothing. radius2BoundaryDetection indicates the radius of boundary point detection. thresholdSmoothing indicates the smoothing threshold.

[0195] In some embodiments, a separate timed metadata track may be defined to store GOF header samples. In this case, such metadata track has a separate processor type (e.g., 'pcgh') and a track reference to the PCC metadata track (e.g., reference type 'pcgr'). If such a track exists and track grouping is used to group texture, geometry, and metadata tracks, such metadata track may also be listed in the track grouping.

[0196] In some embodiments, metadata samples belonging to a GOF may be grouped together to form a sample group, and their associated GOF header information may be stored in a SampleGroupDescriptionEntry in a SampleGroupDescriptionBox. An example syntax is:

[0197]

[0198] In some embodiments, the 'pmsg' sample group, etc., is present in a track containing dynamic PCC metadata. In some embodiments, the 'pmsg' sample group, etc., is present in any video track (eg, a texture video track or a geometry video track).

[0199] In some cases, for example, when encoding and streaming a live feed, it is necessary to include a new GOF header in the bitstream after creating the initial MovieBox. In this case, the new GOF header can be introduced in the SampleGroupDescriptionBox included in the MovieFragmentBox. It should be noted that the introduction of the GOF header in the SampleGroupDescriptionBox included in the MovieFragmentBox can be used for, but not limited to, encoding and streaming of live feeds.

[0200] The PCCMetadata sample may include GOF header information, GOF auxiliary information, and GOF occupancy map information. In some embodiments, the GOF header information, GOF auxiliary information, and GOF occupancy map information are defined as their corresponding parts in the PCC Category 2 coding working draft. The GOF header information is provided above. Example GOF auxiliary information includes the following:

[0201]

[0202]

[0203]

[0204] In some embodiments, to accommodate the situation where only a single video decoder instance is available at a time, only a single video bitstream is fed to the video decoder.

[0205] In some embodiments, a spatial frame packing method may be used to put texture and geometry video frames into a single video frame. A frame packing arrangement SEI message or another SEI message with similar content to the frame packing arrangement SEI message may be used to signal such packing at the video bitstream level. Such packing may be signaled at the file format level using a constrained video scheme.

[0206] An alternative approach could be to perform per-region packing and create a "packed PCC frame". This approach enables different geometric transformations such as rotation, scaling, etc. to be applied to texture and geometry frames. A new type of "per-region packing" restricted scheme type could be defined and signaled in the SchemeInformationBox in ISOBMFF.

[0207] Another alternative can be to use a time interleaving mechanism for texture and geometry frames and ensure that the prediction dependencies of frames only come from frames of the same type (e.g., texture or geometry). In another embodiment, this restriction can be relaxed. Such a coding configuration can be signaled in the SchemeInformationBox of a media track with a new type definition. The sample synthesis time for such an interleaved frame will be the synthesis time of the second frame of the {texture, geometry} pair. In addition, an indicator field can be defined to signal the type of the first video frame.

[0208] In another embodiment, groups of video frames of the same type may be consecutive to each other. In this case, the sample composition time may be signaled on consecutive frames of the same type, and then applied to the same number of frames adjacent to the group of frames in decoding order (e.g., sample timing of N texture frames is applied to the next N geometry frames, then N texture frames, and so on). A new temporal packing type indicator may be defined to signal such an interleaving scheme.

[0209] In some embodiments, the apparatus may include components such as processing circuit 12 for accessing a texture information bitstream, a geometry information bitstream, and an auxiliary metadata bitstream. In some embodiments, the apparatus may include components such as processing circuit 12 for packing the texture information bitstream and the geometry information bitstream into one video media track. In some embodiments, the apparatus may include components such as processing circuit 12 for storing the auxiliary metadata information in a timed metadata track. In some embodiments, the apparatus may include components such as processing circuit 12, memory 14, etc. for associating a video media track and a timed metadata track as a group, and for storing the video media track and the timed metadata track as a group.

[0210] Regarding packing the texture information bitstream and the geometry information bitstream into one video media track, in some embodiments, the texture and geometry video frames in the bitstream can be spatially packed into a single video frame. A new restricted video scheme can be defined for PCC encoding packed texture and geometry video frames. Such a scheme can be identified by the scheme_type 4-character code 'ppvs' in the RestrictedSchemeInfoBox. A new uniform resource identifier (URI) can be defined as scheme_uri and can exist conditionally based on the box flag. Furthermore, the packed PCC video frame information can be included in the SchemeInformationBox of the relevant video track. Such a box can be called PCCPackedVideoBox, and its 4-character code is 'ppvi'. An example of such a box can be as follows:

[0211]

[0212] The following boxes may reside within a Scheme Information Box ('schi') and contain PCC packaged video specific information:

[0213]

[0214] A box called PCCSpatialPackingBox() defines how texture and geometry patches are spatially packed in the video frame, for example:

[0215] aligned(8)class PCCSpatialPackingBox extends FullBox('rwpk',0,0){

[0216] PCCSpatialPackingStruct();

[0217] }

[0218] PCCSpatialPackingStruct() may contain the data structures needed to identify the type, dimensions, and spatial location of each packed region. Such a box may have the following structure:

[0219]

[0220] num_regions defines how many packed regions are defined in this data structure. proj_texture_picture_width defines the width of the input video texture frame. proj_texture_picture_height defines the height of the input video texture frame. proj_geometry_picture_width defines the width of the input geometry texture frame. proj_geometry_picture_height defines the height of the input geometry texture frame. packed_picture_width defines the width of the output packed video frame. packed_picture_height defines the height of the output packed video frame.

[0221] region_type indicates the type of the packed region. region_type=0 may indicate that the source of the region is a texture video frame. region_type=1 indicates that the source of the region is a geometry video frame.

[0222] guard_band_flag indicates whether the region (reason) has a guard band. pack_type indicates the type of per-region packing. RectRegionPacking specifies per-region packing between packing regions and projection regions.

[0223] In another embodiment, a new projection type such as "orthogonal projection" may be added to the ProjectionFormatBox as defined in the MPEGOMAF specification, which resides within the ProjectedOmnivideoBox ('povd') for the projected omnidirectional video ('podv') scheme. In this case, the PCCPackedVideoBox may reside within the ProjectedOmnivideoBox, and its presence may be signaled by having a Projection_type indicating orthogonal projection.

[0224] In some embodiments, texture and geometry video frames are temporally interleaved to form a single video bitstream. In this case, a video prediction scheme can be defined so that texture frames depend only on other texture frames and geometry frames also depend only on other geometry frames. The frame rate can be doubled. The composition time of the latter frame can be used to indicate the composition time of the temporally interleaved frames. In this case, a new box called PCCVideoBox can be defined to indicate the temporal packing information. This box can reside within a SchemeInformationBox when schemeType can be defined as 'pcci' to indicate temporally interleaved PCC texture and geometry frames. An example is:

[0225]

[0226] packing_indication_type indicates the type of temporal or spatial packing. Example packing types and associated values ​​are:

[0227] 0: Spatial packing indicator. This value indicates that a spatial packing scheme is applied and the above spatial packing mechanism is used for texture and geometry PCC video frames.

[0228] 1: Time interleaving, texture frame priority

[0229] 2: Time interleaving, geometry frame priority

[0230] In both temporal interleaving and spatial packing use cases, a track reference mechanism between the auxiliary metadata track and the texture and geometry tracks may be applied. In this case, a single track reference type, which may be referred to as 'pctg', may be used to go from the PCC auxiliary metadata track to / from the packed / interleaved PCC texture and geometry video tracks to the PCC auxiliary metadata track.

[0231] In some embodiments using HEVC, to support this mechanism, some data structures as described herein may be defined.

[0232] A descriptive item property called PCCCPackedFrameProperty may be defined. This property may contain PCCSpatialPackingBox() as defined above. In turn, this property is linked to an image item containing packed texture and geometry image items. An example syntax is:

[0233]

[0234] A descriptive item property called PCCAuxiliaryProperty('pcap') may be defined to include PCC global parameters and GOF header information. An example syntax is:

[0235]

[0236] A new item type called PCC metadata item may be defined to contain PCC auxiliary metadata information, which is referred to as PCC auxiliary metadata sample in the above section. This item may be defined as type 'pcca'.

[0237] If no packing is defined, or if raw PCC encoded geometry and texture images are also stored in the same media file, the following properties may be defined:

[0238] o A descriptive item property called PCCTextureItemProperty ('pcti') may be defined to indicate a PCC coded texture image item. This property may in turn be linked to the associated image item.

[0239] o A descriptive item property called PCCGeometryItemProperty('pcgi') may be defined to indicate a PCC coded geometry image item. This property may in turn be linked to the associated image item.

[0240] o The above described descriptive item property called PCCAuxiliaryProperty('pcap') can be used to store global header and GOF header information.

[0241] o An entity grouping mechanism may be used to group the above PCC metadata items and the two image items with corresponding item texture and geometry item properties together to indicate that they create a PCC encoded 3D representation. Such an entity grouping may have a 'pcce' type in EntitytoGroupBox(), where all relevant item_ids of texture, geometry image and PCC auxiliary metadata items may be listed to indicate the group.

[0242] In another embodiment, the contents of PCCInfoBox, GoFHeaderBox and optional PCC auxiliary metadata information may be listed in EntityToGroupBox (with grouping_type='pcce'). An example of such a data structure is as follows:

[0243]

[0244] In another embodiment, PCCInfoBox etc. is included in an EntityToGroupBox with grouping_type equal to 'pcce' (or some other four-character code indicating PCC), as described above. GofHeaderBox etc. can be included in the file in any manner described in other embodiments, such as including it in a sample group description entry. Similarly, PCC auxiliary metadata information can be included in the file in any manner described in other embodiments.

[0245] In another embodiment, all or part of the PCC metadata may be stored in and signaled as an SEI message, and additionally carried as part of the video bitstream. Such information may include PCC auxiliary sample fields that are mapped to different or single SEI messages and sent with the texture and geometry elementary bitstreams. In this case, the PCC auxiliary metadata SEI message may reside in one of the texture or geometry elementary streams and not be repeated in both. In this embodiment, it may not be necessary to store the PCC auxiliary information or metadata information as a separate feature or metadata image item.

[0246] In another embodiment, a new NAL unit may be defined to contain PCC auxiliary metadata information. Such a NAL unit may in turn also be stored in a format compliant with ISOBMFF as part of a video sample. Again, in this case, such a NAL unit may be stored as part of a texture or geometry video track, or as part of a packed or interleaved texture and geometry track. In this embodiment, there may be no need to store the PCC auxiliary information or metadata information as a separate feature or metadata image item.

[0247] In another embodiment, NAL units containing PCC auxiliary metadata information and video NAL units can be combined into a single PCC stream and stored in a PCC track, where each PCC sample consists of one or more texture, geometry, and auxiliary information NAL units. In this case, a single media track or image item can exist in a PCC media file.

[0248] In some embodiments, subsets of geometry, texture, occupancy map and dynamic time-based metadata may follow the same coding structure. In addition, geometry, texture, occupancy map and dynamic time-based metadata may follow the same timing structure. In this case, a single track architecture may be used. The multiplexer may be able to multiplex the bitstream so that the coded access units of the geometry, texture, occupancy map and dynamic metadata required to reconstruct the point cloud at a certain moment are adjacent to each other. The order of the access unit set varies from implementation to implementation. Furthermore, this multiplexed bitstream is encapsulated into an mdat frame. A single track may treat the continuous access units (geometry, texture, occupancy map and dynamic metadata) of the multiplexed bitstream as a single PCC media sample. A subsample information frame specifically tailored for PCC media samples may be used to indicate the size of an individual access unit.

[0249] In some embodiments of a single track design, video and timed metadata access units may be interleaved. Coded geometry, texture, and timed metadata for one frame may be followed in sequence by other geometry, texture, and timed metadata for another frame. A sample may be coded geometry, texture, occupancy map, and timed metadata that needs to be decoded to be presented at a certain moment in time. An indication of the size of each coded sub-portion of an individual sample may be signaled in a Subsample Information box or stored as Sample Auxiliary Information and referenced via a Sample Auxiliary Information Size box.

[0250] In some embodiments using HEVC, to support this mechanism, some data structures as described herein may be defined.

[0251]

[0252] HEVCDecoderConfigurationRecord() may be as defined in ISO / IEC 14496-15: Information technology - Coding of audiovisual objects - Part 15: Carriage of structured video in network abstraction layer (NAL) units in ISO base media file format. In another embodiment, different codec configurations may be used together. For example, AVC and HEVC codecs may be used for different components and are listed together in PCCConfigurationBox.

[0253] An example HEVCDecoderConfigurationRecord() is:

[0254]

[0255]

[0256] HEVCDecoderConfigurationRecord() includes the size of the length field used in each sample to indicate the length of the NAL unit it contains and the parameter set (if stored in the sample entry). HEVCDecoderConfigurationRecord() is constructed externally (its size must be provided by the structure that contains it).

[0257] The values ​​of general_profile_space, general_tier_flag, general_profile_idc, general_profile_compatibility_flags, general_constraint_indicator_flags, general_level_idc, and min_spatial_segmentation_idc are valid for all parameter sets that are activated when decoding the stream described by this record (referred to as "all parameter sets" in the following sentences of this paragraph). Specifically, the following restrictions apply:

[0258] The value of general_profile_space in all parameter sets may be the same.Tier indication general_tier_flag indicates a tier that is equal to or greater than the highest tier indicated in all parameter sets.Profile indication general_profile_idc may indicate the profile to which the stream associated with this configuration record conforms.

[0259] Explicit indication of chroma format and bit depth, as well as other important format information used by HEVC video elementary streams, may be provided in an HEVC decoder configuration record. If present, each type of such information is identical across all parameter sets in a single HEVC configuration record. If two sequences differ in any type of such information, then two different HEVC configuration records are used. Two different configuration records are also required if two sequences differ in the color space indication in their VUI information. There is a set of arrays to carry initialization NAL units. NAL unit types are limited to indicating SPS, PPS, VPS, and SEINAL units.

[0260] In some embodiments, different codec configurations may be used together. For example, AVC and HEVC codecs may be used for different components and are listed together in PCCConfigurationBox. In an embodiment, PCCConfigurationBox may contain a subset of the above decoders. In turn, PCC samples may be defined by PCCSampleEntry, for example, as:

[0261]

[0262] MPEG4BitRateBox signaling transmits bit rate information.

[0263] MPEG4ExtensionDescriptorsBox includes extension descriptors.

[0264] In some embodiments, when specific track PCC media components are stored in separate tracks, the previously described track grouping and referencing mechanisms may be applied.

[0265] In some embodiments, the storage of a point cloud compression coded bitstream according to an example embodiment is depicted. The embodiment is applied to an instance of a segmented ISOBMFF file. Each PCC media component is carried in a separate media track. The PCC media components can be interlaced with any specific sample count and order. The minimum granularity can be the interlacing of PCC media components one by one. In some embodiments, a graphical representation of example interlaced samples can include texture (T), geometry (G), occupancy (O), and auxiliary (A) metadata samples from different tracks interlaced in an mdat box.

[0266] In some embodiments, the apparatus includes components such as processing circuit 12 for accessing a texture information bitstream, a geometry information bitstream, and an auxiliary metadata bitstream. In some embodiments, the apparatus includes components such as processing circuit 12 for storing the texture information bitstream in a texture video media track. In some embodiments, the apparatus includes components such as processing circuit 12 for storing the geometry information bitstream in a geometry video media track. In some embodiments, the apparatus includes components such as processing circuit 12 for storing the auxiliary metadata information bitstream in a timing metadata track. In some embodiments, the apparatus includes components such as processing circuit 12, memory 14 for constructing and storing an interlacing indicator metadata file indicating an interlacing pattern for one or more components of a texture video media track, a geometry video media track, and a timing metadata track.

[0267] An example moof box may include or be indicated as:

[0268]

[0269] The newly defined InterleavedMediaIndicatorBox() enables signaling of the following features:

[0270] - Media interleaving mode in mdat boxes, which follow the same order as 'traf' boxes in moof boxes. The indicated traf boxes belong to the listed track_IDs and may be listed consecutively to each other.

[0271] - A trun box may be associated with example tr_flags = 0x001000. This indicates that the sample_size field in the trun box indicates the sample size of the represented samples of the track based on the interleaved mode indicated by InterleavedMediaIndicatorBox(), rather than an uninterrupted continuous run of samples.

[0272] An example InterleavedMediaIndicatorBox() is:

[0273]

[0274]

[0275] InterleavedMediaIndicatorBox can be present in "moov" or "moof". If it is present in "moov", the default interleaving strategy is defined. use_default_interleaving when equal to 1 indicates that the media interleaving mode defined in the "moov" box can be used. number_of_consecutive_samples indicates how many samples in the same track exist consecutively. If use_default_interleaving is 1, this field takes the value 0. Otherwise, a value greater than 0 may exist. track_id_count indicates the number of track_IDs listed in the array. track_ID is an integer and provides a track identifier for which random access information is provided.

[0276] The "trun" box may be as defined in the ISOBMFF standard:

[0277]

[0278] The sample_composition_time_offset of the Track Run Box is the presentation time of the access unit in the media processing unit. This value is offset from the start of the MPU. The start presentation time of the media processing unit is specified by the Composition information.

[0279] In some embodiments, other ways may be used to define the interleaving strategy, such as a predefined enumeration or pattern signaled out-of-band or in a file in a predefined box. In some embodiments, if a single track is used to carry PCC media data, a SubSampleInformationBox may be used to access each interleaved PCC component within a PCC media sample. A new SampleGroupDescriptionBox may be defined to indicate the order and configuration of each PCC component.

[0280] In some embodiments where example embodiments have been described with reference to an encoder, it is to be understood that the resulting bitstream and decoder may have corresponding elements configured to perform corresponding functions. In some embodiments, the decoder may include a component such as processing circuit 12 for receiving a point cloud compression encoded bitstream, wherein the point cloud compression encoded bitstream includes a texture information bitstream, a geometry information bitstream, and an auxiliary metadata bitstream. In some embodiments, the decoder may further include a component such as processing circuit 12 for decoding the point cloud compression encoded bitstream.

[0281] In some embodiments where example embodiments have been described with reference to syntax and semantics, certain embodiments also encompass encoders that output bitstream portions according to the syntax and semantics. Likewise, certain embodiments correspondingly encompass decoders that decode bitstream portions according to the syntax and semantics.

[0282] The example embodiments describe the codec according to separate encoder and decoder devices to help understand the processes involved. However, it should be understood that the devices, structures and operations can be implemented as a single encoder-decoder device / structure / operation. In addition, the encoder and decoder can share some or all common elements.

[0283] Regarding Video-based Point Cloud Compression (V-PCC), V-PCC is very flexible, so its encapsulation in ISOBMFF can be done in many ways. Depending on the practical constraints (not necessarily described in the specification), different structural elements can be considered for use in ISOBMFF. Previous discussions on V-PCC content encapsulation have revolved around the use of entity groupings or track groupings, where the relationship between V-PCC tracks and components is described in a common location, where the decoder can easily understand how the content should be decoded and rendered. The benefit of this approach is that very little processing is required on the decoding end to figure out how to play the file. However, the main disadvantage of this system-level approach is the limitations on editing, where this relationship signaling needs to be updated when the track changes.

[0284] Some methods disclosed herein may include track grouping, entity grouping, and track referencing mechanisms to describe the relationship between V-PCC components (metadata and video components), thereby enabling random accessibility of V-PCC content.

[0285] In some embodiments, a method for encapsulating V-PCC data (e.g., content) in ISOBMFF is described, which improves the flexibility of editing V-PCC content by minimizing the amount of cross-references between V-PCC tracks and V-PCC component video tracks, while maintaining the ability to reconstruct a bitstream that conforms to V-PCC. In some embodiments, in order to maintain easy editing capabilities, the amount of repeated information is also limited, and more reconstruction responsibilities are pushed to the decoder component. In some embodiments, methods such as those described herein can make it simple to add new tracks or remove old tracks from a file (e.g., an ISOBMFF file). In some embodiments, such methods generate random access structures, such as file structures, data structures, track structures, information structures, etc., such as with respect to random access to V-PCC tracks. In some embodiments, such random access structures can be used for V-PCC content and can facilitate or indicate how the V-PCC track can or should be processed to encapsulate it in an ISOBMFF file. In some embodiments, if a single track approach is adopted, the method may include methods for how subsample information can be used and how the V-PCC bitstream can be stored in media data.

[0286] As described herein, a V-PCC sequence parameter set (SPS) is a structure that describes the overall configuration for a V-PCC presentation. The SPS may or shall be stored in a V-PCC-track SampleEntries or sampleGroupEntries of an ISOBMFF container.

[0287] In some embodiments, the SPS structure may include any or all of the components in Table 1:

[0288] Table 1

[0289]

[0290]

[0291]

[0292] In some embodiments, the V-PCC track may be a metadata track of a V-PCC package, and its sample data includes a patch data group (PDG). Table 2 provides an example of a PDG structure.

[0293] In many cases, SampleEntry updates are intended to reset the video decoder. This behavior makes it more difficult to store sequence parameter sets in V-PCC-track SampleEntries. In some embodiments, such as for V-PCC content streaming in ISOBMFF format, it may not be desirable to store sequence parameter sets in SampleEntries, but SampleGroups may be considered instead. However, the use of SampleGroups assumes that only one sequence parameter set per sample can be active.

[0294] Table 2

[0295]

[0296] In some embodiments, the component video track may contain the coded attributes, geometry, or occupancy of the video bitstream required for point cloud reconstruction. In some embodiments, the video component track may contain a V-PCC unit header in its sample entry or in its sample group description entry. The V-PCC unit header structure may provide a sequence parameter set identifier, a layer identifier, an attribute identifier, and other related signaling to uniquely identify the purpose of the track in conjunction with the sequence parameter set data stored in the V-PCC-track.

[0297] In some embodiments, a V-PCC bitstream may include a set of V-PCC units. In some embodiments, each V-PCC unit may have a V-PCC unit header and a V-PCC unit payload. In some embodiments, the V-PCC unit header describes the V-PCC unit type. Table 3 describes at least some of the supported V-PCC unit types.

[0298] Table 3

[0299]

[0300] In some embodiments, the attribute video data V-PCC unit header may also specify the attribute type and its index, thereby allowing multiple instances of the same attribute type to be supported. Table 4 specifies the supported V-PCC attribute types.

[0301] Table 4

[0302]

[0303]

[0304] In some embodiments, the occupancy, geometry, and attribute video data unit payloads may correspond to video data units (eg, HEVC NAL units) that may be decoded by a video decoder specified in a corresponding occupancy, geometry, and attribute parameter set V-PCC unit.

[0305] In some embodiments, samples of a video coding V-PCC component track may have extended VisualSampleEntry structures that contain vpcc_unit_header() information. An example data structure may be as follows:

[0306]

[0307] format can be a valid VisualSampleEnty format value for AVC, HEVC, or any other video encoder specific visual sample entry.

[0308] In some embodiments, the data content of vpcc_unit_header() may be explicitly listed in a data structure, such as described above. In some embodiments, such information may be SPS_id, attribute_type, attribute_id, layer_id, etc.

[0309] In some embodiments, the vpcc_unit_header information can be linked to the V-PCC component track samples via a sample grouping mechanism. In some embodiments, a new sample group description of a specific type is defined for this purpose. In the following example, the 4CC code 'vpcu' is given as an example:

[0310]

[0311] In some embodiments, each V-PCC track sample (which may have a vpcc_unit_header() as a header) may be linked to header data via a defined sample grouping mechanism. In some embodiments, the data content of vpcc_unit_header() may be explicitly listed in a data structure, such as described above. In some embodiments, such information may include SPS_id, attribute_type, attribute_id, layer_id, etc.

[0312] Table 5 provides an example of vpcc_unit_header().

[0313] Table 5

[0314]

[0315]

[0316] Referring now to signaling track references, in some embodiments, track references may be used to link one or more V-PCC component video tracks to a main vpcc track (vpcc-track). In some embodiments, a video component track may contain a TrackReferenceBox in a TrackBox with a reference type of 'auxl' and a track_ID matching the vpcc-track. Alternatively, in some embodiments, a new reference_type may be defined for the vpcc-track, such as 'vpcc'. In other embodiments, each V-PCC component track may have a unique track reference type.

[0317] In some embodiments, track references may be defined from a vpcc-track to a V-PCC component video track. New track reference types such as 'pcca', 'pccg', and 'pcco' may be used to reference attribute component video tracks, geometry component video tracks, and occupied component video tracks. A vpcc-track may reference all alternative representations of a V-PCC component track, but a preferred approach may be to signal only the primary representation. Alternative tracks may be established from the alternative_group values ​​within the TrackHeaderBoxes of the individual component video tracks.

[0318] In some embodiments, alternative tracks may be signaled using the alternate_group indicator in the TrackHeaderBox for each video component track or vpcc-track. Tracks containing alternative representations may share the same alternate_group value as described in the ISOBMFF specification. In some embodiments, only one track in an alternative group should be played or streamed at any one time, and must be distinguished from other tracks in the group via attributes such as bitrate, codec, language, packet size, etc. In some embodiments, the decoder / player may determine or be caused to determine which representation should be used. Restrictions may be set on the use of different track combinations in alternative groups, for example, in some embodiments, only alternative tracks with the same resolution or frame rate may or should be used at any given time.

[0319] In some embodiments, in addition to the traditional distinguishable attributes, V-PCC specific attributes may be introduced, such as single stream or multiple stream encapsulation of V-PCC layers. For example, a vpcc-track may contain multiple sequence parameter sets with different layer layouts. In some embodiments, each video component track may be linked to the correct sequence parameter set, so it is again up to the player to decide which representation it should select.

[0320] In some embodiments, at least one benefit of using such a structure is that the file format is easy to edit because the video component track only contains a reference to the vpcc-track (and vice versa). Therefore, in some embodiments, component tracks can be easily added and removed. In addition, the proposed method may include reusing several V-PCC specification structures, which may reduce or minimize the processing of V-PCC content while encapsulating it into ISOBMFF. According to some embodiments, the tradeoff with reducing or minimizing the processing requirements of V-PCC content when encapsulating V-PCC content into ISOBMFF may be or include that the parser will need to understand the V-PCC structure encapsulated in the file format.

[0321] Referring now to signaling track substitutions and recommended representations, in some embodiments, track substitutions may be signaled using the alternate_group value in the TrackHeaderBox. However, in some embodiments, this mechanism fails to provide information about the nature of the alternative grouping (i.e., bitrate, texture difference, resolution, quality, etc.). In some embodiments, to address this shortcoming, the 'altr' entity grouping in the EntitytoGroupBox may be used. In some embodiments, to signal the alternative, a flag of the EntitytoGroupBox may be used. By way of example only, in some embodiments, each bit in the 24-bit flag value may indicate the reason for the substitution. In some embodiments, these may be bitrate, resolution, single PCM or multiple PCM tracks, different textures, encoding quality, etc.

[0322] In some embodiments, since there can be multiple different alternatives in a representation, the content creator can signal a recommended V-PCC representation. In some embodiments, this can be achieved by introducing an EntitytoGroupBox of a specific type (e.g., 'vpcg') within the meta box and listing the V-PCC track and the selected component tracks in the alternative. In some embodiments, there can be multiple such lists for different alternative representation groups.

[0323] Regarding the use of VPCCGroupBox and VPCCDecoderConfigurationRecords, in some embodiments, a VPCCGroupBox may be defined to group vpcc content. In some embodiments, the VPCCGroupBox may exist in a SampleEntry or SampleGroupDescriptionEntry or reside in a VPCCDDecoderConfigurationRecord. In some embodiments, this method for file encapsulation is relatively simple because it is assumed that the V-PCC content uses only a single sequence parameter set. In some embodiments, such a data structure may be linked to each SPS data structure and is valid when the SPS is used to decode a V-PCC frame. In the following example file structure, a method is outlined that uses a VPCCGroupBox to include meaningful high-level details of the V-PCC content in a single location with minimal repetition and maximum flexibility. In some embodiments, at least one benefit of introducing a VPCCGroupBox may be to provide a single location for the decoder to check how the V-PCC content is constructed in order to decide how to decode and render it. In some embodiments, the use of a V-PCC structure such as described herein requires fewer updates to the file format specification itself. However, in some embodiments, the parser may be required to understand these structures, ie, parsing the VPCCGroupBox requires knowledge of the V-PCC structure.

[0324]

[0325] In some embodiments, track_reference_id may indicate the index of the track reference of the referenced track in the 'tref' box of the V-PCC track. In some embodiments, track_id may be used directly in this field. In some embodiments, sps_frame_width and _sps_frame_height may be used to signal the maximum value of sps_frame_width and _sps_frame_height for the entire presentation or representation, for example, if there are several sequence parameter sets. In some embodiments, other vpcc related structures may be present in the VPCCGroupBox.

[0326] In some embodiments, if the sequence parameter set is not expected to change during presentation, the encapsulation structure is convenient, however this is not limited by the specification and therefore the file format should allow multiple sequence parameter sets to be stored for presentation. In some embodiments, because tracks can reference different sequence parameter sets at different times, the time alignment of such references must be maintained. In some embodiments, this makes it less feasible to store the vpcc_unit_header() information of the component video track in the VPCCGroupBox of the vpcc-track. In some embodiments, the method may include storing the vpcc_unit_header() information in the component video track itself. In order to maintain timing information, SampleGroups may be used to store the vpcc_unit_headers() of the component video track, such as:

[0327]

[0328] In some embodiments, there may be a VPCC track decoder configuration record, similar to some conventional storage methods for NAL-based video encoding data.

[0329] The current working draft for V-PCC coded data gives the following definition:

[0330]

[0331] In some embodiments, data entities such as those described above may be encapsulated in a VPCCDDecoderConfigurationRecord or an equivalent data structure, such as:

[0332]

[0333] In some embodiments, the addition of the above listed fields to the data structure may make it easier for the file parser to parse and signal such information without having to parse one or more sequence_parameter_sets.

[0334] In some embodiments, the tracks referenced by the V-PCC track may be listed in the VPCCSampleEntry, such as:

[0335]

[0336] In some embodiments, track_reference_id may indicate the index of the track reference of the referenced track in the 'tref' box of the V-PCC track. In some embodiments, track_id may be used directly in this field.

[0337] Therefore, in some embodiments, the referenced track list and possible vpcc_unit_header() values ​​may be signaled within the VPCCSampleEntry. This provides an easy way for the file parser to understand the nature of the referenced tracks by parsing only the V-PCC track.

[0338] Referring now to the random access aspects of VPCC content, each V-PCC entity may have its own random access interval. In some embodiments, component video tracks may use well-established video-related random access features. Historically, random access of vpcc-tracks can be computationally very complex and time consuming. A vpcc-track typically contains groups of patch data in its sample data. In some embodiments, each vpcc sample may contain patch data for a single vpcc frame. In some embodiments, patch sequence data may be defined as a list of patch_data_group_unit_payloads, where the payload types may be different. Some patch data group unit payloads contain information that is applied to subsequent frames in addition to the frames to which they are implicitly linked and are therefore not frame specific. Such patch sequence unit payloads include at least types 0-5, as provided in Table 6.

[0339] Table 6

[0340]

[0341] A patch data group unit payload with type PDG-PTGLU contains a single patch_tile_group_layer_unit(), which in turn consists of a patch_tile_group_header() and a patch_tile_group_data_unit() pair. The patch_tile_group_header() contains a field ptgh_type which specifies the coding type 1 (intra) or 0 (prediction) for the patch data in the patch tile group data unit. This information can be used to define synchronization samples for vpcc-track.

[0342] In one embodiment, non-frame specific patch data group unit payloads may be extracted from the patch data group stream and stored in the vpcc-track sample group description entry.

[0343]

[0344]

[0345] In some embodiments, patch_data_group_unit_payload() may be constructed as defined in ISO / IEC 23090-5, but is limited to containing only patch data group unit payloads of type 0-5 as defined herein. In some embodiments, the extracted patch data group unit payloads in the sample group description entry may or should be time aligned with the samples from which they were originally extracted, thus maintaining random access capabilities using the existing SampleGroupDescriptionEntry mapping. As a result of this extraction, patch_data_group() shall only contain patch_data_group_unit_payloads of type PDG_PTGLU, meaning that synchronized samples may be defined as samples where the patch tile group layer unit ptgh_type is equal to 1 (intra-frame). During file parsing operations on the decoder side, if two subsequent samples belong to different SampleGroups, the subsequent samples are modified to include the patch_data_group() in the SampleGroupEntry to which they belong.

[0346] In another embodiment, patch_data_group() can contain any pdg_unit_types and no manipulation of patch_data_group() is required.With this method, patch_data_group() containing pdg_unit_types between 0 and 5 (including the endpoints) are copied as is to the SampleGroupEntries of the vpcc track.

[0347] During the file parsing operation at the decoder side, if two subsequent samples belong to different SampleGroups, the subsequent samples are modified to include the patch_data_group() in the SampleGroupEntry to which they belong.

[0348] In another embodiment, all active non-frame specific patch data group unit payloads may or should be replicated in all vpcc sync samples. In this embodiment, a sync (or random access) sample is defined as a sample that does not depend on any previous sample in the track and contains all relevant information to reconstruct a point cloud from time-aligned 2D component video samples. This definition is simpler, but may require that patch data group unit payloads of type 0-5 be replicated for each sync sample, thereby slightly increasing the size of the bitstream, but avoiding the introduction of sample grouping.

[0349] If the v-pcc component track contains multiple layers, this information may need to be signaled within the video track itself. One such mechanism may be to define a new box in SchemeInformationBox to signal the presence of multiple layers. A possible data structure may be as follows:

[0350] Frame name: VPCCMultipleLayerIndicator frame

[0351] Frame type: 'vmli'

[0352] Container: SchemeInformationBox

[0353] Mandatory: yes (when SchemeType is 'pccv')

[0354] Quantity: 1

[0355] The VPCCMultipleLayerIndicator box is used to indicate that the video track consists of multiple layers coded by VPCC with the same properties. This also means that for this V-PCC component, the sps_multiple_layer_streams_present flag in the SPS is set to 1.

[0356]

[0357] In some embodiments, number_of_layers indicates the number of layers that exist and are temporarily packed as frames in a V-PCC component track. Its value may be sps_layer_count_minus1+1.

[0358] In some embodiments, such as when there are gaps in the current number of layers, zero-sized samples may be included in the sample table box.

[0359] In some embodiments, PackedContentInterpretationType may be used for signaling additional information about the packing structure and type. As an example, it may indicate that successive temporal packing frames adopt a predefined layer order (in ascending layer order or descending layer order).

[0360] Now referring to the description about composition time when layers are stored in the same track, when there are time-interleaved vpcc layers in the video component track, the composition time of the samples needs to be described. Considering that samples are not allowed to have the same composition time in ISOBMFF, readers should implicitly set the composition time used for the composed picture between 0 and layer_count_minus1 (including the endpoints) to be consistent with the composition time layer_count_minus1 used for the last composed picture.

[0361] Referring now to a single track representation of V-PCC content, in some embodiments, it is desirable to store the V-PCC encoded bitstream intact in a single track. Potential advantages may include at least:

[0362] Directly store V-PCC coded bitstream without any additional header processing;

[0363] Utilize the time and byte size framework (i.e., sample timing and sample size) of ISOBMFF; and

[0364] Enables and indicates random access position in the V-PCC bitstream.

[0365] In some embodiments, to indicate that a V-PCC track is a "self-contained" media track, the following constraints may apply:

[0366] Each V-PCC access unit may include bitstream data of related V-PCC components only in a predefined or random order, including the V-PCC unit header and SPS. Thus, it will be easy to divide each V-PCC frame in a synthetic time order.

[0367] A V-PCC sample may be equivalent to a V-PCC access unit, which may or may not depend on other V-PCC samples to reconstruct a V-PCC frame corresponding to the same synthesis timestamp of the V-PCC sample.

[0368] A random access or synchronized V-PCC sample is a V-PCC sample that is not dependent on any other V-PCC sample to decode the content.

[0369] The SPS and cell header data structures are kept intact in the coded V-PCC bitstream.

[0370] In some embodiments, an ISOBMFF file storing V-PCC content in a single track may use VolumetricSampleEntry of a specific type (such as 'vpcs'). All possible sequence_parameter_set() may be listed in this sample entry so that a V-PCC decoder can decode it. A possible data structure may be as follows:

[0371]

[0372] Among them, VPCCDecoderConfigurationRecord and VPCCConfigurationBox are defined as follows:

[0373]

[0374]

[0375] In some embodiments, a V-PCC subsample may be defined as a V-PCC component consisting of a V-PCC unit header and a payload. Taking this definition into account, the subsample information box defined in ISOBMFF may be updated as follows:

[0376]

[0377] Among them, codec_specific_parameters may indicate necessary parameters of vpcc_unit mapped to subsamples. Two possible syntaxes of codec_specific_parameters that may be used to interpret codec specific parameter values ​​are shown in Table 7a and Table 7b.

[0378] Table 7a

[0379] codec_specific_parameters value describe 0 Reserve 1 Patch group data 2 Occupancy 3 Geometric shapes 4 property 5,… Reserve

[0380] Table 7b

[0381]

[0382] In some embodiments, the declared parameter types may be syntactically the same as in vpcc_unit_header() and ISO / IEC 23090-5.

[0383] Where pcm_separate_video_data() is as defined in ISO / IEC 23090-5:

[0384]

[0385] In another embodiment, the vpcc_unit_header() may be copied as is to the coded_specific_parameters field. In another embodiment, the subsample_prority field may be set to a non-default value to indicate the priority of the vpcc data unit (as a subsample). In this case, if the decoder does not have enough decoder instances, it may prefer to skip decoding such data units. Similar logic may also be applied to the discardable parameters. A value of 0 may indicate that a subsample is required to represent a V-PCC sample.

[0386] By using the subsample information box, the file parser can extract each V-PCC component, parse its header information, and further identify the second-level information such as attribute_id, layer_id, etc.

[0387] In some embodiments, the file parser may parse codec_specific_parameters() and may further obtain pcc component related information before parsing vpcc_header_data(). This will provide the parser with enough information to push the subsample data to the correct decoder instance.

[0388] In some embodiments, a separate data structure may be incorporated into a new version of the subsample information box, and each V-PCC specific information may be listed in the new data structure.

[0389] In some embodiments, the patch data group unit syntax may include descriptors and terms such as those in Table 8.

[0390] Table 8

[0391]

[0392]

[0393] In some embodiments, a patch data group unit may include a payload and metadata. In some embodiments, the syntax for a patch data group unit payload may include descriptors and terms such as in Table 9.

[0394] Table 9

[0395]

[0396] In some embodiments, the patch sequence parameter set syntax may include descriptors and terms such as those in Table 10.

[0397] Table 10

[0398]

[0399]

[0400] In some embodiments, the patch frame header syntax may include descriptors and terms such as those in Table 12.

[0401] Table 12

[0402]

[0403]

[0404]

[0405] In some embodiments, pfh_address can be added as a placeholder. In some embodiments, the header structure from Table 12 can be modified into a patch_tile header and additional structures for tiles can be defined. In a tiled tiling structure, a process of allocating a "patch tile address" with 2D (x, y) coordinates within the entire frame unit can be performed. In some embodiments, the header may correspond only to a patch tile. In some embodiments, there may be multiple tiles for a frame. In some embodiments, in such a scenario, some information may also be placed in the frame header.

[0406] In some embodiments, aps_attribute_codec_id[i] indicates the identifier of the codec used to compress the attribute video data with index i. aps_attribute_codec_id[i] may or shall be in the range of 0 to 255, including all values ​​and ranges therebetween. In some embodiments, such a value may be predefined or may be defined by a codec or another such component.

[0407] In some embodiments, when present, aps_pcm_attribute_codec_id[i] indicates the identifier of the codec used to compress the attribute video data of a PCM code point of attribute i when the PCM code point is encoded in a separate video stream. aps_pcm_attribute_codec_id[i] may or shall be in the range of 0 to 255, including all values ​​and ranges therebetween.

[0408] In some embodiments, aps_attribute_dimension_minus1[i] plus 1 indicates the total number of dimensions (ie, number of channels) of the attribute with index i. aps_attribute_dimension_minus1[i] may or shall be in the range of 0 to 255, including all values ​​and ranges therebetween.

[0409] In some embodiments, adding 1 to aps_attribute_dimension_partitions_minus1[i] indicates the number of partition groups into which the attribute channels of the attribute with index i should be grouped. aps_attribute_dimension_partitions_minus1[i] can or should be in the range from 0 to 127, including all values and ranges therebetween.

[0410] In some embodiments, adding 1 to aps_attribute_partition_channels_minus1[i][j] indicates the number of channels of the dimension partition group with index j assigned to the attribute with index i. If j < aps_attribute_dimension_partitions_minus1[i], then aps_attribute_partition_channels_minus1[i][j] can or should be in the range from 0 to remainingDimensions–2, including all values and ranges therebetween. If j = aps_attribute_dimension_partitions_minus1[i], then aps_attribute_partition_channels_minus1[i][j] is set to be equal to remainingDimensions–1.

[0411] In some embodiments, aps_attribute_params_enabled_flag[i] indicates whether attribute parameters can be signaled, e.g., for all attributes with index i. aps_attribute_params_enabled_flag[i] being equal to 1 indicates that attribute parameters can be signaled for the attribute with index i. aps_attribute_params_enabled_flag[i] being equal to 0 indicates that for the attribute with index i, attribute parameters may not or should not be signaled.

[0412] In some embodiments, aps_attribute_patch_params_enabled_flag[i] indicates whether attribute patch parameters may be signaled for all attributes with index i. aps_attribute_patch_params_enabled_flag[i] equal to 1 indicates that attribute patch parameters may be signaled, e.g., for the attribute with index i. aps_attribute_patch_params_enabled_flag[i] equal to 0 indicates that attribute patch parameters may not or should not be signaled, e.g., for the attribute with index i.

[0413] In some embodiments, the V-PCC frame may be decoded according to any of the methods described herein.

[0414] In some embodiments, the output of such a decoding process can be a set of decoded video frames or video streams, occupancy, geometry, and attribute (if available) information corresponding to the point cloud sequence, and decoded patch sequence data information, some or all of which are required to perform the 3D reconstruction process.

[0415] In some embodiments, the output of the decoding process is a sequence of decoded point cloud frames. A point cloud frame consists of a set of points with (x, y, z) coordinates. Optionally, in some embodiments, a point cloud frame may also include one or more sets of attributes associated with each point.

[0416] In some embodiments, a decoding process may be specified so that all decoders conforming to a specified profile, layer, and level will produce numerically identical output when a decoding process associated with the profile is invoked for a bitstream conforming to the profile, layer, and level. In some embodiments, such a decoding process may produce output identical to that produced by the process described herein (with the correct output order or output timing, as specified). For best effort decoding purposes, a decoder conforming to a particular profile at a given layer and level may also decode some bitstreams conforming to different layers, levels, or profiles without having to use a decoding process that produces a decoded output that is numerically identical to that produced by the process specified herein (without declaring conformance to other profiles, layers, and levels).

[0417] In some embodiments, the decoding process operates as follows:

[0418] The VPCC bitstream can be parsed and demultiplexed into several substreams, each corresponding to a different set of information. In particular, the VPCC bitstream is demultiplexed into occupancy map, geometry and attribute (if present) video substreams, and in a patch sequence substream. In turn, these substreams can be decoded as specified in subsequent steps.

[0419] The occupancy map video decoding process may be invoked with a substream corresponding to the occupancy map information as input and a decoded occupancy map video frame as output. Each decoded occupancy map video frame may be represented as oFrame[y][x], where y and x are row and column indices, respectively. The variable y may be in the range of 0 to sps_frame_height-1 (inclusive), and the variable x may be in the range of 0 to sps_frame_width-1, including all values ​​and ranges therebetween.

[0420] Furthermore, the geometric video decoding process can be called.

[0421] For each attribute video substream present in the bitstream, the attribute video decoding process may in turn be invoked, such as if the attribute video substream is present in the bitstream.

[0422] Furthermore, a patch sequence decoding process may be invoked, with the substream corresponding to the encoded patch sequence information as input and the decoded patch sequence information as output.

[0423] Optionally, for each point cloud frame, the reconstruction process of the point cloud frame can be called with the array sps_layer_absolute_coding_enabled_flag, with all decoded occupancy, geometry and attribute (if any) frames and decoded patch frame information corresponding to time instance i as input and a list of decoded points in the point cloud frame as output.

[0424] In some embodiments, the geometric video decoding process is performed as follows:

[0425] If sps_layer_count_minus1 is equal to 0, the video decoding process is called using the geometry video bitstream and its associated codec (specified by gps_geometry_codec_id as input). The output of this process is the decoded and displayed / output ordered geometry video frame GeoFrame[layerIdx][orderIdx][compIdx][y][x] and its associated bit depth GeoBitdepth[layerIdx][orderIdx], width GeoWidth[layerIdx][orderIdx], and height GeoHeight[layerIdx][orderIdx], where layerIdx corresponds to the layer index and is equal to 0, orderIdx is the display order index of the decoded geometry frame, compIdx corresponds to the color component index and is equal to 0, y is in the range of 0 to GeoHeight[layerIdx][orderIdx]-1 (inclusive), and x is the column index in the decoded frame and is in the range of 0 to GeoWidth[layerIdx][orderIdx]-1, including all values ​​and ranges therebetween.

[0426] Otherwise, the following applies:

[0427] If sps_multiple_layer_streams_present_flag is equal to 0, the video decoding process is called using the geometry video bitstream and its associated codec (specified by gps_geometry_codec_id as input). The output of this process is the decoded and displayed / output ordered intermediate geometry video frames tempGeoFrame[tempOrderIdx][compIdx][y][x] and their associated bit depth tempGeoBitdepth[tempOrderIdx], width tempGeoWidth[tempOrderIdx], and height tempGeoHeight[tempOrderIdx], where tempOrderIdx is the display order index of all decoded geometry frames, compIdx corresponds to the color component index and is equal to 0, y is in the range of 0 to tempGeoHeight[tempOrderIdx]-1 (inclusive), and x is the column index in the decoded frame and is in the range of 0 to tempGeoWidth[tempOrderIdx]-1, including all values ​​and ranges therebetween. Then, the decoded geometry video frame in display / output order orderIdx for each layer is derived as follows:

[0428]

[0429] Otherwise (if sps_multiple_layer_streams_present_flag is equal to 1), multiple video decoding processes are invoked, each using a geometry video bitstream having a different vpcc_layer_index associated therewith and an associated codec (specified by gps_geometry_codec_id as input). The output of this process is the decoded and displayed / output ordered intermediate geometry video frames tempGeoFrame[layerIdx][orderIdx][compIdx][y][x] and their associated bit depth tempGeoBitdepth[layerIdx][orderIdx], width tempGeoWidth[layerIdx][orderIdx], and height tempGeoHeight[layerIdx][orderIdx], where layerIdx corresponds to the layer index of each frame and is in the range of 0 to sps_layer_count_minus1 (inclusive), orderIdx is the display order index of the decoded geometry frame for each layer, compIdx corresponds to the color component index and is equal to 0, y is in the range of 0 to tempGeoHeight[layerIdx][orderIdx]-1 (inclusive), and x is the column index in the decoded frame and is in the range of 0 to tempGeoWidth[layerIdx][orderIdx]-1, including all values ​​and ranges therebetween. Then, the decoded geometry video frame in display / output order orderIdx for each layer is derived as follows:

[0430]

[0431] In some embodiments, the attribute decoding process may be performed as follows:

[0432] If aps_attribute_count is equal to 0, then no attribute video frame is decoded and no attribute information is associated with the final reconstructed point cloud.

[0433] Otherwise (if aps_attribute_count is not equal to 0), the following applies:

[0434] If sps_layer_count_minus1 is equal to 0, aps_attribute_count video decoding processes are called, each with a different vpcc_attribute_index associated with it and the associated codec (specified by aps_attribute_codec_id[vpcc_attribute_index] as input). The output of this process is the decoded and displayed / output ordered attribute video frame AttrFrame[attrIdx][layerIdx][orderIdx][compIdx][y][x] and its associated bit depth AttrBitdepth[attrIdx][layerIdx][orderIdx], width AttrWidth[attrIdx][layerIdx][orderIdx], and height AttrHeight[attrIdx][layerIdx][orderIdx] information, where attrIdx corresponds to the attribute index and ranges from 0 to aps_attribute_codec_id[vpcc_attribute_index]. unt-1 (including the endpoints), layerIdx corresponds to the layer index and is equal to 0, orderIdx is the display order index of the decoded attribute frame, compIdx corresponds to the attribute component index and is in the range of 0 to aps_attribute_dimension_minus1[attrIdx-1], y is in the range of 0 to AttrHeight[attrIdx][layerIdx][orderIdx]-1 (including the endpoints), x is the column index in the decoded frame and is in the range of 0 to AttrWidth[attrIdx][layerIdx][orderIdx]-1, including all values ​​and ranges in between.

[0435] Otherwise (sps_layer_count_minus1 is not equal to 0), the following applies:

[0436] If sps_multiple_layer_streams_present_flag is equal to 0, aps_attribute_count video decoding processes are called, each with a different vpcc_attribute_index associated with it and the associated codec (specified by aps_attribute_codec_id[vpcc_attribute_index] as input). The output of this process is the decoded and displayed / output ordered intermediate attribute video frames tempAttrFrame[attrIdx][tempOrderIdx][compIdx][y][x] and their associated bit depth tempAttrBitdepth[attrIdx][tempOrderIdx], width tempAttrWidth[attrIdx][tempOrderIdx], and height tempAttrHeight[attrIdx][tempOrderIdx] information, where attrIdx corresponds to the attribute index and ranges from 0 to aps_attrib ute_count-1 (inclusive), tempOrderIdx is the display order index of all decoded attribute frames, compIdx corresponds to the attribute component index and is in the range of 0 to aps_attribute_dimension_minus1[attrIdx-1], y is in the range of 0 to tempAttrHeight[attrIdx][tempOrderIdx]-1 (inclusive), x is the column index in the decoded frame and is in the range of 0 to tempAttrWidth[attrIdx][tempOrderIdx]-1, including all values ​​and ranges therebetween. In turn, the decoded attribute video frame for the attribute indexed by attrIdx in the display / output order orderIdx for each layer is derived as follows:

[0437]

[0438] Otherwise (sps_multiple_layer_streams_present_flag is equal to 1), multiple video decoding processes are called, each using an attribute video bitstream and associated codec (specified by aps_attribute_codec_id[vpcc_attribute_index] as input) with a different vpcc_attribute_index and vpcc_layer_index associated with it. The output of this process is the decoded and displayed / output ordered attribute video frames AttrFrame[attrIdx][layerIdx][orderIdx][compIdx][y][x] and their associated bit depth AttrBitdepth[attrIdx][layerIdx][orderIdx], width AttrWidth[attrIdx][layerIdx][orderIdx], and height AttrHeight[attrIdx][layerIdx][orderIdx] information, where attrIdx corresponds to the attribute index and is in the range of 0 to aps_attribute_count-1, and layerIdx is in the range of 0 to 1. corresponds to the layer index of each frame and is in the range of 0 to sps_layer_count_minus1 (including the endpoints), orderIdx is the display order index of the decoded attribute frame for each layer, compIdx corresponds to the attribute component index and is in the range of 0 to aps_attribute_dimension_minus1[attrIdx-1], y is in the range of 0 to AttrHeight[attrIdx][layerIdx][orderIdx]-1 (including the endpoints), x is the column index in the decoded frame and is in the range of 0 to AttrWidth[attrIdx][layerIdx][orderIdx]-1, including all values ​​and ranges in between.

[0439] Referring now to the process for patch sequence decoding, the input to this process is a bit stream of patch sequence data units corresponding to the VPCC bit stream.

[0440] In some embodiments, the output of this process is decoded patch frames, and any associated geometry and / or attribute parameters contained in patch sequence data units, sorted in ascending order according to their corresponding patch frame sequence counts.

[0441] In some embodiments, for the decoded current patch frame CurrPatchFrame, the decoding process operates as follows:

[0442] Decoding of a patch sequence data unit as discussed herein may have a patch sequence data unit and an associated parameter set unit of a current patch frame as input, and a parsed syntax structure encapsulated within these patch sequence data units as output. Such decoding may be performed by extracting a syntax structure from the patch sequence data unit and then parsing the syntax structure.

[0443] The process may further include deriving variables and functions related to the patch frame sequence count.

[0444] At the beginning of the decoding process for each patch frame, a reference patch frame list construction process such as that described herein may be called to derive a reference patch frame list RefPatchFrameList.

[0445] Furthermore, a reference patch frame marking process such as described herein may be invoked, wherein reference patch frames may be marked as "unused for reference" or "used for long-term reference."

[0446] Some of the processes described herein may specify patch decoding processes according to patch modes, such as the following processes:

[0447] A process for decoding intra-coded patches;

[0448] A process for decoding inter-coded patches; and

[0449] · Process for decoding PCM coded patches.

[0450] In some embodiments, after the current patch frame has been decoded, it may be marked as "used for short-term reference".

[0451] Referring now to the processes for patch sequence data unit decoding, the inputs of these processes are the patch sequence data units of the current patch frame and their associated parameter set units, and the outputs of these processes are the parsed syntax structures encapsulated within these patch sequence data units. In some embodiments, the decoding process for each patch sequence data unit may include extracting the syntax structure from the patch sequence data unit and then parsing the syntax structure.

[0452] Reference is now made to a process for patch frame header decoding, which may include a process for patch frame order count derivation. In some embodiments, a process for patch frame order count derivation may be performed, thereby producing an output of a patch frame order count PatchFrmOrderCntVal for a current patch frame having an index patchFrmIndex. In some embodiments, the patch frame order count may be used to identify and sort patch frames, as well as for decoder consistency checking. In some embodiments, each encoded patch frame is associated with a patch frame order count variable (denoted as PatchFrmOrderCntVal). In some embodiments, when there is no reference patch frame available in the reference patch frame buffer, the variables prevPatchFrmOrderCntLsb and prevPatchFrmOrderCntMsb are derived as follows:

[0453] Let prevPatchFrm be the previous patch frame in decoding order, that is, the patch frame with index patchFrmIndex-1,

[0454] The variable prevPatchFrmOrderCntLsb is set equal to the patch frame order count LSB value of prevPatchFrm, pfh_patch_frame_order_cnt_lsb[patchFrmIndex-1],

[0455] The variable prevPatchFrmOrderCntMsb is set equal to PatchFrmOrderCntMsb of prevPatchFrm, and

[0456] The variable PatchFrmOrderCntMsb of the current patch frame is exported as follows:

[0457] If there is no reference patch frame available in the reference patch frame buffer, PatchFrmOrderCntMsb is set equal to 0.

[0458] Otherwise, PatchFrmOrderCntMsb is derived as follows:

[0459]

[0460]

[0461] In some embodiments, PatchFrmOrderCntVal may be derived as follows: PatchFrmOrderCntVal = PatchFrmOrderCntMsb +

[0462] pfh_patch_frame_order_cnt_lsb[patchFrmIndex]

[0463] In some embodiments, the value of PatchFrmOrderCntVal can be -2 31 To 2 31 In some embodiments, for a CPCS, the values ​​of PatchFrmOrderCntVal of any two coded patch frames may not or should not be the same.

[0464] In some embodiments, at any time during the decoding process, the values ​​of PatchFrmOrderCntVal&(MaxLtPatchFrmOrderCntLsb-1) of any two referenced patch frames in the DPB may not or should not be the same.

[0465] According to at least one embodiment, the function PatchFrmOrderCnt(pFrmX) is defined as follows:

[0466] PatchFrmOrderCnt(pFrmX) = PatchFrmOrderCntVal of patch frame pFrmX

[0467] In some embodiments, the function DiffPatchFrmOrderCnt(pFrmA, pFrmB) is defined as follows:

[0468] DiffPatchFrmOrderCnt(pFrmA,pFrmB)=PatchFrmOrderCnt(pFrmA)–PicOrderCnt(pFrmB)

[0469] In some embodiments, the bitstream may not contain data that would cause the value of DiffPatchFrmOrderCnt(pFrmA, pFrmB) used in the decoding process to be outside the range of -215 to 215-1, including all values ​​and ranges therebetween.

[0470] For example, let X be the current patch frame and Y and Z be two other patch frames in the same CPCS, according to at least some embodiments, when DiffPatchFrmOrderCnt(X,Y) and DiffPatchFrmOrderCnt(X,Z) are both positive or both negative, Y and Z can be considered to be in the same output order direction as X.

[0471] Now referring to the process for building a reference patch frame list, such a process can be called at the beginning of the decoding process for each patch frame. In some embodiments, the reference patch frame can be addressed by a reference index. The reference index is an index to the reference patch frame list. When decoding an I patch frame, the reference patch frame list is not used when decoding patch frame data. When decoding a P patch frame, a single reference patch frame list RefPatchFrmList is used when decoding patch frame data.

[0472] In some embodiments, at the beginning of the decoding process for each patch frame, a reference patch frame list RefPatchFrmList is derived. The reference patch frame list can be used in marking reference patch frames or decoding patch frame data.

[0473] According to some embodiments, the reference patch frame list RefPatchFrmList is constructed as follows:

[0474]

[0475] The first NumRefIdxActive entries in RefPatchFrnList may be referred to as active entries in RefPatchFrmList, and the other entries in RefPatchFrmList may be referred to as inactive entries in RefPatchFrmList.

[0476] According to some embodiments, applying the following constraints may be a requirement for bitstream conformance:

[0477] i) num_ref_entries[RlsIdx] is not less than NumRefIdxActive;

[0478] ii) The patch frame referenced by each active entry in RefPatchFrmList exists in the DPFB;

[0479] iii) The patch frame referenced by each entry in RefPatchFrmList is not the current patch frame;

[0480] iv) the short-term reference patch frame entry and the long-term reference patch frame entry in the RefPatchFrmList of the patch frame do not reference the same patch frame; and

[0481] v) There is no long-term referenced patch frame entry in RefPatchFrmList. For RefPatchFrmList, the difference between the PatchFrmOrderCntVal of the current patch frame and the PatchFrmOrderCntVal of the patch frame referenced by the entry is greater than or equal to 224.

[0482] In some embodiments, let setOfRefPatchFrames be the set of unique patch frames referenced by all entries in RefPatchFrmList. The number of patch frames in setOfRefPatchFrames may be less than or equal to psps_max_dec_patch_frame_buffering_minus1.

[0483] Now referring to the process for reference patch frame marking, such a process can be called once per patch frame after the decoding process of decoding the patch frame header and the reference patch frame list construction for the patch frame but before decoding the patch frame data. This process can cause one or more reference patch frames in the DPFB to be marked as "unused for reference" or "used for long-term reference".

[0484] A decoded patch frame in a DPFB may be marked as "not used for reference", "used for short-term reference", or "used for long-term reference", but at any given moment during the operation of the decoding process, only one of these three is used. Assigning one of these marks to a patch frame will implicitly remove another of these marks when applicable. When a patch frame is referenced to be marked as "used for reference", it can be collectively referred to as the patch frame being marked as "used for short-term reference" or "used for long-term reference" (but not both at the same time).

[0485] A short-term reference patch frame is identified by its PatchFrmOrderCntVal value. A long-term reference patch frame is identified by the least significant bit of Log2(MaxLtPatchFrmOrderCntLsb) of its PatchFrmOrderCntVal value.

[0486] For at least some embodiments, the following applies:

[0487] For each long-term reference patch frame entry in RefPatchFrmList, when the referenced patch frame is a short-term reference patch frame, the patch frame is marked as "used for long-term reference"; and

[0488] Each reference patch frame in the DPFB that is not referenced by any entry in the RefPatchFrmList is marked as "unused for reference".

[0489] Referring now to a process for decoding a patch unit, such as a general process, the input to such a process may include the current patch frame index frmIdx, the current patch index p, and the current patch mode pfdu_patch_mode[frmIdx][p].

[0490] In some embodiments, the output of this process is several parameters associated with the current patch, including its 2D and corresponding 3D position information, such as parameters Patch2dShiftU[frmIdx][p], Patch2dShiftV[frmIdx][p], Patch2dSizeU[frmIdx][p], Patch2dSizeV[frmIdx][p], Patch3dShiftTangentAxis[frmIdx][p], Patch3dShiftBiTangentAxis[frmIdx][p], Patch3dShiftNormalAxis[frmIdx][p], PatchNormalAxis[frmIdx][p], PatchOrientationIndex[frmIdx][p], PatchLod[frmIdx][p], and PatchProjectionMode[frmIdx][p].

[0491] In some embodiments, Patch2dShiftU[frmIdx][p] specifies the x coordinate of the upper left corner of the patch bounding box size of the current patch. The value of Patch2dShiftU[frmIdx][p] can be between 0 and Min((2 pfh _2d_shift_u_bit_count_minus1[frmIdx]+1 –1)*PatchPackingBlockSize,sps_frame_width-1), including all values ​​and ranges in between.

[0492] In some embodiments, Patch2dShiftV[frmIdx][p] specifies the y coordinate of the upper left corner of the patch bounding box size of the current patch. The value of Patch2dShiftV[frmIdx][p] can be between 0 and Min((2 pfh _2d_shift_v_bit_count_minus1[frmIdx]+1 –1)*PatchPackingBlockSize,sps_frame_height-1), including all values ​​and ranges in between.

[0493] In some embodiments, Patch2dSizeU[frmIdx][p] specifies the width of the bounding box of the current patch. The value of Patch2dSizeU[frmIdx][p] can be in the range of 0 to (sps_frame_width-1), including all values ​​and ranges therebetween.

[0494] In some embodiments, Patch2dSizeV[frmIdx][p] specifies the height of the bounding box of the current patch. The value of Patch2dSizeV[frmIdx][p] can be in the range of 0 to (sps_frame_height-1), including all values ​​and ranges therebetween.

[0495] In some embodiments, Patch3dShiftTangentAxis[frmIdx][p] specifies the offset along the tangent axis to be applied to the reconstructed patch points in the current patch. The value of Patch3dShiftTangentAxis[frmIdx][p] can be between 0 and Min(2 pfh_3d_shift_tangent_axis_bit_count_minus1[frmIdx]+1 ,2 gps_geometry_3d_coordinates_bitdepth_minus1+1 -1), including all values ​​and ranges therebetween.

[0496] In some embodiments, Patch3dShiftBiTangentAxis[frmIdx][p] specifies the offset along the bitangent axis to be applied to the reconstructed patch points in the current patch. The value of Patch3dShiftBiTangentAxis[frmIdx][p] can be between 0 and Min(2 pfh_3d_shift_bitangent_axis_bit_count_minus1[frmIdx]+1 ,2 gps _geometry_3d_coordinates_bitdepth_minus1+1 -1), including all values ​​and ranges therebetween.

[0497] In some embodiments, Patch3dShiftNormalAxis[frmIdx][p] specifies an offset along the normal axis to be applied to the reconstructed patch points in the current patch. The value of Patch3dShiftNormalAxis[frmIdx][p] can be between 0 and Min(2 pfh_3d_shift_normal_axis_bit_count_minus1[frmIdx]+1 ,2 gps_geometry_3d_coordinates_bitdepth_minus1+1 -1), including all values ​​and ranges therebetween.

[0498] In some embodiments, PatchNormalAxis[frmIdx][p] is the index of the normal of the projection plane of the current patch. The value of PatchNormalAxis[frmIdx][p] can be in the range of 0 to 2, including all values ​​and ranges therebetween.

[0499] In some embodiments, PatchOrientationIndex[frmIdx][p] specifies the patch orientation index of the current patch.

[0500] In some embodiments, PatchLod[frmIdx][p] specifies the LOD scaling factor to be applied to the current patch. The reconstructed point 3D position of the current patch may be scaled by 2 after it is projected from 2D and before any further transformations are applied. pdu_lod[frmIdx][p] .

[0501] In some embodiments, PatchProjectionMode[frmIdx][p] equal to 0 may specify the plane onto which the current patch is to be projected.

[0502] In some embodiments, if pfdu_patch_mode[frmIdx][p] is equal to I_INTRA or P_INTRA, a process for decoding intra-coded patches may be used, with frmIdx and p as inputs to this process and the output of this process used as output.

[0503] In some embodiments, if pfdu_patch_mode[frmIdx][p] is equal to P_INTER, a process for decoding inter-coded patches in part 0 is used, with frmIdx and p as inputs to this process and the output of this process is used as output.

[0504] In some embodiments, if pfdu_patch_mode[frmIdx][p] is equal to I_PCM or P_PCM, a process for decoding PCM coded patches may be used, with frmIdx and p as inputs to this process and the output of this process used as output.

[0505] Referring now to a process for decoding a patch unit coded in intra mode, the input to this process may include a current patch frame index frmIdx and / or a current patch index p.

[0506] In some embodiments, given a parsed element in a patch data unit, one or more of the following patch-related variables may be first assigned:

[0507] Patch2dShiftU[frmIdx][p]=pdu_2d_shift_u[frmIdx][p]*PatchPackingBlockSize

[0508] Patch2dShiftV[frmIdx][p]=pdu_2d_shift_v[frmIdx][p]*PatchPackingBlockSize

[0509] Patch3dShiftTangentAxis[frmIdx][p]=pdu_3d_shift_tangent_axis[frmIdx][p]

[0510] Patch3dShiftBiTangentAxis[frmIdx][p]=pdu_3d_shift_bitangent_axis[frmIdx][p]

[0511] Patch3dShiftNormalAxis[frmIdx][p]=pdu_3d_shift_normal_axis[frmIdx][p]

[0512] PatchNormalAxis[frmIdx][p]=pdu_normal_axis[frmIdx][p]

[0513] PatchOrientationIndex[frmIdx][p]=pdu_orientation_index[frmIdx][p]

[0514] PatchLod[frmIdx][p]=pdu_lod[frmIdx][p]

[0515] PatchProjectionMode[frmIdx][p]=pdu_projection_mode[frmIdx][p]

[0516] Patch45degreeProjectionRotationAxis[frmIdx][p]=pdu_45degree_projection_rotation_axis[frmIdx][p]

[0517] In some embodiments, the variables Patch2dSizeU[frmIdx][p] and Patch2dSizeV[frmIdx][p] may be derived, such as shown below:

[0518] If p is equal to 0, then:

[0519] Patch2dSizeU[frmIdx][p]=pdu_2d_delta_size_u[frmIdx][p]*PatchPackingBlockSize

[0520] Patch2dSizeV[frmIdx][p]=pdu_2d_delta_size_v[frmIdx][p]*PatchPackingBlockSize

[0521] Otherwise, if (p>0), then:

[0522] Patch2dSizeU[frmIdx][p]=Patch2dSizeU[frmIdx][p–1]+pdu_2d_delta_size_u[frmIdx][p]*PatchPackingBlockSize

[0523] Patch2dSizeV[frmIdx][p]=Patch2dSizeV[frmIdx][p–1]+pdu_2d_delta_size_v[frmIdx][p]*PatchPackingBlockSize

[0524] In some embodiments, the variables PatchTangentAxis[frmIdx][p] and PatchBiTangentAxis[frmIdx][p] may be derived, such as shown below:

[0525] If PatchNormalAxis[frmIdx][p] is equal to "zero", PatchTangentAxis[frmIdx][p] is assigned a value of 2 and PatchBiTangentAxis[frmIdx][p] is assigned a value of 1.

[0526] Otherwise, if PatchNormalAxis[frmIdx][p] is equal to 1, PatchTangentAxis[frmIdx][p] is assigned a value of 2 and PatchBiTangentAxis[frmIdx][p] is assigned a value of 0.

[0527] Otherwise (PatchNormalAxis[frmIdx][p] is equal to 2), PatchTangentAxis[frmIdx][p] is assigned the value 0, and PatchBiTangentAxis[frmIdx][p] is assigned the value 1.

[0528] Signaling Subsample Item Description

[0529] Modern multi-stream encoders such as the Video-based Point Cloud Codec (V-PCC) include multiple independently coded media and metadata streams, such as attribute, geometry, occupancy, and patch data streams. A V-PCC elementary stream consists of many different video coding and metadata streams that may have different coding schemes, codecs, and / or configurations.

[0530] As described above and in U.S. Provisional Patent Application No. 62 / 853,903 filed on May 29, 2019 (hereinafter referred to as the "'903 Provisional Patent", the entire contents of which are incorporated herein by reference for all purposes), the entire elementary stream can be stored in a single track without demultiplexing to multiple tracks. Methods, apparatus, and computer program products for signaling and storing compressed point clouds in video coding are described above and in the '903 Provisional Patent. The methods, apparatus, and computer program products can be used in conjunction with a variety of video formats. With respect to encoding of compressed point clouds, the methods, apparatus, and computer program products access a point cloud compressed coded bitstream and store the point cloud compressed coded bitstream. The point cloud compressed coded bitstream includes one or more V-PCC parameter sets, a video coded attribute bitstream, a video coded occupancy and geometry bitstream, and a non-video coded patch data bitstream. Texture can be one of the attributes and is encoded as an attribute bitstream. All bitstreams can be divided into smaller units, which are called V-PCC units, with a specific unit header. Regarding decoding of compressed point clouds, the method, apparatus and computer program product receive a point cloud compression encoded bit stream and decode the point cloud compression encoded bit stream.

[0531] Such V-PCC coded elementary streams can be stored as multiple tracks in an ISOBMFF file, or as a single track. When they are stored as a single media track, the access units of many different encoders are interleaved in the manner defined by the V-PCC specification, and they are grouped into "V-PCC access units". One or more V-PCC access units correspond to samples represented by a single track.

[0532] It is currently not possible to signal the underlying video codec used in a single track V-PCC storage, as the sample entries of such a single track only declare the high-level V-PCC configuration. The underlying codec is signaled using a specific SEI message or understood by parsing the necessary video NAL units (e.g., video-specific SPS).

[0533] This means that the video decoder cannot be initialized until the V-PCC client receives and parses the subsamples, which is undesirable, not least because many devices want to know if they can decode the stream before actually "downloading" the stream (e.g., by inspecting the DASH MPD or initialization segment and looking at the sample entries).

[0534] In V-PCC, subsamples of a V-PCC sample can have subsamples in any order. Therefore, there is no pattern that can be used to presuppose the order of subsamples.

[0535] In a more general sense, ISOBMFF does not lack support for declaring complex and modern media coding use cases where a "sample" of a track consists of multiple independently coded (and possibly different) media types.

[0536] Thus, example embodiments of devices, systems, apparatus, computer program products, and methods for declaring such information by using subsamples and introducing mechanisms for declaring the type and code parameters of such subsamples are provided herein. In addition, methods, devices, and computer program products are described herein for efficiently interleaving multiple separately encoded media types into a single track (and a single sample) of an ISOBMFF file. This approach provides many benefits, for example, in a V-PCC coding scheme, where a video-based point cloud representation typically includes multiple independently encoded media bitstreams. In this case, one skilled in the art would be motivated to create a multi-track ISOBMFF file, such as described above. However, it is typically easier, more efficient, more economical, less computationally complex, etc. to store the entire elementary stream without demultiplexing it to multiple tracks. A method for storing and signaling a compressed point cloud is described in the '903 provisional patent, wherein the entire elementary stream is stored in a single track without being demultiplexed to multiple tracks.

[0537] However, this single track approach may require the type of the subsample to be declared in the single track metadata level so that the decoder can be initialized correctly and the V-PCC decoder can work properly to decode a single track of the encoded media. For media tracks, such declarations are usually done in the sample entry within the SampleTableBox, and they only cover a single media type. In other words, there is currently no method for declaring different media types (such as two independently encoded video streams).

[0538] This article describes approaches, methods, systems, apparatus, and computer program products for declaring multiple separately coded media types to be efficiently interleaved into a single track (and a single sample) in an ISOBMFF file. A subsample entry is provided that can be used to declare subsamples of multiple different media types in a single media track.

[0539] In various embodiments, a subsample entry is associated with a subsample sequence within a sample sequence. The semantics of a subsample entry may mean that if the subsample sequence is encapsulated in its own track (no other subsamples or other additional header data (such as a V-PCC unit header) is present in the track), then the sample sequence in the track will conform to the same sample entry as the subsample entry.

[0540] This can be done in a number of different ways:

[0541] Option (1): A track-level SubSampleDescriptionBox may be defined in which all subsample entry types are declared in an indexed manner (e.g., similar to sample entries in ISOBMFF). The SubSampleInformationBox may be extended to indicate the subsample entry index for each subsample. For example, the SubSampleDescriptionBox may be defined as follows:

[0542] Sub-Sample Description box

[0543] definition

[0544] Box type: 'ssdb'

[0545] Container: SampleTableBox

[0546] Mandatory: No

[0547] Quantity: Exactly 1

[0548]

[0549]

[0550] The SubSampleEntry abstract class can be instantiated with the actual SampleEntry() format of the individual decoded media bitstream. For example, for H.265 video coding samples, the format field can be one of the sample entry types defined in ISO / IEC 14496-15 ('hvc1', 'hev1', 'hvcC', etc.), and the SubSampleEntry can be a HEVCSampleEntry() as defined in ISO / IEC 14496-15.

[0551] data_reference_index may be set to 0 ("zero") to indicate that it is not relevant and may be omitted.

[0552] entry_count may be set to the number of sample entries used in a subsample of a media sample. For example, if there are three media streams interleaved as access units, there may be at least three sample entries. For example, if a media stream uses multiple sample entries, there may be more than three sample entries.

[0553] Extend SubSampleInformationBox

[0554] In some embodiments, it is necessary to indicate which subsample description entry refers to a subsample. This can be done by extending SubSampleInformationBox as follows:

[0555]

[0556] Here, the same subsample_count value is used in two different loops. This ensures that the number of entries in the subsample and subsample description is the same and they match one-to-one.

[0557] subsample_description_entry_index indicates the index of the subsample description entry as listed in SubSampleDescriptionBox.

[0558] In other embodiments, a flag may be used to signal the presence of a subsample description entry index. In such embodiments, instead of using a version, a flag value may be used instead to signal the presence of a subsample description entry index. In some embodiments, such as in the syntax example provided above, the line "if (version == 2)" may be replaced with "if (flags & 1)".

[0559] In other embodiments, the use of versions or flags may be omitted and the following data structure may be inserted directly:

[0560]

[0561] In another embodiment that can be applied together with or independently of the other embodiments described above, a single subsample_description_entry_index value set is provided and applied to each sample within the scope of the container box. This embodiment can be useful at least when the container box is a TrackFragmentBox. For example, the following syntax can be used:

[0562]

[0563] In some embodiments, the above structure is backwards compatible in that old parsers can safely ignore the new data structure.

[0564] In another embodiment, the data structure may be non-backward compatible, such as shown below:

[0565]

[0566]

[0567] In some embodiments, a flag may also be used instead of a version, however a possible disadvantage of this approach is that it is not backwards compatible and may break existing older parsers.

[0568] In another embodiment, the codec_specific_parameters field of the SubSampleInformationBox may be used to directly signal the subsample description index.

[0569] Option (2): A new box (SubSampleDescriptionIndexBox) may be defined that contains the subsample description entry index within the same container as the SubSampleInformationBox. In some embodiments, the method may require that there be one and only one SubSampleInformationBox in the same container box. In some embodiments, if there are multiple SubSampleInformationBoxes in the same container box, further components may be required that are operable to associate these SubSampleDescriptionIndexBoxes with the corresponding SubSampleInformationBoxes, for example, by adding an identifier field in both the SubSampleInformationBox and the SubSampleDescriptionIndexBox, and associating them together when the values ​​of the identifier are equal.

[0570] SubSampleDescriptionIndexBox:

[0571] Frame type: 'ssdi'

[0572] Container: SampleTableBox or TrackFragmentBox

[0573] Mandatory: No

[0574] Quantity: zero or more

[0575]

[0576] Among them, entry_count and subsample_count have the same values ​​as in SubSampleInformationBox.

[0577] In an alternative syntax, a sample run is associated with a given subsample description entry index. For example, the following syntax can be used:

[0578]

[0579]

[0580] where sample_run specifies the number of samples to which the immediately following set of subsample description entry indices applies.

[0581] In another embodiment with an alternative syntax, a single set of subsample_description_entry_index values ​​is provided and applied to each sample within the scope of the container box.For example, in some embodiments, this method may be relatively more useful when the container box is a TrackFragmentBox.

[0582] Option (3): In some embodiments, Option 3 may be applied for movie fragments, while Option 2 may be applied when the container is a SampleTableBox. In some embodiments, according to Option 3, TrackFragmentHeaderBox may be extended to contain a set of subsample description entry indices that apply to each sample of the track fragment. For example, in some embodiments, the following syntax may be used.

[0583]

[0584] Option (4): The use of track-level sample entries could be extended so that they could also index subsamples. For example, the number of sample entries as required by the file could be declared as SampleEntry() in the SampleDescriptionBox. The SubSampleInformationBox could be enabled to reference the index of the SampleEntry() in the track's SampleDescriptionBox.

[0585] Option (5): In a segmented MP4 file, the subsamples of a movie fragment can be grouped into subsample group descriptions. These descriptions can reside in a new box structure at the SampleTableBox or TrackFragmentBox level. Such subsample group descriptions can be further assigned to subsamples via the SubsampleToGroupMapping box.

[0586] In various embodiments of options 1-5, a subsample entry is associated with a subsample sequence within a sample sequence. The semantics of a subsample entry may mean that if a subsample sequence is encapsulated in its own track without other subsamples or other additional header data (such as a V-PCC unit header) present in the track, then the sample sequence in the track will conform to the same sample entry as the subsample entry. The subsample entry type may be indexed at the track-level subsample description box, and the subsample information box may be extended to indicate the subsample entry index for each subsample. The subsample information box may be extended to include a version or flag that identifies the presence of a subsample description entry index. A subsample description index box may be formed for including the subsample description entry index in the same container as the subsample information box. A track fragment header box for a movie fragment track may be extended to include a set of subsample description entry indexes that apply to each sample of the track fragment. Alternatively, the sample entry at the track level may be extended to index the subsamples in the sample description box, and the subsample information box may reference the index of the sample entry at the track-level sample description box.

[0587] Reference now Figure 4, a method 30 is provided, which includes accessing a point cloud compression coded bitstream at block 31, the point cloud compression coded bitstream being configured to be stored as one or more media tracks. The method 30 shown further includes: at block 32, forming a track-level subsample description box, the track-level subsample description box including a plurality of subsample entry types for corresponding plurality of subsamples. The method 30 shown further includes: at block 33, indicating one or more types of point cloud compression coded bitstream components by including corresponding point cloud unit header information in a codec-specific parameter-related field of the track-level subsample description box. In some embodiments, the method 30 may optionally further include: at block 34, extending the subsample information box to indicate the subsample entry index by indicating the order of existence of the subsample entry in the relevant subsample description box, and further indicating its existence via a version value or setting a flag in the subsample information box. In some embodiments, at least one of the at least one media track having a first packaging type and at least one other media track having a second packaging type, the first packaging type and the second packaging type are signaled via the subsample entry index. In some embodiments, the method may further include: accessing an attribute information bitstream, a geometry information bitstream, an occupancy information bitstream, and a patch information bitstream; detecting that the attribute information bitstream, the geometry information bitstream, the occupancy information bitstream, and the patch information bitstream follow a compatible coding and timing structure; and constructing and storing a video media track based at least on the compatible coding and timing structure. In some embodiments, the point cloud compression coded bitstream includes at least one attribute information bitstream, at least one geometry information bitstream, at least one occupancy information bitstream, and at least one patch information bitstream. In some embodiments, the method may further include: accessing the attribute information bitstream, the geometry information bitstream, the occupancy information bitstream, and the patch information bitstream; storing the attribute information bitstream in an attribute video media track; storing the geometry information bitstream in a geometry video media track; storing the occupancy information bitstream in an occupied video media track; storing the patch information bitstream in another coded media or metadata track; and constructing and storing interleaving indicator metadata in the file, the interleaving indicator metadata indicating an interleaving mode for one or more components of the attribute video media track, the geometry video media track, the occupied video media track, and the separate coded media or metadata track. In some embodiments, each of the one or more components for which an interleaving mode is indicated may be accessed using a SubSampleInformationBox, which defines order and configuration information for the one or more components.In some embodiments, storing includes storing the different video-based point cloud compression structures in an International Organization for Standardization Base Media File Format (ISOBMFF) or any other object structured media file format.

[0588] In some embodiments, an apparatus for performing any of the methods described herein, such as method 30, may be provided. In some embodiments, the apparatus may include components such as at least one processor and at least one memory including computer program code for one or more programs, the components being configured to cause the apparatus to at least: access a point cloud compression coded bitstream, the point cloud compression coded bitstream being configured to be stored as one or more media tracks; form a track-level subsample description box including a plurality of subsample entry types for a corresponding plurality of subsamples; and indicate one or more types of point cloud compression coded bitstream components by including corresponding point cloud unit header information in a codec-specific parameter-related field of the track-level subsample description box. In some embodiments, the apparatus may further include components for causing the apparatus to perform the following operations: extend the subsample information box to indicate the subsample entry index by indicating a version value or setting a flag within the subsample information box.

[0589] In some embodiments, a computer program product includes at least one non-transitory computer-readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions including program code instructions configured to perform the following operations when executed: access a point cloud compression encoded bitstream, the point cloud compression encoded bitstream being configured to be stored as one or more media tracks; form a track-level subsample description box, the track-level subsample description box including a plurality of subsample entry types for corresponding plurality of subsamples; and indicate one or more types of point cloud compression encoded bitstream components by including corresponding point cloud unit header information in a codec-specific parameter-related field of the track-level subsample description box. In some embodiments, the program code instructions may be further configured to, when executed: extend a subsample information box to indicate a subsample entry index by indicating a version value or setting a flag in the subsample information box.

[0590] Reference now Figure 5, method 40 may include: at box 41, accessing a point cloud compression encoded bitstream, the point cloud compression encoded bitstream being configured to be stored as one or more media tracks. In some embodiments, method 40 may further include: at box 42, defining multiple subsample description index boxes, the subsample description index boxes including subsample description entry indexes, the subsample description index boxes being stored in the same container as the corresponding multiple subsample information boxes associated with the multiple subsample description index boxes. In some embodiments, method 40 further includes: at box 43, adding a unique identifier to the corresponding subsample description index boxes of the multiple subsample description index boxes and the corresponding subsample information boxes of the multiple subsample information boxes, the unique identifier being operable to identify a first subsample description index box associated with a first subsample information box.

[0591] Reference now Figure 6 , method 50 may include: at box 51, accessing a point cloud compression encoded bitstream, the point cloud compression encoded bitstream being configured to be stored as one or more media tracks associated with samples of the movie fragment. In some embodiments, method 50 further includes: at box 52, defining a plurality of subsample description index boxes, the subsample description index boxes including subsample description entry indexes, the subsample description index boxes being stored in the same container as the corresponding plurality of subsample information boxes associated with the plurality of subsample description index boxes. Method 50 further includes: at box 53, extending a track fragment header box to include a set of the subsample description entry indexes applied to the samples of the movie fragment.

[0592] Reference now Figure 7 The method 60 may include: at 61, accessing a point cloud compression encoded bitstream, the point cloud compression encoded bitstream being configured to be stored as one or more media tracks associated with samples of a movie fragment. The method 60 further includes: at 62, indexing a plurality of sample entities within a sample description box of the one or more media tracks.

[0593] According to other embodiments, methods, apparatus, and / or computer program products are provided for decoding any point cloud video encoded according to any of the embodiments described herein.

[0594] As mentioned above, Figure 4-7Flowcharts of apparatus 10, methods, and computer program products according to certain exemplary embodiments are included. It will be understood that each of the blocks of these flow charts and the combination of blocks in these flow charts can be implemented in various ways, such as hardware, firmware, processors, circuits, and / or other devices associated with the execution of software including one or more computer program instructions. For example, one or more processes described herein can be embodied by computer program instructions. In this regard, the computer program instructions embodying the processes described herein can be stored by a memory device 14 of an apparatus employing an embodiment of the present invention and executed by the processing circuit 12 of the apparatus. It will be understood that any such computer program instructions can be loaded onto a computer or other programmable device (e.g., hardware) to produce a machine, so that the resulting computer or other programmable device implements the functions specified in the flow chart blocks. These computer program instructions can also be stored in a computer-readable memory, which can guide a computer or other programmable device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product, and its execution implements the functions specified in the flow chart blocks. These computer program instructions may also be loaded onto a computer or other programmable device to cause a series of operations to be performed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide operations for implementing the functions specified in the flowchart blocks.

[0595] Thus, in these examples, a computer program product is defined, wherein computer program instructions (such as computer readable program code portions) are stored by at least one non-transitory computer readable storage medium, wherein the computer program instructions (such as computer readable program code portions) are configured to perform the methods described herein (such as in conjunction with the method) when executed. Figure 4-7 In other embodiments, the computer program instructions (such as computer readable program code portions) need not be stored or otherwise embodied by a non-transitory computer readable storage medium, but may be embodied by a transient medium having computer program instructions (such as computer readable program code portions) that are still configured to perform the functions described herein when executed.

[0596] Therefore, the flowchart blocks support the combination of components for performing the specified functions, and the combination of operations for performing the specified functions. It will also be understood that one or more blocks of the flowchart and the combination of blocks in the flowchart can be implemented by a computer system based on special-purpose hardware that performs the specified functions, or a combination of special-purpose hardware and computer instructions.

[0597] In some embodiments, any or all terms, variables, values, constructs, frameworks, etc. as described herein may be set as outputs of this process.

[0598] In some embodiments, some operations described herein may be modified or further expanded. In addition, in some embodiments, additional optional operations may be included, such as Figure 4-7 The operations represented by the dashed boxes in FIG. Modifications, additions or amplifications to the operations herein may be performed in any order and in any combination.

[0599] Those skilled in the art who benefit from the teachings presented in the foregoing description and the associated drawings will think of many modifications and other embodiments of the invention set forth herein. Therefore, it should be understood that the present invention is not limited to the specific embodiments disclosed, and those modifications and other embodiments are intended to be included within the scope of the appended claims. In addition, although the foregoing description and the associated drawings describe exemplary embodiments in the context of certain exemplary combinations of elements and / or functions, it should be understood that different combinations of elements and / or functions can be provided by alternative embodiments without departing from the scope of the appended claims. In this regard, for example, as may be set forth in some of the appended claims, different combinations of elements and / or functions explicitly described herein can also be conceived. Although specific terms are used herein, they are used only in a general and descriptive sense, and not for the purpose of limitation.

Claims

1. A method for storing and signaling a subsample entry description, comprising: accessing a point cloud compressed encoded bitstream, the point cloud compressed encoded bitstream being configured to be stored as one or more media tracks; forming a track-level sample table box, the track-level sample table box comprising a plurality of subsample entry types for a corresponding plurality of subsamples; Expanding the subsample information box to indicate a subsample entry index for each of the corresponding subsample entries within the sample table box; as well as One or more types of point cloud compression coded bit stream components are indicated by including corresponding point cloud unit header information in the codec parameter related field of the subsample information box.

2. The method according to claim 1, further comprising: The subsample information box is extended to indicate the subsample entry index by using a version value or by setting a flag in the subsample information box to indicate that the subsample entry index exists in the subsample information box.

3. The method according to claim 1 or 2, wherein: Storing the point cloud compression encoding bit stream includes: accessing at least one attribute information bitstream, at least one geometry information bitstream, at least one occupancy information bitstream, and at least one patch information bitstream; detecting that the at least one attribute information bitstream, the at least one geometry information bitstream, the at least one occupancy information bitstream, and the at least one patch information bitstream follow compatible coding and timing structures; and A video media track is constructed and stored based at least on the compatible coding and timing structure.

4. The method according to claim 1 or 2, wherein: The at least one media track has a packetization type signaled via the subsample information box or the subsample entry index.

5. The method according to claim 1 or 2, wherein: The point cloud compression encoding bit stream includes at least one attribute information bit stream, at least one geometry information bit stream, at least one occupancy information bit stream, and at least one patch information bit stream.

6. The method according to claim 5, further comprising: accessing the at least one attribute information bitstream, the at least one geometry information bitstream, the at least one occupancy information bitstream, and the at least one patch information bitstream; storing the at least one attribute information code stream in an attribute video media track; storing the at least one geometry information bitstream in a geometry video media track; storing the at least one occupancy information bitstream in an occupied video media track; storing the at least one patch information bitstream in another coded media or metadata track; as well as Metadata is constructed and stored in a file, the metadata indicating a mode of one or more components of the attribute video media track, the geometry video media track, the occupied video media track, and the another coded media or metadata track.

7. An apparatus for storing and signaling a subsample entry description, comprising: means for accessing a point cloud compressed encoded bitstream, wherein the point cloud compressed encoded bitstream is configured to be stored as one or more media tracks; means for forming a track-level sample table box, wherein the track-level sample table box comprises a plurality of subsample entry types for a corresponding plurality of subsamples; means for extending the subsample information box to indicate a subsample entry index for each of the corresponding subsample entries within the sample table box; and A means for indicating one or more types of point cloud compression coded bit stream components by including corresponding point cloud unit header information in a codec parameter related field of the subsample information box.

8. The apparatus according to claim 7, further comprising: means for extending the subsample information box to indicate the subsample entry index by using a version value or by setting a flag in the subsample information box to indicate the presence of the subsample entry index in the subsample information box.

9. The device according to claim 7 or 8, further comprising: means for accessing at least one attribute information bitstream, at least one geometry information bitstream, at least one occupancy information bitstream, and at least one patch information bitstream; means for detecting that the at least one attribute information bitstream, the at least one geometry information bitstream, the at least one occupancy information bitstream and the at least one patch information bitstream follow compatible coding and timing structures; as well as Means for constructing and storing a video media track based at least on said compatible encoding and timing structures.

10. The device according to claim 7 or 8, wherein: The at least one media track has a packetization type signaled via the subsample information box or the subsample entry index.

11. The device according to claim 7 or 8, wherein: The point cloud compression encoding bit stream includes at least one attribute information bit stream, at least one geometry information bit stream, at least one occupancy information bit stream, and at least one patch information bit stream.

12. The apparatus according to claim 11, further comprising: means for accessing the at least one attribute information bitstream, the at least one geometry information bitstream, the at least one occupancy information bitstream, and the at least one patch information bitstream; A component for storing the at least one attribute information code stream in an attribute video media track; means for storing the at least one geometry information bitstream in a geometry video media track; means for storing the at least one occupancy information bitstream in an occupied video media track; means for storing said at least one patch information bitstream in another coded medium or metadata track; as well as Means for constructing and storing metadata in a file, wherein the metadata indicates a mode of one or more components of the attribute video media track, the geometry video media track, the occupied video media track, and the another coded media or metadata track.

13. The device according to claim 12, wherein: The metadata is interleaving indicator metadata indicating an interleaving pattern of the one or more components of the attribute video media track, the geometry video media track, the occupied video media track, and the another coded media or metadata track.

14. The device according to claim 12, wherein: Each of the one or more components for which the mode is indicated is accessible using the subsample information box, the subsample information box defining a presentation order and configuration information for the one or more components, the subsample information box being configured to indicate the mode.

15. The device according to claim 12, wherein: Storage includes: Different video-based point cloud compression structures are stored in the International Organization for Standardization Base Media File Format (ISOBMFF).

Images