Synchronization of decoded frames before point cloud reconstruction

Abstract

A method and a video-based point cloud compression (V‑PCC) decoder for synchronizing decoded frames before point cloud reconstruction are provided. A V‑PCC bitstream including encoded frames associated with a point cloud sequence is received. A substream of the received V‑PCC bitstream is decoded by a set of video decoders of the V‑PCC decoder to generate V‑PCC components, such as attribute components, geometry components, occupancy map components, and atlas components. Release of the attribute components, geometry components, occupancy map components, and atlas components to a reconstruction unit is delayed based on a first output delay, a second output delay, a third output delay, and a fourth output delay, respectively. The delayed release synchronizes the attribute components, geometry components, occupancy map components, and atlas components with each other before the reconstruction unit reconstructs the point cloud based on the V‑PCC components.

Description

[0001] Citation / Incorporation by Reference of Related Applications

[0002] This application claims priority to U.S. Provisional Patent Application Serial No. 62 / 912,928, filed on October 9, 2019, the entire contents of which are incorporated herein by reference. Technical Field

[0003] Various embodiments of the present disclosure relate to video-based point cloud compression (V-PCC). More specifically, various embodiments of the present disclosure relate to synchronization of decoded frames before point cloud reconstruction. Background Art

[0004] Advances in the field of three-dimensional (3D) scanning technology provide the ability to create dynamic and static 3D geometric representations of 3D objects. Dynamic 3D point clouds are an example of 3D geometric representations that are increasingly used in different applications, such as free viewpoint video (FVV) for sports or live broadcasts, geographic information systems, cultural heritage representations, or autonomous navigation of vehicles. Typically, a dynamic 3D point cloud includes many point cloud frames, where each point cloud frame includes a large number of unstructured 3D points (e.g., each point has X, Y, and Z coordinates) and associated attributes, such as textures including color or reflectivity. The 3D image of the object described by the 3D point cloud may contain multiple orders of magnitude of raw data, which makes data storage, processing, and transmission a challenging task for any practical application. One of the techniques for encoding dynamic 3D point clouds is called video-based point cloud compression (V-PCC), where a conventional video codec is used to encode the geometry and texture of the dynamic 3D point cloud. It may be desirable to efficiently decompress dynamic 3D point clouds.

[0005] Other limitations and disadvantages of conventional traditional approaches will become apparent to those skilled in the art by comparing the described systems with aspects of the present disclosure, as described in the remainder of this application and with reference to the accompanying drawings. Summary of the invention

[0006] As more fully described in the claims, there is provided a video-based point cloud compression (V-PCC) decoder and method for synchronizing decoding frames prior to point cloud reconstruction, substantially as shown in and / or described in conjunction with at least one of the accompanying drawings.

[0007] These and other features and advantages of the present disclosure will be understood from the following detailed description of the disclosure and the accompanying drawings, in which like reference numerals represent like parts. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1is a diagram of an exemplary video-based point cloud compression (V-PCC) decoder for synchronizing decoded frames prior to point cloud reconstruction in accordance with an embodiment of the present disclosure.

[0009] Figure 2 is a diagram of the architecture of a V-PCC decoder according to an embodiment of the present disclosure, illustrating the implementation of an exemplary method for synchronizing decoded frames prior to point cloud reconstruction.

[0010] Figure 3A is a diagram illustrating an example of sub-bitstreams according to an embodiment of the present disclosure, each sub-bitstream containing a single V-PCC component map.

[0011] Figure 3B is a diagram illustrating an example of a single sub-bitstream including an interlaced map according to an embodiment of the present disclosure.

[0012] Figure 4 is a diagram illustrating a hypothetical reference decoder (HRD) for an atlas frame in accordance with at least one embodiment of the present disclosure.

[0013] Figure 5 According to the embodiment of the present disclosure Figure 1 Block diagram of the V-PCC decoder.

[0014] Figure 6 is a flow chart illustrating exemplary operations for synchronizing decoded frames prior to point cloud reconstruction in accordance with an embodiment of the present disclosure. DETAILED DESCRIPTION

[0015] The implementation described below may be found in the disclosed method for synchronizing decoded frames prior to point cloud reconstruction and a video-based point cloud compression (V-PCC) decoder. The present disclosure provides two separate methods for synchronizing decoded frames of different V-PCC components, which are typically not aligned in time due to different group of pictures (GOP) structures. In order to enable a reconstruction unit of a V-PCC decoder to reconstruct a single frame of a point cloud sequence, a conformance point at the output of each video decoder of the V-PCC decoder may require that all V-PCC components be aligned in time before transmitting all V-PCC components to the reconstruction unit. All V-PCC components may be required to reconstruct a full point cloud frame of a point cloud sequence. If these components are not aligned, it may be difficult to meet another conformance point requirement at the output of the reconstruction unit.

[0016] In the present disclosure, two separate methods are provided to temporally align V-PCC components, such as attribute frames, geometry frames, occupancy maps, and atlas frames. In the first method, each video decoder can be configured to delay the release of the corresponding V-PCC component from the decoded picture / atlas buffer of the corresponding decoder so that all V-PCC components are temporally aligned before all components are transmitted to the reconstruction unit for reconstructing the point cloud. In the second method, a group of video decoders can be allowed to output V-PCC components according to their GOP structure, but an external buffer can be used to hold these components for a period of time. When all required V-PCC components are in the external buffer, these components can be removed from the external buffer and released to the reconstruction unit for reconstruction of the point cloud. In this method, the video decoder may not be changed, but additional buffer management may be required. The disclosed method ensures that the decoded frame or V-PCC component meets the requirements for point cloud frame reconstruction, and also provides extensive flexibility in the reconstruction of the point cloud frame.

[0017] Figure 1 is a diagram of an exemplary video-based point cloud compression (V-PCC) decoder for synchronizing decoded frames before point cloud reconstruction according to an embodiment of the present disclosure. Figure 1 , a block diagram 100 of a video-based point cloud compression (V-PCC) decoder 102 is shown. The V-PCC decoder 102 may include a set of video decoders 104 and a reconstruction unit 106. The V-PCC decoder 102 may also include other components (such as a color reconstruction unit), which are omitted in the present disclosure for the sake of brevity.

[0018] The V-PCC decoder 102 may include suitable logic, circuitry, interfaces and / or code that may be configured to receive a V-PCC bitstream 108 from a data source such as a media storage device. The V-PCC bitstream 108 may include coded frames associated with a point cloud sequence. The V-PCC decoder 102 may decode the received V-PCC bitstream 108 and may perform frame-by-frame reconstruction of the point cloud sequence. The frame-by-frame reconstruction process may be repeated until the entire point cloud sequence is reconstructed.

[0019] In an embodiment, the V-PCC decoder 102 may be implemented on a graphics processing unit (GPU), a central processing unit (CPU), a reduced instruction set computing (RISC) processor, an application specific integrated circuit (ASIC) processor, a complex instruction set computing (CISC) processor, a coprocessor, or a combination thereof. In another embodiment, the V-PCC decoder 102 may be implemented as a dedicated hardware decoder. Examples of dedicated hardware decoders may include (but are not limited to) a field programmable gate array (FPGA), a programmable logic device (PLD), an application specific integrated circuit (ASIC), a programmable ASIC (PL-ASIC), an application specific integrated component (ASSP), and a system on a chip (SOC) based on a standard microprocessor (MPU) or a digital signal processor (DSP). In another embodiment, the V-PCC decoder 102 may be implemented as a combination of stored programmable instructions and a dedicated or general decoder circuit.

[0020] Each of the group of video decoders 104 may include appropriate logic, circuits, interfaces and / or codes that can be configured to decode a substream of the V-PCC bitstream 108 to generate a V-PCC component. By decoding the V-PCC bitstream 108, the group of video decoders 104 can generate V-PCC components, such as attribute components (e.g., attribute frames), geometry components (e.g., geometry frames), occupancy map components (e.g., occupancy maps), and atlas components (e.g., atlas frames). In an embodiment, each of the group of video decoders 104 can be implemented based on a standard video codec (e.g., a high-efficiency video coding (HEVC) codec or an advanced video coding (AVC) codec. In another embodiment, each of the group of video decoders 104 can be implemented based on a non-standard or dedicated video codec.

[0021] The reconstruction unit 106 may include appropriate logic, circuitry, interfaces and / or code that may be configured to perform frame-by-frame reconstruction of a point cloud sequence. For example, the reconstruction unit 106 may receive at least one unit size of the generated V-PCC component that may be required to reconstruct a full frame (i.e., a 3D point cloud) of the point cloud sequence. The reconstruction unit 106 may be implemented as a general or dedicated hardware component of the V-PCC decoder 102, stored programmable instructions, or one of their combinations.

[0022] In general, a V-PCC encoder can encode a point cloud sequence to generate a V-PCC bitstream 108. The V-PCC encoder encodes the point cloud sequence by creating a video of the point cloud properties (e.g., texture), geometry, occupancy map, and encoding the video with a video encoder. V-PCC defines a stream structure of metadata information indicating parameters of the point cloud sequence and other relevant information, such as patch size and position.

[0023] For encoding, volume data (i.e., point clouds) can be segmented into regions of connected components called 3D facets. Applying an orthogonal projection along the x-axis, y-axis, or z-axis to a plane to each 3D facet can result in a set of 2D facet points. As part of an atlas component, an atlas frame can refer to a collection of two-dimensional (2D) bounding boxes, which can be projections of their corresponding 3D facet bounding boxes. Packing can refer to placing 2D bounding boxes within each atlas frame by applying a series of transformations (e.g., rotations and translations). Typically, an atlas frame can provide information about the coordinate transformations applied during the 3D to 2D facet generation and packing process.

[0024] From each point cloud in the point cloud sequence, the V-PCC encoder can extract several components. For example, the V-PCC encoder can extract four components, such as binary images, called occupancy maps, geometry frames, attribute frames, and atlas frames. The geometry frame can carry information about the third coordinate (i.e., the distance between the 3D point of the point cloud and the projected 2D point). Similarly, the attribute frame can be used to carry attribute information associated with each point, such as color or reflectivity information. Since a 3D surface element may have multiple points that can all be projected to the same 2D point, several components can be extracted to keep a portion of overlapping points (e.g., near images versus far images to store inner surface points versus outer surface points). The occupancy map can be a binary image, and a pixel intensity of 1 can indicate the presence of at least a single valid point cloud. The atlas frame can include information such as the position and orientation of the surface element, or the block size used in the surface element packing operation. Such information may be required to reconstruct the point cloud of the point cloud sequence.

[0025] In operation, the V-PCC decoder 102 can receive a V-PCC bitstream 108, which can include coded frames associated with a point cloud sequence. In an embodiment, the V-PCC decoder 102 can apply a demultiplexing operation to divide the V-PCC bitstream 108 into substreams. The V-PCC bitstream 108 can include substreams, such as an attribute substream, a geometry substream, an occupancy graph substream, and an atlas substream. Afterwards, each substream can be input to a corresponding decoder in a group of video decoders 104. The group of video decoders 104 can decode the substreams of the V-PCC bitstream 108 to generate V-PCC components, such as an attribute component, a geometry component, an occupancy graph component, and an atlas component.

[0026] V-PCC does not enforce the underlying video encoder to have the same group of pictures (GOP) structure, which may mean that the video frames of the various V-PCC components on the encoder side may be encoded in different orders. In order for a system such as the V-PCC decoder 102 to be able to reconstruct a single frame of a point cloud sequence, a V-PCC unit must be transmitted to the reconstruction unit 106. Here, a V-PCC unit may include V-PCC components that may be required to reconstruct a full point cloud frame of a point cloud sequence. However, a key challenge in the reconstruction process may be related to the need for temporal alignment of the decoding components (i.e., V-PCC components). This may be due to the lack of constraints on the prediction structure across V-PCC substreams, which may result in different frame durations and encoding orders.

[0027] In a basic profile of V-PCC, all elements of a substream may be required to be aligned in time; however, in other profiles, the temporal structure may be different for the sub-bitstreams. This may cause problems with output delay adjustment on the video decoder portion of the V-PCC decoder 102. For example, the V-PCC decoder 102 may include a consistent point "A" at the output of a set of video decoders 104 to ensure output delay adjustment of the V-PCC components.

[0028] Before all V-PCC components are released to the reconstruction unit 106 in order to reconstruct a full point cloud frame of the point cloud sequence, all V-PCC components may have to be aligned in time. Due to the different GOF structures associated with the coded frames within each substream of the V-PCC bitstream 108, these components are usually not aligned in time. In the present disclosure, two separate methods are mentioned to synchronize V-PCC components (such as attribute components, geometry components, occupancy map components, and atlas components). The key difference between these two methods is whether the delay adjustment or the modifications required for synchronization are performed at the decoded picture / atlas buffer or at the V-PCC synthesis buffer level (i.e., the external unit buffer used to hold the V-PCC components before reconstruction).

[0029] In a first approach, after each video decoder in a set of video decoders 104 decodes a corresponding V-PCC substream, the V-PCC decoder 102 may delay releasing each corresponding V-PCC component (i.e., one or more frames decoded from the corresponding V-PCC substream) from a decoded picture buffer / decoded atlas buffer (DPB / DAB) associated with the corresponding video decoder. Such a delay may be applied so that all V-PCC components are aligned in time when they can be provided to the reconstruction unit 106 for reconstruction of the point cloud. Details associated with the first approach are described, for example, in Figure 2 Further provided in.

[0030] In a second approach, a group of video decoders 104 may be allowed to output V-PCC components, such as attribute components, geometry components, occupancy map components, and atlas components, according to their GOP structure, but may use a V-PCC composition buffer (e.g., Figure 2 ) keeps all components for a period of time. When all required V-PCC components are in the V-PCC synthesis buffer (ie, the external buffer), these components can be removed from the V-PCC synthesis buffer and released to the reconstruction unit 106 for reconstruction of the point cloud of the point cloud sequence. Details associated with the second method are described, for example, in Figure 2 Further provided in.

[0031] In both methods, the V-PCC decoder 102 may be configured to delay the release of the attribute component to the reconstruction unit 106 based on the first output delay. Similarly, the V-PCC decoder 102 may be configured to delay the release of the geometry component to the reconstruction unit 106 based on the second output delay. The V-PCC decoder 102 may also be configured to delay the release of the occupancy map component to the reconstruction unit 106 based on the third output delay, and to delay the release of the atlas component to the reconstruction unit 106 based on the fourth output delay.

[0032] Each of the first output delay, the second output delay, the third output delay, or the fourth output delay may be specified in a timing message such as a picture / atlas timing supplemental enhancement information (SEI) message. Such a message may be included in a separate access unit (AU) as part of a corresponding substream of the received V-PCC bitstream 108. The release of the delay may synchronize (i.e., time-align) the V-PCC components (such as attribute components, geometry components, occupancy map components, and atlas components) with each other, and then the reconstruction unit 106 reconstructs the point cloud based on these (time-aligned) V-PCC components.

[0033] Figure 2is a diagram of the architecture of a V-PCC decoder according to an embodiment of the present disclosure, illustrating the implementation of an exemplary method for synchronizing decoded frames prior to point cloud reconstruction. Figure 1 To illustrate the elements in Figure 2 . refer to Figure 2 , a block diagram 200 of the architecture of a V-PCC decoder 102 for implementing an exemplary method of synchronizing (i.e., temporally aligning) various V-PCC components that are typically required to reconstruct a full frame of a point cloud sequence is shown. In the block diagram 200, a demultiplexer 202 and a set of video decoders 104 communicatively coupled to the demultiplexer 202 are shown. The set of video decoders 104 includes an attribute video decoder 204, a geometry video decoder 206, an occupancy map decoder 208, and an atlas frame decoder 210. For the attribute video decoder 204, a first coded picture buffer (CPB) 204A and a first decoded picture buffer (DPB) 204B are shown. For the geometry video decoder 206, a second CPB 206A and a second DPB 206B are shown. For the occupancy map decoder 208, a third CPB 208A and a third DPB 208B are shown. For the atlas frame decoder 210, a coded atlas buffer (CAB) 210A and a decoded atlas buffer (DAB) 210B are shown.

[0034] Also shown in the figure is a V-PCC synthesis buffer 220, a reconstruction unit 106, and two consistency points (A, B). Consistency point A may require output order consistency and output timing consistency at the output of a set of video decoders 104. Similarly, consistency point B may require correct point cloud reconstruction at the output of the reconstruction unit 106 of the V-PCC decoder 102. CPB and DPB are typically implemented with a video decoder (such as a HEVC decoder). Therefore, for the sake of brevity, the details of CPB and DPB are omitted in this disclosure.

[0035] In operation, demultiplexer 202 can receive V-PCC bitstream 108, and can demultiplex the received V-PCC bitstream 108 into different substreams, such as attribute substream, geometry substream, occupancy graph substream and atlas substream. In order to generate each V-PCC component, V-PCC decoder 102 can transmit the coded AU from attribute substream, geometry substream, occupancy graph substream and atlas substream to the first CPB 204A, the second CPB 206A, the third CPB 208A and CAB 210A respectively. From each CPB / CAB, corresponding video decoder can receive corresponding coded AU, and can decode coded AU to generate V-PCC component. After decoding, each such V-PCC component can be transmitted to the corresponding DPB / DAB of V-PCC decoder 102.

[0036] An exemplary first method for output delay adjustment (or time alignment) of a V-PCC component is described here. Conventionally, a hypothetical reference decoder (HRD) in a video codec such as H.264 / AVC or H.265 / HEVC refers to a hypothetical buffer model. The parameters of the HRD may provide a means by which the encoder may signal to the decoder the time at which the coded access unit (AU) bitstream will be available for decoding, and the time at which the decoded AU may then be output, i.e., its synthesis time. Such parameters may include, for example, bitrate, CPB / CAB size, initial CPB / CAB removal delay, and timing information that may be carried by V-PCC video availability information (VUI). SEI messages associated with HRD, such as picture buffering period, atlas buffering period, picture timing, and atlas frame timing, may provide the timing information required to extract CPB and CAB AU removal times and DPB and DAB output synthesis times. Note that the parameters associated with the HRD may be embedded in the V-PCC bitstream 108, or may be provided by external means (e.g., by system-level information).

[0037] The composition time of a decoded AU (or decoded frame) of a V-PCC component may be defined as the removal time (remTime[n][i]) of the coded AU (contained in a substream of the V-PCC bitstream 108) plus the output delay outDelay[n][i], where n and i represent the nth AU of the i-th V-PCC component, respectively. For each V-PCC component such as the attribute component, geometry component, and occupancy map component, the corresponding output delay may be extracted from the syntax elements (e.g., pic_cpb output_delay) in the picture timing SEI message. For a V-PCC component such as the atlas component, the output delay may be determined from the syntax elements (e.g., aft_cab_output_delay_minus1) in the atlas timing SEI message. It may be possible to modify the composition time based on the new output delay value. A first exemplary method is to derive an appropriate delay offset that may ensure that the decoded frames in all V-PCC components are properly aligned before they are input to the V-PCC composition buffer 220 of the AU according to their modified compositionTime[n][i].

[0038] In an embodiment, the V-PCC decoder 102 may receive a first picture timing SEI message associated with an attribute substream of the V-PCC bitstream 108. In order to delay the release of the attribute component 212 (i.e., the first V-PCC component) with respect to other V-PCC components, the V-PCC decoder 102 may extract a first output delay from the received first picture timing SEI message, and may determine a delay offset associated with the attribute component 212 based on the extracted first output delay. The first output delay (also referred to as outDelay[n][i]) may be extracted as the value of the i-th (e.g., i=0) V-PCC component of the syntax element (pic_dpb_output_delay) in the first picture timing SEI message associated with the access unit (n) in the attribute substream. Thereafter, the V-PCC decoder 102 may modify a first composition time (also referred to as compositionTime[n][i]) from which the attribute component 212 will be removed from the first DPB 204B. The first composition time may be modified based on the extracted first output delay and the determined delay offset. Details associated with the determination of the delay offset are described, for example, in Figure 3A-3B The V-PCC decoder 102 may remove the attribute component 212 from the first DPB 204B at the modified first composition time. The removal at the modified first composition time may correspond to a delayed release of the attribute component 212 for reconstruction.

[0039] In an embodiment, the V-PCC decoder 102 may receive a second picture timing SEI message associated with a geometry substream of the V-PCC bitstream 108. In order to delay the release of the geometry component 214 (i.e., the second V-PCC component) with respect to other V-PCC components, the V-PCC decoder 102 may extract a second output delay from the received second picture timing SEI message, and may determine a delay offset associated with the geometry component 214 based on the extracted second output delay. The second output delay (also referred to as outDelay[n][i]) may be extracted as the value of the i-th (e.g., i=1) V-PCC component of the syntax element (pic_dpb_output_delay) in the second picture timing SEI message associated with the access unit (n) in the geometry substream. Thereafter, the V-PCC decoder 102 may modify a second composition time (also referred to as compositionTime[n][i]) from which the geometry component 214 will be removed from the second DPB 206B. The second composition time may be modified based on the extracted second output delay and the determined delay offset. Details associated with the determination of the delay offset are described, for example, in Figure 3A-3BThe V-PCC decoder 102 may remove the geometry component 214 from the second DPB 206B at the modified second synthesis time. The removal at the modified second synthesis time may correspond to a delayed release of the geometry component 214 for reconstruction.

[0040] In an embodiment, the V-PCC decoder 102 may receive a third picture timing SEI message associated with an occupancy map substream of the V-PCC bitstream 108. In order to delay the release of the occupancy map component 216 (i.e., the third V-PCC component) with respect to other V-PCC components, the V-PCC decoder 102 may extract a third output delay from the received third picture timing SEI message, and may determine a delay offset associated with the occupancy map component 216 based on the extracted third output delay. The third output delay (also referred to as outDelay[n][i]) may be extracted as the value of the i-th (e.g., i=2) V-PCC component of the syntax element (pic_dpb_output_delay) in the third picture timing SEI message associated with the access unit (n) in the occupancy map substream. Thereafter, the V-PCC decoder 102 may modify a third composition time (also referred to as compositionTime[n][i]) at which the occupancy map component 216 will be removed from the third DPB 208B. The third composition time may be modified based on the extracted third output delay and the determined delay offset. Details associated with the determination of the delay offset are described, for example, in Figure 3A-3B The V-PCC decoder 102 may remove the occupancy map component 216 from the third DPB 208B at the modified third composition time. The removal at the modified third composition time may correspond to a delayed release of the occupancy map component 216 for reconstruction.

[0041] In an embodiment, the V-PCC decoder 102 may receive an atlas timing SEI message associated with an atlas substream of the V-PCC bitstream 108. To delay the release of the atlas component 218 (i.e., the fourth V-PCC component) with respect to other V-PCC components, the V-PCC decoder 102 may extract a fourth output delay from the received atlas timing SEI message, and may determine a delay offset associated with the atlas component 218 based on the extracted fourth output delay. The fourth output delay (also referred to as outDelay[n][i]) may be extracted as the value of the i-th (e.g., i=3) V-PCC component of the syntax element (aft_cab_output_delay_minus1) in the atlas timing SEI message associated with the access unit (n) in the atlas substream. Thereafter, the V-PCC decoder 102 may modify a fourth composition time (also referred to as compositionTime[n][i]) at which the atlas component 218 will be removed from the DAB 210B. The fourth composition time may be modified based on the extracted fourth output delay and the determined delay offset. Details associated with the determination of the delay offset are described, for example, in Figure 3A-3B The V-PCC decoder 102 may remove the atlas component 218 from the DAB 210B at the modified fourth composition time. The removal at the modified fourth composition time may correspond to a delayed release of the atlas component 218 for reconstruction.

[0042] The delayed release may synchronize (i.e., time-align) the V-PCC components (e.g., the attribute component 212, the geometry component 214, the occupancy map component 216, and the atlas component 218) with each other, after which the reconstruction unit 106 reconstructs the point cloud 222 based on these (time-aligned) V-PCC components.

[0043] In a first approach, the syntax and semantics of SEI and VUI messages (which may be similar to the SEI messages used in the H.264 / AVC standard and / or the H.265 / HEVC standard) are used to refer to HRDs, such as CPB / CAB removal times and DPB / DAB output times. However, due to the codec-agnostic nature of the V-PCC specification, it is likely that for some codecs, such as JPEG, the HRD syntax structure may not be supported or may not even exist. Thus, it may be necessary to rely on any set of syntax elements associated with the coded V-PCC component that carries some information that is comparable to the information carried in the SEI and VUI syntax structures (described in the first approach). This may be independent of whether such information is signaled using any other syntax structure within the V-PCC bitstream 108 or by some other means.

[0044] In an exemplary second method, the modification of the synthesis time (such as in Figure 2 and Figure 3A-3B The difference may be in the buffering scheme, i.e., the decoded frames / pictures from the various V-PCC components may be output from the corresponding DPB / DAB (such as the first DPB 204B, the second DPB 206B, the third DPB 208B and the DAB 210B) and may be directly input into the V-PCC composition buffer 220 without any modification. The release of such frames / pictures from the V-PCC composition buffer 220 to the reconstruction unit 106 may be in accordance with the modified composition time (such as in Figure 2 , 3A -3B for the first method). The second method may not affect the profile / level requirements of the V-PCC codec. Instead, it may rely on a V-PCC unit buffer (such as the V-PCC composition buffer 220) for storage.

[0045] An exemplary second method (also referred to as a codec-agnostic method) for synchronization (i.e., time alignment) of various V-PCC components is described here. After the first substream of the V-PCC bitstream 108 can be decoded, a first V-PCC component such as an attribute component 212 can be output from the first DPB 204B and can be stored in the V-PCC synthesis buffer 220 of the V-PCC decoder 102. Similarly, after the second substream of the V-PCC bitstream 108 can be decoded, a second V-PCC component such as a geometry component 214 can be output from the second DPB 206B and can be stored in the V-PCC synthesis buffer 220. After the third substream of the V-PCC bitstream 108 can be decoded, a third V-PCC component such as an occupancy map component 216 can be output from the third DPB 208B and can be stored in the V-PCC synthesis buffer 220. After the fourth substream of the V-PCC bitstream 108 may be decoded, a fourth V-PCC component such as an atlas component 218 may be output from the DAB 210B and may be stored in the V-PCC composition buffer 220 .

[0046] For each of the stored attribute component 212, the stored geometry component 214, the stored occupancy map component 216, and the stored atlas component 218, the V-PCC decoder 102 may modify the composition time for removal from the V-PCC composition buffer 220. The composition time of the attribute component 212 may be modified based on a first output delay and / or a delay offset associated with the attribute component 212. The composition time of the geometry component 214 may be modified based on a second output delay and / or a delay offset associated with the geometry component 214. The composition time of the occupancy map component 216 may be modified based on a third output delay and / or a delay offset associated with the occupancy map component 216. The composition time of the atlas component 218 may be modified based on a fourth output delay and / or a delay offset associated with the atlas component 218. The details of the modification of the composition time, the extraction of the corresponding output delays, and the determination of the corresponding delay offsets may be the same as those of the first method, and for example in Figure 2 , 3A and 3B.

[0047] When all required V-PCC components are in the V-PCC composition buffer 220, each such component may be removed from the V-PCC composition buffer 220 at a modified composition time associated with the corresponding V-PCC component. Removal of each of the stored attribute component 212, the stored geometry component 214, the stored occupancy map component 216, and the stored atlas component 218 may correspond to a delayed release to the reconstruction unit 106.

[0048] These components may be removed and released to reconstruct a full frame of the point cloud sequence (e.g., point cloud 222). In an embodiment, each of the first output delay, the second output delay, the third output delay, and the fourth output delay may be determined based on an atlas / picture timing SEI message that may be received with an access unit of a corresponding substream of the V-PCC bitstream 108.

[0049] Figure 3A is a diagram illustrating an example of sub-bitstreams according to an embodiment of the present disclosure, each sub-bitstream containing a single V-PCC component graph. Figure 1 and Figure 2 elements to illustrate Figure 3A . refer to Figure 3A, a diagram 300A illustrating an example of a sub-bitstream that each contains a single V-PCC component map is shown. In diagram 300A, a picture order count (POC) of each of the V-PCC components such as an atlas frame 302 (i.e., atlas component), an occupancy map 304 (i.e., occupancy map component), a geometry frame 306 (i.e., geometry component), and an attribute frame 308 (i.e., attribute component) is shown. Each of the four V-PCC components includes 6 frames that are initially decodable in a specific order (according to a specific group of pictures (GOP) structure). Figure 3A In , the term POC may refer to an index that defines the output position of coded pictures and atlas frames, and they may not necessarily be directly associated with the concept of time.

[0050] As shown, a video decoder (such as an atlas frame decoder 210) can decode the atlas substream of the V-PCC bitstream 108 to generate an atlas frame 302 (V-PCC 0) in a first order [0, 1, 2, 3, 4, 5]. After the atlas frame 302 is decoded, the atlas frame 302 can be stored in a DAB 210B. Similarly, another video decoder (such as an occupancy map decoder 208) can decode the occupancy map substream of the V-PCC bitstream 108 to generate an occupancy map 304 (V-PCC1) in a second order [0, 1, 2, 3, 3, 4]. The occupancy map 304 can be stored in a third DPB 208B. Since both the first order and the second order are correct picture orders, there is no need to reorder before outputting these frames / maps to reconstruct the point cloud. Another video decoder (such as the geometry video decoder 206) can decode the geometry substream to generate a geometry frame 306 (V-PCC 2) in a third order [0, 2, 1, 4, 3, 5]. The geometry frame 306 can be stored in the second DPB 206B. Another video decoder (such as the attribute video decoder 204) can decode the attribute substream to generate an attribute frame 308 (V-PCC 3) in a fourth order [0, 4, 2, 1, 3, 5]. The attribute frame 308 can be stored in the first DPB 204B.

[0051] Since the POCs of both the geometry frame 306 and the attribute frame 308 indicate that the frames are out of order, reordering may be required before outputting these frames for reconstruction. As shown in the figure, the reordering causes a delay and the length of the corresponding DPBs of the geometry component and the atlas component is extended by 1 frame and 2 frames, respectively. In other words, the output time (i.e., synthesis time) for removing the geometry frame 306 and the attribute frame 308 from the corresponding DPB, respectively, is delayed compared to the output time for the atlas frame 302 and the occupancy map 304.

[0052] In order to synchronize the atlas frame 302, the occupancy map 304, the geometry frame 306 and the attribute frame 308 in time, a delay offset may be applied to each V-PCC component. As shown, in order to match the output time (i.e., synthesis time) of the geometry frame 306 and the attribute frame 308, the atlas frame 302 may be offset by 2 frames in the DAB 210B, and the occupancy map 304 may be offset by 1 frame in the third DPB 208B. It should be noted that any increase in the DPB / DAB buffer size due to the introduction of the delay offset should be within the limits set by the signaled profile and level indication.

[0053] Here is provided (as in Figure 1 and Figure 2 The details of the modification of the composition time and the determination of the delay offset for each generated V-PCC component (described in ). Let compNum be the total number of V-PCC components (such as atlas components (i.e., atlas frame 302), occupancy map components (i.e., occupancy map 304), geometry components (i.e., geometry frame 306), and attribute components (i.e., attribute frame 308)). Let the array mapCnt(i) represent the number of maps associated with each i-th V-PCC component. Here, mapCnt(i) can be set to 1 if i refers to an atlas component or an occupancy map component, or if the geometry frame and attribute frame are encoded separately. If the maps are staggered in time, mapCnt(i) can be set to the number of geometry frames and attribute frames. Using the HRD video usability information (VUI) timing parameters, the V-PCC decoder 102 can determine the units of the clock (vui_num_units_in_tick) and the time scale (vui_timescale). The clock tick duration (clockTick) can be given by the following equation (1):

[0054]

[0055] where vui_num_units_in_tick may be the number of time units of a clock operating at a frequency vui_time_scale Hz corresponding to one increment of the time tick counter (called a time tick), and

[0056] vui_time_scale may be the number of time units that pass in one second. For example, a time coordinate system that measures time using a 27MHz clock has a vui_time_scale of 27000000. The value of vui_time_scale should be greater than 0.

[0057] vui_num_units_in_tick should be greater than 0. The clock tick in seconds may be equal to the quotient of vui_num_units_in_tick divided by vui_time_scale. For example, when the picture rate of the video signal is 25 Hz, vui_time_scale may be equal to 27000000, and vui_num_units_in_tick may be equal to 1080000, so that the clock tick may be equal to 0.04 seconds.

[0058] The clock tick duration clockSubTick of each V-PCC component can be given by the following equation (2):

[0059]

[0060] Figure 3A In , the variable (i) may represent a component number (ie, an identifier of a V-PCC component) and may be defined on an interval i=[0;compNum).

[0061] For example but not limited to, using equations (1) and (2), the following equation (3) can be used to modify the synthesis time of the i-th V-PCC component:

[0062] compositionTime[n][i]=remTime[n][i]+clockTick×(outDelay[n][i](3)+delayOffset[n][i])+ClockSubTick×i

[0063] in,

[0064] n and i may refer to the nth access unit (AU) of the i-th V-PCC component,

[0065] CompositionTime[n][i] may be the modified composition time after applying the delay offset of each V-PCC component,

[0066] remTime[n][i] may be the time at which a coded AU (contained in a substream of the V-PCC bitstream 108) is removed from the corresponding CPB / CAB of the V-PCC decoder 102,

[0067] outDelay[n][i] may be extracted from the syntax elements in the picture timing SEI message / atlas timing SEI message, and

[0068] delayOffset[n][i] may be the additional output delay (calculated using equation (4)) to be applied in order to synchronize all V-PCC components in time.

[0069] To achieve synchronization, a delay offset (delayOffset[n][i]) may be applied to each V-PCC component, where the initial maximum delay is set to 0. maxInitDelay[n] may be the maximum initial delay that may exist over all output delays (outDelay[n][i]) associated with the nth AU of the i-th V-PCC component. For example, the delay offset may be determined using the following equation (4):

[0070] delayOffset[n][i]=max(maxInitDelay[n][i]-compNum) (4)

[0071] maxInitDelay can be calculated using the following method:

[0072]

[0073] It should be noted that any increase in the DPB buffer size due to the introduction of this additional offset should be within the limits set by the signaled profile and tier indications. Figure 3B When each V-PCC component is released at the modified synthesis time obtained by formula (5) or formula (3), these components can be input into the (unit size) V-PCC synthesis buffer 220. The V-PCC synthesis buffer 220 can output all frames / images from the V-PCC components to the reconstruction unit 106, and the reconstruction unit 106 can reconstruct the point cloud 222 (corresponding to one frame of the point cloud sequence).

[0074] When determining and modifying the composition time of each V-PCC component, it is assumed that all AUs associated with each picture / frame are available. However, in practice, some AUs corresponding to a given picture set AU may be lost. An exemplary method of compensating for lost AUs may require copying the last available AU of the V-PCC component.

[0075] Figure 3B 1 is a diagram illustrating an example of a single sub-bitstream including an interlaced graph according to an embodiment of the present disclosure. Figure 1 , Figure 2 and Figure 3A elements to illustrate Figure 3B . refer to Figure 3B, a diagram 300B illustrating an example of a single sub-bitstream containing an interlaced map is shown. In diagram 300B, the POCs of V-PCC components such as atlas frame 310 (i.e., atlas component) and occupancy map 312 (i.e., occupancy map component) are shown. The POC / atlas frame order count (AFOC) of geometry frame 314 (i.e., geometry component) and attribute frame 316 (i.e., attribute component) are also shown. Each of the four V-PCC components includes 9 frames that can be initially decodable in a specific order (according to a specific group of pictures (GOP) structure). Figure 3B In , the terms POC / AFOC may refer to indices that define the output positions of coded pictures and atlas frames, and they may not necessarily be directly associated with the concept of time.

[0076] As shown, a video decoder (such as an atlas frame decoder 210) can decode an atlas substream of a V-PCC bitstream 108 to generate an atlas frame 310 (V-PCC 0) in a first order [0, 1, 2, 3, 4, 5, 6, 7, 8]. After the atlas frame 310 is decoded, the atlas frame 310 can be stored in a DAB 210B. Similarly, another video decoder (such as an occupancy map decoder 208) can decode an occupancy map substream of a V-PCC bitstream 108 to generate an occupancy map 312 (V-PCC 1) in a second order [0, 1, 2, 3, 3, 4, 5, 6, 7, 8]. The occupancy map 312 can be stored in a third DPB 208B. Since both the first order and the second order are correct picture orders, reordering may not be required before outputting these frames / maps to reconstruct a point cloud.

[0077] Another video decoder (such as the geometry video decoder 206) can decode the geometry substream to generate a geometry frame 314 (V-PCC 2) in a third order [0 / 0, 2 / 1, 1 / 0, 4 / 2, 3 / 1, 6 / 3, 5 / 2, 8 / 4, 7 / 3]. The geometry frame 314 can be stored in the second DPB 206B. Similarly, another video decoder (such as the attribute video decoder 204) can decode the attribute substream to generate an attribute frame 316 (V-PCC 3) in a fourth order [0 / 0, 4 / 2, 2 / 1, 1 / 0, 3 / 1, 8 / 4, 6 / 3, 5 / 2, 7 / 3]. The attribute frame 316 can be stored in the first DPB 204B. Each such frame can represent 2 interlaced images (e.g., as indicated by 0 / 0 (index)).

[0078] Since the POCs of both the geometry frame 314 and the attribute frame 316 indicate that the frames are out of order, reordering may be required before outputting these frames for reconstruction. As shown in the figure, the reordering causes delays and the lengths of the corresponding DPBs of the geometry component and the atlas component are extended by 1 frame and 2 frames, respectively. In other words, the output time (i.e., synthesis time) for removing the geometry frame 314 and the attribute frame 316 from the corresponding DPB, respectively, can be delayed compared to the output time for the atlas frame 310 and the occupancy map 312.

[0079] In order to synchronize the atlas frame 310, the occupancy graph 312, the geometry frame 314 and the attribute frame 316 in time, a delay offset can be applied to each V-PCC component. As shown in the figure, in order to match the output time (i.e., the synthesis time), the geometry frame 314 and the attribute frame 316 are offset by 1 frame in the corresponding DPB. And the atlas frame 310 is offset by 3 frames in the DAB 210B, and the occupancy graph 312 is offset by 3 frames in the third DPB 208B. It should be noted that any increase in the DPB / DAB buffer size caused by the introduction of the delay offset should be within the limits set by the profile and level indications notified by the signal. Here, for the situation that a single substream may contain an interlaced graph, an exemplary modification of the synthesis time based on the delay offset of each V-PCC component is provided. In this case, the V-PCC decoder 102 can call the component set output process. As part of this process, the synthesis time can be modified using the formula (5) that can be given as follows:

[0080] compositionTime[n][i][j]=remTime[n][i]+clockTick×(outDelay[n][i](5)+delayOffset[n][i])+ClockSubTick×(i+j÷mapCnt[i])

[0081] Where j can represent the graph index corresponding to the i-th V-PCC component and the n-th AU of the corresponding graph component. Formula (5) can be used Figure 3A The equations (1), (2) and (4) are derived.

[0082] Figure 4 is a diagram illustrating a hypothetical reference decoder (HRD) for an atlas frame in accordance with at least one embodiment of the present disclosure. Figure 1 , 2 , 3A and 3B elements to illustrate Figure 4 . refer to Figure 4, a diagram 400 of a hypothetical reference decoder (HRD) 402 is shown. The HRD 402 may include a hypothetical stream scheduler (HSS) 404, a coded atlas buffer (CAB) 406, a transient decoding operation 408, and a DAB 410. The HRD 402 may be Figure 2 In this implementation, DAB 410 may be Figure 2 An exemplary implementation of DAB 210B.

[0083] Each access unit may be referred to as access unit [n], where the number n identifies the particular access unit. Access unit 0 is selected by default, and the value of n is incremented by 1 for each subsequent access unit in decoding order. Atlas frame [n] may refer to a coded atlas frame or a decoded atlas frame of access unit [n].

[0084] In operation, HRD 402 may be initialized at decoding unit 0 (i.e., access unit 0), and both CAB 406 and DAB 410 may be set to empty (DAB fullness may be set to equal 0). After initialization, HRD 402 may not be initialized again by a subsequent buffering period SEI message. Data associated with decoding units that flow into CAB 406 based on a specified arrival schedule may be transmitted by HSS 404. Data associated with each decoding unit may be instantaneously removed and decoded by instantaneous decoding operation 408 at the CAB removal time of the decoding unit. Each decoded atlas frame may be placed in the DAB. When a decoded atlas frame becomes no longer needed for inter-frame prediction reference and no longer needed for output, the decoded atlas frame may be removed from the DAB.

[0085] For each atlas bitstream conformance test, the size of the CAB 406 (in bits) and the size of the DAB 410 (in atlas frame storage buffers) may be defined by CabSize[SchedSelIdx] and asps_max_dec_patch_frame_buffering_minus1[HighestTid] + 1, respectively. CabSize[SchedSelIdx] may be the size of the buffer holding the coded atlas data (CAB = coded atlas buffer).

[0086] For example, the operation of CAB 406 and DAB 410, as well as transient decoding operation 408, are specified here. HRD 402 may be initialized as specified by the buffering period SEI message. The timing of removing decoding units from CAB 406 and the timing of outputting decoded atlases from DAB 410 may be specified based on information in the atlas frame timing SEI message. All timing information related to a particular decoding unit may arrive before the CAB removal time of that particular decoding unit.

[0087] The DAB 410 may contain atlas frame storage buffers. Each atlas frame storage buffer may contain decoded atlas frames, which may be marked as "for reference" or may be kept for future output. Removal of an atlas frame from the DAB 410 may occur instantaneously at the CAB removal time (AuCabRemovalTime[n]) of the first decoding unit of access unit n (containing the current atlas frame), before the current atlas frame is decoded (but after the slice header of the first slice of the current atlas frame is parsed). For each atlas frame removed from the DAB 410, the DAB fullness is decremented by 1.

[0088] If AtlasFrameOutputFlag is equal to 1 and DabOutputTime[n] is equal to AuCabRemovalTime[n], the current atlas frame may be output. Otherwise, if AtlasFrameOutputFlag is equal to 0, the current atlas frame may not be output but may be stored in the DAB 410. Otherwise, if AtlasFrameOutputFlag is equal to 1 and DabOutputTime[n] is greater than AuCabRemovalTime[n]), the current atlas frame may then be output and may be stored in the DAB 410. In this case, the current atlas frame may be output at time DabOutputTime[n] unless indicated otherwise by decoding or inference of no_output_of_prior_patch_frames_flag being equal to 1 at a time prior to DabOutputTime[n]. Here, no_output_of_prior_patch_frames_flag may be a flag indicating that the decoded atlas will not be output for display.

[0089] In an embodiment, the current decoded atlas frame may be stored in an empty atlas frame storage buffer DAB 410, and the DAB fullness may be incremented by 1. When asps_long_term_ref_atlas_frames_flag is equal to 1, the atlas frame is marked as "used for long-term reference". asps_long_term_ref_atlas_frames_flag equal to 0 may specify that no long-term reference atlas frame is used for inter-frame prediction of any coded atlas frame in the CAS. asps_long_term_ref_atlas_frames_flag equal to 1 may specify that a long-term reference atlas frame may be used for inter-frame prediction of one or more coded atlas frames in the CAS.

[0090] After all tile groups of the current atlas frame have been decoded, the atlas frame may be marked as "used for short-term reference". It should be noted that, unless more memory is available to store the decoded atlas frame than required by the hierarchy limit, the decoder should start storing the decoded portion of the current atlas frame into the DAB 410 when the first tile group is decoded, and should continue to store more decoded samples as the decoding process proceeds. Immediately after the current atlas frame is decoded, the current decoded atlas frame may be removed from the DAB 410 at the CAB removal time of the last decoding unit of access unit n (containing the current atlas frame), and the DAB fullness may be decremented by 1.

[0091] Figure 5 According to the embodiment of the present disclosure Figure 1 Block diagram of the V-PCC decoder. Figure 1 , 2 , 3A, 3B and 4 elements to illustrate Figure 5 . refer to Figure 5 , a block diagram 500 of a V-PCC decoder 102 is shown. The V-PCC decoder 102 may include a demultiplexer 202 and a circuit 502, which may include a processor 504, a set of video decoders 104 (and associated with CPB / CAB and DPB / DAB) and a reconstruction unit 106. In an embodiment, the V-PCC decoder 102 may also include a memory 506, an input / output (I / O) interface 508 and a V-PCC composition buffer 220.

[0092] The processor 504 may include suitable logic, circuitry, and / or interfaces that may be configured to execute program instructions associated with decoding of the V-PCC bitstream 108 and frame-by-frame reconstruction of a point cloud sequence of V-PCC components obtained from the decoded V-PCC bitstream 108. The processor 504 may be implemented based on many processor technologies known in the art. Exemplary implementations of the processor 504 may include, but are not limited to, a reduced instruction set computing (RISC) processor, an application specific integrated circuit (ASIC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a microcontroller, a coprocessor, a central processing unit (CPU), and / or combinations thereof.

[0093] The memory 506 may include suitable logic, circuitry, and / or interfaces that may be configured to store program instructions that may be executed by the processor 504. The memory 506 may be configured to store information that may be required (or generated) at any stage of operation of the circuit 502. Exemplary implementations of the memory 506 may include (but are not limited to) random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), a hard disk drive (HDD), a solid-state drive (SSD), or a CPU cache.

[0094] The I / O interface 508 may include suitable logic, circuitry, interfaces and / or code that may be configured to act as a communication interface between various components (eg, the demultiplexer 202 , the circuit 502 , the memory 506 ) and the V-PCC synthesis buffer 220 .

[0095] Modifications, additions, or omissions may be made to the block diagram 200 of the V-PCC decoder 102 without departing from the scope of the present disclosure. For example, in some embodiments, the block diagram 500 may include any number of other components that may not be explicitly illustrated or described. Figure 1 , Figure 2 , Figure 3A-3B and Figure 6 The functions or operations described in hereinafter as being performed by V-PCC decoder 102 may be performed by circuit 502 .

[0096] Figure 6 is a flow chart illustrating an exemplary operation for synchronizing decoded frames prior to point cloud reconstruction according to an embodiment of the present disclosure. Figure 1 , 2 , 3A, 3B, 4 and 5 elements to illustrate Figure 6 . refer to Figure 6 , a flowchart 600 is shown. The method illustrated in flowchart 600 may be performed by any computing system, such as by Figure 1 The method may start at 602 and proceed to 604.

[0097] At 604, a V-PCC bitstream 108 may be received. The V-PCC bitstream 108 may include encoded frames associated with a point cloud sequence. In an embodiment, the circuit 502 may be configured to receive a V-PCC bitstream 108 including encoded frames associated with a point cloud sequence.

[0098] At 606, the substreams of the V-PCC bitstream 108 may be decoded to generate V-PCC components, such as the attribute component 212, the geometry component 214, the occupancy map component 216, and the atlas component 218. In an embodiment, the circuit 502 may include a set of video decoders 104 that may be configured to decode the substreams of the V-PCC bitstream 108 to generate the V-PCC components.

[0099] At 608, the release of the attribute component 212 to the reconstruction unit 106 of the V-PCC decoder 102 may be delayed based on the first output delay. In an embodiment, the circuit 502 of the V-PCC decoder 102 may be configured to delay the release of the attribute component 212 to the reconstruction unit 106 based on the first output delay. In an embodiment, the circuit 502 may be configured to delay the release of the attribute frame from the first DPB 204B. In this case, the first output delay may be a DPB output delay, which may be included in a first picture timing SEI message associated with an access unit of the attribute substream of the V-PCC bitstream 108.

[0100] At 610, the release of the geometry component 214 to the reconstruction unit 106 of the V-PCC decoder 102 may be delayed based on the second output delay. In an embodiment, the circuit 502 of the V-PCC decoder 102 may be configured to delay the release of the geometry component 214 to the reconstruction unit 106 based on the second output delay. In an embodiment, the circuit 502 may be configured to delay the release of the geometry frame from the second DPB 206B. In this case, the second output delay may be a DPB output delay included in a second picture timing SEI message associated with an access unit of the geometry substream of the V-PCC bitstream 108.

[0101] At 612, the release of the occupancy map component 216 to the reconstruction unit 106 of the V-PCC decoder 102 may be delayed based on the third output delay. In an embodiment, the circuit 502 of the V-PCC decoder 102 may be configured to delay the release of the occupancy map component 216 to the reconstruction unit 106 based on the third output delay. In an embodiment, the circuit 502 may be configured to delay the release of the occupancy map component 216 from the third DPB 208B. In this case, the third output delay may be a DPB output delay included in a third picture timing SEI message associated with an access unit of the occupancy map substream of the V-PCC bitstream 108.

[0102] At 614, the release of the atlas component 218 to the reconstruction unit 106 of the V-PCC decoder 102 may be delayed based on the fourth output delay. In an embodiment, the circuit 502 of the V-PCC decoder 102 may be configured to delay the release of the atlas component 218 to the reconstruction unit 106 based on the fourth output delay. In an embodiment, the circuit 502 may be configured to delay the release of the atlas component 218 from the DAB 210B. In this case, the fourth output delay may be a DAB output delay contained in an atlas timing SEI message associated with an access unit of the atlas substream of the V-PCC bitstream 108. The release of the delays at 608, 610, 612, and 614 may synchronize the attribute component 212, the geometry component 214, the occupancy map component 216, and the atlas component 218 with each other (i.e., align in time) before the reconstruction unit 106 reconstructs the point cloud 222 based on these released components. Control may be transferred to the end.

[0103] Various embodiments of the present disclosure may provide a non-temporary computer-readable medium and / or storage medium storing instructions that can be executed by a machine and / or computer to operate a system (e.g., a V-PCC decoder 102). The instructions may cause the machine and / or computer to perform operations including receiving a V-PCC bitstream 108 containing encoded frames associated with a point cloud sequence, and decoding a substream of the V-PCC bitstream 108 through a set of video decoders 104 to generate a V-PCC component. Such components include attribute components, geometric components, occupancy map components, and atlas components. The operation also includes delaying the release of the attribute component to the reconstruction unit 106 of the V-PCC decoder 102 based on the first output delay, and delaying the release of the geometric component to the reconstruction unit 106 based on the second output delay. The operation also includes delaying the release of the occupancy map component to the reconstruction unit 106 based on the third output delay, and delaying the release of the atlas component to the reconstruction unit 106 based on the fourth output delay. The delayed release may synchronize the attribute components, the geometry components, the occupancy map components, and the atlas components with each other before the reconstruction unit 106 reconstructs the point cloud based on the released components.

[0104] Exemplary aspects of the present disclosure may provide a V-PCC decoder (e.g., Figure 1The circuit 502 may be configured to receive a V-PCC bitstream 108 containing coded components associated with a point cloud sequence. The circuit 502 may include a set of video decoders 104, which may be configured to decode a substream of the V-PCC bitstream to generate V-PCC components, such as an attribute component 212, a geometry component 214, an occupancy map component 216, and an atlas component 218. The circuit 502 may also be configured to delay the release of the attribute component 212 to a reconstruction unit 106 of the V-PCC decoder 102 based on a first output delay, and to delay the release of the geometry component 214 to the reconstruction unit 106 based on a second output delay. The circuit 502 may also be configured to delay the release of the occupancy map component 216 to the reconstruction unit 106 based on a third output delay, and to delay the release of the atlas component 218 to the reconstruction unit 106 based on a fourth output delay. The delayed release may synchronize (ie, align in time) the attribute component 212 , the geometry component 214 , the occupancy map component 216 , and the atlas component 218 with one another before the reconstruction unit 106 reconstructs a point cloud, such as the point cloud 222 , based on the attribute component 212 , the geometry component 214 , the occupancy map component 216 , and the atlas component 218 .

[0105] In an embodiment, the circuit 502 may also be configured to receive a first picture timing supplemental enhancement information (SEI) message associated with the attribute substream of the V-PCC bitstream 108. The circuit 502 may extract the first output delay from the received first picture timing SEI message, and may determine the delay offset associated with the attribute component 212 based on the extracted first output delay. The circuit 502 may update the first synthesis time at which the attribute component 212 will be removed from the first DPB 204B. The first synthesis time may be updated based on the extracted first output delay and the determined delay offset. The circuit 502 may remove the attribute component 212 from the first DPB 204B at the updated first synthesis time. The removal of the attribute component may correspond to the release of the delay of the attribute component 212.

[0106] In an embodiment, the circuit 502 may also be configured to receive a second picture timing supplemental enhancement information (SEI) message associated with the geometry substream of the V-PCC bitstream 108. The circuit 502 may extract the second output delay from the received second picture timing SEI message, and may determine the delay offset associated with the geometry component 214 based on the extracted second output delay. The circuit 502 may update the second synthesis time at which the geometry component 214 will be removed from the second DPB 206B. The second synthesis time may be updated based on the extracted second output delay and the determined delay offset. The circuit 502 may remove the geometry component 214 from the second DPB 206B at the updated second synthesis time. The removal of the geometry component 214 may correspond to the release of the delay of the geometry component 214.

[0107] In an embodiment, the circuit 502 may also be configured to receive a third picture timing supplemental enhancement information (SEI) message associated with the occupancy diagram substream of the V-PCC bitstream 108. The circuit 502 may extract the third output delay from the received third picture timing SEI message, and may determine the delay offset associated with the occupancy diagram component 216 based on the extracted third output delay. Afterwards, the circuit 502 may update the third synthesis time at which the occupancy diagram component 216 will be removed from the third DPB 208B. The third synthesis time may be updated based on the extracted third output delay and the determined delay offset. The circuit 502 may remove the occupancy diagram component 216 from the third DPB 208B at the updated third synthesis time. The removal of the occupancy diagram component 216 may correspond to the release of the delay of the occupancy diagram component 216.

[0108] In an embodiment, the circuit 502 may also be configured to receive an atlas timing supplemental enhancement information (SEI) message associated with the atlas substream of the V-PCC bitstream 108. The circuit 502 may extract the fourth output delay from the received atlas timing SEI message, and may determine the delay offset associated with the atlas component 218 based on the extracted fourth output delay. Thereafter, the circuit 502 may update the fourth synthesis time at which the atlas component 218 will be removed from the DAB 210B. The fourth synthesis time may be updated based on the extracted fourth output delay and the determined delay offset. The circuit 502 may remove the atlas component 218 from the DAB 210B at the updated fourth synthesis time. The removal of the atlas component 218 may correspond to the release of the delay of the atlas component 218.

[0109] In an embodiment, the circuit 502 may be further configured to store the attribute component 212 in the V-PCC composition buffer 220 of the V-PCC decoder 102 after the attribute substream is decoded. Similarly, the circuit 502 may be further configured to store the geometry component 214, the occupancy map component 216, and the atlas component 218 in the V-PCC composition buffer 220 after each of the geometry substream, the occupancy map substream, and the atlas substream is decoded, respectively.

[0110] For each of the stored attribute component 212, the stored geometry component 214, the stored occupancy map component 216, and the stored atlas component 218, the circuit 502 may be configured to modify a composition time for removal from the V-PCC composition buffer 220. The composition time may be modified based on one of: a first output delay, a second output delay, a third output delay, and a fourth output delay. The circuit 502 may also be configured to remove each of the stored attribute component 212, the stored geometry component 214, the stored occupancy map component 216, and the stored atlas component 218 from the V-PCC composition buffer 220 at the modified composition time associated with the corresponding component. The removal of each of the stored attribute component 212, the stored geometry component 214, the stored occupancy map component 216, and the stored atlas component 218 may correspond to a release of a delay to the reconstruction unit 106.

[0111] The present disclosure may be implemented in hardware or in a combination of hardware and software. The present disclosure may be implemented in a centralized manner, in at least one computer system, or in a distributed manner, where different elements may be distributed across several interconnected computer systems. A computer system or other device suitable for performing the methods described herein may be suitable. The combination of hardware and software may be a general-purpose computer system with a computer program that, when loaded and executed, may control the computer system so that the computer system performs the methods described herein. The present disclosure may be implemented in hardware that includes a portion of an integrated circuit that also performs other functions.

[0112] The present disclosure may also be embedded in a computer program product comprising all the features enabling the implementation of the methods described herein and capable of carrying out these methods when loaded into a computer system. In this context, a computer program means any expression in any language, code or annotation of a set of instructions intended to cause a system with information processing capabilities to perform a specific function, either directly or after one or both of the following: a) conversion into another language, code or annotation; b) reproduction in a different material form.

[0113] Although the present disclosure is described with reference to certain embodiments, it will be appreciated by those skilled in the art that various changes and equivalent substitutions may be made without departing from the scope of the present disclosure. In addition, many modifications may be made to adapt specific circumstances or materials to the teachings of the present disclosure without departing from the scope of the present disclosure. Therefore, it is intended that the present disclosure is not limited to the specific embodiments disclosed, but rather that the present disclosure will include all embodiments falling within the scope of the appended claims.

Claims

1. A method comprising: In the video-based point cloud compression V-PCC decoder: Receiving a V-PCC bitstream, the V-PCC bitstream comprising coded frames associated with a point cloud sequence; Decoding a substream of a V-PCC bitstream by a set of video decoders to generate a plurality of V-PCC components, the plurality of V-PCC components comprising an attribute component, a geometry component, an occupancy map component, and an atlas component; modifying each of a first composition time of the attribute component, a second composition time of the geometry component, a third composition time of the occupancy graph component, and a fourth composition time of the atlas component, wherein the modification of each of the first composition time, the second composition time, the third composition time, and the fourth composition time is based on an output delay of a corresponding V-PCC component of the plurality of V-PCC components, a first clock tick duration, and a second clock tick duration, the first clock tick duration being based on a number of time units of a clock for a particular picture rate of the V-PCC bitstream and an operating frequency of the clock, and the second clock tick duration being a ratio of the first clock tick duration to a total number of the plurality of V-PCC components; delaying an output of the attribute component to a reconstruction unit of the V-PCC decoder based on the modified first synthesis time; delaying output of the geometry component to the reconstruction unit based on the modified second synthesis time; delaying output of the occupancy map component to the reconstruction unit based on the modified third synthesis time; as well as Based on the modified fourth synthesis time, the output of the atlas component to the reconstruction unit is delayed, wherein The delayed output of each of the attribute component, the geometry component, the occupancy map component and the atlas component synchronizes the attribute component, the geometry component, the occupancy map component and the atlas component with each other before the reconstruction unit reconstructs the point cloud based on the plurality of V-PCC components.

2. The method according to claim 1, further comprising: A first picture timing supplemental enhancement information (SEI) message associated with an attribute substream of a V-PCC bitstream is received.

3. The method of claim 2, wherein the output of the delay attribute component comprises: extracting a first output delay from the received first picture timing SEI message, wherein the first output delay corresponds to an output delay of the attribute component; determining, based on the extracted first output delay, an additional output delay associated with the attribute component; modifying a first composition time at which the attribute component will be removed from a first decoded picture buffer associated with the attribute video decoder, wherein the first synthesis time is further modified based on the extracted first output delay and the determined additional output delay; as well as The attribute component is removed from the first decoded picture buffer at the modified first composition time.

4. The method according to claim 1, further comprising: A second picture timing supplemental enhancement information (SEI) message associated with a geometry substream of a V-PCC bitstream is received.

5. The method of claim 4, wherein delaying the output of the geometry component comprises: extracting a second output delay from the received second picture timing SEI message, wherein the second output delay corresponds to an output delay of the geometry component; determining an additional output delay associated with the geometry component based on the extracted second output delay; modifying a second composition time that will remove the geometry component from a second decoded picture buffer associated with the geometry video decoder, wherein the second composition time is further modified based on the extracted second output delay and the determined additional output delay; as well as The geometry component is removed from the second decoded picture buffer at the modified second composition time.

6. The method according to claim 1, further comprising: A third picture timing supplemental enhancement information (SEI) message associated with an occupancy map substream of a V-PCC bitstream is received.

7. The method of claim 6, wherein delaying the output of the occupancy map component comprises: extracting a third output delay from the received third timing SEI message, wherein the third output delay corresponds to an output delay of the occupancy map component; determining an additional output delay associated with the occupancy map component based on the extracted third output delay; modifying a third composition time at which the occupancy map component will be removed from a third decoded picture buffer associated with the occupancy map decoder, wherein the third synthesis time is further modified based on the extracted third output delay and the determined additional output delay; and The occupancy map component is removed from the third decoded picture buffer at the modified third composition time.

8. The method according to claim 1, further comprising: A picture set timing supplemental enhancement information (SEI) message associated with a picture set substream of a V-PCC bitstream is received.

9. The method of claim 8, wherein delaying the output of the atlas component comprises: extracting a fourth output delay from the received atlas timing SEI message, wherein the fourth output delay corresponds to an output delay of the atlas component; determining an additional output delay associated with the atlas component based on the extracted second output delay; modifying a fourth composition time at which an atlas component will be removed from a decoded atlas buffer associated with an atlas frame decoder, wherein the fourth synthesis time is further modified based on the extracted fourth output delay and the determined additional output delay; as well as The atlas component is removed from the decoded atlas buffer at the modified fourth composition time.

10. The method according to claim 1, further comprising: After decoding, storing the attribute components in a V-PCC composition buffer of the V-PCC decoder; After decoding, storing the geometry components in the V-PCC composition buffer; After decoding, storing the occupancy map component in the V-PCC composition buffer; as well as After decoding, the atlas components are stored in the V-PCC composition buffer.

11. The method according to claim 10, further comprising: removing each of the stored attribute component, the stored geometry component, the stored occupancy map component, and the stored atlas component from the V-PCC composition buffer at a modified first composition time, a modified second composition time, a modified third composition time, and a modified fourth composition time, respectively, wherein Removal of each of the stored attributes component, the stored geometry component, the stored occupancy map component, and the stored atlas component corresponds to a delayed output to the reconstruction unit.

12. A video-based point cloud compression V-PCC decoder, comprising: A circuit, the circuit being configured to: Receiving a V-PCC bitstream, the V-PCC bitstream comprising coded frames associated with a point cloud sequence; decoding a substream of a V-PCC bitstream to generate a plurality of V-PCC components, the plurality of V-PCC components comprising an attribute component, a geometry component, an occupancy map component, and an atlas component; modifying each of a first composition time of the attribute component, a second composition time of the geometry component, a third composition time of the occupancy graph component, and a fourth composition time of the atlas component, wherein the modification of each of the first composition time, the second composition time, the third composition time, and the fourth composition time is based on an output delay of a corresponding V-PCC component of the plurality of V-PCC components, a first clock tick duration, and a second clock tick duration, the first clock tick duration being based on a number of time units of a clock for a particular picture rate of the V-PCC bitstream and an operating frequency of the clock, and the second clock tick duration being a ratio of the first clock tick duration to a total number of the plurality of V-PCC components; delaying an output of the attribute component to a reconstruction unit of the V-PCC decoder based on the modified first synthesis time; delaying output of the geometry component to the reconstruction unit based on the modified second synthesis time; delaying output of the occupancy map component to the reconstruction unit based on the modified third synthesis time; as well as Based on the modified fourth synthesis time, the output of the atlas component to the reconstruction unit is delayed, wherein The delayed output of each of the attribute component, the geometry component, the occupancy map component and the atlas component synchronizes the attribute component, the geometry component, the occupancy map component and the atlas component with each other before the reconstruction unit reconstructs the point cloud based on the plurality of V-PCC components.

13. The V-PCC decoder of claim 12, wherein the circuit is further configured to: Receiving a first picture timing supplemental enhancement information (SEI) message associated with an attribute substream of a V-PCC bitstream; extracting a first output delay from the received first picture timing SEI message, wherein the first output delay corresponds to an output delay of the attribute component; determining, based on the extracted first output delay, an additional output delay associated with the attribute component; modifying a first composition time at which the attribute component will be removed from a first decoded picture buffer associated with the attribute video decoder, wherein the first synthesis time is further modified based on the extracted first output delay and the determined additional output delay; and removing the attribute component from the first decoded picture buffer at the modified first composition time, Wherein the removal corresponds to a delayed output of the attribute component.

14. The V-PCC decoder of claim 12, wherein the circuit is further configured to: receiving a second picture timing supplemental enhancement information (SEI) message associated with a geometry substream of a V-PCC bitstream; extracting a second output delay from the received second picture timing SEI message, wherein the second output delay corresponds to an output delay of the geometry component; determining an additional output delay associated with the geometry component based on the extracted second output delay; modifying a second composition time that will remove the geometry component from a second decoded picture buffer associated with the geometry video decoder, wherein the second synthesis time is further modified based on the extracted second output delay and the determined additional output delay; and removing the geometry component from the second decoded picture buffer at the modified second composition time, Wherein the removal corresponds to a delayed output of the geometry component.

15. The V-PCC decoder of claim 12, wherein the circuit is further configured to: receiving a third picture timing supplemental enhancement information (SEI) message associated with an occupancy map substream of a V-PCC bitstream; extracting a third output delay from the received third timing SEI message, wherein the third output delay corresponds to an output delay of the occupancy map component; determining an additional output delay associated with the occupancy map component based on the extracted third output delay; modifying a third composition time at which the occupancy map component will be removed from a third decoded picture buffer associated with the occupancy map decoder, wherein the third synthesis time is further modified based on the extracted third output delay and the determined additional output delay; and removing the occupancy map component from the third decoded picture buffer at a modified third composition time, Wherein the removal corresponds to a delayed output of an occupancy graph component.

16. The V-PCC decoder of claim 12, wherein the circuit is further configured to receive a picture set timing supplemental enhancement information (SEI) message associated with a picture set substream of the V-PCC bitstream.

17. The V-PCC decoder of claim 16, wherein the circuit is further configured to: extracting a fourth output delay from the received atlas timing SEI message, wherein the fourth output delay corresponds to an output delay of the atlas component; determining an additional output delay associated with the atlas component based on the extracted second output delay; modifying a fourth composition time at which an atlas component will be removed from a decoded atlas buffer associated with an atlas frame decoder, wherein the fourth synthesis time is further modified based on the extracted fourth output delay and the determined additional output delay; and removing the atlas component from the decoded atlas buffer at the modified fourth composition time, The removal corresponds to a delayed output of the atlas component.

18. The V-PCC decoder of claim 12, wherein the circuit is further configured to: After decoding, storing the attribute components in a V-PCC composition buffer of the V-PCC decoder; After decoding, storing the geometry components in the V-PCC composition buffer; After decoding, storing the occupancy map component in the V-PCC composition buffer; as well as After decoding, the atlas components are stored in the V-PCC composition buffer.

19. The V-PCC decoder of claim 18, wherein the circuit is further configured to remove each of the stored attribute component, the stored geometry component, the stored occupancy map component, and the stored atlas component from the V-PCC composition buffer at a modified first composition time, a modified second composition time, a modified third composition time, and a modified fourth composition time, respectively, and Wherein the removal of each of the stored attribute component, the stored geometry component, the stored occupancy map component, and the stored atlas component corresponds to a delayed output to the reconstruction unit.

Images