Point cloud data transmitting device, point cloud data transmitting method, point cloud data receiving device, and point cloud data receiving method

The method optimizes point cloud data transmission and reception using V-PCC and viewport signaling, addressing efficiency and complexity issues in processing large 3D data volumes for applications like VR, AR, MR, and autonomous driving.

JP2024159914AActive Publication Date: 2024-11-08LG ELECTRONICS INC
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
JP2024140997
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2020-03-16
Filing Date
2024-08-22
Publication Date
2024-11-08
Estimated Expiration
2040-10-07

AI Technical Summary

Technical Problem

The large volume of point cloud data in 3D space poses challenges in efficient generation, transmission, and reception, requiring significant processing resources and complexity in encoding and decoding.

Method used

A method and device for transmitting and receiving point cloud data that includes encoding point cloud data using video-based point cloud compression (V-PCC) and signaling viewport information, such as camera position and orientation, to optimize data transmission and rendering.

Benefits of technology

Enables high-quality point cloud services with reduced latency and complexity by efficiently transmitting and rendering point cloud content based on viewport information, supporting applications like VR, AR, MR, and autonomous driving.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024159914000001_ABST
    Figure 2024159914000001_ABST
Patent Text Reader

Abstract

To provide a point cloud data transmission method.SOLUTION: A point cloud data transmitting method according to an example includes the steps of encoding point cloud data, and transmitting a bitstream containing point cloud data and signaling information. A point cloud data receiving method according to an embodiment includes the steps of receiving a bitstream including point cloud data and signaling information, decoding the point cloud data, and rendering the decoded point cloud data.SELECTED DRAWING: Figure 43
Need to check novelty before this filing date? Find Prior Art

Description

[Technical Field]

[0001] The embodiment provides a method for providing point cloud content to provide users with various services such as VR (Virtual Reality), AR (Augmented Reality), MR (Mixed Reality) and autonomous driving services. [Background technology]

[0002] A point cloud is a collection of points in 3D space. There is a problem that the number of points in 3D space is large, making it difficult to generate point cloud data.

[0003] There is a problem in that sending and receiving point cloud data requires a large amount of processing. Summary of the Invention [Problem to be solved by the invention]

[0004] The technical objective of the embodiments is to provide a point cloud data transmitting device, a transmitting method, a point cloud data receiving device, and a receiving method for efficiently transmitting and receiving point clouds in order to solve the above-mentioned problems.

[0005] A technical problem related to the embodiments is to provide a point cloud data transmitting device, transmitting method, point cloud data receiving device and receiving method that solves latency and encoding / decoding complexity.

[0006] The technical problem of the embodiment is to provide a point cloud data transmitting device, transmitting method, point cloud data receiving device and receiving method for providing optimized point cloud content to users by signaling information about the viewport of point cloud data.

[0007] The technical problem of the embodiment is to provide a point cloud data transmitting device, transmitting method, point cloud data receiving device and receiving method for providing optimized point cloud content to users by enabling viewport information, recommended viewport information, and initial viewing orientation (i.e., viewpoint) information for data processing and rendering in a V-PCC bitstream to be transmitted within the bitstream.

[0008] However, the scope of the invention is not limited to the above-mentioned technical problems, but can be extended to other technical problems that can be derived by a person skilled in the art based on all the contents of this document. [Means for solving the problem]

[0009] To achieve the above-mentioned objectives and other advantages, a point cloud data transmission method according to an embodiment may include a step of encoding point cloud data, and a step of transmitting a bitstream including the point cloud data and signaling information.

[0010] According to an embodiment, the point cloud data may include geometry data, feature data, and occupancy map data that are encoded using a video-based point cloud compression (V-PCC) scheme.

[0011] According to an embodiment, the signaling information may include information about a viewport for which the viewport is determined depending on the position and orientation of the camera or user.

[0012] According to an embodiment, the information about the viewport may include at least one of coordinate information of the camera or user in 3D space, direction vector information indicating the direction in which the camera or user is looking, up vector information indicating above the camera or user, and right vector information indicating to the right of the camera or user.

[0013] According to an embodiment, the information about the viewport may further include horizontal field of view (FOV) information and vertical FOV information for generating the viewport.

[0014] According to an embodiment, the point cloud data transmitting device may include an encoder for encoding the point cloud data, and a transmitter for transmitting a bitstream including the point cloud data and signaling information.

[0015] According to an embodiment, the point cloud data may include geometry data, feature data, and occupancy map data that are encoded using a video-based point cloud compression (V-PCC) scheme.

[0016] According to an embodiment, the signaling information may include information about a viewport for which the viewport is determined depending on the position and orientation of the camera or user.

[0017] According to an embodiment, the information about the viewport may include at least one of coordinate information of the camera or user in 3D space, direction vector information indicating the direction in which the camera or user is looking, up vector information indicating above the camera or user, and right vector information indicating to the right of the camera or user.

[0018] According to an embodiment, the information about the viewport may further include horizontal FOV information and vertical FOV information for generating the viewport.

[0019] According to an embodiment, a point cloud data receiving method may include receiving a bitstream including point cloud data and signaling information, decoding the point cloud data, and rendering the decoded point cloud data.

[0020] According to an embodiment, the signaling information may include information about a viewport for which the viewport is determined depending on the position and orientation of the camera or user.

[0021] According to an embodiment, the information about the viewport may include at least one of coordinate information of the camera or user in 3D space, direction vector information indicating the direction in which the camera or user is looking, up vector information indicating above the camera or user, and right vector information indicating to the right of the camera or user.

[0022] According to an embodiment, the information about the viewport may further include horizontal FOV information and vertical FOV information for generating the viewport.

[0023] According to an embodiment, the decoded point cloud data may be rendered based on information about the viewport.

[0024] According to an embodiment, a point cloud data receiving device may include a receiver that receives a bitstream including point cloud data and signaling information, a decoder that decodes the point cloud data, and a renderer that renders the decoded point cloud data.

[0025] According to an embodiment, the signaling information may include information about a viewport for which the viewport is determined depending on the position and orientation of the camera or user.

[0026] According to an embodiment, the information about the viewport may include at least one of coordinate information of the camera or user in 3D space, direction vector information indicating the direction in which the camera or user is looking, up vector information indicating above the camera or user, and right vector information indicating to the right of the camera or user.

[0027] According to an embodiment, the information about the viewport may further include horizontal FOV information and vertical FOV information for generating the viewport.

[0028] According to an embodiment, the renderer may render the decoded point cloud data based on information about the viewport. [Effects of the Invention]

[0029] The point cloud data transmitting method, transmitting device, point cloud data receiving method, and receiving device according to the embodiments can provide high-quality point cloud services.

[0030] The point cloud data transmitting method, transmitting device, point cloud data receiving method, and receiving device according to the embodiments can achieve various video codec methods.

[0031] The point cloud data transmitting method, transmitting device, point cloud data receiving method, and receiving device according to the embodiments can provide general-purpose point cloud content such as an autonomous driving service.

[0032] The point cloud data transmission method, transmitting device, point cloud data receiving method, and receiving device according to the embodiments can provide optimal point cloud content services by constructing a V-PCC bitstream and enabling files to be transmitted, received, and stored.

[0033] The point cloud data transmission method, transmitting device, point cloud data receiving method, and receiving device according to the embodiments can provide optimal point cloud content services by including metadata for data processing and rendering in the V-PCC bitstream within the V-PCC bitstream so that it can be transmitted and received.

[0034] The point cloud data transmission method, transmission device, point cloud data receiving method, and receiving device according to the embodiments enable access to the space or part of the point cloud object / content through a user viewport in a player, etc., thereby providing the effect of efficiently accessing and processing the point cloud bitstream through the user's viewport.

[0035] The point cloud data transmission method, transmitting device, point cloud data receiving method, and receiving device according to the embodiments can provide a bounding box and signaling information for partial and / or spatial access to point cloud content, thereby providing the effect of enabling the receiving side to access point cloud content in a variety of ways, taking into account the player or user environment.

[0036] The point cloud data transmission method and transmission device according to the embodiment can provide 3D region information of the point cloud content to assist in accessing a space or portion of the point cloud content depending on a user's viewport, and metadata related to 2D regions on the associated video or atlas frame.

[0037] The point cloud data transmission method and transmission device according to the embodiment can process 3D region information of a point cloud in a point cloud bitstream and information signaling related to 2D regions on the associated video or atlas frames.

[0038] The point cloud data receiving method and receiving device according to the embodiment can efficiently access point cloud content based on, for example, storing and signaling 3D region information of the point cloud in a file and information related to 2D regions on the associated video or atlas frame.

[0039] The point cloud data receiving method and receiving device according to the embodiment can provide point cloud content that takes into account the user environment based on 3D area information of the point cloud associated with an image item in a file and information related to 2D areas on the associated video or atlas frame. [Brief explanation of the drawings]

[0040] The drawings are attached to provide a further understanding of the embodiments and together with the description of the embodiments illustrate the embodiments.

[0041] [Figure 1] 1 illustrates an example of a transmitting / receiving system architecture for providing point cloud content according to an embodiment. [Figure 2] 1 illustrates an example of capturing point cloud data according to an embodiment. [Figure 3] 1 shows an example of a point cloud, geometry, and texture image according to an embodiment. [Figure 4] 10 illustrates an example of a V-PCC encoding process according to an embodiment. [Figure 5] 1 shows an example of a tangent plane and a normal vector of a surface according to an embodiment. [Figure 6] 1 illustrates an example of a bounding box for a point cloud according to an embodiment. [Figure 7] 1 illustrates an example of positioning individual patches of an occupancy map according to an embodiment. [Figure 8] 1 shows an example of the relationship between normal, tangent, and bitangent axes according to an embodiment. [Figure 9] 10 shows an example of configurations of minimum and maximum modes of the projection mode according to an embodiment. [Figure 10] 1 shows an example of an EDD code according to an embodiment. [Figure 11]10 shows an example of recoloring using color values ​​of neighboring points according to an embodiment. [Figure 12] 1 illustrates an example of push-pull background filling according to an embodiment. [Figure 13] 1 shows an example of a possible traversal order for a block of size 4*4 according to an embodiment. [Figure 14] 10 illustrates an example of a best traversal order according to an embodiment. [Figure 15] 1 illustrates an example of a 2D video / image encoder according to an embodiment. [Figure 16] 1 illustrates an example of a V-PCC decoding process according to an embodiment. [Figure 17] 1 illustrates an example of a 2D video / image decoder according to an embodiment. [Figure 18] 10 shows an example of the flow of operations of a transmitting device according to an embodiment. [Figure 19] 10 shows an example of the flow of operations of a receiving device according to an embodiment. [Figure 20] 1 illustrates an example architecture for V-PCC-based point cloud data storage and streaming, according to an embodiment. [Figure 21] 1 illustrates an example of the configuration of a point cloud data storage and transmission device according to an embodiment. [Figure 22] 1 illustrates an example of the configuration of a point cloud data receiving device according to an embodiment. [Figure 23] 1 illustrates an example of a structure that can be linked with a method / apparatus for transmitting and receiving point cloud data according to an embodiment. [Figure 24] 1 illustrates an example of a V-PCC bitstream structure according to an embodiment. [Figure 25] FIG. 1 is a visualization of the structure of a multiple track V-PCC file according to an embodiment. [Figure 26] 10 shows an example of track types of PCC data assigned to data types (data_type) according to an embodiment. [Figure 27] 10 shows an example of an attribute type assigned to an attribute_type according to an embodiment. [Figure 28] 10 illustrates an example of a syntax structure of a profile tier level (profile_tier_level) according to an embodiment. [Figure 29] 10 illustrates an example of PCC data types within a referenced track according to an embodiment. [Figure 30] 10 illustrates an example of a characteristic type according to an embodiment. [Figure 31] 10 illustrates an example of a characteristic type according to an embodiment. [Figure 32] 10 illustrates an example of dividing a pad frame into multiple tiles according to an embodiment. [Figure 33] 10 illustrates an example of the syntax structure of each V-PCC unit according to an embodiment. [Figure 34] 10 illustrates an example of a syntax structure of a V-PCC unit header according to an embodiment. [Figure 35] 10 illustrates an example of a V-PCC unit type assigned to a vpcc_unit_type field according to an embodiment. [Figure 36] 10 illustrates an example of the syntax structure of pcm_separate_video_data(bitCount) according to an embodiment. [Figure 37] 10 illustrates an example of a syntax structure of a V-PCC unit payload according to an embodiment. [Figure 38] 10 illustrates an example of the syntax structure of a sequence parameter set () included in a V-PCC unit payload according to an embodiment. [Figure 39] 10 illustrates an example of a syntax structure of a patch data group according to an embodiment. [Figure 40] 10 illustrates an example of a patch data group type assigned to a pdg_unit_type field of a patch data group according to an embodiment. [Figure 41] 10 illustrates an example syntax structure of a patch data group unit payload according to an embodiment. [Figure 42] 10 illustrates an example of the syntax structure of a Supplemental Enhancement Information (SEI) message (sei_message()) according to an embodiment. [Figure 43] 10 illustrates an example of a V-PCC bitstream structure according to another embodiment of the present disclosure. [Figure 44] 10 illustrates an example of data carried by a sample stream V-PCC unit in a V-PCC bitstream according to an embodiment. [Figure 45] 10 illustrates an example of a syntax structure of a sample stream V-PCC header included in a V-PCC bitstream according to an embodiment. [Figure 46] 10 illustrates an example of a syntax structure of a sample stream V-PCC unit according to an embodiment. [Figure 47] 10 illustrates an example of a syntax structure of a V-PCC unit according to an embodiment. [Figure 48] 10 illustrates an example of a syntax structure of a V-PCC unit header according to an embodiment. [Figure 49] 10 illustrates an example of a V-PCC unit type assigned to a vuh_unit_type field according to an embodiment. [Figure 50] 10 illustrates an example of a syntax structure of a V-PCC unit payload (vpcc_unit_payload()) according to an embodiment. [Figure 51] 10 illustrates an example of an atlas substream structure according to an embodiment. [Figure 52] 10 illustrates an example of the syntax structure of a sample stream NAL header (sample_stream_nal_header()) included in an atlas substream according to an embodiment. [Figure 53] 1 illustrates an example of a syntax structure of a sample stream NAL unit according to an embodiment. [Figure 54]10 illustrates an example of the syntax structure of an SEI message payload (sei_payload()) according to an embodiment. [Figure 55] 10 illustrates an example of a V-PCC sample entry structure according to an embodiment. [Figure 56] 10 illustrates an example of generating a view using a virtual camera and ViewInfoStruct information according to an embodiment. [Figure 57] 10 illustrates an example of generating a view using ObjectRenderingInfoStruct() information according to an embodiment. [Figure 58] 10 illustrates an example of a structure for encapsulating non-timed V-PCC data according to an embodiment. [Figure 59] 1 illustrates an example of how a spatial portion of point cloud data within a viewing frustum is actually displayed on a viewport according to an embodiment. [Figure 60] 1 illustrates an example of a top view of a view frustum according to an embodiment. [Figure 61] 1 illustrates an example of a side view of a view frustum according to an embodiment. [Figure 62] 1 shows a flow of a point cloud data transmission method according to an embodiment. [Figure 63] 1 shows a flow of a point cloud data receiving method according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0042] Preferred embodiments will now be described in detail with reference to the accompanying drawings. The following detailed description, with reference to the accompanying drawings, is intended to illustrate preferred embodiments rather than merely illustrating embodiments that may be implemented by the embodiments. The following detailed description includes details to provide a thorough understanding of the embodiments. However, it will be apparent to those skilled in the art that embodiments may be practiced without such details.

[0043] Although most of the terms used in the examples are common and widely used in the relevant fields, some of them have been arbitrarily selected by the applicant, and their meanings will be explained in detail below as necessary. Therefore, the examples should be understood based on the intended meaning of the terms, rather than the simple names or meanings of the terms.

[0044] FIG. 1 illustrates an example of a transmitting / receiving system architecture for providing point cloud content according to an embodiment.

[0045] This document provides a method for providing point cloud content to provide users with various services such as VR (Virtual Reality), AR (Augmented Reality), MR (Mixed Reality), and autonomous driving services. Point cloud content according to an embodiment refers to data representing objects using points, and may be referred to as a point cloud, point cloud data, point cloud video data, point cloud image data, etc.

[0046] A point cloud data transmission device 10000 according to an embodiment includes a point cloud video acquisition unit 10001, a point cloud video encoder 10002, a file / segment encapsulation unit 10003, and / or a transmitter (or communication module) 10004. The transmission device according to an embodiment can acquire, process, and transmit point cloud video (or point cloud content). Depending on the embodiment, the transmission device may include a fixed station, a base transceiver system (BTS), a network, an AI (artificial intelligence) device and / or system, a robot, an AR / VR / XR device and / or server, etc. Also, depending on the embodiment, the transmitting device 10000 may include devices that communicate with base stations and / or other wireless devices using wireless connection technologies (e.g., 5G NR (New RAT), LTE (Long Term Evolution)), robots, vehicles, AR / VR / XR devices, mobile devices, home appliances, IoT (Internet of Things) devices, AI devices / servers, etc.

[0047] A Point Cloud Video Acquisition unit 10001 according to the embodiment acquires a point cloud video through a process such as capturing, synthesizing or generating a point cloud video.

[0048] A point cloud video encoder 10002 according to an embodiment encodes the point cloud video data acquired by the point cloud video acquisition unit 10001. Depending on the embodiment, the point cloud video encoder 10002 may be referred to as a point cloud encoder, a point cloud data encoder, an encoder, etc. Also, point cloud compression coding (encoding) according to an embodiment is not limited to the above-described embodiment. The point cloud video encoder can output a bitstream including encoded point cloud video data. The bitstream may include not only the encoded point cloud video data but also signaling information related to the encoding of the point cloud video data.

[0049] The point cloud video encoder 10002 according to the embodiment can support both the Geometry-based Point Cloud Compression (G-PCC) encoding method and / or the Video-based Point Cloud Compression (V-PCC) encoding method, and can encode a point cloud (referring to all point cloud data or points) and / or signaling data related to the point cloud.

[0050] A File / Segment Encapsulation module 10003 according to the embodiment encapsulates point cloud data in the form of a file and / or a segment. A point cloud data transmission method / apparatus according to the embodiment can transmit point cloud data in the form of a file and / or a segment.

[0051] According to an embodiment, a transmitter (or communication module) 10004 transmits encoded point cloud video data in the form of a bitstream. Depending on the embodiment, the file or segment may be transmitted to a receiving device via a network or stored on a digital storage medium (e.g., USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc.). According to an embodiment, a transmitter may perform wired / wireless communication with a receiving device (or receiver) via a network such as 4G, 5G, or 6G. The transmitter may also perform necessary data processing operations depending on a network system (e.g., a communication network system such as 4G, 5G, or 6G). The transmitting device may also transmit encapsulated data on an on-demand basis.

[0052] A point cloud data receiving device 10005 according to an embodiment includes a receiver 10006, a file / segment decapsulator 10007, a point cloud video decoder 10008, and / or a renderer 10009. Depending on the embodiment, the receiving device may include a device, robot, vehicle, AR / VR / XR device, mobile device, home appliance, IoT (Internet of Things) device, AI device / server, etc. that communicates with a base station and / or other wireless devices using a wireless connection technology (e.g., 5G NR (New RAT), LTE (Long Term Evolution)).

[0053] A receiver 10006 according to an embodiment receives a bitstream including point cloud video data. According to an embodiment, the receiver 10006 can transmit feedback information to the point cloud data transmitting device 10000.

[0054] The File / Segment Decapsulation module 10007 decapsulates files and / or segments containing point cloud data.

[0055] A point cloud video decoder 10008 decodes the received point cloud video data.

[0056] A Renderer 10009 renders the decoded point cloud video data. According to an embodiment, the Renderer 10009 may transmit feedback information acquired at the receiving end to the point cloud video decoder 10008. According to an embodiment, the point cloud video data may transmit feedback information to the receiver 10006. According to an embodiment, the feedback information received by the point cloud transmitting device may be provided to the point cloud video encoder 10002.

[0057] In the drawing, dotted arrows indicate transmission paths of feedback information acquired by the receiving device 10005. The feedback information is information for reflecting interaction with a user consuming point cloud content and includes user information (e.g., head orientation information, viewport information, etc.). In particular, if the point cloud content is for a service (e.g., an autonomous driving service) that requires interaction with a user, the feedback information may be transmitted to a content transmitting side (e.g., the transmitting device 10000) and / or a service provider. Depending on the embodiment, the feedback information may be used by not only the transmitting device 10000 but also the receiving device 10005, or may not be provided.

[0058] According to an embodiment, head orientation information is information regarding the position, direction, angle, movement, etc. of the user's head. According to an embodiment, the receiving device 10005 can calculate viewport information based on the head orientation information. The viewport information is information regarding the area of ​​the point cloud video viewed by the user. The viewpoint (or orientation) is the point at which the user views the point cloud video and refers to the center of the viewport area. That is, the viewport is an area centered on the viewpoint, and the size, shape, etc. of the area are determined by the FOV (Field of View). In other words, the viewport is determined according to the position and viewpoint (or orientation) of the virtual camera or the user, and the point cloud data is rendered in the viewport based on the viewport information. Therefore, the receiving device 10005 can extract viewport information based on the vertical or horizontal FOV supported by the device, in addition to the head orientation information. The receiving device 10005 also performs gaze analysis to identify the user's point cloud consumption method, the point cloud video area the user gazes at, and the gaze time, etc. According to an embodiment, the receiving device 10005 can transmit feedback information including the results of gaze analysis to the transmitting device 10000. The feedback information according to the embodiment is obtained during the rendering and / or display process. According to the embodiment, the feedback information is obtained by one or more sensors included in the receiving device 10005. According to another embodiment, the feedback information is obtained by the renderer 10009 or another external element (or device, component, etc.). The dotted lines shown in FIG. 1 indicate the transmission process of the feedback information obtained by the renderer 10009. The point cloud content providing system processes (encodes / decodes) the point cloud data based on the feedback information. Therefore, the point cloud video data decoder 10008 can perform a decoding operation based on the feedback information.The receiving device 10005 can also transmit feedback information to the transmitting device. The transmitting device (or the point cloud video data encoder 10002) can perform encoding operations based on the feedback information. Therefore, the point cloud content providing system can efficiently process necessary data (e.g., point cloud data corresponding to the user's head position) based on the feedback information without processing (encoding / decoding) all point cloud data, and provide point cloud content to the user.

[0059] In the embodiment, the sending device 10000 is referred to as an encoder, a sending device, a transmitter, etc., and the receiving device 10005 is referred to as a decoder, a receiving device, a receiver, etc.

[0060] 1 according to an embodiment (processed in a series of processes of acquisition / encoding / transmission / decoding / rendering), is also referred to as point cloud content data or point cloud video data. According to an embodiment, the point cloud content data can be used as a concept including metadata or signaling information related to the point cloud data.

[0061] The elements of the point cloud content providing system shown in FIG. 1 may be embodied in hardware, software, a processor, and / or a combination thereof.

[0062] The embodiment can provide point cloud content to provide users with various services such as VR (Virtual Reality), AR (Augmented Reality), MR (Mixed Reality), and autonomous driving services.

[0063] To provide a point cloud content service, a point cloud video is first acquired. The acquired point cloud video undergoes a series of processes and is then transmitted to a receiving side, where the received data is processed and rendered back into the original point cloud video. This allows the point cloud video to be provided to a user. The embodiment provides a solution for effectively performing this series of processes.

[0064] The overall process for providing point cloud content services (point cloud data transmitting method and / or point cloud data receiving method) includes an acquisition process, an encoding process, a transmission process, a decoding process, a rendering process and / or a feedback process.

[0065] According to an embodiment, the process of providing point cloud content (or point cloud data) is also called a point cloud compression process, which means a video-based point cloud compression (hereinafter referred to as V-PCC) process.

[0066] Each element of the point cloud data transmitting device and the point cloud data receiving device according to the embodiment means hardware, software, a processor, and / or a combination thereof.

[0067] The point cloud compression system may include a transmitting device and a receiving device. According to an embodiment, the transmitting device may be referred to as an encoder, a transmitting device, a transmitter, a point cloud transmitting device, etc. According to an embodiment, the receiving device may be referred to as a decoder, a receiving device, a receiver, a point cloud receiving device, etc. The transmitting device may encode the point cloud video and output a bitstream, which may be transmitted to the receiving device via a digital storage medium or a network in the form of a file or streaming (streaming segment). The digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc.

[0068] The transmitting device may include a point cloud video acquisition unit, a point cloud video encoder, a file / segment encapsulation unit, and a transmission unit (or transmitter) as shown in FIG. 1. The receiving device may generally include a reception unit, a file / segment decapsulation unit, a point cloud video decoder, and a renderer as shown in FIG. 1. The encoder is also referred to as a point cloud video / video / picture / frame encoding device, and the decoder is also referred to as a point cloud video / video / picture / frame decoding device. The renderer may include a display unit, and the renderer and / or the display unit may be configured as separate devices or external components. The transmitting device and the receiving device may further include another internal or external module / unit / component for a feedback process. Each element included in the transmitting device and the receiving device according to the embodiment may be configured as hardware, software, and / or a processor.

[0069] The operation of the receiving device according to the embodiment follows the reverse process of the operation of the transmitting device.

[0070] The point cloud video acquisition unit performs a process of acquiring a point cloud video through a point cloud video capture, synthesis, or generation process. The acquisition process generates 3D position (x, y, z) / characteristic (color, reflectance, transparency, etc.) data for a large number of points, such as a PLY (Polygon File format or the Stanford Triangle format) file. For videos with multiple frames, one or more files can be acquired. Metadata related to the point cloud (e.g., metadata about the capture) is generated during the capture process.

[0071] A point cloud data transmission device according to an embodiment may include an encoder for encoding the point cloud data, and a transmitter for transmitting (or a bitstream including) the point cloud data.

[0072] A point cloud data receiving apparatus according to an embodiment includes a receiver that receives a bitstream including point cloud data, a decoder that decodes the point cloud data, and a renderer that renders the point cloud data.

[0073] The method / apparatus according to the embodiment represents a point cloud data transmitting device and / or a point cloud data receiving device.

[0074] FIG. 2 illustrates an example of capturing point cloud data according to an embodiment.

[0075] The point cloud data (or point cloud video data) according to the embodiment is acquired by a camera, etc. The capturing method according to the embodiment may be, for example, inward-facing and / or outward-facing.

[0076] The inward-facing method according to the embodiment is a capture method in which one or more cameras capture an object of point cloud data from the outside to the inside of the object.

[0077] In the outward facing method according to the embodiment, one or more cameras capture an object of point cloud data from the inside to the outside of the object. For example, in the embodiment, there are four cameras.

[0078] According to an embodiment, point cloud data or point cloud content may be video or still images of objects / environments represented in various forms in 3D space. According to an embodiment, point cloud content may include video / audio / images of objects.

[0079] Equipment for capturing point cloud content consists of a camera capable of acquiring depth (a combination of an infrared pattern projector and an infrared camera) and an RGB camera capable of extracting color information corresponding to the depth information. Alternatively, depth information can be extracted using LiDAR, a radar system that measures the position coordinates of reflectors by emitting a laser pulse and measuring the time it takes for the pulse to reflect back. Geometry, consisting of points in 3D space, can be extracted from the depth information, and attributes representing the color / reflectance of each point can be extracted from the RGB information. Point cloud content may consist of position (x, y, z), color (YCbCr or RGB), or reflectance (r) information for each point. Point cloud content can be created in two ways: outward-facing, which captures the external environment, and inward-facing, which captures a central object. To create point cloud content that allows users to freely view an object (e.g., a character, player, object, actor, or other core object) in a VR / AR environment in 360°, the capture camera is configured in an inward-facing manner. In addition, when creating point cloud content from the current surroundings of a vehicle, such as in an autonomous driving vehicle, the capture camera configuration is outward-facing. Because point cloud content is captured by multiple cameras, a camera calibration process may be required before capturing content to set a global coordinate system between the cameras.

[0080] Point cloud content is video or still footage of objects / environments shown in various forms in 3D space.

[0081] Alternatively, the point cloud content can be obtained by synthesizing any point cloud video based on the captured point cloud video. Alternatively, when providing a point cloud video for a computer-generated virtual space, actual camera capture may not be performed. In this case, the capture process can be replaced by a process that simply generates the relevant data.

[0082] The captured point cloud video requires post-processing to improve the quality of the content. Although the maximum and minimum depth values ​​can be adjusted within the range provided by the camera equipment during the video capture process, point data from undesired areas may still be included. Therefore, post-processing can be performed to remove undesired areas (e.g., background) or fill spatial holes by recognizing connected spaces. Point clouds extracted from cameras sharing a spatial coordinate system can be integrated into a single content by converting each point to a global coordinate system based on the position coordinates of each camera acquired through a calibration process. This allows for the generation of a single point cloud content covering a wide area, or for the acquisition of point cloud content with a high point density.

[0083] The point cloud video encoder 10002 can encode the input point cloud video into one or more video streams. A point cloud video may include multiple frames, with each frame corresponding to a still image / picture. In this document, point cloud video includes point cloud images / frames / pictures / video / audio / images, and point cloud video can be mixed with point cloud images / frames / pictures. The point cloud video encoder 10002 performs a video-based point cloud compression (V-PCC) procedure. For compression and coding efficiency, the point cloud video encoder 10002 performs a series of procedures, such as prediction, transformation, quantization, and entropy coding. The encoded data (encoded video / image information) is output in the form of a bitstream. Based on the V-PCC procedure, the point cloud video encoder 10002 encodes the point cloud video into geometry video, attribute video, occupancy map video, and auxiliary information, as described below. Geometry videos include geometry images, attribute videos include attribute images, and occupancy map videos include occupancy map images. Additional information (also called additional data) includes auxiliary patch information. Attribute videos / images include texture videos / images.

[0084] The encapsulation unit (file / segment encapsulation module) 10003 may encapsulate the encoded point cloud video data and / or point cloud video-related metadata in a format such as a file. Here, the point cloud video-related metadata may be transmitted from a metadata processing unit or the like. The metadata processing unit may be included in the point cloud video encoder 10002 or may be configured as a separate component / module. The encapsulation unit 10003 may encapsulate the corresponding data in a file format such as ISOBMFF, or may process the data in other formats such as DASH segments. According to an embodiment, the encapsulation unit 10003 may include the point cloud video-related metadata in a file format. For example, the point cloud video-related metadata may be included in boxes at various levels in the ISOBMFF file format, or in a separate track within the file. According to an embodiment, the encapsulation unit 10003 may encapsulate the point cloud video-related metadata itself in a file. The transmission processing unit may perform processing for transmission on the point cloud video data encapsulated in the file format. The transmission processing unit may be included in the transmitting unit 10004 or may be configured as a separate component / module. The transmission processing unit processes the point cloud video data according to any transmission protocol. The processing for transmission includes processing for transmission via a broadcast network and processing for transmission via broadband. The transmission processing unit according to the embodiment processes not only the point cloud video data but also the point cloud video-related metadata transmitted from the metadata processing unit for transmission.

[0085] The transmitter 10004 transmits the encoded video / image information or data output in the form of a bitstream to the receiver 10006 of the receiving device in the form of a file or streaming via a digital storage medium or a network. Digital storage media include USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmitter includes elements for generating a media file in a predetermined file format and elements for transmission via a broadcast / communication network. The receiver extracts the bitstream and transmits it to a decoding device.

[0086] The receiver 10006 receives the point cloud video data transmitted by the point cloud video transmitting device of the present invention. Depending on the channel, the receiver may receive the point cloud video data via a broadcast network, broadband, or digital storage medium.

[0087] The receiving processing unit processes the received point cloud video data in accordance with the transmission protocol. The receiving processing unit may be included in the receiver 10006 or may be configured as a separate component / module. Corresponding to the processing for transmission performed on the transmitting side, the receiving processing unit performs the reverse process of the transmitting processing unit described above. The receiving processing unit transmits the acquired point cloud video data to the decapsulating unit 10007, and transmits the acquired point cloud video-related metadata to a metadata processing unit (not shown). The point cloud video-related metadata acquired by the receiving processing unit may be in the form of a signaling table.

[0088] The decapsulation unit (file / segment decapsulation module) 10007 decapsulates the point cloud video data in a file format transmitted from the receiving processing unit. The decapsulation processing unit 10007 decapsulates a file in ISOBMFF format or the like to obtain a point cloud video bitstream or point cloud video-related metadata (metadata bitstream). The obtained point cloud video bitstream is transmitted to the point cloud video decoder 10008, and the obtained point cloud video-related metadata (metadata bitstream) is transmitted to a metadata processing unit (not shown). The point cloud video bitstream may include metadata (metadata bitstream). The metadata processing unit may be included in the point cloud video decoder 10008 or may be configured as a separate component / module. The point cloud video-related metadata obtained by the decapsulation processing unit 10007 may be in the form of a box or track in a file format. If necessary, the decapsulation processing unit 10007 receives metadata required for decapsulation from the metadata processing unit. The point cloud video related metadata may be transmitted to a point cloud video decoder 10008 for use in the point cloud video decoding procedure, or may be transmitted to a renderer 10009 for use in the point cloud video rendering procedure.

[0089] The point cloud video decoder 10008 receives a bitstream and performs operations corresponding to those of the point cloud video encoder to decode video / images. In this case, the point cloud video decoder 10008 separates the point cloud video into geometry video, attribute video, occupancy map video, and auxiliary information, as described below, and decodes it. The geometry video includes geometry images, the attribute video includes attribute images, and the occupancy map video includes occupancy map images. The auxiliary information includes auxiliary patch information. The attribute video / image includes texture video / images.

[0090] The 3D geometry is reconstructed using the decoded geometry image, occupancy map, and additional patch information, and may then undergo a smoothing process. A color point cloud image / picture is reconstructed by assigning color values ​​to the smoothed 3D geometry using a texture image. The renderer 10009 renders the reconstructed geometry and color point cloud image / picture. The rendered video / image is displayed by a display unit (not shown). The user views all or part of the rendered result on a VR / AR display or a general display.

[0091] The feedback process may include a process of transmitting various feedback information obtainable in the rendering / display process to the transmitting side or to a decoder on the receiving side. The feedback process provides interactivity in the consumption of the point cloud video. According to an embodiment, head orientation information, viewport information indicating the area the user is currently viewing, etc. are transmitted in the feedback process. According to an embodiment, a user may interact with something embodied in a VR / AR / MR / autonomous driving environment, and in this case, information related to the interaction may be transmitted to the transmitting side or service provider in the feedback process. Depending on the embodiment, the feedback process may not be performed.

[0092] The head orientation information is information about the position, angle, movement, etc. of the user's head. Based on this information, information about the area the user is currently looking at in the point cloud video, i.e., viewport information, is calculated.

[0093] Viewport information is information about the area the user is currently looking at in the point cloud video. This allows for gaze analysis to determine how the user consumes the point cloud video and how long they gaze at which area of ​​the point cloud video. Gaze analysis is performed on the receiving side and transmitted to the sending side via a feedback channel. Devices such as VR / AR / MR displays extract the viewport area based on the user's head position / direction, the vertical or horizontal FOV supported by the device, etc.

[0094] According to an embodiment, the above-mentioned feedback information may be not only transmitted to the transmitting side but also consumed by the receiving side. That is, the above-mentioned feedback information may be used to perform decoding, rendering, etc. on the receiving side. For example, head orientation information and / or viewport information may be used to prioritize decoding and rendering of only the point cloud video for the area currently viewed by the user.

[0095] Here, the viewport or viewport area refers to the area the user is looking at in the point cloud video. The viewpoint refers to the point the user is looking at in the point cloud video, which is the center of the viewport area. That is, the viewport is an area centered on the viewpoint, and the size and shape of the area are determined by the FOV (Field Of View).

[0096] As mentioned above, this specification relates to point cloud video compression, for example, the methods / embodiments disclosed in this specification may be applied to the Moving Picture Experts Group (MPEG) point cloud compression or point cloud coding (PCC) standard or next generation video / image coding standards.

[0097] In this specification, a picture / frame generally refers to a unit showing one image at a specific time period.

[0098] A pixel or pel is the smallest unit that makes up a picture (or image). The term "sample" is also used as a counterpart to a pixel. A sample generally refers to a pixel or a pixel value, and may refer to only a pixel / pixel value of a luminance (luma) component, only a pixel / pixel value of a chroma component, or only a pixel / pixel value of a depth component.

[0099] A unit refers to a basic unit of image processing. A unit includes at least one of a specific region of a picture and information about that region. The term unit is sometimes used interchangeably with terms such as block, area, or module. In general, an MxN block includes a set (or array) of samples or transform coefficients consisting of M columns and N rows.

[0100] FIG. 3 shows an example of a point cloud, geometry, and texture image according to an embodiment.

[0101] The point cloud according to the embodiment is input to the V-PCC encoding process in Fig. 4, which will be described later, to generate a geometry image and a texture image. According to the embodiment, the point cloud is used interchangeably with point cloud data.

[0102] In Figure 3, the left diagram shows a point cloud, where point cloud objects are located in 3D space and are represented by bounding boxes, etc. The middle diagram in Figure 3 shows a geometry image, and the right diagram shows a texture image (non-padded). In this specification, a geometry image is also referred to as a geometry patch frame / picture or a geometry frame / picture. A texture image is also referred to as a feature patch frame / picture or a feature frame / picture.

[0103] Video-based Point Cloud Compression (V-PCC) is a method for compressing 3D point cloud data based on 2D video codecs such as HEVC (Efficiency Video Coding) and VVC (Versatile Video Coding). The V-PCC compression process generates the following data and information:

[0104] Occupancy map: When the points that make up a point cloud are divided into patches and mapped onto a 2D plane, this refers to a binary map that indicates the presence or absence of data at a corresponding location on the 2D plane with a value of 0 or 1. An occupancy map represents a 2D array that corresponds to an atlas, and the values ​​of the occupancy map indicate whether each sample location in the atlas corresponds to a 3D point. An atlas is an object that contains information about the 2D patches for each point cloud frame. For example, an atlas may include the 2D placement and size of patches, the location of the corresponding 3D region within the 3D points, a projection plan, and LOD (Level of Detail) parameters.

[0105] Patch: A set of points that make up a point cloud. Points that belong to the same patch are adjacent to each other in 3D space and map to the same direction in the six-sided bounding box plane during the mapping process to a 2D image.

[0106] Geometry image: A depth map-style image that represents the geometry of each point in a point cloud, in units of patches. A geometry image consists of a single channel of pixel values. The geometry represents a set of coordinates associated with the point cloud frame.

[0107] Texture image: An image that represents the color information of each point in a point cloud on a patch-by-patch basis. A texture image consists of pixel values ​​for multiple channels (e.g., three channels R, G, B). Textures are included in features. Depending on the embodiment, textures and / or features are interpreted as the same object and / or as containing features.

[0108] Auxiliary patch info: Indicates the metadata required to reconstruct a point cloud from individual patches. The auxiliary patch info includes information about the patch's position in 2D / 3D space, size, etc.

[0109] Point cloud data, eg, V-PCC components, according to an embodiment, include atlases, occupancy maps, geometry, features, etc.

[0110] An atlas represents a collection of 2D bounding boxes, i.e., patches projected into a rectangular frame that correspond to a 3-dimensional bounding box in 3D space, which may represent a subset of a point cloud. In this case, a patch represents a rectangular region in the atlas that corresponds to a rectangular region in the planar projection. Patch data represents the data required to transform the patches contained in the atlas from 2D to 3D. In addition, a group of patch data is also called an atlas.

[0111] An attribute is a scalar or vector associated with each point in the point cloud, such as color, reflectance, surface normal, time stamps, or material ID.

[0112] The point cloud data according to the embodiment represents PCC data in a video-based point cloud compression (V-PCC) format. The point cloud data includes multiple components, such as an occupancy map, patches, geometry, and / or textures.

[0113] FIG. 4 illustrates an example of a point cloud video encoder according to an embodiment.

[0114] Figure 4 shows a V-PCC encoding process for generating and compressing an occupancy map, a geometry image, a texture image, and auxiliary patch information. The V-PCC encoding process in Figure 4 is processed by the point cloud video encoder 10002 in Figure 1. Each component in Figure 4 is implemented by software, hardware, a processor, and / or a combination thereof.

[0115] The patch generation unit 14000 receives a point cloud frame (which may be in the form of a bitstream containing point cloud data). The patch generation unit 40000 generates a patch from the point cloud data. The patch generation unit 40000 also generates patch information containing information related to the generation of the patch.

[0116] The patch packing (or patch packing unit) 40001 packs one or more patches and generates an occupancy map containing information about the patch packing.

[0117] The geometry image generation unit 40002 generates a geometry image based on the point cloud data, patch information (or additional patch information), and / or occupancy map information. The geometry image refers to data including geometry related to the point cloud data (i.e., 3D coordinate values ​​of points), and is also called a geometry frame.

[0118] The texture image generation unit 40003 generates a texture image based on point cloud data, patches, packed patches, patch information (or additional patch information), and / or smoothed geometry. A texture image is also called a feature frame. A texture image can also be generated based on smoothed geometry generated by smoothing a reconstructed geometry image based on patch information.

[0119] The smoothing unit 40004 reduces or removes errors contained in the image data. For example, the reconstructed geometry image can be smoothed based on patch information, i.e., smoothed by gently filtering out portions that may cause errors between data, to generate smoothed geometry. The smoothed geometry is output to the texture image generation unit 40003.

[0120] The auxiliary patch info compression (or auxiliary patch info compression unit) 40005 compresses auxiliary patch information related to the patch information generated in the patch generation process. The auxiliary patch information compressed by the auxiliary patch info compression unit 40005 is transmitted to the multiplexer 40013. The geometry image generation 40002 uses the auxiliary patch information when generating a geometry image.

[0121] Image pads (or image padding units) 40006 and 40007 pad the geometry image and the texture image, respectively, i.e., pad data is padded into the geometry image and the texture image.

[0122] Group dilation 40008 adds data to a texture image, similar to an image pad. Additional patch information is inserted into the texture image.

[0123] Video compression (video compression units) 40009, 40010, 40011 compress the padded geometry image, padded texture image, and / or occupancy map, respectively. In other words, the video compression units 40009, 40010, 40011 compress the input geometry frames, feature frames, and / or occupancy map frames, respectively, and output a geometry video bitstream, a texture image video bitstream, and an occupancy map video bitstream. Video compression encodes geometry information, texture information, occupancy information, etc.

[0124] The entropy compression (entropy compression unit) 40012 compresses the occupancy map based on an entropy method.

[0125] According to an embodiment, if the point cloud data is lossless and / or lossy, entropy compression and / or video compression is performed on the occupancy map frames.

[0126] The multiplexer 40013 multiplexes the video bitstream of the compressed geometry, the video bitstream of the compressed texture image, the video bitstream of the compressed occupancy map, and the bitstream of the compressed additional patch information from each compression unit into one bitstream.

[0127] The blocks described above may be omitted or replaced with blocks having similar or identical functions. Each block shown in Figure 4 operates as at least one of a processor, software, and hardware.

[0128] The detailed operation of each process in FIG. 4 according to the embodiment will be described below.

[0129] Patch generation: 40000

[0130] The patch generation process is the process of dividing a point cloud into patches, which are the units of mapping, in order to map the point cloud to a 2D image. The patch generation process can be divided into three steps: normal value calculation, segmentation, and patch division as follows.

[0131] The process of calculating the normalized value will be specifically described with reference to FIG.

[0132] FIG. 5 shows an example of a tangent plane and a normal vector of a surface according to an embodiment.

[0133] The surface of FIG. 5 is used in the patch generation process 40000 of the V-PCC encoding process of FIG. 4 as follows.

[0134] Normal calculation related to patch generation

[0135] Each point (e.g., a point) that makes up a point cloud has a unique direction, which is represented by a 3D vector called a normal. Using the neighbors of each point, which are found using a KD tree or similar, the tangent plane and normal vector of each point that makes up the surface of the point cloud, as shown in Figure 5, are found. The search range in the process of finding neighboring points is defined by the user.

[0136] Tangent plane: A plane that passes through a point on a surface and completely contains the tangent to a curve on the surface.

[0137] FIG. 6 illustrates an example of a bounding box for a point cloud according to an embodiment.

[0138] The bounding box in this embodiment is a unit box that divides point cloud data based on a hexahedron in 3D space.

[0139] The method / apparatus according to the embodiment, for example, patch generation 4000, uses bounding boxes in the process of generating patches from point cloud data.

[0140] Bounding boxes are used in the process of projecting point cloud objects, which are the target of point cloud data, onto the planes of each hexahedron based on a hexahedron in 3D space. The bounding boxes are generated and processed by the point cloud video acquisition unit 10001 and point cloud video encoder 10002 in Figure 1. Furthermore, patch generation 40000, patch packing 40001, geometry image generation 40002, and texture image generation 40003 in the V-PCC encoding process in Figure 4 are performed based on the bounding boxes.

[0141] Segmentation in relation to patch generation

[0142] Segmentation consists of two processes: initial segmentation and refinement segmentation.

[0143] The point cloud video encoder 10002 according to the embodiment projects points onto one side of a bounding box. Specifically, each point constituting a point cloud is projected onto one side of six bounding boxes surrounding the point cloud, as shown in Fig. 6, and initial segmentation is a process of determining one of the planes of the bounding boxes onto which each point is projected.

[0144] The normal values ​​corresponding to each of the six planes are JPEG2024159914000002.jpg814 is defined as follows:

[0145] (1.0, 0.0, 0.0), (0.0, 1.0, 0.0), (0.0, 0.0, 1.0), (-1.0, 0.0, 0.0), (0.0, -1.0, 0.0), (0.0, 0.0, -1.0).

[0146] The normal value of each point obtained from the normal value calculation process described above is shown in the following formula ( JPEG2024159914000003.jpg89) and The plane with the largest dot product of JPEG2024159914000004.jpg914 is determined to be the projection plane of that plane. That is, the plane with a normal oriented most similarly to the normal of the point is determined to be the projection plane of that point.

[0147] JPEG2024159914000005.jpg1341

[0148] The determined plane is identified as an index-type value (cluster index) of any one of 0 to 5.

[0149] Refine segmentation is a process of improving the projection plane of each point of the point cloud determined in the above-mentioned initial segmentation process by considering the projection planes of adjacent points. In this process, score normal, which indicates the similarity between the normal of each point considered to determine the projection plane in the above-mentioned initial segmentation process and the normal value of each plane of the bounding box, and score smooth, which indicates the degree of agreement between the projection plane of the current point and the projection plane of the adjacent points, are simultaneously considered.

[0150] Score smooth can be considered by assigning weights to score normal, where the weights are defined by the user. The refinement division is performed iteratively, and the number of iterations is also user-defined.

[0151] Segment patches in relation to patch generation

[0152] Patch division is a process of dividing the entire point cloud into patches, which are sets of adjacent points, based on the projection plane information of each point that makes up the point cloud obtained in the initial / improved division process described above. Patch division consists of the following steps:

[0153] (1) Calculate the neighboring points of each point in the point cloud using a KD tree or similar. The maximum number of neighboring points is defined by the user.

[0154] (2) If the neighboring points project onto the same plane as the current point (have the same cluster index value), extract the current point and its neighboring points into one patch.

[0155] (3) Calculate the geometry value of the extracted patch.

[0156] (4) Repeat steps (2) and (3) until there are no more points left unextracted.

[0157] Through the process of patch division, the size of each patch and the occupancy map, geometry image, texture image, etc. of each patch are determined.

[0158] FIG. 7 illustrates an example of positioning individual patches of an occupancy map according to an embodiment.

[0159] The point cloud encoder 10002 according to the embodiment can generate patch packing and occupancy maps.

[0160] Patch packing & Occupancy map generation 40001

[0161] This process determines the positions of individual patches within a 2D image in order to map previously divided patches into a single 2D image. An occupancy map is a type of 2D image and is a binary map that indicates the presence or absence of data at that position with a value of 0 or 1. An occupancy map consists of blocks, and its resolution is determined according to the size of the blocks. For example, if the block size is 1*1, it has a resolution in pixels. The size of the blocks (occupancy packing block size) is determined by the user.

[0162] The process of determining the location of an individual patch in the occupancy map is as follows.

[0163] (1) Set all values ​​in the entire occupancy map to 0.

[0164] (2) Position the patch at a point (u, v) in the occupancy map plane whose horizontal coordinate is in the range [0, occupancySizeU-patch.sizeU0) and whose vertical coordinate is in the range [0, occupancySizeV-patch.sizeV0).

[0165] (3) Set the point (x, y) on the patch plane whose horizontal coordinate is in the range [0, patch.sizeU0) and whose vertical coordinate is in the range [0, patch.sizeV0) as the current point.

[0166] (4) For point (x, y), if the (x, y) coordinate value in the patch occupancy map is 1 (data exists at that point in the patch) and the (u+x, v+y) coordinate value in the global occupancy map is 1 (the occupancy map was filled by the previous patch), change the (x, y) position in raster order and repeat steps (3) to (4). Otherwise, perform step (6).

[0167] (5) Change the (u, v) position in the raster order and repeat the steps (3) to (5).

[0168] (6) (u, v) is determined as the location of the patch, and the occupancy map data of the patch is assigned (copied) to the corresponding part of the overall occupancy map.

[0169] (7) Repeat steps (2) to (6) for the next patch.

[0170] OccupancySizeU: Indicates the width of the occupancy map, and its unit is occupancy packing block size.

[0171] Occupancy Size V (occupancySizeV): Indicates the height of the occupancy map, and its unit is the occupancy packing block size.

[0172] Patch size U0 (patch.sizeU0): Indicates the width of the occupancy map, and its unit is the occupancy packing block size.

[0173] Patch size V0 (patch.sizeV0): Indicates the height of the occupancy map, in units of the occupancy packing block size.

[0174] For example, as shown in FIG. 7, there may be a box corresponding to a patch having an in-box patch size corresponding to the occupied packing size block, and a point (x, y) may be located within the box.

[0175] FIG. 8 shows an example of the relationship between the normal, tangent, and bitangent axes according to an embodiment.

[0176] The point cloud video encoder 10002 according to the embodiment can generate a geometry image. The geometry image means image data containing the geometry information of the point cloud. The generation process of the geometry image uses the three axes (normal, tangent, and bitangent) of the patch in Figure 8.

[0177] Geometry image generation 40002

[0178] In this process, the depth values ​​constituting the geometry image of each individual patch are determined, and the entire geometry image is generated based on the patch positions determined in the above-mentioned patch packing process. The process of determining the depth values ​​constituting the geometry image of each individual patch is configured as follows.

[0179] (1) Calculate parameters related to the position and size of each patch. The parameters include the following information. In one embodiment, the patch position is included in the patch information.

[0180] An index indicating the normal axis: The normal is obtained in the patch generation process described above, the tangent axis is the axis perpendicular to the normal that coincides with the horizontal (u) axis of the patch image, and the bitangent axis is the axis perpendicular to the normal that coincides with the vertical (v) axis of the patch image. The three axes are shown in Figure 8.

[0181] FIG. 9 illustrates an example of minimum and maximum mode configurations of the projection mode according to an embodiment.

[0182] The point cloud video encoder 10002 according to the embodiment performs patch-based projection to generate a geometry image, and the projection modes according to the embodiment include a minimum mode and a maximum mode.

[0183] 3D space coordinates of a patch: Calculated by the minimum size of the bounding box that surrounds the patch. For example, the 3D space coordinates of a patch include the minimum value of the patch tangent direction (patch 3D shift tangent axis), the minimum value of the patch bitangent direction (patch 3D shift bitangent axis), the minimum value of the patch normal direction (patch 3D shift normal axis), etc.

[0184] Patch 2D size: indicates the horizontal and vertical size of the patch when it is packed into a 2D image. The horizontal size (patch 2D size u) is the difference between the maximum and minimum values ​​of the tangent direction of the bounding box, and the vertical size (patch 2D size v) is the difference between the maximum and minimum values ​​of the bitangent direction of the bounding box.

[0185] (2) Determine the projection mode of the patch. The projection mode can be either the minimum mode or the maximum mode. The geometry information of the patch is represented by depth values, and when each point of the patch is projected in the normal direction of the patch, two layer images are generated: one consisting of the maximum depth value and the other consisting of the minimum depth value.

[0186] When generating two layer images d0 and d1 in minimum mode, the minimum depth is set to d0 as shown in Figure 9, and the maximum depth within the surface thickness from the minimum depth is set to d1.

[0187] For example, if the point cloud is located in 2D, as in Figure 9, there may be multiple patches containing multiple points. As in Figure 9, points shown with the same shading belong to the same patch. The process of projecting patches of points shown with blank spaces is shown.

[0188] When projecting a point indicated by a blank space to the left / right side, the depth is calculated by increasing the depth by one from the left side, such as 0, 1, 2, ..6, 7, 8, 9, and then writing the numbers for calculating the depth of the point on the right side.

[0189] The projection mode can be defined by the user, and the same method can be applied to all point clouds, or different methods can be applied to each frame or patch. When different projection modes are applied to each frame or patch, a projection mode that can improve compression efficiency or minimize missed points is adaptively selected.

[0190] (3) Calculate the depth value of each point.

[0191] In minimum mode, the d0 image is constructed with depth0, which is the minimum value of the normal axis at each point minus the minimum value of the normal direction of the patch (patch 3D shift normal axis) calculated in process (1). If there is another depth value within the range of depth0 and the surface thickness at the same position, this value is set to depth1. If not, the value of depth0 is also assigned to depth1. The d1 image is constructed with the value of Depth1.

[0192] For example, in determining the depth of the point d0, the minimum value is calculated (4 2 4 4 0 6 0 0 9 9 0 8 0). In determining the depth of the point d1, the largest value of two or more points is calculated, or if there is only one point, that value is calculated (4 4 4 4 6 6 6 8 9 9 8 8 9). In addition, in the process of encoding and reconstructing the points of the patch, some points are lost (for example, 8 points are lost in the figure).

[0193] In maximum mode, the d0 image is constructed with depth0, which is the maximum value of the normal axis at each point minus the minimum value of the normal direction of the patch (patch 3D shift normal axis) calculated in process (1). If there is another depth value at the same position within the range of depth0 and the surface thickness, this value is set to depth1. If not, the value of depth0 is also assigned to depth1. The d1 image is constructed with the value of Depth1.

[0194] For example, when determining the depth of the point d0, the maximum value is calculated (4 4 4 4 6 6 6 8 9 9 8 8 9). When determining the depth of the point d1, the smaller value of two or more points is calculated, or if there is only one point, that value is calculated (4 2 4 4 5 6 0 6 9 9 0 8 0). Also, in the process of encoding and reconstructing the points of the patch, some points are lost (for example, six points are lost in the figure).

[0195] The overall geometry image can be generated by placing the geometry images of the individual patches generated from the above process into the overall geometry image using the position information of the individual patches generated through the patch packing process described above.

[0196] The d1 layer of the generated overall geometry image can be encoded using various methods. The first method is to encode the depth values ​​of the previously generated d1 image as they are (absolute d1 encoding method). The second method is to encode the difference between the depth values ​​of the previously generated d1 image and the depth values ​​of the d0 image (differential encoding method).

[0197] In this encoding method using the depth values ​​of two layers d0 and d1, if there is another point between the two depths, the geometry information of that point is lost in the encoding process, so an Enhanced-Delta-Depth (EDD) code may be used for lossless coding.

[0198] The EDD code will be specifically described with reference to FIG.

[0199] FIG. 10 shows an example of an EDD code according to an embodiment.

[0200] The point cloud video encoder 10002 and / or some / all of the V-PCC encoding process (e.g., video compression 40009) can encode the geometry information of the points based on the EOD code.

[0201] The EDD code is a method of binary encoding the positions of all points within the surface thickness range, including d1, as shown in Figure 10. As an example, for the points in the second column from the left in Figure 10, there are points in the first and fourth positions above D0, and the second and third positions are empty, so they are represented by an EDD code of 0b1001 (=9). If the EDD code is encoded and transmitted along with D0, the receiving end can recover the geometry information of all points without any loss.

[0202] For example, if a point exists on the reference point, it is 1, and if no point exists, it is 0, and the code is expressed based on four bits.

[0203] Smoothing40004

[0204] Smoothing is the process of removing discontinuities that may occur at pad boundaries due to degradation of image quality resulting from the compression process, and is performed by the point cloud video encoder 10002 or the smoothing unit 40004 using the following process.

[0205] (1) Reconstruct the point cloud from the geometry image. This process can be considered the reverse of the geometry image construction described above. For example, reconstruction is the reverse process of encoding.

[0206] (2) Calculate the neighboring points of each point that makes up the regenerated point cloud using a KD tree or similar.

[0207] (3) For each point, determine whether the point is located on the boundary surface of the patch. For example, if there is an adjacent point with a different projection plane (cluster index) from the current point, the point can be determined to be located on the boundary surface of the patch.

[0208] (4) If a patch boundary exists, move the point to the center of gravity of the neighboring points (located at the average x, y, z coordinates of the neighboring points), i.e., change the geometry value. If no boundary exists, keep the previous geometry value.

[0209] FIG. 11 shows an example of recoloring using color values ​​of neighboring points according to an embodiment.

[0210] The point cloud video encoder 10002 or texture image generator 40003 according to the embodiment can generate a texture image based on decolorization.

[0211] Texture image generation 40003

[0212] The texture image generation process is similar to the geometry image generation process described above, in that it generates texture images for individual patches and places them at predetermined positions to generate the entire texture image. However, in the process of generating texture images for individual patches, an image is generated that has the color values ​​(e.g., R, G, B) of the points that make up the point cloud corresponding to that position, instead of the depth values ​​used for geometry generation.

[0213] The process of calculating the color value of each point that makes up the point cloud uses the geometry that has undergone the smoothing process described above. Since the smoothed point cloud may contain points that have moved from the original point cloud, a color restoration process is required to find a color that matches the changed position. Color restoration is performed using the color values ​​of neighboring points. For example, as shown in Figure 11, a new color value can be calculated by taking into account the color values ​​of the nearest neighboring point and the color values ​​of the neighboring points.

[0214] For example, referring to FIG. 11, the recoloring calculates a suitable color value for the modified position based on the average of the characteristic information of the closest original points to the point and / or the average of the characteristic information of the closest original positions to the point.

[0215] Texture images are also generated in two layers, t0 / t1, just like geometry images, which are generated in two layers, d0 / d1.

[0216] Auxiliary patch info compression 40005

[0217] The point cloud video encoder 10002 or the additional patch information compressor 40005 according to the embodiment can compress the additional patch information (additional information related to the point cloud).

[0218] The additional patch information compression unit 40005 compresses the additional patch information generated in the above-mentioned patch generation, patch packing, geometry generation processes, etc. The additional patch information includes the following parameters:

[0219] Cluster index that identifies the projection plane (normal)

[0220] 3D spatial position of the patch: minimum value of the patch tangent direction (patch 3D shift tangent axis), minimum value of the patch bitangent direction (patch 3D shift bitangent axis), minimum value of the patch normal direction (patch 3D shift normal axis)

[0221] 2D spatial position and size of the patch: horizontal size (patch 2D size u), vertical size (patch 2D size v), horizontal minimum value (patch 2D shift u), vertical minimum value (patch 2D shift u)

[0222] Mapping information for each block and patch: candidate index (when placing patches in order based on the 2D spatial position and size information of the patches mentioned above, multiple patches may be mapped to one block. In this case, the mapped patches form a candidate list, and the index indicates which patch's data is present in the block), local patch index (an index indicating one of the total patches present in the frame). Table 1 shows pseudocode showing the process of matching blocks and patches using the candidate list and local patch index.

[0223] The maximum number of candidate lists is user defined.

[0224] [Table 1]

[0225] FIG. 12 illustrates an example of push-pull background filling according to an embodiment.

[0226] Image padding and group dilation: 40006, 40007, 40008

[0227] The image padder according to the embodiment can fill spaces other than the patch area with meaningless additional data based on a push-pull background filling method.

[0228] Image padding 40006 and 40007 is a process of filling spaces outside the patch area with meaningless data to improve compression efficiency. For image padding, a method is used in which pixel values ​​of the corresponding columns or rows on the boundary side of the patch are copied to fill the empty space. Alternatively, as shown in Figure 12, a push-pull background filling method may be used in which the resolution of the unpadded image is gradually reduced and then the empty space is filled with pixel values ​​from a lower-resolution image in the process of increasing the resolution again.

[0229] Group dilation 40008 is a method for filling empty spaces in a geometry or texture image consisting of two layers, d0 / d1 and t0 / t1. It is a process of filling the empty space values ​​of the two layers calculated by the image padding described above with the average value of the values ​​for the same position in the two layers.

[0230] FIG. 13 shows an example of a possible traversal order for a block of size 4*4 according to an embodiment.

[0231] Occupancy map compression 40012, 40011

[0232] Occupancy map compression according to an embodiment is a process for compressing the occupancy map described above, and there are two methods: video compression for lossy compression and entropy compression for lossless compression. Video compression will be described later.

[0233] The entropy compression process is as follows:

[0234] (1) For each block that makes up the occupancy map, if all blocks are filled, code 1, and repeat the same process for the next block. Otherwise, code 0, and repeat steps (2) to (5).

[0235] (2) Determine the best traversal order for run-length coding of the filled pixels of the block. Figure 13 shows four possible traversal orders for a 4*4 sized block as an example.

[0236] FIG. 14 shows an example of a best traversal order according to an embodiment.

[0237] As described above, the entropy compression unit 40012 according to this embodiment can code (encode) blocks based on the traversal order scheme, as shown in FIG.

[0238] For example, among the possible traversal orders, the best traversal order with the smallest number of runs is selected and its index is encoded. For example, when selecting the third traversal order in FIG. 13, the number of runs can be minimized to 2, and this is selected as the best traversal order.

[0239] The number of runs is encoded. In the example of FIG. 14, since there are two runs, 2 is encoded.

[0240] (4) Encode the occupancy of the first run. In the example of Figure 14, the first run corresponds to an unfilled pixel, so 0 is encoded.

[0241] (5) The length of each run (the number of runs) is coded. In the example of Figure 14, the lengths of the first and second runs, 6 and 10, are coded sequentially.

[0242] Video compression 40009, 40010, 40011

[0243] The video compression units (40009, 40010, 40011) according to the embodiment encode sequences of geometry images, texture images, occupancy map images, etc. generated by the above-mentioned processes using 2D video codecs such as HEVC and VVC.

[0244] FIG. 15 illustrates an example of a 2D video / image encoder according to an embodiment, which is also referred to as an encoding device.

[0245] 15 is an embodiment in which the above-described video compression units 40009, 40010, and 40011 are applied, and shows a schematic block diagram of a 2D video / image encoder 15000 in which video / image signals are encoded. The 2D video / image encoder 15000 may be included in the above-described point cloud video encoder 10002, or may consist of internal / external components. Each component in FIG. 15 corresponds to software, hardware, a processor, and / or a combination thereof.

[0246] Here, the input image may be one of the above-mentioned geometry image, texture image (feature image), and occupancy map image. When the 2D video / image encoder of Fig. 15 is applied to the video compressor 40009, the image input to the 2D video / image encoder 15000 is a padded geometry image, and the bitstream output from the 2D video / image encoder 15000 is a bitstream of compressed geometry image. When the 2D video / image encoder of Fig. 15 is applied to the video compressor 40010, the image input to the 2D video / image encoder 15000 is a padded texture image, and the bitstream output from the 2D video / image encoder 15000 is a bitstream of compressed texture image. When the 2D video / image encoder of Fig. 15 is applied to the video compressor 40011, the image input to the 2D video / image encoder 15000 is an occupancy map image, and the bitstream output from the 2D video / image encoder 15000 is a bitstream of compressed occupancy map image.

[0247] The inter prediction unit 15090 and the intra prediction unit 15100 are collectively referred to as a prediction unit. That is, the prediction unit includes the inter prediction unit 15090 and the intra prediction unit 15100. The transform unit 15030, the quantization unit 15040, the inverse quantization unit 15050, and the inverse transform unit 15060 are collectively referred to as a residual processing unit. The residual processing unit may further include a subtraction unit 15020. According to an embodiment, the image division unit 15010, the subtraction unit 15020, the transform unit 15030, the quantization unit 15040, the inverse quantization unit 15050, the inverse transform unit 15060, the addition unit 155, the filtering unit 15070, the inter prediction unit 15090, the intra prediction unit 15100, and the entropy coding unit 15110 of FIG. 15 are configured as a single hardware component (e.g., an encoder or a processor). The memory 15080 also includes a DPB (decoded picture buffer) and is configured as a digital storage medium.

[0248] The image division unit 15010 divides an input image (or picture or frame) input to the encoding device 15000 into one or more processing units. For example, the processing units are also called coding units (CUs). In this case, the coding units are recursively divided from a coding tree unit (CTU) or a largest coding unit (LCU) according to a quad-tree binary-tree (QTBT) structure. For example, one coding unit is divided into multiple coding units of a deeper depth based on a quad-tree structure and / or a binary-tree structure. In this case, for example, a quad-tree may be applied first, followed by a binary-tree. Alternatively, a binary-tree may be applied first. The coding procedure according to this specification may be performed based on the final coding unit that is not further divided. In this case, the largest coding unit may be used as the final coding unit based on coding efficiency according to video characteristics, or the coding unit may be recursively divided into coding units of lower depth as needed, and the coding unit of the optimal size may be used as the final coding unit. Here, the coding procedure includes procedures such as prediction, transformation, and restoration, which will be described later. As another example, the processing unit may further include a prediction unit (PU) or a transform unit (TU). In this case, each of the prediction unit and the transform unit is divided or partitioned from the final coding unit. The prediction unit is a unit of sample prediction, and the transform unit is a unit for deriving transform coefficients and / or a unit for deriving a residual signal from the transform coefficients.

[0249] The term "unit" is sometimes used interchangeably with terms such as "block," "area," or "module." In general, an MxN block refers to a set of samples or transform coefficients consisting of M columns and N rows. A sample generally refers to a pixel or pixel value, and may refer to only the pixel / pixel value of the luma component, or only the pixel / pixel value of the chroma component. A sample is used as a term corresponding to one picture (or image), which is a pixel or pel.

[0250] The subtractor 15020 of the encoding device 15000 subtracts a prediction signal (predicted block, predicted sample array) output from the inter predictor 15090 or intra predictor 15100 from an input video signal (original block, original sample array) to generate a residual signal (residual signal, residual block, residual sample array), which is then transmitted to the transformer 15030. In this case, as shown, a unit in the encoding device 15000 that subtracts a prediction signal (predicted block, predicted sample array) from an input video signal (original block, original sample array) can be referred to as the subtractor 15020. The predictor performs prediction on a target block (hereinafter referred to as a current block) and generates a predicted block including prediction samples for the current block. The predictor determines whether to apply intra prediction or inter prediction on a current block or CU basis. The prediction unit generates various information related to prediction, such as prediction mode information, as will be described later for each prediction mode, and transmits the information to the entropy coding unit 15110. The prediction information is coded by the entropy coding unit 15110 and output in the form of a bitstream.

[0251] The intra prediction unit 15100 of the prediction unit predicts the current block by referring to samples in the current picture. The referenced samples are located either neighboring or distant from the current block depending on the prediction mode. Prediction modes in intra prediction include multiple non-directional modes and multiple directional modes. Non-directional modes include, for example, DC mode and planar mode. Directional modes include, for example, 33 directional prediction modes or 65 directional prediction modes depending on the precision of the prediction direction. However, this is merely an example, and more or less directional prediction modes may be used depending on the settings. The prediction mode applied to the current block may be determined using the prediction modes applied to neighboring blocks of the intra prediction unit 15100.

[0252] The inter prediction unit 15090 of the prediction unit derives a predicted block for the current block based on a reference block (reference sample array) identified by a motion vector in the reference picture. To reduce the amount of motion information transmitted in inter prediction mode, the motion information is predicted in units of blocks, sub-blocks, or samples based on the correlation between the motion information of neighboring blocks and the current block. The motion information includes a motion vector and a reference picture index. The motion information further includes information on the inter prediction direction (e.g., L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, the neighboring blocks include spatial neighboring blocks in the current picture and temporal neighboring blocks in the reference picture. The reference picture including the reference block and the reference picture including the temporal neighboring blocks may be the same or different. The temporal neighboring block is called a collocated reference block or colCU (colCU), and the reference picture including the temporal neighboring block is also called a collocated picture (colPic). For example, the inter predictor 15090 constructs a motion information candidate list based on neighboring blocks and generates information indicating which candidate is used to derive a motion vector and / or a reference picture index for the current block. Inter prediction is performed based on various prediction modes, such as skip mode and merge mode, in which the inter predictor 15090 uses motion information of neighboring blocks as motion information for the current block. Unlike merge mode, skip mode may not transmit a residual signal. In motion vector prediction (MVP) mode, the motion vector of the neighboring block is used as a motion vector predictor, and the motion vector of the current block is indicated by signaling a motion vector difference.

[0253] The prediction signal generated by the inter prediction unit 15090 or the intra prediction unit 15100 is used to generate a reconstructed signal or to generate a residual signal.

[0254] The transform unit 15030 applies a transform method to the residual signal to generate transform coefficients. For example, the transform method may include at least one of the following: Discrete Cosine Transform (DCT), Discrete Sine Transform (DST), Karhunen-Loeve Transform (KLT), Graph-Based Transform (GBT), or Conditionally Non-Linear Transform (CNT). Here, GBT refers to a transform obtained from a graph representing inter-pixel relationship information. CNT refers to a transform obtained based on a prediction signal generated using all previously reconstructed pixels. The transform process may be applied to square pixel blocks of the same size or non-square variable-size blocks.

[0255] The quantizer 15040 quantizes the transform coefficients and transmits them to the entropy encoder 15110. The entropy encoder 15110 encodes the quantized signal (information about the quantized transform coefficients) and outputs it as a bitstream. The information about the quantized transform coefficients is called residual information. The quantizer 15040 can rearrange the quantized transform coefficients in block form into a one-dimensional vector form based on the coefficient scan order and generate information about the quantized transform coefficients based on the one-dimensional vector form of the quantized transform coefficients.

[0256] The entropy encoder 15110 performs various encoding methods, such as exponential Golomb, context-adaptive variable length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), etc. The entropy encoder 15110 encodes information required for video / image reconstruction (e.g., values ​​of syntax elements) together with or separately from the quantized transform coefficients. The encoded information (e.g., encoded video / image information) is transmitted or stored in the form of a bitstream in units of network abstraction layer (NAL) units.

[0257] The bitstream may be transmitted via a network or stored on a digital storage medium. Here, the network may include a broadcast network and / or a communication network, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. A transmitting unit (not shown) that transmits and / or a storage unit (not shown) that stores the signal output from the entropy encoding unit 15110 may be configured as an internal / external element of the encoding device 15000, or the transmitting unit may be included in the entropy encoding unit 15110.

[0258] The quantized transform coefficients output from the quantizer 15040 are used to generate a prediction signal. For example, the inverse quantizer 15040 and the inverse transformer 15060 apply inverse quantization and inverse transform to the quantized transform coefficients to reconstruct a residual signal (residual block or residual samples). The adder 15200 generates a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the reconstructed residual signal to the prediction signal output from the inter predictor 15090 or the intra predictor 15100. When there is no residual for the current block, such as when skip mode is applied, the predicted block is used as the reconstructed block. The adder 15200 is referred to as a reconstruction unit or reconstructed block generator. The generated reconstructed signal may be used for intra prediction of the next current block in the current picture, or may be used for inter prediction of the next picture after filtering, as described below.

[0259] The filtering unit 15070 may apply filtering to the reconstructed signal output from the adder 15200 to improve subjective / objective image quality. For example, the filtering unit 15070 applies various filtering methods to the reconstructed picture to generate a modified reconstructed picture, and stores the modified reconstructed picture in the memory 15080, specifically in the DPB of the memory 15080. Examples of various filtering methods include deblocking filtering, sample adaptive offset, an adaptive loop filter, and a bilateral filter. The filtering unit 15070 generates various information related to filtering, such as for each filtering method described below, and transmits the information to the entropy coding unit 15110. The entropy coding unit 15110 encodes the filtering information and outputs it in the form of a bitstream.

[0260] The modified reconstructed picture stored in the memory 15080 is used as a reference picture in the inter prediction unit 15090. This allows the encoding device to avoid prediction mismatches in the encoding device 15000 and the decoding device when inter prediction is applied, and also improves encoding efficiency.

[0261] The DPB of the memory 15080 stores the modified reconstructed picture for use as a reference picture in the inter predictor 15090. The memory 15080 stores motion information of the block from which motion information in the current picture is derived (or coded) and / or motion information of blocks in already reconstructed pictures. The stored motion information is transmitted to the inter predictor 15090 to be used as motion information of spatially adjacent blocks or temporally adjacent blocks. The memory 15080 stores reconstructed samples of blocks reconstructed in the current picture and transmits them to the intra predictor 15100.

[0262] At least one of the above-described prediction, transformation, and quantization procedures may be omitted. For example, for a block to which PCM (pulse code modulation) is applied, the prediction, transformation, and quantization procedures may be omitted, and the original sample values ​​may be directly coded and output to a bitstream.

[0263] FIG. 16 illustrates an example of a V-PCC decoding process according to an embodiment.

[0264] The V-PCC decoding process or V-PCC decoder is the inverse process of the V-PCC encoding process (or encoder) of Figure 4. Each component in Figure 16 corresponds to software, hardware, a processor, and / or a combination thereof.

[0265] A demultiplexer 16000 demultiplexes the compressed bitstream and outputs a compressed texture image, a compressed geometry image, a compressed occupancy map image, and compressed additional patch information, respectively.

[0266] Video decompression (video decompression units) 16001 and 16002 decompress the compressed texture images and compressed geometry images, respectively.

[0267] Occupancy map decompression (or occupancy map restorer) 16003 restores the compressed occupancy map image.

[0268] An auxiliary patch information decompression or auxiliary patch information restoration unit 16004 restores compressed auxiliary patch information.

[0269] The geometry reconstruction 16005 reconstructs geometry information based on the reconstructed geometry image, the reconstructed occupancy map, and / or the reconstructed additional patch information, e.g., reconstructing geometry that has changed during the encoding process.

[0270] Smoothing 16006 applies smoothing to the reconstructed geometry, for example, smoothing filtering.

[0271] Texture reconstruction 16007 reconstructs texture from the restored texture image and / or smoothed geometry.

[0272] Color smoothing 16008 smooths the color values ​​from the reconstructed texture, for example by applying smoothing filtering.

[0273] As a result, reconstructed point cloud data is generated.

[0274] Figure 16 shows the decoding process of V-PCC to recover (or decode) the compressed occupancy map, geometry image, texture image, and additional patch information to reconstruct the point cloud.

[0275] Each unit shown in Fig. 16 operates as at least one of a processor, software, and hardware. The detailed operation of each unit in Fig. 16 according to the embodiment is as follows.

[0276] Video decompression 16001, 16002

[0277] This is the inverse process of the video compression described above, in which the geometry image bitstream, compressed texture image bitstream, and / or compressed occupancy map image bitstream generated in the above process are decoded by reversing the video compression process using a 2D video codec such as HEVC or VVC.

[0278] FIG. 17 illustrates an example of a 2D video / image decoder according to an embodiment, which is also called a decoding device.

[0279] The 2D video / image decoder performs the reverse process of the 2D video / image encoder in FIG.

[0280] The 2D video / image decoder in Figure 17 is an example of the video decompression units 16001 and 16002 in Figure 16, and shows a schematic block diagram of a 2D video / image decoder 17000 in which video / image signals are decoded. The 2D video / image decoder 17000 may be included in the point cloud video decoder 10008 described above, or may be configured as an internal / external component. Each component in Figure 17 corresponds to software, hardware, a processor, and / or a combination thereof.

[0281] Here, the input bitstream is one of a geometry image bitstream, a texture image (attribute(s) image) bitstream, and an occupancy map image bitstream. When the 2D video / image decoder of FIG. 17 is applied to the video restoration unit 16001, the bitstream input to the 2D video / image decoder is a compressed texture image bitstream, and the restored image output from the 2D video / image decoder is a restored texture image. When the 2D video / image decoder of FIG. 17 is applied to the video restoration unit 16002, the bitstream input to the 2D video / image decoder is a compressed geometry image bitstream, and the restored image output from the 2D video / image decoder is a restored geometry image. The 2D video / image decoder of FIG. 17 receives and restores a compressed occupancy map image bitstream. The restored image (or output image, decoded image) refers to the restored image for the geometry image, texture image (attribute(s) image), and occupancy map image described above.

[0282] Referring to Figure 17, the inter prediction unit 17070 and the intra prediction unit 17080 are collectively referred to as the prediction unit. That is, the prediction unit includes the inter prediction unit 17070 and the intra prediction unit 17080. The inverse quantization unit 17020 and the inverse transform unit 17030 are collectively referred to as the residual processing unit. That is, the residual processing unit includes the inverse quantization unit 17020 and the inverse transform unit 17030. According to an embodiment, the entropy decoding unit 17010, the inverse quantization unit 17020, the inverse transform unit 17030, the addition unit 17040, the filtering unit 17050, the inter prediction unit 17070, and the intra prediction unit 17080 of Figure 17 are configured as a single hardware component (e.g., a decoder or a processor). Furthermore, the memory 17060 may include a decoded picture buffer (DPB) or may be configured as a digital storage medium.

[0283] When a bitstream including video / image information is input, the decoding device 17000 reconstructs the image corresponding to the process by which the video / image information was processed in the encoding device of Fig. 15. For example, the decoding device 17000 performs decoding using the processing unit applied in the encoding device. Thus, the processing unit for decoding is, for example, a coding unit, and the coding unit is divided from the coding tree unit or the maximum coding unit according to a quad-tree structure and / or a binary-tree structure. Furthermore, the reconstructed video signal decoded and output by the decoding device 17000 is reproduced by a playback device.

[0284] The decoding device 17000 receives a signal output from the encoding device in the form of a bitstream, and the received signal is decoded by the entropy decoding unit 17010. For example, the entropy decoding unit 17010 parses the bitstream to derive information (e.g., video / image information) necessary for image restoration (or picture restoration). For example, the entropy decoding unit 17010 decodes information in the bitstream based on a coding method such as Exponential Golomb Coding, CAVLC, or CABAC, and outputs values ​​of syntax elements necessary for image restoration and quantized values ​​of transform coefficients related to the residual. More specifically, the CABAC entropy decoding method receives bins corresponding to each syntax element in the bitstream, determines a context model using information on the syntax element to be decoded and decoded information on neighboring and current blocks or information on symbols / bins decoded in a previous step, predicts the occurrence probability of the bin according to the determined context model, and performs arithmetic decoding of the bins to generate symbols corresponding to the values ​​of each syntax element. In this case, the CABAC entropy decoding method determines a context model and then updates the context model using information about the decoded symbol / bin for the context model of the next symbol / bin. Prediction-related information from the information decoded by the entropy decoding unit 17010 is provided to prediction units (inter prediction unit 17070 and intra prediction unit 17080), and residual values ​​entropy-decoded by the entropy decoding unit 17010, i.e., quantized transform coefficients and related parameter information, are input to the inverse quantization unit 17020. Filtering-related information from the information decoded by the entropy decoding unit 17010 is provided to the filtering unit 17050. Meanwhile, a receiving unit (not shown) that receives a signal output from the encoding device may be further configured as an internal / external element of the decoding device 17000, and the receiving unit may be a component of the entropy decoding unit 17010.

[0285] The inverse quantization unit 17020 quantizes the quantized transform coefficients and outputs the transform coefficients. The inverse quantization unit 17020 rearranges the quantized transform coefficients into a two-dimensional block. In this case, the rearrangement is performed based on the coefficient scanning order performed by the encoding device. The inverse quantization unit 17020 performs inverse quantization on the quantized transform coefficients using a quantization parameter (e.g., quantization step size information) to obtain transform coefficients.

[0286] The inverse transform unit 17030 inversely transforms the transform coefficients to obtain a residual signal (residual block, residual sample array).

[0287] The prediction unit performs prediction on the current block and generates a predicted block including prediction samples for the current block. The prediction unit determines whether intra prediction or inter prediction is applied to the current block based on prediction information output from the entropy decoding unit 17010, and determines a specific intra / inter prediction mode.

[0288] The intra prediction unit 17080 of the prediction unit predicts the current block by referring to samples in the current picture. The referenced samples may be located neighboring or distant from the current block depending on the prediction mode. Prediction modes in intra prediction include a plurality of non-directional modes and a plurality of directional modes. The intra prediction unit 17080 determines the prediction mode to be applied to the current block using the prediction modes applied to neighboring blocks.

[0289] The inter prediction unit 17070 of the prediction unit derives a predicted block for the current block based on a reference block (reference sample array) identified by a motion vector in a reference picture. To reduce the amount of motion information transmitted in inter prediction mode, the motion information is predicted in units of blocks, sub-blocks, or samples based on the correlation between the motion information of neighboring blocks and the current block. The motion information includes a motion vector and a reference picture index. The motion information further includes information on an inter prediction method (e.g., L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, the neighboring blocks include spatial neighboring blocks in the current picture and temporal neighboring blocks in the reference picture. For example, the inter prediction unit 17070 constructs a motion information candidate list based on the neighboring blocks and derives a motion vector and / or a reference picture index for the current block based on received candidate selection information. Inter prediction is performed based on various prediction modes, and the prediction information includes information indicating the inter prediction mode for the current block.

[0290] The adder 17040 generates a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the residual signal acquired by the inverse transformer 17030 to the predicted signal (predicted block, predicted sample array) output from the inter prediction unit 17070 or intra prediction unit 17080. When there is no residual for the current block, such as when skip mode is applied, the predicted block is used as the reconstructed block.

[0291] The adder 17040 is referred to as a reconstruction unit or a reconstruction block generator. The generated reconstruction signal may be used for intra prediction of the next block to be processed in the current picture, and may also be used for inter prediction of the next picture after filtering, as described below.

[0292] The filtering unit 17050 applies filtering to the reconstructed signal output from the adder 17040 to improve the subjective / objective image quality. For example, the filtering unit 17050 applies various filtering methods to the reconstructed picture to generate a modified reconstructed picture, and transmits the modified reconstructed picture to the memory 17060, specifically to the DPB of the memory 17060. The various filtering methods include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc.

[0293] The (modified) reconstructed picture stored in the DPB of the memory 17060 is used as a reference picture in the inter predictor 17070. The memory 17060 stores motion information of the block from which motion information in the current picture is derived (or decoded) and / or motion information of blocks in already reconstructed pictures. The stored motion information is transmitted to the inter predictor 17070 to be used as motion information of spatially adjacent blocks or temporally adjacent blocks. The memory 17060 stores reconstructed samples of reconstructed blocks in the current picture and transmits them to the intra predictor 17080.

[0294] In this specification, the embodiments described for the filtering unit 15070, inter prediction unit 15090, and intra prediction unit 15100 of the encoding device 15000 in Figure 15 can also be applied to the filtering unit 17050, inter prediction unit 17070, and intra prediction unit 17080 of the decoding device 17000 in the same or corresponding manner.

[0295] Meanwhile, at least one of the above-mentioned prediction, inverse transform, and inverse quantization procedures may be omitted. For example, for a block to which PCM (pulse code modulation) is applied, the prediction, inverse transform, and inverse quantization procedures may be omitted, and the decoded sample values ​​may be used directly as samples of the restored image.

[0296] Occupancy map decompression 16003

[0297] This is the reverse process of the occupancy map compression described above, and is the process of decoding the compressed occupancy map bitstream to restore the occupancy map.

[0298] Auxiliary patch info decompression 16004

[0299] This is the reverse process of the additional patch information compression described above, and is a process of decoding the compressed additional patch information bit stream to restore the additional patch information.

[0300] Geometry reconstruction 16005

[0301] This is the reverse process of generating the geometry image described above. First, patches are extracted from the geometry image using the restored occupancy map, the 2D position / size information of the patch contained in the additional patch information, and the block-to-patch mapping information. Then, the point cloud is restored in 3D space using the extracted patch geometry image and the 3D position information of the patch contained in the additional patch information. The geometry value corresponding to an arbitrary point (u, v) within a patch is called g(u, v), and the coordinate values ​​of the normal, tangent, and bitangent axes of the patch's position in 3D space are (δ0, s0, r0). Then, the coordinate values ​​of the normal, tangent, and bitangent axes of the position in 3D space mapped to point (u, v), δ(u, v), s(u, v), and r(u, v), are expressed as follows:

[0302] δ(u,v) = δ0 + g(u,v)

[0303] s(u,v) = s0 + u

[0304] r(u,v) = r0 + v

[0305] Smoothing16006

[0306] This is similar to the smoothing in the encoding process described above, and is a process for removing discontinuities that may arise from patch boundaries due to image quality degradation that occurs in the compression process.

[0307] Texture reconstruction 16007

[0308] This is the process of restoring a color point cloud by assigning a color value to each point that makes up the smoothed point cloud. Using the mapping information of the geometry image and point cloud reconstructed in the diorama reconstruction process described above, the color value corresponding to the texture image pixel at the same position as the geometry image in 2D space is assigned to the point cloud point that corresponds to the same position in 3D space.

[0309] Color smoothing 16008

[0310] Similar to the geometry smoothing process described above, color smoothing is a process for removing discontinuities in color values ​​that may arise from patch boundaries due to image degradation resulting from the compression process.

[0311] (1) Calculate the neighboring points of each point that constitutes the reconstructed color point cloud using a KD tree, etc. The neighboring point information calculated in the above-mentioned geometry smoothing process may be used as is.

[0312] (2) For each point, determine whether the point is located on the patch boundary surface. The boundary surface information calculated in the above-mentioned geometry smoothing process may be used as is.

[0313] (3) The distribution of color values ​​of adjacent points of a point on the boundary surface is examined to determine whether to perform smoothing. For example, if the entropy of the brightness value is below the boundary value (threshold local entry) (if there are many similar brightness values), it is determined that it is not an edge and smoothing is performed. One method of smoothing is to replace the color value of that point with the average value of the adjacent points.

[0314] FIG. 18 illustrates an example of the operation flow of a transmitting device for compressing and transmitting V-PCC-based point cloud data according to an embodiment.

[0315] The transmitting device according to the embodiment may correspond to or perform some / all of the operations of the transmitting device in Fig. 1, the encoding process in Fig. 4, and the 2D video / image encoder in Fig. 15. Each component of the transmitting device corresponds to software, hardware, a processor, and / or a combination thereof.

[0316] The operation of the transmitting end for compressing and transmitting point cloud data using V-PCC is as shown in the figure.

[0317] The point cloud data transmission device according to the embodiment is referred to as a transmission device, a transmission system, etc.

[0318] The patch generator 18000 receives point cloud data and generates patches for 2D image mapping of the point cloud. Patch information and / or additional patch information are generated as a result of the patch generation, and the generated patch information and / or additional patch information are used in geometry image generation, texture image generation, smoothing, or geometry restoration process for smoothing.

[0319] The patch packing unit 18001 performs a patch packing process to map the patches generated by the patch generation unit 18000 into a 2D image. For example, one or more patches are packed. An occupancy map is generated as a result of the patch packing, and the occupancy map is used in the geometry image generation, geometry image padding, texture image padding, and / or geometry restoration process for smoothing.

[0320] The geometry image generator 18002 generates a geometry image using the point cloud data, patch information (or additional patch information), and / or occupancy map. The generated geometry image is preprocessed by the encoding preprocessor 18003 and then encoded into a single bitstream by the video encoder 18006.

[0321] The pre-encoding processing unit 18003 includes image padding, i.e., padding a portion of the generated geometry image and the generated texture image with meaningless data. The pre-encoding processing unit 18003 may further include a group dilation process for the generated texture image or the texture image that has undergone image padding.

[0322] The geometry reconstruction unit 18010 reconstructs a 3D geometry image using the geometry bitstream, additional patch information, and / or occupancy map encoded by the video encoding unit 18006.

[0323] The smoothing unit 18009 smoothes the 3D geometry image reconstructed and output by the geometry restoration unit 18010 based on the additional patch information, and outputs the smoothed image to the texture image generation unit 18004 .

[0324] The texture image generator 18004 generates a texture image using the smoothed 3D geometry, point cloud data, patches (or packed patches), patch information (or additional patch information), and / or occupancy map. The generated texture image is preprocessed by the encoding preprocessor 18003 and then encoded into a single video bitstream by the video encoder 18006.

[0325] The metadata encoding unit 18005 encodes the additional patch information into one metadata bitstream.

[0326] The video encoder 18006 encodes the geometry image and texture image output from the encoding pre-processor 18003 into separate video bitstreams, and encodes the occupancy map into one video bitstream. In one embodiment, the video encoder 18006 applies the 2D video / image encoder of FIG. 15 to each input image for encoding.

[0327] The multiplexing unit 18007 multiplexes the geometry video bitstream, texture image video bitstream, and occupancy map video bitstream output from the video encoding unit 18006 and the metadata (including additional patch information) bitstream output from the metadata encoding unit 18005 into a single bitstream.

[0328] The transmitter 18008 transmits the bit stream output from the multiplexer 18007 to the receiving end. Alternatively, a file / segment encapsulator may be further provided between the multiplexer 18007 and the transmitter 18008, and the bit stream output from the multiplexer 18007 may be encapsulated in the form of a file and / or segment and output to the transmitter 18008.

[0329] The patch generation unit 18000, patch packing unit 18001, geometry image generation unit 18002, texture image generation unit 18004, metadata encoding unit 18005, and smoothing unit 18009 in Figure 18 correspond to the patch generation unit 40000, patch packing unit 40001, geometry image generation unit 40002, texture image generation unit 40003, additional patch information compression unit 40005, and smoothing unit 40004 in Figure 4, respectively. In addition, the encoding pre-processing unit 18003 in Figure 18 may include image padding units 40006 and 40007 and a group expansion unit 40008 in Figure 4, and the video encoding unit 18006 in Figure 18 may include video compression units 40009, 40010, and 40011 and / or an entropy compression unit 40012 in Figure 4. Therefore, for parts not described in Figure 18, please refer to the descriptions of Figures 4 to 15. The above-described blocks may be omitted or replaced by blocks having similar or identical functions. Each block shown in Fig. 18 may operate as at least one of a processor, software, and hardware. Alternatively, the generated geometry, texture image, and occupancy map video bitstream and additional patch information metadata bitstream may be encapsulated in one or more track data files or segments and transmitted from the transmitting unit to the receiving end.

[0330] Receiver operation process

[0331] FIG. 19 illustrates an example of the operational flow of a receiving device for receiving and restoring V-PCC-based point cloud data according to an embodiment.

[0332] The receiving device according to the embodiment corresponds to or performs some or all of the operations of the receiving device in Fig. 1, the decoding process in Fig. 16, and the 2D video / image encoder in Fig. 17. Each component of the receiving device corresponds to software, hardware, a processor, and / or a combination thereof.

[0333] The operation process of the receiving end for receiving and restoring point cloud data using V-PCC is as shown in the figure. The operation of the V-PCC receiving end is the reverse process of the operation of the V-PCC transmitting end in FIG.

[0334] The point cloud data receiving device according to the embodiment is referred to as a receiving device, a receiving system, etc.

[0335] The receiver receives a point cloud bitstream (i.e., a compressed bitstream), and the demultiplexer 19000 demultiplexes the received point cloud bitstream into a texture image bitstream, a geometry image bitstream, an occupancy map image bitstream, and a metadata (i.e., additional patch information) bitstream. The demultiplexed texture image bitstream, geometry image bitstream, and occupancy map image bitstream are output to a video decoder 19001, and the metadata bitstream is output to a metadata decoder 19002.

[0336] When the transmitting device of Fig. 18 is provided with a file / segment encapsulation unit, one embodiment is to provide a file / segment decapsulation unit between the receiving unit and demultiplexing unit 19000 of the receiving device of Fig. 19. In this case, one embodiment is to encapsulate and transmit the point cloud bitstream in the form of a file and / or segment in the transmitting device, and to receive and decapsulate the file and / or segment containing the point cloud bitstream in the receiving device.

[0337] The video decoding unit 19001 decodes the geometry image bitstream, texture image bitstream, and occupancy map image bitstream into a geometry image, a texture image, and an occupancy map image, respectively. In one embodiment, the video decoding unit 19001 applies the 2D video / image decoder of FIG. 17 to each input bitstream for decoding. The metadata decoding unit 19002 decodes the metadata bitstream into additional patch information and outputs it to the geometry restoration unit 19003.

[0338] The geometry restoration unit 19003 restores (reconstructs) the 3D geometry based on the geometry image, occupancy map, and / or additional patch information output from the video decoding unit 19001 and the metadata decoding unit 19002.

[0339] The smoothing unit 19004 smoothes the 3D geometry reconstructed by the geometry restoration unit 19003 .

[0340] The texture restoration unit 19005 restores texture using the texture image and / or smoothed 3D geometry output from the video decoding unit 19001. That is, the texture restoration unit 19005 restores a color point cloud image / picture by assigning color values ​​to the smoothed 3D geometry using the texture image. Then, to improve objective / subjective visual quality, the color smoothing unit 19006 further performs color smoothing on the color point cloud image / picture. The modified point cloud image / picture derived in this way is shown to the user after the rendering process of the point cloud renderer (19007). Note that the color smoothing process may be omitted in some cases.

[0341] The above-described blocks may be omitted or replaced with blocks having similar or identical functions. Furthermore, each block shown in Figure 19 can operate as at least one of a processor, software, and hardware.

[0342] FIG. 20 illustrates an example architecture for V-PCC-based point cloud data storage and streaming according to an embodiment.

[0343] Part or all of the system in Figure 20 includes part or all of the transmitting / receiving device in Figure 1, the encoding process in Figure 4, the 2D video / image encoder in Figure 15, the decoding process in Figure 16, the transmitting device in Figure 18, and / or the receiving device in Figure 19. Each component in the figure corresponds to software, hardware, a processor, or a combination thereof.

[0344] 20 shows an overall architecture for storing or streaming point cloud data compressed based on video-based point cloud compression (V-PCC). The process of storing and streaming point cloud data may include an acquisition process, an encoding process, a transmission process, a decoding process, a rendering process, and / or a feedback process.

[0345] The embodiment proposes a method for efficiently providing point cloud media / content / data.

[0346] To efficiently provide point cloud media / content / data, the point cloud acquisition unit 20000 first acquires a point cloud video. For example, point cloud data can be acquired using one or more cameras through a point cloud capture, synthesis, or generation process. This acquisition process can acquire a point cloud video including the 3D position of each point (represented by x, y, z position values, etc., hereinafter referred to as geometry) and the characteristics of each point (color, reflectance, transparency, etc.). The acquired point cloud video can also be generated as a file containing the point cloud video, such as a PLY (Polygon File format or the Stanford Triangle format) file. In the case of point cloud data having multiple frames, one or more files can be acquired. Point cloud-related metadata (e.g., metadata related to the capture, etc.) can also be generated during this process.

[0347] The captured point cloud video may require post-processing to improve the quality of the content. During the video capture process, the maximum and minimum depth values ​​may be adjusted within the range provided by the camera equipment. However, even after adjustment, point data from undesired areas may still be included. Therefore, post-processing may be performed to remove undesired areas (e.g., background) or fill spatial holes by recognizing connected spaces. In addition, point clouds extracted from cameras sharing a spatial coordinate system may be integrated into a single content by converting each point to a global coordinate system based on the position coordinates of each camera acquired through calibration. This allows for the acquisition of a point cloud video with a high point density.

[0348] The point cloud pre-processing unit 20001 can generate a point cloud video into one or more pictures / frames. Here, a picture / frame generally refers to a unit representing one image at a specific time period. The point cloud pre-processing unit 20001 can also generate an occupancy map picture / frame, which is a binary map that indicates, with a value of 0 or 1, whether data exists at a position on the 2D plane when dividing the points constituting the point cloud video into one or more patches and mapping them onto a 2D plane. Here, a patch is a set of points constituting a point cloud, and points belonging to the same patch are adjacent to each other in 3D space and are mapped in the same direction on the plane of a six-sided bounding box during the mapping process to a 2D image. The point cloud pre-processing unit 20001 can also generate a geometry picture / frame, which is a depth map-style picture / frame that represents the geometry of each point constituting the point cloud video in units of patches. The point cloud preprocessing unit 20001 can also generate texture pictures / frames, which are pictures / frames that represent the color information of each point constituting the point cloud video on a patch-by-patch basis. In this process, metadata necessary for reconstructing a point cloud from individual patches can be generated, and this metadata includes information about the patches, such as the position and size of each patch in 2D / 3D space (referred to as additional information or additional patch information). Such pictures / frames are generated consecutively in chronological order to form a video stream or metadata stream.

[0349] The point cloud video encoder 20002 can encode one or more video streams related to a point cloud video. A video includes multiple frames, and a frame corresponds to a still image / picture. In this specification, the term point cloud video includes point cloud images / frames / pictures, and may be used interchangeably with point cloud images / frames / pictures. The point cloud video encoder 20002 performs a video-based point cloud compression (V-PCC) procedure. The point cloud video encoder 20002 can perform a series of procedures, such as prediction, transformation, quantization, and entropy coding, for compression and coding efficiency. The encoded data (encoded video / image information) is output in bitstream format. When based on the V-PCC procedure, the point cloud video encoder 20002 can encode the point cloud video into a geometry video, an attribute video, an occupancy map video, and metadata, such as information about patches, as described below. The geometry video may include geometry images, the attribute video may include attribute images, and the occupancy map video may include occupancy map images. The patch data, which is additional information, may include information about the patch. The attribute video / image may include texture video / image.

[0350] The point cloud image encoder 20003 can encode the point cloud into one or more images related to a point cloud video. The point cloud image encoder 20003 performs a video-based point cloud compression (V-PCC) procedure. The point cloud image encoder 20003 can perform a series of procedures such as prediction, transformation, quantization, and entropy coding for compression and coding efficiency. The encoded image is output in bitstream format. When based on the V-PCC procedure, the point cloud image encoder 20003 can encode the point cloud image into a geometry image, an attribute image, an occupancy map image, and metadata, such as information about patches, as described below.

[0351] According to an embodiment, the point cloud video encoder 20002, the point cloud image encoder 20003, the point cloud video decoder 20006, and the point cloud image decoder 20008 may be implemented by a single encoder / decoder as described above, or may be implemented by separate paths as shown in the figure.

[0352] The encapsulation unit (file / segment encapsulation unit) 20004 can encapsulate the encoded point cloud data and / or metadata related to the point cloud in the form of a file or a segment for streaming. Here, the metadata related to the point cloud may be transmitted from a metadata processing unit (not shown). The metadata processing unit may be included in the point cloud video / image encoders 20002, 20003, or may be configured as a separate component / module. The encapsulation unit 20004 encapsulates one bitstream or separate bitstreams including the video / image / metadata in a file format such as ISOBMFF, or processes the bitstream in a format such as a DASH segment. According to an embodiment, the encapsulation unit 20004 can include the metadata related to the point cloud in a file format. For example, the point cloud metadata is included in boxes at various levels in the ISOBMFF file format, or is included in data in a separate track within the file. According to an embodiment, the encapsulation unit 20004 can encapsulate the point cloud-related metadata itself in a file.

[0353] The encapsulation unit 20004 according to the embodiment divides and stores one or more bitstreams in a file, and encapsulates the corresponding signaling information. Also, a patch (or atlas) stream included in the bitstream may be stored in a track in the file, and associated signaling information may be stored. Furthermore, an SEI message present in the bitstream may be stored in a track in the file, and associated signaling information may be stored.

[0354] A transmission processing unit (not shown) may process the encapsulated point cloud data according to a file format for transmission. The transmission processing unit may be included in the transmission unit (not shown) or may be configured as a separate component / module. The transmission processing unit can process the point cloud data according to any transmission protocol. The processing for transmission may include processing for transmission via a broadcast network and processing for transmission via broadband. According to an embodiment, the transmission processing unit may process not only the point cloud data but also point cloud-related metadata transmitted from the metadata processing unit for transmission.

[0355] The transmitter can transmit the point cloud bitstream or a file / segment containing the bitstream to a receiver (not shown) of a receiving device via a digital storage medium or a network. For transmission, processing according to any transmission protocol may be performed. The processed data for transmission is transmitted via a broadcast network and / or broadband. This data is transmitted to the receiver on an on-demand basis. Digital storage media include various media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD. The transmitter includes elements for generating a media file in a predetermined file format and may also include elements for transmission via a broadcast / communication network. The receiver extracts the bitstream and transmits it to a decoding device.

[0356] The receiving unit can receive the point cloud data transmitted by the point cloud data transmitting device according to the present specification. Depending on the transmitted channel, the receiving unit may receive the point cloud data via a broadcast network, may receive the point cloud data via broadband, or may receive the point cloud video data via a digital storage medium. The receiving unit may decode the received data and render it according to the user's viewport, etc.

[0357] The receiving processing unit (not shown) can process the received point cloud video data according to the transmission protocol. The receiving processing unit may be included in the receiving unit or may be configured as a separate component / module. In response to the processing for transmission performed on the transmitting side, the receiving processing unit performs the reverse process of the transmitting processing unit described above. The receiving processing unit transmits the acquired point cloud video to the decapsulating unit 20005 and transmits metadata related to the acquired point cloud to the metadata processing unit (not shown).

[0358] The decapsulating unit (file / segment decapsulation unit) 20005 can decapsulate point cloud data in a file format transmitted from the receiving processing unit. The decapsulating unit 20005 can decapsulate a file in ISOBMFF format or the like to obtain a point cloud bitstream or point cloud-related metadata (or another metadata bitstream). The obtained point cloud bitstream is transmitted to the point cloud video decoder 20006 and the point cloud image decoder 2008, and the obtained point cloud-related metadata (or metadata bitstream) is transmitted to a metadata processing unit (not shown). The point cloud bitstream may include metadata (metadata bitstream). The metadata processing unit may be included in the point cloud video decoder 20006 or may be configured as a separate component / module. The point cloud-related metadata obtained by the decapsulating unit 20005 may be in the form of a box or track in a file format. If necessary, the decapsulating unit 20005 may receive metadata necessary for decapsulation from the metadata processing unit. The point cloud related metadata may be transmitted to the point cloud video decoder 20006 and / or the point cloud image decoder 20008 for use in point cloud decoding, or may be transmitted to the renderer 20009 for use in point cloud rendering.

[0359] The point cloud video decoder 20006 can decode video / images by receiving a bitstream and performing the reverse process corresponding to the operation of the point cloud video encoder 20002. In this case, the point cloud video decoder 20006 can decode the point cloud video by separating it into geometry video, attribute video, occupancy map video, and auxiliary patch information, as described below. The geometry video may include a geometry image, the attribute video may include an attribute image, and the occupancy map video may include an occupancy map image. The auxiliary information may include auxiliary patch information. The attribute video / image may include a texture video / image.

[0360] The point cloud image decoder 20008 receives the bitstream and performs the inverse process corresponding to the operation of the point cloud image encoder 20003. In this case, the point cloud image decoder 20008 can separate and decode the point cloud image into a geometry image, an attribute image, an occupancy map image, and metadata, such as auxiliary patch information.

[0361] The 3D geometry is restored using the decoded geometry video / image, occupancy map, and additional patch information, followed by a smoothing process. A color point cloud video / picture is restored by assigning color values ​​to the smoothed 3D geometry using the texture video / image. The renderer 20009 can render the restored geometry and color point cloud video / picture. The rendered video / image is displayed on the display unit. The user can view all or part of the rendered result on a VR / AR display or a general display.

[0362] The sensing / tracking unit 20007 acquires orientation information and / or user viewport information from the user or the receiving side and transmits it to the receiving unit and / or transmitting unit. The orientation information can indicate information about the position, angle, movement, etc. of the user's head, or information about the position, angle, movement, etc. of the device the user is looking at. Based on this information, information about the area the user is currently looking at in 3D space, i.e., viewport information, can be calculated.

[0363] The viewport information may be information about the area that a user is currently viewing in 3D space through a device or HMD. A device such as a display may extract the viewport area based on orientation information, a vertical or horizontal FOV supported by the device, etc. The orientation or viewport information may be extracted or calculated at the receiving side. The orientation or viewport information analyzed at the receiving side may be transmitted to the transmitting side via a feedback channel.

[0364] The receiving unit can efficiently extract or decode only media data of a specific region, i.e., a region indicated by the orientation information and / or viewport information, from a file using the orientation information and / or viewport information indicating the region the user is currently viewing, acquired by the sensing / tracking unit 20007. Furthermore, the transmitting unit can efficiently encode only media data of a specific region, i.e., a region indicated by the orientation information and / or viewport information, or generate and transmit a file using the orientation information and / or viewport information acquired by the sensing / tracking unit 20007.

[0365] The renderer 20009 can render the decoded point cloud data in 3D space. The rendered video / image is displayed through a display unit. The user can view all or part of the rendered result through a VR / AR display or a general display.

[0366] The feedback process may include transmitting various feedback information obtained from the rendering / display process to the transmitting side or to a decoder on the receiving side. The feedback process may provide interactivity in the consumption of point cloud data. According to an embodiment, head orientation information, viewport information indicating the area the user is currently viewing, etc. may be transmitted during the feedback process. According to an embodiment, a user may interact with something embodied in a VR / AR / MR / autonomous driving environment. In this case, information regarding the interaction may be transmitted to the transmitting side and the service provider during the feedback process. According to an embodiment, the feedback process may be omitted.

[0367] According to an embodiment, the above-mentioned feedback information is not only transmitted to the transmitting side but also consumed by the receiving side. That is, the above-mentioned feedback information may be used to perform decapsulation, decoding, rendering, etc. on the receiving side. For example, orientation information and / or viewport information may be used to prioritize decapsulate, decode, and render point cloud data for the area currently viewed by the user.

[0368] FIG. 21 illustrates an example of the configuration of a point cloud data storage and transmission device according to an embodiment.

[0369] Figure 21 shows a point cloud system according to an embodiment, some or all of which may include the transceiver device of Figure 1, the encoding process of Figure 4, the 2D video / image encoder of Figure 15, the decoding process of Figure 16, the transmitter device of Figure 18 and / or the receiver device of Figure 19, etc. Also, some or all of which may be included in or correspond to the system of Figure 20.

[0370] The point cloud data transmitting device according to the embodiment is configured as shown in the drawing. Each component of the transmitting device may be a module, unit, component, hardware, software, processor, etc.

[0371] The geometry, characteristics, additional data (or information), mesh data, etc. of the point cloud may each be organized into separate streams or stored in different tracks in the file, and may even be contained in different segments.

[0372] The point cloud acquisition unit 21000 acquires point clouds. For example, point cloud data can be acquired through a point cloud capture, synthesis, or generation process using one or more cameras. This acquisition process can acquire point cloud data including the 3D position of each point (represented by x, y, and z position values, hereinafter referred to as geometry) and the characteristics of each point (color, reflectivity, transparency, etc.). This data can be generated, for example, in a PLY (Polygon File format) or Stanford Triangle format file. In the case of point cloud data with multiple frames, one or more files can be acquired. During this process, point cloud-related metadata (e.g., metadata related to the capture) can be generated. The patch generation unit 21001 generates patches from the point cloud data. The patch generation unit 21001 generates point cloud data or point cloud video in one or more pictures / frames. Generally, a picture / frame may refer to a unit that represents one image at a specific time. When the points that make up a point cloud video are divided into one or more patches (a set of points that make up a point cloud, where points belonging to the same patch are adjacent to each other in 3D space and are mapped in the same direction on a six-sided bounding box plane in the mapping process to a 2D image) and mapped onto a 2D plane, an occupancy map picture / frame can be generated, which is a binary map that indicates with a value of 0 or 1 whether data exists at that position on the 2D plane. In addition, a geometry picture / frame can be generated, which is a depth map-style picture / frame that represents the geometry of each point that makes up the point cloud video in units of patches.A texture picture / frame can be generated, which is a picture / frame that represents the color information of each point constituting the point cloud video on a patch-by-patch basis. In this process, metadata necessary for reconstructing a point cloud from individual patches can be generated, and this metadata may include information about each patch, such as its position in 2D / 3D space and size. Such pictures / frames can be generated consecutively in chronological order to form a video stream or metadata stream.

[0373] The patches may also be used for 2D image mapping. For example, point cloud data may be projected onto each face of a cube. After generating the patches, a geometry image, one or more feature images, an occupancy map, additional data, and / or mesh data may be generated based on the generated patches.

[0374] Geometry image generation, attribute image generation, occupancy map generation, auxiliary data generation, and / or mesh data generation are performed by the point cloud pre-processing unit 20001 or a controller (not shown). In one embodiment, the point cloud pre-processing unit 20001 includes a patch generation unit 21001, a geometry image generation unit 21002, an attribute image generation unit 21003, an occupancy map generation unit 21004, an auxiliary data generation unit 21005, and a mesh data generation unit 21006.

[0375] The Geometry Image Generation unit 21002 generates a geometry image based on the results of patch generation. Geometry represents points in 3D space. The geometry image is generated based on the patch using an occupancy map containing information related to the 2D image packing of the patch, additional data (also called additional information, including patch data), and / or mesh data. The geometry image is related to information such as the depth (e.g., closeness, distance) of the patch generated after patch generation.

[0376] The attribute image generation unit 21003 generates an attribute image. For example, the attribute may indicate a texture. The texture may be a color value corresponding to each point. According to an embodiment, an image of multiple (N) attributes (color, reflectance, etc.) including texture may be generated. The multiple attributes may include material (information about material), reflectance, etc. Also, according to an embodiment, the attribute may further include information that the color of the same texture changes depending on the visual or light.

[0377] The Occupancy Map Generation unit 21004 generates an occupancy map from the patch. The occupancy map contains information indicating the presence or absence of data in a pixel, such as its geometry or feature image.

[0378] An auxiliary data generation unit 21005 generates auxiliary data (also referred to as auxiliary patch information) including information about patches. That is, the auxiliary data indicates metadata about patches of a point cloud object. For example, it may indicate information such as a normal vector for the patch. Specifically, according to an embodiment, the auxiliary data includes information necessary for reconstructing a point cloud from the patch (e.g., information about the position and size of the patch in 2D / 3D space, projection plane (normal) identification information, patch mapping information, etc.).

[0379] A mesh data generation unit 21006 generates mesh data from the patches. The mesh represents connectivity information between adjacent points. For example, it may represent triangle data. For example, the mesh data according to the embodiment represents connectivity information between each point.

[0380] The point cloud preprocessing unit 20001 or control unit generates metadata related to patch generation, geometry image generation, feature image generation, occupancy map generation, additional data generation, and mesh data generation.

[0381] The point cloud transmitting device performs video encoding and / or image encoding corresponding to the result generated by the point cloud pre-processing unit 20001. The point cloud transmitting device can generate not only point cloud video data but also point cloud image data. According to an embodiment, the point cloud data may include only video data, only image data, and / or both video data and image data.

[0382] The video encoder 21007 performs geometry video compression, feature video compression, occupancy map video compression, additional data compression, and / or mesh data compression. The video encoder 21007 generates a video stream including each of the encoded video data.

[0383] Specifically, geometry video compression encodes point cloud geometry video data, feature video compression encodes feature video data of point clouds, additional data compression encodes additional data related to point cloud video data, and mesh data compression encodes mesh data of point cloud video data. Each operation of the point cloud video encoding unit is performed in parallel.

[0384] The image encoder 21008 performs geometry image compression, feature image compression, occupancy map image compression, additional data compression, and / or mesh data compression. The image encoder generates an image containing each encoded image data.

[0385] Specifically, geometry image compression encodes point cloud geometry image data, feature image compression encodes feature image data of point clouds, additional data compression encodes additional data associated with point cloud image data, and mesh data compression encodes mesh data associated with point cloud image data. Each operation of the point cloud image encoding unit is performed in parallel.

[0386] The video encoder 21007 and / or the image encoder 21008 can receive metadata from the point cloud preprocessor 20001. The video encoder 21007 and / or the image encoder 21008 can perform their respective encoding processes based on the metadata.

[0387] The file / segment encapsulation unit 21009 encapsulates video streams and / or images into files and / or segments, and performs video track encapsulation, metadata track encapsulation, and / or image encapsulation.

[0388] A video track encapsulation can encapsulate one or more video streams into one or more tracks.

[0389] Metadata track encapsulation can encapsulate metadata related to the video stream and / or images in one or more tracks. The metadata includes data related to the content of the point cloud data, such as initial viewing orientation metadata. Depending on the embodiment, the metadata may be encapsulated in a metadata track, or may be encapsulated together in a video track or an image track.

[0390] Image encapsulation can encapsulate one or more images into one or more tracks or items.

[0391] For example, according to an embodiment, if four video streams and two images are input to the encapsulator, it will encapsulate the four video streams and two images into one file.

[0392] The file / segment encapsulation unit 21009 can receive metadata from the point cloud preprocessing unit 20001. The file / segment encapsulation unit 21009 can perform encapsulation based on the metadata.

[0393] The files and / or segments generated by the file / segment encapsulation are transmitted by a point cloud transmission device or transmission unit. For example, the segments can be delivered based on a DASH-based protocol.

[0394] The sending unit (Delivery) can transmit the point cloud bitstream or a file / segment containing the bitstream to the receiving unit of the receiving device via a digital storage medium or a network. For transmission, processing is performed using any transmission protocol. After processing for transmission, the data can be transmitted via a broadcast network and / or broadband. This data can also be transmitted to the receiving side on an on-demand basis. Digital storage media include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD.

[0395] The encapsulation unit 21009 according to the embodiment can divide and store one bitstream or individual bitstreams into one or more tracks in a file, and encapsulate the signaling information for the divided bitstreams. Also, it can store a patch (or atlas) stream included in the bitstream in a track in the file, and store the associated signaling information. Furthermore, it can store an SEI message present in the bitstream in a track in the file, and store the associated signaling information.

[0396] The transmitting unit may include elements for generating a media file in a predetermined file format and for transmission via a broadcast / communication network. The transmitting unit receives orientation information and / or viewport information from the receiving unit. The transmitting unit may transmit the acquired orientation information and / or viewport information (or information selected by the user) to the point cloud preprocessing unit 20001, the video encoding unit 21007, the image encoding unit 21008, the file / segment encapsulation unit 21009, and / or the point cloud encoding unit. Based on the orientation information and / or viewport information, the point cloud encoding unit may encode all point cloud data or encode point cloud data indicated by the orientation information and / or viewport information. Based on the orientation information and / or viewport information, the file / segment encapsulation unit may encapsulate all point cloud data or encapsulate point cloud data indicated by the orientation information and / or viewport information. Based on the orientation information and / or viewport information, the transmitter can transmit all point cloud data or transmit point cloud data indicated by the orientation information and / or viewport information.

[0397] For example, the point cloud preprocessing unit 20001 may perform the above-described operations on all point cloud data, or may perform the operations on point cloud data indicated by the orientation information and / or viewport information. The video encoding unit 21007 and / or the image encoding unit 21008 may perform the above-described operations on all point cloud data, or may perform the above-described operations on point cloud data indicated by the orientation information and / or viewport information. The file / segment encapsulation unit 21009 may perform the above-described operations on all point cloud data, or may perform the above-described operations on point cloud data indicated by the orientation information and / or viewport information. The transmission unit may perform the above-described operations on all point cloud data, or may perform the above-described operations on point cloud data indicated by the orientation information and / or viewport information.

[0398] FIG. 22 illustrates an example of the configuration of a point cloud data receiving device according to an embodiment.

[0399] Figure 22 shows a point cloud system according to an embodiment, and some or all of the system may include some or all of the transceiver device in Figure 1, the encoding process in Figure 4, the 2D video / image encoder in Figure 15, the decoding process in Figure 16, the transmitter device in Figure 18, and / or the receiver device in Figure 19. It may also be included in or correspond to some or all of the systems in Figures 20 and 21.

[0400] Each component of the receiving device may be a module, unit, component, hardware, software, processor, etc. The delivery client 22006 can receive point cloud data, a point cloud bitstream, or a file / segment containing the bitstream transmitted by the point cloud data transmitting device according to the embodiment. Depending on the transmitted channel, the receiving device may receive point cloud data via a broadcast network or broadband. Alternatively, the receiving device may receive point cloud data via a digital storage medium. The receiving device may include a process for decoding the received data and rendering it according to a user's viewport, etc. The delivery client 22006 (or a receiving processor) can process the received point cloud data according to a transmission protocol. The receiving processor may be included in the receiving unit or may be configured as a separate component / module. Corresponding to the processing for transmission performed on the transmitting side, the receiving processor performs the reverse process of the above-mentioned transmitting processor. The receiving processor can transmit the acquired point cloud data to the file / segment decapsulating unit 22000 and transmit the acquired point cloud-related metadata to a metadata processing unit (not shown).

[0401] The sensing / tracking unit 22005 acquires orientation information and / or viewport information, and can transmit the acquired orientation information and / or viewport information to the transmission client 22006, the file / segment decapsulation unit 22000, the point cloud decoding units 22001 and 22002, and the point cloud processing unit 22003.

[0402] The sending client 22006 may receive all point cloud data based on the orientation information and / or viewport information, or may receive point cloud data indicated by the orientation information and / or viewport information. The file / segment decapsulating unit 22000 can decapsulate all point cloud data based on the orientation information and / or viewport information, or decapsulate the point cloud data indicated by the orientation information and / or viewport information. The point cloud decoding unit (video decoding unit 22001 and / or image decoding unit 22002) can decode all point cloud data based on the orientation information and / or viewport information, or decode the point cloud data indicated by the orientation information and / or viewport information. The point cloud processing unit 22003 can process all point cloud data, or process the point cloud data indicated by the orientation information and / or viewport information.

[0403] The file / segment decapsulation unit 22000 performs video track decapsulation, metadata track decapsulation, and / or image decapsulation. The file / segment decapsulation unit 22000 can decapsulate point cloud data in a file format transmitted by the receiving processing unit. The file / segment decapsulation unit 22000 can decapsulate a file or segment in ISOBMFF format or the like to obtain a point cloud bitstream and point cloud-related metadata (or another metadata bitstream). The obtained point cloud bitstream can be transmitted to point cloud decoding units 22001 and 22002, and the obtained point cloud-related metadata (or metadata bitstream) can be transmitted to a metadata processing unit (not shown). The point cloud bitstream may include metadata (metadata bitstream). The metadata processing unit may be included in the point cloud video decoder or may be configured as a separate component / module. The point cloud-related metadata acquired by the file / segment decapsulating unit 22000 may be in the form of a box or track in a file format. If necessary, the file / segment decapsulating unit 22000 may receive metadata necessary for decapsulation from the metadata processing unit. The point cloud-related metadata may be transmitted to the point cloud decoding units 22001 and 22002 and used for point cloud decoding, or may be transmitted to the point cloud rendering unit 22004 and used for point cloud rendering. The file / segment decapsulating unit 22000 can generate metadata related to point cloud data.

[0404] The Video Track Decapsulation in the File / Segment Decapsulation unit 22000 decapsulates the video tracks contained in the files and / or segments, including the video streams containing geometry video, attribute video, occupancy maps, additional data, and / or mesh data.

[0405] The Metadata Track Decapsulation in the File / Segment Decapsulation unit 22000 decapsulates a bitstream containing metadata and / or additional data related to point cloud data.

[0406] Image Decapsulation in the File / Segment Decapsulation unit 22000 decapsulates images including geometry images, attribute images, occupancy maps, additional data and / or mesh data.

[0407] The decapsulating unit 22000 according to the embodiment splits and parses (decapsulates) one bitstream or individual bitstreams based on one or more tracks in a file, and also decapsulates the corresponding signaling information. It can also decapsulate a patch (or atlas) stream included in the bitstream based on the tracks in the file and parse the associated signaling information. It can also decapsulate an SEI message present in the bitstream based on the tracks in the file and obtain the associated signaling information.

[0408] The video decoding unit 22001 performs geometry video reconstruction, attribute video reconstruction, occupancy map reconstruction, additional data reconstruction, and / or mesh data reconstruction. The video decoding unit decodes the geometry video, attribute video, additional data, and / or mesh data, corresponding to the process of video encoding additional data of the point cloud transmitting device according to the embodiment.

[0409] The image decoding unit 22002 performs geometry image reconstruction, feature image reconstruction, occupancy map reconstruction, additional data reconstruction, and / or mesh data reconstruction. The image decoding unit decodes the geometry image, feature image, additional data, and / or mesh data in accordance with the process performed by the image encoding unit of the point cloud transmitting device according to the embodiment.

[0410] The video decoding unit 22001 and the image decoding unit 22002 according to this embodiment may be processed by a single video / image decoder as described above, or may be performed in separate paths as shown in the figure.

[0411] The video decoder 22001 and / or the image decoder 22002 can generate metadata associated with the video data and / or the image data.

[0412] The Point Cloud Processing unit 22003 performs geometry reconstruction and / or attribute reconstruction.

[0413] Geometry reconstruction reconstructs geometry video and / or geometry images from the decoded video data and / or decoded image data based on the occupancy map, additional data and / or mesh data.

[0414] The feature reconstruction restores the feature video and / or feature image from the decoded feature video and / or the decoded feature image based on the occupancy map, the additional data, and / or the mesh data. According to an embodiment, for example, the feature may be a texture. According to an embodiment, the feature may represent multiple feature information. When there are multiple features, the point cloud processing unit 22003 according to an embodiment performs multiple feature reconstructions.

[0415] The point cloud processing unit 22003 can receive metadata from the video decoding unit 22001, the image decoding unit 22002 and / or the file / segment decapsulating unit 22000, and process the point cloud based on the metadata.

[0416] A point cloud rendering unit 22004 renders the reconstructed point cloud. The point cloud rendering unit 22004 can receive metadata from the video decoding unit 22001, the image decoding unit 22002, and / or the file / segment decapsulating unit 22000, and can render a point cloud based on the metadata.

[0417] The display displays the rendered results on a physical display device.

[0418] According to the method / apparatus of the embodiment, as shown in Figures 20 to 22, a transmitting side encodes point cloud data into a bitstream, encapsulates it in a file and / or segment format, and transmits it, and a receiving side decapsulates the file and / or segment format into a bitstream containing the point cloud, and decodes it into point cloud data. For example, a point cloud data transmitting apparatus according to the embodiment encapsulates point cloud data based on a file, and in this case, the file may include a V-PCC track containing parameters related to the point cloud, a geometry track containing geometry, a feature track containing features, and an occupancy track containing an occupancy map.

[0419] In addition, the point cloud data receiving device according to the embodiment decapsulates point cloud data based on a file, where the file may include a V-PCC track containing parameters related to the point cloud, a geometry track containing geometry, a feature track containing features, and an occupancy track containing an occupancy map.

[0420] The above-mentioned encapsulation operation may be performed by the file / segment encapsulation unit 20004 in Figure 20, the file / segment encapsulation unit 21009 in Figure 21, etc., and the above-mentioned decapsulation operation may be performed by the file / segment decapsulation unit 20005 in Figure 20, the file / segment decapsulation unit 22000 in Figure 22, etc.

[0421] FIG. 23 illustrates an example of a structure that can be linked with a method / apparatus for transmitting and receiving point cloud data according to an embodiment.

[0422] In the structure according to the embodiment, at least one of an AI (Artificial Intelligence) server 23600, a robot 23100, an autonomous vehicle 23200, an XR device 23300, a smartphone 23400, a home appliance 23500, and / or an HMD 23700 connects to a cloud network 23000. Here, the robot 23100, the autonomous vehicle 23200, the XR device 23300, the smartphone 23400, or the home appliance 23500 may be referred to as an apparatus. In addition, the XR device 23300 may correspond to or be linked to a point cloud compressed data (PCC) device according to the embodiment.

[0423] The cloud network 23000 may refer to a network that forms part of a cloud computing infrastructure or exists within a cloud computing infrastructure, where the cloud network 23000 may be configured using a 3G network, a 4G or LTE (Long Term Evolution) network, a 5G network, or the like.

[0424] The AI ​​server 23600 is connected to at least one or more of the robot 23100, autonomous vehicle 23200, XR device 23300, smartphone 23400, home appliance 23500 and / or HMD 23700 via the cloud network 23000, and can assist in at least some of the processing of the connected devices 23100-23700.

[0425] An HMD (Head-Mounted Display) 23700 represents one type of device that can be implemented as an XR device 23300 and / or a PCC device according to an embodiment. An HMD-type device according to an embodiment includes a communication unit, a control unit, a memory unit, an I / O unit, a sensor unit, and a power supply unit.

[0426] Hereinafter, various embodiments of apparatuses 23100 to 23500 to which the above technology is applied will be described. Here, the apparatuses 23100 to 23500 shown in FIG. 23 can be interlocked / connected with the point cloud data transmission / reception apparatus according to the above-described embodiment.

[0427] <PCC+XR>

[0428] The XR / PCC apparatus 23300 is applied with PCC and / or XR (AR+VR) technology and can be implemented in an HMD (Head-Mount Display), a HUD (Head-Up Display) provided in a vehicle, a TV, a mobile phone, a smartphone, a computer, a wearable device, a home appliance, a digital signboard, a vehicle, a fixed robot, a mobile robot, or the like.

[0429] The XR / PCC apparatus 23300 can obtain information on the surrounding space or real objects by analyzing 3D point cloud data or image data acquired from various sensors or an external device to generate position data and characteristic data for 3D points, and can render and output an XR object to be output. For example, the XR / PCC apparatus 23300 can output an XR object including additional information regarding the recognized object corresponding to the recognized object.

[0430] <PCC+Autonomous Driving+XR>

[0431] The autonomous driving vehicle 23200 is applied with PCC technology and XR technology and is implemented in a mobile robot, a vehicle, an unmanned aerial vehicle, or the like.

[0432] The autonomous driving vehicle 23200 to which XR / PCC technology is applied may mean an autonomous driving vehicle equipped with means for providing an XR video or an autonomous driving vehicle that is a target of control / interaction within the XR video. In particular, the autonomous driving vehicle 23200 that is a target of control / interaction within the XR video is distinguished from the XR apparatus 23300 and can be interlocked with each other.

[0433] The autonomous vehicle 23200, which is equipped with a means for providing XR / PCC images, acquires sensor information from sensors including a camera and outputs XR / PCC images generated based on the acquired sensor information. For example, the autonomous vehicle 23200 may be equipped with a HUD and output XR / PCC images, thereby providing passengers with XR / PCC objects corresponding to real objects or objects on a screen.

[0434] In this case, when the XR / PCC object is output to the HUD, at least a portion of the XR / PCC object may be output to overlap with an actual object toward which the passenger's gaze is directed. Meanwhile, when the XR / PCC object is output to a display provided in the autonomous vehicle 23200, at least a portion of the XR / PCC object may be output to overlap with an object within the screen. For example, the autonomous vehicle 23200 may output XR / PCC objects corresponding to objects such as a roadway, another vehicle, a traffic light, a traffic sign, a motorcycle, a pedestrian, a building, etc.

[0435] The VR (Virtual Reality) technology, AR (Augmented Reality) technology, MR (Mixed Reality) technology, and / or PCC (Point Cloud Compression) technology according to the embodiments can be applied to various devices.

[0436] In other words, VR technology is a display technology that presents real objects and backgrounds only as CG images. On the other hand, AR technology is a technology that displays virtual CG images on top of images of real things. MR technology is similar to the above-mentioned AR technology in that it mixes virtual objects into the real world. However, AR technology clearly distinguishes between real objects and virtual objects made of CG images, and uses virtual objects in a way that complements real objects, while MR technology is different from AR technology in that virtual objects and real objects are considered to have the same characteristics. More specifically, for example, hologram services are an application of the above-mentioned MR technology.

[0437] However, recently, rather than clearly distinguishing between VR, AR, and MR technologies, they are referred to as XR (extended reality) technology. Therefore, embodiments of the present invention can be applied to any of VR, AR, MR, and XR technologies. These technologies are applied with encoding / decoding based on PCC, V-PCC, and G-PCC technologies.

[0438] The PCC method / apparatus according to the embodiment can be applied to a vehicle 23200 that provides autonomous driving services.

[0439] An autonomous vehicle 23200 that provides autonomous driving services is connected to the PCC device via wired or wireless communication.

[0440] When the point cloud compressed data (PCC) transceiver according to the embodiment is connected to the autonomous vehicle 23200 via wired or wireless communication, it can receive / process AR / VR / PCC service-related content data that can be provided along with the autonomous driving service and transmit it to the autonomous vehicle 23200. Also, when the point cloud data transceiver is mounted on the autonomous vehicle 23200, the point cloud transceiver can receive / process AR / VR / PCC service-related content data according to a user input signal input through a user interface device and provide it to a user. The vehicle or user interface device according to the embodiment can receive a user input signal. The user input signal according to the embodiment may include a signal instructing an autonomous driving service.

[0441] As described above, the V-PCC-based point cloud video encoder shown in FIG. 1, 4, 18, 20, or 21 generates patches by projecting 3D point cloud data (or content) into 2D space. The patches generated in 2D space are divided into a geometry image (referred to as a geometry frame or geometry patch frame) indicating position information and a texture image (referred to as a feature frame or feature patch frame) indicating color information. The geometry image and texture image are video-compressed for each frame and output as a geometry image video bitstream (referred to as a geometry bitstream) and a texture image video bitstream (referred to as a feature bitstream). In addition, additional patch information (referred to as patch information or metadata), including each patch projection plane information and patch size information required for decoding the 2D patches at the receiving end, is also video-compressed and output as the additional patch information bitstream. In addition, the occupancy map, which indicates the presence or absence of a point for each pixel as 0 or 1, is entropy compressed or video compressed depending on whether it is lossless or lossy mode, and is output as an occupancy map video bitstream (also called occupancy map bitstream). The compressed geometry bitstream, compressed attribute bitstream, compressed additional patch information bitstream, and compressed occupancy map bitstream are multiplexed as the structure of the V-PCC bitstream.

[0442] According to an embodiment, the V-PCC bitstream may be transmitted to the receiving side as is, or may be encapsulated in the form of a file / segment in the file / segment encapsulation unit of Figure 1, Figure 18, Figure 20 or Figure 21 and transmitted to the receiving device, or may be stored in a digital storage medium (e.g., USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc.) In this specification, it is assumed that the file is in ISOBMFF file format as an embodiment.

[0443] Depending on the embodiment, the V-PCC bitstream may be transmitted over multiple tracks of a file or over one single track, as will be described in more detail below.

[0444] Figure 24 shows an example of a V-PCC bitstream structure according to an embodiment. In one embodiment, the V-PCC bitstream in Figure 24 is output from the V-PCC-based point cloud video encoder of Figure 1, Figure 4, Figure 18, Figure 20, or Figure 21.

[0445] A V-PCC bitstream consists of one or more V-PCC units. That is, a V-PCC bitstream is a collection of V-PCC units. Each V-PCC unit consists of a V-PCC unit header and a V-PCC unit payload. This specification classifies the data contained in the V-PCC unit payload by the V-PCC unit header, and therefore the V-PCC unit header includes type information indicating the type of the V-PCC unit. Each V-PCC unit payload can include at least one of geometry video data (i.e., compressed geometry bitstream), attribute video data (i.e., compressed attribute bitstream), occupancy video data (i.e., compressed occupancy map bitstream), patch data group (PDG), and sequence parameter set (SPS) according to the type information of the V-PCC unit header. A patch data group can also be referred to as an atlas. In this specification, atlas can be replaced with patch data group.

[0446] The V-PCC unit payload, which includes at least geometry video data, attribute video data, and occupancy map video data (also referred to as occupancy video data), corresponds to a video data unit (e.g., an HEVC NAL unit) that can be decoded by an appropriate video decoder.

[0447] The geometry video data, feature video data, and occupancy map video data are referred to as 2D video coding information for the geometry, feature, and occupancy map components of the coded point cloud, and the patch data group (or atlas) is referred to as non-video coded information. The patch data group includes additional patch information. The sequence parameter set includes overall coding information for the bitstream and may also be referred to as configuration and metadata information. The sequence parameter set and patch data group may also be referred to as signaling information and may be generated by a metadata processing unit within the point cloud video encoder or by another component / module within the point cloud video encoder. Furthermore, in this specification, the sequence parameter set and patch data group are referred to as initialization information, and the geometry video data, feature video data, and occupancy map video data are referred to as point cloud data.

[0448] As an example, if the type information in the V-PCC unit header indicates a patch data group (VPCC-PDG), the V-PCC unit payload can include a patch data group. Conversely, if the V-PCC unit payload includes a patch data group, the type information in the V-PCC unit header can identify this. Since this is a designer's choice, this specification is not limited to the above example. Also, other V-PCC units can be applied in the same or similar manner.

[0449] A patch data group is included in a V-PCC unit payload in the Patch Data Group Unit() format and can include information on one or more patch tile groups and at least one of a patch sequence parameter set, a patch frame parameter set, a patch frame geometry parameter set, a geometry patch parameter set, a patch frame attribute parameter set, and an attribute patch parameter set.

[0450] Meanwhile, a patch frame (or point cloud object) that is the target of point cloud data may be divided into one or more tiles. According to an embodiment, a tile may represent a certain area in 3D space or a certain area on a 2D plane. Furthermore, a tile may be a rectangular cuboid within a bounding box, a sub-bounding box, or a portion of a patch frame. In this specification, dividing a patch frame (or point cloud object) into one or more tiles may be performed in the point cloud video encoder of FIG. 1, the patch generator of FIG. 18, the point cloud pre-processing unit of FIG. 20, or the patch generator of FIG. 21, or may be performed in a separate component / module.

[0451] According to an embodiment, the V-PCC bitstream having the structure shown in FIG. 24 may be transmitted to the receiving side as is, or may be encapsulated in a file / segment format and transmitted to the receiving side.

[0452] In this specification, an example in which the V-PCC bitstream is encapsulated in a file format and transmitted is taken as an example. For example, the V-PCC bitstream can be encapsulated in a file format based on ISOBMFF (ISO Base Media File Format).

[0453] In one embodiment, the encapsulation of the V-PCC bitstream into a file is performed by the file / segment encapsulation unit 10003 in Fig. 1, the transmission unit 18008 in Fig. 18, the file / segment encapsulation unit 20004 in Fig. 20, or the file / segment encapsulation unit 21009 in Fig. 21. In one embodiment, the decapsulation of the file into a V-PCC bitstream is performed by the file / segment decapsulation unit 10007 in Fig. 1, the reception unit in Fig. 19, the file / segment decapsulation unit 20005 in Fig. 20, or the file / segment decapsulation unit 22000 in Fig. 22.

[0454] Figure 25 is a visualization of a multiple track V-PCC file structure according to an embodiment, i.e., an example of the layout of an ISOBMFF-based file containing multiple tracks.

[0455] An ISOBMFF-based file according to an embodiment may be called a container, a container file, a media file, a V-PCC file, etc. Specifically, the file may be composed of boxes and / or information called ftyp, meta, moov, mdat, etc.

[0456] The ftyp box (file type box) can provide information related to the file type or file compatibility for the file. The receiving side can classify the file by referring to the ftyp box.

[0457] The meta box can contain a vpcg{0, 1, 2, 3} box (V-PCC group box, described in more detail below).

[0458] The mdat box, also called the media data box, can contain a video coded geometry bitstream, a video coded attribute bitstream, a video coded occupancy map bitstream, and a patch data group bitstream.

[0459] The moov box, also known as a movie box, can contain metadata for the media data of the file (e.g., geometry bitstream, attribute bitstream, occupancy map bitstream, etc.). For example, it can contain information necessary for decoding and playing the media data, and can contain information about the samples of the file. The moov box can act as a container for all metadata. The moov box can be the top-layer box among the metadata-related boxes. According to an embodiment, there is only one moov box in a file.

[0460] In one embodiment, in the ISOBMFF container structure as shown in Figure 25, V-PCC units contained in a V-PCC bitstream are mapped to individual tracks within the file based on their type.

[0461] Based on the layout as in Figure 25, an ISOBMFF container for a V-PCC bitstream may include the following:

[0462] 1) A V-PCC track can be included. The V-PCC track includes samples carrying payloads of V-PCC units containing sequence parameter sets and non-video coding information (e.g., type information indicating the type of the V-PCC unit in a V-PCC unit header indicates a sequence parameter set or a patch data group). The V-PCC track can also provide track references to other tracks containing samples carrying payloads of V-PCC units containing video compression information (e.g., type information in a V-PCC unit header indicates geometry video data, attribute video data, or occupancy map video data).

[0463] 2) It may include one or more restricted video scheme tracks for geometry video data. The samples included in this track include NAL units for video-coded elementary streams for geometry video data. In this case, the type information in the V-PCC unit header indicates geometry video data, and the NAL units correspond to payloads that carry the geometry video data.

[0464] 3) It may include one or more constrained video scheme tracks for characteristic video data, whose samples include NAL units for video-coded elementary streams for the characteristic video data, where type information in the V-PCC unit header indicates characteristic video data, and the NAL units correspond to payloads carrying the characteristic video data.

[0465] 4) It may include one restricted video scheme track for occupancy map video data. The samples included in this track include NAL units for the video coded elementary stream for the occupancy map video data. In this case, the type information in the V-PCC unit header indicates the occupancy map video data, and the NAL units correspond to the payloads carrying the occupancy map video data.

[0466] According to an embodiment, tracks containing video coded elementary streams for geometry video data, attribute video data and occupancy map video data are called component tracks.

[0467] In one embodiment, synchronization between elementary streams contained in such component tracks is handled by ISOBMFF track timing structures (stts, ctts, cslg in movie fragments, or equivalent mechanisms). Samples contributing to the same point cloud frame across different video encoded component tracks and V-PCC tracks can have the same composition time.

[0468] The V-PCC parameter set used for this sample can have a decoding time that is the same as the composition time of the frame or an earlier one.

[0469] ISOBMFF files carrying V-PCC content can be classified by V-PCC-defined brand. Tracks of V-PCC content can be grouped in a VPCCGroupBox, a file-level EntityToGroupBox with a V-PCC-specific grouping 4CC value ('vpcg'). A VPCCGroupBox can be provided as an entry point for accessing V-PCC content within a container and can contain initial metadata identifying the V-PCC content. The VPCCGroupBox, an EntityToGroupBox, can be included in a MetaBox box or a Moov box.

[0470] An entity group is a group of items that group tracks. Entities within an entity group may share a particular characteristic or have a particular relationship, as dictated by the grouping type.

[0471] Entity groups are indicated within a GroupsListBox, which is contained within at least one of a file-level MetaBox box, a Movie-level MetaBox box, and a track-level MetaBox box. Entity groups specified in a GroupsListBox of a file-level MetaBox refer to tracks or file-level items. Entity groups specified in a GroupsListBox of a movie-level MetaBox refer to movie-level items. Entity groups specified in a GroupsListBox of a track-level MetaBox refer to the track-level items of that track.

[0472] The GroupsListBox contains EntityToGroupBoxes, each specifying one entity group, as follows:

[0473] Groups List box

[0474] Box Type: 'grpl'

[0475] Container: MetaBox that is not contained in AdditionalMetadataContainerBox

[0476] Mandatory: No

[0477] Quantity: Zero or One

[0478] The GroupsListBox contains the entity groups specified for the file. This box contains a set of full boxes, each called EntityToGroupBox, with a 4-character code indicating the defined grouping type.

[0479] The GroupsListBox does not exist within an AdditionalMetadataContainerBox.

[0480] When a GroupsListBox exists within a file-level MetaBox, there may be no item ID value in the ItemInfoBox in the file-level MetaBox that is the same as the track ID value in the TrackHeaderBox, as shown below.

[0481] aligned(8) class GroupsListBox extends Box('grpl') {

[0482] }

[0483] Box Type: As specified below with the grouping_type value for the EntityToGroupBox

[0484] Container: GroupsListBox

[0485] Mandatory: No

[0486] Quantity: One or more

[0487] The EntityToGroupBox identifies an entity group.

[0488] The box type (grouping_type) indicates the grouping type of the entity group. Each grouping_type code is associated with semantics that describe the grouping. The following explains the grouping_type values:

[0489] 'altr': Items and tracks mapped to this grouping can be substituted for each other, and only one of them can be played (if the mapped items and tracks are part of the presentation, e.g., displayable items or tracks) or processed by other means (if the mapped items or tracks are not part of the presentation, e.g., metadata). The player can select the first entity from the list of entity_id values ​​to process, e.g., decode and play for mapped items and tracks that are part of the presentation, and that suit the needs of the application. An entity_id value shall be mapped to only one grouping of type 'altr'. An alternate group of entities consists of items and tracks that are mapped to the same entity group of type 'altr'.

[0490] Note: EntityToGroupBox includes extensions specific to grouping_type.

[0491] aligned(8) class EntityToGroupBox(grouping_type, version, flags) extends FullBox(grouping_type, version, flags) {

[0492] unsigned int(32) group_id;

[0493] unsigned int(32) num_entities_in_group;

[0494] for(i=0; i <num_entities_in_group; i++)

[0495] unsigned int(32) entity_id;

[0496] }

[0497] In this embodiment, the group_id is a non-negative integer assigned to a particular grouping that is not identical to the group_id of another EntityToGroupBox, the item_ID value of the hierarchical level (file, movie, or track) that contains the GroupsListBox, or the track_ID value (if the GroupsListBox is contained at the file level).

[0498] In one embodiment, num_entities_in_group indicates the number of entity_id values ​​that are mapped to this entity group.

[0499] In this embodiment, entity_id is resolved to an item when an item with item_ID equal to entity_id is present in the hierarchy level (file, movie, or track) that contains the GroupsListBox, when an item with item_ID equal to entity_id is present in the hierarchy level (file, movie, or track) that contains the GroupsListBox, or to a track when a track with track_ID equal to entity_id is present and the GroupsListBox is contained in the file level.

[0500] The V-PCC group box will be described below.

[0501] Box Type: 'vpcg'

[0502] Container: GroupListBox

[0503] Mandatory: Yes

[0504] Quantity: One or more

[0505] The V-PCC group box is included in a GroupsListBox and provides a list of the tracks that comprise a V-PCC content.

[0506] V-PCC content specific information, such as mapping of the attribute types and layers to the related tracks, are listed in this box. This information provides a convenient way to have an initial understanding of the V-PCC content. For flexible configuration of V-PCC content to support various client functions, multiple versions of encoded V-PCC components are listed in this box. V-PCC-defined profile, tier, and level information is also transmitted in this box as follows:

[0507] aligned(8) class VPCCGroupBox() extends EntityToGroupBox('vpcg', version, flags) {

[0508] for (i=0;i <num_entities_in_group;i++){

[0509] unsigned int(4) data_type;

[0510] unsigned int(4) attribute_type;

[0511] unsigned int(1) multiple_layer_present_flag;

[0512] unsigned int(4) layer_count_munus1;

[0513] for (i=0;i <layer_count_minus1+1;i++){

[0514] unsigned int(4) layer_id;

[0515] }

[0516] unsigned int(32) entity_id;

[0517] }

[0518] unsigned int(4) CC_layer_count_minus1;

[0519] vpcc_profile_tier_level()

[0520] }

[0521] The data_type according to the embodiment indicates the track type of the PCC data in the referenced track.

[0522] 26 is a table showing an example of track types of PCC data assigned to data types (data_type) according to an embodiment. For example, a data_type value of 1 indicates a V-PCC track, a value of 2 indicates a geometry video track, a value of 3 indicates a characteristic video track, and a value of 4 indicates an occupancy map video track.

[0523] According to an embodiment, multiple_layer_present_flag indicates whether a single geometry or feature layer or multiple geometries or feature layers are transmitted to an associated entity (or track). For example, when multiple_layer_present_flag is 0, it indicates that a single geometry or feature layer is transmitted to an associated entity (or track), and when it is 1, it indicates that multiple geometries or feature layers are transmitted to an associated entity (or track). According to an embodiment, a V-PCC track (i.e., data_type indicates a V-PCC track) has multiple_layer_present_flag set to 0. As another example, when data_type indicates a geometry video track, when multiple_layer_present_flag is 0, it indicates that the geometry video track transmits a single geometry layer, and when multiple_layer_present_flag is 1, it indicates that the geometry video track transmits multiple geometry layers. As another example, if the data_type indicates a feature video track, if the value of multiple_layer_present_flag is 0, the feature video track transmits a single feature layer, and if the value of multiple_layer_present_flag is 1, the feature video track transmits multiple feature layers.

[0524] In one embodiment, layer_count_minus1 plus 1 indicates the number of geometry and / or feature layers transmitted in the associated entity (or track). A V-PCC track in one embodiment has layer_count_minus1 whose value is 0. For example, if the data_type indicates a geometry video track, layer_count_minus1 plus 1 may indicate the number of geometry layers transmitted in the geometry video track. As another example, if the data_type indicates a feature video track, layer_count_minus1 plus 1 may indicate the number of feature layers transmitted in the feature video track.

[0525] In one embodiment, the layer_id indicates the layer identifier of a geometry and / or attribute layer in the associated entity (or track). In one embodiment, a V-PCC track has a layer_id whose value is 0. The set of layer_id values ​​for a V-PCC component track type is a contiguous set of integers sorted in increasing order, starting from 0. For example, if the data_type indicates a geometry video track, the layer_id may indicate the layer identifier of the geometry layer. As another example, if the data_type indicates a attribute video track, the layer_id may indicate the layer identifier of the attribute layer.

[0526] In this embodiment, pcc_layer_count_munus1 plus 1 indicates the number of layers used to encode the geometry component and / or the feature component of the point cloud stream. For example, if the data_type indicates a geometry video track, pcc_layer_count_munus1 plus 1 may indicate the number of layers used to encode the geometry component of the point cloud stream. As another example, if the data_type indicates a feature video track, pcc_layer_count_munus1 plus 1 may indicate the number of layers used to encode the feature component of the point cloud stream.

[0527] According to the embodiment, attribute_type indicates the attribute type of the attribute video data carried in the referenced entity (or track).

[0528] Figure 27 shows an example of attribute types assigned to attribute_type according to an embodiment. For example, a value of attribute_type of 0 can indicate texture, 1 can indicate material ID, 2 can indicate transparency, 3 can indicate reflectivity, and 4 can indicate normal.

[0529] The entity_id in the example indicates an identifier for the associated item or track. If an item with an item_ID identical to the entity_id exists in the hierarchical level (file, movie or track) that contains the GroupsListBox, it is resolved to the item. Alternatively, if a track with a track_ID identical to the entity_id exists and the GroupsListBox is contained at the file level, it is resolved to the track.

[0530] vpcc_profile_tier_level() is the same as the profile_tier_level() specified in the sequence parameter set (sequence_parameter_set()).

[0531] FIG. 28 illustrates an example of a syntax structure of a profile tier level (profile_tier_level) according to an embodiment.

[0532] The ptl_tier_flag field indicates the codec profile tier used to encode the V-PCC content.

[0533] The ptl_profile_idc field indicates the profile information that the coded point cloud sequence follows.

[0534] The ptl_level_idc field indicates the level of the codec profile to which the coded point cloud sequence conforms.

[0535] The V-PCC track will be described below.

[0536] Each V-PCC content entry point can be represented by a unique V-PCC track. An ISOBMFF file can contain multiple V-PCC contents, so multiple V-PCC tracks exist within the file. A V-PCC track is identified by the Volume Visual Media Box Handler Type 'volm' in the Handler Box of the Media Box.

[0537] Box Type: 'vohd'

[0538] Container: MediaInformationBox

[0539] Mandatory: Yes

[0540] Quantity: Exactly one

[0541] Volumetric tracks use the VolumetricMediaHeaderBox in the MediaInformationBox

[0542] aligned(8) class VolumetricMediaHeaderBox extends FullBox('vohd', version = 0, 1) {

[0543] / / if we don't need anything here, then use Null Media Header

[0544] }

[0545] / / random access point for the patch stream

[0546] A random access point for a V-PCC track may be a sample carrying an I-patch frame in an empty decoder buffer. To indicate a random access point for a V-PCC track, a syncsampleBox exists. This sync sample indicates a sync sample carrying an I-patch frame.

[0547] If some samples in a track, including track fragments, are non-sync samples, but the flag sample_is_non_sync_sample of samples in track fragments is valid and describes the samples, even if the SyncSampleBox is not present, the SyncSampleBox should be present in the SampleTableBox if some samples in the track, including any track fragments, are non-sync samples, but the flag sample_is_non_sync_sample of samples in track fragments is valid and describes the samples, even if the SyncSampleBox is not present. If the track is not fragmented and no SyncSampleBox is present, all samples in the track are sync samples.

[0548] Box Type: 'stbl'

[0549] Container: MediaInformationBox

[0550] Mandatory: Yes

[0551] Quantity: Exactly one

[0552] aligned(8) class SampleTableBox extends Box('stbl') {

[0553] }

[0554] Box Type: 'stss'

[0555] Container: SampleTableBox

[0556] Mandatory: No

[0557] Quantity: Zero or one

[0558] This box provides a compact marking of sync samples in a stream. The table is arranged in strict ascending sample number order. If SyncSampleBox is not present, all samples are sync samples.

[0559] aligned(8) class SyncSampleBox

[0560] extends FullBox('stss', version = 0, 0) {

[0561] unsigned int(32) entry_count;

[0562] int i;

[0563] for (i=0; i < entry_count; i++) {

[0564] unsigned int(32) sample_number;

[0565] }

[0566] }

[0567] version is an integer that indicates the version of this box.

[0568] entry_count is an integer that gives the number of entries in the table below. If the value of entry_count is 0, there are no sink samples in that stream and the table below is empty.

[0569] The sample_number provides the sample number for each sync sample in the stream. In particular, the sample_number provides the sample number for each sync sample carrying an I-patch frame in the V-PCC track.

[0570] The V-PCC track sample entry will be explained below.

[0571] Sample Entry Type: 'vpc1'

[0572] Container: SampleDescriptionBox ('stsd')

[0573] Mandatory: A 'vpc1' sample entry is mandatory

[0574] Quantity: one or more

[0575] The track sample entry type 'vpc1' is used. The V-PCC track sample entry contains a VPCC Configuration Box as defined below, which contains a VPCCDecoderConfigurationRecord as defined below. An optional BitRateBox may be present in the V-PCC track sample entry to signal bitrate information for the V-PCC video stream.

[0576] aligned(8) class VPCCDecoderConfigurationRecord {

[0577] unsigned int(8) configurationVersion = 1;

[0578] unsigned int(8) numOfSequenceParameterSets;

[0579] for (i=0; i<numOfSequenceParameterSets; i++) {

[0580] sequence_parameter_set();

[0581] }

[0582] / / additional fields

[0583] }

[0584] class VPCCConfigurationBox extends Box('vpcc') {

[0585] VPCCDecoderConfigurationRecord() VPCCConfig;

[0586] }

[0587] aligned(8) class VPCCSampleEntry() extends VolumetricSampleEntry ('vpc1') {

[0588] VPCCConfigurationBox config;

[0589] }

[0590] class VolumetricSampleEntry(codingname) extends SampleEntry (codingname){

[0591] }

[0592] The configurationVersion is a version field. Incompatible changes to the record are indicated by a change in the version number.

[0593] numOfSequenceParameterSets indicates the number of V-PCC sequence parameter sets signaled in the decoder configuration record.

[0594] The compressor name in the base class VisualSampleEntry indicates the name of the compressor used, with the recommended value "\012VPCC Coding".

[0595] The V-PCC specification allows multiple instances of the VPCC_SPS unit (within ids 1 to 15). Therefore, a VPCCSampleEntry contains multiple sequence_parameter_set unit payloads.

[0596] The V-PCC sample format will be explained below.

[0597] Each sample in the V-PCC track corresponds to a single point cloud frame.

[0598] The samples corresponding to this frame in the various component tracks have the same composition time as the V-PCC track sample for the frame within the V-PCC track.

[0599] Each V-PCC sample contains only one V-PCC unit payload whose type information in its V-PCC unit header indicates a patch data group (PDG), and this V-PCC unit payload contains one or more patch tile group unit payloads, as shown in Figure 24.

[0600] aligned(8) class vpcc_unit_payload_struct {

[0601] vpcc_unit_payload();

[0602] }

[0603] aligned(8) class VPCCSample {

[0604] vpcc_unit_payload_struct();

[0605] }

[0606] The vpcc_unit_payload() is the payload of a V-PCC unit whose type information in the V-PCC unit header indicates a patch data group (PDG), and contains one patch_data_group_unit() instance.

[0607] For a V-PCC track, the samples carrying the I patch frames are defined as sync samples.

[0608] The V-PCC track reference will now be described.

[0609] To link a V-PCC track to a component video track, the track reference tool in the ISOBMFF standard is used. Three TrackReferenceTypeBoxes are added to the TrackReferenceBox within the V-PCC track's track box, one for each component.

[0610] The TrackReferenceTypeBox contains an array of track_IDs designating the video tracks that reference the V-PCC track.

[0611] The reference_type in a TrackReferenceTypeBox identifies the type of that component (i.e. geometry, attribute, or occupancy map). The 4CC new track reference types include 'pcca', 'pccg', 'pcco', etc.

[0612] In the 'pcca' type, the referenced track contains a video coded attribute V-PCC component.

[0613] In the 'pccg' type, the referenced track contains video coded geometry V-PCC components.

[0614] In the 'pcco' type, the referenced track contains a video coded occupancy map V-PCC component.

[0615] In the following, video encoded V-PCC component tracks will be described.

[0616] Since it makes no sense to display frames decoded from attribute, geometry, or occupancy map tracks without reconstructing a point cloud on the player side, a restricted video scheme type is defined for the video-coded track. A V-PCC video track contains the 4CC identifier 'pccv'. This 4CC identifier 'pccv' is included in the scheme_type field of the SchemeTypeBox in the RestrictedSchemeInfoBox of the restricted sample entry.

[0617] The use of the V-PCC video scheme for the restricted video sample entry type 'resv' indicates that the decoded picture contains point cloud attribute, geometry, or occupancy map data.

[0618] The use of the V-PCC video scheme is represented by a scheme_type equal to 'pccv' (video-based point cloud video) in the SchemeTypeBox of the RestrictedSchemeInfoBox.

[0619] This box is a container that contains boxes that indicate V-PCC specific information for this track. VPCCVideoBox provides PCC specific parameters applicable to all samples in the track.

[0620] Box Type: 'pccv'

[0621] Container: SchemeInformationBox

[0622] Mandatory: Yes, when scheme_type is equal to 'pccc'

[0623] Quantity: Zero or one

[0624] aligned(8) class VPCCVideoBox extends FullBox('vpcc', 0, 0) {

[0625] unsigned int(4) data_type;

[0626] unsigned int(4) attribute_count;

[0627] for(int i=0; i <attribute_count+1;i++) {

[0628] unsigned int(4) attribute_type;

[0629] }

[0630] unsigned int(1) multiple_layer_present_flag;

[0631] unsigned int(4) layer_count_minus1;

[0632] for (i = 0 ; i< layer_count_minus1+1 ; i++) {

[0633] unsigned int(4) layer_id;

[0634] }

[0635] }

[0636] The data_type indicates the data type contained in the video sample included in the corresponding track.

[0637] 29 shows an example of the type of PCC data in a referenced track according to an embodiment. For example, a value of data_type of 1 indicates a V-PCC track, a value of 2 indicates a geometry video track, a value of 3 indicates a characteristic video track, and a value of 4 indicates an occupancy map video track.

[0638] The multiple_layer_present_flag indicates whether a single geometry or feature layer or multiple geometries or feature layers are transmitted in this track. For example, if multiple_layer_present_flag is 0, it indicates that a single geometry or feature layer is transmitted in this track, and if it is 1, it indicates that multiple geometries or feature layers are transmitted in this track. As another example, if the data_type indicates a geometry video track, if the value of multiple_layer_present_flag is 0, the geometry video track transmits a single geometry layer, and if the value of multiple_layer_present_flag is 1, it indicates that the geometry video track transmits multiple geometry layers. As another example, if the data_type indicates a feature video track, if the value of multiple_layer_present_flag is 0, the feature video track transmits a single feature layer, and if the value of multiple_layer_present_flag is 1, it indicates that the feature video track transmits multiple feature layers.

[0639] layer_count_minus1 plus 1 indicates the number of geometry and / or feature layers transmitted in this track. For example, if the data_type indicates a geometry video track, layer_count_minus1 plus 1 indicates the number of geometry layers transmitted in the geometry video track. As another example, if the data_type indicates a feature video track, layer_count_minus1 plus 1 indicates the number of feature layers transmitted in the feature video track.

[0640] The layer_id indicates the layer identifier of the geometry and / or attribute layer associated with the sample in this track. For example, if the data_type indicates a geometry video track, the layer_id indicates the layer identifier of the geometry layer. As another example, if the data_type indicates an attribute video track, the layer_id indicates the layer identifier of the attribute layer. The attribute_count may indicate the number of attribute data of the point cloud stream included in the track. One track may include one or more attribute data. In this case, one example is that the data_type indicates an attribute video track.

[0641] The attribute_type indicates the attribute type of the attribute video data transmitted in the corresponding track.

[0642] Figure 30 shows an example of an attribute type according to an embodiment. For example, the value of the attribute_type field is 0 for texture, 1 for material ID, 2 for transparency, 3 for reflectance, and 4 for normal.

[0643] When the field values ​​present in VPCCVideoBox are included in the signaling related to the item, the above information for the corresponding item (image) can be indicated.

[0644] Attribute sample grouping will now be described.

[0645] If a track contains point cloud feature data, it may contain data of one or more feature types, in which case the data feature types contained in the samples included in that track can be signaled as follows:

[0646] class PCCAttributeSampleGroupEntry extends VisualSampleGroupEntry('pcca') {

[0647] unsigned int(4) attribute_type;

[0648] }

[0649] The attribute_type indicates the attribute type of the attribute video data transmitted in the associated sample in the corresponding track.

[0650] Figure 31 shows an example of an attribute type according to an embodiment. For example, the value of the attribute_type field is 0 for texture, 1 for material ID, 2 for transparency, 3 for reflectivity, and 4 for normal.

[0651] Layer sample grouping will now be described.

[0652] The track point cloud may contain data associated with one or more layers of the same data type (e.g., geometry data, characteristic data). In this case, layer-related information for the data contained in the samples included in the track can be signaled as follows:

[0653] class PCCLayerSampleGroupEntry extends VisualSampleGroupEntry('pccl') {

[0654] unsigned int(4) pcc_layer_count_minus1;

[0655] unsigned int(4) layer_id;

[0656] }

[0657] pcc_layer_count_minus1 plus 1 indicates the number of layers used to encode the geometry and feature components of the point cloud stream.

[0658] layer_id indicates the layer identifier for the associated sample within the track.

[0659] Layer tracking grouping will now be described.

[0660] From the V-PCC point of view, in addition to geometry, all other types of information (other than occupancy and patch information) may have the same number of layers, and all information (geometry / attributes) tagged with the same layer index are related to each other, i.e., they refer to the same reconstructed point in 3D space.

[0661] To reconstruct the feature with layer index M, the geometry layer with index M can also be used. If the geometry layer disappears for some reason, the feature information is not needed since both are 'linked' to each other.

[0662] From a rendering perspective, the information is considered an 'augmenting' reconstructed point cloud, which means that by discarding some layers, a reasonable reconstruction can be obtained, but one example is to not discard layer 0.

[0663] In one embodiment, tracks belonging to the same layer have the same value of track_group_id for track_group_type 'pccl', and the track_group_id of tracks from one layer is different from the track_group_id of tracks from different layers.

[0664] By default, if this track grouping is not indicated for any track in a file, the file is assumed to contain content for only one layer.

[0665] aligned(8) class PCCLayerTrackGroupBox extends TrackGroupTypeBox('pccl') {

[0666] unsigned int(4) pcc_layer_count_minus1;

[0667] unsigned int(4) layer_id;

[0668] }

[0669] pcc_layer_count_minus1 plus 1 indicates the number of layers used to encode the geometry and / or feature components of the point cloud stream.

[0670] layer_id indicates the layer identifier for the associated pcc data (eg, geometry and / or attribute) layer transmitted in the corresponding track.

[0671] This can inform the player of Figure 20 or the renderer of Figure 1, Figure 19, Figure 20, or Figure 22 that data from tracks belonging to the same track group will be used to reconstruct point clouds of the same layer.

[0672] As explained above, this specification provides signaling information in at least one of the V-PCC track and the feature track to enable the client / player to select and use feature non-streams as needed.

[0673] Furthermore, this specification signals signaling information related to random access points among samples in a V-PCC track in at least one of a V-PCC track and a characteristic track.

[0674] Therefore, the client / player can obtain signaling information related to the random access point and obtain and use the required characteristic bitstream based on the signaling information, thereby enabling efficient and faster data processing.

[0675] This specification also signals track grouping information to be used together in at least one of the V-PCC track and the corresponding video track (i.e., geometry track or attribute track) when geometry and / or attribute video data in a file consists of multiple layers.

[0676] In addition, this specification provides that when geometry and / or attribute video data is configured as multiple layers in one track, sample grouping information for this is signaled in at least one of the V-PCC track and the corresponding video track (i.e., the geometry track or the attribute track).

[0677] Furthermore, in this specification, when one track contains multiple feature video data, sample grouping information for the multiple feature video data is signaled in at least one of the V-PCC track and the feature track.

[0678] Meanwhile, a patch frame (or point cloud object or atlas frame) that is a target of point cloud data may be divided into one or more tiles, as described above. According to an embodiment, a tile may represent a certain area in 3D space or a certain area on a 2D plane. Furthermore, a tile may be a rectangular cuboid within a bounding box, a sub-bounding box, or a part of a patch frame. In this specification, dividing a patch frame (or point cloud object) into one or more tiles may be performed in the point cloud video encoder of FIG. 1, the patch generator of FIG. 18, the point cloud pre-processing unit of FIG. 20, or the patch generator of FIG. 21, or may be performed in a separate component / module.

[0679] FIG. 32 illustrates an example of dividing one or more patch frames into a plurality of tiles by dividing one or more patch frames in the row direction and one or more patch frames in the column direction. A tile is a rectangular region within a patch frame, and a tile group can include a number of tiles within the patch frame. In this specification, a tile group includes a plurality of tiles of a patch frame that collectively form a rectangular (or square) region of the patch frame. In particular, the patch frame in FIG. 32 is divided into 24 tiles (= 6 tiles in the column direction * 4 tiles in the row direction), and this illustrates an example of dividing the patch frame into nine rectangular (or square) patch tile groups.

[0680] In this case, if the type information of the PCC unit header in Figure 24 indicates a patch data group (VPCC-PDG) containing at least one patch, the corresponding V-PCC unit payload contains a patch data group, and conversely, if the V-PCC unit payload contains a patch data group, the type information of the corresponding V-PCC unit header can identify this.

[0681] The patch data group may include information for one or more patch tile groups and at least one of a patch sequence parameter set, a patch frame parameter set, a patch frame geometry parameter set, a geometry patch parameter set, a patch frame attribute parameter set, and an attribute patch parameter set.

[0682] Figure 33 shows an example of the syntax structure of each V-PCC unit according to an embodiment. Each V-PCC unit consists of a V-PCC unit header and a V-PCC unit payload. The V-PCC unit of Figure 33 may contain more data, in which case it may further include a trailing_zero_8bits field. The trailing_zero_8bits field according to this embodiment is a byte corresponding to 0x00.

[0683] 34 illustrates an example of a syntax structure of a V-PCC unit header according to an embodiment. In one embodiment, the V-PCC unit header (vpcc_unit_header()) in FIG. 34 includes a vpcc_unit_type field. The vpcc_unit_type field indicates the type of the corresponding V-PCC unit.

[0684] FIG. 35 illustrates an example of a V-PCC unit type assigned to the vpcc_unit_type field according to an embodiment.

[0685] Referring to Figure 35, in one embodiment, if the value of the vpcc_unit_type field is 0, it indicates that the data contained in the V-PCC unit payload of the corresponding V-PCC unit is a sequence parameter set (VPCC_SPS); if it is 1, it indicates that it is a patch data group (VPCC_PDG); if it is 2, it indicates that it is occupied video data (VPCC_OVD); if it is 3, it indicates that it is characteristic video data (VPCC_AVD); and if it is 4, it indicates that it is geometry video data (VPCC_GVD).

[0686] The meaning, procedure, deletion, addition, etc. of the values ​​assigned to the vpcc_unit_type field can be easily changed by those skilled in the art, and therefore the present invention is not limited to the above embodiment.

[0687] At this time, the V-PCC unit payload follows the format of an HEVC NAL unit, i.e., the occupancy, geometry, or attribute video data V-PCC unit payload according to the vpcc_unit_type field value corresponds to a video data unit (e.g., an HEVC NAL unit) that can be decoded by a video decoder specified in the occupancy, geometry, or attribute parameter set V-PCC unit.

[0688] In one embodiment, if the vpcc_unit_type field indicates attribute video data (VPCC_AVD), geometry video data (VPCC_GVD), occupation video data (VPCC_OVD), or patch data group (VPCC_PDG), the corresponding V-PCC unit header further includes a vpcc_sequence_parameter_set_id field.

[0689] The vpcc_sequence_parameter_set_id field specifies the identifier (i.e., sps_sequence_parameter_set_id) of the active sequence parameter set (VPCC SPS). The value of the sps_sequence_parameter_set_id field may be in the range of 0 to 15.

[0690] In one embodiment, if the vpcc_unit_type field indicates attribute video data (VPCC_AVD), the V-PCC unit header further includes a vpcc_attribute_index field and a vpcc_attribute_dimension_index field.

[0691] The vpcc_attribute_index field indicates the index of the attribute video data carried in the attribute video data unit. The value of the vpcc_attribute_index field may range from 0 to (ai_attribute_count-1).

[0692] The vpcc_attribute_dimension_index field indicates the index of the attribute dimension group carried in the attribute video data unit. The value of the vpcc_attribute_dimension_index field may be in the range of 0 to 127.

[0693] The sps_multiple_layer_streams_present_flag field of the V-PCC unit header in Figure 34 indicates whether or not the vpcc_layer_index field and the pcm_separate_video_data(11) field are included.

[0694] For example, if the value of the vpcc_unit_type field indicates characteristic video data (VPCC_AVD) and the value of the sps_multiple_layer_streams_present_flag field is true (e.g., 0), the corresponding V-PCC unit header further includes the vpcc_layer_index field and the pcm_separate_video_data(11) field. In other words, if the value of the sps_multiple_layer_streams_present_flag field is true, it means that multiple layers exist for characteristic video data or geometry video data. In this case, a field indicating the index of the current layer (e.g., vpcc_layer_index) is required.

[0695] The vpcc_layer_index field indicates the index of the current layer of the specific video data. The vpcc_layer_index field has a value between 0 and 15.

[0696] For example, if the value of the vpcc_unit_type field indicates characteristic video data (VPCC_AVD) and the value of the sps_multiple_layer_streams_present_flag field is false (e.g., 1), the corresponding V-PCC unit header further includes a pcm_separate_video_data(15) field. In other words, if the value of the sps_multiple_layer_streams_present_flag field is false, this means that there are no multiple layers for characteristic video data and / or geometry video data. In this case, the field indicating the index of the current layer is not necessary.

[0697] For example, if the value of the vpcc_unit_type field indicates geometry video data (VPCC_GVD) and the value of the sps_multiple_layer_streams_present_flag field is true (e.g., 0), the corresponding V-PCC unit header further includes a vpcc_layer_index field and a pcm_separate_video_data(18) field.

[0698] The vpcc_layer_index field indicates the index of the current layer of geometry video data. The vpcc_layer_index field has a value between 0 and 15.

[0699] For example, if the value of the vpcc_unit_type field indicates geometry video data (VPCC_GVD) and the value of the sps_multiple_layer_streams_present_flag field is false (eg, 1), the corresponding V-PCC unit header further includes a pcm_separate_video_data(22) field.

[0700] For example, if the value of the vpcc_unit_type field indicates dedicated video data (VPCC_OVD) or patch data group (VPCC_PDG), the corresponding V-PCC unit header further includes a vpcc_reserved_zero_23bits field; otherwise, it further includes a vpcc_reserved_zero_27bits field.

[0701] On the other hand, the V-PCC unit header in FIG. 34 may further include a vpcc_pcm_video_flag field.

[0702] For example, a value of 1 in the vpcc_pcm_video_flag field may indicate that the associated geometry video data unit or attribute video data unit contains only Pulse Code Modulation (PCM) coded points. As another example, a value of 0 in the vpcc_pcm_video_flag field may indicate that the associated geometry video data unit or attribute video data unit contains non-PCM coded points. If the vpcc_pcm_video_flag field is not present, its value may be inferred to be 0.

[0703] FIG. 36 illustrates an example of the syntax structure of pcm_separate_video_data(bitCount) included in a V-PCC unit header according to an embodiment.

[0704] In FIG. 36, the bit count of pcm_separate_video_data varies depending on the value of the vpcc_unit_type field in the V-PCC unit header in FIG. 34, as described above.

[0705] pcm_separate_video_data(bitCount) may include the vpcc_pcm_video_flag field and the vpcc_reserved_zero_bitcount_bits field if the value of the sps_pcm_separate_video_present_flag field is true and not the vpcc_layer_index field, otherwise it may include the vpcc_reserved_zero_bitcountplus1_bits field.

[0706] A value of 1 in this vpcc_pcm_video_flag field can indicate that the associated geometry or attribute video data unit is PCM-coded point video only, and a value of 0 in this vpcc_pcm_video_flag field can indicate that the associated geometry or attribute video data unit contains PCM-coded points. If this vpcc_pcm_video_flag field is not present, its value is assumed to be 0.

[0707] FIG. 37 illustrates an example of a syntax structure of a V-PCC unit payload according to an embodiment.

[0708] The V-PCC unit payload in Figure 37 includes one of a sequence parameter set (sequence_parameter_set()), a patch data group (patch_data_group()), or a video data unit (video_data_unit()) depending on the vpcc_unit_type field value of the corresponding V-PCC unit header.

[0709] For example, if the vpcc_unit_type field indicates a sequence parameter set (VPCC_SPS), the V-PCC unit payload includes a sequence parameter set (sequence_parameter_set()), and if it indicates a patch data group (VPCC_PDG), it includes a patch data group (patch_data_group()). In addition, if the vpcc_unit_type field indicates dedicated video data (VPCC_OVD), the V-PCC unit payload includes a dedicated video data unit (video_data_unit()) that transmits the dedicated video data, if it indicates geometry video data (VPCC_GVD), it includes a geometry video data unit (video_data_unit()) that transmits the geometry video data, and if it indicates attribute video data (VPCC_AVD), it includes an attribute video data unit (video_data_unit()) that transmits the attribute video data.

[0710] FIG. 38 illustrates an example of the syntax structure of a sequence parameter set () included in a V-PCC unit payload according to an embodiment.

[0711] The sequence parameter set (SPS) of Figure 38 can be applied to a coded point cloud sequence that includes a sequence of coded geometry video data units, attribute video data units, and occupation video data units.

[0712] The sequence parameter set (SPS) of Figure 38 may include a profile_tier_level(), an sps_sequence_parameter_set_id field, an sps_frame_width field, an sps_frame_height field, and an sps_avg_frame_rate_present_flag field.

[0713] This profile_tier_level() indicates the codec information used to compress the sequence parameter set.

[0714] The sps_sequence_parameter_set_id field provides an identifier for the sequence parameter set for reference by other syntax elements.

[0715] The sps_frame_width field indicates the nominal frame width in terms of integer luma samples.

[0716] The sps_frame_height field indicates the nominal frame height in terms of integer luma samples.

[0717] The sps_avg_frame_rate_present_flag field indicates whether average nominal frame rate information is present in this bitstream. For example, a value of 0 in the sps_avg_frame_rate_present_flag field indicates that average nominal frame rate information is not present in this bitstream. A value of 1 in the sps_avg_frame_rate_present_flag field indicates that average nominal frame rate information is displayed in this bitstream. For example, if the value of the sps_avg_frame_rate_present_flag field is true, i.e., 1, the sequence parameter set may further include an sps_avg_frame_rate field, an sps_enhanced_occupancy_map_for_depth_flag field, and an sps_geometry_attribute_different_layer_flag field.

[0718] The sps_avg_frame_rate field indicates the average nominal point cloud frame rate, in units of point cloud frames per 256 seconds. If the sps_avg_frame_rate field is not present, the value of the field is 0. During the reconstruction phase, the decoded occupancy, geometry, and attribute video are converted to the nominal width, height, and frame rate using appropriate scaling.

[0719] The sps_enhanced_occupancy_map_for_depth_flag field indicates whether the decoded occupancy map video includes information on whether intermediate depth positions between two depth layers are occupied. For example, if the value of this sps_enhanced_occupancy_map_for_depth_flag field is 1, it can indicate whether the decoded occupancy map video includes information on whether intermediate depth positions between two depth layers are occupied. If the value of this sps_enhanced_occupancy_map_for_depth_flag field is 0, it can indicate that the decoded occupancy map video does not include information on whether intermediate depth positions between two depth layers are occupied.

[0720] The sps_geometry_attribute_different_layer_flag field indicates whether the number of layers used to encode the geometry and attribute video data is different. For example, a value of 1 in the sps_geometry_attribute_different_layer_flag field may indicate that the number of layers used to encode the geometry and attribute video data is different. For example, two layers may be used to encode the geometry video data, and one layer may be used to encode the attribute video data. Also, a value of 1 in the sps_geometry_attribute_different_layer_flag field indicates whether the number of layers used to encode the geometry and attribute video data is signaled in the patch sequence data unit.

[0721] The sps_geometry_attribute_different_layer_flag field indicates the presence or absence of the sps_layer_count_geometry_minus1 field and the sps_layer_count_minus1 field. For example, if the value of the sps_geometry_attribute_different_layer_flag field is true (e.g., 1), the sps_layer_count_geometry_minus1 field is also included, and if the value is false (e.g., 0), the sps_layer_count_minus1 field can also be included.

[0722] The sps_layer_count_geometry_minus1 field indicates the number of layers used to encode the geometry video data.

[0723] The sps_layer_count_minus1 field indicates the number of layers used to encode the geometry and attribute video data.

[0724] If the value of the sps_layer_count_minus1 field is greater than 0, the sequence parameter set may further include an sps_multiple_layer_streams_present_flag field and an sps_layer_absolute_coding_enabled_flag[0]=1 field.

[0725] The sps_multiple_layer_streams_present_flag field indicates whether the geometry layers or attribute layers are located in a single video stream or in separate video streams. For example, a value of 0 in the sps_multiple_layer_streams_present_flag field indicates that all geometry layers or attribute layers are placed in a single geometry video stream or a single attribute video stream, respectively. A value of 1 in the sps_multiple_layer_streams_present_flag field indicates that all geometry layers or attribute layers are placed in separate video streams.

[0726] The sequence parameter set (SPS) also includes a repeat statement that is repeated the number of times equal to the value of the sps_layer_count_minus1 field, and this repeat statement includes an sps_layer_absolute_coding_enabled_flag field. In this example, i is initialized to 0 and increments by 1 each time the repeat statement is executed, and the repeat statement is repeated until the i value becomes the value of the sps_layer_count_minus1 field. In addition, if the value of the sps_layer_absolute_coding_enabled_flag field is 0 and the i value is greater than 0, the sequence parameter set (SPS) further includes an sps_layer_predictor_index_diff field; otherwise, the sps_layer_predictor_index_diff field is not included.

[0727] A value of 1 in the sps_layer_absolute_coding_enabled_flag[i] field indicates that the geometry layer with index i is coded without any form of layer prediction. A value of 0 in the sps_layer_absolute_coding_enabled_flag[i] field indicates that the geometry layer with index i is predicted first from another, earlier coded layer before coding.

[0728] The sps_layer_predictor_index_diff[i] field indicates that if the value of the sps_layer_absolute_coding_enabled_flag[i] field is 0, it is used to calculate the predictor of the geometry layer with index i.

[0729] A sequence parameter set (SPS) according to this specification may further include an sps_pcm_patch_enabled_flag field. The sps_pcm_patch_enabled_flag field indicates the presence or absence of an sps_pcm_separate_video_present_flag field, an occupancy_parameter_set(), a geometry_parameter_set(), and an sps_attribute_count field. For example, if the sps_pcm_patch_enabled_flag field is 1, the sequence parameter set (SPS) may further include an sps_pcm_separate_video_present_flag field, an occupancy_parameter_set(), a geometry_parameter_set(), and an sps_attribute_count field. That is, if the sps_pcm_patch_enabled_flag field is 1, it indicates that a patch having PCM-coded points is present in the bitstream.

[0730] The sps_pcm_separate_video_present_flag field indicates whether the PCM-coded geometry video data and the attribute video data are stored in separate video streams. For example, if the value of the sps_pcm_separate_video_present_flag field is 1, it indicates that the PCM-coded geometry video data and the attribute video data are stored in separate video streams.

[0731] occupancy_parameter_set() contains information about the occupancy map.

[0732] The geometry_parameter_set() contains information about the geometry video data.

[0733] The sps_attribute_count field indicates the number of attributes associated with the point cloud.

[0734] Furthermore, the sequence parameter set (SPS) according to this specification includes a repetitive statement that is repeated the number of times equal to the value of the sps_attribute_count field, and in one embodiment, this repetitive statement includes an sps_layer_count_attribute_minus1 field and attribute_parameter_set() if the sps_geometry_attribute_different_layer_flag field is 1. In one embodiment, in this repetitive statement, i is initialized to 0 and incremented by 1 each time the repetitive statement is executed, and the repetition occurs until the value of i becomes the value of the sps_attribute_count field.

[0735] The sps_layer_count_attribute_minus1[i] field indicates the number of layers used to encode the i-th attribute video data associated with the point cloud.

[0736] The attribute_parameter_set(i) contains information about the i-th attribute video data associated with the point cloud.

[0737] In one embodiment, the sequence parameter set (SPS) according to this specification further includes a sps_patch_sequence_orientation_enabled_flag field, a sps_patch_inter_prediction_enabled_flag field, a sps_pixel_deinterleaving_flag field, a sps_point_local_reconstruction_enabled_flag field, a sps_remove_duplicate_point_enabled_flag field, and a byte_alignment( ) field.

[0738] The sps_patch_sequence_orientation_enabled_flag field indicates whether flexible orientation is signaled in the patch sequence data unit. For example, a value of 1 in the sps_patch_sequence_orientation_enabled_flag field indicates that flexible orientation is signaled in the patch sequence data unit, and a value of 0 indicates that flexible orientation is not signaled.

[0739] A value of 1 in the sps_patch_inter_prediction_enabled_flag field indicates that inter prediction for patch information is used, using patch information from a previously coded patch frame.

[0740] A value of 1 in the sps_pixel_deinterleaving_flag field indicates that the decoded geometry and attribute video corresponding to a single stream contains interleaved pixels from two layers. A value of 0 in the sps_pixel_deinterleaving_flag field indicates that the decoded geometry and attribute video corresponding to a single stream contains interleaved pixels only from a single layer.

[0741] A value of 1 in the sps_point_local_reconstruction_enabled_flag field indicates that the local reconstruction mode is to be used during the point cloud reconstruction process.

[0742] A value of 1 in the sps_remove_duplicate_point_enabled_flag field indicates that duplicated points are not reconstructed, where a duplicated point is a point that has the same 2D and 3D geometry coordinates as another point from a lower layer.

[0743] FIG. 39 illustrates an example of a syntax structure of a patch data group (patch_data_group()) according to an embodiment.

[0744] As described above, if the value of the vpcc_unit_type field in the V-PCC unit header indicates a patch data group, the V-PCC unit payload of FIG. 37 includes the patch data group (patch_data_group()) of FIG.

[0745] In one embodiment, the patch data group (patch_data_group()) includes a pdg_unit_type field, a patch_data_group_unit_payload (pdg_unit_type) in which information signaled varies depending on the value of the pdg_unit_type field, and a pdg_terminate_patch_data_group_flag field.

[0746] The pdg_unit_type field indicates the type of patch data group.

[0747] The pdg_terminate_patch_data_group_flag field indicates the end of the patch data group. If the value of the pdg_terminate_patch_data_group_flag field is 0, it indicates that there are more patch data group units in the patch data group. If the value of the pdg_terminate_patch_data_group_flag field is 1, it indicates that there are no more patch data group units in the patch data group and that this is the current end of the patch data group units.

[0748] FIG. 40 shows an example of the type of patch data group assigned to the pdg_unit_type field of the patch data group in FIG.

[0749] For example, a value of 0 in the pdg_unit_type field indicates a patch sequence parameter set (PDG_PSPS), 1 indicates a patch frame parameter set (PDG_PFPS), 2 indicates a patch frame geometry parameter set (PDG_PFGPS), 3 indicates a patch frame attribute parameter set (PDG_PFAPS), 4 indicates a geometry patch parameter set (PDG_GPPS), 5 indicates an attribute patch parameter set (PDG_APPS), 6 indicates a patch tile group layer unit (PDG_PTGLU), 7 indicates a prefix SEI message (PDG_PREFIX_SEI), and 8 indicates a suffix SEI message (PDG_SUFFIX_SEI).

[0750] The patch sequence parameter set (PDG_PSPS) contains sequence-level parameters, the patch frame parameter set (PDG_PFPS) contains frame-level parameters, the patch frame geometry parameter set (PDG_PFGPS) contains frame-level geometry type parameters, the patch frame attribute parameter set (PDG_PFAPS) contains frame-level attribute type parameters, the geometry patch parameter set (PDG_GPPS) contains patch-level geometry type parameters, and the attribute patch parameter set (PDG_APPS) contains patch-level attribute type parameters.

[0751] FIG. 41 illustrates an example of a syntax structure of a patch data group unit payload (patch_data_group_unit_payload(pdg_unit_type)) according to an embodiment.

[0752] If the value of the pdg_unit_type field of the patch data group in Figure 39 indicates a patch sequence parameter set (PDG_PSPS), the patch data group unit payload (patch_data_group_unit_payload()) may include a patch sequence parameter set (patch_sequence_parameter_set()).

[0753] If the value of the pdg_unit_type field indicates a geometry patch parameter set (PDG_GPPS), the patch data group unit payload (patch_data_group_unit_payload()) can contain a geometry patch parameter set (geometry_patch_parameter_set()).

[0754] If the value of the pdg_unit_type field indicates an attribute patch parameter set (PDG_APPS), the patch data group unit payload (patch_data_group_unit_payload()) can include an attribute patch parameter set (attribute_patch_parameter_set()).

[0755] If the value of the pdg_unit_type field indicates a patch frame parameter set (PDG_PFPS), the patch data group unit payload (patch_data_group_unit_payload()) can contain a patch frame parameter set (patch_frame_parameter_set()).

[0756] If the value of the pdg_unit_type field indicates a patch frame attribute parameter set (PDG_PFAPS), the patch data group unit payload (patch_data_group_unit_payload()) can contain a patch frame attribute parameter set (patch_frame_attribute_parameter_set()).

[0757] If the value of the pdg_unit_type field indicates a patch frame geometry parameter set (PDG_PFGPS), the patch data group unit payload (patch_data_group_unit_payload()) can contain a patch frame geometry parameter set (patch_frame_geometry_parameter_set()).

[0758] If the value of the pdg_unit_type field indicates a patch tile group layer unit (PDG_PTGLU), the patch data group unit payload (patch_data_group_unit_payload()) can contain a patch tile group layer unit (patch_tile_group_layer_unit()).

[0759] If the value of the pdg_unit_type field indicates a prefix SEI message (PDG_PREFIX_SEI) or a suffix SEI message (PDG_SUFFIX_SEI), the patch data group unit payload (patch_data_group_unit_payload()) can contain a sei_message().

[0760] Figure 42 shows an example of the syntax structure of a Supplemental Enhancement Information (SEI) message (sei_message()) according to an embodiment. That is, when the value of the pdg_unit_type field included in the patch data group of Figure 39 indicates a prefix SEI message (PDG_PREFIX_SEI) or a suffix SEI message (PDG_SUFFIX_SEI), the patch data group unit payload of Figure 41 includes an SEI message (sei_message()).

[0761] Each SEI message according to the embodiment consists of an SEI message header and an SEI message payload: The SEI message header includes the fields sm_payload_type_byte and sm_payload_size_byte.

[0762] The sm_payload_type_byte field is a byte of the payload type of the corresponding SEI message. For example, it is possible to identify whether the SEI message is a prefix SEI message or a suffix SEI message based on the value of the sm_payload_type_byte field.

[0763] The sm_payload_size_byte field is the payload size in bytes of the corresponding SEI message.

[0764] The SEI message in Figure 42 initializes the payload type (PayloadType) value to 0, then sets the value of the sm_payload_type_byte field in the repeat statement to the payload type value, and terminates the repeat statement if the value of the sm_payload_type_byte field is 0xFF.

[0765] In addition, the SEI message in Figure 42 initializes the payload size value to 0, then sets the value of the sm_payload_size_byte field in the repeat statement to the payload size value, and terminates the repeat statement if the value of the sm_payload_size_byte field is 0xFF.

[0766] Then, the information corresponding to the payload type and payload size set in the above two iterations is signaled by the payload of the SEI message (sei_payload(payloadType, payloadSize)).

[0767] Figure 43 shows an example of a V-PCC bitstream structure according to another embodiment of the present specification. In one embodiment, the V-PCC bitstream in Figure 43 is generated and output from the V-PCC-based point cloud video encoder of Figure 1, Figure 4, Figure 18, Figure 20, or Figure 21.

[0768] According to an embodiment, a V-PCC bitstream contains a coded point cloud sequence (CPCS) and consists of sample stream V-PCC units that carry V-PCC parameter set (VPS) data, an atlas bitstream, a 2D video encoded occupancy map bitstream, a 2D video encoded geometry bitstream, and zero or more 2D video encoded attribute bitstreams.

[0769] 43, a V-PCC bitstream can include one sample stream V-PCC header 40010 and one or more sample stream V-PCC units 40020. For ease of explanation, one or more sample stream V-PCC units 40020 may be referred to as a sample stream V-PCC payload. That is, a sample stream V-PCC payload may be referred to as a collection of sample stream V-PCC units.

[0770] Each sample stream V-PCC unit 40021 consists of V-PCC unit size information 40030 and a V-PCC unit 40040. The V-PCC unit size information 40030 indicates the size of the V-PCC unit 40040. For ease of explanation, the V-PCC unit size information 40030 may be referred to as a sample stream V-PCC unit header, and the V-PCC unit 40040 may be referred to as a sample stream V-PCC unit payload.

[0771] Each V-PCC unit 40040 consists of a V-PCC unit header 40041 and a V-PCC unit payload 40042 .

[0772] This specification distinguishes data included in a corresponding V-PCC unit payload 40042 by a V-PCC unit header 40041, and for this purpose, the V-PCC unit header 40041 includes type information indicating the type of the corresponding V-PCC unit. Each V-PCC unit payload 40042 can include at least one of geometry video data (i.e., a 2D video coded geometry bitstream), attribute video data (i.e., a 2D video coded attribute bitstream), occupancy video data (i.e., a 2D video coded occupancy map bitstream), atlas data, and a V-PCC parameter set (VPS), depending on the type information of the corresponding V-PCC unit header 40041.

[0773] The V-PCC parameter set (VPS) according to the embodiment is also called a sequence parameter set (SPS), and the terms may be used interchangeably.

[0774] The atlas data according to the embodiment may refer to data composed of characteristics (e.g., texture (patch)) and / or depth of point cloud data, and may be referred to as a patch data group.

[0775] FIG. 44 illustrates an example of data carried by a sample stream V-PCC unit in a V-PCC bitstream according to an embodiment.

[0776] The V-PCC bitstream of Figure 44 is an example including a sample stream V-PCC unit transmitting a V-PCC parameter set (VPS), a sample stream V-PCC unit transmitting atlas data (AD), a sample stream V-PCC unit transmitting occupational video data (OVD), a sample stream V-PCC unit transmitting geometry video data (GVD), and a sample stream V-PCC unit transmitting attribute video data (AVD).

[0777] According to an embodiment, each sample stream V-PCC unit includes one type of V-PCC unit: V-PCC parameter set (VPS), atlas data (AD), occupancy video data (OVD), geometry video data (GVD), or attribute video data (AVD).

[0778] FIG. 45 illustrates an example of a syntax structure of a sample stream V-PCC header included in a V-PCC bitstream according to an embodiment.

[0779] A sample stream V-PCC header() according to an embodiment may include ssvh_unit_size_precision_bytes_minus1 and ssvh_reserved_zero_5bits fields.

[0780] The ssvh_unit_size_precision_bytes_minus1 field can be set to indicate the precision, in bytes, of the ssvu_vpcc_unit_size element in all sample stream V-PCC units by adding 1 to the value of this field. The value of this field is in the range 0 to 7.

[0781] The ssvh_reserved_zero_5bits field is a reserved field for future use.

[0782] FIG. 46 illustrates an example of a syntax structure of a sample stream V-PCC unit (sample_stream_vpcc_unit()) according to an embodiment.

[0783] The content of each sample stream V-PCC unit is associated with the same access unit as the V-PCC unit contained in the sample stream V-PCC unit.

[0784] A sample stream V-PCC unit (sample_stream_vpcc_unit()) according to an embodiment may include a ssvu_vpcc_unit_size field and a vpcc_unit (ssvu_vpcc_unit_size).

[0785] The ssvu_vpcc_unit_size field corresponds to the V-PCC unit size information 40030 in Figure 43 and specifies the size in bytes of the subsequent V-PCC unit 40040. The number of bits used to indicate the ssvu_vpcc_unit_size field is (ssvh_unit_size_precision_bytes_minus1 + 1) * 8.

[0786] The vpcc_unit (ssvu_vpcc_unit_size) has a length corresponding to the value of the ssvu_vpcc_unit_size field and transmits one of the V-PCC parameter set (VPS), atlas data (AD), occupation video data (OVD), geometry video data (GVD), and attribute video data (AVD).

[0787] 47 shows an example of a syntax structure of a V-PCC unit according to an embodiment. One V-PCC unit consists of a V-PCC unit header (vpcc_unit_header()) and a V-PCC unit payload (vpcc_unit_payload()). A V-PCC unit according to an embodiment may include more data, in which case it may further include a trailing_zero_8bits field. The trailing_zero_8bits field according to an embodiment is a byte corresponding to 0x00.

[0788] Figure 48 shows an example of a syntax structure of a V-PCC unit header according to an embodiment. In one embodiment, the V-PCC unit header (vpcc_unit_header()) in Figure 48 includes a vuh_unit_type field. The vuh_unit_type field indicates the type of the corresponding V-PCC unit. The vuh_unit_type field according to the embodiment is also referred to as a vpcc_unit_type field.

[0789] FIG. 49 shows an example of a V-PCC unit type assigned to the vuh_unit_type field according to an embodiment.

[0790] Referring to Figure 49, in one embodiment, if the value of the vuh_unit_type field is 0, it indicates that the data contained in the V-PCC unit payload of the corresponding V-PCC unit is a V-PCC parameter set (VPCC_VPS); if it is 1, it indicates that it is atlas data (VPCC_AD); if it is 2, it indicates that it is occupied video data (VPCC_OVD); if it is 3, it indicates that it is geometry video data (VPCC_GVD); and if it is 4, it indicates that it is characteristic video data (VPCC_AVD).

[0791] The meaning, procedure, deletion, addition, etc. of the values ​​assigned to the vuh_unit_type field can be easily changed by those skilled in the art, and therefore the present invention is not limited to the above embodiment.

[0792] A V-PCC unit header according to an embodiment may further include a vuh_vpcc_parameter_set_id field and a vuh_atlas_id field if the vuh_unit_type field indicates attribute video data (VPCC_AVD), geometry video data (VPCC_GVD), occupation video data (VPCC_OVD), or atlas data (VPCC_AD).

[0793] The vuh_vpcc_parameter_set_id field specifies the identifier (ie, vuh_vpcc_parameter_set_id) of the active V-PCC parameter set (VPCC VPS).

[0794] The vuh_atlas_id field specifies the index of the atlas that corresponds to the current V-PCC unit.

[0795] A V-PCC unit header according to an embodiment may further include a vuh_attribute_index field, a vuh_attribute_dimension_index field, a vuh_map_index field, and a vuh_raw_video_flag field if the vuh_unit_type field indicates attribute video data (VPCC_AVD).

[0796] The vuh_attribute_index field indicates the index of the attribute video data carried in the attribute video data unit.

[0797] The vuh_attribute_dimension_index field indicates the index of the attribute dimension group carried in the attribute video data unit.

[0798] The vuh_map_index field, if present, indicates the index of the current attribute stream.

[0799] The vuh_raw_video_flag field can indicate whether or not RAW coded points are present. For example, if the value of the vuh_raw_video_flag field is 1, it can indicate that the associated attribute video data unit includes only RAW coded points. As another example, if the value of the vuh_raw_video_flag field is 0, it can indicate that the associated attribute video data unit includes RAW coded points. Note that if the vuh_raw_video_flag field does not exist, the value of the field can be inferred as 0. According to an embodiment, RAW coded points are also referred to as PCM (Pulse Code Modulation) coded points.

[0800] A V-PCC unit header according to an embodiment may further include a vuh_map_index field, a vuh_raw_video_flag field, and a vuh_reserved_zero_12bits field if the vuh_unit_type field indicates geometry video data (VPCC_GVD).

[0801] The vuh_map_index field, if present, indicates the index of the current geometry stream.

[0802] The vuh_raw_video_flag field may indicate the presence or absence of RAW coded points. For example, if the value of the vuh_raw_video_flag field is 1, it may indicate that the associated geometry video data unit includes only RAW coded points. As another example, if the value of the vuh_raw_video_flag field is 0, it may indicate that the associated geometry video data unit includes RAW coded points. If the vuh_raw_video_flag field does not exist, it may be inferred that the value of the field is 0. According to an embodiment, RAW coded points are also referred to as PCM (Pulse Code Modulation) coded points.

[0803] The vuh_reserved_zero_12bits field is a reserved field for future use.

[0804] According to an embodiment, the V-PCC unit header may further include a vuh_reserved_zero_17bits field if the vuh_unit_type field indicates dedicated video data (VPCC_OVD) or atlas data (VPCC_AD), and may further include a vuh_reserved_zero_27bits field if the vuh_unit_type field does not indicate dedicated video data (VPCC_OVD) or atlas data (VPCC_AD).

[0805] The vuh_reserved_zero_17bits field and the vuh_reserved_zero_27bits field are reserved fields for future use.

[0806] FIG. 50 shows an example of a syntax structure of a V-PCC unit payload (vpcc_unit_payload()) according to an embodiment.

[0807] The V-PCC unit payload in Figure 50 may include one of a V-PCC parameter set (vpcc_parameter_set()), an atlas sub-bitstream (atlas_sub_bitstream()), or a video sub-bitstream (video_sub_bitstream()), depending on the vuh_unit_type field value of the corresponding V-PCC unit header.

[0808] For example, if the vuh_unit_type field indicates a V-PCC parameter set (VPCC_VPS), the V-PCC unit payload includes a V-PCC parameter set (vpcc_parameter_set()) including overall coding information of the bitstream, if it indicates atlas data (VPCC_AD), it includes an atlas sub-bitstream (atlas_sub_bitstream()) transmitting the atlas data, if it indicates dedicated video data (VPCC_OVD), it includes a dedicated video sub-bitstream (video_sub_bitstream()) transmitting the dedicated video data, if it indicates geometry video data (VPCC_GVD), it includes a geometry video sub-bitstream (video_sub_bitstream()) transmitting the geometry video data, and if it indicates attribute video data (VPCC_AVD), it includes an attribute video sub-bitstream (video_sub_bitstream()) transmitting the attribute video data.

[0809] According to the embodiment, the atlas sub-bitstream is also referred to as an atlas sub-stream, the dedicated video sub-bitstream is also referred to as a dedicated video sub-stream, the geometry video sub-bitstream is also referred to as a geometry video sub-stream, and the attribute video sub-bitstream is also referred to as an attribute video sub-stream. The V-PCC unit payload according to the embodiment follows the format of a High Efficiency Video Coding (HEVC) Network Abstraction Layer (NAL) unit.

[0810] Figure 51 shows an example of an atlas substream structure according to an embodiment. In one embodiment, the atlas substream in Figure 51 follows the format of an HEVC NAL unit.

[0811] An atlas substream according to an embodiment consists of a sample stream NAL unit containing an atlas sequence parameter set (ASPS), a sample stream NAL unit containing an atlas frame parameter set (AFPS), one or more sample stream NAL units containing one or more atlas tile group information, and / or one or more sample stream NAL units containing one or more SEI messages.

[0812] The one or more SEI messages according to an embodiment may include a prefix SEI message and a suffix SEI message.

[0813] An atlas substream according to an embodiment may further include a sample stream NAL header before one or more NAL units.

[0814] FIG. 52 illustrates an example of the syntax structure of a sample stream NAL header (sample_stream_nal_header()) included in an atlas substream according to an embodiment.

[0815] The sample stream NAL header() according to the embodiment may include ssnh_unit_size_precision_bytes_minus1 and ssnh_reserved_zero_5bits fields.

[0816] The ssnh_unit_size_precision_bytes_minus1 field can indicate the precision, in bytes, of the ssnu_vpcc_unit_size element in all sample stream NAL units by adding 1 to the value of this field. The value of this field is in the range of 0 to 7.

[0817] The ssnh_reserved_zero_5bits field is a reserved field for future use.

[0818] FIG. 53 illustrates an example of a syntax structure of a sample stream NAL unit (sample_stream_nal_unit()) according to an embodiment.

[0819] A sample stream NAL unit (sample_stream_nal_unit()) according to an embodiment may include a ssnu_nal_unit_size field and a nal_unit (ssnu_nal_unit_size).

[0820] The ssnu_nal_unit_size field specifies the size of the subsequent NAL unit in bytes. The number of bits used to indicate the ssnu_nal_unit_size field is as follows: (ssnh_unit_size_precision_bytes_minus1 + 1) * 8.

[0821] The nal_unit (ssnu_nal_unit_size) has a length corresponding to the value of the ssnu_nal_unit_size field and carries one of an atlas sequence parameter set (ASPS), an atlas frame parameter set (AFPS), atlas style group information, and an SEI message. That is, each sample stream NAL unit may include an atlas sequence parameter set (ASPS), an atlas frame parameter set (AFPS), atlas style group information, and an SEI message. In the embodiment, the atlas sequence parameter set (ASPS), atlas frame parameter set (AFPS), atlas style group information, and SEI message are referred to as atlas data (or metadata for the atlas).

[0822] The SEI messages according to the embodiment may aid in processes related to decoding, reconstruction, display, or other purposes.

[0823] Each SEI message according to the embodiment consists of an SEI message header and an SEI message payload (sei_payload). The SEI message header may include payload type information (payloadType) and payload size information (payloadSize).

[0824] The payload type information (payloadType) indicates the payload type of the corresponding SEI message. For example, it is possible to identify whether the SEI message is a prefix SEI message or a suffix SEI message based on the payload type information (payloadType).

[0825] The payload size information (payloadSize) indicates the payload size of the corresponding SEI message.

[0826] FIG. 54 illustrates an example of a syntax structure of an SEI message payload (sei_payload()) according to an embodiment.

[0827] According to the embodiment, the SEI message may include a prefix SEI message or a suffix SEI message, and each SEI message payload signals information corresponding to payload type information (payloadType) and payload size information (payloadSize) in the SEI message payload (sei_payload(payloadType, payloadSize)).

[0828] The prefix SEI message according to the embodiment may include buffering_period(payloadSize) if payload type information is 0, pic_timing(payloadSize) if it is 1, filler_payload(payloadSize) if it is 2, sei_prefix_indication(payloadSize) if it is 10, and 3D_region_mapping(payloadSize) if it is 13.

[0829] The suffix SEI message according to the embodiment may include filler_payload(payloadSize) if the payload type information is 2, user_data_registered_itu_t_t35(payloadSize) if it is 3, user_data_unregistered(payloadSize) if it is 4, and decoded_pcc_hash(payloadSize) if it is 11.

[0830] In one embodiment, a V-PCC bitstream having the structure shown in Figure 24 or Figure 43 is encapsulated in the ISOBMFF file format in the file / segment encapsulation unit of Figure 1, Figure 18, Figure 20 or Figure 21.

[0831] In this case, the V-PCC stream may be transmitted over multiple tracks of the file, or over one single track.

[0832] An ISOBMFF-based file according to an embodiment can be made up of boxes and / or information also called ftyp, meta, moov, mdat, etc.

[0833] The ftyp box (file type box) can provide information related to the file type or file compatibility of the corresponding file. The receiving side can identify the corresponding file by referring to the ftyp box.

[0834] The meta box may contain a vpcg{0, 1, 2, 3} box (V-PCC Group Box).

[0835] The mdat box, also referred to as the media data box, may contain a video coded geometry bitstream, a video coded attribute bitstream, a video coded occupancy map bitstream, and / or an atlas data bitstream.

[0836] The moov box, also called a movie box, may contain metadata for the media data of the file (e.g., geometry bitstream, attribute bitstream, occupancy map bitstream, etc.). For example, it may contain information necessary for decoding and playing the media data, or may contain information about the samples of the file. The moov box can function as a container for all metadata. The moov box may be the top-layer box among the metadata-related boxes. According to an embodiment, there may be only one moov box in a file.

[0837] According to the embodiment, the box includes a track (trak) box that provides information related to the track of the corresponding file, and the track (trak) box may include a media (mdia) box that provides media information of the corresponding track and a track reference container (tref) box for connecting (reference) the corresponding track with a sample of the file corresponding to the corresponding track.

[0838] The media (mdia) box includes a media information container (minf) box that provides information about the corresponding media data, and the media information container (minf) box may include a sample table (stbl) box that provides metadata related to samples in the mdat box.

[0839] The stbl box may contain a sample description (stsd) box that provides information about the coding type used and any initialization information required for that coding type.

[0840] The stsd box may contain a sample entry for a track that stores a V-PCC bitstream according to an embodiment.

[0841] According to an embodiment, in order to store a V-PCC bitstream in a single track or multiple tracks within a file, the following definitions are made: Volumetric visual track, Volumetric visual media header, Volumetric sample entry, Volumetric samples, V-PCC track samples and sample entries.

[0842] The term V-PCC as used herein is similar to Visual Volumetric Video-based Coding (V3C) and can be referred to complementary to each other.

[0843] According to an embodiment, V-PCC represents a volumetric encoding of point cloud visual information.

[0844] That is, the minf box within the track box of the moov box may further contain a volumetric visual media header box, which contains information about a volumetric visual track that contains a volumetric visual scene.

[0845] Each volumetric visual scene may be represented by a unique volumetric visual track. An ISOBMFF file may contain multiple scenes and therefore multiple volumetric visual tracks may be present in the ISOBMFF file.

[0846] According to an embodiment, a volumetric visual track is identified by the volumetric visual media handler type 'volv' in the HandlerBox of the MediaBox.

[0847] The syntax of the volume visual media header box according to an embodiment is as follows:

[0848] Box Type: 'vvhd'

[0849] Container: MediaInformationBox

[0850] Mandatory: Yes

[0851] Quantity: Exactly one

[0852] A volumetric visual track, depending on the implementation, can use a VolumetricVisualMediaHeaderBox in a MediaInformationBox as follows:

[0853] aligned(8) class VolumetricVisualMediaHeaderBox

[0854] extends FullBox('vvhd', version = 0, 1) {

[0855] }

[0856] The version may be an integer indicating the version of this box.

[0857] A volumetric visual track according to an embodiment can use a VolumetricVisualSampleEntry as follows:

[0858] class VolumetricVisualSampleEntry(codingname)

[0859] extends SampleEntry (codingname){

[0860] unsigned int(8)

[32] compressor_name;

[0861] }

[0862] compressor_name is a name, for informative purposes. It is formatted in a fixed 32-byte field, with the first byte set to the number of bytes to be displayed, followed by that number of bytes of displayable data encoded using UTF-8, and then padding to complete 32 bytes total (including the size byte). This field can be set to 0.

[0863] The format of a volumetric visual sample according to an embodiment may be defined by a coding system.

[0864] According to an embodiment, a V-PCC unit header box may be present in the V-PCC track included in the sample entry and in all video-coded V-PCC component tracks included in the scheme information. The V-PCC unit header box may contain a V-PCC unit header for the data transmitted by each track, as follows:

[0865] aligned(8) class VPCCUnitHeaderBox

[0866] extends FullBox('vunt', version = 0, 0) {

[0867] vpcc_unit_header() unit_header;

[0868] }

[0869] That is, the VPCC unit header box may include a vpcc_unit_header(). Figures 34 and 48 show examples of the syntax structure of a V-PCC unit header (vpcc_unit_header()).

[0870] According to an embodiment, a V-PCC track sample entry may include a VPCC configuration box (VPCCConfigurationBox).

[0871] According to an embodiment, the VPCC Configuration Box (VPCCConfigurationBox) may contain a VPCC Decoder Configuration Record (VPCCDecoderConfigurationRecord) as follows:

[0872] aligned(8) class VPCCDecoderConfigurationRecord {

[0873] unsigned int(8) configurationVersion = 1;

[0874] unsigned int(3) sampleStreamSizeMinusOne;

[0875] unsigned int(5) numOfVPCCParameterSets;

[0876] for (i=0; i< numOfVPCCParameterSets; i++) {

[0877] sample_stream_vpcc_unit VPCCParameterSet;

[0878] }

[0879] unsigned int(8) numOfAtlasSetupUnits;

[0880] for (i=0; i< numOfAtlasSetupUnits; i++) {

[0881] sample_stream_vpcc_unit atlas_setupUnit;

[0882] }

[0883] }

[0884] The configurationVersion included in the VPCC DecoderConfigurationRecord indicates the version field. Incompatible changes to the record are indicated by a change in the version number.

[0885] Adding 1 to the value of sampleStreamSizeMinusOne indicates the precision, in bytes, of the ssvu_vpcc_unit_size element in all sample stream V-PCC units in either this configuration record or a V-PCC sample in the stream to which this configuration record applies.

[0886] numOfVPCCParameterSets indicates the number of VPS (V-PCC parameter sets) signaled in the VPCCDecoderConfigurationRecord.

[0887] VPCCParameterSet is a sample stream V-PCC unit (sample_stream_vpcc_unit()) instance for a V-PCC unit of VPCC_VPS type. The V-PCC unit may include a vpcc_parameter_set(). That is, the VPCCParameterSet array may include a vpcc_parameter_set(). Figure 46 shows an example of the syntax structure of a sample stream V-PCC unit (sample_stream_vpcc_unit()).

[0888] numOfAtlasSetupUnits indicates the number of setup arrays for the atlas stream signaled in the VPCCDecoderConfigurationRecord.

[0889] Atlas_setupUnit is a sample_stream_vpcc_unit() instance that contains an atlas sequence parameter set, an atlas frame parameter set, or an SEI atlas NAL unit. Figure 46 shows an example of the syntax structure of a sample stream V-PCC unit (sample_stream_vpcc_unit()).

[0890] That is, the atlas setup unit (atlas_setupUnit) array may contain atlas parameter sets that are constant for the sample entry in which the VPCCDecoderConfigurationRecord resides and for the stream referenced by the atlas_stream SEI message. According to an embodiment, the atlas setup unit may also be referred to as a setup unit for short.

[0891] According to another embodiment, the VPCC DecoderConfigurationRecord can be represented as follows:

[0892] aligned(8) class VPCCDecoderConfigurationRecord {

[0893] unsigned int(8) configurationVersion = 1;

[0894] unsigned int(3) sampleStreamSizeMinusOne;

[0895] bit(2) reserved = 1;

[0896] unsigned int(3) lengthSizeMinusOne;

[0897] unsigned int(5) numOVPCCParameterSets;

[0898] for (i=0; i< numOVPCCParameterSets; i++) {

[0899] sample_stream_vpcc_unit VPCCParameterSet;

[0900] }

[0901] unsigned int(8) numOfSetupUnitArrays;

[0902] for (j=0; j<numOfSetupUnitArrays; j++) {

[0903] bit(1) array_completeness;

[0904] bit(1) reserved = 0;

[0905] unsigned int(6) NAL_unit_type;

[0906] unsigned int(8) numNALUnits;

[0907] for (i=0; i <numNALUnits; i++) {

[0908] sample_stream_nal_unit setupUnit;

[0909] }

[0910] }

[0911] configurationVersion is a version field. Incompatible changes to the record are indicated by a change in the version number.

[0912] Adding 1 to the value of lengthSizeMinusOne indicates the precision, in bytes, of ssnu_nal_unit_size in all sample stream NAL units contained in the V-PCC samples of the VPCCDecoderConfigurationRecord or the stream to which the VPCCDecoderConfigurationRecord applies. Figure 53 shows an example of the syntax structure of a sample stream NAL unit (sample_stream_nal_unit()) that includes the ssnu_nal_unit_size field.

[0913] numOfVPCCParameterSets indicates the number of VPS (V-PCC parameter sets) signaled in the VPCCDecoderConfigurationRecord.

[0914] VPCCParameterSet is a sample stream V-PCC unit (sample_stream_vpcc_unit()) instance for a V-PCC unit of VPCC_VPS type. The V-PCC unit may include a vpcc_parameter_set(). That is, the VPCCParameterSet array may include a vpcc_parameter_set(). Figure 46 shows an example of the syntax structure of a sample stream V-PCC unit (sample_stream_vpcc_unit()).

[0915] numOfSetupUnitArrays indicates the number of atlas NAL unit arrays of the indicated type.

[0916] The repeat statement that is repeated the number of times equal to the value of numOfSetupUnitArrays may include array_completeness.

[0917] A value of 1 for array_completeness indicates that all atlas NAL units of the given type are in the following array and none are in the stream; a value of 0 for array_completeness indicates that additional atlas NAL units of the indicated type may be in the stream. The default and permitted values ​​are constrained by the sample entry name.

[0918] NAL_unit_type indicates the type of the atlas NAL unit contained in the following array. NAL_unit_type is restricted to take one of the values ​​indicating a NAL_ASPS, NAL_PREFIX_SEI, or NAL_SUFFIX_SEI atlas NAL unit.

[0919] numNALUnits indicates the number of atlas NAL units of the indicated type contained in the VPCCDecoderConfigurationRecord for the stream to which the VPCCDecoderConfigurationRecord applies, i.e., provides information about the entire stream. The SEI array shall only contain SEI messages of a 'declarative' nature. An example of such an SEI is the user-data SEI.

[0920] A setup unit is a sample stream NAL unit (sample_stream_nal_unit()) instance that contains an atlas sequence parameter set, or an atlas frame parameter set, or a declarative SEI atlas NAL unit.

[0921] According to an embodiment, the general layout of a multi-track container (also referred to as a multi-track ISOBMFF V-PCC container) of a V-PCC bitstream, where V-PCC units in a V-PCC elementary stream are mapped to individual tracks within the container file based on their types, has two types of tracks in a multi-track ISOBMFF V-PCC container: V-PCC tracks and V-PCC component tracks.

[0922] A V-PCC track according to the embodiment is a track that carries volumetric visual information within a V-PCC bitstream that includes an atlas sub-bitstream and a sequence parameter set.

[0923] According to the embodiment, a V-PCC component track is a constrained video scheme track that carries 2D video coded data for the occupancy map, geometry, and attribute sub-bitstreams of a V-PCC bitstream. In addition, a V-PCC component track can satisfy the following conditions:

[0924] a) In the sample entry, a new box is inserted into the V-PCC system that describes the role of the video stream contained in this track.

[0925] b) Track references are introduced from V-PCC tracks to V-PCC component tracks to establish the membership of the V-PCC component tracks contained in the particular point cloud represented by the V-PCC track.

[0926] c) The track-header flag is set to 0 to indicate that the track contributes to the V-PCC system but does not directly contribute to the overall layup of the movie.

[0927] Tracks belonging to the same V-PCC sequence are time-aligned. Samples and V-PCC tracks that contribute to the same point cloud frame across different video-encoded V-PCC component tracks have the same presentation time. The V-PCC atlas sequence parameter sets and atlas frame parameter sets used for samples have decoding times equal to or prior to the composition time of the point cloud frame. Furthermore, all tracks belonging to the same V-PCC sequence are time-aligned. Samples that contribute to the same point cloud frame across the different video-encoded V-PCC component tracks and the V-PCC track have the same presentation time. The V-PCC atlas sequence parameter sets and atlas frame parameter sets used for such samples have a decoding time equal to or prior to the composition time of the point cloud frame. In addition, all tracks belonging to the same V-PCC sequence have the same implied or explicit edit lists.

[0928] Note: Synchronization between elementary streams within a component track may be handled by the ISOBMFF track timing structures (stts, ctts, and cslg) or by equivalent mechanisms within a movie fragment.

[0929] Based on such a layout, a V-PCC ISOBMFF container may contain:

[0930] - A V-PCC track containing V-PCC parameter sets included in samples and sample entries that carry payloads of V-PCC units (unit type VPCC_VPS) and atlas V-PCC units (unit type VPCC_AD). The track also contains track references to other tracks that carry payloads of video compressed V-PCC units, such as unit types VPCC_OVD, VPCC_GVD, and VPCC_AVD.

[0931] - A restricted video scheme track with samples containing access units of video coded elementary streams for occupancy map data that are the payload of V-PCC units of type VPCC_OVD.

[0932] - One or more restricted video scheme tracks with samples containing access units of video coded elementary streams of geometry data that are the payload of V-PCC units of type VPCC_GVD.

[0933] - Zero or more restricted video scheme tracks with samples containing access units of a video coded elementary stream of attribute data that are the payload of a V-PCC unit of type VPCC_AVD.

[0934] The V-PCC tracks will be explained below.

[0935] According to an embodiment, the syntax structure of a V-PCC Track Sample Entry is as follows:

[0936] Sample Entry Type: 'vpc1', 'vpcg'

[0937] Container: SampleDescriptionBox

[0938] Mandatory: 'vpc1' or 'vpcg' sample entry is mandatory

[0939] Quantity: There may be one or more sample entries

[0940] V-PCC tracks use VPCCSampleEntry, which extends the volumetric visual sample entry. The sample entry type is 'vpc1' or 'vpcg'.

[0941] The V-PCC sample entry contains a V-PCC configuration box (VPCCConfigurationBox), which contains a V-PCC decoder configuration record (VPCCDecoderConfigurationRecord).

[0942] Under the 'vpc1' sample entry, all atlas sequence parameter sets, atlas frame parameter sets, or V-PCC SEIs are in the setupUnit array.

[0943] Under the 'vpcg' sample entry, the Atlas Sequence Parameter Set, Atlas Frame Parameter Set, and V-PCC SEI may be in this array or stream.

[0944] An optional BitRateBox may be present in a V-PCC Volume Sample Entry to signal the bitrate information of the V-PCC track.

[0945] Volumetric Sequences:

[0946] class VPCCConfigurationBox extends Box('vpcC') {

[0947] VPCCDecoderConfigurationRecord() VPCCConfig;

[0948] }

[0949] aligned(8) class VPCCSampleEntry() extends VolumetricVisualSampleEntry ('vpc1') {

[0950] VPCCConfigurationBox config;

[0951] VPCCUnitHeaderBox unit_header;

[0952] }

[0953] Figure 55 shows an example of a V-PCC sample entry structure according to an embodiment. In Figure 55, a V-PCC sample entry includes one V-PCC parameter set and may include an optional atlas sequence parameter set (ASPS), atlas frame parameter set (AFPS), or SEI.

[0954] A V-PCC bitstream according to an embodiment may further include a sample stream V-PCC header, a sample stream NAL header, and a V-PCC unit header box.

[0955] The V-PCC track sample format will be explained below.

[0956] Each sample in a V-PCC track corresponds to a single point cloud frame. The samples corresponding to this frame in the various component tracks have the same composition time as the V-PCC track sample. Each V-PCC sample contains one or more atlas NAL units, as follows:

[0957] aligned(8) class VPCCSample {

[0958] unsigned int PointCloudPictureLength = sample_size; / / size of samble (eg, from SampleSizeBox)

[0959] for (i=0; i <PointCloudPictureLength; ) {

[0960] sample_stream_nal_unit nalUnit

[0961] i += (VPCCDecoderConfigurationRecord.lengthSizeMinusOne+1) + nalUnit.ssnu_nal_unit_size;

[0962] }

[0963] }

[0964] aligned(8) class VPCCSample

[0965] {

[0966] unsigned int PictureLength = sample_size; / / size of samble (eg, from SampleSizeBox)

[0967] for (i = 0; i < PictureLength; ) / / Signaled until the end of the picture

[0968] {

[0969] unsigned int((VPCCDecoderConfigurationRecord.LengthSizeMinusOne + 1) * 8)

[0970] NALUnitLength;

[0971] <…>​​​​​​​​​​​​​​​​​​​​​​​​​​​ Since it does not make sense to display frames decoded from a feature, geometry, or occupancy map track without reconstructing the point cloud at the player, a restricted video scheme type can be defined for such video-coded tracks.

[0979] In the following, a restricted video scheme is described.

[0980] A V-PCC component video track is represented in the file as restricted video and can be identified by the 'pccv' value in the scheme_type field of the SchemeTypeBox in the RestrictedSchemeInfoBox of the restricted video sample entry.

[0981] There is no restriction on the video codec used to encode the quality, geometry, and occupancy map V-PCC components. Furthermore, the components can be encoded using different video codecs.

[0982] According to an embodiment, scheme information (SchemeInformationBox) is present and may include a VPCCUnitHeaderBox.

[0983] Below we will explain referencing V-PCC component tracks.

[0984] To link a V-PCC track to its component video tracks, three TrackReferenceTypeBoxes may be added to the track reference box within the track box of the V-PCC track for each component. The track reference type box contains an array of track_IDs that store the video tracks for the V-PCC track reference. The reference_type of the TrackReferenceTypeBox identifies the type of component, such as occupancy map, geometry, attribute, or occupancy map. The track reference types are as follows:

[0985] In 'pcco', the referenced track contains a video-coded occupancy map V-PCC component.

[0986] In 'pccg', the referenced track contains the video-coded geometry V-PCC component.

[0987] In 'pcca', the referenced track contains video-coded attributes V-PCC components.

[0988] The type of the V-PCC component carried by the referenced restricted video track and signaled in the track's RestrictedSchemeInfoBox is matched to the reference type of the track reference from the V-PCC track.

[0989] The following describes the single track container of V-PCC bitstream.

[0990] A single-track encapsulation of V-PCC data requires the V-PCC encoded elementary bitstream to be represented by a single-track declaration.

[0991] Single-track encapsulation of PCC data is used for simple ISOBMFF encapsulation of V-PCC encoded bitstreams. Such bitstreams are immediately stored in a single track without further processing. A V-PCC unit header data structure may be present in the bitstream. The single-track container for V-PCC data is provided to the media workflow for further processing (e.g., multi-track file generation, transcoding, DASH segmentation, etc.).

[0992] ISOBMFF files containing single-track encapsulated V-PCC data may include 'pcst' in the compatible_brands[] list of the file type box.

[0993] V-PCC elementary stream track:

[0994] Sample Entry Type: 'vpe1', 'vpeg'

[0995] Container: SampleDescriptionBox

[0996] Mandatory: A 'vpe1' or 'vpeg' sample entry is mandatory

[0997] Quantity: One or more sample entries may be present

[0998] A V-PCC elementary stream track uses volumetric visual sample entries with sample entry type 'vpe1' or 'vpeg'.

[0999] The V-PCC elementary stream sample entry includes a VPCCConfigurationBox.

[1000] Under the 'vpe1' sample entry, all atlas sequence parameter sets, atlas frame parameter sets, and SEIs may be present in the setup unit array. Under the 'vpeg' sample entry, all atlas sequence parameter sets, atlas frame parameter sets, and SEIs may be present in this array or stream.

[1001] Volumetric Sequences:

[1002] class VPCCConfigurationBox extends Box('vpcC') {

[1003] VPCCDecoderConfigurationRecord() VPCCConfig;

[1004] }

[1005] aligned(8) class VPCCElementaryStreamSampleEntry() extends VolumetricVisualSampleEntry ('vpe1') {

[1006] VPCCConfigurationBox config;

[1007] VPCCBoundingInformationBox 3D_bb;

[1008] }

[1009] The V-PCC elementary stream sample format will be described below.

[1010] A V-PCC elementary stream sample may consist of one or more V-PCC units belonging to the same presentation time. Each sample has a unique presentation time, size, and duration. A sample may, for example, be a sync sample or depend on other V-PCC elementary stream samples for decoding.

[1011] The V-PCC elementary stream sync sample will be described below.

[1012] The V-PCC elementary stream sink sample satisfies the following conditions:

[1013] - Independently decodable.

[1014] - Samples that come after the sync sample in decoding order have no decoding dependency on samples before the sync sample.

[1015] All samples that come after the sync sample in decoding order can be successfully decoded.

[1016] The following describes the V-PCC elementary stream sub-sample.

[1017] A V-PCC elementary stream sub-sample is a V-PCC unit contained within a V-PCC elementary stream sample.

[1018] A V-PCC elementary stream track contains a SubSampleInformationBox in the TrackFragmentBox or in each SampleTableBox of the MovieFragmentBoxes that lists the V-PCC elementary stream sub-samples.

[1019] The 32-bit unit header of the V-PCC unit representing the sub-sample is copied into the 32-bit codec_specific_parameters field of the sub-sample entry in the SubSampleInformationBox. The V-PCC unit type of each sub-sample can be identified by parsing the codec_specific_parameters field of the sub-sample entry in the SubSampleInformationBox.

[1020] The rendering of point cloud data will now be described.

[1021] According to an embodiment, rendering of point cloud data is performed by the renderer 10009 of FIG. 1, the point cloud renderer 19007 of FIG. 19, the renderer 20009 of FIG. 20, or the point cloud rendering unit 22004 of FIG. 22. According to an embodiment, point cloud data can be rendered in a 3D space based on metadata. A user can view all or part of the rendered result using a VR / AR display, a general display, or the like. In particular, the point cloud data is rendered using a user's viewport, or the like.

[1022] Here, the term "viewport" and "viewport area" refer to the area a user views in a point cloud video. The term "viewpoint" refers to the point a user views in a point cloud video, which is the center of the viewport area. That is, the viewport is an area centered on the viewpoint, and the size and shape of the area can be determined by the FOV (Field of View).

[1023] According to an embodiment, viewport information is information regarding the area a user is currently viewing in 3D space through a device or HMD. Accordingly, gaze analysis can be performed to determine how a user consumes a point cloud video, and for how long they gaze at which area of ​​the point cloud video. The gaze analysis may be performed at the receiving side and transmitted to the transmitting side via a feedback channel. A display device such as a VR / AR / MR device may extract a viewport area based on orientation information, the user's position / direction, a vertical or horizontal FOV supported by the device, etc. The orientation or viewport information is extracted or calculated at the receiving device. The orientation or viewport information analyzed at the receiving device may be transmitted to the transmitting device via a feedback channel.

[1024] A point cloud video decoder of a receiving device according to an embodiment uses orientation information and / or viewport information indicating the area a user is currently viewing to efficiently extract or decode media data of only a specific area, i.e., the area indicated by the orientation information and / or viewport information, from a file. A point cloud video encoder of a transmitting device according to an embodiment may use orientation information and / or viewport information fed back to the receiving device to encode media data of only a specific area, i.e., the area indicated by the orientation information and / or viewport information, and may encapsulate the encoded media data in a file and transmit it.

[1025] According to an embodiment, the file / segment encapsulation unit of the transmitting device may encapsulate all point cloud data into files / segments based on the orientation information and / or viewport information, or may encapsulate point cloud data indicated by the orientation information and / or viewport information into files / segments.

[1026] According to an embodiment, the file / segment decapsulator of the receiving device may decapsulate a file containing all point cloud data based on the orientation information and / or viewport information, or may decapsulate a file containing point cloud data indicated by the orientation information and / or viewport information.

[1027] According to an embodiment, viewport information is used in the same or similar sense as ViewInfoStruct information, view information.

[1028] According to an embodiment, the viewport information may also be referred to as information about the viewport or metadata related to the viewport information. According to an embodiment, the viewport information may include at least one of viewport information, rendering parameters, and object rendering information.

[1029] According to an embodiment, viewport-related information is generated / encoded by the metadata encoding unit 18005 of the point cloud data transmitting device of Figure 18, or the point cloud pre-processing unit 20001 and / or the video / image encoding units 21007, 21008 of the V-PCC system of Figure 21, and can be acquired / decoded by the metadata decoding unit 19002 of the point cloud data receiving device of Figure 19, or the video / image decoding units 22001, 22002 and / or the point cloud post-processing unit 22003 of the V-PCC system of Figure 22.

[1030] This specification defines metadata associated with viewport (or view) information for point cloud data, and describes embodiments for storing and signaling the metadata associated with that viewport information in a file.

[1031] This specification describes an embodiment for storing metadata related to viewport information for point cloud data that changes dynamically over time in a file.

[1032] This specification defines metadata associated with rendering parameters of point cloud data and describes examples of storing and signaling metadata associated with rendering parameters in files.

[1033] This specification describes an embodiment for storing metadata related to rendering parameters of point cloud data that changes dynamically over time in a file.

[1034] This specification defines metadata related to object rendering information associated with point cloud data in a file, and describes embodiments for storing and signaling metadata related to object rendering information in a file.

[1035] This specification describes an embodiment for storing metadata related to object rendering information of point cloud data that changes dynamically over time in a file.

[1036] FIG. 56 illustrates an example of generating a view using a virtual camera and ViewInfoStruct information according to an embodiment.

[1037] According to an embodiment, the ViewInfoStruct information may include detailed information for generating a rendered view. In particular, the ViewInfoStruct information includes 3D position information of a virtual camera for generating the view, the virtual camera's vertical and horizontal field of view (FOV), the virtual camera's direction vector, and up vector information indicating the direction above the virtual camera. The virtual camera is similar to a user's eye, i.e., the user's vision viewing a 3D region. Based on this information, a view frustum can be inferred. The view frustum refers to a region in 3D space that contains all or a portion of the point cloud data that will actually be rendered and displayed. According to an embodiment, the inferred view frustum is projected in the form of a 2D frame to generate a view (i.e., a 2D image / video frame that will actually be displayed).

[1038] The following is a syntax showing an example of information included in the ViewInfoStruct information.

[1039] aligned(8) class ViewInfoStruct(){

[1040] unsigned int(16) view_pos_x;

[1041] unsigned int(16) view_pos_y;

[1042] unsigned int(16) view_pos_z;

[1043] unsigned int(8) view_vfov;

[1044] unsigned int(8) view_hfov;

[1045] unsigned int(16) view_dir_x;

[1046] unsigned int(16) view_dir_y;

[1047] unsigned int(16) view_dir_z;

[1048] unsigned int(16) view_up_x;

[1049] unsigned int(16) view_up_y;

[1050] unsigned int(16) view_up_z;

[1051] }

[1052] view_pos_x, view_pos_y, view_pos_z indicate the x, y, z coordinate values ​​in 3D space of a virtual camera that can generate a view (e.g., a 2D image / video frame that is actually displayed).

[1053] view_vfov and view_hfov indicate the vertical FOV (vertical field of view) and horizontal FOV (horizontal field of view information) of a virtual camera that can generate a view.

[1054] view_dir_x, view_dir_y, and view_dir_z indicate x, y, and z coordinate values ​​in 3D space that indicate a direction vector that indicates the viewing direction of the virtual camera.

[1055] view_up_x, view_up_y, and view_up_z indicate x, y, and z coordinate values ​​in 3D space that indicate the up vector that indicates the direction above the virtual camera.

[1056] According to an embodiment, ViewInfoStruct() information can be sent in the form of an SEI or the like on the point cloud bitstream, as follows:

[1057] V-PCC view information box

[1058] aligned(8) class VPCCViewInfoBox extends FullBox('vpvi',0,0) {

[1059] ViewInfoStruct();

[1060] }

[1061] The static V-PCC viewport information will be explained below.

[1062] In an embodiment, when the viewport information does not change within a point cloud sequence, the sample entry of the V-PCC track or the sample entry of the V-PCC elementary stream track includes a VPCCViewInfoBox as follows:

[1063] aligned(8) class VPCCSampleEntry() extends VolumetricVisualSampleEntry ('vpc1') {

[1064] VPCCConfigurationBox config;

[1065] VPCCUnitHeaderBox unit_header;

[1066] VPCCViewInfoBox view_info;

[1067] }

[1068] The VPCCViewInfoBox provides detailed information for generating a view that renders the point cloud data associated with the atlas frames stored in the sample track, as follows:

[1069] aligned(8) class VPCCElementaryStreamSampleEntry() extends VolumetricVisualSampleEntry ('vpe1') {

[1070] VPCCConfigurationBox config;

[1071] VPCCViewInfoBox view_info;

[1072] }

[1073] VPCCViewInfoBox contains detailed information for generating the view in which the atlas frames stored in subsamples within the track, and the point cloud data associated with the video frames, are rendered and presented.

[1074] The V-PCC view information sample grouping will be described below.

[1075] According to the embodiment, the 'vpvs' grouping_type for sample grouping indicates the allocation of samples in the V-PCC track to the view information transmitted for this sample group. If a SampleToGroupBox with grouping_type 'vpvs' exists, an accompanying SampleGroupDescriptionBox with the same grouping type exists and contains the ID of this sample group.

[1076] aligned(8) class VPCCViewInfoSampleGroupDescriptionEntry() extends SampleGroupDescriptionEntry('vpvs') {

[1077] ViewInfoStruct();

[1078] }

[1079] Dynamic V-PCC view information will be explained below.

[1080] If a V-PCC track has an associated timed-metadata track with sample entry type 'dyvi', the view information defined for the point cloud stream is carried by the V-PCC track considered as dynamic, i.e., the view information can change dynamically over time.

[1081] The associated timed-metadata track contains a 'cdsc' track reference to the V-PCC carrying the atlas stream.

[1082] If a V-PCC elementary stream track has an associated timed-metadata track with sample entry type 'dyvi', the view information defined for the point cloud stream is carried by the V-PCC elementary track considered as dynamic, i.e., the view information changes dynamically over time.

[1083] The associated timed-metadata track contains a 'cdsc' track reference to the V-PCC basic sample track.

[1084] aligned(8) class DynamicViewInfoSampleEntry extends MetaDataSampleEntry('dyvi') {

[1085] VPCCViewInfoBox init_view_info;

[1086] }

[1087] The init_view_info may include view information() that generates an initial view of the point cloud data.

[1088] The sample syntax for this sample entry type 'dyvi' is as follows:

[1089] aligned(8) DynamicViewInfoSample() {

[1090] VPCCViewInfoBox view_info;

[1091] }

[1092] Each sample may contain view information () that changes over time.

[1093] According to an embodiment, a RenderingParamStruct, which is a rendering parameter for rendering, may include parameter information applicable when rendering point cloud data, such as the size of points when rendering point cloud data, the rendering type of points, how to handle overlapping points (e.g., whether to display them), etc.

[1094] aligned(8) class RenderingParamStruct(){

[1095] unsigned int(16) point_size;

[1096] unsigned int(7) point_type;

[1097] unsigned int(2) duplicated_point;

[1098] }

[1099] Point_size indicates the size of the point being rendered / displayed.

[1100] Point_type can indicate the type of point to be rendered / displayed. For example, a Point_type value of 0 indicates a cuboid, 1 a circle, and 2 a point.

[1101] Duplicated_point indicates how to handle multiple points with the same 3D x, y, and z values. For example, if the value of Duplicated_point is 0, the duplicated points are ignored. In one embodiment, if the value of Duplicated_point is 1, only one point can be randomly selected and rendered / displayed. In another embodiment, if the value of Duplicated_point is 2, the attribute values ​​of the duplicated points can be averaged and rendered / displayed based on the average value.

[1102] RenderingParamStruct() information can be sent on the point cloud bitstream, e.g., in an SEI, as follows:

[1103] V-PCC rendering parameter box

[1104] aligned(8) class VPCCRenderingParamBox extends FullBox('vprp',0,0) {

[1105] RenderingParamStruct();

[1106] }

[1107] The static V-PCC rendering parameters will be explained below.

[1108] According to an embodiment, if the rendering-related parameters do not change within the point cloud sequence, a VPCCRenderingParamBox may be included in the sample entry of the V-PCC track or the sample entry of the V-PCC elementary stream track, as follows:

[1109] aligned(8) class VPCCSampleEntry() extends VolumetricVisualSampleEntry ('vpc1') {

[1110] VPCCConfigurationBox config;

[1111] VPCCUnitHeaderBox unit_header;

[1112] VPCCRenderingParamBox rendering_param;

[1113] }

[1114] VPCCRenderingParamBox may contain detailed information about rendering parameters applicable when rendering point cloud data associated with the atlas frames stored in the samples in the track, as follows:

[1115] aligned(8) class VPCCElementaryStreamSampleEntry() extends VolumetricVisualSampleEntry ('vpe1') {

[1116] VPCCConfigurationBox config;

[1117] VPCCRenderingParamBox rendering_param;

[1118] }

[1119] VPCCRenderingParamBox may contain detailed information about rendering parameters that can be applied when rendering the atlas frames, video frames, and point cloud data stored in subsamples within a track.

[1120] The V-PCC rendering parameter sample grouping will be described below.

[1121] According to an embodiment, the 'vprp' grouping_type for sample grouping indicates the allocation of samples in a V-PCC track to rendering parameters transmitted to a sample group. If a SampleToGroupBox with grouping_type 'vpvs' exists, an accompanying SampleGroupDescriptionBox with the same grouping type may exist and contain the group ID of this sample.

[1122] aligned(8) class VPCCRenderingParamSampleGroupDescriptionEntry() extends SampleGroupDescriptionEntry('vprp') {

[1123] RenderingParamStruct();

[1124] }

[1125] The dynamic V-PCC rendering parameter information will be explained below.

[1126] If a V-PCC track has an associated timed-metadata track with sample entry type 'dyrp', the rendering parameters defined for the point cloud stream are carried by the V-PCC track considered as dynamic rendering parameters, i.e., the rendering parameters change dynamically over time.

[1127] The associated timed-metadata track contains a 'cdsc' track reference to the V-PCC track carrying the atlas stream.

[1128] If a V-PCC elementary stream track has an associated timed-metadata track with sample entry type 'dyrp', the rendering parameters defined for the point cloud stream are carried by the V-PCC elementary stream track considered as dynamic rendering parameters, i.e., the rendering parameters change dynamically over time.

[1129] The associated timed-metadata track contains a 'cdsc' track reference to the V-PCC elementary stream track.

[1130] aligned(8) class DynamicRenderingParamSampleEntry extends MetaDataSampleEntry('dyrp') {

[1131] VPCCRenderingParamBox init_rendering_param;

[1132] }

[1133] init_rendering_param may contain the initial rendering parameters for the point cloud data.

[1134] According to an embodiment, a sample syntax for a sample entry type 'dyvi' is as follows:

[1135] aligned(8) DynamicRenderingParamSample() {

[1136] VPCCRenderingParamBox rendering_param;

[1137] }

[1138] Each sample may contain rendering parameters that change over time.

[1139] FIG. 57 illustrates an example of generating a view using ObjectRenderingInfoStruct() information according to an embodiment.

[1140] According to an embodiment, the object rendering information, ObjectRenderingInfoStruct, may include detailed information applied to point cloud data when rendering the point cloud data. According to an embodiment, the ObjectRenderingInfoStruct may include information such as coordinate system information used by the point cloud data, the position of the point cloud data in 3D space based on the coordinate system, a normal vector of the point cloud data, and an up vector indicating the up direction of the point cloud data, as follows:

[1141] aligned(8) class ObjectRenderingInfoStruct(){

[1142] unsigned int(8) obj_id;

[1143] unsinged int(4) obj_coord_type;

[1144] bit(4) reserved = 1;

[1145] unsigned int(16) obj_pos_x;

[1146] unsigned int(16) obj_pos_y;

[1147] unsigned int(16) obj_pos_z;

[1148] unsigned int(16) obj_dir_x;

[1149] unsigned int(16) obj_dir_y;

[1150] unsigned int(16) obj_dir_z;

[1151] unsigned int(16) obj_up_x;

[1152] unsigned int(16) obj_up_y;

[1153] unsigned int(16) obj_up_z;

[1154] unsigned int(16) obj_scale_x;

[1155] unsigned int(16) obj_scale_y;

[1156] unsigned int(16) obj_scale_z;

[1157] }

[1158] obj_id indicates the identifier of the point cloud data.

[1159] obj_coord_type indicates the coordinate system to use when rendering the point cloud. For example, if the value of obj_coord_type is 0, the global coordinate system will be used. For example, if the value of obj_coord_type is 1, the view coordinate system (i.e. the coordinate system of the virtual camera used to generate the viewport data) will be used.

[1160] obj_pos_x, obj_pos_y, obj_pos_z indicate the x, y, z coordinate values ​​of the origin of the bounding box of the point cloud data on the coordinate system designated by obj_coord_type. According to the embodiment, they indicate the x, y, z coordinate values ​​of the center point of the bounding box of the point cloud data on the coordinate system designated by obj_coord_type.

[1161] obj_dir_x, obj_dir_y, and obj_dir_z indicate x, y, and z coordinate values ​​in 3D space that indicate the direction vector (or normal vector) that points toward the front of the point cloud data.

[1162] obj_up_x, obj_up_y, and obj_up_z indicate x, y, and z coordinate values ​​in 3D space that indicate an up vector that points upward in the point cloud data.

[1163] obj_scale_x, obj_scale_y, obj_scale_z indicate the scaling factors to apply to the x, y and z axes of the source point cloud data when rendering / displaying.

[1164] According to an embodiment, the ObjectRenderingInfoStruct() information can be transmitted on the point cloud bitstream in the form of an SEI or the like.

[1165] According to an embodiment, the V-PCC object rendering information box may include ObjectRenderingInfoStruct() information.

[1166] aligned(8) class VPCCObjectRenderingInfoBox extends FullBox('vpoi',0,0) {

[1167] ObjectRenderingInfoStruct();

[1168] }

[1169] The static V-PCC object rendering information will be described below.

[1170] According to an embodiment, if the object rendering...

Claims

1. 1. A method for encoding point cloud data, comprising the steps of: encoding the point cloud data; encapsulating the bitstream containing the encoded point cloud data in a file; the encoded point cloud data includes encoded geometry data, encoded feature data, and encoded occupancy map data; The file includes a number of tracks, each of the plurality of tracks includes a sample entry and a sample; the samples in at least one of the plurality of tracks include one of the encoded geometry data, the encoded attribute data, and the encoded occupancy map data; the file further comprises signaling information; the signaling information includes information regarding a viewport for the point cloud data; The information about the viewport includes at least information about an initial viewport or information about a dynamic viewport that changes over time; one track of the plurality of tracks is a timed metadata track; a sample entry type of the timed metadata track having a value to indicate that the timed metadata track is for information about the viewport; Information regarding the initial viewport is conveyed via a sample entry in the timed metadata track; A method according to claim 1, wherein information about the dynamic viewport is conveyed via samples of the timed metadata track.

2. The method of claim 1 , wherein the point cloud data is encoded by a video based point cloud compression (V-PCC) scheme.

3. The method of claim 1 , wherein the information about the viewport includes camera position information.

4. The method of claim 3 , wherein the information about the viewport further comprises orientation information of the camera.

5. The method of claim 3 , wherein the information about the viewport further comprises horizontal field of view (FOV) information and vertical FOV information for generating the viewport.

6. 1. An apparatus for encoding point cloud data, comprising: an encoder for encoding the point cloud data; an encapsulation unit for encapsulating a bitstream including the encoded point cloud data into a file, the encoded point cloud data includes encoded geometry data, encoded feature data, and encoded occupancy map data; The file includes a number of tracks, each of the plurality of tracks includes a sample entry and a sample; the samples in at least one of the plurality of tracks include one of the encoded geometry data, the encoded attribute data, and the encoded occupancy map data; the file further comprises signaling information; the signaling information includes information regarding a viewport for the point cloud data; The information about the viewport includes at least information about an initial viewport or information about a dynamic viewport that changes over time; one track of the plurality of tracks is a timed metadata track; a sample entry type of the timed metadata track having a value to indicate that the timed metadata track is for information about the viewport; Information regarding the initial viewport is conveyed via a sample entry in the timed metadata track; An apparatus, wherein information regarding the dynamic viewport is conveyed via samples of the timed metadata track.

7. The apparatus of claim 6 , wherein the point cloud data is encoded by a video based point cloud compression (V-PCC) scheme.

8. The apparatus of claim 6 , wherein the information about the viewport includes camera position information.

9. The apparatus of claim 8 , wherein the information about the viewport further comprises orientation information of the camera.

10. The apparatus of claim 8 , wherein the information about the viewport further comprises horizontal field of view (FOV) information and vertical FOV information for generating the viewport.

11. 1. A method for decoding point cloud data, comprising the steps of: Decapsulating the file into a bitstream containing the encoded point cloud data, the encoded point cloud data includes encoded geometry data, encoded feature data, and encoded occupancy map data; The file includes a number of tracks, each of the plurality of tracks includes a sample entry and a sample; the samples in at least one of the plurality of tracks include one of the encoded geometry data, the encoded attribute data, and the encoded occupancy map data; the file further comprises signaling information; the signaling information includes information regarding a viewport for the point cloud data; and decoding the encoded point cloud data. The information about the viewport includes at least information about an initial viewport or information about a dynamic viewport that changes over time; one track of the plurality of tracks is a timed metadata track; a sample entry type of the timed metadata track having a value to indicate that the timed metadata track is for information about the viewport; Information regarding the initial viewport is conveyed via a sample entry in the timed metadata track; A method according to claim 1, wherein information about the dynamic viewport is conveyed via samples of the timed metadata track.

12. The method of claim 11 , wherein the information about the viewport includes camera position information.

13. The method of claim 12 , wherein the information about the viewport further comprises orientation information of the camera.

14. The method of claim 12 , wherein the information about the viewport further comprises horizontal and vertical field of view (FOV) information for generating the viewport.

15. The method of claim 11 , further comprising: rendering the decoded point cloud data based on information about the viewport.

16. 1. An apparatus for decoding point cloud data, comprising: a decapsulator for decapsulating the file into a bitstream containing encoded point cloud data, the decapsulator comprising: the encoded point cloud data includes encoded geometry data, encoded feature data, and encoded occupancy map data; The file includes a number of tracks, each of the plurality of tracks includes a sample entry and a sample; the samples in at least one of the plurality of tracks include one of the encoded geometry data, the encoded attribute data, and the encoded occupancy map data; the file further comprises signaling information; a decapsulator, the signaling information including information regarding a viewport for the point cloud data; a decoder for decoding the encoded point cloud data, The information about the viewport includes at least information about an initial viewport or information about a dynamic viewport that changes over time; one track of the plurality of tracks is a timed metadata track; a sample entry type of the timed metadata track having a value to indicate that the timed metadata track is for information about the viewport; Information regarding the initial viewport is conveyed via a sample entry in the timed metadata track; An apparatus, wherein information regarding the dynamic viewport is conveyed via samples of the timed metadata track.

17. The apparatus of claim 16 , wherein the information about the viewport includes camera position information.

18. The apparatus of claim 17 , wherein the information about the viewport further comprises orientation information of the camera.

19. The apparatus of claim 17 , wherein the information about the viewport further comprises horizontal and vertical field of view (FOV) information for generating the viewport.

20. The apparatus of claim 16 , further comprising a renderer that renders the decoded point cloud data based on information about the viewport.

Citation Information

Patent Citations

  • Scalable point cloud compression with transform, and corresponding decompression

    US20170347122A1

  • Methods and apparatus for signaling viewports and regions of interest

    US20190297132A1

  • Multiple-viewpoints related metadata transmission and reception method and apparatus

    US20190313081A1

  • Point cloud compression

    WO2019055963A1

  • Three-dimensional data encoding method, three-dimensional data decoding method, three-dimensional data encoding device, and three-dimensional data decoding device

    WO2020162495A1