Point cloud data sending device, point cloud data sending method, point cloud data receiving device and point cloud data receiving method

Point cloud data is encoded through the V-PCC solution and optimized transmission of viewport-related signaling information is used to solve the problem of low point cloud data transmission efficiency and realize efficient and optimized point cloud content services.

CN114503587BActive Publication Date: 2024-11-01LG ELECTRONICS INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202080070476.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2020-03-16
Filing Date
2020-10-07
Publication Date
2024-11-01
Estimated Expiration
2040-10-07

AI Technical Summary

Technical Problem

The prior art is difficult to efficiently transmit and receive point cloud data, resulting in high latency and encoding/decoding complexity, and it is difficult to optimize the transmission of point cloud content.

Method used

Point cloud data is encoded using video-based point cloud compression (V-PCC) scheme, and data transmission is optimized through viewport-related signaling information, allowing the transmission of viewport information and initial viewing orientation in the bitstream to improve point cloud Content transmission efficiency.

Benefits of technology

It realizes efficient point cloud data transmission, reduces latency and encoding/decoding complexity, and provides optimized point cloud content services, suitable for applications such as virtual reality, augmented reality and autonomous driving.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114503587B_ABST
    Figure CN114503587B_ABST
Patent Text Reader

Abstract

The point cloud data transmission method according to the embodiment may include the step of encoding the point cloud data and the step of transmitting the bit stream including the point cloud data and signaling information. The point cloud data reception method according to the embodiment may include the step of receiving the bit stream including the point cloud data and signaling information, the step of decoding the point cloud data and the step of rendering the decoded point cloud data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] An embodiment provides a method of providing point cloud content to provide users with various services such as virtual reality (VR), augmented reality (AR), mixed reality (MR), and autonomous driving services. Background Art

[0002] A point cloud is a collection of points in a three-dimensional (3D) space. Because the number of points in a 3D space is large, it is difficult to generate point cloud data.

[0003] High throughput is required to send and receive point cloud data. Summary of the invention

[0004] Technical issues

[0005] The purpose of the present disclosure is to provide a point cloud data sending device, a point cloud data sending method, a point cloud data receiving device and a point cloud data receiving method for efficiently sending and receiving point clouds.

[0006] Another object of the present disclosure is to provide a point cloud data sending device, a point cloud data sending method, a point cloud data receiving device and a point cloud data receiving method for solving delays and encoding / decoding complexity.

[0007] Another object of the present disclosure is to provide a point cloud data sending device, a point cloud data sending method, a point cloud data receiving device and a point cloud data receiving method for providing optimized point cloud content to users by signaling viewport-related signaling of point cloud data.

[0008] Another object of the present disclosure is to provide a point cloud data sending device, a point cloud data sending method, a point cloud data receiving device and a point cloud data receiving method for providing optimized point cloud content to users by allowing viewport information, recommended viewport information and initial viewing orientation (i.e., viewport) for data processing and rendering in a V-PCC bitstream to be sent in the bitstream.

[0009] Additional advantages, purposes and features of the present disclosure will be partially described in the following description, and will partially become apparent to those of ordinary skill in the art after examining the following or can be learned from the practice of the present disclosure. The purposes and other advantages of the present disclosure can be realized and obtained by the structures specifically pointed out in the written description and claims and the drawings.

[0010] Technical Solution

[0011] To achieve these objectives and other advantages and in accordance with the purposes of the present disclosure, as embodied and broadly described herein, a method for transmitting point cloud data may include the following steps: encoding the point cloud data; and transmitting a bitstream including the point cloud data and signaling information.

[0012] According to an embodiment, the point cloud data may include geometry data, attribute data, and occupancy map data, which are encoded by a video-based point cloud compression (V-PCC) scheme.

[0013] According to an embodiment, the signaling information may include viewport-related information of a viewport determined according to the position and orientation of a camera or a user.

[0014] According to an embodiment, viewport-related information may include coordinate information in a 3-dimensional (3D) space of the camera or user, direction vector information indicating the direction the camera or user is looking, up vector information indicating the upward direction of the camera or user, and right vector information indicating the right direction of the camera or user.

[0015] According to an embodiment, the viewport-related information may include horizontal field of view (FOV) information and vertical FOV information for generating a viewport.

[0016] According to an embodiment, the point cloud data transmitting device may include an encoder for encoding point cloud data and a transmitter for transmitting a bit stream including the point cloud data and signaling information.

[0017] According to an embodiment, the point cloud data may include geometry data, attribute data, and occupancy map data encoded by a video-based point cloud compression (V-PCC) scheme.

[0018] According to an embodiment, the signaling information may include viewport-related information of a viewport determined according to the position and orientation of a camera or a user.

[0019] According to an embodiment, viewport-related information may include coordinate information in a 3-dimensional (3D) space of the camera or user, direction vector information indicating the direction the camera or user is looking, up vector information indicating the upward direction of the camera or user, and right vector information indicating the right direction of the camera or user.

[0020] According to an embodiment, the viewport-related information may include horizontal field of view (FOV) information and vertical FOV information for generating a viewport.

[0021] According to an embodiment, a point cloud data receiving method may include receiving a bit stream including point cloud data and signaling information, decoding the point cloud data, and rendering the decoded point cloud data.

[0022] According to an embodiment, the signaling information may include viewport-related information of a viewport determined according to the position and orientation of a camera or a user.

[0023] According to an embodiment, viewport-related information may include coordinate information in a 3-dimensional (3D) space of the camera or user, direction vector information indicating the direction the camera or user is looking, up vector information indicating the upward direction of the camera or user, and right vector information indicating the right direction of the camera or user.

[0024] According to an embodiment, the viewport-related information may include horizontal field of view (FOV) information and vertical FOV information for generating a viewport.

[0025] According to an embodiment, the decoded point cloud data may be rendered based on viewport related information.

[0026] According to an embodiment, a point cloud data receiving device may include: a receiver for receiving a bit stream including point cloud data and signaling information; a decoder for decoding the point cloud data; and a renderer for rendering the decoded point cloud data.

[0027] According to an embodiment, the signaling information may include viewport-related information of a viewport determined according to the position and orientation of a camera or a user.

[0028] According to an embodiment, viewport-related information may include coordinate information in a 3-dimensional (3D) space of the camera or user, direction vector information indicating the direction the camera or user is looking, up vector information indicating the upward direction of the camera or user, and right vector information indicating the right direction of the camera or user.

[0029] According to an embodiment, the viewport-related information may include horizontal field of view (FOV) information and vertical FOV information for generating a viewport.

[0030] According to an embodiment, the decoded point cloud data may be rendered based on viewport related information.

[0031] Beneficial Effects

[0032] The point cloud data transmitting method, the point cloud data transmitting device, the point cloud data receiving method, and the point cloud data receiving device according to the embodiments can provide a point cloud service with good quality.

[0033] According to the point cloud data sending method, point cloud data sending device, point cloud data receiving method and point cloud data receiving device of the implementation mode, various video encoding and decoding methods can be implemented.

[0034] The point cloud data transmitting method, the point cloud data transmitting device, the point cloud data receiving method, and the point cloud data receiving device according to the embodiments can provide general point cloud contents such as autonomous driving services.

[0035] By using the point cloud data transmitting method, point cloud data transmitting device, point cloud data receiving method and point cloud data receiving device according to the embodiment, V-PCC bit stream can be configured, and files can be transmitted, received and stored. Therefore, the best point cloud content service can be provided.

[0036] By using the point cloud data transmitting method, point cloud data transmitting device, point cloud data receiving method and point cloud data receiving device according to the embodiment, metadata for data processing and rendering in the V-PCC bitstream can be transmitted and received in the V-PCC bitstream. Therefore, the best point cloud content service can be provided.

[0037] By using the point cloud data transmission method, point cloud data transmission device, point cloud data receiving method and point cloud data receiving device according to the embodiment, a player or the like can realize spatial or partial access to point cloud objects / contents according to the user's viewport. Therefore, the point cloud bitstream can be efficiently accessed and processed according to the user's viewport.

[0038] By using the point cloud data transmitting method, point cloud data transmitting device, point cloud data receiving method and point cloud data receiving device according to the embodiment, a bounding box for partial access and / or spatial access to point cloud content and its signaling information can be provided. Therefore, the point cloud content can be accessed in various ways on the receiving side taking into account the player environment or the user environment.

[0039] The point cloud data sending method and sending device according to the embodiment can provide 3D region information about the point cloud content and 2D region related metadata in the video or atlas frame associated therewith to support spatial / partial access to the point cloud content according to the user viewport.

[0040] The point cloud data transmitting method and transmitting apparatus according to the embodiment may process signaling of 3D region information of a point cloud in a point cloud bitstream and 2D region related metadata in a video or atlas frame associated therewith.

[0041] By using the point cloud data receiving method and receiving device according to the embodiment, point cloud content can be efficiently accessed based on the storage and signaling of 3D region information about the point cloud in the point cloud bitstream and 2D region related metadata in the associated video or atlas frame.

[0042] By using the point cloud data receiving method and receiving device according to the embodiment, point cloud content can be provided based on 3D region information about the point cloud associated with the image item in the file and 2D region related information in the video or atlas frame associated therewith in consideration of the user environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this application. The accompanying drawings illustrate embodiments of the present disclosure and together with the description are used to explain the principles of the present disclosure. In the drawings:

[0044] Figure 1 An exemplary structure of a transmission / reception system for providing point cloud contents according to an embodiment is shown.

[0045] Figure 2 Illustration of the capture of point cloud data according to an embodiment.

[0046] Figure 3 Exemplary point cloud, geometry, and texture images are shown in accordance with embodiments.

[0047] Figure 4 An exemplary V-PCC encoding process according to an embodiment is shown.

[0048] Figure 5 Shown are examples of tangent planes and normal vectors of surfaces according to embodiments.

[0049] Figure 6 An exemplary bounding box of a point cloud is shown in accordance with an embodiment.

[0050] Figure 7 An example of determination of individual patch positions on an occupancy map according to an embodiment is shown.

[0051] Figure 8 Exemplary relationships between the normal axis, the tangential axis, and the bi-tangential axes are shown according to embodiments.

[0052] Fig. 9 Exemplary configurations of the minimum mode and the maximum mode of the projection mode according to the embodiment are shown.

[0053] Fig.10 An exemplary EDD code according to an implementation is shown.

[0054] Fig.11 An example of recoloring based on color values ​​of neighboring points according to an embodiment is shown.

[0055] Fig.12 An example of push-pull background fill according to an embodiment is shown.

[0056] Fig.13 An exemplary possible traversal order of a 4*4 block according to an embodiment is shown.

[0057] Fig.14 An exemplary optimal traversal order according to an implementation is shown.

[0058] Fig.15An exemplary 2D video / image encoder according to an embodiment is shown.

[0059] Fig.16 An exemplary V-PCC decoding process according to an embodiment is shown.

[0060] Fig.17 An exemplary 2D video / image decoder according to an embodiment is shown.

[0061] Fig.18 is a flowchart illustrating the operation of the transmitting device according to an embodiment of the present disclosure.

[0062] Fig.19 is a flowchart showing the operation of the receiving device according to the embodiment.

[0063] Fig. 20 An exemplary architecture for V-PCC based storage and streaming of point cloud data is shown, according to an embodiment.

[0064] Fig.21 is an exemplary block diagram of an apparatus for storing and transmitting point cloud data according to an embodiment.

[0065] Fig. 22 is an exemplary block diagram of a point cloud data receiving device according to an embodiment.

[0066] Fig.23 An exemplary structure that can be operated in conjunction with a point cloud data transmitting / receiving method / apparatus according to an embodiment is shown.

[0067] Fig.24 An exemplary structure of a V-PCC bitstream according to an embodiment is shown.

[0068] Fig.25 is a diagram showing the structure of a multi-track V-PCC file according to an embodiment.

[0069] Fig.26 is a table showing an example of track types of PCC data assigned to data_type according to an embodiment.

[0070] Fig. 27 is a table showing examples of attribute types assigned to attribute_type according to an embodiment.

[0071] Fig.28 An example of the syntax structure of profile_tier_level() according to an embodiment is shown.

[0072] Fig.29 is a table showing examples of types of PCC data in a reference track according to an embodiment.

[0073] Fig.30 is a table showing examples of attribute types according to an embodiment.

[0074] Fig.31 is a table showing examples of attribute types according to an embodiment.

[0075] Fig.32 An example of dividing a patch frame into a plurality of tiles by dividing the patch frame into one or more tile rows and one or more tile columns is shown.

[0076] Fig.33 An example of a syntax structure of each V-PCC unit according to an embodiment is shown.

[0077] Fig.34 An example of a syntax structure of a V-PCC unit header according to an embodiment is shown.

[0078] Fig.35 An example of the type of V-PCC unit assigned to the vpcc_unit_type field according to an embodiment is shown.

[0079] Fig.36 An example of a syntax structure of pcm_separate_video_data() included in a V-PCC unit header according to an embodiment is shown.

[0080] Fig.37 An example of a syntax structure of a V-PCC unit payload according to an embodiment is shown.

[0081] Fig.38 An example of a syntax structure of sequence_parameter_set() included in a V-PCC unit payload according to an embodiment is shown.

[0082] Fig.39 An example of a syntax structure of a patch data group according to an embodiment is shown.

[0083] Fig.40 is a table showing an example of the type of patch data group assigned to the pdg_unit_type field of the patch data group according to an embodiment.

[0084] Fig.41 An example of a syntax structure of a patch data group unit payload according to an embodiment is shown.

[0085] Fig.42 An example of a syntax structure of a supplemental enhancement information (SEI) message (sei_message()) according to an embodiment is shown.

[0086] Fig.43An example of a V-PCC bitstream structure according to other embodiments of the present disclosure is shown.

[0087] Fig.44 An example of data carried by a sample stream V-PCC unit in a V-PCC bitstream according to an embodiment is shown.

[0088] Fig.45 An example of a syntax structure of a sample stream V-PCC header included in a V-PCC bitstream according to an embodiment is shown.

[0089] Fig.46 An example of a syntax structure of a sample stream V-PCC unit according to an embodiment is shown.

[0090] Fig.47 An example of a syntax structure of a V-PCC unit according to an embodiment is shown.

[0091] Fig.48 An example of a syntax structure of a V-PCC unit header according to an embodiment is shown.

[0092] Fig.49 An example of a V-PCC unit type assigned to the vuh_unit_type field according to an embodiment is shown.

[0093] Fig.50 An example of a syntax structure of a V-PCC unit payload (vpcc_unit_payload()) according to an embodiment is shown.

[0094] Fig.51 is a diagram showing a structural example of an Atlas substream according to an embodiment.

[0095] Fig.52 An example of a syntax structure of a sample stream NAL header (sample_stream_nal_header()) included in an atlas substream according to an embodiment is shown.

[0096] Fig.53 An example of a syntax structure of a sample stream NAL unit according to an embodiment is shown.

[0097] Fig.54 An example of a syntax structure of a SEI message payload (sei_payload()) according to an embodiment is shown.

[0098] Fig.55 An example of a V-PCC sample entry structure according to an embodiment is shown.

[0099] Fig.56An example of generating a view using a virtual camera and ViewInfoStruct information according to an embodiment is shown.

[0100] Fig.57 An example of creating a view using ObjectRenderingInfoStruct() information according to an embodiment is shown.

[0101] Fig.58 is a diagram showing an exemplary structure for encapsulating non-timed V-PCC data according to an embodiment.

[0102] Fig.59 is a diagram showing an example in which a spatial portion of point cloud data in a viewing frustum is actually displayed on a viewport according to an embodiment.

[0103] Fig.60 An exemplary top view of a viewing cone is shown according to an embodiment.

[0104] Fig.61 An exemplary side view of a viewing cone is shown according to an embodiment.

[0105] Fig.62 An example of a method of transmitting point cloud data according to an embodiment is shown.

[0106] Fig.63 An example of a method of receiving point cloud data according to an embodiment is shown. DETAILED DESCRIPTION

[0107] Reference will now be made in detail to preferred embodiments of the present disclosure, examples of which are shown in the accompanying drawings. The detailed description given below with reference to the accompanying drawings is intended to illustrate exemplary embodiments of the present disclosure, rather than to illustrate the only embodiments that can be implemented according to the present disclosure. The following detailed description includes specific details in order to provide a thorough understanding of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure can be practiced without these specific details.

[0108] Although most of the terms used in the present disclosure are selected from common terms widely used in the art, some terms are arbitrarily selected by the applicant and their meanings are explained in detail in the following description as needed. Therefore, the present disclosure should be understood based on the intended meanings of the terms rather than their simple names or meanings.

[0109] Figure 1 An exemplary structure of a transmission / reception system for providing point cloud contents according to an embodiment is shown.

[0110] The present disclosure provides a method for providing point cloud content to provide users with various services such as virtual reality (VR), augmented reality (AR), mixed reality (MR) and autonomous driving. Point cloud data according to an embodiment represents data representing an object as points, and may be referred to as point cloud, point cloud data, point cloud video data, point cloud image data, etc.

[0111] According to an embodiment, the point cloud data sending device 10000 may include a point cloud video acquisition unit 10001, a point cloud video encoder 10002, a file / segment encapsulation module (file / segment encapsulator) 10003 and / or a transmitter (or communication module) 10004. The sending device according to the embodiment may obtain and process the point cloud video (or point cloud content) and send it. According to an embodiment, the sending device may include a fixed station, a base transceiver system (BTS), a network, an artificial intelligence (AI) device and / or system, a robot, and an AR / VR / XR device and / or a server. According to an embodiment, the sending device 10000 may include a device configured to communicate with a base station and / or other wireless devices using a radio access technology (e.g., 5G new RAT (NR), long term evolution (LTE)), a robot, a vehicle, an AR / VR / XR device, a portable device, a home appliance, an Internet of Things (IoT) device, and an AI device / server.

[0112] According to the embodiment, the point cloud video acquisition unit 10001 acquires the point cloud video by capturing, synthesizing or generating the point cloud video.

[0113] The point cloud video encoder 10002 according to the embodiment encodes the point cloud video data acquired from the point cloud video acquisition unit 10001. According to the embodiment, the point cloud video encoder 10002 may be referred to as a point cloud encoder, a point cloud data encoder, an encoder, etc. The point cloud compression encoding (encoding) according to the embodiment is not limited to the above-mentioned embodiment. The point cloud video encoder may output a bit stream including the encoded point cloud video data. The bit stream may include not only the encoded point cloud video data, but also signaling information related to the encoding of the point cloud video data.

[0114] According to an embodiment, the point cloud video encoder 10002 may support a geometry-based point cloud compression (G-PCC) coding scheme and / or a video-based point cloud compression (V-PCC) coding scheme. In addition, the point cloud video encoder 10002 may encode point clouds (referred to as point cloud data or points) and / or signaling data associated with point clouds.

[0115] The file / segment encapsulation module 10003 according to the embodiment encapsulates the point cloud data in the form of files and / or segments. The point cloud data transmission method / device according to the embodiment can transmit the point cloud data in the form of files and / or segments.

[0116] The transmitter (or communication module) 10004 according to the embodiment transmits the encoded point cloud video data in the form of a bit stream. According to the embodiment, the file or fragment can be sent to the receiving device through the network, or stored in a digital storage medium (e.g., USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc.). The transmitter according to the embodiment can communicate with the receiving device (or receiver) by wire / wireless communication through 4G, 5G, 6G and other networks. In addition, the transmitter can perform necessary data processing operations according to the network system (e.g., 4G, 5G or 6G communication network system). The transmitting device can send the encapsulated data in an on-demand manner.

[0117] According to an embodiment, the point cloud data receiving device 10005 may include a receiver 10006, a file / segment decapsulator (or a file / segment decapsulation module) 10007, a point cloud video decoder 10008, and / or a renderer 10009. According to an embodiment, the receiving device may include a device configured to communicate with a base station and / or other wireless devices using a radio access technology (e.g., 5G New RAT (NR), Long Term Evolution (LTE)), a robot, a vehicle, an AR / VR / XR device, a portable device, a home appliance, an Internet of Things (IoT) device, and an AI device / server.

[0118] According to an embodiment, the receiver 10006 receives a bit stream including point cloud video data. According to an embodiment, the receiver 10006 may send feedback information to the point cloud data transmitting device 10000.

[0119] The file / fragment decapsulation module 10007 decapsulates files and / or fragments containing point cloud data.

[0120] The point cloud video decoder 10008 decodes the received point cloud video data.

[0121] The renderer 10009 renders the decoded point cloud video data. According to an embodiment, the renderer 10009 may send feedback information obtained at the receiving side to the point cloud video decoder 10008. According to an embodiment, the point cloud video data may carry the feedback information to the receiver 10006. According to an embodiment, the feedback information received by the point cloud transmitting device may be provided to the point cloud video encoder 10002.

[0122] The arrow indicated by the dotted line in the figure represents the transmission path of the feedback information obtained by the receiving device 10005. The feedback information is information reflecting the interactivity with the user consuming the point cloud content, and includes user information (e.g., head orientation information, viewport information, etc.). Specifically, when the point cloud content is content for a service that requires interaction with the user (e.g., autonomous driving service, etc.), the feedback information may be provided to the content sender (e.g., sending device 10000) and / or the service provider. According to an embodiment, the feedback information may be used in the receiving device 10005 as well as the sending device 10000, and may not be provided.

[0123] The head orientation information according to the embodiment is information about the user's head position, orientation, angle, movement, etc. The receiving device 10005 according to the embodiment may calculate the viewport information based on the head orientation information. The viewport information may be information about the area of ​​the point cloud video that the user is watching. The viewpoint (or orientation) is the point at which the user watches the point cloud video, and may refer to the center point of the viewport area. That is, the viewport is an area centered on the viewpoint, and the size and shape of the area may be determined by the field of view (FOV). In other words, the viewport is determined according to the position and viewpoint (or orientation) of the visual camera or the user, and the point cloud data is rendered in the viewport based on the viewport information. Therefore, in addition to the head orientation information, the receiving device 10005 may also extract the viewport information based on the vertical or horizontal FOV supported by the device. In addition, the receiving device 10005 performs gaze analysis to check how the user consumes the point cloud, the area where the user gazes in the point cloud video, the gaze time, etc. According to the embodiment, the receiving device 10005 may send feedback information including the gaze analysis result to the sending device 10000. According to an embodiment, the feedback information may be obtained during the rendering and / or display process. According to an embodiment, the feedback information may be obtained by one or more sensors included in the receiving device 10005. In addition, according to an embodiment, the feedback information may be obtained by the renderer 10009 or a separate external element (or device, component, etc.). Figure 1 The dotted line in represents the process of sending the feedback information obtained by the renderer 10009. The point cloud content providing system can process (encode / decode) the point cloud data based on the feedback information. Therefore, the point cloud video decoder 10008 can perform a decoding operation based on the feedback information. The receiving device 10005 can send the feedback information to the sending device. The sending device (or the point cloud video encoder 10002) can perform an encoding operation based on the feedback information. Therefore, the point cloud content providing system can efficiently process necessary data (for example, point cloud data corresponding to the user's head position) based on the feedback information instead of processing (encoding / decoding) all point cloud data, and provide the point cloud content to the user.

[0124] According to an embodiment, the transmitting device 10000 may be referred to as an encoder, a transmitting device, a transmitter, etc., and the receiving device 10005 may be referred to as a decoder, a receiving device, a receiver, etc.

[0125] According to the implementation mode Figure 1 The point cloud data processed in the point cloud content providing system (through a series of processes of acquisition / encoding / transmission / decoding / rendering) may be referred to as point cloud content data or point cloud video data. According to an embodiment, point cloud content data may be used as a concept covering metadata or signaling information related to point cloud data.

[0126] Figure 1 The elements of the point cloud content providing system shown may be implemented by hardware, software, a processor and / or a combination thereof.

[0127] Embodiments may provide a method of providing point cloud content to provide users with various services such as virtual reality (VR), augmented reality (AR), mixed reality (MR), and autonomous driving.

[0128] In order to provide point cloud content services, a point cloud video may be first acquired. The acquired point cloud video may be sent to a receiving side through a series of processes, and the receiving side may process the received data back to the original point cloud video and render the processed point cloud video. Thus, the point cloud video may be provided to a user. An embodiment provides a method for efficiently performing this series of processes.

[0129] All processing used to provide point cloud content services (point cloud data sending method and / or point cloud data receiving method) may include acquisition processing, encoding processing, transmission processing, decoding processing, rendering processing and / or feedback processing.

[0130] According to an embodiment, the process of providing point cloud content (or point cloud data) may be referred to as a point cloud compression process. According to an embodiment, the point cloud compression process may refer to a video-based point cloud compression (V-PCC) process.

[0131] Each element of the point cloud data transmitting device and the point cloud data receiving device according to the embodiment may be hardware, software, a processor and / or a combination thereof.

[0132] The point cloud compression system may include a transmitting device and a receiving device. According to an embodiment, the transmitting device may be referred to as an encoder, a transmitting device, a transmitter, a point cloud data transmitting device, etc. According to an embodiment, the receiving device may be referred to as a decoder, a receiving device, a receiver, a point cloud data receiving device, etc. The transmitting device may output a bit stream by encoding the point cloud video, and transmit it to the receiving device in the form of a file or a stream (streaming segment) through a digital storage medium or a network. The digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD.

[0133] The sending device may include a point cloud video acquisition unit, a point cloud video encoder, a file / segment encapsulator and a sending unit (sender), such as Figure 1 The receiving device may include a receiver, a file / segment decapsulator, a point cloud video decoder and a renderer, such as Figure 1 As shown. The encoder may be referred to as a point cloud video / picture / picture / frame encoder, and the decoder may be referred to as a point cloud video / picture / picture / frame decoding device. The renderer may include a display. The renderer and / or the display may be configured as a separate device or an external component. The transmitting device and the receiving device may also include a separate internal or external module / unit / component for feedback processing. According to an embodiment, each element in the transmitting device and the receiving device may be configured by hardware, software and / or a processor.

[0134] According to an embodiment, the operation of the receiving device may be a reverse process of the operation of the transmitting device.

[0135] The point cloud video acquirer can perform the process of acquiring point cloud video by capturing, arranging or generating the process of point cloud video. In the acquisition process, data of 3D position (x, y, z) / attribute (color, reflectivity, transparency, etc.) of multiple points can be generated, such as polygon file format (PLY) (or Stanford triangle format) files. For videos with multiple frames, one or more files can be acquired. During the capture process, point cloud related metadata (e.g., capture related metadata) can be generated.

[0136] A point cloud data transmitting apparatus according to an embodiment may include an encoder configured to encode point cloud data and a transmitter configured to transmit the point cloud data or a bit stream including the point cloud data.

[0137] A point cloud data receiving apparatus according to an embodiment may include a receiver configured to receive a bit stream including point cloud data, a decoder configured to decode the point cloud data, and a renderer configured to render the point cloud data.

[0138] The method / apparatus according to the embodiment represents a point cloud data sending apparatus and / or a point cloud data receiving apparatus.

[0139] Figure 2 Illustration of the capture of point cloud data according to an embodiment.

[0140] Point cloud data (point cloud video data) according to an embodiment may be acquired by a camera, etc. Capturing techniques according to an embodiment may include, for example, facing inwards and / or facing outwards.

[0141] In facing inward according to an embodiment, one or more cameras facing inwardly toward an object of point cloud data may photograph the object from outside the object.

[0142] In outward facing according to an embodiment, one or more cameras facing outwardly toward an object of point cloud data may photograph the object. For example, according to an embodiment, there may be four cameras.

[0143] Point cloud data or point cloud content according to an embodiment may be a video or still image of an object / environment represented in various types of 3D space. According to an embodiment, the point cloud content may include video / audio / image of an object.

[0144] As a device for capturing point cloud content, a combination of a camera device capable of acquiring depth (a combination of an infrared pattern projector and an infrared camera) and an RGB camera capable of extracting color information corresponding to the depth information may be configured. Alternatively, depth information may be extracted by using LiDAR of a radar system that measures the position coordinates of a reflector by emitting laser pulses and measuring the return time. A geometric shape consisting of points in a 3D space may be extracted from the depth information, and an attribute representing the color / reflectivity of each point may be extracted from the RGB information. Point cloud content may include information about the position (x, y, z) and the color (YCbCr or RGB) or reflectivity (r) of a point. For point cloud content, an outward-facing technology for capturing an external environment and an inward-facing technology for capturing a central object may be used. In a VR / AR environment, when an object (e.g., a core object such as a character, player, thing, or actor) is configured into a point cloud content that a user can view in any direction (360 degrees), the configuration of the capture camera may be based on an inward-facing technology. When the current surrounding environment is configured into the point cloud content in a vehicle mode such as autonomous driving, the configuration of the capture camera may be based on an outward-facing technology. Since point cloud content may be captured by multiple cameras, a camera calibration process may need to be performed to configure a global coordinate system for the cameras before capturing the content.

[0145] Point cloud content can be video or still images of objects / environments existing in various types of 3D spaces.

[0146] In addition, in the point cloud content acquisition method, any point cloud video can be arranged based on the captured point cloud video. Alternatively, when a computer-generated point cloud video of a virtual space is to be provided, capture using an actual camera may not be performed. In this case, the capture process can be simply replaced by a process of generating relevant data.

[0147] The captured point cloud video may need to be post-processed to improve the quality of the content. In the video capture process, the maximum / minimum depth can be adjusted within the range provided by the camera device. Even after the adjustment, point data of unwanted areas may still exist. Therefore, post-processing to remove unwanted areas (e.g., background) or to identify connected spaces and fill spatial holes may be performed. In addition, the point cloud extracted from cameras sharing a spatial coordinate system can be integrated into a piece of content by transforming each point into a global coordinate system based on the position coordinates of each camera obtained through a calibration process. Thus, a piece of point cloud content with a wide range can be generated, or point cloud content with a high density of points can be obtained.

[0148] The point cloud video encoder 10002 may encode the input point cloud video into one or more video streams. A point cloud video may include multiple frames, each of which may correspond to a still image / picture. In the present specification, the point cloud video may include a point cloud image / frame / picture / video / audio. In addition, the term "point cloud video" may be used interchangeably with the point cloud image / frame / picture. The point cloud video encoder 10002 may perform video-based point cloud compression (V-PCC) processing. For compression and coding efficiency, the point cloud video encoder may perform a series of processes such as prediction, transformation, quantization, and entropy coding. The encoded data (encoded video / image information) may be output in the form of a bitstream. Based on the V-PCC processing, the point cloud video encoder may encode the point cloud video by dividing the point cloud video into a geometric video, an attribute video, an occupancy map video, and auxiliary information (or auxiliary data) (to be described later). The geometric video may include a geometric image, the attribute video may include an attribute image, and the occupancy map video may include an occupancy map image. The auxiliary information may include auxiliary patch information. The attribute video / image may include a texture video / image.

[0149] The file / segment encapsulator (file / segment encapsulation module) 10003 may encapsulate the encoded point cloud video data and / or metadata related to the point cloud video in the form of a file, for example. Here, the metadata related to the point cloud video may be received from the metadata processor. The metadata processor may be included in the point cloud video encoder 10002, or may be configured as a separate component / module. The file / segment encapsulator 10003 may encapsulate data in a file format such as ISOBMFF or process data in the form of DASH segments, etc. According to an embodiment, the file / segment encapsulator 10003 may include point cloud video related metadata in a file format. The point cloud video metadata may be included in boxes at various levels of the ISOBMFF file format, for example, or as data in a separate track within a file. According to an embodiment, the file / segment encapsulator 10003 may encapsulate the point cloud video related metadata into a file. The transmission processor may perform transmission processing on the point cloud video data encapsulated according to the file format. The transmission processor may be included in the transmitter 10004, or may be configured as a separate component / module. The transmission processor may process the point cloud video data according to the transmission protocol. The transmission process may include a process of transmitting via a broadcast network and a process of transmitting through a broadband. According to an embodiment, the transmission processor may receive point cloud video related metadata together with the point cloud video data from the metadata processor and perform processing of the point cloud video data for transmission.

[0150] The transmitter 10004 can transmit the encoded video / image information or data output in the form of a bit stream to the receiver 10006 of the receiving device in the form of a file or stream through a digital storage medium or a network. The digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD and SSD. The transmitter may include an element for generating a media file in a predetermined file format, and may include an element for transmission via a broadcast / communication network. The receiver may extract the bit stream and transmit the extracted bit stream to a decoding device.

[0151] The receiver 10006 can receive the point cloud video data transmitted by the point cloud video transmitting device according to the present disclosure. According to the transmission channel, the receiver can receive the point cloud video data via a broadcast network or through broadband. Alternatively, the point cloud video data can be received through a digital storage medium.

[0152] The receiving processor may process the received point cloud video data according to the transmission protocol. The receiving processor may be included in the receiver 10006, or may be configured as a separate component / module. The receiving processor may reversely perform the above-mentioned processing of the sending processor so that the processing corresponds to the transmission processing performed on the sending side. The receiving processor may transmit the acquired point cloud video data to the file / segment decapsulator 10007, and transmit the acquired point cloud video related metadata to the metadata processor (not shown). The point cloud video related metadata acquired by the receiving processor may take the form of a signaling table.

[0153] The file / fragment decapsulator (file / fragment decapsulation module) 10007 can decapsulate the point cloud video data received from the receiving processor in the form of a file. The file / fragment decapsulator 10007 can decapsulate the file according to ISOBMFF, etc., and can obtain a point cloud video bitstream or point cloud video related metadata (metadata bitstream). The acquired point cloud video bitstream can be transmitted to the point cloud video decoder 10008, and the acquired point cloud video related metadata (metadata bitstream) can be transmitted to the metadata processor (not shown). The point cloud video bitstream may include metadata (metadata bitstream). The metadata processor may be included in the point cloud video decoder 10008, or may be configured as a separate component / module. The point cloud video related metadata acquired by the file / fragment decapsulator 10007 may take the form of a box or track in the file format. When necessary, the file / fragment decapsulator 10007 may receive the metadata required for decapsulation from the metadata processor. The point cloud video related metadata may be transmitted to the point cloud video decoder 10008 and used in the point cloud video decoding process, or may be transmitted to the renderer 10009 and used in the point cloud video rendering process.

[0154] The point cloud video decoder 10008 may receive a bitstream and decode the video / image by performing operations corresponding to the operations of the point cloud video encoder. In this case, the point cloud video decoder 10008 may decode the point cloud video by dividing the point cloud video into a geometry video, an attribute video, an occupancy map video, and auxiliary information as described below. The geometry video may include a geometry image, and the attribute video may include an attribute image. The occupancy map video may include an occupancy map image. The auxiliary information may include auxiliary patch information. The attribute video / image may include a texture video / image.

[0155] The 3D geometry may be reconstructed based on the decoded geometry image, occupancy map, and auxiliary patch information, and then may be subjected to smoothing. A color point cloud image / screen may be reconstructed by assigning color values ​​to the smoothed 3D geometry based on the texture image. The renderer 10009 may render the reconstructed geometry and color point cloud image / screen. The rendered video / image may be displayed through a display (not shown). The user may view all or part of the rendering result through a VR / AR display or a typical display.

[0156] Feedback processing may include transmitting various types of feedback information that may be obtained in the rendering / display processing to a decoder on the sending side or the receiving side. Interactivity may be provided through feedback processing when consuming point cloud video. According to an embodiment, head orientation information, viewport information indicating the area currently viewed by the user, etc. may be transmitted to the sending side in the feedback processing. According to an embodiment, the user may interact with things implemented in the VR / AR / MR / autonomous driving environment. In this case, information related to the interaction may be transmitted to the sending side or the service provider during the feedback processing. According to an embodiment, the feedback processing may be skipped.

[0157] The head orientation information may represent information about the position, angle, and motion of the user's head. Based on this information, information about the area of ​​the point cloud video currently viewed by the user (ie, viewport information) may be calculated.

[0158] The viewport information may be information about the area of ​​the point cloud video that the user is currently watching. The viewport information may be used to perform gaze analysis to examine how the user consumes the point cloud video, the area of ​​the point cloud video that the user is looking at, and how long the user is looking at the area. The gaze analysis may be performed on the receiving side, and the analysis results may be transmitted to the sending side on a feedback channel. Devices such as VR / AR / MR displays may extract the viewport area based on the position / orientation of the user's head, the vertical or horizontal FOV supported by the device, and the like.

[0159] According to an embodiment, the feedback information may be not only transmitted to the sending side, but also consumed on the receiving side. That is, decoding and rendering processing on the receiving side may be performed based on the feedback information. For example, only the point cloud video of the area currently viewed by the user may be preferentially decoded and rendered based on the head orientation information and / or the viewport information.

[0160] Here, the viewport or viewport area may represent the area of ​​the point cloud video currently viewed by the user. The viewpoint is the point in the point cloud video viewed by the user, and may represent the center point of the viewport area. That is, the viewport is the area around the viewpoint, and the size and form of the area may be determined by the field of view (FOV).

[0161] The present disclosure relates to point cloud video compression as described above. For example, the methods / implementations disclosed in the present disclosure may be applied to the point cloud compression or point cloud coding (PCC) standard of the Moving Picture Experts Group (MPEG) or the next generation video / image coding standard.

[0162] As used herein, a picture / frame may generally refer to a unit representing one image in a specific time interval.

[0163] A pixel or picture element may be the smallest unit constituting a picture (or image). In addition, "sample" may be used as a term corresponding to a pixel. A sample may generally represent a pixel or a pixel value. It may represent only a pixel / pixel value of a luminance component, only a pixel / pixel value of a chrominance component, or only a pixel / pixel value of a depth component.

[0164] A unit may represent a basic unit of image processing. A unit may include at least one of a specific area of ​​a picture and information related to the area. In some cases, a unit may be used interchangeably with terms such as a block or an area or a module. In general, an M×N block may include samples (or sample arrays) or a set (or array) of transform coefficients arranged in M ​​columns and N rows.

[0165] Figure 3 Examples of point clouds, geometric images, and texture images according to embodiments are shown.

[0166] According to an embodiment, the point cloud may be input to the Figure 4 The V-PCC encoding process is performed to generate a geometric image and a texture image. According to an embodiment, the point cloud may have the same meaning as point cloud data.

[0167] like Figure 3 As shown, the left part shows a point cloud, where the point cloud object is located in a 3D space and can be represented by a bounding box or the like. Figure 3 The middle part of the figure shows a geometric image, and Figure 3 The right part of shows a texture image (non-filling image). In the present disclosure, a geometry image may be referred to as a geometry patch frame / picture or a geometry frame / picture, and a texture image may be referred to as an attribute patch frame / picture or an attribute frame / picture.

[0168] Video-based point cloud compression (V-PCC) according to an embodiment is a method for compressing 3D point cloud data based on a 2D video codec such as High Efficiency Video Coding (HEVC) or Versatile Video Coding (VVC). The data and information that can be generated in the V-PCC compression process are as follows:

[0169] Occupancy map: This is a binary map that uses a value of 0 or 1 to indicate whether data exists at a corresponding position in the 2D plane when dividing the points constituting the point cloud into patches and mapping them to a 2D plane. The occupancy map may represent a 2D array corresponding to an atlas, and the value of the occupancy map may indicate whether each sample position in the atlas corresponds to a 3D point. An atlas refers to an object that includes information about a 2D patch for each point cloud frame. For example, an atlas may include a 2D arrangement and size of the patch, the location of the corresponding 3D area within a 3D point, a projection plan, and level of detail parameters.

[0170] Patch: A collection of points that make up a point cloud, which indicates that points belonging to the same patch are adjacent to each other in 3D space and are mapped in the same direction between the 6-sided bounding box planes in the process of mapping to a 2D image.

[0171] Geometry image: This is an image in the form of a depth map that presents position information (geometry) about the individual points that make up the point cloud on a patch-by-patch basis. A geometry image may consist of one channel of pixel values. The geometry represents a collection of coordinates associated with a point cloud frame.

[0172] Texture image: This is an image that represents the color information about each point constituting the point cloud patch by patch. The texture image may consist of pixel values ​​of multiple channels (e.g., three channels of R, G, and B). Texture is included in the attribute. According to an embodiment, the texture and / or attribute may be interpreted as the same object and / or have a containment relationship.

[0173] Auxiliary patch information: This indicates the metadata required to reconstruct the point cloud using each patch. The auxiliary patch information may include information about the location, size, etc. of the patch in 2D / 3D space.

[0174] Point cloud data, such as a V-PCC component, according to an embodiment may include atlas, occupancy map, geometry, and attributes.

[0175] An atlas represents a set of 2D bounding boxes. It can be a set of patches, for example, patches projected into a rectangular frame corresponding to a 3D bounding box in 3D space, which can represent a subset of a point cloud. In this case, the patch can represent a rectangular area in the atlas corresponding to a rectangular area in a planar projection. In addition, the patch data can represent the data required to perform a transformation from 2D to 3D of the patches included in the atlas. In addition, the patch data group is also called an atlas.

[0176] Attributes can represent scalars or vectors associated with each point in the point cloud. For example, attributes can include color, reflectivity, surface normal, timestamp, material ID.

[0177] Point cloud data according to an embodiment represents PCC data according to a video-based point cloud compression (V-PCC) scheme. The point cloud data may include multiple components. For example, it may include occupancy maps, patches, geometry and / or textures.

[0178] Figure 4 An example of a point cloud video encoder according to an embodiment is shown.

[0179] Figure 4 A V-PCC encoding process for generating and compressing occupancy maps, geometry images, texture images, and auxiliary patch information is shown. Figure 4 The V-PCC coding process can be done by Figure 1 The point cloud video encoder 10002 is processed. Figure 4 Each element of may be executed by software, hardware, a processor and / or a combination thereof.

[0180] The patch generation or patch generator 40000 receives a point cloud frame (which may be in the form of a bitstream containing point cloud data). The patch generator 40000 generates a patch based on the point cloud data. In addition, patch information including information about patch generation is generated.

[0181] Patch packing or patch packer 40001 packs one or more patches. In addition, the patch packer 40001 generates an occupancy map containing information about patch packing.

[0182] The geometric image generation or geometric image generator 40002 generates a geometric image based on the point cloud data, patch information (or auxiliary information) and / or occupancy map information. The geometric image refers to data containing geometry related to the point cloud data (ie, 3D coordinate values ​​of the points), and refers to a geometric framework.

[0183] Texture image generation or texture image generator 40003 generates a texture image based on point cloud data, patches, packed patches, patch information (or auxiliary information) and / or smoothed geometry. The texture image refers to an attribute frame. That is, the texture image can be further generated based on smoothed geometry generated by smoothing based on patch information.

[0184] Smoothing or smoother 40004 can reduce or eliminate errors contained in the image data. For example, the reconstructed geometric image is smoothed based on the patch information. That is, the part that may cause errors between the data can be smoothly filtered out to generate smooth geometry.

[0185] Auxiliary patch information compression or auxiliary patch information compressor 40005 can compress auxiliary patch information related to patch information generated in patch generation. In addition, the auxiliary patch information compressed in the auxiliary patch information compressor 40005 can be sent to the multiplexer 40013. The auxiliary patch information can be used in the geometric image generator 40002.

[0186] Image filling or image fillers 40006 and 40007 can fill geometric images and texture images respectively. Filling data can be filled into geometric images and texture images.

[0187] Group dilation or group dilator 40008 can add data to a texture image in a similar manner to image padding. Auxiliary patch information can be inserted into the texture image.

[0188] Video compression or video compressors 40009, 40010 and 40011 can compress filled geometry images, filled texture images and / or occupancy maps, respectively. In other words, video compressors 40009, 40010 and 40011 can compress input geometry frames, attribute frames and / or occupancy map frames, respectively, to output a video bitstream of a geometry image, a video bitstream of a texture image, and a video bitstream of an occupancy map. Video compression can encode geometry information, texture information and occupancy information.

[0189] Entropy compression or entropy compressor 40012 can compress the occupancy map based on an entropy scheme.

[0190] Depending on the implementation, entropy compression and / or video compression may be performed on the occupancy map frame depending on whether the point cloud data is lossless and / or lossy.

[0191] Multiplexer 40013 multiplexes the compressed geometry video bitstream, the compressed texture image video bitstream, the compressed occupancy map video bitstream and the compressed auxiliary patch information bitstream from each compressor into one bitstream.

[0192] The above blocks may be omitted or may be replaced by blocks having similar or identical functions. Figure 4 Each block shown may function as at least one of a processor, software, and hardware.

[0193] According to the implementation mode Figure 4 The detailed operation of each process is described as follows.

[0194] Patch generation (40000)

[0195] The patch generation process refers to a process of dividing a point cloud into patches (mapping units) in order to map the point cloud to a 2D image. The patch generation process can be divided into three steps: normal value calculation, segmentation, and patch segmentation.

[0196] Will refer to Figure 5 The normal value calculation process is described in detail.

[0197] Figure 5 Shown are examples of tangent planes and normal vectors of surfaces according to embodiments.

[0198] exist Figure 4 The V-PCC coding process is used in the patch generator 40000 as follows Figure 5 surface.

[0199] Normal calculations related to patch generation

[0200] Each point of the point cloud has its own direction, which is represented by a 3D vector called a normal vector. Using the neighbors of each point obtained using a KD tree or the like, we can obtain Figure 5 The tangent plane and normal vector of each point constituting the surface of the point cloud are shown. The search range applied to the process of searching for neighbors can be defined by the user.

[0201] A tangent plane is a plane that passes through a point on a surface and completely includes the tangent line to a curve on the surface.

[0202] Figure 6 An exemplary bounding box of a point cloud is shown in accordance with an embodiment.

[0203] The bounding box according to an embodiment refers to a box for dividing a unit of point cloud data based on a hexahedron in a 3D space.

[0204] The method / apparatus (eg, patch generator 40000 ) according to an embodiment may use a bounding box in a process of generating a patch from point cloud data.

[0205] The bounding box can be used in the process of projecting the target object of the point cloud data onto the plane of each flat face of the hexahedron in the 3D space. The bounding box can be represented by Figure 1 The point cloud video acquisition unit 10001 and the point cloud video encoder 10002 generate and process. In addition, based on the bounding box, Figure 4 The V-PCC encoding process includes patch generation 40000, patch packing 40001, geometry image generation 40002 and texture image generation 40003.

[0206] Segments related to patch generation

[0207] Segmentation is divided into two processes: initial segmentation and refined segmentation.

[0208] The point cloud video encoder 10002 according to the embodiment projects the point onto a surface of the bounding box. Specifically, Figure 6As shown, each point constituting the point cloud is projected onto one of the six faces of a bounding box surrounding the point cloud. Initial segmentation is the process of determining one of the planar faces of the bounding box to which each point is projected.

[0209] It is the normal value corresponding to each of the six planar faces, defined as follows:

[0210] (1.0,0.0,0.0), (0.0,1.0,0.0), (0.0,0.0,1.0), (-1.0,0.0,0.0), (0.0,-1.0,0.0), (0.0,0.0,-1.0).

[0211] As shown in the following formula, the normal vector of each point obtained in the normal value calculation process is and The plane with the largest dot product value is determined as the projection plane of the corresponding point. That is, the plane whose normal vector is most similar to the direction of the normal vector of the point is determined as the projection plane of the point.

[0212]

[0213] The determined plane may be identified by a cluster index (one of 0 to 5).

[0214] Refined segmentation is a process of enhancing the projection plane of each point constituting the point cloud determined in the initial segmentation process by considering the projection plane of the neighboring points. In this process, the score normal and score smoothing can be considered together, the score normal indicating the similarity between the normal vector of each point considered when determining the projection plane in the initial segmentation process and the normal of each flat surface of the bounding box, and the score smoothing indicating the similarity between the projection plane of the current point and the projection plane of the neighboring points.

[0215] Score smoothing can be taken into account by assigning weights to the score normals. In this case, the weight values ​​can be defined by the user. The refinement segmentation can be performed repeatedly, and the number of repetitions can also be defined by the user.

[0216] Patch segmentation related to patch generation

[0217] Patch segmentation is a process of dividing the entire point cloud into patches (a collection of neighboring points) based on the projection plane information about each point constituting the point cloud obtained in the initial / refinement segmentation process. Patch segmentation may include the following steps:

[0218] Use KD tree etc. to calculate the neighboring points of each point that constitutes the point cloud. The maximum number of neighbors can be defined by the user;

[0219] ① When neighboring points are projected onto the same plane as the current point (when they have the same cluster index),

[0220] ② Extract the current point and adjacent points as a patch;

[0221] ③Calculate the geometric value of the extracted patch.

[0222] ④ Repeat operations ② to ③ until there are no unextracted points.

[0223] The occupancy map, geometry image and texture image of each patch and the size of each patch are determined through a patch segmentation process.

[0224] Figure 7 An example of determining positions of various patches on an occupancy map according to an embodiment is shown.

[0225] The point cloud video encoder 10002 according to an embodiment may perform patch packing and generate an occupancy map.

[0226] Patch packing and occupancy map generation (40001)

[0227] This is the process of determining the position of each patch in the 2D image to map the segmented patches to the 2D image. As a 2D image, the occupancy map is a binary map that uses a value of 0 or 1 to indicate whether data exists at the corresponding position. The occupancy map consists of blocks, and its resolution can be determined by the size of the block. For example, when the block is a 1*1 block, pixel-level resolution is obtained. The occupancy packing block size can be determined by the user.

[0228] The process of determining the location of each patch on the occupancy map can be configured as follows:

[0229] ① Set all positions on the occupancy map to 0;

[0230] ② Place the patch at a point (u,v) in the occupancy map plane whose horizontal coordinates are in the range (0,occupancySizeU-patch.sizeU0) and whose vertical coordinates are in the range (0,occupancySizeV-patch.sizeV0);

[0231] ③ Set the point (x, y) in the patch plane whose horizontal coordinates are in the range (0, patch.sizeU0) and whose vertical coordinates are in the range (0, patch.sizeV0) as the current point;

[0232] ④ Change the position of the point (x, y) in raster order, and if the value of the coordinate (x, y) on the patch occupancy map is 1 (there is data at the point in the patch) and the value of the coordinate (u+x, v+y) on the global occupancy map is 1 (fill the occupancy map with the previous patch), repeat operations ③ and ④. Otherwise, proceed to operation ⑥;

[0233] ⑤ Change the position of (u,v) in grating order and repeat operations ③ to ⑤;

[0234] ⑥ Determine (u, v) as the position of the patch and copy the occupancy map data about the patch to the corresponding part on the global occupancy map; and

[0235] ⑦ Repeat steps ② to ⑥ for the next patch.

[0236] occupancySizeU: Indicates the width of the occupancy map. Its unit is the occupancy packing block size.

[0237] occupancySizeV: Indicates the height of the occupancy map. Its unit is the occupancy packing block size.

[0238] patch.sizeU0: Indicates the width of the occupancy map. Its unit is the occupancy packing block size.

[0239] patch.sizeV0: indicates the height of the occupancy map. Its unit is the occupancy packing block size.

[0240] For example, Figure 7 As shown, in a box corresponding to the occupied packed size block, there exists a box corresponding to the patch having the patch size, and the point (x, y) can be located in the box.

[0241] Figure 8 Exemplary relationships between the normal axis, the tangential axis, and the bi-tangential axes are shown according to embodiments.

[0242] The point cloud video encoder 10002 according to an embodiment may generate a geometric image. The geometric image refers to image data including geometric information about the point cloud. The geometric image generation process may use three axes (normal, tangent, and bitangent) of the patch of FIG8.

[0243] Geometric Image Generation (40002)

[0244] In this process, depth values ​​of geometric images constituting each patch are determined, and the entire geometric image is generated based on the positions of the patches determined in the above patch packing process. The process of determining depth values ​​of geometric images constituting each patch can be configured as follows.

[0245] ① Calculate parameters related to the position and size of each patch. The parameters may include the following information: According to an embodiment, the position of the patch is included in the patch information.

[0246] The normal index indicating the normal axis is obtained in the previous patch generation process. The tangential axis is an axis that coincides with the horizontal axis u of the patch image among the axes perpendicular to the normal axis, and the bitangent axis is an axis that coincides with the vertical axis v of the patch image among the axes perpendicular to the normal axis. The three axes can be expressed as Figure 8 shown.

[0247] Fig. 9 Exemplary configurations of the minimum mode and the maximum mode of the projection mode according to the embodiment are shown.

[0248] The point cloud video encoder 10002 according to an embodiment may perform patch-based projection to generate a geometric image, and the projection mode according to an embodiment includes a minimum mode and a maximum mode.

[0249] The 3D space coordinates of the patch may be calculated based on a minimum size bounding box around the patch. For example, the 3D space coordinates may include the minimum tangent value of the patch (on the patch 3d shift tangent axis), the minimum bitangent value of the patch (on the patch 3d shift bitangent axis), and the minimum normal value of the patch (on the patch 3d shift normal axis).

[0250] The 2D size of a patch indicates the horizontal and vertical size of the patch when it is packed into a 2D image. The horizontal size (patch 2d size u) can be obtained as the difference between the maximum tangent value and the minimum tangent value of the bounding box, and the vertical size (patch 2d size v) can be obtained as the difference between the maximum bitangent value and the minimum bitangent value of the bounding box.

[0251] ② Determine the projection mode of the patch. The projection mode can be the minimum mode or the maximum mode. The geometric information about the patch is represented by the depth value. When the points constituting the patch are projected in the normal direction of the patch, two layers of images can be generated, an image constructed using the maximum depth value and an image constructed using the minimum depth value.

[0252] In minimum mode, when generating two-layer images d0 and d1, the minimum depth can be configured for d0, and the maximum depth within the surface thickness from the minimum depth can be configured for d1, such as Fig. 9 shown.

[0253] For example, when the point cloud is located in 2D (such as Fig. 9 As shown in the figure, there are multiple patches including multiple points. As shown in the figure, it is indicated that points marked with the same style of shadow can belong to the same patch. The figure shows the processing of projecting patches of points marked with blanks.

[0254] When a point marked with a blank is projected left / right, the depth may be incremented by 1 relative to the left side, 0, 1, 2, ..., 6, 7, 8, 9, and a number used to calculate the depth of the point may be marked on the right side.

[0255] The same projection mode may be applied to all point clouds, or different projection modes may be applied to each frame or patch according to user definition. When different projection modes are applied to each frame or patch, a projection mode that can enhance compression efficiency or minimize missing points may be adaptively selected.

[0256] ③ Calculate the depth value of each point. In minimum mode, image d0 is constructed using depth0, which is a value obtained by subtracting the minimum normal value of the patch (on the patch 3d shift normal axis) calculated in operation 1) from the minimum normal value of the patch (on the patch 3d shift normal axis) for the minimum normal value of each point. If there is another depth value in the range between depth0 and the surface thickness at the same position, the value is set to depth1. Otherwise, the value of depth0 is assigned to depth1. Image d1 is constructed using the value of depth1.

[0257] For example, when calculating the depth of a point of image d0, the minimum value (4 2 4 4 0 6 0 0 9 9 0 80) may be calculated. When calculating the depth of a point of image d1, a larger value among two or more points may be calculated. When there is only one point, its value may be calculated (4 4 4 4 6 6 6 8 9 9 8 8 9). In the process of encoding and reconstructing the points of the patch, some points may be missing (for example, in the figure, eight points are missing).

[0258] In the maximum mode, image d0 is constructed using depth0, which is a value obtained by subtracting the minimum normal value of the patch (on the patch 3d shift normal axis) calculated in operation 1) from the minimum normal value of the patch (on the patch 3d shift normal axis) for the maximum normal value of each point. If there is another depth value in the range between depth0 and the surface thickness at the same position, the value is set to depth1. Otherwise, the value of depth0 is assigned to depth1. Image d1 is constructed using the value of depth1.

[0259] For example, when calculating the depth of a point of image d0, the minimum value (4 4 4 4 6 6 6 8 9 9 8 89) may be calculated. When calculating the depth of a point of image d1, a smaller value among two or more points may be calculated. When there is only one point, its value may be calculated (4 2 4 4 5 6 0 6 9 9 0 8 0). In the process of encoding and reconstructing the points of the patch, some points may be missing (for example, in the figure, six points are missing).

[0260] The entire geometric image can be generated by placing the geometric images of the respective patches generated by the above-mentioned process onto the entire geometric image based on the patch position information determined in the patch packing process.

[0261] The layer d1 of the entire geometric image generated can be encoded using various methods. The first method (absolute d1 encoding method) encodes the depth value of the previously generated image d1. The second method (differential encoding method) encodes the difference between the depth value of the previously generated image d1 and the depth value of the image d0.

[0262] In the encoding method using depth values ​​of two layers d0 and d1 as described above, if there is another point between the two depths, geometric information about the point is lost in the encoding process, so an Enhanced-Delta-Depth (EDD) code can be used for lossless encoding.

[0263] Below, we will refer to Fig.10 Describe the EDD code in detail.

[0264] Fig.10 An exemplary EDD code according to an implementation is shown.

[0265] In some / all processes of the point cloud video encoder 10002 and / or V-PCC encoding (e.g., video compression 40009), geometric information about points can be encoded based on EOD codes.

[0266] like Fig.10 As shown, the EDD code is used to encode the binary value of the position of all points within the surface thickness range including d1. Fig.10 In the example, since there are points at the first and fourth positions on D0 and the second and third positions are empty, the points included in the second column from the left can be represented by the EDD code 0b1001 (= 9). When the EDD code is encoded and transmitted together with D0, the receiving terminal can restore the geometric information about all points without loss.

[0267] For example, when there is a point above the reference point, the value is 1. When there is no point, the value is 0. Therefore, the code can be expressed based on 4 bits.

[0268] Smooth(40004)

[0269] Smoothing is an operation used to eliminate discontinuities that may appear on patch boundaries due to image quality degradation that occurs during the compression process. Smoothing can be performed by the point cloud video encoder 10002 or the smoother 40004:

[0270] ① Reconstruct the point cloud from the geometric image. This operation can be the opposite of the geometric image generation mentioned above. For example, the reverse process of the encoding can be reconstructed;

[0271] ② Use KD tree etc. to calculate the neighboring points of each point that constitutes the reconstructed point cloud;

[0272] ③ Determine whether each point is on the patch boundary. For example, when there is a neighboring point with a different projection plane (cluster index) from the current point, it can be determined that the point is on the patch boundary;

[0273] ④ If there is a point on the patch boundary, move the point to the centroid of the neighboring points (located at the average x, y, z coordinates of the neighboring points). That is, change the geometry value. Otherwise, maintain the previous geometry value.

[0274] Fig.11 An example of recoloring based on color values ​​of neighboring points according to an embodiment is shown.

[0275] The point cloud video encoder 10002 or the texture image generator 40003 according to an embodiment may generate a texture image based on recoloring.

[0276] Texture Image Generation (40003)

[0277] Similar to the above-mentioned geometric image generation process, the texture image generation process includes generating texture images for each patch and generating the entire texture image by arranging the texture images at a determined position. However, in the operation of generating the texture image for each patch, instead of the depth value used for geometric generation, an image having color values ​​(e.g., R, G, and B values) of points constituting a point cloud corresponding to the position is generated.

[0278] When estimating the color values ​​of the individual points that make up the point cloud, the geometry obtained previously by the smoothing process can be used. In the smoothed point cloud, the positions of some points may have shifted relative to the original point cloud, so a recoloring process may be required to find a color that is appropriate for the changed position. The recoloring can be performed using the color values ​​of neighboring points. For example, Fig.11 As shown, the new color value may be calculated by considering the color value of the nearest neighboring point and the color values ​​of the neighboring points.

[0279] For example, refer to Fig.11 , in recoloring, a suitable color value of the changed position may be calculated based on an average of attribute information about the nearest original point of the point and / or an average of attribute information about the nearest original point of the point.

[0280] Similar to the geometric image generated with two layers d0 and d1, the texture image can also be generated with two layers t0 and t1.

[0281] Auxiliary patch information compression (40005)

[0282] The point cloud video encoder 10002 or the auxiliary patch information compressor 40005 according to an embodiment may compress the auxiliary patch information (auxiliary information about the point cloud).

[0283] The auxiliary patch information compressor 40005 compresses the auxiliary patch information generated in the above patch generation, patch packing and geometry generation processes. The auxiliary patch information may include the following parameters:

[0284] The index (cluster index) used to identify the projection plane (normal plane);

[0285] The 3D spatial position of the patch, i.e., the minimum tangent value of the patch (on the patch 3d shifted tangent axis), the minimum bitangent value of the patch (on the patch 3d shifted bitangent axis), and the minimum normal value of the patch (on the patch 3d shifted normal axis);

[0286] The 2D spatial position and size of the patch, i.e., the horizontal size (patch2d size u), the vertical size (patch2d size v), the minimum horizontal value (patch2d shift u), and the minimum vertical value (patch2d shift u); and

[0287] Mapping information about each block and patch, that is, candidate index (when patches are arranged in order based on 2D spatial position and size information about the patch, multiple patches can be mapped to one block in an overlapping manner. In this case, the mapped patches constitute a candidate list, and the candidate index indicates the sequential position of the patch whose data exists in the block) and a local patch index (indicating an index of a patch existing in the frame). Table 1 shows a pseudo code representing a process of matching between a block and a patch based on the candidate list and the local patch index.

[0288] The maximum number of candidate lists can be defined by the user.

[0289] [Table 1]

[0290]

[0291] Fig.12 Shown is a push-pull background fill according to an embodiment.

[0292] Image filling and group expansion (40006, 40007, 40008)

[0293] The image filler according to the embodiment may fill the space except the patch area with meaningless supplementary data based on a push-pull background filling technique.

[0294] Image padding 40006 and 40007 are processes for filling the space outside the patch area with meaningless data to improve compression efficiency. For image padding, the pixel values ​​in the columns or rows near the border of the patch can be copied to fill the empty space. Alternatively, as Fig.12 As shown, a push-pull background filling method can be used. According to this method, the empty space is filled with pixel values ​​from the low-resolution image in a process of gradually reducing the resolution of the non-filled image and increasing the resolution again.

[0295] Group expansion 40008 is a process of filling the blank space of the geometry image and texture image configured in two layers d0 / d1 and t0 / t1 respectively. In this process, the blank space of the two layers calculated by image filling is filled with the average of the values ​​at the same position.

[0296] Fig.13 An exemplary possible traversal order of a 4*4 block according to an embodiment is shown.

[0297] Occupancy map compression (40012, 40011)

[0298] The occupancy map compressor according to the embodiment can compress the previously generated occupancy map. Specifically, two methods can be used, namely: video compression for lossy compression and entropy compression for lossless compression. Video compression is described below.

[0299] Entropy compression can be performed by the following operations.

[0300] ① If the block constituting the occupancy map is fully occupied, encode 1 and repeat the same operation for the next block of the occupancy map. Otherwise, encode 0 and perform operations 2) to 5).

[0301] ② Determine the optimal traversal order for performing run-length encoding on the occupied pixels of the block. Fig.13 Four possible traversal orders of a 4*4 block are shown.

[0302] Fig.14 An exemplary optimal traversal order according to an implementation is shown.

[0303] As described above, the entropy compressor according to the embodiment can be based on Fig.14 The traversal order scheme shown encodes the blocks (Encoding).

[0304] For example, the best traversal order with the smallest number of runs is selected from the possible traversal orders, and its index is encoded. Fig.13 In the case of the third traversal order in . In the case shown, the number of runs can be minimized to 2, so the third traversal order can be selected as the optimal traversal order.

[0305] ③ Encode the number of runs. Fig.14 In the example of , there are two runs, so 2 is encoded.

[0306] ④ Encode the occupation of the first run. Fig.14 In the example of , 0 is encoded because the first run corresponds to unoccupied pixels.

[0307] ⑤ Encode the length of each run (as many as the number of runs). Fig.14In the example, the lengths 6 and 10 of the first run and the second run are encoded respectively.

[0308] Video compression (40009, 40010, 40011)

[0309] The video compressors 40009, 40010, 40011 according to the embodiment encode the sequences of geometry images, texture images, occupancy map images, etc. generated in the above operations using a 2D video codec such as HEVC or VVC.

[0310] Fig.15 An exemplary 2D video / image encoder according to an embodiment is shown. According to an embodiment, the 2D video / image encoder may be referred to as an encoding device.

[0311] Indicates an embodiment in which the above-mentioned video compressors 40009, 40010 and 40011 are applied Fig.15 2D video / image encoder 15000 is a schematic block diagram of a 2D video / image encoder 15000 configured to encode a video / image signal. 2D video / image encoder 15000 may be included in the above-mentioned point cloud video encoder 10002, or may be configured as an internal / external component. Fig.15 Each component of may correspond to software, hardware, a processor and / or a combination thereof.

[0312] Here, the input image may include one of the above-mentioned geometric image, texture image (attribute image) and occupancy map image. Fig.15 When the 2D video / image encoder 15000 is applied to the video compressor 40009, the image input to the 2D video / image encoder 15000 is a filled geometric image, and the bit stream output from the 2D video / image encoder 15000 is a bit stream of a compressed geometric image. Fig.15 When the 2D video / image encoder 15000 is applied to the video compressor 40010, the image input to the 2D video / image encoder 15000 is a padded texture image, and the bit stream output from the 2D video / image encoder 15000 is a bit stream of the compressed texture image. Fig.15 When the 2D video / image encoder is applied to the video compressor 40011, the image input to the 2D video / image encoder 15000 is an occupancy map image, and the bit stream output from the 2D video / image encoder 15000 is a bit stream of the compressed occupancy map image.

[0313] The inter-frame predictor 15090 and the intra-frame predictor 15100 may be collectively referred to as a predictor. That is, the predictor may include the inter-frame predictor 15090 and the intra-frame predictor 15100. The transformer 15030, the quantizer 15040, the inverse quantizer 15050, and the inverse transformer 15060 may be collectively referred to as a residual processor. The residual processor may also include a subtractor 15020. According to an embodiment, Fig.15 The image segmenter 15010, the subtractor 15020, the transformer 15030, the quantizer 15040, the inverse quantizer 15050, the inverse transformer 15060, the adder 15200, the filter 15070, the inter-frame predictor 15090, the intra-frame predictor 15100 and the entropy encoder 15110 may be configured by one hardware component (e.g., an encoder or a processor). In addition, the memory 15080 may include a decoded picture buffer (DPB) and may be configured by a digital storage medium.

[0314] The image splitter 15010 may split the image (or picture or frame) input to the encoder 15000 into one or more processing units. For example, the processing unit may be referred to as a coding unit (CU). In this case, the CU may be recursively split from a coding tree unit (CTU) or a maximum coding unit (LCU) according to a quadtree binary tree (QTBT) structure. For example, a CU may be split into multiple CUs of a lower depth based on a quadtree structure and / or a binary tree structure. In this case, for example, a quadtree structure may be applied first, and a binary tree structure may be applied later. Alternatively, a binary tree structure may be applied first. The encoding process according to the present disclosure may be performed based on a final CU that is no longer split. In this case, based on encoding efficiency according to the characteristics of the image, the LCU may be used as the final CU. If necessary, the CU may be recursively split into CUs of a lower depth, and a CU of an optimal size may be used as the final CU. Here, the encoding process may include prediction, transformation, and reconstruction (to be described later). As another example, the processing unit may also include a prediction unit (PU) or a transform unit (TU). In this case, the PU and the TU may be partitioned or divided from the above-mentioned final CU. The PU may be a unit of sample prediction, and the TU may be a unit of deriving a transform coefficient and / or a unit of deriving a residual signal from the transform coefficient.

[0315] The term "unit" may be used interchangeably with terms such as block or region or module. In general, an M×N block may represent a set of samples or transform coefficients arranged in M ​​columns and N rows. A sample may generally represent a pixel or a pixel value, and may indicate only a pixel / pixel value of a luma component, or only a pixel / pixel value of a chroma component. "Sample" may be used as a term corresponding to a pixel or a picture element in one picture (or image).

[0316] The subtractor 15020 of the encoding device 15000 may generate a residual signal (residual block or residual sample array) by subtracting a prediction signal (prediction block or prediction sample array) output from the inter predictor 15090 or the intra predictor 15100 from the input image signal (original block or original sample array), and the generated residual signal is sent to the transformer 15030. In this case, as shown in the figure, the unit that subtracts the prediction signal (prediction block or prediction sample array) from the input image signal (original block or original sample array) in the encoding device 15000 may be referred to as a subtractor 15020. The predictor may perform prediction on a processing target block (hereinafter referred to as a current block) and generate a prediction block including prediction samples of the current block. The predictor may determine whether to apply intra prediction or inter prediction based on the current block or CU. As will be described later in the description of each prediction mode, the predictor may generate various types of information (e.g., prediction mode information) about prediction and transmit the generated information to the entropy encoder 15110. Information about the prediction may be encoded by the entropy encoder 15110 and output in the form of a bitstream.

[0317] The intra-frame predictor 15100 of the predictor can predict the current block with reference to samples in the current picture. Depending on the prediction mode, the sample may be adjacent to or far away from the current block. Under intra-frame prediction, the prediction mode may include multiple non-directional modes and multiple directional modes. The non-directional mode may include, for example, a DC mode and a planar mode. Depending on the fineness of the prediction direction, the directional mode may include, for example, 33 directional prediction modes or 65 directional prediction modes. However, this is only an example, and more or fewer directional prediction modes may be used depending on the settings. The intra-frame predictor 15100 may determine the prediction mode to be applied to the current block based on the prediction mode applied to the neighboring blocks.

[0318] The inter-frame predictor 15090 of the predictor may derive the prediction block of the current block based on the reference block (reference sample array) specified by the motion vector on the reference picture. In this case, in order to reduce the amount of motion information sent in the inter-frame prediction mode, the motion information may be predicted based on each block, sub-block or sample based on the correlation of the motion information between the neighboring blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may also include information about the inter-frame prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter-frame prediction, the neighboring blocks may include spatial neighboring blocks present in the current picture and temporal neighboring blocks present in the reference picture. The reference picture including the reference block may be the same as or different from the reference picture including the temporal neighboring block. The temporal neighboring block may be referred to as a collocated reference block or collocated CU (colCU), and the reference picture including the temporal neighboring block may be referred to as a collocated picture (colPic). For example, the inter-frame predictor 15090 may configure a motion information candidate list based on the neighboring blocks and generate information indicating the candidate for deriving the motion vector and / or reference picture index of the current block. Inter-frame prediction may be performed based on various prediction modes. For example, in skip mode and merge mode, the inter-frame predictor 15090 may use motion information about a neighboring block as motion information about a current block. In skip mode, unlike merge mode, a residual signal may not be sent. In motion vector prediction (MVP) mode, the motion vector of a neighboring block may be used as a motion vector predictor, and a motion vector difference may be signaled to indicate the motion vector of the current block.

[0319] The prediction signal generated by the inter-frame predictor 15090 or the intra-frame predictor 15100 may be used to generate a reconstructed signal or to generate a residual signal.

[0320] The transformer 15030 may generate transform coefficients by applying a transform technique to the residual signal. For example, the transform technique may include at least one of a discrete cosine transform (DCT), a discrete sine transform (DST), a Karhunen–Loève transform (KLT), a graph-based transform (GBT), or a conditional nonlinear transform (CNT). Here, GBT refers to a transform obtained from a graph depicting the relationship between pixels. CNT refers to a transform obtained based on a prediction signal generated based on all previously reconstructed pixels. In addition, the transform operation may be applied to square pixel blocks of the same size, or may be applied to blocks of variable sizes other than square.

[0321] The quantizer 15040 may quantize the transform coefficient and send it to the entropy encoder 15110. The entropy encoder 15110 may encode the quantized signal (information about the quantized transform coefficient) and output a bit stream of the encoded signal. The information about the quantized transform coefficient may be referred to as residual information. The quantizer 15040 may rearrange the quantized transform coefficient in the form of a block in the form of a one-dimensional vector based on a coefficient scanning order, and generate information about the quantized transform coefficient based on the quantized transform coefficient in the form of a one-dimensional vector.

[0322] The entropy encoder 15110 may employ various encoding techniques such as, for example, exponential Golomb, context adaptive variable length coding (CAVLC), and context adaptive binary arithmetic coding (CABAC). The entropy encoder 15110 may encode information required for video / image reconstruction (e.g., values ​​of syntax elements) together with or separately from the quantized transform coefficients. The encoded information (e.g., encoded video / image information) may be transmitted or stored in the form of a bitstream based on a network abstraction layer (NAL) unit.

[0323] The bitstream may be sent via a network or may be stored in a digital storage medium. Here, the network may include a broadcast network and / or a communication network, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD. A transmitter (not shown) that sends a signal output from the entropy encoder 15110 and / or a storage unit (not shown) that stores the signal may be configured as an internal / external element of the encoder 15000. Alternatively, the transmitter may be included in the entropy encoder 15110.

[0324] The quantized transform coefficients output from the quantizer 15040 can be used to generate a prediction signal. For example, inverse quantization and inverse transformation can be applied to the quantized transform coefficients by the inverse quantizer 15050 and the inverse transformer 15060 to reconstruct the residual signal (residual block or residual sample). The adder 15200 adds the reconstructed residual signal to the prediction signal output from the inter-frame predictor 15090 or the intra-frame predictor 15100. Thus, a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) can be generated. When there is no residual signal of the processing target block as in the case of applying the skip mode, the prediction block can be used as a reconstructed block. The adder 15200 can be referred to as a reconstructor or a reconstructed block generator. As described below, the generated reconstructed signal can be used for intra-frame prediction of the next processing target block in the current picture, or can be used for inter-frame prediction of the next picture by filtering.

[0325] The filter 15070 can improve the subjective / objective image quality by applying filtering to the reconstructed signal output from the adder 15200. For example, the filter 15070 can generate a modified reconstructed picture by applying various filtering techniques to the reconstructed picture, and the modified reconstructed picture can be stored in the memory 15080 (specifically, the DPB of the memory 15080). For example, various filtering techniques may include deblocking filtering, sample adaptive offset, adaptive loop filtering, and bilateral filtering. As described below in the description of filtering techniques, the filter 15070 can generate various types of information about filtering and transmit the generated information to the entropy encoder 15110. The information about filtering can be encoded by the entropy encoder 15110 and output in the form of a bitstream.

[0326] The modified reconstructed picture stored in the memory 15080 may be used as a reference picture by the inter-frame predictor 15090. Therefore, when the inter-frame prediction is applied, the encoder may avoid prediction mismatch between the encoder 15000 and the decoder and improve encoding efficiency.

[0327] The DPB of the memory 15080 may store the modified reconstructed picture for use as a reference picture by the inter-frame predictor 15090. The memory 15080 may store motion information about a block from which motion information in the current picture is derived (or encoded) and / or motion information about a block that has been reconstructed in the picture. The stored motion information may be transmitted to the inter-frame predictor 15090 to be used as motion information about a spatially neighboring block or motion information about a temporally neighboring block. The memory 15080 may store reconstructed samples of a reconstructed block in the current picture and transmit the reconstructed samples to the intra-frame predictor 15100.

[0328] At least one of the above prediction, transformation and quantization processes may be skipped. For example, for a block to which pulse code modulation (PCM) is applied, the prediction, transformation and quantization processes may be skipped, and the value of the original sample may be encoded and output in the form of a bitstream.

[0329] Fig.16 An exemplary V-PCC decoding process according to an embodiment is shown.

[0330] The V-PCC decoding process or the V-PCC decoder may follow Figure 4 The reverse process of the V-PCC encoding process (or encoder). Fig.16 Each component of may correspond to software, hardware, a processor and / or a combination thereof.

[0331] The demultiplexer 16000 demultiplexes the compressed bit stream to output a compressed texture image, a compressed geometry image, a compressed occupancy map, and compressed auxiliary patch information respectively.

[0332] The video decompression or video decompressor 16001, 16002 decompresses each of the compressed texture image and the compressed geometry image.

[0333] Occupancy map decompression or occupancy map decompressor 16003 decompresses the compressed occupancy map image.

[0334] The auxiliary patch information decompression or auxiliary patch information decompressor 16004 decompresses the compressed auxiliary patch information.

[0335] Geometry reconstruction or geometry reconstructor 16005 restores (reconstructs) geometry information based on the decompressed geometry image, the decompressed occupancy map and / or the decompressed auxiliary patch information. For example, geometry that changed during the encoding process can be reconstructed.

[0336] Smoothing or smoother 16006 can apply smoothing to the reconstructed geometry. For example, a smoothing filter can be applied.

[0337] Texture reconstruction or texture reconstructor 16007 reconstructs textures from decompressed texture images and / or smoothed geometry.

[0338] Color smoothing or color smoother 16008 smoothes color values ​​from the reconstructed texture. For example, a smoothing filter may be applied.

[0339] As a result, reconstructed point cloud data can be generated.

[0340] Fig.16 The decoding process of V-PCC is shown to reconstruct a point cloud by decompressing (decoding) the compressed occupancy map, geometry image, texture image, and auxiliary patch information. Fig.16 Each unit shown may be used as at least one of a processor, software, and hardware. Fig.16 The detailed operation of each unit is described as follows.

[0341] Video decompression (16001, 16002)

[0342] Video decompression is the inverse process of the above-mentioned video compression. It is a process of decoding the bit stream of the geometry image, the bit stream of the compressed texture image and / or the bit stream of the compressed occupancy map image generated in the above-mentioned process using a 2D video codec such as HEVC or VVC.

[0343] Fig.17 An exemplary 2D video / image decoder, also referred to as a decoding device, according to an embodiment is shown.

[0344] 2D video / image decoder can follow Fig.15 The reverse process of the operation of the 2D video / image encoder.

[0345] Fig.17 The 2D video / image decoder is Fig.16 Implementation of video decompressors 16001 and 16002. Fig.17 17000 is a schematic block diagram of a 2D video / image decoder 17000 for decoding a video / image signal. The 2D video / image decoder 17000 may be included in the above-mentioned point cloud video decoder 10008, or may be configured as an internal / external component. Fig.17 Each component of may correspond to software, hardware, a processor and / or a combination thereof.

[0346] Here, the input bitstream may be one of a bitstream of a geometry image, a bitstream of a texture image (attribute image), and a bitstream of an occupancy map image. Fig.17 When the 2D video / image decoder is applied to the video decompressor 16001, the bit stream input to the 2D video / image decoder is the bit stream of the compressed texture image, and the reconstructed image output from the 2D video / image decoder is the decompressed texture image. Fig.17 When the 2D video / image decoder is applied to the video decompressor 16002, the bit stream input to the 2D video / image decoder is a bit stream of a compressed geometric image, and the reconstructed image output from the 2D video / image decoder is a decompressed geometric image. Fig.17 The 2D video / image decoder may receive a bit stream of a compressed occupancy map image and decompress it. The reconstructed image (or output image or decoded image) may represent the reconstructed image of the above-mentioned geometry image, texture image (attribute image) and occupancy map image.

[0347] Reference Fig.17 , the inter-frame predictor 17070 and the intra-frame predictor 17080 may be collectively referred to as a predictor. That is, the predictor may include the inter-frame predictor 17070 and the intra-frame predictor 17080. The inverse quantizer 17020 and the inverse transformer 17030 may be collectively referred to as a residual processor. That is, according to an embodiment, the residual processor may include the inverse quantizer 17020 and the inverse transformer 17030. Fig.17 The entropy decoder 17010, the inverse quantizer 17020, the inverse transformer 17030, the adder 17040, the filter 17050, the inter-frame predictor 17070, and the intra-frame predictor 17080 may be configured by one hardware component (e.g., a decoder or a processor). In addition, the memory 17060 may include a decoded picture buffer (DPB) or may be configured by a digital storage medium.

[0348] When a bit stream containing video / image information is input, the decoder 17000 can Fig.15The encoder processes the video / image information in a corresponding process to reconstruct the image. For example, the decoder 17000 may perform decoding using a processing unit applied in the encoder. Therefore, the processing unit of decoding may be, for example, a CU. The CU may be segmented from the CTU or LCU along a quadtree structure and / or a binary tree structure. Then, the reconstructed video signal decoded and output by the decoder 17000 may be played by a player.

[0349] The decoder 17000 may receive a signal output from the encoder in the form of a bitstream, and the received signal may be decoded by the entropy decoder 17010. For example, the entropy decoder 17010 may parse the bitstream to derive information (e.g., video / image information) required for image reconstruction (or picture reconstruction). For example, the entropy decoder 17010 may decode the information in the bitstream based on a coding technique such as exponential Golomb coding, CAVLC, or CABAC, and output the quantized values ​​of the transformation coefficients of the syntax elements and residuals required for image reconstruction. More specifically, in CABAC entropy decoding, bins corresponding to each syntax element in the bitstream may be received, and a context model may be determined based on the decoding target syntax element information and the decoding information about the adjacent decoding target block or the information about the symbol / bin decoded in the previous step. Then, the probability of occurrence of the bin may be predicted based on the determined context model, and arithmetic decoding of the bin may be performed to generate symbols corresponding to the values ​​of each syntax element. According to CABAC entropy decoding, after determining the context model, the context model may be updated based on information about the symbol / cell decoded for the context model of the next symbol / cell. Information about prediction in the information decoded by the entropy decoder 17010 may be provided to the predictor (inter-frame predictor 17070 and intra-frame predictor 17080), and the residual value (i.e., quantized transform coefficients and related parameter information) that has been entropy decoded by the entropy decoder 17010 may be input to the inverse quantizer 17020. In addition, information about filtering in the information decoded by the entropy decoder 17010 may be provided to the filter 17050. A receiver (not shown) configured to receive a signal output from the encoder may also be configured as an internal / external element of the decoder 17000. Alternatively, the receiver may be a component of the entropy decoder 17010.

[0350] The inverse quantizer 17020 may output a transform coefficient by inversely quantizing the quantized transform coefficient. The inverse quantizer 17020 may rearrange the quantized transform coefficient in the form of a two-dimensional block. In this case, the rearrangement may be performed based on the coefficient scanning order implemented by the encoder. The inverse quantizer 17020 may perform inverse quantization on the quantized transform coefficient using a quantization parameter (e.g., quantization step size information) and obtain the transform coefficient.

[0351] The inverse transformer 17030 obtains a residual signal (a residual block and a residual sample array) by transforming the transform coefficients.

[0352] The predictor may perform prediction on the current block and generate a prediction block including prediction samples of the current block. The predictor may determine whether to apply intra prediction or inter prediction to the current block based on information about prediction output from the entropy decoder 17010, and may determine a specific intra / inter prediction mode.

[0353] The intra predictor 17080 of the predictor may predict the current block with reference to samples in the current picture. Depending on the prediction mode, the sample may be adjacent to or distant from the current block. In intra prediction, the prediction mode may include a plurality of non-directional modes and a plurality of directional modes. The intra predictor 17080 may determine the prediction mode applied to the current block using the prediction mode applied to the neighboring blocks.

[0354] The inter-frame predictor 17070 of the predictor may derive a prediction block of the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. In this case, in order to reduce the amount of motion information sent in the inter-frame prediction mode, the motion information may be predicted based on each block, sub-block or sample based on the correlation of the motion information between the neighboring blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may also include information about the inter-frame prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter-frame prediction, the neighboring blocks may include spatial neighboring blocks present in the current picture and temporal neighboring blocks present in the reference picture. For example, the inter-frame predictor 17070 may configure a motion information candidate list based on neighboring blocks, and derive a motion vector and / or a reference picture index of the current block based on the received candidate selection information. Inter-frame prediction may be performed based on various prediction modes. Information about prediction may include information indicating an inter-frame prediction mode of the current block.

[0355] The adder 17040 may add the residual signal obtained in the inverse transformer 17030 to the prediction signal (prediction block or prediction sample array) output from the inter-frame predictor 17070 or the intra-frame predictor 17080, thereby generating a reconstructed signal (reconstructed picture, reconstructed block or reconstructed sample array). When there is no residual signal for processing the target block as in the case of applying the skip mode, the prediction block may be used as the reconstructed block.

[0356] The adder 17040 may be referred to as a reconstructor or a reconstructed block generator. The generated reconstructed signal may be used for intra prediction of the next processing target block in the current picture, or may be used for inter prediction of the next picture through filtering as described below.

[0357] The filter 17050 may improve the subjective / objective image quality by applying filtering to the reconstructed signal output from the adder 17040. For example, the filter 17050 may generate a modified reconstructed picture by applying various filtering techniques to the reconstructed picture, and may send the modified reconstructed picture to the memory 17060 (specifically, the DPB of the memory 17060). For example, various filtering methods may include deblocking filtering, sample adaptive offset, adaptive loop filtering, and bilateral filtering.

[0358] The reconstructed picture stored in the DPB of the memory 17060 may be used as a reference picture in the inter-frame predictor 17070. The memory 17060 may store motion information about a block in the current picture from which motion information is derived (or decoded) and / or motion information about a block in an already reconstructed picture. The stored motion information may be transmitted to the inter-frame predictor 17070 to be used as motion information about a spatially adjacent block or motion information about a temporally adjacent block. The memory 17060 may store reconstructed samples of the reconstructed block in the current picture and transmit the reconstructed samples to the intra-frame predictor 17080.

[0359] In this disclosure, Fig.15 The embodiments described for the filter 15070, the inter-frame predictor 15090 and the intra-frame predictor 15100 of the encoder 15000 may be respectively applied to the filter 17050, the inter-frame predictor 17070 and the intra-frame predictor 17080 of the decoder 17000 in the same or corresponding manner.

[0360] At least one of the above prediction, inverse transform and inverse quantization processes may be skipped. For example, for a block to which pulse code modulation (PCM) is applied, the prediction, inverse transform and inverse quantization processes may be skipped, and the values ​​of the decoded samples may be used as samples of the reconstructed image.

[0361] Occupancy map decompression (16003)

[0362] This is the inverse process of the occupancy map compression described above. Occupancy map decompression is the process of reconstructing the occupancy map by decompressing the occupancy map bitstream.

[0363] Auxiliary patch information decompression (16004)

[0364] The auxiliary patch information can be reconstructed by performing the inverse process of the auxiliary patch information compression described above and decoding the compressed auxiliary patch information bit stream.

[0365] Geometry Reconstruction (16005)

[0366] This is the inverse process of the above-mentioned geometric image generation. Initially, the patch is extracted from the geometric image using the reconstructed occupancy map, the 2D position / size information about the patch included in the auxiliary patch information, and the information about the mapping between the block and the patch. Then, the point cloud is reconstructed in the 3D space based on the geometric image of the extracted patch and the 3D position information about the patch included in the auxiliary patch information. When the geometric value corresponding to the point (u, v) in the patch is g(u, v), and the position coordinates of the patch on the normal axis, tangential axis and bitangent axis of the 3D space are (δ0, s0, r0), the normal, tangential and bitangent coordinates δ(u, v), s(u, v) and r(u, v) in the 3D space mapped to the position of the point (u, v) can be expressed as follows.

[0367] δ(u,v)=δ0+g(u,v)

[0368] s(u,v)=s0+u

[0369] r(u,v)=r0+v

[0370] Smooth(16006)

[0371] As with smoothing in the above-described encoding process, smoothing is a process for eliminating discontinuities that may appear on patch boundaries due to image quality degradation that occurs during a compression process.

[0372] Texture reconstruction (16007)

[0373] Texture reconstruction is a process of reconstructing a color point cloud by assigning color values ​​to each point constituting a smooth point cloud. This can be performed by assigning color values ​​corresponding to the texture image pixels at the same position in the geometric image in 2D space to the points in the point cloud corresponding to the same position in 3D space based on the mapping information of the geometric image and point cloud reconstructed in the above-mentioned geometric reconstruction process.

[0374] Color Smoothing (16008)

[0375] Color smoothing is a process similar to the geometric smoothing described above. Color smoothing is a process used to eliminate discontinuities that may appear on patch boundaries due to image quality degradation that occurs during the compression process. Color smoothing can be performed by the following operations:

[0376] ① Calculate the neighboring points of each point constituting the reconstructed point cloud using a KD tree, etc. The neighboring point information calculated in the above-mentioned geometric smoothing process can be used.

[0377] ② Determine whether each point is on the patch boundary. These operations can be performed based on the boundary information calculated in the above-mentioned geometric smoothing process.

[0378] ③ Check the distribution of color values ​​of neighboring points of a point existing on the boundary and determine whether to perform smoothing. For example, when the entropy of the brightness value is less than or equal to the threshold local entry (there are many similar brightness values), it can be determined that the corresponding part is not an edge part, and smoothing can be performed. As a smoothing method, the color value of a point can be replaced by the average of the color values ​​of neighboring points.

[0379] Fig.18 is a flowchart illustrating the operation of a transmitting apparatus for compressing and transmitting V-PCC-based point cloud data according to an embodiment of the present disclosure.

[0380] The transmitting device according to the embodiment may correspond to Figure 1 The sending device Figure 4 The encoding process, and Fig.15 2D video / image encoder, or perform some / all of its operations. Each component of the sending device may correspond to software, hardware, a processor and / or a combination thereof.

[0381] The operation process of the sending terminal using V-PCC to compress and transmit point cloud data can be performed as shown in the figure.

[0382] The point cloud data transmitting device according to the embodiment may be referred to as a transmitting device or a transmitting system.

[0383] Regarding the patch generator 18000, a patch for 2D image mapping of a point cloud is generated based on the input point cloud data. As a result of the patch generation, patch information and / or auxiliary patch information are generated. The generated patch information and / or auxiliary patch information can be used in the processing of geometric image generation, texture image generation, smoothing, and geometric reconstruction for smoothing.

[0384] The patch packer 18001 performs a patch packing process of mapping the patches generated by the patch generator 18000 into the 2D image. For example, one or more patches may be packed. As a result of the patch packing, an occupancy map may be generated. The occupancy map may be used in a process of geometric image generation, geometric image filling, texture image filling, and / or for smooth geometric reconstruction.

[0385] The geometric image generator 18002 generates a geometric image based on the point cloud data, patch information (or auxiliary patch information) and / or occupancy map. The generated geometric image is preprocessed by the encoding preprocessor 18003 and then encoded into a bit stream by the video encoder 18006.

[0386] The coding preprocessor 18003 may include an image filling process. In other words, some spaces in the generated geometric image and the generated texture image may be filled with meaningless data. The coding preprocessor 18003 may also include a group expansion process for the generated texture image or the texture image on which image filling has been performed.

[0387] The geometry reconstructor 18010 reconstructs a 3D geometry image based on the geometry bitstream, auxiliary patch information and / or occupancy map encoded by the video encoder 18006 .

[0388] The smoother 18009 smoothes the 3D geometric image reconstructed and output by the geometry reconstructor 18010 based on the auxiliary patch information, and outputs the smoothed 3D geometric image to the texture image generator 18004 .

[0389] The texture image generator 18004 can generate a texture image based on smooth 3D geometry, point cloud data, patches (or packed patches), patch information (or auxiliary patch information) and / or occupancy maps. The generated texture image can be preprocessed by the encoding preprocessor 18003 and then encoded into a video bitstream by the video encoder 18006.

[0390] The metadata encoder 18005 can encode the auxiliary patch information into a metadata bitstream.

[0391] The video encoder 18006 may encode the geometric image and the texture image output from the encoding preprocessor 18003 into corresponding video bit streams, and may encode the occupancy map into a video bit stream. Fig.15 The 2D video / image encoder encodes each input image.

[0392] The multiplexer 18007 multiplexes the video bitstream of geometry, the video bitstream of texture image, the video bitstream of occupancy map output from the video encoder 18006, and the bitstream of metadata (including auxiliary patch information) output from the metadata encoder 18005 into one bitstream.

[0393] The transmitter 18008 transmits the bit stream output from the multiplexer 18007 to the receiving side. Alternatively, a file / segment encapsulator may be provided between the multiplexer 18007 and the transmitter 18008, and the bit stream output from the multiplexer 18007 may be encapsulated into a file and / or segment form and output to the transmitter 18008.

[0394] Fig.18The patch generator 18000, patch packer 18001, geometric image generator 18002, texture image generator 18004, metadata encoder 18005 and smoother 18009 can correspond to patch generation 40000, patch packing 40001, geometric image generation 40002, texture image generation 40003, auxiliary patch information compression 40005 and smoothing 40004 respectively. Fig.18 The encoding preprocessor 18003 may include Figure 4 image fillers 40006 and 40007 and group expander 40008, and Fig.18 The video encoder 18006 may include Figure 4 Video compressors 40009, 40010 and 40011 and / or entropy compressor 40012. For those without reference Fig.18 For the description part, refer to Figures 4 to 15 The above blocks may be omitted or may be replaced by blocks having similar or identical functions. In addition, Fig.18 Each block shown may be used as at least one of a processor, software, or hardware. Alternatively, the generated geometry, texture image, video bitstream of the occupancy map, and metadata bitstream of the auxiliary patch information may be formed into one or more track data in a file, or encapsulated into fragments, and sent to the receiving side through a transmitter.

[0395] Procedure for operating the receiving device

[0396] Fig.19 is a flowchart illustrating the operation of a receiving device for receiving and restoring V-PCC based point cloud data according to an embodiment.

[0397] The receiving device according to the embodiment may correspond to Figure 1 The receiving device Fig.16 The decoding process and Fig.17 2D video / image encoder, or perform some / all of its operations. Each component of the receiving device may correspond to software, hardware, a processor and / or a combination thereof.

[0398] The operation of the receiving terminal using V-PCC to receive and reconstruct point cloud data can be performed as shown in the figure. The operation of the V-PCC receiving terminal can follow Fig.18 The reverse process of the operation of the V-PCC sending terminal.

[0399] The point cloud data receiving device according to the embodiment may be referred to as a receiving device, a receiving system, etc.

[0400] The receiver receives a bit stream of a point cloud (i.e., a compressed bit stream), and the demultiplexer 19000 demultiplexes a bit stream of a texture image, a bit stream of a geometric image, a bit stream of an occupancy map image, and a bit stream of metadata (i.e., auxiliary patch information) from the received point cloud bit stream. The demultiplexed bit streams of the texture image, the geometric image, and the occupancy map image are output to the video decoder 19001, and the bit stream of the metadata is output to the metadata decoder 19002.

[0401] According to an embodiment, when Fig.18 When the sending device is provided with a file / segment encapsulator, the file / segment decapsulator is provided in Fig.19 between the receiver of the receiving device and the demultiplexer 19000. In this case, the sending device encapsulates and sends the point cloud bitstream in the form of files and / or fragments, and the receiving device receives and decapsulates the files and / or fragments containing the point cloud bitstream.

[0402] The video decoder 19001 decodes the bit stream of the geometric image, the bit stream of the texture image, and the bit stream of the occupancy map image into the geometric image, the texture image, and the occupancy map image, respectively. Fig.17 The metadata decoder 19002 decodes the metadata bitstream into auxiliary patch information and outputs the information to the geometry reconstructor 19003.

[0403] The geometry reconstructor 19003 reconstructs 3D geometry based on the geometry image, occupancy map and / or auxiliary patch information output from the video decoder 19001 and the metadata decoder 19002 .

[0404] Smoother 19004 smoothes the 3D geometry reconstructed by geometry reconstructor 19003 .

[0405] The texture reconstructor 19005 reconstructs the texture using the texture image and / or the smoothed 3D geometry output from the video decoder 19001. That is, the texture reconstructor 19005 reconstructs the color point cloud image / picture by assigning color values ​​to the smoothed 3D geometry using the texture image. Thereafter, in order to improve the objective / subjective visual quality, additional color smoothing processing may be performed on the color point cloud image / picture by the color smoother 19006. After the rendering processing in the point cloud renderer 19007, the modified point cloud image / picture derived by the above operation is displayed to the user. In some cases, the color smoothing processing may be omitted.

[0406] The above blocks may be omitted or may be replaced by blocks having similar or identical functions. Fig.19Each block shown may function as at least one of a processor, software, and hardware.

[0407] Fig. 20 An exemplary architecture for V-PCC based storage and streaming of point cloud data is shown, according to an embodiment.

[0408] Fig. 20 Part / all of the system may include Figure 1 The sending device and the receiving device, Figure 4 The encoding process, Fig.15 2D video / image encoder, Fig.16 Decoding processing, Fig.18 The sending device and / or Fig.19 Each component in the figure may correspond to software, hardware, a processor and / or a combination thereof.

[0409] Fig. 20 The overall architecture for storing or streaming point cloud data compressed according to video-based point cloud compression (V-PCC) is shown. The process of storing and streaming point cloud data may include an acquisition process, an encoding process, a transmission process, a decoding process, a rendering process, and / or a feedback process.

[0410] An embodiment proposes a method for efficiently providing point cloud media / content / data.

[0411] In order to efficiently provide point cloud media / content / data, the point cloud acquirer 20000 can acquire point cloud video. For example, one or more cameras can acquire point cloud data by capturing, arranging or generating a point cloud. Through this acquisition process, a point cloud video including the 3D position of each point (which can be represented by x, y and z position values, etc.) (hereinafter referred to as geometry) and the attributes of each point (color, reflectivity, transparency, etc.) can be acquired. For example, a Polygon file format (PLY) (or Stanford triangle format) file containing a point cloud video can be generated. For point cloud data with multiple frames, one or more files can be acquired. In this process, point cloud related metadata (e.g., metadata related to capture, etc.) can be generated.

[0412] Post-processing for improving the quality of the content may be required for the captured point cloud video. In the video capture process, the maximum / minimum depth can be adjusted within the range provided by the camera device. Even after the adjustment, point data in unwanted areas may still exist. Therefore, post-processing to remove unwanted areas (e.g., background) or to identify connected spaces and fill spatial holes may be performed. In addition, the point cloud extracted from cameras sharing a spatial coordinate system can be integrated into one piece of content by transforming each point into a global coordinate system based on the position coordinates of each camera obtained through a calibration process. Thus, a point cloud video with a high density of points can be obtained.

[0413] The point cloud preprocessor 20001 may generate one or more pictures / frames of a point cloud video. Generally, a picture / frame may be a unit representing an image at a specific time interval. In addition, when the points constituting the point cloud video are divided into one or more patches and mapped to a 2D plane, the point cloud preprocessor 20001 may generate an occupancy map picture / frame having a value of 0 or 1, which is a binary map indicating the presence or absence of data at a corresponding position in the 2D plane. Here, a patch is a set of points constituting a point cloud video, wherein points belonging to the same patch are adjacent to each other in a 3D space and are mapped to the same face among the flat faces of a 6-face bounding box when mapped to a 2D image. In addition, the point cloud preprocessor 20001 may generate a geometric picture / frame in the form of a depth map representing information about the position (geometry) of each point constituting the point cloud video patch by patch. The point cloud preprocessor 20001 may also generate a texture picture / frame representing color information about each point constituting the point cloud video patch by patch. In this process, metadata required to reconstruct a point cloud from each patch may be generated. The metadata may contain information about the patches (auxiliary information or auxiliary patch information), such as the position and size of each patch in 2D / 3D space. These pictures / frames may be generated continuously in time sequence to construct a video stream or a metadata stream.

[0414] The point cloud video encoder 20002 may encode one or more video streams related to the point cloud video. A video may include multiple frames, and a frame may correspond to a still image / picture. In the present disclosure, the point cloud video may include a point cloud image / frame / picture, and the term "point cloud video" may be used interchangeably with the point cloud video / frame / picture. The point cloud video encoder 20002 may perform a video-based point cloud compression (V-PCC) process. For compression and coding efficiency, the point cloud video encoder 20002 may perform a series of processes such as prediction, transformation, quantization, and entropy coding. The encoded data (encoded video / image information) may be output in the form of a bitstream. Based on the V-PCC process, as described below, the point cloud video encoder 20002 may encode the point cloud video by dividing the point cloud video into a geometry video, an attribute video, an occupancy map video, and metadata (e.g., information about patches). The geometry video may include a geometry image, the attribute video may include an attribute image, and the occupancy map video may include an occupancy map image. The patch data as auxiliary information may include patch related information. The attribute video / image may include a texture video / image.

[0415] The point cloud image encoder 20003 may encode one or more images related to the point cloud video. The point cloud image encoder 20003 may perform video-based point cloud compression (V-PCC) processing. For compression and coding efficiency, the point cloud image encoder 20003 may perform a series of processes such as prediction, transformation, quantization, and entropy coding. The encoded image may be output in the form of a bitstream. Based on the V-PCC process, as described below, the point cloud image encoder 20003 may encode the point cloud image by dividing the point cloud image into a geometry image, an attribute image, an occupancy map image, and metadata (e.g., information about patches).

[0416] According to an embodiment, the point cloud video encoder 20002, the point cloud image encoder 20003, the point cloud video decoder 20006 and the point cloud image decoder 20008 may be performed by one encoder / decoder as described above and may be performed along separate paths as shown in the figure.

[0417] In the file / segment encapsulator 20004, the encoded point cloud data and / or point cloud related metadata may be encapsulated as a file or segment for streaming. Here, the point cloud related metadata may be received from a metadata processor (not shown) or the like. The metadata processor may be included in the point cloud video / image encoder 20002 / 20003, or may be configured as a separate component / module. The file / segment encapsulator 20004 may encapsulate the corresponding video / image / metadata in a file format such as ISOBMFF or in the form of a DASH segment or the like. According to an embodiment, the file / segment encapsulator 20004 may include point cloud metadata in a file format. The point cloud related metadata may be included in boxes at various levels of, for example, the ISOBMFF file format, or as data in a separate track within a file. According to an embodiment, the file / segment encapsulator 20004 may encapsulate the point cloud related metadata into a file.

[0418] The file / segment encapsulator 20004 according to the embodiment can store a bitstream or a separate bitstream into one or more tracks in the file, and can also encapsulate signaling information for the operation. In addition, the atlas stream (or patch stream) included in the bitstream can be stored as a track in the file, and related signaling information can be stored. In addition, the SEI message present in the bitstream can be stored in the track of the file, and related signaling information can be stored.

[0419] A transmission processor (not shown) may perform transmission processing of the encapsulated point cloud data according to a file format. The transmission processor may be included in a transmitter (not shown), or may be configured as a separate component / module. The transmission processor may process the point cloud data according to a transmission protocol. The transmission processing may include processing for transmission via a broadcast network and processing for transmission via broadband. According to an embodiment, the transmission processor may receive point cloud related metadata and point cloud data from a metadata processor, and perform transmission processing of the point cloud video data.

[0420] The transmitter may send a point cloud bitstream or a file / segment including a bitstream to a receiver (not shown) of a receiving device via a digital storage medium or a network. For transmission, processing according to any transmission protocol may be performed. The data processed for transmission may be transmitted via a broadcast network and / or by broadband. The data may be transmitted to the receiving side in an on-demand manner. The digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD. The transmitter may include an element for generating a media file in a predetermined file format, and may include an element for transmission via a broadcast / communication network. The receiver may extract the bitstream and send the extracted bitstream to a decoder.

[0421] The receiver may receive point cloud data transmitted by the point cloud data transmitting device according to the present disclosure. Depending on the transmission channel, the receiver may receive the point cloud data via a broadcast network or via broadband. Alternatively, the point cloud data may be received via a digital storage medium. The receiver may include a process for decoding the received data and rendering the data according to the user's viewport.

[0422] The receiving processor (not shown) may perform processing on the received point cloud video data according to the transmission protocol. The receiving processor may be included in the receiver, or may be configured as a separate component / module. The receiving processor may reversely perform the processing of the above-mentioned sending processor to correspond to the transmission processing performed on the sending side. The receiving processor may transmit the acquired point cloud video to the file / segment decapsulator 20005, and transmit the acquired point cloud related metadata to the metadata parser.

[0423] The file / fragment decapsulator 20005 can decapsulate the point cloud data received from the receiving processor in the form of a file. The file / fragment decapsulator 20005 can decapsulate the file according to ISOBMFF, etc., and can obtain a point cloud bitstream or point cloud related metadata (or a separate metadata bitstream). The acquired point cloud bitstream can be transmitted to the point cloud video decoder 20006 and the point cloud image decoder 20008, and the acquired point cloud video related metadata (metadata bitstream) can be transmitted to the metadata processor (not shown). The point cloud bitstream may include metadata (metadata bitstream). The metadata processor may be included in the point cloud video decoder 20006, or may be configured as a separate component / module. The point cloud video related metadata acquired by the file / fragment decapsulator 20005 may take the form of a box or track in the file format. If necessary, the file / fragment decapsulator 20005 may receive the metadata required for decapsulation from the metadata processor. The point cloud related metadata may be transmitted to the point cloud video decoder 20006 and / or the point cloud image decoder 20008 and used in the point cloud decoding process, or may be transmitted to the renderer 20009 and used in the point cloud rendering process.

[0424] The point cloud video decoder 20006 may receive a bitstream and decode the video / image by performing an operation corresponding to the operation of the point cloud video encoder 20002. In this case, as described below, the point cloud video decoder 20006 may decode the point cloud video by dividing the point cloud video into a geometry video, an attribute video, an occupancy map video, and auxiliary patch information. The geometry video may include a geometry image, the attribute video may include an attribute image, and the occupancy map video may include an occupancy map image. The auxiliary information may include auxiliary patch information. The attribute video / image may include a texture video / image.

[0425] The point cloud image decoder 20008 may receive the bitstream and perform an inverse process corresponding to the operation of the point cloud image encoder 20003. In this case, the point cloud image decoder 20008 may divide the point cloud image into a geometry image, an attribute image, an occupancy map image, and metadata (which is, for example, auxiliary patch information) to decode them.

[0426] The 3D geometry may be reconstructed based on the decoded geometry video / image, occupancy map, and auxiliary patch information, and then may be subjected to smoothing. A color point cloud image / picture may be reconstructed by assigning color values ​​to the smoothed 3D geometry based on the texture video / image. The renderer 20009 may render the reconstructed geometry and color point cloud image / picture. The rendered video / image may be displayed through a display. All or part of the rendering result may be displayed to the user through a VR / AR display or a typical display.

[0427] The sensor / tracker (sensing / tracking) 20007 acquires orientation information and / or user viewport information from the user or receiving side and transmits the orientation information and / or user viewport information to the receiver and / or transmitter. The orientation information may represent information about the position, angle, movement, etc. of the user's head, or information about the position, angle, movement, etc. of the device through which the user is viewing the video / image. Based on this information, information about the area currently viewed by the user in the 3D space (i.e., viewport information) may be calculated.

[0428] The viewport information may be information about the area in 3D space that the user is currently viewing through the device or HMD. A device such as a display may extract the viewport area based on the orientation information, the vertical or horizontal FOV supported by the device, etc. The orientation or viewport information may be extracted or calculated at the receiving side. The orientation or viewport information analyzed at the receiving side may be sent to the sending side on a feedback channel.

[0429] Based on the orientation information and / or viewport information indicating the area currently viewed by the user acquired by the sensor / tracker 20007, the receiver can effectively extract or decode only the media data of the specific area (i.e., the area indicated by the orientation information and / or viewport information) from the file. In addition, based on the orientation information and / or viewport information acquired by the sensor / tracker 20007, the transmitter can effectively encode only the media data of the specific area (i.e., the area indicated by the orientation information and / or viewport information), or generate and send its file.

[0430] The renderer 20009 can render the decoded point cloud data in 3D space. The rendered video / image can be displayed through a display. The user can view all or part of the rendering result through a VR / AR display or a typical display.

[0431] Feedback processing may include transmitting various feedback information that may be obtained in the rendering / display processing to a decoder on the sending side or the receiving side. Through feedback processing, interactivity may be provided when consuming point cloud data. According to an embodiment, head orientation information, viewport information indicating the area currently viewed by the user, etc. may be transmitted to the sending side in feedback processing. According to an embodiment, the user may interact with content implemented in a VR / AR / MR / autonomous driving environment. In this case, information related to the interaction may be transmitted to the sending side or the service provider in feedback processing. According to an embodiment, feedback processing may be skipped.

[0432] According to an embodiment, the above feedback information may not only be sent to the sending side, but also consumed at the receiving side. That is, the decapsulation processing, decoding and rendering processing of the receiving side may be performed based on the above feedback information. For example, the point cloud data about the area currently viewed by the user may be preferentially decapsulated, decoded and rendered based on the orientation information and / or the viewport information.

[0433] Fig.21 is an exemplary block diagram of an apparatus for storing and transmitting point cloud data according to an embodiment.

[0434] Fig.21 A point cloud system according to an embodiment is shown. Fig.21 Part / all of the system may include Figure 1 The sending device and the receiving device, Figure 4 The encoding process, Fig.15 2D video / image encoder, Fig.16 Decoding processing, Fig.18 The sending device and / or Fig.19 In addition, it may include or correspond to Fig. 20 part / all of the system.

[0435] The point cloud data transmitting device according to the embodiment may be configured as shown in the figure. Each element of the transmitting device may be a module / unit / component / hardware / software / processor.

[0436] The geometry, attributes, occupancy maps, auxiliary data (auxiliary information) and mesh data of the point cloud can each be configured as a separate stream or stored in different tracks in the file. In addition, they can be included in separate fragments.

[0437] The point cloud acquirer 21000 acquires a point cloud. For example, one or more cameras may acquire point cloud data by capturing, arranging or generating a point cloud. Through this acquisition process, point cloud data including the 3D position of each point (which may be represented by x, y and z position values, etc.) (hereinafter referred to as geometry) and the attributes of each point (color, reflectivity, transparency, etc.) may be acquired. For example, a Polygon file format (PLY) (or Stanford triangle format) file including point cloud data may be generated, etc. For point cloud data having multiple frames, one or more files may be acquired. In this process, point cloud related metadata (e.g., metadata related to capture, etc.) may be generated. The patch generator 21001 generates patches from point cloud data. The patch generator 21001 generates point cloud data or point cloud video into one or more pictures / frames. A picture / frame may generally represent a unit representing an image at a specific time interval. When the points constituting the point cloud video are divided into one or more patches (a collection of points constituting the point cloud video, where the points belonging to the same patch are adjacent to each other in the 3D space and are mapped in the same direction between the flat surfaces of the 6-sided bounding box when mapped to the 2D image) and mapped to the 2D plane, an occupancy map screen / frame of a binary map can be generated, which indicates whether there is data at the corresponding position in the 2D plane with 0 or 1. In addition, a geometric screen / frame in the form of a depth map that represents information about the position (geometry) of each point constituting the point cloud video on a patch-by-patch basis can be generated. A texture screen / frame that represents color information about each point constituting the point cloud video on a patch-by-patch basis can be generated. In this process, metadata required to reconstruct the point cloud from each patch can be generated. The metadata may include information about the patch, such as the position and size of each patch in the 2D / 3D space. These screens / frames can be generated continuously in time order to construct a video stream or a metadata stream.

[0438] In addition, the patch can be used for 2D image mapping. For example, the point cloud data can be projected onto each face of a cube. After the patch is generated, a geometric image, one or more attribute images, an occupancy map, auxiliary data, and / or mesh data can be generated based on the generated patch.

[0439] The geometric image generation, attribute image generation, occupancy map generation, auxiliary data generation and / or mesh data generation are performed by the point cloud preprocessor 20001 or a controller (not shown). The point cloud preprocessor 20001 may include a patch generator 21001, a geometric image generator 21002, an attribute image generator 21003, an occupancy map generator 21004, an auxiliary data generator 21005 and a mesh data generator 21006.

[0440] The geometry image generator 21002 generates a geometry image based on the results of patch generation. The geometry represents a point in 3D space. The geometry image is generated using an occupancy map, which includes information related to 2D image packing of patches, auxiliary data (including patch data), and / or mesh data based on the patches. The geometry image is related to information such as the depth (e.g., near, far) of the patches generated after patch generation.

[0441] The attribute image generator 21003 generates an attribute image. For example, an attribute may represent a texture. A texture may be a color value matched to each point. According to an embodiment, an image including multiple attributes (e.g., color and reflectivity) (N attributes) of a texture may be generated. Multiple attributes may include material information and reflectivity. According to an embodiment, an attribute may additionally include information indicating a color, which may vary according to viewing angle and light even for the same texture.

[0442] The occupancy map generator 21004 generates an occupancy map from the patch. The occupancy map includes information indicating whether data exists in a pixel (eg, a corresponding geometry or attribute image).

[0443] The auxiliary data generator 21005 generates auxiliary data (or auxiliary information) including information about the patch. That is, the auxiliary data represents metadata about the patch of the point cloud object. For example, it may represent information such as the normal vector of the patch. Specifically, the auxiliary data may include information required to reconstruct the point cloud from the patch (e.g., information about the position, size, etc. of the patch in 2D / 3D space, as well as projection (normal) plane identification information, patch mapping information, etc.).

[0444] The mesh data generator 21006 generates mesh data from the patch. A mesh represents the connection between adjacent points. For example, it can represent data in the shape of a triangle. For example, mesh data refers to the connectivity between points.

[0445] The point cloud pre-processor 20001 or controller generates metadata related to patch generation, geometric image generation, attribute image generation, occupancy map generation, auxiliary data generation and mesh data generation.

[0446] The point cloud transmitting device performs video encoding and / or image encoding in response to the result generated by the point cloud preprocessor 20001. The point cloud transmitting device may generate point cloud image data and point cloud video data. According to an embodiment, the point cloud data may have only video data, only image data, and / or both video data and image data.

[0447] The video encoder 21007 performs geometry video compression, attribute video compression, occupancy map video compression, auxiliary data compression, and / or mesh data compression. The video encoder 21007 generates a video stream containing encoded video data.

[0448] Specifically, in geometric video compression, point cloud geometric video data is encoded. In attribute video compression, attribute video data of the point cloud is encoded. In auxiliary data compression, auxiliary data associated with the point cloud video data is encoded. In mesh data compression, mesh data of the point cloud video data is encoded. The various operations of the point cloud video encoder can be performed in parallel.

[0449] The image encoder 21008 performs geometric image compression, attribute image compression, occupancy map image compression, auxiliary data compression, and / or mesh data compression. The image encoder generates an image containing encoded image data.

[0450] Specifically, in geometric image compression, point cloud geometric image data is encoded. In attribute image compression, attribute image data of the point cloud is encoded. In auxiliary data compression, auxiliary data associated with the point cloud image data is encoded. In mesh data compression, mesh data associated with the point cloud image data is encoded. The various operations of the point cloud image encoder can be performed in parallel.

[0451] The video encoder 21007 and / or the image encoder 21008 may receive metadata from the point cloud preprocessor 21001. The video encoder 21007 and / or the image encoder 21008 may perform various encoding processes based on the metadata.

[0452] The file / segment encapsulator 21009 encapsulates video streams and / or images in the form of files and / or segments. The file / segment encapsulator 21009 performs video track encapsulation, metadata track encapsulation and / or image encapsulation.

[0453] In video track encapsulation, one or more video streams may be encapsulated into one or more tracks.

[0454] In metadata track encapsulation, metadata related to video streams and / or images may be encapsulated in one or more tracks. Metadata includes data related to the content of the point cloud data. For example, it may include initial viewing orientation metadata. Depending on the implementation, metadata may be encapsulated into a metadata track, or may be encapsulated together in a video track or an image track.

[0455] In image packaging, one or more images can be packaged into one or more tracks or projects.

[0456] For example, according to an embodiment, when four video streams and two images are input to the encapsulator, the four video streams and the two images may be encapsulated in one file.

[0457] The file / segment packager 21009 may receive metadata from the point cloud preprocessor 21001. The file / segment packager 21009 may perform packaging based on the metadata.

[0458] The files and / or segments generated by the file / segment encapsulator 21009 are sent by a point cloud sending device or sender. For example, the segments may be transmitted according to a DASH-based protocol.

[0459] The transmitter may send the point cloud bitstream or a file / segment including the bitstream to a receiver of a receiving device via a digital storage medium or a network. For transmission, processing according to any transmission protocol may be performed. The processed data may be transmitted via a broadcast network and / or by broadband for transmission. The data may be transmitted to the receiving side in an on-demand manner. The digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD.

[0460] The file / segment encapsulator 21009 according to the embodiment may divide and store a bitstream or a separate bitstream into one or more tracks in a file, and may encapsulate signaling information for it. In addition, the patch (or atlas) stream included in the bitstream may be stored as a track in the file, and related signaling information may be stored. In addition, the SEI message present in the bitstream may be stored in the track of the file, and related signaling information may be stored.

[0461] The transmitter may include an element for generating a media file in a predetermined file format, and may include an element for transmission via a broadcast / communication network. The transmitter receives orientation information and / or viewport information from a receiver. The transmitter may transmit the acquired orientation information and / or viewport information (or information selected by a user) to the point cloud preprocessor 21001, the video encoder 21007, the image encoder 21008, the file / segment encapsulator 21009 and / or the point cloud encoder. Based on the orientation information and / or the viewport information, the point cloud encoder may encode all point cloud data or point cloud data indicated by the orientation information and / or the viewport information. Based on the orientation information and / or the viewport information, the file / segment encapsulator may encapsulate all point cloud data or point cloud data indicated by the orientation information and / or the viewport information. Based on the orientation information and / or the viewport information, the transmitter may transmit all point cloud data or point cloud data indicated by the orientation information and / or the viewport information.

[0462] For example, the point cloud preprocessor 21001 may perform the above operations on all point cloud data or on point cloud data indicated by orientation information and / or viewport information. The video encoder 21007 and / or the image encoder 21008 may perform the above operations on all point cloud data or on point cloud data indicated by orientation information and / or viewport information. The file / fragment encapsulator 21009 may perform the above operations on all point cloud data or on point cloud data indicated by orientation information and / or viewport information. The transmitter may perform the above operations on all point cloud data or on point cloud data indicated by orientation information and / or viewport information.

[0463] Fig. 22 is an exemplary block diagram of a point cloud data receiving device according to an embodiment.

[0464] Fig. 22 A point cloud system according to an embodiment is shown. Fig. 22 Part / all of the system may include Figure 1 The sending device and the receiving device, Figure 4 The encoding process, Fig.15 2D video / image encoder, Fig.16 Decoding processing, Fig.18 The sending device and / or Fig.19 In addition, it may include or correspond to Fig. 20 and Fig.21 part / all of the system.

[0465] The various components of the receiving device may be modules / units / components / hardware / software / processors. The transmitting client may receive point cloud data, point cloud bitstreams, or files / fragments including a bitstream sent by a point cloud data sending device according to an embodiment. Depending on the channel used for transmission, the receiver may receive the point cloud data via a broadcast network or by broadband. Alternatively, the point cloud data may be received via a digital storage medium. The receiver may include a process for decoding the received data and rendering the received data according to a user viewport. The transmitting client (receiving processor) 22006 may perform processing on the received point cloud data according to a transmission protocol. The receiving processor may be included in the receiver, or may be configured as a separate component / module. The receiving processor may perform the processing of the above-mentioned sending processor in reverse to correspond to the transmission processing performed by the sending side. The receiving processor may transmit the acquired point cloud data to the file / fragment decapsulator 22000 and transmit the acquired point cloud-related metadata to the metadata processor (not shown).

[0466] The sensor / tracker 22005 acquires orientation information and / or viewport information and may transmit the acquired orientation information and / or viewport information to the transmission client 22006 , the file / fragment decapsulator 22000 , the point cloud decoders 22001 and 22002 , and the point cloud processor 22003 .

[0467] The transmission client 22006 may receive all point cloud data or point cloud data indicated by the orientation information and / or viewport information based on the orientation information and / or viewport information. The file / fragment decapsulator 22000 may decapsulate all point cloud data or point cloud data indicated by the orientation information and / or viewport information based on the orientation information and / or viewport information. The point cloud decoder (video decoder 22001 and / or image decoder 22002) may decode all point cloud data or point cloud data indicated by the orientation information and / or viewport information based on the orientation information and / or viewport information. The point cloud processor 22003 may process all point cloud data or point cloud data indicated by the orientation information and / or viewport information based on the orientation information and / or viewport information.

[0468] The file / fragment decapsulator 22000 performs video track decapsulation, metadata track decapsulation, and / or image decapsulation. The file / fragment decapsulator 22000 may decapsulate point cloud data received from a receiving processor in the form of a file. The file / fragment decapsulator 22000 may decapsulate a file or a fragment according to ISOBMFF, etc. to obtain a point cloud bitstream or point cloud-related metadata (or a separate metadata bitstream). The acquired point cloud bitstream may be transmitted to the point cloud decoders 22001 and 22002, and the acquired point cloud-related metadata (or metadata bitstream) may be transmitted to a metadata processor (not shown). The point cloud bitstream may include metadata (metadata bitstream). The metadata processor may be included in the point cloud video decoder, or may be configured as a separate component / module. The point cloud-related metadata acquired by the file / fragment decapsulator 22000 may take the form of a box or track in a file format. If necessary, the file / fragment decapsulator 22000 may receive metadata required for decapsulation from the metadata processor. The point cloud related metadata may be transmitted to the point cloud decoders 22001 and 22002 and used in the point cloud decoding process, or may be transmitted to the renderer 22004 and used in the point cloud rendering process. The file / fragment depacketizer 22000 may generate metadata related to the point cloud data.

[0469] In video track decapsulation performed by the file / fragment decapsulator 22000, video tracks contained in files and / or fragments are decapsulated. Video streams including geometry video, attribute video, occupancy maps, auxiliary data and / or mesh data are decapsulated.

[0470] In metadata track decapsulation performed by the file / fragment decapsulator 22000, a bitstream including metadata related to point cloud data and / or auxiliary data is decapsulated.

[0471] In the image decapsulation performed by the file / fragment decapsulator 22000, an image including a geometry image, an attribute image, an occupancy map, auxiliary data and / or mesh data is decapsulated.

[0472] The file / fragment decapsulator 22000 according to the embodiment can store a bitstream or a separate bitstream into one or more tracks in the file, and can also decapsulate the signaling information used therefor. In addition, the atlas (patch) stream included in the bitstream can be decapsulated based on the track in the file, and the relevant signaling information can be parsed. In addition, the SEI message present in the bitstream can be decapsulated based on the track in the file, and the relevant signaling information can also be obtained.

[0473] The video decoder 22001 performs geometry video decompression, attribute video decompression, occupancy map decompression, auxiliary data decompression, and / or mesh data decompression. The video decoder 22001 decodes the geometry video, attribute video, auxiliary data, and / or mesh data in a process corresponding to the process performed by the video encoder of the point cloud transmitting device according to the embodiment.

[0474] The image decoder 22002 performs geometric image decompression, attribute image decompression, occupancy map decompression, auxiliary data decompression, and / or mesh data decompression. The image decoder 22002 decodes the geometric image, attribute image, auxiliary data, and / or mesh data in a process corresponding to the process performed by the image encoder of the point cloud transmitting device according to the embodiment.

[0475] The video decoder 22001 and the video decoder 22002 according to an embodiment may be processed by one video / image decoder as described above, and may be executed along separate paths as shown in the figure.

[0476] The video decoder 22001 and / or the image decoder 22002 may generate metadata related to the video data and / or the image data.

[0477] In the point cloud processor 22003, geometric reconstruction and / or attribute reconstruction is performed.

[0478] In geometric reconstruction, a geometric video and / or a geometric image is reconstructed from the decoded video data and / or the decoded image data based on the occupancy map, the auxiliary data and / or the mesh data.

[0479] In attribute reconstruction, an attribute video and / or an attribute image are reconstructed from a decoded attribute video and / or a decoded attribute image based on an occupancy map, auxiliary data, and / or mesh data. According to an embodiment, for example, an attribute may be a texture. According to an embodiment, an attribute may represent multiple pieces of attribute information. When there are multiple attributes, the point cloud processor 22003 according to an embodiment performs multiple attribute reconstructions.

[0480] The point cloud processor 22003 may receive metadata from the video decoder 22001 , the image decoder 22002 , and / or the file / fragment decapsulator 22000 , and process the point cloud based on the metadata.

[0481] The point cloud renderer 22004 renders the reconstructed point cloud. The point cloud renderer 22004 may receive metadata from the video decoder 22001, the image decoder 22002, and / or the file / segment decapsulator 22000, and render the point cloud based on the metadata.

[0482] The display displays the rendering result on the actual display device.

[0483] According to the method / device according to the embodiment, Figure 20 to Figure 22 As shown, the sending side may encode the point cloud data into a bitstream, encapsulate the bitstream into a file and / or a fragment and send it. The receiving side may decapsulate the file and / or the fragment into a bitstream containing the point cloud, and may decode the bitstream into the point cloud data. For example, a point cloud data device according to an embodiment may encapsulate the point cloud data based on a file. The file may include a V-PCC track containing parameters of the point cloud, a geometry track containing geometry, an attribute track containing attributes, and an occupancy track containing an occupancy map.

[0484] In addition, the point cloud data receiving device according to the embodiment decapsulates the point cloud data based on the file. The file may include a V-PCC track containing point cloud parameters, a geometry track containing geometry, an attribute track containing attributes, and an occupancy track containing an occupancy map.

[0485] The above encapsulation operation can be performed by Fig. 20 File / fragment encapsulator 20004, Fig.21 The above decapsulation operation can be performed by the file / segment encapsulator 21009. Fig. 20 File / fragment decapsulator 20005 or Fig. 22 The file / fragment decapsulator 22000 executes.

[0486] Fig.23 An exemplary structure that can be operated in conjunction with the point cloud data transmitting / receiving method / apparatus according to the embodiments is shown.

[0487] In the structure according to the embodiment, at least one of the server 23600, the robot 23100, the self-driving vehicle 23200, the XR device 23300, the smart phone 23400, the home appliance 23500, and / or the head mounted display (HMD) 23700 is connected to the cloud network 23000. Here, the robot 23100, the self-driving vehicle 23200, the XR device 23300, the smart phone 23400, or the home appliance 23500 may be referred to as a device. In addition, the XR device 23300 may correspond to a point cloud data (PCC) device according to the embodiment, or may be operatively connected to the PCC device.

[0488] The cloud network 23000 may represent a network that constitutes a part of a cloud computing infrastructure or exists in a cloud computing infrastructure. Here, the cloud network 23000 may be configured using a 3G network, a 4G or Long Term Evolution (LTE) network, or a 5G network.

[0489] The server 23600 may be connected to at least one of the robot 23100 , the self-driving vehicle 23200 , the XR device 23300 , the smart phone 23400 , the home appliance 23500 , and / or the HMD 23700 via the cloud network 23000 , and may assist at least a portion of the processing of the connected devices 23100 to 23700 .

[0490] HMD 23700 represents one of implementation types of an XR device and / or a PCC device according to an embodiment. The HMD type device according to an embodiment includes a communication unit, a control unit, a memory, an I / O unit, a sensor unit, and a power supply unit.

[0491] Hereinafter, various embodiments of devices 23100 to 23500 to which the above-described technology is applied will be described. Fig.23 The illustrated devices 23100 to 23500 may be operatively connected / coupled to the point cloud data transmitting and receiving devices according to the above-described embodiments.

[0492] <PCC+XR>

[0493] The XR / PCC device 23300 may employ PCC technology and / or XR (AR+VR) technology, and may be implemented as an HMD, a head-up display (HUD) disposed in a vehicle, a television, a mobile phone, a smart phone, a computer, a wearable device, a home appliance, a digital signage, a vehicle, a fixed robot, or a mobile robot.

[0494] The XR / PCC device 23300 can analyze 3D point cloud data or image data obtained through various sensors or from external devices and generate position data and attribute data about 3D points. Thus, the XR / PCC device 23300 can obtain information about the surrounding space or real objects and render and output XR objects. For example, the XR / PCC device 23300 can match an XR object including auxiliary information about the identified object with the identified object and output the matched XR object.

[0495] <PCC + Autonomous Driving + XR>

[0496] The autonomous vehicle 23200 can be implemented as a mobile robot, a vehicle, an unmanned aerial vehicle, etc. by applying PCC technology and XR technology.

[0497] The autonomous vehicle 23200 applying the XR / PCC technology can represent an autonomous vehicle provided with means for providing an XR image, or an autonomous vehicle that is a control / interaction target in the XR image. Specifically, the autonomous vehicle 23200 that is a control / interaction target in the XR image can be distinguished from the XR device 23300 and operably connected thereto.

[0498] The autonomous vehicle 23200 having means for providing an XR / PCC image can obtain sensor information from sensors including a camera and output the generated XR / PCC image based on the obtained sensor information. For example, the autonomous vehicle can have a HUD and output the XR / PCC image thereto to provide an XR / PCC object corresponding to a real object or an object present on the screen to passengers.

[0499] In this case, when the XR / PCC object is output to the HUD, at least a part of the XR / PCC object can be output to overlap with the real object pointed at by the passenger's eyes. On the other hand, when the XR / PCC object is output on a display provided inside the autonomous vehicle, at least a part of the XR / PCC object can be output to overlap with the object on the screen. For example, the autonomous vehicle can output an XR / PCC object corresponding to objects such as a road, another vehicle, a traffic light, a traffic sign, a two-wheeler, a pedestrian, and a building.

[0500] According to an embodiment, virtual reality (VR) technology, augmented reality (AR) technology, mixed reality (MR) technology, and / or point cloud compression (PCC) technology are applicable to various devices.

[0501] In other words, VR technology is a display technology that only provides real-world objects, backgrounds, etc. as CG images. On the other hand, AR technology refers to a technology that displays a CG image virtually created on a real object image. MR technology is similar to the above-mentioned AR technology in that the virtual object to be displayed is mixed and combined with the real world. However, MR technology is different from AR technology. AR technology makes a clear distinction between real objects and virtual objects created as CG images and uses virtual objects as supplementary objects to real objects, while MR technology treats virtual objects as objects with the same properties as real objects. More specifically, an example of the application of MR technology is a holographic service.

[0502] Recently, VR, AR, and MR technologies are often referred to as extended reality (XR) technologies, rather than being clearly distinguished from each other. Therefore, embodiments of the present disclosure are applicable to all VR, AR, MR, and XR technologies. For these technologies, encoding / decoding based on PCC, V-PCC, and G-PCC technologies can be applied.

[0503] The PCC method / apparatus according to the embodiment may be applied to a self-driving vehicle 23200 that provides a self-driving service.

[0504] The self-driving vehicle 23200 providing the self-driving service is connected to the PCC device for wired / wireless communication.

[0505] When the point cloud data compression sending and receiving device (PCC device) according to an embodiment is connected to the self-driving vehicle 23200 for wired / wireless communication, the device may receive and process content data related to the AR / VR / PCC service that can be provided together with the self-driving service and send the processed content data to the self-driving vehicle 23200. In the case where the point cloud data sending and receiving device is installed on the vehicle, the point cloud sending and receiving device may receive and process content data related to the AR / VR / PCC service according to a user input signal input through the user interface device and provide the processed content data to the user. The self-driving vehicle 23200 or the user interface device according to the embodiment may receive the user input signal. The user input signal according to the embodiment may include a signal indicating the self-driving service.

[0506] As mentioned above, Figure 1 , Figure 4 , Fig.18 , Fig. 20 or Fig.21The V-PCC-based point cloud video encoder projects 3D point cloud data (or content) into a 2D space to generate patches. Patches are generated in a 2D space by dividing the data into a geometric image (called a geometric frame or a geometric patch frame) representing position information and a texture image (called an attribute frame or an attribute patch frame) representing color information. The geometric image and the texture image are video compressed for each frame, and a video bitstream of the geometric image (called a geometric bitstream) and a video bitstream of the texture image (called an attribute bitstream) are output. In addition, auxiliary patch information (also called patch information or metadata) including projection plane information and patch size information about each patch (which are required for decoding 2D patches on the receiving side) is also video compressed and the bitstream of the auxiliary patch information is output. In addition, an occupancy map indicating the presence / absence of a point for each pixel as 0 or 1 is entropy compressed or video compressed depending on whether it is in lossless mode or lossy mode, and a video bitstream of the occupancy map (or called an occupancy map bitstream) is output. The compressed geometry bitstream, the compressed attribute bitstream, the compressed auxiliary patch information bitstream and the compressed occupancy map bitstream are multiplexed into the structure of the V-PCC bitstream.

[0507] According to the implementation mode, the V-PCC bit stream can be sent to the receiving side as it is, or can be transmitted by Figure 1 , Fig.18 , Fig. 20 or Fig.21 The file / segment encapsulator encapsulates the data in the form of a file / segment and sends it to a receiving device or stores it in a digital storage medium (e.g., USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc.). According to an embodiment of the present disclosure, the file is in the file format ISOBMFF.

[0508] According to an embodiment, the V-PCC bitstream may be transmitted through a plurality of tracks in a file, or may be transmitted through a single track. The details will be described later.

[0509] Fig.24 An exemplary structure of a V-PCC bitstream according to an embodiment is shown. According to an embodiment, Fig.24 The V-PCC bitstream is from Figure 1 , Figure 4 , Fig.18 , Fig. 20 or Fig.21 Output of the V-PCC based point cloud video encoder.

[0510] The V-PCC bitstream is composed of one or more V-PCC units. That is to say, the V-PCC bitstream is a group of V-PCC units. Each V-PCC unit is composed of a V-PCC unit header and a V-PCC unit payload. In the present disclosure, the data contained in the V-PCC unit payload is distinguished by the V-PCC unit header. To this end, the V-PCC unit header contains type information indicating the V-PCC unit type. Each V-PCC unit payload can contain geometric video data (i.e., compressed geometric bitstream), attribute video data (i.e., compressed attribute bitstream), occupancy video data (i.e., compressed occupancy map bitstream), patch data group (PDG) or sequence parameter set (SPS) according to the type information in the V-PCC unit header. PDG is also called Atlas. In the present disclosure, Atlas can replace PDG.

[0511] The V-PCC unit payload, which contains at least geometry video data, attribute video data and occupancy map video data (also referred to as occupancy video data), corresponds to a video data unit (eg, a HEVC NAL unit) decodable by a suitable video decoder.

[0512] The geometric video data, attribute video data, and occupancy map video data are referred to as 2D video coding information for the geometry, attributes, and occupancy map components of the encoded point cloud, and the PDG (or Atlas) is referred to as non-video coding information. The PDG includes auxiliary patch information. The SPS includes overall coding information about the bitstream and is also referred to as configuration and metadata information. The SPS and PDG may be referred to as signaling information, and may be generated by a metadata processor in a point cloud video encoder, or may be generated by a separate component / module in a point cloud video encoder. In addition, in the present disclosure, the SPS and PDG are referred to as initialization information, and the geometric video data, attribute video data, and occupancy map video data are referred to as point cloud data.

[0513] In one embodiment, when the type information in the V-PCC unit header indicates VPCC_PDG, the corresponding V-PCC unit payload may include a patch data group. Conversely, when the V-PCC unit payload contains a patch data group, the type information in the corresponding V-PCC unit header may identify the same content. This is an option for the designer, so the present disclosure is not limited to the above embodiments. Other V-PCC units are configured in the same or similar manner.

[0514] The patch data group is included in the V-PCC unit payload in the format of patch_data_group(). The patch data group may include information about one or more patch mosaic groups and at least one of a patch sequence parameter set, a patch frame parameter set, a patch frame geometry parameter set, a geometry patch parameter set, a patch frame attribute parameter set, or an attribute patch parameter set.

[0515] The patch frame (also referred to as a point cloud object) as the target of the point cloud data may be divided into one or more tiles. A tile according to an embodiment may represent a specific area in a 3D space or a specific area in a 2D plane. In addition, a tile may be a cuboid in a bounding box, or a portion of a patch frame or a sub-bounding box. In the present disclosure, a tile may be divided into one or more tiles. Figure 1 Point cloud video encoder, Fig.18 Patch generator, Fig. 20 Point cloud preprocessing unit or Fig.21 A patch generator, or a separate component / module, is used to divide a patch frame (or point cloud object) into one or more tiles.

[0516] According to an embodiment, Fig.24 The V-PCC bit stream of the structure shown may be transmitted to the receiving side as it is, or may be encapsulated into a file / segment form and transmitted to the receiving side.

[0517] According to an embodiment of the present disclosure, the V-PCC bitstream is encapsulated into a file format and transmitted. For example, the V-PCC bitstream may be encapsulated into a file format based on ISOBMFF (ISO Base Media File Format).

[0518] According to an embodiment, encapsulating the V-PCC bitstream into a file is performed by Figure 1 File / fragment encapsulator 10003, Fig.18 Transmitter 18008, Fig. 20 File / fragment encapsulator 20004 or Fig.21 The file / segment encapsulator 21009 is executed. According to an embodiment, the file is decapsulated into a V-PCC bit stream by Figure 1 File / fragment decapsulator 10007, Fig.19 Receiver, Fig. 20 File / fragment decapsulator 20005 or Fig. 22 The file / fragment decapsulator 22000 executes.

[0519] Fig.25 : is a diagram showing the structure of a multi-track V-PCC file according to an embodiment. That is, it shows an example of the layout of an ISOBMFF-based file containing multiple tracks.

[0520] An ISOBMFF-based file according to an embodiment may be referred to as a container, a container file, a media file, a V-PCC file, etc. Specifically, a file may be composed of a box and / or information which may be referred to as ftyp, meta, moov, or mdat.

[0521] The ftyp box (file type box) can provide information related to the file compatibility or file type of the file. The receiving side can refer to the ftyp box to identify the file.

[0522] The meta box may include a vpcg{0,1,2,3} box (V-PCC group box), which will be described in detail below.

[0523] The mdat box, also called the media data box, may contain a video coded geometry bitstream, a video coded attribute bitstream, a video coded occupancy map bitstream, and a patch data group bitstream.

[0524] The moov box, also called a movie box, may contain metadata about the media data of the file (e.g., geometry bitstream, attribute bitstream, occupancy map bitstream, etc.). For example, it may contain information required to decode and play the media data, as well as information about the samples of the file. The moov box may be used as a container for all metadata. The moov box may be the highest level box among the metadata related boxes. Depending on the implementation, there may be only one moov box in a file.

[0525] According to an embodiment, in Fig.25 In the ISOBMFF container structure shown, V-PCC units included in the V-PCC bitstream are mapped to various tracks in the file based on their types.

[0526] based on Fig.25 With the layout shown, the ISOBMFF container for the V-PCC bitstream may include the following:

[0527] 1) V-PCC track. The V-PCC track contains sequence parameter sets and samples of payloads of V-PCC units that carry non-video coding information (e.g., type information indicating the type of the corresponding V-PCC unit in the V-PCC unit header indicates SPS or PDG). The V-PCC track may also provide track references to other tracks containing samples that carry payloads of V-PCC units that carry video compression information (e.g., type information in the V-PCC unit header indicates geometry video data, attribute video data, or occupancy map video data).

[0528] 2) One or more constrained video scheme tracks for geometry video data. The samples contained in these tracks contain NAL units of a video coding elementary stream for geometry video data. In this case, the type information in the V-PCC unit header indicates geometry video data, and the NAL unit corresponds to a payload carrying geometry video data.

[0529] 3) One or more restricted video scheme tracks for attribute video data. The samples contained in these tracks contain NAL units of a video coding elementary stream for attribute video data. In this case, the type information in the V-PCC unit header indicates attribute video data, and the NAL unit corresponds to a payload carrying the attribute video data.

[0530] 4) Constrained video scheme tracks for occupancy map data. The samples contained in these tracks contain NAL units of a video coded elementary stream for occupancy map video data. In this case, the type information in the V-PCC unit header indicates occupancy map video data, and the NAL unit corresponds to a payload carrying the occupancy map video data.

[0531] According to an embodiment, tracks comprising video coding elementary streams for geometry video data, attribute video data and occupancy map video data are referred to as component tracks.

[0532] According to an embodiment, synchronization between elementary streams contained in component tracks is handled by the ISOBMFF track timing structure (stts, ctts and cslg, or equivalent mechanisms in movie fragments). Samples contributing to the same point cloud frame across different video coding component tracks and V-PCC tracks can have the same composition time.

[0533] The V-PCC parameter set for such samples may have a decoding time equal to or earlier than the composition time of the frame.

[0534] ISOBMFF files carrying V-PCC content can be distinguished from the brand defined by V-PCC. Tracks of V-PCC content can be grouped within a VPCCGroupBox, which is a file-level EntityToGroupBox with a V-PCC specific grouping 4CC value ('vpcg'). The VPCCGroupBox can be used as an entry point to access V-PCC content in a container and contains initial metadata describing the V-PCC content. A VPCCGroupBox as an EntityToGroupBox can be contained in a MetaBox box or a Moov box.

[0535] An entity group can be a set of items that group a track. The entities in an entity group can share specific characteristics indicated by the grouping type, and can also have specific relationships.

[0536] Entity groups are indicated in the GroupsListBox. The GroupsListBox may be included in at least one of the file-level MetaBox, the movie-level MetaBox, or the track-level MetaBox. Entity groups specified in the GroupsListBox of the file-level MetaBox refer to track- or file-level items. Entity groups specified in the GroupsListBox of the movie-level MetaBox refer to movie-level items. Entity groups specified in the GroupsListBox of the track-level MetaBox refer to track-level items for that track.

[0537] GroupsListBox contains EntityToGroupBoxes, each specifying an entity group as follows.

[0538] Group List Box

[0539] Box type: 'grpl'

[0540] Container: MetaBox not contained in AdditionalMetadataContainerBox

[0541] Mandatory: No

[0542] Quantity: zero or one

[0543] The GroupsListBox contains the entity groups specified for the document. This box contains a complete set of boxes, each called EntityToGroupBox, with a four-character code representing the type of grouping defined.

[0544] The GroupsListBox should not appear in the AdditionalMetadataContainerBox.

[0545] When the GroupsListBox exists in a file-level MetaBox, there may not be an item_ID value in an ItemInfoBox in any file-level MetaBox that is equal to a track_ID value in any TrackHeaderBox.

[0546] aligned(8) class GroupsListBox extends box('grpl') {

[0547] }

[0548] Box Type: As specified below, with a grouping_type value of EntityToGroupBox

[0549] Container: GroupsListBox

[0550] Mandatory: No

[0551] Quantity: One or more

[0552] EntityToGroupBox specifies the entity group.

[0553] The box type (grouping_type) indicates the grouping type of the entity group. Each grouping_type code is associated with a semantic description of the grouping. The following grouping_type values ​​are specified:

[0554] 'altr': The items and tracks mapped to this group are alternatives to each other, and only one of them can be played (when the mapped items and tracks are part of the presentation, e.g., a displayable item or track) or processed in other ways (when the mapped items or tracks are not part of the presentation, but are e.g. metadata). The player can select the first entity from the list of entity_id values ​​that it can handle, e.g., decode and play the mapped items and tracks that are part of the presentation and suitable for the application needs. Any entity_id value should be mapped to only one group of type 'altr'. An alternative group of entities consists of the items and tracks mapped to the same entity group of type 'altr'.

[0555] NOTE: EntityToGroupBox contains grouping_type specific extensions.

[0556] aligned(8) class EntityToGroupBox(grouping_type, version, flags) extends FullBox(grouping_type, version, flags) {

[0557] unsigned int(32)group_id;

[0558] unsigned int(32)num_entities_in_group;

[0559] for(i=0;i <num_entities_in_group;i++)

[0560] unsigned int(32)entity_id;

[0561] }

[0562] According to an embodiment, group_id is a non-negative integer assigned to a specific grouping that is not equal to any group_id value of any other EntityToGroupBox, the containing GroupsListBox, or any track_ID value (when the GroupsListBox is included at the file level).

[0563] According to an embodiment, num_entities_in_group specifies the number of entity_id values ​​mapped to this entity group.

[0564] According to an embodiment, entity_id is resolved to item when an item with item_ID equal to entity_id exists in the hierarchical level (file, movie or track) containing the GroupsListBox, or to track when a track with track_ID equal to entity_id exists and the GroupsListBox is contained in the file level.

[0565] The V-PCC group box is described below.

[0566] Box type: 'vpcg'

[0567] Container: GroupListBox

[0568] Mandatory: Yes

[0569] Quantity: One or more

[0570] The V-PCC group box is included in the GroupsListBox and provides a list of tracks including V-PCC content.

[0571] V-PCC content specific information, such as attribute types and mapping of layers to related tracks, is listed in this box. This information provides a convenient way to get a preliminary understanding of V-PCC content. In order to flexibly configure V-PCC content that supports a variety of different client functions, multiple versions of coded V-PCC components are listed in this box. V-PCC defined archiving, hierarchy and level information are also included in this box.

[0572] aligned(8) class VPCCGroupBox() extends EntityToGroupBox('vpcg', version, flags) {

[0573] for(i=0;i <num_entities_in_group;i++){

[0574] unsigned int(4) data type;

[0575] unsigned int(4) attribute type;

[0576] unsigned int(1)multiple_layer_present_flag;

[0577] unsigned int(4)layer_count_munus1;

[0578] for(i=0;i <layer_count_minus1+1;i++){

[0579] unsigned int(4)layer_id;

[0580] }

[0581] unsigned int(32)entity_id;

[0582] }

[0583] unsigned int(4)PCC_layer_count_minus1;

[0584] vpcc_profile_tier_level()

[0585] }

[0586] According to an embodiment, data_type indicates the track type of PCC data in the reference track.

[0587] Fig.26 is a table showing examples of track types of PCC data assigned to data_type according to an embodiment. For example, data_type equal to 1 may indicate a V-PCC track. data_type equal to 2 may indicate a geometry video track. data_type equal to 3 may indicate an attribute video track. data_type equal to 4 may indicate an occupancy map video track.

[0588] According to an embodiment, multiple_layer_present_flag indicates whether the associated entity (or track) carries a single geometry or attribute layer or multiple geometry or attribute layers. For example, multiple_layer_present_flag equal to 0 indicates that a single geometry or attribute layer is carried in the associated entity (or track). Multiple_layer_present_flag equal to 1 indicates that multiple geometry or attribute layers are carried in the associated entity (or track). According to an embodiment, a V-PCC track (i.e., data_type indicates a V-PCC track) should have multiple_layer_present_flag equal to 0. As another example, when data_type indicates a geometry video track, multiple_layer_present_flag equal to 0 can indicate that the geometry video track carries a single geometry layer, and multiple_layer_present_flag equal to 1 indicates that the geometry video track carries multiple geometry layers. As another example, when data_type indicates an attribute video track, multiple_layer_present_flag equal to 0 can indicate that the attribute video track carries a single attribute layer, and multiple_layer_present_flag equal to 1 can indicate that the attribute video track carries multiple attribute layers.

[0589] According to an embodiment, layer_count_minus1 plus 1 indicates the number of geometry or attribute layers carried in the associated entity (or track). The layer_count_minus1 of the V-PCC track according to an embodiment is equal to 0. For example, when data_type indicates a geometry video track, layer_count_minus1 plus 1 may indicate the number of geometry layers carried in the geometry video track. As another example, when data_type indicates an attribute video track, layer_count_minus1 plus 1 may indicate the number of attribute layers carried by the attribute video track.

[0590] According to an embodiment, layer_id specifies the layer identifier of a geometry or attribute layer in the associated entity (or track). A V-PCC track according to an embodiment has layer_id equal to 0. The set of layer_id values ​​for any V-PCC component track type arranged in ascending order is a set of consecutive integers starting from 0. As an example, when data_type indicates a geometry video track, layer_id may indicate a layer identifier of a geometry layer. As another example, when data_type indicates an attribute video track, layer_id may indicate a layer identifier of an attribute layer.

[0591] According to an embodiment, pcc_layer_count_minus1 plus 1 indicates the number of layers used to encode the geometry and attribute components of the point cloud stream. For example, when data_type indicates a geometry video track, pcc_layer_count_munus1 plus 1 may indicate the number of layers used to encode the geometry component of the point cloud stream. As another example, when data_type indicates an attribute video track, pcc_layer_count_munus1 plus 1 may indicate the number of layers used to encode the attribute component of the point cloud stream.

[0592] According to an embodiment, attribute_type indicates the attribute type of the attribute video data carried in the reference entity (or track).

[0593] Fig. 27 is a table showing examples of attribute types assigned to attribute_type according to implementations. For example, attribute_type equal to 0 may indicate texture, attribute_type equal to 1 may indicate material ID. attribute_type equal to 2 may indicate transparency, attribute_type equal to 3 may indicate reflectivity, and attribute_type equal to 4 may indicate normals.

[0594] Depending on the implementation, entity_id specifies the identifier of the related item or track. When an item with item_ID equal to entity_id appears in the hierarchy level (file, movie or track) containing the GroupsListBox, it is resolved to an item, or when a track with track_ID equal to entity_id exists and the GroupsListBox is contained in the file level, it is resolved to a track.

[0595] vpcc_profile_tier_level() is the same as the profile_tier_level() specified in sequence_parameter_set().

[0596] Fig.28 An example of a syntax structure of profile_tier_level() according to an embodiment is shown.

[0597] The ptl_tier_flag field specifies the codec profile tier used to encode the V-PCC content.

[0598] The ptl_profile_idc field indicates the profile information to which the encoded point cloud sequence conforms.

[0599] The ptl_level_idc field indicates the level of the codec profile to which the encoded point cloud sequence conforms.

[0600] Next, the V-PCC track will be described.

[0601] The entry point of each V-PCC content can be represented by a unique V-PCC track. An ISOBMFF file can contain multiple V-PCC contents, so there can be multiple V-PCC tracks in the file. The V-PCC track should be identified by the volume visual media handler type "volm" in the HandlerBox of the MediaBox.

[0602] Box type: 'vohd'

[0603] Container: MediaInformationBox

[0604] Mandatory: Yes

[0605] Quantity: Exactly one

[0606] Volumetric tracks use VolumetricMediaHeaderBox in MediaInformationBox

[0607] aligned(8) class VolumetricMediaHeaderBox extends FullBox('vohd', version=0, 1) {

[0608] / / If we don't need anything here, use Null Media Header

[0609] }

[0610] / / Random access point of patch stream

[0611] The random access point of a V-PCC track may be a sample carrying an I-patch frame with an empty decoder buffer. To indicate the random access point of a V-PCC track, there is a syncsampleBox. The syncsample indicates a sync sample carrying an I-patch frame.

[0612] If some samples in a track (including any track fragments) are non-sync samples, then SyncSampleBox should be present in SampleTableBox, but the flag sample_is_non_sync_sample for samples in track fragments is valid and describes these samples even if SyncSampleBox is not present. If the track is not fragmented and SyncSampleBox is not present, then all samples in the track are sync samples.

[0613] Box type: 'stbl'

[0614] Container: MediaInformationBox

[0615] Mandatory: Yes

[0616] Quantity: Exactly one

[0617] aligned(8) class SampleTableBox extends Box('stbl') {

[0618] }

[0619] Box type: 'stss'

[0620] Container: SampleTableBox

[0621] Mandatory: No

[0622] Quantity: zero or one

[0623] This box provides a compact marker for sync samples within a stream. The table is arranged in strict ascending sample number order. If SyncSampleBox is not present, every sample is a sync sample.

[0624] aligned(8) class SyncSampleBox

[0625] Extend FullBox('stss', version=0,0) {

[0626] unsigned int(32)entry_count;

[0627] int i;

[0628] for(i=0;i <entry_count;i++){

[0629] unsigned int(32)sample_number;

[0630] }

[0631] }

[0632] 'version' is an integer representing the version of the box.

[0633] entry_count is an integer giving the number of entries in the following table. If entry_count is zero, there are no sync samples in the stream and the following table is empty.

[0634] sample_number gives the sample number for each sync sample in the stream. In particular, for each sync sample carrying an I-patch frame in a V-PCC track, sample_number gives the sample number.

[0635] A V-PCC track sample entry is described below.

[0636] Sample entry type: 'vpc1'

[0637] Container: SampleDescriptionBox('stsd')

[0638] Mandatory: "vpc1" sample entries are mandatory Quantity: one or more

[0639] Track sample entry type 'vpc1' is used. A VPCC track sample entry contains a VPCC configuration box, defined below. This includes a VPCCDDecoderConfigurationRecord defined. An optional BitRateBox may be present in a VPCC track sample entry to signal the bitrate information of the VPCC video stream.

[0640] aligned(8) class VPCCDDecoderConfigurationRecord{

[0641] unsigned int(8)configurationVersion=1;

[0642] unsigned int(8)numOfSequenceParameterSets;

[0643] for(i=0;i <numOfSequenceParameterSets;i++){

[0644] sequence_parameter_set();

[0645] }

[0646] / / Additional fields

[0647] }

[0648] class VPCCConfigurationBox extends Box('vpcc'){

[0649] VPCCDecoderConfigurationRecord()VPCCConfig;

[0650] }

[0651] aligned(8) class VPCCSampleEntry() extends VolumetricSampleEntry('vpc1') {

[0652] VPCCConfigurationBoxconfig;

[0653] }

[0654] class VolumetricSampleEntry(codingname) extends SampleEntry(codingname) {

[0655] }

[0656] configurationVersion is a version field. Incompatible changes to a record are indicated by a change in the version number.

[0657] numOfSequenceParameterSets represents the number of V-PCC sequence parameter sets signaled in the decoder configuration record.

[0658] Compressorname in the base class VisualSampleEntry indicates the name of the compressor recommended to be used with the value "\012VPCCCoding".

[0659] The V-PCC specification allows multiple instances of the VPCC_SPS element (with ids from 1 to 15). Therefore, a VPCCSampleEntry contains multiple sequence_parameter_set element payloads.

[0660] The V-PCC sample format is described below.

[0661] Each sample in the V-PCC track corresponds to a single point cloud frame.

[0662] The samples corresponding to the frame in the respective component tracks have the same composition time as the V-PCC track samples for the frame in the V-PCC track.

[0663] Each V-PCC sample contains only one V-PCC unit payload whose type information in the V-PCC unit header indicates a patch data group (PDG). The V-PCC unit payload includes one or more patch block group unit payloads, such as Fig.24 shown.

[0664] aligned(8) class vpcc_unit_payload_struct{

[0665] vpcc_unit_payload();

[0666] }

[0667] aligned(8) class VPCCSample{

[0668] vpcc_unit_payload_Struct();

[0669] }

[0670] vpcc_unit_payload() is the payload of the V-PCC unit of which the type information in the V-PCC unit header indicates PDG, and contains one instance of patch_data_group_unit().

[0671] For V-PCC tracks, samples carrying I-patch frames are defined as sync samples.

[0672] Next, the V-PCC track reference is described below.

[0673] To link the V-PCC track to the component video track, the track reference facility of the ISOBMFF standard is used. Three TrackReferenceTypeBoxes are added to a TrackReferenceBox inside the TrackBox of the V-PCC track, one for each component.

[0674] TrackReferenceTypeBox contains an array of track_IDs that specify the video tracks that the V-PCC track references.

[0675] The reference_type in TrackReferenceTypeBox identifies the type of component (ie, geometry, attribute, or occupancy map). The 4CCs for new track reference types include "pcca", "pccg", and "pcco".

[0676] In type "pcca", the referenced track contains the attribute V-PCC component of video coding.

[0677] In type "pccg", the referenced track contains the geometry V-PCC component of the video coding.

[0678] In type "pcco", the referenced track contains the video coded occupancy map V-PCC component.

[0679] Next, the V-PCC component track of video coding is described below.

[0680] Since it does not make sense to display decoded frames from attribute, geometry or occupancy map tracks without reconstructing the point cloud on the player side, a restricted video scheme type can be defined for these video coding tracks. The V-PCC video track should contain the 4CC identifier 'pccv'. The 4CC identifier 'pccv' is included in the scheme_type field of the SchemeTypeBox of the RestrictedSchemeInfoBox of the restricted sample entry.

[0681] The restricted video sample entry type 'resv' uses the V-PCC video scheme to indicate that the decoded picture contains attribute, geometry or occupancy map data of a point cloud.

[0682] The use of the V-PCC video scheme is indicated by scheme_type equal to 'pccv' (video-based point cloud video) in the SchemeTypeBox in the RestrictedSchemeInfoBox.

[0683] This box is a container for boxes containing V-PCC specific information indicating this track. VPCCVideoBox provides PCC specific parameters applicable to all samples in the track.

[0684] Box type: 'pccv'

[0685] Container: SchemeInformationBox

[0686] Mandatory: When scheme_type is equal to 'pccv', yes

[0687] Quantity: zero or one

[0688] aligned(8) class VPCCVideoBox extends FullBox('vpcc',0,0) {

[0689] unsigned int(4)data_type;

[0690] unsigned int(4)attribute_count;

[0691] for(int i=0; i <attribute_count+1;i++){

[0692] unsigned int(4)attribute_type;

[0693] }

[0694] unsigned int(1)multiple_layer_present_flag;

[0695] unsigned int(4)layer_count_minus1;

[0696] for(i=0;i <layer_count_minus1+1;i++){

[0697] unsigned int(4)layer_id;

[0698] }

[0699] }

[0700] data_type may indicate the type of data contained in a video sample included in a corresponding track.

[0701] Fig.29 is a table showing examples of the types of PCC data in reference tracks according to an embodiment. For example, data_type indicates a V-PCC track when equal to 1, a geometry video track when equal to 2, an attribute video track when equal to 3, and an occupancy map video track when equal to 4.

[0702] multiple_layer_present_flag indicates whether a single geometry or attribute layer or multiple geometry or attribute layers are carried in the track. For example, multiple_layer_present_flag equal to 0 may indicate that a single geometry or attribute layer is carried in the track. multiple_layer_present_flag equal to 1 may indicate that multiple geometry or attribute layers are carried in the track. As another example, when data_type indicates a geometry video track, multiple_layer_present_flag equal to 0 may indicate that the geometry video track carries a single geometry layer, and multiple_layer_present_flag equal to 1 may indicate that the geometry video track carries multiple geometry layers. As another example, when data_type indicates an attribute video track, multiple_layer_present_flag equal to 0 may indicate that the attribute video track carries a single attribute layer, and multiple_layer_present_flag equal to 1 may indicate that the attribute video track carries multiple attribute layers.

[0703] layer_count_minus1 plus 1 indicates the number of geometry and / or attribute layers carried in the track. For example, when data_type indicates a geometry video track, layer_count_minus1 plus 1 may indicate the number of geometry layers carried by the geometry video track. As another example, when data_type indicates an attribute video track, layer_count_minus1 plus 1 may indicate the number of attribute layers carried in the attribute video track.

[0704] layer_id specifies the layer identifier of the geometry or attribute layer associated with the samples in the track. As an example, when data_type indicates a geometry video track, layer_id may indicate the layer identifier of the geometry layer. As another example, when data_type indicates an attribute video track, layer_id may indicate the layer identifier of the attribute layer. attribute_count may indicate the number of attribute data of the point cloud stream contained in the track. One or more attribute data may be contained in one track. In this case, according to an embodiment, data_type indicates an attribute video track.

[0705] attribute_type indicates the attribute type of the attribute video data carried in the track.

[0706] Fig.30 is a table showing examples of attribute types according to an embodiment. For example, the attribute_type field indicates texture when equal to 0, indicates material ID when equal to 1, indicates transparency when equal to 2, indicates reflectivity when equal to 3, and indicates normal when equal to 4.

[0707] When a field value present in VPCCVideoBox is included in signaling associated with an item, information about the item (image) may be indicated.

[0708] Next, attribute sample grouping is described below.

[0709] When a track contains attribute data for a point cloud, it may contain data of one or more attribute types. In this case, the attribute type of the data contained in the samples contained in the corresponding track may be signaled as follows.

[0710] class PCCAttributeSampleGroupEntry extends VisualSampleGroupEntry('pcca') {

[0711] unsigned int(4)attribute_type;

[0712] }

[0713] attribute_type indicates the attribute type of the attribute video data carried in the associated sample in the track.

[0714] Fig.31 is a table showing examples of attribute types according to an embodiment. For example, the attribute_type field indicates texture when equal to 0, indicates material ID when equal to 1, indicates transparency when equal to 2, indicates reflectivity when equal to 3, and indicates normal when equal to 4.

[0715] Next, layer sample grouping is described below.

[0716] It may include data associated with one or more layers of the same data type (e.g., geometry data, attribute data) of the point cloud of the track. In this case, layer-related information about the data contained in the samples included in the track may be signaled as follows.

[0717] class PCCLayerSampleGroupEntry extends VisualSampleGroupEntry('pccl'){

[0718] unsigned int(4)pcc_layer_count_minus1;

[0719] unsigned int(4)layer_id;

[0720] }

[0721] pcc_layer_count_minus1 plus 1 indicates the number of layers used to encode the geometry and attribute components of the point cloud stream.

[0722] layer_id specifies the layer identifier of the associated sample in the track.

[0723] Next, the layer tracking group is described below.

[0724] From a V-PCC perspective, it is expected that, except for geometry, all other types of information (except occupancy and patch information) can have the same number of layers. In addition, all information (geometry / attributes) marked with the same layer index are related to each other. That is, information marked with the same layer index refers to the same reconstructed point in 3D space.

[0725] In order to construct attributes with layer index M, the geometry layer with index M should also be available. If that geometry layer is missing for some reason, such attribute information will be useless, as the two are "linked" together.

[0726] From a rendering point of view, such information can be considered to "enhance" the reconstructed point cloud. Note that this means that a reasonable reconstruction can still be obtained by discarding some layers. However, according to an embodiment, layer 0 is not discarded.

[0727] According to an embodiment, tracks belonging to the same layer have the same track_group_id value for track_group_type 'pccl', and the track_group_id of a track from one layer is different from the track_group_id of a track from any other layer.

[0728] By default, when track grouping is not indicated for any track in a file, the file is considered to contain only one layer of content.

[0729] aligned(8) class PCCLayerTrackGroupBox extends TrackGroupTypeBox('pccl'){

[0730] unsigned int(4)pcc_layer_count_minus1;

[0731] unsigned int(4)layer_id;

[0732] }

[0733] pcc_layer_count_minus1 plus 1 indicates the number of layers used to encode the geometry and / or attribute components of the point cloud stream.

[0734] layer_id specifies the layer identifier of the associated PCC data (e.g., geometry and / or attributes) layer carried in the track.

[0735] This can inform Fig. 20 Player or Figure 1 , Fig.19 , Fig. 20 or Fig. 22 For a renderer, data from tracks belonging to the same track group can be used to reconstruct the point cloud of the same layer.

[0736] As described above, in the present disclosure, signaling information for allowing a client / player to select and use an attribute bitstream as needed is signaled in at least one of a V-PCC track and an attribute track.

[0737] In addition, according to the present disclosure, signaling information about a random access point among samples in a V-PCC track is signaled in at least one of the V-PCC track and the attribute track.

[0738] Therefore, since the client / player is allowed to obtain signaling information related to the random access point and obtain and use the necessary attribute bitstream based on the signaling information, efficient and faster data processing can be achieved.

[0739] Furthermore, in the present disclosure, when geometry and / or attribute video data in a file consists of multiple layers, track grouping information used together is signaled in at least one of the V-PCC track and the corresponding video track (ie, the geometry track or the attribute track).

[0740] Furthermore, in the present disclosure, when geometry and / or attribute video data consists of multiple layers in one track, sample grouping information therefor is signaled in at least one of the V-PCC track and the corresponding video track (ie, the geometry track or the attribute track).

[0741] Furthermore, in the present disclosure, when a plurality of attribute video data are contained in one track, sample grouping information therefor is signaled in at least one of a V-PCC track and an attribute track.

[0742] The patch frame (or referred to as a point cloud object or atlas frame) as the target of the point cloud data can be divided into one or more tiles, as described above. The tiles according to the embodiment can represent a specific area in the 3D space or a specific area in the 2D plane. In addition, the tiles can be a cuboid in a bounding box, or a part of the patch frame or a sub-bounding box. In the present disclosure, the tiles can be divided into one or more tiles. Figure 1 Point cloud video encoder, Fig.18 Patch generator, Fig. 20 Point cloud preprocessing unit or Fig.21 The partitioning of a patch frame (or point cloud object) into one or more tiles may be performed by a patch generator of the image processing unit, or by a separate component / module.

[0743] Fig.32 An example of dividing a patch frame into a plurality of tiles by dividing the patch frame into one or more tile rows and one or more tile columns is shown. A tile is a rectangular area of ​​a patch frame, and a tile group may contain a plurality of tiles of a patch frame. In the present disclosure, a tile group contains a plurality of tiles of a patch frame that together form a rectangular (quadrilateral) area of ​​a patch frame. In particular, Fig.32 An example is shown in which a patch frame is divided into 24 tiles (6 tile columns and 4 tile rows) and 9 rectangular (quadrilateral) tile groups.

[0744] Here, when Fig.24When the type information in the V-PCC unit header indicates a patch data group (VPCC-PDG) including at least one patch, the corresponding V-PCC unit payload may include a patch data group. Conversely, when the V-PCC unit payload includes a patch data group, the type information in the V-PCC unit header may identify the same content.

[0745] The patch data set may include information about one or more patch tile sets and at least one of a patch sequence parameter set, a patch frame parameter set, a patch frame geometry parameter set, a geometry patch parameter set, a patch frame attribute parameter set, or an attribute patch parameter set.

[0746] Fig.33 An example of a syntax structure of each V-PCC unit according to an embodiment is shown. Each V-PCC unit consists of a V-PCC unit header and a V-PCC unit payload. Fig.33 The V-PCC unit may contain more data. In this case, it may also include a trailing_zero_8bits field. The trailing_zero_8bits field according to an embodiment is a byte corresponding to 0x00.

[0747] Fig.34 An example of the syntax structure of a V-PCC unit header according to an embodiment is shown. In an embodiment, Fig.34 The V-PCC unit header (vpcc_unit_header()) includes a vpcc_unit_type field. The vpcc_unit_type field indicates the type of the V-PCC unit.

[0748] Fig.35 An example of the type of V-PCC unit assigned to the vpcc_unit_type field according to an embodiment is shown.

[0749] Reference Fig.35 According to an embodiment, a vpcc_unit_type field set to 0 indicates that the data contained in the V-PCC unit payload of the V-PCC unit is a sequence parameter set (VPCC_SPS). A vpcc_unit_type field set to 1 indicates that the data is a patch data group (VPCC_PDG). A vpcc_unit_type field set to 2 indicates that the data is occupied video data (VPCC_OVD). A vpcc_unit_type field set to 3 indicates that the data is attribute video data (VPCC_AVD). A vpcc_unit_type field set to 4 indicates that the data is geometry video data = (VPCC_GVD).

[0750] Those skilled in the art may easily change the meaning, order, deletion, addition, etc. of the values ​​assigned to the vpcc_unit_type field, and thus the present disclosure shall not be limited to the above-described embodiments.

[0751] Here, the V-PCC unit payload conforms to the format of the HEVC NAL unit. That is, according to the value of the vpcc_unit_type field, the occupancy, geometry, and attribute video data V-PCC unit payload corresponds to a video data unit decodable by a specified video decoder in the corresponding occupancy, geometry, and attribute parameter set V-PCC unit (e.g., HEVC NAL unit).

[0752] According to an embodiment, when the vpcc_unit_type field indicates attribute video data (VPCC_AVD), geometry video data (VPCC_GVD), occupied temporal video data (VPCC_OVD), or patch data group (VPCC_PDG), the V-PCC unit header further includes a vpcc_sequence_parameter_set_id field.

[0753] The vpcc_sequence_parameter_set_id field specifies an identifier (ie, sps_sequence_parameter_set_id) of an active sequence parameter set (VPCC SPS). The value of the vpcc_sequence_parameter_set_id field may be in the range of 0 to 15, inclusive.

[0754] According to an embodiment, when the vpcc_unit_type field indicates attribute video data (VPCC_AVD), the V-PCC unit header further includes a vpcc_attribute_index field and a vpcc_attribute_dimension_index field.

[0755] The vpcc_attribute_index field indicates the index of attribute video data carried in the attribute video data unit. The value of the vpcc_attribute_index field may be in the range of 0 to (ai_attribute_count-1), inclusive.

[0756] The vpcc_attribute_dimension_index field indicates the index of the attribute dimension group carried in the attribute video data unit. The value of the vpcc_attribute_dimension_index field may be in the range of 0 to 127, inclusive.

[0757] Fig.34 The sps_multiple_layer_streams_present_flag field in the V-PCC unit header indicates whether the vpcc_layer_index field and the pcm_separate_video_data(11) field are included.

[0758] For example, if the value of the vpcc_unit_type field indicates attribute video data (VPCC_AVD) and the value of the sps_multiple_layer_streams_present_flag field is true (e.g., 0), a vpcc_layer_index field and a pcm_separate_video_data(11) field are further included in the V-PCC unit header. That is, when the value of the sps_multiple_layer_streams_present_flag field is true, it indicates that there are multiple layers for attribute video data or geometry video data. In this case, a field indicating the index of the current layer (e.g., vpcc_layer_index) is required.

[0759] The vpcc_layer_index field indicates the index of the current layer of the attribute video data. The value of the vpcc_layer_index field is in the range of 0 to 15.

[0760] For example, when the value of the vpcc_unit_type field indicates attribute video data (VPCC_AVD) and the value of the sps_multiple_layer_streams_present_flag field is false (e.g., 1), the pcm_separate_video_data(15) field is further included in the V-PCC unit header. That is, the sps_multiple_layer_streams_present_flag field is set to false to indicate that there are no multiple layers for attribute video data and / or geometry video data. In this case, the field indicating the index of the current layer is not required.

[0761] For example, when a value of the vpcc_unit_type field indicates geometry video data (VPCC_GVD) and a value of the sps_multiple_layer_streams_present_flag field is true (eg, 0), a vpcc_layer_index field and a pcm_separate_video_data (18) field are further included in the V-PCC unit header.

[0762] The vpcc_layer_index field indicates the index of the current layer of the geometry video data. The value of the vpcc_layer_index field is in the range of 0 to 15.

[0763] For example, when a value of the vpcc_unit_type field indicates geometry video data (VPCC_GVD) and a value of the sps_multiple_layer_streams_present_flag field is false (eg, 1), a pcm_separate_video_data (22) field is further included in the V-PCC unit header.

[0764] For example, if the value of the vpcc_unit_type field indicates occupied video data (VPCC_OVD) or patch data group (VPCC_PDG), a vpcc_reserved_zero_23bits field is further included in the V-PCC unit header. Otherwise, a vpcc_reserved_zero_27bits field is further included.

[0765] Fig.34 The V-PCC unit header may further include a vpcc_pcm_video_flag field.

[0766] For example, a vpcc_pcm_video_flag field set to 1 may indicate that the associated geometry or attribute video data unit contains only PCM (Pulse Code Modulation) code points. As another example, a vpcc_pcm_video_flag field set to 0 may indicate that the associated geometry or attribute video data unit may contain non-PCM code points. When the vpcc_pcm_video_flag field is not present, the value of the field may be inferred to be equal to 0.

[0767] Fig.36 An example of a syntax structure of pcm_separate_video_data(bitCount) included in a V-PCC unit header according to an embodiment is shown.

[0768] exist Fig.36 In the example, the bitCount of pcm_separate_video_data is as described above. Fig.34 The value of the vpcc_unit_type field in the V-PCC unit header varies.

[0769] If the value of the sps_pcm_separate_video_present_flag field is true and the vpcc_layer_index field is not, pcm_separate_video_data(bitCount) includes a vpcc_pcm_video_flag field and a vpcc_reserved_zero_bitcount_bits field. Otherwise, it may include a vpcc_reserved_bits field.

[0770] The vpcc_pcm_video_flag field set to 1 may indicate that the associated geometry or attribute video data unit is only PCM code point video. The vpcc_pcm_video_flag field set to 0 may indicate that the associated geometry or attribute video data unit may contain PCM code points. When the vpcc_pcm_video_flag field does not exist, its value may be considered to be 0.

[0771] Fig.37 An example of a syntax structure of a V-PCC unit payload according to an embodiment is shown.

[0772] Fig.37 The V-PCC unit payload contains one of a sequence parameter set (sequence_parameter_set()), a patch data group (patch_data_group()), and a video data unit (video_data_unit()) according to the value of the vpcc_unit_type field in the V-PCC unit header.

[0773] For example, according to an embodiment, when the vpcc_unit_type field indicates a sequence parameter set (VPCC_SPS), the V-PCC unit payload includes sequence_parameter_set(). When the vpcc_unit_type field indicates a patch data group (VPCC_PDG), the V-PCC unit payload includes patch_data_group(). In addition, when the vpcc_unit_type field indicates occupied video data (VPCC_OVD), the V-PCC unit payload includes an occupied video data unit (video_data_unit()) carrying occupied video data. When the vpcc_unit_type field indicates geometric video data (VPCC_GVD), the V-PCC unit payload includes a geometric video data unit (video_data_unit()) carrying geometric video data. When the vpcc_unit_type field indicates attribute video data (VPCC_AVD), the V-PCC unit payload includes an attribute video data unit (video_data_unit()).

[0774] Fig.38 An example of a syntax structure of sequence_parameter_set() included in a V-PCC unit payload according to an embodiment is shown.

[0775] Fig.38 A sequence parameter set (SPS) may be applied to a coded point cloud sequence comprising a sequence of coded geometry video data units, attribute video data units, and occupancy video data units.

[0776] Fig.38 A sequence parameter set (SPS) may include a profile_tier_level(), a sps_sequence_parameter_set_id field, a sps_frame_width field, a sps_frame_height field, and a sps_avg_frame_rate_present_flag field.

[0777] profile_tier_level() specifies the codec information used to compress SPS.

[0778] The sps_sequence_parameter_set_id field provides an identifier of the VPCC SPS referenced by other syntax elements.

[0779] The sps_frame_width field is the nominal frame width expressed in integer luma samples.

[0780] The sps_frame_height field is the nominal frame height expressed in integer luma samples.

[0781] The sps_avg_frame_rate_present_flag field indicates whether the average nominal frame rate information is included in the bitstream. For example, the sps_avg_frame_rate_present_flag field set to 0 may indicate that there is no average nominal frame rate information in the bitstream. The sps_avg_frame_rate_present_flag field set to 1 may indicate that the average nominal frame rate information should be indicated in the bitstream. For example, when the value of the sps_avg_frame_rate_present_flag field is true, i.e., 1, the sequence parameter set may also include a sps_avg_frame_rate field, a sps_enhanced_occupancy_map_for_depth_flag field, and a sps_geometry_attribute_different_layer_flag field.

[0782] The sps_avg_frame_rate field indicates the average nominal point cloud frame rate, in units of point cloud frames per 256 seconds. When the sps_avg_frame_rate field is not present, the value of this field shall be inferred to be equal to 0. During the reconstruction phase, the decoded occupancy, geometry, and attribute videos may be converted to the nominal width, height, and frame rate using appropriate scaling.

[0783] The sps_enhanced_occupancy_map_for_depth_flag field indicates whether the decoded occupancy map video contains information about whether the intermediate depth positions between two depth layers are occupied. For example, the sps_enhanced_occupancy_map_for_depth_flag field set to 1 may indicate that the decoded occupancy map video contains information about whether the intermediate depth positions between two depth layers are occupied. The sps_enhanced_occupancy_map_for_depth_flag field set to 0 may indicate that the decoded occupancy map video does not contain information about whether the intermediate depth positions between two depth layers are occupied.

[0784] The sps_geometry_attribute_different_layer_flag field indicates whether the number of layers used for encoding of geometry and attribute video data is different. For example, the sps_geometry_attribute_different_layer_flag field set to 1 may indicate that the number of layers used for encoding of geometry and attribute video data is different. For example, two layers may be used to encode geometry video data, and one layer may be used to encode attribute video data. In addition, the sps_geometry_attribute_different_layer_flag field set to 1 may indicate whether the number of layers used for encoding of geometry and attribute data is signaled in the patch sequence data unit.

[0785] The sps_geometry_attribute_different_layer_flag field indicates whether the sps_layer_count_geometry_minus1 field and the sps_layer_count_minus1 field are included. For example, when the value of the sps_geometry_attribute_different_layer_flag field is true (eg, 1), the sps_layer_count_geometry_minus1 field may be further included. When the value is false (eg, 0), the sps_layer_count_minus1 field may be further included.

[0786] The sps_layer_count_geometry_minus1 field indicates the number of layers used for encoding of geometry data.

[0787] The sps_layer_count_minus1 field indicates the number of layers used to encode geometry and attribute video data.

[0788] When the value of the sps_layer_count_minus1 field is greater than 0, the sequence parameter set may further include a sps_multiple_layer_streams_present_flag field and a sps_layer_absolute_coding_enabled_flag[0]=1 field.

[0789] The sps_multiple_layer_streams_present_flag field indicates whether the geometry layers or attribute layers are placed in a single video stream or in separate video streams. For example, the sps_multiple_layer_streams_present_flag field is set to 0 to indicate that all geometry layers or attribute layers are placed in a single geometry or attribute video stream, respectively. The sps_multiple_layer_streams_present_flag field set to 1 can indicate that all geometry or attribute layers are placed in separate video streams.

[0790] In addition, the sequence parameter set (SPS) includes an iteration statement that is repeated as many times as the value of the sps_layer_count_minus1 field. The iteration statement includes a sps_layer_absolute_coding_enabled_flag field. According to an embodiment, a variable i is initialized to 0 and incremented by 1 each time the iteration statement is executed. The iteration statement is repeated until i is equal to the value of the sps_layer_count_minus1 field. In addition, if the value of the sps_layer_absolute_coding_enabled_flag field is 0 and the value of i is greater than 0, the sps_layer_predictor_index_diff field is further included. Otherwise, the sps_layer_predictor_index_diff field is not included.

[0791] The sps_layer_absolute_coding_enabled_flag[i] field set to 1 may indicate that the geometry layer with index i is coded without any form of layer prediction. The sps_layer_absolute_coding_enabled_flag[i] field set to 0 may indicate that the geometry layer with index i is first predicted from another earlier coded layer before coding.

[0792] When the sps_layer_absolute_coding_enabled_flag[i] field is set to 0, the sps_layer_predictor_index_diff[i] field is used to calculate the predictor of the geometry layer with index i.

[0793] The SPS according to the present disclosure may further include a sps_pcm_patch_enabled_flag field. The sps_pcm_patch_enabled_flag field indicates whether the sps_pcm_separate_video_present_flag field, occupancy_parameter_set(), geometry_parameter_set(), and sps_attribute_count fields are included. For example, when the sps_pcm_patch_enabled_flag field is set to 1, the sps_pcm_separate_video_present_flag field, occupancy_parameter_set(), geometry_parameter_set(), and sps_attribute_count fields may be further included. That is, the sps_pcm_patch_enabled_flag field is set to 1 to indicate that a patch with a PCM coding point may be present in the bitstream.

[0794] The sps_pcm_separate_video_present_flag field indicates whether PCM-encoded geometry video data and attribute video data are stored in separate video streams. For example, the sps_pcm_separate_video_present_flag field is set to 1 to indicate that PCM-encoded geometry video data and attribute video data may be stored in separate video streams.

[0795] occupancy_parameter_set() contains information about the occupancy map.

[0796] geometry_parameter_set() contains information about the geometry of the video data.

[0797] The sps_attribute_count field indicates the number of attributes associated with the point cloud.

[0798] In addition, the SPS according to the present disclosure includes an iteration statement, which is repeated as many times as the value of the sps_attribute_count field. According to an embodiment, when the sps_geometry_attribute_different_layer_flag field is set to 1, the iteration statement includes the sps_layer_count_attribute_minus1 field and the attribute_parameter_set(). In the iteration statement, i is initialized to 0 and incremented by 1 each time the iteration statement is executed. The iteration statement is repeated until i is equal to the value of the sps_attribute_count field.

[0799] The sps_layer_count_attribute_minus1[i] field indicates the number of layers used to encode the i-th attribute video data associated with the point cloud.

[0800] attribute_parameter_set(i) contains information about the i-th attribute video data associated with the point cloud.

[0801] According to an embodiment of the present disclosure, the SPS further includes a sps_patch_sequence_orientation_enabled_flag field, a sps_patch_inter_prediction_enabled_flag field, a sps_pixel_deinterleaving_flag field, a sps_point_local_reconstruction_enabled_flag field, a sps_remove_duplicate_point_enabled_flag field, and a byte_alignment() field.

[0802] The sps_patch_sequence_orientation_enabled_flag field indicates whether flexible orientation is signaled in the patch sequence data unit. For example, the sps_patch_sequence_orientation_enabled_flag field is set to 1 to indicate that flexible orientation is signaled in the patch sequence data unit. The sps_patch_sequence_orientation_enabled_flag field is set to 0 to indicate that flexible orientation is not signaled.

[0803] The sps_patch_inter_prediction_enabled_flag field set to 1 indicates that inter prediction by using patch information from a previously encoded patch frame can be used.

[0804] The sps_pixel_deinterleaving_flag field set to 1 indicates that the decoded geometry and attribute video corresponding to a single stream contains interleaved pixels from two layers. The sps_pixel_deinterleaving_flag field set to 0 indicates that the decoded geometry and attribute video corresponding to a single stream contains pixels from only a single layer.

[0805] The sps_point_local_reconstruction_enabled_flag field set to 1 indicates that the local reconstruction mode is used during the point cloud reconstruction process.

[0806] The sps_remove_duplicate_point_enabled_flag field set to 1 indicates that duplicate points should not be reconstructed. Here, a duplicate point is a point that has the same 2D and 3D geometric coordinates as another point from a lower layer.

[0807] Fig.39 An example of a syntax structure of a patch data group (patch_data_group()) according to an embodiment is shown.

[0808] As described above, when the value of the vpcc_unit_type field in the V-PCC unit header indicates a patch data group, Fig.37 The V-PCC unit payload contains Fig.39 patch_data_group().

[0809] According to one embodiment, patch_data_group() includes a pdg_unit_type field, a pdg_data_group_unit_payload(pdg_unit_type) having signaling information that varies according to a value of the pdg_unit_type field, and a pdg_terminate_patch_data_group_flag field.

[0810] The pdg_unit_type field indicates the type of patch data group.

[0811] The pdg_terminate_patch_data_group_flag field indicates the end of the patch data group. The pdg_terminate_patch_data_group_flag field set to 0 may indicate that there are additional patch data group units in the patch data group. The pdg_terminate_patch_data_group_flag field set to 1 may indicate that there are no more patch data group units in the patch data group and this is the end of the current patch data group unit.

[0812] Fig.40 Is to show the assignment to Fig.39 A table of examples of the types of patch data groups of the pdg_unit_type field of the patch data group.

[0813] For example, a pdg_unit_type field set to 0 may indicate a patch sequence parameter set (PDG_PSPS). A pdg_unit_type field set to 1 may indicate a patch frame parameter set (PDG_PFPS). A pdg_unit_type field set to 2 may indicate a patch frame geometry parameter set (PDG_PFGPS). A pdg_unit_type field set to 3 may indicate a patch frame attribute parameter set (PDG_PFAPS). A pdg_unit_type field set to 4 may indicate a geometry patch parameter set (PDG_GPPS). A pdg_unit_type field set to 5 may indicate an attribute patch parameter set (PDG_APPS). A pdg_unit_type field set to 6 may indicate a patch tile group layer unit (PDG_PTGLU). A pdg_unit_type field set to 7 may indicate a prefix SEI message (PDG_PREFIX_SEI). The pdg_unit_type field set to 8 may indicate a suffix SEI message (PDG_SUFFIX_SEI).

[0814] PDG_PSPS may include sequence level parameters, and PDG_PFPS may include frame level parameters. PDG_PFGPS may include frame level geometry type parameters, and PDG_PFAPS may include frame level attribute type parameters. PDG_GPPS may include patch level geometry type parameters, and PDG_APPS may include patch level attribute type parameters.

[0815] Fig.41 An example of a syntax structure of a patch data group unit payload (patch_data_group_unit_payload(pdg_unit_type)) according to an embodiment is shown.

[0816] when Fig.39 When the value of the pdg_unit_type field of the patch data group indicates PDG_PSPS, patch_data_group_unit_payload() may include a patch sequence parameter set (patch_sequence_parameter_set()).

[0817] When the value of the pdg_unit_type field indicates PDG_GPPS, patch_data_group_unit_payload() may include a geometry patch parameter set (geometry_patch_parameter_set()).

[0818] When the value of the pdg_unit_type field indicates PDG_APPS, patch_data_group_unit_payload() may contain an attribute patch parameter set (attribute_patch_parameter_set()).

[0819] When the value of the pdg_unit_type field indicates PDG_PFPS, patch_data_group_unit_payload() may contain a patch frame parameter set (patch_frame_parameter_set()).

[0820] When the value of the pdg_unit_type field indicates PDG_PFAPS, patch_data_group_unit_payload() may include a patch frame attribute parameter set (patch_frame_attribute_parameter_set()).

[0821] When the value of the pdg_unit_type field indicates PDG_PFGPS, patch_data_group_unit_payload() may include a patch frame geometry parameter set (patch_frame_geometry_parameter_set()).

[0822] When the value of the pdg_unit_type field indicates PDG_PTGLU, patch_data_group_unit_payload() may contain a patch tile group layer unit (patch_tile_group_layer_unit()).

[0823] When the value of the pdg_unit_type field indicates PDG_PREFIX_SEI or PDG_SUFFIX_SEI, patch_data_group_unit_payload() may contain sei_message().

[0824] Fig.42 An example of a syntax structure of a supplemental enhancement information (SEI) message (sei_message()) according to an embodiment is shown. That is, when included in Fig.39 When the value of the pdg_unit_type field in the patch data group indicates PDG_PREFIX_SEI or PDG_SUFFIX_SEI, Fig.41 The patch_data_group_unit_payload() contains sei_message().

[0825] Each SEI message according to an embodiment consists of a SEI message header and a SEI message payload. The SEI message header includes a sm_payload_type_byte field and a sm_payload_size_byte field.

[0826] The sm_payload_type_byte field is a byte of the payload type of the SEI message. For example, it may be identified whether the SEI message is a prefix SEI message or a suffix SEI message based on the value of the sm_payload_type_byte field.

[0827] The sm_payload_size_byte field is the bytes of the payload size of the SEI message.

[0828] Fig.42 The SEI message initializes PayloadType to 0, and then sets the value of the sm_payload_type_byte field in the iteration statement to the value of PayloadType. When the value of the sm_payload_type_byte field is 0xFF, the iteration statement terminates.

[0829] also, Fig.42 The SEI message initializes PayloadSize to 0, and then sets the value of the sm_payload_size_byte field in the iteration statement to the value of PayloadSize. When the value of the sm_payload_size_byte field is 0xFF, the iteration statement terminates.

[0830] Then, information corresponding to the payloadType and payloadSize set in the two iteration statements is signaled through the payload (sei_payload(payloadType, payloadSize)) of the SEI message.

[0831] Fig.43 An example of a V-PCC bitstream structure according to other embodiments of the present disclosure is shown. In an embodiment, Fig.43 The V-PCC bitstream is composed of Figure 1 , Figure 4 , Fig.18 , Fig. 20 or Fig.21 Generation and output of V-PCC based point cloud video encoder.

[0832] A V-PCC bitstream containing a coded point cloud sequence (CPCS) according to an embodiment may be composed of a sample stream V-PCC unit. The sample stream V-PCC unit carries V-PCC parameter set (VPS) data, an atlas bitstream, an occupancy map bitstream for 2D video coding, a geometry bitstream for 2D video coding, and zero or more attribute bitstreams for 2D video coding.

[0833] exist Fig.43 In the embodiment of the present invention, the V-PCC bitstream may include a sample stream V-PCC header 40010 and one or more sample stream V-PCC units 40020. For simplicity, one or more sample stream V-PCC units 40020 may be referred to as a sample stream V-PCC payload. In other words, the sample stream V-PCC payload may be referred to as a collection of sample stream V-PCC units.

[0834] Each sample stream V-PCC unit 40021 may include V-PCC unit size information 40030 and a V-PCC unit 40040. The V-PCC unit size information 40030 indicates the size of the V-PCC unit 40040. For simplicity, the V-PCC unit size information 40030 may be referred to as a sample stream V-PCC unit header, and the V-PCC unit 40040 may be referred to as a sample stream V-PCC unit payload.

[0835] Each V-PCC unit 40040 may include a V-PCC unit header 40041 and a V-PCC unit payload 40042 .

[0836] In the present disclosure, data contained in a V-PCC unit payload 40042 is distinguished by a V-PCC unit header 40041. To this end, the V-PCC unit header 40041 contains type information indicating the type of the V-PCC unit. According to the type information in the V-PCC unit header 40041, each V-PCC unit payload 40042 may contain geometry video data (i.e., geometry bitstream for 2D video encoding), attribute video data (i.e., attribute bitstream for 2D video encoding), occupancy video data (i.e., occupancy map bitstream for 2D video encoding), atlas data, or a V-PCC parameter set (VPS).

[0837] A VPS according to an embodiment is also referred to as a sequence parameter set (SPS). These two terms can be used interchangeably.

[0838] Atlas data according to an embodiment may mean data consisting of attributes (eg, texture (patch)) and / or depth of point cloud data, and may also be referred to as a patch data group.

[0839] Fig.44 An example of data carried by a sample stream V-PCC unit in a V-PCC bitstream according to an embodiment is shown.

[0840] exist Fig.44 In the example, the V-PCC bitstream includes a sample stream V-PCC unit carrying a V-PCC parameter set (VPS), a sample stream V-PCC unit carrying atlas data (AD), a sample stream V-PCC unit carrying occupied video data (OVD), a sample stream V-PCC unit carrying geometric video data (GVD), and a sample stream V-PCC unit carrying attribute video data (AVD).

[0841] According to an embodiment, each sample stream V-PCC unit contains one type of V-PCC unit among VPS, AD, OVD, GVD and AVD.

[0842] Fig.45 An example of a syntax structure of a sample stream V-PCC header included in a V-PCC bitstream according to an embodiment is shown.

[0843] The sample_stream_v-pcc_header() according to an embodiment may include a ssvh_unit_size_precision_bytes_minus1 field and a ssvh_reserved_zero_5bits field.

[0844] The value of the ssvh_unit_size_precision_bytes_minus1 field plus 1 may specify the precision (in bytes) of the ssvu_vpcc_unit_size element in all sample stream V-PCC units. The value of this field may be in the range of 0 to 7.

[0845] The ssvh_reserved_zero_5bits field is reserved for future use.

[0846] Fig.46 An example of a syntax structure of a sample stream V-PCC unit (sample_stream_vpcc_unit()) according to an embodiment is shown.

[0847] The content of each sample stream V-PCC unit is associated with the same access unit as the V-PCC units contained in the sample stream V-PCC unit.

[0848] sample_stream_vpcc_unit() according to an embodiment may include a ssvu_vpcc_unit_size field and vpcc_unit (ssvu_vpcc_unit_size).

[0849] The ssvu_vpcc_unit_size field corresponds to Fig.43 The V-PCC unit size information 40030 specifies the size of the subsequent vpcc_unit in bytes. The number of bits used to represent the ssvu_vpcc_unit_size field is equal to (ssvh_unit_size_precision_bytes_minus1+1)*8.

[0850] vpcc_unit (ssvu_vpcc_unit_size) has a length corresponding to the value of the ssvu_vpcc_unit_size field and carries one of VPS, AD, OVD, GVD, and AVD.

[0851] Fig.47 An example of a syntax structure of a V-PCC unit according to an embodiment is shown. A V-PCC unit consists of a V-PCC unit header (vpcc_unit_header()) and a V-PCC unit payload (vpcc_unit_payload()). A V-PCC unit according to an embodiment may contain more data. In this case, it may also include a trailing_zero_8bits field. The trailing_zero_8bits field according to an embodiment is a byte corresponding to 0x00.

[0852] Fig.48 An example of the syntax structure of a V-PCC unit header according to an embodiment is shown. In an embodiment, Fig.48 The vpcc_unit_header() of the embodiment includes a vuh_unit_type field. The vuh_unit_type field indicates the type of the corresponding V-PCC unit. The vuh_unit_type field according to an embodiment is also called a vpcc_unit_type field.

[0853] Fig.49 An example of a V-PCC unit type assigned to the vuh_unit_type field according to an embodiment is shown.

[0854] Reference Fig.49 , according to an embodiment, a vuh_unit_type field set to 0 indicates that the data contained in the V-PCC unit payload of the V-PCC unit is a V-PCC parameter set (VPCC_VPS). A vuh_unit_type field set to 1 indicates that the data is atlas data (VPCC_AD). A vuh_unit_type field set to 2 indicates that the data is occupied video data (VPCC_OVD). A vuh_unit_type field set to 3 indicates that the data is geometric video data (VPCC_GVD). A vuh_unit_type field set to 4 indicates that the data is attribute video data (VPCC_AVD).

[0855] Those skilled in the art may easily change the meaning, order, deletion, addition, etc. of the values ​​assigned to the vuh_unit_type field, and thus the present disclosure shall not be limited to the above-described embodiments.

[0856] When the vuh_unit_type field indicates VPCC_AVD, VPCC_GVD, VPCC_OVD, or VPCC_AD, the V-PCC unit header according to an embodiment may further include a vuh_vpcc_parameter_set_id field and a vuh_atlas_id field.

[0857] The vuh_vpcc_parameter_set_id field specifies the value of the vps_vpcc_parameter_set_id of the active V-PCCVPS.

[0858] The vuh_atlas_id field specifies the index of the atlas corresponding to the current V-PCC unit.

[0859] When the vuh_unit_type field indicates VPCC_AVD, the V-PCC unit header according to an embodiment may further include a vuh_attribute_index field, a vuh_attribute_dimension_index field, a vuh_map_index field, and a vuh_raw_video_flag field.

[0860] The vuh_attribute_index field indicates the index of attribute data carried in the attribute video data unit.

[0861] The vuh_attribute_dimension_index field indicates the index of the attribute dimension group carried in the attribute video data unit.

[0862] When present, the vuh_map_index field may indicate the map index of the current geometry or attribute stream.

[0863] The vuh_raw_video_flag field may indicate whether RAW code points are included. For example, a vuh_raw_video_flag field set to 1 may indicate that the associated attribute video data unit contains only RAW code points. As another example, a vuh_raw_video_flag field set to 0 may indicate that the associated attribute video data unit may contain RAW code points. When the vuh_raw_video_flag field does not exist, its value may be inferred to be equal to 0. According to an embodiment, RAW code points are also referred to as pulse code modulation (PCM) code points.

[0864] When the vuh_unit_type field indicates VPCC_GVD, the V-PCC unit header according to an embodiment may further include a vuh_map_index field, a vuh_raw_video_flag field, and a vuh_reserved_zero_12bits field.

[0865] When present, the vuh_map_index field indicates the index of the current geometry stream.

[0866] The vuh_raw_video_flag field may indicate whether RAW code points are included. For example, a vuh_raw_video_flag field set to 1 may indicate that the associated geometry video data unit contains only RAW code points. As another example, a vuh_raw_video_flag field set to 0 may indicate that the associated geometry video data unit may contain RAW code points. When the vuh_raw_video_flag field does not exist, its value may be inferred to be equal to 0. According to an embodiment, RAW code points are also referred to as PCM code points.

[0867] The vuh_reserved_zero_12bits field is reserved for future use.

[0868] If the vuh_unit_type field indicates VPCC_OVD or VPCC_AD, the V-PCC unit header according to an embodiment may further include a vuh_reserved_zero_17bits field. Otherwise, the V-PCC unit header may further include a vuh_reserved_zero_27bits field.

[0869] The vuh_reserved_zero_17bits field and the vuh_reserved_zero_27bits field are reserved fields for future use.

[0870] Fig.50 An example of a syntax structure of a V-PCC unit payload (vpcc_unit_payload()) according to an embodiment is shown.

[0871] Fig.50 The V-PCC unit payload may contain one of a V-PCC parameter set (vpcc_parameter_set()), an atlas sub-bitstream (atlas_sub_bitstream()), and a video sub-bitstream (video_sub_bitstream()) according to a value of the vuh_unit_type field in the V-PCC unit header.

[0872] For example, when the vuh_unit_type field indicates VPCC_VPS, the V-PCC unit payload contains vpcc_parameter_set(), which contains overall coding information about the bitstream. When the vuh_unit_type field indicates VPCC_AD, the V-PCC unit payload contains atlas_sub_bitstream() carrying atlas data. In addition, according to an embodiment, when the vuh_unit_type field indicates VPCC_OVD, the V-PCC unit payload contains an occupied video sub-bitstream (video_sub_bitstream()) carrying occupied video data. When the vuh_unit_type field indicates VPCC_GVD, the V-PCC unit payload contains a geometry video sub-bitstream (video_sub_bitstream()) carrying geometry video data. When the vuh_unit_type field indicates VPCC_AVD, the V-PCC unit payload contains an attribute video sub-bitstream (video_sub_bitstream()) carrying attribute video data.

[0873] According to an embodiment, the atlas sub-bitstream may be referred to as an atlas sub-stream, and the occupied video sub-bitstream may be referred to as an occupied video sub-stream. The geometry video sub-bitstream may be referred to as a geometry video sub-stream, and the attribute video sub-bitstream may be referred to as an attribute video sub-stream. The V-PCC unit payload according to an embodiment conforms to the format of a High Efficiency Video Coding (HEVC) Network Abstraction Layer (NAL) unit.

[0874] Fig.51 is a diagram showing an example of the structure of an Atlas substream according to an embodiment. In an embodiment, Fig.51 The atlas substream conforms to the format of HEVC NAL units.

[0875] The Atlas substream according to an embodiment may be composed of a sample stream NAL unit containing an Atlas sequence parameter set (ASPS), a sample stream NAL unit containing an Atlas frame parameter set (AFPS), and one or more sample stream NAL units containing information about one or more Atlas tile groups and / or one or more sample stream NAL units containing one or more SEI messages.

[0876] The one or more SEI messages according to an embodiment may include a prefix SEI message and a suffix SEI message.

[0877] The atlas substream according to an embodiment may further include a sample stream NAL header placed in front of one or more NAL units.

[0878] Fig.52 An example of a syntax structure of a sample stream NAL header (sample_stream_nal_header()) included in an atlas substream according to an embodiment is shown.

[0879] sample_stream_nal_header() according to an embodiment may include a ssnh_unit_size_precision_bytes_minus1 field and a ssnh_reserved_zero_5bits field.

[0880] The value of the ssnh_unit_size_precision_bytes_minus1 field plus 1 may specify the precision (in bytes) of the ssnu_nal_unit_size element in all sample stream NAL units. The value of this field may be in the range of 0 to 7.

[0881] The ssnh_reserved_zero_5bits field is reserved for future use.

[0882] Fig.53 An example of a syntax structure of a sample stream NAL unit (sample_stream_nal_unit()) according to an embodiment is shown.

[0883] sample_stream_nal_unit() according to an embodiment may include a ssnu_nal_unit_size field and nal_unit (ssnu_nal_unit_size).

[0884] The ssnu_nal_unit_size field specifies the size of the subsequent NAL_unit in bytes. The number of bits used to represent the ssnu_nal_unit_size field is equal to (ssnh_unit_size_precision_bytes_minus1+1)*8.

[0885] The nal_unit (ssnu_nal_unit_size) has a length corresponding to the value of the ssnu_nal_unit_size field and carries one of an Atlas sequence parameter set (ASPS), an Atlas frame parameter set (AFPS), an Atlas tiling group information, and an SEI message. That is, each sample stream NAL unit may contain an ASPS, an AFPS, an Atlas tiling group information, or an SEI message. According to an embodiment, the ASPS, the AFPS, the Atlas tiling group information, and the SEI message are referred to as Atlas data (or Atlas metadata).

[0886] The SEI message according to an embodiment may assist processes related to decoding, reconstruction, display or other purposes.

[0887] Each SEI message according to an embodiment consists of a SEI message header and a SEI message payload (sei_payload). The SEI message header may contain payload type information (payloadType) and payload size information (payloadSize).

[0888] The payload type indicates the payload type of the SEI message. For example, it can be identified whether the SEI message is a prefix SEI message or a suffix SEI message based on the payloadType.

[0889] Payload Size indicates the payload size of the SEI message.

[0890] Fig.54 An example of a syntax structure of a SEI message payload (sei_payload()) according to an embodiment is shown.

[0891] The SEI message according to the embodiment may include a prefix SEI message or a suffix SEI message. In addition, each SEI message signals information corresponding to payload type information (payloadType) and payload size information (payloadSize) through a SEI message payload (sei_payload(payloadType, payloadSize)).

[0892] When payloadType is equal to 0, the prefix SEI message according to the implementation may include buffering_period(payloadSize). When payloadType is equal to 1, the prefix SEI message may include pic_timing(payloadSize). When payloadType is equal to 2, the prefix SEI message may include filler_payload(payloadSize). When payloadType is equal to 10, the prefix SEI message may include sei_prefix_indication(payloadSize). When payloadType is equal to 13, the prefix SEI message may include 3d_region_mapping(payloadSize).

[0893] When payloadType is equal to 2, the suffix SEI message may include filler_payload(payloadSize). When payloadType is equal to 3, the suffix SEI message may include user_data_registered_itu_t_t35(payloadSize). When payloadType is equal to 4, the suffix SEI message may include user_data_unregistered(payloadSize). When payloadType is equal to 11, the suffix SEI message may include decoded_pcc_hash(payloadSize).

[0894] According to an embodiment, Fig.24 or Fig.43 The V-PCC bit stream with the structure shown is composed of Figure 1 , Fig.18 , Fig. 20 or Fig.21 The file / fragment encapsulator is encapsulated in the ISOBMFF file format.

[0895] In this case, the V-PCC stream may be sent via multiple tracks or a single track of the file.

[0896] An ISOBMFF-based file according to an embodiment may consist of a box and / or information, which may be called ftyp, meta, moov, or mdat.

[0897] The ftyp box (file type box) can provide information related to the file compatibility or file type of the file. The receiving side can refer to the ftyp box to identify the file.

[0898] The meta box may include a vpcg{0,1,2,3} box (V-PCC group box).

[0899] The mdat box, also called the media data box, may contain a video coded geometry bitstream, a video coded attributes bitstream, a video coded occupancy map bitstream and / or an atlas data bitstream.

[0900] The moov box, also called a movie box, may contain metadata about the media data of the file (e.g., geometry bitstream, attribute bitstream, occupancy map bitstream, etc.). For example, it may contain information required to decode and play the media data, as well as information about the samples of the file. The moov box may be used as a container for all metadata. The moov box may be the highest level box among the metadata related boxes. Depending on the implementation, there may be only one moov box in a file.

[0901] The box according to an embodiment may include a track (trak) box providing information related to a track of a file. The track box may include a media (mdia) box providing media information about the track and a track reference container (tref) box for referencing a sample of a track and a file corresponding to the track.

[0902] The mdia box may include a media information container (minf) box that provides information about the media data. The minf box may include a sample table (stbl) box that provides metadata related to the samples of the mdat box.

[0903] The stbl box may include a sample description (stsd) box that provides information about the type of encoding employed and the initialization information required for that encoding type.

[0904] According to an embodiment, the stsd box may include sample entries for storing tracks of the V-PCC bitstream.

[0905] In order to store the V-PCC bitstream according to the embodiments in a single track or multiple tracks in a file, the present disclosure defines a volume visual track, a volume visual media header, a volume sample entry, a volume sample, and samples and simple entries of the V-PCC track.

[0906] The term V-PCC used in this paper is the same as visual volumetric video based coding (V3C). The two terms can be used to complement each other.

[0907] According to an embodiment, video-based point cloud compression (V-PCC) represents volumetric encoding of point cloud visual information.

[0908] That is, the minf box in the trak box of the moov box may also include a volume visual media header box. The volume visual media header box includes information about the volume visual track including the volume visual scene.

[0909] Each volumetric vision scene can be represented by a unique volumetric vision track. An ISOBMFF file can contain multiple scenes, so there can be multiple volumetric vision tracks in an ISOBMFF file.

[0910] According to an embodiment, the volumetric visual track may be identified by the volumetric visual media handler type 'volv' in the HandlerBox of the MediaBox.

[0911] The syntax of the volumetric visual media header box according to an embodiment is configured as follows.

[0912] Box type: 'vvhd'

[0913] Container: MediaInformationBox

[0914] Mandatory: Yes

[0915] Quantity: Exactly one

[0916] A volumetric visual track according to an embodiment may use a volumetric visual media handler box (VolumetricVisualMediaHeaderBox) in a media information box (MediaInformationBox) as follows.

[0917] aligned(8) class VolumetricVisualMediaHeaderBox

[0918] Expand FullBox('vvhd',version=0,1){

[0919] }

[0920] version may be an integer indicating the version of the box.

[0921] A volumetric visual track according to an embodiment may use a volumetric visual sample entry (VolumetricVisualSampleEntry) as follows.

[0922] Class VolumetricVisualSampleEntry(codingname)

[0923] extend SampleEntry(codingname){

[0924] unsigned int(8)

[32] compressor_name;

[0925] }

[0926] compressor_name is a name used for informational purposes. It is formatted as a fixed 32-byte field with the first byte set to the number of bytes to be displayed, followed by the number of bytes of displayable data encoded using UTF-8, and then padding to complete a total of 32 bytes (including the size byte). This field may be set to 0.

[0927] The format of volumetric vision samples according to an embodiment may be defined by the encoding system.

[0928] The V-PCC unit header box according to an embodiment may exist in a V-PCC track included in a sample entry and all video-coded V-PCC component tracks included in scheme information.The V-PCC unit header box may contain a V-PCC unit header for data carried by each track as follows.

[0929] aligned(8) class VPCCUnitHeaderBox

[0930] Extend FullBox('vunt', version=0,0) {

[0931] vpcc_unit_header()unit_header;

[0932] }

[0933] That is, VPCCUnitHeaderBox may include vpcc_unit_header(). Fig.34 and Fig.48 An example of the syntax structure of (vpcc_unit_header()) is shown.

[0934] According to an embodiment, a V-PCC track sample entry may contain a VPCC configuration box (VPCCConfigurationBox).

[0935] According to an embodiment, VPCCConfigurationBox may include a VPCC decoder configuration record (VPCCDecoderConfigurationRecord) as shown below.

[0936] aligned(8) class VPCCDDecoderConfigurationRecord{

[0937] unsigned int(8)configurationVersion=1;

[0938] unsigned int(3)sampleStreamSizeMinusOne;

[0939] unsigned int(5)numOfVPCCParameterSets;

[0940] for(i=0;i <numOfVPCCParameterSets;i++){

[0941] sample_stream_vpcc_unit VPCCParameterSet;

[0942] }

[0943] unsigned int(8)numOfAtlasSetupUnits;

[0944] for(i=0;i <numOfAtlasSetupUnits;i++){

[0945] sample_stream_vpcc_unit atlas_setupUnit;

[0946] }

[0947] }

[0948] The configurationVersion included in the VPCCDDecoderConfigurationRecord indicates the version field. Incompatible changes to the record are indicated by a change in the version number.

[0949] sampleStreamSizeMinusOne plus 1 may indicate the precision of the ssvu_vpcc_unit_size element in all sample stream V-PCC units in this configuration record or in V-PCC samples in the stream to which this configuration record applies.

[0950] numOfVPCCParameterSets specifies the number of V-PCC parameter sets (VPS) signaled in the VPCCDecoderConfigurationRecord.

[0951] VPCCParameterSet is an instance of sample_stream_vpcc_unit() of a V-PCC unit of type VPCC_VPS. A V-PCC unit may include vpcc_parameter_set(). That is, a VPCCParameterSet array may include vpcc_parameter_set(). Fig.46 An example of the syntax structure of a sample stream V-PCC unit (sample_stream_vpcc_unit()) is shown.

[0952] numOfAtlasSetupUnits indicates the number of setup arrays for the atlas stream signaled in the VPCCDecoderConfigurationRecord.

[0953] Atlas_setupUnit is an instance of sample_stream_vpcc_unit() that contains an Atlas sequence parameter set, an Atlas frame parameter set, or a SEI Atlas NAL unit. Fig.46 An example of the syntax structure of a sample stream V-PCC unit (sample_stream_vpcc_unit()) is shown.

[0954] Specifically, the atlas_setupUnit array may include atlas parameter sets that are constant for the streams referred to by the sample entry where the VPCCDecoderConfigurationRecord exists and the atlas stream SEI message. According to an embodiment, atlas_setupUnit may be referred to as a setup unit.

[0955] According to other implementations, VPCCDDecoderConfigurationRecord may be expressed as follows.

[0956] aligned(8) class VPCCDDecoderConfigurationRecord{

[0957] unsigned int(8)configurationVersion=1;

[0958] unsigned int(3)sampleStreamSizeMinusOne;

[0959] bit(2)reserved=1;

[0960] unsigned int(3)lengthSizeMinusOne;

[0961] unsigned int(5)numOfVPCCParameterSets;

[0962] for(i=0;i <numOfVPCCParameterSets;i++){

[0963] sample_stream_vpcc_unit VPCCParameterSet;

[0964] }

[0965] unsigned int(8)numOfSetupUnitArrays;

[0966] for(j=0; j <numOfSetupUnitArrays;j++){

[0967] bit(1)array_completeness;

[0968] bit(1)reserved=0;

[0969] unsigned int(6)NAL_unit_type;

[0970] unsigned int(8)numNALUnits;

[0971] for(i=0;i <numNALUnits;i++){

[0972] sample_stream_nal_unit setupUnit;

[0973] }

[0974] }

[0975] configurationVersion is a version field. Incompatible changes to a record are indicated by a change in the version number.

[0976] The value of lengthSizeMinusOne plus 1 may indicate the precision (in bytes) of the ssnu_nal_unit_size element in all sample stream NAL units in the V-PCC samples in the stream to which the VPCCDecoderConfigurationRecord or the VPCCDecoderConfigurationRecord applies. Fig.53An example of the syntax structure of a sample stream NAL unit (sample_stream_nal_unit()) including a ssnu_nal_unit_size field is shown.

[0977] numOfVPCCParameterSets specifies the number of V-PCC parameter sets (VPS) signaled in the VPCCDecoderConfigurationRecord.

[0978] VPCCParameterSet is a sample_stream_vpcc_unit() instance of a V-PCC unit of type VPCC_VPS. A V-PCC unit may include vpcc_parameter_set(). That is, a VPCCParameterSet array may include vpcc_parameter_set(). Fig.46 An example of the syntax structure of sample_stream_vpcc_unit() is shown.

[0979] numOfSetupUnitArrays indicates the number of atlas NAL unit arrays of the indicated type.

[0980] An iteration statement repeated as many times as the value of numOfSetupUnitArrays may include array_completeness.

[0981] array_completeness equal to 1 indicates that all atlas NAL units of a given type are in the following array, and none are in the stream. array_completeness equal to 0 indicates that additional atlas NAL units of the specified type may be in the stream. The default and allowed values ​​are constrained by the sample entry name.

[0982] NAL_unit_type indicates the type of atlas NAL unit in the following array. NAL_unit_type is constrained to take one of the values ​​indicating a NAL_ASPS, NAL_PREFIX_SEI, or NAL_SUFFIX_SEI atlas NAL unit.

[0983] numNALUnits indicates the number of Atlas NAL units of the indicated type contained in the VPCCDecoderConfigurationRecord for the stream to which the VPCCDecoderConfigurationRecord applies. The SEI array shall contain only SEI messages of a "declarative" nature, i.e. SEI messages that provide information about the entire stream. An example of such a SEI could be a User Data SEI

[0984] setupUnit is an instance of sample_stream_nal_unit(), containing an Atlas sequence parameter set, or an Atlas frame parameter set, or a declarative SEI Atlas NAL unit.

[0985] According to an embodiment, in the general layout of a multi-track container (also referred to as a multi-track ISOBMFF V-PCC container), V-PCC units in a V-PCC elementary stream may be mapped to individual tracks within a container file according to their types. There are two types of tracks in a multi-track ISOBMFF V-PCC container according to an embodiment. One is a V-PCC track, and the other is a V-PCC component track.

[0986] The V-PCC track according to an embodiment is a track that carries volumetric visual information in a V-PCC bitstream, including an atlas sub-bitstream and a sequence parameter set.

[0987] The V-PCC component track according to an embodiment is a restricted video scheme track that carries 2D video coding data for occupancy map, geometry and attribute sub-bitstreams of the V-PCC bitstream. In addition, the V-PCC component track may satisfy the following conditions:

[0988] a) In the sample entry, a new box is inserted, which records the role of the video stream contained in this track in the V-PCC system;

[0989] b) introducing a track reference from the V-PCC track to the V-PCC component track to establish the membership of the V-PCC component track in a particular point cloud represented by the V-PCC track;

[0990] c) Set the track header flag to 0 to indicate that this track does not contribute directly to the overall layout of the movie, but does help the V-PCC system.

[0991] Tracks belonging to the same V-PCC sequence can be time-aligned. V-PCC component tracks coded across different videos and samples contributing to the same point cloud frame by V-PCC tracks have the same presentation time. The decoding time of the V-PCC Atlas sequence parameter set and the Atlas frame parameter set for such samples is equal to or earlier than the composition time of the point cloud frame. In addition, all tracks belonging to the same V-PCC sequence have the same implicit or explicit edit lists.

[0992] NOTE: Synchronization between elementary streams in component tracks is handled by the ISOBMFF track timing structures (stts, ctts and cslg) or equivalent mechanisms in movie fragments.

[0993] Based on this layout, a V-PCC ISOBMFF container may include the following:

[0994] - A V-PCC track, which contains V-PCC parameter sets (in sample entries) and samples that carry the payload of V-PCC parameter sets, V-PCC cells (cell type VPCC_VPS) and Atlas V-PCC cells (cell type VPCC_AD). This track also includes track references to other tracks that carry the payload of video compressed V-PCC cells (i.e., cell types VPCC_OVD, VPCC_GVD, and VPCC_AVD).

[0995] - A constrained video scheme track where the samples contain access units of a video coded elementary stream for occupancy map data (i.e. the payload of a V-PCC unit of type VPCC_OVD).

[0996] - One or more constrained video scheme tracks, where samples contain access units of a video coding elementary stream for geometry data (ie payloads of V-PCC units of type VPCC_GVD).

[0997] - Zero or more constrained video scheme tracks, where samples contain access units of a video coded elementary stream for attribute data (i.e. payloads of V-PCC units of type VPCC_AVD).

[0998] Next, the V-PCC track will be described below.

[0999] The syntax structure of a V-PCC track sample entry according to an embodiment is configured as follows.

[1000] Sample entry types: "vpc1", "vpcg"

[1001] Container: SampleDescriptionBox

[1002] Mandatory: Either the "vpc1" or "vpcg" sample entry is mandatory.

[1003] Quantity: There can be one or more sample entries.

[1004] V-PCC tracks use VPCCSampleEntry, which extends VolumetricVisualSampleEntry. The sample entry type is either "vpc1" or "vpcg".

[1005] The VPCC sample entry contains a VPCCConfigurationBox. This box includes a VPCCDDecoderConfigurationRecord.

[1006] Under the "vpc1" sample entry, all Atlas sequence parameter sets, Atlas frame parameter sets, or V-PCC SEIs are in the setupUnit array.

[1007] Under the "vpcg" sample entry, Atlas sequence parameter set, Atlas frame parameter set, V-PCC SEI may be present in the array or in the stream.

[1008] An optional BitRateBox may be present in the VPCC volume sample entry to signal the bitrate information of the V-PCC track.

[1009] Volume Sequence:

[1010] class VPCCConfigurationBox extends Box('vpcC'){

[1011] VPCCDecoderConfigurationRecord()VPCCConfig;

[1012] }

[1013] aligned(8) class VPCCSampleEntry() extends VolumetricVisualSampleEntry('vpc1') {

[1014] VPCCConfigurationBoxconfig;

[1015] VPCCUnitHeaderBox unit_header;

[1016] }

[1017] Fig.55 An example of a V-PCC sample entry structure according to an embodiment is shown. Fig.55In it, the V-PCC sample entry includes a V-PCC parameter set and may optionally include an Atlas sequence parameter set (ASPS), an Atlas frame parameter set (AFPS), or SEI.

[1018] The V-PCC bitstream according to an embodiment may also contain a sample stream V-PCC header, a sample stream NAL header, and a V-PCC unit header box.

[1019] Next, the V-PCC track sample format is described below.

[1020] Each sample in the V-PCC track corresponds to a single point cloud frame. Samples corresponding to this frame in various component tracks have the same composition time as the V-PCC track sample. Each V-PCC sample includes one or more Atlas NAL units as follows.

[1021] aligned(8) class VPCCSample{

[1022] unsigned int PointCloudPictureLength = sample_size; / / The size of the sample (e.g., from SampleSizeBox)

[1023] for(i = 0; i < PointCloudPictureLength;){

[1024] sample_stream_nal_unit nalUnit

[1025] i += (VPCCDecoderConfigurationRecord.lengthSizeMinusOne + 1) + nalUnit.ssnu_nal_unit_size;

[1026] }

[1027] }

[1028] aligned(8) class VPCCSample

[1029] {

[1030] unsigned int PictureLength = sample_size; / / The size of the sample (e.g., from SampleSizeBox)

[1031] for(i = 0; i < PictureLength;) / / Signal until the end of the picture

[1032] {

[1033] unsigned int((VPCCDDecoderConfigurationRecord.LengthSizeMinusOne+1)*8)

[1034] NALUnitLength;

[1035] bits(NALUnitLength*8)NALUnit;

[1036] i+=(VPCCDDecoderConfigurationRecord.LengthSizeMinusOne+1)+NALUnitLength;

[1037] }

[1038] }

[1039] A sync sample (random access point) in a V-PCC track is a V-PCC IRAP coded patch data access unit of the V-PCC track. If necessary, the Atlas parameter set may be repeated at the sync sample to allow random access.

[1040] Next, the V-PCC component track of video coding is described below.

[1041] The carriage of coded video tracks using MPEG-specified codecs can conform to ISOBMFF-derived specifications. For example, for the carriage of AVC and HEVC-coded video, reference can be made to ISO / IEC 14496-15. ISOBMFF can also provide an extension mechanism if other codec types are required.

[1042] Since it does not make sense to display decoded frames from attribute, geometry or occupancy map tracks without reconstructing the point cloud on the player side, constrained video scheme types can be defined for these video encoding tracks.

[1043] Next, the restricted video scheme is described below.

[1044] A V-PCC component video track may be represented as restricted video in the file and may be identified by 'pccv' in the scheme_type field of the SchemeTypeBox of the RestrictedSchemeInfoBox of the restricted video sample entry.

[1045] It should be noted that there is no restriction on the video codec used for encoding of the attribute, geometry and occupancy map V-PCC components. Furthermore, different video codecs may be used to encode these components.

[1046] Scheme information (SchemeInformationBox) according to the implementation may exist and contain VPCCUnitHeaderBox.

[1047] Next, the reference V-PCC component track is described below.

[1048] To link a V-PCC track to a component video track, three TrackReferenceTypeBoxes can be added to the TrackReferenceBox inside the TrackBox of the V-PCC track, one for each component. The TrackReferenceTypeBox contains an array of track_IDs of the video tracks that the V-PCC track references. The reference_type of the TrackReferenceTypeBox identifies the type of component, such as occupancy map, geometry, or attribute. The track reference types are as follows.

[1049] In "pcco", the reference track contains the occupancy map V-PCC component of the video coding.

[1050] In "pccg", the reference track contains the geometry V-PCC component of the video coding.

[1051] In "pcca", the referenced track contains the attribute V-PCC component of video coding.

[1052] The type of the V-PCC component carried by the referenced restricted video track and signaled in the track's RestrictedSchemeInfoBox shall match the reference type of the track reference from the V-PCC track.

[1053] Next, a single track container of a V-PCC bitstream is described below.

[1054] Single track encapsulation of V-PCC data requires that a V-PCC coded elementary bitstream be represented by a single track declaration.

[1055] In the case of a simple ISOBMFF encapsulation of a V-PCC coded bitstream, a single track encapsulation of PCC data may be used. Such a bitstream may be directly stored as a single track without further processing. The V-PCC unit header data structure may be preserved in the bitstream. A single track container of V-PCC data may be provided to a media workflow for further processing (e.g., multi-track file generation, transcoding, DASH segmentation, etc.).

[1056] ISOBMFF files containing single-track encapsulated V-PCC data can include "pcst" in the compatible_brands[] list of the FileTypeBox.

[1057] V-PCC basic stream track:

[1058] Sample entry types: "vpe1", "vpeg"

[1059] Container: SampleDescriptionBox

[1060] Mandatory: "vpe1" or "vpeg" sample entries are mandatory Quantity: one or more sample entries may be present

[1061] V-PCC elementary stream tracks use VolumetricVisualSampleEntry with sample entry type "vpe1" or "vpeg".

[1062] The VPCC elementary stream sample entry contains a VPCCConfigurationBox.

[1063] Under the 'vpe1' sample entry, all atlas sequence parameter sets, atlas frame parameter sets and SEI can be in the setupUnit array. Under the 'vpeg' sample entry, atlas sequence parameter sets, atlas frame parameter sets and SEI can be present in this array or in the stream.

[1064] Volume Sequence:

[1065] class VPCCConfigurationBox extends Box('vpcC'){

[1066] VPCCDecoderConfigurationRecord()VPCCConfig;

[1067] }

[1068] aligned(8) class VPCCElementaryStreamSampleEntry() extends VolumetricVisualSampleEntry('vpe1') {

[1069] VPCCConfigurationBoxconfig;

[1070] VPCCBoundingInformationBox 3d_bb;

[1071] }

[1072] Next, the V-PCC elementary stream sample format is described below.

[1073] A V-PCC elementary stream sample may include one or more V-PCC units belonging to the same presentation time. Each sample has a unique presentation time, size, and duration. The sample may be a synchronous sample or a decoding method that depends on other V-PCC elementary stream samples.

[1074] Next, a V-PCC elementary stream synchronization sample is described below.

[1075] V-PCC elementary stream synchronization samples can meet all of the following conditions:

[1076] - it is independently decodable;

[1077] - in decoding order, samples after the sync sample do not have any decoding dependencies on any samples before the sync sample; and

[1078] - All samples that appear after the sync sample in decoding order are successfully decodable.

[1079] Next, a V-PCC basic stream sub-sample is described below.

[1080] A V-PCC base stream subsample is a V-PCC unit contained in a V-PCC base stream sample.

[1081] The V-PCC elementary stream track contains a SubSampleInformationBox in the SampleTableBox or in the TrackFragmentBox of each MovieFragmentBox, which lists the V-PCC elementary stream subsamples.

[1082] The 32-bit unit header representing the V-PCC unit of the subsample may be copied to the 32-bit codec_specific_parameters field of the subsample entry in SubSampleInformationBox. The V-PCC unit type of each subsample is identified by parsing the codec_specific_parameters field of the subsample entry in SubSampleInformationBox.

[1083] Next, the rendering of point cloud data is described below.

[1084] According to an embodiment, the rendering of the point cloud data is performed by Figure 1 Renderer 10009, Fig.19 Point cloud renderer 19007, Fig. 20 Renderer 2001 or Fig. 22 The point cloud renderer 22004 is executed. According to an embodiment, the point cloud data can be rendered in a 3D space based on the metadata. The user can view all or part of the rendering result through a VR / AR display or a normal display. In particular, the point cloud data can be rendered according to the user's viewport, etc.

[1085] Here, the viewport or viewport area may refer to the area that the user is viewing in the point cloud video. The viewpoint is the point that the user is viewing in the point cloud video, and may represent the center point of the viewport area. That is, the viewport is the area centered on the viewpoint, and the size, shape, etc. occupied by the area may be determined by the field of view (FOV).

[1086] The viewport information according to the embodiment may be information about the area that the user is currently viewing in 3D space through the device or HMD. The viewport information may be used to perform gaze analysis to check how the user consumes the point cloud video, the area of ​​the point cloud video that the user is looking at, and the time the user is looking at the area. The gaze analysis may be performed on the receiving side, and the analysis results may be transmitted to the sending side on a feedback channel. Display devices such as VR / AR / MR may extract the viewport area based on orientation information, the position / orientation of the user's head, the vertical or horizontal FOV supported by the device, and the like. The direction or viewport information analyzed by the receiving device may be sent to the sending device on a feedback channel.

[1087] The point cloud video decoder of the receiving device according to the embodiment can efficiently extract or decode only the media data of a specific area (i.e., the area indicated by the orientation information and / or viewport information) from the file based on the orientation information and / or viewport information indicating the area currently being viewed by the user. The point cloud video encoder of the sending device according to the embodiment can encode only the media data of the specific area (i.e., the area indicated by the orientation information and / or viewport information) based on the orientation information and / or viewport information fed back from the receiving device, or can encapsulate the encoded media data into a file for transmission.

[1088] According to an embodiment, the file / fragment encapsulator of the sending device may encapsulate all point cloud data into files / fragments based on orientation information and / or viewport information, or may encapsulate point cloud data indicated by orientation information and / or viewport information into files / fragments.

[1089] According to an embodiment, the file / fragment decapsulator of the receiving device may decapsulate a file containing all point cloud data or a file containing point cloud data indicated by the orientation information and / or viewport information based on the orientation information and / or viewport information.

[1090] According to an embodiment, the viewport information may be used in the same or similar meaning as the ViewInfoStruct information or the view information.

[1091] The viewport information according to the embodiment may also be referred to as viewport related information or viewport information related metadata. The viewport related information according to the embodiment may include at least one of viewport information, rendering parameters and object rendering information.

[1092] According to an embodiment, the viewport related information may be provided by Fig.18 The metadata encoder 18005 of the point cloud data sending device or Fig.21 The point cloud preprocessor 20001 and / or the video / image encoders 21007 and 21008 of the V-PCC system generate / encode and can be Fig.19 The metadata decoder 19002 of the point cloud data receiving device or Fig. 22 The video / image decoders 22001 and 22002 and / or the point cloud processor 22003 of the V-PCC system acquire / decode.

[1093] In the present disclosure, embodiments of defining metadata related to viewport (or view) information of point cloud data, and storing and signaling the metadata related to the viewport information in a file are described below.

[1094] In the present disclosure, an embodiment of storing metadata related to viewport information of point cloud data that changes dynamically over time in a file is described below.

[1095] In the present disclosure, embodiments of defining metadata related to rendering parameters of point cloud data and storing and signaling the metadata related to the rendering parameters in a file are described below.

[1096] In the present disclosure, an embodiment of storing metadata related to rendering parameters of point cloud data that dynamically changes according to time in a file is described below.

[1097] In the present disclosure, embodiments of defining metadata related to object rendering information related to point cloud data in a file and storing and signaling the metadata related to the object rendering information in the file are described below.

[1098] In the present disclosure, an embodiment of storing metadata related to object rendering information of point cloud data dynamically changing according to time in a file is described below.

[1099] Fig.56 An example of generating a view using a virtual camera and ViewInfoStruct information according to an embodiment is shown.

[1100] The ViewInfoStruct information according to an embodiment may include detailed information for creating a view to be rendered and provided. Specifically, ViewInfoStruct may include position information about a virtual camera in a 3D space used to create a view, a vertical / horizontal field of view (FOV) of the virtual camera, a direction vector indicating the direction the virtual camera is facing, and an up vector indicating the upward direction of the virtual camera. The virtual camera may be similar to the user's eyes, that is, the user's view of a portion of the 3D space. The view cone can be inferred based on the above information. The view cone refers to an area in 3D space that includes all or part of the point cloud data that is actually rendered and displayed. According to an embodiment, when the inferred view cone is projected in the form of a 2D frame, a view (i.e., a 2D image / video frame to be actually displayed) can be created.

[1101] The following is a syntax showing an example of information included in ViewInfoStruct.

[1102] aligned(8) class ViewInfoStruct(){

[1103] unsigned int(16)view_pos_x;

[1104] unsigned int(16)view_pos_y;

[1105] unsigned int(16)view_pos_z;

[1106] unsigned int(8)view_vfov;

[1107] unsigned int(8)view_hfov;

[1108] unsigned int(16)view_dir_x;

[1109] unsigned int(16)view_dir_y;

[1110] unsigned int(16)view_dir_z;

[1111] unsigned int(16)view_up_x;

[1112] unsigned int(16)view_up_y;

[1113] unsigned int(16)view_up_z;

[1114] }

[1115] view_pos_x, view_pos_y, and view_pos_z may indicate x, y, and z coordinate values ​​in a 3D space of a virtual camera capable of creating a view (eg, a 2D image / video frame to be actually displayed).

[1116] view_vfov and view_hfov may indicate the vertical field of view (FOV) and horizontal field of view (FOV) of the virtual camera that is capable of creating the view.

[1117] view_dir_x, view_dir_y, and view_dir_z may indicate x, y, and z coordinate values ​​in a 3D space for indicating a direction vector representing a direction viewed by the virtual camera.

[1118] view_up_x, view_up_y, and view_up_z may indicate x, y, and z coordinate values ​​in a three-dimensional space to represent an up vector indicating an up direction of the virtual camera.

[1119] The ViewInfoStruct() information according to the embodiment may be transmitted in the form of SEI or the like in a point cloud bitstream as follows.

[1120] V-PCC View Information Box

[1121] aligned(8) class VPCCViewInfoBox extends FullBox('vpvi',0,0) {

[1122] ViewInfoStruct();

[1123] }

[1124] Next, the static V-PCC viewport information is described below.

[1125] When the viewport information according to an embodiment does not change within a point cloud sequence, the VPCCViewInfoBox may be included in a sample entry of a V-PCC track or a sample entry of a V-PCC elementary stream track as shown below.

[1126] aligned(8) class VPCCSampleEntry() extends VolumetricVisualSampleEntry('vpc1') {

[1127] VPCCConfigurationBoxconfig;

[1128] VPCCUnitHeaderBox unit_header;

[1129] VPCCViewInfoBox view_info;

[1130] }

[1131] The VPCCViewInfoBox can contain detailed information for creating a view in which point cloud data associated with atlas frames stored in samples within a track are rendered and served as shown below.

[1132] aligned(8) class VPCCElementaryStreamSampleEntry() extends VolumetricVisualSampleEntry('vpe1') {

[1133] VPCCConfigurationBoxconfig;

[1134] VPCCViewInfoBox view_info;

[1135] }

[1136] A VPCCViewInfoBox can contain detailed information for creating a view in which to render and serve point cloud data associated with a video frame or atlas frame in subsamples stored in a track.

[1137] Next, V-PCC view information sample grouping is described below.

[1138] According to an embodiment, 'vpvs' grouping_type for sample grouping indicates that samples in a V-PCC track are assigned to view information carried in the sample group.When a SampleToGroupBox with grouping_type equal to 'vpvs' is present, an accompanying SampleGroupDescriptionBox with the same grouping type is present and contains the ID to which the group of samples belongs.

[1139] aligned(8) class VPCCViewInfoSampleGroupDescriptionEntry() extends SampleGroupDescriptionEntry('vpvs'){

[1140] ViewInfoStruct();

[1141] }

[1142] Next, dynamic V-PCC view information is described below.

[1143] If a V-PCC track has an associated timed metadata track whose sample entry type is "dyvi", the view information defined for the point cloud stream carried by the V-PCC track is considered dynamic. That is, the view information can change dynamically over time.

[1144] The associated timed metadata track contains a "cdsc" track reference to the V-PCC track carrying the Atlas stream.

[1145] If a V-PCC elementary stream track has an associated timed metadata track whose sample entry type is "dyvi", the view information defined for the point cloud stream carried by the V-PCC elementary stream track is considered to be dynamic. That is, the view information may change dynamically over time.

[1146] The associated timed metadata track contains a "cdsc" track reference to the V-PCC elementary stream track.

[1147] aligned(8) class DynamicViewInfoSampleEntry extends MetaDataSampleEntry('dyvi') {

[1148] VPCCViewInfoBox init_view_info;

[1149] }

[1150] init_view_info may include view information (viewinformation()) for generating an initial view of the point cloud data.

[1151] The sample syntax for this sample entry type "dyvi" may be specified as follows:

[1152] aligned(8)DynamicViewInfoSample(){

[1153] VPCCViewInfoBox view_info;

[1154] }

[1155] Each sample can contain view information that varies over time ().

[1156] According to an embodiment, RenderingParamStruct, which is a rendering parameter for rendering, may include parameter information that can be applied in rendering point cloud data. The parameter may include the size of the point, the rendering type of the point, how to handle duplicate points, for example, whether to display duplicate points in the rendering of the point cloud data, etc., as shown below.

[1157] aligned(8) class RenderingParamStruct(){

[1158] unsigned int(16)point_size;

[1159] unsigned int(7)point_type;

[1160] unsigned int(2)duplicated_point;

[1161] }

[1162] point_size may indicate the size of the points to be rendered / displayed.

[1163] point_type may indicate the type of point to be rendered / displayed. For example, point_type equal to 0 may indicate a cuboid. point_type equal to 1 may indicate a circle. point_type equal to 2 may indicate a point.

[1164] duplicated_point may indicate how to handle multiple points in 3D space with the same x, y, and z values ​​when present. For example, when duplicated_point is equal to 0, duplicate points may be ignored. In an embodiment, when duplicated_point is equal to 1, only one point may be randomly selected and rendered / displayed. In an embodiment, when duplicated_point is equal to 2, the attribute values ​​of duplicate points may be averaged and rendered / displayed based on the average attribute value.

[1165] The RenderingParamStruct() information may be carried in the point cloud bitstream in the form of SEI or the like as follows.

[1166] V-PCC Rendering Parameter Box

[1167] aligned(8) class VPCCRenderingParamBox extends FullBox('vprp',0,0) {

[1168] RenderingParamStruct();

[1169] }

[1170] Next, static V-PCC rendering parameters are described below.

[1171] According to an embodiment, when rendering-related parameters do not change within a point cloud sequence, VPCCRenderingParamBox may be included in a sample entry of a V-PCC track or a sample entry of a V-PCC elementary stream track, as shown below.

[1172] aligned(8) class VPCCSampleEntry() extends VolumetricVisualSampleEntry('vpc1') {

[1173] VPCCConfigurationBoxconfig;

[1174] VPCCUnitHeaderBox unit_header;

[1175] VPCCRenderingParamBox rendering_param;

[1176] }

[1177] A VPCCRenderingParamBox may contain details about rendering parameters that may be applied when rendering point cloud data associated with an atlas frame in samples stored in a track, as described below.

[1178] aligned(8) class VPCCElementaryStreamSampleEntry() extends VolumetricVisualSampleEntry('vpe1') {

[1179] VPCCConfigurationBoxconfig;

[1180] VPCCRenderingParamBox rendering_param;

[1181] }

[1182] VPCCRenderingParamBox may contain detailed information about rendering parameters that may be applied when rendering atlas frames, video frames, and point cloud data in subsamples stored in a track.

[1183] Next, V-PCC rendering parameter sample grouping is described below.

[1184] According to an embodiment, 'vprp' grouping_type for sample grouping indicates that samples in a V-PCC track are assigned to rendering parameters carried in the sample group.When a SampleToGroupBox with grouping_type equal to 'vpvs' is present, an accompanying SampleGroupDescriptionBox with the same grouping type is present and may contain the ID of the group of samples.

[1185] aligned(8) class VPCCRenderingParamSampleGroupDescriptionEntry() extends SampleGroupDescriptionEntry('vprp'){

[1186] RenderingParamStruct();

[1187] }

[1188] Next, dynamic V-PCC rendering parameter information is described below.

[1189] If a V-PCC track has an associated timed metadata track whose sample entry type is "dyrp", then the rendering parameters defined for the point cloud stream carried by the V-PCC track are considered dynamic rendering parameters. That is, the rendering parameters can change dynamically over time.

[1190] The associated timed metadata track contains a "cdsc" track reference to the V-PCC track carrying the Atlas stream.

[1191] If a V-PCC elementary stream track has an associated timed metadata track whose sample entry type is "dyrp", the rendering parameters defined for the point cloud stream carried by the V-PCC elementary stream track are considered dynamic rendering parameters. That is, the rendering parameters can change dynamically over time.

[1192] The associated timed metadata track contains a "cdsc" track reference to the V-PCC elementary stream track.

[1193] aligned(8) class DynamicRenderingParamSampleEntry extends MetaDataSampleEntry('dyrp') {

[1194] VPCCRenderingParamBox init_rendering_param;

[1195] }

[1196] init_rendering_param may contain the initial rendering parameters for the point cloud data.

[1197] A sample syntax for a sample entry type 'dyvi' according to an embodiment may be specified as follows:

[1198] aligned(8)DynamicRenderingParamSample(){

[1199] VPCCRenderingParamBox rendering_param;

[1200] }

[1201] Each sample may include rendering parameters that vary over time.

[1202] Fig.57 An example of creating a view using ObjectRenderingInfoStruct() information according to an embodiment is shown.

[1203] According to an embodiment, ObjectRenderingInfoStruct as object rendering information may include detailed information applied to point cloud data when rendering point cloud data. According to an embodiment, ObjectRenderingInfoStruct may include information such as coordinate system information used by point cloud data, the position of point cloud data in a 3D space based on the coordinate system, a normal vector of point cloud data, and an up vector that may indicate an upward direction of point cloud data.

[1204] aligned(8) classObjectRenderingInfoStruct(){

[1205] unsigned int(8)obj_id;

[1206] unsinged int(4)obj_coord_type;

[1207] bit(4)reserved=1;

[1208] unsigned int(16)obj_pos_x;

[1209] unsigned int(16)obj_pos_y;

[1210] unsigned int(16)obj_pos_z;

[1211] unsigned int(16)obj_dir_x;

[1212] unsigned int(16)obj_dir_y;

[1213] unsigned int(16)obj_dir_z;

[1214] unsigned int(16)obj_up_x;

[1215] unsigned int(16)obj_up_y;

[1216] unsigned int(16)obj_up_z;

[1217] unsigned int(16)obj_scale_x;

[1218] unsigned int(16)obj_scale_y;

[1219] unsigned int(16)obj_scale_z;

[1220] }

[1221] obj_id may indicate an identifier of the point cloud data.

[1222] obj_coord_type may indicate the coordinate system used to render the point cloud. For example, when the value of obj_coord_type is 0, global coordinates may be used. For example, when the value of obj_coord_type is 1, view coordinates (i.e., the coordinate system of a virtual camera that can be used to generate viewport data) may be used.

[1223] obj_pos_x, obj_pos_y, and obj_pos_z may indicate the x, y, and z coordinate values ​​of the origin of the bounding box of the point cloud data in the coordinate system indicated by obj_coord_type. According to an embodiment, the x, y, and z coordinate values ​​of the center point of the bounding box of the point cloud data in the coordinate system indicated by obj_coord_type may be indicated.

[1224] obj_dir_x, obj_dir_y, and obj_dir_z may indicate x, y, and z coordinate values ​​in a 3D space, used to represent a direction vector (or normal vector) indicating a direction in which the point cloud data faces forward.

[1225] obj_up_x, obj_up_y, and obj_up_z may indicate x, y, and z coordinate values ​​in a 3D space for representing an up vector indicating an upward direction of the point cloud data.

[1226] obj_scale_x, obj_scale_y, and obj_scale_z may indicate the scaling factors to be applied to the x, y, and z axes compared to the original source of the point cloud data in the rendering / display.

[1227] According to an embodiment, the ObjectRenderingInfoStruct() information may be carried in the form of SEI or the like in a point cloud bitstream.

[1228] The V-PCC object rendering information box according to an embodiment may include ObjectRenderingInfoStruct() information.

[1229] aligned(8) class VPCCObjectRenderingInfoBox extends FullBox('vpoi',0,0) {

[1230] ObjectRenderingInfoStruct();

[1231] }

[1232] Next, static V-PCC object rendering information is described below.

[1233] According to an embodiment, when object rendering information applied to point cloud data does not change within a point cloud sequence during point cloud data rendering, VPCCObjectRenderingInfoBox may be included in a sample entry of a V-PCC track or a sample entry of a V-PCC elementary stream track.

[1234] aligned(8) class VPCCSampleEntry() extends VolumetricVisualSampleEntry('vpc1') {

[1235] VPCCConfigurationBoxconfig;

[1236] VPCCUnitHeaderBox unit_header;

[1237] VPCCObjectRenderingInfoBox obj_rendering_info;

[1238] }

[1239] The VPCCObjectRenderingInfoBox may contain object rendering information associated with point cloud data associated with an atlas frame in samples stored in a track.

[1240] aligned(8) class VPCCElementaryStreamSampleEntry() extends VolumetricVisualSampleEntry('v...

Claims

1. A method for sending point cloud data, the method comprising the following steps: Encode point cloud data; Encapsulating the bitstream including the encoded point cloud data into a file according to the ISO base media file format ISOBMFF; as well as Send said file, wherein the bitstream is stored in at least one track of a plurality of tracks of the file, each track comprising a sample entry and a sample, The file also includes signaling information. The signaling information includes viewport related information, and the viewport related information includes position information of the viewport for the point cloud data. The viewport related information is classified into dynamic viewport information that changes with time and static viewport information that does not change with time. wherein a track of the plurality of tracks is a timed metadata track, wherein the sample entry type of the timed metadata track has a value for indicating that the timed metadata track is used for the dynamic viewport information, The dynamic viewport information at least includes first viewport information or second viewport information, the first viewport information includes initial viewport position information, and the second viewport information includes viewport position information that changes dynamically over time. wherein the first viewport information is carried by a sample entry of the timed metadata track, wherein the second viewport information is carried by samples of the timed metadata track, and The static viewport information is carried in a track of the plurality of tracks that is different from the timed metadata track.

2. The point cloud data sending method according to claim 1, in, The point cloud data includes geometric data, attribute data and occupancy map data, and the geometric data, the attribute data and the occupancy map data are encoded by a video-based point cloud compression (V-PCC) scheme.

3. The point cloud data sending method according to claim 1, in, The viewport-related information also includes camera direction information.

4. The point cloud data sending method according to claim 1, in, The viewport related information also includes horizontal field of view FOV information and vertical FOV information used to generate the viewport.

5. A point cloud data sending device, the point cloud data sending device comprising: An encoder for encoding point cloud data; A packager, the packager is used to package the bit stream including the encoded point cloud data into a file according to the ISO base media file format ISOBMFF; as well as a sender, the sender being used to send the file, wherein the bitstream is stored in at least one track of a plurality of tracks of the file, each track comprising a sample entry and a sample, The file also includes signaling information. The signaling information includes viewport related information, and the viewport related information includes position information of the viewport for the point cloud data. The viewport related information is classified into dynamic viewport information that changes with time and static viewport information that does not change with time. wherein a track of the plurality of tracks is a timed metadata track, wherein the sample entry type of the timed metadata track has a value for indicating that the timed metadata track is used for the dynamic viewport information, The dynamic viewport information at least includes first viewport information or second viewport information, the first viewport information includes initial viewport position information, and the second viewport information includes viewport position information that changes dynamically over time. wherein the first viewport information is carried by a sample entry of the timed metadata track, wherein the second viewport information is carried by samples of the timed metadata track, and The static viewport information is carried in a track of the plurality of tracks that is different from the timed metadata track.

6. The point cloud data transmitting device according to claim 5, in, The point cloud data includes geometric data, attribute data and occupancy map data, and the geometric data, the attribute data and the occupancy map data are encoded by a video-based point cloud compression (V-PCC) scheme.

7. The point cloud data transmitting device according to claim 5, in, The viewport-related information also includes camera direction information.

8. The point cloud data transmitting device according to claim 5, in, The viewport related information also includes horizontal field of view FOV information and vertical FOV information used to generate the viewport.

9. A method for receiving point cloud data, the method comprising the following steps: Receive files in ISO Base Media File Format ISOBMFF; decapsulating the file into a bitstream comprising the encoded point cloud data, wherein the bitstream is stored in at least one track of a plurality of tracks of the file, wherein each track comprises a sample entry and a sample, wherein the file further comprises signaling information, and wherein the signaling information comprises viewport related information, the viewport related information comprising position information of a viewport for the point cloud data; Decoding the encoded point cloud data; and Rendering the decoded point cloud data based on the viewport related information, The viewport related information is classified into dynamic viewport information that changes with time and static viewport information that does not change with time. wherein a track of the plurality of tracks is a timed metadata track, wherein the sample entry type of the timed metadata track has a value for indicating that the timed metadata track is used for the dynamic viewport information, The dynamic viewport information at least includes first viewport information or second viewport information, the first viewport information includes initial viewport position information, and the second viewport information includes viewport position information that changes dynamically over time. wherein the first viewport information is carried by a sample entry of the timed metadata track, and wherein the second viewport information is carried by samples of the timed metadata track, and The static viewport information is carried in a track of the plurality of tracks that is different from the timed metadata track.

10. The point cloud data receiving method according to claim 9, in, The viewport-related information also includes camera direction information.

11. The point cloud data receiving method according to claim 9, in, The viewport related information also includes horizontal field of view FOV information and vertical FOV information used to generate the viewport.

12. The point cloud data receiving method according to claim 9, in, The decoded point cloud data includes geometric data, attribute data and occupancy map data, and the geometric data, the attribute data and the occupancy map data are decoded by a video-based point cloud compression V-PCC scheme.

13. A point cloud data receiving device, the point cloud data receiving device comprising: A receiver, the receiver being configured to receive a file in ISO base media file format ISOBMFF; a decapsulator for decapsulating the file into a bitstream comprising encoded point cloud data, wherein the bitstream is stored in at least one track of a plurality of tracks of the file, wherein each track comprises a sample entry and a sample, wherein the file further comprises signaling information, and wherein the signaling information comprises viewport related information, the viewport related information comprising position information of a viewport for the point cloud data; A decoder, the decoder being configured to decode the encoded point cloud data; and a renderer, the renderer being used to render the decoded point cloud data based on the viewport related information, The viewport related information is classified into dynamic viewport information that changes with time and static viewport information that does not change with time. wherein a track of the plurality of tracks is a timed metadata track, wherein the sample entry type of the timed metadata track has a value for indicating that the timed metadata track is used for the dynamic viewport information, The dynamic viewport information at least includes first viewport information or second viewport information, the first viewport information includes initial viewport position information, and the second viewport information includes viewport position information that changes dynamically over time. wherein the first viewport information is carried by a sample entry of the timed metadata track, and wherein the second viewport information is carried by samples of the timed metadata track, and The static viewport information is carried in a track of the plurality of tracks that is different from the timed metadata track.

14. The point cloud data receiving device according to claim 13, in, The viewport-related information also includes camera direction information.

15. The point cloud data receiving device according to claim 13, in, The viewport related information also includes horizontal field of view FOV information and vertical FOV information used to generate the viewport.

16. The point cloud data receiving device according to claim 13, in, The decoded point cloud data includes geometric data, attribute data and occupancy map data, and the geometric data, the attribute data and the occupancy map data are decoded by a video-based point cloud compression V-PCC scheme.