Apparatus and method for point cloud data and storage medium thereof

Abstract

The invention relates to an apparatus and method for point cloud data and a storage medium thereof. A point cloud data transmission method according to an embodiment may include a step of encoding point cloud data and a step of transmitting a bitstream including the point cloud data and signaling information. A point cloud data receiving method according to an embodiment may include a step of receiving a bitstream including point cloud data and signaling information, a step of decoding the point cloud data, and a step of rendering the decoded point cloud data.

Description

[0001] This application is a divisional application of the original invention patent application with application number 202080070476.6 (International application number: PCT / KR2020 / 013685, application date: October 7, 2020, invention name: Point cloud data sending device, point cloud data sending method, point cloud data receiving device and point cloud data receiving method). Technical Field

[0002] Embodiments provide a method of providing point cloud content to provide users with various services such as virtual reality (VR), augmented reality (AR), mixed reality (MR), and autonomous driving services. Background Art

[0003] A point cloud is a collection of points in a three-dimensional (3D) space. Generating point cloud data is difficult because the number of points in a 3D space is large.

[0004] High throughput is required to send and receive point cloud data. Summary of the Invention

[0005] Technical issues

[0006] The present disclosure aims to provide a point cloud data transmitting device, a point cloud data transmitting method, a point cloud data receiving device and a point cloud data receiving method for efficiently transmitting and receiving point clouds.

[0007] Another object of the present disclosure is to provide a point cloud data sending device, a point cloud data sending method, a point cloud data receiving device and a point cloud data receiving method for solving delay and encoding / decoding complexity.

[0008] Another object of the present disclosure is to provide a point cloud data sending device, a point cloud data sending method, a point cloud data receiving device and a point cloud data receiving method for providing optimized point cloud content to users by signaling viewport-related signaling of point cloud data.

[0009] Another object of the present disclosure is to provide a point cloud data sending device, a point cloud data sending method, a point cloud data receiving device and a point cloud data receiving method for providing optimized point cloud content to users by allowing viewport information, recommended viewport information and initial viewing orientation (i.e., viewport) for data processing and rendering in a V-PCC bitstream to be sent in the bitstream.

[0010] Additional advantages, objects, and features of the present disclosure will be set forth in part in the following description and will become apparent to those skilled in the art upon examination of the following or may be learned from practice of the present disclosure. The objects and other advantages of the present disclosure may be realized and obtained by the structures particularly pointed out in the written description and claims and the accompanying drawings.

[0011] Technical Solution

[0012] To achieve these objectives and other advantages and in accordance with the purposes of the present disclosure, as embodied and broadly described herein, a method of transmitting point cloud data may include the following steps: encoding the point cloud data; and transmitting a bitstream including the point cloud data and signaling information.

[0013] According to an embodiment, the point cloud data may include geometry data, attribute data, and occupancy map data, which are encoded by a video-based point cloud compression (V-PCC) scheme.

[0014] According to an embodiment, the signaling information may include viewport-related information of a viewport determined according to the position and orientation of the camera or the user.

[0015] According to an embodiment, viewport-related information may include coordinate information in a 3-dimensional (3D) space of the camera or user, direction vector information indicating the direction the camera or user is looking, up vector information indicating the upward direction of the camera or user, and right vector information indicating the right direction of the camera or user.

[0016] According to an embodiment, the viewport-related information may include horizontal field of view (FOV) information and vertical FOV information for generating a viewport.

[0017] According to an embodiment, the point cloud data transmitting apparatus may include an encoder for encoding point cloud data and a transmitter for transmitting a bit stream including the point cloud data and signaling information.

[0018] According to an embodiment, the point cloud data may include geometry data, attribute data, and occupancy map data encoded by a video-based point cloud compression (V-PCC) scheme.

[0019] According to an embodiment, the signaling information may include viewport-related information of a viewport determined according to the position and orientation of the camera or the user.

[0020] According to an embodiment, viewport-related information may include coordinate information in a 3-dimensional (3D) space of the camera or user, direction vector information indicating the direction the camera or user is looking, up vector information indicating the upward direction of the camera or user, and right vector information indicating the right direction of the camera or user.

[0021] According to an embodiment, the viewport-related information may include horizontal field of view (FOV) information and vertical FOV information for generating a viewport.

[0022] According to an embodiment, a point cloud data receiving method may include receiving a bit stream including point cloud data and signaling information, decoding the point cloud data, and rendering the decoded point cloud data.

[0023] According to an embodiment, the signaling information may include viewport-related information of a viewport determined according to the position and orientation of the camera or the user.

[0024] According to an embodiment, viewport-related information may include coordinate information in a 3-dimensional (3D) space of the camera or user, direction vector information indicating the direction the camera or user is looking, up vector information indicating the upward direction of the camera or user, and right vector information indicating the right direction of the camera or user.

[0025] According to an embodiment, the viewport-related information may include horizontal field of view (FOV) information and vertical FOV information for generating a viewport.

[0026] According to an embodiment, the decoded point cloud data may be rendered based on viewport-related information.

[0027] According to an embodiment, the point cloud data receiving device may include: a receiver for receiving a bit stream including point cloud data and signaling information; a decoder for decoding the point cloud data; and a renderer for rendering the decoded point cloud data.

[0028] According to an embodiment, the signaling information may include viewport-related information of a viewport determined according to the position and orientation of the camera or the user.

[0029] According to an embodiment, viewport-related information may include coordinate information in a 3-dimensional (3D) space of the camera or user, direction vector information indicating the direction the camera or user is looking, up vector information indicating the upward direction of the camera or user, and right vector information indicating the right direction of the camera or user.

[0030] According to an embodiment, the viewport-related information may include horizontal field of view (FOV) information and vertical FOV information for generating a viewport.

[0031] According to an embodiment, the decoded point cloud data may be rendered based on viewport-related information.

[0032] Beneficial effects

[0033] The point cloud data transmitting method, point cloud data transmitting device, point cloud data receiving method, and point cloud data receiving device according to the embodiments can provide a point cloud service of good quality.

[0034] The point cloud data transmitting method, point cloud data transmitting device, point cloud data receiving method, and point cloud data receiving device according to the embodiments can implement various video encoding and decoding methods.

[0035] The point cloud data transmitting method, the point cloud data transmitting device, the point cloud data receiving method, and the point cloud data receiving device according to the embodiments can provide general point cloud contents such as autonomous driving services.

[0036] By using the point cloud data transmission method, point cloud data transmission device, point cloud data receiving method, and point cloud data receiving device according to the embodiment, V-PCC bitstream can be configured, and files can be transmitted, received, and stored. Therefore, it is possible to provide an optimal point cloud content service.

[0037] By using the point cloud data transmission method, point cloud data transmission device, point cloud data receiving method, and point cloud data receiving device according to the embodiment, metadata used for data processing and rendering in the V-PCC bitstream can be transmitted and received in the V-PCC bitstream. Therefore, it is possible to provide an optimal point cloud content service.

[0038] Using the point cloud data transmission method, point cloud data transmission device, point cloud data reception method, and point cloud data reception device according to the embodiments, a player or the like can achieve spatial or partial access to point cloud objects / content based on the user's viewport. Therefore, point cloud bitstreams can be efficiently accessed and processed based on the user's viewport.

[0039] The point cloud data transmitting method, point cloud data transmitting device, point cloud data receiving method, and point cloud data receiving device according to the embodiments can provide a bounding box for partial and / or spatial access to point cloud content and its signaling information. Therefore, point cloud content can be accessed in various ways on the receiving side, taking into account the player environment or user environment.

[0040] The point cloud data transmitting method and transmitting apparatus according to the embodiment can provide 3D region information about point cloud content and 2D region-related metadata in the associated video or atlas frame to support spatial / partial access to the point cloud content according to the user viewport.

[0041] The point cloud data transmitting method and transmitting apparatus according to the embodiment can process signaling of 3D region information of a point cloud in a point cloud bitstream and 2D region-related metadata in an associated video or atlas frame.

[0042] By using the point cloud data receiving method and receiving device according to the embodiment, point cloud content can be efficiently accessed based on the storage and signaling of 3D region information about the point cloud in the point cloud bitstream and 2D region-related metadata in the associated video or atlas frame.

[0043] By using the point cloud data receiving method and receiving device according to the embodiment, point cloud content can be provided in consideration of the user environment based on 3D area information about the point cloud associated with the image item in the file and 2D area related information in the video or atlas frame associated therewith. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this application. The accompanying drawings illustrate embodiments of the present disclosure and together with the description serve to explain the principles of the present disclosure. In the drawings:

[0045] Figure 1 An exemplary structure of a transmission / reception system for providing point cloud content according to an embodiment is shown.

[0046] Figure 2 Shown is the capture of point cloud data according to an embodiment.

[0047] Figure 3 Shown are exemplary point clouds, geometry, and texture images according to embodiments.

[0048] Figure 4 An exemplary V-PCC encoding process according to an embodiment is shown.

[0049] Figure 5 Shown are examples of tangent planes and normal vectors of surfaces according to embodiments.

[0050] Figure 6 An exemplary bounding box of a point cloud is shown, according to an embodiment.

[0051] Figure 7 An example of determination of the positions of individual patches on an occupancy map according to an embodiment is shown.

[0052] Figure 8 Exemplary relationships between the normal axis, the tangential axis, and the dual tangential axes according to embodiments are shown.

[0053] Figure 9 Exemplary configurations of the minimum mode and the maximum mode of the projection mode according to the embodiment are shown.

[0054] Figure 10 An exemplary EDD code according to an embodiment is shown.

[0055] Figure 11 An example of recoloring based on color values ​​of neighboring points according to an embodiment is shown.

[0056] Figure 12 An example of push-pull background fill according to an embodiment is shown.

[0057] Figure 13 An exemplary possible traversal order of a 4*4 block according to an embodiment is shown.

[0058] Figure 14 An exemplary optimal traversal order according to an embodiment is shown.

[0059] Figure 15 An exemplary 2D video / image encoder according to an embodiment is shown.

[0060] Figure 16 An exemplary V-PCC decoding process according to an embodiment is shown.

[0061] Figure 17 An exemplary 2D video / image decoder according to an embodiment is shown.

[0062] Figure 18 is a flowchart illustrating the operation of the transmitting device according to an embodiment of the present disclosure.

[0063] Figure 19 is a flowchart illustrating the operation of the receiving device according to the embodiment.

[0064] Figure 20 An exemplary architecture for V-PCC based storage and streaming of point cloud data is shown, according to an embodiment.

[0065] Figure 21 is an exemplary block diagram of an apparatus for storing and transmitting point cloud data according to an embodiment.

[0066] Figure 22 is an exemplary block diagram of a point cloud data receiving apparatus according to an embodiment.

[0067] Figure 23 An exemplary structure that can be operated in conjunction with a method / apparatus for transmitting / receiving point cloud data according to an embodiment is shown.

[0068] Figure 24 An exemplary structure of a V-PCC bitstream according to an embodiment is shown.

[0069] Figure 25 is a diagram showing the structure of a multi-track V-PCC file according to an embodiment.

[0070] Figure 26 is a table showing an example of track types of PCC data assigned to data_type according to an embodiment.

[0071] Figure 27 is a table showing examples of attribute types assigned to attribute_type according to an embodiment.

[0072] Figure 28 An example of the syntax structure of profile_tier_level() according to an embodiment is shown.

[0073] Figure 29 is a table showing examples of types of PCC data in a reference track according to an embodiment.

[0074] Figure 30 is a table showing examples of attribute types according to an embodiment.

[0075] Figure 31 is a table showing examples of attribute types according to an embodiment.

[0076] Figure 32 An example of dividing a patch frame into a plurality of tiles by dividing the patch frame into one or more tile rows and one or more tile columns is shown.

[0077] Figure 33 An example of a syntax structure of each V-PCC unit according to an embodiment is shown.

[0078] Figure 34 An example of a syntax structure of a V-PCC unit header according to an embodiment is shown.

[0079] Figure 35 An example of the type of a V-PCC unit assigned to the vpcc_unit_type field according to an embodiment is shown.

[0080] Figure 36 An example of a syntax structure of pcm_separate_video_data() included in a V-PCC unit header according to an embodiment is shown.

[0081] Figure 37 An example of a syntax structure of a V-PCC unit payload according to an embodiment is shown.

[0082] Figure 38 An example of a syntax structure of sequence_parameter_set() included in a V-PCC unit payload according to an embodiment is shown.

[0083] Figure 39 An example of a syntax structure of a patch data group according to an embodiment is shown.

[0084] Figure 40 is a table showing an example of the type of patch data group assigned to the pdg_unit_type field of the patch data group according to an embodiment.

[0085] Figure 41An example of the syntax structure of a patch data group unit payload according to an embodiment is shown.

[0086] Figure 42 An example of a syntax structure of a supplemental enhancement information (SEI) message (sei_message()) according to an embodiment is shown.

[0087] Figure 43 An example of a V-PCC bitstream structure according to other embodiments of the present disclosure is shown.

[0088] Figure 44 An example of data carried by a sample stream V-PCC unit in a V-PCC bitstream according to an embodiment is shown.

[0089] Figure 45 An example of a syntax structure of a sample stream V-PCC header included in a V-PCC bitstream according to an embodiment is shown.

[0090] Figure 46 An example of a syntax structure of a sample stream V-PCC unit according to an embodiment is shown.

[0091] Figure 47 An example of a syntax structure of a V-PCC unit according to an embodiment is shown.

[0092] Figure 48 An example of a syntax structure of a V-PCC unit header according to an embodiment is shown.

[0093] Figure 49 An example of a V-PCC unit type assigned to the vuh_unit_type field according to an embodiment is shown.

[0094] Figure 50 An example of a syntax structure of a V-PCC unit payload (vpcc_unit_payload()) according to an embodiment is shown.

[0095] Figure 51 is a diagram showing a structural example of an atlas substream according to an embodiment.

[0096] Figure 52 An example of a syntax structure of a sample stream NAL header (sample_stream_nal_header()) included in an atlas substream according to an embodiment is shown.

[0097] Figure 53 An example of the syntax structure of a sample stream NAL unit according to an embodiment is shown.

[0098] Figure 54An example of the syntax structure of the SEI message payload (sei_payload()) according to an embodiment is shown.

[0099] Figure 55 An example of a V-PCC sample entry structure according to an embodiment is shown.

[0100] Figure 56 An example of generating a view using a virtual camera and ViewInfoStruct information according to an embodiment is shown.

[0101] Figure 57 An example of creating a view using ObjectRenderingInfoStruct() information according to an embodiment is shown.

[0102] Figure 58 is a diagram showing an exemplary structure for encapsulating non-timed V-PCC data according to an embodiment.

[0103] Figure 59 is a diagram illustrating an example in which a spatial portion of point cloud data in a viewing frustum is actually displayed on a viewport according to an embodiment.

[0104] Figure 60 An exemplary top view of a viewing cone is shown according to an embodiment.

[0105] Figure 61 An exemplary side view of a viewing cone is shown according to an embodiment.

[0106] Figure 62 An example of a method of transmitting point cloud data according to an embodiment is shown.

[0107] Figure 63 An example of a method of receiving point cloud data according to an embodiment is shown. DETAILED DESCRIPTION

[0108] Reference will now be made in detail to preferred embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. The detailed description given below with reference to the accompanying drawings is intended to illustrate exemplary embodiments of the present disclosure and is not intended to illustrate the only embodiments that can be implemented according to the present disclosure. The following detailed description includes specific details in order to provide a thorough understanding of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure can be practiced without these specific details.

[0109] Although most of the terms used in this disclosure are selected from common terms widely used in the art, some terms are arbitrarily selected by the applicant and their meanings are explained in detail in the following description as needed. Therefore, this disclosure should be understood based on the intended meaning of the terms rather than their simple names or meanings.

[0110] Figure 1 An exemplary structure of a transmission / reception system for providing point cloud content according to an embodiment is shown.

[0111] The present disclosure provides a method for providing point cloud content to provide users with various services such as virtual reality (VR), augmented reality (AR), mixed reality (MR), and autonomous driving. Point cloud data according to an embodiment represents data representing an object as points and can be referred to as point cloud, point cloud data, point cloud video data, point cloud image data, etc.

[0112] According to an embodiment, the point cloud data transmitting device 10000 may include a point cloud video acquisition unit 10001, a point cloud video encoder 10002, a file / segment encapsulation module (file / segment encapsulator) 10003 and / or a transmitter (or communication module) 10004. The transmitting device according to the embodiment may obtain and process the point cloud video (or point cloud content) and transmit it. According to an embodiment, the transmitting device may include a fixed station, a base transceiver system (BTS), a network, an artificial intelligence (AI) device and / or system, a robot, and an AR / VR / XR device and / or server. According to an embodiment, the transmitting device 10000 may include a device configured to communicate with a base station and / or other wireless devices using a radio access technology (e.g., 5G New RAT (NR), Long Term Evolution (LTE)), a robot, a vehicle, an AR / VR / XR device, a portable device, a home appliance, an Internet of Things (IoT) device, and an AI device / server.

[0113] According to an embodiment, the point cloud video acquisition unit 10001 acquires a point cloud video by capturing, synthesizing, or generating a point cloud video.

[0114] The point cloud video encoder 10002 according to the embodiment encodes the point cloud video data acquired from the point cloud video acquisition unit 10001. According to the embodiment, the point cloud video encoder 10002 may be referred to as a point cloud encoder, a point cloud data encoder, an encoder, etc. The point cloud compression encoding (encoding) according to the embodiment is not limited to the above-mentioned embodiment. The point cloud video encoder may output a bitstream including the encoded point cloud video data. The bitstream may include not only the encoded point cloud video data but also signaling information related to the encoding of the point cloud video data.

[0115] According to an embodiment, the point cloud video encoder 10002 may support a geometry-based point cloud compression (G-PCC) encoding scheme and / or a video-based point cloud compression (V-PCC) encoding scheme. In addition, the point cloud video encoder 10002 may encode point clouds (referred to as point cloud data or points) and / or signaling data associated with the point clouds.

[0116] According to an embodiment, the file / segment encapsulation module 10003 encapsulates point cloud data in the form of files and / or segments. According to an embodiment, the point cloud data transmission method / device can transmit point cloud data in the form of files and / or segments.

[0117] The transmitter (or communication module) 10004 according to the embodiment transmits the encoded point cloud video data in the form of a bit stream. According to the embodiment, the file or fragment can be sent to the receiving device via a network or stored in a digital storage medium (e.g., USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc.). The transmitter according to the embodiment can communicate with the receiving device (or receiver) via a 4G, 5G, 6G network or the like via wired / wireless communication. In addition, the transmitter can perform necessary data processing operations based on the network system (e.g., a 4G, 5G or 6G communication network system). The transmitting device can send the encapsulated data in an on-demand manner.

[0118] According to an embodiment, the point cloud data receiving device 10005 may include a receiver 10006, a file / segment decapsulator (or a file / segment decapsulation module) 10007, a point cloud video decoder 10008, and / or a renderer 10009. According to an embodiment, the receiving device may include a device configured to communicate with a base station and / or other wireless devices using a radio access technology (e.g., 5G New RAT (NR), Long Term Evolution (LTE)), a robot, a vehicle, an AR / VR / XR device, a portable device, a home appliance, an Internet of Things (IoT) device, and an AI device / server.

[0119] According to an embodiment, the receiver 10006 receives a bitstream containing point cloud video data. According to an embodiment, the receiver 10006 may send feedback information to the point cloud data transmitting apparatus 10000.

[0120] The file / fragment decapsulation module 10007 decapsulates files and / or fragments containing point cloud data.

[0121] The point cloud video decoder 10008 decodes the received point cloud video data.

[0122] The renderer 10009 renders the decoded point cloud video data. Depending on the embodiment, the renderer 10009 may send feedback information obtained on the receiving side to the point cloud video decoder 10008. Depending on the embodiment, the point cloud video data may carry the feedback information to the receiver 10006. Depending on the embodiment, the feedback information received by the point cloud transmitting device may be provided to the point cloud video encoder 10002.

[0123] The arrow indicated by the dotted line in the figure represents the transmission path of the feedback information obtained by the receiving device 10005. Feedback information is information reflecting the interactivity with the user consuming the point cloud content, and includes user information (e.g., head orientation information, viewport information, etc.). Specifically, when the point cloud content is content for a service that requires interaction with the user (e.g., an autonomous driving service, etc.), the feedback information can be provided to the content sender (e.g., the sending device 10000) and / or the service provider. Depending on the embodiment, the feedback information may be used in the receiving device 10005 and the sending device 10000, and may not be provided.

[0124] According to an embodiment, head orientation information refers to information regarding the user's head position, orientation, angle, movement, and the like. According to an embodiment, the receiving device 10005 may calculate viewport information based on the head orientation information. Viewport information may be information regarding the area of ​​the point cloud video that the user is viewing. The viewpoint (or orientation) is the point from which the user views the point cloud video and may refer to the center point of the viewport area. In other words, the viewport is the area centered on the viewpoint, and the size and shape of the area may be determined by the field of view (FOV). In other words, the viewport is determined based on the position and viewpoint (or orientation) of the visual camera or user, and point cloud data is rendered within the viewport based on the viewport information. Therefore, in addition to head orientation information, the receiving device 10005 may also extract viewport information based on the vertical or horizontal FOV supported by the device. Furthermore, the receiving device 10005 may perform gaze analysis to examine how the user consumes the point cloud, the area within the point cloud video that the user is gazing at, the duration of the gaze, and other information. According to an embodiment, the receiving device 10005 may send feedback information including the gaze analysis results to the sending device 10000. Depending on the embodiment, the feedback information may be obtained during the rendering and / or display process. Depending on the embodiment, the feedback information may be obtained by one or more sensors included in the receiving device 10005. In addition, depending on the embodiment, the feedback information may be obtained by the renderer 10009 or a separate external element (or device, component, etc.). Figure 1 The dotted line in represents the process of sending the feedback information obtained by the renderer 10009. The point cloud content providing system can process (encode / decode) the point cloud data based on the feedback information. Therefore, the point cloud video decoder 10008 can perform a decoding operation based on the feedback information. The receiving device 10005 can send the feedback information to the sending device. The sending device (or the point cloud video encoder 10002) can perform an encoding operation based on the feedback information. Therefore, the point cloud content providing system can efficiently process necessary data (for example, point cloud data corresponding to the user's head position) based on the feedback information instead of processing (encoding / decoding) all the point cloud data, and provide the point cloud content to the user.

[0125] According to an embodiment, the transmitting device 10000 may be referred to as an encoder, a transmitting device, a transmitter, etc., and the receiving device 10005 may be referred to as a decoder, a receiving device, a receiver, etc.

[0126] According to the embodiment Figure 1 Point cloud data processed in a point cloud content providing system (through a series of processes such as acquisition, encoding, transmission, decoding, and rendering) may be referred to as point cloud content data or point cloud video data. Depending on the embodiment, point cloud content data may be used as a concept encompassing metadata or signaling information related to point cloud data.

[0127] Figure 1 The elements of the illustrated point cloud content providing system may be implemented by hardware, software, a processor, and / or a combination thereof.

[0128] Embodiments may provide a method of providing point cloud content to provide users with various services such as virtual reality (VR), augmented reality (AR), mixed reality (MR), and autonomous driving.

[0129] To provide a point cloud content service, a point cloud video must first be acquired. This acquired point cloud video undergoes a series of processing steps and is then sent to a receiving end. The receiving end then processes the received data back into the original point cloud video and renders the processed point cloud video. This point cloud video can then be provided to the user. Embodiments provide a method for efficiently performing this series of processing steps.

[0130] All processing for providing point cloud content services (point cloud data sending method and / or point cloud data receiving method) may include acquisition processing, encoding processing, transmission processing, decoding processing, rendering processing and / or feedback processing.

[0131] According to an embodiment, the process of providing point cloud content (or point cloud data) may be referred to as a point cloud compression process. According to an embodiment, the point cloud compression process may refer to a video-based point cloud compression (V-PCC) process.

[0132] Each element of the point cloud data transmitting apparatus and the point cloud data receiving apparatus according to the embodiment may be hardware, software, a processor and / or a combination thereof.

[0133] The point cloud compression system may include a transmitting device and a receiving device. Depending on the embodiment, the transmitting device may be referred to as an encoder, a transmitting device, a transmitter, a point cloud data transmitting device, etc. Depending on the embodiment, the receiving device may be referred to as a decoder, a receiving device, a receiver, a point cloud data receiving device, etc. The transmitting device may encode the point cloud video to output a bitstream and transmit it to the receiving device in the form of a file or stream (streaming segment) via a digital storage medium or a network. Digital storage media may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD.

[0134] The sending device may include a point cloud video acquisition unit, a point cloud video encoder, a file / segment encapsulator and a sending unit (sender), such as Figure 1 The receiving device may include a receiver, a file / segment decapsulator, a point cloud video decoder and a renderer, as shown in FIG. Figure 1 As shown. The encoder may be referred to as a point cloud video / image / picture / frame encoder, and the decoder may be referred to as a point cloud video / image / picture / frame decoding device. The renderer may include a display. The renderer and / or the display may be configured as a separate device or an external component. The transmitting device and the receiving device may also include a separate internal or external module / unit / component for feedback processing. Depending on the embodiment, each element in the transmitting device and the receiving device may be configured by hardware, software and / or a processor.

[0135] According to an embodiment, the operation of the receiving device may be a reverse process of the operation of the transmitting device.

[0136] The point cloud video acquirer can acquire point cloud video by capturing, editing, or generating point cloud video. During the acquisition process, data representing the 3D positions (x, y, z) and attributes (color, reflectivity, transparency, etc.) of multiple points can be generated, such as files in the Polygon File Format (PLY) (or Stanford Triangle Format). For videos with multiple frames, one or more files can be acquired. During the capture process, point cloud-related metadata (e.g., capture-related metadata) can be generated.

[0137] A point cloud data transmitting apparatus according to an embodiment may include an encoder configured to encode point cloud data and a transmitter configured to transmit the point cloud data or a bit stream including the point cloud data.

[0138] A point cloud data receiving apparatus according to an embodiment may include a receiver configured to receive a bitstream including point cloud data, a decoder configured to decode the point cloud data, and a renderer configured to render the point cloud data.

[0139] The method / apparatus according to the embodiment represents a point cloud data sending apparatus and / or a point cloud data receiving apparatus.

[0140] Figure 2 Shown is the capture of point cloud data according to an embodiment.

[0141] Point cloud data (point cloud video data) according to an embodiment may be acquired by a camera, etc. Capturing techniques according to an embodiment may include, for example, inward-facing and / or outward-facing.

[0142] In the inward facing according to an embodiment, one or more cameras facing inwardly toward an object of point cloud data may photograph the object from outside the object.

[0143] In outward facing according to an embodiment, one or more cameras facing outward toward an object of point cloud data may photograph the object. For example, according to an embodiment, there may be four cameras.

[0144] Point cloud data or point cloud content according to an embodiment may be a video or still image of an object / environment represented in various types of 3D space. According to an embodiment, point cloud content may include video / audio / image of an object.

[0145] Devices for capturing point cloud content can include a combination of a depth-capable camera (a combination of an infrared pattern projector and an infrared camera) and an RGB camera capable of extracting color information corresponding to the depth information. Alternatively, depth information can be extracted using LiDAR, a radar system that measures the position coordinates of reflectors by emitting laser pulses and measuring the return time. Geometric shapes composed of points in 3D space can be extracted from the depth information, and attributes representing the color / reflectivity of each point can be extracted from the RGB information. Point cloud content can include information about the position (x, y, z) and the color (YCbCr or RGB) or reflectivity (r) of each point. Point cloud content can use both outward-facing techniques, which capture the external environment, and inward-facing techniques, which capture the central object. In VR / AR environments, when objects (e.g., core objects such as characters, players, objects, or actors) are incorporated into point cloud content that can be viewed from any direction (360 degrees), the capture camera can be configured using the inward-facing technique. When the current surrounding environment is incorporated into the point cloud content in vehicle modes such as autonomous driving, the capture camera can be configured using the outward-facing technique. Since point cloud content may be captured by multiple cameras, a camera calibration process may need to be performed to configure the global coordinate system for the cameras before capturing the content.

[0146] Point cloud content can be videos or still images of objects / environments existing in various types of 3D spaces.

[0147] Furthermore, in the point cloud content acquisition method, any point cloud video can be compiled based on captured point cloud videos. Alternatively, when providing a computer-generated point cloud video of a virtual space, it may not be necessary to capture the video using an actual camera. In this case, the capture process can simply be replaced by a process that generates the relevant data.

[0148] The captured point cloud video may need to be post-processed to improve the quality of the content. In the video capture process, the maximum / minimum depth can be adjusted within the range provided by the camera device. Even after the adjustment, point data of unwanted areas may still exist. Therefore, post-processing may be performed to remove unwanted areas (e.g., background) or to identify connected spaces and fill spatial holes. In addition, the point clouds extracted from cameras sharing a spatial coordinate system can be integrated into one piece of content by transforming each point into a global coordinate system based on the position coordinates of each camera obtained through a calibration process. Thus, a piece of point cloud content with a wide range can be generated, or point cloud content with a high density of points can be obtained.

[0149] The point cloud video encoder 10002 can encode an input point cloud video into one or more video streams. A point cloud video may include multiple frames, each of which may correspond to a still image / picture. In this specification, a point cloud video may include point cloud images / frames / pictures / video / audio. In addition, the term "point cloud video" may be used interchangeably with point cloud images / frames / pictures. The point cloud video encoder 10002 may perform video-based point cloud compression (V-PCC) processing. For compression and coding efficiency, the point cloud video encoder may perform a series of processes such as prediction, transformation, quantization, and entropy coding. The encoded data (encoded video / image information) may be output in the form of a bitstream. Based on the V-PCC processing, the point cloud video encoder may encode the point cloud video by dividing it into a geometry video, an attribute video, an occupancy map video, and auxiliary information (or auxiliary data) (described later). The geometry video may include a geometry image, the attribute video may include an attribute image, and the occupancy map video may include an occupancy map image. The auxiliary information may include auxiliary patch information. The attribute video / image may include a texture video / image.

[0150] The file / segment encapsulator (file / segment encapsulation module) 10003 can encapsulate the encoded point cloud video data and / or metadata related to the point cloud video in the form of a file, for example. Here, the metadata related to the point cloud video can be received from a metadata processor. The metadata processor may be included in the point cloud video encoder 10002 or configured as a separate component / module. The file / segment encapsulator 10003 can encapsulate data in a file format such as ISOBMFF or process data in the form of DASH segments, etc. Depending on the embodiment, the file / segment encapsulator 10003 may include metadata related to the point cloud video in a file format. The point cloud video metadata may be included in boxes at various levels of the ISOBMFF file format, or as data in separate tracks within a file. Depending on the embodiment, the file / segment encapsulator 10003 may encapsulate the metadata related to the point cloud video in a file. A transmission processor may perform transmission processing on the point cloud video data encapsulated in the file format. The transmission processor may be included in the transmitter 10004 or configured as a separate component / module. The transmission processor may process the point cloud video data according to a transmission protocol. The transmission process may include a process of transmitting via a broadcast network and a process of transmitting through a broadband. According to an embodiment, the transmission processor may receive point cloud video related metadata together with the point cloud video data from the metadata processor and perform processing of the point cloud video data for transmission.

[0151] The transmitter 10004 can transmit the encoded video / image information or data output in the form of a bitstream in the form of a file or stream to the receiver 10006 of the receiving device via a digital storage medium or network. Digital storage media may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD. The transmitter may include components for generating a media file in a predetermined file format and may also include components for transmission via a broadcast / communication network. The receiver may extract the bitstream and transmit the extracted bitstream to a decoding device.

[0152] Receiver 10006 can receive point cloud video data transmitted by the point cloud video transmitting device according to the present disclosure. Depending on the transmission channel, the receiver can receive point cloud video data via a broadcast network or via broadband. Alternatively, the point cloud video data can be received via a digital storage medium.

[0153] The receiving processor may process the received point cloud video data according to the transmission protocol. The receiving processor may be included in the receiver 10006 or may be configured as a separate component / module. The receiving processor may reverse the above-described processing of the transmitting processor so that the processing corresponds to the transmission processing performed on the transmitting side. The receiving processor may transmit the acquired point cloud video data to the file / segment decapsulator 10007 and transmit the acquired point cloud video-related metadata to a metadata processor (not shown). The point cloud video-related metadata acquired by the receiving processor may take the form of a signaling table.

[0154] The file / segment decapsulator (file / segment decapsulation module) 10007 can decapsulate the point cloud video data received from the receiving processor in the form of a file. The file / segment decapsulator 10007 can decapsulate the file according to ISOBMFF, etc., and can obtain a point cloud video bitstream or point cloud video related metadata (metadata bitstream). The obtained point cloud video bitstream can be transmitted to the point cloud video decoder 10008, and the obtained point cloud video related metadata (metadata bitstream) can be transmitted to a metadata processor (not shown). The point cloud video bitstream may include metadata (metadata bitstream). The metadata processor may be included in the point cloud video decoder 10008, or may be configured as a separate component / module. The point cloud video related metadata obtained by the file / segment decapsulator 10007 may take the form of a box or track in the file format. When necessary, the file / segment decapsulator 10007 may receive the metadata required for decapsulation from the metadata processor. The point cloud video related metadata may be transmitted to the point cloud video decoder 10008 and used in the point cloud video decoding process, or may be transmitted to the renderer 10009 and used in the point cloud video rendering process.

[0155] The point cloud video decoder 10008 can receive a bitstream and decode the video / image by performing operations corresponding to those of the point cloud video encoder. In this case, the point cloud video decoder 10008 can decode the point cloud video by separating the point cloud video into a geometry video, an attribute video, an occupancy map video, and auxiliary information as described below. The geometry video may include a geometry image, the attribute video may include an attribute image, the occupancy map video may include an occupancy map image, the auxiliary information may include auxiliary patch information, and the attribute video / image may include a texture video / image.

[0156] The 3D geometry can be reconstructed based on the decoded geometry image, occupancy map, and auxiliary patch information, and then subjected to a smoothing process. A color point cloud image / screen can be reconstructed by assigning color values ​​to the smoothed 3D geometry based on the texture image. The renderer 10009 can render the reconstructed geometry and color point cloud image / screen. The rendered video / image can be displayed on a display (not shown). The user can view all or part of the rendered result on a VR / AR display or a typical display.

[0157] Feedback processing may include transmitting various types of feedback information that can be obtained in the rendering / display process to a decoder on the sending side or the receiving side. Feedback processing can provide interactivity when consuming point cloud videos. Depending on the embodiment, head orientation information, viewport information indicating the area currently viewed by the user, etc. can be transmitted to the sending side during feedback processing. Depending on the embodiment, the user can interact with things implemented in the VR / AR / MR / autonomous driving environment. In this case, information related to the interaction can be transmitted to the sending side or the service provider during the feedback processing. Depending on the embodiment, feedback processing can be skipped.

[0158] The head orientation information may represent information about the position, angle, and motion of the user's head. Based on this information, information about the area of ​​the point cloud video that the user is currently viewing (i.e., viewport information) may be calculated.

[0159] Viewport information can be information about the area of ​​the point cloud video that a user is currently viewing. This information can be used to perform gaze analysis to examine how the user consumes the point cloud video, the area of ​​the point cloud video the user is gazing at, and how long the user gazes at that area. Gaze analysis can be performed on the receiving side, and the analysis results can be transmitted to the sending side over a feedback channel. Devices such as VR / AR / MR displays can extract the viewport area based on the position / orientation of the user's head, the vertical or horizontal FOV supported by the device, and other factors.

[0160] Depending on the implementation, this feedback information can be not only transmitted to the sending side but also consumed by the receiving side. Specifically, decoding and rendering processing on the receiving side can be performed based on this feedback information. For example, based on head orientation information and / or viewport information, only the point cloud video of the area currently being viewed by the user can be prioritized for decoding and rendering.

[0161] Here, the viewport or viewport area may refer to the area of ​​the point cloud video currently being viewed by the user. The viewpoint is the point in the point cloud video that the user is viewing and may represent the center point of the viewport area. In other words, the viewport is the area surrounding the viewpoint, and the size and shape of the area may be determined by the field of view (FOV).

[0162] The present disclosure relates to point cloud video compression as described above. For example, the methods / implementations disclosed in the present disclosure may be applied to the Point Cloud Compression or Point Cloud Coding (PCC) standard of the Moving Picture Experts Group (MPEG) or next-generation video / image coding standards.

[0163] As used herein, a picture / frame may generally refer to a unit representing one image in a specific time interval.

[0164] A pixel or picture element can be the smallest unit that constitutes a picture (or image). In addition, "sample" can be used as a term corresponding to a pixel. A sample can generally represent a pixel or a pixel value. It can represent only the pixel / pixel value of the luma component, only the pixel / pixel value of the chroma component, or only the pixel / pixel value of the depth component.

[0165] A unit may represent a basic unit of image processing. A unit may include at least one of a specific area of ​​a picture and information related to the area. In some cases, a unit may be used interchangeably with terms such as a block, an area, or a module. In general, an M×N block may include samples (or sample arrays) or a set (or array) of transform coefficients arranged in M ​​columns and N rows.

[0166] Figure 3 Examples of point clouds, geometric images, and texture images according to embodiments are shown.

[0167] The point cloud according to the embodiment may be input to the Figure 4 The V-PCC encoding process is performed to generate a geometric image and a texture image. According to an embodiment, the point cloud may have the same meaning as point cloud data.

[0168] like Figure 3 As shown, the left part shows a point cloud, where the point cloud object is located in 3D space and can be represented by a bounding box or the like. Figure 3 The middle part shows the geometric image, and Figure 3 The right portion of FIG shows a texture image (non-fill image). In the present disclosure, a geometry image may be referred to as a geometry patch frame / picture or a geometry frame / picture, and a texture image may be referred to as an attribute patch frame / picture or an attribute frame / picture.

[0169] Video-based point cloud compression (V-PCC) according to an embodiment is a method for compressing 3D point cloud data based on a 2D video codec such as High Efficiency Video Coding (HEVC) or Versatile Video Coding (VVC). The data and information that can be generated in the V-PCC compression process are as follows:

[0170] Occupancy map: This is a binary map that uses a value of 0 or 1 to indicate whether data exists at corresponding locations in the 2D plane when the points that make up the point cloud are divided into patches and mapped to the 2D plane. The occupancy map can represent a 2D array corresponding to an atlas, and the values ​​of the occupancy map can indicate whether each sample location in the atlas corresponds to a 3D point. The atlas is an object that contains information about the 2D patches for each point cloud frame. For example, the atlas can include the 2D arrangement and size of the patches, the location of the corresponding 3D region within the 3D points, the projection plan, and the level of detail parameters.

[0171] Patch: A collection of points that make up a point cloud, which indicates that points belonging to the same patch are adjacent to each other in 3D space and are mapped in the same direction between the 6-sided bounding box planes in the process of mapping to a 2D image.

[0172] Geometry image: This is an image in the form of a depth map that presents positional information (geometry) about the individual points that make up the point cloud, patch by patch. A geometry image can consist of pixel values ​​for one channel. The geometry represents a set of coordinates associated with a point cloud frame.

[0173] Texture Image: This is an image that represents the color information of each point in the point cloud on a patch-by-patch basis. A texture image may consist of pixel values ​​for multiple channels (e.g., R, G, and B). Textures are included in attributes. Depending on the implementation, textures and / or attributes may be interpreted as the same object and / or may contain a context.

[0174] Auxiliary patch information: This indicates the metadata required to reconstruct the point cloud using each patch. The auxiliary patch information may include information about the position, size, etc. of the patch in 2D / 3D space.

[0175] Point cloud data according to an embodiment, eg, a V-PCC component, may include an atlas, an occupancy map, geometry, and attributes.

[0176] An atlas represents a set of 2D bounding boxes. It can be a set of patches, for example, patches projected into a rectangular frame corresponding to a 3D bounding box in 3D space, which can represent a subset of a point cloud. In this case, the patch can represent a rectangular area in the atlas corresponding to a rectangular area in a planar projection. Furthermore, patch data can represent the data required to perform a 2D to 3D transformation on the patches included in the atlas. Furthermore, a patch data set is also referred to as an atlas.

[0177] Attributes can represent scalars or vectors associated with each point in the point cloud. For example, attributes can include color, reflectivity, surface normal, timestamp, material ID.

[0178] Point cloud data according to an embodiment represents PCC data according to a video-based point cloud compression (V-PCC) scheme. Point cloud data may include multiple components. For example, it may include occupancy maps, patches, geometry, and / or textures.

[0179] Figure 4 An example of a point cloud video encoder according to an embodiment is shown.

[0180] Figure 4 A V-PCC encoding process for generating and compressing occupancy maps, geometry images, texture images, and auxiliary patch information is shown. Figure 4 The V-PCC coding process can be done by Figure 1 The point cloud video encoder 10002 processes. Figure 4 Each element may be executed by software, hardware, a processor and / or a combination thereof.

[0181] The patch generator 40000 receives a point cloud frame (which may be in the form of a bitstream containing point cloud data) and generates a patch based on the point cloud data. In addition, the patch generator 40000 generates patch information including information about patch generation.

[0182] Patch packing or patch packer 40001 packs one or more patches. In addition, the patch packer 40001 generates an occupancy map containing information about patch packing.

[0183] The geometric image generation or geometric image generator 40002 generates a geometric image based on the point cloud data, patch information (or auxiliary information) and / or occupancy map information. The geometric image refers to data containing geometry related to the point cloud data (i.e., 3D coordinate values ​​of points) and refers to a geometric framework.

[0184] Texture image generation or texture image generator 40003 generates a texture image based on point cloud data, patches, packed patches, patch information (or auxiliary information), and / or smoothed geometry. The texture image refers to an attribute frame. That is, the texture image can be further generated based on smoothed geometry generated by smoothing based on patch information.

[0185] Smoothing or smoother 40004 can reduce or eliminate errors contained in the image data. For example, the reconstructed geometric image is smoothed based on the patch information. In other words, the parts that may cause errors between the data can be smoothly filtered out to generate smooth geometry.

[0186] Auxiliary patch information compression or auxiliary patch information compressor 40005 can compress auxiliary patch information related to the patch information generated in the patch generation. In addition, the auxiliary patch information compressed in the auxiliary patch information compressor 40005 can be sent to the multiplexer 40013. The auxiliary patch information can be used in the geometric image generator 40002.

[0187] Image filling or image fillers 40006 and 40007 can fill geometric images and texture images respectively. Filling data can be filled into geometric images and texture images.

[0188] Group dilation or group dilator 40008 can add data to a texture image in a similar manner to image padding.Auxiliary patch information can be inserted into the texture image.

[0189] Video compression or video compressors 40009, 40010, and 40011 can compress a filled geometry image, a filled texture image, and / or an occupancy map, respectively. In other words, video compressors 40009, 40010, and 40011 can compress an input geometry frame, an attribute frame, and / or an occupancy map frame, respectively, to output a video bitstream of a geometry image, a video bitstream of a texture image, and a video bitstream of an occupancy map. Video compression can encode geometry information, texture information, and occupancy information.

[0190] Entropy compression or entropy compressor 40012 can compress the occupancy map based on an entropy scheme.

[0191] Depending on the implementation, entropy compression and / or video compression may be performed on the occupancy map frame depending on whether the point cloud data is lossless and / or lossy.

[0192] The multiplexer 40013 multiplexes the compressed geometry video bitstream, the compressed texture image video bitstream, the compressed occupancy map video bitstream, and the compressed auxiliary patch information bitstream from each compressor into one bitstream.

[0193] The above blocks may be omitted or may be replaced by blocks having similar or identical functions. Figure 4 Each block shown may function as at least one of a processor, software, and hardware.

[0194] According to the embodiment Figure 4 The detailed operation of each process is described as follows.

[0195] Patch Generation (40000)

[0196] The patch generation process is a process of dividing a point cloud into patches (mapping units) in order to map the point cloud to a 2D image. The patch generation process can be divided into three steps: normal value calculation, segmentation, and patch segmentation.

[0197] Will refer to Figure 5 The normal value calculation process is described in detail.

[0198] Figure 5 Shown are examples of tangent planes and normal vectors of surfaces according to embodiments.

[0199] exist Figure 4 The V-PCC coding process is processed in the patch generator 40000 as follows Figure 5 surface.

[0200] Normal calculation related to patch generation

[0201] Each point of the point cloud has its own direction, which is represented by a 3D vector called a normal vector. Using the neighbors of each point obtained using a KD tree or the like, we can obtain Figure 5 The tangent plane and normal vector of each point constituting the surface of the point cloud are shown. The search range applied to the process of searching for neighbors can be defined by the user.

[0202] A tangent plane is a plane that passes through a point on a surface and completely includes the tangent line to a curve on the surface.

[0203] Figure 6 An exemplary bounding box of a point cloud is shown, according to an embodiment.

[0204] The bounding box according to the embodiment refers to a box for dividing a unit of point cloud data based on a hexahedron in a 3D space.

[0205] A method / apparatus according to an embodiment (eg, patch generator 40000 ) may use a bounding box in a process of generating a patch from point cloud data.

[0206] The bounding box can be used in the process of projecting the target object of the point cloud data onto the plane of each flat face of the hexahedron in the 3D space. The bounding box can be represented by Figure 1 The point cloud video acquisition unit 10001 and the point cloud video encoder 10002 generate and process. In addition, based on the bounding box, Figure 4 The V-PCC encoding process includes patch generation 40000, patch packing 40001, geometric image generation 40002 and texture image generation 40003.

[0207] Segments related to patch generation

[0208] Segmentation is divided into two processes: initial segmentation and refined segmentation.

[0209] The point cloud video encoder 10002 according to the embodiment projects the point onto a face of the bounding box. Specifically, Figure 6As shown, each point that makes up the point cloud is projected onto one of the six faces of a bounding box surrounding the point cloud. Initial segmentation is the process of determining one of the flat faces of the bounding box to which each point is projected.

[0210] It is the normal value corresponding to each of the six planar faces, defined as follows:

[0211] (1.0,0.0,0.0), (0.0,1.0,0.0), (0.0,0.0,1.0), (-1.0,0.0,0.0), (0.0,-1.0,0.0), (0.0,0.0,-1.0).

[0212] As shown in the following formula, the normal vector of each point obtained in the normal value calculation process is and The plane with the largest dot product value is determined as the projection plane of the corresponding point. That is, the plane with the most similar normal vector to the normal vector of the point is determined as the projection plane of the point.

[0213]

[0214] The determined plane can be identified by a cluster index (one of 0 to 5).

[0215] Refined segmentation is a process that enhances the projection planes of each point that makes up the point cloud, determined in the initial segmentation process, by taking into account the projection planes of neighboring points. In this process, the normal score, which indicates the similarity between the normal vector of each point considered in determining the projection plane in the initial segmentation process and the normals of each flat surface of the bounding box, and the smoothness score, which indicates the similarity between the projection plane of the current point and the projection planes of neighboring points, are considered together.

[0216] Score smoothing can be taken into account by assigning weights to the scoring methods. In this case, the weight values ​​can be user-defined. The segmentation refinement can be performed repeatedly, and the number of repetitions can also be defined by the user.

[0217] Patch segmentation related to patch generation

[0218] Patch segmentation is a process of dividing the entire point cloud into patches (a collection of adjacent points) based on the projection plane information of each point constituting the point cloud obtained in the initial / refinement segmentation process. Patch segmentation may include the following steps:

[0219] Use KD tree etc. to calculate the neighboring points of each point that constitutes the point cloud. The maximum number of neighbors can be defined by the user;

[0220] ① When the neighboring points are projected onto the same plane as the current point (when they have the same cluster index), ② Extract the

[0221] The preceding point and the adjacent point act as a patch;

[0222] ③Calculate the geometric value of the extracted patch.

[0223] ④ Repeat steps ② to ③ until there are no unextracted points.

[0224] The occupancy map, geometry image, and texture image of each patch, as well as the size of each patch, are determined through a patch segmentation process.

[0225] Figure 7 An example of determining positions of individual patches on an occupancy map according to an embodiment is shown.

[0226] The point cloud video encoder 10002 according to an embodiment may perform patch packing and generate an occupancy map.

[0227] Patch packing and occupancy map generation (40001)

[0228] This is the process of determining the locations of individual patches in a 2D image in order to map the segmented patches to the 2D image. As a 2D image, an occupancy map is a binary image, using values ​​of 0 or 1 to indicate the presence or absence of data at the corresponding location. The occupancy map consists of blocks, and its resolution is determined by the block size. For example, a 1x1 block achieves pixel-level resolution. The occupancy packing block size can be determined by the user.

[0229] The process of determining the location of each patch on the occupancy map can be configured as follows:

[0230] ① Set all positions on the occupancy map to 0;

[0231] ② Place the patch at a point (u, v) in the occupancy map plane with horizontal coordinates in the range (0, occupancySizeU - patch.sizeU0) and vertical coordinates in the range (0, occupancySizeV - patch.sizeV0);

[0232] ③ Set the point (x, y) in the patch plane whose horizontal coordinates are in the range (0, patch.sizeU0) and vertical coordinates are in the range (0, patch.sizeV0) as the current point;

[0233] ④ Change the position of the point (x, y) in raster order, and if the value of the coordinate (x, y) on the patch occupancy map is 1 (there is data at the point in the patch) and the value of the coordinate (u+x, v+y) on the global occupancy map is 1 (fill the occupancy map with the previous patch), repeat operations ③ and ④. Otherwise, proceed to operation ⑥;

[0234] ⑤ Change the position of (u,v) in raster order and repeat operations ③ to ⑤;

[0235] ⑥ Determine (u, v) as the location of the patch and copy the occupancy map data about the patch to the corresponding part on the global occupancy map; and

[0236] ⑦ Repeat steps ② to ⑥ for the next patch.

[0237] occupancySizeU: Indicates the width of the occupancy map. Its unit is the occupancy packing block size.

[0238] occupancySizeV: Indicates the height of the occupancy map. Its unit is the occupancy packing block size.

[0239] patch.sizeU0: Indicates the width of the occupancy map. Its unit is the occupancy packing block size.

[0240] patch.sizeV0: indicates the height of the occupancy map. Its unit is the occupancy patch size.

[0241] For example, Figure 7 As shown, in the box corresponding to the occupied packed size block, there exists a box corresponding to the patch having the patch size, and the point (x, y) can be located in the box.

[0242] Figure 8 Exemplary relationships between the normal axis, the tangential axis, and the dual tangential axes according to embodiments are shown.

[0243] The point cloud video encoder 10002 according to an embodiment can generate a geometric image. The geometric image refers to image data including geometric information about the point cloud. The geometric image generation process can use the three axes (normal, tangential, and bitangent) of the patch of Figure 8.

[0244] Geometric Image Generation (40002)

[0245] In this process, the depth values ​​of the geometric images constituting each patch are determined, and the entire geometric image is generated based on the positions of the patches determined in the patch packing process described above. The process of determining the depth values ​​of the geometric images constituting each patch can be configured as follows.

[0246] ① Calculate parameters related to the position and size of each patch. The parameters may include the following information: According to an embodiment, the position of the patch is included in the patch information.

[0247] The normal index indicating the normal axis is obtained in the previous patch generation process. The tangential axis is an axis perpendicular to the normal axis that coincides with the horizontal axis u of the patch image, and the bi-tangential axis is an axis perpendicular to the normal axis that coincides with the vertical axis v of the patch image. The three axes can be represented as Figure 8 shown.

[0248] Figure 9 Exemplary configurations of the minimum mode and the maximum mode of the projection mode according to the embodiment are shown.

[0249] The point cloud video encoder 10002 according to an embodiment may perform patch-based projection to generate a geometric image, and the projection mode according to an embodiment includes a minimum mode and a maximum mode.

[0250] The 3D space coordinates of the patch may be calculated based on a minimum-sized bounding box surrounding the patch. For example, the 3D space coordinates may include the minimum tangent value of the patch (on the patch 3D shift tangent axis), the minimum bitangent value of the patch (on the patch 3D shift bitangent axis), and the minimum normal value of the patch (on the patch 3D shift normal axis).

[0251] The 2D size of a patch indicates the horizontal and vertical size of the patch when it is packed into a 2D image. The horizontal size (patch 2D size u) can be obtained as the difference between the maximum tangent value and the minimum tangent value of the bounding box, and the vertical size (patch 2D size v) can be obtained as the difference between the maximum bitangent value and the minimum bitangent value of the bounding box.

[0252] ② Determine the projection mode of the patch. The projection mode can be minimum or maximum. Geometric information about the patch is represented by depth values. When the points that make up the patch are projected along the normal to the patch, two images are generated: an image constructed using the maximum depth value and an image constructed using the minimum depth value.

[0253] In minimum mode, when generating two-layer images d0 and d1, the minimum depth can be configured for d0, and the maximum depth within the surface thickness from the minimum depth can be configured for d1, such as Figure 9 shown.

[0254] For example, when the point cloud is in 2D (such as Figure 9 As shown in FIG, there are multiple patches including multiple points. As shown in the figure, it is indicated that points marked with the same style of shadow can belong to the same patch. The figure shows the processing of projecting a patch marked with a blank.

[0255] When a point marked with a blank is projected left / right, the depth may be incremented by 1 relative to the left side, 0, 1, 2, ..., 6, 7, 8, 9, and the number used to calculate the depth of the point may be marked on the right side.

[0256] The same projection mode can be applied to all point clouds, or different projection modes can be applied to each frame or patch according to user definition. When different projection modes are applied to each frame or patch, the projection mode that can enhance compression efficiency or minimize missing points can be adaptively selected.

[0257] ③ Calculate the depth value of each point. In minimum mode, construct image d0 using depth0, which is the value obtained by subtracting the minimum normal value of the patch (on the patch 3D shift normal axis) calculated in operation 1) from the minimum normal value of the patch (on the patch 3D shift normal axis) for the minimum normal value of each point. If there is another depth value within the range between depth0 and the surface thickness at the same location, that value is set to depth1. Otherwise, the value of depth0 is assigned to depth1. Construct image d1 using the value of depth1.

[0258] For example, when calculating the depth of a point in image d0, the minimum value (4 2 4 4 0 6 0 0 9 9 0 80) may be calculated. When calculating the depth of a point in image d1, the larger value among two or more points may be calculated. When only one point exists, its value may be calculated as (4 4 4 4 6 66 8 9 9 8 8 9). In the process of encoding and reconstructing the points of the patch, some points may be missing (for example, in the figure, eight points are missing).

[0259] In maximum mode, image d0 is constructed using depth0, which is the value obtained by subtracting the minimum normal value of the patch (on the patch 3D shift normal axis) calculated in operation 1) from the minimum normal value of the patch (on the patch 3D shift normal axis) for each point. If another depth value exists within the range between depth0 and the surface thickness at the same location, that value is set to depth1. Otherwise, the value of depth0 is assigned to depth1. Image d1 is constructed using the value of depth1.

[0260] For example, when calculating the depth of a point in image d0, the minimum value (4 4 4 4 6 6 6 8 9 9 8 89) may be calculated. When calculating the depth of a point in image d1, the smaller value among two or more points may be calculated. When only one point exists, its value may be calculated as (4 2 4 4 5 60 6 9 9 0 8 0). In the process of encoding and reconstructing the points of the patch, some points may be missing (for example, in the figure, six points are missing).

[0261] The entire geometric image can be generated by placing the geometric images of the respective patches generated through the above-described process onto the entire geometric image based on the patch position information determined in the patch packing process.

[0262] The generated layer d1 of the entire geometric image can be encoded using various methods. The first method (absolute d1 encoding method) encodes the depth value of the previously generated image d1. The second method (differential encoding method) encodes the difference between the depth value of the previously generated image d1 and the depth value of image d0.

[0263] In the encoding method using depth values ​​of two layers d0 and d1 as described above, if there is another point between the two depths, geometric information about the point is lost in the encoding process, so Enhanced-Delta-Depth (EDD) code can be used for lossless encoding.

[0264] Below, we will refer to Figure 10 Describe the EDD code in detail.

[0265] Figure 10 An exemplary EDD code according to an embodiment is shown.

[0266] In the point cloud video encoder 10002 and / or some / all processes of V-PCC encoding (e.g., video compression 40009), geometric information about points can be encoded based on EOD codes.

[0267] like Figure 10 As shown, the EDD code is used to encode the binary value of the position of all points within the surface thickness range including d1. Figure 10 In the example, since there are points at the first and fourth positions on D0 and the second and third positions are empty, the points included in the second column from the left can be represented by EDD code 0b1001 (=9). When the EDD code is encoded and transmitted together with D0, the receiving terminal can restore the geometric information about all points without loss.

[0268] For example, when there is a point above the reference point, the value is 1. When there is no point, the value is 0. Therefore, the code can be expressed based on 4 bits.

[0269] Smooth (40004)

[0270] Smoothing is an operation used to eliminate discontinuities that may appear on patch boundaries due to image quality degradation that occurs during the compression process. Smoothing can be performed by the point cloud video encoder 10002 or the smoother 40004:

[0271] ① Reconstructing point clouds from geometric images. This operation can be the opposite of the geometric image generation described above. For example, the reverse process of the encoding can be reconstructed;

[0272] ② Use KD tree to calculate the neighboring points of each point that constitutes the reconstructed point cloud;

[0273] ③ Determine whether each point is on the patch boundary. For example, when there is a neighboring point with a different projection plane (cluster index) than the current point, it can be determined that the point is on the patch boundary;

[0274] ④ If a point exists on the patch boundary, move it to the centroid of the neighboring points (located at the average x, y, z coordinates of the neighboring points). That is, change the geometry value. Otherwise, maintain the previous geometry value.

[0275] Figure 11 An example of recoloring based on color values ​​of neighboring points according to an embodiment is shown.

[0276] The point cloud video encoder 10002 or the texture image generator 40003 according to an embodiment may generate a texture image based on recoloring.

[0277] Texture Image Generation (40003)

[0278] Similar to the above-described geometric image generation process, the texture image generation process includes generating texture images for each patch and arranging the texture images at a determined position to generate the entire texture image. However, in the operation of generating the texture image for each patch, instead of the depth value used for geometric generation, an image having color values ​​(e.g., R, G, and B values) of the points constituting the point cloud corresponding to the position is generated.

[0279] When estimating the color values ​​of the individual points that make up the point cloud, the geometry obtained previously by the smoothing process can be used. In the smoothed point cloud, the positions of some points may have shifted relative to the original point cloud, so it may be necessary to find a recoloring process that is appropriate for the changed position. The recoloring can be performed using the color values ​​of neighboring points. For example, Figure 11 As shown, the new color value can be calculated by considering the color value of the nearest neighboring point and the color values ​​of the neighboring points.

[0280] For example, refer to Figure 11 , in recoloring, an appropriate color value of the changed position may be calculated based on an average of attribute information about the nearest original point of the point and / or an average of attribute information about the nearest original point of the point.

[0281] Similar to the geometric image generated with two layers d0 and d1, the texture image can also be generated with two layers t0 and t1.

[0282] Auxiliary patch information compression (40005)

[0283] The point cloud video encoder 10002 or the auxiliary patch information compressor 40005 according to an embodiment can compress auxiliary patch information (auxiliary information about the point cloud).

[0284] The auxiliary patch information compressor 40005 compresses the auxiliary patch information generated in the patch generation, patch packing and geometry generation processes. The auxiliary patch information may include the following parameters:

[0285] The index (cluster index) used to identify the projection plane (normal plane);

[0286] The 3D spatial position of the patch, i.e., the minimum tangent value of the patch (on the patch 3D shift tangential axis), the minimum bitangent value of the patch (on the patch 3D shift bitangent axis), and the minimum normal value of the patch (on the patch 3D shift normal axis);

[0287] The 2D spatial position and size of the patch, i.e., horizontal size (patch2d size u), vertical size (patch2d size v), minimum horizontal value (patch2d shift u), and minimum vertical value (patch2d shift u); and

[0288] Mapping information about each block and patch, that is, candidate index (when patches are arranged in order based on 2D spatial position and size information about the patches, multiple patches can be mapped to one block in an overlapping manner. In this case, the mapped patches constitute a candidate list, and the candidate index indicates the sequential position of the patch whose data exists in the block) and the local patch index (indicating the index of one patch existing in the frame). Table 1 shows pseudo code representing a process of matching between blocks and patches based on the candidate list and the local patch index.

[0289] The maximum number of candidate lists can be defined by the user.

[0290] [Table 1]

[0291]

[0292] Figure 12 Shown is a push-pull background fill according to an embodiment.

[0293] Image Fill and Group Dilation (40006, 40007, 40008)

[0294] The image filler according to an embodiment may fill the space except the patch area with meaningless supplementary data based on a push-pull background filling technique.

[0295] Image padding 40006 and 40007 is a process of filling the space outside the patch area with meaningless data to improve compression efficiency. For image padding, the pixel values ​​in the columns or rows near the boundaries of the patch can be copied to fill the empty space. Alternatively, as Figure 12 As shown, a push-pull background filling method can be used. According to this method, the empty space is filled with pixel values ​​from the low-resolution image in a process of gradually reducing the resolution of the non-filled image and increasing the resolution again.

[0296] Group expansion 40008 is a process for filling the blank spaces of the geometry image and texture image configured in two layers d0 / d1 and t0 / t1, respectively. In this process, the blank spaces of the two layers calculated by image filling are filled with the average of the values ​​at the same position.

[0297] Figure 13 An exemplary possible traversal order of a 4*4 block according to an embodiment is shown.

[0298] Occupancy map compression (40012, 40011)

[0299] The occupancy map compressor according to an embodiment can compress a previously generated occupancy map. Specifically, two methods can be used, namely: video compression for lossy compression and entropy compression for lossless compression. Video compression is described below.

[0300] Entropy compression can be performed by the following operations.

[0301] ① If the block constituting the occupancy map is fully occupied, encode 1 and repeat the same operation for the next block of the occupancy map. Otherwise, encode 0 and perform operations 2) to 5).

[0302] ② Determine the optimal traversal order for performing run-length encoding on the occupied pixels of the block. Figure 13 Four possible traversal orders of a 4*4 block are shown.

[0303] Figure 14 An exemplary optimal traversal order according to an embodiment is shown.

[0304] As described above, the entropy compressor according to the embodiment can be based on Figure 14 The traversal order scheme shown encodes the blocks (Encoding).

[0305] For example, the best traversal order with the smallest number of runs is selected from the possible traversal orders, and its index is encoded. Figure 13 In the case of the third traversal order in . In the case shown, the number of runs can be minimized to 2, so the third traversal order can be selected as the optimal traversal order.

[0306] ③Encode the number of runs. Figure 14 In the example of , there are two runs, so 2 is encoded.

[0307] ④ Encode the occupation of the first run. Figure 14 In the example of , 0 is encoded because the first run corresponds to an unoccupied pixel.

[0308] ⑤ Encode the length of each run (as many as the number of runs). Figure 14In the example, the lengths 6 and 10 of the first run and the second run are encoded respectively.

[0309] Video compression (40009, 40010, 40011)

[0310] The video compressors 40009 , 40010 , 40011 according to an embodiment encode the sequences of geometry images, texture images, occupancy map images, etc. generated in the above operations using a 2D video codec such as HEVC or VVC.

[0311] Figure 15 An exemplary 2D video / image encoder according to an embodiment is shown. According to an embodiment, the 2D video / image encoder may be referred to as an encoding device.

[0312] Indicates an embodiment in which the above-mentioned video compressors 40009, 40010 and 40011 are applied Figure 15 2D video / image encoder 15000 is a schematic block diagram of a 2D video / image encoder configured to encode a video / image signal. 2D video / image encoder 15000 may be included in the above-mentioned point cloud video encoder 10002, or may be configured as an internal / external component. Figure 15 Each component may correspond to software, hardware, a processor and / or a combination thereof.

[0313] Here, the input image may include one of the above-mentioned geometric image, texture image (attribute image) and occupancy map image. Figure 15 When the 2D video / image encoder is applied to the video compressor 40009, the image input to the 2D video / image encoder 15000 is a filled geometric image, and the bit stream output from the 2D video / image encoder 15000 is a compressed geometric image bit stream. Figure 15 When the 2D video / image encoder 15000 is applied to the video compressor 40010, the image input to the 2D video / image encoder 15000 is a padded texture image, and the bit stream output from the 2D video / image encoder 15000 is a bit stream of the compressed texture image. Figure 15 When the 2D video / image encoder is applied to the video compressor 40011, the image input to the 2D video / image encoder 15000 is an occupancy map image, and the bit stream output from the 2D video / image encoder 15000 is a bit stream of the compressed occupancy map image.

[0314] The inter-frame predictor 15090 and the intra-frame predictor 15100 may be collectively referred to as a predictor. That is, the predictor may include the inter-frame predictor 15090 and the intra-frame predictor 15100. The transformer 15030, the quantizer 15040, the inverse quantizer 15050, and the inverse transformer 15060 may be collectively referred to as a residual processor. The residual processor may further include a subtractor 15020. According to an embodiment, Figure 15 The image segmenter 15010, the subtractor 15020, the transformer 15030, the quantizer 15040, the inverse quantizer 15050, the inverse transformer 15060, the adder 15200, the filter 15070, the inter-frame predictor 15090, the intra-frame predictor 15100, and the entropy encoder 15110 may be configured by one hardware component (e.g., an encoder or a processor). In addition, the memory 15080 may include a decoded picture buffer (DPB) and may be configured by a digital storage medium.

[0315] The image splitter 15010 may split the image (or picture or frame) input to the encoder 15000 into one or more processing units. For example, a processing unit may be referred to as a coding unit (CU). In this case, the CU may be recursively split from a coding tree unit (CTU) or a largest coding unit (LCU) based on a quadtree binary tree (QTBT) structure. For example, a CU may be split into multiple CUs of lower depth based on a quadtree structure and / or a binary tree structure. In this case, for example, the quadtree structure may be applied first, and the binary tree structure may be applied later. Alternatively, the binary tree structure may be applied first. The encoding process according to the present disclosure may be performed based on a final CU that is no longer split. In this case, the LCU may be used as the final CU based on encoding efficiency according to the characteristics of the image. If necessary, the CU may be recursively split into CUs of lower depth, and the optimally sized CU may be used as the final CU. Here, the encoding process may include prediction, transform, and reconstruction (to be described later). As another example, the processing unit may also include a prediction unit (PU) or a transform unit (TU). In this case, the PU and TU may be partitioned or divided from the final CU. A PU may be a unit for sample prediction, and a TU may be a unit for deriving a transform coefficient and / or a unit for deriving a residual signal from the transform coefficient.

[0316] The term "unit" can be used interchangeably with terms such as block, region, or module. In general, an M×N block can represent a set of samples or transform coefficients arranged in M ​​columns and N rows. A sample can generally represent a pixel or pixel value and can indicate only the pixel / pixel value of the luma component or only the pixel / pixel value of the chroma component. "Sample" can be used as a term corresponding to a pixel or picture element in a picture (or image).

[0317] The subtractor 15020 of the encoding device 15000 can generate a residual signal (residual block or residual sample array) by subtracting the prediction signal (prediction block or prediction sample array) output from the inter predictor 15090 or the intra predictor 15100 from the input image signal (original block or original sample array). The generated residual signal is then transmitted to the transformer 15030. In this case, as shown in the figure, the unit in the encoding device 15000 that subtracts the prediction signal (prediction block or prediction sample array) from the input image signal (original block or original sample array) can be referred to as the subtractor 15020. The predictor can perform prediction on a processing target block (hereinafter referred to as the current block) and generate a prediction block including prediction samples of the current block. The predictor can determine whether to apply intra prediction or inter prediction based on the current block or CU. As will be described later in the description of each prediction mode, the predictor can generate various types of information regarding the prediction (e.g., prediction mode information) and transmit the generated information to the entropy encoder 15110. Information about the prediction may be encoded by the entropy encoder 15110 and output in the form of a bitstream.

[0318] The intra-frame predictor 15100 of the predictor can predict the current block with reference to samples in the current picture. Depending on the prediction mode, the samples may be adjacent to or far away from the current block. Under intra-frame prediction, the prediction mode may include multiple non-directional modes and multiple directional modes. The non-directional mode may include, for example, a DC mode and a planar mode. Depending on the fineness of the prediction direction, the directional mode may include, for example, 33 directional prediction modes or 65 directional prediction modes. However, this is only an example, and more or fewer directional prediction modes may be used depending on the settings. The intra-frame predictor 15100 may determine the prediction mode to be applied to the current block based on the prediction mode applied to the neighboring blocks.

[0319] The inter-frame predictor 15090 of the predictor can derive a prediction block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. In this case, to reduce the amount of motion information transmitted in inter-frame prediction mode, motion information can be predicted on a per-block, per-subblock, or per-sample basis based on the correlation of motion information between neighboring blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can also include information regarding the inter-frame prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter-frame prediction, neighboring blocks can include spatially neighboring blocks in the current picture and temporally neighboring blocks in the reference picture. The reference picture containing the reference block can be the same as or different from the reference picture containing the temporally neighboring block. A temporally neighboring block can be referred to as a collocated reference block or collocated CU (colCU), and a reference picture containing a temporally neighboring block can be referred to as a collocated picture (colPic). For example, the inter-frame predictor 15090 can configure a motion information candidate list based on the neighboring blocks and generate information indicating candidates to be used for deriving the motion vector and / or reference picture index for the current block. Inter-frame prediction can be performed based on various prediction modes. For example, in skip mode and merge mode, the inter-frame predictor 15090 may use motion information about a neighboring block as motion information about the current block. In skip mode, unlike merge mode, a residual signal may not be sent. In motion vector prediction (MVP) mode, the motion vector of the neighboring block may be used as a motion vector predictor, and a motion vector difference may be signaled to indicate the motion vector of the current block.

[0320] The prediction signal generated by the inter-frame predictor 15090 or the intra-frame predictor 15100 may be used to generate a reconstructed signal or to generate a residual signal.

[0321] The transformer 15030 may generate transform coefficients by applying a transform technique to the residual signal. For example, the transform technique may include at least one of a discrete cosine transform (DCT), a discrete sine transform (DST), a Karhunen–Loève transform (KLT), a graph-based transform (GBT), or a conditional nonlinear transform (CNT). Here, a GBT refers to a transform obtained from a graph depicting the relationship between pixels. A CNT refers to a transform obtained based on a prediction signal generated from all previously reconstructed pixels. In addition, the transform operation may be applied to square pixel blocks of the same size, or to blocks of variable sizes other than square.

[0322] The quantizer 15040 may quantize the transform coefficients and transmit them to the entropy encoder 15110. The entropy encoder 15110 may encode the quantized signal (information about the quantized transform coefficients) and output a bitstream of the encoded signal. The information about the quantized transform coefficients may be referred to as residual information. The quantizer 15040 may rearrange the quantized transform coefficients in the form of a block in the form of a one-dimensional vector based on a coefficient scanning order, and generate information about the quantized transform coefficients based on the quantized transform coefficients in the form of a one-dimensional vector.

[0323] The entropy encoder 15110 may employ various encoding techniques such as, for example, exponential Golomb, context-adaptive variable length coding (CAVLC), and context-adaptive binary arithmetic coding (CABAC). The entropy encoder 15110 may encode information required for video / image reconstruction (e.g., syntax element values) together with or separately from the quantized transform coefficients. The encoded information (e.g., encoded video / image information) may be transmitted or stored in the form of a bitstream based on a network abstraction layer (NAL) unit.

[0324] The bitstream may be sent via a network or may be stored in a digital storage medium. Here, the network may include a broadcast network and / or a communication network, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD. A transmitter (not shown) that sends a signal output from the entropy encoder 15110 and / or a storage unit (not shown) that stores the signal may be configured as an internal / external element of the encoder 15000. Alternatively, the transmitter may be included in the entropy encoder 15110.

[0325] The quantized transform coefficients output from the quantizer 15040 can be used to generate a prediction signal. For example, inverse quantization and inverse transformation can be applied to the quantized transform coefficients by the inverse quantizer 15050 and the inverse transformer 15060 to reconstruct a residual signal (residual block or residual sample). The adder 15200 adds the reconstructed residual signal to the prediction signal output from the inter-frame predictor 15090 or the intra-frame predictor 15100. Thus, a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) can be generated. When there is no residual signal for the processing target block, as in the case of applying skip mode, the prediction block can be used as a reconstructed block. The adder 15200 can be referred to as a reconstructor or a reconstructed block generator. As described below, the generated reconstructed signal can be used for intra-frame prediction of the next processing target block in the current picture, or can be used for inter-frame prediction of the next picture through filtering.

[0326] The filter 15070 can improve the subjective / objective image quality by applying filtering to the reconstructed signal output from the adder 15200. For example, the filter 15070 can generate a modified reconstructed picture by applying various filtering techniques to the reconstructed picture, and the modified reconstructed picture can be stored in the memory 15080 (specifically, the DPB of the memory 15080). For example, the various filtering techniques may include deblocking filtering, sample adaptive offset, adaptive loop filtering, and bilateral filtering. As described below in the description of the filtering techniques, the filter 15070 can generate various types of information about filtering and transmit the generated information to the entropy encoder 15110. The information about filtering can be encoded by the entropy encoder 15110 and output in the form of a bitstream.

[0327] The modified reconstructed picture stored in the memory 15080 may be used as a reference picture by the inter-frame predictor 15090. Therefore, when inter-frame prediction is applied, the encoder may avoid prediction mismatch between the encoder 15000 and the decoder and improve encoding efficiency.

[0328] The DPB of the memory 15080 may store the modified reconstructed picture for use as a reference picture by the inter-frame predictor 15090. The memory 15080 may store motion information about a block from which motion information in the current picture is derived (or encoded) and / or motion information about a reconstructed block in the picture. The stored motion information may be transmitted to the inter-frame predictor 15090 for use as motion information about spatially neighboring blocks or motion information about temporally neighboring blocks. The memory 15080 may store reconstructed samples of the reconstructed blocks in the current picture and transmit the reconstructed samples to the intra-frame predictor 15100.

[0329] At least one of the above-mentioned prediction, transformation and quantization processes may be skipped. For example, for a block to which pulse code modulation (PCM) is applied, the prediction, transformation and quantization processes may be skipped, and the value of the original sample may be encoded and output in the form of a bitstream.

[0330] Figure 16 An exemplary V-PCC decoding process according to an embodiment is shown.

[0331] The V-PCC decoding process or the V-PCC decoder may follow Figure 4 The reverse process of the V-PCC encoding process (or encoder). Figure 16 Each component may correspond to software, hardware, a processor and / or a combination thereof.

[0332] The demultiplexer 16000 demultiplexes the compressed bit stream to output a compressed texture image, a compressed geometry image, a compressed occupancy map, and compressed auxiliary patch information respectively.

[0333] Video decompression or video decompressor 16001, 16002 decompresses each of the compressed texture image and the compressed geometry image.

[0334] Occupancy map decompression or occupancy map decompressor 16003 decompresses the compressed occupancy map image.

[0335] The auxiliary patch information decompression or auxiliary patch information decompressor 16004 decompresses the compressed auxiliary patch information.

[0336] Geometry reconstruction or geometry reconstructor 16005 restores (reconstructs) geometry information based on the decompressed geometry image, decompressed occupancy map and / or decompressed auxiliary patch information. For example, geometry that changed during the encoding process can be reconstructed.

[0337] Smoothing or smoother 16006 can apply smoothing to the reconstructed geometry. For example, a smoothing filter can be applied.

[0338] Texture reconstruction or texture reconstructor 16007 reconstructs textures from decompressed texture images and / or smoothed geometry.

[0339] Color smoothing or color smoother 16008 smoothes color values ​​from the reconstructed texture. For example, a smoothing filter may be applied.

[0340] As a result, reconstructed point cloud data can be generated.

[0341] Figure 16 The decoding process of V-PCC is shown to reconstruct a point cloud by decompressing (decoding) the compressed occupancy map, geometry image, texture image, and auxiliary patch information. Figure 16 Each unit shown may be used as at least one of a processor, software, and hardware. Figure 16 The detailed operation of each unit is described as follows.

[0342] Video decompression (16001, 16002)

[0343] Video decompression is the inverse process of the above-mentioned video compression. It is a process of decoding the bitstream of the geometry image, the bitstream of the compressed texture image, and / or the bitstream of the compressed occupancy map image generated in the above-mentioned process using a 2D video codec such as HEVC or VVC.

[0344] Figure 17 An exemplary 2D video / image decoder, also referred to as a decoding device, according to an embodiment is shown.

[0345] 2D video / image decoder can follow Figure 15 The reverse process of the operation of the 2D video / image encoder.

[0346] Figure 17 The 2D video / image decoder is Figure 16 Implementation of video decompressors 16001 and 16002. Figure 17 17000 is a schematic block diagram of a 2D video / image decoder 17000 for decoding a video / image signal. The 2D video / image decoder 17000 may be included in the point cloud video decoder 10008 or configured as an internal / external component. Figure 17 Each component may correspond to software, hardware, a processor and / or a combination thereof.

[0347] Here, the input bitstream may be one of a bitstream of a geometry image, a bitstream of a texture image (attribute image), and a bitstream of an occupancy map image. Figure 17 When the 2D video / image decoder is applied to the video decompressor 16001, the bit stream input to the 2D video / image decoder is the bit stream of the compressed texture image, and the reconstructed image output from the 2D video / image decoder is the decompressed texture image. Figure 17 When the 2D video / image decoder is applied to the video decompressor 16002, the bit stream input to the 2D video / image decoder is a bit stream of a compressed geometric image, and the reconstructed image output from the 2D video / image decoder is a decompressed geometric image. Figure 17 The 2D video / image decoder can receive the bitstream of the compressed occupancy map image and decompress it. The reconstructed image (or output image or decoded image) can represent the reconstructed image of the above-mentioned geometric image, texture image (attribute image) and occupancy map image.

[0348] Reference Figure 17 , the inter-frame predictor 17070 and the intra-frame predictor 17080 may be collectively referred to as a predictor. That is, the predictor may include the inter-frame predictor 17070 and the intra-frame predictor 17080. The inverse quantizer 17020 and the inverse transformer 17030 may be collectively referred to as a residual processor. That is, depending on the embodiment, the residual processor may include the inverse quantizer 17020 and the inverse transformer 17030. Figure 17 The entropy decoder 17010, the inverse quantizer 17020, the inverse transformer 17030, the adder 17040, the filter 17050, the inter-frame predictor 17070, and the intra-frame predictor 17080 may be configured by one hardware component (e.g., a decoder or a processor). In addition, the memory 17060 may include a decoded picture buffer (DPB) or may be configured by a digital storage medium.

[0349] When a bit stream containing video / image information is input, the decoder 17000 can Figure 15The decoder 17000 reconstructs the image in a process corresponding to the process of processing video / image information by the encoder. For example, the decoder 17000 can perform decoding using the processing unit used in the encoder. Therefore, the processing unit for decoding can be, for example, a CU. The CU can be divided from the CTU or LCU along a quadtree structure and / or a binary tree structure. The reconstructed video signal decoded and output by the decoder 17000 can then be played by a player.

[0350] The decoder 17000 may receive a signal output from the encoder in the form of a bitstream, and the received signal may be decoded by the entropy decoder 17010. For example, the entropy decoder 17010 may parse the bitstream to derive information (e.g., video / image information) required for image reconstruction (or picture reconstruction). For example, the entropy decoder 17010 may decode the information in the bitstream based on a coding technique such as exponential Golomb coding, CAVLC, or CABAC, and output syntax elements required for image reconstruction and quantized values ​​of residual transform coefficients. More specifically, in CABAC entropy decoding, bins corresponding to various syntax elements in the bitstream may be received, and a context model may be determined based on information about the target syntax element to be decoded, as well as decoded information about adjacent target blocks or information about symbols / bins decoded in previous steps. The probability of occurrence of a bin may then be predicted based on the determined context model, and arithmetic decoding of the bins may be performed to generate symbols corresponding to the values ​​of the various syntax elements. According to CABAC entropy decoding, after determining the context model, the context model can be updated based on information about the symbol / cell decoded for the context model of the next symbol / cell. Information about prediction in the information decoded by the entropy decoder 17010 can be provided to the predictor (inter-frame predictor 17070 and intra-frame predictor 17080), and the residual value (i.e., quantized transform coefficients and related parameter information) that has been entropy decoded by the entropy decoder 17010 can be input to the inverse quantizer 17020. In addition, information about filtering in the information decoded by the entropy decoder 17010 can be provided to the filter 17050. A receiver (not shown) configured to receive a signal output from the encoder can also be configured as an internal / external element of the decoder 17000. Alternatively, the receiver can be a component of the entropy decoder 17010.

[0351] The inverse quantizer 17020 may output a transform coefficient by inversely quantizing the quantized transform coefficient. The inverse quantizer 17020 may rearrange the quantized transform coefficients in the form of two-dimensional blocks. In this case, the rearrangement may be performed based on the coefficient scanning order implemented by the encoder. The inverse quantizer 17020 may perform inverse quantization on the quantized transform coefficient using a quantization parameter (e.g., quantization step size information) and obtain the transform coefficient.

[0352] The inverse transformer 17030 obtains a residual signal (a residual block and a residual sample array) by transforming the transform coefficients.

[0353] The predictor may perform prediction on the current block and generate a prediction block including prediction samples of the current block. The predictor may determine whether to apply intra prediction or inter prediction to the current block based on the information about prediction output from the entropy decoder 17010, and may determine a specific intra / inter prediction mode.

[0354] The intra-frame predictor 17080 of the predictor can predict the current block with reference to samples in the current picture. Depending on the prediction mode, the samples may be adjacent to or distant from the current block. In intra-frame prediction, the prediction mode may include multiple non-directional modes and multiple directional modes. The intra-frame predictor 17080 can use the prediction mode applied to the neighboring blocks to determine the prediction mode applied to the current block.

[0355] The inter-frame predictor 17070 of the predictor can derive a prediction block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. In this case, to reduce the amount of motion information transmitted in inter-frame prediction mode, motion information can be predicted on a per-block, sub-block, or sample basis based on the correlation of motion information between neighboring blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may also include information regarding the inter-frame prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter-frame prediction, neighboring blocks may include spatially neighboring blocks in the current picture and temporally neighboring blocks in the reference picture. For example, the inter-frame predictor 17070 may configure a motion information candidate list based on the neighboring blocks and derive the motion vector and / or reference picture index of the current block based on received candidate selection information. Inter-frame prediction can be performed based on various prediction modes. Information regarding the prediction may include information indicating the inter-frame prediction mode for the current block.

[0356] The adder 17040 may add the residual signal obtained from the inverse transformer 17030 to the prediction signal (prediction block or prediction sample array) output from the inter-frame predictor 17070 or the intra-frame predictor 17080, thereby generating a reconstructed signal (reconstructed picture, reconstructed block, or reconstructed sample array). When there is no residual signal for processing the target block as in the case of applying the skip mode, the prediction block may be used as the reconstructed block.

[0357] The adder 17040 may be referred to as a reconstructor or a reconstructed block generator. The generated reconstructed signal may be used for intra-frame prediction of the next processing target block in the current picture, or may be used for inter-frame prediction of the next picture through filtering as described below.

[0358] The filter 17050 can improve the subjective / objective image quality by applying filtering to the reconstructed signal output from the adder 17040. For example, the filter 17050 can generate a modified reconstructed picture by applying various filtering techniques to the reconstructed picture, and can send the modified reconstructed picture to the memory 17060 (specifically, the DPB of the memory 17060). For example, various filtering methods may include deblocking filtering, sample adaptive offset, adaptive loop filtering, and bilateral filtering.

[0359] The reconstructed picture stored in the DPB of the memory 17060 can be used as a reference picture in the inter-frame predictor 17070. The memory 17060 can store motion information about a block in the current picture from which motion information is derived (or decoded) and / or motion information about a block in an already reconstructed picture. The stored motion information can be transmitted to the inter-frame predictor 17070 to be used as motion information about spatially neighboring blocks or motion information about temporally neighboring blocks. The memory 17060 can store reconstructed samples of the reconstructed block in the current picture and transmit the reconstructed samples to the intra-frame predictor 17080.

[0360] In this disclosure, Figure 15 The embodiments described for the filter 15070, the inter-frame predictor 15090, and the intra-frame predictor 15100 of the encoder 15000 may be applied to the filter 17050, the inter-frame predictor 17070, and the intra-frame predictor 17080 of the decoder 17000, respectively, in the same or corresponding manner.

[0361] At least one of the above-mentioned prediction, inverse transform, and inverse quantization processes may be skipped. For example, for a block to which pulse code modulation (PCM) is applied, the prediction, inverse transform, and inverse quantization processes may be skipped, and the values ​​of the decoded samples may be used as samples of the reconstructed image.

[0362] Occupancy map decompression (16003)

[0363] This is the inverse process of the occupancy map compression described above. Occupancy map decompression is the process of reconstructing the occupancy map by decompressing the occupancy map bitstream.

[0364] Auxiliary patch information decompression (16004)

[0365] The auxiliary patch information can be reconstructed by performing the inverse process of the auxiliary patch information compression described above and decoding the compressed auxiliary patch information bitstream.

[0366] Geometry Reconstruction (16005)

[0367] This is the inverse process of the geometric image generation described above. Initially, patches are extracted from the geometric image using the reconstructed occupancy map, the 2D position / size information about the patches included in the auxiliary patch information, and the information about the mapping between blocks and patches. Then, a point cloud is reconstructed in 3D space based on the geometric image of the extracted patches and the 3D position information about the patches included in the auxiliary patch information. When the geometric value corresponding to a point (u, v) within the patch is g(u, v), and the position coordinates of the patch on the normal, tangential, and bitangential axes of the 3D space are (δ0, s0, r0), the normal, tangential, and bitangential coordinates δ(u, v), s(u, v), and r(u, v) in the 3D space mapped to the position of the point (u, v) can be expressed as follows.

[0368] δ(u,v)=δ0+g(u,v)

[0369] s(u,v)=s0+u

[0370] r(u,v)=r0+v

[0371] Smooth (16006)

[0372] Like smoothing in the above-described encoding process, smoothing is a process for eliminating discontinuities that may appear at patch boundaries due to image quality degradation occurring during the compression process.

[0373] Texture reconstruction (16007)

[0374] Texture reconstruction is a process of reconstructing a color point cloud by assigning color values ​​to each point that makes up the smoothed point cloud. This is performed by assigning color values ​​corresponding to the texture image pixels at the same positions in the geometric image in 2D space to points in the point cloud corresponding to the same positions in 3D space, based on the mapping information between the geometric image and the point cloud reconstructed in the geometric reconstruction process described above.

[0375] Color Smoothing (16008)

[0376] Color smoothing is similar to the process of geometric smoothing described above. Color smoothing is a process used to eliminate discontinuities that may appear at patch boundaries due to image quality degradation that occurs during the compression process. Color smoothing can be performed by the following operations:

[0377] ① Calculate the neighboring points of each point constituting the reconstructed point cloud using a KD tree or other method. The neighboring point information calculated in the above-mentioned geometric smoothing process can be used.

[0378] ② Determine whether each point is on the patch boundary. These operations can be performed based on the boundary information calculated in the above-mentioned geometric smoothing process.

[0379] ③ Check the distribution of color values ​​of neighboring points on the edge and determine whether smoothing should be performed. For example, when the entropy of the luminance value is less than or equal to a threshold local entry (there are many similar luminance values), it can be determined that the corresponding portion is not an edge portion, and smoothing can be performed. As a smoothing method, the color value of the point can be replaced by the average color value of the neighboring points.

[0380] Figure 18 is a flowchart illustrating the operation of a transmitting apparatus for compressing and transmitting V-PCC-based point cloud data according to an embodiment of the present disclosure.

[0381] The transmitting device according to the embodiment may correspond to Figure 1 The sending device, Figure 4 The encoding process, and Figure 15 2D video / image encoder, or perform some / all of its operations. Each component of the transmitting device may correspond to software, hardware, a processor and / or a combination thereof.

[0382] The operation process of the sending terminal using V-PCC to compress and transmit point cloud data can be performed as shown in the figure.

[0383] The point cloud data transmitting device according to the embodiment may be referred to as a transmitting device or a transmitting system.

[0384] Patch generator 18000 generates patches for 2D image mapping of point clouds based on input point cloud data. Patch generation generates patch information and / or auxiliary patch information. This generated patch information and / or auxiliary patch information can be used in geometric image generation, texture image generation, smoothing, and smoothed geometric reconstruction.

[0385] The patch packer 18001 performs a patch packing process to map the patches generated by the patch generator 18000 into a 2D image. For example, one or more patches may be packed. As a result of the patch packing, an occupancy map may be generated. The occupancy map may be used in processes such as geometry image generation, geometry image filling, texture image filling, and / or smooth geometry reconstruction.

[0386] The geometric image generator 18002 generates a geometric image based on the point cloud data, patch information (or auxiliary patch information) and / or occupancy map. The generated geometric image is pre-processed by the encoding pre-processor 18003 and then encoded into a bitstream by the video encoder 18006.

[0387] The encoding preprocessor 18003 may include an image filling process. In other words, some spaces in the generated geometry image and the generated texture image may be filled with meaningless data. The encoding preprocessor 18003 may also include a group dilation process for the generated texture image or the texture image on which image filling has been performed.

[0388] The geometry reconstructor 18010 reconstructs a 3D geometric image based on the geometry bitstream, auxiliary patch information, and / or occupancy map encoded by the video encoder 18006 .

[0389] The smoother 18009 smoothes the 3D geometric image reconstructed and output by the geometry reconstructor 18010 based on the auxiliary patch information, and outputs the smoothed 3D geometric image to the texture image generator 18004 .

[0390] The texture image generator 18004 can generate a texture image based on smoothed 3D geometry, point cloud data, patches (or packed patches), patch information (or auxiliary patch information), and / or an occupancy map. The generated texture image can be pre-processed by the encoding pre-processor 18003 and then encoded into a video bitstream by the video encoder 18006.

[0391] The metadata encoder 18005 can encode the auxiliary patch information into a metadata bitstream.

[0392] The video encoder 18006 can encode the geometric image and texture image output from the encoding preprocessor 18003 into corresponding video bit streams, and can encode the occupancy map into a video bit stream. Figure 15 The 2D video / image encoder encodes each input image.

[0393] The multiplexer 18007 multiplexes the video bitstream of geometry, the video bitstream of texture image, the video bitstream of occupancy map output from the video encoder 18006 and the bitstream of metadata (including auxiliary patch information) output from the metadata encoder 18005 into one bitstream.

[0394] The transmitter 18008 transmits the bit stream output from the multiplexer 18007 to the receiving side. Alternatively, a file / segment encapsulator may be provided between the multiplexer 18007 and the transmitter 18008, and the bit stream output from the multiplexer 18007 may be encapsulated into a file and / or segment form and output to the transmitter 18008.

[0395] Figure 18The patch generator 18000, patch packer 18001, geometric image generator 18002, texture image generator 18004, metadata encoder 18005 and smoother 18009 can correspond to patch generation 40000, patch packing 40001, geometric image generation 40002, texture image generation 40003, auxiliary patch information compression 40005 and smoothing 40004 respectively. Figure 18 The encoding preprocessor 18003 may include Figure 4 Image fillers 40006 and 40007 and group expander 40008, and Figure 18 The video encoder 18006 may include Figure 4 Video compressors 40009, 40010 and 40011 and / or entropy compressor 40012. Figure 18 The description part, refer to Figures 4 to 15 The above blocks may be omitted or may be replaced by blocks having similar or identical functions. In addition, Figure 18 Each block shown can be used as at least one of a processor, software, or hardware. Alternatively, the generated geometry, texture image, occupancy map video bitstream, and metadata bitstream of the auxiliary patch information can be formed into one or more track data in a file, or encapsulated into fragments, and sent to the receiving side via a transmitter.

[0396] Procedure for operating the receiving device

[0397] Figure 19 is a flowchart illustrating the operation of a receiving apparatus for receiving and restoring V-PCC-based point cloud data according to an embodiment.

[0398] The receiving device according to the embodiment may correspond to Figure 1 receiving device, Figure 16 The decoding process and Figure 17 2D video / image encoder, or perform some / all of its operations. Each component of the receiving device may correspond to software, hardware, a processor and / or a combination thereof.

[0399] The operation of the receiving terminal using V-PCC to receive and reconstruct point cloud data can be performed as shown in the figure. The operation of the V-PCC receiving terminal can follow Figure 18 The reverse process of the operation of the V-PCC sending terminal.

[0400] The point cloud data receiving device according to the embodiment may be referred to as a receiving device, a receiving system, etc.

[0401] The receiver receives a point cloud bitstream (i.e., a compressed bitstream), and the demultiplexer 19000 demultiplexes the received point cloud bitstream into a texture image bitstream, a geometry image bitstream, an occupancy map image bitstream, and a metadata bitstream (i.e., auxiliary patch information). The demultiplexed texture image, geometry image, and occupancy map image bitstreams are output to the video decoder 19001, and the metadata bitstream is output to the metadata decoder 19002.

[0402] According to an embodiment, when Figure 18 When the sending device is provided with a file / fragment encapsulator, the file / fragment decapsulator is provided in Figure 19 between the receiver of the receiving device and the demultiplexer 19000. In this case, the sending device encapsulates and sends the point cloud bitstream in the form of files and / or segments, and the receiving device receives and decapsulates the files and / or segments containing the point cloud bitstream.

[0403] The video decoder 19001 decodes the bit stream of the geometric image, the bit stream of the texture image and the bit stream of the occupancy map image into the geometric image, the texture image and the occupancy map image respectively. Figure 17 The metadata decoder 19002 decodes the metadata bitstream into auxiliary patch information and outputs the information to the geometry reconstructor 19003.

[0404] The geometry reconstructor 19003 reconstructs 3D geometry based on the geometry image, occupancy map and / or auxiliary patch information output from the video decoder 19001 and the metadata decoder 19002 .

[0405] The smoother 19004 smoothes the 3D geometry reconstructed by the geometry reconstructor 19003 .

[0406] Texture reconstructor 19005 reconstructs textures using the texture image and / or smoothed 3D geometry output from video decoder 19001. Specifically, texture reconstructor 19005 reconstructs a color point cloud image / image by assigning color values ​​to the smoothed 3D geometry using the texture image. Subsequently, to improve objective / subjective visual quality, additional color smoothing may be performed on the color point cloud image / image by color smoother 19006. After rendering in point cloud renderer 19007, the modified point cloud image / image derived from these operations is displayed to the user. In some cases, color smoothing may be omitted.

[0407] The above blocks may be omitted or may be replaced by blocks having similar or identical functions. Figure 19Each block shown may function as at least one of a processor, software, and hardware.

[0408] Figure 20 An exemplary architecture for V-PCC based storage and streaming of point cloud data is shown, according to an embodiment.

[0409] Figure 20 Part / all of the system may include Figure 1 The sending device and the receiving device, Figure 4 The encoding process, Figure 15 2D video / image encoder, Figure 16 Decoding processing, Figure 18 The sending device and / or Figure 19 Each component in the figure may correspond to software, hardware, processor and / or a combination thereof.

[0410] Figure 20 The overall architecture for storing or streaming point cloud data compressed according to video-based point cloud compression (V-PCC) is shown. The process of storing and streaming point cloud data may include acquisition processing, encoding processing, transmission processing, decoding processing, rendering processing and / or feedback processing.

[0411] Embodiments propose a method for efficiently providing point cloud media / content / data.

[0412] In order to efficiently provide point cloud media / content / data, the point cloud acquirer 20000 can acquire point cloud video. For example, one or more cameras can acquire point cloud data by capturing, arranging or generating a point cloud. Through this acquisition process, a point cloud video including the 3D position of each point (which can be represented by x, y and z position values, etc.) (hereinafter referred to as geometry) and the attributes of each point (color, reflectivity, transparency, etc.) can be acquired. For example, a Polygon file format (PLY) (or Stanford triangle format) file containing a point cloud video can be generated. For point cloud data with multiple frames, one or more files can be acquired. In this process, point cloud-related metadata (e.g., metadata related to capture, etc.) can be generated.

[0413] Post-processing for improving the quality of the content may be required for the captured point cloud video. In the video capture process, the maximum / minimum depth can be adjusted within the range provided by the camera device. Even after the adjustment, point data of unwanted areas may still exist. Therefore, post-processing can be performed to remove unwanted areas (for example, background) or identify connected spaces and fill spatial holes. In addition, the point cloud extracted from cameras sharing a spatial coordinate system can be integrated into a piece of content by transforming each point into a global coordinate system based on the position coordinates of each camera obtained through a calibration process. In this way, a point cloud video with a high density of points can be obtained.

[0414] The point cloud preprocessor 20001 can generate one or more frames of a point cloud video. Generally, a frame can be a unit representing an image at a specific time interval. Furthermore, when dividing the points comprising the point cloud video into one or more patches and mapping them to a 2D plane, the point cloud preprocessor 20001 can generate an occupancy map frame with a value of 0 or 1, which is a binary map indicating the presence or absence of data at corresponding locations in the 2D plane. Here, a patch is a collection of points comprising the point cloud video, where points belonging to the same patch are adjacent to each other in 3D space and, when mapped to a 2D image, are mapped to the same flat surface of a 6-sided bounding box. Furthermore, the point cloud preprocessor 20001 can generate a geometry frame in the form of a depth map that represents information about the position (geometry) of each point comprising the point cloud video on a patch-by-patch basis. The point cloud preprocessor 20001 can also generate a texture frame that represents color information about each point comprising the point cloud video on a patch-by-patch basis. This process generates metadata necessary to reconstruct a point cloud from each patch. The metadata may include information about the patches (auxiliary information or auxiliary patch information), such as the position and size of each patch in 2D / 3D space. These pictures / frames may be generated continuously in time sequence to construct a video stream or metadata stream.

[0415] The point cloud video encoder 20002 can encode one or more video streams related to a point cloud video. A video may include multiple frames, and a frame may correspond to a still image / picture. In this disclosure, a point cloud video may include point cloud images / frames / pictures, and the term "point cloud video" and "point cloud video / frame / picture" may be used interchangeably. The point cloud video encoder 20002 can perform video-based point cloud compression (V-PCC) processing. For compression and coding efficiency, the point cloud video encoder 20002 may perform a series of processes such as prediction, transform, quantization, and entropy coding. The encoded data (encoded video / image information) may be output in the form of a bitstream. Based on the V-PCC processing, as described below, the point cloud video encoder 20002 can encode the point cloud video by separating it into a geometry video, an attribute video, an occupancy map video, and metadata (e.g., information about patches). The geometry video may include a geometry image, the attribute video may include an attribute image, and the occupancy map video may include an occupancy map image. Patch data, as auxiliary information, may include patch-related information. The attribute video / image may include a texture video / image.

[0416] The point cloud image encoder 20003 can encode one or more images related to the point cloud video. The point cloud image encoder 20003 can perform video-based point cloud compression (V-PCC) processing. For compression and coding efficiency, the point cloud image encoder 20003 can perform a series of processes such as prediction, transformation, quantization, and entropy coding. The encoded image can be output in the form of a bitstream. Based on the V-PCC processing, as described below, the point cloud image encoder 20003 can encode the point cloud image by dividing the point cloud image into a geometry image, an attribute image, an occupancy map image, and metadata (e.g., information about patches).

[0417] According to an embodiment, the point cloud video encoder 20002, the point cloud image encoder 20003, the point cloud video decoder 20006 and the point cloud image decoder 20008 can be performed by one encoder / decoder as described above and can be performed along separate paths as shown in the figure.

[0418] In the file / segment encapsulator 20004, the encoded point cloud data and / or point cloud-related metadata may be encapsulated into files or segments for streaming. Here, the point cloud-related metadata may be received from a metadata processor (not shown), etc. The metadata processor may be included in the point cloud video / image encoder 20002 / 20003, or may be configured as a separate component / module. The file / segment encapsulator 20004 may encapsulate the corresponding video / image / metadata in a file format such as ISOBMFF or in the form of DASH segments. Depending on the embodiment, the file / segment encapsulator 20004 may include the point cloud metadata in a file format. The point cloud-related metadata may be included in boxes at various levels of the ISOBMFF file format, for example, or as data in a separate track within the file. Depending on the embodiment, the file / segment encapsulator 20004 may encapsulate the point cloud-related metadata into a file.

[0419] The file / segment encapsulator 20004 according to an embodiment can store a bitstream or separate bitstreams in one or more tracks in a file, and can also encapsulate signaling information used for the operation. In addition, the atlas stream (or patch stream) included in the bitstream can be stored as a track in the file and can store related signaling information. In addition, the SEI message present in the bitstream can be stored in the track of the file and can store related signaling information.

[0420] A transmission processor (not shown) may perform transmission processing of the encapsulated point cloud data according to a file format. The transmission processor may be included in a transmitter (not shown) or may be configured as a separate component / module. The transmission processor may process the point cloud data according to a transmission protocol. The transmission processing may include processing for transmission via a broadcast network and processing for transmission via broadband. Depending on the embodiment, the transmission processor may receive point cloud-related metadata along with the point cloud data from the metadata processor and perform transmission processing of the point cloud video data.

[0421] The transmitter can transmit the point cloud bitstream or the file / segment including the bitstream to the receiver (not shown) of the receiving device via a digital storage medium or a network. For transmission, processing according to any transmission protocol can be performed. The data processed for transmission can be transmitted via a broadcast network and / or by broadband. The data can be transmitted to the receiving side in an on-demand manner. The digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD and SSD. The transmitter may include an element for generating a media file in a predetermined file format, and may include an element for transmission via a broadcast / communication network. The receiver may extract the bitstream and send the extracted bitstream to a decoder.

[0422] A receiver can receive point cloud data transmitted by a point cloud data transmitting device according to the present disclosure. Depending on the transmission channel, the receiver can receive the point cloud data via a broadcast network or broadband. Alternatively, the point cloud data can be received via a digital storage medium. The receiver can include processing to decode the received data and render the data according to the user's viewport.

[0423] A receiving processor (not shown) can process the received point cloud video data according to the transmission protocol. The receiving processor can be included in the receiver or configured as a separate component / module. The receiving processor can reverse the processing of the transmitting processor described above to correspond to the transmission processing performed on the transmitting side. The receiving processor can transmit the acquired point cloud video to the file / segment decapsulator 20005 and transmit the acquired point cloud-related metadata to the metadata parser.

[0424] The file / segment decapsulator 20005 can decapsulate the point cloud data received from the receiving processor in the form of a file. The file / segment decapsulator 20005 can decapsulate the file according to ISOBMFF, etc., and can obtain a point cloud bitstream or point cloud-related metadata (or a separate metadata bitstream). The obtained point cloud bitstream can be transmitted to the point cloud video decoder 20006 and the point cloud image decoder 20008, and the obtained point cloud video-related metadata (metadata bitstream) can be transmitted to a metadata processor (not shown). The point cloud bitstream may include metadata (metadata bitstream). The metadata processor may be included in the point cloud video decoder 20006 or may be configured as a separate component / module. The point cloud video-related metadata obtained by the file / segment decapsulator 20005 may take the form of a box or track in the file format. If necessary, the file / segment decapsulator 20005 may receive the metadata required for decapsulation from the metadata processor. The point cloud related metadata may be transmitted to the point cloud video decoder 20006 and / or the point cloud image decoder 20008 and used in the point cloud decoding process, or may be transmitted to the renderer 20009 and used in the point cloud rendering process.

[0425] The point cloud video decoder 20006 can receive a bitstream and decode the video / image by performing operations corresponding to those of the point cloud video encoder 20002. In this case, as described below, the point cloud video decoder 20006 can decode the point cloud video by separating the point cloud video into a geometry video, an attribute video, an occupancy map video, and auxiliary patch information. The geometry video may include a geometry image, the attribute video may include an attribute image, and the occupancy map video may include an occupancy map image. The auxiliary information may include auxiliary patch information. The attribute video / image may include a texture video / image.

[0426] The point cloud image decoder 20008 may receive the bitstream and perform an inverse process corresponding to the operation of the point cloud image encoder 20003. In this case, the point cloud image decoder 20008 may divide the point cloud image into a geometry image, an attribute image, an occupancy map image, and metadata (which is, for example, auxiliary patch information) to decode them.

[0427] 3D geometry can be reconstructed based on the decoded geometry video / image, occupancy map, and auxiliary patch information, and then smoothed. A color point cloud image / screen can be reconstructed by assigning color values ​​to the smoothed 3D geometry based on the texture video / image. The renderer 20009 can render the reconstructed geometry and color point cloud image / screen. The rendered video / image can be displayed on a display. All or part of the rendering result can be displayed to the user via a VR / AR display or a typical display.

[0428] The sensor / tracker (sensing / tracking) 20007 acquires orientation information and / or user viewport information from the user or receiving side and transmits the orientation information and / or user viewport information to the receiver and / or transmitter. The orientation information may represent information about the position, angle, movement, etc. of the user's head, or information about the position, angle, movement, etc. of the device through which the user is viewing the video / image. Based on this information, information about the area in 3D space that the user is currently viewing (i.e., viewport information) can be calculated.

[0429] Viewport information may be information about the area in 3D space that a user is currently viewing through a device or HMD. A device, such as a display, may extract the viewport area based on orientation information, the vertical or horizontal FOV supported by the device, and so on. Orientation or viewport information may be extracted or calculated on the receiving side. The orientation or viewport information analyzed on the receiving side may be sent to the sending side via a feedback channel.

[0430] Based on the orientation information and / or viewport information indicating the area currently viewed by the user acquired by the sensor / tracker 20007, the receiver can effectively extract or decode only the media data for the specific area (i.e., the area indicated by the orientation information and / or viewport information) from the file. Furthermore, based on the orientation information and / or viewport information acquired by the sensor / tracker 20007, the transmitter can effectively encode only the media data for the specific area (i.e., the area indicated by the orientation information and / or viewport information), or generate and transmit the file thereof.

[0431] The renderer 20009 can render the decoded point cloud data in 3D space. The rendered video / image can be displayed on a display. The user can view all or part of the rendered result on a VR / AR display or a typical display.

[0432] Feedback processing may include transmitting various feedback information that can be obtained during the rendering / display process to a decoder on the sending or receiving side. Feedback processing can provide interactivity when consuming point cloud data. Depending on the embodiment, head orientation information, viewport information indicating the area currently viewed by the user, etc. may be transmitted to the sending side during feedback processing. Depending on the embodiment, the user can interact with content implemented in a VR / AR / MR / autonomous driving environment. In this case, information related to the interaction may be transmitted to the sending side or service provider during feedback processing. Depending on the embodiment, feedback processing may be skipped.

[0433] Depending on the implementation, the feedback information can be not only sent to the sending side but also consumed by the receiving side. That is, the receiving side can perform decapsulation, decoding, and rendering based on the feedback information. For example, point cloud data related to the area currently being viewed by the user can be prioritized for decapsulation, decoding, and rendering based on orientation information and / or viewport information.

[0434] Figure 21 is an exemplary block diagram of an apparatus for storing and transmitting point cloud data according to an embodiment.

[0435] Figure 21 A point cloud system according to an embodiment is shown. Figure 21 Part / all of the system may include Figure 1 The sending device and the receiving device, Figure 4 The encoding process, Figure 15 2D video / image encoder, Figure 16 Decoding processing, Figure 18 The sending device and / or Figure 19 In addition, it may include or correspond to Figure 20 part / all of the system.

[0436] The point cloud data transmitting device according to the embodiment may be configured as shown in the figure. Each element of the transmitting device may be a module / unit / component / hardware / software / processor.

[0437] The point cloud geometry, attributes, occupancy maps, auxiliary data (auxiliary information), and mesh data can each be configured as a separate stream or stored in different tracks in the file. In addition, they can be included in separate fragments.

[0438] The point cloud acquirer 21000 acquires a point cloud. For example, one or more cameras may acquire point cloud data by capturing, arranging, or generating a point cloud. Through this acquisition process, point cloud data including the 3D position of each point (which may be represented by x, y, and z position values, etc.) (hereinafter referred to as geometry) and the attributes of each point (color, reflectivity, transparency, etc.) may be acquired. For example, a Polygon file format (PLY) (or Stanford triangle format) file including point cloud data may be generated. For point cloud data having multiple frames, one or more files may be acquired. In this process, point cloud-related metadata (e.g., metadata related to capture, etc.) may be generated. The patch generator 21001 generates patches from point cloud data. The patch generator 21001 generates point cloud data or point cloud video into one or more pictures / frames. A picture / frame may generally represent a unit representing an image at a specific time interval. When the points constituting a point cloud video are divided into one or more patches (a collection of points constituting a point cloud video, where points belonging to the same patch are adjacent to each other in 3D space and are mapped in the same direction between the flat surfaces of a 6-sided bounding box when mapped to a 2D image) and mapped to a 2D plane, an occupancy map frame / frame of a binary image can be generated, which indicates whether data exists at the corresponding position in the 2D plane as 0 or 1. In addition, a geometric frame / frame in the form of a depth map can be generated, which represents information about the position (geometry) of each point constituting the point cloud video on a patch-by-patch basis. A texture frame / frame can be generated, which represents color information about each point constituting the point cloud video on a patch-by-patch basis. During this process, metadata required to reconstruct a point cloud from each patch can be generated. The metadata can include information about the patches, such as their position and size in 2D / 3D space. These frames / frames can be generated continuously in a temporal sequence to construct a video stream or metadata stream.

[0439] Additionally, patches can be used for 2D image mapping. For example, point cloud data can be projected onto the faces of a cube. After the patch is generated, a geometric image, one or more attribute images, an occupancy map, auxiliary data, and / or mesh data can be generated based on the generated patch.

[0440] The geometric image generation, attribute image generation, occupancy map generation, auxiliary data generation, and / or mesh data generation are performed by the point cloud preprocessor 20001 or a controller (not shown). The point cloud preprocessor 20001 may include a patch generator 21001, a geometric image generator 21002, an attribute image generator 21003, an occupancy map generator 21004, an auxiliary data generator 21005, and a mesh data generator 21006.

[0441] The geometry image generator 21002 generates a geometry image based on the results of patch generation. Geometry represents points in 3D space. The geometry image is generated using an occupancy map and includes information related to 2D image packing of patches, auxiliary data (including patch data), and / or patch-based mesh data. The geometry image also includes information such as the depth (e.g., near, far) of the patch generated after patch generation.

[0442] The attribute image generator 21003 generates an attribute image. For example, an attribute may represent a texture. The texture may be a color value matched to each point. Depending on the embodiment, an image including multiple attributes of the texture (e.g., color and reflectivity) (N attributes) may be generated. The multiple attributes may include material information and reflectivity. Depending on the embodiment, the attributes may also include information indicating the color, which may vary depending on the viewing angle and light even for the same texture.

[0443] The occupancy map generator 21004 generates an occupancy map from the patch. The occupancy map includes information indicating whether data exists in a pixel (eg, corresponding to a geometry or attribute image).

[0444] The auxiliary data generator 21005 generates auxiliary data (or auxiliary information) including information about the patch. Specifically, the auxiliary data represents metadata about the patch of the point cloud object. For example, it may represent information such as the normal vector of the patch. Specifically, the auxiliary data may include information required to reconstruct the point cloud from the patch (e.g., information about the position and size of the patch in 2D / 3D space, as well as projection (normal) plane identification information, patch mapping information, etc.).

[0445] The mesh data generator 21006 generates mesh data from the patch. A mesh represents the connection between adjacent points. For example, it can represent data in the shape of a triangle. For example, mesh data refers to the connectivity between points.

[0446] The point cloud pre-processor 20001 or controller generates metadata related to patch generation, geometric image generation, attribute image generation, occupancy map generation, auxiliary data generation and mesh data generation.

[0447] The point cloud transmitting device performs video encoding and / or image encoding in response to the results generated by the point cloud preprocessor 20001. The point cloud transmitting device can generate point cloud image data as well as point cloud video data. Depending on the embodiment, the point cloud data can include only video data, only image data, and / or both video data and image data.

[0448] The video encoder 21007 performs geometric video compression, attribute video compression, occupancy map video compression, auxiliary data compression, and / or mesh data compression. The video encoder 21007 generates a video stream containing encoded video data.

[0449] Specifically, in geometry video compression, point cloud geometry video data is encoded. In attribute video compression, attribute video data of the point cloud is encoded. In auxiliary data compression, auxiliary data associated with the point cloud video data is encoded. In mesh data compression, mesh data of the point cloud video data is encoded. Each operation of the point cloud video encoder can be performed in parallel.

[0450] The image encoder 21008 performs geometric image compression, attribute image compression, occupancy map image compression, auxiliary data compression, and / or mesh data compression. The image encoder generates an image containing encoded image data.

[0451] Specifically, in geometric image compression, point cloud geometric image data is encoded. In attribute image compression, attribute image data of the point cloud is encoded. In auxiliary data compression, auxiliary data associated with the point cloud image data is encoded. In mesh data compression, mesh data associated with the point cloud image data is encoded. Each operation of the point cloud image encoder can be performed in parallel.

[0452] The video encoder 21007 and / or the image encoder 21008 may receive metadata from the point cloud pre-processor 20001. The video encoder 21007 and / or the image encoder 21008 may perform various encoding processes based on the metadata.

[0453] The file / segment encapsulator 21009 encapsulates video streams and / or images in the form of files and / or segments. The file / segment encapsulator 21009 performs video track encapsulation, metadata track encapsulation, and / or image encapsulation.

[0454] In video track encapsulation, one or more video streams can be encapsulated into one or more tracks.

[0455] In metadata track encapsulation, metadata related to video streams and / or images can be encapsulated in one or more tracks. Metadata includes data related to the content of the point cloud data. For example, it can include initial viewing orientation metadata. Depending on the implementation, metadata can be encapsulated in the metadata track or can be encapsulated together in the video track or image track.

[0456] In image packaging, one or more images can be packaged into one or more tracks or projects.

[0457] For example, according to an embodiment, when four video streams and two images are input to the encapsulator, the four video streams and two images may be encapsulated in one file.

[0458] The file / segment packager 21009 may receive metadata from the point cloud pre-processor 20001. The file / segment packager 21009 may perform packaging based on the metadata.

[0459] The files and / or segments generated by the file / segment encapsulator 21009 are sent by a point cloud sending device or sender. For example, the segments may be transmitted according to a DASH-based protocol.

[0460] The transmitter can transmit the point cloud bitstream or a file / segment containing the bitstream to a receiver in a receiving device via a digital storage medium or a network. For transmission, processing according to any transmission protocol can be performed. Processed data can be transmitted via a broadcast network and / or broadband. Data can be transmitted to the receiving end on an on-demand basis. Digital storage media can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD.

[0461] The file / segment encapsulator 21009 according to an embodiment can divide and store a bitstream or separate bitstreams into one or more tracks in a file, and can encapsulate signaling information therefor. In addition, the patch (or atlas) stream included in the bitstream can be stored as a track in the file, and related signaling information can be stored. In addition, the SEI message present in the bitstream can be stored in the track of the file, and related signaling information can be stored.

[0462] The transmitter may include elements for generating a media file in a predetermined file format and may include elements for transmission via a broadcast / communication network. The transmitter receives orientation information and / or viewport information from a receiver. The transmitter may transmit the obtained orientation information and / or viewport information (or information selected by a user) to the point cloud preprocessor 20001, the video encoder 21007, the image encoder 21008, the file / segment encapsulator 21009, and / or the point cloud encoder. Based on the orientation information and / or viewport information, the point cloud encoder may encode all point cloud data or the point cloud data indicated by the orientation information and / or viewport information. Based on the orientation information and / or viewport information, the file / segment encapsulator may encapsulate all point cloud data or the point cloud data indicated by the orientation information and / or viewport information. Based on the orientation information and / or viewport information, the transmitter may transmit all point cloud data or the point cloud data indicated by the orientation information and / or viewport information.

[0463] For example, the point cloud preprocessor 20001 may perform the aforementioned operations on all point cloud data or on point cloud data indicated by orientation information and / or viewport information. The video encoder 21007 and / or the image encoder 21008 may perform the aforementioned operations on all point cloud data or on point cloud data indicated by orientation information and / or viewport information. The file / segment encapsulator 21009 may perform the aforementioned operations on all point cloud data or on point cloud data indicated by orientation information and / or viewport information. The transmitter may perform the aforementioned operations on all point cloud data or on point cloud data indicated by orientation information and / or viewport information.

[0464] Figure 22 is an exemplary block diagram of a point cloud data receiving apparatus according to an embodiment.

[0465] Figure 22 A point cloud system according to an embodiment is shown. Figure 22 Part / all of the system may include Figure 1 The sending device and the receiving device, Figure 4 The encoding process, Figure 15 2D video / image encoder, Figure 16 Decoding processing, Figure 18 The sending device and / or Figure 19 In addition, it may include or correspond to Figure 20 and Figure 21 part / all of the system.

[0466] The various components of the receiving device may be modules / units / components / hardware / software / processors. The transmitting client may receive point cloud data, point cloud bitstreams or files / fragments including a bitstream transmitted by a point cloud data transmitting device according to an embodiment. Depending on the channel used for transmission, the receiver may receive the point cloud data via a broadcast network or via broadband. Alternatively, the point cloud data may be received via a digital storage medium. The receiver may include processing for decoding the received data and rendering the received data according to a user viewport. The transmitting client (receiving processor) 22006 may perform processing on the received point cloud data according to a transmission protocol. The receiving processor may be included in the receiver or configured as a separate component / module. The receiving processor may perform the processing of the above-mentioned transmitting processor in reverse to correspond to the transmission processing performed by the transmitting side. The receiving processor may transmit the acquired point cloud data to the file / fragment decapsulator 22000 and transmit the acquired point cloud-related metadata to a metadata processor (not shown).

[0467] The sensor / tracker 22005 acquires orientation information and / or viewport information and may transmit the acquired orientation information and / or viewport information to the transmission client 22006 , the file / fragment decapsulator 22000 , the point cloud decoders 22001 and 22002 , and the point cloud processor 22003 .

[0468] The transmitting client 22006 may receive all point cloud data or the point cloud data indicated by the orientation information and / or the viewport information based on the orientation information and / or the viewport information. The file / fragment decapsulator 22000 may decapsulate all point cloud data or the point cloud data indicated by the orientation information and / or the viewport information based on the orientation information and / or the viewport information. The point cloud decoder (video decoder 22001 and / or image decoder 22002) may decode all point cloud data or the point cloud data indicated by the orientation information and / or the viewport information based on the orientation information and / or the viewport information. The point cloud processor 22003 may process all point cloud data or the point cloud data indicated by the orientation information and / or the viewport information based on the orientation information and / or the viewport information.

[0469] The file / segment decapsulator 22000 performs video track decapsulation, metadata track decapsulation, and / or image decapsulation. The file / segment decapsulator 22000 can decapsulate point cloud data received from the receiving processor in the form of a file. The file / segment decapsulator 22000 can decapsulate the file or segment according to ISOBMFF, etc. to obtain a point cloud bitstream or point cloud-related metadata (or a separate metadata bitstream). The obtained point cloud bitstream can be transmitted to the point cloud decoders 22001 and 22002, and the obtained point cloud-related metadata (or metadata bitstream) can be transmitted to a metadata processor (not shown). The point cloud bitstream may include metadata (metadata bitstream). The metadata processor may be included in the point cloud video decoder or may be configured as a separate component / module. The point cloud-related metadata obtained by the file / segment decapsulator 22000 may take the form of a box or track in the file format. If necessary, the file / segment decapsulator 22000 may receive metadata required for decapsulation from the metadata processor. The point cloud related metadata may be transmitted to the point cloud decoders 22001 and 22002 and used in the point cloud decoding process, or may be transmitted to the renderer 22004 and used in the point cloud rendering process. The file / fragment decapsulator 22000 may generate metadata related to the point cloud data.

[0470] In video track decapsulation performed by the file / fragment decapsulator 22000, video tracks contained in files and / or fragments are decapsulated. Video streams including geometry video, attribute video, occupancy maps, auxiliary data, and / or mesh data are decapsulated.

[0471] In metadata track decapsulation performed by the file / fragment decapsulator 22000 , a bitstream including metadata related to point cloud data and / or auxiliary data is decapsulated.

[0472] In the image decapsulation performed by the file / fragment decapsulator 22000, an image including a geometry image, an attribute image, an occupancy map, auxiliary data and / or mesh data is decapsulated.

[0473] The file / fragment decapsulator 22000 according to an embodiment can store a bitstream or separate bitstreams into one or more tracks in a file and can also decapsulate signaling information therefor. In addition, the atlas (patch) stream included in the bitstream can be decapsulated based on the tracks in the file and the relevant signaling information can be parsed. In addition, the SEI message present in the bitstream can be decapsulated based on the tracks in the file and the relevant signaling information can be obtained.

[0474] The video decoder 22001 performs geometry video decompression, attribute video decompression, occupancy map decompression, auxiliary data decompression, and / or mesh data decompression. The video decoder 22001 decodes the geometry video, attribute video, auxiliary data, and / or mesh data in a process corresponding to the process performed by the video encoder of the point cloud transmitting device according to the embodiment.

[0475] The image decoder 22002 performs geometric image decompression, attribute image decompression, occupancy map decompression, auxiliary data decompression, and / or mesh data decompression. The image decoder 22002 decodes the geometric image, attribute image, auxiliary data, and / or mesh data in a process corresponding to the process performed by the image encoder of the point cloud transmitting device according to the embodiment.

[0476] The video decoder 22001 and the video decoder 22002 according to the embodiment can be processed by one video / image decoder as described above and can be performed along separate paths as shown in the figure.

[0477] The video decoder 22001 and / or the image decoder 22002 may generate metadata related to the video data and / or the image data.

[0478] In the point cloud processor 22003 , geometric reconstruction and / or attribute reconstruction is performed.

[0479] In geometric reconstruction, a geometric video and / or a geometric image is reconstructed from the decoded video data and / or the decoded image data based on the occupancy map, the auxiliary data and / or the mesh data.

[0480] In attribute reconstruction, attribute video and / or attribute image are reconstructed from the decoded attribute video and / or decoded attribute image based on the occupancy map, auxiliary data, and / or mesh data. Depending on the embodiment, the attribute may be, for example, a texture. Depending on the embodiment, the attribute may represent multiple pieces of attribute information. When multiple attributes exist, the point cloud processor 22003 according to the embodiment performs multiple attribute reconstruction.

[0481] The point cloud processor 22003 may receive metadata from the video decoder 22001 , the image decoder 22002 , and / or the file / segment decapsulator 22000 , and process the point cloud based on the metadata.

[0482] The point cloud renderer 22004 renders the reconstructed point cloud. The point cloud renderer 22004 may receive metadata from the video decoder 22001, the image decoder 22002, and / or the file / segment decapsulator 22000, and render the point cloud based on the metadata.

[0483] The display displays the rendering result on an actual display device.

[0484] According to the method / device according to the embodiment, Figures 20 to 22 As shown, the sending side can encode the point cloud data into a bitstream, encapsulate the bitstream into a file and / or a fragment, and send it. The receiving side can decapsulate the file and / or fragment into a bitstream containing the point cloud, and can decode the bitstream into point cloud data. For example, the point cloud data device according to the embodiment can encapsulate point cloud data based on a file. The file can include a V-PCC track containing parameters of the point cloud, a geometry track containing geometry, an attribute track containing attributes, and an occupancy track containing an occupancy map.

[0485] In addition, the point cloud data receiving apparatus according to the embodiment decapsulates the point cloud data based on the file. The file may include a V-PCC track containing point cloud parameters, a geometry track containing geometry, an attribute track containing attributes, and an occupancy track containing an occupancy map.

[0486] The above encapsulation operation can be performed by Figure 20 File / fragment encapsulator 20004, Figure 21 The above decapsulation operation can be performed by the file / segment encapsulator 21009. Figure 20 File / Fragment Decapsulator 20005 or Figure 22 The file / fragment decapsulator 22000 is executed.

[0487] Figure 23 An exemplary structure that can be operated in conjunction with the point cloud data transmitting / receiving method / apparatus according to the embodiment is shown.

[0488] In the structure according to the embodiment, at least one of the server 23600, the robot 23100, the self-driving vehicle 23200, the XR device 23300, the smartphone 23400, the home appliance 23500, and / or the head-mounted display (HMD) 23700 is connected to the cloud network 23000. Here, the robot 23100, the self-driving vehicle 23200, the XR device 23300, the smartphone 23400, or the home appliance 23500 may be referred to as a device. In addition, the XR device 23300 may correspond to a point cloud data (PCC) device according to the embodiment, or may be operatively connected to a PCC device.

[0489] The cloud network 23000 may represent a network that constitutes a part of a cloud computing infrastructure or exists in a cloud computing infrastructure. Here, the cloud network 23000 may be configured using a 3G network, a 4G or Long Term Evolution (LTE) network, or a 5G network.

[0490] The server 23600 may be connected to at least one of the robot 23100 , the self-driving vehicle 23200 , the XR device 23300 , the smart phone 23400 , the home appliance 23500 , and / or the HMD 23700 via the cloud network 23000 , and may assist at least a portion of the processing of the connected devices 23100 to 23700 .

[0491] HMD 23700 represents one of implementation types of an XR device and / or a PCC device according to an embodiment. The HMD type device according to an embodiment includes a communication unit, a control unit, a memory, an I / O unit, a sensor unit, and a power supply unit.

[0492] Hereinafter, various embodiments of devices 23100 to 23500 to which the above-described technology is applied will be described. Figure 23 The illustrated devices 23100 to 23500 may be operatively connected / coupled to the point cloud data transmitting and receiving devices according to the above-described embodiments.

[0493] <PCC+XR>

[0494] The XR / PCC device 23300 may employ PCC technology and / or XR (AR+VR) technology, and may be implemented as an HMD, a head-up display (HUD) provided in a vehicle, a television, a mobile phone, a smart phone, a computer, a wearable device, a home appliance, a digital signage, a vehicle, a fixed robot, or a mobile robot.

[0495] The XR / PCC device 23300 can analyze 3D point cloud data or image data obtained through various sensors or from external devices and generate position data and attribute data regarding 3D points. Thereby, the XR / PCC device 23300 can obtain information about the surrounding space or real objects and render and output XR objects. For example, the XR / PCC device 23300 can match an XR object including auxiliary information about the identified object with the identified object and output the matched XR object.

[0496] <PCC + Self-driving + XR>

[0497] The self-driving vehicle 23200 can be implemented as a mobile robot, a vehicle, an unmanned aerial vehicle, etc. by applying PCC technology and XR technology.

[0498] The self-driving vehicle 23200 applying XR / PCC technology can represent an autonomous vehicle provided with means for providing an XR image or an autonomous vehicle as a control / interaction target in the XR image. Specifically, the self-driving vehicle 23200 as a control / interaction target in the XR image can be distinguished from and operatively connected to the XR device 23300.

[0499] The self-driving vehicle 23200 having means for providing an XR / PCC image can obtain sensor information from sensors including a camera and output the generated XR / PCC image based on the obtained sensor information. For example, the self-driving vehicle can have a HUD and output the XR / PCC image thereto to provide an XR / PCC object corresponding to a real object or an object present on the screen to passengers.

[0500] In this case, when the XR / PCC object is output to the HUD, at least a part of the XR / PCC object can be output to overlap with the real object pointed to by the passenger's eyes. On the other hand, when the XR / PCC object is output on a display provided inside the self-driving vehicle, at least a part of the XR / PCC object can be output to overlap with the object on the screen. For example, the self-driving vehicle can output an XR / PCC object corresponding to objects such as a road, another vehicle, a traffic light, a traffic sign, a two-wheeler, a pedestrian, and a building.

[0501] Virtual reality (VR) technology, augmented reality (AR) technology, mixed reality (MR) technology, and / or point cloud compression (PCC) technology according to an embodiment are applicable to various devices.

[0502] In other words, VR technology is a display technology that only provides real-world objects, backgrounds, etc. as CG images. On the other hand, AR technology refers to a technology that displays a CG image created virtually on an image of a real object. MR technology is similar to the above-mentioned AR technology in that the virtual objects to be displayed are mixed and combined with the real world. However, MR technology is different from AR technology. AR technology makes a clear distinction between real objects and virtual objects created as CG images and uses virtual objects as supplementary objects to real objects, while MR technology treats virtual objects as objects with the same characteristics as real objects. More specifically, an example of the application of MR technology is holographic services.

[0503] Recently, VR, AR, and MR technologies are often referred to as extended reality (XR) technologies, rather than being clearly distinguished from each other. Therefore, embodiments of the present disclosure are applicable to all VR, AR, MR, and XR technologies. For these technologies, encoding / decoding based on PCC, V-PCC, and G-PCC technologies can be applied.

[0504] The PCC method / apparatus according to the embodiment may be applied to a self-driving vehicle 23200 that provides a self-driving service.

[0505] The self-driving vehicle 23200 providing the self-driving service is connected to the PCC device for wired / wireless communication.

[0506] When the point cloud data compression transmitting and receiving device (PCC device) according to an embodiment is connected to the self-driving vehicle 23200 for wired / wireless communication, the device can receive and process content data related to AR / VR / PCC services that can be provided together with the self-driving service and transmit the processed content data to the self-driving vehicle 23200. In the case where the point cloud data transmitting and receiving device is installed on the vehicle, the point cloud transmitting and receiving device can receive and process content data related to the AR / VR / PCC service according to a user input signal input through the user interface device and provide the processed content data to the user. The self-driving vehicle 23200 or the user interface device according to the embodiment can receive the user input signal. The user input signal according to the embodiment may include a signal indicating the self-driving service.

[0507] As mentioned above, Figure 1 、 Figure 4 、 Figure 18 、 Figure 20 or Figure 21The V-PCC-based point cloud video encoder projects 3D point cloud data (or content) into 2D space to generate patches. Patches are generated in 2D space by dividing the data into a geometric image representing position information (called a geometry frame or a geometry patch frame) and a texture image representing color information (called an attribute frame or an attribute patch frame). The geometric image and texture image are video compressed for each frame, and a video bitstream of the geometric image (called a geometry bitstream) and a video bitstream of the texture image (called an attribute bitstream) are output. In addition, auxiliary patch information (also called patch information or metadata) including projection plane information and patch size information about each patch (these are required for decoding 2D patches on the receiving side) is also video compressed and a bitstream of the auxiliary patch information is output. In addition, an occupancy map indicating the presence / absence of a point for each pixel as 0 or 1 is entropy compressed or video compressed depending on whether it is in lossless mode or lossy mode, and a video bitstream of the occupancy map (or occupancy map bitstream) is output. The compressed geometry bitstream, the compressed attribute bitstream, the compressed auxiliary patch information bitstream and the compressed occupancy map bitstream are multiplexed into a V-PCC bitstream structure.

[0508] According to the embodiment, the V-PCC bit stream can be sent to the receiving side as it is, or can be transmitted by Figure 1 、 Figure 18 、 Figure 20 or Figure 21 The file / segment encapsulator encapsulates the data in the form of a file / segment and sends it to a receiving device or stores it in a digital storage medium (e.g., USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc.). According to an embodiment of the present disclosure, the file is in the file format ISOBMFF.

[0509] Depending on the embodiment, the V-PCC bitstream may be transmitted through multiple tracks in a file, or may be transmitted through a single track. Details will be described later.

[0510] Figure 24 An exemplary structure of a V-PCC bitstream according to an embodiment is shown. According to an embodiment, Figure 24 The V-PCC bitstream is from Figure 1 、 Figure 4 、 Figure 18 、 Figure 20 or Figure 21 Output of the V-PCC based point cloud video encoder.

[0511] A V-PCC bitstream is composed of one or more V-PCC units. That is to say, a V-PCC bitstream is a group of V-PCC units. Each V-PCC unit is composed of a V-PCC unit header and a V-PCC unit payload. In the present disclosure, the data contained in the V-PCC unit payload is distinguished by the V-PCC unit header. To this end, the V-PCC unit header contains type information indicating the V-PCC unit type. Each V-PCC unit payload can contain geometric video data (i.e., compressed geometric bitstream), attribute video data (i.e., compressed attribute bitstream), occupancy video data (i.e., compressed occupancy map bitstream), patch data group (PDG) or sequence parameter set (SPS) according to the type information in the V-PCC unit header. PDG is also called atlas. In the present disclosure, atlas can replace PDG.

[0512] The V-PCC unit payload, which contains at least geometry video data, attribute video data, and occupancy map video data (also referred to as occupancy video data), corresponds to a video data unit (eg, a HEVC NAL unit) decodable by a suitable video decoder.

[0513] The geometric video data, attribute video data, and occupancy map video data are referred to as 2D video coding information for the geometry, attribute, and occupancy map components of the encoded point cloud, and the PDG (or atlas) is referred to as non-video coding information. The PDG includes auxiliary patch information. The SPS includes overall coding information about the bitstream and is also referred to as configuration and metadata information. The SPS and PDG can be referred to as signaling information and can be generated by a metadata processor in the point cloud video encoder, or can be generated by a separate component / module in the point cloud video encoder. In addition, in the present disclosure, the SPS and PDG are referred to as initialization information, and the geometric video data, attribute video data, and occupancy map video data are referred to as point cloud data.

[0514] In one embodiment, when the type information in the V-PCC unit header indicates VPCC_PDG, the corresponding V-PCC unit payload may include a patch data group. Conversely, when the V-PCC unit payload contains a patch data group, the type information in the corresponding V-PCC unit header may identify the same content. This is a designer's option, and the present disclosure is not limited to the above embodiment. Other V-PCC units are configured in the same or similar manner.

[0515] The patch data group is included in the V-PCC unit payload in the format of patch_data_group(). The patch data group may include information about one or more patch tile groups and at least one of a patch sequence parameter set, a patch frame parameter set, a patch frame geometry parameter set, a geometry patch parameter set, a patch frame attribute parameter set, or an attribute patch parameter set.

[0516] The patch frame (also called point cloud object) as the target of the point cloud data can be divided into one or more tiles. The tile according to the embodiment can represent a specific area in 3D space or a specific area in a 2D plane. In addition, the tile can be a cuboid in a bounding box, or a part of the patch frame or sub-bounding box. In the present disclosure, the patch frame can be divided into one or more tiles. Figure 1 Point cloud video encoder, Figure 18 Patch generator, Figure 20 Point cloud preprocessing unit or Figure 21 The patch generator of the image processing unit or a separate component / module is used to divide the patch frame (or point cloud object) into one or more tiles.

[0517] According to an embodiment, Figure 24 The V-PCC bitstream of the structure shown may be transmitted to the receiving side as it is, or may be encapsulated into a file / segment form and transmitted to the receiving side.

[0518] According to an embodiment of the present disclosure, the V-PCC bitstream is encapsulated into a file format and transmitted. For example, the V-PCC bitstream may be encapsulated into a file format based on ISOBMFF (ISO Base Media File Format).

[0519] According to an embodiment, the V-PCC bitstream is encapsulated into a file by Figure 1 File / fragment encapsulator 10003, Figure 18 Transmitter 18008, Figure 20 File / fragment encapsulator 20004 or Figure 21 The file / segment encapsulator 21009 is executed. According to an embodiment, the file is decapsulated into a V-PCC bit stream by Figure 1 File / fragment decapsulator 10007, Figure 19 Receiver, Figure 20 File / Fragment Decapsulator 20005 or Figure 22 The file / fragment decapsulator 22000 is executed.

[0520] Figure 25 : is a diagram showing the structure of a multi-track V-PCC file according to an embodiment. That is, it shows an example of the layout of an ISOBMFF-based file containing multiple tracks.

[0521] An ISOBMFF-based file according to an embodiment may be referred to as a container, a container file, a media file, a V-PCC file, etc. Specifically, a file may be composed of a box and / or information that may be referred to as ftyp, meta, moov, or mdat.

[0522] The ftyp box (file type box) can provide information related to the file compatibility or file type of the file. The receiving side can refer to the ftyp box to identify the file.

[0523] The meta box may include a vpcg{0,1,2,3} box (V-PCC group box), which will be described in detail below.

[0524] The mdat box, also known as the media data box, can contain a video coded geometry bitstream, a video coded attribute bitstream, a video coded occupancy map bitstream, and a patch data group bitstream.

[0525] The moov box, also known as a movie box, can contain metadata about the file's media data (e.g., geometry bitstream, attribute bitstream, occupancy map bitstream, etc.). For example, it can contain information required to decode and play the media data, as well as information about the file's samples. The moov box can serve as a container for all metadata. The moov box can be the highest-level box among metadata-related boxes. Depending on the implementation, only one moov box may exist in a file.

[0526] According to the embodiment, Figure 25 In the ISOBMFF container structure shown, V-PCC units included in the V-PCC bitstream are mapped to various tracks in the file based on their types.

[0527] based on Figure 25 With the layout shown, the ISOBMFF container for the V-PCC bitstream may include the following:

[0528] 1) V-PCC track. The V-PCC track contains sequence parameter sets and samples that carry the payload of a V-PCC unit including non-video coding information (e.g., the type information indicating the type of the corresponding V-PCC unit in the V-PCC unit header indicates SPS or PDG). The V-PCC track can also provide track references to other tracks containing samples that carry the payload of a V-PCC unit including video compression information (e.g., the type information in the V-PCC unit header indicates geometry video data, attribute video data, or occupancy map video data).

[0529] 2) One or more constrained video scheme tracks for geometry video data. The samples contained in these tracks contain NAL units of a video coding elementary stream for geometry video data. In this case, the type information in the V-PCC unit header indicates geometry video data, and the NAL unit corresponds to the payload carrying the geometry video data.

[0530] 3) One or more constrained video scheme tracks for attribute video data. The samples contained in these tracks contain NAL units of a video coding elementary stream for the attribute video data. In this case, the type information in the V-PCC unit header indicates attribute video data, and the NAL unit corresponds to the payload carrying the attribute video data.

[0531] 4) Constrained video scheme tracks for occupancy map data. The samples contained in these tracks contain NAL units of a video coded elementary stream for occupancy map video data. In this case, the type information in the V-PCC unit header indicates occupancy map video data, and the NAL unit corresponds to the payload carrying the occupancy map video data.

[0532] According to an embodiment, tracks comprising video coding elementary streams for geometry video data, attribute video data and occupancy map video data are referred to as component tracks.

[0533] According to an embodiment, synchronization between elementary streams contained in component tracks is handled by the ISOBMFF track timing structure (stts, ctts and cslg, or equivalent mechanisms in movie fragments). Samples contributing to the same point cloud frame across different video coding component tracks and V-PCC tracks can have the same composition time.

[0534] The V-PCC parameter set for such samples may have a decoding time equal to or earlier than the composition time of the frame.

[0535] ISOBMFF files carrying V-PCC content can be distinguished from V-PCC-defined brands. Tracks of V-PCC content can be grouped within a VPCCGroupBox, which is a file-level EntityToGroupBox with a V-PCC-specific grouping 4CC value ('vpcg'). The VPCCGroupBox can be used as an entry point to access V-PCC content in a container and contains initial metadata describing the V-PCC content. A VPCCGroupBox as an EntityToGroupBox can be contained within a MetaBox box or a Moov box.

[0536] An entity group can be a set of items that group a track. The entities in an entity group can share specific characteristics indicated by the grouping type and can also have specific relationships.

[0537] Entity groups are indicated in the GroupsListBox. The GroupsListBox can be included in at least one of the file-level MetaBox, the movie-level MetaBox, or the track-level MetaBox. Entity groups specified in the GroupsListBox of the file-level MetaBox refer to track- or file-level items. Entity groups specified in the GroupsListBox of the movie-level MetaBox refer to movie-level items. Entity groups specified in the GroupsListBox of the track-level MetaBox refer to track-level items for that track.

[0538] GroupsListBox contains EntityToGroupBoxes, each specifying an entity group as follows.

[0539] Group List Box

[0540] Box type: 'grpl'

[0541] Container: MetaBox not contained in AdditionalMetadataContainerBox

[0542] Mandatory: No

[0543] Quantity: zero or one

[0544] The GroupsListBox contains the entity groups specified for the document. This box contains a complete set of boxes, each called EntityToGroupBox, with a four-character code indicating the type of grouping defined.

[0545] The GroupsListBox should not appear in the AdditionalMetadataContainerBox.

[0546] When the GroupsListBox exists in a file-level MetaBox, there may not be an item_ID value in any ItemInfoBox in any file-level MetaBox that is equal to the track_ID value in any TrackHeaderBox.

[0547] aligned(8) class GroupsListBox extends box('grpl') {

[0548] }

[0549] Box type: specified as follows, with a grouping_type value of EntityToGroupBox

[0550] Container: GroupsListBox

[0551] Mandatory: No

[0552] Quantity: One or more

[0553] EntityToGroupBox specifies the entity group.

[0554] The box type (grouping_type) indicates the grouping type of the entity group. Each grouping_type code is associated with a semantic description of the grouping. The following grouping_type values ​​are specified:

[0555] 'altr': The items and tracks mapped to this group are alternatives to each other, and only one of them can be played (when the mapped items and tracks are part of the presentation, e.g., a displayable item or track) or processed by other means (when the mapped items or tracks are not part of the presentation, but are e.g., metadata). The player can select the first entity from the list of entity_id values ​​that it can handle, e.g., decode and play the mapped items and tracks that are part of the presentation and that suit the needs of the application. Any entity_id value should be mapped to only one group of type 'altr'. An alternative group of entities consists of the items and tracks that are mapped to an entity group of the same type 'altr'.

[0556] NOTE: EntityToGroupBox contains extensions specific to grouping_type.

[0557] aligned(8) class EntityToGroupBox(grouping_type, version, flags) extends FullBox(grouping_type, version, flags) {

[0558] unsigned int(32)group_id;

[0559] unsigned int(32)num_entities_in_group;

[0560] for(i=0;i <num_entities_in_group;i++)

[0561] unsigned int(32)entity_id;

[0562] }

[0563] According to an embodiment, group_id is a non-negative integer assigned to a particular group that is not equal to any group_id value of any other EntityToGroupBox, the containing GroupsListBox, or any track_ID value (when the GroupsListBox is included in the file level).

[0564] According to an embodiment, num_entities_in_group specifies the number of entity_id values ​​mapped to this entity group.

[0565] According to an embodiment, entity_id is resolved to item when an item with item_ID equal to entity_id exists in the hierarchical level (file, movie or track) containing the GroupsListBox, or to track when a track with track_ID equal to entity_id exists and the GroupsListBox is contained in the file level.

[0566] The V-PCC group box is described below.

[0567] Box type: 'vpcg'

[0568] Container: GroupListBox

[0569] Mandatory: Yes

[0570] Quantity: One or more

[0571] The V-PCC group box is included in the GroupsListBox and provides a list of tracks including V-PCC content.

[0572] This box lists V-PCC content-specific information, such as attribute types and the mapping of layers to associated tracks. This information provides a convenient initial understanding of V-PCC content. To flexibly configure V-PCC content to support a variety of client capabilities, this box lists multiple versions of coded V-PCC components. This box also contains V-PCC-defined archiving, tiering, and level information.

[0573] aligned(8) class VPCCGroupBox() extends EntityToGroupBox('vpcg', version, flags) {

[0574] for(i=0;i <num_entities_in_group;i++){

[0575] unsigned int(4) data type;

[0576] unsigned int(4) attribute type;

[0577] unsigned int(1)multiple_layer_present_flag;

[0578] unsigned int(4)layer_count_munus1;

[0579] for(i=0;i <layer_count_minus1+1;i++){

[0580] unsigned int(4)layer_id;

[0581] }

[0582] unsigned int(32)entity_id;

[0583] }

[0584] unsigned int(4)PCC_layer_count_minus1;

[0585] vpcc_profile_tier_level()

[0586] }

[0587] According to an embodiment, data_type indicates the track type of PCC data in the reference track.

[0588] Figure 26 : This is a table showing examples of PCC data track types assigned to data_type according to embodiments. For example, data_type equal to 1 may indicate a V-PCC track. data_type equal to 2 may indicate a geometry video track. data_type equal to 3 may indicate an attribute video track. data_type equal to 4 may indicate an occupancy map video track.

[0589] Depending on the implementation, multiple_layer_present_flag indicates whether the associated entity (or track) carries a single geometry or attribute layer or multiple geometry or attribute layers. For example, multiple_layer_present_flag equal to 0 indicates that a single geometry or attribute layer is carried in the associated entity (or track). Multiple_layer_present_flag equal to 1 indicates that multiple geometry or attribute layers are carried in the associated entity (or track). According to the implementation, the V-PCC track (i.e., data_type indicates a V-PCC track) should have multiple_layer_present_flag equal to 0. As another example, when data_type indicates a geometry video track, multiple_layer_present_flag equal to 0 can indicate that the geometry video track carries a single geometry layer, and multiple_layer_present_flag equal to 1 indicates that the geometry video track carries multiple geometry layers. As another example, when data_type indicates an attribute video track, multiple_layer_present_flag equal to 0 can indicate that the attribute video track carries a single attribute layer, and multiple_layer_present_flag equal to 1 can indicate that the attribute video track carries multiple attribute layers.

[0590] According to an embodiment, layer_count_minus1 plus 1 indicates the number of geometry or attribute layers carried in the associated entity (or track). According to an embodiment, layer_count_minus1 of a V-PCC track is equal to 0. For example, when data_type indicates a geometry video track, layer_count_minus1 plus 1 may indicate the number of geometry layers carried in the geometry video track. As another example, when data_type indicates an attribute video track, layer_count_minus1 plus 1 may indicate the number of attribute layers carried in the attribute video track.

[0591] Depending on the embodiment, layer_id specifies the layer identifier of the geometry or attribute layer in the associated entity (or track). A V-PCC track according to the embodiment has layer_id equal to 0. The set of layer_id values ​​for any V-PCC component track type arranged in ascending order is a set of consecutive integers starting from 0. As an example, when data_type indicates a geometry video track, layer_id may indicate the layer identifier of the geometry layer. As another example, when data_type indicates an attribute video track, layer_id may indicate the layer identifier of the attribute layer.

[0592] According to an embodiment, pcc_layer_count_minus1 plus 1 indicates the number of layers used to encode the geometry and attribute components of the point cloud stream. For example, when data_type indicates a geometry video track, pcc_layer_count_munus1 plus 1 may indicate the number of layers used to encode the geometry component of the point cloud stream. As another example, when data_type indicates an attribute video track, pcc_layer_count_munus1 plus 1 may indicate the number of layers used to encode the attribute component of the point cloud stream.

[0593] According to an embodiment, attribute_type indicates the attribute type of the attribute video data carried in the reference entity (or track).

[0594] Figure 27 is a table showing examples of attribute types assigned to attribute_type according to embodiments. For example, attribute_type equal to 0 may indicate texture, attribute_type equal to 1 may indicate material ID, attribute_type equal to 2 may indicate transparency, attribute_type equal to 3 may indicate reflectivity, and attribute_type equal to 4 may indicate normals.

[0595] Depending on the implementation, entity_id specifies the identifier of the associated item or track. When an item with item_ID equal to entity_id appears in the hierarchy level (file, movie or track) containing the GroupsListBox, it is resolved as an item, or when a track with track_ID equal to entity_id exists and the GroupsListBox is contained in the file level, it is resolved as a track.

[0596] vpcc_profile_tier_level() is the same as profile_tier_level() specified in sequence_parameter_set().

[0597] Figure 28 An example of the syntax structure of profile_tier_level() according to an embodiment is shown.

[0598] The ptl_tier_flag field specifies the codec profile tier used to encode the V-PCC content.

[0599] The ptl_profile_idc field indicates the profile information that the encoded point cloud sequence conforms to.

[0600] The ptl_level_idc field indicates the level of the codec profile to which the encoded point cloud sequence conforms.

[0601] Next, the V-PCC track will be described.

[0602] The entry point of each V-PCC content can be represented by a unique V-PCC track. An ISOBMFF file can contain multiple V-PCC contents, so there can be multiple V-PCC tracks in the file. The V-PCC track should be identified by the volume visual media handler type "volm" in the HandlerBox of the MediaBox.

[0603] Box type: 'vohd'

[0604] Container: MediaInformationBox

[0605] Mandatory: Yes

[0606] Quantity: Exactly one

[0607] Volumetric tracks use VolumetricMediaHeaderBox in MediaInformationBox

[0608] aligned(8) class VolumetricMediaHeaderBox extends FullBox('vohd',version=0,1) {

[0609] / / If we don't need anything here, use Null Media Header

[0610] }

[0611] / / Random access point of patch stream

[0612] The random access point of the V-PCC track can be a sample carrying an I-patch frame with an empty decoder buffer. To indicate the random access point of the V-PCC track, there is a syncsampleBox. The syncsample indicates the sync sample carrying an I-patch frame.

[0613] If some samples in a track (including any track fragments) are non-sync samples, then SyncSampleBox should be present in the SampleTableBox, but the flag sample_is_non_sync_sample for samples in the track fragments is valid and describes these samples even if SyncSampleBox is not present. If the track is not fragmented and SyncSampleBox is not present, then all samples in the track are sync samples.

[0614] Box type: 'stbl'

[0615] Container: MediaInformationBox

[0616] Mandatory: Yes

[0617] Quantity: Exactly one

[0618] aligned(8) class SampleTableBox extends Box('stbl') {

[0619] }

[0620] Box type: 'stss'

[0621] Container: SampleTableBox

[0622] Mandatory: No

[0623] Quantity: zero or one

[0624] This box provides a compact marker for sync samples within a stream. The table is arranged in strictly ascending order of sample number. If SyncSampleBox is not present, every sample is a sync sample.

[0625] aligned(8) class SyncSampleBox

[0626] Extend FullBox('stss',version=0,0){

[0627] unsigned int(32)entry_count;

[0628] int i;

[0629] for(i=0;i <entry_count;i++){

[0630] unsigned int(32)sample_number;

[0631] }

[0632] }

[0633] 'version' is an integer representing the version of the box.

[0634] entry_count is an integer giving the number of entries in the table below. If entry_count is zero, there are no synchronization samples in the stream and the table below is empty.

[0635] sample_number gives the sample number for each sync sample in the stream. In particular, for each sync sample carrying an I-patch frame in a V-PCC track, sample_number gives the sample number.

[0636] The following describes a V-PCC track sample entry.

[0637] Sample entry type: 'vpc1'

[0638] Container: SampleDescriptionBox('stsd')

[0639] Mandatory: 'vpc1' sample entries are mandatory. Quantity: one or more

[0640] The track sample entry type 'vpc1' is used. The VPCC track sample entry contains the VPCC configuration box, defined below. This includes the defined VPCCDDecoderConfigurationRecord. An optional BitRateBox may be present in the VPCC track sample entry to signal the bitrate information of the VPCC video stream.

[0641] aligned(8) class VPCCDDecoderConfigurationRecord{

[0642] unsigned int(8)configurationVersion=1;

[0643] unsigned int(8)numOfSequenceParameterSets;

[0644] for(i=0;i <numOfSequenceParameterSets;i++){

[0645] sequence_parameter_set();

[0646] }

[0647] / / Additional fields

[0648] }

[0649] class VPCCConfigurationBox extends Box('vpcc'){

[0650] VPCCDecoderConfigurationRecord()VPCCConfig;

[0651] }

[0652] aligned(8) class VPCCSampleEntry() extends VolumetricSampleEntry('vpc1') {

[0653] VPCCConfigurationBoxconfig;

[0654] }

[0655] class VolumetricSampleEntry(codingname) extends SampleEntry(codingname) {

[0656] }

[0657] configurationVersion is a version field. Incompatible changes to a record are indicated by a change in the version number.

[0658] numOfSequenceParameterSets represents the number of V-PCC sequence parameter sets signaled in the decoder configuration record.

[0659] Compressorname in the base class VisualSampleEntry indicates the name of the compressor recommended to be used with the value "\012VPCCCoding".

[0660] The V-PCC specification allows multiple instances of the VPCC_SPS element (with ids from 1 to 15). Therefore, a VPCCSampleEntry contains multiple sequence_parameter_set element payloads.

[0661] The V-PCC sample format is described below.

[0662] Each sample in the V-PCC track corresponds to a single point cloud frame.

[0663] The samples corresponding to the frame in the respective component tracks have the same composition time as the V-PCC track samples for the frame in the V-PCC track.

[0664] Each V-PCC sample contains only one V-PCC unit payload whose type information in the V-PCC unit header indicates a patch data group (PDG). The V-PCC unit payload includes one or more patch block group unit payloads, such as Figure 24 shown.

[0665] aligned(8) class vpcc_unit_payload_struct{

[0666] vpcc_unit_payload();

[0667] }

[0668] aligned(8) class VPCCSample{

[0669] vpcc_unit_payload_Struct();

[0670] }

[0671] vpcc_unit_payload() is a payload of a V-PCC unit of which type information in the V-PCC unit header indicates PDG, and contains one instance of patch_data_group_unit().

[0672] For V-PCC tracks, samples carrying I-patch frames are defined as synchronization samples.

[0673] Next, the V-PCC track reference is described below.

[0674] To link the V-PCC track to the component video track, the ISOBMFF standard's track reference facility is used. Three TrackReferenceTypeBoxes are added to a TrackReferenceBox inside the V-PCC track's TrackBox, one for each component.

[0675] TrackReferenceTypeBox contains an array of track_IDs specifying the video tracks that the V-PCC track references.

[0676] The reference_type in TrackReferenceTypeBox identifies the type of component (i.e., geometry, attribute, or occupancy map). The 4CCs for the new track reference types include "pcca", "pccg", and "pcco".

[0677] In type "pcca", the referenced track contains the attribute V-PCC component of video coding.

[0678] In type "pccg", the referenced track contains the geometry V-PCC component of video coding.

[0679] In type "pcco", the referenced track contains the video coded occupancy map V-PCC component.

[0680] Next, the V-PCC component track of video coding is described below.

[0681] Since it does not make sense to display decoded frames from attribute, geometry, or occupancy map tracks without reconstructing the point cloud on the player side, a restricted video scheme type can be defined for these video coding tracks. The V-PCC video track should contain the 4CC identifier "pccv". The 4CC identifier 'pccv' is included in the scheme_type field of the SchemeTypeBox of the RestrictedSchemeInfoBox of the restricted sample entry.

[0682] The restricted video sample entry type 'resv' indicates that the decoded picture contains attribute, geometry or occupancy map data of a point cloud using the V-PCC video scheme.

[0683] The use of the V-PCC video scheme is indicated by scheme_type equal to 'pccv' (video-based point cloud video) within the SchemeTypeBox in the RestrictedSchemeInfoBox.

[0684] This box is a container for boxes containing V-PCC specific information indicating this track. VPCCVideoBox provides PCC specific parameters that are applicable to all samples in the track.

[0685] Box type: 'pccv'

[0686] Container: SchemeInformationBox

[0687] Mandatory: When scheme_type is equal to 'pccv',

[0688] Quantity: zero or one

[0689] aligned(8) class VPCCVideoBox extends FullBox('vpcc',0,0) {

[0690] unsigned int(4)data_type;

[0691] unsigned int(4)attribute_count;

[0692] for(int i=0;i <attribute_count+1;i++){

[0693] unsigned int(4)attribute_type;

[0694] }

[0695] unsigned int(1)multiple_layer_present_flag;

[0696] unsigned int(4)layer_count_minus1;

[0697] for(i=0;i <layer_count_minus1+1;i++){

[0698] unsigned int(4)layer_id;

[0699] }

[0700] }

[0701] data_type may indicate the type of data contained in a video sample included in a corresponding track.

[0702] Figure 29

[0045] is a table showing examples of the types of PCC data in reference tracks according to embodiments. For example, data_type indicates a V-PCC track when it is equal to 1, a geometry video track when it is equal to 2, an attribute video track when it is equal to 3, and an occupancy map video track when it is equal to 4.

[0703] multiple_layer_present_flag indicates whether a single geometry or attribute layer or multiple geometry or attribute layers are carried in the track. For example, multiple_layer_present_flag equal to 0 can indicate that a single geometry or attribute layer is carried in the track. multiple_layer_present_flag equal to 1 can indicate that multiple geometry or attribute layers are carried in the track. As another example, when data_type indicates a geometry video track, multiple_layer_present_flag equal to 0 can indicate that the geometry video track carries a single geometry layer, and multiple_layer_present_flag equal to 1 can indicate that the geometry video track carries multiple geometry layers. As another example, when data_type represents an attribute video track, multiple_layer_present_flag equal to 0 can indicate that the attribute video track carries a single attribute layer, and multiple_layer_present_flag equal to 1 can indicate that the attribute video track carries multiple attribute layers.

[0704] layer_count_minus1 plus 1 indicates the number of geometry and / or attribute layers carried in the track. For example, when data_type indicates a geometry video track, layer_count_minus1 plus 1 may indicate the number of geometry layers carried by the geometry video track. As another example, when data_type indicates an attribute video track, layer_count_minus1 plus 1 may indicate the number of attribute layers carried in the attribute video track.

[0705] layer_id specifies the layer identifier of the geometry or attribute layer associated with the samples in the track. As an example, when data_type indicates a geometry video track, layer_id may indicate the layer identifier of the geometry layer. As another example, when data_type indicates an attribute video track, layer_id may indicate the layer identifier of the attribute layer. attribute_count may indicate the number of attribute data of the point cloud stream included in the track. One or more attribute data may be included in one track. In this case, data_type indicates an attribute video track, depending on the embodiment.

[0706] attribute_type indicates the attribute type of the attribute video data carried in the track.

[0707] Figure 30 is a table showing examples of attribute types according to embodiments. For example, the attribute_type field indicates texture when equal to 0, indicates material ID when equal to 1, indicates transparency when equal to 2, indicates reflectivity when equal to 3, and indicates normal when equal to 4.

[0708] When a field value existing in VPCCVideoBox is included in signaling associated with an item, information about the item (image) may be indicated.

[0709] Next, attribute sample grouping is described below.

[0710] When a track contains attribute data for a point cloud, it may contain data of one or more attribute types. In this case, the attribute type of the data contained in the samples contained in the corresponding track may be signaled as follows.

[0711] class PCCAttributeSampleGroupEntry extends VisualSampleGroupEntry('pcca') {

[0712] unsigned int(4)attribute_type;

[0713] }

[0714] attribute_type indicates the attribute type of attribute video data carried in the associated sample in the track.

[0715] Figure 31 is a table showing examples of attribute types according to embodiments. For example, the attribute_type field indicates texture when equal to 0, indicates material ID when equal to 1, indicates transparency when equal to 2, indicates reflectivity when equal to 3, and indicates normal when equal to 4.

[0716] Next, layer sample grouping is described below.

[0717] It may include data associated with one or more layers of the same data type (e.g., geometric data, attribute data) of the point cloud of the track. In this case, layer-related information about the data contained in the samples included in the track may be signaled as follows.

[0718] class PCCLayerSampleGroupEntry extends VisualSampleGroupEntry('pccl'){

[0719] unsigned int(4)pcc_layer_count_minus1;

[0720] unsigned int(4)layer_id;

[0721] }

[0722] pcc_layer_count_minus1 plus 1 indicates the number of layers used to encode the geometry and attribute components of the point cloud stream.

[0723] layer_id specifies the layer identifier of the associated sample in the track.

[0724] Next, the layer trace packet is described below.

[0725] From a V-PCC perspective, it is expected that all other types of information (except occupancy and patch information) except geometry can have the same number of layers. In addition, all information (geometry / attribute) marked with the same layer index is related to each other. That is, information marked with the same layer index refers to the same reconstructed point in 3D space.

[0726] In order to construct attributes with layer index M, the geometry layer with index M should also be available. If that geometry layer is missing for some reason, such attribute information will be useless, as the two are "linked" together.

[0727] From a rendering perspective, such information can be considered to "enhance" the reconstructed point cloud. Note that this means that a reasonable reconstruction can still be obtained by discarding some layers. However, according to an embodiment, layer 0 is not discarded.

[0728] According to an embodiment, tracks belonging to the same layer have the same track_group_id value for track_group_type 'pccl', and the track_group_id of a track from one layer is different from the track_group_id of a track from any other layer.

[0729] By default, when track grouping is not indicated for any track in a file, the file is considered to contain only one layer of content.

[0730] aligned(8) class PCCLayerTrackGroupBox extends TrackGroupTypeBox('pccl'){

[0731] unsigned int(4)pcc_layer_count_minus1;

[0732] unsigned int(4)layer_id;

[0733] }

[0734] pcc_layer_count_minus1 plus 1 indicates the number of layers used to encode the geometry and / or attribute components of the point cloud stream.

[0735] layer_id specifies the layer identifier of the associated PCC data (e.g., geometry and / or attributes) layer carried in the track.

[0736] This can inform Figure 20 Player or Figure 1 、 Figure 19 、 Figure 20 or Figure 22 For a renderer, data from tracks belonging to the same track group can be used to reconstruct the point cloud of the same layer.

[0737] As described above, in the present disclosure, signaling information for allowing a client / player to select and use an attribute bitstream as needed is signaled in at least one of a V-PCC track and an attribute track.

[0738] In addition, according to the present disclosure, signaling information related to a random access point among samples in a V-PCC track is signaled in at least one of the V-PCC track and the attribute track.

[0739] Therefore, since the client / player is allowed to obtain signaling information related to the random access point and obtain and use necessary attribute bitstreams based on the signaling information, efficient and faster data processing can be achieved.

[0740] Furthermore, in the present disclosure, when geometry and / or attribute video data in a file consists of multiple layers, track grouping information used together is signaled in at least one of the V-PCC track and the corresponding video track (ie, the geometry track or the attribute track).

[0741] Furthermore, in the present disclosure, when geometry and / or attribute video data consists of multiple layers in one track, sample grouping information thereof is signaled in at least one of the V-PCC track and the corresponding video track (ie, the geometry track or the attribute track).

[0742] Furthermore, in the present disclosure, when a plurality of attribute video data are contained in one track, sample grouping information therefor is signaled in at least one of a V-PCC track and an attribute track.

[0743] The patch frame (or referred to as a point cloud object or atlas frame) as the target of the point cloud data can be divided into one or more tiles, as described above. The tiles according to the embodiment can represent a specific area in 3D space or a specific area in a 2D plane. In addition, the tiles can be a cuboid in a bounding box, or a part of a patch frame or a sub-bounding box. In the present disclosure, the tiles can be divided into one or more tiles. Figure 1 Point cloud video encoder, Figure 18 Patch generator, Figure 20 Point cloud preprocessing unit or Figure 21 The partitioning of a patch frame (or point cloud object) into one or more tiles may be performed by the patch generator of the image processing unit, or by a separate component / module.

[0744] Figure 32 An example of dividing a patch frame into a plurality of tiles by dividing the patch frame into one or more tile rows and one or more tile columns is shown. A tile is a rectangular area of ​​a patch frame, and a tile group may contain multiple tiles of a patch frame. In the present disclosure, a tile group contains multiple tiles of a patch frame that together form a rectangular (quadrilateral) area of ​​a patch frame. In particular, Figure 32 An example is shown in which a patch frame is divided into 24 tiles (6 tile columns and 4 tile rows) and 9 rectangular (quadrilateral) tile groups.

[0745] Here, when Figure 24When the type information in the V-PCC unit header indicates a patch data group (VPCC-PDG) including at least one patch, the corresponding V-PCC unit payload may include a patch data group. Conversely, when the V-PCC unit payload includes a patch data group, the type information in the V-PCC unit header may identify the same content.

[0746] The patch data set may include information about one or more patch tile groups and at least one of a patch sequence parameter set, a patch frame parameter set, a patch frame geometry parameter set, a geometry patch parameter set, a patch frame attribute parameter set, or an attribute patch parameter set.

[0747] Figure 33 An example of a syntax structure of each V-PCC unit according to an embodiment is shown. Each V-PCC unit consists of a V-PCC unit header and a V-PCC unit payload. Figure 33 The V-PCC unit may contain more data. In this case, it may also include a trailing_zero_8bits field. The trailing_zero_8bits field according to an embodiment is a byte corresponding to 0x00.

[0748] Figure 34 An example of the syntax structure of the V-PCC unit header according to an embodiment is shown. In an embodiment, Figure 34 The V-PCC unit header (vpcc_unit_header()) includes a vpcc_unit_type field. The vpcc_unit_type field indicates the type of the V-PCC unit.

[0749] Figure 35 An example of the type of a V-PCC unit assigned to the vpcc_unit_type field according to an embodiment is shown.

[0750] Reference Figure 35 According to an embodiment, the vpcc_unit_type field set to 0 indicates that the data contained in the V-PCC unit payload of the V-PCC unit is a sequence parameter set (VPCC_SPS). The vpcc_unit_type field set to 1 indicates that the data is a patch data group (VPCC_PDG). The vpcc_unit_type field set to 2 indicates that the data is occupied video data (VPCC_OVD). The vpcc_unit_type field set to 3 indicates that the data is attribute video data (VPCC_AVD). The vpcc_unit_type field set to 4 indicates that the data is geometry video data (VPCC_GVD).

[0751] Those skilled in the art may easily change the meaning, order, deletion, addition, etc. of the values ​​assigned to the vpcc_unit_type field, and thus the present disclosure shall not be limited to the above-described embodiments.

[0752] Here, the V-PCC unit payload conforms to the format of the HEVC NAL unit. That is, according to the value of the vpcc_unit_type field, the occupancy, geometry, and attribute video data V-PCC unit payload corresponds to a video data unit decodable by a specified video decoder in the corresponding occupancy, geometry, and attribute parameter set V-PCC unit (e.g., HEVC NAL unit).

[0753] According to an embodiment, when the vpcc_unit_type field indicates attribute video data (VPCC_AVD), geometry video data (VPCC_GVD), occupied temporal video data (VPCC_OVD), or patch data group (VPCC_PDG), the V-PCC unit header further includes a vpcc_sequence_parameter_set_id field.

[0754] The vpcc_sequence_parameter_set_id field specifies an identifier (ie, sps_sequence_parameter_set_id) of the active sequence parameter set (VPCC SPS). The value of the vpcc_sequence_parameter_set_id field may be in the range of 0 to 15, inclusive.

[0755] According to an embodiment, when the vpcc_unit_type field indicates attribute video data (VPCC_AVD), the V-PCC unit header further includes a vpcc_attribute_index field and a vpcc_attribute_dimension_index field.

[0756] The vpcc_attribute_index field indicates the index of attribute video data carried in the attribute video data unit.The value of the vpcc_attribute_index field may be in the range of 0 to (ai_attribute_count-1), inclusive.

[0757] The vpcc_attribute_dimension_index field indicates the index of the attribute dimension group carried in the attribute video data unit. The value of the vpcc_attribute_dimension_index field may be in the range of 0 to 127, inclusive.

[0758] Figure 34 The sps_multiple_layer_streams_present_flag field in the V-PCC unit header indicates whether the vpcc_layer_index field and the pcm_separate_video_data(11) field are included.

[0759] For example, if the value of the vpcc_unit_type field indicates attribute video data (VPCC_AVD) and the value of the sps_multiple_layer_streams_present_flag field is true (e.g., 0), the vpcc_layer_index field and the pcm_separate_video_data(11) field are further included in the V-PCC unit header. That is, when the value of the sps_multiple_layer_streams_present_flag field is true, it indicates that there are multiple layers for attribute video data or geometry video data. In this case, a field indicating the index of the current layer (e.g., vpcc_layer_index) is required.

[0760] The vpcc_layer_index field indicates the index of the current layer of the attribute video data. The value of the vpcc_layer_index field ranges from 0 to 15.

[0761] For example, when the value of the vpcc_unit_type field indicates attribute video data (VPCC_AVD) and the value of the sps_multiple_layer_streams_present_flag field is false (e.g., 1), the pcm_separate_video_data (15) field is further included in the V-PCC unit header. That is, the sps_multiple_layer_streams_present_flag field is set to false to indicate that there are no multiple layers for attribute video data and / or geometry video data. In this case, the field indicating the index of the current layer is not required.

[0762] For example, when the value of the vpcc_unit_type field indicates geometry video data (VPCC_GVD) and the value of the sps_multiple_layer_streams_present_flag field is true (eg, 0), a vpcc_layer_index field and a pcm_separate_video_data (18) field are further included in the V-PCC unit header.

[0763] The vpcc_layer_index field indicates the index of the current layer of the geometry video data. The value of the vpcc_layer_index field is in the range of 0 to 15.

[0764] For example, when the value of the vpcc_unit_type field indicates geometry video data (VPCC_GVD) and the value of the sps_multiple_layer_streams_present_flag field is false (eg, 1), a pcm_separate_video_data (22) field is further included in the V-PCC unit header.

[0765] For example, if the value of the vpcc_unit_type field indicates occupied video data (VPCC_OVD) or patch data group (VPCC_PDG), a vpcc_reserved_zero_23bits field is further included in the V-PCC unit header. Otherwise, a vpcc_reserved_zero_27bits field is further included.

[0766] Figure 34 The V-PCC unit header may further include a vpcc_pcm_video_flag field.

[0767] For example, a vpcc_pcm_video_flag field set to 1 may indicate that the associated geometry or attribute video data unit contains only PCM (Pulse Code Modulation) code points. As another example, a vpcc_pcm_video_flag field set to 0 may indicate that the associated geometry or attribute video data unit may contain non-PCM code points. When the vpcc_pcm_video_flag field is not present, the value of this field may be inferred to be equal to 0.

[0768] Figure 36 An example of a syntax structure of pcm_separate_video_data(bitCount) included in a V-PCC unit header according to an embodiment is shown.

[0769] exist Figure 36 In the pcm_separate_video_data, the bitCount is as described above. Figure 34 The value of the vpcc_unit_type field in the V-PCC unit header varies.

[0770] If the value of the sps_pcm_separate_video_present_flag field is true and the vpcc_layer_index field is not, pcm_separate_video_data(bitCount) includes the vpcc_pcm_video_flag field and the vpcc_reserved_zero_bitcount_bits field. Otherwise, it may include the vpcc_reserved_bits field.

[0771] The vpcc_pcm_video_flag field set to 1 may indicate that the associated geometry or attribute video data unit is only PCM code point video. The vpcc_pcm_video_flag field set to 0 may indicate that the associated geometry or attribute video data unit may contain PCM code points. When the vpcc_pcm_video_flag field is not present, its value may be considered to be 0.

[0772] Figure 37 An example of a syntax structure of a V-PCC unit payload according to an embodiment is shown.

[0773] Figure 37 The V-PCC unit payload contains one of a sequence parameter set (sequence_parameter_set()), a patch data group (patch_data_group()), and a video data unit (video_data_unit()) according to the value of the vpcc_unit_type field in the V-PCC unit header.

[0774] For example, according to an embodiment, when the vpcc_unit_type field indicates a sequence parameter set (VPCC_SPS), the V-PCC unit payload includes sequence_parameter_set(). When the vpcc_unit_type field indicates a patch data group (VPCC_PDG), the V-PCC unit payload includes patch_data_group(). Furthermore, when the vpcc_unit_type field indicates occupied video data (VPCC_OVD), the V-PCC unit payload includes an occupied video data unit (video_data_unit()) that carries the occupied video data. When the vpcc_unit_type field indicates geometric video data (VPCC_GVD), the V-PCC unit payload includes a geometric video data unit (video_data_unit()) that carries the geometric video data. When the vpcc_unit_type field indicates attribute video data (VPCC_AVD), the V-PCC unit payload includes an attribute video data unit (video_data_unit()).

[0775] Figure 38 An example of a syntax structure of sequence_parameter_set() included in a V-PCC unit payload according to an embodiment is shown.

[0776] Figure 38 A sequence parameter set (SPS) may be applied to a coded point cloud sequence comprising a sequence of coded geometry video data units, attribute video data units, and occupancy video data units.

[0777] Figure 38 The sequence parameter set (SPS) may include a profile_tier_level(), a sps_sequence_parameter_set_id field, a sps_frame_width field, a sps_frame_height field, and a sps_avg_frame_rate_present_flag field.

[0778] profile_tier_level() specifies the codec information used to compress SPS.

[0779] The sps_sequence_parameter_set_id field provides an identifier of the VPCC SPS referenced by other syntax elements.

[0780] The sps_frame_width field is the nominal frame width expressed in integer luma samples.

[0781] The sps_frame_height field is the nominal frame height expressed in integer luma samples.

[0782] The sps_avg_frame_rate_present_flag field indicates whether the average nominal frame rate information is included in the bitstream. For example, the sps_avg_frame_rate_present_flag field set to 0 may indicate that the average nominal frame rate information is not in the bitstream. The sps_avg_frame_rate_present_flag field set to 1 may indicate that the average nominal frame rate information should be indicated in the bitstream. For example, when the value of the sps_avg_frame_rate_present_flag field is true, that is, 1, the sequence parameter set may further include the sps_avg_frame_rate field, the sps_enhanced_occupancy_map_for_depth_flag field, and the sps_geometry_attribute_different_layer_flag field.

[0783] The sps_avg_frame_rate field indicates the average nominal point cloud frame rate, in units of point cloud frames per 256 seconds. When the sps_avg_frame_rate field is not present, the value of this field shall be inferred to be equal to 0. During the reconstruction phase, the decoded occupancy, geometry, and attribute videos may be converted to the nominal width, height, and frame rate using appropriate scaling.

[0784] The sps_enhanced_occupancy_map_for_depth_flag field indicates whether the decoded occupancy map video contains information about whether intermediate depth positions between two depth layers are occupied. For example, the sps_enhanced_occupancy_map_for_depth_flag field set to 1 may indicate that the decoded occupancy map video contains information about whether intermediate depth positions between two depth layers are occupied. The sps_enhanced_occupancy_map_for_depth_flag field set to 0 may indicate that the decoded occupancy map video does not contain information about whether intermediate depth positions between two depth layers are occupied.

[0785] The sps_geometry_attribute_different_layer_flag field indicates whether the number of layers used for encoding of geometry and attribute video data is different. For example, the sps_geometry_attribute_different_layer_flag field set to 1 may indicate that the number of layers used for encoding of geometry and attribute video data is different. For example, two layers may be used to encode geometry video data, and one layer may be used to encode attribute video data. In addition, the sps_geometry_attribute_different_layer_flag field set to 1 may indicate whether the number of layers used for encoding of geometry and attribute data is signaled in the patch sequence data unit.

[0786] The sps_geometry_attribute_different_layer_flag field indicates whether the sps_layer_count_geometry_minus1 field and the sps_layer_count_minus1 field are included. For example, when the value of the sps_geometry_attribute_different_layer_flag field is true (for example, 1), the sps_layer_count_geometry_minus1 field may be further included. When the value is false (for example, 0), the sps_layer_count_minus1 field may be further included.

[0787] The sps_layer_count_geometry_minus1 field indicates the number of layers used for encoding of geometry data.

[0788] The sps_layer_count_minus1 field indicates the number of layers used to encode geometry and attribute video data.

[0789] When the value of the sps_layer_count_minus1 field is greater than 0, the sequence parameter set may further include a sps_multiple_layer_streams_present_flag field and a sps_layer_absolute_coding_enabled_flag[0]=1 field.

[0790] The sps_multiple_layer_streams_present_flag field indicates whether the geometry layer or attribute layer is placed in a single video stream or in separate video streams. For example, the sps_multiple_layer_streams_present_flag field is set to 0 to indicate that all geometry layers or attribute layers are placed in a single geometry or attribute video stream, respectively. The sps_multiple_layer_streams_present_flag field is set to 1 to indicate that all geometry layers or attribute layers are placed in separate video streams.

[0791] In addition, the sequence parameter set (SPS) includes an iteration statement that is repeated as many times as the value of the sps_layer_count_minus1 field. The iteration statement includes an sps_layer_absolute_coding_enabled_flag field. According to an embodiment, a variable i is initialized to 0 and incremented by 1 each time the iteration statement is executed. The iteration statement is repeated until i equals the value of the sps_layer_count_minus1 field. In addition, if the value of the sps_layer_absolute_coding_enabled_flag field is 0 and the value of i is greater than 0, the sps_layer_predictor_index_diff field is further included. Otherwise, the sps_layer_predictor_index_diff field is not included.

[0792] The sps_layer_absolute_coding_enabled_flag[i] field set to 1 may indicate that the geometry layer with index i is coded without any form of layer prediction. The sps_layer_absolute_coding_enabled_flag[i] field set to 0 may indicate that the geometry layer with index i is first predicted from another earlier coded layer before coding.

[0793] When the sps_layer_absolute_coding_enabled_flag[i] field is set to 0, the sps_layer_predictor_index_diff[i] field is used to calculate the predictor of the geometry layer with index i.

[0794] The SPS according to the present disclosure may further include an sps_pcm_patch_enabled_flag field. The sps_pcm_patch_enabled_flag field indicates whether the sps_pcm_separate_video_present_flag field, occupancy_parameter_set(), geometry_parameter_set(), and sps_attribute_count fields are included. For example, when the sps_pcm_patch_enabled_flag field is set to 1, the sps_pcm_separate_video_present_flag field, occupancy_parameter_set(), geometry_parameter_set(), and sps_attribute_count fields may be further included. That is, the sps_pcm_patch_enabled_flag field is set to 1 to indicate that patches with PCM code points can be present in the bitstream.

[0795] The sps_pcm_separate_video_present_flag field indicates whether PCM-encoded geometry video data and attribute video data are stored in separate video streams. For example, the sps_pcm_separate_video_present_flag field is set to 1 to indicate that PCM-encoded geometry video data and attribute video data can be stored in separate video streams.

[0796] occupancy_parameter_set() contains information about the occupancy map.

[0797] geometry_parameter_set() contains information about the geometry of the video data.

[0798] The sps_attribute_count field indicates the number of attributes associated with the point cloud.

[0799] In addition, the SPS according to the present disclosure includes an iteration statement that is repeated as many times as the value of the sps_attribute_count field. According to an embodiment, when the sps_geometry_attribute_different_layer_flag field is set to 1, the iteration statement includes the sps_layer_count_attribute_minus1 field and attribute_parameter_set(). In the iteration statement, i is initialized to 0 and incremented by 1 each time the iteration statement is executed. The iteration statement is repeated until i equals the value of the sps_attribute_count field.

[0800] The sps_layer_count_attribute_minus1[i] field indicates the number of layers used to encode the i-th attribute video data associated with the point cloud.

[0801] attribute_parameter_set(i) contains information about the i-th attribute video data associated with the point cloud.

[0802] According to an embodiment of the present disclosure, the SPS further includes a sps_patch_sequence_orientation_enabled_flag field, a sps_patch_inter_prediction_enabled_flag field, a sps_pixel_deinterleaving_flag field, a sps_point_local_reconstruction_enabled_flag field, a sps_remove_duplicate_point_enabled_flag field, and a byte_alignment() field.

[0803] The sps_patch_sequence_orientation_enabled_flag field indicates whether flexible orientation is signaled in the patch sequence data unit. For example, the sps_patch_sequence_orientation_enabled_flag field is set to 1 to indicate that flexible orientation is signaled in the patch sequence data unit. The sps_patch_sequence_orientation_enabled_flag field is set to 0 to indicate that flexible orientation is not signaled.

[0804] The sps_patch_inter_prediction_enabled_flag field set to 1 indicates that inter prediction by using patch information from a previously encoded patch frame can be used.

[0805] The sps_pixel_deinterleaving_flag field set to 1 indicates that the decoded geometry and attribute video corresponding to a single stream contains interleaved pixels from two layers. The sps_pixel_deinterleaving_flag field set to 0 indicates that the decoded geometry and attribute video corresponding to a single stream contains pixels from only a single layer.

[0806] The sps_point_local_reconstruction_enabled_flag field set to 1 indicates that the local reconstruction mode is used during the point cloud reconstruction process.

[0807] The sps_remove_duplicate_point_enabled_flag field set to 1 indicates that duplicate points should not be reconstructed. Here, a duplicate point is a point that has the same 2D and 3D geometric coordinates as another point from a lower layer.

[0808] Figure 39 An example of a syntax structure of a patch data group (patch_data_group()) according to an embodiment is shown.

[0809] As described above, when the value of the vpcc_unit_type field in the V-PCC unit header indicates a patch data group, Figure 37 The V-PCC unit payload contains Figure 39 patch_data_group().

[0810] According to one embodiment, patch_data_group() includes a pdg_unit_type field, a pdg_data_group_unit_payload (pdg_unit_type) having signaling information that varies according to a value of the pdg_unit_type field, and a pdg_terminate_patch_data_group_flag field.

[0811] The pdg_unit_type field indicates the type of patch data group.

[0812] The pdg_terminate_patch_data_group_flag field indicates the end of the patch data group. The pdg_terminate_patch_data_group_flag field set to 0 may indicate that there are additional patch data group units in the patch data group. The pdg_terminate_patch_data_group_flag field set to 1 may indicate that there are no more patch data group units in the patch data group and this is the end of the current patch data group unit.

[0813] Figure 40 Is assigned to Figure 39 A table showing examples of the types of patch data groups of the pdg_unit_type field of the patch data group.

[0814] For example, the pdg_unit_type field set to 0 may indicate a patch sequence parameter set (PDG_PSPS). The pdg_unit_type field set to 1 may indicate a patch frame parameter set (PDG_PFPS). The pdg_unit_type field set to 2 may indicate a patch frame geometry parameter set (PDG_PFGPS). The pdg_unit_type field set to 3 may indicate a patch frame attribute parameter set (PDG_PFAPS). The pdg_unit_type field set to 4 may indicate a geometry patch parameter set (PDG_GPPS). The pdg_unit_type field set to 5 may indicate an attribute patch parameter set (PDG_APPS). The pdg_unit_type field set to 6 may indicate a patch tile group layer unit (PDG_PTGLU). The pdg_unit_type field set to 7 may indicate a prefix SEI message (PDG_PREFIX_SEI). The pdg_unit_type field set to 8 may indicate a suffix SEI message (PDG_SUFFIX_SEI).

[0815] PDG_PSPS may include sequence level parameters, and PDG_PFPS may include frame level parameters. PDG_PFGPS may include frame level geometry type parameters, and PDG_PFAPS may include frame level attribute type parameters. PDG_GPPS may include patch level geometry type parameters, and PDG_APPS may include patch level attribute type parameters.

[0816] Figure 41 An example of the syntax structure of a patch data group unit payload (patch_data_group_unit_payload(pdg_unit_type)) according to an embodiment is shown.

[0817] when Figure 39 When the value of the pdg_unit_type field of the patch data group indicates PDG_PSPS, patch_data_group_unit_payload() may include a patch sequence parameter set (patch_sequence_parameter_set()).

[0818] When the value of the pdg_unit_type field indicates PDG_GPPS, patch_data_group_unit_payload() may include a geometry patch parameter set (geometry_patch_parameter_set()).

[0819] When the value of the pdg_unit_type field indicates PDG_APPS, patch_data_group_unit_payload() may contain an attribute patch parameter set (attribute_patch_parameter_set()).

[0820] When a value of the pdg_unit_type field indicates PDG_PFPS, patch_data_group_unit_payload() may contain a patch frame parameter set (patch_frame_parameter_set()).

[0821] When the value of the pdg_unit_type field indicates PDG_PFAPS, patch_data_group_unit_payload() may contain a patch frame attribute parameter set (patch_frame_attribute_parameter_set()).

[0822] When the value of the pdg_unit_type field indicates PDG_PFGPS, patch_data_group_unit_payload() may contain a patch frame geometry parameter set (patch_frame_geometry_parameter_set()).

[0823] When the value of the pdg_unit_type field indicates PDG_PTGLU, patch_data_group_unit_payload() may contain a patch tile group layer unit (patch_tile_group_layer_unit()).

[0824] When the value of the pdg_unit_type field indicates PDG_PREFIX_SEI or PDG_SUFFIX_SEI, patch_data_group_unit_payload() may contain sei_message().

[0825] Figure 42 An example of the syntax structure of the supplemental enhancement information (SEI) message (sei_message()) according to an embodiment is shown. That is, when included in Figure 39 When the value of the pdg_unit_type field in the patch data group indicates PDG_PREFIX_SEI or PDG_SUFFIX_SEI, Figure 41 The patch_data_group_unit_payload() contains sei_message().

[0826] Each SEI message according to an embodiment consists of an SEI message header and an SEI message payload.The SEI message header includes an sm_payload_type_byte field and an sm_payload_size_byte field.

[0827] The sm_payload_type_byte field is a byte of the payload type of the SEI message. For example, it can be identified whether the SEI message is a prefix SEI message or a suffix SEI message based on the value of the sm_payload_type_byte field.

[0828] The sm_payload_size_byte field is the bytes of the payload size of the SEI message.

[0829] Figure 42 The SEI message initializes PayloadType to 0, and then sets the value of the sm_payload_type_byte field in the iteration statement to the value of PayloadType. When the value of the sm_payload_type_byte field is 0xFF, the iteration statement terminates.

[0830] also, Figure 42 The SEI message initializes PayloadSize to 0, and then sets the value of the sm_payload_size_byte field in the iteration statement to the value of PayloadSize. When the value of the sm_payload_size_byte field is 0xFF, the iteration statement terminates.

[0831] Then, information corresponding to the payloadType and payloadSize set in the two iteration statements is signaled through the payload (sei_payload(payloadType, payloadSize)) of the SEI message.

[0832] Figure 43 An example of a V-PCC bitstream structure according to other embodiments of the present disclosure is shown. In an embodiment, Figure 43 The V-PCC bitstream is composed of Figure 1 、 Figure 4 、 Figure 18 、 Figure 20 or Figure 21 Generation and output of V-PCC based point cloud video encoder.

[0833] A V-PCC bitstream containing a coded point cloud sequence (CPCS) according to an embodiment may be composed of a sample stream V-PCC unit. The sample stream V-PCC unit carries V-PCC parameter set (VPS) data, an atlas bitstream, an occupancy map bitstream for 2D video coding, a geometry bitstream for 2D video coding, and zero or more attribute bitstreams for 2D video coding.

[0834] exist Figure 43 In the present invention, a V-PCC bitstream may include a sample stream V-PCC header 40010 and one or more sample stream V-PCC units 40020. For simplicity, one or more sample stream V-PCC units 40020 may be referred to as a sample stream V-PCC payload. In other words, a sample stream V-PCC payload may be referred to as a collection of sample stream V-PCC units.

[0835] Each sample stream V-PCC unit 40021 may include V-PCC unit size information 40030 and a V-PCC unit 40040. The V-PCC unit size information 40030 indicates the size of the V-PCC unit 40040. For simplicity, the V-PCC unit size information 40030 may be referred to as a sample stream V-PCC unit header, and the V-PCC unit 40040 may be referred to as a sample stream V-PCC unit payload.

[0836] Each V-PCC unit 40040 may include a V-PCC unit header 40041 and a V-PCC unit payload 40042 .

[0837] In the present disclosure, data contained in a V-PCC unit payload 40042 is distinguished by a V-PCC unit header 40041. To this end, the V-PCC unit header 40041 includes type information indicating the type of the V-PCC unit. Depending on the type information in the V-PCC unit header 40041, each V-PCC unit payload 40042 may include geometry video data (i.e., a geometry bitstream for 2D video encoding), attribute video data (i.e., an attribute bitstream for 2D video encoding), occupancy video data (i.e., an occupancy map bitstream for 2D video encoding), atlas data, or a V-PCC parameter set (VPS).

[0838] The VPS according to an embodiment is also referred to as a sequence parameter set (SPS). These two terms can be used interchangeably.

[0839] Atlas data according to an embodiment may mean data consisting of attributes (eg, texture (patch)) and / or depth of point cloud data, and may also be referred to as a patch data group.

[0840] Figure 44 An example of data carried by a sample stream V-PCC unit in a V-PCC bitstream according to an embodiment is shown.

[0841] exist Figure 44 In the example, the V-PCC bitstream includes a sample stream V-PCC unit carrying a V-PCC parameter set (VPS), a sample stream V-PCC unit carrying atlas data (AD), a sample stream V-PCC unit carrying occupancy video data (OVD), a sample stream V-PCC unit carrying geometric video data (GVD), and a sample stream V-PCC unit carrying attribute video data (AVD).

[0842] According to an embodiment, each sample stream V-PCC unit contains one type of V-PCC unit among VPS, AD, OVD, GVD and AVD.

[0843] Figure 45 An example of a syntax structure of a sample stream V-PCC header included in a V-PCC bitstream according to an embodiment is shown.

[0844] sample_stream_v-pcc_header() according to an embodiment may include an ssvh_unit_size_precision_bytes_minus1 field and an ssvh_reserved_zero_5bits field.

[0845] The value of the ssvh_unit_size_precision_bytes_minus1 field plus 1 may specify the precision (in bytes) of the ssvu_vpcc_unit_size element in all sample stream V-PCC units. The value of this field may be in the range of 0 to 7.

[0846] The ssvh_reserved_zero_5bits field is reserved for future use.

[0847] Figure 46 An example of a syntax structure of a sample stream V-PCC unit (sample_stream_vpcc_unit()) according to an embodiment is shown.

[0848] The content of each sample stream V-PCC unit is associated with the same access unit as the V-PCC unit contained in the sample stream V-PCC unit.

[0849] sample_stream_vpcc_unit() according to an embodiment may include a ssvu_vpcc_unit_size field and vpcc_unit (ssvu_vpcc_unit_size).

[0850] The ssvu_vpcc_unit_size field corresponds to Figure 43 The V-PCC unit size information 40030 specifies the size of the subsequent vpcc_unit in bytes. The number of bits used to represent the ssvu_vpcc_unit_size field is equal to (ssvh_unit_size_precision_bytes_minus1+1)*8.

[0851] vpcc_unit (ssvu_vpcc_unit_size) has a length corresponding to the value of the ssvu_vpcc_unit_size field and carries one of VPS, AD, OVD, GVD, and AVD.

[0852] Figure 47 An example of the syntax structure of a V-PCC unit according to an embodiment is shown. A V-PCC unit consists of a V-PCC unit header (vpcc_unit_header()) and a V-PCC unit payload (vpcc_unit_payload()). A V-PCC unit according to an embodiment may contain more data. In this case, it may also include a trailing_zero_8bits field. The trailing_zero_8bits field according to an embodiment is a byte corresponding to 0x00.

[0853] Figure 48 An example of the syntax structure of the V-PCC unit header according to an embodiment is shown. In an embodiment, Figure 48 The vpcc_unit_header() includes a vuh_unit_type field. The vuh_unit_type field indicates the type of the corresponding V-PCC unit. The vuh_unit_type field according to an embodiment is also called a vpcc_unit_type field.

[0854] Figure 49 An example of a V-PCC unit type assigned to the vuh_unit_type field according to an embodiment is shown.

[0855] Reference Figure 49 According to an embodiment, the vuh_unit_type field set to 0 indicates that the data contained in the V-PCC unit payload of the V-PCC unit is a V-PCC parameter set (VPCC_VPS). The vuh_unit_type field set to 1 indicates that the data is atlas data (VPCC_AD). The vuh_unit_type field set to 2 indicates that the data is occupied video data (VPCC_OVD). The vuh_unit_type field set to 3 indicates that the data is geometry video data (VPCC_GVD). The vuh_unit_type field set to 4 indicates that the data is attribute video data (VPCC_AVD).

[0856] Those skilled in the art can easily change the meaning, order, deletion, addition, etc. of the values ​​assigned to the vuh_unit_type field, and thus the present disclosure will not be limited to the above-mentioned embodiments.

[0857] When the vuh_unit_type field indicates VPCC_AVD, VPCC_GVD, VPCC_OVD, or VPCC_AD, the V-PCC unit header according to an embodiment may further include a vuh_vpcc_parameter_set_id field and a vuh_atlas_id field.

[0858] The vuh_vpcc_parameter_set_id field specifies the value of the vps_vpcc_parameter_set_id of the active V-PC VPS.

[0859] The vuh_atlas_id field specifies the index of the atlas corresponding to the current V-PCC unit.

[0860] When the vuh_unit_type field indicates VPCC_AVD, the V-PCC unit header according to an embodiment may further include a vuh_attribute_index field, a vuh_attribute_dimension_index field, a vuh_map_index field, and a vuh_raw_video_flag field.

[0861] The vuh_attribute_index field indicates the index of attribute data carried in the attribute video data unit.

[0862] The vuh_attribute_dimension_index field indicates the index of the attribute dimension group carried in the attribute video data unit.

[0863] When present, the vuh_map_index field may indicate the map index of the current geometry or attribute stream.

[0864] The vuh_raw_video_flag field may indicate whether RAW code points are included. For example, a vuh_raw_video_flag field set to 1 may indicate that the associated attribute video data unit contains only RAW code points. As another example, a vuh_raw_video_flag field set to 0 may indicate that the associated attribute video data unit may contain RAW code points. When the vuh_raw_video_flag field is not present, its value may be inferred to be equal to 0. Depending on the embodiment, RAW code points are also referred to as pulse code modulation (PCM) code points.

[0865] When the vuh_unit_type field indicates VPCC_GVD, the V-PCC unit header according to an embodiment may further include a vuh_map_index field, a vuh_raw_video_flag field, and a vuh_reserved_zero_12bits field.

[0866] When present, the vuh_map_index field indicates the index of the current geometry stream.

[0867] The vuh_raw_video_flag field may indicate whether RAW code points are included. For example, a vuh_raw_video_flag field set to 1 may indicate that the associated geometry video data unit contains only RAW code points. As another example, a vuh_raw_video_flag field set to 0 may indicate that the associated geometry video data unit may contain RAW code points. When the vuh_raw_video_flag field is not present, its value may be inferred to be equal to 0. Depending on the embodiment, RAW code points are also referred to as PCM code points.

[0868] The vuh_reserved_zero_12bits field is reserved for future use.

[0869] If the vuh_unit_type field indicates VPCC_OVD or VPCC_AD, the V-PCC unit header according to an embodiment may further include a vuh_reserved_zero_17bits field. Otherwise, the V-PCC unit header may further include a vuh_reserved_zero_27bits field.

[0870] The vuh_reserved_zero_17bits field and the vuh_reserved_zero_27bits field are reserved for future use.

[0871] Figure 50 An example of a syntax structure of a V-PCC unit payload (vpcc_unit_payload()) according to an embodiment is shown.

[0872] Figure 50 The V-PCC unit payload may contain one of a V-PCC parameter set (vpcc_parameter_set()), an atlas sub-bitstream (atlas_sub_bitstream()), and a video sub-bitstream (video_sub_bitstream()) according to a value of the vuh_unit_type field in the V-PCC unit header.

[0873] For example, when the vuh_unit_type field indicates VPCC_VPS, the V-PCC unit payload includes vpcc_parameter_set(), which contains overall coding information about the bitstream. When the vuh_unit_type field indicates VPCC_AD, the V-PCC unit payload includes atlas_sub_bitstream(), which carries atlas data. Furthermore, depending on the embodiment, when the vuh_unit_type field indicates VPCC_OVD, the V-PCC unit payload includes an occupation video sub-bitstream (video_sub_bitstream()) that carries occupation video data. When the vuh_unit_type field indicates VPCC_GVD, the V-PCC unit payload includes a geometry video sub-bitstream (video_sub_bitstream()) that carries geometry video data. When the vuh_unit_type field indicates VPCC_AVD, the V-PCC unit payload includes an attribute video sub-bitstream (video_sub_bitstream()) that carries attribute video data.

[0874] According to an embodiment, the atlas sub-bitstream may be referred to as an atlas sub-stream, and the occupied video sub-bitstream may be referred to as an occupied video sub-stream. The geometry video sub-bitstream may be referred to as a geometry video sub-stream, and the attribute video sub-bitstream may be referred to as an attribute video sub-stream. The V-PCC unit payload according to an embodiment conforms to the format of the High Efficiency Video Coding (HEVC) Network Abstraction Layer (NAL) unit.

[0875] Figure 51 is a diagram showing an example of the structure of an atlas sub-stream according to an embodiment. In an embodiment, Figure 51 The atlas substream conforms to the HEVC NAL unit format.

[0876] The atlas substream according to an embodiment may be composed of a sample stream NAL unit containing an atlas sequence parameter set (ASPS), a sample stream NAL unit containing an atlas frame parameter set (AFPS), and one or more sample stream NAL units containing information about one or more atlas tile groups and / or one or more sample stream NAL units containing one or more SEI messages.

[0877] The one or more SEI messages according to an embodiment may include a prefix SEI message and a suffix SEI message.

[0878] The atlas substream according to an embodiment may further include a sample stream NAL header placed in front of one or more NAL units.

[0879] Figure 52 An example of a syntax structure of a sample stream NAL header (sample_stream_nal_header()) included in an atlas substream according to an embodiment is shown.

[0880] sample_stream_nal_header() according to an embodiment may include a ssnh_unit_size_precision_bytes_minus1 field and a ssnh_reserved_zero_5bits field.

[0881] The value of the ssnh_unit_size_precision_bytes_minus1 field plus 1 may specify the precision (in bytes) of the ssnu_nal_unit_size element in all sample stream NAL units. The value of this field may be in the range of 0 to 7.

[0882] The ssnh_reserved_zero_5bits field is reserved for future use.

[0883] Figure 53 An example of a syntax structure of a sample stream NAL unit (sample_stream_nal_unit()) according to an embodiment is shown.

[0884] sample_stream_nal_unit() according to an embodiment may include a ssnu_nal_unit_size field and nal_unit (ssnu_nal_unit_size).

[0885] The ssnu_nal_unit_size field specifies the size of the subsequent NAL_unit in bytes. The number of bits used to represent the ssnu_nal_unit_size field is equal to (ssnh_unit_size_precision_bytes_minus1+1)*8.

[0886] The nal_unit (ssnu_nal_unit_size) has a length corresponding to the value of the ssnu_nal_unit_size field and carries one of the Atlas Sequence Parameter Set (ASPS), the Atlas Frame Parameter Set (AFPS), the Atlas Tile Group Information, and the SEI message. That is, each sample stream NAL unit may contain the ASPS, AFPS, the Atlas Tile Group Information, or the SEI message. Depending on the embodiment, the ASPS, AFPS, the Atlas Tile Group Information, and the SEI message are referred to as Atlas data (or Atlas metadata).

[0887] SEI messages according to embodiments may assist in processes related to decoding, reconstruction, display, or other purposes.

[0888] Each SEI message according to an embodiment consists of an SEI message header and an SEI message payload (sei_payload). The SEI message header may include payload type information (payloadType) and payload size information (payloadSize).

[0889] The payload type indicates the payload type of the SEI message. For example, it is possible to identify whether the SEI message is a prefix SEI message or a suffix SEI message based on the payloadType.

[0890] Payload Size indicates the payload size of the SEI message.

[0891] Figure 54 An example of the syntax structure of the SEI message payload (sei_payload()) according to an embodiment is shown.

[0892] The SEI message according to the embodiment may include a prefix SEI message or a suffix SEI message. In addition, each SEI message signals information corresponding to payload type information (payloadType) and payload size information (payloadSize) through the SEI message payload (sei_payload (payloadType, payloadSize)).

[0893] When payloadType is equal to 0, the prefix SEI message according to the implementation may include buffering_period(payloadSize). When payloadType is equal to 1, the prefix SEI message may include pic_timing(payloadSize). When payloadType is equal to 2, the prefix SEI message may include filler_payload(payloadSize). When payloadType is equal to 10, the prefix SEI message may include sei_prefix_indication(payloadSize). When payloadType is equal to 13, the prefix SEI message may include 3d_region_mapping(payloadSize).

[0894] When payloadType is equal to 2, the suffix SEI message may include filler_payload(payloadSize). When payloadType is equal to 3, the suffix SEI message may include user_data_registered_itu_t_t35(payloadSize). When payloadType is equal to 4, the suffix SEI message may include user_data_unregistered(payloadSize). When payloadType is equal to 11, the suffix SEI message may include decoded_pcc_hash(payloadSize).

[0895] According to an embodiment, Figure 24 or Figure 43 The V-PCC bit stream of the structure shown is composed of Figure 1 、 Figure 18 、 Figure 20 or Figure 21 The file / segment encapsulator is encapsulated in the ISOBMFF file format.

[0896] In this case, the V-PCC stream may be sent via multiple tracks or a single track of the file.

[0897] An ISOBMFF-based file according to an embodiment may consist of boxes and / or information, which may be referred to as ftyp, meta, moov, or mdat.

[0898] The ftyp box (file type box) can provide information related to the file compatibility or file type of the file. The receiving side can refer to the ftyp box to identify the file.

[0899] The meta box may include a vpcg{0,1,2,3} box (V-PCC group box).

[0900] The mdat box, also called the media data box, may contain a video coding geometry bitstream, a video coding attributes bitstream, a video coding occupancy map bitstream and / or an atlas data bitstream.

[0901] The moov box, also known as a movie box, can contain metadata about the file's media data (e.g., geometry bitstream, attribute bitstream, occupancy map bitstream, etc.). For example, it can contain information required to decode and play the media data, as well as information about the file's samples. The moov box can serve as a container for all metadata. The moov box can be the highest-level box among metadata-related boxes. Depending on the implementation, only one moov box may exist in a file.

[0902] The box according to an embodiment may include a track (trak) box that provides information related to the track of the file. The track box may include a media (mdia) box that provides media information about the track and a track reference container (tref) box for referencing the track and samples of the file corresponding to the track.

[0903] The mdia box may include a media information container (minf) box that provides information about media data. The minf box may include a sample table (stbl) box that provides metadata related to samples of the mdat box.

[0904] The stbl box may include a sample description (stsd) box that provides information about the encoding type employed and the initialization information required for that encoding type.

[0905] According to an embodiment, the stsd box may include sample entries for storing tracks of a V-PCC bitstream.

[0906] In order to store a V-PCC bitstream according to an embodiment in a single track or multiple tracks in a file, the present disclosure defines a volume visual track, a volume visual media header, a volume sample entry, a volume sample, and samples and simple entries of a V-PCC track.

[0907] The term V-PCC used in this paper is the same as visual volumetric video coding (V3C). The two terms can be used to complement each other.

[0908] According to an embodiment, video-based point cloud compression (V-PCC) represents volumetric encoding of point cloud visual information.

[0909] That is, the minf box in the trak box of the moov box may further include a volumetric visual media header box. The volumetric visual media header box includes information about the volumetric visual track including the volumetric visual scene.

[0910] Each volumetric vision scene can be represented by a unique volumetric vision track. An ISOBMFF file can contain multiple scenes, so there can be multiple volumetric vision tracks in an ISOBMFF file.

[0911] According to an embodiment, a volumetric visual track may be identified by a volumetric visual media handler type 'volv' in the HandlerBox of the MediaBox.

[0912] The syntax of the volumetric visual media header box according to an embodiment is configured as follows.

[0913] Box type: 'vvhd'

[0914] Container: MediaInformationBox

[0915] Mandatory: Yes

[0916] Quantity: Exactly one

[0917] The volumetric visual track according to an embodiment may use the volumetric visual media handler box (VolumetricVisualMediaHeaderBox) in the media information box (MediaInformationBox) as follows.

[0918] aligned(8) class VolumetricVisualMediaHeaderBox

[0919] Expand FullBox('vvhd',version=0,1){

[0920] }

[0921] version may be an integer indicating the version of the box.

[0922] A volumetric visual track according to an embodiment may use a volumetric visual sample entry (VolumetricVisualSampleEntry) as follows.

[0923] Class VolumetricVisualSampleEntry(codingname)

[0924] extend SampleEntry(codingname){

[0925] unsigned int(8)

[32] compressor_name;

[0926] }

[0927] The compressor_name is a name used for informational purposes. It is formatted as a fixed 32-byte field with the first byte set to the number of bytes to be displayed, followed by the number of bytes of displayable data encoded using UTF-8, and then padded to complete a total of 32 bytes (including the size byte). This field can be set to 0.

[0928] The format of volumetric vision samples according to an embodiment may be defined by the encoding system.

[0929] The V-PCC unit header box according to an embodiment may be present in the V-PCC track included in the sample entry and all video-coded V-PCC component tracks included in the scheme information. The V-PCC unit header box may contain the following V-PCC unit header for data carried by each track.

[0930] aligned(8) class VPCCUnitHeaderBox

[0931] Extend FullBox('vunt',version=0,0){

[0932] vpcc_unit_header()unit_header;

[0933] }

[0934] That is, VPCCUnitHeaderBox may include vpcc_unit_header(). Figure 34 and Figure 48 An example of the syntax structure of (vpcc_unit_header()) is shown.

[0935] According to an embodiment, a V-PCC track sample entry may contain a VPCC configuration box (VPCCConfigurationBox).

[0936] According to an embodiment, the VPCCConfigurationBox may include a VPCC decoder configuration record (VPCCDecoderConfigurationRecord) as shown below.

[0937] aligned(8) class VPCCDDecoderConfigurationRecord{

[0938] unsigned int(8)configurationVersion=1;

[0939] unsigned int(3)sampleStreamSizeMinusOne;

[0940] unsigned int(5)numOfVPCCParameterSets;

[0941] for(i=0;i <numOfVPCCParameterSets;i++){

[0942] sample_stream_vpcc_unit VPCCParameterSet;

[0943] }

[0944] unsigned int(8)numOfAtlasSetupUnits;

[0945] for(i=0;i <numOfAtlasSetupUnits;i++){

[0946] sample_stream_vpcc_unit atlas_setupUnit;

[0947] }

[0948] }

[0949] The configurationVersion included in the VPCCDDecoderConfigurationRecord indicates the version field. Incompatible changes to the record are indicated by a change in the version number.

[0950] sampleStreamSizeMinusOne plus 1 may indicate the precision of the ssvu_vpcc_unit_size element in all sample stream V-PCC units in this configuration record or in V-PCC samples in the stream to which this configuration record applies.

[0951] numOfVPCCParameterSets specifies the number of V-PCC parameter sets (VPS) signaled in VPCCDecoderConfigurationRecord.

[0952] VPCCParameterSet is an instance of sample_stream_vpcc_unit() of a V-PCC unit of type VPCC_VPS. A V-PCC unit may include vpcc_parameter_set(). That is, a VPCCParameterSet array may include vpcc_parameter_set(). Figure 46 An example of the syntax structure of a sample stream V-PCC unit (sample_stream_vpcc_unit()) is shown.

[0953] numOfAtlasSetupUnits indicates the number of setup arrays for the atlas stream signaled in the VPCCDecoderConfigurationRecord.

[0954] Atlas_setupUnit is an instance of sample_stream_vpcc_unit() that contains an Atlas sequence parameter set, an Atlas frame parameter set, or a SEI Atlas NAL unit. Figure 46 An example of the syntax structure of a sample stream V-PCC unit (sample_stream_vpcc_unit()) is shown.

[0955] Specifically, the atlas_setupUnit array may include atlas parameter sets that are constant for the streams referred to by the sample entries in which the VPCCDecoderConfigurationRecord exists and the atlas stream SEI messages. According to an embodiment, atlas_setupUnit may be simply referred to as setup unit.

[0956] According to other implementations, VPCCDDecoderConfigurationRecord may be represented as follows.

[0957] aligned(8) class VPCCDDecoderConfigurationRecord{

[0958] unsigned int(8)configurationVersion=1;

[0959] unsigned int(3)sampleStreamSizeMinusOne;

[0960] bit(2)reserved=1;

[0961] unsigned int(3)lengthSizeMinusOne;

[0962] unsigned int(5)numOfVPCCParameterSets;

[0963] for(i=0;i <numOfVPCCParameterSets;i++){

[0964] sample_stream_vpcc_unit VPCCParameterSet;

[0965] }

[0966] unsigned int(8)numOfSetupUnitArrays;

[0967] for(j=0;j <numOfSetupUnitArrays;j++){

[0968] bit(1)array_completeness;

[0969] bit(1)reserved=0;

[0970] unsigned int(6)NAL_unit_type;

[0971] unsigned int(8)numNALUnits;

[0972] for(i=0;i <numNALUnits;i++){

[0973] sample_stream_nal_unit setupUnit;

[0974] }

[0975] }

[0976] configurationVersion is a version field. Incompatible changes to a record are indicated by a change in the version number.

[0977] The value of lengthSizeMinusOne plus 1 may indicate the precision (in bytes) of the ssnu_nal_unit_size element in all sample stream NAL units in the V-PCC samples in the stream to which the VPCCDecoderConfigurationRecord or the VPCCDecoderConfigurationRecord applies. Figure 53An example of the syntax structure of a sample stream NAL unit (sample_stream_nal_unit()) including the ssnu_nal_unit_size field is shown.

[0978] numOfVPCCParameterSets specifies the number of V-PCC parameter sets (VPS) signaled in VPCCDecoderConfigurationRecord.

[0979] VPCCParameterSet is a sample_stream_vpcc_unit() instance of a V-PCC unit of type VPCC_VPS. A V-PCC unit may include vpcc_parameter_set(). That is, a VPCCParameterSet array may include vpcc_parameter_set(). Figure 46 An example of the syntax structure of sample_stream_vpcc_unit() is shown.

[0980] numOfSetupUnitArrays indicates the number of atlas NAL unit arrays of the indicated type.

[0981] An iteration statement repeated as many times as the value of numOfSetupUnitArrays may include array_completeness.

[0982] array_completeness equal to 1 indicates that all atlas NAL units of the given type are in the following array and none are in the stream. array_completeness equal to 0 indicates that additional atlas NAL units of the specified type may be in the stream. The default and allowed values ​​are constrained by the sample entry name.

[0983] NAL_unit_type indicates the type of atlas NAL unit in the following array. NAL_unit_type is restricted to take one of the values ​​indicating NAL_ASPS, NAL_PREFIX_SEI, or NAL_SUFFIX_SEI atlas NAL units.

[0984] numNALUnits indicates the number of atlas NAL units of the indicated type contained in the VPCCDecoderConfigurationRecord for the stream to which the VPCCDecoderConfigurationRecord applies. The SEI array shall only contain SEI messages of a "declarative" nature, i.e., SEI messages that provide information about the entire stream. An example of such an SEI could be a user data SEI

[0985] setupUnit is an instance of sample_stream_nal_unit(), containing an atlas sequence parameter set, or an atlas frame parameter set, or a declarative SEI atlas NAL unit.

[0986] According to an embodiment, in the general layout of a multi-track container (also referred to as a multi-track ISOBMFF V-PCC container), V-PCC units in a V-PCC elementary stream can be mapped to individual tracks within the container file according to their type. There are two types of tracks in a multi-track ISOBMFF V-PCC container according to an embodiment: one is a V-PCC track, and the other is a V-PCC component track.

[0987] The V-PCC track according to an embodiment is a track that carries volumetric visual information in a V-PCC bitstream, including an atlas sub-bitstream and a sequence parameter set.

[0988] The V-PCC component track according to an embodiment is a restricted video scheme track that carries 2D video coding data for the occupancy map, geometry, and attribute sub-bitstreams of the V-PCC bitstream. In addition, the V-PCC component track may satisfy the following conditions:

[0989] a) In the sample entry, a new box is inserted, which records the role of the video stream contained in this track in the V-PCC system;

[0990] b) introducing a track reference from the V-PCC track to the V-PCC component track to establish the membership of the V-PCC component track in the specific point cloud represented by the V-PCC track;

[0991] c) Set the track header flag to 0 to indicate that this track does not contribute directly to the overall layout of the movie, but does help the V-PCC system.

[0992] Tracks belonging to the same V-PCC sequence can be time-aligned. Samples of V-PCC component tracks and V-PCC tracks across different video coding that contribute to the same point cloud frame have the same presentation time. The decoding time of the V-PCC atlas sequence parameter set and atlas frame parameter set for such samples is equal to or earlier than the composition time of the point cloud frame. In addition, all tracks belonging to the same V-PCC sequence have the same implicit or explicit edit lists.

[0993] NOTE: Synchronization between elementary streams in component tracks is handled by the ISOBMFF track timing structures (stts, ctts and cslg) or equivalent mechanisms in movie fragments.

[0994] Based on this layout, a V-PCC ISOBMFF vessel can include the following:

[0995] - A V-PCC track, which contains V-PCC parameter sets (in sample entries) and samples that carry the payload of V-PCC parameter set V-PCC units (unit type VPCC_VPS) and Atlas V-PCC units (unit type VPCC_AD). This track also includes track references to other tracks that carry the payload of video compressed V-PCC units (i.e., unit types VPCC_OVD, VPCC_GVD, and VPCC_AVD).

[0996] - A constrained video scheme track where the samples contain access units of a video coded elementary stream for occupancy map data (i.e. the payload of a V-PCC unit of type VPCC_OVD).

[0997] - one or more constrained video scheme tracks, where samples contain access units of a video coding elementary stream for geometry data (i.e. payloads of V-PCC units of type VPCC GVD).

[0998] - Zero or more constrained video scheme tracks, where samples contain access units of a video coding elementary stream for attribute data (i.e. payloads of V-PCC units of type VPCC_AVD).

[0999] Next, the V-PCC track will be described below.

[1000] The syntax structure of a V-PCC track sample entry according to an embodiment is configured as follows.

[1001] Sample entry types: "vpc1", "vpcg"

[1002] Container: SampleDescriptionBox

[1003] Mandatory: The "vpc1" or "vpcg" sample entry is mandatory.

[1004] Quantity: There can be one or more sample entries.

[1005] V-PCC tracks use VPCCSampleEntry, which extends VolumetricVisualSampleEntry. The sample entry type is either "vpc1" or "vpcg".

[1006] The VPCC sample entry contains a VPCCConfigurationBox. This box contains a VPCCDDecoderConfigurationRecord.

[1007] Under the "vpc1" sample entry, all Atlas sequence parameter sets, Atlas frame parameter sets, or V-PCC SEIs are in the setupUnit array.

[1008] Under the "vpcg" sample entry, Atlas sequence parameter set, Atlas frame parameter set, V-PCC SEI may be present in the array or in the stream.

[1009] An optional BitRateBox may be present in a VPCC volume sample entry to signal the bitrate information of the V-PCC track.

[1010] Volume Sequence:

[1011] class VPCCConfigurationBox extends Box('vpcC'){

[1012] VPCCDecoderConfigurationRecord()VPCCConfig;

[1013] }

[1014] aligned(8) class VPCCSampleEntry() extends VolumetricVisualSampleEntry('vpc1') {

[1015] VPCCConfigurationBoxconfig;

[1016] VPCCUnitHeaderBox unit_header;

[1017] }

[1018] Figure 55 An example of a V-PCC sample entry structure according to an embodiment is shown. Figure 55In it, the V-PCC sample entry includes a V-PCC parameter set and may optionally include an Atlas Sequence Parameter Set (ASPS), an Atlas Frame Parameter Set (AFPS), or SEI.

[1019] The V-PCC bitstream according to an embodiment may also contain a sample stream V-PCC header, a sample stream NAL header, and a V-PCC unit header box.

[1020] Next, the V-PCC track sample format is described below.

[1021] Each sample in the V-PCC track corresponds to a single point cloud frame. Samples corresponding to this frame in various component tracks have the same composition time as the V-PCC track sample. Each V-PCC sample includes one or more Atlas NAL units as follows.

[1022] aligned(8) class VPCCSample{

[1023] unsigned int PointCloudPictureLength = sample_size; / / The size of the sample (e.g., from the SampleSizeBox)

[1024] for(i = 0; i < PointCloudPictureLength;){

[1025] sample_stream_nal_unit nalUnit

[1026] i += (VPCCDecoderConfigurationRecord.lengthSizeMinusOne + 1) + nalUnit.ssnu_nal_unit_size;

[1027] }

[1028] }

[1029] aligned(8) class VPCCSample

[1030] {

[1031] unsigned int PictureLength = sample_size; / / The size of the sample (e.g., from the SampleSizeBox)

[1032] for(i = 0; i < PictureLength;) / / Signal until the end of the picture

[1033] {

[1034] unsigned int((VPCCDDecoderConfigurationRecord.LengthSizeMinusOne+1)*8)

[1035] NALUnitLength;

[1036] Bits(NALUnitLength*8)NALUnit;

[1037] i+=(VPCCDDecoderConfigurationRecord.LengthSizeMinusOne+1)+NALUnitLength;

[1038] }

[1039] }

[1040] A sync sample (random access point) in a V-PCC track is a V-PCC IRAP-coded patch data access unit. If necessary, the atlas parameter set can be repeated at the sync sample to allow random access.

[1041] Next, the V-PCC component track of video coding is described below.

[1042] The carriage of coded video tracks using MPEG-specified codecs can conform to ISOBMFF-derived specifications. For example, ISO / IEC 14496-15 can be used for the carriage of AVC and HEVC-coded video. ISOBMFF also provides extension mechanisms if other codec types are required.

[1043] Since it does not make sense to display decoded frames from attribute, geometry or occupancy map tracks without reconstructing the point cloud on the player side, constrained video scheme types can be defined for these video encoding tracks.

[1044] Next, the restricted video scheme is described below.

[1045] A V-PCC component video track may be represented as restricted video in a file and may be identified by 'pccv' in the scheme_type field of the SchemeTypeBox of the RestrictedSchemeInfoBox of the restricted video sample entry.

[1046] It should be noted that there is no restriction on the video codec used for encoding the attribute, geometry and occupancy map V-PCC components. In addition, different video codecs can be used to encode these components.

[1047] Scheme information (SchemeInformationBox) according to an embodiment may exist and include VPCCUnitHeaderBox.

[1048] Next, reference V-PCC component tracks are described below.

[1049] To link a V-PCC track to a component video track, three TrackReferenceTypeBoxes can be added to the TrackReferenceBox inside the V-PCC track's TrackBox, one for each component. The TrackReferenceTypeBox contains an array of track_IDs specifying the video tracks that the V-PCC track references. The reference_type of the TrackReferenceTypeBox identifies the type of component, such as occupancy map, geometry, or attribute. The track reference types are as follows.

[1050] In "pcco", the reference track contains the occupancy map V-PCC component of video coding.

[1051] In "pccg", the reference track contains the geometry V-PCC component of the video coding.

[1052] In "pcca", the referenced track contains the attribute V-PCC component of video coding.

[1053] The type of the V-PCC component carried by the referenced restricted video track and signaled in the track's RestrictedSchemeInfoBox shall match the reference type of the track reference from the V-PCC track.

[1054] Next, a single track container of a V-PCC bitstream is described below.

[1055] Single track encapsulation of V-PCC data requires that a V-PCC coded elementary bitstream be represented by a single track declaration.

[1056] In the case of a simple ISOBMFF encapsulation of a V-PCC coded bitstream, a single track encapsulation of PCC data can be used. Such a bitstream can be stored directly as a single track without further processing. The V-PCC unit header data structure can be preserved in the bitstream. The single track container of V-PCC data can be provided to the media workflow for further processing (e.g., multi-track file generation, transcoding, DASH segmentation, etc.).

[1057] ISOBMFF files containing single-track encapsulated V-PCC data can include "pcst" in the compatible_brands[] list of the FileTypeBox.

[1058] V-PCC basic stream track:

[1059] Sample entry types: "vpe1", "vpeg"

[1060] Container: SampleDescriptionBox

[1061] Mandatory: "vpe1" or "vpeg" sample entries are mandatory Quantity: one or more sample entries may be present

[1062] V-PCC elementary stream tracks use VolumetricVisualSampleEntry with sample entry type "vpe1" or "vpeg".

[1063] The VPCC elementary stream sample entry contains a VPCCConfigurationBox.

[1064] Under the 'vpe1' sample entry, all atlas sequence parameter sets, atlas frame parameter sets and SEI can be in the setupUnit array. Under the 'vpeg' sample entry, atlas sequence parameter sets, atlas frame parameter sets and SEI can be present in this array or stream.

[1065] Volume Sequence:

[1066] class VPCCConfigurationBox extends Box('vpcC'){

[1067] VPCCDecoderConfigurationRecord()VPCCConfig;

[1068] }

[1069] aligned(8) class VPCCElementaryStreamSampleEntry() extends VolumetricVisualSampleEntry('vpe1') {

[1070] VPCCConfigurationBoxconfig;

[1071] VPCCBoundingInformationBox 3d_bb;

[1072] }

[1073] Next, the V-PCC elementary stream sample format is described below.

[1074] A V-PCC elementary stream sample can include one or more V-PCC units belonging to the same presentation time. Each sample has a unique presentation time, size, and duration. Samples can be synchronized or rely on the decoding mode of other V-PCC elementary stream samples.

[1075] Next, a V-PCC elementary stream synchronization sample is described below.

[1076] V-PCC elementary stream synchronization samples can meet all of the following conditions:

[1077] - It is independently decodable;

[1078] - in decoding order, samples following the sync sample do not have any decoding dependencies on any samples preceding the sync sample; and

[1079] - All samples that appear after the sync sample in decoding order are successfully decodable.

[1080] Next, the V-PCC basic stream subsample is described below.

[1081] A V-PCC base stream subsample is a V-PCC unit contained in a V-PCC base stream sample.

[1082] The V-PCC elementary stream track contains a SubSampleInformationBox in the SampleTableBox or in the TrackFragmentBox of each MovieFragmentBox, which lists the V-PCC elementary stream subsamples.

[1083] The 32-bit unit header representing the V-PCC unit of the subsample can be copied to the 32-bit codec_specific_parameters field of the subsample entry in the SubSampleInformationBox. The V-PCC unit type of each subsample is identified by parsing the codec_specific_parameters field of the subsample entry in the SubSampleInformationBox.

[1084] Next, the rendering of point cloud data is described below.

[1085] The rendering of point cloud data according to an embodiment is performed by Figure 1 Renderer 10009, Figure 19 Point Cloud Renderer 19007, Figure 20 Renderer 2001 or Figure 22 The point cloud renderer 22004 executes. Depending on the embodiment, the point cloud data can be rendered in 3D space based on the metadata. The user can view all or part of the rendered area through a VR / AR display or a regular display. In particular, the point cloud data can be rendered based on the user's viewport, etc.

[1086] Here, the viewport or viewport area can refer to the area that the user is viewing in the point cloud video. The viewpoint is the point that the user is viewing in the point cloud video and can represent the center point of the viewport area. In other words, the viewport is the area centered on the viewpoint, and the size and shape of this area can be determined by the field of view (FOV).

[1087] The viewport information according to an embodiment may be information about the area that the user is currently viewing in 3D space through the device or HMD. The viewport information may be used to perform gaze analysis to examine how the user consumes the point cloud video, the area of ​​the point cloud video that the user is viewing, and the time the user is viewing the area. The gaze analysis may be performed on the receiving side, and the analysis results may be transmitted to the sending side on a feedback channel. Display devices such as VR / AR / MR may extract the viewport area based on orientation information, the position / orientation of the user's head, the vertical or horizontal FOV supported by the device, and the like. The direction or viewport information analyzed by the receiving device may be sent to the sending device on a feedback channel.

[1088] The point cloud video decoder of the receiving device according to the embodiment can efficiently extract or decode only the media data of a specific area (i.e., the area indicated by the orientation information and / or viewport information) from the file based on the orientation information and / or viewport information indicating the area currently being viewed by the user. The point cloud video encoder of the sending device according to the embodiment can encode only the media data of the specific area (i.e., the area indicated by the orientation information and / or viewport information) based on the orientation information and / or viewport information fed back from the receiving device, or can encapsulate the encoded media data into a file for transmission.

[1089] According to an embodiment, the file / fragment encapsulator of the sending device may encapsulate all point cloud data into files / fragments based on orientation information and / or viewport information, or may encapsulate point cloud data indicated by orientation information and / or viewport information into files / fragments.

[1090] According to an embodiment, the file / fragment decapsulator of the receiving device may decapsulate a file containing all point cloud data or a file containing point cloud data indicated by the orientation information and / or viewport information based on the orientation information and / or viewport information.

[1091] According to an embodiment, the viewport information may be used in the same or similar meaning as ViewInfoStruct information or view information.

[1092] According to an embodiment, the viewport information may also be referred to as viewport-related information or viewport-related metadata. According to an embodiment, the viewport-related information may include at least one of viewport information, rendering parameters, and object rendering information.

[1093] According to an embodiment, the viewport related information can be obtained by Figure 18 The metadata encoder 18005 of the point cloud data sending device or Figure 21 The point cloud preprocessor 20001 and / or video / image encoders 21007 and 21008 of the V-PCC system generate / encode and can be Figure 19 metadata decoder 19002 of the point cloud data receiving device or Figure 22 The video / image decoders 22001 and 22002 and / or the point cloud processor 22003 of the V-PCC system acquire / decode.

[1094] In the present disclosure, embodiments of defining metadata related to viewport (or view) information of point cloud data and storing and signaling the metadata related to the viewport information in a file are described below.

[1095] In the present disclosure, an embodiment of storing metadata related to viewport information of point cloud data that dynamically changes over time in a file is described below.

[1096] In the present disclosure, embodiments of defining metadata related to rendering parameters of point cloud data and storing and signaling the metadata related to the rendering parameters in a file are described below.

[1097] In the present disclosure, an embodiment of storing metadata related to rendering parameters of point cloud data that dynamically changes according to time in a file is described below.

[1098] In the present disclosure, embodiments of defining metadata related to object rendering information related to point cloud data in a file, and storing and signaling the metadata related to the object rendering information in the file are described below.

[1099] In the present disclosure, an embodiment of storing metadata related to object rendering information of point cloud data that dynamically changes according to time in a file is described below.

[1100] Figure 56 An example of generating a view using a virtual camera and ViewInfoStruct information according to an embodiment is shown.

[1101] The ViewInfoStruct information according to an embodiment may include detailed information for creating a view to be rendered and provided. Specifically, ViewInfoStruct may include position information about the virtual camera in the 3D space used to create the view, the vertical / horizontal field of view (FOV) of the virtual camera, a direction vector indicating the direction the virtual camera is facing, and an up vector indicating the upward direction of the virtual camera. The virtual camera can be similar to the user's eyes, that is, the user's view of a portion of the 3D space. The view cone can be inferred based on the above information. The view cone refers to an area in the 3D space that includes all or part of the point cloud data that is actually rendered and displayed. According to an embodiment, when the inferred view cone is projected in the form of a 2D frame, a view (that is, a 2D image / video frame to be actually displayed) can be created.

[1102] The following is a syntax showing an example of information included in ViewInfoStruct.

[1103] aligned(8) class ViewInfoStruct(){

[1104] unsigned int(16)view_pos_x;

[1105] unsigned int(16)view_pos_y;

[1106] unsigned int(16)view_pos_z;

[1107] unsigned int(8)view_vfov;

[1108] unsigned int(8)view_hfov;

[1109] unsigned int(16)view_dir_x;

[1110] unsigned int(16)view_dir_y;

[1111] unsigned int(16)view_dir_z;

[1112] unsigned int(16)view_up_x;

[1113] unsigned int(16)view_up_y;

[1114] unsigned int(16)view_up_z;

[1115] }

[1116] view_pos_x, view_pos_y, and view_pos_z may indicate x, y, and z coordinate values ​​in a 3D space of a virtual camera capable of creating a view (eg, a 2D image / video frame to be actually displayed).

[1117] view_vfov and view_hfov may indicate the vertical field of view (FOV) and horizontal field of view (FOV) of the virtual camera that is capable of creating the view.

[1118] view_dir_x, view_dir_y, and view_dir_z may indicate x, y, and z coordinate values ​​in a 3D space for indicating a direction vector representing a direction viewed by the virtual camera.

[1119] view_up_x, view_up_y, and view_up_z may indicate x, y, and z coordinate values ​​in a three-dimensional space to represent an up vector indicating an up direction of the virtual camera.

[1120] The ViewInfoStruct() information according to the embodiment may be transmitted in the form of SEI or the like in a point cloud bitstream as follows.

[1121] V-PCC View Information Box

[1122] aligned(8) class VPCCViewInfoBox extends FullBox('vpvi',0,0) {

[1123] ViewInfoStruct();

[1124] }

[1125] Next, the static V-PCC viewport information is described below.

[1126] When the viewport information according to an embodiment does not change within a point cloud sequence, the VPCCViewInfoBox may be included in a sample entry of a V-PCC track or a sample entry of a V-PCC elementary stream track, as shown below.

[1127] aligned(8) class VPCCSampleEntry() extends VolumetricVisualSampleEntry('vpc1') {

[1128] VPCCConfigurationBoxconfig;

[1129] VPCCUnitHeaderBox unit_header;

[1130] VPCCViewInfoBox view_info;

[1131] }

[1132] The VPCCViewInfoBox can contain detailed information for creating a view in which point cloud data associated with atlas frames stored in samples within a track are rendered and served, as shown below.

[1133] aligned(8) class VPCCElementaryStreamSampleEntry() extends VolumetricVisualSampleEntry('vpe1') {

[1134] VPCCConfigurationBoxconfig;

[1135] VPCCViewInfoBox view_info;

[1136] }

[1137] A VPCCViewInfoBox can contain detailed information for creating a view in which to render and serve point cloud data associated with a video frame or atlas frame in subsamples stored in a track.

[1138] Next, V-PCC view information sample grouping is described below.

[1139] According to an embodiment, 'vpvs' grouping_type for sample grouping indicates that samples in a V-PCC track are assigned to view information carried in the sample group. When there is a SampleToGroupBox with grouping_type equal to 'vpvs', an accompanying SampleGroupDescriptionBox with the same grouping type is present and contains the ID to which the group of samples belongs.

[1140] aligned(8) class VPCCViewInfoSampleGroupDescriptionEntry() extends SampleGroupDescriptionEntry('vpvs'){

[1141] ViewInfoStruct();

[1142] }

[1143] Next, dynamic V-PCC view information is described below.

[1144] If a V-PCC track has an associated timed metadata track whose sample entry type is "dyvi", then the view information defined for the point cloud stream carried by the V-PCC track is considered dynamic. That is, the view information can change dynamically over time.

[1145] The associated timed metadata track contains a "cdsc" track reference to the V-PCC track carrying the Atlas stream.

[1146] If a V-PCC elementary stream track has an associated timed metadata track whose sample entry type is "dyvi", then the view information defined for the point cloud stream carried by the V-PCC elementary stream track is considered dynamic. That is, the view information can change dynamically over time.

[1147] The associated timed metadata track contains a "cdsc" track reference to the V-PCC elementary stream track.

[1148] aligned(8) class DynamicViewInfoSampleEntry extends MetaDataSampleEntry('dyvi') {

[1149] VPCCViewInfoBox init_view_info;

[1150] }

[1151] init_view_info may include view information (viewinformation()) for generating an initial view of point cloud data.

[1152] The sample syntax for this sample entry type "dyvi" may be specified as follows:

[1153] aligned(8)DynamicViewInfoSample(){

[1154] VPCCViewInfoBox view_info;

[1155] }

[1156] Each sample can contain view information that changes over time ().

[1157] According to an embodiment, a RenderingParamStruct, which is a rendering parameter used for rendering, may include parameter information that can be applied when rendering point cloud data. This parameter may include point size, point rendering type, how to handle duplicate points (for example, whether to display duplicate points in the rendering of point cloud data), etc., as shown below.

[1158] aligned(8) class RenderingParamStruct(){

[1159] unsigned int(16)point_size;

[1160] unsigned int(7)point_type;

[1161] unsigned int(2)duplicated_point;

[1162] }

[1163] point_size may indicate the size of the point to be rendered / displayed.

[1164] point_type can indicate the type of point to be rendered / displayed. For example, point_type equal to 0 can indicate a cuboid. point_type equal to 1 can indicate a circle. point_type equal to 2 can indicate a point.

[1165] duplicated_point may indicate how to handle multiple points in 3D space with the same x, y, and z values ​​when they exist. For example, when duplicated_point is equal to 0, duplicate points may be ignored. In an embodiment, when duplicated_point is equal to 1, only one point may be randomly selected and rendered / displayed. In an embodiment, when duplicated_point is equal to 2, the attribute values ​​of duplicate points may be averaged and rendered / displayed based on the average attribute value.

[1166] The RenderingParamStruct() information may be carried in the point cloud bitstream in the form of SEI or the like as follows.

[1167] V-PCC Rendering Parameter Box

[1168] aligned(8) class VPCCRenderingParamBox extends FullBox('vprp',0,0) {

[1169] RenderingParamStruct();

[1170] }

[1171] Next, static V-PCC rendering parameters are described below.

[1172] According to an embodiment, when rendering-related parameters do not change within a point cloud sequence, VPCCRenderingParamBox may be included in a sample entry of a V-PCC track or a sample entry of a V-PCC elementary stream track, as shown below.

[1173] aligned(8) class VPCCSampleEntry() extends VolumetricVisualSampleEntry('vpc1') {

[1174] VPCCConfigurationBoxconfig;

[1175] VPCCUnitHeaderBox unit_header;

[1176] VPCCRenderingParamBox rendering_param;

[1177] }

[1178] A VPCCRenderingParamBox may contain details about rendering parameters that may be applied when rendering point cloud data associated with an atlas frame in samples stored in a track, as described below.

[1179] aligned(8) class VPCCElementaryStreamSampleEntry() extends VolumetricVisualSampleEntry('vpe1') {

[1180] VPCCConfigurationBoxconfig;

[1181] VPCCRenderingParamBox rendering_param;

[1182] }

[1183] The VPCCRenderingParamBox may contain detailed information about rendering parameters that may be applied when rendering atlas frames, video frames, and point cloud data in subsamples stored in a track.

[1184] Next, V-PCC rendering parameter sample grouping is described below.

[1185] According to an embodiment, 'vprp' grouping_type for sample grouping indicates that samples in a V-PCC track are assigned to rendering parameters carried in the sample group.When a SampleToGroupBox with grouping_type equal to 'vpvs' is present, an accompanying SampleGroupDescriptionBox with the same grouping type is present and may contain the ID of the group of samples.

[1186] aligned(8) class VPCCRenderingParamSampleGroupDescriptionEntry() extends SampleGroupDescriptionEntry('vprp'){

[1187] RenderingParamStruct();

[1188] }

[1189] Next, dynamic V-PCC rendering parameter information is described below.

[1190] If a V-PCC track has an associated timed metadata track whose sample entry type is "dyrp", then the rendering parameters defined for the point cloud stream carried by the V-PCC track are considered dynamic rendering parameters. That is, the rendering parameters can change dynamically over time.

[1191] The associated timed metadata track contains a "cdsc" track reference to the V-PCC track carrying the Atlas stream.

[1192] If a V-PCC elementary stream track has an associated timed metadata track whose sample entry type is "dyrp", then the rendering parameters defined for the point cloud stream carried by the V-PCC elementary stream are considered dynamic rendering parameters. That is, the rendering parameters can change dynamically over time.

[1193] The associated timed metadata track contains a "cdsc" track reference to the V-PCC elementary stream track.

[1194] aligned(8) class DynamicRenderingParamSampleEntry extends MetaDataSampleEntry('dyrp') {

[1195] VPCCRenderingParamBox init_rendering_param;

[1196] }

[1197] init_rendering_param may contain the initial rendering parameters for the point cloud data.

[1198] A sample syntax for a sample entry type 'dyvi' according to an embodiment may be specified as follows:

[1199] aligned(8)DynamicRenderingParamSample(){

[1200] VPCCRenderingParamBox rendering_param;

[1201] }

[1202] Each sample may include rendering parameters that vary over time.

[1203] Figure 57 An example of creating a view using ObjectRenderingInfoStruct() information according to an embodiment is shown.

[1204] According to an embodiment, ObjectRenderingInfoStruct, which is object rendering information, may include detailed information applied to point cloud data when rendering the point cloud data. According to an embodiment, ObjectRenderingInfoStruct may include information such as coordinate system information used by the point cloud data, the position of the point cloud data in a 3D space based on the coordinate system, a normal vector of the point cloud data, and an up vector that may indicate the upward direction of the point cloud data.

[1205] aligned(8) classObjectRenderingInfoStruct(){

[1206] unsigned int(8)obj_id;

[1207] unsinged int(4)obj_coord_type;

[1208] bit(4)reserved=1;

[1209] unsigned int(16)obj_pos_x;

[1210] unsigned int(16)obj_pos_y;

[1211] unsigned int(16)obj_pos_z;

[1212] unsigned int(16)obj_dir_x;

[1213] unsigned int(16)obj_dir_y;

[1214] unsigned int(16)obj_dir_z;

[1215] unsigned int(16)obj_up_x;

[1216] unsigned int(16)obj_up_y;

[1217] unsigned int(16)obj_up_z;

[1218] unsigned int(16)obj_scale_x;

[1219] unsigned int(16)obj_scale_y;

[1220] unsigned int(16)obj_scale_z;

[1221] }

[1222] obj_id may indicate an identifier of the point cloud data.

[1223] obj_coord_type can indicate the coordinate system used to render the point cloud. For example, when the value of obj_coord_type is 0, global coordinates can be used. For example, when the value of obj_coord_type is 1, view coordinates (i.e., the coordinate system of the virtual camera that can be used to generate viewport data) can be used.

[1224] obj_pos_x, obj_pos_y, and obj_pos_z may indicate the x, y, and z coordinate values ​​of the origin of the bounding box of the point cloud data in the coordinate system indicated by obj_coord_type. Depending on the embodiment, the x, y, and z coordinate values ​​of the center point of the bounding box of the point cloud data may be indicated in the coordinate system indicated by obj_coord_type.

[1225] obj_dir_x, obj_dir_y, and obj_dir_z may indicate x, y, and z coordinate values ​​in a 3D space, used to represent a direction vector (or normal vector) indicating a direction in which the point cloud data faces forward.

[1226] obj_up_x, obj_up_y, and obj_up_z may indicate x, y, and z coordinate values ​​in a 3D space, for representing an up vector indicating an upward direction of the point cloud data.

[1227] obj_scale_x, obj_scale_y, and obj_scale_z may indicate the scaling factors to be applied to the x, y, and z axes compared to the original source of the point cloud data in the rendering / display.

[1228] According to an embodiment, the ObjectRenderingInfoStruct() information may be carried in the form of SEI or the like in the point cloud bitstream.

[1229] The V-PCC object rendering information box according to an embodiment may include ObjectRenderingInfoStruct() information.

[1230] aligned(8) class VPCCObjectRenderingInfoBox extends FullBox('vpoi',0,0) {

[1231] ObjectRenderingInfoStruct();

[1232] }

[1233] Next, static V-PCC object rendering information is described below.

[1234] According to an embodiment, when object rendering information applied to point cloud data does not change within a point cloud sequence during point cloud data rendering, VPCCObjectRenderingInfoBox may be included in a sample entry of a V-PCC track or a sample entry of a V-PCC elementary stream track.

[1235] aligned(8) class VPCCSampleEntry() extends VolumetricVisualSampleEntry('vpc1') {

[1236] VPCCConfigurationBoxconfig;

[1237] VPCCUnitHeaderBox unit_header;

[1238] VPCCObjectRenderingInfoBox obj_rendering_info; [1...

Claims

1. A device for point cloud data, the device for point cloud data comprising: a decapsulator configured to decapsulate a file into a bitstream comprising the point cloud data, wherein the point cloud data comprises geometry data, attribute data, and occupancy map data, wherein the file comprises a plurality of tracks, wherein each of the plurality of tracks comprises a sample entry and a sample, wherein a sample in at least one of the plurality of tracks comprises one of the geometry data, the attribute data, and the occupancy map data, wherein the file further comprises signaling information, and wherein the signaling information comprises parameter set information and viewport related information for the point cloud data; and a decoder configured to decode the point cloud data, The viewport related information is classified into dynamic viewport information that changes with time and static viewport information that does not change with time. wherein a track of the plurality of tracks is a timed metadata track, wherein the sample entry type of the timed metadata track has a value for indicating that the timed metadata track is used for the dynamic viewport information, The dynamic viewport information at least includes first viewport information or second viewport information, the first viewport information includes initial viewport position information, and the second viewport information includes viewport position information that changes dynamically over time. wherein the first viewport information is carried by a sample entry of the timed metadata track, wherein the second viewport information is carried by samples of the timed metadata track, and The static viewport information is carried in a track of the plurality of tracks that is different from the timed metadata track.

2. The device according to claim 1, in, The viewport-related information also includes camera direction information.

3. The device according to claim 1, in, The viewport related information also includes horizontal field of view FOV information and vertical FOV information used to generate the viewport.

4. The device according to claim 1, in, The point cloud data is decoded by a video-based point cloud compression (V-PCC) scheme.

5. A device for point cloud data, the device for point cloud data comprising: an encoder configured to encode the point cloud data; as well as a packager configured to package a bitstream including the encoded point cloud data into a file, The encoded point cloud data includes encoded geometric data, encoded attribute data and encoded occupancy map data, Wherein, the file includes a plurality of tracks, wherein each of the plurality of tracks comprises a sample entry and a sample, wherein samples in at least one track of the plurality of tracks include one of the encoded geometry data, the encoded attribute data, and the encoded occupancy map data, The file also includes signaling information. The signaling information includes parameter set information and viewport related information for the point cloud data. The viewport related information is classified into dynamic viewport information that changes with time and static viewport information that does not change with time. wherein a track of the plurality of tracks is a timed metadata track, wherein the sample entry type of the timed metadata track has a value for indicating that the timed metadata track is used for the dynamic viewport information, The dynamic viewport information at least includes first viewport information or second viewport information, the first viewport information includes initial viewport position information, and the second viewport information includes viewport position information that changes dynamically over time. wherein the first viewport information is carried by a sample entry of the timed metadata track, wherein the second viewport information is carried by samples of the timed metadata track, and The static viewport information is carried in a track of the plurality of tracks that is different from the timed metadata track.

6. The device according to claim 5, in, The point cloud data is encoded by a video-based point cloud compression (V-PCC) scheme.

7. The device according to claim 5, in, The viewport-related information also includes camera direction information.

8. The device according to claim 5, in, The viewport related information also includes horizontal field of view FOV information and vertical FOV information used to generate the viewport.

9. A method for point cloud data, the method for point cloud data comprising the following steps: Encoding the point cloud data; Packing a bit stream including the encoded point cloud data into a file; as well as Send said file, The encoded point cloud data includes encoded geometric data, encoded attribute data and encoded occupancy map data, Wherein, the file includes a plurality of tracks, wherein each of the plurality of tracks comprises a sample entry and a sample, wherein samples in at least one track of the plurality of tracks include one of the encoded geometry data, the encoded attribute data, and the encoded occupancy map data, The file also includes signaling information. The signaling information includes parameter set information and viewport related information for the point cloud data. The viewport related information is classified into dynamic viewport information that changes with time and static viewport information that does not change with time. wherein a track of the plurality of tracks is a timed metadata track, wherein the sample entry type of the timed metadata track has a value for indicating that the timed metadata track is used for the dynamic viewport information, The dynamic viewport information at least includes first viewport information or second viewport information, the first viewport information includes initial viewport position information, and the second viewport information includes viewport position information that changes dynamically over time. wherein the first viewport information is carried by a sample entry of the timed metadata track, and wherein the second viewport information is carried by samples of the timed metadata track, and The static viewport information is carried in a track of the plurality of tracks that is different from the timed metadata track.

10. The method according to claim 9, in, The point cloud data is encoded using a video-based point cloud compression (V-PCC) scheme.

11. The method according to claim 9, in, The viewport-related information also includes camera direction information.

12. The method according to claim 9, in, The viewport related information also includes horizontal field of view FOV information and vertical FOV information used to generate the viewport.

13. A computer-readable storage medium storing a file generated by the method according to claim 9.

Images