Point cloud data sending device, point cloud data sending method, point cloud data receiving device and point cloud data receiving method

Abstract

According to an embodiment, a point cloud data sending method includes the following steps: encoding point cloud data; encapsulating a bit stream including the encoded point cloud data into a file; and sending the file, wherein the bit stream is stored in one or more tracks of the file, the file includes signaling data, and the point cloud data consists of one or more objects; wherein the signaling data may include at least one parameter set and information for partial access to the point cloud data.

Description

Technical Field

[0001] Embodiments provide a method of providing point cloud content to provide users with various services such as virtual reality (VR), augmented reality (AR), mixed reality (MR), and autonomous driving services. Background Art

[0002] A point cloud is a collection of points in a three-dimensional (3D) space. Generating point cloud data is difficult because the number of points in a 3D space is large.

[0003] High throughput is required to send and receive point cloud data. Summary of the Invention

[0004] Technical issues

[0005] An object of the present disclosure is to provide a point cloud data transmitting device, a point cloud data transmitting method, a point cloud data receiving device, and a point cloud data receiving method for efficiently transmitting and receiving point clouds.

[0006] Another object of the present disclosure is to provide a point cloud data sending device, a point cloud data sending method, a point cloud data receiving device and a point cloud data receiving method for solving delay and encoding / decoding complexity.

[0007] Another object of the present disclosure is to provide a point cloud data transmitting device, a point cloud data transmitting method, a point cloud data receiving device, and a point cloud data receiving method for providing optimized point cloud content to a user by signaling information about one or more atlases.

[0008] Another object of the present disclosure is to provide a point cloud data transmitting device, a point cloud data transmitting method, a point cloud data receiving device, and a point cloud data receiving method for providing optimized point cloud content to a user by signaling spatial region information and information related to objects and atlas.

[0009] Another object of the present disclosure is to provide a point cloud data transmitting device, a point cloud data transmitting method, a point cloud data receiving device, and a point cloud data receiving method for providing optimized point cloud content to a user by signaling spatial region information and information related to objects and atlas at a file format level.

[0010] Another object of the present disclosure is to provide a point cloud data sending device, a point cloud data sending method, a point cloud data receiving device, and a point cloud data receiving method for signaling spatial area information that is static or changes over time in a track or project and information related to objects and atlas to efficiently provide point cloud data to users.

[0011] Additional advantages, objects, and features of the present disclosure will be set forth in part in the following description and will become apparent to those skilled in the art upon examination of the following or may be learned from practice of the present disclosure. The objects and other advantages of the present disclosure may be realized and obtained by the structures particularly pointed out in the written description and claims and the accompanying drawings.

[0012] Technical Solution

[0013] To achieve these objects and other advantages and in accordance with the purposes of the present disclosure, as implemented and broadly described herein, a method for sending point cloud data may include the steps of: encoding the point cloud data; encapsulating a bit stream including the encoded point cloud data into a file; and sending the file, the bit stream being stored in one or more tracks of the file, the file also including signaling data, the point cloud data consisting of one or more objects, and the signaling data including at least one parameter set and information for partial access to the point cloud data.

[0014] According to an embodiment, the point cloud data includes geometry data, attribute data and occupancy map data encoded by a video-based encoding scheme.

[0015] According to an embodiment, the information used for partial access is at least one of static information that does not change with time or dynamic information that dynamically changes with time.

[0016] According to an embodiment, the one or more objects are associated with the one or more atlas patches, the one or more atlas patches constitute an atlas frame, and the information for partial access includes mapping information between the one or more objects and the one or more atlas patches.

[0017] According to an embodiment, the mapping information includes object index information for identifying each of the one or more objects, quantity information for identifying the number of atlas tiles associated with the object indicated by the object index information, and atlas identification information for identifying each of the atlas tiles associated with the object.

[0018] According to an embodiment, a point cloud data sending device may include: an encoder for encoding point cloud data; an encapsulator for encapsulating a bit stream including encoded point cloud data into a file; and a transmitter for sending the file, the bit stream being stored in one or more tracks of the file, the file also including signaling data, the point cloud data consisting of one or more objects; and the signaling data including at least one parameter set and information for partial access to the point cloud data.

[0019] According to an embodiment, the point cloud data includes geometry data, attribute data and occupancy map data encoded by a video-based encoding scheme.

[0020] According to an embodiment, the information used for partial access is at least one of static information that does not change with time or dynamic information that dynamically changes with time.

[0021] According to an embodiment, the one or more objects are associated with the one or more atlas patches, the one or more atlas patches constitute an atlas frame, and the information for partial access includes mapping information between the one or more objects and the one or more atlas patches.

[0022] According to an embodiment, the mapping information includes object index information for identifying each of the one or more objects, quantity information for identifying the number of atlas tiles associated with the object indicated by the object index information, and atlas identification information for identifying each of the atlas tiles associated with the object.

[0023] According to an embodiment, a method for receiving point cloud data may include the following steps: receiving a file; decapsulating the file into a bit stream including point cloud data, the bit stream being stored in one or more tracks of the file, and the file also including signaling data; decoding all or a portion of the point cloud data based on the signaling data; and rendering all or a portion of the decoded point cloud data based on the signaling data, the point cloud data consisting of one or more objects, and the signaling data including at least one parameter set and information for partial access to the point cloud data.

[0024] According to an embodiment, the point cloud data includes geometry data, attribute data, and occupancy map data decoded by a video-based encoding scheme.

[0025] According to an embodiment, the information used for partial access is at least one of static information that does not change with time or dynamic information that dynamically changes with time.

[0026] According to an embodiment, the one or more objects are associated with the one or more atlas patches, the one or more atlas patches constitute an atlas frame, and the information for partial access includes mapping information between the one or more objects and the one or more atlas patches.

[0027] According to an embodiment, the mapping information includes object index information for identifying each of the one or more objects, quantity information for identifying the number of atlas tiles associated with the object indicated by the object index information, and atlas identification information for identifying each of the atlas tiles associated with the object.

[0028] According to an embodiment, a point cloud data receiving device may include: a receiver for receiving a file; a decapsulator for decapsulating the file into a bit stream including point cloud data, the bit stream being stored in one or more tracks of the file, and the file also including signaling data; a decoder for decoding all or part of the point cloud data based on the signaling data; and a renderer for rendering all or part of the decoded point cloud data based on the signaling data, the point cloud data consisting of one or more objects; and the signaling data includes at least one parameter set and information for partial access to the point cloud data.

[0029] According to an embodiment, the point cloud data includes geometry data, attribute data, and occupancy map data decoded by a video-based encoding scheme.

[0030] According to an embodiment, the information used for partial access is at least one of static information that does not change with time or dynamic information that dynamically changes with time.

[0031] According to an embodiment, the one or more objects are associated with the one or more atlas patches, the one or more atlas patches constitute an atlas frame, and the information for partial access includes mapping information between the one or more objects and the one or more atlas patches.

[0032] According to an embodiment, the mapping information includes object index information for identifying each of the one or more objects, quantity information for identifying the number of atlas tiles associated with the object indicated by the object index information, and atlas identification information for identifying each of the atlas tiles associated with the object.

[0033] Beneficial effects

[0034] The point cloud data transmitting method, transmitting device, point cloud data receiving method, and receiving device according to the embodiments can provide a point cloud service of good quality.

[0035] According to the embodiment, the point cloud data sending method, sending device, point cloud data receiving method and receiving device can implement various video encoding and decoding schemes.

[0036] The point cloud data transmitting method, transmitting device, point cloud data receiving method, and receiving device according to the embodiments can provide general point cloud content such as self-driving services.

[0037] The point cloud data transmitting method, transmitting device, point cloud data receiving method, and receiving device according to the embodiment can provide optimal point cloud content service by configuring V-PCC bitstream and allowing files to be transmitted, received, and stored.

[0038] By using the point cloud data transmission method, transmission device, point cloud data reception method, and reception device according to the embodiment, V-PCC bitstreams can be efficiently accessed by multiplexing V-PCC bitstreams on a V-PCC unit basis. In addition, the atlas bitstream (or atlas substream) of the V-PCC bitstream can be efficiently stored in a track in a file and thus transmitted and received.

[0039] Using the point cloud data transmission method, transmission device, point cloud data reception method, and reception device according to the embodiments, a V-PCC bitstream can be divided and stored in one or more tracks in a file, and information indicating the relationship between multiple tracks stored in the V-PCC bitstream can be signaled. Therefore, point cloud bitstream files can be efficiently stored and transmitted.

[0040] By using the point cloud data transmission method, point cloud data transmission device, point cloud data receiving method, and point cloud data receiving device according to the embodiment, metadata used for data processing and rendering in the V-PCC bitstream can be transmitted and received in the V-PCC bitstream. Therefore, it is possible to provide optimal point cloud content services.

[0041] Using the point cloud data transmission method, point cloud data transmission device, point cloud data receiving method, and point cloud data receiving device according to the embodiments, it is possible to store and deliver atlas parameter sets in a track or project of a file for decoding and rendering atlas substreams in a V-PCC bitstream. Consequently, a V-PCC decoder / player can operate efficiently when decoding V-PCC bitstreams and atlas substreams or parsing and processing bitstreams in a track / project. Furthermore, even when an atlas subbitstream is divided into one or more tracks and stored, necessary atlas data and related video data can be efficiently selected, extracted, and decoded.

[0042] Using the point cloud data transmission method, point cloud data transmission device, point cloud data receiving method, and point cloud data receiving device according to the embodiment, point cloud data can be divided into multiple spatial regions for processing for partial access and / or spatial access to point cloud content. Therefore, encoding and transmission operations on the transmitting side and decoding and rendering operations on the receiving side can be performed in real time and can be processed with low latency.

[0043] The point cloud data transmission method, point cloud data transmission device, point cloud data reception method, and point cloud data reception device according to the embodiments can provide spatial region information about spatial regions segmented from point cloud content. Therefore, point cloud content can be accessed in various ways, taking into account the player or user environment on the receiving side.

[0044] By using the point cloud data transmission method, point cloud data transmission device, point cloud data reception method, and point cloud data reception device according to the embodiment, spatial area information for data processing and rendering in the V-PCC bitstream can be transmitted and received at the file format level through tracks. Therefore, it is possible to provide optimal point cloud content services.

[0045] Using the point cloud data transmission method, point cloud data transmission device, point cloud data receiving method, and point cloud data receiving device according to the embodiments, the association between one or more objects divided from point cloud content and one or more atlas tiles of an atlas frame can be delivered in a sample, a sample group, a sample entry, a track group, an entity group within a track, or a timed metadata track. This enables object-based access and access to atlas tiles associated with an object or associated point cloud data.

[0046] Using the point cloud data transmission method, point cloud data transmission device, point cloud data receiving method, and point cloud data receiving device according to the embodiments, the association between the object and the atlas can be delivered in samples, sample groups, sample entries, track groups, entity groups within a track, or in timed metadata tracks, depending on the degree of association between one or more objects divided from point cloud content and one or more atlases. Therefore, atlases associated with the objects can be efficiently extracted.

[0047] Using the point cloud data transmission method, point cloud data transmission device, point cloud data receiving method, and point cloud data receiving device according to the embodiments, SEI messages used for processing and rendering data in a V-PCC bitstream can be stored and delivered to a track or project in a file. Therefore, a PCC decoder / player can operate efficiently when decoding a V-PCC bitstream or parsing and processing the corresponding bitstream in a track / project. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The accompanying drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this application. The accompanying drawings illustrate embodiments of the present disclosure and together with the description serve to explain the principles of the present disclosure. In the drawings:

[0049] Figure 1 An exemplary structure of a transmission / reception system for providing point cloud content according to an embodiment is shown.

[0050] Figure 2 Shown is the capture of point cloud data according to an embodiment.

[0051] Figure 3 Shown are exemplary point clouds, geometry, and texture images according to embodiments.

[0052] Figure 4 An exemplary V-PCC encoding process according to an embodiment is shown.

[0053] Figure 5 Shown are examples of tangent planes and normal vectors of surfaces according to embodiments.

[0054] Figure 6 An exemplary bounding box of a point cloud is shown, according to an embodiment.

[0055] Figure 7 An example of determination of the positions of individual patches on an occupancy map according to an embodiment is shown.

[0056] Figure 8 Exemplary relationships between the normal axis, the tangential axis, and the dual tangential axes according to embodiments are shown.

[0057] Figure 9 Exemplary configurations of the minimum mode and the maximum mode of the projection mode according to the embodiment are shown.

[0058] Figure 10 An exemplary EDD code according to an embodiment is shown.

[0059] Figure 11 An example of recoloring based on color values ​​of neighboring points according to an embodiment is shown.

[0060] Figure 12 An example of push-pull background fill according to an embodiment is shown.

[0061] Figure 13 An exemplary possible traversal order of a 4*4 block according to an embodiment is shown.

[0062] Figure 14 An exemplary optimal traversal order according to an embodiment is shown.

[0063] Figure 15 An exemplary 2D video / image encoder according to an embodiment is shown.

[0064] Figure 16 An exemplary V-PCC decoding process according to an embodiment is shown.

[0065] Figure 17 An exemplary 2D video / image decoder according to an embodiment is shown.

[0066] Figure 18 is a flowchart illustrating the operation of the transmitting device according to an embodiment of the present disclosure.

[0067] Figure 19 is a flowchart illustrating the operation of the receiving device according to the embodiment.

[0068] Figure 20 An exemplary architecture for V-PCC based storage and streaming of point cloud data is shown, according to an embodiment.

[0069] Figure 21 is an exemplary block diagram of an apparatus for storing and transmitting point cloud data according to an embodiment.

[0070] Figure 22 is an exemplary block diagram of a point cloud data receiving apparatus according to an embodiment.

[0071] Figure 23 An exemplary structure that can be operated in conjunction with a method / apparatus for transmitting / receiving point cloud data according to an embodiment is shown.

[0072] Figure 24 Shows an example of the relationship between objects and atlas tiles according to an embodiment.

[0073] Figure 25 is a diagram showing an exemplary V-PCC bitstream structure according to an embodiment.

[0074] Figure 26 An example of data carried by a sample stream V-PCC unit in a V-PCC bitstream according to an embodiment is shown.

[0075] Figure 27 An exemplary syntax structure of a sample stream V-PCC header included in a V-PCC bitstream according to an embodiment is shown.

[0076] Figure 28 An exemplary syntax structure of a sample stream V-PCC unit according to an embodiment is shown.

[0077] Figure 29 An exemplary syntax structure of a V-PCC unit according to an embodiment is shown.

[0078] Figure 30 An exemplary syntax structure of a V-PCC unit header according to an embodiment is shown.

[0079] Figure 31 An exemplary V-PCC unit type assigned to the vuh_unit_type field according to an embodiment is shown.

[0080] Figure 32 An exemplary syntax structure of a V-PCC unit payload (vpcc_unit_payload()) according to an embodiment is shown.

[0081] Figure 33 An exemplary syntax structure of a V-PCC parameter set included in a V-PCC unit payload according to an embodiment is shown.

[0082] Figure 34 An example of dividing an atlas frame into multiple tiles according to an embodiment is shown.

[0083] Figure 35 is a diagram illustrating an exemplary atlas sub-stream structure according to an embodiment.

[0084] Figure 36 An exemplary syntax structure of a sample stream NAL header included in an atlas substream according to an embodiment is shown.

[0085] Figure 37 An exemplary syntax structure of a sample stream NAL unit according to an embodiment is shown.

[0086] Figure 38 An embodiment of the syntax structure of nal_unit (NumBytesInNalUnit) according to an embodiment is shown.

[0087] Figure 39 An embodiment of the syntax structure of the NAL unit header according to an embodiment is shown.

[0088] Figure 40 An example of the type of RBSP data structure assigned to the nal_unit_type field according to an embodiment is shown.

[0089] Figure 41 A syntax structure illustrating the syntax of an Atlas sequence parameter set according to an embodiment.

[0090] Figure 42 The syntax structure of the Atlas frame parameter set according to an embodiment is shown.

[0091] Figure 43 FIG. 2 shows the syntax structure of the atlas frame patch information according to an embodiment.

[0092] Figure 44 Shown is the syntax structure of an Atlas adaptation parameter set according to an embodiment.

[0093] Figure 45 A syntax structure of camera parameters according to an embodiment is shown.

[0094] Figure 46 An example of a camera model assigned to the acp_camera_model field according to an embodiment is shown.

[0095] Figure 47 Shown is the syntax structure of the Atlas tiling layer according to an embodiment.

[0096] Figure 48 A syntax structure of an atlas tiling (or patch) header included in an atlas tiling layer according to an embodiment is shown.

[0097] Figure 49 An example of a coding type assigned to an atgh_type field according to an embodiment is shown.

[0098] Figure 50 An implementation of a ref_list_struct() syntax structure according to an implementation is shown.

[0099] Figure 51 An atlas tiling (or patch) data unit is shown according to an embodiment.

[0100] Figure 52 An example of a patch mode type allocated to the atgdu_patch_mode field when the atgh_type field indicates I_TILE_GRP according to an embodiment is shown.

[0101] Figure 53 An example of a patch mode type assigned to the atgdu_patch_mode field when the atgh_type field indicates P_TILE_GRP according to an embodiment is shown.

[0102] Figure 54 An example of a patch mode type assigned to the atgdu_patch_mode field when the atgh_type field indicates SKIP_TILE_GRP according to an embodiment is shown.

[0103] Figure 55 Patch information data according to an embodiment is shown.

[0104] Figure 56 The syntax structure of the patch data unit according to an embodiment is shown.

[0105] Figure 57 Rotation and offset with respect to patch orientation are shown according to an embodiment.

[0106] Figure 58 The syntax structure of SEI information according to an embodiment is shown.

[0107] Figure 59 An exemplary syntax structure of an SEI message payload according to an embodiment is shown.

[0108] Figure 60 is a table showing an example of a sample entry structure and a sample format according to a sample entry type according to an embodiment.

[0109] Figure 61 is a diagram showing an example of transmission of one or more atlas tiles through one V3C track according to an embodiment.

[0110] Figure 62 is a diagram illustrating an example of transmission of one or more atlas tiles over multiple V3C tracks according to an embodiment.

[0111] Figure 63 is a diagram illustrating an exemplary track structure for storing a single atlas (or a single atlas bitstream) having a single atlas tile, according to an embodiment.

[0112] Figure 64 is a diagram illustrating an exemplary track structure for storing a single atlas having multiple atlas tiles, according to an embodiment.

[0113] Figure 65 is a diagram illustrating an exemplary track structure for storing a plurality of atlases according to an embodiment.

[0114] Figure 66 is a diagram illustrating an exemplary track structure for storing a plurality of atlases according to an embodiment.

[0115] Figure 67 is a table showing an example of the relationship between sample entry types and track references according to an embodiment.

[0116] Figure 68 is a diagram showing an exemplary structure for encapsulating non-timed V-PCC data according to an embodiment.

[0117] Figure 69 is a flowchart of a method for transmitting point cloud data according to an embodiment.

[0118] Figure 70 is a flowchart illustrating a method for receiving point cloud data according to an embodiment. DETAILED DESCRIPTION

[0119] Reference will now be made in detail to preferred embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. The detailed description given below with reference to the accompanying drawings is intended to illustrate exemplary embodiments of the present disclosure and is not intended to illustrate the only embodiments that can be implemented according to the present disclosure. The following detailed description includes specific details in order to provide a thorough understanding of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure can be practiced without these specific details.

[0120] Although most of the terms used in this disclosure are selected from common terms widely used in the art, some terms are arbitrarily selected by the applicant and their meanings are explained in detail in the following description as needed. Therefore, this disclosure should be understood based on the intended meaning of the terms rather than their simple names or meanings.

[0121] Figure 1 An exemplary structure of a transmission / reception system for providing point cloud content according to an embodiment is shown.

[0122] The present disclosure provides a method for providing point cloud content to provide users with various services such as virtual reality (VR), augmented reality (AR), mixed reality (MR), and autonomous driving. Point cloud data according to an embodiment represents data representing an object as points and can be referred to as point cloud, point cloud data, point cloud video data, point cloud image data, etc.

[0123] According to an embodiment, the point cloud data transmitting device 10000 may include a point cloud video acquisition unit 10001, a point cloud video encoder 10002, a file / segment encapsulation module (file / segment encapsulator) 10003 and / or a transmitter (or communication module) 10004. The transmitting device according to the embodiment may obtain and process the point cloud video (or point cloud content) and transmit it. According to an embodiment, the transmitting device may include a fixed station, a base transceiver system (BTS), a network, an artificial intelligence (AI) device and / or system, a robot, and an AR / VR / XR device and / or server. According to an embodiment, the transmitting device 10000 may include a device configured to communicate with a base station and / or other wireless devices using a radio access technology (e.g., 5G New RAT (NR), Long Term Evolution (LTE)), a robot, a vehicle, an AR / VR / XR device, a portable device, a home appliance, an Internet of Things (IoT) device, and an AI device / server.

[0124] According to an embodiment, the point cloud video acquisition unit 10001 acquires a point cloud video by capturing, synthesizing, or generating a point cloud video.

[0125] The point cloud video encoder 10002 according to the embodiment encodes the point cloud video data acquired from the point cloud video acquisition unit 10001. According to the embodiment, the point cloud video encoder 10002 may be referred to as a point cloud encoder, a point cloud data encoder, an encoder, etc. The point cloud compression encoding (encoding) according to the embodiment is not limited to the above-mentioned embodiment. The point cloud video encoder may output a bitstream including the encoded point cloud video data. The bitstream may include not only the encoded point cloud video data but also signaling information related to the encoding of the point cloud video data.

[0126] According to an embodiment, the point cloud video encoder 10002 may support a geometry-based point cloud compression (G-PCC) encoding scheme and / or a video-based point cloud compression (V-PCC) encoding scheme. In addition, the point cloud video encoder 10002 may encode point clouds (referred to as point cloud data or points) and / or signaling data associated with the point clouds.

[0127] According to an embodiment, the file / segment encapsulation module 10003 encapsulates point cloud data in the form of files and / or segments. According to an embodiment, the point cloud data transmission method / device can transmit point cloud data in the form of files and / or segments.

[0128] The transmitter (or communication module) 10004 according to the embodiment transmits the encoded point cloud video data in the form of a bit stream. According to the embodiment, the file or fragment can be sent to the receiving device via a network or stored in a digital storage medium (e.g., USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc.). The transmitter according to the embodiment can communicate with the receiving device (or receiver) via a 4G, 5G, 6G network or the like via wired / wireless communication. In addition, the transmitter can perform necessary data processing operations based on the network system (e.g., a 4G, 5G or 6G communication network system). The transmitting device can send the encapsulated data in an on-demand manner.

[0129] According to an embodiment, the point cloud data receiving device 10005 may include a receiver 10006, a file / segment decapsulator (or a file / segment decapsulation module) 10007, a point cloud video decoder 10008, and / or a renderer 10009. According to an embodiment, the receiving device may include a device configured to communicate with a base station and / or other wireless devices using a radio access technology (e.g., 5G New RAT (NR), Long Term Evolution (LTE)), a robot, a vehicle, an AR / VR / XR device, a portable device, a home appliance, an Internet of Things (IoT) device, and an AI device / server.

[0130] According to an embodiment, the receiver 10006 receives a bitstream containing point cloud video data. According to an embodiment, the receiver 10006 may send feedback information to the point cloud data transmitting apparatus 10000.

[0131] The file / fragment decapsulation module 10007 decapsulates files and / or fragments containing point cloud data.

[0132] The point cloud video decoder 10008 decodes the received point cloud video data.

[0133] The renderer 10009 renders the decoded point cloud video data. Depending on the embodiment, the renderer 10009 may send feedback information obtained on the receiving side to the point cloud video decoder 10008. Depending on the embodiment, the point cloud video data may carry the feedback information to the receiver 10006. Depending on the embodiment, the feedback information received by the point cloud transmitting device may be provided to the point cloud video encoder 10002.

[0134] The arrow indicated by the dotted line in the figure represents the transmission path of the feedback information obtained by the receiving device 10005. Feedback information is information reflecting the interactivity with the user consuming the point cloud content, and includes user information (e.g., head orientation information, viewport information, etc.). Specifically, when the point cloud content is content for a service that requires interaction with the user (e.g., an autonomous driving service, etc.), the feedback information can be provided to the content sender (e.g., the sending device 10000) and / or the service provider. Depending on the embodiment, the feedback information may be used in the receiving device 10005 and the sending device 10000, and may not be provided.

[0135] According to an embodiment, head orientation information refers to information regarding the user's head position, orientation, angle, movement, and the like. According to an embodiment, the receiving device 10005 may calculate viewport information based on the head orientation information. Viewport information may be information regarding the area of ​​the point cloud video that the user is viewing. The viewpoint (or orientation) is the point from which the user views the point cloud video and may refer to the center point of the viewport area. In other words, the viewport is the area centered on the viewpoint, and the size and shape of the area may be determined by the field of view (FOV). In other words, the viewport is determined based on the position and viewpoint (or orientation) of the visual camera or user, and point cloud data is rendered within the viewport based on the viewport information. Therefore, in addition to head orientation information, the receiving device 10005 may also extract viewport information based on the vertical or horizontal FOV supported by the device. Furthermore, the receiving device 10005 may perform gaze analysis to examine how the user consumes the point cloud, the area within the point cloud video that the user is gazing at, the duration of the gaze, and other information. According to an embodiment, the receiving device 10005 may send feedback information including the gaze analysis results to the sending device 10000. Depending on the embodiment, the feedback information may be obtained during the rendering and / or display process. Depending on the embodiment, the feedback information may be obtained by one or more sensors included in the receiving device 10005. In addition, depending on the embodiment, the feedback information may be obtained by the renderer 10009 or a separate external element (or device, component, etc.). Figure 1 The dotted line in represents the process of sending the feedback information obtained by the renderer 10009. The point cloud content providing system can process (encode / decode) the point cloud data based on the feedback information. Therefore, the point cloud video decoder 10008 can perform a decoding operation based on the feedback information. The receiving device 10005 can send the feedback information to the sending device. The sending device (or the point cloud video encoder 10002) can perform an encoding operation based on the feedback information. Therefore, the point cloud content providing system can efficiently process necessary data (for example, point cloud data corresponding to the user's head position) based on the feedback information instead of processing (encoding / decoding) all the point cloud data, and provide the point cloud content to the user.

[0136] According to an embodiment, the transmitting device 10000 may be referred to as an encoder, a transmitting device, a transmitter, etc., and the receiving device 10005 may be referred to as a decoder, a receiving device, a receiver, etc.

[0137] According to the embodiment Figure 1 Point cloud data processed in a point cloud content providing system (through a series of processes such as acquisition / encoding / transmission / decoding / rendering) may be referred to as point cloud content data or point cloud video data. Depending on the embodiment, point cloud content data may be used as a concept that encompasses metadata or signaling information related to point cloud data.

[0138] Figure 1 The elements of the illustrated point cloud content providing system may be implemented by hardware, software, a processor, and / or a combination thereof.

[0139] Embodiments may provide a method of providing point cloud content to provide users with various services such as virtual reality (VR), augmented reality (AR), mixed reality (MR), and autonomous driving.

[0140] To provide a point cloud content service, a point cloud video must first be acquired. This acquired point cloud video undergoes a series of processing steps and is then sent to a receiving end. The receiving end then processes the received data back into the original point cloud video and renders the processed point cloud video. This point cloud video can then be provided to the user. Embodiments provide a method for efficiently performing this series of processing steps.

[0141] All processing for providing point cloud content services (point cloud data sending method and / or point cloud data receiving method) may include acquisition processing, encoding processing, transmission processing, decoding processing, rendering processing and / or feedback processing.

[0142] According to an embodiment, the process of providing point cloud content (or point cloud data) may be referred to as a point cloud compression process. According to an embodiment, the point cloud compression process may refer to a video-based point cloud compression (V-PCC) process.

[0143] Each element of the point cloud data transmitting apparatus and the point cloud data receiving apparatus according to the embodiment may be hardware, software, a processor and / or a combination thereof.

[0144] The point cloud compression system may include a transmitting device and a receiving device. Depending on the embodiment, the transmitting device may be referred to as an encoder, a transmitting device, a transmitter, a point cloud data transmitting device, etc. Depending on the embodiment, the receiving device may be referred to as a decoder, a receiving device, a receiver, a point cloud data receiving device, etc. The transmitting device may encode the point cloud video to output a bitstream and transmit it to the receiving device in the form of a file or stream (streaming segment) via a digital storage medium or a network. Digital storage media may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD.

[0145] The sending device may include a point cloud video acquisition unit, a point cloud video encoder, a file / segment encapsulator and a sending unit (sender), such as Figure 1 The receiving device may include a receiver, a file / segment decapsulator, a point cloud video decoder and a renderer, as shown in FIG. Figure 1 As shown. The encoder may be referred to as a point cloud video / image / picture / frame encoder, and the decoder may be referred to as a point cloud video / image / picture / frame decoding device. The renderer may include a display. The renderer and / or the display may be configured as a separate device or an external component. The transmitting device and the receiving device may also include a separate internal or external module / unit / component for feedback processing. Depending on the embodiment, each element in the transmitting device and the receiving device may be configured by hardware, software and / or a processor.

[0146] According to an embodiment, the operation of the receiving device may be a reverse process of the operation of the transmitting device.

[0147] The point cloud video acquirer can acquire point cloud video by capturing, editing, or generating point cloud video. During the acquisition process, data representing the 3D positions (x, y, z) and attributes (color, reflectivity, transparency, etc.) of multiple points can be generated, such as files in the Polygon File Format (PLY) (or Stanford Triangle Format). For videos with multiple frames, one or more files can be acquired. During the capture process, point cloud-related metadata (e.g., capture-related metadata) can be generated.

[0148] A point cloud data transmitting apparatus according to an embodiment may include an encoder configured to encode point cloud data and a transmitter configured to transmit the point cloud data or a bit stream including the point cloud data.

[0149] A point cloud data receiving apparatus according to an embodiment may include a receiver configured to receive a bitstream including point cloud data, a decoder configured to decode the point cloud data, and a renderer configured to render the point cloud data.

[0150] The method / apparatus according to the embodiment represents a point cloud data sending apparatus and / or a point cloud data receiving apparatus.

[0151] Figure 2 Shown is the capture of point cloud data according to an embodiment.

[0152] Point cloud data (point cloud video data) according to an embodiment may be acquired by a camera, etc. Capturing techniques according to an embodiment may include, for example, inward-facing and / or outward-facing.

[0153] In the inward facing according to an embodiment, one or more cameras facing inwardly toward an object of point cloud data may photograph the object from outside the object.

[0154] In outward facing according to an embodiment, one or more cameras facing outward toward an object of point cloud data may photograph the object. For example, according to an embodiment, there may be four cameras.

[0155] Point cloud data or point cloud content according to an embodiment may be a video or still image of an object / environment represented in various types of 3D space. According to an embodiment, point cloud content may include video / audio / image of an object.

[0156] Devices for capturing point cloud content can include a combination of a depth-capable camera (a combination of an infrared pattern projector and an infrared camera) and an RGB camera capable of extracting color information corresponding to the depth information. Alternatively, depth information can be extracted using LiDAR, a radar system that measures the position coordinates of reflectors by emitting laser pulses and measuring the return time. Geometric shapes composed of points in 3D space can be extracted from the depth information, and attributes representing the color / reflectivity of each point can be extracted from the RGB information. Point cloud content can include information about the position (x, y, z) and the color (YCbCr or RGB) or reflectivity (r) of each point. Point cloud content can use both outward-facing techniques, which capture the external environment, and inward-facing techniques, which capture the central object. In VR / AR environments, when objects (e.g., core objects such as characters, players, objects, or actors) are incorporated into point cloud content that can be viewed from any direction (360 degrees), the capture camera can be configured using the inward-facing technique. When the current surrounding environment is incorporated into the point cloud content in vehicle modes such as autonomous driving, the capture camera can be configured using the outward-facing technique. Since point cloud content may be captured by multiple cameras, a camera calibration process may need to be performed to configure the global coordinate system for the cameras before capturing the content.

[0157] Point cloud content can be videos or still images of objects / environments existing in various types of 3D spaces.

[0158] Furthermore, in the point cloud content acquisition method, any point cloud video can be compiled based on captured point cloud videos. Alternatively, when providing a computer-generated point cloud video of a virtual space, it may not be necessary to capture the video using an actual camera. In this case, the capture process can simply be replaced by a process that generates the relevant data.

[0159] The captured point cloud video may need to be post-processed to improve the quality of the content. In the video capture process, the maximum / minimum depth can be adjusted within the range provided by the camera device. Even after the adjustment, point data of unwanted areas may still exist. Therefore, post-processing may be performed to remove unwanted areas (e.g., background) or to identify connected spaces and fill spatial holes. In addition, the point clouds extracted from cameras sharing a spatial coordinate system can be integrated into one piece of content by transforming each point into a global coordinate system based on the position coordinates of each camera obtained through a calibration process. Thus, a piece of point cloud content with a wide range can be generated, or point cloud content with a high density of points can be obtained.

[0160] The point cloud video encoder 10002 can encode an input point cloud video into one or more video streams. A point cloud video may include multiple frames, each of which may correspond to a still image / picture. In this specification, a point cloud video may include point cloud images / frames / pictures / video / audio. In addition, the term "point cloud video" may be used interchangeably with point cloud images / frames / pictures. The point cloud video encoder 10002 may perform video-based point cloud compression (V-PCC) processing. For compression and coding efficiency, the point cloud video encoder may perform a series of processes such as prediction, transformation, quantization, and entropy coding. The encoded data (encoded video / image information) may be output in the form of a bitstream. Based on the V-PCC processing, the point cloud video encoder may encode the point cloud video by dividing it into a geometry video, an attribute video, an occupancy map video, and auxiliary information (or auxiliary data) (described later). The geometry video may include a geometry image, the attribute video may include an attribute image, and the occupancy map video may include an occupancy map image. The auxiliary information may include auxiliary patch information. The attribute video / image may include a texture video / image.

[0161] The file / segment encapsulator (file / segment encapsulation module) 10003 can encapsulate the encoded point cloud video data and / or metadata related to the point cloud video in the form of a file, for example. Here, the metadata related to the point cloud video can be received from a metadata processor. The metadata processor may be included in the point cloud video encoder 10002 or configured as a separate component / module. The file / segment encapsulator 10003 can encapsulate data in a file format such as ISOBMFF or process data in the form of DASH segments, etc. Depending on the embodiment, the file / segment encapsulator 10003 may include metadata related to the point cloud video in a file format. The point cloud video metadata may be included in boxes at various levels of the ISOBMFF file format, or as data in separate tracks within a file. Depending on the embodiment, the file / segment encapsulator 10003 may encapsulate the metadata related to the point cloud video in a file. A transmission processor may perform transmission processing on the point cloud video data encapsulated in the file format. The transmission processor may be included in the transmitter 10004 or configured as a separate component / module. The transmission processor may process the point cloud video data according to a transmission protocol. The transmission process may include a process of transmitting via a broadcast network and a process of transmitting through a broadband. According to an embodiment, the transmission processor may receive point cloud video related metadata together with the point cloud video data from the metadata processor and perform processing of the point cloud video data for transmission.

[0162] The transmitter 10004 can transmit the encoded video / image information or data output in the form of a bitstream in the form of a file or stream to the receiver 10006 of the receiving device via a digital storage medium or network. Digital storage media may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD. The transmitter may include components for generating a media file in a predetermined file format and may also include components for transmission via a broadcast / communication network. The receiver may extract the bitstream and transmit the extracted bitstream to a decoding device.

[0163] Receiver 10006 can receive point cloud video data transmitted by the point cloud video transmitting device according to the present disclosure. Depending on the transmission channel, the receiver can receive point cloud video data via a broadcast network or via broadband. Alternatively, the point cloud video data can be received via a digital storage medium.

[0164] The receiving processor may process the received point cloud video data according to the transmission protocol. The receiving processor may be included in the receiver 10006 or may be configured as a separate component / module. The receiving processor may reverse the above-described processing of the transmitting processor so that the processing corresponds to the transmission processing performed on the transmitting side. The receiving processor may transmit the acquired point cloud video data to the file / segment decapsulator 10007 and transmit the acquired point cloud video-related metadata to a metadata processor (not shown). The point cloud video-related metadata acquired by the receiving processor may take the form of a signaling table.

[0165] The file / segment decapsulator (file / segment decapsulation module) 10007 can decapsulate the point cloud video data received from the receiving processor in the form of a file. The file / segment decapsulator 10007 can decapsulate the file according to ISOBMFF, etc., and can obtain a point cloud video bitstream or point cloud video related metadata (metadata bitstream). The obtained point cloud video bitstream can be transmitted to the point cloud video decoder 10008, and the obtained point cloud video related metadata (metadata bitstream) can be transmitted to a metadata processor (not shown). The point cloud video bitstream may include metadata (metadata bitstream). The metadata processor may be included in the point cloud video decoder 10008, or may be configured as a separate component / module. The point cloud video related metadata obtained by the file / segment decapsulator 10007 may take the form of a box or track in the file format. When necessary, the file / segment decapsulator 10007 may receive the metadata required for decapsulation from the metadata processor. The point cloud video related metadata may be transmitted to the point cloud video decoder 10008 and used in the point cloud video decoding process, or may be transmitted to the renderer 10009 and used in the point cloud video rendering process.

[0166] The point cloud video decoder 10008 can receive a bitstream and decode the video / image by performing operations corresponding to those of the point cloud video encoder. In this case, the point cloud video decoder 10008 can decode the point cloud video by separating the point cloud video into a geometry video, an attribute video, an occupancy map video, and auxiliary information as described below. The geometry video may include a geometry image, the attribute video may include an attribute image, the occupancy map video may include an occupancy map image, the auxiliary information may include auxiliary patch information, and the attribute video / image may include a texture video / image.

[0167] The 3D geometry can be reconstructed based on the decoded geometry image, occupancy map, and auxiliary patch information, and then subjected to a smoothing process. A color point cloud image / screen can be reconstructed by assigning color values ​​to the smoothed 3D geometry based on the texture image. The renderer 10009 can render the reconstructed geometry and color point cloud image / screen. The rendered video / image can be displayed on a display (not shown). The user can view all or part of the rendered result on a VR / AR display or a typical display.

[0168] Feedback processing may include transmitting various types of feedback information that can be obtained in the rendering / display process to a decoder on the sending side or the receiving side. Feedback processing can provide interactivity when consuming point cloud videos. Depending on the embodiment, head orientation information, viewport information indicating the area currently viewed by the user, etc. can be transmitted to the sending side during feedback processing. Depending on the embodiment, the user can interact with things implemented in the VR / AR / MR / autonomous driving environment. In this case, information related to the interaction can be transmitted to the sending side or the service provider during the feedback processing. Depending on the embodiment, feedback processing can be skipped.

[0169] The head orientation information may represent information about the position, angle, and motion of the user's head. Based on this information, information about the area of ​​the point cloud video that the user is currently viewing (i.e., viewport information) may be calculated.

[0170] Viewport information can be information about the area of ​​the point cloud video that a user is currently viewing. This information can be used to perform gaze analysis to examine how the user consumes the point cloud video, the area of ​​the point cloud video the user is gazing at, and how long the user gazes at that area. Gaze analysis can be performed on the receiving side, and the analysis results can be transmitted to the sending side over a feedback channel. Devices such as VR / AR / MR displays can extract the viewport area based on the position / orientation of the user's head, the vertical or horizontal FOV supported by the device, and other factors.

[0171] Depending on the implementation, this feedback information can be not only transmitted to the sending side but also consumed by the receiving side. That is, decoding and rendering processing on the receiving side can be performed based on this feedback information. For example, based on head orientation information and / or viewport information, only the point cloud video of the area currently being viewed by the user can be prioritized for decoding and rendering.

[0172] Here, the viewport or viewport area may refer to the area of ​​the point cloud video that the user is currently viewing. The viewpoint is the point in the point cloud video that the user is viewing and may represent the center point of the viewport area. In other words, the viewport is the area surrounding the viewpoint, and the size and shape of the area may be determined by the field of view (FOV).

[0173] The present disclosure relates to point cloud video compression as described above. For example, the methods / implementations disclosed in the present disclosure may be applied to the Point Cloud Compression or Point Cloud Coding (PCC) standard of the Moving Picture Experts Group (MPEG) or next-generation video / image coding standards.

[0174] As used herein, a picture / frame may generally refer to a unit representing one image in a specific time interval.

[0175] A pixel or picture element can be the smallest unit that constitutes a picture (or image). In addition, "sample" can be used as a term corresponding to a pixel. A sample can generally represent a pixel or a pixel value. It can represent only the pixel / pixel value of the luma component, only the pixel / pixel value of the chroma component, or only the pixel / pixel value of the depth component.

[0176] A unit may represent a basic unit of image processing. A unit may include at least one of a specific area of ​​a picture and information related to the area. In some cases, a unit may be used interchangeably with terms such as a block, an area, or a module. In general, an M×N block may include samples (or sample arrays) or a set (or array) of transform coefficients arranged in M ​​columns and N rows.

[0177] Figure 3 Examples of point clouds, geometric images, and texture images according to embodiments are shown.

[0178] The point cloud according to the embodiment may be input to the Figure 4 The V-PCC encoding process is performed to generate a geometric image and a texture image. According to an embodiment, the point cloud may have the same meaning as point cloud data.

[0179] like Figure 3 As shown, the left part shows a point cloud, where the point cloud object is located in 3D space and can be represented by a bounding box or the like. Figure 3 The middle part shows the geometric image, and Figure 3 The right portion of FIG shows a texture image (non-fill image). In the present disclosure, a geometry image may be referred to as a geometry patch frame / picture or a geometry frame / picture, and a texture image may be referred to as an attribute patch frame / picture or an attribute frame / picture.

[0180] Video-based point cloud compression (V-PCC) according to an embodiment is a method for compressing 3D point cloud data based on a 2D video codec such as High Efficiency Video Coding (HEVC) or Versatile Video Coding (VVC). The data and information that can be generated in the V-PCC compression process are as follows:

[0181] Occupancy map: This is a binary map that uses a value of 0 or 1 to indicate whether data exists at corresponding locations in the 2D plane when the points that make up the point cloud are divided into patches and mapped to the 2D plane. The occupancy map can represent a 2D array corresponding to an atlas, and the values ​​of the occupancy map can indicate whether each sample location in the atlas corresponds to a 3D point. The atlas is an object that contains information about the 2D patches for each point cloud frame. For example, the atlas can include the 2D arrangement and size of the patches, the location of the corresponding 3D region within the 3D points, the projection plan, and the level of detail parameters.

[0182] Patch: A collection of points that make up a point cloud, which indicates that points belonging to the same patch are adjacent to each other in 3D space and are mapped in the same direction between the 6-sided bounding box planes in the process of mapping to a 2D image.

[0183] Geometry image: This is an image in the form of a depth map that presents positional information (geometry) about the individual points that make up the point cloud, patch by patch. A geometry image can consist of pixel values ​​for one channel. The geometry represents a set of coordinates associated with a point cloud frame.

[0184] Texture Image: This is an image that represents the color information of each point in the point cloud on a patch-by-patch basis. A texture image may consist of pixel values ​​for multiple channels (e.g., R, G, and B). Textures are included in attributes. Depending on the implementation, textures and / or attributes may be interpreted as the same object and / or may contain a context.

[0185] Auxiliary patch information: This indicates the metadata required to reconstruct the point cloud using each patch. The auxiliary patch information may include information about the position, size, etc. of the patch in 2D / 3D space.

[0186] Point cloud data according to an embodiment, eg, a V-PCC component, may include atlas, occupancy map, geometry, and attributes.

[0187] An atlas represents a set of 2D bounding boxes. It can be a set of patches, for example, patches projected into a rectangular frame corresponding to a 3D bounding box in 3D space, which can represent a subset of a point cloud. In this case, the patch can represent a rectangular area in the atlas corresponding to a rectangular area in a planar projection. Furthermore, patch data can represent the data required to perform a 2D to 3D transformation on the patches included in the atlas. Furthermore, a patch data set is also referred to as an atlas.

[0188] Attributes can represent scalars or vectors associated with each point in the point cloud. For example, attributes can include color, reflectivity, surface normal, timestamp, material ID.

[0189] Point cloud data according to an embodiment represents PCC data according to a video-based point cloud compression (V-PCC) scheme. Point cloud data may include multiple components. For example, it may include occupancy maps, patches, geometry, and / or textures.

[0190] Figure 4 An example of a point cloud video encoder according to an embodiment is shown.

[0191] Figure 4 A V-PCC encoding process for generating and compressing occupancy maps, geometry images, texture images, and auxiliary patch information is shown. Figure 4 The V-PCC coding process can be done by Figure 1 The point cloud video encoder 10002 is processed. Figure 4 Each element may be executed by software, hardware, a processor and / or a combination thereof.

[0192] The patch generator 14000 receives a point cloud frame (which may be in the form of a bitstream containing point cloud data) and generates a patch based on the point cloud data. In addition, the patch generator 14000 generates patch information including information about patch generation.

[0193] Patch packing or patch packer 14001 packs one or more patches. In addition, the patch packer 14001 generates an occupancy map containing information about patch packing.

[0194] The geometric image generation or geometric image generator 14002 generates a geometric image based on the point cloud data, patch information (or auxiliary information) and / or occupancy map information. The geometric image refers to data containing geometry related to the point cloud data (i.e., 3D coordinate values ​​of points) and refers to a geometric framework.

[0195] Texture image generation or texture image generator 14003 generates a texture image based on point cloud data, patches, packed patches, patch information (or auxiliary information), and / or smoothed geometry. The texture image refers to an attribute frame. In other words, the texture image can also be generated based on smoothed geometry generated by smoothing based on patch information.

[0196] Smoothing or smoother 14004 can reduce or eliminate errors contained in the image data. For example, the reconstructed geometric image is smoothed based on the patch information. In other words, the parts that may cause errors between the data can be smoothly filtered out to generate smooth geometry.

[0197] Auxiliary patch information compression or the auxiliary patch information compressor 14005 can compress auxiliary patch information related to the patch information generated in the patch generation. In addition, the auxiliary patch information compressed by the auxiliary patch information compressor 14005 can be sent to the multiplexer 14013. The auxiliary patch information can be used in the geometric image generator 14002. Depending on the embodiment, the compressed auxiliary patch information can be referred to as a compressed auxiliary patch information bitstream, an auxiliary patch information bitstream, a compressed atlas bitstream, or an atlas bitstream.

[0198] Image filling or image fillers 14006 and 14007 can fill the geometric image and texture image respectively. Filling data can be filled into the geometric image and texture image.

[0199] Group dilation or group dilator 14008 can add data to a texture image in a similar manner to image padding.Auxiliary patch information can be inserted into the texture image.

[0200] Video compression or video compressors 14009, 14010, and 14011 can compress a filled geometry image, a filled texture image, and / or an occupancy map, respectively. In other words, video compressors 14009, 14010, and 14011 can compress an input geometry frame, an attribute frame, and / or an occupancy map frame, respectively, to output a video bitstream of a geometry image, a video bitstream of a texture image, and a video bitstream of an occupancy map. Video compression can encode geometry information, texture information, and occupancy information. Depending on the embodiment, the compressed geometry video bitstream can be referred to as a 2D video coded geometry bitstream, a compressed geometry bitstream, a video coded geometry bitstream, or geometry video data, etc. Depending on the embodiment, the compressed texture image video bitstream can be referred to as a 2D video coded attribute bitstream, a compressed attribute bitstream, a video coded attribute bitstream, or attribute video data, etc.

[0201] Entropy compression or entropy compressor 14012 can compress the occupancy map based on an entropy scheme.

[0202] Depending on the embodiment, entropy compression and / or video compression may be performed on the occupancy map frame depending on whether the point cloud data is lossless and / or lossy. Depending on the embodiment, the entropy and / or video compression map may be referred to as a compressed occupancy map video bitstream, a 2D video encoded occupancy map bitstream, an occupancy map bitstream, a compressed occupancy map bitstream, a video encoded occupancy map bitstream, or occupancy map video data, etc.

[0203] The multiplexer 14013 multiplexes the compressed geometry video bitstream, the compressed texture image video bitstream, the compressed occupancy map video bitstream, and the compressed auxiliary patch information bitstream from each compressor into one bitstream.

[0204] The above blocks may be omitted or may be replaced by blocks having similar or identical functions. Figure 4 Each block shown may function as at least one of a processor, software, and hardware.

[0205] According to the embodiment Figure 4 The detailed operation of each process is described below.

[0206] Patch Generation (14000)

[0207] The patch generation process is a process of dividing a point cloud into patches (mapping units) in order to map the point cloud to a 2D image. The patch generation process can be divided into three steps: normal value calculation, segmentation, and patch segmentation.

[0208] Will refer to Figure 5 The normal value calculation process is described in detail.

[0209] Figure 5 Shown are examples of tangent planes and normal vectors of surfaces according to embodiments.

[0210] exist Figure 4 The patch generator 14000 of the V-PCC coding process is used as follows Figure 5 surface.

[0211] Normal calculation related to patch generation

[0212] Each point of the point cloud has its own direction, which is represented by a 3D vector called a normal vector. Using the neighbors of each point obtained using a KD tree or the like, we can obtain Figure 5 The tangent plane and normal vector of each point constituting the surface of the point cloud are shown. The search range applied to the process of searching for neighbors can be defined by the user.

[0213] A tangent plane is a plane that passes through a point on a surface and completely includes the tangent line to a curve on the surface.

[0214] Figure 6 An exemplary bounding box of a point cloud is shown, according to an embodiment.

[0215] The bounding box according to the embodiment refers to a box for dividing a unit of point cloud data based on a hexahedron in a 3D space.

[0216] A method / apparatus according to an embodiment (eg, patch generator 14000 ) may use a bounding box in a process of generating a patch from point cloud data.

[0217] The bounding box can be used in the process of projecting the target object of the point cloud data onto the plane of each flat face of the hexahedron in the 3D space. The bounding box can be represented by Figure 1The point cloud video acquisition unit 10001 and the point cloud video encoder 10002 generate and process. In addition, based on the bounding box, it is possible to perform Figure 4 The V-PCC encoding process includes patch generation 14000, patch packing 14001, geometric image generation 14002 and texture image generation 14003.

[0218] Segments related to patch generation

[0219] Segmentation is divided into two processes: initial segmentation and refined segmentation.

[0220] The point cloud video encoder 10002 according to the embodiment projects the point onto a face of the bounding box. Specifically, Figure 6 As shown, each point that makes up the point cloud is projected onto one of the six faces of a bounding box surrounding the point cloud. Initial segmentation is the process of determining one of the flat faces of the bounding box to which each point is projected.

[0221] It is the normal value corresponding to each of the six planar faces, defined as follows:

[0222] (1.0,0.0,0.0), (0.0,1.0,0.0), (0.0,0.0,1.0), (-1.0,0.0,0.0), (0.0,-1.0,0.0), (0.0,0.0,-1.0).

[0223] As shown in the following formula, the normal vector of each point obtained in the normal value calculation process is and The plane with the largest dot product value is determined as the projection plane of the corresponding point. In other words, the plane whose normal vector is most similar to the direction of the normal vector of the point is determined as the projection plane of the point.

[0224]

[0225] The determined plane can be identified by a cluster index (one of 0 to 5).

[0226] Refined segmentation is a process that enhances the projection planes of each point that makes up the point cloud, determined in the initial segmentation process, by taking into account the projection planes of neighboring points. In this process, the normal score, which indicates the similarity between the normal vector of each point considered in determining the projection plane in the initial segmentation process and the normals of each flat surface of the bounding box, and the smoothness score, which indicates the similarity between the projection plane of the current point and the projection planes of neighboring points, are considered together.

[0227] Score smoothing can be taken into account by assigning weights to the scoring methods. In this case, the weight values ​​can be user-defined. The segmentation refinement can be performed repeatedly, and the number of repetitions can also be defined by the user.

[0228] Patch segmentation related to patch generation

[0229] Patch segmentation is a process of dividing the entire point cloud into patches (a collection of adjacent points) based on the projection plane information of each point constituting the point cloud obtained in the initial / refinement segmentation process. Patch segmentation may include the following steps:

[0230] Use KD tree etc. to calculate the neighboring points of each point that constitutes the point cloud. The maximum number of neighbors can be defined by the user;

[0231] ① When neighboring points are projected onto the same plane as the current point (when they have the same cluster index),

[0232] ②Extract the current point and adjacent points as a patch;

[0233] ③Calculate the geometric value of the extracted patch.

[0234] ④ Repeat steps ② to ③ until there are no unextracted points.

[0235] The occupancy map, geometry image, and texture image of each patch, as well as the size of each patch, are determined through a patch segmentation process.

[0236] Figure 7 An example of determining positions of individual patches on an occupancy map according to an embodiment is shown.

[0237] The point cloud video encoder 10002 according to an embodiment may perform patch packing and generate an occupancy map.

[0238] Patch packing and occupancy map generation (14001)

[0239] This is the process of determining the locations of individual patches in a 2D image in order to map the segmented patches to the 2D image. As a 2D image, an occupancy map is a binary image, using values ​​of 0 or 1 to indicate the presence or absence of data at the corresponding location. The occupancy map consists of blocks, and its resolution is determined by the block size. For example, a 1x1 block achieves pixel-level resolution. The occupancy packing block size can be determined by the user.

[0240] The process of determining the location of each patch on the occupancy map can be configured as follows:

[0241] ① Set all positions on the occupancy map to 0;

[0242] ② Place the patch at a point (u, v) in the occupancy map plane with horizontal coordinates in the range (0, occupancySizeU - patch.sizeU0) and vertical coordinates in the range (0, occupancySizeV - patch.sizeV0);

[0243] ③ Set the point (x, y) in the patch plane whose horizontal coordinates are in the range (0, patch.sizeU0) and vertical coordinates are in the range (0, patch.sizeV0) as the current point;

[0244] ④ Change the position of the point (x, y) in raster order, and if the value of the coordinate (x, y) on the patch occupancy map is 1 (there is data at the point in the patch) and the value of the coordinate (u+x, v+y) on the global occupancy map is 1 (fill the occupancy map with the previous patch), repeat operations ③ and ④. Otherwise, proceed to operation ⑥;

[0245] ⑤ Change the position of (u,v) in raster order and repeat operations ③ to ⑤;

[0246] ⑥ Determine (u, v) as the location of the patch and copy the occupancy map data about the patch to the corresponding part on the global occupancy map; and

[0247] ⑦ Repeat steps ② to ⑥ for the next patch.

[0248] occupancySizeU: Indicates the width of the occupancy map. Its unit is the occupancy packing block size.

[0249] occupancySizeV: Indicates the height of the occupancy map. Its unit is the occupancy packing block size.

[0250] patch.sizeU0: Indicates the width of the occupancy map. Its unit is the occupancy packing block size.

[0251] patch.sizeV0: indicates the height of the occupancy map. Its unit is the occupancy patch size.

[0252] For example, Figure 7 As shown, in the box corresponding to the occupied packed size block, there exists a box corresponding to the patch having the patch size, and the point (x, y) can be located in the box.

[0253] Figure 8 Exemplary relationships between the normal axis, the tangential axis, and the dual tangential axes according to embodiments are shown.

[0254] The point cloud video encoder 10002 according to an embodiment can generate a geometric image. The geometric image refers to image data including geometric information about the point cloud. The geometric image generation process can use the three axes (normal, tangential, and bitangent) of the patch of Figure 8.

[0255] Geometric Image Generation (14002)

[0256] In this process, the depth values ​​of the geometric images constituting each patch are determined, and the entire geometric image is generated based on the positions of the patches determined in the patch packing process described above. The process of determining the depth values ​​of the geometric images constituting each patch can be configured as follows.

[0257] ① Calculate parameters related to the position and size of each patch. The parameters may include the following information: According to an embodiment, the position of the patch is included in the patch information.

[0258] The normal index indicating the normal axis is obtained in the previous patch generation process. The tangential axis is an axis perpendicular to the normal axis that coincides with the horizontal axis u of the patch image, and the bi-tangential axis is an axis perpendicular to the normal axis that coincides with the vertical axis v of the patch image. The three axes can be represented as Figure 8 shown.

[0259] Figure 9 Exemplary configurations of the minimum mode and the maximum mode of the projection mode according to the embodiment are shown.

[0260] The point cloud video encoder 10002 according to an embodiment may perform patch-based projection to generate a geometric image, and the projection mode according to an embodiment includes a minimum mode and a maximum mode.

[0261] The 3D space coordinates of the patch may be calculated based on a minimum-sized bounding box surrounding the patch. For example, the 3D space coordinates may include the minimum tangent value of the patch (on the patch 3D shift tangent axis), the minimum bitangent value of the patch (on the patch 3D shift bitangent axis), and the minimum normal value of the patch (on the patch 3D shift normal axis).

[0262] The 2D size of a patch indicates the horizontal and vertical size of the patch when it is packed into a 2D image. The horizontal size (patch 2D size u) can be obtained as the difference between the maximum tangent value and the minimum tangent value of the bounding box, and the vertical size (patch 2D size v) can be obtained as the difference between the maximum bitangent value and the minimum bitangent value of the bounding box.

[0263] ② Determine the projection mode of the patch. The projection mode can be minimum or maximum. Geometric information about the patch is represented by depth values. When the points that make up the patch are projected along the normal to the patch, two images are generated: an image constructed using the maximum depth value and an image constructed using the minimum depth value.

[0264] In minimum mode, when generating two-layer images d0 and d1, the minimum depth can be configured for d0, and the maximum depth within the surface thickness from the minimum depth can be configured for d1, such as Figure 9 shown.

[0265] For example, when the point cloud is in 2D (such as Figure 9As shown in FIG, there are multiple patches including multiple points. As shown in the figure, it is indicated that points marked with the same style of shadow can belong to the same patch. The figure shows the processing of projecting a patch marked with a blank.

[0266] When a point marked with a blank is projected left / right, the depth may be incremented by 1 relative to the left side, 0, 1, 2, ..., 6, 7, 8, 9, and the number used to calculate the depth of the point may be marked on the right side.

[0267] The same projection mode can be applied to all point clouds, or different projection modes can be applied to each frame or patch according to user definition. When different projection modes are applied to each frame or patch, the projection mode that can enhance compression efficiency or minimize missing points can be adaptively selected.

[0268] ③Calculate the depth value of each point.

[0269] In minimum mode, image d0 is constructed using depth0, which is the value obtained by subtracting the minimum normal value of the patch (on the patch 3D shift normal axis) calculated in operation 1) from the minimum normal value of the patch (on the patch 3D shift normal axis) for each point. If another depth value exists at the same location within the range between depth0 and the surface thickness, that value is set to depth1. Otherwise, the value of depth0 is assigned to depth1. Image d1 is constructed using the value of depth1.

[0270] For example, when calculating the depth of a point in image d0, the minimum value (4 2 4 4 0 6 0 0 9 9 0 80) may be calculated. When calculating the depth of a point in image d1, the larger value among two or more points may be calculated. When only one point exists, its value may be calculated as (4 4 4 4 6 6 6 8 9 9 8 8 9). In the process of encoding and reconstructing the points of the patch, some points may be missing (for example, in the figure, eight points are missing).

[0271] In maximum mode, image d0 is constructed using depth0, which is the value obtained by subtracting the minimum normal value of the patch (on the patch 3D shift normal axis) calculated in operation 1) from the minimum normal value of the patch (on the patch 3D shift normal axis) for each point. If another depth value exists within the range between depth0 and the surface thickness at the same location, that value is set to depth1. Otherwise, the value of depth0 is assigned to depth1. Image d1 is constructed using the value of depth1.

[0272] For example, when calculating the depth of a point in image d0, the minimum value (4 4 4 4 6 6 6 8 9 9 8 89) may be calculated. When calculating the depth of a point in image d1, the smaller value among two or more points may be calculated. When only one point exists, its value may be calculated as (4 2 4 4 5 6 0 6 9 9 0 8 0). In the process of encoding and reconstructing the points of the patch, some points may be missing (for example, in the figure, six points are missing).

[0273] The entire geometric image can be generated by placing the geometric images of the respective patches generated through the above-described process onto the entire geometric image based on the patch position information determined in the patch packing process.

[0274] The generated layer d1 of the entire geometric image can be encoded using various methods. The first method (absolute d1 encoding method) encodes the depth value of the previously generated image d1. The second method (differential encoding method) encodes the difference between the depth value of the previously generated image d1 and the depth value of image d0.

[0275] In the encoding method using depth values ​​of two layers d0 and d1 as described above, if there is another point between the two depths, geometric information about the point is lost in the encoding process, so Enhanced-Delta-Depth (EDD) code can be used for lossless encoding.

[0276] Below, we will refer to Figure 10 Describe the EDD code in detail.

[0277] Figure 10 An exemplary EDD code according to an embodiment is shown.

[0278] In the point cloud video encoder 10002 and / or some / all processes of V-PCC encoding (e.g., video compression 14009), geometric information about points can be encoded based on EOD codes.

[0279] like Figure 10 As shown, the EDD code is used to encode the binary value of the position of all points within the surface thickness range including d1. Figure 10 In the example, since there are points at the first and fourth positions on D0 and the second and third positions are empty, the points included in the second column from the left can be represented by EDD code 0b1001 (=9). When the EDD code is encoded and transmitted together with D0, the receiving terminal can restore the geometric information about all points without loss.

[0280] For example, when there is a point above the reference point, the value is 1. When there is no point, the value is 0. Therefore, the code can be expressed based on 4 bits.

[0281] Smooth (14004)

[0282] Smoothing is an operation used to eliminate discontinuities that may appear on patch boundaries due to image quality degradation that occurs during the compression process. Smoothing can be performed by the point cloud video encoder 10002 or the smoother 14004:

[0283] ① Reconstructing point clouds from geometric images. This operation can be the opposite of the geometric image generation described above. For example, the reverse process of the encoding can be reconstructed;

[0284] ② Use KD tree to calculate the neighboring points of each point that constitutes the reconstructed point cloud;

[0285] ③ Determine whether each point is on the patch boundary. For example, when there is a neighboring point with a different projection plane (cluster index) than the current point, it can be determined that the point is on the patch boundary;

[0286] ④ If a point exists on the patch boundary, move it to the centroid of the neighboring points (located at the average x, y, z coordinates of the neighboring points). In other words, change the geometry value. Otherwise, maintain the previous geometry value.

[0287] Figure 11 An example of recoloring based on color values ​​of neighboring points according to an embodiment is shown.

[0288] The point cloud video encoder 10002 or the texture image generator 14003 according to an embodiment may generate a texture image based on recoloring.

[0289] Texture Image Generation (14003)

[0290] Similar to the above-described geometric image generation process, the texture image generation process includes generating texture images for each patch and arranging the texture images at a determined position to generate the entire texture image. However, in the operation of generating the texture image for each patch, instead of the depth value used for geometric generation, an image having color values ​​(e.g., R, G, and B values) of the points constituting the point cloud corresponding to the position is generated.

[0291] When estimating the color values ​​of the individual points that make up the point cloud, the geometry obtained previously by the smoothing process can be used. In the smoothed point cloud, the positions of some points may have shifted relative to the original point cloud, so it may be necessary to find a recoloring process that is appropriate for the changed position. The recoloring can be performed using the color values ​​of neighboring points. For example, Figure 11 As shown, the new color value can be calculated by considering the color value of the nearest neighboring point and the color values ​​of the neighboring points.

[0292] For example, refer to Figure 11, in recoloring, an appropriate color value of the changed position may be calculated based on an average of attribute information about the nearest original point of the point and / or an average of attribute information about the nearest original point of the point.

[0293] Similar to the geometric image generated with two layers d0 and d1, the texture image can also be generated with two layers t0 and t1.

[0294] Auxiliary patch information compression (14005)

[0295] The point cloud video encoder 10002 or the auxiliary patch information compressor 14005 according to an embodiment may compress auxiliary patch information (auxiliary information about a point cloud).

[0296] The auxiliary patch information compressor 14005 compresses the auxiliary patch information generated in the patch generation, patch packing and geometry generation processes. The auxiliary patch information may include the following parameters:

[0297] The index (cluster index) used to identify the projection plane (normal plane);

[0298] The 3D spatial position of the patch, i.e., the minimum tangent value of the patch (on the patch 3D shift tangential axis), the minimum bitangent value of the patch (on the patch 3D shift bitangent axis), and the minimum normal value of the patch (on the patch 3D shift normal axis);

[0299] The 2D spatial position and size of the patch, i.e., horizontal size (patch2d size u), vertical size (patch2d size v), minimum horizontal value (patch2d shift u), and minimum vertical value (patch2d shift u); and

[0300] Mapping information about each block and patch, that is, candidate index (when patches are arranged in order based on 2D spatial position and size information about the patches, multiple patches can be mapped to one block in an overlapping manner. In this case, the mapped patches constitute a candidate list, and the candidate index indicates the sequential position of the patch whose data exists in the block) and the local patch index (indicating the index of one patch existing in the frame). Table 1 shows pseudo code representing a process of matching between blocks and patches based on the candidate list and the local patch index.

[0301] The maximum number of candidate lists can be defined by the user.

[0302] [Table 1]

[0303]

[0304] Figure 12 Shown is a push-pull background fill according to an embodiment.

[0305] Image Fill and Group Dilation (14006, 14007, 14008)

[0306] The image filler according to an embodiment may fill the space except the patch area with meaningless supplementary data based on a push-pull background filling technique.

[0307] Image padding 14006 and 14007 is a process of filling the space outside the patch area with meaningless data to improve compression efficiency. For image padding, the pixel values ​​in the columns or rows near the boundaries of the patch can be copied to fill the empty space. Alternatively, as Figure 12 As shown, a push-pull background filling method can be used. According to this method, the empty space is filled with pixel values ​​from the low-resolution image in a process of gradually reducing the resolution of the non-filled image and increasing the resolution again.

[0308] Group expansion 14008 is a process for filling the blank spaces of the geometry image and texture image configured in two layers d0 / d1 and t0 / t1, respectively. In this process, the blank spaces of the two layers calculated by image filling are filled with the average of the values ​​at the same position.

[0309] Figure 13 An exemplary possible traversal order of a 4*4 block according to an embodiment is shown.

[0310] Occupancy map compression (14012, 14011)

[0311] The occupancy map compressor according to an embodiment can compress a previously generated occupancy map. Specifically, two methods can be used, namely: video compression for lossy compression and entropy compression for lossless compression. Video compression is described below.

[0312] Entropy compression can be performed by the following operations.

[0313] ① If the block constituting the occupancy map is fully occupied, encode 1 and repeat the same operation for the next block of the occupancy map. Otherwise, encode 0 and perform operations 2) to 5).

[0314] ② Determine the optimal traversal order for performing run-length encoding on the occupied pixels of the block. Figure 13 Four possible traversal orders of a 4*4 block are shown.

[0315] Figure 14 An exemplary optimal traversal order according to an embodiment is shown.

[0316] As described above, the entropy compressor according to the embodiment can be based on Figure 14 The traversal order scheme shown encodes the blocks (Encoding).

[0317] For example, the best traversal order with the smallest number of runs is selected from the possible traversal orders, and its index is encoded. Figure 13 In the case of the third traversal order in . In the case shown, the number of runs can be minimized to 2, so the third traversal order can be selected as the optimal traversal order.

[0318] ③Encode the number of runs. Figure 14 In the example of , there are two runs, so 2 is encoded.

[0319] ④ Encode the occupation of the first run. Figure 14 In the example of , 0 is encoded because the first run corresponds to an unoccupied pixel.

[0320] ⑤ Encode the length of each run (as many as the number of runs). Figure 14 In the example, the lengths 6 and 10 of the first run and the second run are encoded respectively.

[0321] Video compression (14009, 14010, 14011)

[0322] The video compressors 14009 , 14010 , 14011 according to an embodiment encode the sequences of geometry images, texture images, occupancy map images, etc. generated in the above operations using a 2D video codec such as HEVC or VVC.

[0323] Figure 15 An exemplary 2D video / image encoder according to an embodiment is shown. According to an embodiment, the 2D video / image encoder may be referred to as an encoding device.

[0324] Indicates an embodiment in which the above-mentioned video compressors 14009, 14010 and 14011 are applied Figure 15 2D video / image encoder 15000 is a schematic block diagram of a 2D video / image encoder configured to encode a video / image signal. 2D video / image encoder 15000 may be included in the above-mentioned point cloud video encoder 10002, or may be configured as an internal / external component. Figure 15 Each component may correspond to software, hardware, a processor and / or a combination thereof.

[0325] Here, the input image may include one of the above-mentioned geometric image, texture image (attribute image) and occupancy map image. Figure 15 When the 2D video / image encoder is applied to the video compressor 14009, the image input to the 2D video / image encoder 15000 is a filled geometric image, and the bit stream output from the 2D video / image encoder 15000 is a bit stream of the compressed geometric image. Figure 15When the 2D video / image encoder is applied to the video compressor 14010, the image input to the 2D video / image encoder 15000 is a padded texture image, and the bit stream output from the 2D video / image encoder 15000 is a bit stream of the compressed texture image. Figure 15 When the 2D video / image encoder is applied to the video compressor 14011, the image input to the 2D video / image encoder 15000 is an occupancy map image, and the bit stream output from the 2D video / image encoder 15000 is a bit stream of the compressed occupancy map image.

[0326] The inter-frame predictor 15090 and the intra-frame predictor 15100 may be collectively referred to as a predictor. That is, the predictor may include the inter-frame predictor 15090 and the intra-frame predictor 15100. The transformer 15030, the quantizer 15040, the inverse quantizer 15050, and the inverse transformer 15060 may be collectively referred to as a residual processor. The residual processor may further include a subtractor 15020. According to an embodiment, Figure 15 The image segmenter 15010, the subtractor 15020, the transformer 15030, the quantizer 15040, the inverse quantizer 15050, the inverse transformer 15060, the adder 15200, the filter 15070, the inter-frame predictor 15090, the intra-frame predictor 15100, and the entropy encoder 15110 may be configured by one hardware component (e.g., an encoder or a processor). In addition, the memory 15080 may include a decoded picture buffer (DPB) and may be configured by a digital storage medium.

[0327] The image splitter 15010 may split the image (or picture or frame) input to the encoder 15000 into one or more processing units. For example, a processing unit may be referred to as a coding unit (CU). In this case, the CU may be recursively split from a coding tree unit (CTU) or a largest coding unit (LCU) based on a quadtree binary tree (QTBT) structure. For example, a CU may be split into multiple CUs of lower depth based on a quadtree structure and / or a binary tree structure. In this case, for example, the quadtree structure may be applied first, and the binary tree structure may be applied later. Alternatively, the binary tree structure may be applied first. The encoding process according to the present disclosure may be performed based on a final CU that is no longer split. In this case, the LCU may be used as the final CU based on encoding efficiency according to the characteristics of the image. If necessary, the CU may be recursively split into CUs of lower depth, and the optimally sized CU may be used as the final CU. Here, the encoding process may include prediction, transform, and reconstruction (to be described later). As another example, the processing unit may also include a prediction unit (PU) or a transform unit (TU). In this case, the PU and TU may be partitioned or divided from the final CU. A PU may be a unit for sample prediction, and a TU may be a unit for deriving a transform coefficient and / or a unit for deriving a residual signal from the transform coefficient.

[0328] The term "unit" can be used interchangeably with terms such as block, region, or module. In general, an M×N block can represent a set of samples or transform coefficients arranged in M ​​columns and N rows. A sample can generally represent a pixel or pixel value and can indicate only the pixel / pixel value of the luma component or only the pixel / pixel value of the chroma component. "Sample" can be used as a term corresponding to a pixel or picture element in a picture (or image).

[0329] The subtractor 15020 of the encoder 15000 can generate a residual signal (residual block or residual sample array) by subtracting the prediction signal (prediction block or prediction sample array) output from the inter predictor 15090 or the intra predictor 15100 from the input image signal (original block or original sample array). The generated residual signal is then transmitted to the transformer 15030. In this case, as shown in the figure, the unit in the encoder 15000 that subtracts the prediction signal (prediction block or prediction sample array) from the input image signal (original block or original sample array) can be referred to as the subtractor 15020. The predictor can perform prediction on a processing target block (hereinafter referred to as the current block) and generate a prediction block including prediction samples of the current block. The predictor can determine whether to apply intra prediction or inter prediction based on the current block or CU. As will be described later in the description of each prediction mode, the predictor can generate various types of information regarding the prediction (e.g., prediction mode information) and transmit the generated information to the entropy encoder 15110. Information about the prediction may be encoded by the entropy encoder 15110 and output in the form of a bitstream.

[0330] The intra-frame predictor 15100 of the predictor can predict the current block with reference to samples in the current picture. Depending on the prediction mode, the samples may be adjacent to or far away from the current block. Under intra-frame prediction, the prediction mode may include multiple non-directional modes and multiple directional modes. The non-directional mode may include, for example, a DC mode and a planar mode. Depending on the fineness of the prediction direction, the directional mode may include, for example, 33 directional prediction modes or 65 directional prediction modes. However, this is only an example, and more or fewer directional prediction modes may be used depending on the settings. The intra-frame predictor 15100 may determine the prediction mode to be applied to the current block based on the prediction mode applied to the neighboring blocks.

[0331] The inter-frame predictor 15090 of the predictor can derive a prediction block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. In this case, to reduce the amount of motion information transmitted in inter-frame prediction mode, motion information can be predicted on a per-block, per-subblock, or per-sample basis based on the correlation of motion information between neighboring blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can also include information regarding the inter-frame prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter-frame prediction, neighboring blocks can include spatially neighboring blocks in the current picture and temporally neighboring blocks in the reference picture. The reference picture containing the reference block can be the same as or different from the reference picture containing the temporally neighboring block. A temporally neighboring block can be referred to as a collocated reference block or collocated CU (colCU), and a reference picture containing a temporally neighboring block can be referred to as a collocated picture (colPic). For example, the inter-frame predictor 15090 can configure a motion information candidate list based on the neighboring blocks and generate information indicating candidates to be used for deriving the motion vector and / or reference picture index for the current block. Inter-frame prediction can be performed based on various prediction modes. For example, in skip mode and merge mode, the inter-frame predictor 15090 may use motion information about a neighboring block as motion information about the current block. In skip mode, unlike merge mode, a residual signal may not be sent. In motion vector prediction (MVP) mode, the motion vector of the neighboring block may be used as a motion vector predictor, and a motion vector difference may be signaled to indicate the motion vector of the current block.

[0332] The prediction signal generated by the inter-frame predictor 15090 or the intra-frame predictor 15100 may be used to generate a reconstructed signal or to generate a residual signal.

[0333] The transformer 15030 may generate transform coefficients by applying a transform technique to the residual signal. For example, the transform technique may include at least one of a discrete cosine transform (DCT), a discrete sine transform (DST), a Karhunen–Loève transform (KLT), a graph-based transform (GBT), or a conditional nonlinear transform (CNT). Here, a GBT refers to a transform obtained from a graph depicting the relationship between pixels. A CNT refers to a transform obtained based on a prediction signal generated from all previously reconstructed pixels. In addition, the transform operation may be applied to square pixel blocks of the same size, or to blocks of variable sizes other than square.

[0334] The quantizer 15040 may quantize the transform coefficients and transmit them to the entropy encoder 15110. The entropy encoder 15110 may encode the quantized signal (information about the quantized transform coefficients) and output a bitstream of the encoded signal. The information about the quantized transform coefficients may be referred to as residual information. The quantizer 15040 may rearrange the quantized transform coefficients in the form of a block in the form of a one-dimensional vector based on a coefficient scanning order, and generate information about the quantized transform coefficients based on the quantized transform coefficients in the form of a one-dimensional vector.

[0335] The entropy encoder 15110 may employ various encoding techniques such as, for example, exponential Golomb, context-adaptive variable length coding (CAVLC), and context-adaptive binary arithmetic coding (CABAC). The entropy encoder 15110 may encode information required for video / image reconstruction (e.g., syntax element values) together with or separately from the quantized transform coefficients. The encoded information (e.g., encoded video / image information) may be transmitted or stored in the form of a bitstream based on a network abstraction layer (NAL) unit.

[0336] The bitstream may be sent via a network or may be stored in a digital storage medium. Here, the network may include a broadcast network and / or a communication network, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD. A transmitter (not shown) that sends a signal output from the entropy encoder 15110 and / or a storage unit (not shown) that stores the signal may be configured as an internal / external element of the encoder 15000. Alternatively, the transmitter may be included in the entropy encoder 15110.

[0337] The quantized transform coefficients output from the quantizer 15040 can be used to generate a prediction signal. For example, inverse quantization and inverse transformation can be applied to the quantized transform coefficients by the inverse quantizer 15050 and the inverse transformer 15060 to reconstruct a residual signal (residual block or residual sample). The adder 15200 adds the reconstructed residual signal to the prediction signal output from the inter-frame predictor 15090 or the intra-frame predictor 15100. Thus, a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) can be generated. When there is no residual signal for the processing target block, as in the case of applying skip mode, the prediction block can be used as a reconstructed block. The adder 15200 can be referred to as a reconstructor or a reconstructed block generator. As described below, the generated reconstructed signal can be used for intra-frame prediction of the next processing target block in the current picture, or can be used for inter-frame prediction of the next picture through filtering.

[0338] The filter 15070 can improve the subjective / objective image quality by applying filtering to the reconstructed signal output from the adder 15200. For example, the filter 15070 can generate a modified reconstructed picture by applying various filtering techniques to the reconstructed picture, and the modified reconstructed picture can be stored in the memory 15080 (specifically, the DPB of the memory 15080). For example, the various filtering techniques may include deblocking filtering, sample adaptive offset, adaptive loop filtering, and bilateral filtering. As described below in the description of the filtering techniques, the filter 15070 can generate various types of information about filtering and transmit the generated information to the entropy encoder 15110. The information about filtering can be encoded by the entropy encoder 15110 and output in the form of a bitstream.

[0339] The modified reconstructed picture stored in the memory 15080 may be used as a reference picture by the inter-frame predictor 15090. Therefore, when inter-frame prediction is applied, the encoder may avoid prediction mismatch between the encoder 15000 and the decoder and improve encoding efficiency.

[0340] The DPB of the memory 15080 may store the modified reconstructed picture for use as a reference picture by the inter-frame predictor 15090. The memory 15080 may store motion information about a block from which motion information in the current picture is derived (or encoded) and / or motion information about a reconstructed block in the picture. The stored motion information may be transmitted to the inter-frame predictor 15090 for use as motion information about spatially neighboring blocks or motion information about temporally neighboring blocks. The memory 15080 may store reconstructed samples of the reconstructed blocks in the current picture and transmit the reconstructed samples to the intra-frame predictor 15100.

[0341] At least one of the above-mentioned prediction, transformation and quantization processes may be skipped. For example, for a block to which pulse code modulation (PCM) is applied, the prediction, transformation and quantization processes may be skipped, and the value of the original sample may be encoded and output in the form of a bitstream.

[0342] Figure 16 An exemplary V-PCC decoding process according to an embodiment is shown.

[0343] The V-PCC decoding process or the V-PCC decoder may follow Figure 4 The reverse process of the V-PCC encoding process (or encoder). Figure 16 Each component may correspond to software, hardware, a processor and / or a combination thereof.

[0344] The demultiplexer 16000 demultiplexes the compressed bit stream to output a compressed texture image, a compressed geometry image, a compressed occupancy map, and compressed auxiliary patch information respectively.

[0345] Video decompression or video decompressor 16001, 16002 decompresses each of the compressed texture image and the compressed geometry image.

[0346] Occupancy map decompression or occupancy map decompressor 16003 decompresses the compressed occupancy map image.

[0347] The auxiliary patch information decompression or auxiliary patch information decompressor 16004 decompresses the compressed auxiliary patch information.

[0348] Geometry reconstruction or geometry reconstructor 16005 restores (reconstructs) geometry information based on the decompressed geometry image, decompressed occupancy map and / or decompressed auxiliary patch information. For example, geometry that changed during the encoding process can be reconstructed.

[0349] Smoothing or smoother 16006 can apply smoothing to the reconstructed geometry. For example, a smoothing filter can be applied.

[0350] Texture reconstruction or texture reconstructor 16007 reconstructs textures from decompressed texture images and / or smoothed geometry.

[0351] Color smoothing or color smoother 16008 smoothes color values ​​from the reconstructed texture. For example, a smoothing filter may be applied.

[0352] As a result, reconstructed point cloud data can be generated.

[0353] Figure 16 The decoding process of V-PCC is shown to reconstruct a point cloud by decompressing (decoding) the compressed occupancy map, geometry image, texture image, and auxiliary patch information.

[0354] Figure 16 Each unit shown may be used as at least one of a processor, software, and hardware. Figure 16 The detailed operation of each unit is described as follows.

[0355] Video decompression (16001, 16002)

[0356] Video decompression is the inverse process of the above-mentioned video compression. It is a process of decoding the bitstream of the geometry image, the bitstream of the compressed texture image, and / or the bitstream of the compressed occupancy map image generated in the above-mentioned process using a 2D video codec such as HEVC or VVC.

[0357] Figure 17 An exemplary 2D video / image decoder, also referred to as a decoding device, according to an embodiment is shown.

[0358] 2D video / image decoder can follow Figure 15 The reverse process of the operation of the 2D video / image encoder.

[0359] Figure 17 The 2D video / image decoder is Figure 16 Implementation of video decompressors 16001 and 16002. Figure 17 17000 is a schematic block diagram of a 2D video / image decoder 17000 for decoding a video / image signal. The 2D video / image decoder 17000 may be included in the point cloud video decoder 10008 or configured as an internal / external component. Figure 17 Each component may correspond to software, hardware, a processor and / or a combination thereof.

[0360] Here, the input bitstream may be one of a bitstream of a geometry image, a bitstream of a texture image (attribute image), and a bitstream of an occupancy map image. Figure 17 When the 2D video / image decoder is applied to the video decompressor 16001, the bit stream input to the 2D video / image decoder is the bit stream of the compressed texture image, and the reconstructed image output from the 2D video / image decoder is the decompressed texture image. Figure 17 When the 2D video / image decoder is applied to the video decompressor 16002, the bit stream input to the 2D video / image decoder is a bit stream of a compressed geometric image, and the reconstructed image output from the 2D video / image decoder is a decompressed geometric image. Figure 17 The 2D video / image decoder can receive the bitstream of the compressed occupancy map image and decompress it. The reconstructed image (or output image or decoded image) can represent the reconstructed image of the above-mentioned geometric image, texture image (attribute image) and occupancy map image.

[0361] Reference Figure 17 , the inter-frame predictor 17070 and the intra-frame predictor 17080 may be collectively referred to as a predictor. That is, the predictor may include the inter-frame predictor 17070 and the intra-frame predictor 17080. The inverse quantizer 17020 and the inverse transformer 17030 may be collectively referred to as a residual processor. That is, depending on the embodiment, the residual processor may include the inverse quantizer 17020 and the inverse transformer 17030. Figure 17 The entropy decoder 17010, the inverse quantizer 17020, the inverse transformer 17030, the adder 17040, the filter 17050, the inter-frame predictor 17070, and the intra-frame predictor 17080 may be configured by one hardware component (e.g., a decoder or a processor). In addition, the memory 17060 may include a decoded picture buffer (DPB) or may be configured by a digital storage medium.

[0362] When a bit stream containing video / image information is input, the decoder 17000 can Figure 15 The decoder 17000 reconstructs the image in a process corresponding to the process of processing video / image information by the encoder. For example, the decoder 17000 can perform decoding using the processing unit used in the encoder. Therefore, the processing unit for decoding can be, for example, a CU. The CU can be divided from the CTU or LCU along a quadtree structure and / or a binary tree structure. The reconstructed video signal decoded and output by the decoder 17000 can then be played by a player.

[0363] The decoder 17000 may receive a signal output from the encoder in the form of a bitstream, and the received signal may be decoded by the entropy decoder 17010. For example, the entropy decoder 17010 may parse the bitstream to derive information (e.g., video / image information) required for image reconstruction (or picture reconstruction). For example, the entropy decoder 17010 may decode the information in the bitstream based on a coding technique such as exponential Golomb coding, CAVLC, or CABAC, and output syntax elements required for image reconstruction and quantized values ​​of residual transform coefficients. More specifically, in CABAC entropy decoding, bins corresponding to various syntax elements in the bitstream may be received, and a context model may be determined based on information about the target syntax element to be decoded, as well as decoded information about adjacent target blocks or information about symbols / bins decoded in previous steps. The probability of occurrence of a bin may then be predicted based on the determined context model, and arithmetic decoding of the bins may be performed to generate symbols corresponding to the values ​​of the various syntax elements. According to CABAC entropy decoding, after determining the context model, the context model can be updated based on information about the symbol / cell decoded for the context model of the next symbol / cell. Information about prediction in the information decoded by the entropy decoder 17010 can be provided to the predictor (inter-frame predictor 17070 and intra-frame predictor 17080), and the residual value (i.e., quantized transform coefficients and related parameter information) that has been entropy decoded by the entropy decoder 17010 can be input to the inverse quantizer 17020. In addition, information about filtering in the information decoded by the entropy decoder 17010 can be provided to the filter 17050. A receiver (not shown) configured to receive a signal output from the encoder can also be configured as an internal / external element of the decoder 17000. Alternatively, the receiver can be a component of the entropy decoder 17010.

[0364] The inverse quantizer 17020 may output a transform coefficient by inversely quantizing the quantized transform coefficient. The inverse quantizer 17020 may rearrange the quantized transform coefficients in the form of two-dimensional blocks. In this case, the rearrangement may be performed based on the coefficient scanning order implemented by the encoder. The inverse quantizer 17020 may perform inverse quantization on the quantized transform coefficient using a quantization parameter (e.g., quantization step size information) and obtain the transform coefficient.

[0365] The inverse transformer 17030 obtains a residual signal (a residual block and a residual sample array) by transforming the transform coefficients.

[0366] The predictor may perform prediction on the current block and generate a prediction block including prediction samples of the current block. The predictor may determine whether to apply intra prediction or inter prediction to the current block based on the information about prediction output from the entropy decoder 17010, and may determine a specific intra / inter prediction mode.

[0367] The intra-frame predictor 17080 of the predictor can predict the current block with reference to samples in the current picture. Depending on the prediction mode, the samples may be adjacent to or distant from the current block. In intra-frame prediction, the prediction mode may include multiple non-directional modes and multiple directional modes. The intra-frame predictor 17080 can use the prediction mode applied to the neighboring blocks to determine the prediction mode applied to the current block.

[0368] The inter-frame predictor 17070 of the predictor can derive a prediction block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. In this case, to reduce the amount of motion information transmitted in inter-frame prediction mode, motion information can be predicted on a per-block, sub-block, or sample basis based on the correlation of motion information between neighboring blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may also include information regarding the inter-frame prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter-frame prediction, neighboring blocks may include spatially neighboring blocks in the current picture and temporally neighboring blocks in the reference picture. For example, the inter-frame predictor 17070 may configure a motion information candidate list based on the neighboring blocks and derive the motion vector and / or reference picture index of the current block based on received candidate selection information. Inter-frame prediction can be performed based on various prediction modes. Information regarding the prediction may include information indicating the inter-frame prediction mode for the current block.

[0369] The adder 17040 may add the residual signal obtained from the inverse transformer 17030 to the prediction signal (prediction block or prediction sample array) output from the inter-frame predictor 17070 or the intra-frame predictor 17080, thereby generating a reconstructed signal (reconstructed picture, reconstructed block, or reconstructed sample array). When there is no residual signal for processing the target block as in the case of applying the skip mode, the prediction block may be used as the reconstructed block.

[0370] The adder 17040 may be referred to as a reconstructor or a reconstructed block generator. The generated reconstructed signal may be used for intra-frame prediction of the next processing target block in the current picture, or may be used for inter-frame prediction of the next picture through filtering as described below.

[0371] The filter 17050 can improve the subjective / objective image quality by applying filtering to the reconstructed signal output from the adder 17040. For example, the filter 17050 can generate a modified reconstructed picture by applying various filtering techniques to the reconstructed picture, and can send the modified reconstructed picture to the memory 17060 (specifically, the DPB of the memory 17060). For example, various filtering methods may include deblocking filtering, sample adaptive offset, adaptive loop filtering, and bilateral filtering.

[0372] The reconstructed picture stored in the DPB of the memory 17060 can be used as a reference picture in the inter-frame predictor 17070. The memory 17060 can store motion information about a block in the current picture from which motion information is derived (or decoded) and / or motion information about a block in an already reconstructed picture. The stored motion information can be transmitted to the inter-frame predictor 17070 to be used as motion information about spatially neighboring blocks or motion information about temporally neighboring blocks. The memory 17060 can store reconstructed samples of the reconstructed block in the current picture and transmit the reconstructed samples to the intra-frame predictor 17080.

[0373] In this disclosure, Figure 15 The embodiments described for the filter 15070, the inter-frame predictor 15090, and the intra-frame predictor 15100 of the encoder 15000 may be applied to the filter 17050, the inter-frame predictor 17070, and the intra-frame predictor 17080 of the decoder 17000, respectively, in the same or corresponding manner.

[0374] At least one of the above-mentioned prediction, inverse transform, and inverse quantization processes may be skipped. For example, for a block to which pulse code modulation (PCM) is applied, the prediction, inverse transform, and inverse quantization processes may be skipped, and the values ​​of the decoded samples may be used as samples of the reconstructed image.

[0375] Occupancy map decompression (16003)

[0376] This is the inverse process of the occupancy map compression described above. Occupancy map decompression is the process of reconstructing the occupancy map by decompressing the occupancy map bitstream.

[0377] Auxiliary patch information decompression (16004)

[0378] The auxiliary patch information can be reconstructed by performing the inverse process of the auxiliary patch information compression described above and decoding the compressed auxiliary patch information bitstream.

[0379] Geometry Reconstruction (16005)

[0380] This is the inverse process of the geometric image generation described above. Initially, patches are extracted from the geometric image using the reconstructed occupancy map, the 2D position / size information about the patches included in the auxiliary patch information, and the information about the mapping between blocks and patches. Then, a point cloud is reconstructed in 3D space based on the geometric image of the extracted patches and the 3D position information about the patches included in the auxiliary patch information. When the geometric value corresponding to a point (u, v) within the patch is g(u, v), and the position coordinates of the patch on the normal, tangential, and bitangential axes of the 3D space are (δ0, s0, r0), the normal, tangential, and bitangential coordinates δ(u, v), s(u, v), and r(u, v) in the 3D space mapped to the position of the point (u, v) can be expressed as follows.

[0381] δ(u,v)=δ0+g(u,v)

[0382] s(u,v)=s0+u

[0383] r(u,v)=r0+v

[0384] Smooth (16006)

[0385] Like smoothing in the above-described encoding process, smoothing is a process for eliminating discontinuities that may appear at patch boundaries due to image quality degradation occurring during the compression process.

[0386] Texture reconstruction (16007)

[0387] Texture reconstruction is a process of reconstructing a color point cloud by assigning color values ​​to each point that makes up the smoothed point cloud. This is performed by assigning color values ​​corresponding to the texture image pixels at the same positions in the geometric image in 2D space to points in the point cloud corresponding to the same positions in 3D space, based on the mapping information between the geometric image and the point cloud reconstructed in the geometric reconstruction process described above.

[0388] Color Smoothing (16008)

[0389] Color smoothing is similar to the process of geometric smoothing described above. Color smoothing is a process used to eliminate discontinuities that may appear at patch boundaries due to image quality degradation that occurs during the compression process. Color smoothing can be performed by the following operations:

[0390] ① Use a KD tree or other method to calculate the neighboring points of each point constituting the reconstructed point cloud. The neighboring point information calculated in the above-mentioned geometric smoothing process can be used.

[0391] ② Determine whether each point is on the patch boundary. These operations can be performed based on the boundary information calculated in the above-mentioned geometric smoothing process.

[0392] ③ Check the distribution of color values ​​of neighboring points on the edge and determine whether smoothing should be performed. For example, when the entropy of the luminance value is less than or equal to a threshold local entry (there are many similar luminance values), it can be determined that the corresponding portion is not an edge portion, and smoothing can be performed. As a smoothing method, the color value of the point can be replaced by the average color value of the neighboring points.

[0393] Figure 18 1 is a flowchart illustrating the operation of a transmitting apparatus for compressing and transmitting V-PCC-based point cloud data according to an embodiment of the present disclosure.

[0394] The transmitting device according to the embodiment may correspond to Figure 1 The sending device, Figure 4The encoding process, and Figure 15 2D video / image encoder, or perform some / all of its operations. Each component of the transmitting device may correspond to software, hardware, a processor and / or a combination thereof.

[0395] The operation process of the sending terminal using V-PCC to compress and transmit point cloud data can be performed as shown in the figure.

[0396] The point cloud data transmitting device according to the embodiment may be referred to as a transmitting device or a transmitting system.

[0397] Patch generator 18000 generates patches for 2D image mapping of point clouds based on input point cloud data. Patch generation generates patch information and / or auxiliary patch information. This generated patch information and / or auxiliary patch information can be used in geometric image generation, texture image generation, smoothing, and smoothed geometric reconstruction.

[0398] The patch packer 18001 performs a patch packing process to map the patches generated by the patch generator 18000 into a 2D image. For example, one or more patches may be packed. As a result of the patch packing, an occupancy map may be generated. The occupancy map may be used in processes such as geometry image generation, geometry image filling, texture image filling, and / or smooth geometry reconstruction.

[0399] The geometric image generator 18002 generates a geometric image based on the point cloud data, patch information (or auxiliary patch information) and / or occupancy map. The generated geometric image is pre-processed by the encoding pre-processor 18003 and then encoded into a bitstream by the video encoder 18006.

[0400] The encoding preprocessor 18003 may include an image filling process. In other words, some spaces in the generated geometry image and the generated texture image may be filled with meaningless data. The encoding preprocessor 18003 may also include a group dilation process for the generated texture image or the texture image on which image filling has been performed.

[0401] The geometry reconstructor 18010 reconstructs a 3D geometric image based on the geometry bitstream, auxiliary patch information, and / or occupancy map encoded by the video encoder 18006 .

[0402] The smoother 18009 smoothes the 3D geometric image reconstructed and output by the geometry reconstructor 18010 based on the auxiliary patch information, and outputs the smoothed 3D geometric image to the texture image generator 18004 .

[0403] The texture image generator 18004 can generate a texture image based on smoothed 3D geometry, point cloud data, patches (or packed patches), patch information (or auxiliary patch information), and / or an occupancy map. The generated texture image can be pre-processed by the encoding pre-processor 18003 and then encoded into a video bitstream by the video encoder 18006.

[0404] The metadata encoder 18005 can encode the auxiliary patch information into a metadata bitstream.

[0405] The video encoder 18006 can encode the geometric image and texture image output from the encoding preprocessor 18003 into corresponding video bit streams, and can encode the occupancy map into a video bit stream. Figure 15 The 2D video / image encoder encodes each input image.

[0406] The multiplexer 18007 multiplexes the video bitstream of geometry, the video bitstream of texture image, the video bitstream of occupancy map output from the video encoder 18006 and the bitstream of metadata (including auxiliary patch information) output from the metadata encoder 18005 into one bitstream.

[0407] The transmitter 18008 transmits the bit stream output from the multiplexer 18007 to the receiving side. Alternatively, a file / segment encapsulator may be provided between the multiplexer 18007 and the transmitter 18008, and the bit stream output from the multiplexer 18007 may be encapsulated into a file and / or segment form and output to the transmitter 18008.

[0408] Figure 18 The patch generator 18000, patch packer 18001, geometric image generator 18002, texture image generator 18004, metadata encoder 18005 and smoother 18009 can correspond to patch generation 14000, patch packing 14001, geometric image generation 14002, texture image generation 14003, auxiliary patch information compression 14005 and smoothing 14004 respectively. Figure 18 The encoding preprocessor 18003 may include Figure 4 Image fillers 14006 and 14007 and group expander 14008, and Figure 18 The video encoder 18006 may include Figure 4 Video compressors 14009, 14010 and 14011 and / or entropy compressor 14012. Figure 18 The description part, refer to Figures 4 to 15 The above blocks may be omitted or may be replaced by blocks having similar or identical functions. In addition, Figure 18 Each block shown can be used as at least one of a processor, software, or hardware. Alternatively, the generated geometry, texture image, occupancy map video bitstream, and metadata bitstream of the auxiliary patch information can be formed into one or more track data in a file, or encapsulated into fragments, and sent to the receiving side via a transmitter.

[0409] Procedure for operating the receiving device

[0410] Figure 19 is a flowchart illustrating the operation of a receiving apparatus for receiving and restoring V-PCC-based point cloud data according to an embodiment.

[0411] The receiving device according to the embodiment may correspond to Figure 1 receiving device, Figure 16 The decoding process and Figure 17 2D video / image encoder, or perform some / all of its operations. Each component of the receiving device may correspond to software, hardware, a processor and / or a combination thereof.

[0412] The operation of the receiving terminal using V-PCC to receive and reconstruct point cloud data can be performed as shown in the figure. The operation of the V-PCC receiving terminal can follow Figure 18 The reverse process of the operation of the V-PCC sending terminal.

[0413] The point cloud data receiving device according to the embodiment may be referred to as a receiving device, a receiving system, etc.

[0414] The receiver receives a point cloud bitstream (i.e., a compressed bitstream), and the demultiplexer 19000 demultiplexes the received point cloud bitstream into a texture image bitstream, a geometry image bitstream, an occupancy map image bitstream, and a metadata bitstream (i.e., auxiliary patch information). The demultiplexed texture image, geometry image, and occupancy map image bitstreams are output to the video decoder 19001, and the metadata bitstream is output to the metadata decoder 19002.

[0415] According to an embodiment, when Figure 18 When the sending device is provided with a file / fragment encapsulator, the file / fragment decapsulator is provided in Figure 19 between the receiver of the receiving device and the demultiplexer 19000. In this case, the sending device encapsulates and sends the point cloud bitstream in the form of files and / or segments, and the receiving device receives and decapsulates the files and / or segments containing the point cloud bitstream.

[0416] The video decoder 19001 decodes the bit stream of the geometric image, the bit stream of the texture image and the bit stream of the occupancy map image into the geometric image, the texture image and the occupancy map image respectively. Figure 17 The metadata decoder 19002 decodes the metadata bitstream into auxiliary patch information and outputs the information to the geometry reconstructor 19003.

[0417] The geometry reconstructor 19003 reconstructs 3D geometry based on the geometry image, occupancy map and / or auxiliary patch information output from the video decoder 19001 and the metadata decoder 19002 .

[0418] The smoother 19004 smoothes the 3D geometry reconstructed by the geometry reconstructor 19003 .

[0419] Texture reconstructor 19005 reconstructs textures using the texture image and / or smoothed 3D geometry output from video decoder 19001. Specifically, texture reconstructor 19005 reconstructs a color point cloud image / image by assigning color values ​​to the smoothed 3D geometry using the texture image. Subsequently, to improve objective / subjective visual quality, additional color smoothing may be performed on the color point cloud image / image by color smoother 19006. After rendering in point cloud renderer 19007, the modified point cloud image / image derived from these operations is displayed to the user. In some cases, color smoothing may be omitted.

[0420] The above blocks may be omitted or may be replaced by blocks having similar or identical functions. Figure 19 Each block shown may function as at least one of a processor, software, and hardware.

[0421] Figure 20 An exemplary architecture for V-PCC based storage and streaming of point cloud data is shown, according to an embodiment.

[0422] Figure 20 Part / all of the system may include Figure 1 The sending device and the receiving device, Figure 4 The encoding process, Figure 15 2D video / image encoder, Figure 16 Decoding processing, Figure 18 The sending device and / or Figure 19 Each component in the figure may correspond to software, hardware, processor and / or a combination thereof.

[0423] Figure 20The overall architecture for storing or streaming point cloud data compressed according to video-based point cloud compression (V-PCC) is shown. The process of storing and streaming point cloud data may include acquisition processing, encoding processing, transmission processing, decoding processing, rendering processing and / or feedback processing.

[0424] Embodiments propose a method for efficiently providing point cloud media / content / data.

[0425] In order to efficiently provide point cloud media / content / data, the point cloud acquirer 20000 can acquire point cloud video. For example, one or more cameras can acquire point cloud data by capturing, arranging or generating a point cloud. Through this acquisition process, a point cloud video including the 3D position of each point (which can be represented by x, y and z position values, etc.) (hereinafter referred to as geometry) and the attributes of each point (color, reflectivity, transparency, etc.) can be acquired. For example, a Polygon file format (PLY) (or Stanford triangle format) file containing a point cloud video can be generated. For point cloud data with multiple frames, one or more files can be acquired. In this process, point cloud related metadata (for example, metadata related to capture, etc.) can be generated.

[0426] Post-processing for improving the quality of the content may be required for the captured point cloud video. In the video capture process, the maximum / minimum depth can be adjusted within the range provided by the camera device. Even after the adjustment, point data of unwanted areas may still exist. Therefore, post-processing can be performed to remove unwanted areas (for example, background) or identify connected spaces and fill spatial holes. In addition, the point cloud extracted from cameras sharing a spatial coordinate system can be integrated into a piece of content by transforming each point into a global coordinate system based on the position coordinates of each camera obtained through a calibration process. In this way, a point cloud video with a high density of points can be obtained.

[0427] The point cloud preprocessor 20001 can generate one or more frames of a point cloud video. Generally, a frame can be a unit representing an image at a specific time interval. Furthermore, when dividing the points comprising the point cloud video into one or more patches and mapping them to a 2D plane, the point cloud preprocessor 20001 can generate an occupancy map frame with a value of 0 or 1, which is a binary map indicating the presence or absence of data at corresponding locations in the 2D plane. Here, a patch is a collection of points comprising the point cloud video. In this context, points belonging to the same patch are adjacent to each other in 3D space and, when mapped to a 2D image, are mapped to the same flat surface of a 6-sided bounding box. Furthermore, the point cloud preprocessor 20001 can generate a geometric frame in the form of a depth map that represents information about the position (geometry) of each point comprising the point cloud video on a patch-by-patch basis. The point cloud preprocessor 20001 can also generate a texture frame that represents color information about each point comprising the point cloud video on a patch-by-patch basis. In this process, metadata required to reconstruct a point cloud from each patch can be generated. The metadata may include information about the patches (auxiliary information or auxiliary patch information), such as the position and size of each patch in 2D / 3D space. These pictures / frames may be generated continuously in time sequence to construct a video stream or metadata stream.

[0428] The point cloud video encoder 20002 can encode one or more video streams related to a point cloud video. A video may include multiple frames, and a frame may correspond to a still image / picture. In this disclosure, a point cloud video may include point cloud images / frames / pictures, and the term "point cloud video" and "point cloud video / frame / picture" may be used interchangeably. The point cloud video encoder 20002 can perform video-based point cloud compression (V-PCC) processing. For compression and coding efficiency, the point cloud video encoder 20002 may perform a series of processes such as prediction, transform, quantization, and entropy coding. The encoded data (encoded video / image information) may be output in the form of a bitstream. Based on the V-PCC processing, as described below, the point cloud video encoder 20002 can encode the point cloud video by separating it into a geometry video, an attribute video, an occupancy map video, and metadata (e.g., information about patches). The geometry video may include a geometry image, the attribute video may include an attribute image, and the occupancy map video may include an occupancy map image. Patch data, as auxiliary information, may include patch-related information. The attribute video / image may include a texture video / image.

[0429] The point cloud image encoder 20003 can encode one or more images related to the point cloud video. The point cloud image encoder 20003 can perform video-based point cloud compression (V-PCC) processing. For compression and coding efficiency, the point cloud image encoder 20003 can perform a series of processes such as prediction, transformation, quantization, and entropy coding. The encoded image can be output in the form of a bitstream. Based on the V-PCC processing, as described below, the point cloud image encoder 20003 can encode the point cloud image by dividing the point cloud image into a geometry image, an attribute image, an occupancy map image, and metadata (e.g., information about patches).

[0430] According to an embodiment, the point cloud video encoder 20002, the point cloud image encoder 20003, the point cloud video decoder 20006 and the point cloud image decoder 20008 can be performed by one encoder / decoder as described above and can be performed along separate paths as shown in the figure.

[0431] In the file / segment encapsulator 20004, the encoded point cloud data and / or point cloud-related metadata may be encapsulated into files or segments for streaming. Here, the point cloud-related metadata may be received from a metadata processor (not shown), etc. The metadata processor may be included in the point cloud video / image encoder 20002 / 20003, or may be configured as a separate component / module. The file / segment encapsulator 20004 may encapsulate the corresponding video / image / metadata in a file format such as ISOBMFF or in the form of DASH segments. Depending on the embodiment, the file / segment encapsulator 20004 may include the point cloud metadata in a file format. The point cloud-related metadata may be included in boxes at various levels of the ISOBMFF file format, for example, or as data in a separate track within the file. Depending on the embodiment, the file / segment encapsulator 20004 may encapsulate the point cloud-related metadata into a file.

[0432] The file / segment encapsulator 20004 according to an embodiment can store a bitstream or separate bitstreams in one or more tracks in a file, and can also encapsulate signaling information used for the operation. In addition, the atlas stream (or patch stream) included in the bitstream can be stored as a track in the file and can store related signaling information. In addition, the SEI message present in the bitstream can be stored in the track of the file and can store related signaling information.

[0433] A transmission processor (not shown) may perform transmission processing of the encapsulated point cloud data according to a file format. The transmission processor may be included in a transmitter (not shown) or may be configured as a separate component / module. The transmission processor may process the point cloud data according to a transmission protocol. The transmission processing may include processing for transmission via a broadcast network and processing for transmission via broadband. Depending on the embodiment, the transmission processor may receive point cloud-related metadata along with the point cloud data from the metadata processor and perform transmission processing of the point cloud video data.

[0434] The transmitter can transmit the point cloud bitstream or the file / segment including the bitstream to the receiver (not shown) of the receiving device via a digital storage medium or a network. For transmission, processing according to any transmission protocol can be performed. The data processed for transmission can be transmitted via a broadcast network and / or by broadband. The data can be transmitted to the receiving side in an on-demand manner. The digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD and SSD. The transmitter may include an element for generating a media file in a predetermined file format, and may include an element for transmission via a broadcast / communication network. The receiver may extract the bitstream and send the extracted bitstream to a decoder.

[0435] A receiver can receive point cloud data transmitted by a point cloud data transmitting device according to the present disclosure. Depending on the transmission channel, the receiver can receive the point cloud data via a broadcast network or broadband. Alternatively, the point cloud data can be received via a digital storage medium. The receiver can include processing to decode the received data and render the data according to the user's viewport.

[0436] A receiving processor (not shown) can process the received point cloud video data according to the transmission protocol. The receiving processor can be included in the receiver or configured as a separate component / module. The receiving processor can reverse the processing of the transmitting processor described above to correspond to the transmission processing performed on the transmitting side. The receiving processor can transmit the acquired point cloud video to the file / segment decapsulator 20005 and transmit the acquired point cloud-related metadata to the metadata parser.

[0437] The file / segment decapsulator 20005 can decapsulate the point cloud data received from the receiving processor in the form of a file. The file / segment decapsulator 20005 can decapsulate the file according to ISOBMFF, etc., and can obtain a point cloud bitstream or point cloud-related metadata (or a separate metadata bitstream). The obtained point cloud bitstream can be transmitted to the point cloud video decoder 20006 and the point cloud image decoder 20008, and the obtained point cloud video-related metadata (metadata bitstream) can be transmitted to a metadata processor (not shown). The point cloud bitstream may include metadata (metadata bitstream). The metadata processor may be included in the point cloud video decoder 20006 or may be configured as a separate component / module. The point cloud video-related metadata obtained by the file / segment decapsulator 20005 may take the form of a box or track in the file format. If necessary, the file / segment decapsulator 20005 may receive the metadata required for decapsulation from the metadata processor. The point cloud related metadata may be transmitted to the point cloud video decoder 20006 and / or the point cloud image decoder 20008 and used in the point cloud decoding process, or may be transmitted to the renderer 20009 and used in the point cloud rendering process.

[0438] The point cloud video decoder 20006 can receive a bitstream and decode the video / image by performing operations corresponding to those of the point cloud video encoder 20002. In this case, as described below, the point cloud video decoder 20006 can decode the point cloud video by separating the point cloud video into a geometry video, an attribute video, an occupancy map video, and auxiliary patch information. The geometry video may include a geometry image, the attribute video may include an attribute image, and the occupancy map video may include an occupancy map image. The auxiliary information may include auxiliary patch information. The attribute video / image may include a texture video / image.

[0439] The point cloud image decoder 20008 may receive the bitstream and perform an inverse process corresponding to the operation of the point cloud image encoder 20003. In this case, the point cloud image decoder 20008 may divide the point cloud image into a geometry image, an attribute image, an occupancy map image, and metadata (which is, for example, auxiliary patch information) to decode them.

[0440] 3D geometry can be reconstructed based on the decoded geometry video / image, occupancy map, and auxiliary patch information, and then smoothed. A color point cloud image / screen can be reconstructed by assigning color values ​​to the smoothed 3D geometry based on the texture video / image. The renderer 20009 can render the reconstructed geometry and color point cloud image / screen. The rendered video / image can be displayed on a display. All or part of the rendering result can be displayed to the user via a VR / AR display or a typical display.

[0441] The sensor / tracker (sensing / tracking) 20007 acquires orientation information and / or user viewport information from the user or receiving side and transmits the orientation information and / or user viewport information to the receiver and / or transmitter. The orientation information may represent information about the position, angle, movement, etc. of the user's head, or information about the position, angle, movement, etc. of the device through which the user is viewing the video / image. Based on this information, information about the area in 3D space that the user is currently viewing (i.e., viewport information) can be calculated.

[0442] Viewport information may be information about the area in 3D space that a user is currently viewing through a device or HMD. A device, such as a display, may extract the viewport area based on orientation information, the vertical or horizontal FOV supported by the device, and so on. Orientation or viewport information may be extracted or calculated on the receiving side. The orientation or viewport information analyzed on the receiving side may be sent to the sending side via a feedback channel.

[0443] Based on the orientation information and / or viewport information indicating the area currently viewed by the user acquired by the sensor / tracker 20007, the receiver can effectively extract or decode only the media data for the specific area (i.e., the area indicated by the orientation information and / or viewport information) from the file. Furthermore, based on the orientation information and / or viewport information acquired by the sensor / tracker 20007, the transmitter can effectively encode only the media data for the specific area (i.e., the area indicated by the orientation information and / or viewport information), or generate and transmit the file thereof.

[0444] The renderer 20009 can render the decoded point cloud data in 3D space. The rendered video / image can be displayed on a display. The user can view all or part of the rendered result on a VR / AR display or a typical display.

[0445] Feedback processing may include transmitting various feedback information that can be obtained during the rendering / display process to a decoder on the sending or receiving side. Feedback processing can provide interactivity when consuming point cloud data. Depending on the embodiment, head orientation information, viewport information indicating the area currently viewed by the user, etc. may be transmitted to the sending side during feedback processing. Depending on the embodiment, the user can interact with content implemented in a VR / AR / MR / autonomous driving environment. In this case, information related to the interaction may be transmitted to the sending side or service provider during feedback processing. Depending on the embodiment, feedback processing may be skipped.

[0446] Depending on the implementation, the feedback information can be not only sent to the sending side but also consumed by the receiving side. That is, the receiving side can perform decapsulation, decoding, and rendering based on the feedback information. For example, point cloud data related to the area currently being viewed by the user can be prioritized for decapsulation, decoding, and rendering based on orientation information and / or viewport information.

[0447] Figure 21 is an exemplary block diagram of an apparatus for storing and transmitting point cloud data according to an embodiment.

[0448] Figure 21 A point cloud system according to an embodiment is shown. Figure 21 Part / all of the system may include Figure 1 The sending device and the receiving device, Figure 4 The encoding process, Figure 15 2D video / image encoder, Figure 16 Decoding processing, Figure 18 The sending device and / or Figure 19 In addition, it may include or correspond to Figure 20 part / all of the system.

[0449] The point cloud data transmitting device according to the embodiment may be configured as shown in the figure. Each element of the transmitting device may be a module / unit / component / hardware / software / processor.

[0450] The point cloud geometry, attributes, occupancy maps, auxiliary data (auxiliary information), and mesh data can each be configured as a separate stream or stored in different tracks in the file. In addition, they can be included in separate fragments.

[0451] The point cloud acquirer 21000 acquires a point cloud. For example, one or more cameras may acquire point cloud data by capturing, arranging, or generating a point cloud. Through this acquisition process, point cloud data including the 3D position of each point (which may be represented by x, y, and z position values, etc.) (hereinafter referred to as geometry) and the attributes of each point (color, reflectivity, transparency, etc.) may be acquired. For example, a Polygon file format (PLY) (or Stanford triangle format) file including point cloud data may be generated. For point cloud data having multiple frames, one or more files may be acquired. In this process, point cloud-related metadata (e.g., metadata related to capture, etc.) may be generated. The patch generator 21001 generates patches from the point cloud data. The patch generator 21001 generates point cloud data or point cloud video into one or more pictures / frames. A picture / frame may generally represent a unit representing an image at a specific time interval. When the points comprising a point cloud video are divided into one or more patches (a collection of points comprising a point cloud video, where points belonging to the same patch are adjacent to each other in 3D space and are mapped in the same direction between the flat surfaces of a 6-sided bounding box when mapped to a 2D image) and mapped to a 2D plane, an occupancy map frame / frame can be generated as a binary image, indicating the presence or absence of data at corresponding locations in the 2D plane as 0 or 1. Furthermore, a geometry frame / frame in the form of a depth map can be generated, representing information about the position (geometry) of each point comprising the point cloud video on a patch-by-patch basis. A texture frame / frame can also be generated, representing color information about each point comprising the point cloud video on a patch-by-patch basis. In this process, metadata required to reconstruct a point cloud from each patch can be generated. This metadata can include information about the patches, such as their position and size in 2D / 3D space. These frames / frames can be generated sequentially in time to construct a video stream or metadata stream.

[0452] Additionally, patches can be used for 2D image mapping. For example, point cloud data can be projected onto the faces of a cube. After the patch is generated, a geometric image, one or more attribute images, an occupancy map, auxiliary data, and / or mesh data can be generated based on the generated patch.

[0453] The geometric image generation, attribute image generation, occupancy map generation, auxiliary data generation, and / or mesh data generation are performed by the point cloud preprocessor 20001 or a controller (not shown). The point cloud preprocessor 20001 may include a patch generator 21001, a geometric image generator 21002, an attribute image generator 21003, an occupancy map generator 21004, an auxiliary data generator 21005, and a mesh data generator 21006.

[0454] The geometry image generator 21002 generates a geometry image based on the results of patch generation. Geometry represents points in 3D space. The geometry image is generated using an occupancy map and includes information related to 2D image packing of patches, auxiliary data (including patch data), and / or patch-based mesh data. The geometry image also includes information such as the depth (e.g., near, far) of the patch generated after patch generation.

[0455] The attribute image generator 21003 generates an attribute image. For example, an attribute may represent a texture. The texture may be a color value matched to each point. Depending on the embodiment, an image including multiple attributes of the texture (e.g., color and reflectivity) (N attributes) may be generated. The multiple attributes may include material information and reflectivity. Depending on the embodiment, the attributes may also include information indicating the color, which may vary depending on the viewing angle and light even for the same texture.

[0456] The occupancy map generator 21004 generates an occupancy map from the patch. The occupancy map includes information indicating whether data exists in a pixel (eg, corresponding to a geometry or attribute image).

[0457] The auxiliary data generator 21005 generates auxiliary data (or auxiliary information) including information about the patch. That is, the auxiliary data represents metadata about the patch of the point cloud object. For example, it may represent information such as the normal vector of the patch. Specifically, the auxiliary data may include information required to reconstruct the point cloud from the patch (e.g., information about the position and size of the patch in 2D / 3D space, as well as projection (normal) plane identification information, patch mapping information, etc.).

[0458] The mesh data generator 21006 generates mesh data from the patch. A mesh represents the connection between adjacent points. For example, it can represent data in the shape of a triangle. For example, mesh data refers to the connectivity between points.

[0459] The point cloud pre-processor 20001 or controller generates metadata related to patch generation, geometric image generation, attribute image generation, occupancy map generation, auxiliary data generation and mesh data generation.

[0460] The point cloud transmitting device performs video encoding and / or image encoding in response to the results generated by the point cloud preprocessor 20001. The point cloud transmitting device can generate point cloud image data as well as point cloud video data. Depending on the embodiment, the point cloud data can include only video data, only image data, and / or both video data and image data.

[0461] The video encoder 21007 performs geometric video compression, attribute video compression, occupancy map video compression, auxiliary data compression, and / or mesh data compression. The video encoder 21007 generates a video stream containing encoded video data.

[0462] Specifically, in geometry video compression, point cloud geometry video data is encoded. In attribute video compression, attribute video data of the point cloud is encoded. In auxiliary data compression, auxiliary data associated with the point cloud video data is encoded. In mesh data compression, mesh data of the point cloud video data is encoded. Each operation of the point cloud video encoder can be performed in parallel.

[0463] The image encoder 21008 performs geometric image compression, attribute image compression, occupancy map image compression, auxiliary data compression, and / or mesh data compression. The image encoder generates an image containing encoded image data.

[0464] Specifically, in geometric image compression, point cloud geometric image data is encoded. In attribute image compression, attribute image data of the point cloud is encoded. In auxiliary data compression, auxiliary data associated with the point cloud image data is encoded. In mesh data compression, mesh data associated with the point cloud image data is encoded. Each operation of the point cloud image encoder can be performed in parallel.

[0465] The video encoder 21007 and / or the image encoder 21008 may receive metadata from the point cloud pre-processor 21001. The video encoder 21007 and / or the image encoder 21008 may perform various encoding processes based on the metadata.

[0466] The file / segment encapsulator 21009 encapsulates video streams and / or images in the form of files and / or segments. The file / segment encapsulator 21009 performs video track encapsulation, metadata track encapsulation, and / or image encapsulation.

[0467] In video track encapsulation, one or more video streams can be encapsulated into one or more tracks.

[0468] In metadata track encapsulation, metadata related to video streams and / or images can be encapsulated in one or more tracks. Metadata includes data related to the content of the point cloud data. For example, it can include initial viewing orientation metadata. Depending on the implementation, metadata can be encapsulated in the metadata track or can be encapsulated together in the video track or image track.

[0469] In image packaging, one or more images can be packaged into one or more tracks or projects.

[0470] For example, according to an embodiment, when four video streams and two images are input to the encapsulator, the four video streams and two images may be encapsulated in one file.

[0471] The file / segment packager 21009 may receive metadata from the point cloud pre-processor 21001. The file / segment packager 21009 may perform packaging based on the metadata.

[0472] The files and / or segments generated by the file / segment encapsulator 21009 are sent by a point cloud sending device or sender. For example, the segments may be transmitted according to a DASH-based protocol.

[0473] The transmitter can transmit the point cloud bitstream or a file / segment containing the bitstream to a receiver in a receiving device via a digital storage medium or a network. For transmission, processing according to any transmission protocol can be performed. Processed data can be transmitted via a broadcast network and / or broadband. Data can be transmitted to the receiving end on an on-demand basis. Digital storage media can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD.

[0474] The file / segment encapsulator 21009 according to an embodiment can divide and store a bitstream or separate bitstreams into one or more tracks in a file, and can encapsulate signaling information therefor. In addition, the patch (or atlas) stream included in the bitstream can be stored as a track in the file, and related signaling information can be stored. In addition, the SEI message present in the bitstream can be stored in the track of the file, and related signaling information can be stored.

[0475] The transmitter may include elements for generating a media file in a predetermined file format and may include elements for transmission via a broadcast / communication network. The transmitter receives orientation information and / or viewport information from a receiver. The transmitter may transmit the obtained orientation information and / or viewport information (or information selected by a user) to the point cloud preprocessor 21001, the video encoder 21007, the image encoder 21008, the file / segment encapsulator 21009, and / or the point cloud encoder. Based on the orientation information and / or viewport information, the point cloud encoder may encode all point cloud data or the point cloud data indicated by the orientation information and / or viewport information. Based on the orientation information and / or viewport information, the file / segment encapsulator may encapsulate all point cloud data or the point cloud data indicated by the orientation information and / or viewport information. Based on the orientation information and / or viewport information, the transmitter may transmit all point cloud data or the point cloud data indicated by the orientation information and / or viewport information.

[0476] For example, the point cloud preprocessor 21001 may perform the aforementioned operations on all point cloud data or on point cloud data indicated by orientation information and / or viewport information. The video encoder 21007 and / or the image encoder 21008 may perform the aforementioned operations on all point cloud data or on point cloud data indicated by orientation information and / or viewport information. The file / segment encapsulator 21009 may perform the aforementioned operations on all point cloud data or on point cloud data indicated by orientation information and / or viewport information. The transmitter may perform the aforementioned operations on all point cloud data or on point cloud data indicated by orientation information and / or viewport information.

[0477] Figure 22 is an exemplary block diagram of a point cloud data receiving apparatus according to an embodiment.

[0478] Figure 22 A point cloud system according to an embodiment is shown. Figure 22 Part / all of the system may include Figure 1 The sending device and the receiving device, Figure 4 The encoding process, Figure 15 2D video / image encoder, Figure 16 Decoding processing, Figure 18 The sending device and / or Figure 19 In addition, it may include or correspond to Figure 20 and Figure 21 part / all of the system.

[0479] The various components of the receiving device may be modules / units / components / hardware / software / processors. The transmitting client may receive point cloud data, point cloud bitstreams or files / fragments including a bitstream transmitted by a point cloud data transmitting device according to an embodiment. Depending on the channel used for transmission, the receiver may receive the point cloud data via a broadcast network or via broadband. Alternatively, the point cloud data may be received via a digital storage medium. The receiver may include processing for decoding the received data and rendering the received data according to a user viewport. The transmitting client (receiving processor) 22006 may perform processing on the received point cloud data according to a transmission protocol. The receiving processor may be included in the receiver or configured as a separate component / module. The receiving processor may perform the processing of the above-mentioned transmitting processor in reverse to correspond to the transmission processing performed by the transmitting side. The receiving processor may transmit the acquired point cloud data to the file / fragment decapsulator 22000 and transmit the acquired point cloud-related metadata to a metadata processor (not shown).

[0480] The sensor / tracker 22005 acquires orientation information and / or viewport information and may transmit the acquired orientation information and / or viewport information to the transmission client 22006 , the file / fragment decapsulator 22000 , the point cloud decoders 22001 and 22002 , and the point cloud processor 22003 .

[0481] The transmitting client 22006 may receive all point cloud data or the point cloud data indicated by the orientation information and / or the viewport information based on the orientation information and / or the viewport information. The file / fragment decapsulator 22000 may decapsulate all point cloud data or the point cloud data indicated by the orientation information and / or the viewport information based on the orientation information and / or the viewport information. The point cloud decoder (video decoder 22001 and / or image decoder 22002) may decode all point cloud data or the point cloud data indicated by the orientation information and / or the viewport information based on the orientation information and / or the viewport information. The point cloud processor 22003 may process all point cloud data or the point cloud data indicated by the orientation information and / or the viewport information based on the orientation information and / or the viewport information.

[0482] The file / segment decapsulator 22000 performs video track decapsulation, metadata track decapsulation, and / or image decapsulation. The file / segment decapsulator 22000 can decapsulate point cloud data received from the receiving processor in the form of a file. The file / segment decapsulator 22000 can decapsulate the file or segment according to ISOBMFF, etc. to obtain a point cloud bitstream or point cloud-related metadata (or a separate metadata bitstream). The obtained point cloud bitstream can be transmitted to the point cloud decoders 22001 and 22002, and the obtained point cloud-related metadata (or metadata bitstream) can be transmitted to a metadata processor (not shown). The point cloud bitstream may include metadata (metadata bitstream). The metadata processor may be included in the point cloud video decoder or may be configured as a separate component / module. The point cloud-related metadata obtained by the file / segment decapsulator 22000 may take the form of a box or track in the file format. If necessary, the file / segment decapsulator 22000 may receive metadata required for decapsulation from the metadata processor. The point cloud related metadata may be transmitted to the point cloud decoders 22001 and 22002 and used in the point cloud decoding process, or may be transmitted to the renderer 22004 and used in the point cloud rendering process. The file / fragment decapsulator 22000 may generate metadata related to the point cloud data.

[0483] In video track decapsulation performed by the file / fragment decapsulator 22000, video tracks contained in files and / or fragments are decapsulated. Video streams including geometry video, attribute video, occupancy maps, auxiliary data, and / or mesh data are decapsulated.

[0484] In metadata track decapsulation performed by the file / fragment decapsulator 22000 , a bitstream including metadata related to point cloud data and / or auxiliary data is decapsulated.

[0485] In the image decapsulation performed by the file / fragment decapsulator 22000, an image including a geometry image, an attribute image, an occupancy map, auxiliary data and / or mesh data is decapsulated.

[0486] The file / fragment decapsulator 22000 according to an embodiment can store a bitstream or separate bitstreams into one or more tracks in a file and can also decapsulate signaling information therefor. In addition, the atlas (patch) stream included in the bitstream can be decapsulated based on the tracks in the file and the relevant signaling information can be parsed. In addition, the SEI message present in the bitstream can be decapsulated based on the tracks in the file and the relevant signaling information can be obtained.

[0487] The video decoder 22001 performs geometry video decompression, attribute video decompression, occupancy map decompression, auxiliary data decompression, and / or mesh data decompression. The video decoder 22001 decodes the geometry video, attribute video, auxiliary data, and / or mesh data in a process corresponding to the process performed by the video encoder of the point cloud transmitting device according to the embodiment.

[0488] The image decoder 22002 performs geometric image decompression, attribute image decompression, occupancy map decompression, auxiliary data decompression, and / or mesh data decompression. The image decoder 22002 decodes the geometric image, attribute image, auxiliary data, and / or mesh data in a process corresponding to the process performed by the image encoder of the point cloud transmitting device according to the embodiment.

[0489] The video decoder 22001 and the video decoder 22002 according to the embodiment can be processed by one video / image decoder as described above and can be performed along separate paths as shown in the figure.

[0490] The video decoder 22001 and / or the image decoder 22002 may generate metadata related to the video data and / or the image data.

[0491] In the point cloud processor 22003 , geometric reconstruction and / or attribute reconstruction is performed.

[0492] In geometric reconstruction, a geometric video and / or a geometric image is reconstructed from the decoded video data and / or the decoded image data based on the occupancy map, the auxiliary data and / or the mesh data.

[0493] In attribute reconstruction, attribute video and / or attribute image are reconstructed from the decoded attribute video and / or decoded attribute image based on the occupancy map, auxiliary data, and / or mesh data. Depending on the embodiment, the attribute may be, for example, a texture. Depending on the embodiment, the attribute may represent multiple pieces of attribute information. When multiple attributes exist, the point cloud processor 22003 according to the embodiment performs multiple attribute reconstruction.

[0494] The point cloud processor 22003 may receive metadata from the video decoder 22001 , the image decoder 22002 , and / or the file / segment decapsulator 22000 , and process the point cloud based on the metadata.

[0495] The point cloud renderer 22004 renders the reconstructed point cloud. The point cloud renderer 22004 may receive metadata from the video decoder 22001, the image decoder 22002, and / or the file / segment decapsulator 22000, and render the point cloud based on the metadata.

[0496] The display displays the rendering result on an actual display device.

[0497] According to the method / device according to the embodiment, Figures 20 to 22 As shown, the sending side can encode the point cloud data into a bitstream, encapsulate the bitstream into a file and / or a fragment, and send it. The receiving side can decapsulate the file and / or fragment into a bitstream containing the point cloud, and can decode the bitstream into point cloud data. For example, the point cloud data device according to the embodiment can encapsulate point cloud data based on a file. The file can include a V-PCC track containing parameters of the point cloud, a geometry track containing geometry, an attribute track containing attributes, and an occupancy track containing an occupancy map.

[0498] In addition, the point cloud data receiving apparatus according to the embodiment decapsulates the point cloud data based on the file. The file may include a V-PCC track containing point cloud parameters, a geometry track containing geometry, an attribute track containing attributes, and an occupancy track containing an occupancy map.

[0499] The above encapsulation operation can be performed by Figure 20 File / fragment encapsulator 20004, Figure 21 The above decapsulation operation can be performed by the file / segment encapsulator 21009. Figure 20 File / Fragment Decapsulator 20005 or Figure 22 The file / fragment decapsulator 22000 is executed.

[0500] Figure 23 An exemplary structure that can be operated in conjunction with the point cloud data transmitting / receiving method / apparatus according to the embodiment is shown.

[0501] In the structure according to the embodiment, at least one of the server 23600, the robot 23100, the self-driving vehicle 23200, the XR device 23300, the smartphone 23400, the home appliance 23500, and / or the head-mounted display (HMD) 23700 is connected to the cloud network 23000. Here, the robot 23100, the self-driving vehicle 23200, the XR device 23300, the smartphone 23400, or the home appliance 23500 may be referred to as a device. In addition, the XR device 23300 may correspond to a point cloud data (PCC) device according to the embodiment, or may be operatively connected to a PCC device.

[0502] The cloud network 23000 may represent a network that constitutes a part of a cloud computing infrastructure or exists in a cloud computing infrastructure. Here, the cloud network 23000 may be configured using a 3G network, a 4G or Long Term Evolution (LTE) network, or a 5G network.

[0503] The server 23600 may be connected to at least one of the robot 23100 , the self-driving vehicle 23200 , the XR device 23300 , the smart phone 23400 , the home appliance 23500 , and / or the HMD 23700 via the cloud network 23000 , and may assist at least a portion of the processing of the connected devices 23100 to 23700 .

[0504] HMD 23700 represents one of implementation types of an XR device and / or a PCC device according to an embodiment. The HMD type device according to an embodiment includes a communication unit, a control unit, a memory, an I / O unit, a sensor unit, and a power supply unit.

[0505] Hereinafter, various embodiments of devices 23100 to 23500 to which the above-described technology is applied will be described. Figure 23 The illustrated devices 23100 to 23500 may be operatively connected / coupled to the point cloud data transmitting and receiving devices according to the above-described embodiments.

[0506] <PCC+XR>

[0507] The XR / PCC device 23300 may employ PCC technology and / or XR (AR+VR) technology, and may be implemented as an HMD, a head-up display (HUD) provided in a vehicle, a television, a mobile phone, a smart phone, a computer, a wearable device, a home appliance, a digital signage, a vehicle, a fixed robot, or a mobile robot.

[0508] The XR / PCC device 23300 can analyze 3D point cloud data or image data obtained through various sensors or from external devices and generate position data and attribute data of 3D points. Thus, the XR / PCC device 23300 can obtain information about the surrounding space or real objects and render and output XR objects. For example, the XR / PCC device 23300 can match an XR object including auxiliary information about the identified object with the identified object and output the matched XR object.

[0509] <PCC + Autonomous Driving + XR>

[0510] The autonomous driving vehicle 23200 can be implemented as a mobile robot, vehicle, unmanned aerial vehicle, etc. by applying PCC technology and XR technology.

[0511] The autonomous driving vehicle 23200 applying XR / PCC technology can represent an autonomous vehicle provided with means for providing an XR image or an autonomous vehicle as a control / interaction target in the XR image. Specifically, the autonomous driving vehicle 23200 as a control / interaction target in the XR image can be distinguished from the XR device 23300 and operatively connected thereto.

[0512] The autonomous driving vehicle 23200 having means for providing an XR / PCC image can obtain sensor information from sensors including a camera and output the generated XR / PCC image based on the obtained sensor information. For example, the autonomous driving vehicle can have a HUD and output the XR / PCC image thereto to provide an XR / PCC object corresponding to a real object or an object present on the screen to passengers.

[0513] In this case, when the XR / PCC object is output to the HUD, at least a part of the XR / PCC object can be output to overlap with a real object pointed by a passenger's eyes. On the other hand, when the XR / PCC object is output on a display provided inside the autonomous driving vehicle, at least a part of the XR / PCC object can be output to overlap with an object on the screen. For example, the autonomous driving vehicle can output an XR / PCC object corresponding to objects such as roads, other vehicles, traffic lights, traffic signs, two-wheel vehicles, pedestrians, and buildings.

[0514] Virtual reality (VR) technology, augmented reality (AR) technology, mixed reality (MR) technology, and / or point cloud compression (PCC) technology according to an embodiment are applicable to various devices.

[0515] In other words, VR technology is a display technology that only provides real-world objects, backgrounds, etc. as CG images. On the other hand, AR technology refers to a technology that displays a CG image created virtually on an image of a real object. MR technology is similar to the above-mentioned AR technology in that the virtual objects to be displayed are mixed and combined with the real world. However, MR technology is different from AR technology. AR technology makes a clear distinction between real objects and virtual objects created as CG images and uses virtual objects as supplementary objects to real objects, while MR technology treats virtual objects as objects with the same characteristics as real objects. More specifically, an example of the application of MR technology is holographic services.

[0516] Recently, VR, AR, and MR technologies are often referred to as extended reality (XR) technologies, rather than being clearly distinguished from each other. Therefore, embodiments of the present disclosure are applicable to all VR, AR, MR, and XR technologies. For these technologies, encoding / decoding based on PCC, V-PCC, and G-PCC technologies can be applied.

[0517] The PCC method / apparatus according to the embodiment may be applied to a self-driving vehicle 23200 that provides a self-driving service.

[0518] The self-driving vehicle 23200 providing the self-driving service is connected to the PCC device for wired / wireless communication.

[0519] When the point cloud data compression transmitting and receiving device (PCC device) according to an embodiment is connected to the self-driving vehicle 23200 for wired / wireless communication, the device can receive and process content data related to AR / VR / PCC services that can be provided together with the self-driving service and transmit the processed content data to the self-driving vehicle 23200. In the case where the point cloud data transmitting and receiving device is installed on the vehicle, the point cloud transmitting and receiving device can receive and process content data related to the AR / VR / PCC service according to a user input signal input through the user interface device and provide the processed content data to the user. The self-driving vehicle 23200 or the user interface device according to the embodiment can receive the user input signal. The user input signal according to the embodiment may include a signal indicating the self-driving service.

[0520] As mentioned above, Figure 1 、 Figure 4 、 Figure 18 、 Figure 20 or Figure 21The V-PCC-based point cloud video encoder projects 3D point cloud data (or content) into 2D space to generate patches. Patches are generated in 2D space by dividing the data into a geometric image representing position information (called a geometry frame or geometry patch frame) and a texture image representing color information (called an attribute frame or attribute patch frame). The geometric image and texture image are video compressed for each frame, and a video bitstream of the geometric image (called a geometry bitstream) and a video bitstream of the texture image (called an attribute bitstream) are output. In addition, auxiliary patch information (also called patch information, metadata, or atlas data), including projection plane information and patch size information about each patch (required for decoding 2D patches on the receiving side), is also video compressed, and a bitstream of the auxiliary patch information is output. In addition, an occupancy map, which indicates the presence / absence of a point as 0 or 1 for each pixel, is entropy compressed or video compressed depending on whether it is in lossless mode or lossy mode, and a video bitstream of the occupancy map (or occupancy map bitstream) is output. The compressed geometry bitstream, the compressed attribute bitstream, the compressed auxiliary patch information bitstream (also called compressed atlas bitstream) and the compressed occupancy map bitstream are multiplexed into the structure of the V-PCC bitstream.

[0521] According to the embodiment, the V-PCC bit stream can be sent to the receiving side as it is, or can be transmitted by Figure 1 、 Figure 18 、 Figure 20 or Figure 21 The file / segment encapsulator encapsulates the data in the form of a file / segment and sends it to a receiving device or stores it in a digital storage medium (e.g., USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc.). According to an embodiment of the present disclosure, the file is in the file format ISOBMFF.

[0522] Depending on the embodiment, the V-PCC bitstream may be transmitted through multiple tracks in a file, or may be transmitted through a single track. Details will be described later.

[0523] In the present disclosure, point cloud data (i.e., V-PCC data) represents a volumetric encoding of a point cloud consisting of a sequence of point cloud frames. In a point cloud sequence (which is a sequence of point cloud frames), each point cloud frame comprises a collection of points. Each point can have a 3D position (i.e., geometric information) and multiple attributes such as color, reflectivity, surface normal, etc. That is, each point cloud frame represents a collection of 3D points specified by their Cartesian coordinates (x, y, z) (i.e., position) and zero or more attributes at a specific time instance.

[0524] As used herein, the term video-based point cloud compression (V-PCC) is the same as visual volumetric video-based coding (V3C). Depending on the embodiment, the terms V-PCC and V3C may have the same meaning and may be used interchangeably.

[0525] According to an embodiment, point cloud content (or referred to as V-PCC content or V3C content) refers to volumetric media (or point cloud data) encoded using V-PCC.

[0526] According to an embodiment, the point cloud content (or referred to as a volumetric scene) represents 3D data and may be divided into (or composed of) one or more objects.

[0527] That is, a volumetric scene is an area or unit consisting of one or more objects that constitute volumetric media. Furthermore, when encapsulating a V-PCC bitstream in a file format, the area obtained by dividing the bounding box of the entire volumetric media according to a spatial criterion is referred to as a 3D spatial region. Depending on the embodiment, a 3D spatial region may be referred to as a 3D region or a spatial region.

[0528] Depending on the embodiment, an object may represent a segment of point cloud data, volumetric media, V3C content, or V-PCC content. Furthermore, an object may be divided into multiple objects based on spatial criteria, etc. In the present disclosure, each of the divided objects is referred to as a sub-object or simply an object. Depending on the embodiment, a 3D bounding box may represent information indicating the positional information of an object in 3D space, and a 2D bounding box may represent a rectangular area surrounding a patch corresponding to an object in a 2D frame. That is, the data generated after projecting an object onto a 2D plane (which is one of the processes for encoding an object) is a patch, and the box surrounding the patch may be referred to as a 2D bounding box. That is, since an object may be composed of multiple patches during the encoding process, an object is associated with a patch. Furthermore, an object in 3D space is represented by 3D bounding box information surrounding the object, and an atlas frame includes patch information corresponding to each object and 3D bounding box information in 3D space. Therefore, an object is associated with a 3D bounding box, an atlas patch, or a 3D spatial region.

[0529] According to an embodiment, during the step of encapsulating a V-PCC bitstream in a file format, the 3D bounding box of the point cloud data may be segmented into one or more 3D regions, and each segmented 3D region may include one or more objects. Furthermore, during the step of encapsulating patches in an atlas frame, the patches are collected on an object-by-object basis and encapsulated (mapped) into one or more atlas tile regions in the atlas frame. That is, a 3D region (i.e., at the file level) may be associated with one or more objects (i.e., at the bitstream level), and an object may be associated with one or more 3D regions. Furthermore, since each object is associated with one or more atlas tiles, a 3D region may be associated with one or more atlas tiles. An atlas tile (or patch) according to an embodiment represents an independently decodable rectangular region of an atlas frame.

[0530] Figure 24 Shows an example of a relationship between an object and an atlas tile (or tiles) according to an embodiment.

[0531] According to an embodiment, it is assumed that the point cloud content (or referred to as point cloud data or V3C content or V-PCC content) is divided into three objects (object #1, object #2, object #3), and the atlas frame consists of five atlas tiles (atlas tile #1, atlas tile #2, atlas tile #3, atlas tile #4, atlas tile #5).

[0532] Depending on the embodiment, 3D areas may overlap with each other. As an example, 3D (space) area #1 may include object #1, and 3D area #2 may include object #2 and object #3. As another example, 3D area #1 may include object #1 and object #2, and 3D area #2 may include object #2 and object #3. In other words, object #2 may be associated with 3D area #1 and 3D area #2. In addition, since the same object (e.g., object #2) may be included in different 3D areas (e.g., 3D area #1 and 3D area #2), the patch corresponding to object #2 may be assigned to (included in) different 3D areas (3D area #1 and 3D area #2).

[0533] According to an embodiment, the atlas frame is composed of one or more atlas tiles (or tiles). An object in the point cloud content can be associated with one or more atlas tiles in the atlas frame. For example, Figure 24In the embodiment of the present invention, assuming that the point cloud content (or point cloud data) is divided into three objects (object #1, object #2, object #3), and the atlas frame consists of five atlas tiles (atlas tile #1, atlas tile #2, atlas tile #3, atlas tile #4, atlas tile #5), object #1 may be associated with atlas tile #1 and atlas tile #2, object #2 may be associated with atlas tile #3, and object #3 may be associated with atlas tile #4 and atlas tile #5. This is merely an example for better understanding of the present disclosure, and the number of objects into which the point cloud content is divided and the atlas tile associated with each object may vary.

[0534] The present disclosure proposes a method for extracting, decoding, and rendering only a V-PCC sub-bitstream related to necessary objects from a V-PCC bitstream.

[0535] Depending on the implementation, an object may be associated with one or more atlas sub-bitstreams (or atlas substreams). Depending on the implementation, an atlas sub-bitstream may be defined as a sub-bitstream extracted from a V-PCC bitstream that includes a portion of an atlas NAL unit. In an implementation, an atlas sub-bitstream may correspond to a single atlas. If there are multiple atlases, multiple atlas sub-bitstreams may be generated. Furthermore, a video sub-bitstream may contain occupancy, geometry, and attribute components for each atlas or each atlas patch.

[0536] For example, one or more atlases (or atlas sub-bitstreams) may be included in a V-PCC bitstream to support 3DOF+ video. The present disclosure proposes a method for efficiently using one or more atlases received by a receiver. Specifically, the present disclosure proposes a method for extracting, decoding, and rendering only necessary sub-bitstreams from a V-PCC bitstream, even when multiple atlas sub-bitstreams are present.

[0537] Depending on the implementation, the atlas sub-bitstream carries a portion or all of the atlas data.

[0538] According to an embodiment, Atlas data is signaling information that includes an Atlas Sequence Parameter Set (ASPS), an Atlas Frame Parameter Set (AFPS), an Atlas Adaptation Parameter Set (AAPS), Atlas tiling information (also known as Atlas tiling information), and SEI messages, and can be referred to as metadata about Atlas. According to an embodiment, an ASPS is a syntax structure containing syntax elements that apply to zero or more complete Coded Atlas Sequences (CASs), as determined by the contents of syntax elements in the ASPS (referred to as syntax elements in each tiling (or tiling) header). According to an embodiment, an AFPS is a syntax structure containing syntax elements that apply to zero or more complete Coded Atlas frames, as determined by the contents of syntax elements in each tiling (or tiling) header. According to an embodiment, an AAPS can include camera parameters related to a portion of an Atlas sub-bitstream, such as camera position, rotation, scale, and camera model. In this disclosure, for simplicity, ASPS, AFPS, and AAPS are referred to as Atlas Parameter Sets.

[0539] Depending on the embodiment, as used herein, the term syntax element may have the same meaning as a field or a parameter.

[0540] According to an embodiment, an atlas represents a collection of 2D bounding boxes and may be a patch projected onto a rectangular frame.

[0541] According to an embodiment, an atlas frame is a 2D rectangular array of atlas samples to which a patch is projected, and additional information corresponding to a volumetric frame associated with the patch. An atlas sample is a location in the rectangular frame to which a patch associated with the atlas is projected.

[0542] According to an embodiment, an atlas frame may be divided into one or more rectangular partitions, which may be referred to as tile partitions or tiles. Alternatively, two or more tile partitions may be grouped and referred to as tiles. In other words, one or more tile partitions may constitute a tile. In the present disclosure, a tile has the same meaning as an atlas tile. A tile is a division unit of signaling information of point cloud data called an atlas. According to an embodiment, tiles in an atlas frame do not overlap with each other, and one atlas frame may include areas unrelated to the tiles (i.e., one or more tile partitions). In addition, the height and width of each tile included in one atlas frame may differ between tiles.

[0543] Depending on the embodiment, some point cloud data corresponding to a specific 3D spatial region among all point cloud data may be associated with one or more 2D regions. Thus, one 3D region may correspond to one atlas frame and may be associated with multiple 2D regions. Depending on the embodiment, a 2D region represents one or more video frames or atlas frames containing data associated with the point cloud data in the 3D region.

[0544] According to an embodiment, a patch is a collection of points that make up a point cloud, indicating that points belonging to the same patch are adjacent to each other in 3D space and are mapped along the same direction in the six-sided bounding box plane when mapping to a 2D image. A patch is signaling information about the configuration of point cloud data.

[0545] The receiving device according to the embodiment can reconstruct attribute video data, geometry video data, and occupancy video data, which are actual video data with the same presentation time based on atlas (tiles, patches).

[0546] For spatial access or partial access to point cloud data, it is necessary to access a portion of the point cloud data based on a 3D (spatial) region or object. To this end, in the present disclosure, mapping information between a 3D region and a patch, mapping information between a 3D region and an object, mapping information between an object and an atlas, or mapping information between an object and a patch is signaled.

[0547] That is, in the present disclosure, to support the decoding and / or rendering / display of a specific object in a PCC decoder / player of a receiving device, a transmitting device may generate signaling information for identifying the relationship between the object and the atlas, the relationship between the object and an atlas patch, or the relationship between the object and an atlas sub-bitstream, and transmit the signaling information to the receiving side. This signaling information may be static or time-varying. In the present disclosure, the signaling information may be transmitted via a V-PCC bitstream, via sample entries and / or samples of a file carrying the V-PCC bitstream, or in the form of metadata. Depending on the embodiment, the signaling information may be stored in samples, sample entries, sample groups, or track groups within a track, or in separate metadata tracks within a file. Specifically, a portion of the signaling information may be stored in a sample entry of a track in the form of a box or a full box. Details of the signaling information related to objects and atlas, as well as methods for storing / transmitting the signaling information, will be described later.

[0548] When the user zooms in or changes the viewport, a portion of the entire point cloud object / data may be rendered or displayed on the changed viewport. In this case, it may be more efficient for the PCC decoder / player to decode or process the video or atlas data associated with the portion of the point cloud data rendered or displayed on the user's viewport and skip decoding or processing the video or atlas data associated with the portion / area of ​​point cloud data that is not rendered or displayed. To this end, it is necessary to support partial access to point cloud data.

[0549] In order to achieve spatial access or partial access to the rendered / displayed point cloud data when the PCC decoder / player of the receiving device presents / displays the point cloud data, the sending device can send 3D region information about the point cloud (which can be static or time-varying) through a V-PCC bitstream, signal the 3D region information about the point cloud in a sample entry and / or sample of a file carrying the V-PCC bitstream, or send the 3D region information about the point cloud in the form of metadata. In the present disclosure, information signaled in a V-PCC bitstream or file or sent in the form of metadata is referred to as signaling information.

[0550] Depending on the implementation, signaling information may be stored in a sample, sample entry, sample group, or track group within a track, or in a separate metadata track within a file. Specifically, a portion of the signaling information may be stored in a sample entry in the form of a box or a full box. Details of the signaling information related to a 3D spatial region and methods for storing / transmitting the signaling information will be described later.

[0551] According to an embodiment, the signaling information may be generated by a metadata generator of the sending device (e.g., Figure 18 The signaling information may be generated by a metadata encoder 18005) and then signaled in a sample, sample entry, sample group, or track group of a file or in a separate metadata track by a file / segment encapsulator, or may be generated by a file / segment encapsulator and signaled in a sample, sample entry, sample group, or track group of a track or in a separate metadata track by a file / segment encapsulator. In the present disclosure, the signaling information may include metadata related to the point cloud data (e.g., setting values, etc.). Depending on the application, the signaling information may be defined at the system level (such as file format, HTTP Dynamic Adaptive Streaming (DASH), or MPEG Media Transport (MMT)) or at the wired interface level (such as High Definition Multimedia Interface (HDMI), DisplayPort, Video Electronics Standards Association (VESA), or CTA).

[0552] Figure 25 An example of a V-PCC bitstream structure according to other embodiments of the present disclosure is shown. In an embodiment, Figure 25 The V-PCC bitstream is composed of Figure 1 、 Figure 4 、 Figure 18 、 Figure 20 or Figure 21 Generation and output of V-PCC based point cloud video encoder.

[0553] A V-PCC bitstream containing a coded point cloud sequence (CPCS) according to an embodiment may be composed of a sample stream V-PCC unit or V-PCC units. The sample stream V-PCC unit or V-PCC unit carries V-PCC parameter set (VPS) data, an atlas bitstream, a 2D video coding occupancy map bitstream, a 2D video coding geometry bitstream, and zero or more 2D video coding attribute bitstreams.

[0554] exist Figure 25 In the example, the V-PCC bitstream may include a sample stream V-PCC header 40010 and one or more sample stream V-PCC units 40020. For simplicity, one or more sample stream V-PCC units 40020 may be referred to as a sample stream V-PCC payload. In other words, the sample stream V-PCC payload may be referred to as a sample stream V-PCC unit set. Figure 27 A detailed description of the sample stream V-PCC header 40010 is described in.

[0555] Each sample stream V-PCC unit 40021 may include V-PCC unit size information 40030 and a V-PCC unit 40040. The V-PCC unit size information 40030 indicates the size of the V-PCC unit 40040. For simplicity, the V-PCC unit size information 40030 may be referred to as a sample stream V-PCC unit header, and the V-PCC unit 40040 may be referred to as a sample stream V-PCC unit payload.

[0556] Each V-PCC unit 40040 may include a V-PCC unit header 40041 and a V-PCC unit payload 40042 .

[0557] In the present disclosure, data contained in a V-PCC unit payload 40042 is distinguished by a V-PCC unit header 40041. To this end, the V-PCC unit header 40041 includes type information indicating the type of the V-PCC unit. Depending on the type information in the V-PCC unit header 40041, each V-PCC unit payload 40042 may contain at least one of geometry video data (i.e., 2D video coding geometry bitstream), attribute video data (i.e., 2D video coding attribute bitstream), occupancy video data (i.e., 2D video coding occupancy map bitstream), atlas data, or a V-PCC parameter set (VPS).

[0558] The VPS according to an embodiment is also referred to as a sequence parameter set (SPS). These two terms can be used interchangeably.

[0559] According to an embodiment, Atlas data is signaling information including an Atlas Sequence Parameter Set (ASPS), an Atlas Frame Parameter Set (AFPS), an Atlas Adaptation Parameter Set (AAPS), Atlas Patch Group Information (or Atlas Patch Information), and an SEI message, and is referred to as an Atlas bitstream or patch data set. Furthermore, the ASPS, AFPS, and AAPS are also referred to as Atlas Parameter Sets.

[0560] Figure 26 An example of data carried by a sample stream V-PCC unit in a V-PCC bitstream according to an embodiment is shown.

[0561] exist Figure 26 In the example, the V-PCC bitstream includes a sample stream V-PCC unit carrying a V-PCC parameter set (VPS), a sample stream V-PCC unit carrying atlas data (AD), a sample stream V-PCC unit carrying occupancy video data (OVD), a sample stream V-PCC unit carrying geometric video data (GVD), and a sample stream V-PCC unit carrying attribute video data (AVD).

[0562] According to an embodiment, each sample stream V-PCC unit contains one type of V-PCC unit among VPS, AD, OVD, GVD and AVD.

[0563] A field, which is a term used in the syntax of the present disclosure described below, may have the same meaning as a parameter or an element (or syntax element).

[0564] Figure 27 An example of the syntax structure of a sample stream V-PCC header 40010 included in a V-PCC bitstream according to an embodiment is shown.

[0565] sample_stream_v-pcc_header() according to an embodiment may include an ssvh_unit_size_precision_bytes_minus1 field and an ssvh_reserved_zero_5bits field.

[0566] The value of the ssvh_unit_size_precision_bytes_minus1 field plus 1 may specify the precision (in bytes) of the ssvu_vpcc_unit_size element in all sample stream V-PCC units. The value of this field may be in the range of 0 to 7.

[0567] The ssvh_reserved_zero_5bits field is reserved for future use.

[0568] Figure 28 An example of a syntax structure of a sample stream V-PCC unit (sample_stream_vpcc_unit()) according to an embodiment is shown.

[0569] The content of each sample stream V-PCC unit is associated with the same access unit as the V-PCC unit contained in the sample stream V-PCC unit.

[0570] sample_stream_vpcc_unit() according to an embodiment may include a ssvu_vpcc_unit_size field and vpcc_unit (ssvu_vpcc_unit_size).

[0571] The ssvu_vpcc_unit_size field corresponds to Figure 25 The V-PCC unit size information 40030 specifies the size of the subsequent vpcc_unit in bytes. The number of bits used to represent the ssvu_vpcc_unit_size field is equal to (ssvh_unit_size_precision_bytes_minus1+1)*8.

[0572] vpcc_unit (ssvu_vpcc_unit_size) has a length corresponding to the value of the ssvu_vpcc_unit_size field and carries one of VPS, AD, OVD, GVD, and AVD.

[0573] Figure 29 An example of the syntax structure of a V-PCC unit according to an embodiment is shown. A V-PCC unit consists of a V-PCC unit header (vpcc_unit_header()) 40041 and a V-PCC unit payload (vpcc_unit_payload()) 40042. A V-PCC unit according to an embodiment may contain more data. In this case, it may also include a trailing_zero_8bits field. The trailing_zero_8bits field according to an embodiment is a byte corresponding to 0x00.

[0574] Figure 30 An example of the syntax structure of the V-PCC unit header 40041 according to an embodiment is shown. In an embodiment, Figure 30The vpcc_unit_header() includes a vuh_unit_type field. The vuh_unit_type field indicates the type of the corresponding V-PCC unit. The vuh_unit_type field according to an embodiment is also called a vpcc_unit_type field.

[0575] Figure 31 An example of a V-PCC unit type assigned to the vuh_unit_type field according to an embodiment is shown.

[0576] Reference Figure 31 According to an embodiment, the vuh_unit_type field set to 0 indicates that the data contained in the V-PCC unit payload of the V-PCC unit is a V-PCC parameter set (VPCC_VPS). The vuh_unit_type field set to 1 indicates that the data is atlas data (VPCC_AD). The vuh_unit_type field set to 2 indicates that the data is occupied video data (VPCC_OVD). The vuh_unit_type field set to 3 indicates that the data is geometry video data (VPCC_GVD). The vuh_unit_type field set to 4 indicates that the data is attribute video data (VPCC_AVD).

[0577] Those skilled in the art can easily change the meaning, order, deletion, addition, etc. of the values ​​assigned to the vuh_unit_type field, and thus the present disclosure will not be limited to the above-mentioned embodiments.

[0578] When the vuh_unit_type field indicates VPCC_AVD, VPCC_GVD, VPCC_OVD, or VPCC_AD, the V-PCC unit header according to an embodiment may further include a vuh_vpcc_parameter_set_id field and a vuh_atlas_id field.

[0579] The vuh_vpcc_parameter_set_id field specifies the value of vps_vpcc_parameter_set_id of the active V-PCC VPS.

[0580] The vuh_atlas_id field specifies the index of the atlas corresponding to the current V-PCC unit.

[0581] When the vuh_unit_type field indicates VPCC_AVD, the V-PCC unit header according to an embodiment may further include a vuh_attribute_index field, a vuh_attribute_partition_index field, a vuh_map_index field, and a vuh_auxiliary_video_flag field.

[0582] The vuh_attribute_index field indicates the index of attribute data carried in the attribute video data unit.

[0583] The vuh_attribute_partition_index field indicates the index of the attribute dimension group carried in the attribute video data unit.

[0584] When present, the vuh_map_index field may indicate the map index of the current geometry or attribute stream.

[0585] When the vuh_auxiliary_video_flag field is set to 1, it may indicate that the associated attribute video data unit contains only RAW and / or EOM (enhanced occupancy mode) code points. As another example, when the vuh_auxiliary_video_flag field is set to 0, it may indicate that the associated attribute video data unit may contain RAW and / or EOM code points. When the vuh_auxiliary_video_flag field is not present, its value may be inferred to be equal to 0. Depending on the embodiment, RAW and / or EOM code points are also referred to as pulse code modulation (PCM) code points.

[0586] When the vuh_unit_type field indicates VPCC_GVD, the V-PCC unit header according to an embodiment may further include a vuh_map_index field, a vuh_auxiliary_video_flag field, and a vuh_reserved_zero_12bits field.

[0587] When present, the vuh_map_index field indicates the index of the current geometry stream.

[0588] When the vuh_auxiliary_video_flag field is set to 1, it may indicate that the associated geometry video data unit contains only RAW and / or EOM code points. As another example, when the vuh_auxiliary_video_flag field is set to 0, it may indicate that the associated geometry video data unit may contain RAW and / or EOM code points. When the vuh_auxiliary_video_flag field is not present, its value may be inferred to be equal to 0. Depending on the embodiment, RAW and / or EOM code points are also referred to as PCM code points.

[0589] The vuh_reserved_zero_12bits field is reserved for future use.

[0590] If the vuh_unit_type field indicates VPCC_OVD or VPCC_AD, the V-PCC unit header according to an embodiment may further include a vuh_reserved_zero_17bits field. Otherwise, the V-PCC unit header may further include a vuh_reserved_zero_27bits field.

[0591] The vuh_reserved_zero_17bits field and the vuh_reserved_zero_27bits field are reserved for future use.

[0592] Figure 32 An example of a syntax structure of a V-PCC unit payload (vpcc_unit_payload()) according to an embodiment is shown.

[0593] Figure 32 The V-PCC unit payload may contain one of a V-PCC parameter set (vpcc_parameter_set()), an atlas sub-bitstream (atlas_sub_bitstream()), and a video sub-bitstream (video_sub_bitstream()) according to a value of the vuh_unit_type field in the V-PCC unit header.

[0594] For example, when the vuh_unit_type field indicates VPCC_VPS, the V-PCC unit payload includes vpcc_parameter_set(), which contains overall coding information about the bitstream. When the vuh_unit_type field indicates VPCC_AD, the V-PCC unit payload includes atlas_sub_bitstream(), which carries atlas data. Furthermore, depending on the embodiment, when the vuh_unit_type field indicates VPCC_OVD, the V-PCC unit payload includes an occupation video sub-bitstream (video_sub_bitstream()) that carries occupation video data. When the vuh_unit_type field indicates VPCC_GVD, the V-PCC unit payload includes a geometry video sub-bitstream (video_sub_bitstream()) that carries geometry video data. When the vuh_unit_type field indicates VPCC_AVD, the V-PCC unit payload includes an attribute video sub-bitstream (video_sub_bitstream()) that carries attribute video data.

[0595] According to an embodiment, the atlas sub-bitstream may be referred to as an atlas sub-stream, and the occupied video sub-bitstream may be referred to as an occupied video sub-stream. The geometry video sub-bitstream may be referred to as a geometry video sub-stream, and the attribute video sub-bitstream may be referred to as an attribute video sub-stream. The V-PCC unit payload according to an embodiment may conform to the format of a High Efficiency Video Coding (HEVC) Network Abstraction Layer (NAL) unit.

[0596] Figure 33 An example of a syntax structure of a V-PCC parameter set included in a V-PCC unit payload according to an embodiment is shown.

[0597] exist Figure 33 In

[15] , profile_tier_level() contains information about the V-PCC codec profile and specifies constraints on the bitstream. It represents restrictions on the capabilities required to decode the bitstream. Profiles, tiers, and levels can be used to indicate interoperability points between separate decoder implementations.

[0598] The vps_vpcc_parameter_set_id field provides an identifier of the V-PCC VPS for reference by other syntax elements.

[0599] The value of the vps_atlas_count_minus1 field plus 1 indicates the total number of atlases supported in the current bitstream.

[0600] An iteration statement is also included in the V-PCC parameter set, and the iteration statement is repeated as many times as the value of the vps_atlas_count_minus1 field (i.e., the total number of atlases). In an embodiment, in the iteration statement, j may be initialized to 0 and incremented by 1 each time the iteration statement is executed until j reaches the value of the vps_atlas_count_minus1 field + 1.

[0601] In an embodiment, the iteration statement includes the following fields. In addition to the following fields (not shown), the iteration statement may also include an atlas identifier for identifying the atlas with index j. In one embodiment, index j may be an identifier for identifying the jth atlas.

[0602] The vps_frame_width[j] field indicates the V-PCC frame width in terms of integer luma samples for the atlas with index j. This frame width is the nominal width associated with all V-PCC components of the atlas with index j.

[0603] The vps_frame_height[j] field indicates the V-PCC frame height in terms of integer luma samples of the atlas with index j. This frame height is the nominal height associated with all V-PCC components of the atlas with index j.

[0604] The vps_map_count_minus1[j] field plus 1 indicates the number of maps used to encode the geometry and attribute data of the atlas with index j.

[0605] When the vps_map_count_minus1[j] field is greater than 0, the parameter set may also include the following parameters.

[0606] Depending on the value of the vps_map_count_minus1[j] field, the parameter set may also include the following parameters.

[0607] The vps_multiple_map_streams_present_flag[j] field equal to 0 indicates that all geometry or attribute maps of the atlas with index j are placed in a single geometry or attribute video stream, respectively. The vps_multiple_map_streams_present_flag[j] field equal to 1 indicates that all geometry or attribute maps of the atlas with index j are placed in separate video streams.

[0608] If the vps_multiple_map_streams_present_flag[j] field is equal to 1, the vps_map_absolute_coding_enabled_flag[j][i] field may also be included in the parameter set. Otherwise, the vps_map_absolute_coding_enabled_flag[j][i] field may be 1.

[0609] The vps_map_absolute_coding_enabled_flag[j][i] field equal to 1 indicates that the geometry map with index i of the atlas with index j is coded without any form of map prediction. The vps_map_absolute_coding_enabled_flag[j][i] field equal to 0 indicates that the geometry map with index i of the atlas with index j is first predicted from another earlier coded map before coding.

[0610] The vps_map_absolute_coding_enabled_flag[j][0] field equal to 1 indicates that the geometry map with index 0 is coded without map prediction.

[0611] If the vps_map_absolute_coding_enabled_flag[j][i] field is 0 and i is greater than 0, the vps_map_predictor_index_diff[j][i] field may also be included in the parameter set. Otherwise, the vps_map_predictor_index_diff[j][i] field may be 0.

[0612] When the vps_map_absolute_coding_enabled_flag[j][i] field is equal to 0, the vps_map_predictor_index_diff[j][i] field is used to calculate the predictor of the geometry map with index i of the atlas with index j.

[0613] The vps_auxiliary_video_present_flag[j] field is equal to 1 to indicate that the auxiliary information of the atlas with index j, i.e., RAW or EOM patch data, can be stored in a separate video stream, called the auxiliary video stream. The vps_auxiliary_video_present_flag[j] field is equal to 0 to indicate that the auxiliary information of the atlas with index j is not stored in a separate video stream.

[0614] occupancy_information() includes occupancy video related information.

[0615] geometry_information() includes relevant geometric video information.

[0616] attribute_information() includes attribute video related information.

[0617] That is, the V-PCC parameter set may include occupancy_information(), geometry_information(), and attribute_information() for each atlas.

[0618] The V-PCC parameter set may also include a vps_extension_present_flag field.

[0619] The vps_extension_present_flag field equal to 1 specifies that the vps_extension_length field is present in the vpcc_parameter_set. The vps_extension_present_flag field equal to 0 specifies that the vps_extension_length field is not present.

[0620] The vps_extension_length_minus1 field plus 1 specifies the number of vps_extension_data_byte elements that follow this syntax element.

[0621] According to the vps_extension_length_minus1 field, extension data (vps_extension_data_byte) can also be included in the parameter set.

[0622] The vps_extension_data_byte field can have any value.

[0623] An atlas frame (or point cloud object or patch frame) that is a target of point cloud data may be divided (or split) into one or more tiles or one or more atlas tiles. According to an embodiment, a tile may represent a specific area in 3D space or a specific area in a 2D plane. In addition, a tile may be a rectangular cuboid in a bounding box, a sub-bounding box, or a portion of an atlas frame. According to other embodiments, an atlas frame may be divided into one or more rectangular partitions, which may be referred to as tile partitions or tiles. Alternatively, two or more tile partitions may be grouped and referred to as tiles. In the present disclosure, dividing an atlas frame (or point cloud object) into one or more tiles may be performed by Figure 1 Point cloud video encoder, Figure 18 Patch generator, Figure 20 Point cloud preprocessor or Figure 21 The patch generator may be executed by the , or it may be executed by a separate component / module.

[0624] Figure 34 An example of dividing an atlas frame (or patch frame) 43010 into a plurality of tiles by dividing the atlas frame (or patch frame) 43010 into one or more tile rows and one or more tile columns is shown. A tile 43030 is a rectangular area of ​​the atlas frame, and a tile group 43050 may include a plurality of tiles in the atlas frame. In the present disclosure, a tile group 43050 includes a plurality of tiles of the atlas frame, which together form a rectangular (quadrilateral) area of ​​the atlas frame. In the present disclosure, a tile and a tile group may not be distinguished from each other, and one tile group may correspond to one tile. For example, Figure 34 An example is shown in which an atlas frame (or patch frame) is divided into 24 tiles (ie, 6 tile columns and 4 tile rows) and 9 rectangular (quadrangular) tile groups.

[0625] According to other embodiments, Figure 34 In the embodiment of the present invention, tile group 43050 may be referred to as a tile, and tile 43030 may be referred to as a tile partition. The term signaling information may also be varied and referenced according to the above complementary relationship.

[0626] Figure 35 is a diagram showing an example of the structure of the Atlas sub-stream as described above. In an embodiment, Figure 35 The atlas substream conforms to the HEVC NAL unit format.

[0627] The atlas substream according to an embodiment may include a sample stream NAL header and one or more sample stream NAL units. Figure 35In the example, one or more sample stream NAL units may be referred to as a sample stream NAL payload. That is, a sample stream NAL payload may be referred to as a collection of sample stream NAL units.

[0628] According to an embodiment, one or more sample stream NAL units may be composed of a sample stream NAL unit containing an Atlas sequence parameter set (ASPS), a sample stream NAL unit containing an Atlas frame parameter set (AFPS), and one or more sample stream NAL units containing information about one or more Atlas tile groups (or Atlas tiles) and / or one or more sample stream NAL units containing one or more SEI messages.

[0629] The one or more SEI messages according to an embodiment may include a prefix SEI message and a suffix SEI message.

[0630] Figure 36 An example of a syntax structure of a sample stream NAL header (sample_stream_nal_header()) included in an atlas substream according to an embodiment is shown.

[0631] sample_stream_nal_header() according to an embodiment may include a ssnh_unit_size_precision_bytes_minus1 field and a ssnh_reserved_zero_5bits field.

[0632] The value of the ssnh_unit_size_precision_bytes_minus1 field plus 1 may specify the precision (in bytes) of the ssnu_nal_unit_size element in all sample stream NAL units. The value of this field may be in the range of 0 to 7.

[0633] The ssnh_reserved_zero_5bits field is reserved for future use.

[0634] Figure 37 An example of a syntax structure of a sample stream NAL unit (sample_stream_nal_unit()) according to an embodiment is shown.

[0635] sample_stream_nal_unit() according to an embodiment may include a ssnu_nal_unit_size field and nal_unit (ssnu_nal_unit_size).

[0636] The ssnu_nal_unit_size field specifies the size of the subsequent NAL_unit in bytes. The number of bits used to represent the ssnu_nal_unit_size field is equal to (ssnh_unit_size_precision_bytes_minus1+1)*8.

[0637] The nal_unit (ssnu_nal_unit_size) has a length corresponding to the value of the ssnu_nal_unit_size field and carries one of the Atlas Sequence Parameter Set (ASPS), the Atlas Frame Parameter Set (AFPS), the Atlas Adaptation Parameter Set (AAPS), Atlas Tile Group Information (or Atlas Tile Information), and the SEI message. That is, one sample stream NAL unit may contain one of the ASPS, AFPS, AAPS, Atlas Tile Group Information (or Atlas Tile Information), or the SEI message. Depending on the embodiment, the ASPS, AFPS, AAPS, Atlas Tile Group Information (or Atlas Tile Information), and the SEI message are referred to as Atlas data (or Atlas-related metadata).

[0638] SEI messages according to embodiments may assist in processes related to decoding, reconstruction, display, or other purposes.

[0639] Figure 38 Show Figure 37 An example of the syntax structure of nal_unit(NumBytesInNalUnit).

[0640] exist Figure 38 In NAL, NumBytesInNalUnit indicates the size of the NAL unit (in bytes). Figure 37 The value of the ssnu_nal_unit_size field.

[0641] According to an embodiment, a NAL unit may include a NAL unit header (nal_unit_header()) and a NumBytesInRbsp field. The NumBytesInRbsp field is initialized to 0 and indicates the bytes belonging to the payload of the NAL unit.

[0642] The NAL unit includes an iteration statement that is repeated as many times as the value of NumBytesInNalUnit. In an embodiment, the iteration statement includes rbsp_byte[NumBytesInRbsp++]. According to an embodiment, in the iteration statement, i is initialized to 2 and incremented by 1 each time the iteration statement is executed. The iteration statement is repeated until i reaches the value of NumBytesInNalUnit.

[0643] rbsp_byte[NumBytesInRbsp++] is the i-th byte of the raw byte sequence payload (RBSP) that carries Atlas data. The RBSP is specified as a continuous sequence of bytes. In other words, rbsp_byte[NumBytesInRbsp++] carries one of the Atlas Sequence Parameter Set (ASPS), Atlas Frame Parameter Set (AFPS), Atlas Adaptation Parameter Set (AAPS), Atlas Tile Group Information (Atlas Tile Information), and SEI messages.

[0644] Figure 39 Show Figure 38 An example of a syntax structure of a NAL unit header. The NAL unit header may include a nal_forbidden_zero_bit field, a nal_unit_type field, a nal_layer_id field, and a nal_temporal_id_plus1 field.

[0645] The nal_forbidden_zero_bit is used for error detection in NAL units and must be 0.

[0646] nal_unit_type specifies the type of RBSP data structure contained in the NAL unit. Figure 40 An example of the RBSP data structure according to the value of the nal_unit_type field is described.

[0647] nal_layer_id specifies the identifier of the layer to which an Atlas Coding Layer (ACL) NAL unit belongs or the identifier of the layer to which a non-ACL NAL unit applies.

[0648] nal_temporal_id_plus1 minus 1 specifies the temporal identifier of the NAL unit.

[0649] Figure 40 An example of the type of the RBSP data structure assigned to the nal_unit_type field is shown. That is, this figure shows the type of the nal_unit_type field in the NAL unit header of the NAL unit included in the sample stream NAL unit.

[0650] exist Figure 40 In the NAL unit, NAL_TRAIL indicates that a coded tile group of non-TSA, non-STSA trailing atlas frames is included in the NAL unit. The RBSP syntax structure of the NAL unit is atlas_tile_group_layer_rbsp() or atlas_tile_layer_rbsp(). The type class of the NAL unit is ACL. Depending on the embodiment, the tile group may be referred to as a tile.

[0651] NAL TSA indicates that the coded tile group of the TSA atlas frame is included in the NAL unit. The RBSP syntax structure of the NAL unit is atlas_tile_group_layer_rbsp() or atlas_tile_layer_rbsp(). The type class of the NAL unit is ACL.

[0652] NAL_STSA indicates that the coded tile group of the STSA atlas frame is included in the NAL unit. The RBSP syntax structure of the NAL unit is atlas_tile_group_layer_rbsp() or atlas_tile_layer_rbsp(). The type class of the NAL unit is ACL.

[0653] NAL_RADL indicates that the coded tile group of the RADL atlas frame is included in the NAL unit. The RBSP syntax structure of the NAL unit is atlas_tile_group_layer_rbsp() or atlas_tile_layer_rbsp(). The type class of the NAL unit is ACL.

[0654] NAL_RASL indicates that the coded tile group of the RASL atlas frame is included in the NAL unit. The RBSP syntax structure of the NAL unit is atlas_tile_group_layer_rbsp() or aatlas_tile_layer_rbsp(). The type class of the NAL unit is ACL.

[0655] NAL_SKIP indicates that the coded tile group of the skipped atlas frame is included in the NAL unit. The RBSP syntax structure of the NAL unit is atlas_tile_group_layer_rbsp() or atlas_tile_layer_rbsp(). The type class of the NAL unit is ACL.

[0656] NAL_RSV_ACL_6 to NAL_RSV_ACL_9 indicate that a reserved non-IRAP ACL NAL unit type is included in the NAL unit. The type class of the NAL unit is ACL.

[0657] NAL_BLA_W_LP, NAL_BLA_W_RADL, and NAL_BLA_N_LP indicate that the coded tile group of the BLA atlas frame is included in the NAL unit. The RBSP syntax structure of the NAL unit is atlas_tile_group_layer_rbsp() or atlas_tile_layer_rbsp(). The type class of the NAL unit is ACL.

[0658] NAL_GBLA_W_LP, NAL_GBLA_W_RADL, and NAL_GBLA_N_LP indicate that a coded tile group of a GBLA atlas frame may be included in a NAL unit. The RBSP syntax structure of a NAL unit is atlas_tile_group_layer_rbsp() or atlas_tile_layer_rbsp(). The type class of the NAL unit is ACL.

[0659] NAL_IDR_W_RADL and NAL_IDR_N_LP indicate that the coded tile group of the IDR atlas frame is included in the NAL unit. The RBSP syntax structure of the NAL unit is atlas_tile_group_layer_rbsp() or atlas_tile_layer_rbsp(). The type class of the NAL unit is ACL.

[0660] NAL_GIDR_W_RADL and NAL_GIDR_N_LP indicate that the coded tile group of the GIDR atlas frame is included in the NAL unit. The RBSP syntax structure of the NAL unit is atlas_tile_group_layer_rbsp() or atlas_tile_layer_rbsp(). The type class of the NAL unit is ACL.

[0661] NAL_CRA indicates that the coded tile group of the CRA atlas frame is included in the NAL unit. The RBSP syntax structure of the NAL unit is atlas_tile_group_layer_rbsp() or atlas_tile_layer_rbsp(). The type class of the NAL unit is ACL.

[0662] NAL_GCRA indicates that the coded tile group of the GCRA atlas frame is included in the NAL unit. The RBSP syntax structure of the NAL unit is atlas_tile_group_layer_rbsp() or atlas_tile_layer_rbsp(). The type class of the NAL unit is ACL.

[0663] NAL_IRAP_ACL_22 and NAL_IRAP_ACL_23 indicate that the reserved IRAP ACL NAL unit type is included in the NAL unit. The type class of the NAL unit is ACL.

[0664] NAL_RSV_ACL_24 to NAL_RSV_ACL_31 indicate that a reserved non-IRAP ACL NAL unit type is included in the NAL unit. The type class of the NAL unit is ACL.

[0665] NAL_ASPS indicates that the NAL unit includes ASPS. The RBSP syntax structure of the NAL unit is atlas_sequence_parameter_set_rbsp(). The type class of the NAL unit is non-ACL.

[0666] NAL_AFPS indicates that the NAL unit includes AFPS. The RBSP syntax structure of the NAL unit is atlas_frame_parameter_set_rbsp(). The type class of the NAL unit is non-ACL.

[0667] NAL_AUD indicates that an access unit delimiter is included in the NAL unit. The RBSP syntax structure of the NAL unit is access_unit_delimiter_rbsp(). The type class of the NAL unit is non-ACL.

[0668] NAL_VPCC_AUD indicates that a V-PCC access unit delimiter is included in the NAL unit. The RBSP syntax structure of the NAL unit is access_unit_delimiter_rbsp(). The type class of the NAL unit is non-ACL.

[0669] NAL_EOS indicates that the NAL unit type may be the end of a sequence. The RBSP syntax structure of the NAL unit is end_of_seq_rbsp(). The type class of the NAL unit is non-ACL.

[0670] NAL_EOB indicates that the NAL unit type may be the end of the bitstream. The RBSP syntax structure of the NAL unit is end_of_atlas_sub_bitstream_rbsp(). The type class of the NAL unit is non-ACL.

[0671] NAL_FD fill indication fill_data_rbsp() is included in the NAL unit. The type class of the NAL unit is non-ACL.

[0672] NAL_PREFIX_NSEI and NAL_SUFFIX_NSEI indicate that the NAL unit includes non-essential supplementary enhancement information. The RBSP syntax structure of the NAL unit is sei_rbsp(). The type class of the NAL unit is non-ACL.

[0673] NAL_PREFIX_ESEI and NAL_SUFFIX_ESEI indicate that the necessary supplemental enhancement information is included in the NAL unit. The RBSP syntax structure of the NAL unit is sei_rbsp(). The type class of the NAL unit is non-ACL.

[0674] NAL_AAPS indicates that the NAL unit includes the Atlas Adaptation Parameter Set. The RBSP syntax structure of the NAL unit is atlas_adaptation_parameter_set_rbsp(). The type of the NAL unit is non-ACL.

[0675] NAL_RSV_NACL_44 to NAL_RSV_NACL_47 indicate that the NAL unit type can be a reserved non-ACL NAL unit type. The type class of the NAL unit is non-ACL.

[0676] NAL_UNSPEC_48 to NAL_UNSPEC_63 indicate that the NAL unit type may be an unspecified non-ACL NAL unit type. The type class of the NAL unit is non-ACL.

[0677] Figure 41 The syntax structure of the Atlas sequence parameter set according to an embodiment is shown.

[0678] Figure 41 Shows the syntax of the RBSP data structure included in a NAL unit when the NAL unit type is atlas sequence parameter.

[0679] Each sample stream NAL unit may contain one of an atlas parameter set (eg, ASPS, AAPS, or AFPS), one or more atlas patch groups (or atlas patches), and a SEI.

[0680] The ASPS may contain syntax elements that apply to zero or more complete Coded Atlas Sequences (CAS) as determined by the contents of the syntax elements found in the ASPS (referred to as syntax elements in each tiling (or tile) header). Depending on the implementation, a syntax element may have the same meaning as a field or parameter.

[0681] exist Figure 41In the asps_atlas_sequence_parameter_set_id field, the asps_atlas_sequence_parameter_set_id field can provide an identifier for identifying the Atlas sequence parameter set for reference by other syntax elements.

[0682] The asps_frame_width field indicates the atlas frame width in terms of an integer number of samples, where a sample corresponds to a luma sample of the video component.

[0683] The asps_frame_height field indicates the atlas frame height in terms of an integer number of samples, where a sample corresponds to a luma sample of the video component.

[0684] The asps_log2_patch_packing_block_size field specifies the value of the variable PatchPackingBlockSize, which is used for the horizontal and vertical placement of patches within the atlas.

[0685] The asps_log2_max_atlas_frame_order_cnt_lsb_minus4 field specifies the value of the variable MaxAtlasFrmOrderCntLsb in the decoding process for the Atlas frame order count.

[0686] The asps_max_dec_atlas_frame_buffering_minus1 field plus 1 specifies the maximum size required for the decoded atlas frame buffer of CAS, in units of atlas frame storage buffers.

[0687] The asps_long_term_ref_atlas_frames_flag field equal to 0 specifies that the long-term reference Atlas frame is not used for inter prediction of any coded Atlas frame in CAS. The asps_long_term_ref_atlas_frames_flag field equal to 1 specifies that the long-term reference Atlas frame may be used for inter prediction of one or more coded Atlas frames in CAS.

[0688] The asps_num_ref_atlas_frame_lists_in_asps field specifies the number of ref_list_struct(rlsIdx) syntax structures included in the atlas sequence parameter set.

[0689] ASAP includes an iteration statement that repeats as many times as the value of the asps_num_ref_atlas_frame_lists_in_asps field. In an embodiment, the iteration statement includes ref_list_struct(i). According to an embodiment, in the iteration statement, i is initialized to 0 and incremented by 1 each time the iteration statement is executed. The iteration statement is repeated until i reaches the value of the asps_num_ref_atlas_frame_lists_in_asps field.

[0690] Will refer to Figure 50 Describe ref_list_struct(i) in detail.

[0691] When the asps_num_ref_atlas_frame_lists_in_asps field is greater than 0, the atgh_ref_atlas_frame_list_sps_flag field may be included in the atlas tiling (tile) header. When the asps_num_ref_atlas_frame_lists_in_asps field is greater than 1, the atgh_ref_atlas_frame_list_idx field may be included in the atlas tiling (tile) header.

[0692] The asps_use_eight_orientations_flag field equal to 0 specifies that the patch orientation index pdu_orientation_index[i][j] field for the patch with index j in the frame with index i is in the range of 0 to 1, inclusive. The asps_use_eight_orientations_flag field equal to 1 specifies that the patch orientation index pdu_orientation_index[i][j] field for the patch with index j in the frame with index i is in the range of 0 to 7, inclusive.

[0693] The asps_extended_projection_enabled_flag field equal to 0 specifies that patch projection information is not signaled for the current atlas tiling. The asps_extended_projection_enabled_flag field equal to 1 specifies that patch projection information is signaled for the current atlas tiling.

[0694] The asps_normal_axis_limits_quantization_enabled_flag field is equal to 1 to specify that the quantization parameter should be signaled and used to quantize the normal-axis related elements of the patch data unit, the merged patch data unit, or the inter-patch data unit. If the asps_normal_axis_limits_quantization_enabled_flag field is equal to 0, quantization is not applied to any normal-axis related elements of the patch data unit, the merged patch data unit, or the inter-patch data unit.

[0695] When the asps_normal_axis_limits_quantization_enabled_flag field is 1, the atgh_pos_min_z_quantizer field may be included in the atlas tiling (or tile) header.

[0696] The asps_normal_axis_max_delta_value_enabled_flag field is equal to 1, which specifies that the maximum nominal shift value of the normal axis that can be present in the geometric information of the patch with index i in the frame with index j shall be indicated in the bitstream of each patch data unit, merged patch data unit, or inter-patch data unit. If the asps_normal_axis_max_delta_value_enabled_flag field is equal to 0, the maximum nominal shift value of the normal axis that can be present in the geometric information of the patch with index i in the frame with index j shall not be indicated in the bitstream of each patch data unit, merged patch data unit, or inter-patch data unit.

[0697] When the asps_normal_axis_max_delta_value_enabled_flag field is 1, the atgh_pos_delta_max_z_quantizer field may be included in the atlas tiling (or patch) header.

[0698] The asps_remove_duplicate_point_enabled_flag field equal to 1 indicates that duplicate points are not constructed for the current atlas, where a duplicate point is a point that has the same 2D and 3D geometric coordinates as another point from a lower index map. The asps_remove_duplicate_point_enabled_flag field equal to 0 indicates that all points are reconstructed.

[0699] The asps_max_dec_atlas_frame_buffering_minus1 field plus 1 specifies the maximum size required for the decoded atlas frame buffer of CAS, in units of atlas frame storage buffers.

[0700] The asps_pixel_deinterleaving_flag field equal to 1 indicates that the decoded geometry and attribute video of the current atlas contains spatially interleaved pixels from two pictures. The asps_pixel_deinterleaving_flag field equal to 0 indicates that the decoded geometry and attribute video corresponding to the current atlas contains pixels from only a single picture.

[0701] The asps_patch_precedence_order_flag field is equal to 1 and indicates that the patch priority of the current atlas is the same as the decoding order. The asps_patch_precedence_order_flag field is equal to 0 and indicates that the patch priority of the current atlas is opposite to the decoding order.

[0702] The asps_patch_size_quantizer_present_flag field equal to 1 indicates that the patch size quantizer parameter is present in the atlas patch group header. The asps_patch_size_quantizer_present_flag field equal to 0 indicates that the patch size quantizer parameter is not present.

[0703] When the asps_patch_size_quantizer_present_flag field is equal to 1, the atgh_patch_size_x_info_quantizer field and the atgh_patch_size_y_info_quantizer field may be included in the atlas patch group (or patch) header.

[0704] The asps_eom_patch_enabled_flag field equal to 1 indicates that the decoded occupancy map video of the current atlas contains information about whether intermediate depth positions between two depth maps are occupied. The asps_eom_patch_enabled_flag field equal to 0 indicates that the decoded occupancy map video does not contain information about whether intermediate depth positions between two depth maps are occupied.

[0705] The asps_raw_patch_enabled_flag field is equal to 1 to indicate that the decoded geometry and attribute video of the current atlas contains information related to RAW code points. The asps_raw_patch_enabled_flag field is equal to 0 to indicate that the decoded geometry and attribute video does not contain information related to RAW code points.

[0706] When the asps_eom_patch_enabled_flag field or the asps_raw_patch_enabled_flag field is equal to 1, the asps_auxiliary_video_enabled_flag field may be included in the atlas sequence parameter set syntax.

[0707] The asps_auxiliary_video_enabled_flag field equal to 1 indicates that information related to RAW and EOM patch types can be placed in the auxiliary video sub-bitstream. The asps_auxiliary_video_enabled_flag field equal to 0 indicates that information related to RAW and EOM patch types can only be placed in the primary video sub-bitstream.

[0708] The asps_point_local_reconstruction_enabled_flag field equal to 1 indicates that point local reconstruction mode information may be present in the bitstream of the current atlas. The asps_point_local_reconstruction_enabled_flag field equal to 0 indicates that information related to the point local reconstruction mode is not present in the bitstream of the current atlas.

[0709] The asps_map_count_minus1 field plus 1 indicates the number of maps that can be used to encode the geometry and attribute data of the current atlas.

[0710] When the asps_pixel_deinterleaving_enabled_flag field is equal to 1, for each value of the asps_map_count_minus1 field, the asps_pixel_deinterleaving_map_flag[j] field may be included in the ASPS.

[0711] The asps_pixel_deinterleaving_map_flag[i] field equal to 1 indicates that the decoded geometry and attribute video corresponding to the picture indexed i in the current atlas contains spatially interleaved pixels corresponding to two pictures. The asps_pixel_deinterleaving_map_flag[i] field equal to 0 indicates that the decoded geometry and attribute video corresponding to the picture indexed i in the current atlas contains pixels corresponding to a single picture.

[0712] When the asps_eom_patch_enabled_flag field and the asps_map_count_minus1 field are equal to 0, the ASPS may further include an asps_eom_fix_bit_count_minus1 field.

[0713] The asps_eom_fix_bit_count_minus1 field plus 1 indicates the bit size of the EOM codeword.

[0714] When the asps_point_local_reconstruction_enabled_flag field is equal to 1, asps_point_local_reconstruction_information (asps_map_count_minus1) may be included in the ASPS and transmitted.

[0715] When the asps_pixel_deinterleaving_enabled_flag field or the asps_point_local_reconstruction_enabled_flag field is equal to 1, the ASPS may further include an asps_surface_thickness_minus1 field.

[0716] The asps_surface_thickness_minus1 field plus 1 specifies the maximum absolute difference between the explicitly coded depth value and the interpolated depth value when the asps_pixel_deinterleaving_enabled_flag field or the asps_point_local_reconstruction_enabled_flag field is equal to 1.

[0717] The asps_vui_parameters_present_flag field equal to 1 indicates that the vui_parameters() syntax structure is present. The asps_vui_parameters_present_flag field equal to 0 indicates that the vui_parameters() syntax structure is not present.

[0718] The asps_extension_flag field equal to 0 indicates that the asps_extension_data_flag field is not present in the ASPS RBSP syntax structure.

[0719] The asps_extension_data_flag field indicates that data for extension is included in the ASPS RBSP syntax structure.

[0720] The rbsp_trailing_bits field is used to fill the remaining bits with 0 for byte alignment after adding 1 (which is a stop bit) to indicate the end of RBSP data.

[0721] Figure 42 An Atlas frame parameter set according to an embodiment is shown.

[0722] Figure 42 Shows that when the NAL unit type (nal_unit_type) is Figure 40 The NAL_AFPS shown is the syntax structure of the atlas frame parameter set included in the NAL unit.

[0723] exist Figure 42 In [1], an Atlas Frame Parameter Set (AFPS) contains a syntax structure that includes syntax elements that apply to all zero or more fully coded Atlas frames.

[0724] The afps_atlas_frame_parameter_set_id field may provide an identifier for identifying the Atlas frame parameter set for reference by other syntax elements. In other words, an identifier that can be referenced by other syntax elements may be provided by the AFPS Atlas frame parameter set.

[0725] The afps_atlas_sequence_parameter_set_id field specifies the value of the asps_atlas_sequence_parameter_set_id of the active Atlas sequence parameter set.

[0726] Will refer to Figure 43 Describe atlas_frame_tile_information().

[0727] The afps_output_flag_present_flag field equal to 1 indicates that the atgh_atlas_output_flag or ath_atlas_output_flag field is present in the associated tiling (or patch) header. The afps_output_flag_present_flag field equal to 0 indicates that the atgh_atlas_output_flag field or ath_atlas_output_flag field is not present in the associated tiling (or patch) header.

[0728] The afps_num_ref_idx_default_active_minus1 field plus 1 specifies the inferred value of the variable NumRefIdxActive for tiles with the atgh_num_ref_idx_active_override_flag field equal to 0.

[0729] The afps_additional_lt_afoc_lsb_len field specifies the value of the variable MaxLtAtlasFrmOrderCntLsb used in the decoding process of the reference Atlas frame.

[0730] The afps_3d_pos_x_bit_count_minus1 field plus 1 specifies the number of bits in the fixed-length representation of the pdu_3d_pos_x[j] field of the patch with index j in the atlas tiling set that references the afps_atlas_frame_parameter_set_id field.

[0731] The afps_3d_pos_y_bit_count_minus1 field plus 1 specifies the number of bits in the fixed-length representation of the pdu_3d_pos_y[j] field of the patch with index j in the atlas tiling set that references the afps_atlas_frame_parameter_set_id field.

[0732] The afps_lod_mode_enabled_flag field equal to 1 indicates that LOD parameters may be present in a patch. The afps_lod_mode_enabled_flag field equal to 0 indicates that LOD parameters are not present in a patch.

[0733] The afps_override_eom_for_depth_flag field equal to 1 indicates that the values ​​of the afps_eom_number_of_patch_bit_count_minus1 field and the afps_eom_max_bit_count_minus1 field are explicitly present in the bitstream. The afps_override_eom_for_depth_flag field equal to 0 indicates that the values ​​of the afps_eom_number_of_patch_bit_count_minus1 field and the afps_eom_max_bit_count_minus1 field are implicitly derived.

[0734] The afps_eom_number_of_patch_bit_count_minus1 field plus 1 specifies the number of bits used to represent the number of geometry patches associated in the EOM attribute patch in the Atlas frame associated with this Atlas frame parameter set.

[0735] The afps_eom_max_bit_count_minus1 field plus 1 specifies the number of bits used to represent the number of EOM points for each geometry patch associated with the EOM attribute patch in the Atlas frame associated with this Atlas frame parameter set.

[0736] The afps_raw_3d_pos_bit_count_explicit_mode_flag field equal to 1 indicates that the number of bits in the fixed-length representation of the rpdu_3d_pos_x field, the rpdu_3d_pos_y field, and the rpdu_3d_pos_z field is explicitly coded by the atgh_raw_3d_pos_axis_bit_count_minus1 field in the atlas tile group header that references the afps_atlas_frame_parameter_set_id field. The afps_raw_3d_pos_bit_count_explicit_mode_flag field equal to 0 indicates that the value of the atgh_raw_3d_pos_axis_bit_count_minus1 field is implicitly derived.

[0737] When the afps_raw_3d_pos_bit_count_explicit_mode_flag field is equal to 1, the atgh_raw_3d_pos_axis_bit_count_minus1 field may be included in the atlas tile group (or patch) header.

[0738] afps_fixed_camera_model_flag indicates whether a fixed camera model exists.

[0739] The afps_extension_flag field equal to 0 indicates that the afps_extension_data_flag field is not present in the AFPS RBSP syntax structure.

[0740] The afps_extension_data_flag field may contain extension related data.

[0741] Figure 43 The syntax structure of atlas_frame_tile_information according to an embodiment is shown.

[0742] Figure 43 Show Figure 42 The syntax of atlas_frame_tile_information contained in .

[0743] The afti_single_tile_in_atlas_frame_flag field equal to 1 specifies that there is only one tile in each Atlas frame referencing the AFPS. The afti_single_tile_in_atlas_frame_flag field equal to 0 indicates that there is more than one tile in each Atlas frame referencing the AFPS.

[0744] The afti_uniform_tile_spacing_flag field equal to 1 specifies that tile column and row boundaries are uniformly distributed across the Atlas frame and is signaled using the syntax elements afti_tile_cols_width_minus1 and afti_tile_rows_height_minus1, respectively. The afti_uniform_tile_spacing_flag field equal to 0 specifies that tile column and row boundaries may or may not be uniformly distributed across the Atlas frame and is signaled using the afti_num_tile_columns_minus1 and afti_num_tile_rows_minus1 fields and the lists of syntax elements afti_tile_column_width_minus1[i] and afti_tile_row_height_minus1[i].

[0745] The afti_tile_cols_width_minus1 field plus 1 specifies the width of tile columns excluding the rightmost tile column of the atlas frame in units of 64 samples when the afti_uniform_tile_spacing_flag field is equal to 1.

[0746] The afti_tile_rows_height_minus1 field plus 1 specifies the height of tile rows excluding the bottom tile row of the atlas frame in units of 64 samples when the afti_uniform_tile_spacing_flag field is equal to 1.

[0747] The afti_num_tile_columns_minus1 field plus 1 specifies the number of tile columns that the atlas frame is partitioned into when the afti_uniform_tile_spacing_flag field is equal to 0.

[0748] The afti_num_tile_rows_minus1 field plus 1 specifies the number of tile rows that divide the atlas frame when the pti_uniform_tile_spacing_flag field is equal to 0.

[0749] The afti_tile_column_width_minus1[i] field plus 1 specifies the width of the i-th tile column in units of 64 samples.

[0750] The afti_tile_row_height_minus1[i] field plus 1 specifies the height of the i-th tile row in units of 64 samples.

[0751] The afti_single_tile_per_tile_group_flag field equal to 1 specifies that each tile group referencing this AFPS includes one tile (or tile partition). The afti_single_tile_per_tile_group_flag field equal to 0 specifies that the tile group referencing this AFPS may include more than one tile (or tile partition).

[0752] When the afti_single_tile_per_tile_group_flag field is equal to 0, the afti_num_tile_groups_in_atlas_frame_minus1 field is carried in the atlas frame tile information.

[0753] The afti_num_tile_groups_in_atlas_frame_minus1 field plus 1 specifies the number of tile groups (or tiles) in each atlas frame referencing the AFPS.

[0754] For each value of the afti_num_tile_groups_in_atlas_frame_minus1 field, an afti_top_left_tile_idx[i] field and an afti_bottom_right_tile_idx_delta[i] field may also be included in the AFPS.

[0755] The afti_top_left_tile_idx[i] field specifies the tile index of the tile located at the top left corner of the i-th tile group.

[0756] The afti_bottom_right_tile_idx_delta[i] field specifies the difference between the tile index of the tile located at the bottom right corner of the i-th tile group and the afti_top_left_tile_idx[i] field.

[0757] The afti_signalled_tile_group_id_flag field equal to 1 specifies that the tile group ID for each tile group or the tile ID for each tile is signaled.

[0758] When the afti_signalled_tile_group_id_flag field is 1, the afti_signalled_tile_group_id_length_minus1 field and the afti_tile_group_id[i] field may be carried in the atlas frame tile information.

[0759] The afti_signalled_tile_group_id_length_minus1 field plus 1 specifies the number of bits used to represent the syntax element afti_tile_group_id[i] field (when present) and the syntax element atgh_address in the tile group header.

[0760] The afti_tile_group_id[i] field specifies the tile group ID of the i-th tile group. The length of the afti_tile_group_id[i] field is afti_signalled_tile_group_id_length_minus1+1 bits.

[0761] exist Figure 43In

[0045] , a tile group may be referred to as a tile, and a tile may be referred to as a partition (or tile partition). In this case, afti_tile_group_id[i] indicates the ID of the i-th tile.

[0762] Figure 44 An atlas adaptation parameter set (atlas_adaptation_parameter_set_rbsp()) according to an embodiment is shown.

[0763] Figure 44 The syntax structure of the Atlas Adaptation Parameter Set (AAPS) carried by the NAL unit when the NAL unit type (nal_unit_type) is NAL_AAPS is shown.

[0764] The AAPS RBSP includes parameters that can be referenced by one or more coded patch group (or patch) NAL units of a coded Atlas frame. At any given moment during the operation of the decoding process, at most one AAPS RBSP is considered active, and activation of any particular AAPS RBSP causes deactivation of the previously active AAPS RBSP.

[0765] exist Figure 44 In the aaps_atlas_adaptation_parameter_set_id field, an identifier for identifying the Atlas adaptation parameter set is provided for reference by other syntax elements.

[0766] The aaps_atlas_sequence_parameter_set_id field specifies the value of the asps_atlas_sequence_parameter_set_id field of the active Atlas sequence parameter set.

[0767] The aaps_camera_parameters_present_flag field is equal to 1 to specify that the camera parameters (atlas_camera_parameters) are present in the current atlas adaptation parameter set. The aaps_camera_parameters_present_flag field is equal to 0 to specify that the camera parameters of the current adaptation parameter set do not exist. Figure 45 Provide a description.

[0768] The aaps_extension_flag field equal to 0 indicates that the aaps_extension_data_flag field is not present in the AAPS RBSP syntax structure.

[0769] The aaps_extension_data_flag field may indicate whether AAPS contains extension-related data.

[0770] Figure 45 Shows atlas_camera_parameters according to an embodiment.

[0771] Figure 45 Show Figure 44 The detailed syntax of atlas_camera_parameters.

[0772] exist Figure 45 The acp_camera_model field indicates the camera model of the point cloud frame associated with the current atlas adaptation parameter set.

[0773] Figure 46 is a table showing an example of a camera model assigned to the acp_camera_model field according to an embodiment.

[0774] For example, an acp_camera_model field equal to 0 indicates that the camera model is unspecified.

[0775] The acp_camera_model field equal to 1 indicates that the camera model is an orthographic camera model.

[0776] When the acp_camera_model field is 2-255, the camera model can be reserved.

[0777] According to an embodiment, when the value of the acp_camera_model field is equal to 1, the camera parameters may further include an acp_scale_enabled_flag field, an acp_offset_enabled_flag field, and / or an acp_rotation_enabled_flag field related to scaling, offset, and rotation.

[0778] The acp_scale_enabled_flag field equal to 1 indicates that the scaling parameters of the current camera model exist. The acp_scale_enabled_flag field equal to 0 indicates that the scaling parameters of the current camera model do not exist.

[0779] When acp_scale_enabled_flagfield is equal to 1, the acp_scale_on_axis[d] field may be included in the atlas camera parameters for each d value.

[0780] The acp_scale_on_axis[d] field specifies the scale value Scale[d] of the current camera model along the d axis. The value of d ranges from 0 to 2 (inclusive), where 0, 1, and 2 correspond to the X, Y, and Z axes respectively.

[0781] The acp_offset_enabled_flag field is equal to 1, indicating that the offset parameters of the current camera model exist. The acp_offset_enabled_flag field is equal to 0, indicating that the offset parameters of the current camera model do not exist.

[0782] When the acp_offset_enabled_flag field is equal to 1, the acp_offset_on_axis[d] field may be included in the atlas camera parameters for each d value.

[0783] The acp_offset_on_axis[d] field indicates the value of the offset Offset[d] of the current camera model along the d axis, where d is in the range of 0 to 2 (inclusive). The value of d is equal to 0, 1, and 2, corresponding to the X, Y, and Z axes, respectively.

[0784] The acp_rotation_enabled_flag field is equal to 1, indicating that the rotation parameters of the current camera model exist. The acp_rotation_enabled_flag field is equal to 0, indicating that the rotation parameters of the current camera model do not exist.

[0785] When acp_rotation_enabled_flagfield is equal to 1, the atlas camera parameters may further include an acp_rotation_qx field, an acp_rotation_qy field, and an acp_rotation_qz field.

[0786] The acp_rotation_qx field specifies the x-component qX for the rotation of the current camera model using a quaternion representation.

[0787] The acp_rotation_qy field specifies the y component qY for the rotation of the current camera model using a quaternion representation.

[0788] The acp_rotation_qz field specifies the z component qZ for the rotation of the current camera model using a quaternion representation.

[0789] atlas_camera_parameters() as described above may be included in at least one SEI message and transmitted.

[0790] Figure 47An atlas_tile_group_layer according to an embodiment is shown.

[0791] Figure 47 Show the basis Figure 40 The NAL unit type shown indicates the syntax structure of atlas_tile_group_layer or atlas_tile_layer carried in a NAL unit.

[0792] According to an embodiment, a tiling may correspond to a tile. In the present disclosure, the term "tiling" may be referred to as the term "tile." Similarly, the term "atgh" may be interpreted as the term "ath."

[0793] The atlas_tile_group_layer field or the atlas_tile_layer field may include the atlas_tile_group_header (atlas_tile_group_header) or the atlas_tile_header (atlas_tile_header). Figure 48 Describes the atlas tile group (or tile) header (atlas_tile_group_header or atlas_tile_header).

[0794] When the atgh_type field of the atlas tile group (or tile) is not SKIP_TILE_GRP, the atlas tile group (or tile) data may be contained in atlas_tile_group_layer or atlas_tile_layer.

[0795] Figure 48 An example of a syntax structure of an atlas tile group (or tile) header (atlas_tile_group_header() or atlas_tile_header()) included in an atlas tile group layer according to an embodiment is shown.

[0796] exist Figure 48 In

[0064] , the atgh_atlas_frame_parameter_set_id field specifies the value of the afps_atlas_frame_parameter_set_id field of the active Atlas frame parameter set of the current Atlas block group.

[0797] The atgh_atlas_adaptation_parameter_set_id field specifies the value of the aaps_atlas_adaptation_parameter_set_id field of the active Atlas adaptation parameter set for the current Atlas tiling.

[0798] The atgh_address field specifies the tile group (or tile) address of the tile group (or tile). When not present, the value of the atgh_address field is inferred to be equal to 0. The tile group (or tile) address is the tile group ID (or tile ID) of the tile group (or tile). The length of the atgh_address field is the afti_signalled_tile_group_id_length_minus1 field + 1 bit. If the afti_signalled_tile_group_id_flag field is equal to 0, the value of the atgh_address field is in the range of 0 to the afti_num_tile_groups_in_atlas_frame_minus1 field, inclusive. Otherwise, the value of the atgh_address field is in the range of 0 to 2(afti_signalled_tile_group_id_length_minus1field+1)-1, inclusive. The AFTI includes an afti_signalled_tile_group_id_length_minus1 field and an afti_signalled_tile_group_id_flag field.

[0799] The atgh_type field specifies the coding type of the current atlas tile group (tile).

[0800] Figure 49 An example of a coding type assigned to an atgh_type field according to an embodiment is shown.

[0801] For example, when the value of the atgh_type field is 0, the coding type of the atlas tile group (or tile) is P_TILE_GRP (inter-frame atlas tile group (or tile)).

[0802] When the value of the atgh_type field is 1, the coding type of the atlas tile group (or tile) is I_TILE_GRP (intra-atlas tile group (or tile)).

[0803] When the value of the atgh_type field is 2, the coding type of the atlas tile group (or tile) is SKIP_TILE_GRP (skip atlas tile group (or tile)).

[0804] The value of the afps_output_flag_present_flag field included in the AFTI is equal to 1, and the atlas tile group (or tile) header may further include an atgh_atlas_output_flag field.

[0805] The atgh_atlas_output_flag field affects the atlas output and removal process after decoding.

[0806] The atgh_atlas_frm_order_cnt_lsb field specifies the Atlas frame order count modulo MaxAtlasFrmOrderCntLsb of the current Atlas tile group.

[0807] According to an embodiment, if the value of the asps_num_ref_atlas_frame_lists_in_asps field included in the Atlas Sequence Parameter Set (ASPS) is greater than 1, the Atlas tile group (or patch) header may further include the atgh_ref_atlas_frame_list_sps_flag field. asps_num_ref_atlas_frame_lists_in_asps specifies the number of ref_list_struct(rlsIdx) syntax structures included in the ASPS.

[0808] The atgh_ref_atlas_frame_list_sps_flag field equal to 1 specifies that the reference atlas frame list of the current atlas tiling (or patch) is derived based on one of the ref_list_struct(rlsIdx) syntax structures in the active ASPS. The atgh_ref_atlas_frame_list_sps_flag field equal to 0 specifies that the reference atlas frame list of the current atlas tiling list is derived based on the ref_list_struct(rlsIdx) syntax structure directly included in the patch header of the current atlas tiling.

[0809] According to an embodiment, when the atgh_ref_atlas_frame_list_sps_flag field is equal to 0, the atlas patch group (or patch) header includes ref_list_struct (asps_num_ref_atlas_frame_lists_in_asps), and when atgh_ref_atlas_frame_list_sps_flag is greater than 1, the atlas patch group (or patch) header includes the atgh_ref_atlas_frame_list_sps_flag field.

[0810] The atgh_ref_atlas_frame_list_idx field specifies the index of the ref_list_struct(rlsIdx) syntax structure used to derive the reference atlas frame list for the current atlas tile group (or patch). The reference atlas frame list is a list of ref_list_struct(rlsIdx) syntax structures included in the active ASPS.

[0811] According to an embodiment, depending on the value of the NumLtrAtlasFrmEntries field, the atlas tile group (or patch) header further includes an atgh_additional_afoc_lsb_present_flag[j] field, and if atgh_additional_afoc_lsb_present_flag[j] is equal to 1, it may further include an atgh_additional_afoc_lsb_val[j] field.

[0812] atgh_additional_afoc_lsb_val[j] specifies the value of FullAtlasFrmOrderCntLsbLt[RlsIdx][j] of the current atlas tile group (or tile).

[0813] According to an embodiment, if the atgh_type field does not indicate SKIP_TILE_GRP, the atlas tile group (or tile) header may further include an atgh_pos_min_z_quantizer field, an atgh_pos_delta_max_z_quantizer field, an atgh_patch_size_x_info_quantizer field, an atgh_patch_size_y_info_quantizer field, an atgh_raw_3d_pos_axis_bit_count_minus1 field, and / or an atgh_num_ref_idx_active_minus1 field, depending on information included in the ASPS or AFPS.

[0814] According to an embodiment, when the value of the asps_normal_axis_limits_quantization_enabled_flag field included in the ASPS is 1, the atgh_pos_min_z_quantizer field is included. When the values ​​of both the asps_normal_axis_limits_quantization_enabled_flag and asps_axis_max_enabled_flag fields included in the ASPS are 1, the atgh_pos_delta_max_z_quantizer field is included.

[0815] According to an embodiment, when the value of the asps_patch_size_quantizer_present_flag field included in the ASPS is 1, the atgh_patch_size_x_info_quantizer field and the atgh_patch_size_y_info_quantizer field are included. When the value of the afps_raw_3d_pos_bit_count_explicit_mode_flag field included in the AFPS is 1, the atgh_raw_3d_pos_axis_bit_explicit_minus1 field is included.

[0816] According to an embodiment, when the atgh_type field indicates P_TILE_GRP and num_ref_entries[RlsIdx] is greater than 1, the atlas tile group (or tile) header further includes an atgh_num_ref_idx_active_override_flag field. When the value of the atgh_num_ref_idx_active_override_flag field is 1, the atgh_num_ref_idx_active_minus1 field is included in the atlas tile group (or tile) header.

[0817] atgh_pos_min_z_quantizer specifies the quantizer to be applied to pdu_3d_pos_min_z[p] for the patch with index p. If the atgh_pos_min_z_quantizer field is not present, its value may be inferred to be equal to 0.

[0818] The atgh_pos_delta_max_z_quantizer field specifies the quantizer to be applied to the value of the pdu_3d_pos_delta_max_z[p] field for the patch with index p. If the atgh_pos_delta_max_z_quantizer field is not present, its value may be inferred to be equal to 0.

[0819] The atgh_patch_size_x_info_quantizer field specifies the value of the quantizer PatchSizeXQuantizer to be applied to the variables pdu_2d_size_x_minus1[p], mpdu_2d_delta_size_x[p], ipdu_2d_delta_size_x[p], rpdu_2d_size_x_minus1[p], and epdu_2d_size_x.minus1[p] for the patch with index p. If the atgh_patch_size_x_info_quantizer field is not present, its value may be inferred to be equal to the asps_log2_patch_packing_block_size field.

[0820] The atgh_patch_size_y_info_quantizer field specifies the value of the quantizer PatchSizeYQuantizer to be applied to the variables pdu_2d_size_y_minus1[p], mpdu_2d_delta_size_y[p], ipdu_2d_delta_size_y[p], rpdu_2d_size_y_minus1[p], and epdu_2d_size_y_minus1[p] for the patch with index p. If the atgh_patch_size_y_info_quantizer field is not present, its value may be inferred to be equal to the asps_log2_patch_packing_block_size field.

[0821] The atgh_raw_3d_pos_axis_bit_count_minus1 field plus 1 specifies the number of bits in the fixed length representation of rpdu_3d_pos_x, rpdu_3d_pos_y, and rpdu_3d_pos_z.

[0822] The atgh_num_ref_idx_active_override_flag field is equal to 1 and specifies that the syntax element atgh_num_ref_idx_active_minus1 is present in the current atlas tile group (or tile). The atgh_num_ref_idx_active_override_flag field is equal to 0 and specifies that the syntax element atgh_num_ref_idx_active_minus1 is not present. If the atgh_num_ref_idx_active_override_flag field is not present, its value is inferred to be 0.

[0823] The atgh_num_ref_idx_active_minus1 field plus 1 specifies the maximum reference index of the reference Atlas frame list that can be used to decode the current Atlas tiling group. When the value of the atgh_num_ref_idx_active_minus1 field is equal to 0, the reference index of the reference Atlas frame list may not be used to decode the current Atlas tiling group (or patch).

[0824] byte_alignment is used to fill the remaining bits with 0 after adding 1 (which is a stop bit) for byte alignment to indicate the end of data.

[0825] As described above, one or more ref_list_struct(rlsIdx) syntax structures may be included in the ASPS and / or may be included directly in the atlas tilset (or tile) header.

[0826] Figure 50 An example of the syntax structure of ref_list_struct() according to an embodiment is shown.

[0827] exist Figure 50 In the ref_list_struct(rlsIdx) syntax structure, the num_ref_entries[rlsIdx] field specifies the number of entries in the ref_list_struct(rlsIdx) syntax structure.

[0828] As many of the following elements as the value of the num_ref_entries[rlsIdx] field may be included in the reference list structure.

[0829] When the asps_long_term_ref_atlas_frames_flag field is equal to 1, a reference atlas frame flag (st_ref_atlas_frame_flag[rlsIdx][i]) may be included in the reference list structure.

[0830] When the i-th entry is the first short-term reference Atlas frame entry in the ref_list_struct(rlsIdx) syntax structure, the abs_delta_afoc_st[rlsIdx][i] field specifies the absolute difference between the Atlas frame sequence count values ​​of the current Atlas tile group and the Atlas frame referenced by the i-th entry. When the i-th entry is a short-term reference Atlas frame entry but not the first short-term reference Atlas frame entry in the ref_list_struct(rlsIdx), this field specifies the absolute difference between the Atlas frame sequence count values ​​of the Atlas frame referenced by the i-th entry and the previous short-term reference Atlas frame entry in the ref_list_struct(rlsIdx) syntax structure.

[0831] When the st_ref_atlas_frame_flag[rlsIdx][i] field is equal to 1, the abs_delta_afoc_st[rlsIdx][i] field may be included in the reference list structure.

[0832] The abs_delta_afoc_st[rlsIdx][i] field specifies the absolute difference between the Atlas frame sequence count values ​​of the current Atlas tile group and the Atlas frame referenced by the i-th entry when the i-th entry is the first short-term reference Atlas frame entry in the ref_list_struct(rlsIdx) syntax structure, or specifies the absolute difference between the Atlas frame sequence count values ​​of the Atlas frame referenced by the i-th entry in the ref_list_struct(rlsIdx) syntax structure and the previous short-term reference Atlas frame entry when the i-th entry is a short-term reference Atlas frame entry but not the first short-term reference Atlas frame entry in the ref_list_struct(rlsIdx) syntax structure.

[0833] When the value of the abs_delta_afoc_st[rlsIdx][i] field is greater than 0, an entry sign flag (strpf_entry_sign_flag[rlsIdx][i]) field may be included in the reference list structure.

[0834] A strpf_entry_sign_flag[rlsIdx][i] field equal to 1 specifies that the i-th entry in the syntax structure ref_list_struct(rlsIdx) field has a value greater than or equal to 0. A strpf_entry_sign_flag[rlsIdx][i] field equal to 0 specifies that the i-th entry in the syntax structure ref_list_struct(rlsIdx) has a value less than 0. When not present, the value of the strpf_entry_sign_flag[rlsIdx][i] field is inferred to be equal to 1.

[0835] When asps_long_term_ref_atlas_frames_flag included in the ASPS is equal to 0, the afoc_lsb_lt[rlsIdx][i] field may be included in the reference list structure.

[0836] The afoc_lsb_lt[rlsIdx][i] field specifies the value of the Atlas frame order count modulo MaxAtlasFrmOrderCntLsb of the Atlas frame referenced by the i-th entry in the ref_list_struct(rlsIdx) field. The length of the afoc_lsb_lt[rlsIdx][i] field is asps_log2_max_atlas_frame_order_cnt_lsb_minus4+4 bits.

[0837] Figure 51Atlas tile group data (atlas_tile_group_data_unit) according to an embodiment is shown.

[0838] Figure 51 Shown included in Figure 47 The syntax of atlas tile group data (atlas_tile_group_data_unit()) in the atlas tile group (or tile) layer of . The atlas tile group data may correspond to atlas tile data, and a tile group may be referred to as a tile.

[0839] exist Figure 51 , as p increases from 0 to 1, the atlas-related elements (or fields) according to index p may be included in the atlas tiling (or tile) data.

[0840] The atgdu_patch_mode[p] field indicates the patch mode of the patch with index p in the current atlas tile group. When the atgh_type field included in the atlas tile group (or tile) header indicates SKIP_TILE_GRP, this indicates that the entire tile group (or tile) information is directly copied from the tile group (or tile) having the same atgh_address as the current tile group (or tile) corresponding to the first reference atlas frame.

[0841] When the atgdu_patch_mode[p] field is not I_END and atgdu_patch_mode[p] is not P_END, patch_information_data and atgdu_patch_mode[p] may be included in the atlas patch group data (or atlas patch data) of each index p.

[0842] Figure 52 An example of a patch mode type assigned to the atgdu_patch_mode field when the atgh_type field indicates I_TILE_GRP according to an embodiment is shown.

[0843] For example, the atgdu_patch_mode field equal to 0 indicates a non-predictive patch mode with an identifier of I_INTRA.

[0844] The atgdu_patch_mode field equal to 1 indicates a RAW point patch mode with an identifier of I_RAW.

[0845] The atgdu_patch_mode field equal to 2 indicates the EOM point patch mode with identifier I_EOM.

[0846] The atgdu_patch_mode field equal to 14 indicates the patch termination mode with identifier I_END.

[0847] Figure 53 An example of a patch mode type assigned to the atgdu_patch_mode field when the atgh_type field indicates P_TILE_GRP according to an embodiment is shown.

[0848] For example, the atgdu_patch_mode field equal to 0 indicates a patch skip mode with an identifier of P_SKIP.

[0849] The atgdu_patch_mode field equal to 1 indicates the patch merging mode with identifier P_MERGE.

[0850] The atgdu_patch_mode field equal to 2 indicates the inter-frame prediction patch mode with identifier P_INTER.

[0851] The atgdu_patch_mode field equal to 3 indicates a non-predictive patch mode with identifier P_INTRA.

[0852] The atgdu_patch_mode field equal to 4 indicates a RAW point patch mode with an identifier of P_RAW.

[0853] The atgdu_patch_mode field equal to 5 indicates the EOM point patch mode with identifier P_EOM.

[0854] The atgdu_patch_mode field equal to 14 indicates the patch termination mode with identifier P_END.

[0855] Figure 54 An example of a patch mode type assigned to the atgdu_patch_mode field when the atgh_type field indicates SKIP_TILE_GRP according to an embodiment is shown.

[0856] For example, atgdu_patch_mode equal to 0 indicates a patch skip mode with identifier P_SKIP.

[0857] According to an embodiment, the atlas patch group (or patch) data unit may further include an AtgduTotalNumberOfPatches field. The AtgduTotalNumberOfPatches field indicates the number of patches and may be set to the final value of p.

[0858] Figure 55Patch information data (patch_information_data(patchIdx, patchMode)) according to an embodiment is shown.

[0859] Figure 55 Shown included in Figure 51 An example of the syntax structure of patch information data (patch_information_data(p, atgdu_patch_mode[p])) in the tile group (or tile) data unit. Figure 51 In patch_information_data(p,atgdu_patch_mode[p]), p corresponds to Figure 55 patchIdx, and atgdu_patch_mode[p] corresponds to Figure 55 patchMode.

[0860] For example, when the atgh_type field indicates SKIP_TILE_GR, skip_patch_data_unit (patchIdx) is included as patch information data.

[0861] When the atgh_type field indicates P_TILE_GR, one of skip_patch_data_unit (patchIdx), merge_patch_data_unit (patchIdx), patch_data_unit (patchIdx), inter_patch_data_unit (patchIdx), raw_patch_data_unit (patchIdx), and eom_patch_data_unit (patchIdx) may be included as patch information data according to patchMode.

[0862] For example, when patchMode indicates patch skip mode (P_SKIP), skip_patch_data_unit(patchIdx) is included. When patchMode indicates patch merge mode (P_MERGE), merge_patch_data_unit(patchIdx) is included. When patchMode indicates P_INTRA, patch_data_unit(patchIdx) is included. When patchMode indicates P_INTER, inter_patch_data_unit(patchIdx) is included. When patchMode indicates RAW point patch mode (P_RAW), raw_patch_data_unit(patchIdx) is included. When patchMode is EOM point patch mode (P_EOM), eom_patch_data_unit(patchIdx) is included.

[0863] When the atgh_type field indicates I_TILE_GR, one of patch_data_unit (patchIdx), raw_patch_data_unit (patchIdx), and eom_patch_data_unit (patchIdx) may be included as the patch information data according to patchMode.

[0864] For example, when patchMode indicates I_INTRA, patch_data_unit (patchIdx) is included. When patchMode indicates RAW point patch mode (I_RAW), raw_patch_data_unit (patchIdx) is included. When patchMode indicates EOM point patch mode (I_EOM), eom_patch_data_unit (patchIdx) is included.

[0865] Figure 56 The syntax structure of the patch data unit (patch_data_unit(patchIdx)) according to the embodiment is shown. As described above, when the atgh_type field indicates P_TILE_GR and the patchMode indicates P_INTRA, or when the atgh_type field indicates I_TILE_GR and the patchMode indicates I_INTRA, the patch_data_unit(patchIdx) may be included as the patch information data.

[0866] exist Figure 56In

[15] , the pdu_2d_pos_x[p] field indicates the x-coordinate (or left offset) of the top-left corner of the patch bounding box of the patch with index p in the current atlas tile group (or tile) (tileGroupIdx). An atlas tile group (or tile) may have a tile group (or tile) index (tileGroupIdx). The tileGroupIdx may be expressed as a multiple of the PatchPackingBlockSize.

[0867] The pdu_2d_pos_y[p] field indicates the y coordinate (or top offset) of the top left corner of the tile bounding box of the tile with index p in the current atlas tile group (or tile) (tileGroupIdx). tileGroupIdx can be expressed as a multiple of PatchPackingBlockSize.

[0868] The pdu_2d_size_x_minus1[p] field plus 1 specifies the quantized width value of the patch with index p in the current atlas tile group (or tile) tileGroupIdx.

[0869] The pdu_2d_size_y_minus1[p] field plus 1 specifies the quantized height value of the patch with index p in the current atlas tile group (or tile) tileGroupIdx.

[0870] The pdu_3d_pos_x[p] field specifies the offset along the tangent axis to be applied to the reconstructed patch point in the patch with index p of the current atlas tiling (or patch).

[0871] The pdu_3d_pos_y[p] field specifies the offset along the bitangent axis to be applied to the reconstructed patch point in the patch with index p of the current atlas tiling (or patch).

[0872] The pdu_3d_pos_min_z[p] field specifies the offset along the normal axis to be applied to the reconstructed patch point in the patch with index p of the current atlas tiling (or patch).

[0873] When a value of the asps_normal_axis_max_delta_value_enabled_flag field included in the ASPS is equal to 1, the pdu_3d_pos_delta_max_z[patchIdx] field may be included in the patch data unit.

[0874] If present, the pdu_3d_pos_delta_max_z[p] field specifies the nominal maximum value of the displacement along the normal axis that is expected to be present in the reconstructed bitdepth patch geometry samples in the patch at index p of the current atlas tiling group (or patch), after conversion to their nominal representation.

[0875] The pdu_projection_id[p] field specifies the projection mode of the patch with index p of the current atlas tiling group (or tile) and the value of the index of the normal of the projection plane.

[0876] The pdu_orientation_index[p] field indicates the patch orientation index of the patch with index p of the current atlas tiling group (or tile). Figure 57 Provide a description.

[0877] When a value of the afps_lod_mode_enabled_flag field included in the AFPS is equal to 1, pdu_lod_enabled_flag[patchIndex] may be included in the patch data unit.

[0878] When the pdu_lod_enabled_flag[patchIndex] field is greater than 0, the pdu_lod_scale_x_minus1[patchIndex] field and the pdu_lod_scale_y[patchIndex] field may be included in the patch data unit.

[0879] When the pdu_lod_enabled_flag[patchIndex] field is equal to 1 and patchIndex is p, it specifies that LOD parameters exist for the current patch with index p. If the pdu_lod_enabled_flag[p] field is equal to 0, LOD parameters do not exist for the current patch.

[0880] The pdu_lod_scale_x_minus1[p] field plus 1 specifies the LOD scaling factor to be applied to the local x coordinate of the point in the patch with index p of the current atlas tiling group (or patch) before adding it to the patch coordinate Patch3dPosX[p].

[0881] The pdu_lod_scale_y[p] field specifies the LOD scaling factor to be applied to the local y coordinate of the point in the patch with index p of the current atlas tiling group (or tile) before adding it to the patch coordinate Patch3dPosY[p].

[0882] When a value of the asps_point_local_reconstruction_enabled_flag field included in the ASPS is equal to 1, point_local_reconstruction_data (patchIdx) may be included in the patch data unit.

[0883] According to an embodiment, point_local_reconstruction_data(patchIdx) may contain information allowing a decoder to recover points lost due to compression loss or the like.

[0884] Figure 57 Rotation and offset relative to patch orientation are shown according to an embodiment.

[0885] Figure 57 Show assigned to Figure 56 The rotation matrix and offset of the patch orientation index (pdu_orientation_index[p] field).

[0886] The method / apparatus according to the embodiment can perform an orientation operation on point cloud data. This operation can be performed using Figure 57 The identifiers, rotations and offsets shown are performed.

[0887] According to an embodiment, the NAL unit may include SEI information. For example, non-essential supplementary enhancement information or essential supplementary enhancement information may be included in the NAL unit according to nal_unit_type.

[0888] Figure 58 The syntax structure of SEI information including sei_message() according to an embodiment is exemplarily shown.

[0889] SEI messages facilitate processes related to decoding, reconstruction, display, or other purposes.Depending on the implementation, there may be two types of SEI messages: essential and non-essential.

[0890] The decoding process may not require non-essential SEI messages. A conforming decoder may not be required to process this information for output order consistency.

[0891] The necessary SEI message can be an integral part of the V-PCC bitstream and should not be removed from the bitstream. The necessary SEI message can be classified into the following two types:

[0892] Type A required SEI messages contain the information required to check bitstream conformance and output timing decoder conformance. Every V-PCC decoder that complies with point A may not discard any Type A required SEI messages and consider them for bitstream conformance and output timing decoder conformance.

[0893] Type B mandatory SEI messages: A V-PCC decoder conforming to a specific reconstruction profile may not discard any Type B mandatory SEI messages and consider them for 3D point cloud reconstruction and conformance purposes.

[0894] According to an embodiment, the SEI message consists of an SEI message header and an SEI message payload. The SEI message header includes an sm_payload_type_byte field and an sm_payload_size_byte field.

[0895] The sm_payload_type_byte field indicates the payload type of the SEI message. For example, it may be identified whether the SEI message is a prefix SEI message or a suffix SEI message based on the value of the sm_payload_type_byte field.

[0896] The sm_payload_size_byte field indicates the payload size of the SEI message.

[0897] According to an embodiment, the sm_payload_type_byte field is set to the value of PayloadType in the SEI message payload, and the sm_payload_size_byte field is set to the value of PayloadSize in the SEI message payload.

[0898] Figure 59 An example of the syntax structure of the SEI message payload (sei_payload(payloadType, payloadSize)) according to an embodiment is shown.

[0899] In an embodiment, when nal_unit_type is NAL_PREFIX_NSEI or NAL_PREFIX_ESEI, the SEI message payload may include sei(payloadSize) according to PayloadType.

[0900] In another embodiment, when nal_unit_type is NAL_SUFFIX_NSEI or NAL_SUFFIX_ESEI, the SEI message payload may include sei(payloadSize) according to PayloadType.

[0901] In addition, Figure 25 The V-PCC bit stream (also called V3C bit stream) with the structure shown can be sent to the receiving side as it is, or it can be converted by Figure 1 、 Figure 20 or Figure 21 The file / segment encapsulator encapsulates it into the ISOBMFF file format and sends it to the receiving side.

[0902] In the latter case, the V-PCC bitstream can be sent via multiple tracks in the file or via a single track. In this case, the file can be sent via Figure 1 、 Figure 20 or Figure 22 The file / segment decapsulator of the receiving device decapsulates it into a V-PCC bitstream.

[0903] For example, a V-PCC bitstream carrying a V-PCC parameter set, a geometry bitstream, an occupancy map bitstream, an attribute bitstream, and / or an atlas bitstream may be Figure 1 、 Figure 20 or Figure 21 The V-PCC bitstream may be stored in a single track or multiple tracks in a file based on ISOBMFF (ISO Base Media File Format) according to an embodiment.

[0904] According to an embodiment, a file based on ISOBMFF may be referred to as a container, a container file, a media file, a V-PCC file, etc. Specifically, a file may be composed of a box and / or information that may be referred to as ftyp, meta, moov, or mdat.

[0905] The ftyp box (file type box) can provide information related to the file compatibility or file type of the file. The receiving side can refer to the ftyp box to identify the file.

[0906] The meta box may include a vpcg{0,1,2,3} box (V-PCC group box).

[0907] The mdat box, also known as the media data box, contains the actual media data. Depending on the implementation, the geometry bitstream for video coding, the attribute bitstream for video coding, the occupancy map bitstream for video coding, and the atlas bitstream are contained in samples of the mdat box of the file. Depending on the implementation, the samples may be called V-PCC samples.

[0908] The moov box, also known as a movie box, can contain metadata about the file's media data (e.g., geometry bitstream, attribute bitstream, occupancy map bitstream, etc.). For example, it can contain information required to decode and play the media data, as well as information about the file's samples. The moov box can serve as a container for all metadata. The moov box can be the highest-level box among metadata-related boxes. Depending on the implementation, only one moov box may exist in a file.

[0909] The box according to an embodiment may include a track (trak) box that provides information related to the track of the file. The track box may include a media (mdia) box that provides media information about the track and a track reference container (tref) box for referencing the track and samples of the file corresponding to the track.

[0910] The mdia box may include a media information container (minf) box providing information about corresponding media data and a handler (hdlr) box (HandlerBox) indicating a type of stream.

[0911] The minf box may include a sample table (stbl) box that provides metadata related to the samples of the mdat box.

[0912] The stbl box may include a sample description (stsd) box that provides information about the encoding type employed and the initialization information required for that encoding type.

[0913] According to an embodiment, the stsd box may include sample entries for storing tracks of a V-PCC bitstream.

[0914] The term V-PCC used herein has the same meaning as the term visual volumetric video based coding (V3C). The two terms can be used to complement each other.

[0915] In the present disclosure, in order to store the V-PCC bitstream according to an embodiment in a single track or multiple tracks in a file, a volumetric visual track, a volumetric visual media header, a volumetric visual sample entry, a volumetric visual sample, samples and sample entries of a V-PCC track (or called a V3C track), samples and sample entries of a V-PCC video component track (or called a V3C video component track), etc. can be defined as follows.

[0916] A volumetric visual track (or simply referred to as a volume track) is a track with a handler type reserved for describing a volumetric visual track. That is, a volumetric visual track can be identified by the volumetric visual media handler type 'volv' included in the HandlerBox of the MediaBox and / or the volumetric visual media header (vvhd) in the minf box of the media box (MediaBox).

[0917] V3C tracks refer to V3C bitstream track, V3C atlas track, and V3C atlas tile track.

[0918] In the case of a single-track container, the V3C bitstream track is a volumetric visual track containing the V3C bitstream.

[0919] In the case of a multi-track container, the V3C Atlas track is a volumetric visual track that contains the V3C Atlas bitstream.

[0920] In the case of a multi-track container, a V3C Atlas patch track is a volumetric visual track that contains a portion of the V3C Atlas bitstream corresponding to one or more patches.

[0921] The V3C video component track is a video track that carries 2D video coding data corresponding to one of an occupancy video bitstream, a geometry video bitstream, and an attribute video bitstream in a V3C bitstream.

[0922] According to an embodiment, video-based point cloud compression (V-PCC) represents volumetric encoding of point cloud visual information.

[0923] That is, the minf box in the trak box of the moov box may further include a volumetric visual media header box. The volumetric visual media header box includes information about the volumetric visual track including the volumetric visual scene.

[0924] Each volumetric vision scene can be represented by a unique volumetric vision track. An ISOBMFF file can contain multiple scenes, so there can be multiple volumetric vision tracks in an ISOBMFF file.

[0925] Depending on the implementation, a volumetric visual track can be identified by the volumetric visual media handler type 'volv' in the HandlerBox of the MediaBox and / or the volumetric visual media header (vvhd) in the minf box of the mdia box (MediaBox). The minf box can be referred to as a media information container or media information box. The minf box is included in the mdia box. The mdia box is included in the trak box. The trak box is included in the moov box of the file. A single volumetric visual track or multiple volumetric visual tracks can exist in a file.

[0926] Depending on the implementation, the volumetric visual track may use the VolumetricVisualMediaHeaderBox in the MediaInformationBox. The MediaInformationBox is referred to as a minf box, while the VolumetricVisualMediaHeaderBox is referred to as a vvhd box.

[0927] According to an embodiment, the syntax of the volumetric visual media header (vvhd) box may be defined as follows.

[0928] Box type: 'vvhd'

[0929] Container: MediaInformationBox

[0930] Mandatory: Yes

[0931] Quantity: Exactly one

[0932] The syntax of a volumetric visual media header box (ie, a vvhd type box) according to an embodiment is as follows.

[0933] aligned(8)class VolumetricVisualMediaHeaderBox

[0934] extends FullBox('vvhd',version=0,1){

[0935] }

[0936] version may be an integer indicating the version of the box.

[0937] Depending on the implementation, the volumetric visual track may use VolumetricVisualSampleEntry to send signaling information and VolumetricVisualSample to send the actual data.

[0938] According to embodiments, a volumetric vision sample entry may be referred to as a sample entry or a V-PCC sample entry, and a volumetric vision sample may be referred to as a sample or a V-PCC sample.

[0939] Depending on the implementation, a single volume visual track or multiple volume visual tracks can exist in a file. Depending on the implementation, a single volume visual track can be referred to as a single track or a V-PCC single track, and multiple volume visual tracks can be referred to as multiple tracks or multiple V-PCC tracks.

[0940] An example of the syntax structure of VolumetricVisualSampleEntry is shown below.

[0941] Class VolumetricVisualSampleEntry(codingname)extends SampleEntry(codingname){

[0942] unsigned int(8)

[32] compressor_name;

[0943] }

[0944] The compressor_name field is the name of the compressor for informational purposes. It is formatted as a fixed 32-byte field with the first byte set to the number of bytes to be displayed, followed by the number of bytes of displayable data encoded using UTF-8, and then padded to complete a total of 32 bytes (including the size byte). This field can be set to 0.

[0945] According to an embodiment, the format of the volumetric vision samples may be defined by an encoding system.

[0946] According to an embodiment, a V-PCC unit header box including a V-PCC unit header may be present in sample entries of a V-PCC track and sample entries of all video-coded V-PCC component tracks included in the scheme information. The V-PCC unit header box may contain the following V-PCC unit headers for data carried by each track.

[0947] aligned(8) class VPCCUnitHeaderBox extends FullBox('vunt',version=0,0){

[0948] vpcc_unit_header()unit_header;

[0949] }

[0950] That is, VPCCUnitHeaderBox may include vpcc_unit_header().

[0951] Figure 30 An example of the syntax structure of vpcc_unit_header() is shown.

[0952] According to an embodiment, the sample entry from which VolumetricVisualSampleEntry is inherited (ie, the superclass of VolumetricVisualSampleEntry) includes a VPCC decoder configuration box (VPCCConfigurationBox).

[0953] According to an embodiment, the VPCCConfigurationBox may include a VPCCDecoderConfigurationRecord as shown below.

[0954] Class VPCCConfigurationBox extends Box('VpcC'){

[0955] VPCCDecoderConfigurationRecord()VPCCConfig;

[0956] }

[0957] According to an embodiment, the syntax of VPCCDDecoderConfigurationRecord() may be defined as follows.

[0958] aligned(8) class VPCCDDecoderConfigurationRecord{

[0959] unsigned int(8)configurationVersion=1;

[0960] unsigned int(2)lengthSizeMinusOne;

[0961] bit(1)reserved=1;

[0962] unsigned int(5)numOfVPCCParameterSets;

[0963] for(i=0;i <numOfVPCCParameterSets;i++){

[0964] unsigned int(16)VPCCParameterSetLength;

[0965] vpcc_unit(VPCCParameterSetLength)vpccParameterSet;

[0966] }

[0967] unsigned int(8)numOfSetupUnitArrays;

[0968] for(j=0;j <numOfSetupUnitArrays;j++){

[0969] bit(1)array_completeness;

[0970] bit(1)reserved=0;

[0971] unsigned int(6)NAL_unit_type;

[0972] unsigned int(8)numNALUnits;

[0973] for(i=0;i <numNALUnits;i++){

[0974] unsigned int(16)SetupUnitLength;

[0975] nal_unit(SetupUnitLength)setupUnit;

[0976] }

[0977] }

[0978] }

[0979] configurationVersion is a version field. Incompatible changes to a record are indicated by a change in the version number.

[0980] When the value of the lengthSizeMinusOne field is incremented by 1, it indicates the length (in bytes) of the NALUnitLength field included in the V-PCC sample of the VPCCDecoderConfigurationRecord or the stream to which the VPCCDecoderConfigurationRecord is applied. For example, a size of 1 byte is indicated by "0". The value of this field is the same as the value of the ssnh_unit_size_precision_bytes_minus1 field in the sample_stream_nal_header() of the Atlas substream. Figure 36 An example of a syntax structure of sample_stream_nal_header() including a ssnh_unit_size_precision_bytes_minus1 field is shown. The value of the ssnh_unit_size_precision_bytes_minus1 field plus 1 may indicate the precision of the ssnu_vpcc_unit_size element in all sample stream NAL units, in bytes.

[0981] numOfVPCCParameterSets specifies the number of V-PCC parameter sets (VPS) signaled in VPCCDecoderConfigurationRecord.

[0982] The VPCC parameter set is an instance of sample_stream_vpcc_unit() of a V-PCC unit of type VPCC_VPS. A V-PCC unit may include vpcc_parameter_set(). That is, a VPCCParameterSet array may include vpcc_parameter_set(). Figure 28 An example of the syntax structure of sample_stream_vpcc_unit() is shown.

[0983] numOfSetupUnitArrays indicates the number of atlas NAL unit arrays of the specified type.

[0984] An n-iteration statement repeated as many times as the value of numOfSetupUnitArrays may include array_completeness.

[0985] array_completeness equal to 1 indicates that all atlas NAL units of the given type are in the following array and none are in the stream. array_completeness equal to 0 indicates that additional atlas NAL units of the specified type may be in the stream. The default and allowed values ​​are constrained by the sample entry name.

[0986] NAL_unit_type indicates the type of atlas NAL unit in the following array: NAL_unit_type is constrained to take one of the values ​​indicating a NAL_ASPS, NAL_AFPS, NAL_AAPS, NAL_PREFIX_ESEI, NAL_SUFFIX_ESEI, NAL_PREFIX_NSEI, or NAL_SUFFIX_NSEI atlas NAL unit.

[0987] The numNALUnits field indicates the number of atlas NAL units of the specified type included in the VPCCDDecoderConfigurationRecord of the stream to which the VPCCDDecoderConfigurationRecord applies. The SEI array shall contain only SEI messages.

[0988] The SetupUnitLength field indicates the size of the setupUnit field in bytes. This field includes the size of both the NAL unit header and the NAL unit payload, but does not include the length field itself.

[0989] setupUnit is an instance of sample_stream_nal_unit() containing a NAL unit of type NAL_ASPS, NAL_AFPS, NAL_AAPS, NAL_PREFIX_ESEI, NAL_PREFIX_NSEI, NAL_SUFFIX_ESEI, or NAL_SUFFIX_NSEI.

[0990] According to an embodiment, the SetupUnit array may include an Atlas parameter set that is constant for the stream referenced by the sample entry in which the VPCCDDecoderConfigurationRecord exists and includes an Atlas substream required or non-required SEI message. According to an embodiment, an Atlas setup unit may be simply referred to as a setup unit.

[0991] Depending on the embodiment, the file / segment encapsulator of the present disclosure can perform sample grouping, track grouping, single-track encapsulation of a V-PCC bitstream, or multi-track encapsulation of a V-PCC bitstream. Furthermore, the file / segment encapsulator can add signaling information including SEI messages, signaling information related to patches and objects, and / or signaling information for supporting spatial access to sample entries or separate metadata tracks in the form of boxes or FullBoxes. Each box will be described in detail below.

[0992] Next, a description will be given of information related to 3D regions signaled in samples, sample entries, sample groups, or track groups included in at least one track of a file or signaled in separate metadata tracks, signaling information including SEI messages, and / or information related to patches and objects.

[0993] SEI information structure

[0994] According to an embodiment, the SEI information structure (referred to as VPCCSEIInfoStruct or V3CSEIInfoStruct) includes required SEI atlas NAL units and / or non-essential SEI atlas NAL units as shown below.

[0995] aligned(8)class VPCCSEIInfoStruct(){

[0996] unsigned int(16)numEssentialSEIs;

[0997] for(i=0;i <numEssentialSEIs;i++){

[0998] unsigned int(16)ESEI_type;

[0999] unsigned int(16)ESEI_length

[1000] nal_unit(ESEI_length)ESEI_byte;

[1001] }

[1002] unsigned int(16)numNonEssentialSEIs;

[1003] for(i=0;i <numNonEssentialSEIs;i++){

[1004] unsigned int(16)NSEI_type;

[1005] unsigned int(16)NSEI_length

[1006] nal_unit(NSEI_length)NSEI_byte;

[1007] }

[1008] }

[1009] The numEssentialSEIs field indicates the number of essential SEI messages signaled in this structure.

[1010] The ESEI_type field indicates the type of the following essential SEI message.

[1011] The ESEI_length field indicates the byte length of the following essential SEI message.

[1012] The ESEI_byte field contains the mandatory SEI atlas NAL unit with nal_unit_type equal to NAL_PREFIX_ESEI or NAL_SUFFIX_ESEI.

[1013] numNonEssentialSEIs indicates the number of non-essential SEI messages signaled in this structure.

[1014] The NSEI_type field indicates the type of the following non-essential SEI message.

[1015] The NSEI_length field indicates the byte length of the following non-essential SEI message.

[1016] The NSEI_byte field contains the non-essential SEI atlas NAL unit with nal_unit_type equal to NAL_PREFIX_NSEI or NAL_SUFFIX_NSEI.

[1017] SEI Information Box

[1018] The SEI information box (referred to as VPCCSEIInfoBox or V3CSEIInfoBox) according to an embodiment may include the above-described SEI information structure (VPCCSEIInfo Struct()).

[1019] aligned(8)class VPCCSEIInfoBox extends Box('vsei'){

[1020] VPCCSEIInfoStruct();

[1021] }

[1022] Spatial region information structure

[1023] According to an embodiment, a 3D spatial region information structure (3DSpatialRegionStruct) and a 3D bounding box information structure (3DBoundingBoxStruct) provide information about a spatial region of volumetric media. The information about the spatial region of volumetric media includes the x, y, and z offsets of the spatial region, the width, height, and depth of the region in 3D space, and bounding box information about the volumetric media.

[1024] 3D point information structure

[1025] According to an embodiment, the 3D point information may include position information about each point as follows.

[1026] aligned(8)class 3DPoint(){

[1027] unsigned int(16)x;

[1028] unsigned int(16)y;

[1029] unsigned int(16)z;

[1030] }

[1031] 3DPoint() can contain an x ​​field, a y field, and a z field.

[1032] The x field, y field, and z field may indicate x, y, and z coordinate values ​​in Cartesian coordinates.

[1033] Cube area information structure

[1034] According to an embodiment, the cube region information may include cube region information associated with an anchor point signaled in a Cartesian coordinate system, as described below.

[1035] aligned(8)class CuboidRegionStruct(){

[1036] unsigned int(16)cuboid_dx;

[1037] unsigned int(16)cuboid_dy;

[1038] unsigned int(16)cubo...

Claims

1. A method for sending point cloud data, the method comprising the following steps: Encode point cloud data; Packing a bit stream including the encoded point cloud data into a file; as well as Send said file, wherein the bitstream is stored in one or more tracks of the file, Wherein, the file also includes signaling data, Wherein, the point cloud data consists of one or more objects. The signaling data includes at least one parameter set and information for partial access to the point cloud data, wherein the one or more objects are associated with one or more atlas tiles, wherein the one or more atlas tiles constitute an atlas frame, wherein the information for partial access includes mapping information between the one or more objects and the one or more atlas tiles, and The mapping information includes object index information for identifying each of the one or more objects, quantity information for identifying the quantity of atlas tiles associated with the object, and atlas identification information for identifying each of the atlas tiles associated with the object.

2. The point cloud data sending method according to claim 1, wherein: The point cloud data includes geometry data, attribute data, and occupancy map data encoded by a video-based encoding scheme.

3. The point cloud data sending method according to claim 1, in, The information used for partial access is at least one of static information that does not change with time or dynamic information that dynamically changes with time.

4. A point cloud data sending device, the point cloud data sending device comprising: An encoder, wherein the encoder encodes the point cloud data; A packager that packages a bit stream including the encoded point cloud data into a file; as well as a sender, the sender sending the file, wherein the bitstream is stored in one or more tracks of the file, Wherein, the file also includes signaling data, Wherein, the point cloud data consists of one or more objects. The signaling data includes at least one parameter set and information for partial access to the point cloud data, wherein the one or more objects are associated with one or more atlas tiles, wherein the one or more atlas tiles constitute an atlas frame, wherein the information for partial access includes mapping information between the one or more objects and the one or more atlas tiles, and The mapping information includes object index information for identifying each of the one or more objects, quantity information for identifying the quantity of atlas tiles associated with the object, and atlas identification information for identifying each of the atlas tiles associated with the object.

5. The point cloud data transmitting device according to claim 4, wherein: The point cloud data includes geometry data, attribute data, and occupancy map data encoded by a video-based encoding scheme.

6. The point cloud data transmitting device according to claim 4, in, The information used for partial access is at least one of static information that does not change with time or dynamic information that dynamically changes with time.

7. A method for receiving point cloud data, the method comprising the following steps: Receive documents; decapsulating the file into a bitstream comprising point cloud data, wherein the bitstream is stored in one or more tracks of the file, and wherein the file further comprises signaling data; Decoding all or part of the point cloud data based on the signaling data; and rendering all or part of the decoded point cloud data based on the signaling data, Wherein, the point cloud data consists of one or more objects. The signaling data includes at least one parameter set and information for partial access to the point cloud data, wherein the one or more objects are associated with one or more atlas tiles, wherein the one or more atlas tiles constitute an atlas frame, wherein the information for partial access includes mapping information between the one or more objects and the one or more atlas tiles, and The mapping information includes object index information for identifying each of the one or more objects, quantity information for identifying the quantity of atlas tiles associated with the object, and atlas identification information for identifying each of the atlas tiles associated with the object.

8. The point cloud data receiving method according to claim 7, wherein: The point cloud data includes geometry data, attribute data, and occupancy map data decoded by a video-based encoding scheme.

9. The point cloud data receiving method according to claim 7, in, The information used for partial access is at least one of static information that does not change with time or dynamic information that dynamically changes with time.

10. A point cloud data receiving device, the point cloud data receiving device comprising: a receiver, the receiver receiving the file; a decapsulator that decapsulates the file into a bitstream comprising point cloud data, wherein the bitstream is stored in one or more tracks of the file, and wherein the file also includes signaling data; A decoder, the decoder decoding all or part of the point cloud data based on the signaling data; and a renderer, the renderer rendering all or part of the decoded point cloud data based on the signaling data, Wherein, the point cloud data consists of one or more objects. The signaling data includes at least one parameter set and information for partial access to the point cloud data, wherein the one or more objects are associated with one or more atlas tiles, wherein the one or more atlas tiles constitute an atlas frame, wherein the information for partial access includes mapping information between the one or more objects and the one or more atlas tiles, and The mapping information includes object index information for identifying each of the one or more objects, quantity information for identifying the quantity of atlas tiles associated with the object, and atlas identification information for identifying each of the atlas tiles associated with the object.

11. The point cloud data receiving device according to claim 10, wherein: The point cloud data includes geometry data, attribute data, and occupancy map data decoded by a video-based encoding scheme.

12. The point cloud data receiving device according to claim 10, in, The information used for partial access is at least one of static information that does not change with time or dynamic information that dynamically changes with time.

Images