Point cloud data sending device, point cloud data sending method, point cloud data receiving device and point cloud data receiving method

Abstract

The point cloud data transmission method according to the embodiment may include the following steps: encoding the point cloud data; and transmitting the point cloud data. The point cloud data reception method according to the embodiment may include the following steps: receiving the point cloud data; decoding the point cloud data; and rendering the point cloud data.

Description

Technical Field

[0001] Embodiments provide a method of providing point cloud content to provide users with various services such as virtual reality (VR), augmented reality (AR), mixed reality (MR), and autonomous driving services. Background Art

[0002] A point cloud is a collection of points in a three-dimensional (3D) space. Generating point cloud data is difficult because the number of points in a 3D space is large.

[0003] High throughput is required to send and receive point cloud data. Summary of the Invention

[0004] Technical issues

[0005] An object of the present disclosure is to provide a point cloud data transmitting device, a point cloud data transmitting method, a point cloud data receiving device, and a point cloud data receiving method for efficiently transmitting and receiving point clouds.

[0006] Another object of the present disclosure is to provide a point cloud data sending device, a point cloud data sending method, a point cloud data receiving device and a point cloud data receiving method for solving delay and encoding / decoding complexity.

[0007] The embodiments are not limited to the above-mentioned purposes, and the scope of the embodiments can be extended to other purposes that can be inferred by those skilled in the art based on the entire content of this disclosure.

[0008] Technical Solution

[0009] To achieve these objectives and other advantages, and in one aspect of the present disclosure, a method for transmitting point cloud data may include the following steps: encoding the point cloud data; encapsulating the point cloud data; and transmitting the point cloud data. In another aspect of the present disclosure, an apparatus for receiving point cloud data may include: a receiver configured to receive the point cloud data; a decapsulator configured to decapsulate the point cloud data; and a decoder configured to decode the point cloud data.

[0010] Beneficial effects

[0011] The point cloud data transmitting method, point cloud data transmitting device, point cloud data receiving method, and point cloud data receiving device according to the embodiments can provide a point cloud service of good quality.

[0012] The point cloud data transmitting method, point cloud data transmitting device, point cloud data receiving method, and point cloud data receiving device according to the embodiments can implement various video encoding and decoding methods.

[0013] The point cloud data transmitting method, the point cloud data transmitting apparatus, the point cloud data receiving method, and the point cloud data receiving apparatus according to the embodiments can provide general point cloud content such as autonomous driving services. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this application. The accompanying drawings illustrate embodiments of the present disclosure and together with the description serve to explain the principles of the present disclosure. In the drawings:

[0015] Figure 1 An exemplary structure of a transmission / reception system for providing point cloud content according to an embodiment is shown.

[0016] Figure 2 Shown is the capture of point cloud data according to an embodiment.

[0017] Figure 3 Shown are exemplary point clouds, geometry, and texture images according to embodiments.

[0018] Figure 4 An exemplary V-PCC encoding process according to an embodiment is shown.

[0019] Figure 5 Shown are examples of tangent planes and normal vectors of surfaces according to embodiments.

[0020] Figure 6 An exemplary bounding box of a point cloud is shown, according to an embodiment.

[0021] Figure 7 An example of determination of the positions of individual patches on an occupancy map according to an embodiment is shown.

[0022] Figure 8 Exemplary relationships between the normal axis, the tangential axis, and the dual tangential axes according to embodiments are shown.

[0023] Figure 9 Exemplary configurations of the minimum mode and the maximum mode of the projection mode according to the embodiment are shown.

[0024] Figure 10 An exemplary EDD code according to an embodiment is shown.

[0025] Figure 11 An example of recoloring based on color values ​​of neighboring points according to an embodiment is shown.

[0026] Figure 12 An example of push-pull background fill according to an embodiment is shown.

[0027] Figure 13 An exemplary possible traversal order of a 4*4 block according to an embodiment is shown.

[0028] Figure 14 An exemplary optimal traversal order according to an embodiment is shown.

[0029] Figure 15 An exemplary 2D video / image encoder according to an embodiment is shown.

[0030] Figure 16 An exemplary V-PCC decoding process according to an embodiment is shown.

[0031] Figure 17 An exemplary 2D video / image decoder according to an embodiment is shown.

[0032] Figure 18 is a flowchart illustrating the operation of the transmitting device according to an embodiment of the present disclosure.

[0033] Figure 19 is a flowchart illustrating the operation of the receiving device according to the embodiment.

[0034] Figure 20 An exemplary architecture for V-PCC based storage and streaming of point cloud data is shown, according to an embodiment.

[0035] Figure 21 is an exemplary block diagram of an apparatus for storing and transmitting point cloud data according to an embodiment.

[0036] Figure 22 is an exemplary block diagram of a point cloud data receiving apparatus according to an embodiment.

[0037] Figure 23 An exemplary structure that can be operated in conjunction with a method / apparatus for transmitting / receiving point cloud data according to an embodiment is shown.

[0038] Figure 24 An exemplary structure of a V-PCC bitstream according to an embodiment is shown.

[0039] Figure 25 The structure of a bitstream including point cloud data according to an embodiment is shown.

[0040] Figure 26 A V-PCC cell and a V-PCC cell head according to an embodiment are shown.

[0041] Figure 27 The payload of a V-PCC unit according to an embodiment is shown.

[0042] Figure 28 A V-PCC parameter set according to an embodiment is shown.

[0043] Figure 29An atlas frame according to an embodiment is shown.

[0044] Figure 30 The structure of an atlas bitstream according to an embodiment is shown.

[0045] Figure 31 Schematic diagram showing a sample stream NAL unit, a sample stream NAL unit header, a NAL unit, and a NAL unit header included in a bitstream containing point cloud data according to an embodiment.

[0046] Figure 32 NAL unit types according to embodiments are shown.

[0047] Figure 33 An atlas sequence parameter set according to an embodiment is shown.

[0048] Figure 34 An atlas frame parameter set according to an embodiment is shown.

[0049] Figure 35 Atlas_frame_tile_information according to an embodiment is shown.

[0050] Figure 36 An atlas_adaptation_parameter_set according to an embodiment is shown.

[0051] Figure 37 Shows atlas_camera_parameters according to an embodiment.

[0052] Figure 38 An atlas_tile_group_layer and an atlas_tile_group_header according to an embodiment are shown.

[0053] Figure 39 A reference list structure (ref_list_struct) according to an embodiment is shown.

[0054] Figure 40 Atlas tile group data (atlas_tile_group_data_unit) according to an embodiment is shown.

[0055] Figure 41 A patch pattern according to an embodiment is shown.

[0056] Figure 42 Patch_information_data according to an embodiment is shown.

[0057] Figure 43 Patch information according to an embodiment is shown.

[0058] Figure 44 An SEI message according to an embodiment is shown.

[0059] Figure 45 An encapsulated V-PCC data container structure according to an embodiment is shown.

[0060] Figure 46 The structure of a file according to an embodiment is shown.

[0061] Figure 47 An overview of the structure used to encapsulate non-timed V-PCC data according to an embodiment is shown.

[0062] Figure 48 The structure of a V3C project according to an embodiment is shown.

[0063] Figure 49 A V3C project according to an embodiment is shown.

[0064] Figures 50 to 51 A V3C project according to an embodiment is shown.

[0065] Figures 52 to 53 A V3C project according to an embodiment is shown.

[0066] Figure 54 A method of transmitting point cloud data according to an embodiment is shown.

[0067] Figure 55 A method of receiving point cloud data according to an embodiment is shown. DETAILED DESCRIPTION

[0068] Reference will now be made in detail to preferred embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. The detailed description given below with reference to the accompanying drawings is intended to illustrate exemplary embodiments of the present disclosure and is not intended to illustrate the only embodiments that can be implemented according to the present disclosure. The following detailed description includes specific details in order to provide a thorough understanding of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure can be practiced without these specific details.

[0069] Although most of the terms used in this disclosure are selected from general terms widely used in the art, some terms are arbitrarily selected by the applicant, and their meanings are explained in detail in the following description as needed. Therefore, this disclosure should be understood based on the intended meaning of the terms rather than their simple names or meanings.

[0070] Figure 1 An exemplary structure of a transmission / reception system for providing point cloud content according to an embodiment is shown.

[0071] The present disclosure provides a method for providing point cloud content to provide users with various services such as virtual reality (VR), augmented reality (AR), mixed reality (MR), and autonomous driving. Point cloud content according to an embodiment represents data representing an object as points and may be referred to as point cloud, point cloud data, point cloud video data, point cloud image data, etc.

[0072] According to an embodiment, the point cloud data transmitting device 10000 may include a point cloud video acquirer 10001, a point cloud video encoder 10002, a file / segment encapsulation module 10003 and / or a transmitter (or communication module) 10004. The transmitting device according to the embodiment may ensure and process the point cloud video (or point cloud content) and transmit it. According to an embodiment, the transmitting device may include a fixed station, a base transceiver system (BTS), a network, an artificial intelligence (AI) device and / or system, a robot, and an AR / VR / XR device and / or a server. According to an embodiment, the transmitting device 10000 may include a device robot, a vehicle, an AR / VR / XR device, a portable device, a home appliance, an Internet of Things (IoT) device, and an AI device / server, which is configured to communicate with a base station and / or other wireless devices using a radio access technology (e.g., 5G New RAT (NR), Long Term Evolution (LTE)).

[0073] The point cloud video acquirer 10001 according to the embodiment acquires point cloud video by capturing, synthesizing or generating a process of point cloud video.

[0074] The point cloud video encoder 10002 according to an embodiment encodes point cloud video data. Depending on the embodiment, the point cloud video encoder 10002 may be referred to as a point cloud encoder, a point cloud data encoder, an encoder, etc. Point cloud compression encoding (encoding) according to an embodiment is not limited to the above-described embodiment. The point cloud video encoder may output a bitstream containing the encoded point cloud video data. The bitstream may include not only the encoded point cloud video data but also signaling information related to the encoding of the point cloud video data.

[0075] According to embodiments, the encoder may support both the geometry-based point cloud compression (G-PCC) coding scheme and / or the video-based point cloud compression (V-PCC) coding scheme. Furthermore, the encoder may encode point clouds (referring to point cloud data or points) and / or signaling data associated with point clouds. Specific encoding operations according to embodiments are described below.

[0076] As used herein, the term V-PCC may stand for Video-Based Point Cloud Compression (V-PCC). The term V-PCC may be synonymous with Visual Volumetric Video Coding (V3C). These terms may be used complementary.

[0077] According to an embodiment, the file / segment encapsulation module 10003 encapsulates point cloud data in the form of files and / or segments. According to an embodiment, the point cloud data transmission method / device can transmit point cloud data in the form of files and / or segments.

[0078] The transmitter (or communication module) 10004 according to an embodiment transmits the encoded point cloud video data in the form of a bitstream. According to an embodiment, the file or clip can be sent to the receiving device via a network or stored in a digital storage medium (e.g., USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc.). The transmitter according to an embodiment can communicate wirelessly / wirelessly with the receiving device (or receiver) via a 4G, 5G, 6G, etc. network. In addition, the transmitter can perform necessary data processing operations based on the network system (e.g., a 4G, 5G, or 6G communication network system). The transmitting device can send the encapsulated data in an on-demand manner.

[0079] According to an embodiment, the point cloud data receiving device 10005 may include a receiver 10006, a file / segment decapsulation module 10007, a point cloud video decoder 10008, and / or a renderer 10009. According to an embodiment, the receiving device may include a device robot, a vehicle, an AR / VR / XR device, a portable device, a home appliance, an Internet of Things (IoT) device, and an AI device / server, which is configured to communicate with a base station and / or other wireless devices using a radio access technology (e.g., 5G New RAT (NR), Long Term Evolution (LTE)).

[0080] According to an embodiment, the receiver 10006 receives a bitstream containing point cloud video data. According to an embodiment, the receiver 10006 may send feedback information to the point cloud data transmitting apparatus 10000.

[0081] The file / segment decapsulation module 10007 decapsulates the file and / or segment containing the point cloud data. The decapsulation module according to the embodiment may perform the reverse process of the encapsulation process according to the embodiment.

[0082] The point cloud video decoder 10008 decodes the received point cloud video data. The decoder according to the embodiment may perform the inverse process of the encoding according to the embodiment.

[0083] The renderer 10007 renders the decoded point cloud video data. Depending on the embodiment, the renderer 10007 may send feedback information obtained at the receiving end to the point cloud video decoder 10008. Depending on the embodiment, the point cloud video data may carry the feedback information to the receiver. Depending on the embodiment, the feedback information received by the point cloud transmitting device may be provided to the point cloud video encoder.

[0084] The arrow indicated by the dotted line in the figure represents the transmission path of the feedback information obtained by the receiving device 10005. Feedback information is information used to reflect the interactivity with the user consuming the point cloud content, and includes user information (e.g., head orientation information), viewport information, etc.). Specifically, when the point cloud content is a service that requires interaction with the user (e.g., an autonomous driving service, etc.), the feedback information can be provided to the content sending side (e.g., the sending device 10000) and / or the service provider. Depending on the embodiment, the feedback information can be used in the receiving device 10005 as well as the sending device 10000, and may not be provided.

[0085] According to an embodiment, head orientation information refers to information regarding the user's head position, orientation, angle, movement, and the like. According to an embodiment, the receiving device 10005 may calculate viewport information based on the head orientation information. Viewport information may be information regarding the area of ​​the point cloud video being viewed by the user. The viewpoint is the point at which the user is viewing the point cloud video and may refer to the center of the viewport area. In other words, the viewport is the area centered on the viewpoint, and the size and shape of the area may be determined by the field of view (FOV). Therefore, in addition to head orientation information, the receiving device 10005 may extract viewport information based on the vertical or horizontal FOV supported by the device. Furthermore, the receiving device 10005 may perform gaze analysis to examine how the user consumes the point cloud, the area within the point cloud video the user is gazing at, the duration of the gaze, and other information. According to an embodiment, the receiving device 10005 may transmit feedback information including the gaze analysis results to the transmitting device 10000. According to an embodiment, the feedback information may be obtained during the rendering and / or display process. According to an embodiment, the feedback information may be provided by one or more sensors included in the receiving device 10005. In addition, depending on the embodiment, the feedback information may be ensured by the renderer 10009 or a separate external element (or device, component, etc.). Figure 1 The dotted line in represents the processing of feedback information ensured by the sending renderer 10009. The point cloud content providing system can process (encode / decode) the point cloud data based on the feedback information. Therefore, the point cloud video decoder 10008 can perform a decoding operation based on the feedback information. The receiving device 10005 can send feedback information to the sending device. The sending device (or the point cloud video encoder 10002) can perform an encoding operation based on the feedback information. Therefore, the point cloud content providing system can effectively process necessary data (for example, point cloud data corresponding to the user's head position) based on the feedback information, rather than processing (encoding / decoding) all point cloud data, and provide the point cloud content to the user.

[0086] According to an embodiment, the transmitting device 10000 may be referred to as an encoder, a transmitting device, a transmitter, etc., and the receiving device 10005 may be referred to as a decoder, a receiving device, a receiver, etc.

[0087] According to the embodiment Figure 1 The point cloud data processed in the point cloud content providing system (through a series of processes such as acquisition / encoding / transmission / decoding / rendering) may be referred to as point cloud content data or point cloud video data. Depending on the embodiment, point cloud content data may be used as a concept that includes metadata or signaling information related to point cloud data.

[0088] Figure 1 The elements of the illustrated point cloud content providing system may be implemented by hardware, software, a processor, and / or a combination thereof.

[0089] Embodiments may provide a method of providing point cloud content to provide users with various services such as virtual reality (VR), augmented reality (AR), mixed reality (MR), and autonomous driving.

[0090] To provide point cloud content services, a point cloud video must first be acquired. This acquired point cloud video can then be sent through a series of processing steps. The receiving end can then process the received data back into the original point cloud video and render the processed point cloud video. This point cloud video can then be provided to the user. Embodiments provide a method for efficiently performing this series of processing steps.

[0091] All processing for providing point cloud content services (point cloud data sending method and / or point cloud data receiving method) may include acquisition processing, encoding processing, transmission processing, decoding processing, rendering processing and / or feedback processing.

[0092] According to an embodiment, the process of providing point cloud content (or point cloud data) may be referred to as a point cloud compression process. According to an embodiment, the point cloud compression process may refer to a geometry-based point cloud compression process.

[0093] Each element of the point cloud data transmitting apparatus and the point cloud data receiving apparatus according to the embodiment may be hardware, software, a processor and / or a combination thereof.

[0094] To provide point cloud content services, a point cloud video can be acquired. This acquired point cloud video is sent through a series of processing steps. The receiving end then processes the received data back into the original point cloud video and renders the processed point cloud video. This point cloud video can then be provided to users. Embodiments provide a method for efficiently performing this series of processing steps.

[0095] All processing for providing point cloud content services may include acquisition processing, encoding processing, transmission processing, decoding processing, rendering processing and / or feedback processing.

[0096] The point cloud compression system may include a transmitting device and a receiving device. The transmitting device may encode the point cloud video to output a bitstream, which is then transmitted to the receiving device via a digital storage medium or a network in the form of a file or stream (streaming segment). Digital storage media may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD.

[0097] The transmitting device may include a point cloud video acquirer, a point cloud video encoder, a file / segment encapsulator, and a transmitter. The receiving device may include a receiver, a file / segment decapsulator, a point cloud video decoder, and a renderer. The encoder may be referred to as a point cloud video / image / image / frame encoder, and the decoder may be referred to as a point cloud video / image / image / frame decoding device. The transmitter may be included in the point cloud video encoder. The receiver may be included in the point cloud video decoder. The renderer may include a display. The renderer and / or display may be configured as separate devices or external components. The transmitting device and the receiving device may also include separate internal or external modules / units / components for feedback processing.

[0098] According to an embodiment, the operation of the receiving device may be the inverse process of the operation of the transmitting device.

[0099] The point cloud video acquirer can acquire point cloud video by capturing, arranging, or generating point cloud video. During the acquisition process, data representing the 3D positions (x, y, z) and attributes (color, reflectivity, transparency, etc.) of multiple points can be generated, such as files in the polygon file format (PLY) (or Stanford Triangle format). For videos with multiple frames, one or more files can be acquired. During the capture process, point cloud-related metadata (e.g., capture-related metadata) can be generated.

[0100] According to an embodiment, a point cloud data transmitting apparatus may include an encoder configured to encode point cloud data and a transmitter configured to transmit the point cloud data. The data may be transmitted in the form of a bit stream containing the point cloud.

[0101] A point cloud data receiving apparatus according to an embodiment may include a receiver configured to receive point cloud data, a decoder configured to decode the point cloud data, and a renderer configured to render the point cloud data.

[0102] The method / apparatus according to the embodiment represents a point cloud data sending apparatus and / or a point cloud data receiving apparatus.

[0103] Figure 2 Shown is the capture of point cloud data according to an embodiment.

[0104] Point cloud data according to an embodiment may be acquired by a camera, etc. Capturing techniques according to an embodiment may include, for example, facing inward and / or facing outward.

[0105] In the inward facing according to an embodiment, one or more cameras facing inwardly toward an object of point cloud data may photograph the object from outside the object.

[0106] In outward facing according to an embodiment, one or more cameras facing outward toward an object of point cloud data may photograph the object. For example, according to an embodiment, there may be four cameras.

[0107] Point cloud data or point cloud content according to an embodiment may be a video or still image of an object / environment represented in various types of 3D space. According to an embodiment, point cloud content may include video / audio / image of an object.

[0108] To capture point cloud content, a camera device capable of acquiring depth (a combination of an infrared pattern projector and an infrared camera) and an RGB camera capable of extracting color information corresponding to the depth information can be configured. Alternatively, depth information can be extracted using LiDAR, a radar system that measures the position coordinates of reflectors by emitting laser pulses and measuring the return time. The geometric shape composed of points in 3D space can be extracted from the depth information, and attributes representing the color / reflectivity of each point can be extracted from the RGB information. Point cloud content can include information about the position (x, y, z) and the color (YCbCr or RGB) or reflectivity (r) of each point. For point cloud content, both outward-facing techniques, which capture the external environment, and inward-facing techniques, which capture the central object, can be used. In VR / AR environments, when objects (e.g., core objects such as characters, players, objects, or actors) are incorporated into point cloud content that can be viewed from any direction (360 degrees), the capture camera configuration can be based on the inward-facing technique. When the current surrounding environment is incorporated into the point cloud content in vehicle modes such as autonomous driving, the capture camera configuration can be based on the outward-facing technique. Since point cloud content may be captured by multiple cameras, a camera calibration process may need to be performed to configure the global coordinate system for the cameras before capturing the content.

[0109] Point cloud content can be videos or still images of objects / environments existing in various types of 3D spaces.

[0110] Furthermore, in the point cloud content acquisition method, any point cloud video can be compiled based on captured point cloud videos. Alternatively, when providing a computer-generated point cloud video of a virtual space, it may not be necessary to capture the video using an actual camera. In this case, the capture process can simply be replaced by a process that generates the relevant data.

[0111] The captured point cloud video may need to be post-processed to improve the quality of the content. In the video capture process, the maximum / minimum depth can be adjusted within the range provided by the camera device. Even after the adjustment, point data of unwanted areas may still exist. Therefore, post-processing may be performed to remove unwanted areas (e.g., background) or to identify connected spaces and fill spatial holes. In addition, the point clouds extracted from cameras sharing a spatial coordinate system can be integrated into one piece of content by transforming each point into a global coordinate system based on the position coordinates of each camera obtained through a calibration process. Thus, a piece of point cloud content with a wide range can be generated, or point cloud content with a high density of points can be obtained.

[0112] The point cloud video encoder can encode the input point cloud video into one or more video streams. A video may include multiple frames, and each frame may correspond to a still image / picture. In this specification, the point cloud video may include a point cloud image / frame / picture. In addition, the term "point cloud video" may be used interchangeably with point cloud image / frame / picture / video / audio. The point cloud video encoder may perform video-based point cloud compression (V-PCC) processing. For compression and coding efficiency, the point cloud video encoder may perform a series of processes such as prediction, transformation, quantization, and entropy coding. The encoded data (encoded video / image information) may be output in the form of a bitstream. Based on the V-PCC processing, the point cloud video encoder may encode the point cloud video by dividing it into a geometry video, an attribute video, an occupancy map video, and auxiliary information (to be described later). The geometry video may include a geometry image, the attribute video may include an attribute image, and the occupancy map video may include an occupancy map image. The auxiliary information may include auxiliary patch information. The attribute video / image may include a texture video / image.

[0113] The encapsulation processor (file / segment encapsulation module) 10003 can encapsulate the encoded point cloud video data and / or metadata related to the point cloud video, for example, in the form of a file. Here, the metadata related to the point cloud video can be received from the metadata processor. The metadata processor may be included in the point cloud video encoder or configured as a separate component / module. The encapsulation processor can encapsulate data in a file format such as ISOBMFF or process data in the form of DASH segments, etc. Depending on the embodiment, the encapsulation processor may include the point cloud video-related metadata in a file format. The point cloud video metadata may be included in boxes at various levels of the ISOBMFF file format, for example, or as data in separate tracks within the file. Depending on the embodiment, the encapsulation processor may encapsulate the point cloud video-related metadata into a file. The transmission processor may perform transmission processing on the point cloud video data encapsulated in the file format. The transmission processor may be included in the transmitter or configured as a separate component / module. The transmission processor may process the point cloud video data according to a transmission protocol. The transmission processing may include processing for transmission via a broadcast network and processing for transmission via broadband. According to an embodiment, the transmission processor may receive point cloud video related metadata together with the point cloud video data from the metadata processor and perform processing of the point cloud video data for transmission.

[0114] Transmitter 10004 can transmit the encoded video / image information or data output in the form of a bitstream in the form of a file or stream to a receiver of a receiving device via a digital storage medium or network. Digital storage media may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD. The transmitter may include components for generating a media file in a predetermined file format and may also include components for transmission via a broadcast / communication network. The receiver may extract the bitstream and transmit the extracted bitstream to a decoding device.

[0115] Receiver 10006 can receive point cloud video data transmitted by the point cloud video transmitting device according to the present disclosure. Depending on the transmission channel, the receiver can receive point cloud video data via a broadcast network or via broadband. Alternatively, the point cloud video data can be received via a digital storage medium.

[0116] The receiving processor can process the received point cloud video data according to the transmission protocol. The receiving processor can be included in the receiver or configured as a separate component / module. The receiving processor can reverse the processing of the transmitting processor, so that the processing corresponds to the transmission processing performed on the transmitting side. The receiving processor can transmit the acquired point cloud video data to the decapsulation processor and transmit the acquired point cloud video-related metadata to the metadata parser. The point cloud video-related metadata acquired by the receiving processor can take the form of a signaling table.

[0117] The decapsulation processor (file / segment decapsulation module) 10007 can decapsulate the point cloud video data received from the receiving processor in the form of a file. The decapsulation processor can decapsulate the file according to ISOBMFF, etc., and can obtain a point cloud video bitstream or point cloud video related metadata (metadata bitstream). The obtained point cloud video bitstream can be transmitted to the point cloud video decoder, and the obtained point cloud video related metadata (metadata bitstream) can be transmitted to the metadata processor. The point cloud video bitstream may include metadata (metadata bitstream). The metadata processor may be included in the point cloud video decoder, or may be configured as a separate component / module. The point cloud video related metadata obtained by the decapsulation processor may take the form of a box or track in the file format. When necessary, the decapsulation processor may receive the metadata required for decapsulation from the metadata processor. The point cloud video related metadata may be transmitted to the point cloud video decoder and used in the point cloud video decoding process, or may be transmitted to the renderer and used in the point cloud video rendering process.

[0118] The point cloud video decoder may receive a bitstream and decode the video / image by performing operations corresponding to those of the point cloud video encoder. In this case, the point cloud video decoder may decode the point cloud video by separating the point cloud video into a geometry video, an attribute video, an occupancy map video, and auxiliary information as described below. The geometry video may include a geometry image, the attribute video may include an attribute image, the occupancy map video may include an occupancy map image, the auxiliary information may include auxiliary patch information, and the attribute video / image may include a texture video / image.

[0119] The 3D geometry can be reconstructed based on the decoded geometry image, occupancy map, and auxiliary patch information, and then smoothed. A color point cloud image / screen can be reconstructed by assigning color values ​​to the smoothed 3D geometry based on the texture image. A renderer can render the reconstructed geometry and color point cloud image / screen. The rendered video / image can be displayed on a display. The user can view all or part of the rendered result on a VR / AR display or a typical display.

[0120] Feedback processing may include transmitting various types of feedback information that can be obtained in the rendering / display process to a decoder on the sending side or the receiving side. Feedback processing can provide interactivity when consuming point cloud videos. Depending on the embodiment, head orientation information, viewport information indicating the area currently viewed by the user, etc. can be transmitted to the sending side during feedback processing. Depending on the embodiment, the user can interact with things implemented in the VR / AR / MR / autonomous driving environment. In this case, information related to the interaction can be transmitted to the sending side or the service provider during the feedback processing. Depending on the embodiment, feedback processing can be skipped.

[0121] The head orientation information may represent information about the position, angle, and motion of the user's head. Based on this information, information about the area of ​​the point cloud video that the user is currently viewing (i.e., viewport information) may be calculated.

[0122] Viewport information can be information about the area of ​​the point cloud video that a user is currently viewing. This information can be used to perform gaze analysis to examine how the user consumes the point cloud video, the area of ​​the point cloud video the user is gazing at, and how long the user gazes at that area. Gaze analysis can be performed on the receiving side, and the analysis results can be transmitted to the sending side over a feedback channel. Devices such as VR / AR / MR displays can extract the viewport area based on the position / orientation of the user's head, the vertical or horizontal FOV supported by the device, and other factors.

[0123] Depending on the implementation, this feedback information can be not only transmitted to the sending side but also consumed by the receiving side. Specifically, decoding and rendering processing on the receiving side can be performed based on this feedback information. For example, based on head orientation information and / or viewport information, only the point cloud video of the area currently being viewed by the user can be prioritized for decoding and rendering.

[0124] Here, the viewport or viewport area may refer to the area of ​​the point cloud video currently being viewed by the user. The viewpoint is the point in the point cloud video that the user is viewing and may represent the center point of the viewport area. In other words, the viewport is the area surrounding the viewpoint, and the size and shape of the area may be determined by the field of view (FOV).

[0125] The present disclosure relates to point cloud video compression as described above. For example, the methods / implementations disclosed in the present disclosure may be applied to the Point Cloud Compression or Point Cloud Coding (PCC) standard of the Moving Picture Experts Group (MPEG) or next-generation video / image coding standards.

[0126] As used herein, a picture / frame may generally refer to a unit representing one image in a specific time interval.

[0127] A pixel or picture element may be the smallest unit constituting a picture element (or image). Furthermore, "sample" may be used as a term corresponding to a pixel. A sample may generally represent a pixel or a pixel value, or may represent only a pixel / pixel value of a luma component, only a pixel / pixel value of a chroma component, or only a pixel / pixel value of a depth component.

[0128] A unit may represent a basic unit of image processing. A unit may include at least one of a specific area of ​​a picture and information related to the area. In some cases, a unit may be used interchangeably with terms such as a block or an area. In general, an M×N block may include samples (or sample arrays) or a set (or array) of transform coefficients arranged in M ​​columns and N rows.

[0129] Figure 3Examples of point clouds, geometric images, and texture images according to embodiments are shown.

[0130] The point cloud according to the embodiment may be input to a Figure 4 According to an embodiment, the point cloud may have the same meaning as point cloud data.

[0131] As shown in the figure, the left part shows a point cloud, where objects are located in 3D space and can be represented by a bounding box, etc. The middle part shows the geometry, and the right part shows a texture image (non-filled image).

[0132] Video-based point cloud compression (V-PCC) according to an embodiment can provide a method for compressing 3D point cloud data based on a 2D video codec such as HEVC or VVC. The data and information that can be generated in the V-PCC compression process are as follows:

[0133] Occupancy map: This is a binary map that uses a value of 0 or 1 to indicate whether data exists at corresponding locations in the 2D plane when the points constituting the point cloud are divided into patches and mapped to the 2D plane. The occupancy map can represent a 2D array corresponding to an atlas, and the values ​​of the occupancy map can indicate whether each sample location in the atlas corresponds to a 3D point.

[0134] An atlas is a collection of 2D bounding boxes and information related thereto, located in a rectangular frame, that corresponds to a 3D bounding box in the 3D space of the rendered volume data.

[0135] An atlas bitstream is a bitstream of one or more atlas frames and related data constituting an atlas.

[0136] An atlas frame is a 2D rectangular array of atlas samples onto which a patch is projected.

[0137] An atlas sample is the location of the rectangular frame to which the patch associated with the atlas is projected.

[0138] The atlas frame can be divided into tiles. A tile is a unit into which a 2D frame is divided. That is, a tile is a unit for dividing signaling information of point cloud data called an atlas.

[0139] Patch: A collection of points that make up a point cloud, which indicates that points belonging to the same patch are adjacent to each other in 3D space and are mapped in the same direction between the 6-sided bounding box planes in the process of mapping to a 2D image.

[0140] A patch is a unit that a tile is divided into. A patch is signaling information about the configuration of point cloud data.

[0141] The receiving device according to the embodiment can restore attribute video data, geometry video data, and occupancy video data (actual video data having the same presentation time) based on the atlas (tiles, patches).

[0142] Geometry image: This is an image in the form of a depth map that presents positional information (geometry) about the individual points that make up the point cloud, patch by patch. A geometry image can consist of a single channel of pixel values. The geometry represents the set of coordinates associated with a point cloud frame.

[0143] Texture Image: This is an image that represents the color information of each point in the point cloud on a patch-by-patch basis. A texture image may consist of pixel values ​​for multiple channels (e.g., R, G, and B). Textures are included in attributes. Depending on the implementation, textures and / or attributes may be interpreted as the same object and / or may contain a context.

[0144] Auxiliary patch information: This indicates the metadata required to reconstruct the point cloud using each patch. The auxiliary patch information may include information about the position, size, etc. of the patch in 2D / 3D space.

[0145] Point cloud data (eg, V-PCC components) according to embodiments may include an atlas, an occupancy map, geometry, and attributes.

[0146] An atlas represents a set of 2D bounding boxes. It can be a patch, for example, projected onto a rectangular frame. An atlas can correspond to a 3D bounding box in 3D space and can represent a subset of a point cloud.

[0147] Attributes can represent scalars or vectors associated with each point in the point cloud. For example, attributes can include color, reflectivity, surface normal, timestamp, material ID.

[0148] Point cloud data according to an embodiment represents PCC data according to a video-based point cloud compression (V-PCC) scheme. Point cloud data may include multiple components. For example, it may include occupancy maps, patches, geometry, and / or textures.

[0149] Figure 4 A V-PCC encoding process according to an embodiment is shown.

[0150] The figure shows the V-PCC encoding process for generating and compressing occupancy maps, geometry images, texture images and auxiliary patch information. Figure 4 The V-PCC coding process can be done by Figure 1 The point cloud video encoder 10002 processes. Figure 4 The various elements of the present invention may be executed by software, hardware, a processor and / or a combination thereof.

[0151] The patch generator 40000 receives a point cloud frame (which may be in the form of a bitstream containing point cloud data). The patch generator 40000 generates patches from the point cloud data. In addition, patch information including information about patch generation is generated.

[0152] Patch packing or patch packer 40001 packs patches for point cloud data. For example, one or more patches may be packed. In addition, the patch packer generates an occupancy map containing information about the patch packing.

[0153] The geometric image generation or geometric image generator 40002 generates a geometric image based on point cloud data, patches and / or packed patches. The geometric image refers to data containing geometry related to the point cloud data.

[0154] Texture image generation or texture image generator 40003 generates a texture image based on point cloud data, patches and / or packed patches. In addition, the texture image can be further generated based on smooth geometry generated by smoothing based on patch information.

[0155] Smoothing or smoother 40004 can reduce or eliminate errors contained in the image data. For example, based on the patched reconstructed geometric image, parts between the data that may cause errors can be smoothly filtered out to generate smooth geometry.

[0156] Auxiliary patch information compression or auxiliary patch information compressor 40005 compresses auxiliary patch information related to the patch information generated in patch generation. In addition, the compressed auxiliary patch information can be sent to the multiplexer. The auxiliary patch information can be generated in geometric image generation 40002.

[0157] Image filling or image fillers 40006 and 40007 can fill geometric images and texture images respectively. Filling data can be filled into geometric images and texture images.

[0158] Group expansion or group expander 40008 can add data to the texture image in a similar manner to image filling. The added data can be inserted into the texture image.

[0159] Video compression or video compressors 40009, 40010, 40011 can compress the filled geometry image, filled texture image and / or occupancy map, respectively. Compression can encode geometry information, texture information, occupancy information, etc.

[0160] Entropy compression or entropy compressor 40012 can compress (e.g., encode) the occupancy map based on an entropy scheme.

[0161] According to an embodiment, entropy compression and / or video compression may be performed depending on whether the point cloud data is lossless and / or lossy, respectively.

[0162] Multiplexer 40013 multiplexes the compressed geometry image, compressed texture image, and compressed occupancy map into a bitstream.

[0163] Described below Figure 4 Specific operations in each process.

[0164] Patch generation 40000

[0165] The patch generation process is a process of dividing a point cloud into patches (mapping units) in order to map the point cloud to a 2D image. The patch generation process can be divided into three steps: normal value calculation, segmentation, and patch segmentation.

[0166] Will refer to Figure 5 The normal value calculation process is described in detail.

[0167] Figure 5 Shown are examples of tangent planes and normal vectors of surfaces according to embodiments.

[0168] Figure 5 The surface is used as follows Figure 4 The patch generation process of the V-PCC encoding process is in 40000.

[0169] Normal calculation related to patch generation:

[0170] Each point in a point cloud has its own orientation, represented by a 3D vector called a normal vector. Using the neighbors of each point, obtained using a KD tree or other method, it is possible to determine the tangent plane and normal vector for each point that makes up the surface of the point cloud, as shown in the figure. The search range used in the neighbor search process can be defined by the user.

[0171] A tangent plane is a plane that passes through a point on a surface and completely includes the tangent line to a curve on the surface.

[0172] Figure 6 An exemplary bounding box of a point cloud is shown, according to an embodiment.

[0173] Methods / apparatuses according to embodiments (eg, patch generation) may employ bounding boxes when generating patches from point cloud data.

[0174] The bounding box according to the embodiment refers to a box for dividing cells of point cloud data based on hexahedrons in a 3D space.

[0175] The bounding box can be used in the process of projecting the target object of the point cloud data onto the plane of each flat face of the hexahedron in the 3D space. Figure 1 The point cloud video acquirer 10001 and the point cloud video encoder 10002 generate and process. In addition, based on the bounding box, it can be performed Figure 2The V-PCC encoding process includes patch generation 40000, patch packing 40001, geometric image generation 40002 and texture image generation 40003.

[0176] Segments related to patch generation

[0177] Segmentation is divided into two processes: initial segmentation and refined segmentation.

[0178] The point cloud encoder 10002 according to an embodiment projects the points onto one of the faces of the bounding box. Specifically, each point constituting the point cloud is projected onto one of the six faces of the bounding box surrounding the point cloud. Initial segmentation is the process of determining one of the flat faces of the bounding box to which each point is projected.

[0179] It is the normal value corresponding to each of the six planar faces, defined as follows:

[0180] (1.0,0.0,0.0), (0.0,1.0,0.0), (0.0,0.0,1.0), (-1.0,0.0,0.0), (0.0,-1.0,0.0), (0.0,0.0,-1.0).

[0181] As shown in the following formula, the normal vector of each point obtained in the normal value calculation process is and The plane with the largest dot product value is determined as the projection plane of the corresponding point. That is, the plane with the most similar normal vector to the normal vector of the point is determined as the projection plane of the point.

[0182]

[0183] The determined plane can be identified by a cluster index (one of 0 to 5).

[0184] Refined segmentation is a process that enhances the projection planes of each point that makes up the point cloud, determined in the initial segmentation process, by taking into account the projection planes of neighboring points. In this process, the normal score, which indicates the similarity between the normal vector of each point considered in determining the projection plane in the initial segmentation process and the normals of each flat surface of the bounding box, and the smoothness score, which indicates the similarity between the projection plane of the current point and the projection planes of neighboring points, are considered together.

[0185] Score smoothing can be taken into account by assigning weights to the scoring methods. In this case, the weight values ​​can be user-defined. The segmentation refinement can be performed repeatedly, and the number of repetitions can also be defined by the user.

[0186] Patch segmentation related to patch generation

[0187] Patch segmentation is a process of dividing the entire point cloud into patches (sets of adjacent points) based on the projection plane information of each point constituting the point cloud obtained in the initial / refinement segmentation process. Patch segmentation may include the following steps.

[0188] ① Use KD tree or other methods to calculate the neighboring points of each point that constitutes the point cloud. The maximum number of neighbors can be defined by the user.

[0189] ②When the neighboring points are projected onto the same plane as the current point (when they have the same cluster index),

[0190] Extract the current point and its neighboring points as a patch.

[0191] ③ Calculate the geometric value of the extracted patch. The details are described below.

[0192] ④ Repeat steps ② to ④ until there are no unextracted points.

[0193] The occupancy map, geometry image, and texture image of each patch, as well as the size of each patch, are determined through a patch segmentation process.

[0194] Figure 7 An example of determining positions of individual patches on an occupancy map according to an embodiment is shown.

[0195] The point cloud encoder 10002 according to an embodiment may perform patch packing and generate an occupancy map.

[0196] Patch packing and occupancy map generation (40001)

[0197] This is the process of determining the locations of individual patches in a 2D image in order to map the segmented patches to the 2D image. As a 2D image, an occupancy map is a binary image, using values ​​of 0 or 1 to indicate the presence or absence of data at the corresponding location. The occupancy map consists of blocks, and its resolution is determined by the block size. For example, a 1x1 block achieves pixel-level resolution. The occupancy packing block size can be determined by the user.

[0198] The process of determining the location of each patch on the occupancy map may be configured as follows.

[0199] ① Set all positions on the occupancy map to 0.

[0200] ② Place the patch at a point (u,v) in the occupancy map plane with horizontal coordinates in the range (0,occupancySizeU-patch.sizeU0) and vertical coordinates in the range (0,occupancySizeV-patch.sizeV0).

[0201] ③ Set the point (x, y) in the patch plane whose horizontal coordinates are in the range (0, patch.sizeU0) and whose vertical coordinates are in the range (0, patch.sizeV0) as the current point.

[0202] ④ Change the position of point (x, y) in raster order, and if the value of coordinate (x, y) on the patch occupancy map is 1 (data exists at the point in the patch) and the value of coordinate (u+x, v+y) on the global occupancy map is 1 (the occupancy map is filled with the previous patch), repeat operations ③ and ④. Otherwise, proceed to operation ⑥.

[0203] ⑤ Change the position of (u,v) in raster order and repeat operations ③ to ⑤.

[0204] ⑥ Determine (u, v) as the position of the patch and copy the occupancy map data about the patch to the corresponding part on the global occupancy map.

[0205] ⑦ Repeat steps ② to ⑦ for the next patch.

[0206] occupancySizeU: Indicates the width of the occupancy map. Its unit is the occupancy packing block size.

[0207] occupancySizeV: Indicates the height of the occupancy map. Its unit is the occupancy packing block size.

[0208] patch.sizeU0: Indicates the width of the occupancy map. Its unit is the occupancy packing block size.

[0209] patch.sizeV0: indicates the height of the occupancy map. Its unit is the occupancy patch size.

[0210] For example, Figure 7 As shown, a box corresponding to a patch having a patch size exists in a box corresponding to an occupied packed size block, and a point (x, y) may be located in the box.

[0211] Figure 8 Exemplary relationships between the normal axis, the tangential axis, and the dual tangential axes according to embodiments are shown.

[0212] The point cloud encoder 10002 according to an embodiment may generate a geometric image. A geometric image refers to image data including geometric information about a point cloud. The geometric image generation process may be performed using Figure 8 The three axes of the patch in (normal, tangential, and bitangential).

[0213] Geometric Image Generation (40002)

[0214] In this process, the depth values ​​of the geometric images constituting each patch are determined, and the entire geometric image is generated based on the positions of the patches determined in the patch packing process described above. The process of determining the depth values ​​of the geometric images constituting each patch can be configured as follows.

[0215] ① Calculate parameters related to the position and size of each patch. The parameters may include the following information.

[0216] The normal index indicating the normal axis is obtained in the previous patch generation process. The tangential axis is the axis perpendicular to the normal axis that coincides with the horizontal axis u of the patch image, and the bi-tangential axis is the axis perpendicular to the normal axis that coincides with the vertical axis v of the patch image. The three axes can be shown in the figure.

[0217] Figure 9 Exemplary configurations of the minimum mode and the maximum mode of the projection mode according to the embodiment are shown.

[0218] The point cloud encoder 10002 according to an embodiment may perform patch-based projection to generate a geometric image, and the projection mode according to an embodiment includes a minimum mode and a maximum mode.

[0219] The 3D space coordinates of the patch may be calculated based on a minimum-sized bounding box surrounding the patch. For example, the 3D space coordinates may include the minimum tangent value of the patch (on the patch 3D shift tangent axis), the minimum bitangent value of the patch (on the patch 3D shift bitangent axis), and the minimum normal value of the patch (on the patch 3D shift normal axis).

[0220] The 2D size of a patch indicates the horizontal and vertical size of the patch when it is packed into a 2D image. The horizontal size (patch 2D size u) can be obtained as the difference between the maximum tangent value and the minimum tangent value of the bounding box, and the vertical size (patch 2D size v) can be obtained as the difference between the maximum bitangent value and the minimum bitangent value of the bounding box.

[0221] ② Determine the projection mode of the patch. The projection mode can be minimum or maximum. Geometric information about the patch is represented by depth values. When the points that make up the patch are projected along the normal to the patch, two images are generated: an image constructed using the maximum depth value and an image constructed using the minimum depth value.

[0222] In minimum mode, when generating two-layer images d0 and d1, a minimum depth can be configured for d0, and a maximum depth within the surface thickness from the minimum depth can be configured for d1, as shown in the figure.

[0223] For example, when a point cloud is in 2D as shown in the figure, there may be multiple patches containing multiple points. As shown in the figure, points marked with the same shading pattern may belong to the same patch. The figure shows the processing of a patch with a point marked blank.

[0224] When a point marked as blank is projected left / right, the depth may be increased by 1 to 0, 1, 2, ..., 6, 7, 8, 9 relative to the left, and the number used to calculate the depth of the point may be marked on the right.

[0225] The same projection mode can be applied to all point clouds, or different projection modes can be applied to each frame or patch according to user definition. When different projection modes are applied to each frame or patch, the projection mode that can enhance compression efficiency or minimize missing points can be adaptively selected.

[0226] ③Calculate the depth value of each point.

[0227] In minimum mode, image d0 is constructed using depth0, which is the value obtained by subtracting the minimum normal value of the patch (on the patch 3D shift normal axis) calculated in operation ① from the minimum normal value of the patch (on the patch 3D shift normal axis) for each point. If another depth value exists at the same location within the range between depth0 and the surface thickness, that value is set to depth1. Otherwise, the value of depth0 is assigned to depth1. Image d1 is constructed using the value of depth1.

[0228] For example, when determining the depth of a point in image d0, the minimum value (4 2 4 4 0 6 0 0 9 9 0 8 0) may be calculated. When determining the depth of a point in image d1, the larger value among two or more points may be calculated. When only one point exists, its value (4 4 4 4 6 6 6 8 9 9 8 8 9) may be calculated. In the process of encoding and reconstructing the points of the patch, some points may be lost (for example, in the figure, eight points are lost).

[0229] In maximum mode, image d0 is constructed using depth0, which is the value obtained by subtracting the minimum normal value of the patch (on the patch 3D shift normal axis) calculated in operation 1) from the minimum normal value of the patch (on the patch 3D shift normal axis) for each point. If another depth value exists within the range between depth0 and the surface thickness at the same location, that value is set to depth1. Otherwise, the value of depth0 is assigned to depth1. Image d1 is constructed using the value of depth1.

[0230] For example, when determining the depth of a point in image d0, the maximum value (4 4 4 4 6 6 6 8 9 9 8 8 9) may be calculated. Furthermore, when determining the depth of a point in image d1, the lower value among two or more points may be calculated. When only one point exists, its value (4 2 4 4 5 6 0 6 9 9 0 8 0) may be calculated. In the process of encoding and reconstructing the points of a patch, some points may be lost (for example, six points are lost in the figure).

[0231] The entire geometric image can be generated by placing the geometric images of the respective patches generated through the above-described process onto the entire geometric image based on the patch position information determined in the patch packing process.

[0232] The generated layer d1 of the entire geometric image can be encoded using various methods. The first method (absolute d1 method) encodes the depth value of the previously generated image d1. The second method (differential method) encodes the difference between the depth value of the previously generated image d1 and the depth value of image d0.

[0233] In the encoding method using depth values ​​of two layers d0 and d1 as described above, if there is another point between the two depths, geometric information about the point is lost in the encoding process, so Enhanced-Delta-Depth (EDD) code can be used for lossless encoding.

[0234] Below, we will refer to Figure 10 Describe the EDD code in detail.

[0235] Figure 10 An exemplary EDD code according to an embodiment is shown.

[0236] In some / all processing of the point cloud encoder 10002 and / or V-PCC encoding (e.g., video compression 40009), geometric information about points may be encoded based on EOD codes.

[0237] As shown in the figure, the EDD code is used to binary encode the positions of all points within the surface thickness range including d1. For example, in the figure, since points exist at the first and fourth positions on D0 and are empty at the second and third positions, the points included in the second column from the left can be represented by the EDD code 0b1001 (=9). When the EDD code is encoded and transmitted along with D0, the receiving terminal can losslessly recover the geometric information about all points.

[0238] For example, when a point exists above the reference point, the value is 1. When no point exists, the value is 0. Therefore, the code can be expressed based on 4 bits.

[0239] Smooth (40004)

[0240] Smoothing is an operation used to eliminate discontinuities that may appear on patch boundaries due to image quality degradation that occurs during the compression process. Smoothing can be performed by a point cloud encoder or a smoother:

[0241] ① Reconstruct the point cloud from the geometric image. This operation can be the opposite of the geometric image generation described above. For example, it can be the inverse process of the reconstruction encoding.

[0242] ② Use KD tree or other methods to calculate the neighboring points of each point that constitutes the reconstructed point cloud.

[0243] ③ Determine whether each point is on the patch boundary. For example, when there is a neighboring point with a different projection plane (cluster index) than the current point, it can be determined that the point is on the patch boundary.

[0244] ④ If a point exists on the patch boundary, move it to the centroid of the neighboring points (located at the average x, y, z coordinates of the neighboring points). That is, change the geometry value. Otherwise, maintain the previous geometry value.

[0245] Figure 11 An example of recoloring based on color values ​​of neighboring points according to an embodiment is shown.

[0246] The point cloud encoder or texture image generator 40003 according to an embodiment may generate a texture image based on recoloring.

[0247] Texture Image Generation (40003)

[0248] Similar to the above-described geometric image generation process, the texture image generation process includes generating texture images for each patch and arranging the texture images at a determined position to generate the entire texture image. However, in the operation of generating the texture image for each patch, instead of the depth value used for geometric generation, an image having color values ​​(e.g., R, G, and B values) of the points constituting the point cloud corresponding to the position is generated.

[0249] When estimating the color values ​​of each point making up the point cloud, the geometry previously obtained through smoothing can be used. In a smoothed point cloud, the positions of some points may have shifted relative to the original point cloud, necessitating a recoloring process to find a color appropriate for the changed positions. Recoloring can be performed using the color values ​​of neighboring points. For example, as shown in the figure, the color value of the nearest neighbor and the color values ​​of neighboring points can be considered to calculate a new color value.

[0250] For example, referring to the drawings, in recoloring, an appropriate color value of a changed position may be calculated based on an average of attribute information about original points closest to the point and / or an average of attribute information about original positions closest to the point.

[0251] Similar to the geometric image generated with two layers d0 and d1, the texture image can also be generated with two layers t0 and t1.

[0252] Auxiliary patch information compression (40005)

[0253] The point cloud encoder or the auxiliary patch information compressor according to an embodiment may compress auxiliary patch information (auxiliary information about a point cloud).

[0254] The auxiliary patch information compressor compresses the auxiliary patch information generated in the patch generation, patch packing and geometry generation processes. The auxiliary patch information may include the following parameters:

[0255] The index (cluster index) used to identify the projection plane (normal plane);

[0256] The 3D spatial position of the patch, i.e., the minimum tangent value of the patch (on the patch 3D shift tangential axis), the minimum bitangent value of the patch (on the patch 3D shift bitangent axis), and the minimum normal value of the patch (on the patch 3D shift normal axis);

[0257] The 2D spatial position and size of the patch, i.e., horizontal size (patch2d size u), vertical size (patch2d size v), minimum horizontal value (patch2d shift u), and minimum vertical value (patch2d shift u); and

[0258] Mapping information about each block and patch, that is, candidate index (when patches are arranged in order based on 2D spatial position and size information about the patches, multiple patches can be mapped to one block in an overlapping manner. In this case, the mapped patches constitute a candidate list, and the candidate index indicates the sequential position of the patch whose data exists in the block) and the local patch index (indicating the index of one patch existing in the frame). Table X shows pseudo code representing a process of matching between blocks and patches based on the candidate list and the local patch index.

[0259] The maximum number of candidate lists can be defined by the user.

[0260] Table 1-1 Pseudocode for mapping blocks to patches

[0261]

[0262] Figure 12 Shown is a push-pull background fill according to an embodiment.

[0263] Image Fill and Group Dilation (40006, 40007, 40008)

[0264] The image filler according to an embodiment may fill a space other than a patch area with meaningless supplementary data based on a push-pull background filling technique.

[0265] Image padding is the process of filling the space outside the patch area with meaningless data to improve compression efficiency. For image padding, pixel values ​​in columns or rows near the patch's boundaries are copied to fill the empty space. Alternatively, as shown in the figure, a push-pull background padding method can be used. This method uses pixel values ​​from the lower-resolution image to fill the empty space, gradually reducing the resolution of the non-filled image and then increasing it again.

[0266] Group dilation is a process that fills the empty spaces of the geometry image and texture image configured in two layers d0 / d1 and t0 / t1, respectively. In this process, the empty spaces of the two layers calculated by image filling are filled with the average of the values ​​at the same position.

[0267] Figure 13 An exemplary possible traversal order of a 4*4 block according to an embodiment is shown.

[0268] Occupancy map compression (40012, 40011)

[0269] The occupancy map compressor according to the embodiment can compress a previously generated occupancy map. Specifically, two methods can be used, namely, video compression for lossy compression and entropy compression for lossless compression. Video compression is described below.

[0270] Entropy compression can be performed by the following operations.

[0271] ① If the block constituting the occupancy map is fully occupied, encode 1 and repeat the same operation for the next block of the occupancy map. Otherwise, encode 0 and perform operations ② to ⑤.

[0272] ② Determine the best traversal order for performing run-length encoding on the occupied pixels of the block. The figure shows four possible traversal orders for a 4*4 block.

[0273] Figure 14 An exemplary optimal traversal order according to an embodiment is shown.

[0274] As described above, the entropy compressor according to the embodiment may encode a block based on the traversal order scheme as described above.

[0275] For example, the best traversal order with the smallest number of runs is selected from the possible traversal orders, and its index is encoded. Figure 13 In the case of the third traversal order in . In the case shown, the number of runs can be minimized to 2, so the third traversal order can be selected as the optimal traversal order.

[0276] Encodes the number of runs. Figure 14 In the example of , there are two runs, so 2 is encoded.

[0277] ④ Encode the occupation of the first run. Figure 14 In the example of , 0 is encoded because the first run corresponds to an unoccupied pixel.

[0278] ⑤ Encode the length of each run (as many as the number of runs). Figure 14 In the example, the lengths 6 and 10 of the first run and the second run are encoded respectively.

[0279] Video compression (40009, 40010, 40011)

[0280] The video compressor according to the embodiment encodes the sequence of geometry images, texture images, occupancy map images, etc. generated in the above-mentioned operations using a 2D video codec such as HEVC or VVC.

[0281] Figure 15 An exemplary 2D video / image encoder according to an embodiment is shown.

[0282] This figure illustrates an embodiment in which the aforementioned video compression or video compressors 40009, 40010, and 40011 are applied, and is a schematic block diagram of a 2D video / image encoder 15000 configured to encode a video / image signal. The 2D video / image encoder 15000 may be included in the aforementioned point cloud video encoder, or may be configured as an internal / external component. Figure 15 The various components may correspond to software, hardware, a processor and / or a combination thereof.

[0283] Here, the input image may include the aforementioned geometric image, texture image (attribute image), and occupancy map image. The output bitstream of the point cloud video encoder (i.e., point cloud video / image bitstream) may include the output bitstream of each input image (i.e., geometric image, texture image (attribute image), occupancy map image, etc.).

[0284] The inter-frame predictor 15090 and the intra-frame predictor 15100 may be collectively referred to as a predictor. That is, the predictor may include the inter-frame predictor 15090 and the intra-frame predictor 15100. The transformer 15030, the quantizer 15040, the inverse quantizer 15050, and the inverse transformer 15060 may be included in the residual processor. The residual processor may also include a subtractor 15020. According to an embodiment, the above-mentioned image segmenter 15010, the subtractor 15020, the transformer 15030, the quantizer 15040, the inverse quantizer 15050, the inverse transformer 15060, the adder 155, the filter 15070, the inter-frame predictor 15090, the intra-frame predictor 15100, and the entropy encoder 15110 may be configured by a hardware component (e.g., an encoder or a processor). In addition, the memory 15080 may include a decoded picture buffer (DPB) and may be configured by a digital storage medium.

[0285] The image splitter 15010 may split the image (or picture or frame) input to the encoder 15000 into one or more processing units. For example, a processing unit may be referred to as a coding unit (CU). In this case, the CU may be recursively split from a coding tree unit (CTU) or a largest coding unit (LCU) based on a quadtree binary tree (QTBT) structure. For example, a CU may be split into multiple CUs of lower depth based on a quadtree structure and / or a binary tree structure. In this case, for example, the quadtree structure may be applied first, and the binary tree structure may be applied later. Alternatively, the binary tree structure may be applied first. The encoding process according to the present disclosure may be performed based on a final CU that is no longer split. In this case, the LCU may be used as the final CU based on encoding efficiency according to the characteristics of the image. If necessary, the CU may be recursively split into CUs of lower depth, and the optimally sized CU may be used as the final CU. Here, the encoding process may include prediction, transform, and reconstruction (to be described later). As another example, the processing unit may also include a prediction unit (PU) or a transform unit (TU). In this case, the PU and TU may be partitioned or divided from the final CU. A PU may be a unit for sample prediction, and a TU may be a unit for deriving a transform coefficient and / or a unit for deriving a residual signal from the transform coefficient.

[0286] The term "unit" can be used interchangeably with terms such as block or region. In general, an M×N block can represent a set of samples or transform coefficients arranged in M ​​columns and N rows. A sample can generally represent a pixel or pixel value and can indicate only the pixel / pixel value of the luma component or only the pixel / pixel value of the chroma component. "Sample" can be used as a term corresponding to a pixel or picture element in a picture (or image).

[0287] The encoder 15000 can generate a residual signal (residual block or residual sample array) by subtracting the prediction signal (prediction block or prediction sample array) output from the inter predictor 15090 or the intra predictor 15100 from the input image signal (original block or original sample array), and the generated residual signal is sent to the transformer 15030. In this case, as shown in the figure, the unit in the encoder 15000 that subtracts the prediction signal (prediction block or prediction sample array) from the input image signal (original block or original sample array) can be referred to as a subtractor 15020. The predictor can perform prediction on a processing target block (hereinafter referred to as the current block) and generate a prediction block including prediction samples of the current block. The predictor can determine whether to apply intra prediction or inter prediction based on the current block or CU. As will be described later in the description of each prediction mode, the predictor can generate various types of information related to the prediction (e.g., prediction mode information) and transmit the generated information to the entropy encoder 15110. The information related to the prediction can be encoded by the entropy encoder 15110 and output in the form of a bitstream.

[0288] The intra-frame predictor 15100 can predict the current block with reference to samples in the current picture. Depending on the prediction mode, the samples may be adjacent to or far away from the current block. Under intra-frame prediction, the prediction mode may include multiple non-directional modes and multiple directional modes. The non-directional mode may include, for example, a DC mode and a planar mode. Depending on the fineness of the prediction direction, the directional mode may include, for example, 33 directional prediction modes or 65 directional prediction modes. However, this is only an example, and more or fewer directional prediction modes may be used depending on the settings. The intra-frame predictor 15100 may determine the prediction mode to be applied to the current block based on the prediction mode applied to the neighboring blocks.

[0289] The inter-frame predictor 15090 can derive a prediction block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. In this case, to reduce the amount of motion information transmitted in inter-frame prediction mode, motion information can be predicted on a per-block, per-subblock, or per-sample basis based on the correlation of motion information between neighboring blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can also include information regarding the inter-frame prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter-frame prediction, neighboring blocks can include spatially neighboring blocks in the current picture and temporally neighboring blocks in the reference picture. The reference picture containing the reference block can be the same as or different from the reference picture containing the temporally neighboring block. A temporally neighboring block can be referred to as a collocated reference block or collocated CU (colCU), and a reference picture containing a temporally neighboring block can be referred to as a collocated picture (colPic). For example, the inter-frame predictor 15090 can configure a motion information candidate list based on the neighboring blocks and generate information indicating candidates to be used for deriving the motion vector and / or reference picture index for the current block. Inter-frame prediction can be performed based on various prediction modes. For example, in skip mode and merge mode, the inter-frame predictor 15090 may use motion information about a neighboring block as motion information about the current block. In skip mode, unlike merge mode, a residual signal may not be sent. In motion vector prediction (MVP) mode, the motion vector of the neighboring block may be used as a motion vector predictor, and a motion vector difference may be signaled to indicate the motion vector of the current block.

[0290] The prediction signal generated by the inter-frame predictor 15090 or the intra-frame predictor 15100 may be used to generate a reconstructed signal or to generate a residual signal.

[0291] The transformer 15030 may generate transform coefficients by applying a transform technique to the residual signal. For example, the transform technique may include at least one of a discrete cosine transform (DCT), a discrete sine transform (DST), a Karhunen–Loève transform (KLT), a graph-based transform (GBT), or a conditional nonlinear transform (CNT). Here, a GBT refers to a transform obtained from a graph depicting the relationship between pixels. A CNT refers to a transform obtained based on a prediction signal generated from all previously reconstructed pixels. In addition, the transform operation may be applied to square pixel blocks of the same size, or to blocks of variable sizes other than square.

[0292] The quantizer 15040 may quantize the transform coefficients and send them to the entropy encoder 15110. The entropy encoder 15110 may encode the quantized signal (information about the quantized transform coefficients) and output a bitstream of the encoded signal. The information about the quantized transform coefficients may be referred to as residual information. The quantizer 15040 may rearrange the quantized transform coefficients in block form into a one-dimensional vector based on the coefficient scanning order and generate information about the quantized transform coefficients based on the quantized transform coefficients in the one-dimensional vector form. The entropy encoder 15110 may employ various coding techniques such as exponential Golomb, context-adaptive variable length coding (CAVLC), and context-adaptive binary arithmetic coding (CABAC). The entropy encoder 15110 may encode information required for video / image reconstruction (e.g., syntax element values) together with or separately from the quantized transform coefficients. The encoded information (e.g., encoded video / image information) may be transmitted or stored in the form of a bitstream based on a network abstraction layer (NAL) unit. The bitstream may be transmitted over a network or stored in a digital storage medium. Here, the network may include a broadcast network and / or a communication network, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD. A transmitter (not shown) that transmits a signal output from the entropy encoder 15110 and / or a storage unit (not shown) that stores the signal may be configured as an internal / external element of the encoder 15000. Alternatively, the transmitter may be included in the entropy encoder 15110.

[0293] The quantized transform coefficients output from the quantizer 15040 can be used to generate a prediction signal. For example, inverse quantization and inverse transformation can be applied to the quantized transform coefficients by the inverse quantizer 15050 and the inverse transformer 15060 to reconstruct a residual signal (residual block or residual sample). The adder 155 adds the reconstructed residual signal to the prediction signal output from the inter-frame predictor 15090 or the intra-frame predictor 15100. Thus, a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) can be generated. When there is no residual signal for the processing target block, as in the case of applying skip mode, the prediction block can be used as a reconstructed block. The adder 155 can be referred to as a reconstructor or a reconstructed block generator. As described below, the generated reconstructed signal can be used for intra-frame prediction of the next processing target block in the current picture, or can be used for inter-frame prediction of the next picture through filtering.

[0294] The filter 15070 can improve the subjective / objective image quality by applying filtering to the reconstructed signal. For example, the filter 15070 can generate a modified reconstructed picture by applying various filtering techniques to the reconstructed picture, and the modified reconstructed picture can be stored in the memory 15080 (specifically, the DPB of the memory 15080). For example, various filtering techniques may include deblocking filtering, sample adaptive offset, adaptive loop filtering, and bilateral filtering. As described below in the description of the filtering technique, the filter 15070 can generate various types of information about filtering and transmit the generated information to the entropy encoder 15110. The information about filtering can be encoded by the entropy encoder 15110 and output in the form of a bitstream.

[0295] The modified reconstructed picture transmitted to the memory 15080 may be used as a reference picture by the inter-frame predictor 15090. Therefore, when inter-frame prediction is applied, the encoder may avoid prediction mismatch between the encoder 15000 and the decoder and improve encoding efficiency.

[0296] The DPB of the memory 15080 may store the modified reconstructed picture for use as a reference picture by the inter-frame predictor 15090. The memory 15080 may store motion information about a block from which motion information in the current picture is derived (or encoded) and / or motion information about a reconstructed block in the picture. The stored motion information may be transmitted to the inter-frame predictor 15090 for use as motion information about spatially neighboring blocks or motion information about temporally neighboring blocks. The memory 15080 may store reconstructed samples of the reconstructed blocks in the current picture and transmit the reconstructed samples to the intra-frame predictor 15100.

[0297] At least one of the above-mentioned prediction, transformation and quantization processes may be skipped. For example, for a block to which pulse coding mode (PCM) is applied, the prediction, transformation and quantization processes may be skipped, and the value of the original sample may be encoded and output in the form of a bitstream.

[0298] Figure 16 An exemplary V-PCC decoding process according to an embodiment is shown.

[0299] The V-PCC decoding process or the V-PCC decoder may follow Figure 4 The inverse process of the V-PCC encoding process (or encoder). Figure 16 The various components in may correspond to software, hardware, processors and / or a combination thereof.

[0300] The demultiplexer 16000 demultiplexes the compressed bit stream to output a compressed texture image, a compressed geometry image, a compressed occupancy map, and compressed auxiliary patch information.

[0301] The video decompression or video decompressor 16001, 16002 decompresses (or decodes) each of the compressed texture image and the compressed geometry image.

[0302] Occupancy map decompression or occupancy map decompressor 16003 decompresses the compressed occupancy map.

[0303] The auxiliary patch information decompression or the auxiliary patch information decompressor 16004 decompresses the auxiliary patch information.

[0304] Geometry reconstruction or geometry reconstructor 16005 restores (reconstructs) geometry information based on the decompressed geometry image, decompressed occupancy map and / or decompressed auxiliary patch information. For example, geometry that changed during the encoding process can be reconstructed.

[0305] Smoothing or smoother 16006 can apply smoothing to the reconstructed geometry. For example, a smoothing filter can be applied.

[0306] Texture reconstruction or texture reconstructor 16007 reconstructs textures from decompressed texture images and / or smoothed geometry.

[0307] Color smoothing or color smoother 16008 smoothes color values ​​from the reconstructed texture. For example, a smoothing filter may be applied.

[0308] As a result, reconstructed point cloud data can be generated.

[0309] The figure shows a decoding process of V-PCC for reconstructing a point cloud by decoding a compressed occupancy map, a geometry image, a texture image, and auxiliary patch information. Each process according to an embodiment operates as follows.

[0310] Video decompression (1600, 16002)

[0311] Video decompression is the inverse process of the above-mentioned video compression. In video decompression, a 2D video codec such as HEVC or VVC is used to decode the compressed bitstream containing the geometry image, texture image, and occupancy map image generated in the above-mentioned process.

[0312] Figure 17 An exemplary 2D video / image decoder according to an embodiment is shown.

[0313] 2D video / image decoder can follow Figure 15 The inverse process of the 2D video / image encoder.

[0314] Figure 17 The 2D video / image decoder is Figure 16 Video decompression or video decompressor. Figure 17is a schematic block diagram of a 2D video / image decoder 17000 that performs decoding of a video / image signal. The 2D video / image decoder 17000 may be included in Figure 1 The point cloud video decoder can be configured as an internal / external component. Figure 17 The various components in may correspond to software, hardware, processors and / or a combination thereof.

[0315] Here, the input bitstream may include the bitstreams of the geometric image, texture image (attribute image), and occupancy map image. The reconstructed image (or output image or decoded image) may represent the reconstructed image of the geometric image, texture image (attribute image), and occupancy map image.

[0316] Referring to this figure, the inter-frame predictor 17070 and the intra-frame predictor 17080 may be collectively referred to as a predictor. That is, the predictor may include the inter-frame predictor 17070 and the intra-frame predictor 17080. The inverse quantizer 17020 and the inverse transformer 17030 may be collectively referred to as a residual processor. That is, depending on the embodiment, the residual processor may include the inverse quantizer 17020 and the inverse transformer 17030. The entropy decoder 17010, inverse quantizer 17020, inverse transformer 17030, adder 17040, filter 17050, inter-frame predictor 17070, and intra-frame predictor 17080 described above may be configured by a single hardware component (e.g., a decoder or processor). In addition, the memory 170 may include a decoded picture buffer (DPB) or may be configured by a digital storage medium.

[0317] When a bitstream containing video / image information is input, decoder 17000 can reconstruct the image using a process corresponding to the process of processing the video / image information by the encoder in Figure 0.2-1. For example, decoder 17000 can use the processing unit applied in the encoder to perform decoding. Therefore, the processing unit for decoding can be, for example, a CU. The CU can be divided from the CTU or LCU along a quadtree structure and / or a binary tree structure. The reconstructed video signal decoded and output by decoder 17000 can then be played by a player.

[0318] The decoder 17000 can receive a signal output from the encoder in the form of a bitstream, and the received signal can be decoded by the entropy decoder 17010. For example, the entropy decoder 17010 can parse the bitstream to derive information required for image reconstruction (or picture reconstruction) (e.g., video / image information). For example, the entropy decoder 17010 can decode the information in the bitstream based on a coding technique such as exponential Golomb coding, CAVLC, or CABAC, and output the syntax elements required for image reconstruction and the quantized values ​​of the residual transform coefficients. More specifically, in CABAC entropy decoding, bins corresponding to each syntax element in the bitstream can be received, and a context model can be determined based on the decoding target syntax element information and decoding information about adjacent decoding target blocks or information about symbols / bins decoded in previous steps. The probability of the occurrence of the bin can then be predicted based on the determined context model, and arithmetic decoding of the bins can be performed to generate symbols corresponding to the values ​​of each syntax element. According to CABAC entropy decoding, after determining the context model, the context model can be updated based on information about the symbol / cell decoded for the context model of the next symbol / cell. Information about prediction in the information decoded by the entropy decoder 17010 can be provided to the predictor (inter-frame predictor 17070 and intra-frame predictor 17080), and the residual value (i.e., quantized transform coefficients and related parameter information) that has been entropy decoded by the entropy decoder 17010 can be input to the inverse quantizer 17020. In addition, information about filtering in the information decoded by the entropy decoder 17010 can be provided to the filter 17050. A receiver (not shown) configured to receive a signal output from the encoder can also be configured as an internal / external element of the decoder 17000. Alternatively, the receiver can be a component of the entropy decoder 17010.

[0319] The inverse quantizer 17020 may output a transform coefficient by inversely quantizing the quantized transform coefficient. The inverse quantizer 17020 may rearrange the quantized transform coefficients in the form of two-dimensional blocks. In this case, the rearrangement may be performed based on the coefficient scanning order implemented by the encoder. The inverse quantizer 17020 may perform inverse quantization on the quantized transform coefficient using a quantization parameter (e.g., quantization step size information) and obtain the transform coefficient.

[0320] The inverse transformer 17030 obtains a residual signal (a residual block and a residual sample array) by performing inverse transformation on the transformation coefficients.

[0321] The predictor may perform prediction on the current block and generate a prediction block including prediction samples of the current block. The predictor may determine whether to apply intra prediction or inter prediction to the current block based on the information about prediction output from the entropy decoder 17010, and may determine a specific intra / inter prediction mode.

[0322] The intra-frame predictor 265 can predict the current block with reference to samples in the current picture. Depending on the prediction mode, the samples may be adjacent to or distant from the current block. In intra-frame prediction, the prediction modes may include multiple non-directional modes and multiple directional modes. The intra-frame predictor 17080 can use the prediction modes applied to neighboring blocks to determine the prediction mode applied to the current block.

[0323] The inter-frame predictor 17070 can derive a prediction block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. In this case, to reduce the amount of motion information transmitted in inter-frame prediction mode, motion information can be predicted on a per-block, sub-block, or sample basis based on the correlation of motion information between neighboring blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may also include information regarding the inter-frame prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter-frame prediction, neighboring blocks may include spatially neighboring blocks in the current picture and temporally neighboring blocks in the reference picture. For example, the inter-frame predictor 17070 may configure a motion information candidate list based on the neighboring blocks and derive the motion vector and / or reference picture index of the current block based on received candidate selection information. Inter-frame prediction can be performed based on various prediction modes. Information regarding the prediction may include information indicating the inter-frame prediction mode for the current block.

[0324] The adder 17040 may add the obtained residual signal to the prediction signal (prediction block or prediction sample array) output from the inter-frame predictor 17070 or the intra-frame predictor 17080 to generate a reconstructed signal (reconstructed picture, reconstructed block or reconstructed sample array). When there is no residual signal for processing the target block as in the case of applying the skip mode, the prediction block may be used as the reconstructed block.

[0325] The adder 17040 may be referred to as a reconstructor or a reconstructed block generator. The generated reconstructed signal may be used for intra-frame prediction of the next processing target block in the current picture, or may be used for inter-frame prediction of the next picture through filtering as described below.

[0326] The filter 17050 can improve the subjective / objective image quality by applying filtering to the reconstructed signal. For example, the filter 17050 can generate a modified reconstructed picture by applying various filtering techniques to the reconstructed picture, and can send the modified reconstructed picture to the memory 250 (specifically, the DPB of the memory 17060). For example, various filtering methods may include deblocking filtering, sample adaptive offset, adaptive loop filtering, and bilateral filtering.

[0327] The reconstructed picture stored in the DPB of the memory 17060 can be used as a reference picture in the inter-frame predictor 17070. The memory 17060 can store motion information about a block in the current picture from which motion information is derived (or decoded) and / or motion information about a block in an already reconstructed picture. The stored motion information can be transmitted to the inter-frame predictor 17070 to be used as motion information about spatially neighboring blocks or motion information about temporally neighboring blocks. The memory 17060 can store reconstructed samples of the reconstructed block in the current picture and transmit the reconstructed samples to the intra-frame predictor 17080.

[0328] In the present disclosure, the embodiments described with respect to the filter 160, the inter-frame predictor 180, and the intra-frame predictor 185 of the encoder 100 may be applied to the filter 17050, the inter-frame predictor 17070, and the intra-frame predictor 17080 of the decoder 17000, respectively, in the same or corresponding manner.

[0329] At least one of the above-mentioned prediction, transformation and quantization processes may be skipped. For example, for a block to which pulse coding mode (PCM) is applied, the prediction, transformation and quantization processes may be skipped, and the values ​​of the decoded samples may be used as samples of the reconstructed image.

[0330] Occupancy map decompression (16003)

[0331] This is the inverse process of the occupancy map compression described above. Occupancy map decompression is the process of reconstructing the occupancy map by decompressing the occupancy map bitstream.

[0332] Auxiliary patch information decompression (16004)

[0333] The auxiliary patch information may be reconstructed by performing the inverse process of the auxiliary patch information compression described above and decoding the compressed auxiliary patch information bitstream.

[0334] Geometry Reconstruction (16005)

[0335] This is the inverse process of the geometric image generation described above. Initially, patches are extracted from the geometric image using the reconstructed occupancy map, the 2D position / size information about the patches included in the auxiliary patch information, and the information about the mapping between blocks and patches. Then, a point cloud is reconstructed in 3D space based on the geometric image of the extracted patches and the 3D position information about the patches included in the auxiliary patch information. When the geometric value corresponding to the point (u, v) within the patch is g(u, v), and the position coordinates of the patch on the normal axis, tangential axis, and bitangential axis of the 3D space are (d0, s0, r0), the normal, tangential, and bitangential coordinates d(u, v), s(u, v), and r(u, v) in the 3D space mapped to the position of the point (u, v) can be expressed as follows:

[0336] d(u,v)=d0+g(u,v);

[0337] s(u,v)=s0+u;

[0338] r(u,v)=r0+v.

[0339] Smooth (16006)

[0340] Like smoothing in the above-described encoding process, smoothing is a process for eliminating discontinuities that may appear at patch boundaries due to image quality degradation occurring during the compression process.

[0341] Texture reconstruction (16007)

[0342] Texture reconstruction is a process of reconstructing a color point cloud by assigning color values ​​to each point that makes up the smoothed point cloud. This is performed by assigning color values ​​corresponding to the texture image pixels at the same positions in the geometric image in 2D space to points in the point cloud corresponding to the same positions in 3D space, based on the mapping information between the geometric image and the point cloud in the geometric reconstruction process described above.

[0343] Color Smoothing (16008)

[0344] Color smoothing is similar to the process of geometric smoothing described above. Color smoothing is a process used to eliminate discontinuities that may appear at patch boundaries due to image quality degradation that occurs during the compression process. Color smoothing can be performed by the following operations:

[0345] ① Use a KD tree or other method to calculate the neighboring points of each point that constitutes the reconstructed point cloud. The neighboring point information calculated in the geometric smoothing process described in Section 2.5 can be used.

[0346] ② Determine whether each point is on the patch boundary. These operations can be performed based on the boundary information calculated in the above-mentioned geometric smoothing process.

[0347] ③ Check the distribution of color values ​​of neighboring points on the edge and determine whether smoothing should be performed. For example, when the entropy of the luminance value is less than or equal to a threshold local entry (there are many similar luminance values), it can be determined that the corresponding portion is not an edge portion, and smoothing can be performed. As a smoothing method, the color value of the point can be replaced by the average color value of the neighboring points.

[0348] Figure 18 is a flowchart illustrating the operation of the transmitting device according to an embodiment of the present disclosure.

[0349] The transmitting device according to the embodiment may correspond to Figure 1 The sending device, Figure 4 The encoding process and Figure 152D video / image encoder, or perform some / all operations thereof. Each component of the transmitting device may correspond to software, hardware, a processor and / or a combination thereof.

[0350] The operation process of the sending terminal using V-PCC to compress and transmit point cloud data can be performed as shown in the figure.

[0351] The point cloud data transmitting device according to the embodiment may be referred to as a transmitting device.

[0352] The patch generator 18000 generates patches for 2D image mapping of point clouds. As a result of patch generation, auxiliary patch information is generated. This information can be used in processes such as geometry image generation, texture image generation, and smooth geometry reconstruction.

[0353] The patch packer 18001 performs a patch packing process to map the generated patches into a 2D image. As a result of the patch packing, an occupancy map can be generated. The occupancy map can be used in the processes of generating geometry images, generating texture images, and for smooth geometry reconstruction.

[0354] The geometric image generator 18002 generates a geometric image based on the auxiliary patch information and the occupancy map. The generated geometric image is encoded into a bit stream through video coding.

[0355] The encoding preprocessor 18003 may include an image filling process. The geometry image regenerated by decoding the generated geometry image or the coded geometry bitstream can be used for 3D geometry reconstruction and then subjected to a smoothing process.

[0356] The texture image generator 18004 can generate a texture image based on (smoothed) 3D geometry, point cloud, auxiliary patch information and occupancy map. The generated texture image can be encoded into a video bitstream.

[0357] The metadata encoder 18005 can encode the auxiliary patch information into a metadata bitstream.

[0358] The video encoder 18006 can encode the occupancy map into a video bitstream.

[0359] The multiplexer 18007 may multiplex the generated video bitstreams of the geometry image, texture image, and occupancy map and the metadata bitstream of the auxiliary patch information into one bitstream.

[0360] The transmitter 18008 may transmit the bitstream to the receiving terminal. Alternatively, the generated video bitstream of the geometry image, texture image, and occupancy map and the metadata bitstream of the auxiliary patch information may be processed into one or more track data files or encapsulated into fragments and transmitted to the receiving terminal through the transmitter.

[0361] Figure 19 is a flowchart illustrating the operation of the receiving device according to the embodiment.

[0362] The receiving device according to the embodiment may correspond to Figure 1 receiving device, Figure 16 The decoding process and Figure 17 2D video / image encoder, or perform some / all operations thereof. Each component of the receiving device may correspond to software, hardware, a processor and / or a combination thereof.

[0363] The operation of the receiving terminal using V-PCC to receive and reconstruct point cloud data can be performed as shown in the figure. The operation of the V-PCC receiving terminal can be performed as follows Figure 18 The reverse process of the operation of the V-PCC sending terminal.

[0364] The point cloud data receiving device according to the embodiment may be referred to as a receiving device.

[0365] After file / fragment decapsulation, the received point cloud bitstream is demultiplexed by demultiplexer 19000 into a compressed video bitstream of geometry images, texture images, and occupancy maps, and a metadata bitstream of auxiliary patch information. Video decoder 19001 and metadata decoder 19002 decode the demultiplexed video bitstream and metadata bitstream. 3D geometry is reconstructed by geometry reconstructor 19003 based on the decoded geometry images, occupancy maps, and auxiliary patch information, and then subjected to smoothing by smoother 19004. Texture reconstructor 19005 can reconstruct a color point cloud image / image by assigning color values ​​to the smoothed 3D geometry based on the texture images. Color smoothing can then be performed to improve subjective / objective visual quality, and the modified point cloud image / image derived from the color smoothing process is displayed to the user through a rendering process (e.g., by a point cloud renderer). In some cases, the color smoothing process can be skipped.

[0366] Figure 20 An exemplary architecture for V-PCC based storage and streaming of point cloud data is shown, according to an embodiment.

[0367] Figure 20 Part / all of the system may include Figure 1 The sending device and the receiving device, Figure 4 The encoding process, Figure 15 2D video / image encoder, Figure 16 Decoding processing, Figure 18 The sending device and / or Figure 19 Each component in the figure may correspond to software, hardware, processor and / or a combination thereof.

[0368] Figures 20 to 22 1 is a diagram showing a structure in which a system according to an embodiment is additionally connected to a transmitting device and a receiving device. The transmitting device and the receiving device according to the embodiment may be referred to as a transmitting / receiving apparatus according to the embodiment.

[0369] In accordance with Figures 20 to 22 In the device of the embodiment shown, Figure 18 The corresponding sending device may generate a container in a data format suitable for transmitting a bit stream containing the encoded point cloud data.

[0370] The V-PCC system according to the embodiment may create a container including point cloud data, and may further add additional data required for efficient transmission / reception to the container.

[0371] The receiving device according to the embodiment may be based on Figures 20 to 22 The system shown receives and parses the container. Figure 19 A corresponding receiving device can decode and recover the point cloud data from the parsed bit stream.

[0372] The figure shows an overall architecture for storing or streaming point cloud data compressed according to video-based point cloud compression (V-PCC). The processing of storing and streaming point cloud data may include acquisition processing, encoding processing, transmission processing, decoding processing, rendering processing and / or feedback processing.

[0373] Embodiments propose a method for efficiently providing point cloud media / content / data.

[0374] In order to efficiently provide point cloud media / content / data, the point cloud acquirer 20000 can acquire point cloud video. For example, one or more cameras can acquire point cloud data by capturing, arranging or generating a point cloud. Through this acquisition process, a point cloud video including the 3D position of each point (which can be represented by x, y and z position values, etc.) (hereinafter referred to as geometry) and the attributes of each point (color, reflectivity, transparency, etc.) can be acquired. For example, a Polygon file format (PLY) (or Stanford triangle format) file containing a point cloud video can be generated. For point cloud data with multiple frames, one or more files can be acquired. In this process, point cloud-related metadata (e.g., metadata related to capture, etc.) can be generated.

[0375] Post-processing for improving the quality of the content may be required for the captured point cloud video. In the video capture process, the maximum / minimum depth can be adjusted within the range provided by the camera device. Even after the adjustment, point data of unwanted areas may still exist. Therefore, post-processing can be performed to remove unwanted areas (for example, background) or identify connected spaces and fill spatial holes. In addition, the point cloud extracted from cameras sharing a spatial coordinate system can be integrated into a piece of content by transforming each point into a global coordinate system based on the position coordinates of each camera obtained through a calibration process. In this way, a point cloud video with a high density of points can be obtained.

[0376] The point cloud preprocessor 20001 can generate one or more frames of a point cloud video. A frame can generally represent a unit representing an image at a specific time interval. When the points comprising the point cloud video are divided into one or more patches (a collection of points comprising the point cloud video, where points belonging to the same patch are adjacent to each other in 3D space and, when mapped to a 2D image, are mapped in the same direction between the flat surfaces of a 6-sided bounding box) and mapped to a 2D plane, an occupancy map frame can be generated, which indicates the presence of data at corresponding locations in the 2D plane using a value of 0 or 1. Furthermore, a geometry frame can be generated in the form of a depth map, which represents information about the position (geometry) of each point comprising the point cloud video on a patch-by-patch basis. A texture frame can also be generated, which represents color information about each point comprising the point cloud video on a patch-by-patch basis. During this process, metadata required to reconstruct a point cloud from each patch can be generated. This metadata can include information about the patches, such as their position and size in 2D / 3D space. These pictures / frames may be generated continuously in time sequence to construct a video stream or metadata stream.

[0377] The point cloud video encoder 20002 can encode one or more video streams related to a point cloud video. A video may include multiple frames, and a frame may correspond to a still image / picture. In this disclosure, a point cloud video may include point cloud images / frames / pictures, and the term "point cloud video" and "point cloud video / frame / picture" may be used interchangeably. The point cloud video encoder may perform video-based point cloud compression (V-PCC) processing. For compression and coding efficiency, the point cloud video encoder may perform a series of processes such as prediction, transform, quantization, and entropy coding. The encoded data (encoded video / image information) may be output as a bitstream. Based on the V-PCC processing, as described below, the point cloud video encoder can encode the point cloud video by separating it into a geometry video, an attribute video, an occupancy map video, and metadata (e.g., information about patches). The geometry video may include a geometry image, the attribute video may include an attribute image, and the occupancy map video may include an occupancy map image. Patch data, serving as auxiliary information, may include patch-related information. The attribute video / image may include a texture video / image.

[0378] The point cloud image encoder 20003 can encode one or more images related to the point cloud video. The point cloud image encoder can perform video-based point cloud compression (V-PCC) processing. For compression and coding efficiency, the point cloud image encoder can perform a series of processes such as prediction, transformation, quantization, and entropy coding. The encoded image can be output in the form of a bitstream. Based on the V-PCC process, as described below, the point cloud image encoder can encode the point cloud image by dividing the point cloud image into a geometry image, an attribute image, an occupancy map image, and metadata (e.g., information about patches).

[0379] The point cloud video encoder and / or the point cloud image encoder according to an embodiment may generate a PCC bitstream (G-PCC and / or V-PCC bitstream) according to an embodiment.

[0380] According to an embodiment, the video encoder 2002, the image encoder 20003, the video decoding 20006, and the image decoding may be performed by one encoder / decoder as described above, and may be performed along separate paths as shown in the figure.

[0381] In file / segment encapsulation 20004, the encoded point cloud data and / or point cloud-related metadata may be encapsulated into files or segments for streaming. Here, the point cloud-related metadata may be received from a metadata processor, etc. The metadata processor may be included in the point cloud video / image encoder, or may be configured as a separate component / module. The encapsulation processor may encapsulate the corresponding video / image / metadata in a file format such as ISOBMFF or in the form of DASH segments, etc. Depending on the embodiment, the encapsulation processor may include the point cloud metadata in a file format. The point cloud-related metadata may be included in boxes at various levels of the ISOBMFF file format, for example, or as data in a separate track within the file. Depending on the embodiment, the encapsulation processor may encapsulate the point cloud-related metadata into a file.

[0382] The encapsulator or packager according to an embodiment may divide a G-PCC / V-PCC bitstream into one or more tracks and store them in a file, and may also encapsulate signaling information used for this operation. In addition, the atlas stream included in the G-PCC / V-PCC bitstream may be stored as a track in the file, and related signaling information may be stored. In addition, SEI messages present in the G-PCC / V-PCC bitstream may be stored in a track in the file, and related signaling information may be stored.

[0383] The transmission processor can perform transmission processing of the encapsulated point cloud data according to the file format. The transmission processor can be included in the transmitter or configured as a separate component / module. The transmission processor can process the point cloud data according to the transmission protocol. The transmission processing can include processing for transmission via a broadcast network and processing for transmission via broadband. Depending on the embodiment, the transmission processor can receive point cloud-related metadata along with the point cloud data from the metadata processor and perform transmission processing of the point cloud video data.

[0384] The transmitter can transmit the point cloud bitstream or the file / segment including the bitstream to the receiver of the receiving device via a digital storage medium or a network. For transmission, processing according to any transmission protocol can be performed. The data processed for transmission can be transmitted via a broadcast network and / or by broadband. The data can be transmitted to the receiving side in an on-demand manner. The digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD and SSD. The transmitter may include an element for generating a media file in a predetermined file format, and may include an element for transmission via a broadcast / communication network. The receiver may extract the bitstream and transmit the extracted bitstream to a decoder.

[0385] A receiver can receive point cloud data transmitted by a point cloud data transmitting device according to the present disclosure. Depending on the transmission channel, the receiver can receive the point cloud data via a broadcast network or broadband. Alternatively, the point cloud data can be received via a digital storage medium. The receiver can include processing to decode the received data and render the data according to the user's viewport.

[0386] The receiving processor processes the received point cloud video data according to the transmission protocol. The receiving processor may be included in the receiver or configured as a separate component / module. The receiving processor performs the reverse processing of the transmitting processor described above, corresponding to the transmission processing performed on the transmitting side. The receiving processor transmits the acquired point cloud video to the decapsulation processor and transmits the acquired point cloud-related metadata to the metadata parser.

[0387] The decapsulation processor (file / segment decapsulation) 20005 can decapsulate the point cloud data received from the receiving processor in the form of a file. The decapsulation processor can decapsulate the file according to ISOBMFF, etc., and can obtain a point cloud bitstream or point cloud-related metadata (or a separate metadata bitstream). The obtained point cloud bitstream can be transmitted to the point cloud decoder, and the obtained point cloud video-related metadata (metadata bitstream) can be transmitted to the metadata processor. The point cloud bitstream may include metadata (metadata bitstream). The metadata processor may be included in the point cloud decoder, or may be configured as a separate component / module. The point cloud video-related metadata obtained by the decapsulation processor may take the form of a box or track in the file format. If necessary, the decapsulation processor may receive metadata required for decapsulation from the metadata processor. The point cloud-related metadata may be transmitted to the point cloud decoder and used in the point cloud decoding process, or may be transmitted to the renderer and used in the point cloud rendering process.

[0388] The point cloud video decoder 20006 can receive a bitstream and decode the video / image by performing operations corresponding to those of the point cloud video encoder. In this case, as described below, the point cloud video decoder can decode the point cloud video by separating the point cloud video into a geometry video, an attribute video, an occupancy map video, and auxiliary patch information. The geometry video may include a geometry image, the attribute video may include an attribute image, and the occupancy map video may include an occupancy map image. The auxiliary information may include auxiliary patch information. The attribute video / image may include a texture video / image.

[0389] The 3D geometry can be reconstructed based on the decoded geometry image, occupancy map, and auxiliary patch information, and then smoothed. A color point cloud image / screen can be reconstructed by assigning color values ​​to the smoothed 3D geometry based on the texture image. A renderer can render the reconstructed geometry and color point cloud image / screen. The rendered video / image can be displayed on a display. All or part of the rendered result can be displayed to the user via a VR / AR display or a typical display.

[0390] The sensor / tracker (sensing / tracking) 20007 acquires orientation information and / or user viewport information from the user or receiving side and transmits the orientation information and / or user viewport information to the receiver and / or transmitter. The orientation information may represent information about the position, angle, movement, etc. of the user's head, or information about the position, angle, movement, etc. of the device through which the user is viewing the video / image. Based on this information, information about the area in 3D space that the user is currently viewing (i.e., viewport information) can be calculated.

[0391] Viewport information may be information about the area in 3D space that a user is currently viewing through a device or HMD. A device, such as a display, may extract the viewport area based on orientation information, the vertical or horizontal FOV supported by the device, and so on. Orientation or viewport information may be extracted or calculated on the receiving side. The orientation or viewport information analyzed on the receiving side may be sent to the sending side via a feedback channel.

[0392] Based on the orientation information acquired by the sensor / tracker and / or the viewport information indicating the area currently viewed by the user, the receiver can effectively extract or decode only the media data for the specific area (i.e., the area indicated by the orientation information and / or the viewport information) from the file. Furthermore, based on the orientation information and / or the viewport information acquired by the sensor / tracker, the transmitter can effectively encode only the media data for the specific area (i.e., the area indicated by the orientation information and / or the viewport information), or generate and transmit the file thereof.

[0393] The renderer can render the decoded point cloud data in 3D space. The rendered video / image can be displayed on a monitor. Users can view all or part of the rendered result on a VR / AR display or a typical monitor.

[0394] Feedback processing may include transmitting various types of feedback information that can be obtained during the rendering / display process to a decoder on the sending or receiving side. Feedback processing can provide interactivity when consuming point cloud data. Depending on the embodiment, head orientation information, viewport information indicating the area currently viewed by the user, etc. may be transmitted to the sending side during feedback processing. Depending on the embodiment, the user can interact with content implemented in a VR / AR / MR / autonomous driving environment. In this case, information related to the interaction may be transmitted to the sending side or service provider during feedback processing. Depending on the embodiment, feedback processing may be skipped.

[0395] Depending on the implementation, the feedback information can be not only sent to the sending side but also consumed by the receiving side. That is, the receiving side can perform decapsulation, decoding, and rendering based on the feedback information. For example, point cloud data related to the area currently being viewed by the user can be prioritized for decapsulation, decoding, and rendering based on orientation information and / or viewport information.

[0396] A method for transmitting point cloud data according to an embodiment may include the following steps: encoding the point cloud data; packaging the point cloud data; and transmitting the point cloud data.

[0397] A method for receiving point cloud data according to an embodiment may include the following steps: receiving point cloud data; decapsulating the point cloud data; and decoding the point cloud data.

[0398] Figure 21 is an exemplary block diagram of an apparatus for storing and transmitting point cloud data according to an embodiment.

[0399] Figure 21 A point cloud system according to an embodiment is shown. Part / all of the system may include Figure 1 The sending device and the receiving device, Figure 4 The encoding process, Figure 15 2D video / image encoder, Figure 16 Decoding processing, Figure 18 The sending device and / or Figure 19 In addition, it may include or correspond to Figure 20 part / all of the system.

[0400] The point cloud data transmitting device according to the embodiment may be configured as shown in the figure. Each element of the transmitting device may be a module / unit / component / hardware / software / processor.

[0401] The point cloud's geometry, attributes, auxiliary data, and mesh data can each be configured as a separate stream or stored in different tracks within the file. Additionally, they can be included in separate fragments.

[0402] The point cloud acquirer (Point Cloud Acquisition) 21000 acquires a point cloud. For example, one or more cameras may acquire point cloud data by capturing, arranging, or generating a point cloud. This acquisition process acquires point cloud data including the 3D position of each point (which may be represented by x, y, and z position values, etc.) (hereinafter referred to as geometry) and the attributes of each point (color, reflectivity, transparency, etc.). For example, a Polygon File Format (PLY) (or Stanford Triangle Format) file containing point cloud data may be generated. For point cloud data having multiple frames, one or more files may be acquired. During this process, metadata related to the point cloud (e.g., metadata related to the capture, etc.) may be generated.

[0403] The patch generator (or patch generation) 21002 generates patches from point cloud data. The patch generator generates point cloud data or point cloud video into one or more frames. A frame generally represents a unit representing an image at a specific time interval. When the points constituting the point cloud video are divided into one or more patches (a collection of points constituting the point cloud video, where points belonging to the same patch are adjacent to each other in 3D space and are mapped in the same direction between the flat surfaces of a 6-sided bounding box when mapped to a 2D image) and mapped to a 2D plane, an occupancy map frame / frame can be generated as a binary image, which indicates whether data exists at corresponding locations in the 2D plane as 0 or 1. In addition, a geometric frame / frame in the form of a depth map can be generated, which represents information about the position (geometry) of each point constituting the point cloud video on a patch-by-patch basis. A texture frame / frame can also be generated, which represents color information about each point constituting the point cloud video on a patch-by-patch basis. During this process, metadata required to reconstruct the point cloud from each patch can be generated. The metadata may include information about the patches, such as the position and size of each patch in 2D / 3D space.These pictures / frames may be generated continuously in a time sequence to construct a video stream or a metadata stream.

[0404] Additionally, patches can be used for 2D image mapping. For example, point cloud data can be projected onto the faces of a cube. After the patch is generated, a geometric image, one or more attribute images, an occupancy map, auxiliary data, and / or mesh data can be generated based on the generated patch.

[0405] The geometry image generation, the attribute image generation, the occupancy map generation, the auxiliary data generation and / or the mesh data generation are performed by a pre-processor or a controller.

[0406] In geometry image generation 21002, a geometry image is generated based on the results of patch generation. Geometry represents points in 3D space. The geometry image is generated using an occupancy map, which includes information related to 2D image packing of patches, auxiliary data (patch data), and / or patch-based mesh data. The geometry image is associated with information such as the depth (e.g., near, far) of the patches generated after patch generation.

[0407] In attribute image generation 21003, an attribute image is generated. For example, an attribute may represent texture. The texture may be a color value matched to each point. Depending on the embodiment, an image including multiple attributes of the texture (e.g., color and reflectivity) (N attributes) may be generated. The multiple attributes may include material information and reflectivity. Depending on the embodiment, the attributes may also include information indicating color, which may vary depending on the viewing angle and light even for the same texture.

[0408] An occupancy map is generated from the patch in occupancy map generation 21004. The occupancy map includes information indicating whether data is present in a pixel (eg, corresponding to a geometry or attribute image).

[0409] In Auxiliary Data Generation 21005 , auxiliary data including information about the patch is generated. Specifically, the auxiliary data represents metadata about the patch of the point cloud object. For example, it may represent information such as the normal vector of the patch. Specifically, the auxiliary data may include information required to reconstruct a point cloud from the patch (e.g., information about the patch's position and size in 2D / 3D space, as well as projection (normal) plane identification information, patch mapping information, etc.).

[0410] In mesh data generation 21006, mesh data is generated from the patch. A mesh represents the connection between adjacent points. For example, it can represent data in the shape of a triangle. For example, mesh data refers to the connectivity between points.

[0411] The point cloud pre-processor or controller generates metadata related to patch generation, geometry image generation, attribute image generation, occupancy map generation, auxiliary data generation, and mesh data generation.

[0412] The point cloud transmitting device performs video encoding and / or image encoding in response to the results generated by the preprocessor. The point cloud transmitting device can generate point cloud image data and point cloud video data. Depending on the embodiment, the point cloud data can include only video data, only image data, and / or both video data and image data.

[0413] The video encoder 21007 performs geometric video compression, attribute video compression, occupancy map compression, auxiliary data compression, and / or mesh data compression. The video encoder generates a video stream containing encoded video data.

[0414] Specifically, in geometry video compression, point cloud geometry video data is encoded. In attribute video compression, attribute video data of the point cloud is encoded. In auxiliary data compression, auxiliary data associated with the point cloud video data is encoded. In mesh data compression, mesh data of the point cloud video data is encoded. Each operation of the point cloud video encoder can be performed in parallel.

[0415] The image encoder 21008 performs geometric image compression, attribute image compression, occupancy map compression, auxiliary data compression, and / or mesh data compression. The image encoder generates an image containing encoded image data.

[0416] Specifically, in geometric image compression, point cloud geometric image data is encoded. In attribute image compression, attribute image data of the point cloud is encoded. In auxiliary data compression, auxiliary data associated with the point cloud image data is encoded. In mesh data compression, mesh data associated with the point cloud image data is encoded. Each operation of the point cloud image encoder can be performed in parallel.

[0417] The video encoder and / or the image encoder may receive the metadata from the pre-processor and may perform various encoding processes based on the metadata.

[0418] The file / segment encapsulator (File / Segment Encapsulation) 21009 encapsulates video streams and / or images in the form of files and / or segments. The file / segment encapsulator performs video track encapsulation, metadata track encapsulation, and / or image encapsulation.

[0419] In video track encapsulation, one or more video streams can be encapsulated into one or more tracks.

[0420] In metadata track encapsulation, metadata related to video streams and / or images can be encapsulated in one or more tracks. Metadata includes data related to the content of the point cloud data. For example, it can include initial viewing orientation metadata. Depending on the implementation, metadata can be encapsulated in the metadata track or can be encapsulated together in the video track or image track.

[0421] In image packaging, one or more images can be packaged into one or more tracks or projects.

[0422] For example, according to an embodiment, when four video streams and two images are input to the encapsulator, the four video streams and two images may be encapsulated in one file.

[0423] The point cloud video encoder and / or the point cloud image encoder according to an embodiment may generate a G-PCC / V-PCC bitstream according to an embodiment.

[0424] The file / segment encapsulator may receive metadata from the preprocessor and may perform encapsulation based on the metadata.

[0425] The files and / or segments generated by the file / segment encapsulation are transmitted by a point cloud transmitting device or transmitter. For example, the segments may be transmitted according to a DASH-based protocol.

[0426] The encapsulator or packager according to an embodiment may divide the V-PCC bitstream into one or more tracks and store them in a file, and may also encapsulate signaling information used for this operation. In addition, the atlas stream included in the V-PCC bitstream may be stored as a track in the file, and related signaling information may be stored. In addition, the SEI message present in the V-PCC bitstream may be stored in a track in the file, and related signaling information may be stored.

[0427] The transmitter can transmit a point cloud bitstream or a file / segment including the bitstream to a receiver of a receiving device via a digital storage medium or a network. For transmission, processing according to any transmission protocol can be performed. Processed data can be transmitted via a broadcast network and / or broadband for transmission. Data can be transmitted to the receiving side on demand. Digital storage media can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD. The transmitter can include elements for generating a media file in a predetermined file format and can include elements for transmission via a broadcast / communication network. The transmitter receives orientation information and / or viewport information from the receiver. The transmitter can transmit the acquired orientation information and / or viewport information (or information selected by the user) to a preprocessor, a video encoder, an image encoder, a file / segment encapsulator, and / or a point cloud encoder. Based on the orientation information and / or viewport information, the point cloud encoder can encode all point cloud data or the point cloud data indicated by the orientation information and / or viewport information. Based on the orientation information and / or the viewport information, the file / fragment encapsulator may encapsulate all point cloud data or the point cloud data indicated by the orientation information and / or the viewport information. Based on the orientation information and / or the viewport information, the transmitter may transmit all point cloud data or the point cloud data indicated by the orientation information and / or the viewport information.

[0428] For example, the preprocessor may perform the above operations on all point cloud data or on point cloud data indicated by orientation information and / or viewport information. The video encoder and / or image encoder may perform the above operations on all point cloud data or on point cloud data indicated by orientation information and / or viewport information. The file / fragment encapsulator may perform the above operations on all point cloud data or on point cloud data indicated by orientation information and / or viewport information. The transmitter may perform the above operations on all point cloud data or on point cloud data indicated by orientation information and / or viewport information.

[0429] Figure 22 is an exemplary block diagram of a point cloud data receiving apparatus according to an embodiment.

[0430] Figure 22 A point cloud system according to an embodiment is shown. Part / all of the system may include Figure 1 The sending device and the receiving device, Figure 4 The encoding process, Figure 15 2D video / image encoder, Figure 16 Decoding processing, Figure 18 The sending device and / or Figure 19 In addition, it may include or correspond to Figure 20 and Figure 21 part / all of the system.

[0431] The various components of the receiving device may be modules / units / components / hardware / software / processors. The transmitting client may receive point cloud data, point cloud bitstreams or files / fragments including a bitstream transmitted by a point cloud data transmitting device according to an embodiment. Depending on the channel used for transmission, the receiver may receive point cloud data via a broadcast network or via broadband. Alternatively, point cloud video data may be received via a digital storage medium. The receiver may include processing for decoding the received data and rendering the received data according to a user viewport. The receiving processor may perform processing on the received point cloud data according to a transmission protocol. The receiving processor may be included in the receiver or configured as a separate component / module. The receiving processor may perform the processing of the above-mentioned transmitting processor in reverse to correspond to the transmission processing performed by the transmitting side. The receiving processor may transmit the acquired point cloud data to the decapsulation processor and transmit the acquired point cloud-related metadata to the metadata parser.

[0432] The sensor / tracker (sensing / tracking) acquires orientation information and / or viewport information and may transmit the acquired orientation information and / or viewport information to the delivery client, the file / fragment decapsulator, and the point cloud decoder.

[0433] The transmitting client may receive all point cloud data or the point cloud data indicated by the orientation information and / or viewport information based on the orientation information and / or viewport information. The file / fragment decapsulator may decapsulate all point cloud data or the point cloud data indicated by the orientation information and / or viewport information based on the orientation information and / or viewport information. The point cloud decoder (video decoder and / or image decoder) may decode all point cloud data or the point cloud data indicated by the orientation information and / or viewport information based on the orientation information and / or viewport information. The point cloud processor may process all point cloud data or the point cloud data indicated by the orientation information and / or viewport information based on the orientation information and / or viewport information.

[0434] The file / segment decapsulator (file / segment decapsulation) 22000 performs video track decapsulation, metadata track decapsulation, and / or image decapsulation. The decapsulation processor (file / segment decapsulation) can decapsulate point cloud data received from the receiving processor in the form of a file. The decapsulation processor (file / segment decapsulation) can decapsulate the file or segment according to ISOBMFF, etc. to obtain a point cloud bitstream or point cloud-related metadata (or a separate metadata bitstream). The obtained point cloud bitstream can be transmitted to the point cloud decoder, and the obtained point cloud-related metadata (or metadata bitstream) can be transmitted to the metadata processor. The point cloud bitstream may include metadata (metadata bitstream). The metadata processor may be included in the point cloud video decoder or may be configured as a separate component / module. The point cloud-related metadata obtained by the decapsulation processor may take the form of a box or track in the file format. If necessary, the decapsulation processor may receive metadata required for decapsulation from the metadata processor. The point cloud-related metadata may be transmitted to the point cloud decoder and used in the point cloud decoding process, or may be transmitted to the renderer and used in the point cloud rendering process. The file / fragment decapsulator may generate metadata associated with the point cloud data.

[0435] In video track decapsulation, video tracks contained in files and / or fragments are decapsulated. Video streams including geometry video, attribute video, occupancy maps, auxiliary data, and / or mesh data are decapsulated.

[0436] In metadata track decapsulation, the bitstream containing metadata related to the point cloud data and / or auxiliary data is decapsulated.

[0437] In image depacketization, an image including a geometry image, an attribute image, an occupancy map, auxiliary data and / or mesh data is depacketized.

[0438] The decapsulator or decapsulator according to an embodiment can divide and parse (decapsulate) the G-PCC / V-PCC bitstream based on one or more tracks in the file, and can also decapsulate its signaling information. In addition, the atlas stream included in the G-PCC / V-PCC bitstream can be decapsulated based on the tracks in the file, and the relevant signaling information can be parsed. In addition, the SEI message present in the G-PCC / V-PCC bitstream can be decapsulated based on the tracks in the file, and the relevant signaling information can also be obtained.

[0439] Video decoding or video decoder 22001 performs geometry video decompression, attribute video decompression, occupancy map decompression, auxiliary data decompression, and / or mesh data decompression. The video decoder decodes the geometry video, attribute video, auxiliary data, and / or mesh data in a process corresponding to the process performed by the video encoder of the point cloud transmitting device according to the embodiment.

[0440] Image decoding or image decoder 22002 performs geometric image decompression, attribute image decompression, occupancy map decompression, auxiliary data decompression, and / or mesh data decompression. The image decoder decodes the geometric image, attribute image, auxiliary data, and / or mesh data in a process corresponding to the process performed by the image encoder of the point cloud transmitting device according to the embodiment.

[0441] Video decoding and image decoding according to an embodiment may be processed by one video / image decoder as described above, and may be performed along separate paths as shown in the figure.

[0442] Video decoding and / or image decoding may generate metadata related to the video data and / or image data.

[0443] The point cloud video encoder and / or the point cloud image encoder according to an embodiment may decode a G-PCC / V-PCC bitstream according to an embodiment.

[0444] In point cloud processing 22003 , geometric reconstruction and / or attribute reconstruction are performed.

[0445] In geometric reconstruction, a geometric video and / or a geometric image is reconstructed from the decoded video data and / or the decoded image data based on the occupancy map, the auxiliary data and / or the mesh data.

[0446] In attribute reconstruction, an attribute video and / or attribute image are reconstructed from a decoded attribute video and / or decoded attribute image based on an occupancy map, auxiliary data, and / or mesh data. Depending on the embodiment, an attribute may be, for example, a texture. Depending on the embodiment, an attribute may represent multiple pieces of attribute information. When multiple attributes are present, the point cloud processor according to the embodiment performs multiple attribute reconstructions.

[0447] The point cloud processor may receive metadata from the video decoder, the image decoder, and / or the file / segment decapsulator and process the point cloud based on the metadata.

[0448] Point cloud rendering or a point cloud renderer renders the reconstructed point cloud. The point cloud renderer may receive metadata from a video decoder, an image decoder, and / or a file / segment decapsulator and render the point cloud based on the metadata.

[0449] The display actually displays the rendered result on the monitor.

[0450] like Figures 15 to 19 As shown, in the method / device according to the embodiment Figures 15 to 19 After encoding / decoding the point cloud data as shown, the bitstream containing the point cloud data can be packaged and / or depackaged in the form of files and / or fragments.

[0451] For example, the point cloud data transmitting apparatus according to the embodiment may package the point cloud data based on a file, which may include a V-PCC track containing point cloud parameters, a geometry track containing geometry, an attribute track containing attributes, and an occupancy track containing an occupancy map.

[0452] In addition, the point cloud data receiving apparatus according to the embodiment decapsulates the point cloud data based on the file. The file may include a V-PCC track containing point cloud parameters, a geometry track containing geometry, an attribute track containing attributes, and an occupancy track containing an occupancy map.

[0453] The above operations can be performed by Figure 20 File / fragment encapsulator 20004, file / fragment decapsulator 20005, Figure 21 File / fragment encapsulator 21009 and Figure 22 The file / fragment decapsulator 22000 is executed.

[0454] Figure 23 An exemplary structure that can be operated in conjunction with the point cloud data transmitting / receiving method / apparatus according to the embodiment is shown.

[0455] In the structure according to the embodiment, at least one of the server 2360, the robot 2310, the self-driving vehicle 2320, the XR device 2330, the smartphone 2340, the home appliance 2350, and / or the head-mounted display (HMD) 2370 is connected to the cloud network 2300. Here, the robot 2310, the self-driving vehicle 2320, the XR device 2330, the smartphone 2340, or the home appliance 2350 may be referred to as a device. In addition, the XR device 1730 may correspond to a point cloud data (PCC) device according to the embodiment, or may be operatively connected to the PCC device.

[0456] The cloud network 2300 may represent a network that constitutes a part of a cloud computing infrastructure or exists in a cloud computing infrastructure. Here, the cloud network 2300 may be configured using a 3G network, a 4G or Long Term Evolution (LTE) network, or a 5G network.

[0457] The server 2360 may be connected to at least one of the robot 2310 , the self-driving vehicle 2320 , the XR device 2330 , the smart phone 2340 , the home appliance 2350 , and / or the HMD 2370 via the cloud network 2300 , and may assist at least a portion of the processing of the connected devices 2310 to 2370 .

[0458] The HMD 2370 represents one of implementation types of an XR device and / or a PCC device according to an embodiment. The HMD type device according to an embodiment includes a communication unit, a control unit, a memory, an I / O unit, a sensor unit, and a power supply unit.

[0459] Hereinafter, various embodiments of apparatuses 2310 to 2350 to which the above technologies are applied will be described. Figure 23 The apparatuses 2310 to 2350 shown can be operably connected / linked to the point cloud data sending and receiving apparatus according to the above embodiments.

[0460] <PCC+XR>

[0461] The XR / PCC apparatus 2330 can adopt PCC technology and / or XR (AR + VR) technology, and can be implemented as an HMD, a head-up display (HUD) installed in a vehicle, a television, a mobile phone, a smartphone, a computer, a wearable device, a household appliance, a digital signage, a vehicle, a stationary robot, or a mobile robot.

[0462] The XR / PCC apparatus 2330 can analyze 3D point cloud data or image data obtained through various sensors or from external apparatuses and generate position data and attribute data regarding 3D points. Thus, the XR / PCC apparatus 2330 can obtain information about the surrounding space or real objects, and render and output XR objects. For example, the XR / PCC apparatus 2330 can cause an XR object including auxiliary information about the recognized object to match the recognized object and output the matched XR object.

[0463] <PCC+XR+Mobile Phone>

[0464] The XR / PCC apparatus 2330 can be implemented as a mobile phone 2340 by applying PCC technology.

[0465] The mobile phone 2340 can decode and display point cloud content based on PCC technology.

[0466] <PCC+Autopilot+XR>

[0467] The self-driving vehicle 2320 can be implemented as a mobile robot, a vehicle, an unmanned aerial vehicle, etc. by applying PCC technology and XR technology.

[0468] The self-driving vehicle 2320 to which XR / PCC technology is applied can represent an autonomous vehicle provided with means for providing an XR image, or an autonomous vehicle that is a control / interaction target in an XR image. Specifically, the self-driving vehicle 2320 that is a control / interaction target in an XR image can be distinguished from the XR apparatus 2330 and can be operably connected thereto.

[0469] A self-driving vehicle 2320 having a means for providing XR / PCC images may acquire sensor information from sensors including cameras and output generated XR / PCC images based on the acquired sensor information. For example, a self-driving vehicle may have a HUD and output XR / PCC images thereto to provide passengers with XR / PCC objects corresponding to real objects or objects present on a screen.

[0470] In this case, when the XR / PCC object is output to the HUD, at least a portion of the XR / PCC object may be output to overlap with the real object that the passenger's eyes are pointing at. On the other hand, when the XR / PCC object is output on a display provided inside the self-driving vehicle, at least a portion of the XR / PCC object may be output to overlap with the object on the screen. For example, the self-driving vehicle may output XR / PCC objects corresponding to objects such as a road, another vehicle, a traffic light, a traffic sign, a two-wheeled vehicle, a pedestrian, and a building.

[0471] Virtual reality (VR) technology, augmented reality (AR) technology, mixed reality (MR) technology, and / or point cloud compression (PCC) technology according to an embodiment are applicable to various devices.

[0472] In other words, VR technology is a display technology that only provides real-world objects, backgrounds, etc. as CG images. On the other hand, AR technology refers to a technology that displays a CG image created virtually on an image of a real object. MR technology is similar to the above-mentioned AR technology in that the virtual objects to be displayed are mixed and combined with the real world. However, MR technology is different from AR technology. AR technology makes a clear distinction between real objects and virtual objects created as CG images and uses virtual objects as supplementary objects to real objects, while MR technology treats virtual objects as objects with the same characteristics as real objects. More specifically, an example of the application of MR technology is holographic services.

[0473] Recently, VR, AR, and MR technologies are often referred to as extended reality (XR) technologies, rather than being clearly distinguished from each other. Therefore, embodiments of the present disclosure are applicable to all VR, AR, MR, and XR technologies. For these technologies, encoding / decoding based on PCC, V-PCC, and G-PCC technologies can be applied.

[0474] The PCC method / apparatus according to the embodiment may be applied to a vehicle providing a self-driving service.

[0475] Vehicles providing self-driving services are connected to the PCC device for wired / wireless communication.

[0476] When the point cloud data sending and receiving device (PCC device) according to the embodiment is connected to the vehicle for wired / wireless communication, the device can receive and process content data related to AR / VR / PCC services that can be provided together with the self-driving service and send the processed content data to the vehicle. In the case where the point cloud data sending and receiving device is installed on the vehicle, the point cloud sending and receiving device can receive and process content data related to the AR / VR / PCC service according to the user input signal input through the user interface device and provide the processed content data to the user. The vehicle or the user interface device according to the embodiment can receive the user input signal. The user input signal according to the embodiment may include a signal indicating the self-driving service.

[0477] The method / apparatus according to the embodiment refers to a point cloud data transmitting apparatus (e.g., Figure 1 The sending device 10000, Figure 18 The sending device), the encoder of the sending device (for example, Figure 1 Point cloud video encoder 10002, Figure 4 The encoder, Figure 15 ), a file encapsulator of the sending device (e.g., Figure 20 The file / segment encapsulator 20004 of the system to which the sending device is connected), Figure 21 The preprocessor of the encoder, the file / segment encapsulator 21009 of the system to which the encoder is connected, the point cloud data receiving device according to the embodiment (for example, Figure 1 The receiving device 10005, Figure 19 receiving device), a decoder of the receiving device ( Figure 1 Point cloud video decoder 10008, Figure 16 decoder, Figure 17 decoder), a file decapsulator of the receiving device (e.g., Figure 20 The receiving device is connected to the system's file / fragment decapsulator 20005, Figure 22 The processor of the system, the file / segment decapsulator 22000 of the decoder connected to the decoder, etc. Each element may be implemented by hardware, software, firmware or a combination thereof including one or more processors or integrated circuits configured to communicate with one or more memories.

[0478] The video-based point cloud compression (V-PCC) described in this specification is the same as the visual volumetric video-based coding (V3C). The terms V-PCC and V3C according to the embodiments may be used interchangeably and may have the same meaning.

[0479] The method / apparatus according to the embodiment may provide a file format signaling method for non-timed V3C data including a plurality of atlases.

[0480] The method / apparatus according to the embodiment may provide a file format signaling method for non-timed V3C data when the non-timed V3C data includes a plurality of atlases.

[0481] The method / apparatus according to the embodiment may provide a file format signaling method in the case where non-timed V3C data includes multiple video data units.

[0482] The method / apparatus according to the embodiment may provide a transmitter or receiver for providing a point cloud content service, which efficiently stores a V-PCC bitstream in a track of a file and provides signaling thereof.

[0483] The method / apparatus according to an embodiment may support efficient access to point cloud data by storing and transmitting a V-PCC bitstream in a format according to an embodiment.

[0484] The method / apparatus according to an embodiment may include a file storage scheme for efficiently storing and signaling a V-PCC bitstream in a file track and supporting efficient access to the stored V-PCC bitstream, and provides a scheme for dividing and storing a V-PCC bitstream into one or more tracks in a file.

[0485] Definitions of terms according to the embodiments are as follows. VPS: V-PCC parameter set; AD: atlas data; OVD: occupancy video data; GVD: geometry video data; AVD: attribute video data; ACL: atlas coding layer; AAPS: atlas adaptation parameter set. AAPS may include camera parameters associated with part of the atlas sub-bitstream, such as camera position, rotation, scale, and camera model. These parameters may be associated with part of the atlas sub-bitstream ( Figure 30ASPS: Atlas Sequence Parameter Set; contains the syntax structure of syntax elements that apply to zero or more complete coded atlas sequences (CAS) as determined by the contents of syntax elements found in the ASPS referenced by syntax elements found in each tile group header; AFPS: Atlas Frame Parameter Set; contains the syntax structure of syntax elements that apply to zero or more complete coded atlas frames as determined by the contents of syntax elements found in tile group header 0; SEI: Supplementary Enhancement Information; Atlas: This can be a 2D bounding box (i.e., a patch) projected onto a rectangular frame corresponding to a 3D bounding box of a subset of a representable point cloud in 3D space. A collection of 2D bounding boxes, or a collection of 2D bounding boxes and their associated information placed on a rectangular frame and corresponding to the volume of rendered volume data in 3D space; atlas subbitstream: a subbitstream extracted from a V-PCC bitstream that contains a portion of the atlas NAL bitstream; V-PCC content: a point cloud encoded using Video Coding Point Cloud Compression (V-PCC); V-PCC track: a volumetric vision track that carries the atlas bitstream of a V-PCC bitstream; V-PCC component track: a video track that carries 2D video coded data for any of the occupancy map, geometry, or attribute component video bitstreams of a V-PCC bitstream.

[0486] The V3C image data may refer to non-timed V3C data. According to an embodiment, the non-timed V3C data may be composed of multiple atlas data and / or multiple atlas tiles to support partial access, etc.

[0487] In order to separate the non-timed V3C data into atlases and / or atlas tiles ( Figures 20 to 22 ) to decode and render point cloud data, which can be achieved by submitting to the V3C project ( Figures 47 to 53 )Add new item types to configure the data structure and define the corresponding new V3C item properties.

[0488] Methods / apparatus according to embodiments may provide file format signaling related to the carriage of encoded data based on non-timed visual volumetric video.

[0489] Figure 24 An exemplary structure of a V-PCC bitstream according to an embodiment is shown.

[0490] According to Figure 24 The bitstream of the illustrated embodiment can be obtained by Figure 1 Point cloud video encoder 10002, Figure 4 The encoding process, Figure 15 The encoder, Figure 18 The sending device, Figure 20 Video / image encoders 20002 and 20003, Figure 21 Video / image encoders 21007 and 21008,Figure 23 XR devices etc. can encode point cloud data to generate. Figure 24 The bit stream shown can be used on the receiving side by Figure 1 Point cloud video decoder 10008, Figure 16 Decoding processing, Figure 17 decoder, Figure 19 The receiving device and Figure 20 Video / image decoder 20006, Figure 22 Video / image decoders 22001 and 2202, Figure 23 The XR device 1230 decodes the point cloud data to reconstruct the point cloud data.

[0491] The method / apparatus according to an embodiment is a transmitter, a receiver, and / or a processor for providing a point cloud content service for efficiently storing a V-PCC (=V3C) bitstream in a track of a file and providing signaling thereof.

[0492] The method / apparatus according to the embodiment provides a data format for storing a V-PCC bitstream containing point cloud data. Thus, a data storage and signaling scheme is provided that allows the receiving method / apparatus according to the embodiment to receive point cloud data and efficiently access the point cloud data. Thus, based on storage technology for efficiently accessing files containing point cloud data, a transmitter and / or receiver can provide point cloud content services.

[0493] Figure 24 The structure of a point cloud bitstream included in data transmitted and received by a method / apparatus according to an embodiment is shown.

[0494] The compression and decompression techniques of point cloud data according to the embodiments represent volumetric encoding and decoding of point cloud visual information.

[0495] The point cloud bitstream containing the coded point cloud sequence (CPCS) (which may be referred to as a V-PCC bitstream or V3C bitstream) 25000 may include a sample stream V-PCC unit. The sample stream V-PCC unit may carry V-PCC parameter set (VPS) data, an atlas bitstream, a 2D video coding occupancy map bitstream, a 2D video coding geometry bitstream, and zero or one or more 2D video coding attribute bitstreams.

[0496] The point cloud bitstream may include a sample stream VPCC header.

[0497] ssvh_unit_size_precision_bytes_minus1: The value obtained by adding 1 to this value specifies the precision (in bytes) of the ssvu_vpcc_unit_size element in all sample stream V-PCC units. ssvh_unit_size_precision_bytes_minus1 can be in the range of 0 to 7.

[0498] The syntax of the sample stream V-PCC unit is configured as follows. Each sample stream V-PCC unit may include one of the VPS, AD, OVD, GVD, and AVD V-PCC unit types. The content of each sample stream V-PCC unit may be associated with the same access unit as the V-PCC unit included in the sample stream V-PCC unit.

[0499] ssvu_vpcc_unit_size: specifies the size (in bytes) of the subsequent vpcc_unit. The number of bits used to represent ssvu_vpcc_unit_size is equal to (ssvh_unit_size_precision_bytes_minus1+1)*8.

[0500] According to the method / apparatus of the embodiment, receiving Figure 24 The bit stream including the encoded point cloud data is generated by the encapsulators 20004 and 21009. Figures 45 to 53 The file shown.

[0501] According to the embodiment, the method / apparatus receives Figures 45 to 53 The file shown is read and the point cloud data is decoded by a file / fragment decapsulator 22000 or the like.

[0502] VPS and / or AD can be encapsulated in track 4 (V3C track). OVD can be encapsulated in track 2 (occupied track). GVD can be encapsulated in track 3 (geometry track). AVD can be encapsulated in track 1 (attribute track).

[0503] Figure 25 The structure of a bitstream including point cloud data according to an embodiment is shown.

[0504] Figure 25 The bit stream may correspond to Figure 24 bitstream. Figure 24 and Figure 25 The bitstream is composed of Figure 1 The sending device 10000, Figure 1 Point cloud video encoder 10002, Figure 4 The encoder, Figure 15 The encoder, Figure 18 The sending device,Figure 20 Processor 20001 and Figure 20 Video / image encoder 20002, Figure 21 Processors 21001 to 21006, Figure 21 The video / image encoders 21007 and 21008 etc. generate.

[0505] Figure 24 and Figure 25 The bitstream is Figure 1 File / fragment wrapper, Figure 20 File / fragment encapsulator 20004, Figure 21 The file / fragment encapsulator 21009 is stored in the container ( Figures 45 to 53 , etc.).

[0506] Figure 24 and Figure 25 The bitstream in Figure 1 Transmitter 10004 and others send.

[0507] Figure 1 The receiving device 10005, the receiver 10006, etc. receive the data including Figure 24 and Figure 25 The bitstream container ( Figures 45 to 53 documents, etc.).

[0508] Depend on Figure 1 File / fragment decapsulator 10007, Figure 20 File / fragment decapsulator 20005, Figure 22 The file / fragment decapsulator 22000 etc. is parsed from the container Figure 24 and Figure 25 bitstream.

[0509] Figure 24 and Figure 25 The bitstream is composed of Figure 1 Point cloud video decoder 10008, Figure 16 decoder, Figure 17 decoder, Figure 19 receiving device, Figure 20 Video / image decoder 20006, Figure 22 Video / image decoders 22001 and 22002, Figure 22 The processor 22003 decodes and reconstructs it to provide it to the user.

[0510] A sample stream V-PCC unit included in a bitstream related to point cloud data according to an embodiment may include a V-PCC unit size and a V-PCC unit.

[0511] Each V-PCC unit may include a V-PCC unit header and a V-PCC unit payload. The V-PCC unit header may describe the V-PCC unit type. The V-PCC unit header of attribute video data may describe the attribute type, its index, supported multiple instances of the same attribute type, etc.

[0512] The unit payload of occupancy, geometry, and attribute video data may correspond to a video data unit. For example, the occupancy video data, geometry video data, and attribute video data may be HEVC NAL units. Such video data may be decoded by a video decoder according to an embodiment.

[0513] Figure 26 A V-PCC cell and a V-PCC cell head according to an embodiment are shown.

[0514] Figure 26 Show Figure 25 The syntax of the V-PCC unit and the V-PCC unit header are shown.

[0515] A V-PCC bitstream according to an embodiment may contain a series of V-PCC sequences.

[0516] A vpcc unit type with a value of vuh_unit_type equal to VPCC_VPS is expected to be the first V-PCC unit type in a V-PCC sequence. All other V-PCC unit types follow this unit type without any additional restrictions on their coding order. The V-PCC unit payload of a V-PCC unit carrying occupancy video, attribute video, or geometry video consists of one or more NAL units.

[0517] A VPCC unit may include a header and a payload.

[0518] The VPCC unit header may include the following information based on the VUH unit type.

[0519] vuh_unit_type indicates the type of the V-PCC unit as follows.

[0520] [Table 1]

[0521] vuh_unit_type Identifier V-PCC Unit Type Description 0 VPCC_VPS V-PCC Parameter Set V-PCC Level Parameter 1 VPCC_AD Atlas Data Atlas Information 2 VPCC_OVD Occupied Video Data Occupancy Information 3 VPCC_GVD Geometric Video Data Geometric Information 4 VPCC_AVD Attribute Video Data Attribute Information 5…31 VPCC_RSVD Reservation -

[0522] When vuh_unit_type indicates attribute video data (VPCC_AVD), geometry video data (VPCC_GVD), occupancy video data (VPCC_OVD), or atlas data (VPCC_AD), vuh_vpcc_parameter_setID and vuh_atlas_id are carried in the unit header. Parameter set ID and atlas ID associated with the V-PCC unit may be transmitted.

[0523] When the unit type is atlas video data, the unit header may carry an attribute index (vuh_attribute_index), an attribute partition index (vuh_attribute_partition_index), a map index (vuh_map_index) and an auxiliary video flag (vuh_auxiliary_video_flag).

[0524] When the unit type is geometry video data, it can carry vuh_map_index and vuh_auxiliary_video_flag.

[0525] When the unit type is occupied video data or atlas data, the unit header may contain additional reserved bits.

[0526] vuh_vpcc_parameter_set_id specifies the value of vps_vpcc_parameter_set_id of the active V-PCC VPS. The vpcc_parameter_set_id in the header of the current V-PCC unit can be used to know the ID of the VPS parameter set and declare the relationship between the V-PCC unit and the V-PCC parameter set.

[0527] vuh_atlas_id specifies the index of the atlas corresponding to the current V-PCC unit. The atlas index can be known through vuh_atlas_id in the header of the current V-PCC unit, and the atlas corresponding to the V-PCC unit can be announced.

[0528] vuh_attribute_index indicates the index of attribute data carried in the attribute video data unit.

[0529] vuh_attribute_partition_index indicates the index of the attribute dimension group carried in the attribute video data unit.

[0530] When present, vuh_map_index indicates the map index of the current geometry or attribute stream.

[0531] vuh_auxiliary_video_flag equal to 1 indicates that the associated geometry or attribute video data unit is only RAW and / or EOM code point video. vuh_auxiliary_video_flag equal to 0 indicates that the associated geometry or attribute video data unit may contain RAW and / or EOM code points.

[0532] Figure 27 The payload of a V-PCC unit according to an embodiment is shown.

[0533] Figure 27 The syntax of the payload of the V-PCC unit is shown.

[0534] When vuh_unit_type is V-PCC parameter set (VPCC_VPS), the V-PCC unit payload contains vpcc_parameter_set().

[0535] When vuh_unit_type is V-PCC atlas data (VPCC_AD), the V-PCC unit payload contains atlas_sub_bitstream().

[0536] When vuh_unit_type is V-PCC Accumulated Video Data (VPCC_OVD), Geometric Video Data (VPCC_GVD), or Attribute Video Data (VPCC_AVD), the V-PCC unit payload contains video_sub_bitstream().

[0537] Hereinafter, information contained in the payload of the V-PCC unit will be described.

[0538] Figure 28 A V-PCC parameter set according to an embodiment is shown.

[0539] Figure 28 It is shown that when the payload of a unit of a bitstream according to an embodiment contains Figure 27 The parameter set shown is the syntax of the parameter set.

[0540] Figure 28 A VPS may include the following elements.

[0541] profile_tier_level() specifies restrictions on the bitstream, and therefore restrictions on the capabilities required to decode the bitstream. Profiles, tiers, and levels can also be used to indicate interoperability points between various decoder implementations.

[0542] vps_vpcc_parameter_set_id may provide an identifier of the V-PCC VPS for reference by other syntax elements.

[0543] vps_atlas_count_minus1 plus 1 indicates the total number of atlases supported in the current bitstream.

[0544] Depending on the number of atlases, the parameter set may further include the following parameters.

[0545] vps_frame_width[j] indicates the V-PCC frame width in terms of integer luma samples for the atlas of index j. This frame width is the nominal width associated with all V-PCC components of the atlas of index j.

[0546] vps_frame_height[j] indicates the V-PCC frame height in terms of integer luma samples for the atlas of index j. This frame height is the nominal height associated with all V-PCC components of the atlas of index j.

[0547] vps_map_count_minus1[j] plus 1 indicates the number of maps used to encode the geometry and attribute data of the atlas of index j.

[0548] When vps_map_count_minus1[j] is greater than 0, the parameter set may further include the following parameters.

[0549] Depending on the value of vps_map_count_minus1[j], the parameter set may further include the following parameters.

[0550] vps_multiple_map_streams_present_flag[j] equal to 0 indicates that all geometry or attribute maps of the atlas of index j are placed in a single geometry or attribute video stream, respectively. vps_multiple_map_streams_present_flag[j] equal to 1 indicates that all geometry or attribute maps of the atlas of index j are placed in separate video streams.

[0551] If vps_multiple_map_streams_present_flag[j] indicates 1, vps_map_absolute_coding_enabled_flag[j][i] may be further included in the parameter set. Otherwise, vps_map_absolute_coding_enabled_flag[j][i] may have a value of 1.

[0552] vps_map_absolute_coding_enabled_flag[j][i] equal to 1 indicates that the geometry map of index i of the atlas of index j is coded without any form of map prediction. vps_map_absolute_coding_enabled_flag[j][i] equal to 0 indicates that the geometry map of index i of the atlas of index j is first predicted from another earlier coded map before coding.

[0553] vps_map_absolute_coding_enabled_flag[j][0] equal to 1 indicates that the geometry map of index 0 is coded without map prediction.

[0554] If vps_map_absolute_coding_enabled_flag[j][i] is 0 and i is greater than 0, the parameter set may further include vps_map_predictor_index_diff[j][i]. Otherwise, vps_map_predictor_index_diff[j][i] may be 0.

[0555] When vps_map_absolute_coding_enabled_flag[j][i] is equal to 0, vps_map_predictor_index_diff[j][i] is used to calculate the predictor of the geometry map with index i of the atlas with index j.

[0556] vps_auxiliary_video_present_flag[j] equal to 1 indicates that auxiliary information (ie, RAW or EOM patch data) of the atlas of index j can be stored in a separate video stream (referred to as the auxiliary video stream). vps_auxiliary_video_present_flag[j] equal to 0 indicates that auxiliary information of the atlas of index j is not stored in a separate video stream.

[0557] occupancy_information() includes a set of occupancy video related parameters.

[0558] geometry_information() includes a set of geometric video related parameters.

[0559] attribute_information() includes attribute video related parameter sets.

[0560] vps_extension_present_flag equal to 1 specifies that the syntax element vps_extension_length is present in the vpcc_parameter_set syntax structure. vps_extension_present_flag equal to 0 specifies that the syntax element vps_extension_length is not present.

[0561] vps_extension_length_minus1 plus 1 specifies the number of vps_extension_data_byte elements that follow this syntax element.

[0562] According to vps_extension_length_minus1, the parameter set may further include extension data.

[0563] vps_extension_data_byte may have any value that may be included by the extension.

[0564] Figure 29 An atlas frame according to an embodiment is shown.

[0565] Figure 29 Shown including Figure 1 Encoder 10002, Figure 4 The encoder, Figure 15 The encoder, Figure 18 The sending device, Figure 20 and Figure 21 The figure shows an atlas frame of a mosaic block encoded by a system such as Figure 1 Decoder 10008, Figure 16 and Figure 17 decoder, Figure 19 receiving device, Figure 23 The atlas frame of the mosaic is decoded by the system.

[0566] An atlas frame is divided into one or more tile rows and one or more tile columns. A tile is a rectangular area of ​​an atlas frame. A tile group contains multiple tiles of an atlas frame. Tiles and tile groups are indistinguishable from each other, and a tile group may correspond to a tile. Only rectangular tile groups are supported. In this mode, a tile group (or tile) may contain multiple tiles of an atlas frame, which together form a rectangular area of ​​the atlas frame. Figure 29 Shown is a patch or tile group partitioning of an atlas frame according to an embodiment. Figure 29 The atlas frame is shown to be divided into 24 tiles (6 tile columns and 4 tile rows) and 9 rectangular tile groups. According to an embodiment, tile group may be used as a term corresponding to a tile, without distinguishing between tile groups and tiles.

[0567] That is, according to the embodiment, a tile group may correspond to a tile and may be referred to as a tile. In addition, a tile may correspond to a tile partition and may be referred to as a tile partition. According to the complementary relationship as described above, the term signaling information may also be changed and referred to.

[0568] Figure 30 The structure of an atlas bitstream according to an embodiment is shown.

[0569] Figure 30 Show Figure 25 An example of a bitstream unit in which the payload carries an atlas bitstream.

[0570] According to an embodiment, the term "sub" may be interpreted as meaning a portion. According to an embodiment, a sub-bitstream may be interpreted as a bitstream.

[0571] According to an embodiment, an atlas bitstream may include a sample stream NAL header and a sample stream NAL unit.

[0572] Each sample stream NAL unit according to an embodiment may include an atlas sequence parameter set (ASPS) (see Figure 33 ), Atlas Adaptation Parameter Set (AAPS) (see Figure 36 ), Atlas Frame Parameter Set (AFPS) (see Figure 34 ), one or more atlas tile groups, one or more required SEIs and one or more non-essential SEIs.

[0573] Reference Figure 30 The V-PCC unit payload of a V-PCC unit carrying an atlas sub-bitstream may consist of one or more sample stream NAL units.

[0574] Hereinafter, the Figure 30 The syntax of the information contained in the atlas bitstream.

[0575] [[ID=281 Schematic diagram showing a sample stream NAL unit, a sample stream NAL unit header, a NAL unit, and a NAL unit header included in a bitstream containing point cloud data according to an embodiment.

[0576] ​ Show ​ The syntax of the data contained in the atlas bitstream.

[0577] ssnh_unit_size_precision_bytes_minus1 plus 1 specifies the precision (in bytes) of the ssnu_nal_unit_size element in all sample stream NAL units. ssnh_unit_size_precision_bytes_minus1 can be in the range of 0 to 7.

[0578] ssnu_nal_unit_size specifies the size (in bytes) of the subsequent NAL_unit. The number of bits used to represent ssnu_nal_unit_size may be equal to (ssnh_unit_size_precision_bytes_minus1+1)*8.

[0579] NumBytesInNalUnit indicates the size of the NAL unit in bytes.

[0580] NumBytesInRbsp indicates the number of bytes belonging to the payload of the NAL unit and may be initialized to 0.

[0581] rbsp_byte[i] is the i-th byte of RBSP. RBSP can be represented as an ordered sequence of bytes.

[0582] nal_forbidden_zero_bit may be equal to zero (0). This field is used for error detection in the NAL unit and must be zero.

[0583] nal_unit_type can have ​ The value shown. nal_unit_type specifies the type of RBSP data structure contained in the NAL unit.

[0584] nal_layer_id specifies the identifier of the layer to which the ACL NAL unit belongs or the identifier of the layer to which the non-ACL NAL unit applies.

[0585] nal_temporal_id_plus1 minus 1 specifies the temporary identifier of the NAL unit.

[0586] ​ NAL unit types according to embodiments are shown.

[0587] ​ Show ​ nal_unit_type included in the NAL unit header of the sample stream NAL unit.

[0588] NAL_TRAIL: The coded tile group of non-TSA and non-STSA trailing atlas frames can be included in the NAL unit. The RBSP syntax structure of the NAL unit is atlas_tile_group_layer_rbsp() or atlas_tile_layer_rbsp(). The type class of the NAL unit is ACL. Depending on the embodiment, the tile group can correspond to a tile.

[0589] NAL TSA: The coded tile group of a TSA atlas frame can be included in a NAL unit. The RBSP syntax structure of the NAL unit is atlas_tile_group_layer_rbsp() or atlas_tile_layer_rbsp(). The type class of the NAL unit is ACL.

[0590] NAL_STSA: The coded tile group of the STSA atlas frame can be included in the NAL unit. The RBSP syntax structure of the NAL unit is atlas_tile_group_layer_rbsp() or atlas_tile_layer_rbsp(). The type class of the NAL unit is ACL.

[0591] NAL_RADL: The coded tile group of a RADL atlas frame can be included in a NAL unit. The RBSP syntax structure of the NAL unit is atlas_tile_group_layer_rbsp() or atlas_tile_layer_rbsp(). The type class of the NAL unit is ACL.

[0592] NAL_RASL: The coded tile group of a RASL atlas frame can be included in a NAL unit. The RBSP syntax structure of the NAL unit is atlas_tile_group_layer_rbsp() or aatlas_tile_layer_rbsp(). The type class of the NAL unit is ACL.

[0593] NAL_SKIP: The coded tile group of the skipped atlas frame can be included in the NAL unit. The RBSP syntax structure of the NAL unit is atlas_tile_group_layer_rbsp() or atlas_tile_layer_rbsp(). The type class of the NAL unit is ACL.

[0594] NAL_RSV_ACL_6 to NAL_RSV_ACL_9: Reserved non-IRAP ACL NAL unit types can be included in NAL units. The type class of the NAL unit is ACL.

[0595] NAL_BLA_W_LP, NAL_BLA_W_RADL, NAL_BLA_N_LP: The coded tile group of the BLA atlas frame can be included in the NAL unit. The RBSP syntax structure of the NAL unit is atlas_tile_group_layer_rbsp() or atlas_tile_layer_rbsp(). The type class of the NAL unit is ACL.

[0596] NAL_GBLA_W_LP, NAL_GBLA_W_RADL, NAL_GBLA_N_LP: The coded tile group of the GBLA atlas frame can be included in the NAL unit. The RBSP syntax structure of the NAL unit is atlas_tile_group_layer_rbsp() or atlas_tile_layer_rbsp(). The type class of the NAL unit is ACL.

[0597] NAL_IDR_W_RADL, NAL_IDR_N_LP: The coded tile group of the IDR atlas frame can be included in the NAL unit. The RBSP syntax structure of the NAL unit is atlas_tile_group_layer_rbsp() or atlas_tile_layer_rbsp(). The type class of the NAL unit is ACL.

[0598] NAL_GIDR_W_RADL, NAL_GIDR_N_LP: The coded tile group of the GIDR atlas frame can be included in the NAL unit. The RBSP syntax structure of the NAL unit is atlas_tile_group_layer_rbsp() or atlas_tile_layer_rbsp(). The type class of the NAL unit is ACL.

[0599] NAL_CRA: The coded tile group of a CRA atlas frame can be included in a NAL unit. The RBSP syntax structure of the NAL unit is atlas_tile_group_layer_rbsp() or atlas_tile_layer_rbsp(). The type class of the NAL unit is ACL.

[0600] NAL_GCRA: The coded tile group of a GCRA atlas frame can be included in a NAL unit. The RBSP syntax structure of the NAL unit is atlas_tile_group_layer_rbsp() or atlas_tile_layer_rbsp(). The type class of the NAL unit is ACL.

[0601] NAL_IRAP_ACL_22, NAL_IRAP_ACL_23: Reserved IRAP ACL NAL unit type can be included in the NAL unit. The type class of the NAL unit is ACL.

[0602] NAL_RSV_ACL_24 to NAL_RSV_ACL_31: Reserved non-IRAP ACL NAL unit types that can be included in NAL units. The type class of the NAL unit is ACL.

[0603] NAL_ASPS: The atlas sequence parameter set can be included in the NAL unit. The RBSP syntax structure of the NAL unit is atlas_sequence_parameter_set_rbsp(). The type class of the NAL unit is non-ACL.

[0604] NAL_AFPS: The atlas frame parameter set can be included in the NAL unit. The RBSP syntax structure of the NAL unit is atlas_frame_parameter_set_rbsp(). The type class of the NAL unit is non-ACL.

[0605] NAL_AUD: Access unit delimiter can be included in the NAL unit. The RBSP syntax structure of the NAL unit is access_unit_delimiter_rbsp(). The type class of the NAL unit is non-ACL.

[0606] NAL_VPCC_AUD: The V-PCC access unit delimiter may be included in the NAL unit. The RBSP syntax structure of the NAL unit is access_unit_delimiter_rbsp(). The type class of the NAL unit is non-ACL.

[0607] NAL_EOS: The NAL unit type can be end of sequence. The RBSP syntax structure of the NAL unit is end_of_seq_rbsp(). The type class of the NAL unit is non-ACL.

[0608] NAL_EOB: The NAL unit type can be the end of the bitstream. The RBSP syntax structure of the NAL unit is end_of_atlas_sub_bitstream_rbsp(). The type class of the NAL unit is non-ACL.

[0609] NAL_FD Filler: The NAL unit type can be filler_data_rbsp(). The type class of the NAL unit is non-ACL.

[0610] NAL_PREFIX_NSEI, NAL_SUFFIX_NSEI: The NAL unit type can be non-essential supplementary enhancement information. The RBSP syntax structure of the NAL unit is sei_rbsp(). The type class of the NAL unit is non-ACL.

[0611] NAL_PREFIX_ESEI, NAL_SUFFIX_ESEI: The NAL unit type can be required supplementary enhancement information. The RBSP syntax structure of the NAL unit is sei_rbsp(). The type class of the NAL unit is non-ACL.

[0612] NAL_AAPS: The NAL unit type can be an atlas adaptation parameter set. The RBSP syntax structure of the NAL unit is atlas_adaptation_parameter_set_rbsp(). The type class of the NAL unit is non-ACL.

[0613] NAL_RSV_NACL_44 to NAL_RSV_NACL_47: The NAL unit type can be a reserved non-ACL NAL unit type. The type class of the NAL unit is non-ACL.

[0614] NAL_UNSPEC_48 to NAL_UNSPEC_63: The NAL unit type may be an unspecified non-ACL NAL unit type. The type class of the NAL unit is non-ACL.

[0615] ​ An atlas sequence parameter set according to an embodiment is shown.

[0616] ​ Show ​ The atlas sequence parameter set (ASPS) included in the atlas bitstream.

[0617] Each sample stream NAL unit may include one of the atlas parameter sets (eg, ASPS, AAPS, or AFPS), information about one or more atlas tiles, and SEI.

[0618] An ASPS may include syntax elements that apply to zero or more complete coded atlas sequences (CAS) determined by the content of syntax elements found in the ASPS referenced by syntax elements found in respective patch group headers.

[0619] ASPS may include the following elements.

[0620] asps_atlas_sequence_parameter_set_id may provide the identifier of the atlas sequence parameter set for reference by other syntax elements.

[0621] asps_frame_width indicates the atlas frame width in terms of integer luma samples of the current atlas.

[0622] asps_frame_height indicates the atlas frame height in terms of integer luma samples of the current atlas.

[0623] asps_log2_patch_packing_block_size specifies the value of the variable PatchPackingBlockSize used for the horizontal and vertical placement of patches within the atlas.

[0624] asps_log2_max_atlas_frame_order_cnt_lsb_minus4 specifies the value of the variable MaxAtlasFrmOrderCntLsb used in the decoding process of the atlas frame order count.

[0625] asps_max_dec_atlas_frame_buffering_minus1 plus 1 specifies the maximum required size of the decoded atlas frame buffer for CAS in units of atlas frame storage buffers.

[0626] asps_long_term_ref_atlas_frames_flag equal to 0 specifies that long-term reference atlas frames are not used for inter prediction of any coded atlas frames in CAS. asps_long_term_ref_atlas_frames_flag equal to 1 specifies that long-term reference atlas frames may be used for inter prediction of one or more coded atlas frames in CAS.

[0627] asps_num_ref_atlas_frame_lists_in_asps specifies the number of ref_list_struct(rlsIdx) syntax structures included in the atlas sequence parameter set.

[0628] Depending on the value of asps_num_ref_atlas_frame_lists_in_asps, ref_list_struct(i) may be included in the atlas sequence parameter set.

[0629] asps_use_eight_orientations_flag equal to 0 specifies that the patch orientation index pdu_orientation_index[i][j] of the patch with index j in the frame with index i is in the range of 0 to 1, inclusive. asps_use_eight_orientations_flag equal to 1 specifies that the patch orientation index pdu_orientation_index[i][j] of the patch with index j in the frame with index i is in the range of 0 to 7, inclusive.

[0630] ASPS extended projection enabled flag (asps_extended_projection_enabled_flag): If the value is 0, it indicates that patch projection information is not signaled for the current atlas tile group or current atlas tile. If the flag is 1, it indicates that patch projection information is signaled for the current atlas tile group or current atlas tile.

[0631] asps_normal_axis_limits_quantization_enabled_flag equal to 1 specifies that the signaled quantization parameter should be used to quantize the normal-axis related elements of a patch data unit, a merged patch data unit, or an inter-patch data unit. If asps_normal_axis_limits_quantization_enabled_flag is equal to 0, quantization is not applied to any normal-axis related elements of a patch data unit, a merged patch data unit, or an inter-patch data unit. When asps_normal_axis_limits_quantization_enabled_flag is 1, atgh_pos_min_z_quantizer may be included in the atlas tile group (or tile) header.

[0632] asps_normal_axis_max_delta_value_enabled_flag equal to 1 specifies that the maximum nominal shift value of the normal axis that may be present in the geometric information for the patch with index i in the frame with index j shall be indicated in the bitstream for each patch data unit, merged patch data unit, or inter-patch data unit. If asps_normal_axis_max_delta_value_enabled_flag is equal to 0, the maximum nominal shift value of the normal axis that may be present in the geometric information for the patch with index i in the frame with index j shall not be indicated in the bitstream for each patch data unit, merged patch data unit, or inter-patch data unit. When asps_normal_axis_max_delta_value_enabled_flag is equal to 1, atgh_pos_delta_max_z_quantizer may be included in the atlas tile group (or patch) header.

[0633] asps_remove_duplicate_point_enabled_flag equal to 1 indicates that duplicate points are not reconstructed for the current atlas, where a duplicate point is a point that has the same 2D and 3D geometric coordinates as another point from a lower-indexed atlas. asps_remove_duplicate_point_enabled_flag equal to 0 indicates that all points are reconstructed.

[0634] asps_max_dec_atlas_frame_buffering_minus1 plus 1 specifies the maximum required size of the decoded atlas frame buffer for CAS in units of atlas frame storage buffers.

[0635] asps_pixel_deinterleaving_flag equal to 1 indicates that the decoded geometry and attribute video of the current atlas contains spatially interleaved pixels from two pictures. asps_pixel_deinterleaving_flag equal to 0 indicates that the decoded geometry and attribute video corresponding to the current atlas contains pixels from only a single picture.

[0636] asps_patch_precedence_order_flag equal to 1 indicates that the patch priority of the current atlas is in the same order as the decoding order. asps_patch_precedence_order_flag equal to 0 indicates that the patch priority of the current atlas is in the opposite order as the decoding order.

[0637] asps_patch_size_quantizer_present_flag equal to 1 indicates that the patch size quantizer parameter is present in the atlas tile group header. asps_patch_size_quantizer_present_flag equal to 0 indicates that the patch size quantizer parameter is not present.

[0638] When asps_patch_size_quantizer_present_flag is equal to 1, atgh_patch_size_x_info_quantizer and atgh_patch_size_y_info_quantizer may be included in the atlas tile group (or patch) header.

[0639] asps_point_local_reconstruction_enabled_flag equal to 1 indicates that point local reconstruction mode information may be present in the bitstream of the current atlas. asps_point_local_reconstruction_enabled_flag equal to 0 indicates that no information related to point local reconstruction mode is present in the bitstream of the current atlas. When asps_point_local_reconstruction_enabled_flag is equal to 1, asps_point_local_reconstruction_information may be carried in the atlas sequence parameter set.

[0640] asps_map_count_minus1 plus 1 indicates the number of maps available for encoding the geometry and attribute data of the current atlas.

[0641] asps_map_pixel_deinterleaving_flag[i] equal to 1 indicates that the decoded geometry and attribute video corresponding to the picture with index i in the current atlas contains spatially interleaved pixels corresponding to two pictures. asps_map_pixel_deinterleaving_flag[i] equal to 0 indicates that the decoded geometry and attribute video corresponding to the picture with index i in the current atlas contains pixels corresponding to a single picture. When not present, the value of asps_map_pixel_deinterleaving_flag[i] may be inferred to be 0.

[0642] ASPS EOM fixed bit count (asps_eom_fix_bit_count_minus1): If this value is increased by 1, it indicates the bit size of the EOM codeword.

[0643] ASPS surface thickness (asps_surface_thickness_minus1): When asps_pixel_deinterleaving_enabled_flag or asps_point_local_reconstruction_enabled_flag is equal to 1, plus 1 specifies the maximum absolute difference between the explicitly encoded depth value and the interpolated depth value.

[0644] ASPS vui parameter present flag (asps_vui_parameters_present_flag): equal to 1 specifies that the vui_parameters() syntax structure is present. asps_vui_parameters_present_flag equal to 0 specifies that the vui_parameters() syntax structure is not present.

[0645] asps_extension_flag equal to 0 specifies that the asps_extension_data_flag syntax element is not present in the ASPS RBSP syntax structure.

[0646] asps_extension_data_flag indicates that data for extension is included in the ASPS RBSP syntax structure.

[0647] rbsp_trailing_bits is used to fill the remaining bits with 0s after adding the stop bit 1 to indicate the end of the RBSP data to facilitate byte alignment.

[0648] ​ An atlas frame parameter set according to an embodiment is shown.

[0649] ​ Show​ The detailed syntax of the atlas frame parameter set included in the atlas bitstream.

[0650] An atlas frame parameter set (AFPS) comprises a syntax structure containing syntax elements that apply to zero or more complete coded atlas frames.

[0651] afps_atlas_frame_parameter_set_id identifies the atlas frame parameter set for reference by other syntax elements.

[0652] afps_atlas_sequence_parameter_set_id specifies the value of asps_atlas_sequence_parameter_set_id of the active atlas sequence parameter set.

[0653] Will refer to ​ Describe atlas_frame_tile_information().

[0654] afps_output_flag_present_flag equal to 1 indicates that the atgh_frame_output_flag syntax element is present in the associated patch group header. afps_output_flag_present_flag equal to 0 indicates that the atgh_frame_output_flag syntax element is not present in the associated patch group header.

[0655] afps_num_ref_idx_default_active_minus1 plus 1 specifies the inferred value of the variable NumRefIdxActive for tiles or tilings with atgh_num_ref_idx_active_override_flag equal to 0.

[0656] afps_additional_lt_afoc_lsb_len specifies the value of the variable MaxLtAtlasFrmOrderCntLsb used for the decoding process of the reference atlas frame.

[0657] afps_3d_pos_x_bit_count_minus1 plus 1 specifies the number of bits in the fixed-length representation of pdu_3d_pos_x[j] for the patch of index j in the atlas tiling set referenced by afps_atlas_frame_parameter_set_id.

[0658] afps_3d_pos_y_bit_count_minus1 plus 1 specifies the number of bits in the fixed-length representation of pdu_3d_pos_y[j] for the patch of index j in the atlas tiling set referenced by afps_atlas_frame_parameter_set_id.

[0659] afps_lod_mode_enabled_flag equal to 1 indicates that LOD parameters may be present in a patch. afps_lod_mode_enabled_flag equal to 0 indicates that LOD parameters are not present in a patch.

[0660] afps_override_eom_for_depth_flag equal to 1 indicates that the values ​​of afps_eom_number_of_patch_bit_count_minus1 and afps_eom_max_bit_count_minus1 are explicitly present in the bitstream. afps_override_eom_for_depth_flag equal to 0 indicates that the values ​​of afps_eom_number_of_patch_bit_count_minus1 and afps_eom_max_bit_count_minus1 are implicitly derived.

[0661] afps_eom_number_of_patch_bit_count_minus1 plus 1 specifies the number of bits used to represent the number of geometry patches associated in the EOM attribute patch in the atlas frame associated with this atlas frame parameter set.

[0662] afps_eom_max_bit_count_minus1 plus 1 specifies the number of bits used to represent the number of EOM points per geometry patch associated in the EOM attribute patch in the atlas frame associated with this atlas frame parameter set.

[0663] afps_raw_3d_pos_bit_count_explicit_mode_flag equal to 1 indicates that the number of bits in the fixed-length representation of rpdu_3d_pos_x, rpdu_3d_pos_y, and rpdu_3d_pos_z is explicitly coded by atgh_raw_3d_pos_axis_bit_count_minus1 in the atlas tile group header that references afps_atlas_frame_parameter_set_id. afps_raw_3d_pos_bit_count_explicit_mode_flag equal to 0 indicates that the value of atgh_raw_3d_pos_axis_bit_count_minus1 is implicitly derived.

[0664] afps_extension_flag equal to 0 specifies that the afps_extension_data_flag syntax element is not present in the AFPS RBSP syntax structure.

[0665] afps_extension_data_flag may include extension related data.

[0666] ​ Atlas_frame_tile_information according to an embodiment is shown.

[0667] ​ Show ​ The syntax of atlas frame patch information included in the atlas bitstream.

[0668] afti_single_tile_in_atlas_frame_flag equal to 1 specifies that there is only one tile-reference AFPS in each atlas frame. afti_single_tile_in_atlas_frame_flag equal to 0 specifies that there are more than one tile-reference AFPS in each atlas frame.

[0669] afti_uniform_tile_spacing_flag equal to 1 specifies that tile column and row boundaries are uniformly distributed across the atlas frame and are signaled using the syntax elements afti_tile_cols_width_minus1 and afti_tile_rows_height_minus1, respectively. afti_uniform_tile_spacing_flag equal to 0 specifies that tile column and row boundaries may or may not be uniformly distributed across the atlas frame and are signaled using the syntax elements afti_num_tile_columns_minus1 and afti_num_tile_rows_minus1 and a list of syntax element pairs afti_tile_column_width_minus1[i] and afti_tile_row_height_minus1[i].

[0670] When afti_uniform_tile_spacing_flag is equal to 1, afti_tile_cols_width_minus1 plus 1 specifies the width of tile columns except the rightmost tile column of the atlas frame in units of 64 samples.

[0671] When afti_uniform_tile_spacing_flag is equal to 1, afti_tile_rows_height_minus1 plus 1 specifies the height of tile rows except the bottom tile row of the atlas frame in units of 64 samples.

[0672] When afti_uniform_tile_spacing_flag is not equal to 1, the atlas frame tile information includes the following elements.

[0673] When afti_uniform_tile_spacing_flag is equal to 0, afti_num_tile_columns_minus1 plus 1 specifies the number of tile columns to partition the atlas frame.

[0674] When pti_uniform_tile_spacing_flag is equal to 0, afti_num_tile_rows_minus1 plus 1 specifies the number of tile rows to partition the atlas frame.

[0675] afti_tile_column_width_minus1[i] plus 1 specifies the width of the i-th tile column in units of 64 samples.

[0676] Depending on the value of afti_num_tile_columns_minus1, afti_tile_column_width_minus1[i] is included in the atlas frame tile information.

[0677] afti_tile_row_height_minus1[i] plus 1 specifies the height of the i-th tile row in units of 64 samples.

[0678] Depending on the value of afti_num_tile_rows_minus1, afti_tile_row_height_minus1[i] is included in the atlas frame tile information.

[0679] afti_single_tile_per_tile_group_flag equal to 1 specifies that each tile group (or tile) referencing this AFPS includes one tile. afti_single_tile_per_tile_group_flag equal to 0 specifies that the tile group referencing this AFPS may include more than one tile. When not present, the value of afti_single_tile_per_tile_group_flag may be inferred to be equal to 1.

[0680] Based on the value of afti_num_tile_groups_in_atlas_frame_minus1, afti_tile_idx[i] is included in the atlas frame tile information.

[0681] When afti_single_tile_per_tile_group_flag is equal to 0, afti_num_tile_groups_in_atlas_frame_minus1 is carried in the atlas frame tile information.

[0682] afti_num_tile_groups_in_atlas_frame_minus1 plus 1 specifies the number of tile groups in each atlas frame of the reference AFPS. The value of afti_num_tile_groups_in_atlas_frame_minus1 can be in the range of 0 to NumTilesInAtlasFrame-1 (inclusive). When not present and afti_single_tile_per_tile_group_flag is equal to 1, the value of afti_num_tile_groups_in_atlas_frame_minus1 can be inferred to be equal to NumTilesInAtlasFrame-1.

[0683] Depending on the value of afti_num_tile_groups_in_atlas_frame_minus1, the atlas frame tile information includes the following elements.

[0684] afti_top_left_tile_idx[i] specifies the tile index of the tile located at the top left corner of the i-th tile group (tile). For any i not equal to j, the value of afti_top_left_tile_idx[i] is not equal to the value of afti_top_left_tile_idx[j]. When not present, the value of afti_top_left_tile_idx[i] can be inferred to be equal to i. The length of the afti_top_left_tile_idx[i] syntax element may be Ceil(Log2(NumTilesInAtlasFrame) bits.

[0685] afti_bottom_right_tile_idx_delta[i] specifies the difference between the tile index of the tile located at the bottom right corner of the i-th tile group (tile) and afti_top_left_tile_idx[i]. When afti_single_tile_per_tile_group_flag is equal to 1, the value of afti_bottom_right_tile_idx_delta[i] is inferred to be equal to 0. The length of the afti_bottom_right_tile_idx_delta[i] syntax element is Ceil(Log2(NumTilesInAtlasFrame-afti_top_left_tile_idx[i])) bits.

[0686] afti_signalled_tile_group_id_flag equal to 1 specifies that the tile group ID is signaled for each tile group.

[0687] When afti_signalled_tile_group_id_flag is equal to 1, afti_signalled_tile_group_id_length_minus1 and afti_tile_group_id[i] may be carried in the atlas frame tile information. When the value of this flag is 0, the tile group ID may not be signaled.

[0688] afti_signalled_tile_group_id_length_minus1 plus 1 specifies the number of bits used to represent the syntax element afti_tile_group_id[i] (when present) and the syntax element atgh_address in the tile group header.

[0689] afti_tile_group_id[i] specifies the tile group ID of the i-th tile group. The length of the afti_tile_group_id[i] syntax element is afti_signalled_tile_group_id_length_minus1+1 bits.

[0690] Depending on the value of afti_num_tile_groups_in_atlas_frame_minus1, afti_tile_group_id[i] is included in the atlas frame tile information.

[0691] ​ An atlas_adaptation_parameter_set according to an embodiment is shown.

[0692] ​ Show ​ The atlas adaptation parameter set included in the atlas bitstream.

[0693] An AAPS RBSP includes parameters referenced by one or more coded patch set NAL units of a coded atlas frame. At any given moment during the decoding process, at most one AAPS RBSP is considered active. Activation of any particular AAPS RBSP results in deactivation of the previously active AAPS RBSP.

[0694] aaps_atlas_adaptation_parameter_set_id may identify the atlas adaptation parameter set for reference by other syntax elements.

[0695] aaps_atlas_sequence_parameter_set_id specifies the value of asps_atlas_sequence_parameter_set_id of the active atlas sequence parameter set.

[0696] aaps_camera_parameters_present_flag is equal to 1 and specifies that the camera parameters are present in the current atlas adaptation parameter set. aaps_camera_parameters_present_flag is equal to 0 and specifies that the camera parameters are not present in the current adaptation parameter set.

[0697] aaps_extension_flag equal to 0 specifies that the aaps_extension_data_flag syntax element is not present in the AAPS RBSP syntax structure.

[0698] aaps_extension_data_flag can have various values.

[0699] ​ Shows atlas_camera_parameters according to an embodiment.

[0700] ​ Show ​ atlas_camera_parameters included in .

[0701] acp_camera_model indicates the camera model of the point cloud frame associated with the current adaptation parameter set.

[0702] When acp_camera_model is equal to 0, the name of acp_camera_model is not specified.

[0703] When acp_camera_model is equal to 1, acp_camera_model indicates an orthographic camera model.

[0704] When acp_camera_model has a value of 2 to 255, acp_camera_model may be reserved for future use.

[0705] acp_scale_enabled_flag equal to 1 indicates that the scale parameters of the current camera model exist. acp_scale_enabled_flag equal to 0 indicates that the scale parameters of the current camera model do not exist.

[0706] acp_offset_enabled_flag is equal to 1, indicating that the offset parameters of the current camera model exist. acp_offset_enabled_flag is equal to 0, indicating that the offset parameters of the current camera model do not exist.

[0707] acp_rotation_enabled_flag is equal to 1, indicating that the rotation parameters of the current camera model exist. acp_rotation_enabled_flag is equal to 0, indicating that the rotation parameters of the current camera model do not exist.

[0708] acp_scale_on_axis[d] specifies the value of scale[d] along the d-axis of the current camera model. The value of d is in the range of 0 to 2 (inclusive), with values ​​0, 1, and 2 corresponding to the X, Y, and Z axes respectively.

[0709] acp_offset_on_axis[d] indicates the value of the offset along the d-axis of the current camera model, where d is in the range of 0 to 2 (inclusive). Values ​​of d equal to 0, 1, and 2 may correspond to the X, Y, and Z axes, respectively.

[0710] acp_rotation_qx specifies the x component qX of the rotation of the current camera model using a quaternion.

[0711] acp_rotation_qy specifies the y component qY of the rotation of the current camera model using a quaternion.

[0712] acp_rotation_qz specifies the z component qZ of the rotation of the current camera model using a quaternion.

[0713] ​ An atlas_tile_group_layer and an atlas_tile_group_header according to an embodiment are shown.

[0714] ​ Show ​ The atlas patch group layer and atlas patch group header are included in the atlas bitstream.

[0715] atgh_atlas_frame_parameter_set_id specifies the value of afps_atlas_frame_parameter_set_id of the active atlas frame parameter set for the current atlas tile group.

[0716] atgh_atlas_adaptation_parameter_set_id specifies the value of aaps_atlas_adaptation_parameter_set_id of the active atlas adaptation parameter set for the current atlas tile set.

[0717] atgh_address specifies the tile group address of the tile group. When not present, the value of atgh_address is inferred to be equal to 0. The tile group address is the tile group ID of the tile group. The length of atgh_address may be afti_signalled_tile_group_id_length_minus1+1 bits. When afti_signalled_tile_group_id_flag is equal to 0, the value of atgh_address is in the range of 0 to afti_num_tile_groups_in_atlas_frame_minus1 (inclusive). Additionally, the value of atgh_address may be in the range of 0 to 2 (afti_signalled_tile_group_id_length_minus1+1) In the range of -1 (inclusive).

[0718] atgh_type specifies the encoding type of the current atlas tile group.

[0719] When atgh_type is equal to 0, the type is inter-atlas tile group (P_TILE_GRP).

[0720] When atgh_type is equal to 1, the type is intra-atlas tile group (I_TILE_GRP).

[0721] When atgh_type is equal to 2, the type is SKIP atlas tile group (SKIP_TILE_GRP).

[0722] When atgh_type is equal to 3, the type is reserved.

[0723] atgh_atlas_output_flag may affect the decoded atlas output and removal process.

[0724] atgh_atlas_frm_order_cnt_lsb specifies the atlas frame order count modulo MaxAtlasFrmOrderCntLsb of the current atlas tile group.

[0725] atgh_ref_atlas_frame_list_sps_flag equal to 1 specifies that the reference atlas frame list of the current atlas patch may be derived based on one of the ref_list_struct(rlsIdx) syntax structures in the active ASPS. atgh_ref_atlas_frame_list_sps_flag equal to 0 specifies that the reference atlas frame list of the current atlas patch may be derived based on the ref_list_struct(rlsIdx) syntax structure directly included in the patch header of the current atlas patch.

[0726] atgh_ref_atlas_frame_list_idx specifies an index into the list of ref_list_struct(rlsIdx) syntax structures included in the active ASPS of the ref_list_struct(rlsIdx) syntax structure used for deriving the reference atlas frame list for the current atlas patch.

[0727] atgh_additional_afoc_lsb_present_flag[j] equal to 1 specifies that atgh_additional_afoc_lsb_val[j] is present for the current atlas tile set. atgh_additional_afoc_lsb_present_flag[j] equal to 0 specifies that atgh_additional_afoc_lsb_val[j] is not present.

[0728] atgh_additional_afoc_lsb_val[j] specifies the value of FullAtlasFrmOrderCntLsbLt[RlsIdx][j] of the current atlas tile group.

[0729] atgh_pos_min_z_quantizer specifies the quantizer to be applied to the pdu_3d_pos_min_z[p] value for patch p. If atgh_pos_min_z_quantizer is not present, its value is inferred to be equal to 0.

[0730] atgh_pos_delta_max_z_quantizer specifies the quantizer to be applied to the pdu_3d_pos_delta_max_z[p] value for the patch at index p. When atgh_pos_delta_max_z_quantizer is not present, its value may be inferred to be equal to 0.

[0731] atgh_patch_size_x_info_quantizer specifies the value of the quantizer PatchSizeXQuantizer to be applied to the variables pdu_2d_size_x_minus1[p], mpdu_2d_delta_size_x[p], ipdu_2d_delta_size_x[p], rpdu_2d_size_x_minus1[p], and epdu_2d_size_x_minus1[p] for the patch of index p. When atgh_patch_size_x_info_quantizer is not present, its value can be inferred to be equal to asps_log2_patch_packing_block_size.

[0732] atgh_patch_size_y_info_quantizer specifies the value of the quantizer PatchSizeYQuantizer for the variables pdu_2d_size_y_minus1[p], mpdu_2d_delta_size_y[p], ipdu_2d_delta_size_y[p], rpdu_2d_size_y_minus1[p], and epdu_2d_size_y_minus1[p] to be applied to the patch of index p. When atgh_patch_size_y_info_quantizer is not present, its value can be inferred to be equal to asps_log2_patch_packing_block_size.

[0733] atgh_raw_3d_pos_axis_bit_count_minus1 plus 1 specifies the number of bits in the fixed length representation of rpdu_3d_pos_x, rpdu_3d_pos_y, and rpdu_3d_pos_z.

[0734] atgh_num_ref_idx_active_override_flag equal to 1 specifies that the syntax element atgh_num_ref_idx_active_minus1 is present for the current atlas tile. atgh_num_ref_idx_active_override_flag equal to 0 specifies that the syntax element atgh_num_ref_idx_active_minus1 is not present. When atgh_num_ref_idx_active_override_flag is not present, its value may be inferred to be equal to 0.

[0735] atgh_num_ref_idx_active_minus1 specifies the maximum reference index for referencing an atlas frame list that can be used to decode the current atlas patch set. When the value of NumRefIdxActive is equal to 0, no reference index of a reference atlas frame list can be used to decode the current atlas patch set.

[0736] byte_alignment is used to fill the remaining bits with 0 after adding 1 as a stop bit to indicate the end of data to facilitate byte alignment.

[0737] ​ A reference list structure (ref_list_struct) according to an embodiment is shown.

[0738] ​Show Figure 38 The reference list structure included in .

[0739] num_ref_entries[rlsIdx] specifies the number of entries in the ref_list_struct(rlsIdx) syntax structure.

[0740] st_ref_atlas_frame_flag[rlsIdx][i] equal to 1 specifies that the i-th entry in the ref_list_struct(rlsIdx) syntax structure is a short-term reference atlas frame entry. st_ref_atlas_frame_flag[rlsIdx][i] equal to 0 specifies that the i-th entry in the ref_list_struct(rlsIdx) syntax structure is a long-term reference atlas frame entry. When not present, the value of st_ref_atlas_frame_flag[rlsIdx][i] can be inferred to be equal to 1.

[0741] abs_delta_afoc_st[rlsIdx][i] specifies the absolute difference between the atlas frame order count values ​​of the current atlas tile group and the atlas frame referenced by the i-th entry when the i-th entry is the first short-term reference atlas frame entry in the ref_list_struct(rlsIdx) syntax structure, or specifies the absolute difference between the atlas frame order count values ​​of the i-th entry and the atlas frame referenced by the previous short-term reference atlas frame entry in the ref_list_struct(rlsIdx) syntax structure when the i-th entry is a short-term reference atlas frame entry but not the first short-term reference atlas frame entry in the ref_list_struct(rlsIdx) syntax structure.

[0742] strpf_entry_sign_flag[rlsIdx][i] equal to 1 specifies that the i-th entry in the syntax structure ref_list_struct(rlsIdx) has a value greater than or equal to 0. strpf_entry_sign_flag[rlsIdx][i] equal to 0 specifies that the i-th entry in the syntax structure ref_list_struct(rlsIdx) has a value less than 0. When not present, the value of strpf_entry_sign_flag[rlsIdx][i] may be inferred to be equal to 1.

[0743] afoc_lsb_lt[rlsIdx][i] specifies the value of the atlas frame order count modulo MaxAtlasFrmOrderCntLsb of the atlas frame referenced by the i-th entry in the ref_list_struct(rlsIdx) syntax structure. The length of the afoc_lsb_lt[rlsIdx][i] syntax element may be asps_log2_max_atlas_frame_order_cnt_lsb_minus4+4 bits.

[0744] Figure 40 Atlas tile group data (atlas_tile_group_data_unit) according to an embodiment is shown.

[0745] Figure 40 Show Figure 30 The atlas patch group data included in the atlas bitstream.

[0746] atgdu_patch_mode[p] indicates the patch mode of the patch of index p in the current atlas patch. A patch of atgh_type=SKIP_TILE_GRP means that the entire patch information is directly copied from the patch with the same atgh_address as the current patch corresponding to the first reference atlas frame.

[0747] Figure 41 A patch pattern according to an embodiment is shown.

[0748] Figure 41 Shown with Figure 40 A detailed description of the patch mode is provided.

[0749] The patch mode type of the I_TILE_GRP type atlas tile group can be specified as follows.

[0750] atgdu_patch_mode equal to 0 indicates the non-prediction patch mode with identifier I_INTRA.

[0751] atgdu_patch_mode equal to 1 indicates the RAW point patch mode with identifier I_RAW.

[0752] atgdu_patch_mode equal to 2 indicates the EOM point patch mode with identifier I_EOM.

[0753] The value of atgdu_patch_mode equal to 3 to 13 indicates the reservation mode with the identifier I_RESERVED.

[0754] atgdu_patch_mode equal to 14 indicates the patch termination mode with identifier I_END.

[0755] The patch mode type for a P_TILE_GRP type atlas tile group can be specified as follows.

[0756] atgdu_patch_mode equal to 0 indicates patch skip mode with identifier P_SKIP.

[0757] atgdu_patch_mode equal to 1 indicates the patch merging mode with identifier P_MERGE.

[0758] atgdu_patch_mode equal to 2 indicates the inter-frame prediction patch mode with identifier P_INTER.

[0759] atgdu_patch_mode equal to 3 indicates the non-predictive patch mode with identifier P_INTRA.

[0760] atgdu_patch_mode equal to 4 indicates the RAW point patch mode with the identifier P_RAW.

[0761] atgdu_patch_mode equal to 5 indicates the EOM point patch mode with identifier P_EOM.

[0762] The value of atgdu_patch_mode equal to 6 to 13 indicates the reservation mode with the identifier P_RESERVED.

[0763] atgdu_patch_mode equal to 14 indicates the patch termination mode with identifier P_END.

[0764] The patch mode type for an atlas tile group of type SKIP_TILE_GRP can be specified as follows.

[0765] atgdu_patch_mode equal to 0 indicates patch skip mode with identifier P_SKIP.

[0766] Figure 42 Patch_information_data according to an embodiment is shown.

[0767] Figure 42 Shown included in Figure 40 patch_information_data in .

[0768] patch_information_data can carry the following information according to patchIdx and patchMode.

[0769] When atgh_type is SKIP_TILE_GR, it can carry skip_patch_data_unit (patchIdx).

[0770] When atgh_type is P_TILE_GR, the following elements can be carried. Specifically, when patchMode is P_SKIP, skip_patch_data_unit(patchIdx) can be provided. When patchMode is P_MERGE, merge_patch_data_unit(patchIdx) can be provided. When patchMode is P_INTRA, patch_data_unit(patchIdx) can be carried. When patchMode is P_INTER, inter_patch_data_unit(patchIdx) can be carried. When patchMode is P_RAW, raw_patch_data_unit(patchIdx) can be carried. When patchMode is P_EOM, eom_patch_data_unit(patchIdx) can be carried.

[0771] When atgh_type is I_TILE_GR, the following elements can be carried. Specifically, when patchMode is I_INTRA, patch_data_unit (patchIdx) can be carried. When patchMode is I_RAW, raw_patch_data_unit (patchIdx) can be carried. When patchMode is I_EOM, eom_patch_data_unit (patchIdx) can be carried.

[0772] Figure 42 A patch_data_unit according to an embodiment is shown.

[0773] pdu_2d_pos_x[p] specifies the x-coordinate (or left offset) of the top-left corner of the patch bounding box of patch p in the current atlas tile group tileGroupIdx, expressed as a multiple of PatchPackingBlockSize.

[0774] pdu_2d_pos_y[p] specifies the y coordinate (or top offset) of the top-left corner of the patch bounding box of patch p in the current atlas tile group tileGroupIdx, expressed as a multiple of PatchPackingBlockSize.

[0775] pdu_2d_size_x_minus1[p] plus 1 specifies the quantized width value of the patch with index p in the current atlas tile group tileGroupIdx.

[0776] pdu_2d_size_y_minus1[p] plus 1 specifies the quantized height value of the patch with index p in the current atlas tile group tileGroupIdx.

[0777] pdu_3d_pos_x[p] specifies the offset along the tangential axis to be applied to the reconstructed patch point in the patch at index p of the current atlas tiling.

[0778] pdu_3d_pos_y[p] specifies the offset along the bitangent axis to be applied to the reconstructed patch point in the patch at index p of the current atlas tiling.

[0779] pdu_3d_pos_min_z[p] specifies the offset along the normal axis to be applied to the reconstructed patch point in the patch at index p of the current atlas tiling.

[0780] If present, pdu_3d_pos_delta_max_z[p] specifies the nominal maximum value along the normal axis of the offset in the patch of index p of the current atlas tile set that is expected to be present in the reconstructed bit-depth patch geometry samples after conversion to its nominal representation.

[0781] pdu_projection_id[p] specifies the projection mode and the index of the projection plane normal for the patch at index p of the current atlas tile group.

[0782] pdu_orientation_index[p] indicates the patch orientation index of the patch of index p in the current atlas tile group.

[0783] pdu_lod_enabled_flag[p] equal to 1 specifies that LOD parameters exist for the current patch p. When pdu_lod_enabled_flag[p] equal to 0, LOD parameters do not exist for the current patch.

[0784] pdu_lod_scale_x_minus1[p] specifies the LOD scaling factor to be applied to the local x-coordinate of the point in the patch at index p of the current atlas tile set before being added to the patch coordinate Patch3dPosX[p].

[0785] pdu_lod_scale_y[p] specifies the LOD scale factor to be applied to the local y coordinate of the point in the patch at index p of the current atlas tile set before being added to the patch coordinate Patch3dPosY[p].

[0786] point_local_reconstruction_data(patchIdx) may include information for enabling a decoder to reconstruct a point lost due to compression loss or the like.

[0787] Figure 43 Patch information according to an embodiment is shown.

[0788] Figure 43 Show Figure 42 The patch orientation index of the patch of atlas tiling index p.

[0789] Depending on the value of x, the identifier, rotation, and offset information can be as follows Figure 43 Configuration shown. The patch can have an orientation based on rotation and offset.

[0790] Figure 44 An SEI message according to an embodiment is shown.

[0791] Figure 44 Shown included in Figure 30 Syntax of SEI messages in atlas sub-bitstreams.

[0792] According to the embodiment, the method / apparatus may generate and transmit SEI messages. According to the embodiment, the method / apparatus may assist in processing related to decoding, reconstruction, display, or other purposes based on SEI messages. According to the embodiment, there may be two types of SEI messages: required SEI messages and non-essential SEI messages.

[0793] The decoding process may not require non-essential SEI messages. The requirements may vary depending on the performance of the decoder.

[0794] The required SEI messages may form part of the V-PCC bitstream ( Figure 24 It may be a required element of the bitstream configuration. Required SEI messages can be categorized into the following two types.

[0795] Type A mandatory SEI messages: These SEIs may contain information required to check bitstream conformance and output timing decoder conformance. A V-PCC decoder specifying A shall not discard any relevant Type A mandatory SEI messages and may use such information for bitstream decoding.

[0796] Type B mandatory SEI messages: A V-PCC decoder conforming to a specific reconstruction profile shall not discard any relevant Type B mandatory SEI messages and use this information for 3D point cloud reconstruction.

[0797] sei_message() may include one or more sei_payloads.

[0798] sei can have various payload configurations depending on the NAL unit type.

[0799] Figure 45 An encapsulated V-PCC data container structure according to an embodiment is shown.

[0800] Figure 1 The point cloud video encoder 10002 of the sending device 10000, Figure 4 and Figure 15 The encoder, Figure 18 The sending device, Figure 29 Video / image encoders 20002 and 20003, Figure 21 Processors and encoders 21000 to 21008 and Figure 23 The XR device 2330 generates a bitstream containing point cloud data according to an embodiment.

[0801] Figure 1 File / fragment encapsulator 10003, Figure 20 File / fragment encapsulator 20004, Figure 21 File / fragment encapsulator 21009 and Figure 23 The XR device formats the bitstream as Figure 24 and Figure 25 The file structure.

[0802] Similarly, Figure 1 The file / segment decapsulation module 10007 of the receiving device 10005, Figures 20 to 23 File / fragment decapsulators 20005, 21009, and 22000 and Figure 23 The XR device 2330 receives and decapsulates the file and parses the bitstream. Figure 1 Point cloud video decoder 10008, Figure 16 and Figure 17 decoder, Figure 19 receiving device, Figures 20 to 23 Video / image decoders 20006, 21007, 21008, 22001, and 22002 and Figure 23 The XR device 2330 decodes to restore the point cloud data.

[0803] Figure 45 and Figure 46 Shows the structure of a point cloud data container according to the ISOBMFF file format.

[0804] Figure 45 and Figure 46 A structure for transporting a container of point clouds based on multiple tracks is shown.

[0805] The method / apparatus according to the embodiment may transmit / receive a container file including point cloud data and additional data related to the point cloud data based on a plurality of tracks.

[0806] Track 1 70000 is the attribute track and may contain Figure 1 、 Figure 4 、 Figure 15 、 Figure 18 The attribute data 70040 encoded as shown.

[0807] Track 2 70010 is an occupied track and may contain Figure 1 、 Figure 4 、 Figure 15 、 Figure 18 The encoded geometric data 70050 is shown in FIG.

[0808] Track 3 70020 is a geometric track and may contain Figure 1 、 Figure 4 、 Figure 15 、 Figure 18 The occupancy data 70060 is encoded as shown.

[0809] Track 4 70030 is a v-pcc (v3c) track and may include an atlas bitstream 70070 containing data related to point cloud data.

[0810] Each track consists of a sample entry and a sample. A sample is a unit corresponding to a frame. To decode the Nth frame, a sample or sample entry corresponding to the Nth frame is required. A sample entry may contain information describing the sample.

[0811] Figure 46 The structure of a file according to an embodiment is shown.

[0812] The v3c track 71000 corresponds to track 4 70030. Data contained in the v3c track 71000 may have a format of a data container called a box. The v3c track 71000 contains reference information about V3C component tracks 71010 to 71030.

[0813] The receiving method / device according to the embodiment may receive a message including: Figure 46 The container of point cloud data shown (which may be referred to as a file) and parses the V3C track, and occupancy data, geometry data, and attribute data may be decoded and reconstructed based on reference information contained in the V3C track.

[0814] Occupancy track 71010 corresponds to track 2 70010 and contains occupancy data. Geometry track 71020 corresponds to track 3 70020 and contains geometry data. Attribute track 71030 corresponds to track 1 70000 and contains attribute data.

[0815] V-PCC system

[0816] V-PCC system refers to the Figure 45 and Figure 46 The files (single and / or multiple tracks) shown are transmitted as Figures 20 to 22 Elements of the encoded point cloud data are shown.

[0817] Video-based point cloud compression represents the volumetric coding of point cloud visual information. A V-PCC bitstream containing a coded point cloud sequence (CPCS) may consist of a V-PCC parameter set (VPS), a coded atlas bitstream, a 2D video coded occupancy map bitstream, a 2D video coded geometry bitstream, and zero or more 2D video coded attribute bitstreams.

[0818] The method / device according to the embodiment can be implemented by a file / segment encapsulator ( Figure 20 20004, Figure 21 21009) received Figure 25 and Figure 26 V-PCC bitstream. When the file ( Figures 20 to 22 ) or point cloud data is an image, the method / device can be based on the project ( Figure 47 ) container structure to encapsulate and transmit the following information.

[0819] When the file ( Figures 20 to 22 ) or when the point cloud data is an image, the method / device according to the embodiment may be implemented by a file / fragment decapsulator ( Figure 20 20005, Figure 22 22000) to receive and decapsulate the project ( Figure 47 ).

[0820] Volumetric Vision Track

[0821] A volumetric visual track can be identified by the volumetric visual media handler type "volv" in the HandlerBox of the MediaBox.

[0822] Volumetric Visual Media Header

[0823] Frame type: "vvhd"

[0824] Container: MediaInformationBox

[0825] Mandatory: Yes

[0826] Quantity: Exactly one

[0827] The volumetric visual track uses the VolumetricVisualMediaHeaderBox in the MediaInformationBox.

[0828] aligned(8)class VolumetricVisualMediaHeaderBox

[0829] extends FullBox('vvhd',version=0,1){

[0830] }

[0831] "version" is an integer specifying the version of the box.

[0832] Volumetric Vision Sample Entries

[0833] Volumetric Visual Tracks should use VolumetricVisualSampleEntry.

[0834] class VolumetricVisualSampleEntry(codingname)extends SampleEntry(codingname){

[0835] unsigned int(8)

[32] compressor_name;

[0836] }

[0837] compressor_name is a name used for informational purposes. It is formatted as a fixed 32-byte field, with the first byte set to the number of bytes to display, followed by the number of bytes of displayable data encoded using UTF-8, and then padded to complete the total of 32 bytes (including the size byte). This field can be set to 0.

[0838] Volumetric Vision Sample

[0839] The format of volumetric vision samples is defined by the encoding system.

[0840] Hereinafter, detailed information about a common data structure included in a volumetric vision track according to the V-PCC system will be described.

[0841] Common data structures

[0842] V-PCC unit head frame

[0843] This box may be present in the sample entry of a V-PCC track and in the scheme information of all video coding V-PCC component tracks. This box may contain the V-PCC unit header for the data carried by the corresponding track.

[0844] aligned(8)class VPCCUnitHeaderBox extends FullBox('vunt',version=0,0){

[0845] vpcc_unit_header()unit_header;

[0846] }

[0847] This box contains vpcc_unit_header() as above.

[0848] V-PCC decoder configuration box

[0849] The V-PCC decoder configuration box may include a VPCCDecoderConfigurationRecord.

[0850] class VPCCConfigurationBox extends Box('vpcC'){

[0851] VPCCDecoderConfigurationRecord()VPCCConfig;

[0852] }

[0853] The record contains a version field. The version field can define version 1 of the record. Incompatible changes to the record can be indicated by a change in the version number. A decoder can determine whether to decode the record or bitstream based on the version number.

[0854] The VPCCParameterSet array includes vpcc_parameter_set().

[0855] The SetupUnit array includes the atlas parameter sets that are constant for the stream referenced by the sample entry, where the decoder configuration record and atlas substream SEI message are present.

[0856] The SetupUnit array may be constant for the stream referenced by the sample entry.Decoder configuration records may be present as well as atlas substream SEI messages.

[0857] aligned(8)class VPCCDecoderConfigurationRecord{

[0858] unsigned int(8)ConfigurationVersion=1;

[0859] unsigned int(2)lengthSizeMinusOne;

[0860] bit(1)reserved=1;

[0861] unsigned int(5)numOfVPCCParameterSets;

[0862] for(i=0;i <numOfVPCCParameterSets;i++){

[0863] unsigned int(16)VPCCParameterSetLength;

[0864] vpcc_unit(VPCCParameterSetLength)vpccParameterSet; / / The parameter vpccParameterSet is as defined in the standard document ISO / IEC 23090.

[0865] }

[0866] For example, vpccParameterSet may include information about point cloud data, such as vps_v3c_parameter_set_id, vps_atlas_id, vps_frame_width[j], vps_frame_height[j], vps_multiple_map_streams_present_flag[j], occupancy_information(j), geometry_information(j), and attribute_information(j).

[0867] unsigned int(8)numOfSetupUnitArrays;

[0868] for(j=0;j <numOfSetupUnitArrays;j++){

[0869] bit(1)array_completeness;

[0870] bit(1)reserved=0;

[0871] unsigned int(6)NAL_unit_type;

[0872] unsigned int(8)numNALUnits;

[0873] for(i=0;i <numNALUnits;i++){

[0874] unsigned int(16)SetupUnitLength;

[0875] nal_unit(SetupUnitLength) setupUnit; / / setupUnit is as defined in ISO / IEC 23090-5.

[0876] }

[0877] }

[0878] }

[0879] configurationVersion is a version field. Incompatible changes to a record can be indicated by a change in the version number.

[0880] lengthSizeMinusOne plus 1 indicates the length (in bytes) of the NALUnitLength field in the V-PCC samples in the stream to which this configuration record applies.

[0881] For example, the size of one byte may be indicated by a value of 0. The value of this field may be equal to ssnh_unit_size_precision_bytes_minus1 in sample_stream_nal_header() of the atlas substream.

[0882] numOfVPCCParameterSets specifies the number of V-PCC parameter set elements signaled in the decoder configuration record.

[0883] VPCCParameterSetLength indicates the size (in bytes) of the vpccParameterSet field.

[0884] vpccParameterSet is a V-PCC element of type VPCC_VPS that carries vpcc_parameter_set().

[0885] numOfSetupUnitArrays indicates the number of arrays of atlas NAL units of the indicated type.

[0886] array_completeness equal to 1 indicates that all atlas NAL units of the given type are in the following array, and none are in the stream. array_completeness equal to 0 indicates that additional atlas NAL units of the indicated type may be in the stream. The default and allowed values ​​are constrained by the sample entry name.

[0887] NAL_unit_type indicates the type of the atlas NAL unit in the following array. It can be one of the values ​​indicating NAL_ASPS, NAL_PREFIX_SEI, or NAL_SUFFIX_SEI atlas NAL units.

[0888] numNALUnits indicates the number of atlas NAL units of the indicated type included in the configuration record of the stream to which this configuration record applies. The SEI array may contain only SEI messages.

[0889] SetupUnitLength indicates the size of the setupUnit field in bytes.The length field may include the size of both the NAL unit header and the NAL unit payload, but not the length field itself.

[0890] A setupUnit may contain a NAL unit of type NAL_ASPS, NAL_AFPS, NAL_PREFIX_ESEI, NAL_PREFIX_NSEI, NAL_SUFFIX_ESEI, or NAL_SUFFIX_NSEI.

[0891] When not present in setupUnit, units of type NAL_ASPS (atlas sequence parameter set), NAL_AFPS (atlas frame parameter set), NAL_AAPS (atlas adaptation parameters), NAL_PREFIX_ESEI, NAL_PREFIX_NSEI, NAL_SUFFIX_ESEI, or NAL_SUFFIX_NSEI may be included. NAL_PREFIX_ESEI (required supplemental enhancement information, sei_rbsp()), NAL_PREFIX_NSEI (non-required supplemental enhancement information, sei_rbsp()), NAL_SUFFIX_ESEI (required supplemental enhancement information, sei_rbsp()), or NAL_SUFFIX_NSEI (non-required supplemental enhancement information, sei_rbsp()) may contain SEI messages. SEI messages may provide information about the stream as a whole. An example of such an SEI may be a user data SEI.

[0892] Sample Group

[0893] The method / apparatus according to the embodiment may transmit point cloud data and related data in a sample group. For example, atlas-related parameter information according to the embodiment may be grouped and transmitted.

[0894] Track Grouping

[0895] The method / apparatus according to the embodiment can efficiently transmit point cloud data and related data by grouping related data based on track grouping. For example, to support partial access to a spatial region of point cloud data, tracks carrying the spatial region of point cloud data can be grouped and identified.

[0896] Entity Grouping

[0897] The method / apparatus according to the embodiment may efficiently transmit point cloud data and related data by grouping related data based on entity grouping.

[0898] Multi-track container for V-PCC bitstreams

[0899] Multi-track container for V-PCC bitstreams

[0900] In the general layout of a multi-track ISOBMFF V-PCC container, V-PCC units in a V-PCC bitstream can be mapped to individual tracks within the container file based on their type. Figure 40 and Figure 41 Configuration shown. There can be two types of tracks in a multi-track ISOBMFF V-PCC container: V-PCC tracks and V-PCC assembly tracks.

[0901] The V-PCC component track is a video scheme track that carries the 2D video coding data of the occupancy map, geometry, and attribute sub-bitstreams of the V-PCC bitstream. In addition, the V-PCC component track meets the following conditions:

[0902] a) In the sample entry, a new box may be inserted in the V-PCC system, which records the role of the video stream;

[0903] b) a track reference may be introduced from a V-PCC track to a V-PCC component track to establish a membership relationship of a V-PCC component track in a specific point cloud represented by the V-PCC track;

[0904] c) The track header flag may be set to 0 to indicate that the track does not contribute directly to the overall lay-out of the movie, but rather contributes to the V-PCC system.

[0905] Tracks belonging to the same V-PCC sequence are time aligned. Samples contributing to the same point cloud frame across different video coded V-PCC component tracks and V-PCC tracks have the same presentation time. The decoding time of the V-PCC atlas sequence parameter set and atlas frame parameter set for these samples is equal to or earlier than the editing time of the point cloud frame. In addition, all tracks belonging to the same V-PCC sequence have the same implicit or explicit edit lists. Synchronization between elementary streams in component tracks is handled by the ISOBMGG track timing structure (stts, ctts and cslg) or equivalent mechanisms in movie fragments.

[0906] The synchronization samples in the V-PCC track and the V-PCC component track may be time aligned or time misaligned. In the absence of time alignment, random access may involve pre-rolling the various tracks from different synchronization start times to allow starting at the desired time.

[0907] In the case of time alignment (e.g., as required by a V-PCC profile such as the Basic Toolset Profile defined in V-PCC), the synchronization samples of a V-PCC track may be considered as random access points to the V-PCC content, and random access may be performed by referring only to the synchronization sample information of the V-PCC track.

[0908] Based on this layout, a V-PCC ISOBMFF container can contain the following:

[0909] The container file may include one or more V-PCC tracks containing V-PCC parameter sets, atlas sub-bitstream parameter sets (in sample entries), and samples carrying atlas sub-bitstream NAL units. The track also includes track references to other tracks that carry payloads of video compression V-PCC units (i.e., unit types VPCC_OVD, VPCC_GVD, and VPCC_AVD).

[0910] The container file may contain a video scheme track, where the samples contain access units of a video coded elementary stream (ie payloads of V-PCC units of type VPCC_OVD) of occupancy map data.

[0911] A container file may contain one or more video scheme tracks, where samples contain access units of a video coded elementary stream of geometry data (ie payloads of V-PCC units of type VPCC_GVD).

[0912] A container file may contain zero or more video scheme tracks, where samples contain access units of a video coded elementary stream of attribute data (ie payloads of V-PCC units of type VPCC_AVD).

[0913] V-PCC track

[0914] According to the V-PCC system documents or projects ( Figures 45 to 53 etc.) can include the following information in the track.

[0915] V-PCC Track Sample Entry

[0916] Sample entry types: "vpc1", "vpcg"

[0917] Container: SampleDescriptionBox

[0918] Mandatory: "vpc1" or "vpcg" sample entries are mandatory

[0919] Quantity: There can be one or more sample entries

[0920] A track according to an embodiment may include sample entry types called "vpc1," "vpcg," and "vpga."

[0921] A V-PCC track may use VPCCSampleEntry that extends VolumetricVisualSampleEntry with a sample entry type of "vpc1" or "vpcg".

[0922] A VPCC track sample entry may contain a VPCCConfigurationBox.

[0923] Under the "vpc1" sample entry, all atlas sequence parameter sets, atlas frame parameter sets, or V-PCC SEI may be present in the setupUnit array.

[0924] Under the "vpcg" sample entry, atlas sequence parameter set, atlas frame parameter set, V-PCC SEI may be present in the array or in the stream.

[0925] An optional BitRateBox may be present in the VPCC volume sample entry to signal the bitrate information of the V-PCC track.

[0926]

[0927] V-PCC track sample format

[0928] Each sample in a V-PCC track may correspond to a single coded atlas access unit.

[0929] The samples in the various component tracks corresponding to the frame may have the same arrangement timing as the V-PCC track samples.

[0930] Each V-PCC sample may contain one V-PCC unit payload of type VPCC_AD, which may include one or more map set NAL units.

[0931]

[0932] A nalUnit may contain a single atlas NAL unit in the NAL unit sample stream format.

[0933] Video Coding V-PCC component track

[0934] Since it does not make sense to display frames decoded from attribute, geometry or occupancy map tracks without reconstructing the point cloud on the player side, a restricted video scheme type may be defined for these video coding tracks.

[0935] A V-PCC component video track may be represented in a file as restricted video and may be identified by "pccv" in the scheme_type field of the SchemeTypeBox of the RestrictedSchemeInfoBox of its restricted video sample entry.

[0936] It should be noted that there is no restriction on the video codec used to encode the attribute, geometry and occupancy map V-PCC components. These components can be encoded using different video codecs.

[0937] Program Information

[0938] The SchemeInformationBox may exist and contain the VPCCUnitHeaderBox.

[0939] Reference V-PCC Component Track

[0940] To link a V-PCC track to a component video track, add three TrackReferenceTypeBoxes to the TrackReferenceBox within the V-PCC track's TrackBox, one for each component. The TrackReferenceTypeBox contains an array of track_IDs specifying the video tracks that the V-PCC track references. The reference_type of the TrackReferenceTypeBox identifies the type of component (i.e., occupancy map, geometry, or attributes or occupancy map). The 4CCs for these track reference types can be as follows:

[0941] "pcco": The reference track contains the Video Coding Occupancy Map V-PCC component.

[0942] "pccg": The reference track contains the Video Coding Geometry V-PCC component.

[0943] "pcca": The reference track contains the video coding attributes V-PCC component.

[0944] The type of the V-PCC component carried by the reference restricted video track and signaled in the track's RestrictedSchemeInfoBox may match the reference type of the track reference from the V-PCC track.

[0945] Single track container for V-PCC bitstream

[0946] Single track encapsulation of V-PCC data requires that a V-PCC coded elementary bit stream be represented by a single track.

[0947] Single-track encapsulation of PCC data can be utilized with simple ISOBMFF encapsulation of a V-PCC coded bitstream. This bitstream can be stored directly as a single track without further processing. The V-PCC unit header data structure can be preserved in the bitstream. The single-track container can be provided to the media workflow for further processing (e.g., multi-track file generation, transcoding, DASH segmentation, etc.).

[0948] V-PCC bitstream track

[0949] Sample entry types: "vpe1", "vpeg"

[0950] Container: SampleDescriptionBox

[0951] Mandatory: "vpe1" or "vpeg" sample entries are mandatory

[0952] Quantity: There can be one or more sample entries

[0953] V-PCC bitstream tracks use VolumetricVisualSampleEntry with sample entry type "vpe1" or "vpeg".

[0954] The VPCC bitstream sample entry contains a VPCCConfigurationBox.

[0955] Under the "vpe1" sample entry, all atlas sequence parameter sets, atlas frame parameter sets, SEIs may be present in the setupUnit array.

[0956] Under the "vpeg" sample entry, atlas sequence parameter set, atlas frame parameter set, SEI may be present in the array or in the stream.

[0957]

[0958] V-PCC bitstream sample format

[0959] A V-PCC bitstream sample contains one or more V-PCC units belonging to the same presentation time, i.e., one V-PCC access unit. A sample can be self-contained (like a synchronization sample) or decoded from other samples of a V-PCC bitstream track.

[0960] V-PCC bitstream synchronization samples

[0961] The V-PCC bitstream synchronization samples meet all the following conditions:

[0962] Can be decoded independently.

[0963] The decoding of samples following the sync sample in decoding order does not depend on any samples preceding the sync sample.

[0964] All samples following the sync sample in decoding order can be decoded successfully.

[0965] V-PCC bitstream subsample

[0966] A V-PCC bitstream subsample is a V-PCC unit contained in a V-PCC bitstream sample.

[0967] The V-PCC bitstream track should contain a SubSampleInformationBox in the SampleTableBox or in the TrackFragmentBox of each MovieFragmentBox, which lists the V-PCC bitstream subsamples. The 32-bit unit header of the V-PCC unit representing the subsample can be copied to the 32-bit codec_specific_parameters field of the subsample entry in the SubSampleInformationBox. The V-PCC unit type of each subsample can be identified by parsing the codec_specific_parameters field of the subsample entry in the SubSampleInformationBox.

[0968] Timed metadata track

[0969] The method / apparatus according to an embodiment may send metadata related to time-varying point cloud data in a timed metadata track of a container (file, track). The timed metadata track may include sample entries containing entry information and samples containing specific information.

[0970] Figure 47 An overview of the structure used to encapsulate non-timed V-PCC data according to an embodiment is shown.

[0971] Figure 47 show when Figures 20 to 22 A structure for sending non-scheduled V-PCC data when a point cloud data-related device processes non-scheduled point cloud data.

[0972] According to the embodiment, the point cloud data transmitting / receiving method / device and the system included in the transmitting / receiving device can be Figure 47 As shown, non-timed V-PCC data is encapsulated to transmit and receive the non-timed V-PCC data.

[0973] When the point cloud data according to the embodiment is an image, Figure 1 Point cloud video encoder 10002 (or Figure 4 The encoder, Figure 15 The encoder, Figure 18 The sending device, Figure 20 Processor 20001, Figure 20 Encoder 20003, Figure 21 processor, Figure 21 The image encoder 21008) can encode the image, and the file / segment encapsulator 10003 (or Figure 20 File / fragment encapsulator 20004, Figure 21 The file / segment encapsulator 21009 can store images and image related information in Figure 47 The sender 10004 can send the container.

[0974] Similarly, Figure 1 The receiver receives Figure 47 Container, and file / fragment decapsulator 10007 (or Figure 20 The file / fragment decapsulator 20005 and the file / fragment decapsulator 22000) parse the container. Figure 1 Point cloud video decoder 10008 (or Figure 16 decoder, Figure 17 decoder, Figure 19 The receiving device, image decoder 20006 or image decoder 22002) can decode the image contained in the project and provide the decoded image to the user.

[0975] The image according to the embodiment may be a still image. The method / apparatus according to the embodiment may send / receive point cloud data about the image. The method / apparatus according to the embodiment may be based on Figure 47 The data container structure shown stores the image in the project and sends / receives it. In addition, attribute information about the image can be stored in the image properties, etc.

[0976] The non-timed V-PCC data is stored in the file as a picture item. Two new item types (ie, V-PCC item and V-PCC unit item) are defined for encapsulating the non-timed V-PCC data.

[0977] A new handler type 4 CC code "vpcc" is defined and stored in the HandlerBox of the MetaBox to indicate the presence of V-PCC items, V-PCC unit items and other V-PCC encoded content representation information.

[0978] V-PCC Project

[0979] V-PCC Item: A V-PCC item represents an independently decodable V-PCC access unit. The item type "vpci" is defined to identify a V-PCC item. A V-PCC item stores the V-PCC unit payload of an atlas sub-bitstream. When the PrimaryItemBox exists, the item_id in this box is set to indicate the V-PCC item.

[0980] According to an embodiment, a project type 4CC code "v3ci" may be defined to identify a V3C project. A V3C project may store an atlas sub-bitstream according to an embodiment ( Figure 30 etc.) of the V3C unit payload.

[0981] Additionally, V3C projects with a new project type "v3cb" can be added to Figure 47 to support multiple atlases and can be constructed together with V3C configuration properties that may include atlas parameter set data Figure 47 structure.

[0982] V3C Block Project

[0983] When the V3C atlas data includes a plurality of atlas tiles, the V3C tile item may be an item for encapsulating the atlas tile data.

[0984] According to an embodiment, an item type 4CC code "v3at" may be defined to identify a V3C patch item. A V3C patch item may store a portion of a V3C unit payload of an atlas sub-bitstream. A V3C patch item may be associated with corresponding V3C patch item properties.

[0985] According to embodiments, multiple atlases refer to situations where multiple atlases exist. For example, point cloud data may be moving. Alternatively, multiple atlases may mean that the camera acquiring the point cloud data is not static but dynamic. Furthermore, when the viewport of the camera acquiring the point cloud data moves, as in 3DOF+, multiple atlases may be required to represent the data corresponding to the direction of movement. When a user moves their head to a specific position, the point cloud data may need to change, and therefore multiple atlases may be required for the point cloud data.

[0986] V3C Unit Project

[0987] according to Figure 47 The v3c item of the embodiment is an item representing V3C unit data. The V3C unit item can store the V3C unit payload of the occupancy, geometry, and attribute video data unit. The V3C unit item can only store data related to one V3C access unit.

[0988] The item type 4CC code of the V3C unit item may be set according to the codec used to encode the corresponding video data unit. The V3C unit item may be associated with corresponding V3C unit header item properties and codec specific configuration item properties.

[0989] A V3C unit item can be marked as a hidden item when displaying it independently does not make sense. To indicate the relationship between a V3C item and a V3C unit, three item reference times (4CC codes "v3vo," "v3vg," and "v3va") can be defined. An item reference can be defined from a V3C item to a related V3C unit item. The following 4CC codes for item reference types are described.

[0990] "v3vo": Reference V3C unit item contains occupied video data unit.

[0991] A reference V3C unit entry may contain occupied video data units.

[0992] "v3vg": Reference V3C unit item containing geometry video data unit.

[0993] A reference V3C unit item may contain a geometry video data unit.

[0994] "v3va": Reference V3C unit item contains attribute video data unit.

[0995] A reference V3C unit item may contain attribute video data units.

[0996] V3c Configuration Project Nature

[0997] Frame type: "v3cp"

[0998] Property type: Descriptive item properties

[0999] Container: ItemPropertyContainerBox

[1000] Mandatory (per project): For V3C projects of type "v3ci", yes

[1001] Quantity (per item): For V3C items of type "v3ci", one or more

[1002] A V3C parameter set may be stored as a descriptive item property and may be associated with a V3C item. Depending on the embodiment, only one V3C parameter set may be allowed to be stored in a V3C configuration item property.

[1003] A property according to an embodiment may be defined as a container of information including additional description of the corresponding item.

[1004] For the V3CP item property, Essential may be set to 1. A V-PCC parameter set may be stored as a descriptive item property and may be associated with a V-PCC item.

[1005]

[1006] v3c_unit_payload_size specifies the size (in bytes) of v3c_unit_payload().

[1007] v3c_unit_payload() may be a V3C unit of type V3C_VPS and may conform to the description in ISO / IEC 23090-5 [V3C].

[1008] To support multiple atlases, V3C_AD can be added to the V3C unit type of v3c_unit_paylad(), which is intended to include atlas parameter set information and / or SEI information. v3c_unit_payload() can include atlas parameter data according to the NAL unit type of the atlas data. That is, a data structure can be added to v3c_unit_payload_struct() as follows.

[1009]

[1010]

[1011] v3c_unit_type can indicate V3C_AD or V3C_VPS depending on the usage.

[1012] When v3c_unit_type is V3C_AD, num_of_setup_units may indicate the number of included atlas parameter data.

[1013] NAL_unit_type may indicate the type of non-ACL NAL unit of the atlas NAL unit.

[1014] num_of_v3c_unit_items may indicate the number of V3C unit items associated with the atlas data.

[1015] entry_v3c_unit_item is a flag that can have a value of true (1) or false (0). When set to true (1), it can indicate that the item is an entry V3C unit item. One or more V3C unit items can have a value of true (1).

[1016] v3c_unit_item_type may indicate the type of the V3C unit item, ie, occupancy (0), geometry (1), or attribute (2).

[1017] reference_v3c_unit_item_id may indicate the id value of the V3C unit item.

[1018] v3c_unit_payload_size specifies the size (in bytes) of v3c_unit_payload().

[1019] v3c_unit_payload() may be a V3C unit of type V3C_VPS as defined in ISO / IEC 23090-5 [V3C].

[1020] V3C unit head project nature

[1021] Frame type: "vunt"

[1022] Property type: Descriptive item properties

[1023] Container: ItemPropertyContainerBox

[1024] Mandatory (per project): For V3C projects of type "v3ci" and for V3C unit projects,

[1025] Quantity (per item): one

[1026] The V3C unit header may be stored as a descriptive item property and may be associated with a V3C item and a V3C unit item.

[1027] For the "vunt" item property, essential can be set to 1.

[1028]

[1029] V3C Block Project Nature

[1030] Frame type: "v3tp"

[1031] Property type: Descriptive item properties

[1032] Container: ItemPropertyContainerBox

[1033] Mandatory (per item): For V3C tile items of type "v3at",

[1034] Quantity (per item): For V3C tile items of type "v3at", one or more

[1035] V3C atlas tile information may be stored as descriptive item properties and may be associated with a V3C tile item of type "v3at".

[1036] For the "v3tp" item property, essential can be set to 1.

[1037]

[1038] num_atlas_tiles indicates the number of tiles contained in the associated V3C project.

[1039] atlas_tile_id indicates the tile address of a tile existing in the relevant V3C item.

[1040] V3C tile area information item properties

[1041] Frame type: "v3rp"

[1042] Property type: Descriptive item properties

[1043] Container: ItemPropertyContainerBox

[1044] Mandatory (per item): For V3C tile items of type "v3at",

[1045] Quantity (per item): For V3C tile items of type "v3at", one or more

[1046] The V3C atlas tile region information may be stored as a descriptive item property and may be associated with a V3C tile item of type “v3at.” The V3C atlas tile region information may be used by a receiver to support partial access to V3C content.

[1047] For the "v3tp" item property, essential can be set to 1.

[1048]

[1049]

[1050] num_regions indicates the number of 3D spatial regions in the volumetric media.

[1051] num_atlas_tiles indicates the number of atlas tiles associated with the 3D spatial region.

[1052] atlas_tile_id is the identifier of the atlas tile associated with the 3D space region.

[1053] num_objects indicates the total number of objects associated with the 3D space region.

[1054] obj_idx_length is the length (in bytes) of the object index.

[1055] soi_object_idx indicates the value of each object index defined by the scene object information SEI message.

[1056] 3d_region_id is an identifier of a 3D spatial region.

[1057] x, y, and z specify the x, y, and z coordinates of the 3D point in the Cartesian coordinate system.

[1058] cuboid_x, cuboid_y, and cuboid_dz ​​indicate the dimensions of the cube subregion in Cartesian coordinates relative to the anchor point along the x, y, and z axes, respectively.

[1059] An anchor is a 3D point in a Cartesian coordinate system that serves as an anchor for a region of 3D space.

[1060] dimensions_included_flag is a flag indicating whether the size of the 3D space region is signaled.

[1061] In the point cloud data transmitting method / apparatus according to the embodiment, the point cloud data may be encoded by an encoder according to the embodiment, etc. Figure 1 File / fragment encapsulator 10003, Figure 20 File / fragment encapsulator 20004, Figure 21 File / fragment encapsulator 21009, Figure 23 The XR device 1230 can encapsulate the point cloud data as follows Figures 45 to 53 The data structure shown and send the data.

[1062] In the point cloud data receiving method / device according to the embodiment, point cloud data can be received. Figures 45 to 53 The point cloud data in Figure 1 File / fragment decapsulator 10007, Figure 20 File / fragment decapsulator 20005, Figure 21 File / fragment decapsulator 22000, Figure 23 The XR device 1230 or the like decapsulates the data and can be decoded by a decoder according to an embodiment.

[1063] Hereinafter, a V3C item structure according to an embodiment will be described.

[1064] Figure 48 The structure of a V3C project according to an embodiment is shown.

[1065] Figure 48 Shown is a format for carrying a single atlas with multiple atlas tiles.

[1066] The method / apparatus according to an embodiment may add a V3C tile item containing information related to an atlas tile according to an embodiment to a file format.

[1067] Figure 48 A V3C project can use the reference type "v3ct" to reference a related V3C tile project from a V3C project.

[1068] The method / device for receiving point cloud data according to the embodiment may receive the following: Figure 48 The project shown decodes point cloud data (geometry, attributes, occupancy and its parameter information) and patch projects (geometry, attributes, occupancy and its parameter information) to efficiently render multiple point cloud data.

[1069] Figure 49 A V3C project according to an embodiment is shown.

[1070] Figure 49 Shown in Figure 48 An example of a relationship is provided in the illustrated V3C project structure. The point cloud data method / apparatus according to an embodiment may use a reference type "vbas" to indicate a relationship from a V3C tile project to a V3C project.

[1071] Figures 50 to 51 A V3C project according to an embodiment is shown.

[1072] The method / apparatus according to an embodiment may provide a plurality of atlases having a plurality of atlas tiles, such as Figure 50 shown.

[1073] Multiple atlases with multiple atlas tiles

[1074] Reference Figure 50 , atlas data can be respectively packaged into multiple V3C items, and a V3C item of item type "v3cb" with common atlas parameter set information of the V3C item can be generated, and a V3C item of item type "v3ci" can be referenced using a reference type "v3cm". Figure 49 The relationship between the V3C project and the V3C tile project is established as shown.

[1075] Figures 52 to 53 A V3C project according to an embodiment is shown.

[1076] The method / apparatus according to the embodiment may use the reference type “vbas” to indicate a relationship from a V3C tile item to a V3C item, and may use the reference type “v3bs” to indicate a relationship between V3C items corresponding to a plurality of atlas data.

[1077] according to Figures 48 to 53 , the point cloud data receiving method / device of the embodiment can decode multiple atlas blocks based on a single atlas and / or multiple atlases.

[1078] Will refer to Figures 48 to 53 Describe the details.

[1079] According to an embodiment, the V-PCC bitstream or point cloud data may contain only one atlas with multiple atlas tiles.

[1080] According to an embodiment, an atlas patch may include atlas-related data about the patch based on the patch ID, for example, it may include information about patch mode, patch data, etc.

[1081] According to an embodiment, a patch may refer to a region of an atlas frame, and an encoding / decoding unit may be configured for each patch.

[1082] An atlas frame according to an embodiment may be a collection of tiles.

[1083] According to an embodiment, an atlas bitstream corresponding to a patch may be carried in an atlas patch track.

[1084] According to an embodiment, the non-timed VC3 content may be stored in a file. The non-timed V3C content may be packaged into a V3C atlas item, a V3C atlas tile item, a V3C component item, etc.

[1085] The method / apparatus according to an embodiment may encapsulate non-timed V3C data in a single atlas with a single atlas tile.

[1086] The method / apparatus according to an embodiment may encapsulate non-timed V3C data into a single atlas with multiple atlas tiles (see Figure 48 and Figure 49 wait).

[1087] The method / apparatus according to the embodiment may encapsulate the non-timed V3C data in multiple atlases with multiple atlas tiles (see Figure 50 and Figure 51 wait).

[1088] Reference Figure 48 , non-timing data can consist of a single atlas. A single atlas can have multiple atlas tiles, and information about multiple atlas tiles can be sent based on V3C Tile Items. Tile Items for a single atlas tile can be referenced by items that carry component data (e.g., geometry, attributes, occupancy) and reference information, and can contain property information about tile areas, tile properties, etc. Component items can also contain configuration item properties that describe information about the component.

[1089] The point cloud data receiving device according to the embodiment can analyze Figure 48 The structure shown is used to reconstruct point cloud data through an atlas of multiple tiles.

[1090] Figure 49 It shows how a tile item according to an embodiment refers to a project according to an embodiment. A project according to an embodiment indicates a project structure including common information of a plurality of tile items.

[1091] The point cloud data receiving apparatus according to the embodiment may acquire related items based on reference information when receiving a tile item.

[1092] Reference Figure 50 and Figure 51 The point cloud data transmitting device according to the embodiment can transmit multiple atlases. The point cloud data receiving device according to the embodiment can decode the multiple atlases to access and provide various point cloud data.

[1093] Items according to embodiments may carry non-timed point cloud data.Items may include properties indicating public information.

[1094] A project can reference multiple projects based on reference information.

[1095] The project may include properties containing public information and may reference tile projects related to the atlas tile based on reference information. It may reference information about the atlas tile and component (geometry, attributes, occupancy) information about the atlas tile based on the reference information.

[1096] Tile items according to embodiments may be connected to items based on reference information, and items may be connected to common items based on reference information.

[1097] The point cloud data receiving apparatus according to the embodiment may acquire a plurality of atlases having a plurality of atlas tiles, access a desired atlas based on reference information, and reconstruct point cloud data.

[1098] Values ​​indicating reference information, item types, etc. according to the embodiment are not limited to the values ​​shown in the drawings, and various values ​​may be used to indicate a target object.

[1099] A flowchart representing a file format transmission signaling operation of the file / segment encapsulator 10003, 20004, or 21009 according to an embodiment will be described. The file / segment encapsulator may create a file format to transmit atlas data as follows.

[1100] 1) The file encapsulator receives the V3C bitstream generated by the point cloud encoder according to an embodiment ( Figure 24 、 Figure 25 and Figure 30 ).

[1101] 2) Can parse the atlas information contained in the V3C bitstream ( Figure 30 and Figures 33 to 44 ).

[1102] 3) The number of atlases, atlas ID, number of atlas tiles and atlas tile ID information can be identified.

[1103] 4) V3C tile items and V3C items that can be configured to suit the item types of each atlas data and atlas tiles ( Figure 47 ).

[1104] 5) By parsing the atlas parameter set and V3C unit header from the input bitstream, the V3C configuration item properties and V3C unit header item properties can be configured respectively ( Figures 48 to 53 ).

[1105] 6) By configuring reference information between each V3C project and V3C tile project, you can generate package output files in the form of .heif files ( Figures 48 to 53 ).

[1106] The method / apparatus according to an embodiment may transmit an encapsulated file.

[1107] A flowchart representing a file format reception signaling operation of the file / segment decapsulator 10007, 20005, or 22000 according to an embodiment will be described. The file / segment decapsulator may process the file format to receive atlas data as follows.

[1108] 1) The file decapsulator receives the encapsulated file in the form of a .heif file encapsulated in the V3C file format ( Figures 45 to 47 ).

[1109] 2) The receiving device can parse V3C items and / or V3C tile items according to each item type ( Figure 47 ).

[1110] 3) When a V3C project with project type "v3cb" exists, the receiver can determine that the project contains multiple atlas data ( Figure 50 and Figure 51 ).

[1111] 4) When there is no V3C item of type "v3cb", the receiver can determine that a single atlas data is contained ( Figure 48 and Figure 49 ).

[1112] 5) V3C configuration item properties and / or V3C unit header item properties included in each V3C item can be parsed.

[1113] 6) V3C parameter set information and V3C unit header information may be obtained as a result of parsing in step 5) above.

[1114] 7) It can be determined whether the V3C tile items referenced by each V3C item exist.

[1115] Alternatively, a V3C item having an item_id associated with a reference type of "vbas" may be identified from the V3C tile item.

[1116] 8) When there are V3C tile items referenced by each V3C item, information such as the number of atlas tiles and tile IDs can be obtained based on the included V3C tile property information.

[1117] 9) When a plurality of atlas data (ie, a plurality of V3C items) are included, it is possible to identify whether there is a V3C item referenced by each V3C item.

[1118] Alternatively, a V3C item having an item_id associated with a reference type of "v3cm" may be determined from the V3C items.

[1119] 10) Through the above operations, V3C configuration information, V3C parameter set information and V3C unit header information used to configure the non-scheduled V3C bitstream can be obtained.

[1120] Figure 54 A method of transmitting point cloud data according to an embodiment is shown.

[1121] 5200: According to an embodiment, the method for transmitting point cloud data may include encoding the point cloud data. According to an embodiment, encoding may include Figure 1 Point cloud video encoder 10002, Figure 4 The encoding, Figure 15 The encoder, Figure 18 The sending device, Figure 20 Video / image encoders 20002 and 20003, Figure 21 The operation of the video / image encoders 21007 and 21008, etc.

[1122] 5201: The method for transmitting point cloud data according to an embodiment may further include encapsulating a bit stream containing point cloud data based on a file. According to an embodiment, encapsulation may include Figure 1 File / fragment encapsulator 10003, Figure 20 File / fragment encapsulator 20004, Figure 21 Operations of the file / fragment encapsulator 21009, etc.

[1123] 5202: According to an embodiment, the method of sending point cloud data may further include sending point cloud data. According to an embodiment, sending may include Figure 1 The operation of transmitter 10004, etc.

[1124] Figure 55 A method of receiving point cloud data according to an embodiment is shown.

[1125] 5300: According to an embodiment, a method of receiving point cloud data may include receiving a bit stream containing point cloud data. According to an embodiment, receiving may include Figure 1 The operation of the receiver 10006 and the like.

[1126] 5301: According to an embodiment, the method for receiving point cloud data may further include decapsulating the point cloud data based on the file. According to an embodiment, decapsulation may include Figure 1 File / fragment decapsulator 10007, Figure 20 File / fragment decapsulator 20005, Figure 22 The operation of the file / fragment decapsulator 22000 and the like.

[1127] 5302: According to an embodiment, the method for receiving point cloud data may further include decoding the point cloud data. According to an embodiment, decoding may include Figure 1 Point cloud video decoder 10008, Figure 16 Decoding, Figure 17 decoder, Figure 19 receiving device, Figure 20 Video / image decoder 20006 and Figure 22 The operation of the video / image decoders 22001 and 22002, etc.

[1128] A method of transmitting point cloud data according to an embodiment may include: encoding the point cloud data; packaging the point cloud data based on a file; and transmitting the point cloud data.

[1129] The point cloud data may include non-timed point cloud data. When the non-timed point cloud data includes multiple atlases, the multiple atlases may be packaged based on one or more atlas items in the file.

[1130] According to an embodiment, the file may further include an atlas tile item. The atlas item and the atlas tile item may be referenced based on the reference information.

[1131] According to an embodiment, the file may further include tile property information. The tile property information may include information indicating the number of tiles in the atlas tile item and information indicating an identifier of the tile.

[1132] When the non-timed point cloud data includes a single atlas, the single atlas may be packaged based on one or more atlas items in the file.

[1133] According to an embodiment, an apparatus for receiving point cloud data may include: a receiver configured to receive point cloud data; a decapsulator configured to decapsulate the point cloud data based on a file; and a decoder configured to decode the point cloud data.

[1134] When the untimed point cloud data includes a single atlas, the single atlas may be unpacked based on one or more atlas items in the file.

[1135] When point cloud data (e.g., image type data), i.e., non-timed V3C data (static images) consists of multiple atlases and / or multiple atlas tiles, the method / apparatus according to the embodiment can effectively support partial access to V3C image data and other operations by defining an effective file format structure and file format signaling for file encapsulation.

[1136] In the method / apparatus according to the embodiment, a transmitter or a receiver configured to provide a point cloud content service may configure a V-PCC bitstream and store the file as described above.

[1137] The methods / apparatuses according to the embodiments can efficiently multiplex VV-PCC bitstreams and support efficient access to bitstreams based on V-PCC units. These methods / apparatuses can also allow the atlas stream of the V-PCC bitstream to be efficiently stored on tracks in a file and transmitted.

[1138] Therefore, according to the embodiment, the transmitting device (or file encapsulator) can generate and transmit non-timed V3C data containing multiple atlases and / or multiple atlas tiles in an efficient file format. Therefore, according to the embodiment, the receiving device (or file decapsulator) can provide V3C image data to the user based on partial access.

[1139] In other words, the above data representation scheme can allow efficient access to the point cloud bitstream. In addition, the information required for data processing and rendering of the point cloud bitstream can be efficiently accessed.

[1140] The transmitting device according to the embodiment may divide and store the V-PCC bitstream into one or more tracks in a file and may generate relevant parameter information as signaling information in the file. This device may generate signaling information indicating the relationship between multiple tracks including the V-PCC bitstream stored as parameter information in a container structure such as a file. Files in point cloud bitstreams can be efficiently stored and transmitted by indicating alternative V-PCC tracks stored in the file.

[1141] The embodiments have been described in terms of methods and / or devices. The description of the method and the description of the device may complement each other.

[1142] Although the embodiments have been described with reference to the various drawings for simplicity, new embodiments may be designed by combining the embodiments shown in the drawings. If a person skilled in the art designs a computer-readable recording medium having recorded thereon a program for executing the embodiments mentioned in the above description, it may fall within the scope of the appended claims and their equivalents. The apparatus and method may not be limited to the configuration and method of the above-described embodiments. The above-described embodiments may be configured by selectively combining them in whole or in part with each other to achieve various modifications. Although preferred embodiments have been described with reference to the drawings, it will be understood by those skilled in the art that various modifications and changes may be made to the embodiments without departing from the spirit or scope of the present disclosure described in the appended claims. These modifications should not be understood solely from the technical ideas or viewpoints of the embodiments.

[1143] The various elements of the device of the embodiment can be implemented by hardware, software, firmware or a combination thereof. The various elements in the embodiment can be implemented by a single chip (e.g., a single hardware circuit). According to the embodiment, the components according to the embodiment can be implemented as separate chips respectively. According to the embodiment, at least one or more components of the device according to the embodiment may include one or more processors capable of executing one or more programs. One or more programs can execute any one or more operations / methods according to the embodiment, or include instructions for executing them. The executable instructions for executing the method / operation of the device according to the embodiment can be stored in a non-transitory CRM or other computer program product configured to be executed by one or more processors, or can be stored in a temporary CRM or other computer program product configured to be executed by one or more processors. In addition, the memory according to the embodiment can be used as not only covering volatile memory (e.g., RAM), but also covering the concepts of non-volatile memory, flash memory and PROM. In addition, it can also be implemented in the form of a carrier wave (e.g., transmission via the Internet). In addition, the processor-readable recording medium can be distributed to computer systems connected via a network so that the processor-readable code can be stored and executed in a distributed manner.

[1144] In this document, the terms " / " and "," should be interpreted as indicating "and / or". For example, the expression "A / B" may mean "A and / or B". In addition, "A, B" may mean "A and / or B". In addition, "A / B / C" may mean "at least one of A, B and / or C". In addition, "A, B, C" may also mean "at least one of A, B and / or C". In addition, in this document, the term "or" should be interpreted as "and / or". For example, the expression "A or B" may mean 1) only A, 2) only B and / or 3) both A and B. In other words, in this document, the term "or" should be interpreted as "in addition or alternatively".

[1145] Terms such as first and second may be used to describe various elements of an embodiment. However, various components according to an embodiment should not be limited by the above terms. These terms are only used to distinguish one element from another. For example, a first user input signal may be referred to as a second user input signal. Similarly, a second user input signal may be referred to as a first user input signal. The use of these terms should be interpreted as not departing from the scope of various embodiments. Both the first user input signal and the second user input signal are user input signals, but unless otherwise clearly provided above, they do not mean the same user input signals.

[1146] The terms used to describe the embodiments are only for the purpose of describing specific embodiments and are not intended to limit the embodiments. As used in the description of the embodiments and in the claims, unless clearly otherwise provided above, the singular form includes plural referents. The expression "and / or" is used to include all possible combinations of terms. Terms such as "including" or "having" are intended to indicate the presence of graphics, quantities, steps, elements and / or components and should be understood as not excluding the possibility of the presence of additional graphics, quantities, steps, elements and / or components. As used herein, conditional expressions such as "if" and "when" are not limited to optional cases and are intended to be interpreted as performing the relevant operation when a specific condition is met or interpreting the relevant definition based on the specific condition.

[1147] The operations according to the embodiments described in this specification may be performed by a transmitting / receiving device including a memory and / or a processor according to the embodiments. The memory may store a program for processing / controlling the operations according to the embodiments, and the processor may control the various operations described in this specification. The processor may be referred to as a controller, etc. In the embodiments, the operations may be performed by firmware, software, and / or a combination thereof. The firmware, software, and / or a combination thereof may be stored in the processor or the memory.

[1148] The operations according to the above-described embodiments may be performed by a transmitting device and / or a receiving device according to the embodiments. The transmitting / receiving device may include: a transmitter / receiver configured to transmit and receive media data; a memory configured to store instructions (program code, algorithm, flow chart, and / or data) for processing according to the embodiments; and a processor configured to control the operations of the transmitting / receiving device.

[1149] The processor may be referred to as a controller, etc., and may correspond to, for example, hardware, software, and / or a combination thereof. The operations according to the above-described embodiments may be performed by the processor. In addition, the processor may be implemented as an encoder / decoder for the operations of the above-described embodiments.

[1150] Mode of the present disclosure

[1151] As described above, relevant details are described in the best mode for carrying out the embodiment.

[1152] Industrial Applicability

[1153] As described above, the embodiments are fully or partially applicable to point cloud data transmitting / receiving devices and systems.

[1154] Those skilled in the art can change or modify the embodiments in various ways within the scope of the embodiments.

[1155] The embodiments may include changes / modifications within the scope of the claims and their equivalents.

Claims

1. A method for sending point cloud data, the method comprising the following steps: Encode point cloud data; Encapsulating the point cloud data based on a file; as well as sending the point cloud data, wherein the file comprises a first component item containing geometric data of the point cloud data, a second component item containing attribute data of the point cloud data, a third component item containing occupancy data of the point cloud data, an atlas item containing atlas data of the point cloud data, and an atlas tile item containing atlas tiles, wherein the atlas tile item is associated with property information including information for indicating the number of tiles and information for indicating an identifier of the tile, The relationship between the atlas item and the atlas tile item is indicated based on the item reference information.

2. The method according to claim 1, in, When the non-timed point cloud data includes a single atlas, the single atlas is packaged based on one or more atlas items in the file.

3. A device for sending point cloud data, the device comprising: an encoder configured to encode point cloud data; A packager configured to package the point cloud data based on a file; as well as a transmitter configured to transmit the point cloud data, wherein the file comprises a first component item containing geometric data of the point cloud data, a second component item containing attribute data of the point cloud data, a third component item containing occupancy data of the point cloud data, an atlas item containing atlas data of the point cloud data, and an atlas tile item containing atlas tiles, wherein the atlas tile item is associated with property information including information for indicating the number of tiles and information for indicating an identifier of the tile, The relationship between the atlas item and the atlas tile item is indicated based on the item reference information.

4. The device according to claim 3, in, When the non-timed point cloud data includes a single atlas, the single atlas is packaged based on one or more atlas items in the file.

5. A method for receiving point cloud data, the method comprising the following steps: Receiving point cloud data; Decapsulating the point cloud data based on the file; as well as Decoding the point cloud data, wherein the file comprises a first component item comprising geometric data of the point cloud data, a second component item comprising attribute data of the point cloud data, a third component item comprising occupancy data of the point cloud data, an atlas item comprising atlas data of the point cloud data, and an atlas tile item comprising atlas tiles, wherein the atlas tile item is associated with property information comprising information for indicating the number of tiles and information for indicating an identifier of the tile, The relationship between the atlas item and the atlas tile item is indicated based on the item reference information.

6. The method according to claim 5, in, When the non-timed point cloud data includes a single atlas, the single atlas is packaged based on one or more atlas items in the file.

7. A device for receiving point cloud data, the device comprising: a receiver configured to receive point cloud data; A decapsulator configured to decapsulate the point cloud data based on a file; as well as a decoder configured to decode the point cloud data, wherein the file comprises a first component item containing geometric data of the point cloud data, a second component item containing attribute data of the point cloud data, a third component item containing occupancy data of the point cloud data, an atlas item containing atlas data of the point cloud data, and an atlas tile item containing atlas tiles, wherein the atlas tile item is associated with property information including information for indicating the number of tiles and information for indicating an identifier of the tile, The relationship between the atlas item and the atlas tile item is indicated based on the item reference information.

8. The device according to claim 7, in, When the non-timed point cloud data includes a single atlas, the single atlas is packaged based on one or more atlas items in the file.

Images