Point cloud data transmission device, point cloud data transmission method, point cloud data reception device, and point cloud data reception method

Abstract

A point cloud data transmission method according to an embodiment may comprise the steps of: encoding geometric data of point cloud data; encoding attribute data of the point cloud data based on the geometric data; and transmitting the encoded geometry data, the encoded attribute data, and the signaling information.

Description

Technical Field

[0001] Embodiments relate to a method and apparatus for processing point cloud content. Background Art

[0002] Point cloud content is content represented by a point cloud, which is a collection of points belonging to a coordinate system representing a three-dimensional space (or volume). Point cloud content can express media configured in three dimensions and is used to provide various services such as virtual reality (VR), augmented reality (AR), mixed reality (MR), XR (extended reality), and self-driving services. However, tens of thousands to hundreds of thousands of point data are required to represent point cloud content. Therefore, a method of efficiently processing large amounts of point data is needed.

[0003] In other words, high throughput is required to transmit and receive point cloud data. Therefore, in the process of transmitting and receiving point cloud data (in which encoding for compression and decoding for decompression are performed), the calculation operation is complicated and time-consuming due to the large capacity of point cloud data. Summary of the invention

[0004] Technical issues

[0005] An object of the present disclosure designed to solve the above-mentioned problems is to provide an apparatus and method for efficiently transmitting / receiving a point cloud.

[0006] Another object of the present disclosure is to provide an apparatus and method for solving delay and encoding / decoding complexity.

[0007] Another object of the present disclosure is to provide an apparatus and method for providing an efficient scalable service by encoding / decoding point cloud data based on layer groups.

[0008] The embodiments are not limited to the above-mentioned purposes, and the scope of the embodiments may cover other purposes that can be inferred by those skilled in the art based on the overall content of the present disclosure.

[0009] Technical Solution

[0010] To achieve these objectives and other advantages and in accordance with the purposes of the present disclosure, as implemented and broadly described herein, a method for sending point cloud data may include the following steps: encoding geometric data of the point cloud data, encoding attribute data of the point cloud data based on the geometric data, and sending the encoded geometric data, encoded attribute data, and signaling information.

[0011] According to an embodiment, the step of encoding the attribute data may include: segmenting the attribute data into one or more subgroups having a tree structure, and generating one or more levels of detail (LoDs) from the subgroup-based tree structure; searching and selecting one or more nearest neighbors of a neighbor search target node based on at least one of the one or more LoDs, and compressing the attribute information based on the selected one or more nearest neighbors of the neighbor search target node.

[0012] According to an embodiment, the sub-group based tree structure may include at least one parent sub-group and one or more child sub-groups of the at least one parent sub-group.

[0013] Depending on the implementation, each of the subgroups may be mapped to one or more LoDs.

[0014] According to an embodiment, the step of generating one or more LoDs may include comparing at least one parent-child group with a corresponding child group of the geometry data, detecting at least one missing node included in the at least one parent-child group, and appending the detected node to the corresponding LoD in the LoD.

[0015] According to an embodiment, the step of searching and selecting the nearest neighbor may include: when the neighbor search target node belongs to one of one or more sub-subgroups, searching for the nearest neighbor by referring to a node of the sub-subgroup to which the neighbor search target node belongs that is included in the one or more sub-subgroups.

[0016] According to an embodiment, the step of searching for and selecting a nearest neighbor may further include searching for the nearest neighbor by referring to a node included in a parent sub-group of a sub-sub-group to which the neighbor search target node belongs.

[0017] According to an embodiment, the signaling information may include at least subgroup identification information for identifying each of the subgroups or bounding box information related to each of the subgroups.

[0018] According to an embodiment, a device for sending point cloud data may include: a geometry encoder configured to encode geometry data of the point cloud data; an attribute encoder configured to encode attribute data of the point cloud data based on the geometry data; and a transmitter configured to send the encoded geometry data, the encoded attribute data and signaling information.

[0019] According to an embodiment, the attribute encoder may include: a LoD generator configured to: segment attribute data into one or more subgroups having a tree structure, and generate one or more levels of detail (LoDs) from the subgroup-based tree structure; and search and select one or more nearest neighbors of a neighbor search target node based on at least one of the one or more LoDs; and a compressor configured to compress attribute information based on the selected one or more nearest neighbors of the neighbor search target node.

[0020] According to an embodiment, the sub-group based tree structure may include at least one parent sub-group and one or more child sub-groups of the at least one parent sub-group.

[0021] Depending on the implementation, each of the subgroups may be mapped to one or more LoDs.

[0022] According to an embodiment, the LoD generator may be configured to compare at least one parent-child group with a corresponding child group of geometry data, detect at least one missing node included in the at least one parent-child group, and append the detected node to the corresponding LoD in the LoD.

[0023] According to an embodiment, when the neighbor search target node belongs to one of one or more sub-subgroups, the LoD generator may search for the nearest neighbor by referring to a node of the sub-subgroup to which the neighbor search target node belongs included in the one or more sub-subgroups.

[0024] According to an embodiment, the LoD generator may search for the nearest neighbor by referring to a node included in a parent sub-group of a sub-sub-group to which the neighbor search target node belongs.

[0025] Beneficial Effects

[0026] The apparatus and method according to the embodiments may provide quality point cloud services.

[0027] The device and method according to the embodiment may implement various video codec methods.

[0028] The apparatus and method according to the embodiment may provide general point cloud content such as autonomous driving services.

[0029] The apparatus and method according to the embodiment may perform spatially adaptive segmentation of point cloud data for independent encoding and decoding of the point cloud data, thereby improving parallel processing and providing scalability.

[0030] The apparatus and method according to the embodiment may perform encoding and decoding by segmenting point cloud data in units of tiles and / or slices, and signaling necessary data thereof, thereby improving encoding and decoding performance of point clouds.

[0031] The device and method according to the embodiment can divide and transmit the compressed data of the point cloud data according to a predetermined criterion. In addition, when layered coding is used, the compressed data can be divided and transmitted according to the layer. Therefore, the storage and transmission efficiency of the transmitting device can be improved.

[0032] When performing scalable transmission and / or scalable decoding, the device and method according to the embodiment apply the same LoD generation and NN search performed by the attribute encoder in the transmitting device to the attribute decoder in the receiving device. Specifically, by using the attribute encoder of the transmitting device and the attribute decoder of the receiving device to perform the same position correction and NN search, the neighbor candidates considered by the attribute encoder of the transmitting device and the neighbor candidates considered by the attribute decoder of the receiving device are not different, and the order of the nodes does not change. Therefore, decoder prediction errors may not occur in operations such as predictive transformations for performing attribute predictions from neighbor nodes.

[0033] The apparatus and method according to the embodiment may improve the efficiency of scalable attribute coding by performing LoD generation and neighbor search in consideration of the presence of missing slices (or subgroups). BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this application, illustrate embodiments of the present disclosure, and together with the description serve to explain the principles of the present disclosure. In the drawings:

[0035] Figure 1 An exemplary point cloud content providing system according to an embodiment is shown;

[0036] Figure 2 is a block diagram illustrating a point cloud content providing operation according to an embodiment;

[0037] Figure 3 An exemplary point cloud encoder according to an embodiment is illustrated;

[0038] Figure 4 shows examples of octrees and occupancy codes according to an embodiment;

[0039] Figure 5 illustrates an example of point configuration in each LOD according to an embodiment;

[0040] Figure 6 illustrates an example of point configuration in each LOD according to an embodiment;

[0041] Figure 7 A point cloud decoder according to an embodiment is illustrated;

[0042] Figure 8A transmitting device according to an embodiment is illustrated;

[0043] Figure 9 A receiving device according to an embodiment is illustrated;

[0044] Figure 10 An exemplary structure that can be operated in conjunction with a point cloud data transmitting / receiving method / apparatus according to an embodiment is illustrated;

[0045] Figure 11 and Figure 12 is a diagram illustrating a process for encoding, transmitting, and decoding point cloud data according to an embodiment;

[0046] Figure 13 illustrates a layer-based configuration of point cloud data according to an embodiment;

[0047] Figure 14 (a) illustrates a bitstream structure for geometry data according to an embodiment, and Figure 14 (b) illustrates a bitstream structure for attribute data according to an embodiment;

[0048] Figure 15 is a diagram illustrating an example of a configuration of a bitstream for dividing a bitstream into layers for delivery according to an embodiment;

[0049] Figure 16 An exemplary bitstream ordering method used when a geometry bitstream and an attribute bitstream are multiplexed into a single bitstream according to an embodiment is illustrated;

[0050] Figure 17 Another exemplary bitstream ordering method used when a geometry bitstream and an attribute bitstream are multiplexed into a single bitstream according to an embodiment is illustrated;

[0051] Figure 18 (a) to Figure 18 (c) illustrates an example of symmetric geometry-property selection according to an embodiment;

[0052] Figure 19 (a) to Figure 19 (c) illustrates an example of asymmetric geometry-attribute selection according to an embodiment;

[0053] Figure 20 (a) to Figure 20 (c) illustrates an exemplary method of configuring a slice including point cloud data according to an embodiment;

[0054] Figure 21 (a) and Figure 21 (b) illustrates a geometric coding layer structure according to an embodiment;

[0055] Figure 22 illustrates layer group and subgroup structures according to embodiments;

[0056] Figure 23 (a) to Figure 23 (c) illustrates a representation of point cloud data based on layer groups according to an embodiment;

[0057] Figure 24 A point cloud data transmitting / receiving device / method according to an embodiment is illustrated;

[0058] Figure 25 is a flow chart illustrating an exemplary LoD generation method according to an embodiment;

[0059] Figure 26 is a diagram illustrating an exemplary LoD generation method used when there is a missing subgroup according to an embodiment;

[0060] Figure 27 is a diagram illustrating an exemplary method of searching for nearest neighbors according to an embodiment;

[0061] Figure 28 is a diagram illustrating another exemplary method of searching for nearest neighbors according to an embodiment;

[0062] Figure 29 is a diagram illustrating yet another exemplary method of searching for nearest neighbors according to an embodiment;

[0063] Figure 30 Another exemplary point cloud transmitting device according to an embodiment is illustrated;

[0064] Figure 31 is a flow chart illustrating an exemplary method of encoding point cloud data according to an embodiment;

[0065] Figure 32 Another exemplary point cloud receiving device according to an embodiment is illustrated;

[0066] Figure 33 is a flow chart illustrating an exemplary method of decoding point cloud data according to an embodiment;

[0067] Figure 34 is a flowchart illustrating an exemplary bitstream structure for transmitting / receiving point cloud data according to an embodiment;

[0068] Figure 35 An embodiment of a syntax structure of a sequence parameter set according to an embodiment is illustrated;

[0069] Figure 36An embodiment of a syntax structure of a header of a related attribute data unit according to an embodiment is illustrated;

[0070] Figure 37 is a diagram illustrating an example of compressing geometry and attributes of point cloud data for a service;

[0071] Figure 38 is a diagram illustrating another example of compressing geometry and attributes of point cloud data for a service according to an embodiment;

[0072] Figure 39 is a diagram illustrating yet another example of compressing geometry and attributes of point cloud data for a service according to an embodiment;

[0073] Figure 40 is a flowchart of a method of transmitting point cloud data according to an embodiment; and

[0074] Figure 41 is a flowchart of a method of receiving point cloud data according to an embodiment. DETAILED DESCRIPTION

[0075] Reference will now be made in detail to the preferred embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. The detailed description given below with reference to the accompanying drawings is intended to illustrate exemplary embodiments of the present disclosure, rather than to show the only embodiments that can be implemented according to the present disclosure. The following detailed description includes specific details in order to provide a thorough understanding of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure can be practiced without these specific details.

[0076] Although most of the terms used in the present disclosure are selected from common terms widely used in the art, some terms are arbitrarily selected by the applicant and their meanings are explained in detail in the following description as needed. Therefore, the present disclosure should be understood based on the intended meanings of the terms rather than their simple names or meanings.

[0077] Figure 1 An exemplary point cloud content providing system according to an embodiment is shown.

[0078] Figure 1 The point cloud content providing system shown may include a transmitting device 10000 and a receiving device 10004. The transmitting device 10000 and the receiving device 10004 can transmit and receive point cloud data via wired or wireless communication.

[0079] According to an embodiment, the point cloud data sending device 10000 can obtain and process point cloud video (or point cloud content) and send it. According to an embodiment, the sending device 10000 may include a fixed station, a base transceiver system (BTS), a network, an artificial intelligence (AI) device and / or system, a robot, an AR / VR / XR device and / or a server. According to an embodiment, the sending device 10000 may include a device configured to communicate with a base station and / or other wireless devices using a radio access technology (e.g., 5G new RAT (NR), long term evolution (LTE)), a robot, a vehicle, an AR / VR / XR device, a portable device, a home appliance, an Internet of Things (IoT) device, and an AI device / server.

[0080] According to the implementation manner, the sending device 10000 includes a point cloud video acquisition unit 10001, a point cloud video encoder 10002 and / or a transmitter (or communication module) 10003.

[0081] The point cloud video acquisition unit 10001 according to the embodiment acquires the point cloud video through a processing process such as capture, synthesis or generation. The point cloud video is a point cloud content represented by a point cloud, which is a set of points located in a 3D space and can be referred to as point cloud video data, point cloud data, etc. The point cloud video according to the embodiment may include one or more frames. A frame represents a still image / picture. Therefore, the point cloud video may include a point cloud image / frame / picture and may be referred to as a point cloud image, frame or picture.

[0082] The point cloud video encoder 10002 according to the embodiment encodes the acquired point cloud video data. The point cloud video encoder 10002 may encode the point cloud video data based on point cloud compression coding. The point cloud compression coding according to the embodiment may include geometry-based point cloud compression (G-PCC) coding and / or video-based point cloud compression (V-PCC) coding or next generation coding. The point cloud compression coding according to the embodiment is not limited to the above-mentioned embodiment. The point cloud video encoder 10002 may output a bit stream containing the encoded point cloud video data. The bit stream may include not only the encoded point cloud video data, but also signaling information related to the encoding of the point cloud video data.

[0083] The transmitter 10003 according to the embodiment transmits a bitstream containing encoded point cloud video data. The bitstream according to the embodiment is encapsulated in a file or a fragment (e.g., a stream fragment) and is transmitted via various networks such as a broadcast network and / or a broadband network. Although not shown in the figure, the transmitting device 10000 may include a packager (or a package module) configured to perform a packaging operation. According to an embodiment, the packager may be included in the transmitter 10003. According to an embodiment, the file or fragment may be sent to the receiving device 10004 via a network, or stored in a digital storage medium (e.g., USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc.). The transmitter 10003 according to the embodiment can communicate with the receiving device 10004 (or the receiver 10005) via a 4G, 5G, 6G or other network by wire / wireless communication. In addition, the transmitter may perform necessary data processing operations according to the network system (e.g., a 4G, 5G or 6G communication network system). The transmitting device 10000 may transmit the encapsulated data in an on-demand manner.

[0084] The receiving device 10004 according to an embodiment includes a receiver 10005, a point cloud video decoder 10006 and / or a renderer 10007. According to an embodiment, the receiving device 10004 may include a device configured to communicate with a base station and / or other wireless devices using a radio access technology (e.g., 5G New RAT (NR), Long Term Evolution (LTE)), a robot, a vehicle, an AR / VR / XR device, a portable device, a home appliance, an Internet of Things (IoT) device, and an AI device / server.

[0085] The receiver 10005 according to the embodiment receives a bitstream containing point cloud video data or a file / segment encapsulated with a bitstream from a network or a storage medium. The receiver 10005 may perform necessary data processing according to a network system (e.g., a communication network system of 4G, 5G, 6G, etc.). The receiver 10005 according to the embodiment may decapsulate the received file / segment and output a bitstream. According to the embodiment, the receiver 10005 may include a decapsulator (or decapsulation module) configured to perform a decapsulation operation. The decapsulator may be implemented as an element (or component) separated from the receiver 10005.

[0086] The point cloud video decoder 10006 decodes the bitstream containing the point cloud video data. The point cloud video decoder 10006 can decode the point cloud video data according to the method in which the point cloud video data is encoded (for example, in the reverse process of the operation of the point cloud video encoder 10002). Therefore, the point cloud video decoder 10006 can decode the point cloud video data by performing point cloud decompression encoding (the reverse process of point cloud compression). The point cloud decompression encoding includes G-PCC encoding.

[0087] The renderer 10007 renders the decoded point cloud video data. In one embodiment, the renderer 10007 may render the decoded point cloud video data according to a viewport, etc. The renderer 10007 may render not only the point cloud video data but also the audio data to output the point cloud content. According to an embodiment, the renderer 10007 may include a display configured to display the point cloud content. According to an embodiment, the display may be implemented as a separate device or component rather than included in the renderer 10007.

[0088] The arrow indicated by the dotted line in the figure represents the transmission path of the feedback information obtained by the receiving device 10004. Feedback information is information that reflects the interactivity with the user who consumes the point cloud content, and includes information about the user (e.g., head orientation information, viewport information, etc.). Specifically, when the point cloud content is content for a service that requires interaction with the user (e.g., self-driving service, etc.), the feedback information may be provided to the content sender (e.g., sending device 10000) and / or the service provider. Depending on the embodiment, the feedback information may be used in the receiving device 10004 and the sending device 10000, or may not be provided.

[0089] The head orientation information according to the embodiment may represent information about the position, orientation, angle and movement of the user's head. The receiving device 10004 according to the embodiment may calculate the viewport information based on the head orientation information. The viewport information is information about the area of ​​the point cloud video that the user is watching (that is, the area that the user is currently watching). That is, the viewport information is information about the area that the user is currently watching in the point cloud video. In other words, the viewport or viewport area may represent the area that the user is watching in the point cloud video. The viewpoint is the point that the user is watching in the point cloud video, and may represent the center point of the viewport area. That is, the viewport is an area centered on the viewpoint, and the size and shape of the area may be determined by the field of view (FOV). Therefore, the receiving device 10004 may extract the viewport information based on the vertical or horizontal FOV supported by the device and the head orientation information. In addition, the receiving device 10004 may perform gaze analysis based on the head orientation information and / or the viewport information to determine the way the user consumes the point cloud video, the area that the user is looking at in the point cloud video, and the gaze time. According to an embodiment, the receiving device 10004 may send feedback information including the gaze analysis result to the sending device 10000. According to an embodiment, a device such as a VR / XR / AR / MR display may extract a viewport area based on the position / orientation of the user's head and the vertical or horizontal FOV supported by the device. According to an embodiment, the head orientation information and the viewport information may be referred to as feedback information, signaling information, or metadata.

[0090] According to an embodiment, feedback information may be obtained during rendering and / or display. The feedback information may be obtained by one or more sensors included in the receiving device 10004. According to an embodiment, the feedback information may be ensured by the renderer 10007 or a separate external element (or device, component, etc.). Figure 1 The dotted line in represents the process of sending feedback information ensured by the renderer 10007. Feedback information can not only be sent to the sending side, but also be consumed on the receiving side. That is, the point cloud content providing system can process (encode / decode / render) point cloud data based on the feedback information. For example, the point cloud video decoder 10006 and the renderer 10007 can prioritize decoding and rendering of point cloud videos only for the area currently viewed by the user based on the feedback information (i.e., head orientation information and / or viewport information).

[0091] In addition, the receiving device 10004 may send feedback information to the sending device 10000. The sending device 10000 (or the point cloud video data encoder 10002) may perform an encoding operation based on the feedback information. Therefore, the point cloud content providing system may efficiently process necessary data (e.g., point cloud data corresponding to the user's head position) based on the feedback information instead of processing (encoding / decoding) the entire point cloud data, and provide the point cloud content to the user.

[0092] According to an implementation, the transmitting device 10000 may be referred to as an encoder, a transmitting device, a transmitter, a transmitting system, etc., and the receiving device 10004 may be referred to as a decoder, a receiving device, a receiver, a receiving system, etc.

[0093] According to the implementation Figure 1 The point cloud data processed in the point cloud content providing system (through a series of processes of acquisition / encoding / transmission / decoding / rendering) may be referred to as point cloud content data or point cloud video data. According to an embodiment, point cloud content data may be used as a concept covering metadata or signaling information related to point cloud data.

[0094] Figure 1 The elements of the point cloud content providing system shown may be implemented by hardware, software, a processor and / or a combination thereof.

[0095] Figure 2 is a block diagram illustrating point cloud content providing operations according to an embodiment.

[0096] Figure 2 The block diagram shows Figure 1 The operation of the point cloud content providing system described in . As described above, the point cloud content providing system can process point cloud data based on point cloud compression coding (e.g., G-PCC).

[0097] According to an embodiment, a point cloud content providing system (e.g., a point cloud sending device 10000 or a point cloud video acquisition unit 10001) can acquire a point cloud video (20000). The point cloud video is represented by a point cloud belonging to a coordinate system for expressing a 3D space. According to an embodiment, a point cloud video may include a Ply (Polygon file format or Stanford Triangle format) file. When a point cloud video has one or more frames, the acquired point cloud video may include one or more Ply files. A Ply file contains point cloud data such as point geometry and / or attributes. Geometry includes the position of a point. The position of each point may be represented by a parameter (e.g., the value of the X, Y, and Z axes) representing a three-dimensional coordinate system (e.g., a coordinate system consisting of X, Y, and Z axes). Attributes include attributes of the point (e.g., information about the texture, color (YCbCr or RGB), reflectivity r, transparency, etc. of each point). A point has one or more attributes. For example, a point may have a color attribute or two attributes, color and reflectivity. According to an embodiment, geometry may be referred to as position, geometry information, geometry data, position information, position data, etc., and attribute may be referred to as attribute, attribute information, attribute data, etc. The point cloud content providing system (e.g., the point cloud transmitting device 10000 or the point cloud video acquiring unit 10001) may obtain point cloud data from information related to a point cloud video acquiring process (e.g., depth information, color information, etc.).

[0098] According to an embodiment, a point cloud content providing system (e.g., a transmitting device 10000 or a point cloud video encoder 10002) may encode point cloud data (20001). The point cloud content providing system may encode point cloud data based on point cloud compression coding. As described above, point cloud data may include geometric information and attribute information about points. Therefore, the point cloud content providing system may perform geometric coding for encoding geometry and output a geometric bitstream. The point cloud content providing system may perform attribute coding for encoding attributes and output an attribute bitstream. According to an embodiment, the point cloud content providing system may perform attribute coding based on geometric coding. The geometric bitstream and the attribute bitstream according to the embodiment may be multiplexed and output as one bitstream. The bitstream according to the embodiment may also include signaling information related to geometric coding and attribute coding.

[0099] According to the embodiment, the point cloud content providing system (eg, the transmitting device 10000 or the transmitter 10003) may transmit the encoded point cloud data (20002). Figure 1As shown, the encoded point cloud data can be represented by a geometry bitstream and an attribute bitstream. In addition, the encoded point cloud data can be sent in the form of a bitstream together with signaling information related to the encoding of the point cloud data (e.g., signaling information related to geometry encoding and attribute encoding). The point cloud content providing system can encapsulate the bitstream carrying the encoded point cloud data and send it in the form of a file or a fragment.

[0100] The point cloud content providing system (eg, receiving device 10004 or receiver 10005) according to the embodiment may receive a bit stream containing encoded point cloud data. In addition, the point cloud content providing system (eg, receiving device 10004 or receiver 10005) may demultiplex the bit stream.

[0101] The point cloud content providing system (e.g., the receiving device 10004 or the point cloud video decoder 10005) may decode the encoded point cloud data (e.g., geometry bitstream, attribute bitstream) sent in the bitstream. The point cloud content providing system (e.g., the receiving device 10004 or the point cloud video decoder 10005) may decode the point cloud video data based on the signaling information related to the encoding of the point cloud video data contained in the bitstream. The point cloud content providing system (e.g., the receiving device 10004 or the point cloud video decoder 10005) may decode the geometry bitstream to reconstruct the position (geometry) of the point. The point cloud content providing system may reconstruct the attributes of the point by decoding the attribute bitstream based on the reconstructed geometry. The point cloud content providing system (e.g., the receiving device 10004 or the point cloud video decoder 10005) may reconstruct the point cloud video based on the position according to the reconstructed geometry and the decoded attributes.

[0102] According to an embodiment, a point cloud content providing system (e.g., receiving device 10004 or renderer 10007) may render decoded point cloud data (20004). The point cloud content providing system (e.g., receiving device 10004 or renderer 10007) may use various rendering methods to render the geometry and attributes decoded by the decoding process. Points in the point cloud content may be rendered as vertices with a specific thickness, cubes with a specific minimum size centered at the corresponding vertex position, or circles centered at the corresponding vertex position. All or part of the rendered point cloud content is provided to the user through a display (e.g., a VR / AR display, a general display, etc.).

[0103] The point cloud content providing system (e.g., receiving device 10004) according to the embodiment may obtain feedback information (20005). The point cloud content providing system may encode and / or decode the point cloud data based on the feedback information. Feedback Information and Operations of the Point Cloud Content Providing System According to the Embodiment and Reference Figure 1 The feedback information and operations described are the same, so their detailed description is omitted.

[0104] Figure 3 An exemplary point cloud encoder according to an implementation is shown.

[0105] Figure 3 Show Figure 1 An example of a point cloud video encoder 10002. The point cloud encoder reconstructs and encodes point cloud data (e.g., the location and / or attributes of the points) to adjust the quality of the point cloud content (e.g., lossless, lossy, or near-lossless) according to network conditions or applications. When the total size of the point cloud content is large (e.g., 60 Gbps of point cloud content given for 30 fps), the point cloud content providing system may not be able to stream the content in real time. Therefore, the point cloud content providing system may reconstruct the point cloud content based on the maximum target bit rate to provide the point cloud content according to the network environment, etc.

[0106] As reference Figure 1 and Figure 2 As described, the point cloud encoder can perform geometry encoding and attribute encoding. Geometry encoding is performed before attribute encoding.

[0107] The point cloud video encoder according to the embodiment includes a coordinate transformer (transform coordinates) 30000, a quantizer (quantization and removal of points (voxelization)) 30001, an octree analyzer (analysis of octree) 30002, a surface approximation analyzer (analysis of surface approximation) 30003, an arithmetic encoder (arithmetic coding) 30004, a geometry reconstructor (reconstruction of geometry) 30005, a color transformer (transformation of color) 30006, an attribute transformer (transformation of attributes) 30007, a RAHT transformer 30008, a LOD generator (generated LOD) 30009, a lifting transformer (lifting) 30010, a coefficient quantizer (quantization of coefficients) 30011 and / or an arithmetic encoder (arithmetic coding) 30012. Figure 3 In the point cloud encoder of FIG. 30000, the coordinate transformer 30001, the quantizer 30002, the surface approximation analyzer 30003, the arithmetic encoder 30004 and the geometry reconstructor 30005 can be grouped together and referred to as a geometry encoder. The color transformer 30006, the attribute transformer 30007, the RAHT transformer 30008, the LOD generator 30009, the lifting transformer 30010, the coefficient quantizer 30011 and / or the arithmetic encoder 30012 can be grouped together and referred to as an attribute encoder.

[0108] The coordinate transformer 30000, the quantizer 30001, the octree analyzer 30002, the surface approximation analyzer 30003, the arithmetic encoder 30004 and the geometry reconstructor 30005 may perform geometry coding. The geometry coding according to the embodiment may include octree geometry coding, prediction tree geometry coding, direct coding, triplet geometry coding and entropy coding. Direct coding and triplet geometry coding are applied selectively or in combination. The geometry coding is not limited to the above examples.

[0109] As shown in the figure, the coordinate transformer 30000 according to the embodiment receives the position and transforms it into coordinates. For example, the position can be transformed into position information in a three-dimensional space (e.g., a three-dimensional space represented by an XYZ coordinate system). The position information in the three-dimensional space according to the embodiment can be referred to as geometric information.

[0110] The quantizer 30001 according to the embodiment quantizes the geometry. For example, the quantizer 30001 may quantize the points based on the minimum position value of all points (e.g., the minimum value on each of the X, Y, and Z axes). The quantizer 30001 performs a quantization operation: the difference between the minimum position value and the position value of each point is multiplied by a preset quantization scaling value, and then the nearest integer value is found by rounding the value obtained by the multiplication. Therefore, one or more points may have the same quantized position (or position value). The quantizer 30001 according to the embodiment performs voxelization based on the quantized position to reconstruct the quantized point. As in the case of a pixel (the smallest unit containing 2D image / video information), a point of a point cloud content (or 3D point cloud video) according to the embodiment may be included in one or more voxels. As a composite of volume and pixel, the term voxel refers to a 3D cubic space generated when a 3D space is divided into units (unit = 1.0) based on an axis representing the 3D space (e.g., X-axis, Y-axis, and Z-axis). The quantizer 30001 may match a point group in a 3D space with a voxel. According to an embodiment, a voxel may include only one point. According to an embodiment, a voxel may include one or more points. In order to represent a voxel as a point, the position of the center of the voxel may be set based on the position of one or more points included in the voxel. In this case, the properties of all positions included in a voxel may be combined and assigned to the voxel.

[0111] The octree analyzer 30002 according to the embodiment performs octree geometry encoding (or octree encoding) to present voxels in an octree structure. The octree structure represents points matched to voxels based on an octal tree structure.

[0112] The surface approximation analyzer 30003 according to an embodiment may analyze and approximate an octree. The octree analysis and approximation according to an embodiment is a process of analyzing a region including a plurality of points to efficiently provide an octree and voxelization.

[0113] The arithmetic encoder 30004 according to the embodiment performs entropy coding on the octree and / or the approximate octree. For example, the coding scheme includes arithmetic coding. As a result of the coding, a geometry bitstream is generated.

[0114] The color converter 30006, the attribute converter 30007, the RAHT converter 30008, the LOD generator 30009, the lifting converter 30010, the coefficient quantizer 30011 and / or the arithmetic encoder 30012 perform attribute encoding. As described above, a point may have one or more attributes. The attribute encoding according to the embodiment is also applied to the attributes of a point. However, when the attribute (e.g., color) includes one or more elements, the attribute encoding is applied independently to each element. The attribute encoding according to the embodiment includes color transform encoding, attribute transform encoding, regional adaptive hierarchical transform (RAHT) encoding, interpolation-based hierarchical nearest neighbor prediction (prediction transform) encoding, and interpolation-based hierarchical nearest neighbor prediction (lifting transform) encoding with an update / lifting step. Depending on the point cloud content, the above-mentioned RAHT encoding, prediction transform encoding, and lifting transform encoding may be selectively used, or a combination of one or more encoding schemes may be used. The attribute encoding according to the embodiment is not limited to the above examples.

[0115] The color converter 30006 according to an embodiment performs color conversion encoding to convert the color value (or texture) included in the attribute. For example, the color converter 30006 can convert the format of the color information (e.g., from RGB to YCbCr). Optionally, the operation of the color converter 30006 according to an embodiment can be applied according to the color value included in the attribute.

[0116] The geometry reconstructor 30005 according to an embodiment reconstructs (decompresses) an octree and / or an approximate octree. The geometry reconstructor 30005 reconstructs an octree / voxel based on the result of analyzing the point distribution. The reconstructed octree / voxel may be referred to as a reconstructed geometry (restored geometry).

[0117] The attribute transformer 30007 according to the embodiment performs attribute transformation to transform the attribute based on the reconstructed geometry and / or the position where the geometry encoding is not performed. As described above, since the attribute depends on the geometry, the attribute transformer 30007 can transform the attribute based on the reconstructed geometry information. For example, based on the position value of the point included in the voxel, the attribute transformer 30007 can transform the attribute of the point at the position. As described above, when the center position of the voxel is set based on the position of one or more points included in the voxel, the attribute transformer 30007 transforms the attribute of one or more points. When triplet geometry encoding is performed, the attribute transformer 30007 can transform the attribute based on the triplet geometry encoding.

[0118] The attribute converter 30007 can perform attribute conversion by calculating the average of the attributes or attribute values ​​(e.g., the color or reflectivity of each point) of the neighboring points within a specific position / radius from the center position (or position value) of each voxel. The attribute converter 30007 can apply weights according to the distance from the center to each point when calculating the average. Therefore, each voxel has a position and a calculated attribute (or attribute value).

[0119] The attribute converter 30007 can search for neighboring points within a specific position / radius from the center position of each voxel based on a KD tree or a Morton code. The KD tree is a binary search tree and supports a data structure capable of managing points based on position, so that a nearest neighbor search (NNS) can be performed quickly. The Morton code is generated by presenting the coordinates (e.g., (x, y, z)) representing the 3D positions of all points as bit values ​​and mixing the bits. For example, when the coordinates representing the point position are (5, 9, 1), the bit values ​​of the coordinates are (0101, 1001, 0001). Mixing the bit values ​​in the order of z, y, and x according to the bit index produces 010001000111. This value is represented as a decimal number 1095. That is, the Morton code value of the point with coordinates (5, 9, 1) is 1095. The attribute converter 30007 can sort the points based on the Morton code value and perform NNS through a depth-first traversal process. After the attribute transform operation, KD tree or Morton code is used when NNS is needed in another transform process for attribute encoding.

[0120] As shown, the transformed attributes are input to the RAHT transformer 30008 and / or the LOD generator 30009.

[0121] The RAHT transformer 30008 according to an embodiment performs RAHT encoding for predicting attribute information based on the reconstructed geometric information. For example, the RAHT transformer 30008 may predict attribute information of a higher-level node in the octree based on attribute information associated with a lower-level node in the octree.

[0122] The LOD generator 30009 according to an embodiment generates a level of detail (LOD) to perform predictive transform coding. The LOD according to an embodiment is the degree of detail of the point cloud content. As the LOD value decreases, the details of the point cloud content are degraded. As the LOD value increases, the details of the point cloud content are enhanced. Points can be classified by LOD.

[0123] The lifting transformer 30010 according to an embodiment performs lifting transform coding that transforms point cloud attributes based on weights. As described above, lifting transform coding can be optionally applied.

[0124] The coefficient quantizer 30011 according to the embodiment quantizes the attribute of the attribute encoding based on the coefficient.

[0125] The arithmetic encoder 30012 according to the embodiment encodes quantized properties based on arithmetic coding.

[0126] Although not shown in the figure, Figure 3 The elements of the point cloud encoder may be implemented by hardware, software, firmware, or a combination thereof including one or more processors or integrated circuits configured to communicate with one or more memories included in the point cloud providing device. The one or more processors may execute the above Figure 3 At least one of the operations and / or functions of the elements of the point cloud encoder. In addition, one or more processors may be operable or executable to perform Figure 3 The software program and / or instruction set for the operation and / or function of the elements of the point cloud encoder. According to the embodiment, one or more memories may include high-speed random access memory, or include non-volatile memory (e.g., one or more disk storage devices, flash memory devices or other non-volatile solid-state memory devices).

[0127] Figure 4 Examples of octrees and occupancy codes are shown according to an embodiment.

[0128] As reference Figures 1 to 3 As described, the point cloud content providing system (point cloud video encoder 10002) or the point cloud encoder (e.g., octree analyzer 30002) performs octree geometry encoding (or octree encoding) based on the octree structure to efficiently manage the area and / or position of voxels.

[0129] Figure 4 The upper part of FIG. 1 shows the octree structure. The 3D space of the point cloud content according to the embodiment is represented by the axes (eg, X-axis, Y-axis, and Z-axis) of the coordinate system. d ,2 d ,2 d ) to create an octree structure. Here, 2d can be set to the value of the smallest bounding box that surrounds all points of the point cloud content (or point cloud video). Here, d represents the depth of the octree. The value of d is determined in the following formula. In the following formula, (x int n ,y int n ,z int n ) represents the position (or position value) of the quantized point.

[0130]

[0131] like Figure 4As shown in the middle of the upper part of , the entire 3D space can be divided into eight spaces according to the partition. Each divided space is represented by a cube with six faces. Figure 4 As shown in the upper right part of , each of the eight spaces is divided again based on the axis of the coordinate system (e.g., X-axis, Y-axis, and Z-axis). Therefore, each space is divided into eight smaller spaces. The divided smaller spaces are also represented by cubes with six faces. This division scheme is applied until the leaf nodes of the octree become voxels.

[0132] Figure 4 The lower part shows the octree occupancy code. The occupancy code of the octree is generated to indicate whether each of the eight divided spaces generated by dividing one space contains at least one point. Therefore, a single occupancy code is represented by eight child nodes. Each child node represents the occupancy of the divided space, and the child node has a 1-bit value. Therefore, the occupancy code is represented as an 8-bit code. That is, when the space corresponding to the child node contains at least one point, the node is assigned a value of 1. When the space corresponding to the child node does not contain a point (the space is empty), the node is assigned a value of 0. Since Figure 4 The occupancy code shown is 00100001, thus indicating that the spaces corresponding to the third and eighth subnodes among the eight subnodes each contain at least one point. As shown in the figure, each of the third and eighth subnodes has eight subnodes, and the subnodes are represented by 8-bit occupancy codes. The accompanying drawings show that the occupancy code of the third subnode is 10000111, and the occupancy code of the eighth subnode is 01001111. The point cloud encoder (e.g., arithmetic encoder 30004) according to an embodiment may perform entropy encoding on the occupancy code. In order to increase compression efficiency, the point cloud encoder may perform intra / inter encoding on the occupancy code. The receiving device (e.g., receiving device 10004 or point cloud video decoder 10006) according to an embodiment reconstructs the occupancy code based on the occupancy code.

[0133] According to an embodiment, a point cloud encoder (e.g., Figure 4 The point cloud encoder or octree analyzer 30002) can perform voxelization and octree encoding to store point locations. However, points are not always evenly distributed in the 3D space, so there may be specific areas where fewer points exist. Therefore, performing voxelization on the entire 3D space is inefficient. For example, when a specific area contains few points, voxelization does not need to be performed in the specific area.

[0134] Therefore, for the above-mentioned specific area (or nodes other than the leaf nodes of the octree), the point cloud encoder according to the embodiment may skip voxelization and perform direct encoding to directly encode the point positions included in the specific area. The coordinates of the directly encoded points according to the embodiment are called direct coding mode (DCM). The point cloud encoder according to the embodiment may also perform triplet geometry encoding based on the surface model, which is to reconstruct the point positions in the specific area (or node) based on voxels. Triplet geometry encoding is a geometry encoding that represents an object as a series of triangular meshes. Therefore, the point cloud decoder can generate a point cloud from the mesh surface. Direct encoding and triplet geometry encoding according to the embodiment can be performed selectively. In addition, direct encoding and triplet geometry encoding according to the embodiment can be performed in combination with octree geometry encoding (or octree encoding).

[0135] In order to perform direct encoding, the option of using direct mode to apply direct encoding should be enabled. The node to which direct encoding is to be applied is not a leaf node, and there should be points less than a threshold value within a specific node. In addition, the total number of points to which direct encoding is to be applied should not exceed a preset threshold. When the above conditions are met, the point cloud encoder (or arithmetic encoder 30004) according to the embodiment can perform entropy encoding on the point position (or position value).

[0136] A point cloud encoder (e.g., surface approximation analyzer 30003) according to an embodiment may determine a specific level of the octree (a level less than the depth d of the octree), and may use a surface model starting from this level to perform triplet geometry encoding to reconstruct point positions in the node area based on voxels (triplet mode). A point cloud encoder according to an embodiment may specify a level to which triplet geometry encoding is to be applied. For example, when a specific level is equal to the depth of the octree, the point cloud encoder does not operate in triplet mode. In other words, a point cloud encoder according to an embodiment may operate in triplet mode only when the specified level is less than the depth value of the octree. A 3D cubic area of ​​a node at a specified level according to an embodiment is referred to as a block. A block may include one or more voxels. A block or voxel may correspond to a block. Geometry is represented as a surface within each block. A surface according to an embodiment may intersect each edge of a block at most once.

[0137] A block has 12 edges, so there are at least 12 intersections in a block. Each intersection is called a vertex. When there is at least one occupied voxel adjacent to the edge among all blocks sharing the edge, the vertex existing along the edge is detected. An occupied voxel according to an embodiment refers to a voxel containing a point. The vertex position detected along the edge is the average position of the edge of all voxels adjacent to the edge among all blocks along the shared edge.

[0138] Once the vertex is detected, the point cloud encoder according to the embodiment can perform entropy encoding on the start point (x, y, z) of the edge, the direction vector (Δx, Δy, Δz) of the edge, and the vertex position value (relative position value within the edge). When triplet geometry encoding is applied, the point cloud encoder according to the embodiment (e.g., geometry reconstructor 30005) can generate restored geometry (reconstructed geometry) by performing triangle reconstruction, upsampling, and voxelization processes.

[0139] The vertices located at the edges of the block determine the surface passing through the block. The surface according to an embodiment is a non-planar polygon. In the triangle reconstruction process, the surface represented by the triangle is reconstructed based on the starting point of the edge, the direction vector of the edge and the position value of the vertex. The triangle reconstruction process is performed by i) calculating the centroid value of each vertex, ii) subtracting the center value from each vertex value, and iii) estimating the sum of the squares of the values ​​obtained by subtraction.

[0140] ① ② ③

[0141] Estimate the minimum value of the sum, and perform the projection process according to the axis with the minimum value. For example, when the element x is the smallest, each vertex is projected on the x-axis relative to the center of the block and projected on the (y,z) plane. When the value obtained by the projection on the (y,z) plane is (ai,bi), the value of θ is estimated by atan2(bi,ai), and the vertices are sorted based on the value of θ. The following table shows the vertex combination for creating triangles according to the number of vertices. The vertices are sorted from 1 to n. Table 1 below shows that for four vertices, two triangles can be constructed according to the vertex combination. The first triangle can be composed of vertices 1, 2, and 3 among the sorted vertices, and the second triangle can be composed of vertices 3, 4, and 1 among the sorted vertices.

[0142] [Table 1]

[0143] [Table 1] Triangles formed from vertices sorted in 1,…,n order

[0144] n Triangle 3 (1,2,3) 4 (1,2,3),(3,4,1) 5 (1,2,3),(3,4,5),(5,1,3) 6 (1,2,3),(3,4,5),(5,6,1),(1,3,5) 7 (1,2,3),(3,4,5),(5,6,7),(7,1,3),(3,5,7) 8 (1,2,3),(3,4,5),(5,6,7),(7,8,1),(1,3,5),(5,7,1) 9 (1,2,3),(3,4,5),(5,6,7),(7,8,9),(9,1,3),(3,5,7),(7,9,3) 10 (1,2,3),(3,4,5),(5,6,7),(7,8,9),(9,10,1),(1,3,5),(5,7,9),(9,1,5) 11 (1,2,3),(3,4,5),(5,6,7),(7,8,9),(9,10,11),(11,1,3),(3,5,7),(7,9,11),(11,3,7) 12 (1,2,3),(3,4,5),(5,6,7),(7,8,9),(9,10,11),(11,12,1),(1,3,5),(5,7,9),(9,11,1),(1,5,9)

[0145] An upsampling process is performed to add points in the middle along the edges of the triangle, and voxelization is performed. The added points are generated based on the upsampling factor and the width of the block. The added points are called refined vertices. The point cloud encoder according to an embodiment may voxelize the refined vertices. In addition, the point cloud encoder may perform attribute encoding based on the voxelized position (or position value).

[0146] Figure 5 An example of point configuration in each LOD according to an embodiment is shown.

[0147] As reference Figures 1 to 4 As described, the encoded geometry is reconstructed (decompressed) before performing attribute encoding. When direct encoding is applied, the geometry reconstruction operation may include changing the placement of the directly encoded points (e.g., placing the directly encoded points in front of the point cloud data). When triplet geometry encoding is applied, the geometry reconstruction process is performed by triangle reconstruction, upsampling, and voxelization. Since the attributes depend on the geometry, attribute encoding is performed based on the reconstructed geometry.

[0148] The point cloud encoder (e.g., LOD generator 30009) can classify (or reorganize) the points by LOD. The figure shows the point cloud content corresponding to the LOD. The leftmost screen in the figure shows the original point cloud content. The second screen from the left in the figure shows the distribution of points in the lowest LOD, and the rightmost screen in the figure shows the distribution of points in the highest LOD. That is, the points in the lowest LOD are sparsely distributed, and the points in the highest LOD are densely distributed. That is, as the LOD increases in the direction indicated by the arrow indicated at the bottom of the figure, the space (or distance) between the points becomes narrower.

[0149] Figure 6 An example of point configuration for each LOD is shown according to an embodiment.

[0150] As reference Figures 1 to 5 As described, the point cloud content providing system or point cloud encoder (eg, point cloud video encoder 10002, Figure 3 The point cloud encoder or LOD generator 30009) can generate LOD. LOD is generated by reorganizing points into a set of refinement levels according to the set LOD distance value (or Euclidean distance set). The LOD generation process is performed not only by the point cloud encoder, but also by the point cloud decoder.

[0151] Figure 6 The upper part of shows examples of points (P0 to P9) of point cloud content distributed in 3D space. Figure 6 In , the original order represents the order of points P0 to P9 before LOD generation. Figure 6 In the LOD-based order, the order of points generated according to the LOD is shown. Points are reorganized by LOD. In addition, a high LOD contains points belonging to a lower LOD. Figure 6 As shown, LOD0 contains P0, P5, P4, and P2. LOD1 contains the points of LOD0, P1, P6, and P3. LOD2 contains the points of LOD0, the points of LOD1, P9, P8, and P7.

[0152] As reference Figure 3 As described, the point cloud encoder according to the embodiment may selectively or in combination perform prediction transform coding, lifting transform coding, and RAHT transform coding.

[0153] The point cloud encoder according to the embodiment may generate a predictor for a point to perform predictive transform coding for setting a prediction attribute (or a prediction attribute value) of each point. That is, N predictors may be generated for N points. The predictor according to the embodiment may calculate a weight (=1 / distance) based on the LOD value of each point, index information about neighboring points that exist within a set distance of each LOD, and the distance to the neighboring point.

[0154] The predicted attribute (or attribute value) according to the embodiment is set to the average of the values ​​obtained by multiplying the attribute (or attribute value) (e.g., color, reflectivity, etc.) of the neighboring points set in the predictor of each point by the weight (or weight value) calculated based on the distance to each neighboring point. The point cloud encoder (e.g., coefficient quantizer 30011) according to the embodiment can quantize and inverse quantize the residual (which can be called residual attribute, residual attribute value, attribute prediction residual, etc.) obtained by subtracting the predicted attribute (or attribute value) of each point from the attribute (attribute value) of each point. Tables 2 and 3 below show the quantization process.

[0155] [Table 2]

[0156] int PCCQuantization(int value, int quantStep) { if (value >= 0) { return floor(value / quantStep + 1.0 / 3.0); } else { return -floor(-value / quantStep + 1.0 / 3.0); } }

[0157] [Table 3]

[0158] int PCCInverseQuantization(int value, int quantStep) { if (quantStep == 0) { return value; } else { return value * quantStep; } }

[0159] When the predictor of each point has a neighboring point, the point cloud encoder (e.g., the arithmetic encoder 30012) according to the embodiment may perform entropy encoding on the quantized and inverse quantized residual values ​​as described above. When the predictor of each point has no neighboring point, the point cloud encoder (e.g., the arithmetic encoder 30012) according to the embodiment may perform entropy encoding on the attributes of the corresponding point without performing the above operation.

[0160] The point cloud encoder (e.g., lifting transformer 30010) according to an embodiment may generate a predictor for each point, set the calculated LOD and register neighbor points in the predictor, and set weights according to the distance to the neighbor points to perform lifting transform encoding. The lifting transform encoding according to an embodiment is similar to the above-mentioned predictive transform encoding, but is different in that weights are cumulatively applied to attribute values. The process of cumulatively applying weights to attribute values ​​according to an embodiment is configured as follows.

[0161] 1) Create an array quantized weight (QW) for storing weight values ​​of each point. The initial value of all elements of QW is 1.0. Multiply the QW value of the predictor index of the neighbor node registered in the predictor by the weight of the predictor of the current point, and add the values ​​obtained by multiplication.

[0162] 2) Boosting prediction process: The value obtained by multiplying the attribute value of the point by the weight is subtracted from the existing attribute value to calculate the predicted attribute value.

[0163] 3) Create temporary arrays called updateweight and update, and initialize the temporary arrays to zero.

[0164] 4) Accumulate the weight calculated by multiplying the weight calculated for all predictors by the weight stored in the QW corresponding to the predictor index with the updateweight array as the index of the neighbor node. Accumulate the value obtained by multiplying the attribute value of the neighbor node index by the calculated weight with the update array.

[0165] 5) Boosting update process: Divide the attribute values ​​of the update array of all predictors by the weight value of the updateweight array of the predictor index, and add the existing attribute value to the value obtained by the division.

[0166] 6) For all predictors, the predicted attribute is calculated by multiplying the attribute value updated by the lifting update process by the weight (stored in QW) updated by the lifting prediction process. The point cloud encoder (e.g., coefficient quantizer 30011) according to the embodiment quantizes the predicted attribute value. In addition, the point cloud encoder (e.g., arithmetic encoder 30012) performs entropy encoding on the quantized attribute value.

[0167] A point cloud encoder according to an embodiment (e.g., RAHT transformer 30008) may perform RAHT transform coding, in which attributes associated with nodes at a lower level in an octree are used to predict attributes of nodes at a higher level. RAHT transform coding is an example of attribute intra-frame coding by scanning backward through an octree. A point cloud encoder according to an embodiment scans the entire area starting from voxels and repeats a merging process of merging voxels into larger blocks at each step until a root node is reached. The merging process according to an embodiment is performed only on occupied nodes. The merging process is not performed on empty nodes. The merging process is performed on the upper node directly above the empty node.

[0168] The following formula represents the RAHT transformation matrix. In this formula, Represents the average attribute value of voxels at level 1. Can be based on and to calculate. and The weight of and

[0169]

[0170] here, Is the low-pass value and is used during the next highest level of merging. represents the high pass coefficient 0000000000. The high pass coefficients of each step are quantized and subjected to entropy coding (e.g., encoded by arithmetic encoder 30012). The weights are calculated as pass and Create the root node as follows.

[0171]

[0172] The values ​​of gDC are also quantized and subjected to entropy decoding like the high-pass coefficients.

[0173] Figure 7 A point cloud decoder according to an embodiment is shown.

[0174] Figure 7 The point cloud decoder shown is an example of a point cloud decoder and can perform a decoding operation, which is Figures 1 to 6 The inverse process of the encoding operation of the point cloud encoder shown.

[0175] As reference Figure 1 and Figure 6 As described, the point cloud decoder can perform geometry decoding and attribute decoding. Geometry decoding is performed before attribute decoding.

[0176] According to an embodiment, a point cloud decoder includes an arithmetic decoder (arithmetic decoding) 7000, an octree synthesizer (synthesized octree) 7001, a surface approximation synthesizer (synthesized surface approximation) 7002 and a geometry reconstructor (reconstructed geometry) 7003, an inverse coordinate transformer (inverse transformed coordinates) 7004, an arithmetic decoder (arithmetic decoding) 7005, an inverse quantizer (inverse quantization) 7006, a RAHT transformer 7007, an LOD generator (generate LOD) 7008, an inverse lifting (inverse lifting) 7009 and / or an inverse color transformer (inverse transformed color) 7010.

[0177] The arithmetic decoder 7000, the octree synthesizer 7001, the surface approximation synthesizer 7002, the geometry reconstructor 7003, and the coordinate inverse transformer 7004 may perform geometry decoding. The geometry decoding according to the embodiment may include direct decoding and triplet geometry decoding. Direct encoding and triplet geometry decoding are selectively applied. The geometry decoding is not limited to the above examples, and as a reference Figures 1 to 6 The inverse process of the geometric encoding described is performed.

[0178] The arithmetic decoder 7000 according to the embodiment decodes the received geometry bitstream based on arithmetic coding. The operation of the arithmetic decoder 7000 corresponds to the inverse process of the arithmetic encoder 30004.

[0179] The octree synthesizer 7001 according to the embodiment may generate an octree by acquiring an occupancy code (or information about the geometry obtained as a result of decoding) from the decoded geometry bitstream. Figures 1 to 6 Describe the configuration in detail.

[0180] When triplet geometry encoding is applied, the surface approximation synthesizer 7002 according to an embodiment may synthesize a surface based on the decoded geometry and / or the generated octree.

[0181] According to an embodiment, the geometry reconstructor 7003 may regenerate the geometry based on the surface and / or the decoded geometry. Figures 1 to 9 As described, direct coding and triplet geometry coding are selectively applied. Therefore, the geometry reconstructor 7003 directly imports the position information about the points to which direct coding is applied and adds them. When triplet geometry coding is applied, the geometry reconstructor 7003 can reconstruct the geometry by performing the reconstruction operations (e.g., triangle reconstruction, upsampling, and voxelization) of the geometry reconstructor 30005. Details and References Figure 6 Those described are the same, so their description is omitted. The reconstructed geometry may include a point cloud screen or frame that does not contain attributes.

[0182] The coordinate inverse transformer 7004 according to an embodiment may acquire a point position by transforming the coordinates based on the reconstructed geometry.

[0183] The arithmetic decoder 7005, the inverse quantizer 7006, the RAHT transformer 7007, the LOD generator 7008, the inverse lifter 7009 and / or the color inverse transformer 7010 may perform attribute decoding. Attribute decoding according to an embodiment includes regional adaptive hierarchical transform (RAHT) decoding, hierarchical nearest neighbor prediction (prediction transform) decoding based on interpolation, and hierarchical nearest neighbor prediction (lifting transform) decoding based on interpolation with an update / lifting step. The above three decoding schemes may be selectively used, or a combination of one or more decoding schemes may be used. Attribute decoding according to an embodiment is not limited to the above examples.

[0184] The arithmetic decoder 7005 according to the embodiment decodes the attribute bit stream through arithmetic coding.

[0185] The inverse quantizer 7006 according to the embodiment inversely quantizes information about the decoded attribute bitstream or the attribute obtained as a result of decoding, and outputs the inversely quantized attribute (or attribute value). The inverse quantization may be selectively applied based on the attribute encoding of the point cloud encoder.

[0186] According to an embodiment, the RAHT transformer 7007, the LOD generator 7008 and / or the inverse lifter 7009 may process the reconstructed geometry and inverse quantized attributes. As described above, the RAHT transformer 7007, the LOD generator 7008 and / or the inverse lifter 7009 may selectively perform a decoding operation corresponding to the encoding of the point cloud encoder.

[0187] The color inverse transformer 7010 according to an embodiment performs inverse transform encoding to inversely transform a color value (or texture) included in the decoded attribute. The operation of the color inverse transformer 7010 may be selectively performed based on the operation of the color transformer 30006 of the point cloud encoder.

[0188] Although not shown in the figure, Figure 7 The elements of the point cloud decoder may be implemented by hardware, software, firmware, or a combination thereof including one or more processors or integrated circuits configured to communicate with one or more memories included in the point cloud providing device. The one or more processors may execute the above Figure 7 At least one or more of the operations and / or functions of the elements of the point cloud decoder. In addition, one or more processors may be operable or executed to perform Figure 7 A software program and / or instruction set for the operation and / or functionality of the elements of a point cloud decoder.

[0189] Figure 8 A transmitting device according to an embodiment is shown.

[0190] Figure 8 The sending device shown is Figure 1 The sending device 10000 (or Figure 3 Example of a point cloud encoder). Figure 8 The sending device shown can be executed with reference to Figures 1 to 6 The transmitting device according to the embodiment may include a data input unit 8000, a quantization processor 8001, a voxelization processor 8002, an octree occupancy code generator 8003, a surface model processor 8004, an intra-frame / inter-frame coding processor 8005, an arithmetic encoder 8006, a metadata processor 8007, a color transformation processor 8008, an attribute transformation processor 8009, a prediction / lifting / RAHT transformation processor 8010, an arithmetic encoder 8011 and / or a transmission processor 8012.

[0191] The data input unit 8000 receives or acquires point cloud data according to an embodiment. The data input unit 8000 may perform the same operation and / or acquisition method as the point cloud video acquisition unit 10001 (or refer to Figure 2 The acquisition process 20000) described above has the same or similar operations and / or acquisition methods.

[0192] The data input unit 8000, the quantization processor 8001, the voxelization processor 8002, the octree occupancy code generator 8003, the surface model processor 8004, the intra / inter coding processor 8005 and the arithmetic encoder 8006 perform geometric coding. Figures 1 to 9 The geometrical encodings described are the same or similar, so their detailed description is omitted.

[0193] The quantization processor 8001 according to an embodiment quantizes geometry (e.g., position values ​​of points). The operation and / or quantization of the quantization processor 8001 is similar to the reference Figure 3 The operation and / or quantization of the quantizer 30001 described above are the same or similar. Figures 1 to 9 Same as those described.

[0194] According to an embodiment, the voxelization processor 8002 voxelizes the quantized position value of the point. The voxelization processor 8002 may perform the same as the reference Figure 3 The operation and / or voxelization process of the quantizer 30001 described above is the same or similar to the operation and / or process described above. Figures 1 to 6 Same as those described.

[0195] The octree occupancy code generator 8003 according to the embodiment performs octree encoding on the voxelized position of the point based on the octree structure. The octree occupancy code generator 8003 may generate an occupancy code. The octree occupancy code generator 8003 may perform the same as the reference code. Figure 3 and Figure 4 The operations and / or methods of the point cloud encoder (or octree analyzer 30002) described herein are the same or similar to the operations and / or methods described herein. Figures 1 to 6 Same as those described.

[0196] According to an embodiment, the surface model processor 8004 may perform triplet geometry encoding based on the surface model to reconstruct the point position in a specific area (or node) based on voxels. Figure 3 The operations and / or methods of the point cloud encoder (e.g., surface approximation analyzer 30003) described above are the same or similar to the operations and / or methods. Figures 1 to 6 Same as those described.

[0197] The intra / inter encoding processor 8005 according to an embodiment may perform intra / inter encoding on the point cloud data. The intra / inter encoding processor 8005 may perform encoding that is the same as or similar to intra / inter encoding. According to an embodiment, the intra / inter encoding processor 8005 may be included in the arithmetic encoder 8006.

[0198] According to an embodiment, the arithmetic encoder 8006 performs entropy coding on the octree and / or approximate octree of the point cloud data. For example, the coding scheme includes arithmetic coding. The arithmetic encoder 8006 performs the same or similar operations and / or methods as the operations and / or methods of the arithmetic encoder 30004.

[0199] The metadata processor 8007 according to an embodiment processes metadata (e.g., set values) about the point cloud data and provides it to necessary processing processes such as geometry coding and / or attribute coding. In addition, the metadata processor 8007 according to an embodiment may generate and / or process signaling information related to geometry coding and / or attribute coding. The signaling information according to an embodiment may be encoded separately from the geometry coding and / or attribute coding. The signaling information according to an embodiment may be interleaved.

[0200] The color conversion processor 8008, the attribute conversion processor 8009, the prediction / lifting / RAHT conversion processor 8010 and the arithmetic encoder 8011 perform attribute encoding. Figures 1 to 6 The attribute codes described are the same or similar, so their detailed description is omitted.

[0201] The color transform processor 8008 according to an embodiment performs color transform encoding to transform the color value included in the attribute. The color transform processor 8008 may perform color transform encoding based on the reconstructed geometry. The reconstructed geometry is compared with the reference Figures 1 to 9 In addition, its execution is the same as that of reference Figure 3 The operations and / or methods of the color converter 30006 described above are the same as or similar to the operations and / or methods described above, and detailed descriptions thereof are omitted.

[0202] The attribute transformation processor 8009 according to an embodiment performs attribute transformation to transform attributes based on reconstructed geometry and / or locations where geometry encoding is not performed. The attribute transformation processor 8009 performs the attribute transformation with reference to Figure 3 The operations and / or methods of the attribute converter 30007 described in the foregoing description are the same as or similar to those of the attribute converter 30007. Detailed description thereof is omitted. The prediction / lifting / RAHT transform processor 8010 according to the embodiment may encode the transformed attribute by any one or a combination of RAHT coding, prediction transform coding, and lifting transform coding. The prediction / lifting / RAHT transform processor 8010 performs the same as or similar to the reference. Figure 3 The operations of the RAHT transformer 30008, the LOD generator 30009 and the lifting transformer 30010 described above are the same or similar to at least one operation. In addition, the prediction transform coding, the lifting transform coding and the RAHT transform coding are the same as those of the reference Figures 1 to 9 Those described are the same, so their detailed description is omitted.

[0203] The arithmetic encoder 8011 according to the embodiment may encode the encoded attribute based on arithmetic coding. The arithmetic encoder 8011 performs the same or similar operation and / or method as that of the arithmetic encoder 300012.

[0204] The transmission processor 8012 according to the embodiment may send individual bitstreams containing encoded geometry and / or encoded attributes or metadata information, or send a bitstream containing encoded geometry and / or encoded attributes and metadata information. When the encoded geometry and / or encoded attributes and metadata information according to the embodiment are configured as a bitstream, the bitstream may include one or more sub-bitstreams. The bitstream according to the embodiment may include signaling information and slice data, and the signaling information includes a sequence parameter set (SPS) for sequence level signaling, a geometry parameter set (GPS) for geometry information coding signaling, an attribute parameter set (APS) for attribute information coding signaling, and a patch parameter set (TPS) for patch level signaling. Slice data may include information about one or more slices. A slice according to the embodiment may include a geometry bitstream Geom00 and one or more attribute bitstreams Attr00 and Attr10.

[0205] A slice is a sequence of syntactic elements that represents the entire or a portion of a coded point cloud frame.

[0206] The TPS according to an embodiment may include information about each of the one or more tiles (e.g., coordinate information and height / size information about a bounding box). The geometry bitstream may include a header and a payload. The header of the geometry bitstream according to an embodiment may include a parameter set identifier (geom_parameter_set_id), a tile identifier (geom_tile_id), and a slice identifier (geom_slice_id) included in the GPS, as well as information about the data contained in the payload. As described above, the metadata processor 8007 according to an embodiment may generate and / or process signaling information and send it to the transmission processor 8012. According to an embodiment, an element that performs geometry encoding and an element that performs attribute encoding may share data / information with each other as indicated by the dotted lines. The transmission processor 8012 according to an embodiment may perform operations and / or transmission methods that are the same or similar to those of the transmitter 10003. Details and References Figure 1 and Figure 2 Those described are the same, so their description is omitted.

[0207] Figure 9 A receiving device according to an embodiment is illustrated.

[0208] Figure 9 The receiving device shown isFigure 1 An example of a receiving device 10004. Figure 9 The receiving device shown can execute and refer to Figures 1 to 8 One or more operations and methods that are the same or similar to those of the described point cloud decoder.

[0209] The receiving device according to the embodiment may include a receiver 9000, a receiving processor 9001, an arithmetic decoder 9002, an octtree reconstruction processor based on an occupancy code 9003, a surface model processor (triangle reconstruction, upsampling, voxelization) 9004, an inverse quantization processor 9005, a metadata parser 9006, an arithmetic decoder 9007, an inverse quantization processor 9008, a prediction / lifting / RAHT inverse transform processor 9009, a color inverse transform processor 9010 and / or a renderer 9011. Each decoding element according to the embodiment may perform an inverse process of the operation of the corresponding encoding element according to the embodiment.

[0210] The receiver 9000 according to the embodiment receives the point cloud data. The receiver 9000 may perform the same Figure 1 The operation and / or receiving method of the receiver 10005 is the same as or similar to the operation and / or receiving method of the receiver 10005. The detailed description thereof is omitted.

[0211] The reception processor 9001 according to an embodiment may acquire a geometry bitstream and / or an attribute bitstream from the received data. The reception processor 9001 may be included in the receiver 9000 .

[0212] The arithmetic decoder 9002, the octtree reconstruction processor 9003 based on the occupancy code, the surface model processor 9004, and the inverse quantization processor 9005 may perform geometric decoding. Figures 1 to 10 The geometric decoding described is the same or similar, so its detailed description is omitted.

[0213] The arithmetic decoder 9002 according to the embodiment may decode the geometry bitstream based on arithmetic coding. The arithmetic decoder 9002 performs the same or similar operations and / or encoding as those of the arithmetic decoder 7000.

[0214] The octree reconstruction processor 9003 based on the occupancy code according to the embodiment can reconstruct the octree by obtaining the occupancy code from the decoded geometry bitstream (or information about the geometry obtained as a result of decoding). The octree reconstruction processor 9003 based on the occupancy code performs the same or similar operations and / or methods as the operations of the octree synthesizer 7001 and / or the octree generation method. When triplet geometry coding is applied, the surface model processor 9004 according to the embodiment can perform triplet geometry decoding and related geometry reconstruction (e.g., triangle reconstruction, upsampling, voxelization) based on the surface model method. The surface model processor 9004 performs the same or similar operations as the operations of the surface approximation synthesizer 7002 and / or the geometry reconstructor 7003.

[0215] The inverse quantization processor 9005 according to an embodiment may inversely quantize the decoded geometry.

[0216] According to an embodiment, the metadata parser 9006 may parse metadata (eg, setting values) contained in the received point cloud data. The metadata parser 9006 may pass the metadata to the geometry decoding and / or attribute decoding. Figure 8 The metadata described is the same, so a detailed description thereof is omitted.

[0217] The arithmetic decoder 9007, the inverse quantization processor 9008, the prediction / lifting / RAHT inverse transform processor 9009 and the color inverse transform processor 9010 perform attribute decoding. Figures 1 to 8 The attribute decoding described in at least one of them is the same or similar, so its detailed description is omitted.

[0218] The arithmetic decoder 9007 according to the embodiment may decode the attribute bitstream by arithmetic coding. The arithmetic decoder 9007 may decode the attribute bitstream based on the reconstructed geometry. The arithmetic decoder 9007 performs the same or similar operation and / or encoding as that of the arithmetic decoder 7005.

[0219] The inverse quantization processor 9008 according to an embodiment may inversely quantize the decoded attribute bitstream. The inverse quantization processor 9008 performs the same or similar operation and / or method as the inverse quantizer 7006 and / or the inverse quantization method.

[0220] The prediction / lifting / RAHT inverse transform processor 9009 according to an embodiment can process the reconstructed geometric and inverse quantized properties. The prediction / lifting / RAHT inverse transform processor 9009 performs the same Figure 7The color inverse transform processor 9010 according to the embodiment performs inverse transform encoding to inversely transform the color value (or texture) included in the decoded attribute. The color inverse transform processor 9010 performs the same or similar operations and / or decoding as the RAHT transformer 7007, the LOD generator 7008 and / or the inverse lifter 7009 of the embodiment. Figure 7 The operations and / or inverse transform encoding of the color inverse transformer 7010 are the same as or similar to the operations and / or inverse transform encoding of the color inverse transformer 7010. The renderer 9011 according to the embodiment may render point cloud data.

[0221] Figure 10 An exemplary structure that can be operated in conjunction with a point cloud data transmitting / receiving method / apparatus according to an embodiment is shown.

[0222] Figure 10 The structure of 1060 represents a configuration in which at least one of the server 1060, the robot 1010, the self-driving vehicle 1020, the XR device 1030, the smart phone 1040, the home appliance 1050, and / or the head mounted display (HMD) 1070 is connected to the cloud network 1000. The robot 1010, the self-driving vehicle 1020, the XR device 1030, the smart phone 1040, or the home appliance 1050 is referred to as a device. In addition, the XR device 1030 may correspond to a point cloud data (PCC) device according to an embodiment or may be operably connected to a PCC device.

[0223] The cloud network 1000 may represent a network that constitutes a part of a cloud computing infrastructure or exists in a cloud computing infrastructure. Here, the cloud network 1000 may be configured using a 3G network, a 4G or Long Term Evolution (LTE) network, or a 5G network.

[0224] The server 1060 may be connected to at least one of the robot 1010 , the self-driving vehicle 1020 , the XR device 1030 , the smart phone 1040 , the home appliance 1050 , and / or the HMD 1070 via the cloud network 1000 , and may assist in at least a portion of the processing of the connected devices 1010 to 1070 .

[0225] The HMD 1070 represents one of implementation types of an XR device and / or a PCC device according to an embodiment. The HMD type device according to an embodiment includes a communication unit, a control unit, a memory, an I / O unit, a sensor unit, and a power supply unit.

[0226] Hereinafter, various embodiments of devices 1010 to 1050 to which the above-described technology is applied will be described. Figure 10 The illustrated devices 1010 to 1050 may be operably connected / coupled to the point cloud data sending device and receiving device according to the above-described embodiments.

[0227] <PCC + XR>

[0228] The XR / PCC device 1030 may employ PCC technology and / or XR (AR+VR) technology, and may be implemented as an HMD, a head-up display (HUD) disposed in a vehicle, a television, a mobile phone, a smart phone, a computer, a wearable device, a home appliance, a digital signage, a vehicle, a fixed robot, or a mobile robot.

[0229] The XR / PCC device 1030 may analyze 3D point cloud data or image data acquired through various sensors or from an external device and generate position data and attribute data about 3D points. Thus, the XR / PCC device 1030 may acquire information about the surrounding space or real objects, and render and output XR objects. For example, the XR / PCC device 1030 may match an XR object including auxiliary information about the recognized object with the recognized object and output the matched XR object.

[0230] <PCC + XR + Mobile Phone>

[0231] The XR / PCC device 1030 may be implemented as a smart phone 1040 by applying the PCC technology.

[0232] Smart phone 1040 may decode and display point cloud content based on PCC technology.

[0233] <PCC + Self - driving + XR>

[0234] The self-driving vehicle 1020 may be implemented as a mobile robot, a vehicle, an unmanned aerial vehicle, etc. by applying PCC technology and XR technology.

[0235] The self-driving vehicle 1020 to which the XR / PCC technology is applied may represent a self-driving vehicle provided with a means for providing an XR image, or a self-driving vehicle as a control / interaction target in the XR image. Specifically, as a control / interaction target in the XR image, the self-driving vehicle 1020 may be distinguished from the XR device 1030 and may be operably connected thereto.

[0236] The self-driving vehicle 1020 having means for providing XR / PCC images may acquire sensor information from sensors including cameras, and output generated XR / PCC images based on the acquired sensor information. For example, the self-driving vehicle 1020 may have a HUD and output XR / PCC images thereto, thereby providing passengers with XR / PCC objects corresponding to real objects or objects presented on the screen.

[0237] When the XR / PCC object is output to the HUD, at least a portion of the XR / PCC object may be output to overlap with the real object to which the passenger's eyes are directed. On the other hand, when the XR / PCC object is output on a display provided in the self-driving vehicle, at least a portion of the XR / PCC object may be output to overlap with the object on the screen. For example, the self-driving vehicle 1220 may output XR / PCC objects corresponding to objects such as a road, another vehicle, a traffic light, a traffic sign, a two-wheeled vehicle, a pedestrian, and a building.

[0238] Virtual reality (VR) technology, augmented reality (AR) technology, mixed reality (MR) technology, and / or point cloud compression (PCC) technology according to an embodiment are applicable to various devices.

[0239] In other words, VR technology is a display technology that only provides CG images of real-world objects, backgrounds, etc. On the other hand, AR technology refers to a technology that displays a virtually created CG image on an image of a real object. MR technology is similar to the above-mentioned AR technology in that the virtual objects to be displayed are mixed and combined with the real world. However, MR technology is different from AR technology in that AR technology clearly distinguishes between real objects and virtual objects created as CG images and uses virtual objects as supplementary objects to real objects, while MR technology treats virtual objects as objects with properties equivalent to real objects. More specifically, an example of the application of MR technology is a holographic service.

[0240] Recently, VR, AR, and MR technologies are often referred to as extended reality (XR) technologies, rather than being clearly distinguished from each other. Therefore, embodiments of the present disclosure are applicable to any of VR, AR, MR, and XR technologies. Encoding / decoding based on PCC, V-PCC, and G-PCC technologies are applicable to such technologies.

[0241] The PCC method / apparatus according to the embodiment may be applied to a vehicle that provides a self-driving service.

[0242] Vehicles providing self-driving services are connected to the PCC device for wired / wireless communication.

[0243] When a point cloud data (PCC) transmitting / receiving device according to an embodiment is connected to a vehicle for wired / wireless communication, the device may receive / process content data related to an AR / VR / PCC service (which may be provided together with a self-driving service) and send it to the vehicle. In the case where the PCC transmitting / receiving device is installed on the vehicle, the PCC transmitting / receiving device may receive / process content data related to the AR / VR / PCC service and provide it to the user based on a user input signal input through a user interface device. The vehicle or user interface device according to the embodiment may receive a user input signal. The user input signal according to the embodiment may include a signal indicating a self-driving service.

[0244] The point cloud data transmission method / device according to the embodiment may be interpreted as referring to Figure 1 The transmitting device 10000, the point cloud video encoder 10002 and the transmitter 10003, Figure 2 Acquisition 20000-Encoding 20001-Transmission 20002, Figure 3 Point cloud video encoder, Figure 8 The sending device Figure 10 Devices Figure 30 A term for a sending device, etc.

[0245] The point cloud data receiving method / device according to the embodiment may be interpreted as referring to Figure 1 The receiving device 10004, the receiver 10005 and the point cloud video decoder 10006, Figure 2 Transmission 20002-Decoding 20003-Rendering 20004, Figure 7 Point cloud video decoder, Figure 9 The receiving device Figure 10 Devices Figure 32 A term for a receiving device, etc.

[0246] The point cloud data transmitting / receiving method / apparatus according to the embodiment may be referred to as the method / apparatus according to the embodiment for short.

[0247] According to the embodiment, geometric data, geometric information, position information, etc. constituting point cloud data are interpreted as having the same meaning. Attribute data, attribute information, etc. constituting point cloud data are interpreted as having the same meaning.

[0248] The method / apparatus according to the embodiment may process point cloud data taking into account scalable transmission.

[0249] Regarding the method / device according to the implementation mode, the present invention discloses a method for efficiently supporting selective decoding of partial data according to receiver performance or transmission speed when sending / receiving point cloud data. Specifically, the present disclosure proposes a technology for improving the efficiency of scalable coding, wherein an encoder at a transmitting end can selectively transmit information required by a decoder at a receiving side for compressed data, and the decoder can decode it, wherein the coding unit can be configured as an independent slice in the sense of a tree level, LOD, layer group unit, etc.

[0250] Specifically, the present disclosure proposes a method for improving the efficiency of scalable coding in point cloud data compression. Here, scalable coding is a technology for gradually changing the resolution of data according to the request / processing speed / performance / transmission bandwidth of the receiver, which can allow compressed data to be efficiently transmitted on the transmitter side and decoded on the receiving side. To this end, segmentation grouping can be applied together with the technology of the present disclosure to efficiently transmit point cloud data based on layer configuration, and can be used as a compression method for efficient storage and transmission of large-capacity point cloud data with wide distribution and high-density points.

[0251] References Figure 3 and Figure 7 The point cloud data sending / receiving device (which may be referred to as an encoder / decoder) of the embodiment shown, the point cloud data consists of a set of points. Each of the points includes geometric information (or geometry or geometric data) and attribute information (or attributes or attribute data). The geometric information is the three-dimensional position information (xyz) about each point. That is, the position of each point is represented by parameters in a coordinate system representing the three-dimensional space (for example, parameters (x, y, z) representing the three axes of the space (for example, the X-axis, the Y-axis, and the Z-axis)). The attribute information represents the color (RGB, YUV, etc.), reflectivity, normal vector, transparency, etc. of the point. In point cloud compression (PCC), octree-based compression is performed to efficiently compress the non-uniform distribution in the three-dimensional space, and the attribute information is compressed according to the octree-based compression. Figure 3 The point cloud video encoder and Figure 7 The point cloud video decoder shown in can process operations according to the implementation mode through corresponding components.

[0252] According to an embodiment, a transmitting device compresses geometric information (e.g., position) and attribute information (e.g., color / brightness / reflectivity, etc.) about point cloud data, and transmits the compressed information to a receiving device. The point cloud data may be configured according to an octree structure having layers according to a degree of detail or level of detail (LoD). Then, scalable point cloud data encoding and representation may be performed based on the configuration. In this case, only a portion of the point cloud data may be decoded or represented according to the performance or transmission rate of the receiving device. However, there is currently no method for removing unnecessary data in advance.

[0253] In other words, in the case where only a portion of the scalable point cloud compressed bitstream needs to be sent (e.g., only a portion of the decoded layer in scalable decoding), the necessary portion cannot be selected and sent. Figure 11 As shown, the transmitting device should re-encode the necessary parts after decoding, or as Figure 12 As shown, when the complete data is delivered to the receiving device, the receiving device should selectively apply necessary data after decoding.

[0254] However, in Figure 11 In this case, delays may occur due to the time used for decoding and re-encoding. Figure 12 In the case of fixed bandwidth, bandwidth efficiency may be reduced due to sending unnecessary data to the receiving device. In addition, when using a fixed bandwidth, it may be necessary to reduce data quality to send data.

[0255] Therefore, the method / apparatus according to an embodiment may provide slicing so that a point cloud may be divided into regions for processing.

[0256] Specifically, in the case of octree-based position compression, entropy-based compression and direct compression can be used together. In this regard, this paper proposes a slice configuration for efficiently exploiting scalability.

[0257] Furthermore, with respect to the method / apparatus according to the embodiments, a slice segment structure of point cloud data may be defined, and a scalable layer and a slice structure for scalable transmission may be signaled.

[0258] The method / apparatus according to the embodiment may divide and process a bitstream in specific units for efficient bitstream delivery and decoding.

[0259] The method / apparatus according to the embodiment can realize the selective transmission and decoding of layered point cloud data in bit stream units.

[0260] The unit according to the embodiment may be referred to as LOD, layer, slice, etc. LOD is the same term as LOD in attribute data encoding, but in another sense may mean a data unit for the hierarchical structure of a bitstream. LOD may be a concept corresponding to a bundle of one depth or two or more depths (e.g., the depth (level) of an octree or multiple trees) of a layer structure based on point cloud data. Similarly, a layer is provided to generate a unit of a sub-bitstream. It is a concept corresponding to a bundle of one depth or two or more depths, and may correspond to one LOD or two or more LODs. In addition, a slice is a unit for configuring a unit of a sub-bitstream, and may correspond to one depth, a part of one depth, or two or more depths. In addition, a slice may correspond to one LOD, a part of one LOD, or two or more LODs. According to the embodiment, LOD, layer, and slice may correspond to each other, or one of LOD, layer, and slice may be included in another. In addition, the unit according to the embodiment may include LOD, layer, slice, layer group, or subgroup, and may be referred to interchangeably with each other. According to an embodiment, in the octree structure, layer, depth, level, and depth level may have the same meaning.

[0261] Figure 13 is a diagram illustrating an example of layer-based point cloud data configuration according to an embodiment. Figure 13 An example of an octree structure is illustrated, in which the depth level of the root node is set to 0 and the depth level of the leaf node is set to 7.

[0262] The method / apparatus according to the embodiment can be as follows Figure 13 The illustrated configuration is based on layer-based point cloud data to encode and decode the point cloud data.

[0263] The stratification of point cloud data according to the implementation manner may have a layer structure in terms of SNR, spatial resolution, color, temporal frequency, bit depth, etc. according to the application field, and the layers may be configured in the direction of increasing data density based on an octree structure or a LOD structure.

[0264] That is, when LOD is generated based on the octree structure, LOD may be defined as increasing in a direction of increasing detail (ie, in a direction of increasing octave depth level). In the present disclosure, a layer may have the same meaning as a level, a depth, and a depth level.

[0265] Reference Figure 13, for example, in an octree structure having 7 depth levels other than a root node level (or root level), LOD 0 is configured to include levels from the root node level to octree depth level 4, and LOD 1 is configured to include levels from the root node level to octree depth level 5, and LOD 2 is configured to include levels from the root node level to octree depth level 7.

[0266] Figure 13 (a) illustrates a bit stream structure of geometry data according to an embodiment, and Figure 13 (b) illustrates a bit stream structure of attribute data according to an embodiment.

[0267] The method / apparatus according to the embodiment can be based on Figure 13 The octree structure shown in the figure is hierarchical to generate LOD, and as Figure 14 (a) and Figure 14 The geometry bitstream and attribute bitstream are configured as shown in (b).

[0268] The transmitting device according to the embodiment may divide the bit stream obtained by point cloud compression into a geometry bit stream and an attribute bit stream according to the data type to transmit the bit stream.

[0269] In this case, each bitstream may be composed of slices and transmitted. According to an embodiment, the geometry bitstream (e.g., Figure 14 (a)) and attribute bitstreams (e.g., Figure 14 Each of (b)) can be configured as a slice and delivered regardless of the layer information or LoD information. In this case, in order to use only some layers or LoDs, an operation of decoding the bitstream, an operation of selecting only the portion to be used and removing unnecessary portions, and an operation of re-encoding based only on necessary information should be performed.

[0270] The present disclosure proposes a method to divide a bitstream into layers (or LoDs) to avoid such unnecessary intermediate operations.

[0271] Figure 15 is a diagram illustrating an example of a bitstream configuration for delivering a bitstream divided into layers (or LoDs) according to an embodiment.

[0272] For example, in the case of the LoD-based PCC technology, the lower LoD is included in the higher LoD. That is, the higher LoD includes all points of the lower LoD. In addition, when information about points included in the current LoD but not included in the previous LoD (i.e., new points added to each LoD) is defined as R (static or reserved), the transmitting device may divide the initial LoD information and the information R newly included in each LoD into independent units (e.g., slices) and transmit them, such as Figure 15 shown.

[0273] In other words, a set of new points added to configure each LoD compared to the previous LoD may be defined as information R. Figure 15 An example is illustrated in which LoD1 includes LoD0 and information R1 , and LoD2 includes LoD1 and information R2 .

[0274] According to an embodiment, points sampled for one or more octree depth levels may be determined as data belonging to information R. That is, a set of points sampled for one or more octree depth levels (that is, points matched with occupied nodes) may be defined as information R. According to another embodiment, points sampled for one octree depth level may be divided into multiple pieces of information R according to a predetermined criterion. In this case, various criteria for dividing one octree depth level into multiple pieces of information R may be considered as follows. For example, when one octree depth level is divided into M pieces of information R, the M pieces of information R may be configured so that the data in the information R may have a continuous Morton code, or may be grouped so that the data in the information R may have the same remainder obtained by dividing the Morton code sequential index by M. Alternatively, when grouped into sibling nodes, the M pieces of information R may be configured by grouping data at the same position. According to another embodiment, if necessary, some sampling points at multiple octree depth levels may be determined as information R.

[0275] exist Figure 15 In the example of , according to an embodiment, the geometry bitstream and the attribute bitstream are each divided into three slices. Each slice includes a header and a payload (also called a data unit) containing actual data (e.g., geometry data, attribute data). The header may contain information about the corresponding slice. In addition, the header may also contain reference information related to a previous slice, a previous LoD, or a previous layer for LoD configuration.

[0276] For example, refer to Figure 15, the geometry bitstream is divided into slices carrying geometry data belonging to LoD0, slices carrying geometry data belonging to information R1, and slices carrying geometry data belonging to information R2. The attribute bitstream is divided into slices carrying attribute data belonging to LoD0, slices carrying attribute data belonging to information R1, and slices carrying attribute data belonging to information R2.

[0277] The reception method / apparatus according to the embodiment may receive a bitstream divided into LODs or layers, and may efficiently decode only data to be used without a complicated intermediate process.

[0278] In this regard, various implementations may be applied to transmit the bitstream.

[0279] For example, the geometry bitstream and the attribute bitstream may be delivered separately. Alternatively, the geometry bitstream and the attribute bitstream may be multiplexed into one bitstream to be delivered.

[0280] When each bitstream contains LoD0 and one or more pieces of information R, the delivery order of LoD0 and the one or more pieces of information R may vary.

[0281] exist Figure 15 In the example of , the geometry bitstream and the attribute bitstream are delivered separately. In this case, LoD0 including the geometry bitstream and two pieces of information R (R1, R2) is delivered in sequence, and LoD0 including the attribute bitstream and two pieces of information R (R1, R2) is delivered in sequence.

[0282] Figure 16 An exemplary bitstream ordering used when a geometry bitstream and an attribute bitstream are multiplexed into one bitstream according to an embodiment is illustrated.

[0283] The sending method / device according to the embodiment can send a bit stream as follows: Figure 16 As shown, the geometric data and attribute data are sent serially. In this operation, according to the data type, the entire geometric data (or geometric information) can be sent first, and then the attribute data (or attribute information) can be sent. In this case, the geometric data can be quickly reconstructed based on the sent bit stream information.

[0284] Reference Figure 16 , for example, a layer (LOD) containing geometry data may be positioned first in the bitstream, and a layer (LOD) containing attribute data may be positioned after the geometry layer. Since the attribute data depends on the geometry data, the layer (LOD) containing geometry data may be positioned before the layer (LOD) containing attribute data. Figure 16An example is illustrated in which LoD0 including geometry data and two pieces of information R (R1, R2) is sequentially transmitted, and then LoD0 including attribute data and two pieces of information R (R1, R2) is sequentially transmitted. In this regard, the position may be changed according to the implementation. In addition, references may be made between geometry headers, and references may also be made between attribute headers and geometry headers.

[0285] Figure 17 Another exemplary bitstream sorting method when a geometry bitstream and an attribute bitstream are multiplexed into one bitstream according to an embodiment is illustrated.

[0286] When sending a bit stream, the sending method / device according to the embodiment may be as follows: Figure 18 As shown, the geometry data and the attribute data are sent serially. In this case, the bit stream constituting the same layer containing the geometry data and the attribute data can be bundled and sent. In this case, when a compression technique for parallel decoding of geometry and attributes is used, the decoding execution time can be shortened. In this regard, the information that needs to be processed first can be set first (low LoD and geometry should be before attributes).

[0287] Figure 17 An example is illustrated in which LoD0 containing geometry data, LoD0 containing attribute data, information R1 containing geometry data, information R1 containing attribute data, information R2 containing geometry data, and information R2 containing attribute data are transmitted in this order. In this case, the positions may be changed according to the implementation. In addition, reference may be made between geometry headers, and reference may also be made between attribute headers and geometry headers.

[0288] The transmission / reception method / device according to the embodiment can efficiently select the desired layer (or LoD) in the application field at the bitstream level when transmitting and receiving the bitstream. Figure 16 When the geometry information is grouped and sent as shown, there may be an empty part in the middle after the bitstream level is selected. In this case, the bitstream may need to be rearranged.

[0289] like Figure 18 (a) to Figure 18 (c) or Figure 19 (a) to Figure 19 As shown in (c), by Figure 17 The layers shown bundle and transmit geometric data and attribute data, and may selectively transmit necessary information and / or selectively remove unnecessary information according to the application field.

[0290] Figure 18 (a) to Figure 18(c) is a diagram illustrating an example of symmetric geometry-property selection according to an embodiment.

[0291] For example, refer to Figure 18 (a) to Figure 18 (c), when a portion of the bitstream needs to be selected according to the implementation mode, the transmitting device selects and transmits only data up to LoD1 (i.e., LoD0+R1), and removes information R2 corresponding to the upper layer (i.e., the new portion of LoD2) from the bitstream and does not transmit the information R2. In the case of symmetric geometry-attribute selection, geometry data and attribute data of the same layer are simultaneously selected and transmitted, or simultaneously selected and removed.

[0292] Figure 19 (a) to Figure 19 (c) is a diagram illustrating an example of asymmetric geometry-attribute selection according to an embodiment. In the case of asymmetric geometry-attribute selection, only one of geometry data and attribute data of the same layer is selected and transmitted or removed.

[0293] For example, refer to Figure 19 (a) to Figure 19 (c), when a part of the bitstream needs to be selected according to the implementation mode, the transmitting device selects and transmits LoD1 (LoD0+R1) containing geometric data and attribute data, LoD1 (LoD0+R1) containing attribute data, and R2 containing geometric data, and removes R2 containing attribute data from the bitstream so as not to be transmitted. In other words, for attribute data, data other than the data (R2) of the upper layer is selected and transmitted. For geometric data, data of all layers (from level 0 (root level) to level 7 (leaf level) in the octree structure) are transmitted.

[0294] When a portion of the bitstream needs to be selected according to the implementation, it can be used Figure 18 (a) to Figure 18 (c) Symmetric geometry-attribute selection method, Figure 19 (a) to Figure 19 (c) an asymmetric geometry-attribute selection method or a combination of a symmetric geometry-attribute selection method and an asymmetric geometry-attribute selection method to select a portion of the bit stream.

[0295] The above-mentioned bitstream segments, selection of a portion of the bitstream, etc. are intended to support the scalability of point cloud data.

[0296] Reference Figure 13 , when point cloud data is represented in an octree structure and divided into LODs (or layers), scalable encoding / decoding (scalability) can be supported.

[0297] The scalability functionality according to an embodiment may include slice level scalability and / or octree level scalability.

[0298] The LoD according to an embodiment may be used as a unit indicating a set of one or more octree layers. In addition, the LoD may mean a bundle of octree layers to be configured on a slice-by-slice basis.

[0299] In addition to the meaning of LOD in attribute encoding / decoding, LOD according to an embodiment may be used in a broad sense, such as a unit for dividing data in detail.

[0300] That is, the spatial scalability of the actual octree layer (or scalable attribute layer) may be provided for each octree layer. However, when the scalability is configured at the slice level before bitstream parsing, the selection may be performed at the LoD level.

[0301] For example, refer to Figure 13 , in the octree structure, levels from the root level to level 4 correspond to LoD0, and levels from the root level to level 5 correspond to LoD1. In addition, levels from the root level to level 8 (ie, leaf level) correspond to LoD2.

[0302] In other words, Figure 13 In the example of , when scalability is used at a slice level, the scalable operations provided correspond to three steps of LoD0, LoD1, and LoD2, and the scalable operations that can be provided in the decoding operation through the octree structure correspond to 8 steps from the root level to the leaf level.

[0303] According to an embodiment, when LoD0 to LoD2 are composed of corresponding slices, the transcoder of the receiver or transmitter (see Figure 11 ) can select only LoD0, only LoD1, or only LoD2 for scalable processing. Figure 13 , LoD1 includes LoD0, and LOD2 includes LoD1 and LoD2.

[0304] For example, when only LoD0 is selected, the maximum octree level is 4, and a scalable layer can be selected from among octree layers 0 to 4 in a decoding operation. In this case, the receiving device can regard the node size obtainable through the maximum octree level (or depth) as a leaf node, and can send the node size through signaling information.

[0305] For example, when LoD1 is selected, layer 5 may be added. Therefore, the maximum octree level may be 5, and a scalable layer may be selected from among octree layers 0 to 5 in a decoding operation. In this case, the receiving device may regard the node size obtainable through the maximum octree level (or depth) as a leaf node, and may transmit the node size through signaling information. According to an embodiment, the octree depth, octree layer, and octree level may be units into which data is divided in detail.

[0306] For example, when LoD2 is selected, layers 6 and 7 may be added. Therefore, the maximum octree level may be 7, and a scalable layer may be selected from among octree layers 0 to 7 in a decoding operation. In this case, the receiving device may regard a node size obtainable through the maximum octree level (or depth) as a leaf node, and may transmit the node size through signaling information.

[0307] Figure 20 (a) to Figure 20 (c) illustrates an exemplary method of configuring a slice including point cloud data according to an embodiment.

[0308] The transmission method / apparatus / encoder according to the embodiment may configure the G-PCC bitstream by segmenting the bitstream in a slice structure. The data unit for detailed data representation may be a slice.

[0309] For example, one or more octree levels (or depths) may match one slice.

[0310] The transmission method / device (e.g., encoder) according to the embodiment may configure a bitstream based on a slice 41001 by scanning the nodes (points) included in the octree in the direction of the scan order 41000. The slice may include nodes of one or more levels in the octree structure, may include only nodes of a specific level, or may include only some nodes of a specific level. Alternatively, it may include only some nodes of one or more levels.

[0311] Figure 20 (a) of FIG. 4 illustrates an exemplary octree structure composed of 7 slices. In this example, slice 41002 may include nodes of levels 0 to 4, and slice 41003 may include some nodes of level 5. Slice 41004 may include some other nodes of level 5, and slice 41005 may include some other nodes of level 5. That is, in Figure 20 In (a), level 5 is divided into three slices. Similarly, in Figure 20 In (a), level 6 (i.e., leaf level) is also divided into three slices. In other words, a slice can be composed of some nodes at a specific level.

[0312] Figure 20 (b) illustrates an exemplary octree structure consisting of four slices. In this example, one slice includes nodes of levels 0 to 3 and some nodes of level 4, and another slice includes other nodes of level 4 and some nodes of level 5. In addition, another slice includes other nodes of level 5 and some nodes of level 6, and another slice includes other nodes of level 6.

[0313] Figure 20 (c) illustrates an exemplary octree structure consisting of five slices. One slice consists of nodes at level 0 to level 3, and four slices consist of nodes at level 4 to level 6. That is, a slice includes some nodes at level 4, some nodes at level 5, and some nodes at level 6. In other words, in levels 4 to 6, a slice may include some data at level 4 and data at level 5 or level 6 corresponding to a child node of the data.

[0314] In other words, Figure 20 (b) and Figure 20 As shown in (c), when multiple octree layers are matched with a slice, only some nodes of each of the layers may be included in the slice. When multiple slices constitute a geometry / attribute frame in this way, information required for the receiving device to configure the layer may be sent to the receiving device through signaling information. For example, the signaling information may include information about the layers included in each slice and information about the nodes included in each layer.

[0315] The encoder according to the embodiment and a device corresponding to the encoder may encode point cloud data, and generate and transmit a bit stream including the encoded data and signaling information (or parameter information) related to the point cloud data.

[0316] In addition, when generating a bitstream, the bitstream structure (for example, see Figures 14 to 20 Therefore, a receiving device, a decoder, a corresponding device, etc. according to an embodiment can receive and parse a bit stream configured to be suitable for selective decoding of some data, thereby decoding only a portion of the point cloud data and efficiently providing it.

[0317] Next, scalable transmission of point cloud data will be described.

[0318] The point cloud data transmitting method / apparatus according to the embodiment can scalably transmit a bit stream containing point cloud data, and the point cloud data receiving method / apparatus according to the embodiment can scalably receive and decode the bit stream.

[0319] When having Figures 11 to 20When the bitstream of the structure shown is used for scalable transmission, signaling information for selecting the slice required by the receiving device can be sent to the receiving device. Scalable transmission may not mean sending or decoding the entire bitstream, but may mean sending or decoding only a portion of the bitstream. Therefore, low-resolution point cloud data can be provided.

[0320] When scalable transmission is applied to octree-based geometry bitstream according to an embodiment, it should be allowed to be based only on each octree layer ( Figure 13 )’s bitstream to configure the point cloud data.

[0321] To this end, the target octree level should not have dependencies on lower octree level information. This can be a constraint applied to common geometry encoding / attribute encoding.

[0322] In addition, in scalable transmission, it is necessary to transmit a scalable structure for the transmitting / receiving device to select a scalable layer to the receiving device. Considering the octree structure according to the embodiment, all octree layers may support scalable transmission, or scalable transmission may be allowed only for specific octree layers and lower layers. For example, when some of the octree layers are included, signaling information may be delivered to the receiving device to indicate the scalable layer including slices. Therefore, the receiving device can determine whether slices are necessary / unnecessary in the bitstream stage. Figure 20 In the example of (a), level 0 (i.e., root level) to level 4 41002 may constitute a scalable layer without supporting scalable transmission, and the lower octree layers may be matched with the scalable layers in a one-to-one correspondence. Generally, scalability may be supported for the portion corresponding to the leaf node. Figure 20 As shown in (c), when multiple octree layers are included in a slice, it can be defined that a scalable layer should be configured for the layer.

[0323] In this case, scalable transmission and scalable decoding can be used separately according to the purpose. According to an embodiment, scalable transmission can be used so that the transmitting / receiving device selects information up to a specific layer without involving a decoder. According to an embodiment, scalable decoding can be used to select a specific layer during encoding. That is, scalable transmission can support the selection of necessary information in a compressed state without involving a decoder (i.e., at the bitstream stage) so that the transmitting or receiving device can determine a specific layer. On the other hand, in the case of scalable decoding, in the encoding / decoding process, encoding / decoding can be supported only for information up to a necessary part. Therefore, scalable decoding can be used for operations such as scalable representation.

[0324] In this case, the layer configuration for scalable transmission may be different from the layer configuration for scalable decoding. For example, the lower three octree layers including the leaf nodes may constitute one layer in terms of scalable transmission. On the other hand, in terms of scalable decoding, when all layer information is included, scalable decoding may be enabled for each of the leaf node layer, leaf node layer-1, and leaf node layer-2.

[0325] Figure 21 (a) and Figure 21 (b) illustrates a geometric coding layer structure according to an embodiment. Specifically, Figure 21 (a) illustrates an example of three slices generated by a layer group structure at an encoder on the transmitter side, and Figure 21 (b) illustrates an example of a partially decoded output using two slices at a decoder on the receiving side.

[0326] When fine granularity slicing (FGS) is enabled, the G-PCC bitstream can be sliced ​​into multiple sub-bitstreams. Here, fine granularity slicing can be referred to as layer group-based slicing. In order to effectively use the hierarchical structure of G-PCC, each slice can include coded data from a portion of the coding layer or a portion of the region. Using segmentation or partitioning of slices paired with the coding layer structure, use cases for scalable transmission or spatial random access can be supported in an efficient manner.

[0327] Layer - group - based slice segment

[0328] In fine-grained slicing, each slice segment may contain data encoded from layer groups defined as follows.

[0329] A layer group can be defined as a set of consecutive tree layers, where the start depth and end depth of the tree layer group can be any number in the tree depth, and the start depth can be less than the end depth. The order of data encoded in a slice segment can be the same as the order of data encoded in a single slice.

[0330] For example, consider Figure 21 In the geometric coding layer structure with eight coding layers shown in (a), there are three layer groups, each layer group matches a different slice. More specifically, layer group 1 for coding layers 0 to 4 matches slice 1, layer group 2 for coding layer 54 matches slice 2, and layer group 3 for coding layers 6 and 74 matches slice 3. When the first two slices (i.e., slice 1 and slice 2) are sent or selected, the decoded output will be partial layers 0 to 5, as shown in FIG. Figure 21 By using slices in the layer group structure, partial decoding of the coded layer can be supported without accessing the entire bitstream.

[0331] The bitstream and point cloud data according to the embodiment can be generated based on the slice segments based on the coding layer. By slicing the bitstream at the end of the coding layer during the encoding process, the method / apparatus according to the embodiment can select relevant slices to support scalable transmission or partial decoding.

[0332] Figure 21 (a) shows a geometric coding layer structure with 8 layers, where each slice corresponds to a layer group. Layer group 1 includes coding layers 0 to 4. Layer group 2 includes coding layer 5. Layer group 3 is a group for coding layers 6 and 7. When the geometry (or attribute) has a tree structure including eight levels (depths), the bitstream can be hierarchically configured by grouping data corresponding to one or more levels (depths). Each group can be included in one slice.

[0333] Figure 21 (b) shows the output of decoding when two slices are selected from three slices. When the decoder selects group 1 and group 2, partial layers of layers (depths) 0 to 5 of the tree are selected. That is, even if the entire bitstream is not accessed, partial decoding of the coded layer can be supported by using slices of the layer group structure.

[0334] For the partial decoding process according to the embodiment, the encoder may generate three slices based on the layer group structure. The decoder according to the embodiment may select two slices from among the three slices and perform partial decoding.

[0335] The bitstream according to the embodiment may include slices based on layer groups. Each slice may include a header containing signaling information related to the point cloud data (i.e., geometric data and / or attribute data) included in the slice. The receiving method / apparatus according to the embodiment may select a slice and decode the point cloud data contained in the payload of the slice based on the header included in the selected slice.

[0336] In addition to the layer group structure, the method / apparatus according to the embodiment may also divide the layer group into multiple subgroups taking into account the use case of spatial random access. The subgroups are mutually exclusive, and the set of subgroups may be the same as the layer group. Since the points of each subgroup form a boundary in the spatial domain, the subgroup may be represented by subgroup bounding box information. Based on the spatial information, the layer group and subgroup structure may support access to the region of interest (ROI) by selecting a slice covering the ROI. Spatial random access within a frame or patch may be supported by efficiently comparing the ROI with the bounding box information about each slice.

[0337] like Figure 21 As shown in (a), the method / apparatus according to an embodiment may be configured to transfer slices of point cloud data.

[0338] According to an embodiment, the entire coded bitstream may be included in a single slice. For multiple slices, each slice may contain a sub-bitstream. The order of the slices may be the same as the order of the sub-bitstreams. In addition, each slice may match a layer group in a tree structure.

[0339] Furthermore, a slice may not affect previous slices, just as higher levels of a geometry tree do not affect lower levels.

[0340] The segmented slices according to the embodiments are efficient in terms of error robustness, efficient transmission, support of regions of interest, etc.

[0341] 1) Error recovery

[0342] Compared to a single slice structure, segmented slices may be more robust to errors. In other words, when a slice contains the entire bitstream of a frame, data loss may affect the entire frame data. On the other hand, when the bitstream is segmented into multiple slices, at least one slice that is not affected by the loss can be decoded even when at least one slice is lost.

[0343] 2) Scalable sending

[0344] In the present disclosure, multiple decoders with different capabilities may be supported.

[0345] When the encoded point cloud data (i.e., point cloud compression (PCC) bitstream) is contained in a single slice, the LOD of the encoded point cloud data can be determined before encoding. Therefore, multiple pre-encoded bitstreams of point cloud data with different resolutions can be sent independently, which may be inefficient in terms of large bandwidth or storage space.

[0346] When the coded point cloud data (i.e., PCC bitstream) is contained in segmented slices, a single bitstream can support decoders at different levels. From the decoder's perspective, the receiving device can select the target layer and can pass a partially selected bitstream to the decoder. Similarly, by using a single bitstream without splitting the entire bitstream, the transmitting device can efficiently generate partial bitstreams.

[0347] 3) Region-based spatial scalability

[0348] In terms of G-PCC requirements according to embodiments, region-based spatial scalability may be defined as follows: the compressed bitstream consists of one or more layers, so a specific ROI may have a higher density with additional layers, and layers may be predicted from lower layers.

[0349] To support this requirement, it is necessary to support different detailed representations of regions by region. For example, in VR / AR applications, distant objects can be represented with lower precision, and closer objects can be represented with higher precision. In addition, the decoder can increase the resolution of the region of interest when requested. This operation can be implemented using the scalable structure of the geometric octree and G-PCC (e.g., a scalable attribute coding scheme).

[0350] According to an embodiment, the decoder should access the entire bitstream based on the current slice structure containing the entire geometry or attributes, which may lead to inefficiencies in bandwidth, memory, and decoder. On the other hand, when the bitstream is segmented into multiple slices and each slice contains a sub-bitstream according to a scalable layer, the decoder according to an embodiment can efficiently select a slice before parsing the bitstream as needed.

[0351] The method / apparatus according to the embodiment may generate a layer group using a tree structure (or layer structure) of point cloud data.

[0352] Reference Figure 21 (a) As an example, there are eight layers in a geometric coding layer structure (e.g., an octree structure), and three slices can be used to contain one or more layers. A group represents a group of layers. When scalable attribute coding is used, the tree structure is the same as the geometric tree structure. The same octree slice map can be used to create attribute slice segments.

[0353] The layer group according to the embodiment means a bundle of layer structure units (eg, octree layers or LoD layers) generated in G-PCC encoding.

[0354] A subgroup can be represented as a set of adjacent nodes within a layer group. For example, a subgroup can be configured as a set of nodes that are adjacent to each other in Morton code order, a set of adjacent nodes based on distance, or a set of adjacent nodes based on coding order. Nodes in a parent-child relationship can also exist within a subgroup.

[0355] When subgroups are defined, the boundary occurs in the middle of the layer, and a parameter such as entropy_continution_enabled_flag may be signaled to indicate whether entropy continuation is maintained at the boundary. Continuation may also be maintained by referencing the previous slice using ref_slice_id.

[0356] The tree structure according to the embodiment may be an octree structure, and the attribute layer structure or attribute coding tree according to the embodiment may include a level of detail (LOD) structure. In other words, the tree structure for point cloud data may include layers corresponding to depths or levels, and the layers may be grouped.

[0357] According to the method / apparatus of the embodiment (e.g.,Figure 3 Octree analyzer 30002 or LOD generator 30009, Figure 7 The octree synthesizer 7002 or LOD generator 7008) can generate an octree structure of geometry or an LOD tree structure of attributes. In addition, the point cloud data can be grouped based on the layer of the tree structure.

[0358] Reference Figure 21 (a), a plurality of layers are grouped into first to third groups. The groups may be further divided to form subgroups.

[0359] According to an embodiment, a slice may contain encoded data from a layer group. Here, a layer group is defined as a set of consecutive tree layers. The start depth and end depth of a tree layer may be specific numbers in the tree depth, where the number for the start is less than the number for the end.

[0360] Although Figure 21 (a) illustrates a geometric coding layer structure as an example of a tree structure, but a coding layer structure for attributes can also be created in a similar manner.

[0361] Figure 22 is a diagram illustrating a layer group and a sub-group structure according to an embodiment.

[0362] Reference Figure 22 ,Point cloud data and bitstream can be distinguished and represented by bounding boxes.

[0363] exist Figure 22 , a subgroup structure and a bounding box corresponding to the subgroup are shown. Layer group 2 is divided into two subgroups (group 2-1 and group 2-2) included in different slices, and layer group 3 is divided into four subgroups (group 3-1, group 3-2, group 3-3, and group 3-4) included in different slices. Given the slices and bounding box information of the layer group and subgroup, spatial access can be performed in the following manner: 1) compare the bounding box of each slice with the ROI, 2) select the slice whose subgroup bounding box overlaps with the ROI, and 3) decode the selected slice.

[0364] When considering the ROI in region 3-3, slice 1, slice 3, and slice 6 are selected as the subgroup bounding boxes of layer group 1 and subgroup 2-2 and subgroup 3-3 covering the ROI. For efficient spatial access, it is assumed that there is no dependency between subgroups from the same layer group. In real-time streaming or low-latency use cases, time efficiency can be improved by performing selection and decoding as each slice segment is received.

[0365] When encoding geometry and / or attributes, the method / apparatus according to an embodiment may present the data as a tree 2200 consisting of layers (which may be referred to as depth, level, etc.). Point cloud data corresponding to the layers (depth / level) may be grouped into layer groups (or groups) 45000. Layer group 2 may be further divided (segmented) into two subgroups 45002, and layer group 3 may be further divided (segmented) into four subgroups 45003. Each subgroup may be configured as a slice to generate a bitstream.

[0366] A receiving device according to an embodiment may receive a bitstream, select a specific slice from the bitstream, and decode a bounding box corresponding to a subgroup included in the selected slice. For example, when slice 1 is selected, a bounding box 45004 corresponding to layer group 1 may be decoded. Layer group 1 may be data corresponding to a maximum area. When a detailed area for layer group 1 is additionally displayed, a method / device according to an embodiment may select slice 3 and / or slice 6, and hierarchically and partially access bounding boxes (point cloud data) of subgroup 2-2 and / or subgroup 3-3 of the detailed area included in the area of ​​layer group 1.

[0367] use Figure 22 The encoding and decoding of point cloud data of layer groups and subgroups can be done by Figure 1 The sending / receiving device Figure 2 Encoding and decoding, Figure 3 A sending device / method, Figure 7 Receiving device / method, Figure 8 and Figure 9 Transmitting / receiving device / method, Figure 10 Devices Figure 30 and Figure 32 Send / receive method, Figure 31 and Figure 33 The sending / receiving device and Figure 40 and Figure 41 The send / receive method is executed.

[0368] Figure 23 (a) to Figure 23 (c) illustrates representation of point cloud data based on layer groups according to an embodiment.

[0369] The device / method according to the embodiment can provide efficient access to large-scale point cloud data or dense point cloud data through layer group slicing based on scalability and spatial accessibility. Due to the large number of points and large size of data, point cloud data may take a lot of time to render or display content. Therefore, as an alternative, the level of detail can be adjusted based on the viewer's interest. For example, when the viewer is away from the scene or object, structural or global area information is more important than local details. On the other hand, when the viewer moves closer to a specific area or object, detailed information about the ROI is required. Using an adaptive method, the renderer according to the embodiment can efficiently provide data with sufficient quality to the viewer. Figure 23 (a) to Fig.23 (c) illustrates an example of increasing detail at three viewing distance levels based on ROI changes.

[0370] Fig.23 The high-level view in (a) shows coarse details, Fig.23 The mid-level view in (b) shows mid-level details, and Fig.23 The low-level view in (c) shows fine-grained details.

[0371] Fig.24 A point cloud data transmitting / receiving apparatus / method according to an embodiment is illustrated.

[0372] When layer group slices are used to generate a G-PCC bitstream, multi-resolution ROIs may be supported.

[0373] Reference Fig.24 , which can support multi-resolution ROIs through the scalability and spatial accessibility of hierarchical slices. Fig.24 In the embodiment, the encoder 47001 on the transmitting side can generate bitstream slices of spatial subgroups of each layer group or bitstream slices of octree layer groups. Upon request, slices matching the ROI of each resolution are selected and sent to the receiving side. Compared with the patch-based method, the size of the entire bitstream is reduced because it does not include details beyond the requested ROI. On the receiving side, the decoder 47004 can combine the slices to produce three outputs, for example, 1) a high-level view output from the layer group; 2) a mid-level view output from selected subgroups of layer group 1 and layer group 2; and 3) a low-level view output with good quality details from layer group 1 and selected subgroups of layer group 2 and layer group 3. The outputs can be generated progressively, so the receiver can provide a viewing experience such as zooming that progressively increases the resolution from a high-level view to a low-level view.

[0374] Encoder 47001 may correspond to a geometry encoder and / or an attribute encoder as a point cloud encoder according to an embodiment. The encoder may slice the point cloud data based on layer groups (or groups). The layers may be referred to as the depth of the tree, the level of LOD, etc. As shown in section 47002, the depth of the octree of the geometry and / or the level of the attribute layer may be divided into layer groups (or subgroups).

[0375] The slice selector 47003 in conjunction with the encoder 47001 can select segmented slices (or sub-slices) and selectively and partially send data such as layer group 1 to layer group 3.

[0376] The decoder 47004 can decode the point cloud data that is selectively and partially transmitted. For example, the decoder 47004 can decode the layer group 1 for the high-level view (which has a high depth / layer / level or index 0, or is close to the root). In addition, decoding can be performed by separately increasing the index of the depth / level on the layer group 1 based on the layer group 1 and the layer group 2 for the mid-level view. In addition, decoding can be performed based on the layer group 1 to the layer group 3 for the low-level view.

[0377] Reference Fig.24 , according to an embodiment, the encoder 47001 can receive point cloud data as input and slice it into layer groups. That is, the point cloud data can be hierarchically structured and divided into layer groups. The hierarchical structure can refer to an octree structure or a level of detail (LoD). Part 47002 represents the point cloud data divided into layer groups. The slice selector 47003 can select a layer group (or a corresponding slice) and send the selected slice to the decoder 47004 on the receiving side. The decoder 47004 can reconstruct only layer group 1, reconstruct layer group 1 and layer group 2, or reconstruct all received layer groups by combining the received slices according to the needs of the user. The layer groups are hierarchical and have different levels of detail. Reconstructing only layer group 1 can result in a wider range of reconstruction and coarse details. Reconstructing all layer groups 1 to 3 can result in a narrower range of reconstruction and fine details.

[0378] Scalable encoding and decoding of geometric data has been described above.

[0379] Depending on the implementation, scalable encoding and decoding may also be applied to attribute data.

[0380] The present disclosure describes a layer-based attribute compression method for improving the transmission efficiency of scalable attribute compression.

[0381] Specifically, the present disclosure proposes a LoD generation method for partial encoding or partial decoding of attribute data.

[0382] More specifically, the present disclosure proposes a method for efficiently performing LoD generation and nearest neighbor (NN) search in the case of missing slices.

[0383] Furthermore, the present disclosure proposes a method for performing NN search and position correction on a per-subgroup basis.

[0384] In the present disclosure, a layer group can be further divided into multiple subgroups. The subgroups according to the embodiment are mutually exclusive, and the set of subgroups can be the same as the layer group. The points in each subgroup are defined in a spatial region, so the subgroup can be indicated by the subgroup bounding box information. In addition, the layer group can be referred to as a subgroup. In the present disclosure, the term subgroup can be used interchangeably with slice or FGS. Therefore, a missing (or disappeared) slice can be referred to as a missing subgroup or a missing FGS.

[0385] For example, a subgroup (or subgroup or subgroup) can be represented as a set of adjacent nodes in a layer group. In this case, the subgroup can be configured as a set of nodes adjacent to each other in Morton code order, a set of nodes adjacent to each other based on distance, or a set of nodes adjacent to each other according to coding order.

[0386] A subgroup may correspond to one LOD, or may correspond to two or more LODs. In addition, a slice may correspond to one LOD, or may correspond to two or more LODs. Therefore, in the present disclosure, a subgroup may be used interchangeably with a slice or FGS. According to an embodiment, the terms layer, depth, level, and depth level may be used interchangeably in the octree structure.

[0387] In the present disclosure, a subgroup may have a parent-child relationship. In other words, a parent subgroup may have one or more sub-subgroups. In addition, sub-subgroups of the same level may be referred to as brother subgroups.

[0388] Furthermore, when octree-based geometry coding proceeds from the root level toward the leaf level using gradual segmentation, the subsampling performed to generate LoD proceeds from the leaf level to the root level. Fig.26 In , for example, the arrow indicates the direction of subsampling. LOD is the level of detail of the point cloud content, where a decrease in the value of LOD indicates a degradation of the detail of the point cloud content, and an increase in the value of LOD indicates an enhancement of the detail of the point cloud content. Fig.26 , LoD N-1 is the lowest level of detail, and LoD N+2 is the highest level of detail.

[0389] In one embodiment, the LoD generation and NN search performed by the transmitting device of the present disclosure is performed by Figure 3 LoD Generator 30009 or Figure 8The prediction / lifting / RAHT transformation processor 8010 is executed. This embodiment is only intended to provide understanding for those skilled in the art, and LoD generation and NN search can be performed by corresponding blocks or modules. In one embodiment, LoD generation and NN search can be performed by Figure 7 LoD Generator 7008 or Fig. 9 The prediction / lifting / RAHT inverse transform processor 9009 is executed in the receiving device of the present disclosure. This embodiment is only intended to provide understanding for those skilled in the art, and LoD generation and NN search can be performed by corresponding boxes or modules.

[0390] This disclosure describes a method for generating LoD when geometric information is encoded (or decoded) based on a geometric tree (e.g., an octree) and there are missing slices (i.e., missing FGS). In addition, a NN search method used in this case is described.

[0391] Fig.25 is a flow chart illustrating an exemplary LoD generation method according to an embodiment.

[0392] exist Fig.25 In the example, it is assumed that the input geometry encoded (or decoded) points are used for attribute encoding (or attribute decoding). In this case, the LoD can be the maximum depth of the geometry tree.

[0393] That is, subsampling for LOD generation is performed (operation 48001). Then, it is determined whether the current LoD is the finest LOD of the current layer group (operation 48002). Here, FGS (or layer group-based slicing) refers to dividing a slice (i.e., a layer group) into a plurality of slices (i.e., subgroups). In operation 48002, it is determined whether the current LoD is the bottom in the layer group.

[0394] When the LoD is not the finest LOD of the current layer group, operation 48001 is performed. When the LoD is the finest LOD of the current layer group, operation 48003 is performed to determine whether a sub-FGS (also called a sub-slice or a sub-sub-group) is lost (operation 48003). That is, it is determined whether there is a missing FGS (or slice or sub-group).

[0395] When it is determined in operation 48003 that there is no missing FGS, operation 48001 is performed. When it is determined that there is a missing FGS, the missing node is detected (operation 48004). Then, an alternative node for the detected missing node is added to the corresponding LOD (operation 48005). In other words, if there is a missing FGS, the node required for LoD generation is searched.

[0396] Then, it is determined whether the end of the LoD has been reached (operation 48006). When it is determined that the end of the LoD has not been reached, operation 48001 is performed. When it is determined that the end of the LoD has been reached, LOD generation is terminated, and NN search is performed based on the generated LoD.

[0397] Operation 48003 and operation 48004 are described in detail below.

[0398] As described above, the point cloud data sending method / apparatus can send a bit stream containing point cloud data in a scalable manner on a per-slice (or subgroup) basis, and the point cloud data receiving method / apparatus according to an embodiment can receive and decode the bit stream in a scalable manner on a per-slice (or subgroup) basis.

[0399] Therefore, in the decoding operation, the decoder on the receiving side may have all the slices or may have some missing slices. For example, when the decoder performs partial decoding, some slices (or subgroups or FGS) may not exist. These slices may be referred to as missing (ie, lost) slices (or subgroups or FGS).

[0400] Therefore, the encoder on the transmission side according to the present disclosure generates LoD in consideration of this situation.

[0401] Fig.26 is a diagram illustrating an exemplary LoD generation method used when there is a missing FGS (ie, a missing subgroup).

[0402] exist Fig.26 , 48010 represents a parent subgroup, and 48021 and 48025 represent sub-subgroups of the parent subgroup 48010. For simplicity, 48021 will be referred to as the first sub-subgroup, and 48025 will be referred to as the second sub-subgroup. It is assumed that the second sub-subgroup 48025 is missing.

[0403] As described above, subsampling is performed from the leaf level to the root level. Therefore, in the subsampling, the nodes in the portion 48022-1 in the layer LoD N+2 in the first sub-subgroup 48021 are sibling nodes, and one of the three sibling nodes is selected as the parent node 48022. Similarly, in the right portion, one of the four sibling nodes is selected as the parent node. Then, one of the two parent nodes in the layer LoD N+1 is selected as the parent node 48020 of the parent subgroup 48010. In other words, the node 48020 in the parent subgroup 48010 is a node generated by subsampling the node in the sub-subgroup in the lower layer.

[0404] However, the second sub-group 48025 is missing. Therefore, nodes 48023 and 48024 in the parent sub-group 48010 may not be generated by subsampling the nodes in the sub-group in the lower layer. In other words, nodes 48023 and 48024 are missing nodes in the parent sub-group, which may not be generated by subsampling because the sub-group is missing.

[0405] In one embodiment, when a child FGS is missing or lost, the missing node may be detected at the finest LoD of the parent child group (i.e., the bottom LoD of the parent child group). For this process, the temporary (or temporary) output node of the geometry FGS corresponding to the parent child group of the current attribute may be used as a reference node list (see operation 48004). In other words, in one embodiment, when there is a missing subgroup (or FGS or slice) associated with attribute information, the corresponding subgroup of geometry (or FGS of geometry) may be used to detect at least one missing node of the parent child group. For this purpose, it is assumed in the present disclosure that the layer group structure and / or subgroup structure is the same for geometry and attributes.

[0406] According to an embodiment, after comparing the subsampled nodes from the current child subgroup and the temporary geometry slice output nodes, the nodes located in the geometry output but not in the subsampled output are determined to be missing nodes, and the nodes of the geometry corresponding to the missing nodes are appended to the LoD of the current subgroup. In other words, LoD N is generated by the subsampled nodes from LoD N+1 and / or the missing nodes of the current layer group. The missing nodes of the current layer group are detected from the nodes of the corresponding geometry layer group and appended to the LoD. In other words, LoD N is generated using the subsampled nodes from the child subgroup of the current attribute and / or the nodes from the geometry parent subgroup corresponding to the parent subgroup of the current attribute.

[0407] In the present disclosure, the subsampled nodes from LoD N+1 are indexed to LoD N by the octree-based subsampling process in G-PCCv1. Then, the missing nodes caused by the missing sub-FGS are detected from the geometry FGS temporary output. The detected missing nodes are appended to LoD N. In the present disclosure, it is assumed that the LOD of the attribute corresponds to the octree level of the geometry. In other words, the LOD N of the attribute corresponds to the octree level N of the geometry. Therefore, in the present disclosure, one of the nodes in the octree N of the geometry is set to the missing node in the LOD N of the attribute.

[0408] Therefore, since the missing subgroup of the attribute exists in the layer group of the geometry, the missing nodes of the parent subgroup (LoD N+1) of the missing subgroup (LoD N) among the nodes of the corresponding subgroup of the geometry are used to generate LoD N. In other words, the nodes from the corresponding subgroup of the geometry are appended to LoD N.

[0409] In other words, LoD N of an attribute may include at least one sub-sampled node from LoD N+1 and / or at least one node from a corresponding sub-group of the geometry (ie, a missing node).

[0410] The code below is about detecting missing nodes in a subgroup of attributes to generate a LoD N and appending the nodes corresponding to the missing nodes from the corresponding subgroup of geometry to the LoD N.

[0411] void

[0412] PCCTMC3Decoder3::decodeAttributeBrick(const PayloadBuffer&buf)

[0413] {

[0414] assert(buf.type==PayloadType::kAttributeBrick);

[0415] / / todo(df): replace assertions with error handling

[0416] assert(_sps);

[0417] assert(_gps);

[0418] / / Verify that it corresponds to the correct geometry slice

[0419] AttributeBrickHeader abh=parseAbhIds(buf);

[0420] assert(abh.attr_geom_slice_id==_sliceId);

[0421] / / todo(df): Verify that sps activation is not changed by APS

[0422] const auto it_attr_aps=_apss.find(abh.attr_attr_parameter_set_id);

[0423] assert(it_attr_aps!=_apss.cend());

[0424] const auto&attr_aps=it_attr_aps->second;

[0425] assert(abh.attr_sps_attr_idx<_sps->attributeSets.size());

[0426] const auto&attr_sps=_sps->attributeSets[abh.attr_sps_attr_idx];

[0427] const auto&label=attr_sps.attributeLabel;

[0428] / / Sanity check for loss detection

[0429] if(_gbh.entropy_continuation_flag)

[0430] assert(_gbh.prev_slice_id==_ctxtMemAttrSliceIds[abh.attr_sps_attr_idx]);

[0431] / / Make sure the context array is an allocated context array

[0432] / / todo(df): move it to sps to activate

[0433] _ctxtMemAttrSliceIds.resize(_sps->attributeSets.size());

[0434] _ctxtMemAttrs.resize(_sps->attributeSets.size());

[0435] / / To make sure the attribute decoder is reusable,

[0436] / / abh must be checked.

[0437] int abhSize;

[0438] abh=parseAbh(*_sps,attr_aps,buf,&abhSize);

[0439] if(_sps->layer_group_enabled_flag){

[0440] int layerIdx = 0;

[0441] for(int i=0;i<=_layerGroupParams.numLayerGroupsMinus1;i++){

[0442] _layerGroupParams.subgrpBboxOrigin[i].resize(_layerGroupParams.numSubgroupsMinus1[i]+1);

[0443] _layerGroupParams.subgrpBboxSize[i].resize(_layerGroupParams.numSubgroupsMinus1[i]+1);

[0444] _layerGroupParams.numNodes[i].resize(_layerGroupParams.numSubgroupsMinus1[i]+1);

[0445] _layerGroupParams.refLayerGroupId[i].resize(_layerGroupParams.numSubgroupsMinus1[i]+1);

[0446] _layerGroupParams.refSubgroupId[i].resize(_layerGroupParams.numSubgroupsMinus1[i]+1);

[0447] _layerGroupParams.parentSubgroupId[i].resize(_layerGroupParams.numSubgroupsMinus1[i]+1);

[0448] for(int j=0;j<_layerGroupParams.numLayersPerLayerGroup[i];j++,layerIdx++)

[0449] _layerGroupParams.dcmNodesIdx[layerIdx].resize(_layerGroupParams.numSubgroupsMinus1[i]+1);

[0450] }

[0451] _abh = abh;

[0452] _layerGroupParams.rootNodeSizeLog2=_sps->root_node_size_log2;

[0453] int numCodedLayers = 0;

[0454] int numCodedLayerGroups=_sps->num_layer_groups_minus1-_params.numSkipLayerGroups;

[0455] for(int i=0;i<=numCodedLayerGroups;i++){

[0456] numCodedLayers+=_layerGroupParams.numLayersPerLayerGroup[i];

[0457] }

[0458] _layerGroupParams.rootNodeSizeLog2_coded=numCodedLayers;

[0459] }

[0460] pcc::chrono::Stopwatch<pcc::chrono::utime_inc_children_clock> clock_user;

[0461] / / If it is not compatible, please replace the attribute decoder

[0462] if(!_attrDecoder||!_attrDecoder->isReusable(attr_aps,abh))

[0463] _attrDecoder=makeAttributeDecoder();

[0464] clock_user.start();

[0465] / / Convert Cartesian positions to spherical for use in attribute encoding.

[0466] / / NOTE: This preserves the original Cartesian position so it can be restored later

[0467] std::vector<pcc::point_t> altPositions;

[0468] if(attr_aps.spherical_coord_flag

[0469] &! _sps->layer_group_enabled_flag){

[0470] / / If predgeom is used, reuse the internal position instead

[0471] / / Recalculate.

[0472] Box3 <int>bboxRpl;

[0473] if(_gps->predgeom_enabled_flag){

[0474] altPositions=_posSph;

[0475] bboxRpl=Box3 <int>(altPositions.begin(),altPositions.end());

[0476] }else{

[0477] altPositions.resize(_currentPointCloud.getPointCount());

[0478] auto laserOrigin=_gbh.geomAngularOrigin(*_gps);

[0479] bboxRpl=convertXyzToRpl(

[0480] laserOrigin,_gps->angularTheta.data(),_gps->angularTheta.size(),

[0481] &_currentPointCloud[0],

[0482] &_currentPointCloud[0]+_currentPointCloud.getPointCount(),

[0483] altPositions.data());

[0484] }

[0485] offsetAndScale(

[0486] bboxRpl.min,attr_aps.attr_coord_scale,altPositions.data(),

[0487] altPositions.data()+altPositions.size());

[0488] _currentPointCloud.swapPoints(altPositions);

[0489] }

[0490] / / The portion not included in the current subgroup used to select the nodes belonging to the parent subgroup boolsliceSelectionEnabled = 0;

[0491] int maxTempNodes = 0;

[0492] for(int i=0;i<_layerGroupParams.sliceSelectionIndicationFlag.size();i++)

[0493] for(int j=0;j<_layerGroupParams.sliceSelectionIndicationFlag[i].size();j++){if(_layerGroupParams.sliceSelectionIndicationFlag[i][j]){

[0494] int curLayerGroup=i;

[0495] int curSubgroup=j;

[0496] maxTempNodes+=_layerGroupParams.pointCloud_tempNodes[curLayerGroup][curSubgroup].getPointCount();

[0497] }

[0498] }

[0499] if(maxTempNodes)

[0500] sliceSelectionEnabled=true;

[0501] PCCPointSet3 pointCloudWithTempNodes;

[0502] if(maxTempNodes){

[0503] int pointCount=_currentPointCloud.getPointCount();

[0504] _layerGroupParams.pointCount_output=pointCount;

[0505] PCCPointSet3 tempNodes;

[0506] tempNodes=_currentPointCloud;

[0507] tempNodes.resize(0);

[0508] tempNodes.resize(maxTempNodes);

[0509] int addPointCount=0;

[0510] int accLayers=0;

[0511] int maxLayers=_layerGroupParams.rootNodeSizeLog2.max();

[0512] for(int i=0;i<_layerGroupParams.numLayerGroupsMinus1;i++){

[0513] accLayers+=_layerGroupParams.numLayersPerLayerGroup[i];

[0514] int shift=maxLayers-accLayers;

[0515] for(int j=0;j<=_layerGroupParams.numSubgroupsMinus1[i];j++){

[0516] if(_layerGroupParams.sliceSelectionIndicationFlag[i][j]){

[0517] int curLayerGroup=i;

[0518] int curSubgroup=j;

[0519] int child_LayerGroup=curLayerGroup+1;

[0520] auto parentNodes=_layerGroupParams.pointCloud_tempNodes[curLayerGroup][curSubgroup];

[0521] for(int m=0;m<parentNodes.getPointCount();m++){

[0522] auto pos=parentNodes[m]< <shift;

[0523] bool alreadyPresentFlag=false;

[0524] for(int k=0;k<=_layerGroupParams.numSubgroupsMinus1[child_LayerGroup];k++){

[0525] if(_layerGroupParams.sliceSelectionIndicationFlag[child_LayerGroup][k]){

[0526] int child_subgroup = k;

[0527] auto bbox_min=_layerGroupParams.subgrpBboxOrigin[child_LayerGroup][child_Subgroup];

[0528] auto bbox_max=bbox_min+_layerGroupParams.subgrpBboxSize[child_LayerGroup][child_Subgroup];

[0529] / / If the position (pos) of a node belonging to a parent child group falls within the bounding box of the current child group, it is indicated by alreadyPresentFlag and is not included in the input (tempNodes) of LoD generation.

[0530] if(pos.x()>=bbox_min.x()&&pos.x() <bbox_max.x()

[0531] &&pos.y()>=bbox_min.y()&&pos.y() <bbox_max.y()

[0532] &&pos.z()>=bbox_min.z()&&pos.z() <bbox_max.z()){

[0533] alreadyPresentFlag = true;

[0534] break;

[0535] }

[0536] }

[0537] }

[0538] if (!alreadyPresentFlag)

[0539] tempNodes[addPointCount++] = pos;

[0540] }

[0541] }

[0542] }

[0543] }

[0544] tempNodes.resize(addPointCount);

[0545] pointCloudWithTempNodes = _currentPointCloud;

[0546] pointCloudWithTempNodes.resize(pointCount + addPointCount);

[0547] for (int i = 0; i < addPointCount; i++)

[0548] pointCloudWithTempNodes[pointCount + i] = tempNodes[i];

[0549] std::swap(_currentPointCloud, pointCloudWithTempNodes);

[0550] pointCloudWithTempNodes.clear();

[0551] }

[0552] / / Function to perform decoding

[0553] auto& ctxtMemAttr = _ctxtMemAttrs.at(abh.attr_sps_attr_idx);

[0554] _attrDecoder->decode(

[0555] *_sps,attr_sps,attr_aps,abh,_gbh.footer.geom_num_points_minus1,

[0556] _params.minGeomNodeSizeLog2,buf.data()+abhSize,buf.size()-abhSize,

[0557] ctxtMemAttr,_currentPointCloud,_layerGroupParams,0,0);

[0558] if(_sps->layer_group_enabled_flag){

[0559] / / Part related to storing context memory to achieve continuity between parent-child subgroups

[0560] if(_ctxtMemSavedAttr.size()==0)

[0561] _ctxtMemSavedAttr.resize(_sps->attributeSets.size());

[0562] _ctxtMemSavedAttr[abh.attr_sps_attr_idx].push_back(ctxtMemAttr);

[0563] _refIdxToSavedArrayIdxAttr.resize(_sps->num_layer_groups_minus1+1);

[0564] _refIdxToSavedArrayIdxAttr[0].push_back(0);

[0565] _accNumSubgroupsPerLayerGroupAttr.resize(_sps->num_layer_groups_minus1+1);

[0566] _accNumSubgroupsPerLayerGroupAttr[0]=1;

[0567] }

[0568] if(attr_aps.spherical_coord_flag

[0569] &&! _sps->layer_group_enabled_flag)

[0570] _currentPointCloud.swapPoints(altPositions);

[0571] / / Note the current sliceID for loss detection

[0572] _ctxtMemAttrSliceIds[abh.attr_sps_attr_idx]=_sliceId;

[0573] clock_user.stop();

[0574] std::cout< <label<<"s bitstream size"<<buf.size()<<"B\n";

[0575] auto total_user=

[0576] std::chrono::duration_cast <std::chrono::milliseconds>(clock_user.count());

[0577] std::cout<

[0578] <<"s processing time(user):"< <total_user.count() / 1000.0

[0579] <<"s\n";

[0580] std::cout< <std::endl;

[0581] }

[0582] / / --------------------------------------------------------------------------

[0583] Next, a process of performing a nearest neighbor (NN) search based on the LoD generated as described above is described below.

[0584] In other words, according to the present disclosure, when performing an NN search, only points located in the same layer group (or subgroup or slice) are used to search the NN of a point. This is because, as described above, there may be missing slices (or missing FGS or missing subgroups). For example, a slice may be missing because it was not sent to the receiving side during scalable transmission. And, when there are missing slices, nodes in the missing slices cannot be referenced during the NN search. Therefore, in one embodiment, an NN search can be performed by referring only to the nodes in the subgroup to which the current point to be decoded (or encoded) belongs.

[0585] Fig. 27 is a diagram illustrating an exemplary method of searching for nearest neighbors according to an embodiment.

[0586] In the NN search, the index subgroup includes at least one LOD to which the point to be encoded (or decoded) belongs, and the reserved subgroup includes at least one lower LOD based on the at least one LOD included in the index subgroup. According to an embodiment, the points of at least one LOD in the index subgroup are sorted in ascending order based on the size of the Morton code, and the points of at least one LoD in the reserved subgroup are also sorted in ascending order based on the size of the Morton code. In one embodiment, the NN of the current point in the index subgroup (i.e., the point to be encoded or decoded) is searched among the points belonging to at least one LOD in the reserved subgroup and / or the points belonging to at least one LOD in the index subgroup.

[0587] exist Fig. 27 ​48030 is the current point to be encoded (or decoded) in the index subgroup. In one embodiment, when performing the NN search, the NN of the current point 48030 may be searched among the points belonging to at least one LOD in the reserved subgroup in the subgroup 48033 to which the current point 48030 belongs and / or the points belonging to the subgroup to which the current point 48030 belongs. In other words, when searching for the NN of the current point 48030, the points belonging to another subgroup 48031 to which the current point 48030 does not belong are not referenced.

[0588] In other words, when performing the NN search, a point (i.e., a node) outside the boundary of the subgroup to which the current point 48030 belongs (i.e., outside the boundary of the subgroup to which the current point 48030 belongs) that searches for at least one neighbor node is not considered as a neighbor of the current point 48030. This is because the subgroup 48031 may be missing.

[0589] Therefore, in the present disclosure, searching for the NN of point 48030 belonging to the current LoD includes searching for prediction candidates among neighbor nodes selected from at least one LoD higher than the current LoD and the encoded nodes in the current LoD. Specifically, in one embodiment, for fast search, the search range is defined by a cubic boundary and the number of points in the Morton coding order. In this regard, when the neighbor candidate is in the same layer group as the current node, the present disclosure proposes an additional constraint that considers the subgroup boundary (referred to as the intra-layer group search boundary). In other words, the search for the NN of the current point is performed only in the subgroup to which the current point belongs, which is referred to as an intra-layer group search.

[0590] Search boundary within a layer group: When neighbor candidates are in the same layer group as the current node, restrict the use of neighbor nodes in the same subgroup boundary.

[0591] In other words, in the present disclosure, neighbor nodes 48037 and 48038 outside the subgroup bounding box 48033 cannot be neighbor candidates (ie, neighbor nodes or neighbor points) of the current node 48030 .

[0592] In addition, in the present disclosure, the layer group adaptation position can be used for subgroup estimation and calculation of neighbor distances. Considering the case of missing sub-subgroups, this can prevent mismatch between the encoder and the decoder.

[0593] In other words, for scalable transmission, the attribute encoder creates a downsampled geometric mesh by performing sampling on the remaining layers except for one or more skipped layers, and corrects the positions of occupied nodes including neighbor search target nodes by performing upsampling based on information related to the skipped layers. Then, NN search is performed based on the corrected positions to obtain the distance between the nodes. Then, attribute prediction and compression can be performed based on the obtained NN nodes.

[0594] According to an embodiment, the layer group adaptation position is the encoded position, which is right-shifted by the number of LoDs in the encoded descendant (e.g., child) sub-groups, and then left-shifted by the number of LODs in the encoded descendant sub-groups and skipped descendant sub-groups.

[0595] The following code shows an example of generating LoD and performing NN search according to the present disclosure. Specifically, as a preprocessing for NN search, it shows an example of dividing nodes based on the parent-child group boundary for the parent LoD and storing them in the reserved subgroup. It also shows an example of including IDCM nodes belonging to the subgroup used as an ancestor (i.e., parent) as a target for NN search. In addition, it shows an example of dividing nodes according to the subgroup for the current LoD and storing them in the index subgroup.

[0596] Then, according to the present disclosure, an NN search is performed for each subgroup. In other words, an NN search is performed using the subgroup of the parent LoD related to the subgroup of the current LoD. In this case, the NN search for the current point is performed within the subgroup to which the current point belongs, and nodes in the subgroup to which the current point does not belong are not referenced.

[0597] / / LoD generation function

[0598] inline void

[0599] buildPredictorsFast(

[0600] const AttributeParameterSet&aps,

[0601] const AttributeBrickHeader&abh,

[0602] const PCCPointSet3&pointCloud,

[0603] int32_t minGeomNodeSizeLog2,

[0604] int geom_num_points_minus1,

[0605] std::vector <pccpredictor>&predictors,

[0606] std::vector<uint32_t>&numberOfPointsPerLevelOfDetail,

[0607] std::vector<uint32_t>&indexes,

[0608] LayerGroupSlicingParams&layerGroupParams)

[0609] {

[0610] const int32_t pointCount=int32_t(pointCloud.getPointCount());

[0611] assert(pointCount);

[0612] std::vector <mortoncodewithindex>packedVoxel;

[0613] computeMortonCodesUnsorted(pointCloud,aps.lodNeighBias,packedVoxel);

[0614] if(!aps.canonical_point_order_flag)

[0615] std::sort(packedVoxel.begin(),packedVoxel.end());

[0616] std::vector<uint32_t>retained,input,pointIndexToPredictorIndex;

[0617] pointIndexToPredictorIndex.resize(pointCount);

[0618] retained.reserve(pointCount);

[0619] std::vector<uint32_t>pointIdxToPackedVoxelIdx;

[0620] std::vector<uint32_t>dcmNodesList;

[0621] if(layerGroupParams.layerGroupEnabledFlag){

[0622] pointIdxToPackedVoxelIdx.resize(pointCount);

[0623] for(uint32_t i=0;i<pointCount;i++){

[0624] auto pointIdx=packedVoxel[i].index;

[0625] pointIdxToPackedVoxelIdx[pointIdx]=i;

[0626] }

[0627] / / Preprocessing to manage IDCM nodes and non-IDCM nodes separately / / All IDCM nodes are collected in dcmNodesList. int layerIdx = 0;

[0628] int dcmNodesCount = 0;

[0629] int maxNumLayer = 0;

[0630] for(int i=0; i<=layerGroupParams.numLayerGroupsMinus1;++)

[0631] maxNumLayer+=layerGroupParams.numLayersPerLayerGroup[i];

[0632] for(int i=0;i<=layerGroupParams.numLayerGroupsMinus1;++){

[0633] for(int j=0; j <layerGroupParams.numLayersPerLayerGroup[i];j++,layerIdx++){

[0634] for(int sbgrIdx=0; sbgrIdx<=layerGroupParams.numSubgroupsMinus1[i]; bgrIdx++){

[0635] if(layerGroupParams.dcmNodesIdx[layerIdx][sbgrIdx].size()&&layerIdx <maxNumLayer-1){for(int m=0;m<layerGroupParams.dcmNodesIdx[layerIdx][sbgrIdx].size();m++){

[0636] auto pointIdx=layerGroupParams.dcmNodesIdx[layerIdx][sbgrIdx][m];

[0637] auto packedVoxelIndex=pointIdxToPackedVoxelIdx[pointIdx];

[0638] dcmNodesList.push_back(packedVoxelIndex);

[0639] }

[0640] }

[0641] }

[0642] }

[0643] }

[0644] / / Separate IDCM nodes from non-IDCM nodes

[0645] if(dcmNodesList.size()){

[0646] std::sort(dcmNodesList.begin(),dcmNodesList.end());

[0647] input.resize(pointCount-dcmNodesList.size());

[0648] int dcmIdx = 0;

[0649] int nonIdcmIdx = 0;

[0650] for(uint32_t i=0; i <pointCount;++i){

[0651] auto packedVoxelIndex=dcmNodesList[dcmIdx];

[0652] if(packedVoxel[packedVoxelIndex].mortonCode==packedVoxel[i].mortonCode)

[0653] dcmIdx++; / / IDCM node

[0654] else{

[0655] input[nonIdcmIdx++]=i; / / non-IDCM node

[0656] }

[0657] }

[0658] }

[0659] else{

[0660] input.resize(pointCount);

[0661] for(uint32_t i = 0; i < pointCount; ++i)

[0662] input[i] = i;

[0663] }

[0664] }

[0665] / / Prepare the output buffer

[0666] predictors.resize(pointCount);

[0667] numberOfPointsPerLevelOfDetail.resize(0);

[0668] indexes.resize(0);

[0669] indexes.reserve(pointCount);

[0670] numberOfPointsPerLevelOfDetail.reserve(21);

[0671] numberOfPointsPerLevelOfDetail.push_back(pointCount);

[0672] bool concatenateLayers = aps.scalable_lifting_enabled_flag;

[0673] if(layerGroupParams.layerGroupEnabledFlag)

[0674] concatenateLayers = false;

[0675] std::vector<uint32_t> indexesOfSubsample;

[0676] if(concatenateLayers)

[0677] indexesOfSubsample.reserve(pointCount);

[0678] std::vector<Box3<int32_t>> bBoxes;

[0679] const int32_t log2CubeSize = 7;

[0680] MortonIndexMap3d atlas;

[0681] atlas.resize(log2CubeSize);

[0682] atlas.init();

[0683] auto maxNumDetailLevels = aps.maxNumDetailLevels();

[0684] int32_t predIndex = int32_t(pointCount);

[0685] int treeLvlGap = 0;

[0686] int maxNumLayers = 0;

[0687] if (layerGroupParams.layerGroupEnabledFlag)

[0688] treeLvlGap = layerGroupParams.rootNodeSizeLog2.max() - layerGroupParams.rootNodeSizeLog2_coded.max();

[0689] for (int m = 0; m <= layerGroupParams.numLayerGroupsMinus1; m++)

[0690] maxNumLayers += layerGroupParams.numLayersPerLayerGroup[m];

[0691] maxNumDetailLevels = maxNumLayers + 1;

[0692] }

[0693] int maxLoD = 0;

[0694] / / Subsample at lower LoD to construct higher LoD

[0695] for(auto lodIndex=minGeomNodeSizeLog2;

[0696] !input.empty()&&lodIndex<maxNumDetailLevels;++lodIndex){

[0697] const int32_t startIndex=indexes.size();

[0698] if(lodIndex==maxNumDetailLevels-1){

[0699] for(const auto index:input){

[0700] indexes.push_back(index);

[0701] }

[0702] }else{

[0703] subsample(

[0704] aps,abh,pointCloud,packedVoxel,input,lodIndex,retained,indexes,

[0705] atlas,layerGroupParams);

[0706] }

[0707] const int32_t endIndex=indexes.size();

[0708] int curLayerGroup=0;

[0709] int curSubgroup=0;

[0710] int prtLayerGroup=0;

[0711] int prtSubgroup=0;

[0712] std::vector<std::vector<uint32_t>>retained_subgroup,indexes_subgroup;

[0713] if(layerGroupParams.layerGroupEnabledFlag){

[0714] int layerIdx=maxNumLayers-lodIndex;

[0715] int accNumLayers=0;

[0716] for(int m=0;m<=layerGroupParams.numLayerGroupsMinus1;m++){

[0717] accNumLayers+=layerGroupParams.numLayersPerLayerGroup[m];

[0718] if(accNumLayers>=layerIdx){

[0719] curLayerGroup=m;

[0720] break;

[0721] }

[0722] }

[0723] if(accNumLayers-layerGroupParams.numLayersPerLayerGroup[curLayerGroup]+1==layerIdx&&curLayerGroup>0)

[0724] prtLayerGroup=curLayerGroup-1;

[0725] else

[0726] prtLayerGroup=curLayerGroup;

[0727] retained_subgroup.clear();

[0728] indexes_subgroup.clear();

[0729] retained_subgroup.resize(layerGroupParams.numSubgroupsMinus1[prtLayerGroup]+1);

[0730] indexes_subgroup.resize(layerGroupParams.numSubgroupsMinus1[curLayerGroup]+1);

[0731] std::vector <std::vector<uint32_t> >dcmNodeList_ParentSubgroup;

[0732] int layerIdx_minus1=layerIdx-1;

[0733] if(dcmNodesList.size()){

[0734] / / The process of obtaining the set of IDCM nodes belonging to the parent-child group

[0735] dcmNodeList_ParentSubgroup.resize(layerGroupParams.numSubgroupsMinus1[prtLayerGroup]+1);

[0736] int lyrGrpIdx = 0;

[0737] if(layerIdx_minus1>0){

[0738] for(int curLodIndex=0; curLodIndex <layerIdx_minus1;curLodIndex++){

[0739] lyrGrpIdx = 0;

[0740] int accNumLayers = 0;

[0741] for(int m=0; m<=layerGroupParams.numLayerGroupsMinus1; m++){

[0742] accNumLayers+=layerGroupParams.numLayersPerLayerGroup[m];

[0743] if(accNumLayers>curLodIndex){

[0744] lyrGrpIdx=m;

[0745] break;

[0746] }

[0747] }

[0748] for(int m=0;m<=layerGroupParams.numSubgroupsMinus1[lyrGrpIdx];m++){

[0749] if layerGroupParams.sliceSelectionIndicationFlag[lyrGrpIdx][m]){

[0750] auto bbox_min=layerGroupParams.subgrpBboxOrigin[lyrGrpIdx][m];

[0751] auto bbox_max=bbox_min+layerGroupParams.subgrpBboxSize[lyrGrpIdx][m];

[0752] for(int subgrpIdx=0;subgrpIdx<=layerGroupParams.numSubgroupsMinus1[prtLayerGroup];

[0753] subgrpIdx++){

[0754] auto retained_subgrp_bbox_min=layerGroupParams.subgrpBboxOrigin[prtLayerGroup][subgrpIdx];

[0755] auto retained_subgrp_bbox_max=retained_subgrp_bbox_min+layerGroupParams.subgrpBboxSize[prtLayerGroup][subgrpIdx];

[0756] if(retained_subgrp_bbox_min[0]>=bbox_min[0]&&retained_subgrp_bbox_max[0]<=bbox_max[0]

[0757] &&retained_subgrp_bbox_min[1]>=bbox_min[1]&&retained_subgrp_bbox_max[1]<=bbox_max[1]

[0758] &&retained_subgrp_bbox_min[2]>=bbox_min[2]&&retained_subgrp_bbox_max[2]<=bbox_max[2]){

[0759] for(int k=0;k <layerGroupParams.dcmNodesIdx[curLodIndex][m].size();k++){

[0760] auto pointCloudIndex=layerGroupParams.dcmNodesIdx[curLodIndex][m][k];

[0761] auto packedVoxelIndex=pointIdxToPackedVoxelIdx[pointCloudIndex];

[0762] dcmNodeList_ParentSubgroup[subgrpIdx].push_back(packedVoxelIndex);

[0763] }

[0764] }

[0765] }

[0766] }

[0767] }

[0768] }

[0769] }

[0770] }

[0771] / / NN search preprocessing: divide nodes according to the parent-child group boundary for the parent LoD and store them in retained_subgroup.

[0772] for(int i=0; i <retained.size();i++){

[0773] auto pos=(pointCloud[packedVoxel[retained[i]].index]>>lodIndex)<<(lodIndex+treeLvlGap);

[0774] prtSubgroup=0;

[0775] for(int m=0;m<=layerGroupParams.numSubgroupsMinus1[prtLayerGroup];m++){

[0776] if(layerGroupParams.sliceSelectionIndicationFlag[prtLayerGroup][m]){

[0777] auto bbox_min=layerGroupParams.subgrpBboxOrigin[prtLayerGroup][m];

[0778] auto bbox_max=bbox_min+layerGroupParams.subgrpBboxSize[prtLayerGroup][m];

[0779] if(pos.x()>=bbox_min[0]&&pos.x()<bbox_max[0]

[0780] &&pos.y()>=bbox_min[1]&&pos.y()<bbox_max[1]

[0781] &&pos.z()>=bbox_min[2]&&pos.z()<bbox_max[2]){

[0782] prtSubgroup=m;

[0783] break;

[0784] }

[0785] }

[0786] }

[0787] retained_subgroup[prtSubgroup].push_back(retained[i]);

[0788] }

[0789] / / NN search pre-processing: IDCM nodes belonging to the subgroup used as ancestors are included as targets for NN search.

[0790] if(dcmNodesList.size()){

[0791] for(int i=0;i<=layerGroupParams.numSubgroupsMinus1[prtLayerGroup];i++){

[0792] if(layerGroupParams.sliceSelectionIndicationFlag[prtLayerGroup][i]

[0793] &&dcmNodeList_ParentSubgroup[i].size()){

[0794] for(int j=0; j <dcmNodeList_ParentSubgroup[i].size();j++)

[0795] retained_subgroup[i].push_back(dcmNodeList_ParentSubgroup[i][j]);

[0796] std::sort(retained_subgroup[i].begin(),retained_subgroup[i].end());

[0797] }

[0798] }

[0799] if (layerIdx>=2){

[0800] auto prev=retained.size();

[0801] int parentLayerIndex=layerIdx-2;

[0802] for(int i=0;i<=layerGroupParams.numSubgroupsMinus1[prtLayerGroup];i++){

[0803] if(layerGroupParams.sliceSelectionIndicationFlag[prtLayerGroup][i]){

[0804] for(int k=0;k <layerGroupParams.dcmNodesIdx[parentLayerIndex][i].size();k++){

[0805] auto pointCloudIndex=layerGroupParams.dcmNodesIdx[parentLayerIndex][i][k];

[0806] auto packedVoxelIndex=pointIdxToPackedVoxelIdx[pointCloudIndex];

[0807] retained.push_back(packedVoxelIndex);

[0808] }

[0809] }

[0810] }

[0811] std::sort(retained.begin(),retained.end());

[0812] }

[0813] }

[0814] / / NN search preprocessing: nodes are divided according to subgroups for the current LoD and stored in indexes_subgroup. for(int i = startIndex; i <endIndex;i++){

[0815] auto pos=(pointCloud[packedVoxel[indexes[i]].index]>>lodIndex)<<(lodIndex+treeLvlGap); curSubgroup=0;

[0816] for(int m=0; m<=layerGroupParams.numSubgroupsMinus1[curLayerGroup];++){

[0817] if(layerGroupParams.sliceSelectionIndicationFlag[curLayerGroup][m]){

[0818] auto bbox_min=layerGroupParams.subgrpBboxOrigin[curLayerGroup][m];

[0819] auto bbox_max=bbox_min+layerGroupParams.subgrpBboxSize[curLayerGroup][m];if(pos.x()>=bbox_min[0]&&pos.x()<bbox_max[0]

[0820] &&pos.y()>=bbox_min[1]&&pos.y()<bbox_max[1]

[0821] &&pos.z()>=bbox_min[2]&&pos.z()<bbox_max[2]){

[0822] curSubgroup=m;

[0823] break;

[0824] }

[0825] }

[0826] }

[0827] if(layerGroupParams.sliceSelectionIndicationFlag[curLayerGroup][curSubgroup])

[0828] indexes_subgroup[curSubgroup].push_back(indexes[i]);

[0829] }

[0830] }

[0831] if(concatenateLayers&&!layerGroupParams.layerGroupEnabledFlag){

[0832] indexesOfSubsample.resize(endIndex);

[0833] if (startIndex != endIndex) {

[0834] for (int32_t i = startIndex; i < endIndex; i++)

[0835] indexesOfSubsample[i] = indexes[i];

[0836] int32_t numOfPointInSkipped = geom_num_points_minus1 + 1 - pointCount;

[0837] if (endIndex - startIndex <= startIndex + numOfPointInSkipped) {

[0838] concatenateLayers = false;

[0839] } else {

[0840] for (int32_t i = 0; i < startIndex; i++)

[0841] indexes[i] = indexesOfSubsample[i];

[0842] / / Reset predIndex

[0843] predIndex = pointCount;

[0844] for (int lod = 0; lod < lodIndex - minGeomNodeSizeLog2; lod++) {

[0845] int divided_startIndex =

[0846] pointCount - numberOfPointsPerLevelOfDetail[lod];

[0847] int divided_endIndex =

[0848] pointCount - numberOfPointsPerLevelOfDetail[lod + 1];

[0849] computeNearestNeighbors(

[0850] aps,abh,packedVoxel,retained,divided_startIndex,

[0851] divided_endIndex,lod+minGeomNodeSizeLog2,indexes,predictors,pointIndexToPredictorIndex,predIndex,atlas,layerGroupParams,curLayerGroup,curSubgroup);}

[0852] }

[0853] }

[0854] }

[0855] / / Perform a NN search for each subgroup. Use the subgroups of the parent LoD that are related to the subgroup of the current LoD.

[0856] nt numIndexedNodes = 0;

[0857] if(layerGroupParams.layerGroupEnabledFlag){

[0858] auto indexes_tmp = indexes;

[0859] indexes.resize(startIndex);

[0860] int count = 0;

[0861] for(int i=0;i<=layerGroupParams.numSubgroupsMinus1[curLayerGroup];i++){

[0862] if(layerGroupParams.sliceSelectionIndicationFlag[curLayerGroup][i]){

[0863] if(curLayerGroup==prtLayerGroup)

[0864] prtSubgroup = i;

[0865] else{

[0866] auto cur_bbox_min=layerGroupParams.subgrpBboxOrigin[curLayerGroup][i];

[0867] auto cur_bbox_max=cur_bbox_min+layerGroupParams.subgrpBboxSize[curLayerGroup][i];

[0868] for(int m=0;m<=layerGroupParams.numSubgroupsMinus1[prtLayerGroup];m++){

[0869] auto bbox_min=layerGroupParams.subgrpBboxOrigin[prtLayerGroup][m];

[0870] auto bbox_max=bbox_min+layerGroupParams.subgrpBboxSize[prtLayerGroup][m];

[0871] if(cur_bbox_min.x()>=bbox_min[0]&&cur_bbox_max.x()<=bbox_max[0]

[0872] &&cur_bbox_min.y()>=bbox_min[1]&&cur_bbox_max.y()<=bbox_max[1]

[0873] &&cur_bbox_min.z()>=bbox_min[2]&&cur_bbox_max.z()<=bbox_max[2]){

[0874] prtSubgroup=m;

[0875] break;

[0876] }

[0877] }

[0878] }

[0879] int startIndex_subgroup=0;

[0880] int endIndex_subgroup = indexes_subgroup[i].size();

[0881] int curSubgroup = i;

[0882] / / NN search function

[0883] computeNearestNeighbors(

[0884] aps, abh, packedVoxel, retained_subgroup[prtSubgroup], startIndex_subgroup, endIndex_subgroup, lodIndex, indexes_subgroup[i],

[0885] predictors, pointIndexToPredictorIndex, predIndex, atlas

[0886] , layerGroupParams, curLayerGroup, curSubgroup);

[0887] for (int m = startIndex_subgroup; m < endIndex_subgroup; m++)

[0888] indexes.push_back(indexes_subgroup[i][m]);

[0889] numIndexedNodes += indexes_subgroup[i].size();

[0890] layerGroupParams.numberOfPointsPerLodPerSubgroups.push_back(indexes_subgroup[i].size());

[0891] }

[0892] else

[0893] layerGroupParams.numberOfPointsPerLodPerSubgroups.push_back(0);

[0894] }

[0895] }

[0896] else

[0897] computeNearestNeighbors(

[0898] aps,abh,packedVoxel,retained,startIndex,endIndex,lodIndex,indexes,

[0899] predictors,pointIndexToPredictorIndex,predIndex,atlas,layerGroupParams,curLayerGroup,curSubgroup);

[0900] if(layerGroupParams.layerGroupEnabledFlag){

[0901] if(pointCount>indexes.size())

[0902] numberOfPointsPerLevelOfDetail.push_back(pointCount-indexes.size());

[0903] }

[0904] else{

[0905] if(!retained.empty()){

[0906] numberOfPointsPerLevelOfDetail.push_back(retained.size());

[0907] }

[0908] }

[0909] input.resize(0);

[0910] std::swap(retained,input);

[0911] }

[0912] std::reverse(indexes.begin(),indexes.end());

[0913] updatePredictors(pointIndexToPredictorIndex,predictors);

[0914] std::reverse(

[0915] numberOfPointsPerLevelOfDetail.begin(),

[0916] numberOfPointsPerLevelOfDetail.end());

[0917] std::reverse(

[0918] layerGroupParams.numberOfPointsPerLodPerSubgroups.begin(),

[0919] layerGroupParams.numberOfPointsPerLodPerSubgroups.end());

[0920] }

[0921] In other embodiments of the present disclosure, when performing an NN search for a current point to find at least one neighbor node, nodes belonging to the parent slice of the slice (i.e., sub-subgroup) to which the current point belongs can be used as reference nodes, even if they are outside the boundaries of the sub-slice (i.e., subgroup).

[0922] Fig.28 is a diagram illustrating another exemplary method of searching for nearest neighbors according to an embodiment.

[0923] In one embodiment, the NN search may be performed by referring to the node 48054 in the subgroup 48053 to which the current point 48050 belongs and the nodes 48041 and 48043 in the parent subgroup 48040 of the subgroup 48053 to which the current point 48050 belongs. This NN search is referred to herein as an inter-layer group search. In this case, the node 48052 in another subgroup 48051 in the same layer as the subgroup 48053 to which the current point 48050 belongs (i.e., the sibling subgroup of the subgroup to which the current point 48050 belongs) is not considered (or used) as a reference node in the NN search for the current point 48050.

[0924] In other words, when the current LoD (LoD N) 48055 is at the coarsest level of the current layer group, the neighbor candidate can be located in another layer group. In this case, if the node (or neighbor candidate) is located in the parent-child group boundary, it is allowed to find neighbors across the child group boundary.

[0925] According to an embodiment, for the inter-layer group search boundary, when the neighbor candidate is located in the parent layer group, the use of neighbor nodes in the parent-child group boundary is restricted.

[0926] In addition, Fig.28 In

[0064] , when layer group adapted positions are used, the encoded positions of nodes within parent nodes may be handled differently when the node is within the child group bounding box of the current node.

[0927] The layer group adaptation position is described below.

[0928] Nodes within the sub-subgroup bounding box: The encoding position is right-shifted by the number of LoDs in the encoded descendant subgroups of the current subgroup, and then left-shifted by the number of LoDs in the encoded descendant subgroups of the current subgroup and the skipped descendant subgroups.

[0929] Nodes outside the child subgroup bounding box: The encoding position is right-shifted by the number of LoDs in the encoded descendant (e.g., child) subgroups of the parent subgroup, and then left-shifted by the number of LoDs in the encoded descendant subgroups and skipped descendant subgroups of the parent subgroup.

[0930] Fig.29 is a diagram illustrating yet another exemplary method of searching for nearest neighbors according to an embodiment.

[0931] exist Fig.29 , 48060 represents a parent subgroup, 48061 represents a neighbor candidate outside the bounding box of subgroup 48070, 48070 represents a current point to be encoded (or decoded) within subgroup 48070, and 48072 represents a neighbor candidate within the bounding box of subgroup 48070. In other words, 48061 represents a node outside the bounding box of the sub-subgroup, and 48072 represents a node within the bounding box of the sub-subgroup.

[0932] According to an embodiment, in the NN search, when the neighbor candidate is within the subgroup bounding box (48072), the details of the geometric position are considered down to the finest level (e.g., bottom level) of the child subgroup (48070). On the other hand, when the neighbor candidate is outside the subgroup bounding box (48061), the details of the geometric position are considered down to the finest level (e.g., bottom level) of the parent subgroup. In other words, when the neighbor candidate is outside the subgroup bounding box (48061), the child subgroup is not considered when determining the position of the point in the NN search.

[0933] Fig.30 Another exemplary point cloud transmitting device according to an embodiment is illustrated. Fig.30 The elements of the point cloud sending device shown may be implemented as hardware, software, a processor and / or a combination thereof.

[0934] According to an embodiment, the point cloud sending device may include a data input unit 51001, a signaling processor 51002, a geometry encoder 51003, an attribute encoder 51004 and a transmission processor 51005.

[0935] The geometry encoder 51003 and the attribute encoder 51004 can perform Figure 1 Point cloud video encoder 10002, Figure 2 Code 20001, Figure 3 Point cloud video encoder, Figure 8 Point cloud video encoder, Fig.25 The attribute encoding and Fig.31 The attributes encode some or all of the operations described.

[0936] According to an embodiment, the data input unit 51001 receives or acquires point cloud data. The data input unit 51001 may perform Figure 1 Some or all of the operations of the point cloud video acquisition unit 10001 may be performed Figure 8 Some or all of the operations of the data input unit 8000.

[0937] The data input unit 51001 outputs the position of the point in the point cloud data to the geometry encoder 51003, and outputs the attribute of the point in the point cloud data to the attribute encoder 51004. In addition, the data input unit 51001 outputs the parameter to the signaling processor 51002. According to an embodiment, the parameter may be provided to the geometry encoder 51003 and the attribute encoder 51004.

[0938] The geometry encoder 51003 constructs an octree based on the position of the input point and performs geometry compression based on the octree. The geometry encoder 51003 performs entropy coding on the compressed geometry information and outputs the coded information to the transmission processor 51005 in the form of a geometry bitstream.

[0939] The geometry encoder 51003 reconstructs the geometry information based on the position changed by compression, and outputs the reconstructed (or decoded) geometry information to the attribute encoder 51004.

[0940] According to an embodiment, the geometry encoder 51003 constructs an octree based on the position of the input point, performs layer group-based slicing in the octree, selects one or more slices, and then compresses the geometry information about the one or more selected slices. Figures 11 to 24 The layer group-based slices and slice-by-slice geometry compression according to the embodiments are described in detail, and thus a description thereof is skipped below to avoid redundancy.

[0941] The attribute encoder 51004 compresses the attribute information based on the position and / or reconstructed geometry information for which geometry encoding has not been performed. According to an embodiment, the attribute information may be encoded using any one or a combination of one or more of RAHT encoding, LOD-based predictive transform encoding, and lifting transform encoding. The attribute encoder 51004 performs entropy encoding on the compressed attribute information and outputs the information to the transmission processor 51005 in the form of an attribute bitstream.

[0942] According to an embodiment, the attribute encoder 51004 performs a reference Figures 11 to 29 In other words, in one embodiment, when there is a missing subgroup (or FGS or slice) associated with the attribute information, the attribute encoder 51004 can use the corresponding subgroup of geometry (or FGS of geometry) to detect at least one missing node of the parent subgroup. According to an embodiment, after comparing the subsampled nodes from the current subgroup with the temporary geometry slice output nodes, the missing nodes in the geometry output but not in the subsampled output are appended to the LoD of the current subgroup. In other words, the LoDN can include subsampled nodes from the subgroup of the current attribute and / or nodes from the geometry parent group corresponding to the parent group of the current attribute.

[0943] Furthermore, in the NN search for the current point, neighbors may be selected based on distance, which may include neighbor nodes outside the subgroup to which the current point belongs.

[0944] According to an embodiment, when a neighbor node belongs to a sibling subgroup of a subgroup (i.e., a subgroup of the same level), the attribute encoder 51004 does not regard neighbor nodes outside the subgroup to which the current point belongs as neighbor nodes of the current point. In other words, based on the sub-level subgroup boundary, nodes outside the boundary are not selected as neighbors. Therefore, by imposing a restriction that only nodes within the same subgroup bounding box (or boundary) are applied to generate predictors based on the aforementioned criteria, accurate decoding can be performed even in the case of partial decoding of segmented slices. In this case, it is possible to allow reference to the root layer group in all cases.

[0945] According to an embodiment, when a neighbor node outside the subgroup to which the current point belongs belongs to a parent-subgroup of the subgroup, the attribute encoder 51004 regards the neighbor node as a neighbor node of the current point.

[0946] According to an embodiment, when performing the NN search, when the neighbor candidate is within the subgroup bounding box, the attribute encoder 51004 considers the details of the geometric position down to the finest level (e.g., the bottom level) of the child subgroup. On the other hand, when the neighbor candidate is outside the subgroup bounding box, the details of the geometric position are considered down to the finest level (e.g., the bottom level) of the parent subgroup.

[0947] The attribute encoder 51004 then performs attribute prediction and compression based on the NN nodes obtained as described above.

[0948] The signaling processor 51002 may generate and / or process signaling information required for encoding / decoding / rendering of geometry information and attribute information, and provide the processed information to the geometry encoder 51003, the attribute encoder 51004, and / or the transmission processor 51005. Alternatively, the signaling processor 51002 may be provided with signaling information generated by the geometry encoder 51003, the attribute encoder 51004, and / or the transmission processor 51005. The signaling processor 51002 may provide information fed back from the receiving device (e.g., head orientation information and / or viewport information) to the geometry encoder 51003, the attribute encoder 51004, and / or the transmission processor 51005.

[0949] In the present disclosure, signaling information including information related to slices based on layer groups and / or attribute data units (i.e., attribute slices) may be signaled and transmitted at the level of parameter sets (sequence parameter set (SPS), geometry parameter set (GPS), attribute parameter set (APS), patch parameter set (TPS) (or patch list), etc.). In other words, signaling information including information related to skipped layers may also be signaled and transmitted per coding unit (or compression unit or prediction unit) of an image (such as a slice or patch).

[0950] The transport processor 51005 can execute Figure 8 The operations and / or sending methods of the transmission processor 8012 are the same or similar to the operations and / or sending methods of the transmission processor 8012, and perform the same or similar operations and / or sending methods as those of the transmission processor 8012. Figure 1 The operation and / or transmission method of the transmitter 10003 is the same as or similar to the operation and / or transmission method of the transmitter 10003. For details that will not be described below, refer to Figure 1 or Figure 8 Description.

[0951] The transport processor 51005 may multiplex the geometry bitstream output from the geometry encoder 51003, the attribute bitstream output from the attribute encoder 51004, and the signaling bitstream output from the signaling processor 51002 into one bitstream. The multiplexed bitstream may be transmitted without change or may be encapsulated in a file or a fragment and transmitted. In one embodiment of the present disclosure, the file may be in the ISOBMFF file format.

[0952] According to an embodiment, the file or segment may be sent to a receiving device or stored in a digital storage medium (e.g., a USB drive, SD, CD, DVD, Blu-ray disc, HDD, SSD, etc.). The transmission processor 51005 according to an embodiment may communicate with the receiving device through a network such as 4G, 5G or 6G through wired / wireless communication. In addition, the transmission processor 51005 may perform necessary data processing according to a network system (e.g., a 4G, 5G or 6G communication network system). The transmission processor 51005 may send encapsulated data in an on-demand manner.

[0953] Fig.31 is a flowchart illustrating an exemplary method for encoding point cloud data according to an embodiment. More specifically, Fig.31 is an example of Fig.30 Flowchart of an exemplary encoding method in which the attribute encoder 51004 performs LoD-based attribute encoding.

[0954] Fig.31 The encoding of the point cloud data can be obtained by Figure 1 The sending device Figure 2 The encoding, Figure 3 A sending device / method, Figure 8 A sending device / method, Fig.10 Devices Fig.15 The encoder, Fig.30 It is executed by a sending device or a combination thereof.

[0955] In the encoding method according to the embodiment, parameters such as a sequence parameter set (SPS), a geometry parameter set (GPS), an attribute parameter set (APS), and a LGSI are generated, and a layer group structure is configured. The configuration of the layer group structure may be performed during or before the generation of the LGSI. After performing geometry encoding, LoD generation is performed based on the restored (or reconstructed) geometry node (operation 48081).

[0956] Specifically, in operation 48081, reference is performed Figures 11 to 30 The LoD generation and NN search described above. According to one embodiment, operation 48081 may include detecting at least one missing node of the parent subgroup using the corresponding subgroup of geometry (or FGS of geometry) based on the presence of missing subgroups (or FGS or slices) associated with the attribute information. According to an embodiment, after comparing the subsampled nodes from the current subgroup and the temporary geometry slice output nodes, the missing nodes in the geometry output but not in the subsampled output are appended to the LoD of the current subgroup. In other words, LoD N is generated by the subsampled nodes from LoD N+1 and / or the missing nodes of the current layer group. The missing nodes of the current layer group are detected from the nodes of the corresponding geometry layer group. In other words, LoD N is generated using the subsampled nodes from the subgroup of the current attribute and / or the nodes from the geometry parent group corresponding to the parent group of the current attribute. Then, in the NN search for the current point, neighbors can be selected based on distance, which may include neighbor nodes outside the subgroup to which the current point belongs.

[0957] According to an embodiment, when a neighbor node outside the subgroup to which the current point belongs belongs to a sibling subgroup of the subgroup (i.e., a subgroup at the same level), the neighbor node is not considered to be a neighbor of the current point. In other words, based on the sub-level subgroup boundary, nodes outside the boundary are not selected as neighbors. Therefore, by imposing a restriction that only nodes within the same subgroup boundary box are applied to generate predictors based on the aforementioned criteria, accurate decoding can be performed even in the case of partial decoding of segmented slices. In this case, it is possible to allow reference to the root layer group in all cases.

[0958] According to an embodiment, when a neighbor node outside the subgroup to which the current point belongs belongs to a parent subgroup of the subgroup, the neighbor node is considered to be a neighbor node of the current point.

[0959] After performing LoD generation and NN search in operation 48081, LoDs are mapped to layer groups (operation 48082).

[0960] In other words, by applying the layer group slice to the attribute bitstream, an attribute corresponding to the partially decoded geometry bitstream can be provided. In this case, considering that the layer group is configured based on the geometry coding layer, the attribute coding layer can be configured as follows. According to an embodiment, in the case where partial decoding is performed by matching the LoD with the layer group based on the layer group structure configured according to the geometry coding layer, the geometry output and the attribute output can be matched.

[0961] As an example, a geometry coding layer may be matched with an attribute coding layer in one-to-one correspondence. In this case, octree-based subsampling and RAHT-based attribute coding may be performed. As another example, a layer group may be matched with an attribute coding layer in one-to-one correspondence, and the layer group may include a plurality of attribute coding layers. In other words, in the case of LoD-based attribute coding, since performance may vary due to changes in the nearest neighbor (NN) depending on the LoD configuration, the method of generating the layer group may affect compression efficiency.

[0962] Then, quantization weights are derived based on the LoD (operation 48083).

[0963] For example, when nodes are passed in a separated state due to layer group slicing, subgroup boundaries and layer group boundaries should be considered even for quantization weights. The present disclosure can support partial decoding by limiting the quantization weight update to be performed only within the subgroup boundaries in the leaf layer group. In other words, in the present disclosure, parent-child subgroup boundaries can be aligned, and restrictions can be applied to select nodes that only exist within the child-subgroup boundaries as neighbors. In this regard, for layer group slicing, the root layer group may not be subject to subdivision, and fixed weights may be used for the root layer group as a method of supporting partial decoding. As another example, quantization weights can be adaptively applied on a per-subgroup basis.

[0964] Then, when each attribute coding layer is changed, the layer group is checked for changes, and the subgroup is checked for changes for each node within the attribute coding layer (operation 48086). When a subgroup or layer group changes, the attribute encoder information related to the previous subgroup can be stored, and the attribute encoder for the current subgroup can be used to ensure the continuity of context, zero jumps, etc. within the subgroup and to ensure independence from neighboring subgroups. Specifically, when zeros appear continuously, zero jumps can provide greater compression efficiency. By providing continuity for zero jumps within the same layer group and subgroup, additional compression efficiency can be provided even when the attribute coding layer changes. After compressing the nodes belonging to all attribute coding layers, the encoded bitstreams for the corresponding subgroups are packaged into corresponding slices.

[0965] Although the attribute encoding based on LoD describes Fig.31 , but the encoding according to the embodiment can be applied to other attribute encoding methods such as RAHT.

[0966] The transmitting device / method according to the embodiment may perform operations including generating LoD (operation 48081), mapping LoD to layer groups (operation 48082), and deriving quantization weights (operation 48083). The operations may also include determining layer groups (operation 48084), determining subgroups (operation 48085), checking whether subgroups have changed (operation 48086), storing and loading context information (operation 48087), and encoding attribute information (operation 48088). The above operations may be performed at the node level or at the LoD level.

[0967] The step of generating LoD (operation 48081) may include subsampling, constructing a predictor, or performing a neighbor search according to an embodiment. When generating LoD, the construction of the predictor or the neighbor search may be performed based on the boundaries of the subgroup. For example, the neighbors of the first node may be selected from the nodes within the boundaries of the subgroup to which the first node belongs.

[0968] Fig.32 Another exemplary point cloud receiving device according to the embodiment is illustrated. Fig.32 The elements of the point cloud receiving device shown in can be implemented as hardware, software, a processor and / or a combination thereof.

[0969] According to an embodiment, the point cloud receiving device may include a receiving processor 61001, a signaling processor 61002, a geometry decoder 61003, an attribute decoder 61004 and a post-processor 61005.

[0970] The receiving processor 61001 may receive one bitstream, or may receive each of a geometry bitstream, an attribute bitstream, and a signaling bitstream. Upon receiving a file and / or a fragment, the receiving processor 61001 may decapsulate the received file and / or fragment and output a bitstream for the file and / or fragment.

[0971] When receiving (or decapsulating) a bitstream, the receiving processor 61001 can demultiplex the geometry bitstream, attribute bitstream and / or signaling bitstream from a bitstream, output the demultiplexed signaling bitstream to the signaling processor 61002, output the geometry bitstream to the geometry decoder 61003, and output the attribute bitstream to the attribute decoder 61004.

[0972] When receiving (or decapsulating) a geometry bitstream, an attribute bitstream and / or a signaling bitstream respectively, the receiving processor 61001 may pass the signaling bitstream to the signaling processor 61002, the geometry bitstream to the geometry decoder 61003, and the attribute bitstream to the attribute decoder 61004.

[0973] The signaling processor 61002 may parse and process information included in signaling information (e.g., such as SPS, GPS, APS, TPS) or metadata from an input signaling bitstream, and provide the parsed and processed information to the geometry decoder 61003, the attribute decoder 61004, and the post-processor 61005. According to another embodiment, the signaling information included in the geometry data unit header and / or the attribute slice data unit header may be pre-parsed by the signaling processor 61002 before decoding the corresponding slice data.

[0974] According to an embodiment, the signaling processor 61002 may also parse and process information signaled in a sequence parameter set and / or attribute data unit header (e.g., information related to layer group-based slices), and provide the processed information to the attribute decoder 61004.

[0975] According to an embodiment, the geometry decoder 61003 can perform a decode operation on the compressed geometry bitstream based on the signaling information. Fig.30 The inverse process of the operation of the geometry encoder 51003 is performed to reconstruct the geometry. The geometry information recovered (or reconstructed) by the geometry decoder 61003 is provided to the attribute decoder 61004. The attribute decoder 61004 can perform a compression on the compressed attribute bitstream based on the signaling information and the reconstructed geometry information. Fig.30 The inverse process of the operation of the attribute encoder 51004 is performed to restore the attributes.

[0976] According to an embodiment, the post-processor 61005 may match the geometry information (ie, position) reconstructed and output by the geometry decoder 61003 with the attribute information recovered and output by the attribute decoder 61004 to reconstruct and display / render the point cloud data.

[0977] According to an embodiment, the attribute decoder 61004 may also include a LoD generator configured to perform LoD generation and NN search. The details of the LoD generation and NN search process have been referred to above. Figures 1 to 32 A description has been made and will not be given below to avoid redundancy.

[0978] That is, the above-mentioned LoD generation and NN search can be equally applied to the attribute encoder 51004 of the transmitting device and the attribute decoder 61004 of the receiving device.

[0979] Fig.33 A method for decoding point cloud data according to an embodiment is illustrated. More specifically, Fig.33 is an example of a method for Fig.32 Flowchart of an exemplary decoding method in which the attribute decoder 61004 performs LoD-based attribute decoding.

[0980] Fig.33 The decoding of the point cloud data in can be done by Figure 1 The receiving device Figure 2 Decoding, Figure 7 Receiving device / method, Fig. 9 Receiving device / method, Fig.10 Devices Fig.32 decoders or any combination thereof.

[0981] The input to the attribute decoder 61004 is the attribute fine granularity slice bitstream or the attribute layer group slice bitstream, the decoded geometry data and the layer group structure. After geometry decoding, LoD generation and NN search are performed based on the decoded geometry data. Here, reference has been made to Figures 1 to 31 The LoD generation and NN search are described in detail, so their description will be omitted below to avoid redundancy. That is, the LoD generation and NN search can be performed in the same way by the encoder on the transmitting side and the decoder on the receiving side. In other words, the LOD generation and NN search process is performed not only by the point cloud encoder, but also by the point cloud decoder.

[0982] Then, weight derivation is performed based on the relationship of the nodes according to the LoD. The information generated at this time can be used for the subsequent attribute decoding process. The first attribute slice can be operated in the same way as the previous attribute slice. Then, when the layer group slice is enabled, the relevant attribute data unit header can be parsed, and the layer group identification information (dadu_layer_group_id) and the subgroup identification information (dadu_subgroup_id) can be obtained. When necessary, information about the LoD can be signaled. Based on this information, a layer group that matches the current LoD can be searched, and a parent-child group can be selected. When the layer group structure is applied to geometry and attributes in the same manner according to the present disclosure, the reference and parent information used in geometry decoding can be used. Based on this information, the information in the relevant attribute data unit can be decoded, and the point cloud data can be reconstructed.

[0983] According to an embodiment, the LoD generation and weight derivation method used by encoder 51004 can be used for decoding.

[0984] Reference Fig.33 , a method for decoding point cloud data according to an embodiment may include generating LoD (operation 48091) and deriving quantization weights (operation 48092). Operations 48091 and 48092 may correspond to LoD generation, NN search, and weight derivation performed by the attribute encoder 51004 of the transmitting device. In the decoding method according to an embodiment, as LoD generation, NN search, and weight derivation performed during encoding, LoD may be generated and quantization weights may be derived while considering the boundaries of subgroups. In this case, the generation of LoD may include generating a predictor and searching for NN. Reference has been made to Figures 25 to 31 Predictor generation and neighbor search are described in detail, so their detailed description will be skipped below.

[0985] The decoding method according to the embodiment may include parsing the attribute data unit header (operation 48093), decoding the attribute data unit (operation 48094), determining whether the layer group slice is enabled (operation 48095), and parsing the relevant attribute data unit header based on the layer group slice being enabled (operation 48096). It may also include mapping the LoD to the layer group (operation 48097), selecting the parent-child group (operation 48098), and decoding the relevant attribute data unit.

[0986] exist Fig.33 In the operation 48095, the layer_group_enabled_flag may be determined. The layer_group_enabled_flag equal to 1 indicates that layer group slicing is performed. The layer group slicing being performed may indicate that the point cloud data (geometry or attributes) is grouped into layer groups or subgroups for processing.

[0987] When the associated attribute data unit header is parsed in operation 48096, the LoD may be mapped to a layer group (operation 48097), and a parent-child group may be selected (operation 48098) to decode the attributes contained in the associated data unit.

[0988] The related attribute data unit header may include layer group identification information (dadu_layer_group_id) and subgroup identification information (dadu_subgroup_id), thereby specifying the layer group index and subgroup index. The related attribute data unit header may also include bounding box origin information and size information related to each subgroup. The related attribute data unit header may also include information about the reference layer group and reference subgroup and quantization weight information related to the current subgroup.

[0989] According to an embodiment, dadu_layer_group_id and dadu_subgroup_id may be used to map LoDs to layer groups (operation 48097), and bounding box origin information and size information associated with subgroups may be used to compare bounding box information when selecting a parent subgroup (operation 48098).

[0990] The decoding method according to the embodiment may perform decoding by referring to context information based on the layer group and the subgroup. The context information may be referred to as context state information, state information, etc. The context information may be referred to by using signaling information such as information about the reference layer group and the reference subgroup or by deriving the parent-child group based on bounding box information related to the subgroup.

[0991] Fig.34 An exemplary bitstream structure of point cloud data for transmission / reception according to an embodiment is illustrated. According to an embodiment, from Figure 1 , Figure 2 , Figure 3 , Figure 8 and Fig.30 The bitstream output by any of the point cloud video encoders can be Fig.34 form.

[0992] According to an embodiment, the bitstream of the point cloud data provides tiles or slices to allow the point cloud data to be divided into regions to be processed. Each region of the bitstream according to an embodiment may have different importance. Therefore, when the point cloud data is divided into tiles, different filters (encoding methods) and different filter units may be applied to the tiles respectively. When the point cloud data is divided into slices, different filters and different filter units may be applied to the slices respectively.

[0993] The transmitting device according to the embodiment may be configured as follows: Fig.34 The point cloud data is transmitted by using the bitstream structure shown, so a method of applying different encoding operations according to importance and using an encoding method with good quality in an important area can be provided. In addition, efficient encoding and transmission according to the characteristics of the point cloud data can be supported, and attribute values ​​according to user requirements can be provided.

[0994] The receiving device according to the embodiment may be configured as follows: Fig.34 The point cloud data is received by using the bit stream structure shown in the figure, so different filtering methods (decoding methods) can be applied to each area (area divided into tiles or slices) instead of using a complex decoding (filtering) method for the entire point cloud data according to the processing capability of the receiving device. Therefore, better image quality can be ensured in areas that are important to users, and appropriate latency can be ensured on the system.

[0995] When the geometry bitstream, attribute bitstream and / or signaling bitstream (or signaling information) according to the embodiment are constituted as follows Fig.34 When a bitstream (or G-PCC bitstream) is shown, the bitstream may include one or more sub-bitstreams. According to an embodiment, the bitstream may include a sequence parameter set (SPS) for signaling a sequence level, a geometry parameter set (GPS) for signaling geometry information coding, one or more attribute parameter sets (APS) for signaling attribute information coding, and one or more attribute parameter sets (APS) for signaling attribute information coding. 0 and APS 1 ), a patch list (also referred to as TPS) for signaling at the patch level, and one or more slices (slice 0 to slice n). That is, the bitstream of point cloud data according to the embodiment may include one or more patches, wherein each patch may be a slice group including one or more slices (slice 0 to slice n). The patch list (i.e., TPS) may include information about each of the one or more patches (e.g., coordinate value information and height / size information related to the patch bounding box). Each slice may include a geometry bitstream (Geom0) and / or one or more attribute bitstreams (Attr0 and Attr1). For example, slice 0 may include a geometry bitstream (Geom0 0 ) and one or more attribute bitstreams (Attr0 0 and Attr1 0 ).

[0996] The geometry bitstream in each slice may include a geometry slice header (geom_slice_header) and geometry slice data (geom_slice_data). According to an embodiment, the geometry bitstream in each slice may be referred to as a geometry data unit, and the geometry slice header may be referred to as a geometry data unit header. In addition, the geometry slice data may be referred to as geometry data unit data. According to an embodiment, the geometry slice header (or geometry data unit header) may include identification information (geom_parameter_set_id) related to a parameter set included in GPS, a tile identifier (geom_tile_id), a slice identifier (geom_slice_id), and information related to data included in the geometry slice data (geom_slice_data) (geomBoxOrigin, geom_box_log2_scale, geom_max_node_size_log2, geom_num_points). geomBoxOrigin is geometry box origin information indicating the box origin of geometry slice data, geom_box_log2_scale is information indicating the log scale of geometry slice data, geom_max_node_size_log2 is information indicating the size of the root geometry octree node, and geom_num_points is information related to the number of points of geometry slice data. Geometry slice data (or geometry data unit data) may include geometry information (or geometry data) related to point cloud data in a corresponding slice.

[0997] Each attribute bitstream in each slice may include an attribute slice header (attr_slice_header) and attribute slice data (attr_slice_data). According to an embodiment, the attribute bitstream in each slice may be referred to as an attribute data unit. In addition, the attribute slice header may be referred to as an attribute data unit header, and the attribute slice data may be referred to as attribute data unit data. According to an embodiment, the attribute slice header (or attribute data unit header) may include information about the corresponding attribute slice data (or the corresponding attribute data unit), and the attribute slice data may include attribute information (also referred to as attribute data or attribute value) related to the point cloud data in the corresponding slice. When there are multiple attribute bitstreams in a slice, each attribute bitstream may include different attribute information. For example, one attribute bitstream may include attribute information corresponding to color, and another attribute bitstream may include attribute information corresponding to reflectivity.

[0998] According to an embodiment, the parameters required for encoding and / or decoding the point cloud data may be defined in a parameter set of the point cloud data (e.g., SPS, GPS, APS, and TPS (also referred to as a patch list)) and / or a header of the corresponding slice. For example, in encoding and / or decoding of geometric information, parameters may be added to GPS. In patch-based encoding and / or decoding, parameters may be added to patch and / or slice headers.

[0999] According to an embodiment, information related to layer group based slices may be signaled in the SPS and / or the APS.

[1000] According to an embodiment, information related to layer group based slices may be signaled in a TPS and / or attribute data unit header.

[1001] According to an embodiment, when the syntax elements defined below are applicable to multiple point cloud data streams as well as the current point cloud data stream, information related to layer group-based slices may be carried in a higher parameter set.

[1002] According to an embodiment, information related to slices based on layer groups may be defined in a corresponding location or a separate location according to an application or system to use different application scopes, different application methods, etc. As used in the syntax described in this document, a field may have the same meaning as a parameter or a syntax element.

[1003] According to an embodiment, a parameter containing information related to a slice based on a layer group (which may be referred to as metadata, signaling information, etc.) may be generated by a metadata processor (or metadata generator) or a signaling processor of a transmitting device, and may be passed to a receiving device for decoding / reconstruction operations. For example, the parameter generated and transmitted by the transmitting device may be acquired by a metadata parser of the receiving device.

[1004] Fig.35 An embodiment of the syntax structure of a sequence parameter set (seq_parameter_set()) (SPS) according to the present disclosure is illustrated. The SPS may include sequence information related to a point cloud data bitstream. Specifically, the SPS is exemplarily shown to include information related to slices based on layer groups.

[1005] Fig.35 The syntax can be included in Fig.34 and can be generated by a point cloud encoder and decoded by a point cloud decoder according to an implementation.

[1006] exist Fig.35 In the simple_profile_compatibility_flag field, the bitstream may indicate whether the bitstream is compatible with the main profile. For example, simple_profile_compatibility_flag equal to 1 may indicate that the bitstream is compatible with the simple profile. For example, simple_profile_compatibility_flag equal to 0 may indicate that the bitstream is compatible with profiles other than the simple profile.

[1007] When unique_point_positions_constraint_flag is equal to 1, all output points can have unique positions in each point cloud frame referenced by the current SPS. When unique_point_positions_constraint_flag is equal to 0, two or more output points can have the same position in any point cloud frame referenced by the current SPS. For example, even when all points are unique in the corresponding slice, slices and other points in the frame may overlap. In this case, unique_point_positions_constraint_flag is set to 0.

[1008] level_idc indicates the level to which the bitstream conforms.

[1009] sps_seq_parameter_set_id provides an identifier of the SPS for reference by other syntax elements.

[1010] sps_num_attribute_sets indicates the number of coded attributes in the bitstream.

[1011] An SPS according to an embodiment includes an iteration statement that iterates as many times as the value of the sps_num_attribute_sets field. In an embodiment, i is initialized to 0 and incremented by 1 each time the iteration statement is executed. The iteration statement is repeated until the value of i becomes equal to the value of the sps_num_attribute_sets field. The iteration statement may include an attribute_dimension_minus1[i] field and an attribute_instance_id[i] field. attribute_dimension_minus1[i] plus 1 indicates the number of components of the i-th attribute.

[1012] The attribute_instance_id[i] field specifies the instance ID of the i-th attribute.

[1013] The known_attribute_label_flag[i] field indicates whether the known_attribute_label[i] field or the attribute_label_four_bytes[i] field is signaled for the i-th attribute. For example, when known_attribute_label_flag[i] is equal to 0, it indicates that the known_attribute_label[i] field is signaled for the i-th attribute. When known_attribute_label_flag[i] is equal to 1, it indicates that the attribute_label_four_bytes[i] field is signaled for the i-th attribute.

[1014] known_attribute_label[i] specifies the type of the i-th attribute. For example, a known_attribute_label[i] of 0 specifies that the i-th attribute is color. A known_attribute_label[i] of 1 specifies that the i-th attribute is reflectivity. A known_attribute_label[i] of 2 specifies that the i-th attribute is a frame index. Additionally, a known_attribute_label[i] of 4 specifies that the i-th attribute is transparency. A known_attribute_label[i] of 5 specifies that the i-th attribute is a normal.

[1015] axis_coding_order indicates the correspondence between the X output axis label, the Y output axis label, and the Z output axis label and the three position components in the reconstructed point cloud RecPic[pointedx][axis], and axis = 0...2.

[1016] sps_bypass_stream_enabled_flag equal to 1 may indicate that the bypass encoding mode is used for reading the bitstream. As another example, sps_bypass_stream_enabled_flag equal to 0 may indicate that the bypass encoding mode is not used when reading the bitstream.

[1017] exist Fig.35 In the sps_extension_flag field, the value of the sps_extension_present_flag field indicates whether the sps_extension_data syntax structure exists in the SPS syntax structure. For example, the value of the sps_extension_present_flag field equals 1, indicating that the sps_extension_data syntax structure exists in the SPS syntax structure. The value of the sps_extension_present_flag field equals 0, indicating that the syntax structure does not exist. The following is an example of information related to layer group-based slices included in the sps_extension_data when the value of the sps_extension_present_flag field is 1.

[1018] layer_group_enabled_flag equal to 1 indicates that the geometry (or attribute) bitstream of a slice is contained in multiple slices matching the coding layer group or a subgroup thereof. layer_group_enabled_flag equal to 0 indicates that the geometry (or attribute) bitstream is contained in a single slice.

[1019] When layer_group_enabled_flag is equal to 1, num_layer_groups_minus1 plus 1 indicates the number of layer groups, which represents groups of consecutive tree layers that are part of the geometric coding tree structure. num_layer_groups_minus1 ranges from 0 to the number of coding tree layers.

[1020] The loop iterating according to the value of num_layer_groups_minus1 may include a layer_group_id[i] field, a num_layers_minus1[i] field, and a subgroup_enabled_flag[i] field.

[1021] layer_group_id[i] indicates the identifier of the i-th layer group. layer_group_id is in the range of 0 to num_layer_groups_minus1.

[1022] num_layers_minus1[i]+1 indicates the number of coded layers included in the i-th layer group. The total number of layer groups may be derived by adding all (num_layers_minus1[i]+1) for i from 0 to num_layer_groups_minus1.

[1023] subgroup_enabled_flag[i] equal to 1 indicates that the i-th layer group is divided into two or more subgroups. In this case, the point set in the subgroup is the same as the point set in the layer group. When subgroup_enabled_flag[i] of the i-th layer group is equal to 1, and j is greater than or equal to i, the subgroup_enabled_flag of the j-th layer group shall be equal to 1. subgroup_enabled_flag equal to 0 indicates that the current layer group is contained in a single slice and is not divided into multiple subgroups.

[1024] When layer_group_enabled_flag is equal to 1, the SPS may include subgroup_bbox_origin_bits_minus1 and subgroup_bbox_size_bits_minus1.

[1025] subgroup_bbox_origin_bits_minus1+1 indicates the length in bits of the syntax element (ie, field) subgroup_bbox_origin.

[1026] subgroup_bbox_size_bits_minus1+1 indicates the length in bits of the syntax element subgroup_bbox_size.

[1027] According to an embodiment, when layer_group_enabled_flag is equal to 1, the SPS may further include root_subgroup_bbox_origin and root_subgroup_bbox_size.

[1028] root_subgroup_bbox_origin may indicate the location of the origin of the subgroup bounding box of the root subgroup.

[1029] root_subgroup_bbox_size may indicate the size of the subgroup bounding box of the root subgroup.

[1030] The SPS according to an embodiment may further include a sps_extension_data_flag field when sps_extension_flag is equal to 1. The sps_extension_data_flag field may have any value.

[1031] Fig.36 An embodiment of a syntax structure of a dependent attribute data unit header (dependent_attribute_data_unit_header()) according to the present disclosure is illustrated. Fig.36 An example in which a related attribute data unit header contains information related to layer group-based slices is illustrated.

[1032] Fig.36 The syntax can be included in Fig.34 and, depending on the implementation, can be generated by a point cloud encoder and decoded by a point cloud decoder.

[1033] According to an embodiment, a related attribute data unit header may be included in the attribute data unit.The term attribute data unit may be used interchangeably with related attribute data unit or attribute slice.

[1034] According to an embodiment, the relevant attribute data unit header may be included in at least one of SPS, APS, GPS or TPS.

[1035] exist Fig.36 In the slice, dadu_attribute_parameter_set_id indicates the active APS indicated by aps_attr_parameter_set_id. The value of dgdu_attribute_parameter_set_id is equal to the value of adu_attribute_parameter_set_id in the corresponding slice.

[1036] dadu_sps_attr_idx identifies the coded attribute by its index into the active SPS attribute list.

[1037] dadu_slice_id specifies the attribute slice to which the current related attribute data unit belongs.

[1038] dadu_layer_group_id may be an indicator of the layer group of a slice. dadu_layer_group_id may be in the range of 0 to num_layer_groups_minus1. When not present, it is inferred to be 0.

[1039] dadu_subgroup_id indicates an indicator of a subgroup of the layer group referenced by dadu_layer_group_id. dadu_subgroup_id may be in the range of 0 to num_subgroups_minus1[dadu_layer_group_id]. dadu_subgroup_id may indicate the order of slices with the same dadu_layer_group_id. When not present, it is inferred to be 0.

[1040] The receiving method / device according to the embodiment may have the following effects.

[1041] The present disclosure describes a method for dividing and sending compressed data based on specific criteria for point cloud data. Specifically, in one application of the present disclosure, when layered coding (or layered coding) is used, the compressed data can be divided according to layers for transmission, which can improve the storage and transmission efficiency of the sending side. Specifically, when scalable attribute coding is used, the entire encoded geometric data should be sent, which is a disadvantage. In the present disclosure, only the geometric layers that match the tree level used in the scalable attribute coding can be transmitted, which can further improve the transmission efficiency.

[1042] Fig.37 is a diagram illustrating an example of compressing the geometry and attributes of point cloud data for a service. In other words, in a PCC-based service, the compression rate or amount of data can be adjusted and sent according to the receiver performance or the transmission environment. Fig.37 In the case where point cloud data is bundled on a slice-by-slice basis as shown, the receiver performance or the transmission environment may change. In this case, it is necessary to 1) pre-transcode the bitstream into a form suitable for each environment, store it separately, and select the bitstream when sending, or 2) perform transcoding before sending. In this case, when the number of receiver environments to be supported increases or the transmission environment changes frequently, problems related to storage space or delays caused by transcoding may arise.

[1043] Fig.38 is a diagram illustrating another example of compressing geometry and attributes of point cloud data for a service according to an embodiment.

[1044] As proposed in the present disclosure, dividing the compressed data according to the layers for transmission can make it possible to selectively pass only the necessary parts of the pre-compressed data through the bitstream selector at the bitstream level without the need for a separate transcoding operation. This is efficient in terms of storage space because only one storage space is required per stream. It is also efficient in terms of bandwidth because the bitstream selector selects only the necessary layers before transmission.

[1045] The effects of the features of the present disclosure from the perspective of the receiving side are described below. In one application of the present disclosure, when layered coding (or layered coding) is used, the compressed data can be divided according to the layers, which can improve the efficiency of the receiving side. Specifically, in the case of scalable attribute coding, such as Fig.39 As shown, receiving and decoding the entire coded geometry data may cause delays and increase the receiver computational burden. According to the present disclosure, only the geometry layer that matches the tree level used in the scalable attribute encoding can be decoded, which can reduce the delay factor and improve the efficiency of the decoder by saving the computational power required for decoding.

[1046] Fig.39 is a diagram illustrating the operation of the transmitting side / receiving side when transmitting layered point cloud data. In this case, if information for reconstructing the entire point cloud data is transmitted regardless of the performance of the receiver, the receiver may need to reconstruct the point cloud data by decoding and then select only the point cloud data corresponding to the necessary layer (e.g., data selection or subsampling). In this case, since the transmitted bit stream has been decoded, a delay may occur in a receiver targeting low latency, or decoding may not be performed depending on the receiver performance.

[1047] However, as proposed in this article, when a receiver receives only compressed data of a necessary layer, the receiver is allowed to selectively decode a specific layer. Thus, the decoder efficiency can be improved and decoders of different performances can be supported.

[1048] Fig.40 is a flow chart of a method of transmitting point cloud data according to an embodiment.

[1049] The method of transmitting point cloud data according to an embodiment may include acquiring point cloud data (71001), encoding the point cloud data (71002), and transmitting the encoded point cloud data and signaling information (71003). In this case, a bit stream containing the encoded point cloud data and signaling information may be encapsulated in a file and transmitted.

[1050] When acquiring point cloud data (71001), you can execute Figure 1 Some or all operations of the point cloud video acquisition unit 10001 or Figure 8 Some or all operations of the data input unit 8000.

[1051] When encoding point cloud data (71002), you can execute Figure 1 Point cloud video encoder 10002, Figure 2 Code 20001, Figure 3 Point cloud video encoder, Figure 8 Point cloud video encoder, Fig.25 The attribute code, Fig.30 The geometry encoder and attribute encoder of Fig.31 Some or all of the attribute encoding operations to encode geometric information and attribute information.

[1052] According to an embodiment, encoding point cloud data (71002) may include compressing geometric information related to the input point cloud data and compressing attribute information.

[1053] According to an embodiment, compressing the geometric information may include compressing the geometric information on a per-layer group and / or per-subgroup basis.

[1054] The compressed geometry information can be entropy coded and output in the form of a geometry bitstream.

[1055] According to an embodiment, compressing the attribute information includes compressing the attribute information based on the position and / or reconstructed geometry information for which geometry coding has not been performed. In one embodiment, the attribute information may be encoded using any one or a combination of one or more of RAHT coding, LOC-based predictive transform coding, and lifting transform coding.

[1056] According to an embodiment, compressing the attribute information includes performing a reference Figures 11 to 30 LoD generation and NN search described. In one embodiment, compression of attribute information may include detecting at least one missing node of a parent subgroup using a corresponding subgroup of geometry (or FGS of geometry) based on the presence of missing subgroups (or FGSs or slices) associated with the attribute information. According to an embodiment, after comparing the subsampled nodes from the current subgroup and the temporary geometry slice output nodes, the missing nodes in the geometry output but not in the subsampled output are appended to the LoD of the current subgroup. In other words, LoD N is generated using subsampled nodes from the subgroup of the current attribute and / or nodes from the geometry parent group corresponding to the parent group of the current attribute. Then, in the NN search for the current point, neighbors may be selected based on distance, which may include neighbor nodes outside the subgroup to which the current point belongs.

[1057] According to an embodiment, when compressing attribute information, when a neighbor node outside the subgroup to which the current point belongs belongs to a sibling subgroup of the subgroup (i.e., a subgroup at the same level), the neighbor node is not considered to be a neighbor of the current point. In other words, based on the sub-level subgroup boundary, nodes outside the boundary are not selected as neighbors. Therefore, by imposing a restriction that only nodes within the same subgroup boundary box are applied to generate a predictor based on the aforementioned criteria, accurate decoding can be performed even in the case of partial decoding of segmented slices. In this case, it is possible to allow reference to the root layer group in all cases.

[1058] According to an embodiment, when compressing attribute information, when a neighbor node outside the subgroup to which the current point belongs belongs to the parent subgroup of the subgroup, the neighbor node is considered to be a neighbor node of the current point.

[1059] According to an embodiment, when performing a NN search when compressing attribute information, when a neighbor candidate is within the subgroup bounding box, the attribute encoder 51004 considers the details of the geometric position to be down to the finest level (e.g., the bottom level) of the child subgroup. On the other hand, when the neighbor candidate is outside the subgroup bounding box, the details of the geometric position are considered to be down to the finest level (e.g., the bottom level) of the parent subgroup.

[1060] The compressed attribute information is entropy encoded and output in the form of an attribute bitstream.

[1061] In the present disclosure, the signaling information may include geometric information at a layer group and / or sub-group level and information related to layer group-based slices for encoding of attribute information and decoding at a receiving side.

[1062] The information related to the layer group-based slice may be included in at least one of the SPS, the APS, and / or the attribute data unit header (or the related attribute data unit header) and transmitted to the receiving side. Fig.35 and Fig.36 , further details regarding the information included in the information related to the layer group-based slices are omitted below.

[1063] Fig.41 is a flowchart of a method of receiving point cloud data according to an embodiment.

[1064] According to an embodiment, a method for receiving point cloud data may include receiving encoded point cloud data and signaling information (81001), decoding the point cloud data based on the signaling information (81002), and rendering the decoded point cloud data (81003).

[1065] According to the embodiment, the receiving of point cloud data and signaling information (81001) can be performed by Figure 1 Receiver 10005, Figure 2 Transmission 20002 or decoding 20003 or Fig. 9 The receiver 9000 or the receiving processor 9001 executes.

[1066] When decoding point cloud data (81002) according to an embodiment, it is possible to perform Figure 1 Point cloud video decoder 10006, Figure 2 Decoding 20003, Figure 8 Point cloud video decoder, Fig. 9 Point cloud video decoder, Fig.32 Geometry decoder and attribute decoder or Fig.33 Some or all of the attribute decoding operations to decode the geometry information and attribute information.

[1067] According to an implementation, decoding point cloud data (81002) includes decoding geometric information and decoding attribute information.

[1068] Decoding the geometric information may include decoding (ie, reconstructing) the geometric information on a per-layer group and / or sub-group basis based on information related to layer group-based slices included in the signaling information.

[1069] Decoding the attribute information may include decoding (i.e., decompressing) the attribute information based on layer groups and / or subgroups based on the reconstructed geometric information and information related to slices based on layer groups included in the signaling information. In one embodiment, the attribute information may be decoded using one or a combination of one or more of RAHT coding, LOD-based prediction transform coding, and lifting transform coding.

[1070] In rendering (81003) according to an embodiment, point cloud data can be reconstructed based on the restored (or reconstructed) geometric information and attribute information, and the point cloud data can be rendered according to various rendering schemes. For example, points in the point cloud content can be rendered as vertices with a specific thickness, cubes with a specific minimum size centered at the corresponding vertex position, circles centered at the vertex position, etc. All or part of the rendered point cloud content can be presented to the user through a display (e.g., a VR / AR display, a general display, etc.). Rendering point cloud data (81003) according to an embodiment can be performed by Figure 1 Renderer 10007 or Figure 2 Rendering 20004 or Fig. 9 The renderer 9011 is used to execute.

[1071] Therefore, the present disclosure can improve the efficiency of compression of attribute information by performing LoD generation and NN search while considering missing slices (or subgroups, or FGS) during attribute encoding / decoding.

[1072] Each of the above components, modules or units can be software, processors or hardware components that execute the continuous process stored in the memory (or storage unit). Each of the steps described in the above embodiments can be performed by a processor, a software component or a hardware component. Each module / block / unit described in the above embodiments can be operated as a processor, software or hardware. In addition, the method presented by the embodiment can be executed as a code. The code can be written on a processor-readable storage medium, so it is read by a processor provided by the device.

[1073] In this specification, when a component "includes" or "comprises" an element, unless otherwise specified, it means that the component also includes or comprises another element. In addition, the term "...module (or unit)" disclosed in the specification refers to a unit for processing at least one function or operation, and can be implemented by hardware, software, or a combination of hardware and software.

[1074] Although the embodiments have been explained with reference to each of the accompanying drawings for simplicity, new embodiments may be designed by combining the embodiments illustrated in the accompanying drawings. If a computer-readable recording medium having a program for executing the embodiments mentioned in the foregoing description is designed by a person skilled in the art, it may fall within the scope of the appended claims and their equivalents.

[1075] The device and method may not be limited to the configuration and method of the above-mentioned embodiments. The above-mentioned embodiments may be configured by selectively combining with each other in whole or in part to achieve various modifications.

[1076] Although preferred embodiments of the embodiments have been shown and described, the embodiments are not limited to the specific embodiments described above, and various modifications may be made by a person of ordinary skill in the art without departing from the spirit of the embodiments claimed in the claims, and these modifications should not be understood separately from the technical concept or view of the embodiments.

[1077] Various elements of the device of the embodiment can be implemented by hardware, software, firmware or a combination thereof. Various elements in the embodiment can be implemented by a single chip (e.g., a single hardware circuit). According to the embodiment, the components according to the embodiment can be implemented as separate chips respectively. According to the embodiment, at least one or more components of the device according to the embodiment may include one or more processors capable of executing one or more programs. One or more programs may execute any one or more operations / methods according to the embodiment or include instructions for executing them. The executable instructions for executing the method / operation of the device according to the embodiment may be stored in a non-transient CRM or other computer program product configured to be executed by one or more processors, or may be stored in a temporary CRM or other computer program product configured to be executed by one or more processors. In addition, the memory according to the embodiment can be used as not only covering volatile memory (e.g., RAM), but also covering the concepts of non-volatile memory, flash memory and PROM. In addition, it can also be implemented in the form of a carrier (e.g., transmission via the Internet). In addition, the processor-readable recording medium can be distributed to a computer system connected via a network so that the processor-readable code can be stored and executed in a distributed manner.

[1078] In this document, the terms " / " and "", should be interpreted as indicating "and / or". For example, the expression "A / B" may mean "A and / or B". Furthermore, "A, B" may mean "A and / or B". Furthermore, "A / B / C" may mean "at least one of A, B, and / or C". In addition, "A / B / C" may mean "at least one of A, B, and / or C". Furthermore, in this document, the term "or" should be interpreted as indicating "and / or". For example, the expression "A or B" may include 1) only A, 2) only B, and / or 3) both A and B. In other words, the term "or" in this document should be interpreted as indicating "in addition or alternatively".

[1079] The various elements of the embodiments can be implemented by hardware, software, firmware or a combination thereof. The various elements of the embodiments can be performed by a single chip such as a single hardware circuit. According to the embodiments, the elements can be selectively performed by separate chips respectively. According to the embodiments, at least one of the elements of the embodiments can be performed in one or more processors including instructions for performing operations according to the embodiments.

[1080] The operation according to the embodiment described in this specification can be performed by a sending / receiving device including one or more memories and / or one or more processors according to the embodiment. One or more memories can store programs for processing / controlling the operation according to the embodiment, and one or more processors can control the various operations described in this specification. One or more processors can be referred to as controllers, etc. In an embodiment, the operation can be performed by firmware, software, and / or a combination thereof. Firmware, software, and / or a combination thereof can be stored in a processor or a memory.

[1081] Terms such as first and second can be used to describe various elements of the embodiments. However, various components according to the embodiments should not be limited by the above terms. These terms are only used to distinguish one element from another element. For example, a first user input signal can be referred to as a second user input signal. Similarly, a second user input signal can be referred to as a first user input signal. The use of these terms should not be interpreted without departing from the scope of various embodiments. Both the first user input signal and the second user input signal are user input signals, but unless the context clearly stipulates otherwise, they do not mean the same user input signal.

[1082] The terms used to describe the embodiments are used only to describe specific embodiments and are not intended to limit the embodiments. As used in the description of the embodiments and in the claims, unless the context clearly stipulates otherwise, the singular form includes plural referents. The expression "and / or" is used to include all possible combinations of terms. Terms such as "including" or "having" are intended to indicate the presence of graphics, quantities, steps, elements and / or components, and should be understood as not excluding the possibility of additional presence of graphics, quantities, steps, elements and / or components. As used herein, conditional expressions such as "if" and "when" are not limited to optional situations, and are intended to be interpreted as performing relevant operations or interpreting relevant definitions according to specific conditions when specific conditions are met. The embodiments may include changes / modifications within the scope of the claims and their equivalents.

[1083] Invention Mode

[1084] As described above, the relevant contents are described in the best mode for carrying out the embodiment.

[1085] Industrial Applicability

[1086] As described above, the embodiments may be applied in whole or in part to point cloud data sending / receiving devices and systems. It is obvious to those skilled in the art that various changes or modifications may be made to the embodiments within the scope of the embodiments. Therefore, the embodiments are intended to cover modifications and variations as long as they fall within the scope of the attached claims and their equivalents.< / mortoncodewithindex> < / pccpredictor> < / std::chrono::milliseconds> < / int> < / int>

Claims

1. A method for sending point cloud data, the method comprising the following steps: Encoding geometric data of the point cloud data; encoding attribute data of the point cloud data based on the geometric data; as well as Sends encoded geometry data, encoded attribute data, and signaling information.

2. The method according to claim 1, wherein: The step of encoding the attribute data comprises: Segmenting the attribute data into one or more subgroups having a tree structure, and generating one or more levels of detail (LoDs) from the subgroup-based tree structure; searching for and selecting one or more nearest neighbors of a neighbor search target node based on at least one of the one or more LoDs; and Attribute information is compressed based on the selected one or more nearest neighbors of the neighbor search target node.

3. The method according to claim 2, wherein: The sub-group based tree structure includes at least one parent sub-group and one or more child sub-groups of the at least one parent sub-group.

4. The method according to claim 3, wherein: Each of the subsets is mapped to one or more LoDs.

5. The method according to claim 4, wherein: The step of generating the one or more LoDs comprises: comparing the at least one parent-child group with a corresponding child group of the geometric data; detecting at least one missing node included in the at least one parent-child group; and The detected nodes are appended to corresponding ones of the LoDs.

6. The method according to claim 4, wherein: The steps of searching and selecting the nearest neighbor include: When the neighbor search target node belongs to one of the one or more sub-subgroups, the nearest neighbor is searched by referring to a node included in the sub-subgroup to which the neighbor search target node belongs among the one or more sub-subgroups.

7. The method according to claim 6, wherein: The step of searching and selecting the nearest neighbor also includes: The nearest neighbor is searched by referring to a node included in a parent subgroup of the sub-subgroup to which the neighbor search target node belongs.

8. The method according to claim 4, wherein: The signaling information includes at least subgroup identification information for identifying each of the subgroups or bounding box information related to each of the subgroups.

9. A device for transmitting point cloud data, the device comprising: A geometry encoder configured to encode geometry data of the point cloud data; an attribute encoder configured to encode attribute data of the point cloud data based on the geometric data; as well as A transmitter is configured to transmit encoded geometry data, encoded attribute data and signaling information.

10. The device according to claim 9, wherein: The attribute encoder comprises: A level of detail LoD generator, the LoD generator is configured to: Segmenting the attribute data into one or more subgroups having a tree structure, and generating one or more LoDs from the subgroup-based tree structure; and searching for and selecting one or more nearest neighbors of a neighbor search target node based on at least one of the one or more LoDs; and A compressor is configured to compress attribute information based on the selected one or more nearest neighbors of the neighbor search target node.

11. The device according to claim 10, wherein: The sub-group based tree structure includes at least one parent sub-group and one or more child sub-groups of the at least one parent sub-group.

12. The device according to claim 11, wherein Each of the subsets is mapped to one or more LoDs.

13. The device according to claim 12, wherein: The LoD generator is configured to: comparing the at least one parent-child group with a corresponding child group of the geometric data; detecting at least one missing node included in the at least one parent-child group; as well as The detected nodes are appended to corresponding ones of the LoDs.

14. The device according to claim 12, wherein: When the neighbor search target node belongs to one of the one or more sub-sub-groups, the LoD generator searches for the nearest neighbor by referring to a node included in the sub-sub-group to which the neighbor search target node belongs among the one or more sub-sub-groups.

15. The device according to claim 14, wherein: The LoD generator searches for the nearest neighbor by referring to a node included in a parent sub-group of the sub-sub-group to which the neighbor search target node belongs.

Images