Point cloud data transmission device, point cloud data transmission method, point cloud data reception device, and point cloud data reception method

Abstract

A point cloud data transmission method according to an embodiment may comprise the steps of: encoding point cloud data; and transmitting the bit stream including the point cloud data. A point cloud data receiving method according to an embodiment may comprise the steps of: receiving a bitstream including point cloud data; and decoding the point cloud data.

Description

Technical Field

[0001] Embodiments relate to a method and apparatus for processing point cloud content. Background Art

[0002] Point cloud content is content represented by a point cloud, which is a collection of points belonging to a coordinate system representing a three-dimensional space. Point cloud content can express media configured in three dimensions and is used to provide various services such as virtual reality (VR), augmented reality (AR), mixed reality (MR), and self-driving services. However, tens of thousands to hundreds of thousands of point data are required to represent point cloud content. Therefore, a method of efficiently processing a large amount of point data is needed. Summary of the invention

[0003] Technical issues

[0004] The embodiment provides an apparatus and method for efficiently processing point cloud data. The embodiment provides a method and apparatus for processing point cloud data to solve waiting time and encoding / decoding complexity.

[0005] The technical scope of the embodiments is not limited to the above-mentioned technical objectives, and can be extended to other technical objectives that can be inferred by those skilled in the art based on the entire content disclosed in this document.

[0006] Technical Solution

[0007] In one aspect of the present disclosure, a method for transmitting point cloud data may include: encoding the point cloud data; and transmitting a bit stream containing the point cloud data. In another aspect of the present disclosure, a method for receiving point cloud data may include: receiving a bit stream containing the point cloud data; and decoding the point cloud data.

[0008] Beneficial Effects

[0009] The apparatus and method according to the embodiments can efficiently process point cloud data.

[0010] The apparatus and method according to the embodiments can provide high-quality point cloud services.

[0011] The device and method according to the embodiment may provide point cloud content for providing general services such as VR service and self-driving service. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are included to provide a further understanding of the present disclosure and are incorporated into and constitute a part of this application, illustrate embodiments of the present disclosure, and together with this description, are used to explain the principles of the present disclosure. In order to better understand the various embodiments described below, reference should be made to the description of the following embodiments in conjunction with the accompanying drawings. The same reference numerals will be used throughout the drawings to refer to the same or similar parts.

[0013] Figure 1 An exemplary point cloud content providing system according to an embodiment is shown;

[0014] Figure 2 is a block diagram illustrating point cloud content providing operations according to an embodiment;

[0015] Figure 3 An exemplary point cloud encoder according to an embodiment is illustrated;

[0016] Figure 4 shows examples of octrees and occupancy codes according to an embodiment;

[0017] Figure 5 illustrates an example of point configuration in each LOD according to an embodiment;

[0018] Figure 6 illustrates an example of point configuration in each LOD according to an embodiment;

[0019] Figure 7 A point cloud decoder according to an embodiment is illustrated;

[0020] Figure 8 A transmitting device according to an embodiment is illustrated;

[0021] Figure 9 A receiving device according to an embodiment is illustrated;

[0022] Figure 10 An exemplary structure that can be operated in conjunction with a point cloud data transmitting / receiving method / apparatus according to an embodiment is illustrated;

[0023] Figure 11 The process of encoding, transmitting and decoding point cloud data according to an embodiment is illustrated;

[0024] Figure 12 illustrates a layer-based point cloud data configuration according to an embodiment and a geometry and attribute bitstream structure according to an embodiment;

[0025] Figure 13 illustrates a bitstream configuration according to an embodiment;

[0026] Figure 14 A bitstream sorting method according to an embodiment is illustrated;

[0027] Figure 15 A method of selecting geometric data and attribute data according to an embodiment is illustrated;

[0028] Figure 16 A method of configuring a slice containing point cloud data according to an embodiment is illustrated;

[0029] Figure 17 illustrates a geometric coding layer structure according to an embodiment;

[0030] Figure 18 illustrates layer group and subgroup structures according to embodiments;

[0031] Figure 19 illustrates a multi-resolution, multi-size ROI according to an embodiment;

[0032] Figure 20 illustrates layer group slicing according to an embodiment;

[0033] Figure 21 A context reference method for fine grained slicing (FGS) according to an embodiment is illustrated;

[0034] Figure 22 An attribute encoding method according to an embodiment is illustrated;

[0035] Figure 23 An attribute decoding method according to an embodiment is illustrated;

[0036] Figure 24 illustrates a bitstream including parameters and encoded point cloud data according to an embodiment;

[0037] Figure 25 illustrates a set of attribute parameters according to an embodiment;

[0038] Figure 26 illustrates a dependency attribute data unit header according to an embodiment;

[0039] Figure 27 A process of sending partial point cloud data according to an embodiment is illustrated;

[0040] Figure 28 A point cloud data transmitting / receiving method according to an embodiment is illustrated;

[0041] Figure 29 A point cloud data transmitting / receiving method according to an embodiment is illustrated;

[0042] Figure 30 A method of transmitting point cloud data according to an embodiment is illustrated; and

[0043] Figure 31 A method of receiving point cloud data according to an embodiment is illustrated. DETAILED DESCRIPTION

[0044] Reference will now be made in detail to the preferred embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. The detailed description given below with reference to the accompanying drawings is intended to illustrate exemplary embodiments of the present disclosure, rather than to show the only embodiments that can be implemented according to the present disclosure. The following detailed description includes specific details in order to provide a thorough understanding of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure can be practiced without these specific details.

[0045] Although most of the terms used in the present disclosure are selected from common terms widely used in the art, some terms are arbitrarily selected by the applicant and their meanings are explained in detail in the following description as needed. Therefore, the present disclosure should be understood based on the intended meanings of the terms rather than their simple names or meanings.

[0046] Figure 1 An exemplary point cloud content providing system according to an embodiment is shown.

[0047] Figure 1 The point cloud content providing system shown may include a transmitting device 10000 and a receiving device 10004. The transmitting device 10000 and the receiving device 10004 can transmit and receive point cloud data via wired or wireless communication.

[0048] According to an embodiment, the point cloud data sending device 10000 can obtain and process point cloud video (or point cloud content) and send it. According to an embodiment, the sending device 10000 may include a fixed station, a base transceiver system (BTS), a network, an artificial intelligence (AI) device and / or system, a robot, an AR / VR / XR device and / or a server. According to an embodiment, the sending device 10000 may include a device configured to communicate with a base station and / or other wireless devices using a radio access technology (e.g., 5G New RAT (NR), Long Term Evolution (LTE)), a robot, a vehicle, an AR / VR / XR device, a portable device, a home appliance, an Internet of Things (IoT) device, and an AI device / server.

[0049] According to the implementation manner, the sending device 10000 includes a point cloud video acquirer 10001, a point cloud video encoder 10002 and / or a transmitter (or communication module) 10003.

[0050] The point cloud video acquirer 10001 according to the embodiment acquires the point cloud video through a processing process such as capture, synthesis or generation. The point cloud video is a point cloud content represented by a point cloud, which is a set of points located in a 3D space and can be referred to as point cloud video data, point cloud data, etc. The point cloud video according to the embodiment may include one or more frames. A frame represents a still image / picture. Therefore, the point cloud video may include a point cloud image / frame / picture and may be referred to as a point cloud image, frame or picture.

[0051] The point cloud video encoder 10002 according to the embodiment encodes the acquired point cloud video data. The point cloud video encoder 10002 may encode the point cloud video data based on point cloud compression coding. The point cloud compression coding according to the embodiment may include geometry-based point cloud compression (G-PCC) coding and / or video-based point cloud compression (V-PCC) coding or next generation coding. The point cloud compression coding according to the embodiment is not limited to the above-mentioned embodiment. The point cloud video encoder 10002 may output a bit stream containing the encoded point cloud video data. The bit stream may include not only the encoded point cloud video data, but also signaling information related to the encoding of the point cloud video data.

[0052] The transmitter 10003 according to the embodiment transmits a bitstream containing encoded point cloud video data. The bitstream according to the embodiment is encapsulated in a file or a fragment (e.g., a stream fragment) and is transmitted via various networks such as a broadcast network and / or a broadband network. Although not shown in the figure, the transmitting device 10000 may include a packager (or a package module) configured to perform a packaging operation. According to an embodiment, the packager may be included in the transmitter 10003. According to an embodiment, the file or fragment may be sent to the receiving device 10004 via a network, or stored in a digital storage medium (e.g., USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc.). The transmitter 10003 according to the embodiment can communicate with the receiving device 10004 (or the receiver 10005) via a 4G, 5G, 6G or other network by wire / wireless communication. In addition, the transmitter may perform necessary data processing operations according to the network system (e.g., a 4G, 5G or 6G communication network system). The transmitting device 10000 may transmit the encapsulated data in an on-demand manner.

[0053] The receiving device 10004 according to an embodiment includes a receiver 10005, a point cloud video decoder 10006 and / or a renderer 10007. According to an embodiment, the receiving device 10004 may include a device configured to communicate with a base station and / or other wireless devices using a radio access technology (e.g., 5G New RAT (NR), Long Term Evolution (LTE)), a robot, a vehicle, an AR / VR / XR device, a portable device, a home appliance, an Internet of Things (IoT) device, and an AI device / server.

[0054] The receiver 10005 according to the embodiment receives a bitstream containing point cloud video data or a file / segment encapsulated with a bitstream from a network or a storage medium. The receiver 10005 may perform necessary data processing according to a network system (e.g., a communication network system of 4G, 5G, 6G, etc.). The receiver 10005 according to the embodiment may decapsulate the received file / segment and output a bitstream. According to the embodiment, the receiver 10005 may include a decapsulator (or decapsulation module) configured to perform a decapsulation operation. The decapsulator may be implemented as an element (or component) separated from the receiver 10005.

[0055] The point cloud video decoder 10006 decodes the bitstream containing the point cloud video data. The point cloud video decoder 10006 can decode the point cloud video data according to the method in which the point cloud video data is encoded (for example, in the reverse process of the operation of the point cloud video encoder 10002). Therefore, the point cloud video decoder 10006 can decode the point cloud video data by performing point cloud decompression encoding (the reverse process of point cloud compression). The point cloud decompression encoding includes G-PCC encoding.

[0056] The renderer 10007 renders the decoded point cloud video data. The renderer 10007 may output the point cloud content by rendering not only the point cloud video data but also the audio data. According to an embodiment, the renderer 10007 may include a display configured to display the point cloud content. According to an embodiment, the display may be implemented as a separate device or component rather than being included in the renderer 10007.

[0057] The arrow indicated by the dotted line in the figure represents the transmission path of the feedback information obtained by the receiving device 10004. Feedback information is information that reflects the interactivity with the user who consumes the point cloud content, and includes information about the user (e.g., head orientation information, viewport information, etc.). Specifically, when the point cloud content is content for a service that requires interaction with the user (e.g., self-driving service, etc.), the feedback information may be provided to the content sender (e.g., sending device 10000) and / or the service provider. Depending on the embodiment, the feedback information may be used in the receiving device 10004 and the sending device 10000, or may not be provided.

[0058] The head orientation information according to the embodiment is information about the user's head position, orientation, angle, movement, etc. The receiving device 10004 according to the embodiment may calculate the viewport information based on the head orientation information. The viewport information may be information about the area of ​​the point cloud video that the user is watching. The viewpoint is the point through which the user watches the point cloud video, and may refer to the center point of the viewport area. That is, the viewport is an area centered on the viewpoint, and the size and shape of the area may be determined by the field of view (FOV). Therefore, in addition to the head orientation information, the receiving device 10004 may also extract the viewport information based on the vertical or horizontal FOV supported by the device. In addition, the receiving device 10004 performs gaze analysis, etc. to check the way the user consumes the point cloud, the area where the user gazes in the point cloud video, the gaze time, etc. According to the embodiment, the receiving device 10004 may send feedback information including the gaze analysis result to the sending device 10000. The feedback information according to the embodiment may be obtained during the rendering and / or display process. The feedback information according to the embodiment may be obtained by one or more sensors included in the receiving device 10004. According to an embodiment, the feedback information may be obtained by the renderer 10007 or a separate external element (or device, component, etc.). Figure 1 The dotted line in represents the process of sending the feedback information obtained by the renderer 10007. The point cloud content providing system can process (encode / decode) the point cloud data based on the feedback information. Therefore, the point cloud video data decoder 10006 can perform a decoding operation based on the feedback information. The receiving device 10004 can send the feedback information to the sending device 10000. The sending device 10000 (or the point cloud video data encoder 10002) can perform an encoding operation based on the feedback information. Therefore, the point cloud content providing system can efficiently process necessary data (for example, point cloud data corresponding to the user's head position) based on the feedback information instead of processing (encoding / decoding) the entire point cloud data, and provide the point cloud content to the user.

[0059] According to an embodiment, the transmitting device 10000 may be referred to as an encoder, a transmitting device, a transmitter, etc., and the receiving device 10004 may be referred to as a decoder, a receiving device, a receiver, etc.

[0060] According to the implementation mode Figure 1 The point cloud data processed in the point cloud content providing system (through a series of processes of acquisition / encoding / transmission / decoding / rendering) may be referred to as point cloud content data or point cloud video data. According to an embodiment, point cloud content data may be used as a concept covering metadata or signaling information related to point cloud data.

[0061] Figure 1 The elements of the point cloud content providing system shown may be implemented by hardware, software, a processor and / or a combination thereof.

[0062] Figure 2 is a block diagram illustrating point cloud content providing operations according to an embodiment.

[0063] Figure 2 The block diagram shows Figure 1 The operation of the point cloud content providing system described in . As described above, the point cloud content providing system can process point cloud data based on point cloud compression coding (e.g., G-PCC).

[0064] According to an embodiment, a point cloud content providing system (e.g., a point cloud sending device 10000 or a point cloud video acquirer 10001) can acquire a point cloud video (20000). The point cloud video is represented by a point cloud belonging to a coordinate system for expressing a 3D space. According to an embodiment, a point cloud video may include a Ply (Polygon file format or Stanford Triangle format) file. When a point cloud video has one or more frames, the acquired point cloud video may include one or more Ply files. A Ply file contains point cloud data such as point geometry and / or attributes. Geometry includes the position of a point. The position of each point may be represented by a parameter (e.g., the value of the X, Y, and Z axes) representing a three-dimensional coordinate system (e.g., a coordinate system consisting of X, Y, and Z axes). Attributes include attributes of the point (e.g., information about the texture, color (YCbCr or RGB), reflectivity r, transparency, etc. of each point). A point has one or more attributes. For example, a point may have a color attribute or two attributes, color and reflectivity. According to an embodiment, geometry may be referred to as position, geometry information, geometry data, position information, position data, etc., and attributes may be referred to as attributes, attribute information, attribute data, etc. The point cloud content providing system (e.g., the point cloud transmitting device 10000 or the point cloud video acquirer 10001) may obtain point cloud data from information related to a point cloud video acquisition process (e.g., depth information, color information, etc.).

[0065] According to an embodiment, a point cloud content providing system (e.g., a transmitting device 10000 or a point cloud video encoder 10002) may encode point cloud data (20001). The point cloud content providing system may encode point cloud data based on point cloud compression coding. As described above, point cloud data may include geometric information and attribute information about points. Therefore, the point cloud content providing system may perform geometric coding for encoding geometry and output a geometric bitstream. The point cloud content providing system may perform attribute coding for encoding attributes and output an attribute bitstream. According to an embodiment, the point cloud content providing system may perform attribute coding based on geometric coding. The geometric bitstream and the attribute bitstream according to the embodiment may be multiplexed and output as one bitstream. The bitstream according to the embodiment may also include signaling information related to geometric coding and attribute coding.

[0066] According to the embodiment, the point cloud content providing system (eg, the transmitting device 10000 or the transmitter 10003) may transmit the encoded point cloud data (20002). Figure 1 As shown, the encoded point cloud data can be represented by a geometry bitstream and an attribute bitstream. In addition, the encoded point cloud data can be sent in the form of a bitstream together with signaling information related to the encoding of the point cloud data (e.g., signaling information related to geometry encoding and attribute encoding). The point cloud content providing system can encapsulate the bitstream carrying the encoded point cloud data and send it in the form of a file or a fragment.

[0067] The point cloud content providing system (eg, receiving device 10004 or receiver 10005) according to the embodiment may receive a bit stream containing encoded point cloud data. In addition, the point cloud content providing system (eg, receiving device 10004 or receiver 10005) may demultiplex the bit stream.

[0068] The point cloud content providing system (e.g., the receiving device 10004 or the point cloud video decoder 10005) may decode the encoded point cloud data (e.g., geometry bitstream, attribute bitstream) sent in the bitstream. The point cloud content providing system (e.g., the receiving device 10004 or the point cloud video decoder 10005) may decode the point cloud video data based on the signaling information related to the encoding of the point cloud video data contained in the bitstream. The point cloud content providing system (e.g., the receiving device 10004 or the point cloud video decoder 10005) may decode the geometry bitstream to reconstruct the position (geometry) of the point. The point cloud content providing system may reconstruct the attributes of the point by decoding the attribute bitstream based on the reconstructed geometry. The point cloud content providing system (e.g., the receiving device 10004 or the point cloud video decoder 10005) may reconstruct the point cloud video based on the position according to the reconstructed geometry and the decoded attributes.

[0069] According to an embodiment, a point cloud content providing system (e.g., receiving device 10004 or renderer 10007) may render decoded point cloud data (20004). The point cloud content providing system (e.g., receiving device 10004 or renderer 10007) may use various rendering methods to render the geometry and attributes decoded by the decoding process. Points in the point cloud content may be rendered as vertices with a specific thickness, cubes with a specific minimum size centered at the corresponding vertex position, or circles centered at the corresponding vertex position. All or part of the rendered point cloud content is provided to the user through a display (e.g., a VR / AR display, a general display, etc.).

[0070] The point cloud content providing system (e.g., receiving device 10004) according to the embodiment may obtain feedback information (20005). The point cloud content providing system may encode and / or decode the point cloud data based on the feedback information. Feedback Information and Operations of the Point Cloud Content Providing System According to the Embodiment and Reference Figure 1 The feedback information and operations described are the same, so their detailed description is omitted.

[0071] Figure 3 An exemplary point cloud encoder according to an implementation is shown.

[0072] Figure 3 Show Figure 1 An example of a point cloud video encoder 10002. The point cloud encoder reconstructs and encodes point cloud data (e.g., the location and / or attributes of the points) to adjust the quality of the point cloud content (e.g., lossless, lossy, or near-lossless) according to network conditions or applications. When the total size of the point cloud content is large (e.g., 60 Gbps of point cloud content given for 30 fps), the point cloud content providing system may not be able to stream the content in real time. Therefore, the point cloud content providing system may reconstruct the point cloud content based on the maximum target bit rate to provide the point cloud content according to the network environment, etc.

[0073] As reference Figure 1 and Figure 2 As described, the point cloud encoder can perform geometry encoding and attribute encoding. Geometry encoding is performed before attribute encoding.

[0074] According to an embodiment, the point cloud encoder includes a coordinate transformer (transforming coordinates) 30000, a quantizer (quantizing and removing points (voxelization)) 30001, an octree analyzer (analyzing octree) 30002 and a surface approximation analyzer (analyzing surface approximation) 30003, an arithmetic encoder (arithmetic coding) 30004, a geometry reconstructor (reconstructing geometry) 30005, a color transformer (transforming color) 30006, an attribute transformer (transforming attributes) 30007, a RAHT transformer (RAHT) 30008, an LOD generator (generating LOD) 30009, a lifting transformer (lifting) 30010, a coefficient quantizer (quantizing coefficients) 30011 and / or an arithmetic encoder (arithmetic coding) 30012.

[0075] The coordinate transformer 30000, the quantizer 30001, the octree analyzer 30002, the surface approximation analyzer 30003, the arithmetic encoder 30004 and the geometry reconstructor 30005 may perform geometry coding. The geometry coding according to the embodiment may include octree geometry coding, prediction tree geometry coding, direct coding, triplet geometry coding and entropy coding. Direct coding and triplet geometry coding are applied selectively or in combination. The geometry coding is not limited to the above examples.

[0076] As shown in the figure, the coordinate transformer 30000 according to the embodiment receives the position and transforms it into coordinates. For example, the position can be transformed into position information in a three-dimensional space (e.g., a three-dimensional space represented by an XYZ coordinate system). The position information in the three-dimensional space according to the embodiment can be referred to as geometric information.

[0077] The quantizer 30001 according to the embodiment quantizes the geometry. For example, the quantizer 30001 may quantize the points based on the minimum position value of all points (e.g., the minimum value on each of the X, Y, and Z axes). The quantizer 30001 performs a quantization operation: the difference between the minimum position value and the position value of each point is multiplied by a preset quantization scaling value, and then the nearest integer value is found by rounding the value obtained by the multiplication. Therefore, one or more points may have the same quantized position (or position value). The quantizer 30001 according to the embodiment performs voxelization based on the quantized position to reconstruct the quantized point. As in the case of a pixel (the smallest unit containing 2D image / video information), a point of a point cloud content (or 3D point cloud video) according to the embodiment may be included in one or more voxels. As a composite of volume and pixel, the term voxel refers to a 3D cubic space generated when a 3D space is divided into units (unit = 1.0) based on an axis representing the 3D space (e.g., X-axis, Y-axis, and Z-axis). The quantizer 30001 may match a point group in a 3D space with a voxel. According to an embodiment, a voxel may include only one point. According to an embodiment, a voxel may include one or more points. In order to represent a voxel as a point, the position of the center of the voxel may be set based on the position of one or more points included in the voxel. In this case, the properties of all positions included in a voxel may be combined and assigned to the voxel.

[0078] The octree analyzer 30002 according to the embodiment performs octree geometry encoding (or octree encoding) to present voxels in an octree structure. The octree structure represents points matched to voxels based on an octal tree structure.

[0079] The surface approximation analyzer 30003 according to an embodiment may analyze and approximate an octree. The octree analysis and approximation according to an embodiment is a process of analyzing a region including a plurality of points to efficiently provide an octree and voxelization.

[0080] The arithmetic encoder 30004 according to the embodiment performs entropy coding on the octree and / or the approximate octree. For example, the coding scheme includes arithmetic coding. As a result of the coding, a geometry bitstream is generated.

[0081] The color converter 30006, the attribute converter 30007, the RAHT converter 30008, the LOD generator 30009, the lifting converter 30010, the coefficient quantizer 30011 and / or the arithmetic encoder 30012 perform attribute encoding. As described above, a point may have one or more attributes. The attribute encoding according to the embodiment is also applied to the attributes of a point. However, when the attribute (e.g., color) includes one or more elements, the attribute encoding is applied independently to each element. The attribute encoding according to the embodiment includes color transform encoding, attribute transform encoding, regional adaptive hierarchical transform (RAHT) encoding, interpolation-based hierarchical nearest neighbor prediction (prediction transform) encoding, and interpolation-based hierarchical nearest neighbor prediction (lifting transform) encoding with an update / lifting step. Depending on the point cloud content, the above-mentioned RAHT encoding, prediction transform encoding, and lifting transform encoding may be selectively used, or a combination of one or more encoding schemes may be used. The attribute encoding according to the embodiment is not limited to the above examples.

[0082] The color converter 30006 according to an embodiment performs color conversion encoding to convert the color value (or texture) included in the attribute. For example, the color converter 30006 can convert the format of the color information (e.g., from RGB to YCbCr). Optionally, the operation of the color converter 30006 according to an embodiment can be applied according to the color value included in the attribute.

[0083] The geometry reconstructor 30005 according to an embodiment reconstructs (decompresses) an octree and / or an approximate octree. The geometry reconstructor 30005 reconstructs an octree / voxel based on the result of analyzing the point distribution. The reconstructed octree / voxel may be referred to as a reconstructed geometry (restored geometry).

[0084] The attribute transformer 30007 according to the embodiment performs attribute transformation to transform the attribute based on the reconstructed geometry and / or the position where the geometry encoding is not performed. As described above, since the attribute depends on the geometry, the attribute transformer 30007 can transform the attribute based on the reconstructed geometry information. For example, based on the position value of the point included in the voxel, the attribute transformer 30007 can transform the attribute of the point at the position. As described above, when the center position of the voxel is set based on the position of one or more points included in the voxel, the attribute transformer 30007 transforms the attribute of one or more points. When triplet geometry encoding is performed, the attribute transformer 30007 can transform the attribute based on the triplet geometry encoding.

[0085] The attribute converter 30007 can perform attribute conversion by calculating the average of the attributes or attribute values ​​(e.g., the color or reflectivity of each point) of the neighboring points within a specific position / radius from the center position (or position value) of each voxel. The attribute converter 30007 can apply weights according to the distance from the center to each point when calculating the average. Therefore, each voxel has a position and a calculated attribute (or attribute value).

[0086] The attribute converter 30007 can search for neighboring points within a specific position / radius from the center position of each voxel based on a KD tree or a Morton code. The KD tree is a binary search tree and supports a data structure capable of managing points based on position, so that a nearest neighbor search (NNS) can be performed quickly. The Morton code is generated by presenting the coordinates (e.g., (x, y, z)) representing the 3D positions of all points as bit values ​​and mixing the bits. For example, when the coordinates representing the point position are (5, 9, 1), the bit values ​​of the coordinates are (0101, 1001, 0001). Mixing the bit values ​​in the order of z, y, and x according to the bit index produces 010001000111. This value is represented as a decimal number 1095. That is, the Morton code value of the point with coordinates (5, 9, 1) is 1095. The attribute converter 30007 can sort the points based on the Morton code value and perform NNS through a depth-first traversal process. After the attribute transform operation, KD tree or Morton code is used when NNS is needed in another transform process for attribute encoding.

[0087] As shown, the transformed attributes are input to the RAHT transformer 30008 and / or the LOD generator 30009.

[0088] The RAHT transformer 30008 according to an embodiment performs RAHT encoding for predicting attribute information based on the reconstructed geometric information. For example, the RAHT transformer 30008 may predict attribute information of a higher-level node in the octree based on attribute information associated with a lower-level node in the octree.

[0089] The LOD generator 30009 according to an embodiment generates a level of detail (LOD) to perform predictive transform coding. The LOD according to an embodiment is the degree of detail of the point cloud content. As the LOD value decreases, the details of the point cloud content are degraded. As the LOD value increases, the details of the point cloud content are enhanced. Points can be classified by LOD.

[0090] The lifting transformer 30010 according to an embodiment performs lifting transform coding that transforms point cloud attributes based on weights. As described above, lifting transform coding can be optionally applied.

[0091] The coefficient quantizer 30011 according to the embodiment quantizes the attribute of the attribute encoding based on the coefficient.

[0092] The arithmetic encoder 30012 according to the embodiment encodes quantized properties based on arithmetic coding.

[0093] Although not shown in the figure, Figure 3 The elements of the point cloud encoder may be implemented by hardware, software, firmware, or a combination thereof including one or more processors or integrated circuits configured to communicate with one or more memories included in the point cloud providing device. The one or more processors may execute the above Figure 3 At least one of the operations and / or functions of the elements of the point cloud encoder. In addition, one or more processors may be operable or executed to perform Figure 3 A software program and / or set of instructions for the operation and / or functionality of the elements of the point cloud encoder. One or more memories according to an embodiment may include high-speed random access memory, or include non-volatile memory (e.g., one or more disk storage devices, flash memory devices, or other non-volatile solid-state memory devices).

[0094] Figure 4 Examples of octrees and occupancy codes are shown according to an embodiment.

[0095] As reference Figures 1 to 3 As described, the point cloud content providing system (point cloud video encoder 10002) or the point cloud encoder (e.g., octree analyzer 30002) performs octree geometry encoding (or octree encoding) based on the octree structure to efficiently manage the area and / or position of voxels.

[0096] Figure 4 The upper part of FIG. 1 shows the octree structure. The 3D space of the point cloud content according to the embodiment is represented by the axes (eg, X-axis, Y-axis, and Z-axis) of the coordinate system. d ,2 d ,2 d ) to create an octree structure. Here, 2d can be set to the value of the smallest bounding box that constitutes all points around the point cloud content (or point cloud video). Here, d represents the depth of the octree. The value of d is determined in the following formula. In the following formula, (x int n ,y int n ,z int n ) represents the position (or position value) of the quantized point.

[0097] d=Ceil(Log2(Max(x_n^int,y_n^int,z_n^in,n=1,…,N)+1))

[0098] like Figure 4 As shown in the middle of the upper part of , the entire 3D space can be divided into eight spaces according to the partition. Each divided space is represented by a cube with six faces. Figure 4 As shown in the upper right part of , each of the eight spaces is divided again based on the axis of the coordinate system (e.g., X-axis, Y-axis, and Z-axis). Therefore, each space is divided into eight smaller spaces. The divided smaller spaces are also represented by cubes with six faces. This division scheme is applied until the leaf nodes of the octree become voxels.

[0099] Figure 4 The lower part shows the octree occupancy code. The occupancy code of the octree is generated to indicate whether each of the eight divided spaces generated by dividing one space contains at least one point. Therefore, a single occupancy code is represented by eight child nodes. Each child node represents the occupancy of the divided space, and the child node has a 1-bit value. Therefore, the occupancy code is represented as an 8-bit code. That is, when the space corresponding to the child node contains at least one point, the node is assigned a value of 1. When the space corresponding to the child node does not contain a point (the space is empty), the node is assigned a value of 0. Since Figure 4 The occupancy code shown is 00100001, thus indicating that the spaces corresponding to the third and eighth subnodes among the eight subnodes each contain at least one point. As shown in the figure, each of the third and eighth subnodes has eight subnodes, and the subnodes are represented by 8-bit occupancy codes. The accompanying drawings show that the occupancy code of the third subnode is 10000111, and the occupancy code of the eighth subnode is 01001111. The point cloud encoder (e.g., arithmetic encoder 30004) according to an embodiment may perform entropy encoding on the occupancy code. In order to increase compression efficiency, the point cloud encoder may perform intra / inter encoding on the occupancy code. The receiving device (e.g., receiving device 10004 or point cloud video decoder 10006) according to an embodiment reconstructs the occupancy code based on the occupancy code.

[0100] According to an embodiment, a point cloud encoder (e.g., Figure 4 The point cloud encoder or octree analyzer 30002) can perform voxelization and octree encoding to store point locations. However, points are not always evenly distributed in the 3D space, so there may be specific areas where fewer points exist. Therefore, performing voxelization on the entire 3D space is inefficient. For example, when a specific area contains few points, voxelization does not need to be performed in the specific area.

[0101] Therefore, for the above-mentioned specific area (or nodes other than the leaf nodes of the octree), the point cloud encoder according to the embodiment may skip voxelization and perform direct encoding to directly encode the point positions included in the specific area. The coordinates of the directly encoded points according to the embodiment are called direct coding mode (DCM). The point cloud encoder according to the embodiment may also perform triplet geometry encoding based on the surface model, which is to reconstruct the point positions in the specific area (or node) based on voxels. Triplet geometry encoding is a geometry encoding that represents an object as a series of triangular meshes. Therefore, the point cloud decoder can generate a point cloud from the mesh surface. Direct encoding and triplet geometry encoding according to the embodiment can be performed selectively. In addition, direct encoding and triplet geometry encoding according to the embodiment can be performed in combination with octree geometry encoding (or octree encoding).

[0102] In order to perform direct encoding, the option of using direct mode to apply direct encoding should be enabled. The node to which direct encoding is to be applied is not a leaf node, and there should be points less than a threshold value within a specific node. In addition, the total number of points to which direct encoding is to be applied should not exceed a preset threshold. When the above conditions are met, the point cloud encoder (or arithmetic encoder 30004) according to the embodiment can perform entropy encoding on the point position (or position value).

[0103] A point cloud encoder (e.g., surface approximation analyzer 30003) according to an embodiment may determine a specific level of the octree (a level less than the depth d of the octree), and may use a surface model starting from this level to perform triplet geometry encoding to reconstruct point positions in the node area based on voxels (triplet mode). A point cloud encoder according to an embodiment may specify a level to which triplet geometry encoding is to be applied. For example, when a specific level is equal to the depth of the octree, the point cloud encoder does not operate in triplet mode. In other words, a point cloud encoder according to an embodiment may operate in triplet mode only when the specified level is less than the depth value of the octree. A 3D cubic area of ​​a node at a specified level according to an embodiment is referred to as a block. A block may include one or more voxels. A block or voxel may correspond to a block. Geometry is represented as a surface within each block. A surface according to an embodiment may intersect each edge of a block at most once.

[0104] A block has 12 edges, so there are at least 12 intersections in a block. Each intersection is called a vertex. When there is at least one occupied voxel adjacent to the edge among all blocks sharing the edge, the vertex existing along the edge is detected. An occupied voxel according to an embodiment refers to a voxel containing a point. The vertex position detected along the edge is the average position of the edge of all voxels adjacent to the edge among all blocks along the shared edge.

[0105] Once the vertex is detected, the point cloud encoder according to the embodiment can perform entropy encoding on the start point (x, y, z) of the edge, the direction vector (Δx, Δy, Δz) of the edge, and the vertex position value (relative position value within the edge). When triplet geometry encoding is applied, the point cloud encoder according to the embodiment (e.g., geometry reconstructor 30005) can generate restored geometry (reconstructed geometry) by performing triangle reconstruction, upsampling, and voxelization processes.

[0106] The vertices located at the edges of the block determine the surface passing through the block. The surface according to an embodiment is a non-planar polygon. In the triangle reconstruction process, the surface represented by the triangle is reconstructed based on the starting point of the edge, the direction vector of the edge and the position value of the vertex. The triangle reconstruction process is performed by i) calculating the centroid value of each vertex, ii) subtracting the center value from each vertex value, and iii) estimating the sum of the squares of the values ​​obtained by subtraction.

[0107]

[0108] The minimum value of the sum is estimated, and the projection process is performed according to the axis with the minimum value. For example, when the element x is the smallest, each vertex is projected on the x-axis relative to the center of the block and projected on the (y,z) plane. When the value obtained by the projection on the (y,z) plane is (ai,bi), the value of θ is estimated by atan2(bi,ai), and the vertices are sorted based on the value of θ. The following table shows the vertex combination for creating a triangle according to the number of vertices. The vertices are sorted from 1 to n. The following table shows that for four vertices, two triangles can be constructed according to the vertex combination. The first triangle can be composed of vertices 1, 2, and 3 among the sorted vertices, and the second triangle can be composed of vertices 3, 4, and 1 among the sorted vertices.

[0109] Table 2-1. Triangles formed from vertices ordered 1,…,n

[0110] n triangle

[0111] 3(1,2,3)

[0112] 4(1,2,3),(3,4,1)

[0113] 5(1,2,3),(3,4,5),(5,1,3)

[0114] 6(1,2,3),(3,4,5),(5,6,1),(1,3,5)

[0115] 7(1,2,3),(3,4,5),(5,6,7),(7,1,3),(3,5,7)

[0116] 8(1,2,3),(3,4,5),(5,6,7),(7,8,1),(1,3,5),(5,7,1)

[0117] 9(1,2,3),(3,4,5),(5,6,7),(7,8,9),(9,1,3),(3,5,7),(7,9,3)

[0118] 10(1,2,3),(3,4,5),(5,6,7),(7,8,9),(9,10,1),(1,3,5),(5,7,9),(9,1,5)

[0119] 11(1,2,3),(3,4,5),(5,6,7),(7,8,9),(9,10,11),(11,1,3),(3,5,7),(7,9,11),(11,3,7)

[0120] 12(1,2,3),(3,4,5),(5,6,7),(7,8,9),(9,10,11),(11,12,1),(1,3,5),(5,7,9),(9,11,1),(1,5,9)

[0121] An upsampling process is performed to add points in the middle along the edges of the triangle, and voxelization is performed. The added points are generated based on the upsampling factor and the width of the block. The added points are called refined vertices. The point cloud encoder according to an embodiment may voxelize the refined vertices. In addition, the point cloud encoder may perform attribute encoding based on the voxelized position (or position value).

[0122] Figure 5 An example of point configuration in each LOD according to an embodiment is shown.

[0123] As reference Figures 1 to 4 As described, the encoded geometry is reconstructed (decompressed) before performing attribute encoding. When direct encoding is applied, the geometry reconstruction operation may include changing the placement of the directly encoded points (e.g., placing the directly encoded points in front of the point cloud data). When triplet geometry encoding is applied, the geometry reconstruction process is performed by triangle reconstruction, upsampling, and voxelization. Since the attributes depend on the geometry, attribute encoding is performed based on the reconstructed geometry.

[0124] The point cloud encoder (e.g., LOD generator 30009) can classify (or reorganize) the points by LOD. The figure shows the point cloud content corresponding to the LOD. The leftmost screen in the figure shows the original point cloud content. The second screen from the left in the figure shows the distribution of points in the lowest LOD, and the rightmost screen in the figure shows the distribution of points in the highest LOD. That is, the points in the lowest LOD are sparsely distributed, and the points in the highest LOD are densely distributed. That is, as the LOD increases in the direction indicated by the arrow indicated at the bottom of the figure, the space (or distance) between the points becomes narrower.

[0125] Figure 6 An example of point configuration for each LOD is shown according to an embodiment.

[0126] As reference Figures 1 to 5 As described, the point cloud content providing system or point cloud encoder (eg, point cloud video encoder 10002, Figure 3 The point cloud encoder or LOD generator 30009) can generate LOD. LOD is generated by reorganizing points into a set of refinement levels according to the set LOD distance value (or Euclidean distance set). The LOD generation process is performed not only by the point cloud encoder, but also by the point cloud decoder.

[0127] Figure 6 The upper part of shows examples of points (P0 to P9) of point cloud content distributed in 3D space. Figure 6 In , the original order represents the order of points P0 to P9 before LOD generation. Figure 6 In the LOD-based order, the order of points generated according to the LOD is shown. Points are reorganized by LOD. In addition, a high LOD contains points belonging to a lower LOD. Figure 6 As shown, LOD0 contains P0, P5, P4 and P2. LOD1 contains the points of LOD0, P1, P6 and P3. LOD2 contains the points of LOD0, the points of LOD1, P9, P8 and P7.

[0128] As reference Figure 3 As described, the point cloud encoder according to the embodiment may selectively or in combination perform prediction transform coding, lifting transform coding, and RAHT transform coding.

[0129] The point cloud encoder according to the embodiment may generate a predictor for a point to perform predictive transform coding for setting a prediction attribute (or a prediction attribute value) of each point. That is, N predictors may be generated for N points. The predictor according to the embodiment may calculate a weight (=1 / distance) based on the LOD value of each point, index information about neighboring points that exist within a set distance of each LOD, and the distance to the neighboring point.

[0130] The predicted attribute (or attribute value) according to the embodiment is set to the average of the values ​​obtained by multiplying the attribute (or attribute value) (e.g., color, reflectivity, etc.) of the neighboring points set in the predictor of each point by the weight (or weight value) calculated based on the distance to each neighboring point. The point cloud encoder (e.g., coefficient quantizer 30011) according to the embodiment may quantize and inverse quantize the residual (which may be referred to as residual attribute, residual attribute value or attribute prediction residual, attribute residual) obtained by subtracting the predicted attribute (attribute value) from the attribute (attribute value) of each point. The quantization process is configured as shown in the following table.

[0131] Table. Attribute prediction residual quantization pseudo code

[0132] int PCCQuantization(int value,int quantStep){

[0133] if(value>=0){

[0134] return floor(value / quantStep+1.0 / 3.0);

[0135] }else{

[0136] return-floor(-value / quantStep+1.0 / 3.0);

[0137] }

[0138] }

[0139] Table. Pseudocode for inverse quantization of attribute prediction residuals

[0140] int PCCInverseQuantization(int value,int quantStep){

[0141] if(quantStep==0){

[0142] return value;

[0143] }else{

[0144] return value*quantStep;

[0145] }

[0146] }

[0147] When the predictor of each point has a neighboring point, the point cloud encoder (e.g., the arithmetic encoder 30012) according to the embodiment may perform entropy encoding on the quantized and inverse quantized residual values ​​as described above. When the predictor of each point has no neighboring point, the point cloud encoder (e.g., the arithmetic encoder 30012) according to the embodiment may perform entropy encoding on the attributes of the corresponding point without performing the above operation.

[0148] The point cloud encoder (e.g., lifting transformer 30010) according to an embodiment may generate a predictor for each point, set the calculated LOD and register neighbor points in the predictor, and set weights according to the distance to the neighbor points to perform lifting transform encoding. The lifting transform encoding according to an embodiment is similar to the above-mentioned predictive transform encoding, but is different in that weights are cumulatively applied to attribute values. The process of cumulatively applying weights to attribute values ​​according to an embodiment is configured as follows.

[0149] 1) Create an array quantized weight (QW) for storing weight values ​​of each point. The initial value of all elements of QW is 1.0. Multiply the QW value of the predictor index of the neighbor node registered in the predictor by the weight of the predictor of the current point, and add the values ​​obtained by multiplication.

[0150] 2) Boosting prediction process: The value obtained by multiplying the attribute value of the point by the weight is subtracted from the existing attribute value to calculate the predicted attribute value.

[0151] 3) Create temporary arrays called updateweight and update, and initialize the temporary arrays to zero.

[0152] 4) Accumulate the weight calculated by multiplying the weight calculated for all predictors by the weight stored in the QW corresponding to the predictor index with the updateweight array as the index of the neighbor node. Accumulate the value obtained by multiplying the attribute value of the neighbor node index by the calculated weight with the update array.

[0153] 5) Boosting update process: Divide the attribute values ​​of the update array of all predictors by the weight value of the updateweight array of the predictor index, and add the existing attribute value to the value obtained by the division.

[0154] 6) For all predictors, the predicted attribute is calculated by multiplying the attribute value updated by the lifting update process by the weight (stored in QW) updated by the lifting prediction process. The point cloud encoder (e.g., coefficient quantizer 30011) according to the embodiment quantizes the predicted attribute value. In addition, the point cloud encoder (e.g., arithmetic encoder 30012) performs entropy encoding on the quantized attribute value.

[0155] A point cloud encoder according to an embodiment (e.g., RAHT transformer 30008) may perform RAHT transform coding, in which attributes associated with nodes at a lower level in an octree are used to predict attributes of nodes at a higher level. RAHT transform coding is an example of attribute intra-frame coding by scanning backward through an octree. A point cloud encoder according to an embodiment scans the entire area starting from voxels and repeats a merging process of merging voxels into larger blocks at each step until a root node is reached. The merging process according to an embodiment is performed only on occupied nodes. The merging process is not performed on empty nodes. The merging process is performed on the upper node directly above the empty node.

[0156] The following formula represents the RAHT transformation matrix. In this formula, Represents the average attribute value of voxels at level l. Can be based on and to calculate. and The weight of and

[0157]

[0158] Here, Is the low-pass value and is used during the next highest level of merging. Denotes the high-pass coefficient. The high-pass coefficient of each step is quantized and subjected to entropy coding (eg, encoded by arithmetic encoder 300012). The weight is calculated as Create the root node by g1 0,0,0 and g1 0,0,1 as follows.

[0159]

[0160] The values ​​of gDC are also quantized and subjected to entropy decoding like the high-pass coefficients.

[0161] Figure 7 A point cloud decoder according to an embodiment is shown.

[0162] Figure 7 The point cloud decoder shown is an example of a point cloud decoder and can perform a decoding operation, which is Figures 1 to 6 The inverse process of the encoding operation of the point cloud encoder shown.

[0163] As reference Figure 1 and Figure 6 As described, the point cloud decoder can perform geometry decoding and attribute decoding. Geometry decoding is performed before attribute decoding.

[0164] According to an embodiment, a point cloud decoder includes an arithmetic decoder (arithmetic decoding) 7000, an octree synthesizer (synthesized octree) 7001, a surface approximation synthesizer (synthesized surface approximation) 7002 and a geometry reconstructor (reconstructed geometry) 7003, an inverse coordinate transformer (inverse transformed coordinates) 7004, an arithmetic decoder (arithmetic decoding) 7005, an inverse quantizer (inverse quantization) 7006, a RAHT transformer 7007, an LOD generator (generate LOD) 7008, an inverse lifting (inverse lifting) 7009 and / or an inverse color transformer (inverse transformed color) 7010.

[0165] The arithmetic decoder 7000, the octree synthesizer 7001, the surface approximation synthesizer 7002, the geometry reconstructor 7003, and the coordinate inverse transformer 7004 may perform geometry decoding. The geometry decoding according to the embodiment may include direct decoding and triplet geometry decoding. Direct encoding and triplet geometry decoding are selectively applied. The geometry decoding is not limited to the above examples, and as a reference Figures 1 to 6 The inverse process of the geometric encoding described is performed.

[0166] The arithmetic decoder 7000 according to the embodiment decodes the received geometry bitstream based on arithmetic coding. The operation of the arithmetic decoder 7000 corresponds to the inverse process of the arithmetic encoder 30004.

[0167] The octree synthesizer 7001 according to the embodiment may generate an octree by acquiring an occupancy code (or information about the geometry obtained as a result of decoding) from the decoded geometry bitstream. Figures 1 to 6 Describe the configuration in detail.

[0168] When triplet geometry encoding is applied, the surface approximation synthesizer 7002 according to an embodiment may synthesize a surface based on the decoded geometry and / or the generated octree.

[0169] According to an embodiment, the geometry reconstructor 7003 may regenerate the geometry based on the surface and / or the decoded geometry. Figures 1 to 9 As described, direct coding and triplet geometry coding are selectively applied. Therefore, the geometry reconstructor 7003 directly imports the position information about the points to which direct coding is applied and adds them. When triplet geometry coding is applied, the geometry reconstructor 7003 can reconstruct the geometry by performing the reconstruction operations (e.g., triangle reconstruction, upsampling, and voxelization) of the geometry reconstructor 30005. Details and References Figure 6 Those described are the same, so their description is omitted. The reconstructed geometry may include a point cloud screen or frame that does not contain attributes.

[0170] The coordinate inverse transformer 7004 according to an embodiment may acquire a point position by transforming the coordinates based on the reconstructed geometry.

[0171] The arithmetic decoder 7005, the inverse quantizer 7006, the RAHT transformer 7007, the LOD generator 7008, the inverse lifting 7009 and / or the color inverse transformer 7010 may perform reference Figure 6 Attribute decoding described. Attribute decoding according to an embodiment includes regional adaptive hierarchical transform (RAHT) decoding, hierarchical nearest neighbor prediction (prediction transform) decoding based on interpolation, and hierarchical nearest neighbor prediction (lifting transform) decoding based on interpolation with an update / lifting step. The above three decoding schemes can be used selectively, or a combination of one or more decoding schemes can be used. Attribute decoding according to an embodiment is not limited to the above examples.

[0172] The arithmetic decoder 7005 according to the embodiment decodes the attribute bit stream through arithmetic coding.

[0173] The inverse quantizer 7006 according to the embodiment inversely quantizes information about the decoded attribute bitstream or the attribute obtained as a result of decoding, and outputs the inversely quantized attribute (or attribute value). The inverse quantization may be selectively applied based on the attribute encoding of the point cloud encoder.

[0174] According to an embodiment, the RAHT transformer 7007, the LOD generator 7008 and / or the inverse lifter 7009 may process the reconstructed geometry and inverse quantized attributes. As described above, the RAHT transformer 7007, the LOD generator 7008 and / or the inverse lifter 7009 may selectively perform a decoding operation corresponding to the encoding of the point cloud encoder.

[0175] The color inverse transformer 7010 according to an embodiment performs inverse transform encoding to inversely transform a color value (or texture) included in the decoded attribute. The operation of the color inverse transformer 7010 may be selectively performed based on the operation of the color transformer 30006 of the point cloud encoder.

[0176] Although not shown in the figure, Figure 7 The elements of the point cloud decoder may be implemented by hardware, software, firmware, or a combination thereof including one or more processors or integrated circuits configured to communicate with one or more memories included in the point cloud providing device. The one or more processors may execute the above Figure 7 At least one or more of the operations and / or functions of the elements of the point cloud decoder. In addition, one or more processors may be operable or executed to perform Figure 7 A software program and / or instruction set for the operation and / or functionality of the elements of a point cloud decoder.

[0177] Figure 8 A transmitting device according to an embodiment is shown.

[0178] Figure 8The sending device shown is Figure 1 The sending device 10000 (or Figure 3 Example of a point cloud encoder). Figure 8 The sending device shown can be executed with reference to Figures 1 to 6 The transmitting device according to the embodiment may include a data input unit 8000, a quantization processor 8001, a voxelization processor 8002, an octree occupancy code generator 8003, a surface model processor 8004, an intra-frame / inter-frame coding processor 8005, an arithmetic encoder 8006, a metadata processor 8007, a color transformation processor 8008, an attribute transformation processor 8009, a prediction / lifting / RAHT transformation processor 8010, an arithmetic encoder 8011 and / or a transmission processor 8012.

[0179] The data input unit 8000 according to the embodiment receives or acquires point cloud data. The data input unit 8000 may perform the same operation and / or acquisition method as the point cloud video acquirer 10001 (or refer to Figure 2 The acquisition process 20000) described above has the same or similar operations and / or acquisition methods.

[0180] The data input unit 8000, the quantization processor 8001, the voxelization processor 8002, the octree occupancy code generator 8003, the surface model processor 8004, the intra / inter coding processor 8005 and the arithmetic encoder 8006 perform geometric coding. Figures 1 to 9 The geometrical encodings described are the same or similar, so their detailed description is omitted.

[0181] The quantization processor 8001 according to an embodiment quantizes geometry (e.g., position values ​​of points). The operation and / or quantization of the quantization processor 8001 is similar to the reference Figure 3 The operation and / or quantization of the quantizer 30001 described above are the same or similar. Figures 1 to 9 Same as those described.

[0182] According to an embodiment, the voxelization processor 8002 voxelizes the quantized position value of the point. The voxelization processor 8002 may perform the same as the reference Figure 3 The operation and / or voxelization process of the quantizer 30001 described above is the same or similar to the operation and / or process described above. Figures 1 to 6 Same as those described.

[0183] The octree occupancy code generator 8003 according to the embodiment performs octree encoding on the voxelized position of the point based on the octree structure. The octree occupancy code generator 8003 may generate an occupancy code. The octree occupancy code generator 8003 may perform the same as the reference code. Figure 3 and Figure 4 The operations and / or methods of the point cloud encoder (or octree analyzer 30002) described herein are the same or similar to the operations and / or methods described herein. Figures 1 to 6 Same as those described.

[0184] According to an embodiment, the surface model processor 8004 may perform triplet geometry encoding based on the surface model to reconstruct the point position in a specific area (or node) based on voxels. Figure 3 The operations and / or methods of the point cloud encoder (e.g., surface approximation analyzer 30003) described above are the same or similar to the operations and / or methods. Figures 1 to 6 Same as those described.

[0185] According to an embodiment, the intra-frame / inter-frame encoding processor 8005 may perform intra-frame / inter-frame encoding on the point cloud data. The intra-frame / inter-frame encoding processor 8005 may perform the same as the reference. Figure 7 The intra / inter coding described is the same or similar to the coding. Details and references Figure 7 The intra / inter encoding processor 8005 may be included in the arithmetic encoder 8006 according to an embodiment.

[0186] According to an embodiment, the arithmetic encoder 8006 performs entropy coding on the octree and / or approximate octree of the point cloud data. For example, the coding scheme includes arithmetic coding. The arithmetic encoder 8006 performs the same or similar operations and / or methods as the operations and / or methods of the arithmetic encoder 30004.

[0187] The metadata processor 8007 according to an embodiment processes metadata (e.g., set values) about the point cloud data and provides it to necessary processing processes such as geometry coding and / or attribute coding. In addition, the metadata processor 8007 according to an embodiment may generate and / or process signaling information related to geometry coding and / or attribute coding. The signaling information according to an embodiment may be encoded separately from the geometry coding and / or attribute coding. The signaling information according to an embodiment may be interleaved.

[0188] The color conversion processor 8008, the attribute conversion processor 8009, the prediction / lifting / RAHT conversion processor 8010 and the arithmetic encoder 8011 perform attribute encoding. Figures 1 to 6 The attribute codes described are the same or similar, so their detailed description is omitted.

[0189] The color transform processor 8008 according to an embodiment performs color transform encoding to transform the color value included in the attribute. The color transform processor 8008 may perform color transform encoding based on the reconstructed geometry. The reconstructed geometry is compared with the reference Figures 1 to 9 In addition, its execution is the same as that of reference Figure 3 The operations and / or methods of the color converter 30006 described above are the same as or similar to the operations and / or methods described above, and detailed descriptions thereof are omitted.

[0190] The attribute transformation processor 8009 according to an embodiment performs attribute transformation to transform attributes based on reconstructed geometry and / or locations where geometry encoding is not performed. The attribute transformation processor 8009 performs the attribute transformation with reference to Figure 3 The operations and / or methods of the attribute converter 30007 described in the foregoing description are the same as or similar to those of the attribute converter 30007. Detailed description thereof is omitted. The prediction / lifting / RAHT transform processor 8010 according to the embodiment may encode the transformed attribute by any one or a combination of RAHT coding, prediction transform coding, and lifting transform coding. The prediction / lifting / RAHT transform processor 8010 performs the same as or similar to the reference. Figure 3 The operations of the RAHT transformer 30008, the LOD generator 30009 and the lifting transformer 30010 described above are the same or similar to at least one operation. In addition, the prediction transform coding, the lifting transform coding and the RAHT transform coding are the same as those of the reference Figures 1 to 9 Those described are the same, so their detailed description is omitted.

[0191] The arithmetic encoder 8011 according to the embodiment may encode the encoded attribute based on arithmetic coding. The arithmetic encoder 8011 performs the same or similar operation and / or method as that of the arithmetic encoder 300012.

[0192] The transmission processor 8012 according to the embodiment may send individual bitstreams containing encoded geometry and / or encoded attribute and metadata information, or send a bitstream configured with encoded geometry and / or encoded attribute and metadata information. When the encoded geometry and / or encoded attribute and metadata information according to the embodiment is configured into a bitstream, the bitstream may include one or more sub-bitstreams. The bitstream according to the embodiment may include signaling information and slice data, and the signaling information includes a sequence parameter set (SPS) for sequence level signaling, a geometry parameter set (GPS) for geometry information coding signaling, an attribute parameter set (APS) for attribute information coding signaling, and a patch parameter set (TPS) for patch level signaling. Slice data may include information about one or more slices. A slice according to the embodiment may include a geometry bitstream Geom00 and one or more attribute bitstreams Attr00 and Attr10.

[0193] A slice is a sequence of syntactic elements that represents the entirety or a portion of an encoded point cloud frame.

[0194] The TPS according to an embodiment may include information about each of the one or more tiles (e.g., coordinate information and height / size information about a bounding box). The geometry bitstream may include a header and a payload. The header of the geometry bitstream according to an embodiment may include a parameter set identifier (geom_parameter_set_id), a tile identifier (geom_tile_id), and a slice identifier (geom_slice_id) included in the GPS, as well as information about the data contained in the payload. As described above, the metadata processor 8007 according to an embodiment may generate and / or process signaling information and send it to the transmission processor 8012. According to an embodiment, an element that performs geometry encoding and an element that performs attribute encoding may share data / information with each other as indicated by the dotted lines. The transmission processor 8012 according to an embodiment may perform operations and / or transmission methods that are the same or similar to those of the transmitter 10003. Details and References Figure 1 and Figure 2 Those described are the same, so their description is omitted.

[0195] Figure 9 A receiving device according to an embodiment is shown.

[0196] Figure 9 The receiving device shown is Figure 1 The receiving device 10004 (or Figure 10 and Figure 11 Example of a point cloud decoder. Figure 9 The receiving device shown can execute and refer to Figures 1 to 11 One or more operations and methods that are the same or similar to those of the described point cloud decoder.

[0197] The receiving device according to the embodiment may include a receiver 9000, a receiving processor 9001, an arithmetic decoder 9002, an octtree reconstruction processor based on an occupancy code 9003, a surface model processor (triangle reconstruction, upsampling, voxelization) 9004, an inverse quantization processor 9005, a metadata parser 9006, an arithmetic decoder 9007, an inverse quantization processor 9008, a prediction / lifting / RAHT inverse transform processor 9009, a color inverse transform processor 9010 and / or a renderer 9011. Each decoding element according to the embodiment may perform an inverse process of the operation of the corresponding encoding element according to the embodiment.

[0198] The receiver 9000 according to the embodiment receives the point cloud data. The receiver 9000 may perform the same Figure 1The operation and / or receiving method of the receiver 10005 is the same as or similar to the operation and / or receiving method of the receiver 10005. The detailed description thereof is omitted.

[0199] The reception processor 9001 according to an embodiment may acquire a geometry bitstream and / or an attribute bitstream from the received data. The reception processor 9001 may be included in the receiver 9000 .

[0200] The arithmetic decoder 9002, the octtree reconstruction processor 9003 based on the occupancy code, the surface model processor 9004, and the inverse quantization processor 9005 may perform geometric decoding. Figures 1 to 10 The geometric decoding described is the same or similar, so its detailed description is omitted.

[0201] The arithmetic decoder 9002 according to the embodiment may decode the geometry bitstream based on arithmetic coding. The arithmetic decoder 9002 performs the same or similar operations and / or encoding as those of the arithmetic decoder 7000.

[0202] The octree reconstruction processor 9003 based on the occupancy code according to the embodiment can reconstruct the octree by obtaining the occupancy code from the decoded geometry bitstream (or information about the geometry obtained as a result of decoding). The octree reconstruction processor 9003 based on the occupancy code performs the same or similar operations and / or methods as the operation of the octree synthesizer 7001 and / or the octree generation method. When triplet geometry coding is applied, the surface model processor 9004 according to the embodiment can perform triplet geometry decoding and related geometry reconstruction (e.g., triangle reconstruction, upsampling, voxelization) based on the surface model method. The surface model processor 9004 performs the same or similar operations as the surface approximation synthesizer 7002 and / or the geometry reconstructor 7003.

[0203] The inverse quantization processor 9005 according to an embodiment may inversely quantize the decoded geometry.

[0204] According to an embodiment, the metadata parser 9006 may parse metadata (eg, setting values) contained in the received point cloud data. The metadata parser 9006 may pass the metadata to the geometry decoding and / or attribute decoding. Figure 8 The metadata described is the same, so a detailed description thereof is omitted.

[0205] The arithmetic decoder 9007, the inverse quantization processor 9008, the prediction / lifting / RAHT inverse transform processor 9009 and the color inverse transform processor 9010 perform attribute decoding. Figures 1 to 10 The attribute decoding described is the same or similar, so the detailed description thereof is omitted.

[0206] The arithmetic decoder 9007 according to the embodiment may decode the attribute bitstream by arithmetic coding. The arithmetic decoder 9007 may decode the attribute bitstream based on the reconstructed geometry. The arithmetic decoder 9007 performs the same or similar operation and / or encoding as that of the arithmetic decoder 7005.

[0207] The inverse quantization processor 9008 according to an embodiment may inversely quantize the decoded attribute bitstream. The inverse quantization processor 9008 performs the same or similar operation and / or method as the inverse quantizer 7006 and / or the inverse quantization method.

[0208] The prediction / lifting / RAHT inverse transform processor 9009 according to an embodiment may process the reconstructed geometry and inverse quantized attributes. The prediction / lifting / RAHT inverse transform processor 9009 performs one or more operations and / or decoding that are the same or similar to the operations and / or decoding of the RAHT converter 7007, the LOD generator 7008 and / or the inverse lifter 7009. The color inverse transform processor 9010 according to an embodiment performs inverse transform encoding to inversely transform the color value (or texture) included in the decoded attribute. The color inverse transform processor 9010 performs the same or similar operations and / or inverse transform encoding as the operations and / or inverse transform encoding of the color inverse converter 7010. The renderer 9011 according to an embodiment may render point cloud data.

[0209] Figure 10 An exemplary structure that can be operated in conjunction with a point cloud data transmitting / receiving method / apparatus according to an embodiment is shown.

[0210] Figure 10 The structure of 1060 represents a configuration in which at least one of the server 1060, the robot 1010, the self-driving vehicle 1020, the XR device 1030, the smart phone 1040, the home appliance 1050, and / or the head mounted display (HMD) 1070 is connected to the cloud network 1000. The robot 1010, the self-driving vehicle 1020, the XR device 1030, the smart phone 1040, or the home appliance 1050 is referred to as a device. In addition, the XR device 1030 may correspond to a point cloud data (PCC) device according to an embodiment or may be operably connected to a PCC device.

[0211] The cloud network 1000 may represent a network that constitutes a part of a cloud computing infrastructure or exists in a cloud computing infrastructure. Here, the cloud network 1000 may be configured using a 3G network, a 4G or Long Term Evolution (LTE) network, or a 5G network.

[0212] The server 1060 may be connected to at least one of the robot 1010 , the self-driving vehicle 1020 , the XR device 1030 , the smart phone 1040 , the home appliance 1050 , and / or the HMD 1070 via the cloud network 1000 , and may assist in at least a portion of the processing of the connected devices 1010 to 1070 .

[0213] The HMD 1070 represents one of implementation types of an XR device and / or a PCC device according to an embodiment. The HMD type device according to an embodiment includes a communication unit, a control unit, a memory, an I / O unit, a sensor unit, and a power supply unit.

[0214] Hereinafter, various embodiments of devices 1010 to 1050 to which the above-described technology is applied will be described. Figure 10 The illustrated devices 1010 to 1050 may be operably connected / coupled to the point cloud data sending device and receiving device according to the above-described embodiments.

[0215] <PCC + XR>

[0216] The XR / PCC device 1030 may employ PCC technology and / or XR (AR+VR) technology, and may be implemented as an HMD, a head-up display (HUD) disposed in a vehicle, a television, a mobile phone, a smart phone, a computer, a wearable device, a home appliance, a digital signage, a vehicle, a fixed robot, or a mobile robot.

[0217] The XR / PCC device 1030 may analyze 3D point cloud data or image data acquired through various sensors or from an external device and generate position data and attribute data about 3D points. Thus, the XR / PCC device 1030 may acquire information about the surrounding space or real objects, and render and output XR objects. For example, the XR / PCC device 1030 may match an XR object including auxiliary information about the recognized object with the recognized object and output the matched XR object.

[0218] <PCC + XR + Mobile Phone>

[0219] The XR / PCC device 1030 may be implemented as a mobile phone 1040 by applying the PCC technology.

[0220] The mobile phone 1040 may decode and display point cloud content based on the PCC technology.

[0221] <PCC + Self - Driving + XR>

[0222] The self-driving vehicle 1020 may be implemented as a mobile robot, a vehicle, an unmanned aerial vehicle, etc. by applying PCC technology and XR technology.

[0223] The self-driving vehicle 1020 to which the XR / PCC technology is applied may represent a self-driving vehicle provided with a means for providing an XR image, or a self-driving vehicle as a control / interaction target in the XR image. Specifically, as a control / interaction target in the XR image, the self-driving vehicle 1020 may be distinguished from the XR device 1030 and may be operably connected thereto.

[0224] The self-driving vehicle 1020 having means for providing XR / PCC images may acquire sensor information from sensors including cameras, and output generated XR / PCC images based on the acquired sensor information. For example, the self-driving vehicle 1020 may have a HUD and output XR / PCC images thereto, thereby providing passengers with XR / PCC objects corresponding to real objects or objects presented on the screen.

[0225] When the XR / PCC object is output to the HUD, at least a portion of the XR / PCC object may be output to overlap with the real object to which the passenger's eyes are directed. On the other hand, when the XR / PCC object is output on a display provided in the self-driving vehicle, at least a portion of the XR / PCC object may be output to overlap with the object on the screen. For example, the self-driving vehicle 1220 may output XR / PCC objects corresponding to objects such as a road, another vehicle, a traffic light, a traffic sign, a two-wheeled vehicle, a pedestrian, and a building.

[0226] Virtual reality (VR) technology, augmented reality (AR) technology, mixed reality (MR) technology, and / or point cloud compression (PCC) technology according to an embodiment are applicable to various devices.

[0227] In other words, VR technology is a display technology that only provides CG images of real-world objects, backgrounds, etc. On the other hand, AR technology refers to a technology that displays a virtually created CG image on an image of a real object. MR technology is similar to the above-mentioned AR technology in that the virtual objects to be displayed are mixed and combined with the real world. However, MR technology is different from AR technology in that AR technology clearly distinguishes between real objects and virtual objects created as CG images and uses virtual objects as supplementary objects to real objects, while MR technology treats virtual objects as objects with properties equivalent to real objects. More specifically, an example of the application of MR technology is a holographic service.

[0228] Recently, VR, AR, and MR technologies are often referred to as extended reality (XR) technologies, rather than being clearly distinguished from each other. Therefore, embodiments of the present disclosure are applicable to any of VR, AR, MR, and XR technologies. Encoding / decoding based on PCC, V-PCC, and G-PCC technologies are applicable to such technologies.

[0229] The PCC method / apparatus according to the embodiment may be applied to a vehicle that provides a self-driving service.

[0230] Vehicles providing self-driving services are connected to the PCC device for wired / wireless communication.

[0231] When a point cloud data (PCC) transmitting / receiving device according to an embodiment is connected to a vehicle for wired / wireless communication, the device may receive / process content data related to an AR / VR / PCC service (which may be provided together with a self-driving service) and send it to the vehicle. In the case where the PCC transmitting / receiving device is installed on the vehicle, the PCC transmitting / receiving device may receive / process content data related to the AR / VR / PCC service and provide it to the user based on a user input signal input through a user interface device. The vehicle or user interface device according to the embodiment may receive a user input signal. The user input signal according to the embodiment may include a signal indicating a self-driving service.

[0232] The point cloud data transmission method / device according to the embodiment can be interpreted as involving Figure 1 The sending device 10000, Figure 1 Point cloud video encoder 10002, Figure 1 Transmitter 10003, Figure 2 Acquisition 20000 / Encoding 20001 / Transmission 20002, Figure 3 The encoder, Figure 8 The sending device Figure 10 Devices Figure 12 , Figure 21 , Figure 28 and Figure 29 The encoder, Figure 23 and Figure 30 Terms such as the method of sending.

[0233] The point cloud data receiving method / device according to the embodiment can be interpreted as involving Figure 1 The receiving device 10004, Figure 1 Receiver 10005, Figure 1 Point cloud video decoder 10006, Figure 2 Transmission 20002 / decoding 20003 / rendering 20004, Figure 7 The decoder, Figure 9 The receiving device Figure 10 Devices Figure 11 , Figure 21 , Figure 28 and Figure 29 The decoder, Figure 24 and Figure 31 Terms such as reception method, etc.

[0234] In addition, the point cloud data sending / receiving method / apparatus according to the implementation manner may be simply referred to as method / apparatus.

[0235] According to an embodiment, geometric data, geometric information, position information and geometry constituting point cloud data will be interpreted as having the same meaning. Attribute data and attribute information constituting point cloud data will be interpreted as having the same meaning.

[0236] The method / apparatus according to an embodiment may include and perform implicit and explicit context referencing for point cloud compression based on attribute layer group slicing.

[0237] Embodiments include a method for efficiently supporting selective decoding of a portion of data when necessary due to receiver performance or transmission speed of sending and receiving point cloud data. Embodiments include a method for selecting necessary information or removing unnecessary information in a bitstream unit by dividing geometry and attribute data delivered per data unit into semantic units such as geometry octrees and levels of detail (LoD).

[0238] Embodiments include a method of configuring a data structure composed of a point cloud. More specifically, embodiments disclose a packaging and signaling method for efficiently transmitting layer-configured point cloud compression (PCC) data, and include a method of applying the packaging and signaling method to a scalable PCC-based service. In particular, embodiments include a method of configuring slice segments and sending and receiving the slice segments so as to be more suitable for scalable PCC services when direct compression mode is used for position compression. Specifically, embodiments can provide a compression structure for efficiently storing and sending large point cloud data with wide distribution and high point density.

[0239] Reference Figure 3 , point cloud data consists of the location (geometry) (e.g., XYZ coordinates) and attributes (e.g., color, reflectivity, intensity, grayscale, opacity, etc.) of each data. In point cloud compression (PCC), octree-based compression is performed to efficiently compress the distribution characteristics of uneven distribution in 3D space, and attribute information is compressed based on this. Figure 3 The operations according to the embodiment may be processed by corresponding components of the transmitting side and the receiving side of the PCC.

[0240] Figure 11 The process of encoding, transmitting, and decoding point cloud data according to an embodiment is illustrated.

[0241] Figure 11 Each component of may correspond to hardware, software, a processor and / or a combination thereof.

[0242] The point cloud encoder 15000 is a transmission device that performs the transmission method according to the embodiment, and can scalably encode and transmit point cloud data.

[0243] The point cloud decoder 15010 is a receiving device that performs the receiving method according to the embodiment and can decode the point cloud data in a scalable manner.

[0244] The source data received by encoder 15000 may include geometric data and / or attribute data.

[0245] The encoder 15000 encodes the point cloud data scalably, but does not immediately generate a partial PCC bitstream. Instead, when it receives full geometry data and full attribute data, it stores the data in a storage device connected to the encoder. The encoder can then perform transcoding on the partial encoding, generate and send a partial PCC bitstream. The decoder 15010 can receive and decode the partial PCC bitstream to reconstruct partial geometry and / or partial attributes.

[0246] Upon receiving the full geometry and full attributes, the encoder 15000 may store the data in a storage device connected to the encoder and transcode the point cloud data using a low quantization parameter (QP) to generate and send a complete PCC bitstream. The decoder 15010 may receive and decode the complete PCC bitstream to reconstruct the full geometry and / or full attributes. The decoder 15010 may select partial geometry and / or partial attributes from the complete PCC bitstream through data selection.

[0247] The method / apparatus according to the present embodiment compresses and transmits point cloud data by dividing the position information about the data points as point cloud data and the characteristic information such as color / brightness / reflectivity into geometric information and attribute information. In this case, an octree structure with layers may be configured according to the degree of detail, or PCC data may be configured according to the level of detail (LoD). Then, scalable point cloud data encoding and representation may be performed based on the configured structure or data. In this case, only a portion of the point cloud data may be decoded or represented due to the performance of the receiver or the transmission rate.

[0248] In this process, unnecessary data can be removed in advance according to the method / apparatus of the embodiment. In other words, when only a portion of the scalable PCC bit stream needs to be sent (i.e., only some layers are decoded in scalable decoding), it is not possible to select and send only the necessary portion. Therefore, 1) the necessary portion needs to be re-encoded after decoding (15020), or 2) the receiver must selectively apply the operation (15030) after the entire data is transmitted to the receiver. However, in case 1), a delay may occur due to the time used for decoding and re-encoding (15020). In case 2), bandwidth efficiency may be reduced due to the transmission of unnecessary data. In addition, when a fixed bandwidth is used, it may be necessary to reduce data quality for transmission (15030).

[0249] Therefore, the method / apparatus according to the embodiment may define a slice segmentation structure of point cloud data, and signal a scalable layer and slice structure for scalable transmission.

[0250] In an embodiment, to ensure efficient bitstream delivery and decoding, the bitstream may be divided into specific units to be processed.

[0251] For octree-based geometry compression, the method / apparatus according to the embodiment may use entropy-based encoding and direct encoding together. In this case, a slice configuration for efficiently utilizing scalability is required.

[0252] The unit according to the embodiment may be referred to as LOD, layer, slice, etc. LOD is the same term as LOD in attribute data encoding, but may mean a data unit for the hierarchical structure of a bitstream. It may be a concept of a bundle corresponding to a depth or two or more depths based on the hierarchical structure of point cloud data, such as the depth (level) of an octree or multiple trees. Similarly, a layer is provided to generate a unit of a sub-bitstream, and a layer is a bundle corresponding to a depth or two or more depths and may correspond to a LOD or a concept of two or more LODs. In addition, a slice is a unit for configuring a sub-bitstream unit, and may correspond to a depth, a part of a depth, or two or more depths. Moreover, it may correspond to a LOD, a part of a LOD, or two or more LODs. According to the embodiment, LOD, layer and slice may correspond to each other, or one of LOD, layer and slice may be included in another. In addition, the unit according to the embodiment may include LOD, layer, slice, layer group or subgroup, and may be referred to as complementary to each other.

[0253] In addition, in the case of large point clouds with wide distribution and high density of points, latency issues may arise due to the many bitstreams that need to be processed to access the area of ​​interest. The method / apparatus according to the embodiment can solve the latency issue. For a hierarchical point cloud structure for compressing large point cloud data with low latency according to the embodiment, a tile / brick / slice layer group / slice subgroup, tile / slice layer group / slice subgroup, slice layer group / slice subgroup, or layer group bbox->subgroup bbox relationship can be used.

[0254] Figure 12 A layer-based point cloud data configuration and the structure of geometry and attribute bitstreams according to an embodiment are shown.

[0255] The sending method / device according to the embodiment may be configured as follows Figure 12 Layer-based point cloud data is shown to encode and decode point cloud data.

[0256] The embodiments are directed to efficient transmission and decoding by selectively transmitting and decoding data in bitstream units of point cloud data configured by layers.

[0257] According to the application field, the stratification of point cloud data can have a layer structure in terms of SNR, spatial resolution, color, temporal frequency, bit depth, etc., and the layers can be constructed in the direction of increasing data density based on the octree structure or the LoD structure.

[0258] The method / apparatus according to the embodiment can be based on Figure 16 The layering shown is used to configure, encode and decode geometry bitstreams and attribute bitstreams.

[0259] The bit stream obtained by the sending device / encoder through point cloud compression according to the embodiment of the present invention can be divided into a geometric data bit stream and an attribute data bit stream according to the type of data and sent.

[0260] Each bitstream according to an embodiment may be composed of slices. Regardless of the layer information or LoD information, the geometry data bitstream and the attribute data bitstream may each be configured as a slice and delivered. In this case, when only a portion of the layer or LoD is used, the following operations should be performed: 1) decode the bitstream, 2) select only the desired portion and remove unnecessary portions, and 3) perform encoding again based only on necessary information.

[0261] Figure 13 A bitstream configuration according to an embodiment is shown.

[0262] The sending method / device according to the implementation mode may generate Figure 13 The bit stream shown in FIG. 1 and the receiving method / device according to the embodiment can be used for the bit stream as shown in FIG.Figure 13 The point cloud data included in the bitstream shown is decoded.

[0263] Bitstream Configuration According to Implementation

[0264] In an embodiment, in order to avoid unnecessary intermediate processing, the bitstream may be divided into layers (or LoDs) and transmitted.

[0265] For example, in a LoD-based PCC structure, a lower LoD is included in a higher LoD. Information included in the current LoD but not included in the previous LoD (i.e., information newly included in each LoD) can be referred to as R (remaining part). Figure 17 As shown, the initial LoD information and the information R newly included in each LoD may be divided and transmitted in each independent unit.

[0266] The transmission method / device according to the embodiment may encode the geometry data and generate a geometry bitstream. The geometry bitstream may be configured for each LOD or layer. The geometry bitstream may include a header (geometry header) for each LOD or layer. The header may include reference information for the next LOD or next layer. The current LOD (layer) may also include information R (geometry data) not included in the previous LOD (layer).

[0267] The receiving method / device according to the embodiment may encode the attribute data and generate an attribute bitstream. The attribute bitstream may be configured for each LOD or layer, and the attribute bitstream may include a header (attribute header) for each LOD or layer. The header may include reference information for the next LOD or next layer. The current LOD (layer) may also include information R (attribute data) not included in the previous LOD (layer).

[0268] The receiving method / apparatus according to the embodiment can receive a bitstream composed of LODs or layers and efficiently decode only necessary data without complicated intermediate processing.

[0269] Figure 14 A bitstream sorting method according to an embodiment is illustrated.

[0270] like Figure 14 As shown, the method / device according to the embodiment can be used for Figure 13 Sort the bit stream.

[0271] Bitstream sorting method according to an embodiment

[0272] like Figure 14As shown, when sending a bitstream, the sending method / device according to the embodiment can send geometry and attributes serially. In this case, depending on the data type, the entire geometry information (geometry data) can be sent first, and then the attribute information (attribute data) can be sent. In this case, the geometry information can be quickly reconstructed based on the sent bitstream information.

[0273] exist Figure 14 In (a), for example, a layer (LOD) containing geometry data may be positioned first in the bitstream, and a layer (LOD) containing attribute data may be positioned after the geometry layer. Since the attribute data depends on the geometry data, the geometry layer may be positioned first. Furthermore, the positions may be changed differently depending on the implementation. References may also be made between geometry headers and between attribute headers and geometry headers.

[0274] Reference Figure 14 (b), a bitstream constituting the same layer including geometry data and attribute data can be collected and delivered. In this case, the decoding execution time can be reduced by using compression techniques that can decode geometry and attributes in parallel. In this case, the information that needs to be processed first (lower LoD, where geometry must precede attributes) can be placed first.

[0275] The first layer 1800 includes the geometry data and attribute data corresponding to the lowest LOD0 (layer 0) together with respective headers. The second layer 1810 includes LOD0 (layer 0) and also includes the geometry data and attribute data of the points of the new and more detailed layer 1 (LOD1) as information R1, which are not included in LOD0 (layer 0). The third layer 1820 can then be placed in a similar manner.

[0276] The transmission / reception method / device according to the embodiment can efficiently select the desired layer (or LoD) in the application field at the bitstream level when transmitting and receiving the bitstream. In the bitstream sorting method according to the embodiment, the geometric information ( Figure 18 ) may generate empty parts in the middle after bitstream level selection. In this case, the bitstream may need to be rearranged. In the case of bundling and delivering geometry and attributes according to each layer ( Figure 14 ), unnecessary information can be selectively removed according to the application field as follows.

[0277] Figure 15 A method of selecting geometric data and attribute data according to an embodiment is illustrated.

[0278] The bitstream selection of the method / apparatus according to an embodiment is performed as follows.

[0279] When it is necessary to select a bitstream as described above, the method / apparatus according to the implementation mode can select data at the bitstream level: 1) symmetric selection of geometry and attributes; 2) asymmetric selection of geometry and attributes; or 3) a combination of the above two methods.

[0280] 1) Symmetric selection of geometry and attributes

[0281] Reference Figure 15 , which illustrates a case where only LoDs up to LoD1 (LOD0+R1) are selected (19000) and transmitted or decoded, and information corresponding to R2 (new part in LOD2) corresponding to the upper layer is removed for transmission / decoding.

[0282] 2) Asymmetric selection of geometry and attributes

[0283] The method / apparatus according to an embodiment may send geometry and attributes asymmetrically. Only the upper layer attributes (attribute R2) are removed (19001), and the full geometry (from level 0 (root level) to level 7 (leaf level) in the triangle octree structure) may be selected and sent / decoded (19011).

[0284] Reference Figure 12 , when point cloud data is represented in an octree structure and hierarchically divided into LODs (or layers), scalable encoding / decoding (scalability) can be supported.

[0285] Scalability functionality according to embodiments may include slice level scalability and / or octree level scalability.

[0286] The LoD (Level of Detail) according to the embodiment may be used as a unit representing a set of one or more octree layers. In addition, it may mean a bundle of octree layers to be configured as slices.

[0287] In attribute encoding / decoding, the LOD according to the embodiment may be extended in a broader sense and used as a unit of dividing data in detail.

[0288] That is, the spatial scalability of the actual octree layer (or scalable attribute layer) can be provided for each octree layer. However, when scalability is configured per slice before bitstream parsing, a selection can be made in LoD according to the implementation.

[0289] In the octree structure, LOD0 may correspond to the root level to level 4, LOD1 may correspond to the root level to level 5, and LOD2 may correspond to the root level to level 7, with level 7 being a leaf level.

[0290] That is to say, Figure 12As shown, when scalability is utilized by slices, such as in the case of scalable transmission, the scalable step size provided can correspond to three steps of LoD0, LoD1 and LoD2, and the scalable step size provided by the octree structure in the decoding operation can correspond to eight steps from root to leaf.

[0291] According to an embodiment, for example, Figure 12 In the example, when LoD0 to LoD2 are configured as corresponding slices, the transcoder ( Figure 11 The transcoder 15040) can select 1) only LoD0, 2) LoD0 and LoD1, or 3) LoD0, LoD1 and LoD2 for scalable processing.

[0292] Example 1: When only LoD0 is selected, the maximum octree level may be 4, and a scalable layer may be selected from octree layers 0 to 4 in the decoding process. In this case, the receiver may regard the node size that can be obtained by the maximum octree depth as a leaf node, and may transmit the node size through signaling information.

[0293] Example 2: When LoD0 and LoD1 are selected, layer 5 may be added. Therefore, the maximum octree level may be 5, and a scalable layer may be selected from octree layers 0 to 5 in the decoding process. In this case, the receiver may regard the node size that can be obtained by the maximum octree depth as a leaf node, and may transmit the node size through signaling information.

[0294] According to an embodiment, an octree depth, an octree layer, and an octree level may be units in which data is divided in detail.

[0295] Example 3: When LoD0, LoD1, and LoD2 are selected, layers 6 and 7 may be added. Therefore, the maximum octree level may be 7, and a scalable layer may be selected from octree layers 0 to 7 in the decoding process. In this case, the receiver may regard the node size that can be obtained by the maximum octree depth as a leaf node, and may transmit the node size through signaling information.

[0296] Figure 16 A method of configuring a slice including point cloud data according to an embodiment is illustrated.

[0297] Slice Configuration According to Implementation

[0298] The transmission method / apparatus / encoder according to the embodiment may configure a G-PCC bitstream by segmenting the bitstream in a slice structure. The data unit for detailed data representation may be a slice.

[0299] A slice according to an embodiment may mean a data unit for partitioning point cloud data. That is, a slice represents a portion of point cloud data. A slice may be referred to as a term representing a certain portion or unit.

[0300] For example, one or more octree levels may be matched to one slice.

[0301] According to the transmission method / apparatus of the embodiment, such as an encoder, a bit stream based on slice 2001 may be configured by scanning nodes (points) included in the octree in the direction of the scanning order 2000 .

[0302] exist Figure 16 In (a), some nodes in the octree layer can be included in one slice.

[0303] The octree layers (e.g., level 0 to level 4) may constitute a slice 2002.

[0304] Partial data of the octree layer (e.g., level 5) may constitute each slice 2003, 2004, 2005.

[0305] Partial data of the octree layer (e.g., level 6) can constitute each slice.

[0306] exist Figure 16 In (b) and (c) of 16, when multiple octree layers match one slice, only some nodes of each layer may be included. In this way, when multiple slices constitute one geometry / attribute frame, information necessary for configuring the layer may be delivered to the receiver. The information may include information about the layers included in each slice and information about the nodes included in each layer.

[0307] exist Figure 16 In (b), the octree layers (e.g., level 0 to level 3) and partial data of level 4 can be configured as one slice.

[0308] The octree layers (eg, partial data of level 4 and partial data of level 5) may be configured as one slice.

[0309] The octree layers (eg, partial data of level 5 and partial data of level 6) may be configured as one slice.

[0310] An octree layer (e.g., partial data at level 6) can be configured as a slice.

[0311] exist Figure 16 In (c), the octree layers (e.g., data of level 0 to level 4) can be configured as a slice.

[0312] Partial data from each of octree layers level 5, level 6, and level 7 may be configured as one slice.

[0313] The encoder and the device corresponding to the encoder according to the embodiment may encode point cloud data, and may generate and transmit a bitstream including the encoded data and parameter information related to the point cloud data.

[0314] In addition, when generating a bitstream, the bitstream structure according to the embodiment may be used. Figures 12 to 16 ) to generate a bit stream. Therefore, the receiving device, decoder and corresponding device according to the embodiment can receive and parse the bit stream configured to be suitable for selective partial data decoding, and partially decode and efficiently provide point cloud data (see Figure 11 ).

[0315] Scalable transmission according to embodiments

[0316] The point cloud data transmitting method / apparatus according to the embodiment can scalably transmit a bit stream including point cloud data, and the point cloud data receiving method / apparatus according to the embodiment can scalably receive and decode the bit stream.

[0317] When based on Figures 12 to 16 When the bitstream of the illustrated embodiment is used for scalable transmission, the information required to select the slice required by the receiver can be sent to the receiver. Scalable transmission can mean sending or decoding only a portion of the bitstream, rather than decoding the entire bitstream, and the result can be low-resolution point cloud data.

[0318] When applying scalable transmission to octree-based geometry bitstreams, point cloud data may need to be configured with only one octree layer ( Figure 16 )'s bitstream to a specific octree layer range.

[0319] To this end, the target octree level should not depend on information about lower octree levels. This can be a constraint that applies jointly to geometry encoding and attribute encoding.

[0320] In addition, in scalable transmission, it is necessary to deliver a scalable structure for the transmitter / receiver to select a scalable layer. Considering the octree structure according to the embodiment, all octree layers may support scalable transmission, or scalable transmission may be allowed only for a specific octree layer or a lower layer. When a slice includes some layers in the octree layer, the scalable layer in which the slice is included may be indicated. Therefore, it may be determined whether a slice is necessary / unnecessary in the bitstream stage. Figure 16In the example of (a), the yellow part starting from the root node constitutes a scalable layer, but does not support scalable transmission. Subsequent octree layers can be matched with scalable layers in a one-to-one correspondence. Generally, the part corresponding to the leaf node can support scalability. When multiple octree layers are included in a slice, it can be defined that a scalable layer should be configured for these layers.

[0321] In this case, scalable transmission and scalable decoding can be used separately according to the purpose. Scalable transmission can be used on the sending / receiving side for the purpose of selecting information up to a specific layer without involving a decoder. Scalable decoding is used to select a specific layer during encoding. That is, scalable transmission can support the selection of necessary information in a compressed state (at the bitstream stage) without involving a decoder, so that the information can be sent or determined by the receiver. On the other hand, scalable decoding can support encoding / decoding only up to the required part of the data in the encoding / decoding process, and can therefore be used as a scalable representation in this case.

[0322] In this case, the layer configuration for scalable transmission may be different from the layer configuration for scalable decoding. For example, the three bottom octree layers including leaf nodes may constitute one layer in terms of scalable transmission. However, when all layer information is included in terms of scalable decoding, scalable decoding may be performed for each of leaf node layer n, leaf node layer n-1, and leaf node layer n-2.

[0323] The method / apparatus according to the embodiment may perform fine-grained slicing.

[0324] When fine-grained slicing is enabled, in the encoding operation of the point cloud transmission method according to the embodiment, the G-PCC bitstream (see Figure 24 In order to efficiently use the hierarchical structure of G-PCC, each slice may include coded data for a portion of the coding layer or a portion of the region. Using partitioning of slices paired with the coding layer structure, use cases for scalable transmission or spatial random access can be supported in an efficient manner.

[0325] Figure 17 A geometric coding layer structure according to an embodiment is illustrated.

[0326] According to the point cloud data sending method / device of the embodiment (ie, Figure 1 The sending device 10000, Figure 1 Point cloud video encoder 10002, Figure 1 Transmitter 10003, Figure 2 Acquisition 20000 / Encoding 20001 / Transmission 20002, Figure 3 The encoder, Figure 8 The sending device Figure 10Devices Figure 12 , Figure 21 , Figure 28 and Figure 29 The encoder, Figure 23 and Figure 30 The point cloud can be encoded into a layer structure to generate a layer-based bit stream, such as Figure 17 shown.

[0327] According to the point cloud data receiving method / device of the embodiment (ie, Figure 1 The receiving device 10004, Figure 1 Receiver 10005, Figure 1 Point cloud video decoder 10006, Figure 2 Transmission 20002 / decoding 20003 / rendering 20004, Figure 7 The decoder, Figure 9 The receiving device Figure 10 Devices Figure 11 , Figure 21 , Figure 28 and Figure 29 The decoder, Figure 24 and Figure 31 The receiving method of the present invention can receive layer-based point cloud data and bit streams, and selectively decode the data, such as Figure 17 shown.

[0328] The bitstream and point cloud data according to the embodiment can be generated according to the slice segmentation based on the coding layer. By slicing the bitstream at the end of the coding layer of the coding process, the method / apparatus according to the embodiment can select relevant slices to support scalable transmission or partial decoding.

[0329] Figure 17 (a) illustrates a geometric coding layer structure with 8 layers, where each slice corresponds to a layer group. Layer group 1 3900 includes coding layers 0 to 4. Layer group 2 3901 includes coding layer 5. Layer group 3 3902 is a group for coding layers 6 and 7. When the geometry (or attribute) has a tree structure of eight levels (depths), the bitstream can be hierarchically configured by grouping data corresponding to one or more levels (depths). Each group can be included in one slice.

[0330] Figure 17 (b) illustrates the decoding output when 2 slices are selected from 3 groups. When the decoder selects group 1 and group 2, partial layers of tree levels (depths) 0 to 5 can be selected. That is, partial decoding of the coded layer can be supported by using slices of the layer group structure without accessing the entire bitstream.

[0331] For the partial decoding process according to the embodiment, the encoder may generate three slices generated based on the layer group structure, and the decoder may perform partial decoding by selecting two slices from the three slices.

[0332] A bitstream according to an embodiment ( Figure 24 ) may include layer-based groups / slices 3903. Each slice may include a header including signaling information related to point cloud data (geometric data and / or attribute data) included in the slice. The receiving method / apparatus according to the embodiment may select a portion of the slices and decode the point cloud data included in the payload of the slice based on the header included in the slice.

[0333] In addition to the layer group structure, the method / device according to the embodiment may also consider the use case of spatial random access and divide the layer group into several subgroups. The subgroups are mutually exclusive, and the set of subgroups may be the same as the layer group. Since the points of each subgroup form a boundary in the spatial domain, the subgroup can be represented by the subgroup bounding box information. Based on the spatial information, the layer group and subgroup structure can support spatial access. By efficiently comparing the region of interest (ROI) with the bounding box information about each slice, spatial random access within a frame or tile can be supported.

[0334] The method / apparatus according to an embodiment may divide layer group 1 3900 , layer group 2 3901 , and layer group 3 3902 into one or more sub-groups.

[0335] Although Figure 17 A geometry coding layer structure is shown as an example, but an attribute coding layer structure can be generated in a similar manner.

[0336] The method / apparatus according to an embodiment may perform slice segmentation based on layer groups. In fine-grained slicing, each slice may contain coded data from layer groups defined as follows.

[0337] A layer group is defined as a set of contiguous tree layers whose start and end depths can be any number in the tree depth and whose start is less than its end. The order of the coded data in a slice segment can be the same as the order of the coded data in a single slice.

[0338] For example, suppose a geometric coding layer structure with eight layers is configured, such as Figure 17 In this example, there are three layer groups, each of which matches a different slice: layer group 1 for coding layers 0 to 4, layer group 2 for coding layer 5, and layer group 3 for coding layer. When the first two slices are sent or selected, the decoded output will be partial layers 0 to 5, as shown in Figure 17By using slices in the layer group structure, partial decoding of the coded layer can be supported even without access to the entire bitstream.

[0339] A subgroup is a subset of a layer group, and the points in its subgroup are adjacent to each other. Subgroups of a layer group are mutually exclusive, and the set of points in a subgroup of a layer group can be the same as the set of points in the layer group. The points in each subgroup are bounded in a spatial region, and thus the subgroup bounding box information can be used to describe the boundaries of the subgroup. Using the spatial information, the layer group and subgroup structure can support efficient access to the region of interest (ROI) by selecting slices that cover the ROI.

[0340] Figure 18 A layer group structure and a sub-group structure according to an embodiment are illustrated.

[0341] Figure 17 The layer-based point cloud data and bit stream shown can be represented as Figure 18 The bounding box shown.

[0342] The subgroup structure and the bounding boxes corresponding to the subgroups are shown. Layer group 2 is divided into two subgroups (group 2-1 and group 2-2), and layer group 3 is divided into four subgroups (group 3-1, group 3-2, group 3-3, and group 3-4). The subgroups of layer group 2 and the subgroups of layer group 3 are included in different slices. Given the slices of a layer group and the subgroups with bounding box information, 1) the bounding box of each slice can be compared with the ROI, and 2) the slice whose subgroup bounding box is related to the ROI can be selected and spatial access can be performed. Then, 3) the selected slice is selected. When considering the ROI in area 3-3, slice 1, slice 3, and slice 6 are selected as the subgroup bounding boxes of layer group 1 and subgroup 2-2 and subgroup 3-3 to cover the ROI area. For efficient spatial access, it is assumed that there is no dependency between subgroups in the same layer group. In real-time streaming or low-latency use cases, when each slice segment is received, selection and decoding operations can be performed to improve time efficiency.

[0343] The method / apparatus according to the embodiment may represent data as a plurality of layers (which may be referred to as depth or level) as a layer tree 4000 during geometry and / or attribute encoding. Point cloud data corresponding to the layer (depth / level) may be grouped into layer groups (or groups) 4001, as shown in FIG. 39. Each layer group may be further divided (segmented) into subgroups 4002. A bitstream may be generated by configuring each subgroup as a slice. A receiving device according to the embodiment may receive a bitstream, select a specific slice, decode the subgroups included in the slice, and decode the bounding box corresponding to the subgroup. For example, when slice 1 is selected, a bounding box 4003 corresponding to group 1 may be decoded. Group 1 may be data corresponding to the largest area. When a user wishes to additionally view a refined area of ​​group 1, a method / apparatus according to the embodiment may select slice 3 and / or slice 6, and may partially and hierarchically access the bounding boxes (point cloud data) of group 2-2 and / or group 3-3 for the refined area included in the area of ​​group 1.

[0344] Figure 19 A multi-resolution, multi-size ROI is illustrated according to an embodiment.

[0345] Layer group slicing can provide efficient access to large-scale point cloud data or dense point cloud data based on scalability and spatial access capabilities. Due to the large number of points and large size of data, rendering or displaying content may take a lot of time. As an alternative, the level of detail can be adjusted based on the viewer's interest. For example, when the viewer is far away from the scene or object, structural or global area information is more important than local details. On the other hand, when the viewer moves closer to a specific area or object, detailed information about the ROI is required. Using an adaptive approach, the renderer can efficiently provide data of sufficient quality to the viewer. Figure 19 An example incremental detail change of three viewing distance levels is illustrated, where the viewing distance changes based on the ROI. For example, the viewing distance can be 1) high-level view (coarse detail), 2) medium-level view (medium-level detail), and 3) low-level view (fine-grained detail).

[0346] Figure 20 Layer group slicing according to an embodiment is illustrated.

[0347] According to the point cloud data sending method / device of the embodiment (ie, Figure 1 The sending device 10000, Figure 1 Point cloud video encoder 10002, Figure 1 Transmitter 10003, Figure 2 Acquisition 20000 / Encoding 20001 / Transmission 20002, Figure 3 The encoder, Figure 8 The sending device Figure 10 DevicesFigure 12 , Figure 21 , Figure 28 and Figure 29 The encoder, Figure 23 and Figure 30 The point cloud data can be sliced ​​based on the layer group, such as Figure 20 shown.

[0348] According to the point cloud data receiving method / device of the embodiment (ie, Figure 1 The receiving device 10004, Figure 1 Receiver 10005, Figure 1 Point cloud video decoder 10006, Figure 2 Transmission 20002 / decoding 20003 / rendering 20004, Figure 7 The decoder, Figure 9 The receiving device Figure 10 Devices Figure 11 , Figure 21 , Figure 28 and Figure 29 The decoder, Figure 24 and Figure 31 The receiving method of the layer group slice can receive and decode the data based on the layer group slice, such as Figure 20 shown.

[0349] The method / apparatus according to an embodiment may support high-resolution ROI based on scalability of hierarchical slicing and spatial accessibility.

[0350] Reference Figure 20 , the encoder can generate bitstream slices of octree layer groups or spatial subgroups of each layer group. Upon request, slices matching the ROI of each resolution can be selected and sent. Since detailed information other than the requested ROI is not included in the bitstream, the entire bitstream size can be smaller than that in the tile-based approach. The receiver's decoder can combine slices to generate three outputs. For example, 1) a high-level view output can be generated based on layer group slice 1; 2) a mid-level view output can be generated in layer group slice 1 and selected subgroups of layer group 2; 3) a low-level view of fine detailed output can be generated in layer group 1 and selected subgroups of layer groups 2 and 3. Since the output can be generated progressively, the receiver can provide a viewing experience such as zooming in / out. The resolution can be gradually increased from a high-level view to a low-level view.

[0351] According to an embodiment, encoder 60000 may correspond to a geometry encoder and an attribute encoder as a point cloud encoder. The encoder may slice the point cloud data based on layer groups (or groups). The layer may be referred to as the depth of the tree, the level of the layer, etc. As shown in part 60000-1, the depth of the octree of the geometry and / or the level of the attribute layer may be divided into layer groups (or subgroups).

[0352] The slice selector 60001 in conjunction with the encoder 60000 can select partitioned slices (or sub-slices) to selectively and partially send data, such as layer group 1 to layer group 3.

[0353] The decoder 6002 can decode the selectively and partially transmitted point cloud data. For example, it can decode the high-level view of layer group 1 (which has a high depth / layer / level or index 0, or close to the root). Thereafter, the mid-level view can be decoded by increasing the index of the depth / level above layer group 1 based on layer group 1 and layer group 2 alone. The low-level view can be decoded based on layer groups 1 to 3.

[0354] Figure 21 A context referencing method for fine grained slicing (FGS) according to an embodiment is illustrated.

[0355] According to the point cloud data sending method / device of the embodiment (ie, Figure 1 The sending device 10000, Figure 1 Point cloud video encoder 10002, Figure 1 Transmitter 10003, Figure 2 Acquisition 20000 / Encoding 20001 / Transmission 20002, Figure 3 The encoder, Figure 8 The sending device Figure 10 Devices Figure 12 , Figure 21 , Figure 28 and Figure 29 The encoder, Figure 23 and Figure 30 The sending method of point cloud data can be efficiently encoded by referring to the context when encoding the geometry and / or attributes, such as Figure 21 shown.

[0356] According to the point cloud data receiving method / device of the embodiment (ie, Figure 1 The receiving device 10004, Figure 1 Receiver 10005, Figure 1 Point cloud video decoder 10006, Figure 2 Transmission 20002 / decoding 20003 / rendering 20004, Figure 7 The decoder, Figure 9The receiving device Figure 10 Devices Figure 11 , Figure 21 , Figure 28 and Figure 29 The decoder, Figure 24 and Figure 31 The receiving method of the point cloud data can be decoded efficiently by referring to the context when decoding the geometry and / or attributes, such as ​ shown.

[0357] The gray part is the header, subgroup 0 of layer group 0. The arrows indicate the signaling related to context reference, and the reason for context reference: context improves the bitstream compression performance by utilizing the characteristics of the current compression target in a slice (e.g., probability distribution of 0, probability distribution of 1, etc.) to obtain better compression performance, but as the slices are divided, each slice becomes less efficient. A reference technique is needed.

[0358] Context reference signaling for geometry FGS may provide an alternative method for context reference for the following FGS. It is contemplated that context reference signaling in attribute layer group slices may be aligned with context reference in geometry.

[0359] However, in the absence of FGS, an independent attribute context reference should be used for geometry. One reason for missing FGS is due to the subsampling process in LoD generation. When all points in the finest LoD of a subgroup are subsampled to a higher subgroup, there are no remaining points in the subgroup and no FGS is encoded.

[0360] Reference ​ , you can see an example of a parent content reference with a missing FGS. ​ As shown in (b), the FGS in layer group 1 and sub-group 1 is missing. Since no coded data is generated from the FGS, the context reference is not stored in the attribute context buffer. This causes an error when encoding the sub-FGS in layer group 2 and sub-group 1 because there is no context to reference.

[0361] Reference ​ , 2200 represents data in an FGS unit. FGS may be a data unit obtained by further dividing a slice. Although all point cloud data may be included in one slice for encoding and decoding, one slice may be partitioned into smaller units, and point cloud data may be encoded and decoded based on segmented slices, which may allow random access, partial decoding, and the like. In addition, the method according to the embodiments described herein is needed to address technical issues such as missing points and increased complexity that may occur due to the need to process finely divided slices. According to an embodiment, segmented slices may be generated based on layer groups or subgroups, etc., as described above.

[0362] The first FGS 2200 (FGS0) exemplarily represents subgroup 0 of layer group 0 in one or more layer groups including one or more subgroups. The first FGS 2200 may be referred to as a segmented slice, or various other terms implying a unit smaller than a slice. The first FGS 2200 may include a header at a leading position. FGS1 may be subgroup 0 of layer group 1, and FGS2 may be subgroup 1 of layer group 1. The arrows indicate the relationship of context references. For example, FGS N+1 corresponding to subgroup 0 of layer group 2 may be similar to the parent FSG1 to which FSG N+1 belongs, so the context of FSGN+1 can be efficiently encoded and decoded by referring to the context included in the header of FSG1.

[0363] Context can have a significant impact on the performance of bitstream compression. Context indicates the characteristics of the current compression target (e.g., the probability distribution of 0, 1, etc.). Including all point cloud data in one slice for encoding and decoding can improve compression performance because repeated probability rules can be used within one slice. When the slice is further segmented into layer groups and / or subgroups to encode and decode each segmentation unit (FGS), recompressing and reconstructing the context for each segmentation unit (FGS) may reduce efficiency. Therefore, a method of referring to context as described in the embodiment is needed.

[0364] like ​ As shown in (a), by defining a signaling method, the context can be referenced according to the parent / child FGS relationship, and the context contained in the header of the parent FSG, the parent's parent FSG, etc. can be referenced. Specifically, the context can be stored in a geometry and / or attribute context buffer, and the context to be referenced can be loaded. By following the reference relationship upward through the parent's parent, all child FSGs can reference the context of (0,0) that is ultimately associated with the root node.

[0365] However, if ​ As shown in (b), a certain FSG in the middle, such as (1,1), may be missing. This is because (1,1) may not be processed during random access or partial encoding / decoding. In this case, the buffer does not store the context to be used for encoding / decoding in (1,1), and therefore the subsequent encoding / decoding of the lower-level FSG may not utilize the information with the most similar context from (1,1).

[0366] In addition, if ​ As shown in (c), if (1,1) does not exist or is missing as described above, the parent of (1,1) can be searched, and the context information used in the parent can be loaded.

[0367] In other words, to solve the above problem, the grandparent context can be used in the absence of a parent FGS. The grandparent of the FGS in layer group 2 and subgroup 1 is the FGS of the root layer group. In this case, the context reference can be indicated in both explicit and implicit ways. The method / apparatus according to the embodiment may include the following two signaling methods to provide a general solution for future applications: 1) explicit signaling of attribute context reference, and 2) implicit signaling of attribute context reference.

[0368] In the explicit signaling method, the reference layer group ID (ref_layer_group_id) and the reference subgroup ID (ref_subgroup_id) can be signaled from the dependency attribute data unit header included in the bitstream to provide an explicit indication of the context reference for a more general use case. The use of the explicit signaling method can be indicated by the indication flag aps_attr_ref_id_present_flag. For example, aps_attr_ref_id_present_flag equal to 1 can indicate the use of the explicit signaling method, while aps_attr_ref_id_present_flag equal to 0 can indicate the use of the implicit signaling method described later. In addition, ref_layer_group_id and ref_subgroup_id can be explicitly indicated as signaling information, so that the attribute context can be referenced from the dependency header (such as the parent's header, the parent's parent, etc.). In addition, considering both the dependency and neighbor relationships, the ID to be referenced can be indicated in the header, such as the parent or the neighbor's parent. A device that is to encode or decode the current FSG may load attribute context information related to a parent or neighbor from a buffer by referring to the identification information in the header, thereby encoding and decoding the current context efficiently and quickly.

[0369] In the implicit signaling method, for implicit signaling, the context reference of the current attribute FGS can inherit the context reference of the corresponding geometry FGS, that is, the geometry FGS whose layer_group_id and subgroup_id are the same as the ID of the dependency attribute data unit header contained in the bitstream. Geometry and attributes have similar structures. Therefore, the similarity of geometry may lead to the similarity of attributes. In other words, the reference target can be implicitly set on the sending side and the receiving side. It can be set to the parent, or the parent of the parent. Explicit signaling fields may not be used to indicate the target. Instead, the attribute context can be referenced in the same way as the geometry context reference signaling.

[0370] When a parent attribute FGS is missing, the context reference of the current attribute FGS inherits from the context reference of the context reference of the geometry FGS corresponding to the missing parent attribute FGS.

[0371] The method / apparatus according to the embodiment may make an implicit indication for a reference.

[0372] When FGS is missing, the encoder can send an empty FGS. For example, it can send only the dependency attribute data unit header. The decoder can usually use the context reference indicator of the geometry FGS corresponding to the attribute FGS. When delivering the sub-attribute FGS of the empty attribute FGS, the context reference is found in the following process.

[0373] 1) Get ref_layer_group_id A and ref_subgroup_id A` delivered in the data unit header of the geometry FGS corresponding to the attribute FGS.

[0374] 2) Get ref_layer_group_id B and ref_ssubgroup_id B` delivered in the geometry FGS indicated by ref_layer_group_id A and ref_subgroup_id A`.

[0375] 3) The context state of the attribute FGS indicated by ref_layer_group_id B and ref_subgroup_id B` can be used.

[0376] The process is represented as the following program instructions:

[0377] void

[0378] PCCTMC3Decoder3::decodeDependentAttributeBrick(const PayloadBuffer&buf)

[0379] {

[0380] assert(buf.type==PayloadType::kDependentAttributeDataUnit);

[0381] assert(_sps->layer_group_enabled_flag);

[0382] const auto it_attr_aps=_apss.find(_abh.attr_attr_parameter_set_id);

[0383] assert(it_attr_aps!=_apss.cend());

[0384] const auto&attr_aps=it_attr_aps->second;

[0385] assert(_abh.attr_sps_attr_idx<_sps->attributeSets.size());

[0386] const auto&attr_sps=_sps->attributeSets[_abh.attr_sps_attr_idx];

[0387] const auto&label=attr_sps.attributeLabel;

[0388] int abhSize;

[0389] DependentAttributeDataUnitHeader dep_abh;

[0390] dep_abh=parseDepAbh(*_sps,buf,&abhSize);

[0391] int curLayerGroup=dep_abh.layer_group_id;

[0392] int curSubgroup=_subgroupIdToArrayIdx[dep_abh.layer_group_id][dep_abh.subgroup_id];

[0393] if(!_layerGroupParams.sliceSelectionIndicationFlag[curLayerGroup].size()

[0394] ||!_layerGroupParams.sliceSelectionIndicationFlag[curLayerGroup][curSubgroup]){

[0395] return;

[0396] }

[0397] int refLayerGroup=_layerGroupParams.refLayerGroupId[curLayerGroup][curSubgroup];

[0398] int refSubgroup=_layerGroupParams.refSubgroupId[curLayerGroup][curSubgroup];

[0399] int refArrayIdx=_refIdxToSavedArrayIdxAttr[refLayerGroup][refSubgroup];

[0400] int parentLayerGroup=curLayerGroup-1;

[0401] int parentSubgroup=0;

[0402] while(refArrayIdx==-1){

[0403] bool detectedFlag=false;

[0404] auto bbox_min=_layerGroupParams.subgrpBboxOrigin[curLayerGroup][curSubgroup];

[0405] auto bbox_max=bbox_min+

[0406] _layerGroupParams.subgrpBboxSize[curLayerGroup][curSubgroup];

[0407] for(int i=0;i<=_layerGroupParams.numSubgroupsMinus1[parentLayerGroup];i++){

[0408] auto parent_bbox_min=_layerGroupParams.subgrpBboxOrigin[parentLayerGroup][i];

[0409] auto parent_bbox_max=parent_bbox_min+

[0410] _layerGroupParams.subgrpBboxSize[parentLayerGroup][i];

[0411] if(bbox_min.x()>=parent_bbox_min.x()&&bbox_max.x()<=parent_bbox_max.x()

[0412] &&bbox_min.y()>=parent_bbox_min.y()&&bbox_max.y()<=parent_bbox_max.y()

[0413] &&bbox_min.z()>=parent_bbox_min.z()&&bbox_max.z()<=parent_bbox_max.z()){

[0414] parentSubgroup=i;

[0415] detectedFlag=true;

[0416] break;

[0417] }

[0418] }

[0419] if(detectedFlag){

[0420] refLayerGroup=

[0421] _layerGroupParams.refLayerGroupId[parentLayerGroup][parentSubgroup];

[0422] refSubgroup=_layerGroupParams.refSubgroupId[parentLayerGroup][parentSubgroup];

[0423] refArrayIdx=_refIdxToSavedArrayIdxAttr[refLayerGroup][refSubgroup];

[0424] }

[0425] else if(parentSubgroup==0)

[0426] refArrayIdx=0;

[0427] else

[0428] parentSubgroup--;

[0429] }

[0430] / / Specifically, the above process represents an operation of encoding and decoding the current attribute FSG by referring to the context from the parent attribute FSG based on the identification information related to the parent attribute FSG of each child group of the layer group to refer to the attribute FSG.

[0431] auto ctxtMemAttr=_ctxtMemSavedAttr[_abh.attr_sps_attr_idx][refArrayIdx];

[0432] _attrDecoder->decode(

[0433] *_sps,attr_sps,attr_aps,_abh,_gbh.footer.geom_num_points_minus1,

[0434] _params.minGeomNodeSizeLog2,buf.data()+abhSize,buf.size()-abhSize,

[0435] ctxtMemAttr,_currentPointCloud,

[0436] _layerGroupParams,curLayerGroup,curSubgroup);

[0437] if(_layerGroupParams.ctxtRefIdcFlag[curLayerGroup][curSubgroup]){

[0438] if(_accNumSubgroupsPerLayerGroupAttr[curLayerGroup]==0){

[0439] _refIdxToSavedArrayIdxAttr[curLayerGroup].resize(_refIdxToSavedArrayIdx[curLayerGroup].siz e(),-1);

[0440] _accNumSubgroupsPerLayerGroupAttr[curLayerGroup]=1;

[0441] }

[0442] else

[0443] _accNumSubgroupsPerLayerGroupAttr[curLayerGroup]++;

[0444] _refIdxToSavedArrayIdxAttr[curLayerGroup][dep_abh.subgroup_id]=_ctxtMemSavedAttr[_abh.attr_sps_attr_idx].size();

[0445] _ctxtMemSavedAttr[_abh.attr_sps_attr_idx].push_back(ctxtMemAttr);

[0446] }

[0447] }

[0448] ​ An attribute encoding method according to an embodiment is illustrated.

[0449] according to ​ The point cloud data sending method / device of the embodiment corresponds to ​ The sending device 10000, ​ Point cloud video encoder 10002, ​ Transmitter 10003, ​ Acquisition 20000 / Encoding 20001 / Transmission 20002, ​ The encoder, ​ The sending device ​ Devices ​ , ​ , ​ and ​ The encoder, ​ and ​ The sending method, etc.

[0450] ​is an exemplary flow chart of a layer group slice encoder for LoD-based attribute coding. First, parameters such as SPS, GPS, APS, LGSI, etc. are generated and the layer group structure is configured. The layer group structure can be configured when or before generating the layer group slice list (LGSI). After performing geometric coding, LoD generation is performed based on the geometric nodes. The attribute coding layer (e.g., LoD, RAHT coding layer) is matched with the layer group, and then the quantization weight is obtained based on the LoD. Then, for each attribute coding layer, it is checked whether the layer group has changed, and for the nodes in the attribute coding layer, it is checked whether the subgroup has changed. When a subgroup or layer group changes, the attribute encoder for the previous subgroup can be stored, and the attribute encoder for the current subgroup can be used to ensure the continuity of context, zero stroke, etc. within the subgroup and to ensure independence from neighboring subgroups. After compressing the nodes in all attribute coding layers, the coded bitstream of each subgroup is packaged into each slice.

[0451] Although the examples described above are based on LoD-based attribute coding, they may also be applied to other attribute coding included in the embodiments, such as RAHT.

[0452] Reference ​ , the encoder can be based on ​The flow chart of is used to encode the point cloud data. Depending on the encoding configuration of the encoder, the value of the parameter can be generated. Here, the parameters of SPS, GPS, APS, LGSI, etc. can be generated together with the encoded point cloud data and sent in the bitstream. The encoder can first encode the geometric data of the point cloud data through a geometric encoder. After the geometric encoding, the encoder can generate LOD through an attribute encoder. During LOD generation, the attribute can be represented by a layer group (which can be called a layer group including subgroups, an FGS including subgroups, etc.) structure according to the above-mentioned embodiment. LOD can be mapped to a layer group. One or more indexes of LOD (which can be called layers, levels, depths, etc.) can be grouped (or mapped) into layer groups. Weights can be assigned to each layer group. For example, when the highest level layer group is important, it can be assigned a larger weight. On the other hand, when the lowest level layer group is important, it can be assigned a larger weight. Depending on the desired configuration, the weight is variable, such as random access to data about detail levels and partial encoding. Layer groups can be determined. Here, the determined layer group may be a layer group to be partially encoded or partially sent. A subgroup may be determined in a subgroup belonging to the layer group. The determined subgroup may be a subgroup to be partially encoded or partially sent. Check whether the subgroup has changed. In the case where the subgroup has changed, context information may be saved when encoding the subgroup, and necessary context information may be loaded. The attributes of the node are encoded. In the case where the subgroup has not changed, the attributes of the node may be encoded immediately. When the end of the node and the end of the LOD are reached, an attribute data unit header may be generated. The attribute bitstream may be generated and sent together with the geometry bitstream.

[0453] ​ An attribute decoding method according to an embodiment is illustrated.

[0454] according to ​ The point cloud data receiving method / device of the embodiment corresponds to ​ The receiving device 10004, ​ Receiver 10005, ​ Point cloud video decoder 10006, ​ Transmission 20002 / decoding 20003 / rendering 20004, ​ The decoder, ​ The receiving device ​ Devices ​ , ​ , ​ and ​ The decoder, ​ and ​ receiving method, etc.

[0455] The input of the attribute decoder is the attribute fine-grained slice bitstream, the decoded geometry data and the layer group structure. After geometry decoding, LoD generation is performed based on the decoded geometry data provided, and weight derivation is performed based on the relationship of the nodes according to the LoD. The information generated by the operation is used for attribute decoding in subsequent operations. The first attribute slice can be operated in the same way as the previous attribute slice. Then, when the layer group slice is enabled, the dependency attribute data unit header can be parsed, and the information in layer_group_id and subgroup_id can be obtained. When necessary, information about the LoD can be notified by signal. Based on this information, a layer group that matches the current LoD can be searched, and a parent-child group can be selected. When the layer group structure is applied to geometry and attributes in the same way, the reference and parent information used in geometry decoding can be used. Based on this, the information in the dependency attribute data unit can be decoded and the point cloud data can be reconstructed.

[0456] When performing decoding, the LoD generation and weight derivation methods used by the encoder can be used. Thus, the LoD can be mapped to layer groups.

[0457] Reference ​ , the decoder can be based on ​ The flowchart shown is used to decode the point cloud data. Parameters contained in the bitstream, such as SPS, GPS, APS and LGSI, can be parsed. The geometry decoder of the decoder can decode the geometry data. The attribute decoder of the decoder can generate LOD. When generating LOD, the layer group structure (FGS) according to the embodiment can be used. LOD specific weights can be derived. Weights can be applied to attributes in the reverse process of the operation of the encoder. The attribute data unit header in the bitstream can be parsed. The attribute data unit in the bitstream can be decoded. Check whether layer group slicing is enabled. When layer group slicing is enabled, parse the dependency attribute data unit header. LOD can be mapped to layer groups. A higher subgroup of the current subgroup of the current layer group can be selected. The higher subgroup can refer to the parent subgroup of the current subgroup, the parent of the parent subgroup, etc. The dependency attribute data unit can be decoded. Check whether the end of the geometry bitstream is reached. The attribute bitstream corresponding to the geometry bitstream can be decoded. Both geometry and attributes are decoded, and a point cloud output is generated. Then, the decoding process is terminated. When layer group slicing is not enabled, decoding can be terminated without any layer group-based operation.

[0458] ​ A bitstream including parameters and encoded point cloud data according to an embodiment is illustrated.

[0459] Point cloud data transmission method / device according to an embodiment (corresponding to ​ The sending device 10000, ​Point cloud video encoder 10002, ​ Transmitter 10003, ​ Acquisition 20000 / Encoding 20001 / Transmission 20002, ​ The encoder, ​ The sending device ​ Devices ​ , ​ , ​ , ​ and ​ The encoder, ​ and ​ The sending method, etc.) can generate ​ The bit stream is shown.

[0460] according to ​ Point cloud data receiving method / device of an embodiment of the present invention (corresponding to ​ The receiving device 10004, ​ Receiver 10005, ​ Point cloud video decoder 10006, ​ Transmission 20002 / decoding 20003 / rendering 20004, ​ The decoder, ​ The receiving device ​ Devices ​ , ​ , ​ and ​ The decoder, ​ and ​ The receiving method, etc.) can be parsed as ​ The bit stream is shown.

[0461] Information about separated slices (which refer to layer groups, subgroups, FSGs, etc.) according to an embodiment may be defined in parameter sets and SEI messages as follows. It may be defined in a sequence parameter set, a geometry parameter set, an attribute parameter set, a geometry slice header, and an attribute slice header, and may be defined at a corresponding position or a separate position according to an application or system definition to use different application scopes, different application methods, etc. In other words, it may have different meanings depending on the location of the signal carrier. When information is defined in SPS, the information may be applied equally throughout the sequence. When it is defined in GPS, this may indicate that the information is used for position reconstruction. When it is defined in APS, it may indicate that the information is applied to attribute reconstruction. When it is defined in TPS, it may indicate that signaling is applied only to points within a tile. When delivered on a slice basis, it may indicate that signaling is applied only to corresponding slices. In addition, it may be defined at a corresponding position or a separate position according to the application and system definition to use different application scopes, different application methods, etc. In addition, when the syntax elements defined below are applicable to multiple point cloud data streams as well as the current point cloud data stream, the information may be carried in an upper-level parameter set or the like.

[0462] Each abbreviation has the following meaning. Each abbreviation may be referred to by another term within the equivalent meaning: SPS: Sequence Parameter Set; GPS: Geometry Parameter Set; APS: Attribute Parameter Set; TPS: Tile Parameter Set; Geom: Geometry Bitstream = Geometry Slice Header + Geometry Slice Data; Attr: Attribute Bitstream = Attribute Brick Header + Attribute Brick Data.

[0463] Although it has been described that the information is defined independently of the coding technique, the information can be defined in conjunction with the coding method. It can be defined in a tile parameter set to support different scalability across regions. In addition, when the syntax elements defined below apply to multiple point cloud data streams as well as the current point cloud data stream, the information can be carried in a parent parameter set, etc.

[0464] Alternatively, a network abstraction layer (NAL) unit may be defined, and relevant information (eg, layer_id) for selecting a layer may be delivered so that a bitstream may be selected at a system level.

[0465] Parameters according to an embodiment (which may be referred to as metadata, signaling information, etc.) may be generated in a process of a transmitter according to each embodiment described below, and delivered to a receiver according to an embodiment for use in a reconstruction process.

[0466] For example, the parameters according to the embodiment may be generated in a metadata processor (or metadata generator) of a transmitting device according to the embodiment described below, and acquired by a metadata parser of a receiving device according to the embodiment.

[0467] Fig.25 A set of attribute parameters according to an embodiment is illustrated.

[0468] Fig.25 Examples include Fig.24 A set of attribute parameters in the bitstream.

[0469] When layer_group_enabled_flag indicates enable (Enable), the attribute parameter set includes aps_attr_ref_id_present_flag.

[0470] aps_attr_ref_id_present_flag: aps_attr_ref_id_present_flag equal to 1 indicates that the context reference of the attribute slice is indicated (and signaled) by attr_ref_layer_group_id and attr_ref_subgroup_id, and the use of the context state of the subsequent attribute slice is indicated (and signaled) by attr_context_reference_indication_flag in the dependency attribute data unit header. When aps_attr_ref_id_present_flag is equal to 0, the use of the context reference and context state is inherited from the geometry slice corresponding to the layer_group_id and subgroup_id of the current dependency attribute slice (see Fig. 22 ).

[0471] Fig.26 A dependency attribute data unit header according to an embodiment is illustrated.

[0472] Fig.26 Examples are included in Fig.24 The dependency attribute data unit header in the bitstream.

[0473] dadu_attribute_parameter_set_id: Indicates the identifier of the attribute parameter set referenced by the dependency attribute data unit header.

[0474] dadu_sps_attr_idx: indicates the SPS attribute index. It indicates the index in the attribute list of the sequence parameter set.

[0475] dadu_slice_id: Indicates the ID of the slice associated with the dependency attribute data unit header.

[0476] layer_group_id: Indicates the ID of the layer group to which the dependency attribute data unit header is related.

[0477] When aps_attr_ref_id_present_flag indicates that a reference ID is present, the attribute reference layer group ID may be included in the dependency attribute data unit header.

[0478] attr_ref_layer_group_id: an indicator indicating a layer group identifier for the context reference of the current dependency attribute data unit. attr_ref_layer_group_id may be in the range of 0 to attr_layer_group_id of the current dependency attribute data unit.

[0479] When the subgroup enable flag of the layer group corresponding to layer_group_id indicates enable (Enabled), subgroup_id may be provided.

[0480] subgroup_id indicates a subgroup ID associated with the dependency attribute data unit header.

[0481] When aps_attr_ref_id_present_flag indicates presence, the following elements are provided.

[0482] attr_ref_subgroup_id: attr_ref_subgroup_id indicates an indicator of a reference subgroup of the layer group indicated by attr_ref_layer_group_id. attr_ref_subgroup_id may be in the range of 0 to num_subgroup_id_minus1 of the layer group indicated by attr_ref_layer_group_id.

[0483] attr_context_reference_indication_flag: attr_context_reference_indication_flag equal to 1 indicates that the context state of the current dependency slice is inherited by one or more subsequent dependency slices. attr_context_reference_indication_flag equal to 0 indicates that the context state of the current dependent slice is not inherited by subsequent dependent slices.

[0484] The decoder can use attr_context_reference_indication_flag to manage the context buffer. When attr_context_reference_indication_flag is equal to 1, the context state of the current dependent slice is stored in the context buffer at the end of decoding. When attr_context_reference_indication_flag is equal to 0, the context state of the current dependent slice is not stored in the context buffer.

[0485] Fig. 27 A process of transmitting partial point cloud data according to an embodiment is illustrated.

[0486] Point cloud data transmission method / device according to an embodiment (corresponding to Figure 1 The sending device 10000, Figure 1 Point cloud video encoder 10002, Figure 1 Transmitter 10003, Figure 2 Acquisition 20000 / Encoding 20001 / Transmission 20002, Figure 3 The encoder, Figure 8 The sending device Fig.10 Devices Fig.12 , Fig.21 , Fig. 22 , Fig.28 and Fig.29 The encoder, Fig.23 and Fig.30 The transmission method of the present invention can scalably encode the entire point cloud data, store it in a storage device, perform transcoding for partial encoding, and then transmit the partial bit stream, such as Fig. 27 shown.

[0487] According to the point cloud data receiving method / device of the embodiment (corresponding to Figure 1 The receiving device 10004, Figure 1 Receiver 10005, Figure 1 Point cloud video decoder 10006, Figure 2 Transmission 20002 / decoding 20003 / rendering 20004, Figure 7 The decoder, Fig. 9 The receiving device Fig.10 Devices Fig.11 , Figure 21 to Figure 23 , Fig.28 and Fig.29 The decoder, Fig.24 and Fig.31 The receiving method, etc.) can receive and decode the bit stream of part of the point cloud data to reconstruct part of the geometry and / or part of the attributes.

[0488] Embodiments include methods for dividing and transmitting compressed data according to specific criteria for point cloud data. When layered encoding is used, compressed data may be divided and transmitted according to layers, which may improve efficiency on the receiving side.

[0489] The geometry and attributes of point cloud data can be compressed to provide services. In PCC-based services, the compression rate or amount of data for transmission can be adjusted according to the receiver performance or the transmission environment.

[0490] When configuring point cloud data on a slice-by-slice basis, receiver performance or a transmission environment may change. In this case, it is necessary to 1) pre-transcode the bitstream into a form suitable for each environment, store the bitstream separately, and select a portion when transmitting, or 2) perform transcoding before transmission. In this case, if the number of receiver environments to be supported increases or the transmission environment changes frequently, problems related to storage space or delays caused by transcoding may arise.

[0491] Fig.28 A method of transmitting and receiving point cloud data according to an embodiment is illustrated.

[0492] Point cloud data transmission method / device according to an embodiment (corresponding to Figure 1 The sending device 10000, Figure 1 Point cloud video encoder 10002, Figure 1 Transmitter 10003, Figure 2 Acquisition 20000 / Encoding 20001 / Transmission 20002, Figure 3 The encoder, Figure 8 The sending device Fig.10 Devices Fig.12 , Fig.21 , Fig. 22 , Fig.28 and Fig.29 The encoder, Fig.23 and Fig.30 The sending method, etc.) can be as follows Fig.28 The entire point cloud data can be scalably encoded and stored in a storage device (storage can be skipped), and the bit stream can be selected to send part of the point cloud immediately, which is similar to Fig. 27 Different.

[0493] According to the point cloud data receiving method / device of the embodiment (corresponding to Figure 1 The receiving device 10004, Figure 1 Receiver 10005, Figure 1 Point cloud video decoder 10006, Figure 2 Transmission 20002 / decoding 20003 / rendering 20004, Figure 7 The decoder, Fig. 9 The receiving device Fig.10 Devices Fig.11 , Figure 21 to Figure 23 , Fig.28 and Fig.29 The decoder, Fig.24 and Fig.31 The receiving method, etc.) can immediately receive and decode the bit stream of partial point cloud data to reconstruct partial geometry and / or partial attributes.

[0494] According to an embodiment, when compressed data is divided and transmitted according to layers, only necessary parts of pre-compressed data can be selectively transmitted in the bitstream step without a separate transcoding operation. This is efficient even in terms of storage space because only one storage space is required for each stream. In addition, since only necessary layers are selectively transmitted, efficient transmission can be performed even in terms of (bitstream selector) bandwidth.

[0495] The point cloud data receiving method / device according to the embodiment of the present invention can provide the following effects:

[0496] Fig.29 A method of transmitting and receiving point cloud data according to an embodiment is illustrated.

[0497] Point cloud data transmission method / device according to an embodiment (corresponding to Figure 1 The sending device 10000, Figure 1 Point cloud video encoder 10002, Figure 1 Transmitter 10003, Figure 2 Acquisition 20000 / Encoding 20001 / Transmission 20002, Figure 3 The encoder, Figure 8 The sending device Fig.10 Devices Fig.12 , Fig.21 , Fig. 22 , Fig.28 and Fig.29 The encoder, Fig.23 and Fig.30 The entire point cloud data may be scalably encoded and stored in a storage device (storage may be skipped), transcoded, and then the entire point cloud may be sent, such as Fig.28 shown.

[0498] According to the point cloud data receiving method / device of the embodiment (corresponding to Figure 1 The receiving device 10004, Figure 1 Receiver 10005, Figure 1 Point cloud video decoder 10006, Figure 2Transmission 20002 / decoding 20003 / rendering 20004, Figure 7 The decoder, Fig. 9 The receiving device Fig.10 Devices Fig.11 , Figure 21 to Figure 23 , Fig.28 and Fig.29 The decoder, Fig.24 and Fig.31 The receiving method of the present invention can receive the entire point cloud, decode it scalably, and select data by subsampling to perform a method of reconstructing the entire geometry / attributes or reconstructing part of the geometry / attributes.

[0499] In other words, when layered point cloud data is transmitted, beneficial effects can be achieved on both the transmitting and receiving sides. In this case, if information that enables the entire PCC data to be reconstructed is transmitted regardless of the performance of the receiver, the receiver needs to select data corresponding to a desired layer (i.e., perform data selection or subsampling) after reconstructing the point cloud data by decoding. In this case, since the delivered bit stream has been decoded, a delay may occur in a receiver targeting low latency, or decoding may not be performed depending on the receiver performance.

[0500] According to an embodiment, when the bitstream is divided into slices and delivered, the receiver can selectively deliver the bitstream to the decoder according to the decoder performance or the density of the point cloud data to be represented according to the application field. In this case, by performing the selection before decoding, the decoder efficiency can be improved and decoders with various performances can be supported.

[0501] Fig.30 A method of transmitting point cloud data according to an embodiment is illustrated.

[0502] S3000: The point cloud data sending method according to the implementation may include encoding the point cloud data.

[0503] The encoding operation according to an embodiment includes Figure 1 The point cloud video acquirer 10001, the point cloud video encoder 10002 and the transmitter 10003 of the transmitting device 10000 may also include Figure 2 Get 20000 / code 20001, Figure 3 , Figure 8 , Fig.11 , Fig.21 , Fig.28 and Fig.29 The encoder, Figures 4 to 6 The encoding, Fig.10 XR device 1030, Figures 12 to 20 The encoding, Fig. 22 and Fig.23The encoding and Figure 25 to Figure 27 The parameters and bitstream generation operations.

[0504] S3001: The point cloud data transmission method according to the embodiment may further include transmitting a bit stream containing the point cloud data. The bit stream may include the encoded point cloud and generated parameters.

[0505] Fig.31 A method of receiving point cloud data according to an embodiment is illustrated.

[0506] S3100: The point cloud data receiving method according to an embodiment may include receiving a bit stream containing point cloud data.

[0507] Receiving operations according to embodiments may include receiving a bitstream including the encoded point cloud and the generated parameters.

[0508] S3101: According to the implementation scheme, the point cloud data receiving method may further include decoding the point cloud data.

[0509] The decoding operation according to an embodiment may include Figure 1 The receiving device 10004, the renderer 10007, the point cloud video decoder 10006 and the receiver 10005, Figure 2 Transmission 20002-Decoding 20003-Rendering 20004, Figure 7 , Fig. 9 , Fig.11 , Fig.21 , Fig.28 and Fig.29 The decoder, Fig.10 XR device 1030, Figures 12 to 20 Layer group based decoding, Fig. 22 and Fig.24 Segment-based decoding, Figure 25 to Figure 27 Operations such as bit stream parsing.

[0510] Reference Figure 1 According to an embodiment, a method for sending a point cloud may include encoding point cloud data, and sending a bit stream containing the point cloud data.

[0511] Reference Fig.21 Regarding slices and FGS (or segmented slice units), encoding point cloud data in any of the above methods may include encoding point cloud data based on a layer group structure. The layer group structure may include one or more subgroups, wherein the subgroups may be delivered based on segmented slices.

[0512] Reference Fig.21, with respect to FGS and context reference, the segmented slice of the subgroup in any of the above methods includes information for a reference context. The information for the reference context indicates at least one of a parent layer group or a parent child group of the subgroup related to the segmented slice, wherein the segmented slice is encoded based on the information for the reference context.

[0513] Reference Fig.21 , with respect to FGS, context reference, and explicit signaling, the information for the reference context in any of the above methods may include at least one of a reference layer group ID or a reference subgroup ID. That is, it may include a reference layer group ID, or include a reference subgroup ID, or include both.

[0514] Reference Fig.21 , with respect to FGS, context reference, and implicit signaling, the segmented slice for the subgroup in any of the above methods includes information for a reference context. Alternatively, based on the absence of information for a reference context in the segmented slice, the segmented slice is encoded based on reference information associated with the segmented slice of the geometric data of the point cloud data.

[0515] Reference Fig. 22 , with respect to the flow chart of encoding (from the perspective of LOD generation), encoding point cloud data by any of the above methods includes encoding geometric data of the point cloud data, encoding attribute data of the point cloud data. Encoding the attribute data of the point cloud data includes generating a level of detail (LoD) for the attribute data, and mapping the LoD to a layer group, wherein the attribute data is encoded based on a subgroup of the layer group. Generate an attribute data unit header related to the encoding of the attribute data.

[0516] The point cloud data sending method is executed by a sending device. The sending device may include an encoder, or may include a memory and a processor. The instructions stored in the memory may be configured to enable the processor to perform an encoding operation.

[0517] The transmitting device may include: an encoder configured to encode point cloud data; and a transmitter configured to transmit a bit stream including the point cloud data.

[0518] The point cloud data receiving method may perform the opposite process of the sending method. The receiving method may include receiving a bit stream containing point cloud data and decoding the point cloud data.

[0519] Decoding the point cloud data in any of the above methods includes decoding the point cloud data based on a layer group structure. The layer group structure includes one or more subgroups, wherein the subgroups are delivered based on segmented slices.

[0520] The segmented slice of the subgroup in any of the above methods includes information for reference context. The information for reference context indicates at least one of a parent layer group or a parent subgroup of the subgroup related to the segmented slice, wherein the segmented slice is decoded based on the information for reference context.

[0521] The information for the reference context in any of the above methods may include at least one of a reference layer group ID or a reference subgroup ID. That is, it may include a reference layer group ID, or include a reference subgroup ID, or include both.

[0522] The segmented slices of the subset in any of the above methods include information for reference context. Alternatively, based on the absence of information for reference context in the segmented slices, the segmented slices are decoded based on reference information associated with the segmented slices of the geometric data of the point cloud data.

[0523] Reference Fig.24 , with respect to a flowchart of decoding, decoding the point cloud data in any of the above methods includes decoding the geometric data of the point cloud data, and decoding the attribute data of the point cloud data. Decoding the attribute data of the point cloud data includes generating a level of detail (LoD) for the attribute data, and mapping the LoD to a layer group, wherein the attribute data is encoded based on a subgroup of the layer group. Generate an attribute data unit header related to the decoding of the attribute data.

[0524] The point cloud data receiving method is performed by a receiving device. The receiving device may include a decoder, and may also include a memory and a processor. The instructions stored in the memory may be configured to cause the processor to perform a decoding operation. The receiving device may include: a receiver configured to receive a bit stream containing point cloud data, and a decoder configured to decode the point cloud data.

[0525] Therefore, the method / device according to the embodiment can process the point cloud data into a single slice, or the slice can be further segmented (e.g., segmented slices, slices based on layer groups, FGS, etc.), and the related data can be compressed and reconstructed. The method / device according to the embodiment can quickly and accurately support random access and / or partial coding. The segmented slice structure is used for geometry and attributes. Therefore, in order to solve the complexity and latency caused by encoding and decoding each segmentation unit, a context-referenced signaling method can be used. In order to efficiently store and load the context stored in the context buffer, the geometry segmented slice (layer group, subgroup) can refer to the header of the neighbor segmented slice. Similarly, the attribute segmented slice (layer group, subgroup) can refer to the header of the neighbor segmented slice. In other words, by signaling the layer group identification information and the subgroup identification information to indicate the parent-child group, the current segmented slice can be efficiently compressed and reconstructed. In addition, the signaling method can be supported in various modes, thereby allowing the creation and delivery of explicit signaling fields. In addition, when the explicit signaling mode is not enabled, the implicit signaling method can be used to reference the context by specifying the parent-child group.

[0526] The embodiments are described in terms of methods and / or devices, and the description of the methods and the description of the devices can be applied complementary to each other.

[0527] Although the drawings are described separately for simplicity, new embodiments can be designed by combining the embodiments shown in the corresponding figures. Designing a recording medium that can be read by a computer and records the program for executing the above-mentioned embodiments according to the needs of those skilled in the art also falls within the scope of the attached claims and their equivalents. The apparatus and method according to the embodiment may not be limited to the configuration and method of the above-mentioned embodiment. Various modifications can be made to the embodiment by selectively combining all or some of the embodiments. Although preferred embodiments are described with reference to the drawings, it will be understood by those skilled in the art that various modifications and changes can be made to the embodiment without departing from the spirit or scope of the present disclosure described in the attached claims. These modifications should not be understood separately from the technical ideas or viewpoints of the embodiments.

[0528] Various elements of the device of the embodiment can be implemented by hardware, software, firmware or a combination thereof. Various elements in the embodiment can be implemented by a single chip (e.g., a single hardware circuit). According to the embodiment, the components according to the embodiment can be implemented as separate chips respectively. According to the embodiment, at least one or more components of the device according to the embodiment may include one or more processors capable of executing one or more programs. One or more programs may execute any one or more operations / methods according to the embodiment or include instructions for executing them. The executable instructions for executing the method / operation of the device according to the embodiment may be stored in a non-transitory CRM or other computer program product configured to be executed by one or more processors, or may be stored in a temporary CRM or other computer program product configured to be executed by one or more processors. In addition, the memory according to the embodiment can be used as not only covering volatile memory (e.g., RAM), but also covering the concepts of non-volatile memory, flash memory and PROM. In addition, it can also be implemented in the form of a carrier (e.g., transmission via the Internet). In addition, the processor-readable recording medium can be distributed to a computer system connected via a network so that the processor-readable code can be stored and executed in a distributed manner.

[0529] In the present disclosure, " / " and "," should be interpreted as indicating "and / or". For example, the expression "A / B" may mean "A and / or B". Furthermore, "A, B" may mean "A and / or B". Furthermore, "A / B / C" may mean "at least one of A, B, and / or C". In addition, "A / B / C" may mean "at least one of A, B, and / or C". Furthermore, in the present specification, the term "or" should be interpreted as indicating "and / or". For example, the expression "A or B" may mean 1) only A, 2) only B, or 3) both A and B. In other words, the term "or" used in this document should be interpreted as indicating "in addition or alternatively".

[0530] Terms such as first and second can be used to describe various elements of the embodiments. However, various components according to the embodiments should not be limited by the above terms. These terms are only used to distinguish one element from another element. For example, a first user input signal can be referred to as a second user input signal. Similarly, a second user input signal can be referred to as a first user input signal. The use of these terms should not be interpreted without departing from the scope of various embodiments. Both the first user input signal and the second user input signal are user input signals, but unless the context clearly stipulates otherwise, they do not mean the same user input signal.

[0531] The terms used to describe the embodiments are used to describe specific embodiments and are not intended to limit the embodiments. As used in the description of the embodiments and the claims, unless the context clearly stipulates otherwise, the singular form includes plural referents. The expression "and / or" is used to include all possible combinations of terms. Terms such as "including" or "having" are intended to indicate the presence of graphics, quantities, steps, elements and / or components, and should be understood as not excluding the possibility of additional presence of graphics, quantities, steps, elements and / or components. As used herein, conditional expressions such as "if" and "when" are not limited to optional situations, and are intended to perform related operations or interpret related definitions according to specific conditions when specific conditions are met.

[0532] The operations according to the embodiments described in this specification may be performed by a transmitting / receiving device including a memory and / or a processor according to the embodiments. The memory may store a program for processing / controlling operations according to the embodiments, and the processor may control the various operations described in this specification. The processor may be referred to as a controller, etc. In an embodiment, the operations may be performed by firmware, software, and / or a combination thereof. Firmware, software, and / or a combination thereof may be stored in a processor or a memory.

[0533] The operations according to the above-described embodiments may be performed by a sending device and / or a receiving device according to the embodiments. The sending / receiving device may include a sender / receiver configured to send and receive media data, a memory configured to store instructions (program code, algorithm, flow chart and / or data) for the process according to the embodiments, and a processor configured to control the operation of the sending / receiving device.

[0534] The processor may be referred to as a controller, etc., and may correspond to, for example, hardware, software, and / or a combination thereof. The operations according to the above-described embodiments may be performed by the processor. In addition, the processor may be implemented as an encoder / decoder for the operations of the above-described embodiments.

[0535] Invention Mode

[0536] As described above, relevant details are described in the best mode for carrying out the embodiment.

[0537] Industrial Applicability

[0538] As described above, the embodiments may be applied in whole or in part to a point cloud data transmitting / receiving device and system.

[0539] A skilled person can change or modify the embodiments in various ways within the scope of the embodiments.

[0540] The embodiments may include changes / modifications within the scope of the claims and their equivalents.

Claims

1. A method for sending point cloud data, the method comprising: Encode point cloud data; as well as A bitstream containing the point cloud data is sent.

2. The point cloud data sending method according to claim 1, wherein: Encoding the point cloud data includes: Encoding the point cloud data based on a layer group structure, Wherein, the layer group structure includes one or more subgroups, Therein, the subgroups are delivered based on segmented slices.

3. The point cloud data sending method according to claim 2, wherein: The segmented slices of the subgroup contain information for a reference context, The information for the reference context indicates at least one of a parent layer group or a parent child group of the subgroup related to the segmented slice, Wherein, the segmented slice is encoded based on the information used for the reference context.

4. The point cloud data transmission method according to claim 3, wherein: The information for referring to the context includes at least one of a reference layer group ID or a reference sub-group ID.

5. The point cloud data transmission method according to claim 2, wherein: The segmented slices of the subset contain information for a reference context, or Wherein, based on the absence of the information for the reference context in the segmented slice, the segmented slice is encoded based on reference information related to the segmented slice of the geometric data of the point cloud data.

6. The point cloud data sending method according to claim 1, wherein: Encoding the point cloud data includes: Encoding the geometric data of the point cloud data; and encoding the attribute data of the point cloud data, The step of encoding the attribute data comprises: generating a level of detail (LoD) for the attribute data; and Mapping the LoDs to layer groups, wherein the attribute data is encoded based on a subgroup of the layer group, Therein, an attribute data unit header related to the encoding of the attribute data is generated.

7. A point cloud data sending device, the point cloud data sending device comprising: an encoder configured to encode point cloud data; as well as A transmitter is configured to transmit a bit stream containing the point cloud data.

8. A method for receiving point cloud data, the method comprising: receiving a bitstream containing point cloud data; as well as The point cloud data is decoded.

9. The point cloud data receiving method according to claim 8, wherein: Decoding the point cloud data includes: Decoding the point cloud data based on the layer group structure, Wherein, the layer group structure includes one or more subgroups, Therein, the subgroups are delivered based on segmented slices.

10. The point cloud data receiving method according to claim 9, wherein: The segmented slices of the subgroup contain information for a reference context, The information for the reference context indicates at least one of a parent layer group or a parent child group of the subgroup related to the segmented slice, Wherein, the segmented slice is decoded based on the information used for the reference context.

11. The point cloud data receiving method according to claim 10, wherein: The information for the reference context includes at least one of a reference layer group ID or a reference sub-group ID.

12. The point cloud data receiving method according to claim 9, wherein: The segmented slices of the subset contain information for a reference context, or Wherein, based on the absence of the information for the reference context in the segmented slice, the segmented slice is decoded based on reference information related to the segmented slice of the geometric data of the point cloud data.

13. The point cloud data receiving method according to claim 1, wherein: Decoding the point cloud data includes: Decoding the geometric data of the point cloud data; and Decoding the attribute data of the point cloud data, Wherein, decoding the attribute data comprises: generating a level of detail (LoD) for the attribute data; and Mapping the LoDs to layer groups, The attribute data is decoded based on a subgroup of the layer group.

14. A point cloud data receiving device, the point cloud data receiving device comprising: a receiver configured to receive a bitstream containing point cloud data; as well as A decoder is configured to decode the point cloud data.

Images