Point cloud data transmission device, point cloud data transmission method, point cloud data reception device, and point cloud data reception method

By encoding and transmitting point cloud data through bitstreams and utilizing compression techniques, the method addresses the inefficiencies in processing large point cloud data, enhancing service quality and responsiveness.

CN119096547APending Publication Date: 2024-12-06LG ELECTRONICS INC
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202280094392.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-04-22
Filing Date
2022-12-22
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

Existing technologies face challenges in efficiently processing large amounts of point cloud data required for applications like virtual reality, augmented reality, and self-driving services, leading to delays and encoding/decoding complexity.

Method used

A method and apparatus for encoding and transmitting point cloud data through bitstreams, followed by decoding and rendering, utilizing techniques such as geometry-based and video-based point cloud compression, and incorporating feedback information to optimize processing based on user interaction.

Benefits of technology

This approach enables efficient processing of point cloud data, providing high-quality services by reducing delays and complexity, and adapting to user interactions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119096547A_ABST
    Figure CN119096547A_ABST
Patent Text Reader

Abstract

A point cloud data transmission method according to an embodiment may comprise the steps of: encoding point cloud data; and transmitting the bit stream including the point cloud data. A point cloud data receiving method according to an embodiment may comprise the steps of: receiving a bitstream including point cloud data; and decoding the point cloud data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Embodiments relate to a method and apparatus for processing point cloud content. Background Art

[0002] Point cloud content is content represented by a point cloud, which is a collection of points belonging to a coordinate system representing a three-dimensional space. Point cloud content can express media configured in three dimensions and is used to provide various services such as virtual reality (VR), augmented reality (AR), mixed reality (MR), and self-driving services. However, tens of thousands to hundreds of thousands of point data are required to represent point cloud content. Therefore, a method of efficiently processing a large amount of point data is needed. Summary of the invention

[0003] Technical issues

[0004] The embodiment provides an apparatus and method for efficiently processing point cloud data. The embodiment provides a method and apparatus for processing point cloud data to solve delay and encoding / decoding complexity.

[0005] The technical scope of the embodiments is not limited to the above-mentioned technical purposes, and can be extended to other technical purposes that can be inferred by those skilled in the art based on the entire contents disclosed herein.

[0006] Technical Solution

[0007] To achieve these objects and other advantages, and in accordance with the purposes of the present disclosure, as embodied and broadly described herein, a method of transmitting point cloud data may include: encoding the point cloud data; and transmitting a bitstream containing the point cloud data. In another aspect of the present disclosure, a method of receiving point cloud data may include: receiving a bitstream containing the point cloud data; and decoding the point cloud data.

[0008] Beneficial Effects

[0009] The apparatus and method according to the embodiments can efficiently process point cloud data.

[0010] The apparatus and method according to the embodiments can provide high-quality point cloud services.

[0011] The device and method according to the embodiment may provide point cloud content for providing general services such as VR service and self-driving service. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings, which are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of the present application, illustrate embodiments of the present disclosure and are used together with the specification to explain the principles of the present disclosure. In order to better understand the various embodiments described below, reference should be made to the description of the following embodiments in conjunction with the accompanying drawings. The same reference numerals will be used throughout the drawings to refer to the same or similar parts. In the drawings:

[0013] Figure 1 An exemplary point cloud content providing system according to an embodiment is shown;

[0014] Figure 2 is a block diagram illustrating point cloud content providing operations according to an embodiment;

[0015] Figure 3 An exemplary process for capturing point cloud video according to an embodiment is shown;

[0016] Figure 4 An exemplary point cloud encoder according to an embodiment is shown;

[0017] Figure 5 shows an example of a voxel according to an embodiment;

[0018] Figure 6 shows examples of octrees and occupancy codes according to an embodiment;

[0019] Figure 7 shows an example of a neighbor node pattern according to an embodiment;

[0020] Figure 8 shows examples of point configurations in various LODs according to embodiments;

[0021] Fig. 9 shows examples of point configurations in various LODs according to embodiments;

[0022] Fig.10 A point cloud decoder according to an embodiment is shown;

[0023] Fig.11 A point cloud decoder according to an embodiment is shown;

[0024] Fig.12 A transmitting device according to an embodiment is shown;

[0025] Fig.13 A receiving device according to an embodiment is shown;

[0026] Fig.14 An exemplary structure that can be operated in conjunction with a point cloud data transmitting / receiving method / apparatus according to an embodiment is shown;

[0027] Fig.15 shows inter-frame prediction (inter-frame prediction) according to an embodiment;

[0028] Fig.16 shows a compact bounding box according to an embodiment;

[0029] Fig.17 A method of connecting compact bounding boxes according to an embodiment is shown;

[0030] Fig.18 A bitstream containing point cloud data and parameters according to an embodiment is shown;

[0031] Fig.19 shows the syntax of a sequence parameter set (SPS) according to an embodiment;

[0032] Fig. 20 shows the syntax of a geometry parameter set (GPS) according to an embodiment;

[0033] Fig.21 shows a matching mode according to an embodiment;

[0034] Fig. 22 shows matching criteria according to an embodiment;

[0035] Fig.23 shows a geometry slice header according to an embodiment;

[0036] Fig.24 shows matching criteria according to an embodiment;

[0037] Fig.25 shows matching criteria according to an embodiment;

[0038] Fig.26 shows an attribute parameter set (APS) according to an embodiment;

[0039] Fig. 27 shows an attribute data unit header according to an embodiment;

[0040] Fig.28 A point cloud data transmitting device according to an embodiment is shown;

[0041] Fig.29 A point cloud data receiving device according to an embodiment is shown;

[0042] Fig.30 A method of classifying static and dynamic objects of an object compact bounding box according to an embodiment is shown;

[0043] Fig.31 shows a reference bounding box in a reference frame and a current bounding box in a current frame according to an embodiment;

[0044] Fig.32 shows the syntax of the SPS according to an embodiment;

[0045] Fig.33 shows the syntax of GPS according to an embodiment;

[0046] Fig.34 shows a standard schema for dynamic objects according to an embodiment;

[0047] Fig.35 shows the syntax of a geometry slice header according to an embodiment;

[0048] Fig.36 A point cloud data transmitting device according to an embodiment is shown;

[0049] Fig.37 A point cloud data receiving device according to an embodiment is shown;

[0050] Fig.38 A point cloud data transmission method according to an embodiment is shown; and

[0051] Fig.39 A point cloud data receiving method according to an embodiment is shown. DETAILED DESCRIPTION

[0052] Reference will now be made in detail to preferred embodiments of the present disclosure, examples of which are shown in the accompanying drawings. The detailed description given below with reference to the accompanying drawings is intended to illustrate exemplary embodiments of the present disclosure, rather than to illustrate the only embodiments that can be implemented according to the present disclosure. The following detailed description includes specific details in order to provide a thorough understanding of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure can be practiced without these specific details.

[0053] Although most of the terms used in the present disclosure are selected from common terms widely used in the art, some terms are arbitrarily selected by the applicant and their meanings are explained in detail in the following description as needed. Therefore, the present disclosure should be understood based on the intended meanings of the terms rather than their simple names or meanings.

[0054] Figure 1 An exemplary point cloud content providing system according to an embodiment is shown.

[0055] Figure 1 The point cloud content providing system shown may include a transmitting device 10000 and a receiving device 10004. The transmitting device 10000 and the receiving device 10004 can transmit and receive point cloud data via wired or wireless communication.

[0056] According to an embodiment, the point cloud data sending device 10000 can obtain and process point cloud video (or point cloud content) and send it. According to an embodiment, the sending device 10000 may include a fixed station, a base transceiver system (BTS), a network, an artificial intelligence (AI) device and / or system, a robot, an AR / VR / XR device and / or a server. According to an embodiment, the sending device 10000 may include a device configured to communicate with a base station and / or other wireless devices using a radio access technology (e.g., 5G new RAT (NR), long term evolution (LTE)), a robot, a vehicle, an AR / VR / XR device, a portable device, a home appliance, an Internet of Things (IoT) device, and an AI device / server.

[0057] According to the implementation manner, the sending device 10000 includes a point cloud video acquirer 10001, a point cloud video encoder 10002 and / or a transmitter (or communication module) 10003.

[0058] The point cloud video acquirer 10001 according to the embodiment acquires the point cloud video through a processing process such as capture, synthesis, generation, etc. The point cloud video is a point cloud content represented by a point cloud, which is a collection of points located in a 3D space and can be referred to as point cloud video data, point cloud data, etc. The point cloud video according to the embodiment can include one or more frames. A frame represents a still image / picture. Therefore, the point cloud video can include a point cloud image / frame / picture and can be referred to as a point cloud image, frame or picture.

[0059] The point cloud video encoder 10002 according to the embodiment encodes the acquired point cloud video data. The point cloud video encoder 10002 may encode the point cloud video data based on point cloud compression coding. The point cloud compression coding according to the embodiment may include geometry-based point cloud compression (G-PCC) coding and / or video-based point cloud compression (V-PCC) coding or next generation coding. The point cloud compression coding according to the embodiment is not limited to the above-mentioned embodiment. The point cloud video encoder 10002 may output a bit stream containing the encoded point cloud video data. The bit stream may include not only the encoded point cloud video data, but also signaling information related to the encoding of the point cloud video data.

[0060] The transmitter 10003 according to the embodiment transmits a bitstream containing encoded point cloud video data. The bitstream according to the embodiment is encapsulated in a file or a fragment (e.g., a stream fragment) and is transmitted via various networks such as a broadcast network and / or a broadband network. Although not shown in the figure, the transmitting device 10000 may include a packager (or a package module) configured to perform a packaging operation. According to the embodiment, the packager may be included in the transmitter 10003. According to the embodiment, the file or fragment may be sent to the receiving device 10004 via a network, or stored in a digital storage medium (e.g., USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc.). The transmitter 10003 according to the embodiment can communicate with the receiving device 10004 (or the receiver 10005) via a 4G, 5G, 6G or other network by wire / wireless communication. In addition, the transmitter may perform necessary data processing operations according to the network system (e.g., a 4G, 5G or 6G communication network system). The transmitting device 10000 may transmit the encapsulated data in an on-demand manner.

[0061] The receiving device 10004 according to an embodiment includes a receiver 10005, a point cloud video decoder 10006 and / or a renderer 10007. According to an embodiment, the receiving device 10004 may include a device configured to communicate with a base station and / or other wireless devices using a radio access technology (e.g., 5G New RAT (NR), Long Term Evolution (LTE)), a robot, a vehicle, an AR / VR / XR device, a portable device, a home appliance, an Internet of Things (IoT) device, and an AI device / server.

[0062] The receiver 10005 according to the embodiment receives a bitstream containing point cloud video data or a file / segment encapsulated with a bitstream from a network or a storage medium. The receiver 10005 may perform necessary data processing according to a network system (e.g., a communication network system of 4G, 5G, 6G, etc.). The receiver 10005 according to the embodiment may decapsulate the received file / segment and output a bitstream. According to the embodiment, the receiver 10005 may include a decapsulator (or decapsulation module) configured to perform a decapsulation operation. The decapsulator may be implemented as an element (or component) separate from the receiver 10005.

[0063] The point cloud video decoder 10006 decodes a bitstream containing point cloud video data. The point cloud video decoder 10006 can decode the point cloud video data according to a method for encoding the point cloud video data (e.g., in the reverse process of the operation of the point cloud video encoder 10002). Therefore, the point cloud video decoder 10006 can decode the point cloud video data by performing point cloud decompression encoding (as the reverse process of point cloud compression). The point cloud decompression encoding includes G-PCC encoding.

[0064] The renderer 10007 renders the decoded point cloud video data. The renderer 10007 may output the point cloud content by rendering not only the point cloud video data but also the audio data. According to an embodiment, the renderer 10007 may include a display configured to display the point cloud content. According to an embodiment, the display may be implemented as a separate device or component rather than being included in the renderer 10007.

[0065] The arrow indicated by the dotted line in the figure represents the transmission path of the feedback information obtained by the receiving device 10004. Feedback information is information that reflects the interactivity with the user who consumes the point cloud content, and includes information about the user (e.g., head orientation information, viewport information, etc.). Specifically, when the point cloud content is content for a service that requires interaction with the user (e.g., self-driving service, etc.), the feedback information may be provided to the content sender (e.g., sending device 10000) and / or the service provider. Depending on the embodiment, the feedback information may be used in the receiving device 10004 and the sending device 10000, or may not be provided.

[0066] The head orientation information according to the embodiment is information about the user's head position, orientation, angle, movement, etc. The receiving device 10004 according to the embodiment may calculate the viewport information based on the head orientation information. The viewport information may be information about the area of ​​the point cloud video that the user is watching. The viewpoint is the point through which the user watches the point cloud video, and may refer to the center point of the viewport area. That is, the viewport is an area centered on the viewpoint, and the size and shape of the area may be determined by the field of view (FOV). Therefore, in addition to the head orientation information, the receiving device 10004 may also extract the viewport information based on the vertical or horizontal FOV supported by the device. In addition, the receiving device 10004 performs gaze analysis, etc. to check the way the user consumes the point cloud, the area where the user gazes in the point cloud video, the gaze time, etc. According to the embodiment, the receiving device 10004 may send feedback information including the gaze analysis result to the sending device 10000. The feedback information according to the embodiment may be obtained during the rendering and / or display process. The feedback information according to the embodiment may be obtained by one or more sensors included in the receiving device 10004. According to an embodiment, the feedback information may be obtained by the renderer 10007 or a separate external element (or device, component, etc.). Figure 1The dotted line in represents the process of sending the feedback information obtained by the renderer 10007. The point cloud content providing system can process (encode / decode) the point cloud data based on the feedback information. Therefore, the point cloud video data decoder 10006 can perform a decoding operation based on the feedback information. The receiving device 10004 can send the feedback information to the sending device 10000. The sending device 10000 (or the point cloud video data encoder 10002) can perform an encoding operation based on the feedback information. Therefore, the point cloud content providing system can efficiently process necessary data (for example, point cloud data corresponding to the user's head position) based on the feedback information instead of processing (encoding / decoding) the entire point cloud data, and provide the point cloud content to the user.

[0067] According to an embodiment, the transmitting device 10000 may be referred to as an encoder, a transmitting device, a transmitter, etc., and the receiving device 10004 may be referred to as a decoder, a receiving device, a receiver, etc.

[0068] According to the implementation mode Figure 1 The point cloud data processed in the point cloud content providing system (through a series of processes of acquisition / encoding / transmission / decoding / rendering) may be referred to as point cloud content data or point cloud video data. According to an embodiment, point cloud content data may be used as a concept covering metadata or signaling information related to point cloud data.

[0069] Figure 1 The elements of the point cloud content providing system shown may be implemented by hardware, software, a processor and / or a combination thereof.

[0070] Figure 2 is a block diagram illustrating point cloud content providing operations according to an embodiment.

[0071] Figure 2 The block diagram shows Figure 1 The operation of the point cloud content providing system described in . As described above, the point cloud content providing system can process point cloud data based on point cloud compression coding (e.g., G-PCC).

[0072] According to the embodiment, the point cloud content providing system (e.g., the point cloud sending device 10000 or the point cloud video acquirer 10001) can acquire a point cloud video (20000). The point cloud video is represented by a point cloud belonging to a coordinate system for representing a 3D space. The point cloud video provided according to the embodiment may include a Ply (Polygon file format or Stanford Triangle format) file. When the point cloud video has one or more frames, the acquired point cloud video may include one or more Ply files. The Ply file contains point cloud data, such as point geometry and / or attributes. The geometry includes the position of the point. The position of each point can be represented by a parameter (e.g., the value of the X, Y and Z axes) representing a three-dimensional coordinate system (e.g., a coordinate system composed of X, Y and Z axes). The attributes include the attributes of the point (e.g., information about the texture of each point, color (in YCbCr or RGB), reflectivity r, transparency, etc.). A point has one or more attributes. For example, a point may have an attribute as a color, or two attributes as color and reflectivity. According to an embodiment, geometry may be referred to as position, geometry information, geometry data, position information, position data, etc., and attributes may be referred to as attributes, attribute information, attribute data, etc. The point cloud content providing system (e.g., the point cloud transmitting device 10000 or the point cloud video acquirer 10001) may protect the point cloud data from information related to the acquisition process of the point cloud video (e.g., depth information, color information, etc.).

[0073] According to an embodiment, a point cloud content providing system (e.g., a transmitting device 10000 or a point cloud video encoder 10002) may encode point cloud data (20001). The point cloud content providing system may encode point cloud data based on point cloud compression coding. As described above, point cloud data may include geometry and attributes of points. Therefore, the point cloud content providing system may perform geometry coding for encoding geometry and output a geometry bitstream. The point cloud content providing system may perform attribute coding for encoding attributes and output an attribute bitstream. According to an embodiment, the point cloud content providing system may perform attribute coding based on geometry coding. The geometry bitstream and the attribute bitstream according to the embodiment may be multiplexed and output as one bitstream. The bitstream according to the embodiment may also include signaling information related to geometry coding and attribute coding.

[0074] According to the embodiment, the point cloud content providing system (eg, the transmitting device 10000 or the transmitter 10003) may transmit the encoded point cloud data (20002). Figure 1As shown, the encoded point cloud data may be represented by a geometry bitstream and an attribute bitstream. In addition, the encoded point cloud data may be sent in the form of a bitstream together with signaling information related to encoding of the point cloud data (e.g., signaling information related to geometry encoding and attribute encoding). The point cloud content providing system may encapsulate the bitstream carrying the encoded point cloud data and send it in the form of a file or a fragment.

[0075] The point cloud content providing system (eg, receiving device 10004 or receiver 10005) according to the embodiment may receive a bit stream containing encoded point cloud data. In addition, the point cloud content providing system (eg, receiving device 10004 or receiver 10005) may demultiplex the bit stream.

[0076] The point cloud content providing system (e.g., the receiving device 10004 or the point cloud video decoder 10005) may decode the encoded point cloud data (e.g., geometry bitstream, attribute bitstream) sent in the bitstream. The point cloud content providing system (e.g., the receiving device 10004 or the point cloud video decoder 10005) may decode the point cloud video data based on the signaling information related to the encoding of the point cloud video data contained in the bitstream. The point cloud content providing system (e.g., the receiving device 10004 or the point cloud video decoder 10005) may decode the geometry bitstream to reconstruct the position (geometry) of the point. The point cloud content providing system may reconstruct the attributes of the point by decoding the attribute bitstream based on the reconstructed geometry. The point cloud content providing system (e.g., the receiving device 10004 or the point cloud video decoder 10005) may reconstruct the point cloud video based on the position according to the reconstructed geometry and the decoded attributes.

[0077] According to an embodiment, a point cloud content providing system (e.g., receiving device 10004 or renderer 10007) may render decoded point cloud data (20004). The point cloud content providing system (e.g., receiving device 10004 or renderer 10007) may use various rendering methods to render the geometry and attributes decoded by the decoding process. Points in the point cloud content may be rendered as vertices with a specific thickness, cubes with a specific minimum size centered at the corresponding vertex position, or circles centered at the corresponding vertex position. All or part of the rendered point cloud content is provided to the user through a display (e.g., a VR / AR display, a general display, etc.).

[0078] The point cloud content providing system (e.g., receiving device 10004) according to the embodiment may obtain feedback information (20005). The point cloud content providing system may encode and / or decode the point cloud data based on the feedback information. Feedback Information and Operations of the Point Cloud Content Providing System According to the Embodiment and Reference Figure 1 The feedback information and operations described are the same, so their detailed description is omitted.

[0079] Figure 3 An exemplary process of capturing point cloud video according to an embodiment is shown.

[0080] Figure 3 Show reference Figure 1 to Figure 2 An exemplary point cloud video capture process of a point cloud content providing system is described.

[0081] Point cloud content includes point cloud videos (images and / or videos) representing objects and / or environments located in various 3D spaces (e.g., 3D spaces representing real environments, 3D spaces representing virtual environments, etc.). Therefore, a point cloud content providing system according to an embodiment may use one or more cameras (e.g., an infrared camera capable of obtaining depth information, an RGB camera capable of extracting color information corresponding to the depth information, etc.), a projector (e.g., an infrared pattern projector that obtains depth information), LiDAR, etc. to capture point cloud videos. A point cloud content providing system according to an embodiment may extract a geometric shape composed of points in a 3D space from the depth information and extract the attributes of each point from the color information to obtain point cloud data. Images and / or videos according to an embodiment may be captured based on at least one of an inward-facing technique and an outward-facing technique.

[0082] Figure 3 The left portion of shows inward-facing techniques. Inward-facing techniques refer to techniques that utilize one or more cameras (or camera sensors) positioned around a central object to capture an image of the central object. Inward-facing techniques may be used to generate point cloud content that provides a 360-degree image of a key object to a user (e.g., VR / AR content that provides a 360-degree image of an object (e.g., a key object such as a character, player, object, or actor) to a user).

[0083] Figure 3 The right portion of shows outward-facing techniques. Outward-facing techniques refer to techniques that utilize one or more cameras (or camera sensors) positioned around a central object to capture the environment of the central object rather than an image of the central object. Outward-facing techniques can be used to generate point cloud content that provides the surrounding environment as it appears from the user's perspective (e.g., content representing the external environment that can be provided to a user of a self-driving vehicle).

[0084] As shown in the figure, the point cloud content may be generated based on the capture operation of one or more cameras. In this case, the coordinate system may be different between the cameras, so the point cloud content providing system may calibrate one or more cameras to set the global coordinate system before the capture operation. In addition, the point cloud content providing system may generate point cloud content by synthesizing any image and / or video with the image and / or video captured by the above-mentioned capture technology. The point cloud content providing system may not perform when generating point cloud content representing the virtual space. Figure 3The point cloud content providing system according to the embodiment may perform post-processing on the captured image and / or video. In other words, the point cloud content providing system may remove unwanted areas (e.g., background), identify the space to which the captured image and / or video is connected, and when there is a space hole, perform an operation to fill the space hole.

[0085] The point cloud content providing system can generate a point cloud content by performing coordinate transformation on the points of the point cloud video obtained from each camera. The point cloud content providing system can perform coordinate transformation on the points based on the position coordinates of each camera. Therefore, the point cloud content providing system can generate content representing a wide range, or can generate point cloud content with high density points.

[0086] Figure 4 An exemplary point cloud encoder according to an implementation is shown.

[0087] Figure 4 Show Figure 1 An example of a point cloud video encoder 10002. The point cloud encoder reconstructs and encodes point cloud data (e.g., the location and / or attributes of the points) to adjust the quality of the point cloud content (e.g., lossless, lossy, or near-lossless) according to network conditions or applications. When the total size of the point cloud content is large (e.g., 60 Gbps of point cloud content given for 30 fps), the point cloud content providing system may not be able to stream the content in real time. Therefore, the point cloud content providing system may reconstruct the point cloud content based on the maximum target bit rate to provide the point cloud content according to the network environment, etc.

[0088] As reference Figure 1 and Figure 2 As described, the point cloud encoder can perform geometry encoding and attribute encoding. The geometry encoding is performed before the attribute encoding.

[0089] According to an embodiment, the point cloud encoder includes a coordinate transformer (transforming coordinates) 40000, a quantizer (quantizing and removing points (voxelization)) 40001, an octree analyzer (analyzing octree) 40002 and a surface approximation analyzer (analyzing surface approximation) 40003, an arithmetic encoder (arithmetic coding) 40004, a geometry reconstructor (reconstructing geometry) 40005, a color transformer (transforming color) 40006, an attribute transformer (transforming attributes) 40007, a RAHT transformer (RAHT) 40008, an LOD generator (generating LOD) 40009, a lifting transformer (lifting) 40010, a coefficient quantizer (quantizing coefficients) 40011 and / or an arithmetic encoder (arithmetic coding) 40012.

[0090] The coordinate transformer 40000, the quantizer 40001, the octree analyzer 40002, the surface approximation analyzer 40003, the arithmetic encoder 40004 and the geometry reconstructor 40005 may perform geometry coding. The geometry coding according to the embodiment may include octree geometry coding, direct coding, triplet geometry coding and entropy coding. Direct coding and triplet geometry coding are applied selectively or in combination. The geometry coding is not limited to the above examples.

[0091] As shown in the figure, the coordinate transformer 40000 according to the embodiment receives the position and transforms it into coordinates. For example, the position can be transformed into position information in a three-dimensional space (e.g., a three-dimensional space represented by an XYZ coordinate system). The position information in the three-dimensional space according to the embodiment can be referred to as geometric information.

[0092] The quantizer 40001 according to the embodiment quantizes the geometry. For example, the quantizer 40001 may quantize the points based on the minimum position value of all points (e.g., the minimum value on each of the X, Y, and Z axes). The quantizer 40001 performs a quantization operation: the difference between the minimum position value and the position value of each point is multiplied by a preset quantization scaling value, and then the nearest integer value is found by rounding the value obtained by the multiplication. Therefore, one or more points may have the same quantized position (or position value). The quantizer 40001 according to the embodiment performs voxelization based on the quantized position to reconstruct the quantized point. As in the case of a pixel (the smallest unit containing 2D image / video information), the point of the point cloud content (or 3D point cloud video) according to the embodiment may be included in one or more voxels. As a composite of volume and pixel, the term voxel refers to a 3D cubic space generated when the 3D space is divided into units (unit = 1.0) based on the axes representing the 3D space (e.g., X-axis, Y-axis, and Z-axis). The quantizer 40001 may match a point group in a 3D space with a voxel. According to an embodiment, a voxel may include only one point. According to an embodiment, a voxel may include one or more points. In order to represent a voxel as a point, the position of the center of the voxel may be set based on the position of one or more points included in the voxel. In this case, the properties of all positions included in a voxel may be combined and assigned to the voxel.

[0093] The octree analyzer 40002 according to the embodiment performs octree geometry encoding (or octree encoding) to present voxels in an octree structure. The octree structure represents points matched to voxels based on an octal tree structure.

[0094] The surface approximation analyzer 40003 according to an embodiment may analyze and approximate an octree. The octree analysis and approximation according to an embodiment is a process of analyzing a region including a plurality of points to efficiently provide an octree and voxelization.

[0095] The arithmetic encoder 40004 according to the embodiment performs entropy coding on the octree and / or the approximate octree. For example, the coding scheme includes arithmetic coding. As a result of the coding, a geometry bitstream is generated.

[0096] The color converter 40006, the attribute converter 40007, the RAHT converter 40008, the LOD generator 40009, the lifting converter 40010, the coefficient quantizer 40011 and / or the arithmetic encoder 40012 perform attribute encoding. As described above, a point may have one or more attributes. The attribute encoding according to the embodiment is also applied to the attributes of a point. However, when the attribute (e.g., color) includes one or more elements, the attribute encoding is applied independently to each element. The attribute encoding according to the embodiment includes color transform encoding, attribute transform encoding, regional adaptive hierarchical transform (RAHT) encoding, interpolation-based hierarchical nearest neighbor prediction (prediction transform) encoding, and interpolation-based hierarchical nearest neighbor prediction (lifting transform) encoding with an update / lifting step. Depending on the point cloud content, the above-mentioned RAHT encoding, prediction transform encoding, and lifting transform encoding may be selectively used, or a combination of one or more encoding schemes may be used. The attribute encoding according to the embodiment is not limited to the above examples.

[0097] The color converter 40006 according to the embodiment performs color conversion encoding to convert the color value (or texture) included in the attribute. For example, the color converter 40006 can convert the format of the color information (e.g., from RGB to YCbCr). Optionally, the operation of the color converter 40006 according to the embodiment can be applied according to the color value included in the attribute.

[0098] The geometry reconstructor 40005 according to an embodiment reconstructs (decompresses) an octree and / or an approximate octree. The geometry reconstructor 40005 reconstructs an octree / voxel based on the result of analyzing the point distribution. The reconstructed octree / voxel may be referred to as a reconstructed geometry (restored geometry).

[0099] The attribute transformer 40007 according to the embodiment performs attribute transformation to transform the attribute based on the reconstructed geometry and / or the position where the geometry encoding is not performed. As described above, since the attribute depends on the geometry, the attribute transformer 40007 can transform the attribute based on the reconstructed geometry information. For example, based on the position value of the point included in the voxel, the attribute transformer 40007 can transform the attribute of the point at the position. As described above, when the center position of the voxel is set based on the position of one or more points included in the voxel, the attribute transformer 40007 transforms the attribute of one or more points. When triplet geometry encoding is performed, the attribute transformer 40007 can transform the attribute based on the triplet geometry encoding.

[0100] The attribute converter 40007 can perform attribute conversion by calculating the average of the attributes or attribute values ​​(e.g., the color or reflectivity of each point) of the neighboring points within a specific position / radius from the center position (or position value) of each voxel. The attribute converter 40007 can apply weights according to the distance from the center to each point when calculating the average. Therefore, each voxel has a position and a calculated attribute (or attribute value).

[0101] Attribute converter 40007 can search for neighboring points within a specific position / radius of the position of the center of each voxel based on a KD tree or a Morton code. The KD tree is a binary search tree and supports a data structure capable of managing points based on position, so that a nearest neighbor search (NNS) can be performed quickly. The Morton code is generated by presenting the coordinates (e.g., (x, y, z)) representing the 3D position of all points as bit values ​​and mixing these bits. For example, when the coordinates representing the position of the point are (5, 9, 1), the bit value of the coordinate is (0101, 1001, 0001). Mixing the bit values ​​in the order of z, y, and x according to the bit index generates 010001000111, which is represented as a decimal number 1095. That is, the Morton code value of the point with coordinates (5, 9, 1) is 1095. Attribute converter 40007 can sort the points based on the Morton code value and perform NNS through a depth-first traversal process. After the attribute transformation operation, when NNS is needed in another transformation process for attribute encoding, KD tree or Morton code is used.

[0102] As shown, the transformed attributes are input to the RAHT transformer 40008 and / or the LOD generator 40009.

[0103] The RAHT transformer 40008 according to an embodiment performs RAHT encoding for predicting attribute information based on the reconstructed geometric information. For example, the RAHT transformer 40008 may predict attribute information of a higher-level node in the octree based on attribute information associated with a lower-level node in the octree.

[0104] The LOD generator 40009 according to an embodiment generates a level of detail (LOD) to perform predictive transform coding. The LOD according to an embodiment is the degree of detail of the point cloud content. As the LOD value decreases, the details of the point cloud content are degraded. As the LOD value increases, the details of the point cloud content are enhanced. Points can be classified by LOD.

[0105] The lifting transformer 40010 according to an embodiment performs lifting transform coding that transforms point cloud attributes based on weights. As described above, lifting transform coding can be optionally applied.

[0106] The coefficient quantizer 40011 according to the embodiment quantizes the attribute of the attribute encoding based on the coefficient.

[0107] The arithmetic encoder 40012 according to the embodiment encodes quantized properties based on arithmetic coding.

[0108] Although not shown in the figure, Figure 4 Elements of the point cloud encoder may be implemented by hardware including one or more processors or integrated circuits configured to communicate with one or more memories included in the point cloud providing device, software, firmware, or a combination thereof. The one or more processors may perform the above Figure 4 At least one of the operations and / or functions of the elements of the point cloud encoder. In addition, one or more processors may operate or execute to perform Figure 4 A set of software programs and / or instructions for the operation and / or functionality of the elements of the point cloud encoder. One or more memories according to the embodiment may include high-speed random access memory, or include non-volatile memory (e.g., one or more disk storage devices, flash memory devices, or other non-volatile solid-state storage devices).

[0109] Figure 5 An example of a voxel according to an embodiment is shown.

[0110] Figure 5 The voxels are shown in a 3D space represented by a coordinate system consisting of three axes (X axis, Y axis and Z axis). Figure 4 As described above, the point cloud encoder (eg, quantizer 40001) may perform voxelization. A voxel refers to a 3D cubic space generated when a 3D space is divided into units (unit=1.0) based on axes representing the 3D space (eg, X-axis, Y-axis, and Z-axis). Figure 5 An example of voxels generated by an octree structure is shown, where a cubic axis-aligned bounding box defined by two poles (0,0,0) and (2d,2d,2d) is recursively subdivided. A voxel includes at least one point. The spatial coordinates of the voxel can be estimated from the positional relationship with the voxel group. As described above, the voxel has properties similar to the pixels of a 2D image / video (e.g., color or reflectivity). The details of the voxel are similar to those of the reference Figure 4 Those described are the same, so their description is omitted.

[0111] Figure 6 Examples of octrees and occupancy codes are shown according to an embodiment.

[0112] As reference Figures 1 to 4 As described, the point cloud content providing system (point cloud video encoder 10002) or the point cloud encoder (e.g., octree analyzer 40002) performs octree geometry encoding (or octree encoding) based on the octree structure to efficiently manage the area and / or position of voxels.

[0113] Figure 6 The upper part of FIG. 1 shows the octree structure. The 3D space of the point cloud content according to the embodiment is represented by the axes (eg, X-axis, Y-axis, and Z-axis) of the coordinate system. d ,2 d ,2 d ) to create an octree structure. Here, 2 d It can be set to the value of the minimum bounding box that surrounds all points of the point cloud content (or point cloud video). Here, d represents the depth of the octree. The value of d is determined in the following formula. In the following formula, (x int n ,y int n ,z int n ) represents the position (or position value) of the quantized point.

[0114]

[0115] like Figure 6 As shown in the middle of the upper part of , the entire 3D space can be divided into eight spaces according to the partition. Each divided space is represented by a cube with six faces. Figure 6 As shown in the upper right part of , each of the eight spaces is divided again based on the axis of the coordinate system (e.g., X-axis, Y-axis, and Z-axis). Therefore, each space is divided into eight smaller spaces. The divided smaller spaces are also represented by cubes with six faces. This partitioning scheme is applied until the leaf nodes of the octree become voxels.

[0116] Figure 6 The lower part shows the octree occupancy code. The occupancy code of the octree is generated to indicate whether each of the eight divided spaces generated by dividing one space contains at least one point. Therefore, a single occupancy code is represented by eight child nodes. Each child node represents the occupancy of the divided space, and the child node has a 1-bit value. Therefore, the occupancy code is represented as an 8-bit code. That is, when the space corresponding to the child node contains at least one point, the node is assigned a value of 1. When the space corresponding to the child node does not contain a point (the space is empty), the node is assigned a value of 0. Since Figure 6The occupancy code shown is 00100001, so it indicates that the spaces corresponding to the third and eighth subnodes among the eight subnodes each contain at least one point. As shown in the figure, each of the third and eighth subnodes has eight subnodes, and the subnodes are represented by 8-bit occupancy codes. The accompanying drawings show that the occupancy code of the third subnode is 10000111, and the occupancy code of the eighth subnode is 01001111. The point cloud encoder (e.g., arithmetic encoder 40004) according to the embodiment may perform entropy encoding on the occupancy code. In order to increase the compression efficiency, the point cloud encoder may perform intra-frame / inter-frame encoding on the occupancy code. The receiving device (e.g., receiving device 10004 or point cloud video decoder 10006) according to the embodiment reconstructs the occupancy code based on the occupancy code.

[0117] According to an embodiment, a point cloud encoder (e.g., Figure 4 The point cloud encoder or octree analyzer 40002) may perform voxelization and octree encoding to store point locations. However, points are not always evenly distributed in the 3D space, so there may be specific areas where fewer points exist. Therefore, performing voxelization on the entire 3D space is inefficient. For example, when a specific area contains very few points, voxelization does not need to be performed in the specific area.

[0118] Therefore, for the above-mentioned specific area (or nodes other than the leaf nodes of the octree), the point cloud encoder according to the embodiment may skip voxelization and perform direct encoding to directly encode the point positions included in the specific area. The coordinates of the directly encoded points according to the embodiment are called direct coding mode (DCM). The point cloud encoder according to the embodiment may also perform triplet geometry encoding based on the surface model, which is to reconstruct the point positions in the specific area (or node) based on voxels. Triplet geometry encoding is a geometry encoding that represents an object as a series of triangular meshes. Therefore, the point cloud decoder can generate a point cloud from the mesh surface. Direct encoding and triplet geometry encoding according to the embodiment can be performed selectively. In addition, direct encoding and triplet geometry encoding according to the embodiment can be performed in combination with octree geometry encoding (or octree encoding).

[0119] In order to perform direct encoding, the option of using direct mode to apply direct encoding should be enabled. The node to which direct encoding is to be applied is not a leaf node, and there should be points less than a threshold value within a specific node. In addition, the total number of points to which direct encoding is to be applied should not exceed a preset threshold. When the above conditions are met, the point cloud encoder (or arithmetic encoder 40004) according to the embodiment can perform entropy encoding on the point position (or position value).

[0120] A point cloud encoder (e.g., surface approximation analyzer 40003) according to an embodiment may determine a specific level of the octree (a level less than the depth d of the octree), and may use a surface model starting from this level to perform triplet geometry encoding to reconstruct point positions in the node area based on voxels (triplet mode). A point cloud encoder according to an embodiment may specify a level to which triplet geometry encoding is to be applied. For example, when a specific level is equal to the depth of the octree, the point cloud encoder does not operate in triplet mode. In other words, a point cloud encoder according to an embodiment may operate in triplet mode only when the specified level is less than the depth value of the octree. A 3D cubic area of ​​a node at a specified level according to an embodiment is referred to as a block. A block may include one or more voxels. A block or voxel may correspond to a block. Geometry is represented as a surface within each block. A surface according to an embodiment may intersect each edge of a block at most once.

[0121] A block has 12 edges, so there are at least 12 intersections in a block. Each intersection is called a vertex. When there is at least one occupied voxel adjacent to the edge among all blocks sharing the edge, the vertex existing along the edge is detected. An occupied voxel according to an embodiment refers to a voxel containing a point. The vertex position detected along the edge is the average position of the edge of all voxels adjacent to the edge among all blocks along the shared edge.

[0122] Once the vertex is detected, the point cloud encoder according to the embodiment can perform entropy encoding on the start point (x, y, z) of the edge, the direction vector (Δx, Δy, Δz) of the edge, and the vertex position value (relative position value within the edge). When triplet geometry encoding is applied, the point cloud encoder according to the embodiment (e.g., geometry reconstructor 40005) can generate restored geometry (reconstructed geometry) by performing triangle reconstruction, upsampling, and voxelization processes.

[0123] The vertices located at the edges of the block determine the surface passing through the block. The surface according to an embodiment is a non-planar polygon. In the triangle reconstruction process, the surface represented by the triangle is reconstructed based on the starting point of the edge, the direction vector of the edge and the position value of the vertex. The triangle reconstruction process is performed by i) calculating the centroid value of each vertex, ii) subtracting the center value from each vertex value, and iii) estimating the sum of the squares of the values ​​obtained by subtraction.

[0124]

[0125] The minimum value of the sum is estimated, and the projection process is performed according to the axis with the minimum value. For example, when the element x is the smallest, each vertex is projected on the x-axis relative to the center of the block and projected on the (y,z) plane. When the value obtained by the projection on the (y,z) plane is (ai,bi), the value of θ is estimated by atan2(bi,ai), and the vertices are sorted based on the value of θ. The following table shows the vertex combination for creating a triangle according to the number of vertices. The vertices are sorted from 1 to n. The following table shows that for four vertices, two triangles can be constructed according to the vertex combination. The first triangle can be composed of vertices 1, 2, and 3 among the sorted vertices, and the second triangle can be composed of vertices 3, 4, and 1 among the sorted vertices.

[0126] Table 2-1 Triangles formed from vertices sorted 1,…,n

[0127] n triangle

[0128] 3(1,2,3)

[0129] 4(1,2,3),(3,4,1)

[0130] 5(1,2,3),(3,4,5),(5,1,3)

[0131] 6(1,2,3),(3,4,5),(5,6,1),(1,3,5)

[0132] 7(1,2,3),(3,4,5),(5,6,7),(7,1,3),(3,5,7)

[0133] 8(1,2,3),(3,4,5),(5,6,7),(7,8,1),(1,3,5),(5,7,1)

[0134] 9(1,2,3),(3,4,5),(5,6,7),(7,8,9),(9,1,3),(3,5,7),(7,9,3)

[0135] 10(1,2,3),(3,4,5),(5,6,7),(7,8,9),(9,10,1),(1,3,5),(5,7,9),(9,1,5)

[0136] 11(1,2,3),(3,4,5),(5,6,7),(7,8,9),(9,10,11),(11,1,3),(3,5,7),(7,9,11),(11,3,7)

[0137] 12(1,2,3),(3,4,5),(5,6,7),(7,8,9),(9,10,11),(11,12,1),(1,3,5),(5,7,9),(9,11,1),(1,5,9)

[0138] An upsampling process is performed to add points in the middle along the edges of the triangle, and voxelization is performed. The added points are generated based on the upsampling factor and the width of the block. The added points are called refined vertices. The point cloud encoder according to an embodiment may voxelize the refined vertices. In addition, the point cloud encoder may perform attribute encoding based on the voxelized position (or position value).

[0139] Figure 7 An example of a neighbor node pattern according to an embodiment is shown.

[0140] In order to increase the compression efficiency of the point cloud video, the point cloud encoder according to an embodiment may perform entropy coding based on context adaptive arithmetic coding.

[0141] As reference Figures 1 to 6 As described, the point cloud content providing system or point cloud encoder (eg, point cloud video encoder 10002, Figure 4 The point cloud encoder or arithmetic encoder 40004) may immediately perform entropy encoding on the occupancy code. In addition, the point cloud content providing system or the point cloud encoder may perform entropy encoding (intra-frame encoding) based on the occupancy code of the current node and the occupancy of the neighboring nodes, or perform entropy encoding (inter-frame encoding) based on the occupancy code of the previous frame. The frame representation according to the embodiment is a collection of point cloud videos generated simultaneously. The compression efficiency of the intra-frame coding / inter-frame coding according to the embodiment may depend on the number of neighboring nodes referenced. When the number of bits increases, the operation becomes complicated, but the encoding can be biased to one side, which can increase the compression efficiency. For example, when a 3-bit context is given, 2 bits need to be used. 3 = 8 ways to perform encoding. The division of parts for encoding affects the implementation complexity. Therefore, it is necessary to meet appropriate levels of compression efficiency and complexity.

[0142] Figure 7 The process of obtaining an occupancy pattern based on the occupancy of neighbor nodes is shown. The point cloud encoder according to the embodiment determines the occupancy of neighbor nodes of each node of the octree and obtains the value of the neighbor pattern. The neighbor node pattern is used to infer the occupancy pattern of the node. Figure 7 The left part of the diagram shows a cube corresponding to a node (the cube in the middle) and six cubes (neighboring nodes) that share at least one face with the cube. The nodes shown in the diagram are nodes at the same depth. The numbers shown in the diagram represent the weights (1, 2, 4, 8, 16, and 32) associated with the six nodes, respectively. The weights are assigned in sequence according to the positions of the neighboring nodes.

[0143] Figure 7 The right part of shows the neighbor node pattern value. The neighbor node pattern value is the sum of the values ​​multiplied by the weights of the occupied neighbor nodes (neighbor nodes with points). Therefore, the neighbor node pattern value is 0 to 63. When the neighbor node pattern value is 0, it indicates that there is no node with a point among the neighbor nodes of the node (no occupied node). When the neighbor node pattern value is 63, it indicates that all neighbor nodes are occupied nodes. As shown in the figure, since the neighbor nodes assigned weights 1, 2, 4 and 8 are occupied nodes, the neighbor node pattern value is 15 (the sum of 1, 2, 4 and 8). The point cloud encoder can perform encoding according to the neighbor node pattern value (for example, when the neighbor node pattern value is 63, 64 types of encoding can be performed). According to an embodiment, the point cloud encoder can reduce the encoding complexity by changing the neighbor node pattern value (for example, based on a table that changes 64 to 10 or 6).

[0144] Figure 8 An example of point configuration in each LOD according to an embodiment is shown.

[0145] As reference Figures 1 to 7 As described, the encoded geometry is reconstructed (decompressed) before performing attribute encoding. When direct encoding is applied, the geometry reconstruction operation may include changing the placement of the directly encoded points (e.g., placing the directly encoded points in front of the point cloud data). When triplet geometry encoding is applied, the geometry reconstruction process is performed by triangle reconstruction, upsampling, and voxelization. Since the attributes depend on the geometry, attribute encoding is performed based on the reconstructed geometry.

[0146] The point cloud encoder (e.g., LOD generator 40009) can classify (or reorganize) the points by LOD. The figure shows the point cloud content corresponding to the LOD. The leftmost screen in the figure shows the original point cloud content. The second screen from the left in the figure shows the distribution of points in the lowest LOD, and the rightmost screen in the figure shows the distribution of points in the highest LOD. That is, the points in the lowest LOD are sparsely distributed, and the points in the highest LOD are densely distributed. That is, as the LOD increases in the direction indicated by the arrow indicated at the bottom of the figure, the space (or distance) between the points becomes narrower.

[0147] Fig. 9 An example of point configuration for each LOD is shown according to an embodiment.

[0148] As reference Figures 1 to 8 As described, the point cloud content providing system or point cloud encoder (eg, point cloud video encoder 10002, Figure 4The point cloud encoder or LOD generator 40009) can generate LOD. LOD is generated by reorganizing points into a set of refinement levels according to the set LOD distance value (or Euclidean distance set). The LOD generation process is performed not only by the point cloud encoder, but also by the point cloud decoder.

[0149] Fig. 9 The upper part of shows examples of points (P0 to P9) of point cloud content distributed in 3D space. Fig. 9 In , the original order represents the order of points P0 to P9 before LOD generation. Fig. 9 In the LOD-based order, the order of points generated according to the LOD is shown. Points are reorganized by LOD. In addition, a high LOD contains points belonging to a lower LOD. Fig. 9 As shown, LOD0 contains P0, P5, P4, and P2. LOD1 contains the points of LOD0, P1, P6, and P3. LOD2 contains the points of LOD0, the points of LOD1, P9, P8, and P7.

[0150] As reference Figure 4 As described, the point cloud encoder according to the embodiment may selectively or in combination perform prediction transform coding, lifting transform coding, and RAHT transform coding.

[0151] The point cloud encoder according to the embodiment may generate a predictor for a point to perform predictive transform coding for setting a prediction attribute (or a prediction attribute value) of each point. That is, N predictors may be generated for N points. The predictor according to the embodiment may calculate a weight (=1 / distance) based on the LOD value of each point, index information about neighboring points that exist within a set distance of each LOD, and the distance to the neighboring point.

[0152] The predicted attribute (or attribute value) according to the embodiment is set to the average of the values ​​obtained by multiplying the attribute (or attribute value) (e.g., color, reflectivity, etc.) of the neighboring points set in the predictor of each point by the weight (or weight value) calculated based on the distance to each neighboring point. The point cloud encoder (e.g., coefficient quantizer 40011) according to the embodiment may quantize and inversely quantize the residual (which may be referred to as residual attribute, residual attribute value, or attribute prediction residual) obtained by subtracting the predicted attribute (attribute value) from the attribute (attribute value) of each point. The quantization process is configured as shown in the following table.

[0153] Table attribute prediction residual quantization pseudo code

[0154] int PCCQuantization(int value,int quantStep){

[0155] if(value>=0){

[0156] return floor(value / quantStep+1.0 / 3.0);

[0157] }else{

[0158] return-floor(-value / quantStep+1.0 / 3.0);

[0159] }

[0160] }

[0161] Table attribute prediction residual inverse quantization pseudo code

[0162] int PCCInverseQuantization(int value,int quantStep){

[0163] if(quantStep==0){

[0164] return value;

[0165] }else{

[0166] return value*quantStep;

[0167] }

[0168] }

[0169] When the predictor of each point has a neighboring point, the point cloud encoder (e.g., the arithmetic encoder 40012) according to the embodiment may perform entropy encoding on the quantized and inverse quantized residual values ​​as described above. When the predictor of each point has no neighboring point, the point cloud encoder (e.g., the arithmetic encoder 40012) according to the embodiment may perform entropy encoding on the attributes of the corresponding point without performing the above operation.

[0170] The point cloud encoder (e.g., lifting transformer 40010) according to an embodiment may generate a predictor for each point, set the calculated LOD and register neighbor points in the predictor, and set weights according to the distance to the neighbor points to perform lifting transform encoding. The lifting transform encoding according to an embodiment is similar to the above-mentioned predictive transform encoding, but is different in that weights are cumulatively applied to attribute values. The process of cumulatively applying weights to attribute values ​​according to an embodiment is configured as follows.

[0171] 1) Create an array quantized weight (QW) for storing weight values ​​of each point. The initial value of all elements of QW is 1.0. Multiply the QW value of the predictor index of the neighbor node registered in the predictor by the weight of the predictor of the current point, and add the values ​​obtained by multiplication.

[0172] 2) Boosting prediction process: The value obtained by multiplying the attribute value of the point by the weight is subtracted from the existing attribute value to calculate the predicted attribute value.

[0173] 3) Create temporary arrays called updateweight and update, and initialize the temporary arrays to zero.

[0174] 4) Accumulate the weight calculated by multiplying the weight calculated for all predictors by the weight stored in the QW corresponding to the predictor index with the updateweight array as the index of the neighbor node. Accumulate the value obtained by multiplying the attribute value of the neighbor node index by the calculated weight with the update array.

[0175] 5) Boosting update process: Divide the attribute values ​​of the update array of all predictors by the weight value of the updateweight array of the predictor index, and add the existing attribute value to the value obtained by the division.

[0176] 6) For all predictors, the predicted attribute is calculated by multiplying the attribute value updated by the lifting update process by the weight updated by the lifting prediction process (stored in QW). The point cloud encoder (e.g., coefficient quantizer 40011) according to the embodiment quantizes the predicted attribute value. In addition, the point cloud encoder (e.g., arithmetic encoder 40012) performs entropy encoding on the quantized attribute value.

[0177] A point cloud encoder according to an embodiment (e.g., RAHT transformer 40008) may perform RAHT transform coding, in which attributes associated with nodes at a lower level in an octree are used to predict attributes of nodes at a higher level. RAHT transform coding is an example of attribute intra-frame coding by scanning backward through an octree. A point cloud encoder according to an embodiment scans the entire area starting from voxels and repeats a merging process of merging voxels into larger blocks at each step until the root node is reached. The merging process according to an embodiment is performed only on occupied nodes. The merging process is not performed on empty nodes. The merging process is performed on the upper node directly above the empty node.

[0178] The following formula represents the RAHT transformation matrix. In this formula, Represents the average attribute value of voxels at level l. Can be based on and to calculate. and The weight of and

[0179]

[0180] here, Is the low-pass value and is used during the next highest level of merging. Denotes the high-pass coefficient. The high-pass coefficient of each step is quantized and undergoes entropy coding (eg, encoded by arithmetic encoder 400012). The weight is calculated as pass and Create the root node as follows.

[0181]

[0182] Fig.10 A point cloud decoder according to an embodiment is shown.

[0183] Fig.10 The point cloud decoder shown is Figure 1 An example of a point cloud video decoder 10006 described in Figure 1 The operations of the point cloud video decoder 10006 shown are the same or similar operations. As shown, the point cloud decoder can receive a geometry bitstream and an attribute bitstream contained in one or more bitstreams. The point cloud decoder includes a geometry decoder and an attribute decoder. The geometry decoder performs geometry decoding on the geometry bitstream and outputs the decoded geometry. The attribute decoder performs attribute decoding based on the decoded geometry and attribute bitstreams, and outputs the decoded attributes. The decoded geometry and the decoded attributes are used to reconstruct the point cloud content (decoded point cloud).

[0184] Fig.11 A point cloud decoder according to an embodiment is shown.

[0185] Fig.11 The point cloud decoder shown is Fig.10 An example of a point cloud decoder is shown, and a decoding operation can be performed, which is Figures 1 to 9 The inverse process of the encoding operation of the point cloud encoder shown.

[0186] As reference Figure 1 and Fig.10 As described, the point cloud decoder can perform geometry decoding and attribute decoding. Geometry decoding is performed before attribute decoding.

[0187] According to an embodiment, a point cloud decoder includes an arithmetic decoder (arithmetic decoding) 11000, an octree synthesizer (synthesized octree) 11001, a surface approximation synthesizer (synthesized surface approximation) 11002 and a geometry reconstructor (reconstructed geometry) 11003, an inverse coordinate transformer (inverse transformed coordinates) 11004, an arithmetic decoder (arithmetic decoding) 11005, an inverse quantizer (inverse quantization) 11006, a RAHT transformer 11007, an LOD generator (generate LOD) 11008, an inverse lifting (inverse lifting) 11009 and / or an inverse color transformer (inverse transformed color) 11010.

[0188] The arithmetic decoder 11000, the octree synthesizer 11001, the surface approximation synthesizer 11002, the geometry reconstructor 11003, and the coordinate inverse transformer 11004 may perform geometry decoding. The geometry decoding according to the embodiment may include direct encoding and triplet geometry decoding. Direct encoding and triplet geometry decoding are selectively applied. The geometry decoding is not limited to the above examples, and as a reference Figures 1 to 9 The inverse process of the geometric encoding described is performed.

[0189] The arithmetic decoder 11000 according to the embodiment decodes the received geometry bitstream based on arithmetic coding. The operation of the arithmetic decoder 11000 corresponds to the inverse process of the arithmetic encoder 40004.

[0190] The octree synthesizer 11001 according to the embodiment may generate an octree by acquiring an occupancy code (or information about the geometry obtained as a result of decoding) from the decoded geometry bitstream. Figures 1 to 9 Describe the configuration in detail.

[0191] When triplet geometry encoding is applied, a surface approximation synthesizer 11002 according to an embodiment may synthesize a surface based on the decoded geometry and / or the generated octree.

[0192] According to an embodiment, the geometry reconstructor 11003 can regenerate the geometry based on the surface and / or the decoded geometry. Figures 1 to 9 As described, direct coding and triplet geometry coding are selectively applied. Therefore, the geometry reconstructor 11003 directly imports the position information about the points to which direct coding is applied and adds them. When triplet geometry coding is applied, the geometry reconstructor 11003 can reconstruct the geometry by performing the reconstruction operations (e.g., triangle reconstruction, upsampling, and voxelization) of the geometry reconstructor 40005. Details and References Figure 6 Those described are the same, so their description is omitted. The reconstructed geometry may include a point cloud screen or frame that does not contain attributes.

[0193] The coordinate inverse transformer 11004 according to an embodiment may acquire a point position by transforming the coordinates based on the reconstructed geometry.

[0194] The arithmetic decoder 11005, the inverse quantizer 11006, the RAHT transformer 11007, the LOD generator 11008, the inverse lifter 11009 and / or the color inverse transformer 11010 may perform reference Fig.10 Attribute decoding described. Attribute decoding according to an embodiment includes regional adaptive hierarchical transform (RAHT) decoding, hierarchical nearest neighbor prediction (prediction transform) decoding based on interpolation, and hierarchical nearest neighbor prediction (lifting transform) decoding based on interpolation with an update / lifting step. The above three decoding schemes can be used selectively, or a combination of one or more decoding schemes can be used. Attribute decoding according to an embodiment is not limited to the above examples.

[0195] The arithmetic decoder 11005 according to the embodiment decodes the attribute bit stream through arithmetic coding.

[0196] The inverse quantizer 11006 according to the embodiment inversely quantizes information about the decoded attribute bitstream or the attribute obtained as a result of decoding, and outputs the inversely quantized attribute (or attribute value). The inverse quantization may be selectively applied based on the attribute encoding of the point cloud encoder.

[0197] According to an embodiment, the RAHT transformer 11007, the LOD generator 11008 and / or the inverse lifter 11009 may process the reconstructed geometry and inverse quantized attributes. As described above, the RAHT transformer 11007, the LOD generator 11008 and / or the inverse lifter 11009 may selectively perform a decoding operation corresponding to the encoding of the point cloud encoder.

[0198] The color inverse transformer 11010 according to an embodiment performs inverse transform encoding to inversely transform a color value (or texture) included in the decoded attribute. The operation of the color inverse transformer 11010 may be selectively performed based on the operation of the color transformer 40006 of the point cloud encoder.

[0199] Although not shown in the figure, Fig.11 The elements of the point cloud decoder may be implemented by hardware, software, firmware, or a combination thereof including one or more processors or integrated circuits configured to communicate with one or more memories included in the point cloud providing device. The one or more processors may execute the above Fig.11 At least one or more of the operations and / or functions of the elements of the point cloud decoder. In addition, one or more processors may be operable or executed to perform Fig.11 A software program and / or instruction set for the operation and / or functionality of the elements of a point cloud decoder.

[0200] Fig.12 A transmitting device according to an embodiment is shown.

[0201] Fig.12 The sending device shown is Figure 1 The sending device 10000 (or Figure 4 Example of a point cloud encoder). Fig.12 The sending device shown can be executed with reference to Figures 1 to 9 The transmitting device according to the embodiment may include a data input unit 12000, a quantization processor 12001, a voxelization processor 12002, an octree occupancy code generator 12003, a surface model processor 12004, an intra-frame / inter-frame coding processor 12005, an arithmetic encoder 12006, a metadata processor 12007, a color transformation processor 12008, an attribute transformation processor 12009, a prediction / lifting / RAHT transformation processor 12010, an arithmetic encoder 12011 and / or a transmitting processor 12012.

[0202] The data input unit 12000 according to the embodiment receives or acquires point cloud data. The data input unit 12000 may perform the same operation and / or acquisition method as the point cloud video acquirer 10001 (or refer to Figure 2 The acquisition process 20000) described above has the same or similar operations and / or acquisition methods.

[0203] The data input unit 12000, the quantization processor 12001, the voxelization processor 12002, the octree occupancy code generator 12003, the surface model processor 12004, the intra / inter coding processor 12005 and the arithmetic encoder 12006 perform geometric coding. Figures 1 to 9 The geometrical encodings described are the same or similar, so their detailed description is omitted.

[0204] According to an embodiment, the quantization processor 12001 quantizes geometry (e.g., position values ​​of points). The operation and / or quantization of the quantization processor 12001 is similar to the reference Figure 4 The operation and / or quantization of the quantizer 40001 described above are the same or similar. Figures 1 to 9 Same as those described.

[0205] According to an embodiment, the voxelization processor 12002 voxelizes the quantized position value of the point. The voxelization processor 12002 may perform the same as the reference Figure 4 The operation and / or voxelization process of the quantizer 40001 described above is the same or similar to the operation and / or process described above. Figures 1 to 9 Same as those described.

[0206] The octree occupancy code generator 12003 according to the embodiment performs octree encoding on the voxelized position of the point based on the octree structure. The octree occupancy code generator 12003 may generate an occupancy code. The octree occupancy code generator 12003 may perform the same as the reference code. Figure 4 and Figure 6 The operations and / or methods of the point cloud encoder (or octree analyzer 40002) described above are the same or similar to the operations and / or methods. Figures 1 to 9 Same as those described.

[0207] According to an embodiment, the surface model processor 12004 may perform triplet geometry encoding based on the surface model to reconstruct the point position in a specific area (or node) based on voxels. Figure 4 The operations and / or methods described herein are the same or similar to the operations and / or methods of the point cloud encoder (e.g., surface approximation analyzer 40003). Figures 1 to 9 Same as those described.

[0208] According to the embodiment, the intra-frame / inter-frame encoding processor 12005 may perform intra-frame / inter-frame encoding on the point cloud data. The intra-frame / inter-frame encoding processor 12005 may perform the same as the reference. Figure 7 The intra / inter coding described is the same or similar to the coding. Details and references Figure 7 The intra / inter encoding processor 12005 may be included in the arithmetic encoder 12006 according to an embodiment.

[0209] According to an embodiment, the arithmetic encoder 12006 performs entropy coding on the octree and / or approximate octree of the point cloud data. For example, the coding scheme includes arithmetic coding. The arithmetic encoder 12006 performs the same or similar operations and / or methods as the operations and / or methods of the arithmetic encoder 40004.

[0210] The metadata processor 12007 according to an embodiment processes metadata (e.g., set values) about the point cloud data and provides it to necessary processing processes such as geometry coding and / or attribute coding. In addition, the metadata processor 12007 according to an embodiment may generate and / or process signaling information related to geometry coding and / or attribute coding. The signaling information according to an embodiment may be encoded separately from the geometry coding and / or attribute coding. The signaling information according to an embodiment may be interleaved.

[0211] The color conversion processor 12008, the attribute conversion processor 12009, the prediction / lifting / RAHT conversion processor 12010, and the arithmetic encoder 12011 perform attribute encoding. Figures 1 to 9The attribute codes described are the same or similar, so their detailed description is omitted.

[0212] The color transform processor 12008 according to an embodiment performs color transform encoding to transform the color value included in the attribute. The color transform processor 12008 may perform color transform encoding based on the reconstructed geometry. The reconstructed geometry is compared with the reference Figures 1 to 9 In addition, its execution is the same as that of reference Figure 4 The operations and / or methods of the color converter 40006 described above are the same as or similar to the operations and / or methods described above, and their detailed description is omitted.

[0213] The attribute transformation processor 12009 according to an embodiment performs attribute transformation to transform attributes based on reconstructed geometry and / or locations where geometry encoding is not performed. The attribute transformation processor 12009 performs the attribute transformation with reference to Figure 4 The operations and / or methods of the attribute converter 40007 described in the foregoing description are the same as or similar to those of the attribute converter 40007. Detailed description thereof is omitted. The prediction / lifting / RAHT transform processor 12010 according to the embodiment may encode the transformed attribute by any one or a combination of RAHT coding, prediction transform coding, and lifting transform coding. The prediction / lifting / RAHT transform processor 12010 performs the same as the reference. Figure 4 The operations of the RAHT transformer 40008, the LOD generator 40009 and the lifting transformer 40010 described above are the same or similar to at least one operation. In addition, the prediction transform coding, the lifting transform coding and the RAHT transform coding are the same as those of the reference Figures 1 to 9 Those described are the same, so their detailed description is omitted.

[0214] The arithmetic encoder 12011 according to the embodiment may encode the encoded attribute based on arithmetic coding. The arithmetic encoder 12011 performs the same or similar operation and / or method as that of the arithmetic encoder 400012.

[0215] The sending processor 12012 according to the embodiment may send individual bitstreams containing encoded geometry and / or encoded attributes and metadata information, or send a bitstream configured with encoded geometry and / or encoded attributes and metadata information. When the encoded geometry and / or encoded attributes and metadata information according to the embodiment are configured into a bitstream, the bitstream may include one or more sub-bitstreams. The bitstream according to the embodiment may include signaling information and slice data, and the signaling information includes a sequence parameter set (SPS) for sequence level signaling, a geometry parameter set (GPS) for geometry information coding signaling, an attribute parameter set (APS) for attribute information coding signaling, and a patch parameter set (TPS) for patch level signaling. Slice data may include information about one or more slices. A slice according to the embodiment may include a geometry bitstream Geom00 and one or more attribute bitstreams Attr00 and Attr10.

[0216] A slice is a sequence of syntactic elements that represents all or part of a coded point cloud frame.

[0217] The TPS according to an embodiment may include information about each of the one or more tiles (e.g., coordinate information and height / size information about a bounding box). The geometry bitstream may include a header and a payload. The header of the geometry bitstream according to an embodiment may include a parameter set identifier (geom_parameter_set_id), a tile identifier (geom_tile_id), and a slice identifier (geom_slice_id) included in the GPS, as well as information about the data contained in the payload. As described above, the metadata processor 12007 according to an embodiment may generate and / or process signaling information and send it to the transmission processor 12012. According to an embodiment, an element that performs geometry encoding and an element that performs attribute encoding may share data / information with each other as indicated by the dotted lines. The transmission processor 12012 according to an embodiment may perform operations and / or transmission methods that are the same or similar to those of the transmitter 10003. Details and References Figure 1 and Figure 2 Those described are the same, so their description is omitted.

[0218] Fig.13 A receiving device according to an embodiment is shown.

[0219] Fig.13 The receiving device shown is Figure 1 The receiving device 10004 (or Fig.10 and Fig.11 Example of a point cloud decoder. Fig.13 The receiving device shown can execute and refer to Figures 1 to 11One or more operations and methods that are the same or similar to those of the described point cloud decoder.

[0220] The receiving device according to the embodiment may include a receiver 13000, a receiving processor 13001, an arithmetic decoder 13002, an octtree reconstruction processor based on an occupancy code 13003, a surface model processor (triangle reconstruction, upsampling, voxelization) 13004, an inverse quantization processor 13005, a metadata parser 13006, an arithmetic decoder 13007, an inverse quantization processor 13008, a prediction / lifting / RAHT inverse transform processor 13009, a color inverse transform processor 13010 and / or a renderer 13011. Each decoding element according to the embodiment may perform an inverse process of the operation of the corresponding encoding element according to the embodiment.

[0221] The receiver 13000 according to the embodiment receives point cloud data. The receiver 13000 may perform the same Figure 1 The operation and / or receiving method of the receiver 10005 is the same as or similar to the operation and / or receiving method of the receiver 10005. The detailed description thereof is omitted.

[0222] The reception processor 13001 according to an embodiment may obtain a geometry bitstream and / or an attribute bitstream from the received data. The reception processor 13001 may be included in the receiver 13000.

[0223] The arithmetic decoder 13002, the octtree reconstruction processor 13003 based on the occupancy code, the surface model processor 13004, and the inverse quantization processor 13005 may perform geometric decoding. Figures 1 to 10 The geometric decoding described is the same or similar, so its detailed description is omitted.

[0224] The arithmetic decoder 13002 according to the embodiment may decode the geometry bitstream based on arithmetic coding. The arithmetic decoder 13002 performs the same or similar operations and / or encoding as those of the arithmetic decoder 11000.

[0225] The octree reconstruction processor 13003 based on the occupancy code according to the embodiment can reconstruct the octree by obtaining the occupancy code from the decoded geometry bitstream (or information about the geometry obtained as a result of decoding). The octree reconstruction processor 13003 based on the occupancy code performs the same or similar operations and / or methods as the operations of the octree synthesizer 11001 and / or the octree generation method. When triplet geometry coding is applied, the surface model processor 13004 according to the embodiment can perform triplet geometry decoding and related geometry reconstruction (e.g., triangle reconstruction, upsampling, voxelization) based on the surface model method. The surface model processor 13004 performs the same or similar operations as the operations of the surface approximation synthesizer 11002 and / or the geometry reconstructor 11003.

[0226] The inverse quantization processor 13005 according to an embodiment may inversely quantize the decoded geometry.

[0227] According to an embodiment, the metadata parser 13006 may parse metadata (eg, setting values) contained in the received point cloud data. The metadata parser 13006 may pass the metadata to the geometry decoding and / or attribute decoding. Fig.12 The metadata described is the same, so a detailed description thereof is omitted.

[0228] The arithmetic decoder 13007, the inverse quantization processor 13008, the prediction / lifting / RAHT inverse transform processor 13009 and the color inverse transform processor 13010 perform attribute decoding. Figures 1 to 10 The attribute decoding described is the same or similar, so the detailed description thereof is omitted.

[0229] The arithmetic decoder 13007 according to the embodiment may decode the attribute bitstream by arithmetic coding. The arithmetic decoder 13007 may decode the attribute bitstream based on the reconstructed geometry. The arithmetic decoder 13007 performs the same or similar operation and / or encoding as that of the arithmetic decoder 11005.

[0230] The inverse quantization processor 13008 according to an embodiment may inversely quantize the decoded attribute bitstream. The inverse quantization processor 13008 performs the same or similar operation and / or method as the inverse quantizer 11006 and / or the inverse quantization method.

[0231] The prediction / lifting / RAHT inverse transform processor 13009 according to an embodiment may process the reconstructed geometry and inverse quantized attributes. The prediction / lifting / RAHT inverse transform processor 13009 performs one or more operations and / or decoding that are the same or similar to the operations and / or decoding of the RAHT transformer 11007, the LOD generator 11008 and / or the inverse lifter 11009. The color inverse transform processor 13010 according to an embodiment performs inverse transform encoding to inversely transform the color value (or texture) included in the decoded attribute. The color inverse transform processor 13010 performs the same or similar operations and / or inverse transform encoding as the operations and / or inverse transform encoding of the color inverse transformer 11010 and / or inverse transform encoding. The renderer 13011 according to an embodiment may render point cloud data.

[0232] Fig.14 An exemplary structure that can be operated in conjunction with a point cloud data transmitting / receiving method / apparatus according to an embodiment is shown.

[0233] Fig.14 The structure of represents a configuration in which at least one of the server 1460, the robot 1410, the self-driving vehicle 1420, the XR device 1430, the smart phone 1440, the home appliance 1450, and / or the head mounted display (HMD) 1470 is connected to the cloud network 1400. The robot 1410, the self-driving vehicle 1420, the XR device 1430, the smart phone 1440, or the home appliance 1450 is referred to as a device. In addition, the XR device 1430 may correspond to a point cloud data (PCC) device according to an embodiment or may be operably connected to a PCC device.

[0234] The cloud network 1400 may represent a network that constitutes a part of a cloud computing infrastructure or exists in a cloud computing infrastructure. Here, the cloud network 1400 may be configured using a 3G network, a 4G or Long Term Evolution (LTE) network, or a 5G network.

[0235] The server 1460 may be connected to at least one of the robot 1410 , the self-driving vehicle 1420 , the XR device 1430 , the smart phone 1440 , the home appliance 1450 , and / or the HMD 1470 via the cloud network 1400 , and may assist in at least a portion of the processing of the connected devices 1410 to 1470 .

[0236] The HMD 1470 represents one of implementation types of an XR device and / or a PCC device according to an embodiment. The HMD type device according to an embodiment includes a communication unit, a control unit, a memory, an I / O unit, a sensor unit, and a power supply unit.

[0237] Hereinafter, various embodiments of devices 1410 to 1450 to which the above-mentioned technology is applied will be described. Fig.14The apparatuses 1410 to 1450 shown can be operably connected / coupled to the point cloud data transmitting apparatus and the receiving apparatus according to the above-described embodiments.

[0238] <PCC+XR>

[0239] The XR / PCC apparatus 1430 can adopt PCC technology and / or XR (AR+VR) technology, and can be implemented as an HMD, a head-up display (HUD) provided in a vehicle, a television, a mobile phone, a smartphone, a computer, a wearable device, a household appliance, a digital signage, a vehicle, a stationary robot, or a mobile robot.

[0240] The XR / PCC apparatus 1430 can analyze 3D point cloud data or image data obtained through various sensors or from an external device and generate position data and attribute data regarding 3D points. Thus, the XR / PCC apparatus 1430 can obtain information about the surrounding space or real objects, and render and output XR objects. For example, the XR / PCC apparatus 1430 can match an XR object including auxiliary information about the identified object with the identified object and output the matched XR object.

[0241] <PCC+XR+mobile phone>。

[0242] The XR / PCC apparatus 1430 can be implemented as a mobile phone 1440 by applying PCC technology.

[0243] The mobile phone 1440 can decode and display point cloud content based on PCC technology.

[0244] <PCC+self-driving+XR>

[0245] The self-driving vehicle 1420 can be implemented as a mobile robot, a vehicle, an unmanned aerial vehicle, etc. by applying PCC technology and XR technology.

[0246] The self-driving vehicle 1420 applying XR / PCC technology can represent a self-driving vehicle provided with means for providing an XR image, or a self-driving vehicle that is a control / interaction target in an XR image. Specifically, as a control / interaction target in an XR image, the self-driving vehicle 1420 can be distinguished from the XR apparatus 1430 and can be operably connected thereto.

[0247] The self-driving vehicle 1420 having means for providing an XR / PCC image can obtain sensor information from a sensor including a camera, and output the generated XR / PCC image based on the obtained sensor information. For example, the self-driving vehicle 1420 can have an HUD and output the XR / PCC image thereto, thereby providing an XR / PCC object corresponding to a real object or an object presented on the screen to a passenger.

[0248] When the XR / PCC object is output to the HUD, at least a portion of the XR / PCC object may be output to overlap with the real object to which the passenger's eyes are directed. On the other hand, when the XR / PCC object is output on a display provided in the self-driving vehicle, at least a portion of the XR / PCC object may be output to overlap with the object on the screen. For example, the self-driving vehicle 1220 may output XR / PCC objects corresponding to objects such as a road, another vehicle, a traffic light, a traffic sign, a two-wheeled vehicle, a pedestrian, and a building.

[0249] Virtual reality (VR) technology, augmented reality (AR) technology, mixed reality (MR) technology, and / or point cloud compression (PCC) technology according to an embodiment are applicable to various devices.

[0250] In other words, VR technology is a display technology that only provides CG images of real-world objects, backgrounds, etc. On the other hand, AR technology refers to a technology that displays a virtually created CG image on an image of a real object. MR technology is similar to the above-mentioned AR technology in that the virtual objects to be displayed are mixed and combined with the real world. However, MR technology is different from AR technology in that AR technology clearly distinguishes between real objects and virtual objects created as CG images and uses virtual objects as supplementary objects to real objects, while MR technology treats virtual objects as objects with properties equivalent to real objects. More specifically, an example of the application of MR technology is a holographic service.

[0251] Recently, VR, AR, and MR technologies are often referred to as extended display (XR) technologies rather than being clearly distinguished from each other. Therefore, embodiments of the present disclosure are applicable to any of VR, AR, MR, and XR technologies. Encoding / decoding based on PCC, V-PCC, and G-PCC technologies are applicable to such technologies.

[0252] The PCC method / apparatus according to the embodiment may be applied to a vehicle that provides a self-driving service.

[0253] Vehicles providing self-driving services are connected to the PCC device for wired / wireless communication.

[0254] When a point cloud data (PCC) transmitting / receiving device according to an embodiment is connected to a vehicle for wired / wireless communication, the device may receive / process content data related to an AR / VR / PCC service (which may be provided together with a self-driving service) and send it to the vehicle. In the case where the PCC transmitting / receiving device is installed on the vehicle, the PCC transmitting / receiving device may receive / process content data related to the AR / VR / PCC service and provide it to the user based on a user input signal input through a user interface device. The vehicle or user interface device according to the embodiment may receive a user input signal. The user input signal according to the embodiment may include a signal indicating a self-driving service.

[0255] The point cloud data transmitting method / apparatus according to the embodiment may be interpreted as a term referring to the following items: Figure 1 The sending device 10000, Figure 1 Point cloud video encoder 10002, Figure 1 Transmitter 10003, Figure 2 Get 20000 / encode 20001 / send 20002, Figure 4 The encoder, Fig.12 The sending device Fig.14 Devices Figures 15 to 17 The encoding, Figures 18 to 27 Bitstream generation, Fig.28 The encoding, Fig.30 and Fig.31 The encoding, Figure 32 to Figure 35 bitstream and parameter generation, Fig.36 The encoding, Fig.38 The sending method, etc.

[0256] The point cloud data receiving method / apparatus according to the embodiment may be interpreted as a term referring to the following items: Figure 1 The receiving device 10004, Figure 1 Receiver 10005, Figure 1 Point cloud video decoder 10006, Figure 2 Send 20002 / decode 20003 / render 20004, Fig.10 and Fig.11 The decoder, Fig.13 The receiving device Fig.14 Devices Figures 15 to 17 Decoding, Figures 18 to 27 Bitstream parsing, Fig.29 Decoding, Fig.30 and Fig.31 Decoding, Figure 32 to Figure 35 Bitstream parsing, Fig.37 Decoding, Fig.39 receiving method, etc.

[0257] The method / apparatus for sending or receiving point cloud data according to an embodiment may be simply referred to as method / apparatus.

[0258] According to the embodiment, the geometric data, geometric information, position information and geometric structure constituting the point cloud data shall be interpreted as having the same meaning. The attribute data, attribute information and attribute constituting the point cloud data shall be interpreted as having the same meaning.

[0259] The method / apparatus according to an embodiment may include and perform a method of matching regions for inter-frame prediction of point cloud content (compact bounding box matching method for inter-frame prediction of point cloud).

[0260] In order to increase the compression efficiency of point cloud data having one or more frames, the method / apparatus according to the embodiment may use a compact bounding box to predict the position to predict the positional relationship of nodes in the prediction tree geometry using an inter-frame prediction technique.

[0261] The method / apparatus according to the embodiment can compress and reconstruct data composed of point clouds. Specifically, in order to effectively compress a point cloud having one or more frames, important regions can be extracted, and position matching between regions can be performed to predict prediction tree nodes between frames.

[0262] Fig.15 Inter-frame prediction (inter prediction) according to an embodiment is shown.

[0263] Point cloud data sending method / device according to an embodiment ( Figure 1 The sending device 10000, Figure 1 Point cloud video encoder 10002, Figure 1 Transmitter 10003, Figure 2 Get 20000 / encode 20001 / send 20002, Figure 4 The encoder, Fig.12 The sending device Fig.14 Devices Figures 15 to 17 The encoding, Figures 18 to 27 Bitstream generation, Fig.28 The encoding, Fig.30 and Fig.31 The encoding, Figure 32 to Figure 35 bitstream and parameter generation, Fig.36 The encoding and Fig.38 The sending method can be based on Fig.15 The inter-frame prediction shown is used to encode the point cloud data.

[0264] Point cloud data receiving method / device according to an embodiment ( Figure 1 The receiving device 10004, Figure 1 Receiver 10005, Figure 1 Point cloud video decoder 10006, Figure 2 Send 20002 / decode 20003 / render 20004, Fig.10 and Fig.11 The decoder, Fig.13 The receiving device Fig.14 Devices Figures 15 to 17 Decoding, Figures 18 to 27 Bitstream parsing, Fig.29 Decoding, Fig.30 and Fig.31 Decoding, Figure 32 to Figure 35 Bitstream parsing, Fig.37 Decoding and Fig.39 The receiving method) can be based on Fig.15 Inter-frame prediction is used to decode point cloud data.

[0265] The inter-frame prediction in the prediction tree structure takes an immediately preceding coded frame as a reference frame, searches for a point whose azimuth is most similar to that of a previously decoded point in the current frame and whose laser ID is located at the same position in the reference frame, and uses the next closest point or the closest point among the points whose azimuth is greater than that point as the predictor of the current point. The case of using the closest point as the predictor and the case of using the next closest point as the predictor can be distinguished by a flag that will be signaled. In the case of inter-frame prediction, point information to be used as the position of the current point is determined, and the corresponding predictor information is delivered to the receiver.

[0266] Reference Fig.15 , when encoding or decoding the current frame containing a point cloud acquired by a camera and / or sensor, points acquired in increasing directions of laser ID and azimuth according to the laser ID may be located in the current frame. When encoding (or decoding) the current point, the point has been previously decoded, and therefore a prediction value may be sought in the reference frame of the current point to predict the current point. When a point in the reference frame whose data is most similar to the geometric data and / or attribute data of the current point is found, it is not directly encoded as the value of the current point. Instead, the residual between the predicted value and the current value may be encoded and sent, and the receiver may reconstruct the original point value by summing the residual and the predicted value in the same manner. For accurate and fast prediction, a reference point having an azimuth and laser ID that is the same as or most similar to the azimuth and laser ID of a point located before the current point may be searched in the reference frame. Since the reference frame also contains the same or similar points as the current frame, a point whose azimuth is greater than and closest to the azimuth of the reference point (i.e., the closest azimuth) can be set as the predictor of the current point, or a point whose azimuth is second closest to the azimuth of the reference frame can be set as the predictor of the current point. In other words, the geometric data and / or attribute data of the predictor can be the most similar to the geometric data and / or attribute data of the current point.

[0267] When looking for a point to reference in a reference frame to predict the current point in the current frame, the point is searched in a predefined list of mapping points that simply has the first point encoded for each azimuth and laser ID as each element of the list. For the azimuth, the actual azimuth value is not used as is. Instead, they are grouped into quantized values ​​by range so that points with different detailed azimuth values ​​are grouped into the same quantized azimuth. This scheme has the following disadvantages: because points that are significantly different from previously decoded points in the current frame are grouped together due to azimuth quantization, the residual becomes larger after the prediction that depends on the position. In addition, since this method that only considers inter-point prediction is applied, positions belonging to other objects may be predicted only because they are nearby points, even if each object has different local motion in content. Predictive geometry coding is point-by-point prediction, and there is no method for object-level or block-level prediction.

[0268] Therefore, the method / apparatus according to an embodiment includes a method of using a compact bounding box (compact bbox) representing an object area to match an object compact bbox at a highly correlated location in order to efficiently perform inter-frame prediction of the prediction geometry.

[0269] The method / apparatus according to an embodiment may define a region divided on an object-by-object basis in the form of a compact frame.

[0270] The method / apparatus according to an embodiment may signal inter-frame correlation region information after defining the region divided on an object-by-object basis in the form of a minimum frame.

[0271] The method / apparatus according to an embodiment may find and define each compact bounding box after dividing the point cloud into PUs.

[0272] The method / apparatus according to an embodiment may signal matching criteria based on coordinate characteristics of a box defining an area of ​​an object for classification and matching.

[0273] Fig.16 A compact bounding box is shown according to an embodiment.

[0274] Point cloud data sending method / device according to an embodiment ( Figure 1 The sending device 10000, Figure 1 Point cloud video encoder 10002, Figure 1 Transmitter 10003, Figure 2 Get 20000 / encode 20001 / send 20002, Figure 4 The encoder, Fig.12 The sending device Fig.14 Devices Figures 15 to 17 The encoding, Figures 18 to 27 Bitstream generation, Fig.28 The encoding, Fig.30 and Fig.31 The encoding, Figure 32 to Figure 35 bitstream and parameter generation, Fig.36 The encoding and Fig.38 The sending method) can generate Fig.16 The (compact) bounding box shown is used, and the point cloud data is encoded based on the bounding box.

[0275] Point cloud data receiving method / device according to an embodiment ( Figure 1 The receiving device 10004, Figure 1 Receiver 10005, Figure 1 Point cloud video decoder 10006, Figure 2 Send 20002 / decode 20003 / render 20004, Fig.10 and Fig.11 The decoder, Fig.13 The receiving device Fig.14 Devices Figures 15 to 17 Decoding, Figures 18 to 27 Bitstream parsing, Fig.29 Decoding, Fig.30 and Fig.31 Decoding, Figure 32 to Figure 35 Bitstream parsing, Fig.37 Decoding and Fig.39 The receiving method) can generate Fig.16 The (compact) bounding box shown, and the point cloud data can be decoded based on the bounding box.

[0276] A compact bounding box for connecting object cells can be generated, and a list can be configured. The compact bounding box can be simply referred to as a bounding box.

[0277] In order to define the core area of ​​each frame or the area of ​​each object, any method for classifying the points constituting each object can be used. The result of one of the HDBSCAN, DBSCAN, kd tree and GMM as clustering methods can be delivered to set the area of ​​each object. Each object can be extracted based on the relationship of the distance between the points calculated when building the prediction geometry tree. The area of ​​the compact bounding box of each object found by the correlation between the points can be defined by setting the maximum value of the bounding box by collecting the maximum value along each axis of the three dimensions and setting the minimum value of the bounding box by collecting the minimum value along each axis.

[0278] For example, Fig.16As shown, for a compact bounding box of an area classified as an intermediate object, a minimum bounding box value (BboxMin) and a maximum bounding box value (BboxMax) which are the maximum and minimum values ​​on each axis of the position information classified as the same object can be found, and the coordinates of the rest of the box can be found by a combination of the two points. The compact bounding box found for each unit separated by an object in the point cloud can be as follows Fig.16 As shown.

[0279] Once the area of ​​each object is defined, a list of compact bboxes can be generated by transforming the list into spherical coordinates and sorting the coordinates based on the radius of the maximum bounding box value (BboxMax) or the minimum bounding box value (BboxMin) present in the same frame. When necessary, the Cartesian coordinates can be treated as is and sorted based on a specific axis. In the case where the frame is divided into prediction units (PUs) and a compact bbox is defined for each object contained in each PU, a list of compact bboxes for the frame can be generated by sorting the compact bboxes within the PU according to a specific criterion after each PU is sorted.

[0280] The two coordinates (min, max) of the compact bounding box (compact bbox) found for each object can be tied together to create a sortable form.

[0281] Reference Fig.16 , the points that make up the objects in the frame can be found based on the characteristics of the points. When generating a compact bounding box for an object unit, a compact bounding box can be generated based on the maximum and minimum values ​​along each of the x-axis, y-axis, and z-axis. Objects can be distinguished, a compact bounding box can be generated for each object, and a list of compact bounding boxes can be sorted according to specific criteria. The coordinates can be transformed into spherical coordinates to be sorted by radius. Sorting can be performed based on a specific axis in a Cartesian coordinate system without transforming the coordinates. In the case of applying predictive coding, objects within a prediction unit can be distinguished, and compact bounding boxes can be generated and sorted.

[0282] Fig.17 A method of connecting compact bounding boxes according to an embodiment is shown.

[0283] The method / apparatus according to the embodiment can be as follows Fig.16 A compact bounding box is generated as shown, and Fig.17 Connect the compact bounding boxes as shown.

[0284] The per-object compact bounding boxes between frames can be connected as follows.

[0285] The compact bbox list of objects generated in each frame can be compared with the reference value set by the system or user to generate a new compact bbox matching list that can match the information about the compact bbox with the closest value. If there is a list of compact bboxes that can be defined as objects in the current frame and the reference frame, the closest value in terms of a specific reference (e.g., the radius in the coordinates where the bbox is the smallest) can be found in the object compact bbox list of the reference frame for each object in the current frame. Then the value closest to the minimum coordinate value of the bounding box according to the radius can be entered. The reference value for comparison can be changed, and the object compact bbox of the reference frame with the closest difference in terms of radius, azimuth, laser ID or elevation can be selected and added to the matching list. If necessary, the list can be sorted based on a specific axis of Cartesian coordinates, the closest value based on the distance difference between the values ​​of BboxMin (Euclidean, Manhattan, etc.), the closest value based on the distance difference between the values ​​of BboxMax, the closest value along a specific axis (X, Y or Z), or the closest value between the Min or Max that can define the box after projection can be selected. Information about the object compact bbox in the selected reference frame is removed from the list to avoid duplication, and the object compact bbox list of the remaining reference frames can be used to find the matching compact bbox information of the next object compact bbox in the current frame. When the difference with the reference value is greater than a predetermined threshold, it is assumed that there is no matching compact bbox information, and information about the corresponding object compact bbox may not be input, or (0,0,0) information may be input to indicate that there is no matching compact bbox information. A matching list may be generated so that its size is equal to the number of object lists in the current frame.

[0286] Matching regions can be used for prediction in geometry coding and attribute coding. These regions can also be applied to octree geometry coding by implementing block-by-block prediction instead of point-by-point prediction of predictive geometry. In other words, geometric inter-frame prediction can be performed based on the prediction tree and / or the octree (occupancy tree).

[0287] Reference Fig.17, the current frame contains n objects, and a bounding box can be generated for each object based on a bounding box maximum value and a bounding box minimum value by distinguishing the objects included in the frame. A list of n bounding boxes can be generated. Similarly, the reference frame of the current frame can contain m objects and have a list of m bounding boxes of objects. A matching list can be generated by comparing each bounding box in the current frame with each bounding box in the reference frame and finding a bounding box in the reference frame that is close to each bounding box in the current frame. Comparing the bounding boxes of objects between the two frames indicates that there are similar areas that are close to each other and point cloud data exists in these areas. Therefore, when predicting and encoding the current frame, the matching list can be used to perform efficient inter-frame prediction. As described above, there may be various criteria for finding a bounding box (area) that is most similar to the current area in the reference frame. For example, in the minimum coordinate value of the bounding box, a value closest to the bounding box of the current frame based on the radius can be found, and the bounding box with the closest radius in the maximum coordinate value of the bounding box can be added to the matching list. In addition to the radius, a bounding box in the reference frame having at least one of the azimuth, laser ID, and elevation values ​​most similar to the bounding box in the current frame can be found and registered in the matching list. A matching list can be generated by sorting the bounding boxes based on a Cartesian coordinate system rather than a specific axis in a spherical coordinate system and selecting the bounding box with the smallest distance difference between the minimum coordinates of the bounding boxes. A matching list can be generated by selecting the bounding box with the smallest distance difference between the maximum coordinates of the bounding boxes. A matching list can be generated by selecting the bounding box with the closest value on any one of the x, y, or z axes. A matching list can be generated by projecting data and selecting the bounding box with the closest maximum value or the closest minimum value of the projected frame. The bounding box information selected from the reference frame of the current frame can be removed from the list to avoid duplication. When a box similar to the bounding box of the first object in the current frame is found in the reference frame, the bounding box in the reference frame can be removed from the list of objects in the reference frame and registered in the matching list. A box similar to the bounding box of the second object in the current frame can be found in the list of remaining objects in the reference frame. The number of bounding boxes stored in the matching list can be equal to the number of objects in the list of objects in the current frame.

[0288] Fig.18 A bitstream containing point cloud data and parameters according to an embodiment is shown.

[0289] Point cloud data sending method / device according to an embodiment ( Figure 1 The sending device 10000, Figure 1 Point cloud video encoder 10002, Figure 1 Transmitter 10003, Figure 2 Get 20000 / encode 20001 / send 20002, Figure 4 The encoder, Fig.12 The sending device Fig.14 Devices Figures 15 to 17 The encoding, Figures 18 to 27 Bitstream generation, Fig.28 The encoding, Fig.30 and Fig.31 The encoding, Figure 32 to Figure 35 bitstream and parameter generation, Fig.36 The encoding and Fig.38 The sending method can encode point cloud data, generate relevant parameters, and generate and send Fig.18 The bit stream is shown.

[0290] Point cloud data receiving method / device according to an embodiment ( Figure 1 The receiving device 10004, Figure 1 Receiver 10005, Figure 1 Point cloud video decoder 10006, Figure 2 Send 20002 / decode 20003 / render 20004, Fig.10 and Fig.11 The decoder, Fig.13 The receiving device Fig.14 Devices Figures 15 to 17 Decoding, Figures 18 to 27 Bitstream parsing, Fig.29 Decoding, Fig.30 and Fig.31 Decoding, Figure 32 to Figure 35 Bitstream parsing, Fig.37 Decoding and Fig.39 The receiving method) can receive and parse Fig.18 The bit stream shown is decoded based on the parameters.

[0291] When generating a bitstream, a method / apparatus according to an embodiment may define a compact bounding box for defining an object for each frame, and may define a component for matching an inter-frame object compact bounding box. A sequence parameter set of a bitstream according to an embodiment indicates that a compact bounding box (compact bbox) of an object has been matched for inter-frame prediction. A sequence parameter set may be referred to as sequence signaling information, etc. A geometry parameter set (GPS) carries initial parameters for connecting compact bounding boxes of objects. GPS may be referred to as geometry signaling information, etc. A geometry slice unit may carry detailed information for connecting compact bounding boxes of objects.

[0292] Although the signaling (parameter) information according to the embodiment has been described with respect to its use in geometry, the information may be shared for attribute encoding, and the location of the signaling may be added to a higher / lower level syntax structure depending on the purpose.

[0293] Relevant information may be signaled to add / execute an implementation. In the following, parameters according to an implementation (which may be referred to as metadata, signaling information, etc.) may be generated in a process of a transmitter according to an implementation described below, and delivered to a receiver according to an implementation for use in a reconstruction process. For example, parameters according to an implementation may be generated in a metadata processor (or metadata generator) of a transmitting device according to an implementation described below, and acquired by a metadata parser of a receiving device according to an implementation described below. The configuration of the encoded point cloud will be described with reference to each figure.

[0294] The following abbreviations are used in this article: SPS: Sequence Parameter Set; GPS: Geometry Parameter Set; APS: Attribute Parameter Set; TPS: Tile Parameter Set; Geom: Geometry Bitstream = Geometry Slice Header + Geometry Slice Data; Attr: Attribute Bitstream = Attribute Brick Header + Attribute Brick Data. Slice, Brick, and Data Unit are encoding / decoding units of point cloud data. These terms may be interpreted as corresponding to each other.

[0295] Reference Fig.18 , the bitstream may include SPS, GPS, APS, TPS and TPS. There may be multiple APSs. TPS may include information about the tile bounding boxes of multiple tiles. For example, it may include position (coordinate) information, size information (width, depth, height), etc. about the tile bounding box. The bitstream may contain geometric information (data) and attribute information (data) based on each slice (data unit). Because the point cloud data is encoded on a per-slice (data unit) basis, the bitstream may contain multiple slices (data units). A slice (data unit) may include geometric information (position) about a point and one or more attribute information (color, reflectivity, etc.). For example, slice (data unit) 0 may include geometric data, and the geometric data may include a geometric slice header and geometric slice data. The geometric slice header may contain information about geometry. For example, it may contain information such as a geometric parameter set ID, a geometric tile ID, a geometric slice ID, the origin of a box (bounding box) containing geometry, a logarithmic scale of the box, a maximum node size, and the number of points.

[0296] Fig.19 The syntax of a sequence parameter set (SPS) according to an embodiment is shown.

[0297] Fig.19 Shown is included in Fig.18 The SPS in the bitstream of the .SPS may contain information for configuring object compact bounding box matching. The information for configuring object compact bounding box matching may be added to the SPS and signaled.

[0298] sps_compact_bbox_matching_enable: When inter prediction is applied to a sequence, it can signal whether box-by-box prediction for defining object regions is enabled. When sps_compact_bbox_matching_enable is TRUE, it can indicate that prediction has been performed for each object region. When it is FALSE, it can indicate that prediction has been performed using conventional methods.

[0299] A sequence parameter set may also contain the following elements:

[0300] simple_profile_compatibility_flag: simple_profile_compatibility_flag equal to 1 specifies that the bitstream conforms to the simple profile. simple_profile_compatibility_flag equal to 0 specifies that the bitstream conforms to a profile other than the simple profile.

[0301] dense_profile_compatibility_flag: When equal to 1, it specifies that the bitstream conforms to the dense profile. density_profile_compatibility_flag equal to 0 specifies that the bitstream conforms to a profile other than the dense profile.

[0302] predictive_profile_compatibility_flag: When equal to 1, it specifies that the bitstream conforms to a predictive profile. predictive_profile_compatibility_flag equal to 0 specifies that the bitstream conforms to a profile other than the predictive profile.

[0303] main_profile_compatibility_flag: When equal to 1, it specifies that the bitstream conforms to the main profile (mainprofile). main_profile_compatibility_flag equal to 0 specifies that the bitstream conforms to a profile other than the main profile.

[0304] reserved_profile_compatibility_18bits: shall be equal to 0 in bitstreams conforming to this version of this document. Other values ​​of reserved_profile_compatibility_18bits are reserved for future use by ISO / IEC. A decoder shall ignore the value of reserved_profile_compatibility_18bits.

[0305] slice_reordering_constraint_flag: when equal to 1, indicates that the bitstream is sensitive to reordering and removal of data units. slice_reordering_constraint_flag equal to 0 indicates that the bitstream is not sensitive to reordering and removal of data units.

[0306] unique_point_positions_constraint_flag: When equal to 1, indicates that all output points have unique positions in each point cloud frame referenced to the current SPS. unique_point_positions_constraint_flag equal to 0 indicates that two or more output points can have the same position in any point cloud frame referenced to the current SPS.

[0307] level_idc: Indicates the level specified in Annex A to which the bitstream conforms. The bitstream shall not contain any value for level_idc other than the value specified in Annex A. Other values ​​of level_idc are reserved for future use by ISO / IEC.

[0308] sps_seq_parameter_set_id: Provides an identifier for the SPS so that it can be referenced by other syntax elements. sps_seq_parameter_set_id shall be 0 in a bitstream conforming to this version of this document. Other values ​​of sps_seq_parameter_set_id are reserved for future use by ISO / IEC.

[0309] frame_ctr_lsb_bits: specifies the length of the syntax element frame_ctr_lsb in bits.

[0310] slice_tag_bits: specifies the length of the syntax element slice_tag in bits.

[0311] seq_origin_bits: specifies the length of the syntax element seq_origin_xyz[k] in bits.

[0312] seq_origin_xyz[k] and seq_origin_log2_scale: Specifies the origin of the sequence's local coordinate system. Index k is the kth X, Y, or Z component of the origin coordinate. When not present, the values ​​of seq_origin_xyz[k] and seq_origin_log2_scale are inferred to be 0.

[0313] The array SeqOrigin is the origin of the sequence local coordinate system:

[0314] SeqOrigin[k]=seq_origin_xyz[k]< <seq_origin_log2_scale

[0315] seq_bounding_box_size_bits: The length of the syntax element seq_bounding_box_size_minus1_xyz[k] in bits.

[0316] seq_bounding_box_size_xyz_minus1[k] plus 1 specifies the kth component of the width, height, and depth of the coded volume dimensions in the output coordinate system, respectively. When not present, the coded volume dimensions are undefined.

[0317] seq_unit_numerator_minus1, seq_unit_denominator_minus1, and seq_unit_in_metres_flag: specify the length of the X, Y, and Z unit vectors in the output coordinate system.

[0318] seq_global_scale_factor_log2, seq_global_scale_refinement_num_bits, and seq_global_scale_refinement_factor: specify the fixed decimal scaling factor used to derive the output point positions from their positions in the sequence's local coordinate system.

[0319] seq_global_scale_factor_log2: Used to derive a global scaling factor to apply to locations in the point cloud.

[0320] seq_global_scale_refinement_num_bits: length of the syntax element seq_global_scale_refinement_factor in bits. When seq_global_scale_refinement_num_bits is equal to 0, no segmentation is applied.

[0321] seq_global_scale_refinement_factor: Specifies the refinement of the global scaling value. When not present, seq_global_scale_refinement_factor is inferred to be equal to 0.

[0322] sps_num_attributes: Indicates the number of attributes in the coded point cloud. It is a requirement for bitstream conformance that each slice has an attribute data unit corresponding to each attribute component listed in the SPS.

[0323] attribute_dimension_minus1[attrId] plus 1 specifies the number of components of the attrId-th attribute.

[0324] attribute_instance_id[attrId]: specifies the instance identifier of the attrId-th attribute.

[0325] attribute_bitdepth_minus1[attrId] plus 1 specifies the bit depth of each component of the attrId-th (one or more) attribute signals.

[0326] known_attribute_label_flag[attrId], known_attribute_label_flag[attrId]l, and attribute_label_oid[attrId]: Identify the type of data passed in the attrId-th attribute. known_attribute_label_flag[attrId] indicates whether the attribute is identified by the value of known_attribute_label[attrId] or by the object identifier attribute_label_oid[attrId].

[0327] The attribute type identified by known_attribute_label may be specified. When the value of known_attribute_label is not specified, it is reserved for future use by ISO / IEC.

[0328] The attribute type may indicate color, reflectivity, opacity, frame index, frame number, material identifier, normal vector, etc.

[0329] num_attribute_parameters: indicates the number of attribute parameter sets in the bitstream. Attribute parameters signaled in a sequence parameter set are applied to all data units in the coded point cloud sequence.

[0330] axis_coding_order: Specifies the correspondence between the X, Y, and Z output axis labels and the three position components of all points in the reconstructed point cloud.

[0331] bypass_stream_enabled_flag equal to 1 indicates that the bypass coding mode can be used when reading the bitstream. bypass_stream_enabled_flag equal to 0 indicates that the bypass coding mode is not used when reading the bitstream.

[0332] entropy_continuation_enabled_flag equal to 1 indicates that the initial entropy context state of a slice may depend on the final entropy context state of the previous slice. entropy_continution_enabled_flag equal to 0 specifies that the initial entropy context state of each slice is independent. entropy_continuation_enabled_flag equal to 0 is a requirement for bitstream conformance when slice_reordering_constaint_flag is equal to 0.

[0333] sps_extension_flag equal to 0 specifies that the syntax element sps_extension_data_flag is not present in the SPS syntax structure. sps_extension_flag shall be equal to 0 in bitstreams conforming to this version of this document. The value of sps_extension_flag of 1 is reserved for future use by ISO / IEC. A decoder shall ignore any syntax element sps_extension_data_flag following the value of 1 of sps_extension_flag in the SPS syntax structure.

[0334] sps_extension_data_flag can have any value. Its presence and its value do not affect the conformity of a decoder to the profile specified in Annex A. Decoders conforming to this version of this document shall ignore all syntax elements sps_extension_data_flag.

[0335] Fig. 20 The syntax of the geometry parameter set (GPS) according to an embodiment is shown.

[0336] Fig. 20 Shown is included in Fig.18 The GPS in the bitstream of the .GPS can include information for configuring compact bounding box matching of objects. Information for configuring compact bounding box matching of objects can be added to the GPS.

[0337] gps_compact_bbox_matching_enable: When sps_compact_bbox_matching_enable is TRUE, gps_compact_bbox_matching_enable can signal whether prediction of boxes defining object areas is performed in geometry coding on a frame-by-frame basis. When sps_compact_bbox_matching_enable information is omitted at the SPS level, gps_compact_bbox_matching_enable can operate independently and can be applied when prediction is required for each object area or each specific area, regardless of the geometry coding method. When gps_compact_bbox_matching_enable is TRUE, it can indicate that prediction has been performed for each object area. When it is FALSE, it can indicate that prediction has been performed using a conventional method.

[0338] gps_compact_bbox_matching_coordinates: When gps_compact_bbox_matching_enable is TRUE, gps_compact_bbox_matching_coordinates signals the type of the value of the coordinate information about the object's compact bounding box. When it is TRUE, it indicates that the value is stored based on spherical coordinates. When it is FALSE, it may indicate that the value is stored based on Cartesian coordinates.

[0339] Creteria_point: can signal whether the matching between boxes should be based on the minimum (min) or maximum (max) value of the bounding box. When it is TRUE, it can indicate that the matching is performed based on the minimum value of the bounding box. When it is FALSE, it can indicate that the matching is performed based on the maximum value of the bounding box.

[0340] numPUminus1: Indicates the number of segmentations to be performed when searching for an object after segmenting each region of the point cloud to define a compact bounding box for the object. When equal to 0, it may indicate no segmentation. The point cloud may be divided into numPUminus1+1 segments.

[0341] gps_compact_bbox_matching_criteria_sph: When gps_compact_bbox_matching_enable is TRUE, gps_compact_bbox_matching_criteria_sph may indicate that the matching list is found based on spherical coordinates, and for spherical coordinates, it may indicate the parameters that form the basis for matching (see Fig.21 ).

[0342] gps_compact_bbox_matching_criteria: When gps_compact_bbox_matching_enable is FALSE, gps_compact_bbox_matching_criteria may indicate that a matching list is found based on Cartesian coordinates, and for Cartesian coordinates, it may indicate the parameters that form the basis for making the match (see Fig. 22 ).

[0343] GPS can also contain the following information.

[0344] gps_geom_parameter_set_id identifies the GPS for reference by other DUs.

[0345] gps_seq_parameter_set_id identifies the active SPS with sps_seq_parameter_set_id.

[0346] slice_geom_origin_scale_present indicates whether slice_geom_origin_log2_scale is present in the GDU header (when equal to 1) or not (when equal to 0). slice_geom_origin_scale_present equal to 0 indicates that the slice origin scale is indicated by gps_geom_origin_log2_scale.

[0347] gps_geom_origin_log2_scale specifies the scaling factor used to derive the slice origin from slice_geom_origin_xyz when slice_geom_origin_scale_present is equal to 0.

[0348] geom_dup_point_count_enabled specifies whether duplicate points can be signaled in the GDU by per-point duplicate count (when equal to 1) or not by per-point duplicate count (when equal to 0).

[0349] geom_dup_point_count_enabled equal to 0 does not prohibit the encoding of the same point position multiple times within a single slice by means other than the syntax elements direct_dup_point_cnt, occ_dup_point_cnt or ptn_dup_point_cnt.

[0350] geom_tree_type equal to 0 indicates that the tile geometry is encoded using occupancy trees (7.3.3.4). geom_tree_type equal to 1 indicates that the tile geometry is encoded using prediction trees (7.3.3.8).

[0351] gps_extension_present indicates whether the syntax element gps_extension_data is present (when equal to 1) or absent (when equal to 0) in the GPS syntax structure. gps_extension_present is 0 in bitstreams conforming to this version of this document.

[0352] gps_extension_data can have any value. Its presence and its value do not affect the decoder's compliance with the profile specified in this version of this document. A decoder can ignore all syntax elements of gps_extension_data.

[0353] geom_angular_enabled indicates whether tile geometry is encoded (when equal to 1) or not (when equal to 0) based on information about the beam set positioned and rotated along the V axis of the angular origin. When geom_angular_enabled is enabled, point positions are assumed to be sampled along rays cast by the beams.

[0354] slice_angular_origin_present specifies whether (when equal to 1) or not (when equal to 0) the slice relative angular origin is signaled in the GDU header. slice_angular_origin_present equal to 0 indicates that the angular origin is gps_angular_origin_xyz. When not present, slice_angular_origin_present is inferred to be 0.

[0355] gps_angular_origin_bits_minus1 plus 1 indicates the length of each syntax element gps_angular_origin_xyz in bits.

[0356] gps_angular_origin_xyz[k] indicates the kth XYZ coordinate of the angular origin in the coded coordinate system.

[0357] num_beam_minus 1 plus 1 indicates the number of beams listed by the GPS.

[0358] beam_elevation_init and beam_elevation_diff[i] together indicate the beam elevation as the slope above the ST plane. The elevation slope of the ith beam is represented by the expression BeamElev[i]. It is a binary fixed point value with 18 decimal bits.

[0359] beam_voffset_init and beam_voffset_diff[i] together indicate the V-axis offset of the listed beam from the angular origin. The offset is specified in units of the encoding coordinate system. The offset of the i-th beam is specified by the expression BeamOffsetV[i].

[0360] beam_steps_per_rotation_init_minus1 and beam_steps_per_rotation_diff[i] indicate the number of steps taken per rotation of the rotation beam. The value for the i-th beam is indicated by the expression BeamStepsPerRev[i].

[0361] ptree_ang_azimuth_pi_bits_minus11 plus 11 indicates the number of bits that indicate a half rotation of the beam around the V axis. A half rotation is π radians.

[0362] ptree_ang_radius_scale_log2 indicates the factor used to scale the radial angular coordinates of a point during transformation to Cartesian coordinates.

[0363] ptree_ang_azimuth_step_minus1 plus 1 indicates the expected change in the azimuth of the beam rotated between code points. The azimuth prediction residual used to encode the angular prediction tree may be encoded as a multiple of ptree_ang_azimuth_step_minus1+1, and the remainder.

[0364] occtree_point_cnt_list_present indicates whether the GDU footer lists the number of points per occupancy tree level (when equal to 1) or does not list them (when equal to 0). When not present, occtree_point_cnt_list_present is inferred to be 0.

[0365] occtree_direct_coding_mode greater than 0 indicates that the point position may be encoded by an appropriate direct node in the occtree. occtree_direct_coding_mode equal to 0 indicates that direct nodes should not be present in the occupancy tree.

[0366] The rate of direct node eligibility generally increases as the value of occtree_direct_coding_mode increases.

[0367] occtree_direct_joint_coding_enabled indicates that the direct node used to encode two points should jointly encode the positions (when equal to 1) or not jointly encode (when equal to 0) according to the specific order of the points.

[0368] occtree_coded_axis_list_present equal to 1 indicates that the GDU header contains the syntax element occtree_coded_axis for deriving the node size of each occtree level. occtree_coded_axis_list_present equal to 0 indicates that the syntax element occtree_coded_axis is not present in the GDU syntax and the occupancy tree indicates a three-dimensional volume indicated by the tree depth.

[0369] occtree_neigh_window_log2_minus1 plus 1 indicates the number of occupied tree node positions that form each availability window within the tree level. Nodes outside the window are not available in processes related to nodes within the window. occtree_neigh_window_log2_minus1 equal to 0 indicates that only sibling nodes should be considered available to the current node.

[0370] occtree_adjacent_child_enabled indicates whether the adjacent children of the adjacent occupancy tree node are used for bit occupancy contextualization (when equal to 1) or not used for bit occupancy contextualization (when equal to 0). When not present, occtree_adjacent_child_enabled is inferred to be 0.

[0371] occtree_intra_pred_max_nodesize_log2 minus 1 indicates the maximum size of an occupancy tree node suitable for intra-slice occupancy prediction. When not present, occtree_intra_pred_max_nodesize_log2 is inferred to be 0.

[0372] occtree_bitwise_coding indicates that the node occupancy bitmap is encoded using the syntax element occupancy_bit (when equal to 1) or using the precoded syntax element occupancy_byte (when equal to 0).

[0373] occtree_planar_enabled indicates whether the encoding of the node occupancy bitmap is performed partially by signaling of occupied and unoccupied planes (when equal to 1) or not (when equal to 0). When not present, occtree_planar_enabled is inferred to be 0.

[0374] occtree_planar_threshold[i] indicates a threshold used in part to determine axis-specific suitability for plane occupancy coding. The thresholds are expressed from the most likely plane axis (i=0) to the least likely plane axis (i=2). Each threshold represents the minimum likelihood of a qualifying axis for which occ_single_plane is expected to be 1. The range of occtree_planar_threshold [8,120] corresponds to the likelihood interval [0,1).

[0375] When present, occtree_direct_node_rate_minus1 indicates that only occtree_direct_node_rate_minus1+1 nodes out of all 32 eligible nodes may be encoded as direct nodes.

[0376] occtree_planar_buffer_disabled indicates whether per-node occupancy plane position contextualization using the plane position of the previously coded node should be disabled (when equal to 1) or not disabled (when equal to 0). When not present, occtree_planar_buffer_disabled is inferred to be 0.

[0377] geom_scaling_enabled indicates whether the encoded geometry should be scaled (when equal to 1) or not scaled (when equal to 0) during the geometry decoding process.

[0378] geom_qp indicates the geometry QP before adding per-slice and per-node offsets.

[0379] geom_qp_mul_log2 indicates the scaling factor applied to the geometry QP. Each time the scaling step size doubles, Exp2(3-geom_qp_mul_log2) QP values ​​are given.

[0380] ptree_qp_period_log2 indicates the period of signaling prediction tree node QP offsets. For all Exp2(ptree_qp_period_log2) nodes, this period is one.

[0381] occtree_direct_node_qp_offset indicates the offset relative to the slice geometry QP, which is used to scale the point positions for direct node coding.

[0382] Fig.21 A matching mode according to an embodiment is shown.

[0383] exist Fig.21 It is shown in Fig. 20 The matching criteria indicated by gps_compact_bbox_matching_criteria_sph in gps_compact_bbox_matching_criteria_sph.

[0384] The matching between bounding boxes may be performed based on a proximity reference value, based on at least one of a signaled radius, azimuth, elevation / laser ID, radius+azimuth, radius+elevation / laser ID, and azimuth+elevation / laser ID (see Fig.17 wait).

[0385] Fig. 22 Matching criteria according to an embodiment are shown.

[0386] Fig. 22 Shows Fig. 20 The value of gps_compact_bbox_matching_criteria.

[0387] Based on at least one criterion such as X-axis, Y-axis, Z-axis, X-axis and Y-axis, Y-axis and Z-axis, X-axis and Z-axis, Euclidean distance or Manhattan distance, matching can be performed between bounding boxes having values ​​closest to the criterion (see Fig.17 wait).

[0388] Fig.23 A geometry slice header is shown according to an embodiment.

[0389] Fig.23 Shown included in Fig.18 A slice header in a bit stream, where the slice header can be called a data unit header, etc.

[0390] The geometry slice header may contain information for configuring tight bounding box matching of objects. Information for configuring tight bounding box matching of objects may be added to the geometry slice header.

[0391] gsh_compact_bbox_matching_enable: When gps_compact_bbox_matching_enable is TRUE, gsh_compact_bbox_matching_enable can signal whether prediction according to the box defining the object area is performed in the geometry encoding on a slice-by-slice basis. When the gps_compact_bbox_matching_enable information is omitted at the GPS level, gsh_compact_bbox_matching_enable can operate independently and can be applied when prediction is required according to the object area or according to a specific area, regardless of the geometry encoding method. When gsh_compact_bbox_matching_enable is TRUE, it can indicate that prediction has been performed according to the object area. When it is FALSE, it can indicate that prediction has been performed using conventional methods.

[0392] gps_compact_bbox_matching_coordinates: When gsh_compact_bbox_matching_enable is TRUE, gps_compact_bbox_matching_coordinates signals the type of the value of the coordinate information about the object's compact bounding box. When it is TRUE, it indicates that the value is stored based on spherical coordinates. When it is FALSE, it may indicate that the value is stored based on Cartesian coordinates.

[0393] Creteria_point: It can signal whether the matching between boxes should be based on the minimum (min) or maximum (max) value of the bounding box. When it is TRUE, it can indicate that the matching is performed based on the minimum value of the bounding box. When it is FALSE, it can indicate that the matching is performed based on the maximum value of the bounding box.

[0394] numPUminus1: Indicates the number of segmentations to be performed when searching for objects after segmenting each region of the point cloud to define a compact bounding box for the object. When equal to 0, no segmentation may be indicated. The point cloud may be divided into numPUminus1+1 segments.

[0395] gsh_compact_bbox_matching_criteria_sph: When gsh_compact_bbox_matching_enable is TRUE, gps_compact_bbox_matching_criteria_sph may indicate that the matching list is found based on spherical coordinates, and for spherical coordinates, it may indicate the parameters that form the basis for making the match. In the mode value, 0 may indicate that the radius is considered, 1 may indicate that the azimuth is considered, 2 may indicate that the elevation or laser ID is considered, 3 may indicate that both the radius and azimuth are considered, and 4 may indicate that both the radius and the elevation or laser ID are considered, and 5 indicates that both the azimuth and the elevation or laser ID are considered. Any parameters based on spherical coordinates may be added to the criteria to be considered.

[0396] gsh_compact_bbox_matching_criteria: When gsh_compact_bbox_matching_enable is FALSE, gsh_compact_bbox_matching_criteria may indicate that a matching list is found based on Cartesian coordinates, and for Cartesian coordinates, it may indicate the parameters that form the basis for matching. In the value of mode, 0 may indicate that the comparison is performed based on the x-axis, 1 may indicate that the comparison is performed based on the y-axis, 2 may indicate that the comparison is performed based on a single axis, the z-axis, and 3, 4, and 5 may indicate that the comparison is performed based on two of the three axes (e.g., the x-axis and the y-axis, the y-axis and the z-axis, and the x-axis and the z-axis), 6 and 7 may indicate that the comparison is performed based on distance to consider all three axes, where 6 may indicate that the comparison is performed based on the Euclidean distance, and 7 may indicate that the comparison is performed based on the Manhattan distance.

[0397] The geometry slice header may be referred to as a geometry data unit header.

[0398] The geometry data unit header may also contain the following information.

[0399] gdu_geometry_parameter_set_id indicates the active GPS with gps_geom_parameter_set_id.

[0400] slice_id identifies the slice to be referenced by other DUs.

[0401] slice_tag identifies the slice as a member of a slice group with the same value of slice_tag. When a tile inventory DU is present, a slice group is a patch identified by a patch ID. When a patch inventory DU is not present, the interpretation of slice_tag is application dependent.

[0402] frame_ctr_lsb specifies the LSB of frame_ctr_lsb_bits of the conceptual frame counter FrameCtr.

[0403] slice_entropy_continution equal to 1 indicates that the entropy parsing state recovery process should be applied at the beginning of all ADUs and GDUs of the slice. slice_entropy_continution equal to 0 specifies that the parsing of all ADUs and GDUs of the slice is independent of the parsing of other slices. When not present, slice_entropy_continuation is inferred to be 0.

[0404] It is a bitstream conformance requirement that slice_entropy_continution shall be equal to 0 when the GDU is the first in a coded point cloud frame. The decoder shall ignore (remove from the bitstream and discard) any slice in a coded point cloud frame with slice_entropy_continution equal to 1 that is not preceded by a slice with slice_entropy_continution equal to 0 in the same frame.

[0405] prev_slice_id shall be the same as the GDU slice_id of the previous slice in the bitstream order. The decoder shall ignore (remove from the bitstream and discard) slices in the same frame whose prev_slice_id is not equal to the slice_id of the previous slice.

[0406] When slice_tag is not equal to the slice_tag of the GDU identified by prev_slice_id, it is recommended that slice_entropy_continution is 0. For example, when slice_tag is used to select a subset of slices, decoding may be prevented if there is a dependency on a non-selected slice.

[0407] slice_geom_origin_bits_minus1+1 specifies the length in bits of each syntax element slice_geom_origin_xyz.

[0408] slice_geom_origin_xyz[] and slice_geom_origin_log2_scale specify the []th XYZ coordinate of the slice origin in the coded coordinate system. The slice origin in STV coordinates is specified by the expression SliceOrigin[]. When not present, slice_geom_origin_log2_scale is inferred to be gps_geom_origin_log2_scale.

[0409] slice_angular_origin_bits_minus1+1 specifies the bit length of each syntax element slice_angular_origin_xyz.

[0410] slice_angular_origin_xyz[] specifies the []th XYZ coordinate relative to the angular origin of the slice's coordinate system. When not present, slice_angular_origin_xyz[] should be inferred to be 0.

[0411] The slice angle origin of STV coordinates is specified by the expression AngularOrigin[].

[0412] slice_geom_qp_offset specifies the slice geometry QP as an offset from the GPS geom_qp. When not present, slice_geom_qp_offset is inferred to be 0.

[0413] Geometry Data Unit Footer Semantics

[0414] As specified in 11.2.4, the start of the GDU footer is determined to be at the end of the GDU.

[0415] slice_num_points_minus1+1 indicates the number of coded points in a DU. It is a requirement for bitstream conformance that slice_num_points_minus1+1 is equal to the number of decodable points in a DU.

[0416] Fig.24 Matching criteria according to an embodiment are shown.

[0417] Fig.24 It is shown in Fig.23 The matching criteria mode associated with gsh_compact_bbox_matching_criteria_sph is described in gsh_compact_bbox_matching_criteria_sph.

[0418] Fig.25 Matching criteria according to an embodiment are shown.

[0419] Fig.25 It is shown in Fig.23 The matching criteria pattern associated with gsh_compact_bbox_matching_criteria described in gsh_compact_bbox_matching_criteria.

[0420] Fig.26 An attribute parameter set (APS) according to an embodiment is shown.

[0421] Fig.26 Shown is included in Fig.18 The attribute parameter set in the bitstream.

[0422] The attribute parameter set may also contain information for configuring tight bounding box matching of objects.

[0423] aps_compact_bbox_matching_enable: When sps_compact_bbox_matching_enable is TRUE, aps_compact_bbox_matching_enable can signal whether prediction according to the box defining the object area is performed in the attribute encoding. It can signal whether this information is inherited in the attribute encoding. When aps_compact_bbox_matching_enable is TRUE, it can indicate that the matching list of the object compact bounding box is inherited for attribute encoding. When it is FALSE, it can indicate that the attributes are encoded using the normal method.

[0424] The attribute parameter set can also contain the following information:

[0425] aps_attr_parameter_set_id identifies the APS to be referenced by other DUs.

[0426] aps_seq_parameter_set_id identifies the active SPS based on sps_seq_parameter_set_id.

[0427] attr_coding_type specifies the attribute encoding method.

[0428] attr_primary_qp_minus4+4 specifies the QP for the primary attribute component before adding per-slice, per-region, and per-transform level offsets.

[0429] attr_secondary_qp_offset specifies the offset to be applied to the primary attribute QP to derive the QP of the secondary attribute component.

[0430] attr_qp_offset_present indicates whether (when equal to 1) or not (when equal to 0) the QP offset attr_qp_offset[] according to the slice attribute is present in the ADU header.

[0431] attr_coord_conv_enabled specifies whether to use scaled angular coordinates (when equal to 1) or slice-relative STV point locations (when equal to 0). attr_coord_conv_enabled equal to 0 when geom_angular_enabled equals 0 is a requirement for bitstream conformance.

[0432] attr_coord_conv_scale_bits_minus1[]+1 specifies the length in bits of the syntax element attr_coord_conv_scale[].

[0433] attr_coord_conv_scale[] specifies the scaling factor used to scale the []th corner coordinate of the point used for attribute encoding. The scaling factor is in units of 2^(-8).

[0434] Fig. 27 An attribute data unit header according to an embodiment is shown.

[0435] Fig. 27 Shown included in Fig.18 The attribute data unit header (attribute slice header) in the bit stream.

[0436] The attribute data unit header may contain information for configuring tight bounding box matching of the object.

[0437] matching_list: When aps_compact_bbox_matching_enable is TRUE, the matching list can be inherited from the result of geometry encoding, and the compact bounding box list can be stored in the attribute data unit header for attribute encoding.

[0438] The attribute data unit header may also include the following information.

[0439] adu_attr_parameter_set_id specifies the active APS with aps_attr_parameter_set_id.

[0440] In bitstreams conforming to this version of this document, adu_reserved_zero_3bits is equal to 0.

[0441] adu_sps_attr_idx identifies the attribute that is encoded as an index into the active SPS attribute list.

[0442] At the beginning of each ADU, the variable AttrIdx is set to adu_sps_attr_idx.

[0443] Attributes encoded by the ADU have up to three components unless attr_coding_type is equal to 3.

[0444] adu_slice_id indicates the value of the previous GDU slice_id.

[0445] defattr_seq_parameter_set_id specifies the active SPS with sps_seq_parameter_set_id.

[0446] defattr_sps_attr_idx identifies the attribute that is encoded as an index into the active SPS attribute list.

[0447] The variable AttrIdx is set to defattr_sps_attr_idx at the beginning of each basic attribute data unit.

[0448] defattr_slice_id indicates the value of the previous GDU slice_id.

[0449] defattr_value[] specifies the value of the []th attribute component of all points in the slice. The length of defattr_value[] in bits is AttrBitDepth.

[0450] Fig.28 A point cloud data transmitting device according to an embodiment is shown.

[0451] Point cloud data sending method / device according to an embodiment ( Figure 1 The sending device 10000, Figure 1 Point cloud video encoder 10002, Figure 1 Transmitter 10003, Figure 2 Get 20000 / encode 20001 / send 20002, Figure 4 The encoder, Fig.12 The sending device Fig.14 Devices Figures 15 to 17 The encoding, Figures 18 to 27 Bitstream generation, Fig.28 The encoding, Fig.30 and Fig.31 The encoding, Figure 32 to Figure 35 bitstream and parameter generation, Fig.36 The encoding and Fig.38 The sending method) can be configured as follows Fig.28 The sending device shown. Fig.28 Each component in may correspond to software, hardware, a processor and / or a combination thereof.

[0452] The transmitting device according to the embodiment may be referred to as a point cloud encoder according to the embodiment, may be referred to as a device including an encoder and a transmitter, or may be referred to as a device including a memory and a processor.

[0453] The transmitter first receives point cloud data as input, separates the points into positions (or geometry) and attribute values, and performs a coordinate transformation into a codable form. If the frame is used for intra prediction, an intra prediction method is applied (see Figures 1 to 14 ) and perform entropy encoding to generate a bitstream. If the frame is not used for intra-frame prediction, check whether a compact bounding box match list is used for prediction. If a compact bounding box match list is not used, perform inter-frame prediction. If a compact bounding box match list is used, check whether the match is performed after transformation to spherical coordinates. If spherical coordinates are used, generate a match list after the transformation. Otherwise, Cartesian coordinates can be used and sorted by predefined criteria. Then, a match list can be generated. The generated list can be used to perform block-by-block prediction during inter-frame prediction. After the inter-frame prediction is completed, entropy encoding can be performed and then an output bitstream can be generated.

[0454] Reference Fig.28 According to an embodiment, a transmitting device or an encoder may include a predictor and an entropy encoder. The predictor may perform prediction on the point cloud data, and the entropy encoder may encode the point cloud data encoded by the prediction based on an entropy method to output a bit stream ( Fig.18 ).

[0455] The device according to the embodiment receives point cloud data including geometry and attributes as input data. The input data may include parameters related to the encoding of the point cloud data. The parameters may be information configured in the encoder system. Determine whether the mode is intra-frame prediction. When the mode is intra-frame prediction, the point cloud data can be predicted based on the intra-frame prediction, and the residual between the current point cloud data and the predicted data can be generated based on the predicted data, thereby reducing the size of the bit stream to be encoded. The prediction can be applied to at least one of the geometry encoding and the attribute encoding of the point cloud data, or to both the geometry and the attributes.

[0456] When the mode is not intra prediction, that is, when the mode is inter prediction mode, the prediction can be performed according to Figures 15 to 17 The point cloud data contained in the current frame is predicted based on the reference frame. The device according to the embodiment checks compact_bbox_matching_enable (see Figures 18 to 26 ). When compact_bbox_matching_enable is FALSE, inter prediction is performed (see Fig.15When compact_bbox_matching_enable is TRUE, check whether compact_bbox_matching_criteria_sph is TRUE or FALSE. When compact_bbox_matching_criteria_sph is TRUE, coordinates may be transformed. Cartesian coordinates may be transformed into spherical coordinates, and the next process may be performed based on the spherical coordinates. When compact_bbox_matching_criteria_sph is FALSE, the next process may be performed based on Cartesian coordinates. The device according to the embodiment may generate a matching box list (see Fig.17 ). Based on the matching box list, inter-frame prediction can be performed. Figure 15 to Figure 17 Processes for generating compact bounding boxes, generating matching box lists, and performing inter-frame prediction are described.

[0457] Fig.29 A point cloud data receiving device according to an embodiment is shown.

[0458] Point cloud data receiving method / device according to an embodiment ( Figure 1 The receiving device 10004, Figure 1 Receiver 10005, Figure 1 Point cloud video decoder 10006, Figure 2 Send 20002 / decode 20003 / render 20004, Fig.10 and Fig.11 The decoder, Fig.13 The receiving device Fig.14 Devices Figures 15 to 17 Decoding, Figures 18 to 27 Bitstream parsing, Fig.29 Decoding, Fig.30 and Fig.31 Decoding, Figure 32 to Figure 35 Bitstream parsing, Fig.37 Decoding and Fig.39 The receiving method) can be configured as follows Fig.29 The receiving device shown. Fig.29 Each component of may correspond to software, hardware, a processor and / or a combination thereof.

[0459] The receiving device according to the embodiment may be referred to as a point cloud decoder according to the embodiment, may be referred to as a device including a receiver and a decoder, or may be referred to as a device including a memory and a processor. Fig.29 The receiving device may correspond to Fig.28 A sending device, and can perform operations corresponding to the sending operation and / or its inverse process.

[0460] The receiver performs entropy decoding, inverse quantization, and inverse transformation on the geometry and attribute bitstreams sent by the transmitter. Then, for a slice / frame unit, if the corresponding frame is an intra-predicted frame, the receiver performs intra-prediction (see Figures 1 to 14 ) to reconstruct the point cloud. If the frame is not an intra-predicted frame, the receiver checks whether prediction has already been performed based on the compact bounding box matching list. If so, the receiver performs inter-prediction on a block-by-block basis using the matching compact bounding boxes and stores the unmatched compact bounding boxes for use in predicting the next frame. Inter-prediction may be performed on points outside the compact bounding boxes used for inter-prediction. Inter-prediction may be performed without using a compact bounding box matching list (see Fig.15 After inter prediction, reconstruction can be performed and the points can then be used for attribute encoding and rendering.

[0461] Reference Fig.29 According to an embodiment, a receiving device or a decoder may include a predictor and a reconstructor. The predictor may perform prediction on the point cloud data, and the reconstructor may reconstruct the point cloud data based on the predicted data.

[0462] The receiving device according to the embodiment can receive Fig.18 The device may decode the bitstream based on an entropy method. The device according to the embodiment may determine a mode for performing prediction. When the mode is an intra-frame prediction mode, intra-frame prediction may be performed (see Figures 1 to 14 The point cloud data may be reconstructed by summing the predicted point cloud data and the received residual value. When the mode is not the intra prediction mode, that is, when the mode is the inter prediction mode, the device according to the embodiment may check the compact bounding box enable information. When the enable information is FALSE, inter prediction is performed. When the enable information is TRUE, a matching box list is generated. Based on the generated matching box list, inter prediction may be performed (see Figures 15 to 17 etc.), and the point cloud data can be reconstructed.

[0463] An embodiment includes a method comprising: analyzing relationships between points within a frame; defining a compact bounding box that can represent a region on an object-by-object basis to match regions on a block-by-block basis when a core point is a meaningful set of points; configuring a list of compact bounding boxes; and connecting regions / blocks by matching similar regions in a corresponding list of compact bounding boxes during inter-frame prediction. Local motion estimation can be achieved by performing frame prediction between regions, rather than simple sampling or stereo segmentation. This can result in higher accuracy, reduced residuals, and prediction on a point set basis rather than a point-by-point basis. Therefore, encoding / decoding time can be reduced.

[0464] The operation according to the embodiment described in this disclosure can be performed by a sending / receiving device including a memory and / or a processor according to the embodiment. The memory can store a program for processing / controlling the operation according to the embodiment, and the processor can control the various operations described in this disclosure. The processor can be referred to as a controller, etc. In an embodiment, the operation can be performed by firmware, software, and / or a combination thereof. Firmware, software, and / or a combination thereof can be stored in a processor or a memory.

[0465] The method / apparatus according to an embodiment may include and perform classification of static objects and dynamic objects for inter-frame prediction of point cloud content and a method for motion estimation of dynamic objects.

[0466] As a method for improving the compression efficiency of point cloud data having one or more frames, the method / apparatus according to an embodiment may include a method of selecting dynamic objects in a frame for predicting the positional relationship of each node from a prediction tree geometry structure using inter-frame prediction technology and performing motion estimation and compensation by connecting the dynamic objects between frames.

[0467] In order to efficiently perform inter-frame prediction of prediction geometry, the method / apparatus according to an embodiment may include a method of separating a dynamic area from a static area based on position information about a compact bounding box that can define a prediction object and performing estimation and compensation on the separated motion.

[0468] The method / apparatus according to an embodiment may perform motion estimation and compensation based on the position difference between the minimum regions matched on an object-by-object basis. Dynamic and static regions may be separated based on the degree of motion. For each object classified as a dynamic region, a different motion vector may be found and applied. For an object classified as a static region, a single motion vector may be found and applied.

[0469] Fig.30 A method of classifying static objects and dynamic objects of an object compact bounding box according to an embodiment is shown.

[0470] follow Fig.15 and Fig.16 , the method / apparatus according to an embodiment may include and perform a method for classifying static objects and dynamic objects in an object compact bounding box connected between frames.

[0471] Based on the difference in azimuth, radius, or laser ID between connected boxes according to the criteria for connecting object boxes and the threshold, boxes having values ​​between a first threshold (threshold 1) and a second threshold (threshold 2) or greater than a single threshold can be classified as dynamic objects.

[0472] Reference Fig.16 and Fig.30, the top view shows only the boxes whose azimuth difference in the matched object compact bounding box is between threshold 1 and threshold 2. The points in area 3000 may represent the point cloud of the reference frame, and the points in area 3001 may represent the point cloud of the current frame. In the case where the box is classified as a dynamic box when the azimuth difference of the maximum position of the box is between two thresholds, the object classified as a dynamic object in the reference frame may be separated from the list of matched object boxes (see Fig.17 wait).

[0473] When only isolated dynamic objects in the reference frame are marked, e.g. Fig.30 The points shown labeled 3002 may represent the area of ​​each corresponding object. The remaining objects not classified as dynamic objects may be classified as static objects to allow motion estimation and compensation to be performed differently than for dynamic objects.

[0474] Reference Fig.30 , showing a top view of a point and a bounding box containing the point. As described above, the method / apparatus according to the embodiment can match object compact bounding boxes and classify dynamic objects and static objects based on whether a specific reference (e.g., the difference in azimuth, radius, or laser ID between matched or connected boxes) is within a specific range or satisfies a specific standard. Fig.30 is an example of a top view of points in a reference frame and a current frame, showing the distribution of points in the reference / current frames labeled 3000 and 30001. Once the dynamic object boxes are determined in the points, the objects classified as dynamic objects in the reference frame can be separated from the list of matching object boxes. When separated, only the dynamic objects in the reference frame are retained, and motion estimation and compensation can be performed on the current frame based on the points classified as dynamic objects. Motion estimation and compensation can be performed on the current frame based on the points classified as static objects in the reference frame. Since the characteristics of dynamic objects are different from those of static objects, motion estimation and compensation can be performed more accurately by classifying objects according to different criteria.

[0475] Fig.31 A reference bounding box in a reference frame and a current bounding box in a current frame are shown according to an embodiment.

[0476] like Fig.30 As shown, dynamic boxes and static boxes can be classified in object compact bounding boxes connected between frames, and bounding boxes of dynamic boxes can be represented in reference frames and current frames.

[0477] The method / apparatus according to an embodiment may predict the motion of dynamic objects in compact bounding boxes of the objects connected between frames.

[0478] Objects separated as dynamic objects may have motion independent of the motion of other objects. Different motion estimation and compensation may be performed for each dynamic object.

[0479] The difference between the eight vertex positions of the compact bounding box of each dynamic object in the reference frame and the object compact bounding box in the current frame connected to the dynamic object and the median value of a box (i.e., the difference between the positions of a total of nine points) can be used as a candidate motion vector. When each of the eight motion vectors is applied to the dynamic object in the reference frame, the value that can minimize the difference with the points contained in the connected object bounding box (object Bbox) in the current frame can be used as a motion vector.

[0480] like Fig.31 The bounding boxes (bboxes) of the dynamic objects found in the reference frame and the current frame shown can be referred to as the reference bounding box (refBbox) and the current bounding box (currBbox), respectively. The combination of coordinate information about the Min and Max positions of each box can be used to determine the reference value for finding motion vector candidates.

[0481] The difference between the positions may be a candidate value for determining a motion vector as follows.

[0482] MV_candidate=(P_(ref,candidate)-P_(curr,candidate))

[0483] After applying the motion vector candidate MVcandidate to the point corresponding to the dynamic object in the reference frame, the MVcandidate with the smallest sum of differences with all combinations of points contained in the matching bbox in the current frame can be selected as the motion vector.

[0484] The method / apparatus according to an embodiment may predict the motion of static objects in a compact bounding box of the object connected between frames.

[0485] All points in the object bbox except the object bbox classified as a dynamic object are classified as static objects, and the remaining boxes except the object bbox matching the dynamic object in the current frame can undergo motion estimation of static objects.

[0486] To ensure that the same motion vector is applied to all points classified as static objects, least mean squares (LMS) or robust fitting can be used to find the motion vector that produces the smallest difference with the points matched to the static objects in the current frame when a 3×3 matrix reflecting rotation, translation, and scale is applied to the points of the static objects in the reference frame.

[0487] Reference Fig.18In an embodiment of the present disclosure, a compact bounding box for defining an object is defined for each frame. Components for matching object compact bounding boxes between frames and for classifying objects into dynamic objects and static objects for different motion vector compensation may be defined. The sequence parameter set in the bitstream signals that the object has been separated into dynamic / static objects for performing inter-frame prediction and motion compensation. The GPS in the bitstream carries initial parameters for connecting object compact bounding boxes. The geometry slice unit in the bitstream may signal detailed information for connecting object compact bounding boxes, as well as box position information about dynamic objects and motion vectors for each dynamic object.

[0488] Although the information has been described with respect to its use in a geometry structure, the information may be shared for attribute encoding and the location of the signaling may be added to a higher / lower level syntax structure depending on the purpose.

[0489] Relevant information may be signaled to add / execute an implementation. In the following, parameters according to an implementation (which may be referred to as metadata, signaling information, etc.) may be generated in a process of a transmitter according to an implementation described below, and delivered to a receiver according to an implementation for use in a reconstruction process. For example, parameters according to an implementation may be generated in a metadata processor (or metadata generator) of a transmitting device according to an implementation described below, and acquired by a metadata parser of a receiving device according to an implementation described below. The configuration of the encoded point cloud and the syntax of the parameters will be described with reference to each figure.

[0490] Fig.32 The syntax of the SPS according to an embodiment is shown.

[0491] Fig.32 Corresponds to Fig.19 SPS, and Fig.19 and Fig.32 The fields described in can be included in Fig.18 The SPS of the bitstream.

[0492] The SPS may include information for classifying dynamic / static objects for motion vector estimation.

[0493] sps_dynamic_obj_enable: When inter prediction is applied to a sequence, it may indicate that separate motion vector estimation / compensation has been applied to dynamic objects. When sps_dynamic_obj_enable is TRUE, it may indicate that motion vector compensation has been performed for each dynamic object region. When it is FALSE, it may indicate that prediction has been performed using a conventional method of performing motion compensation without dynamic objects.

[0494] Fig.33 The syntax of GPS according to the embodiment is shown.

[0495] Fig.33 Corresponds to Fig. 20 GPS. Fig. 20 and Fig.33 The information in can be included in the GPS in the bitstream.

[0496] GPS may include information for motion vector estimation by classifying dynamic / static objects.

[0497] gps_dynamic_obj_enable: When sps_dynamic_obj_enable in the SPS is TRUE, it can be signaled that motion vector compensation has been performed on a dynamic object-by-dynamic object basis during geometry encoding on a frame-by-frame basis. The unit of the acquired cloud data can be a sequence, and the sequence can include one or more frames. The encoder and decoder can encode and decode point cloud data in data units or slices of a frame. When the sps_dynamic_obj_enable information is omitted at the SPS level, gps_dynamic_obj_enable can operate independently. When motion vector compensation is required on a per-dynamic object basis, it can be applied regardless of the geometry encoding method. When gps_dynamic_obj_enable is TRUE, it can indicate that prediction / compensation has been performed according to the dynamic object area. When it is FALSE, it can indicate that prediction has been performed using a conventional method for performing motion compensation in the absence of dynamic objects.

[0498] dynamic_obj_threshold[2]: The threshold for separating dynamic objects may be signaled. One or two thresholds may be signaled. When a single threshold is signaled, an object is classified as a dynamic object if the reference value for classification of dynamic objects is greater than the threshold. When two thresholds are signaled, an object may be separated as a dynamic object if the reference value for classification of dynamic objects is greater than a first threshold (threshold[1]) and less than a second threshold (threshold[2]).

[0499] dynamic_obj_criteria_axis: Axis information used as a criterion for separating dynamic objects can be signaled. When the coordinate information representing the object frame is Cartesian coordinates, the values ​​of the x, y, and z axes are signaled as mode 0, mode 0, and mode 2 (or mode 0, mode 1, and mode 2), respectively; when the coordinate information is spherical coordinates, the values ​​of the radius, azimuth, and laser ID (or elevation) axes can be signaled as mode 0, mode 1, and mode 2, respectively.

[0500] Fig.34 A standard schema of a dynamic object according to an embodiment is shown.

[0501] As Fig.33 Continue, such as Fig.34 As shown, the mode set to the first value can indicate the X axis (Cartesian) / radius (spherical), the mode set to the second value can indicate the Y axis (Cartesian) / azimuth (spherical), and the mode set to the third value can indicate the Z axis (Cartesian) / elevation (laser ID) (spherical). The fourth value can be used to indicate a reserved value.

[0502] Fig.35 The syntax of the geometry slice header according to an embodiment is shown.

[0503] Fig.35 Corresponds to Fig.23 The geometric slice head of Fig.23 and Fig.35 The geometry slice header can be included in Fig.18 in the bit stream.

[0504] The geometry slice header may add information for performing motion vector estimation by classifying dynamic / static objects.

[0505] When gsh_dynamic_obj_enable is TRUE, it may indicate that prediction / compensation has been performed according to dynamic object regions. When it is FALSE, it may indicate that prediction has been performed using a conventional method of performing motion compensation without dynamic objects (see the section on conventional methods in this document). Fig.15 ).

[0506] numDynamicObj: When gsh_dynamic_obj_enable is TRUE, the total number of dynamic objects can be signaled.

[0507] DynamicObj_Min[i][3]: The minimum position of the dynamic object compact bounding box that can be signaled for each dynamic object. It can signal as many positions as numDynamicObj.

[0508] DynamicObj_Max[i][3]: The maximum position of the dynamic object compact bounding box that can be signaled for each dynamic object. It can signal as many positions as numDynamicObj.

[0509] DynamicObj_MV[i][3]: can signal the motion vector of each dynamic object. It can signal as many vectors as numDynamicObj.

[0510] StatisticObj_MV[3][4]: may signal a motion vector used for motion compensation of a static object and may have a 3×4 matrix structure.

[0511] Fig.36 A point cloud data transmitting device according to an embodiment is shown.

[0512] Fig.36 The sending device corresponds to Figure 1 The sending device 10000, Figure 1 Point cloud video encoder 10002, Figure 1 Transmitter 10003, Figure 2 Get 20000 / encode 20001 / send 20002, Figure 4 The encoder, Fig.12 The sending device Fig.14 Devices Figures 15 to 17 The encoding, Figures 18 to 27 Bitstream generation, Fig.28 The encoding, Fig.30 and Fig.31 The encoding, Figure 32 to Figure 35 bitstream and parameter generation, Fig.38 The sending method, etc. Fig.36 Each component of may correspond to hardware, software, a processor and / or a combination thereof.

[0513] The transmitting device according to the embodiment may correspond to a point cloud data encoder. The transmitting device may include a Fig.36 The sending device may include a processor configured to perform the operations in the block diagram and a memory configured to store relevant information. Fig.36 The operation of the encoder in the block diagram and the transmitter configured to send the encoded bit stream. The transmitting device according to the embodiment can perform Fig.36 Flowchart of the process.

[0514] The structure of the transmitting device according to the embodiment may have the structure of the point cloud encoder according to the embodiment. The point cloud encoder for processing point cloud compression may specifically represent Fig.39 As shown.

[0515] The transmitter first receives point cloud data as input data, separates the point into position (or geometry) and attribute values, and performs a coordinate transformation to an encodable form. If the frame is used for intra-frame prediction, the intra-frame prediction method is applied and entropy coding is performed to generate a bitstream. If the frame is not used for intra-frame prediction, it is checked whether the dynamic / static objects are separated to perform motion estimation. If the dynamic / static objects are not separated, inter-frame prediction is performed without separation. When dynamic / static objects are separated for motion vector estimation / compensation, a matching list is obtained by matching the object compact bounding box between the current frame and the reference frame, and the matched objects are separated into dynamic / static objects based on the signaled reference value. Motion vector estimation can be performed for each point group belonging to each separated static object and each separated dynamic object, which can be used for inter-frame prediction. After the inter-frame prediction is completed, entropy coding is performed and an output bitstream is generated.

[0516] Reference Fig.36 , the encoder receives point cloud data as input. It checks whether the mode used to encode the point cloud data is intra prediction. When the mode is intra prediction, intra prediction is performed and entropy coding is performed. Then the bitstream is generated. When the mode is not intra prediction, the dynamic object enable information is checked to be TRUE or FALSE. If dynamic object enable is FALSE, the bitstream is generated by entropy coding after inter prediction. If dynamic object enable is true, the object compact bounding box is found and box-to-box matching is performed (see Fig.16 and Fig.17 In the generated and matched object compact bounding boxes, dynamic and static objects can be classified (see Fig.30 , Fig.31 When the object is a static object, motion estimation according to the static object is performed. When the object is a dynamic object, motion estimation according to the dynamic object is performed as described above. Compensation is performed based on the estimated motion, and inter-frame prediction is performed. Then, a bit stream is output after entropy coding.

[0517] Fig.37 A point cloud data receiving device according to an embodiment is shown.

[0518] Fig.37 The receiving device may correspond to Figure 1 The receiving device 10004, Figure 1 Receiver 10005, Figure 1 Point cloud video decoder 10006, Figure 2 Send 20002 / decode 20003 / render 20004, Fig.10 and Fig.11 The decoder, Fig.13 The receiving device Fig.14 Devices Figures 15 to 17Decoding, Figures 18 to 27 Bitstream parsing, Fig.29 Decoding, Fig.30 and Fig.31 Decoding, Figure 32 to Figure 35 Bitstream parsing, Fig.39 receiving method, etc. Fig.37 Each component of may correspond to hardware, software, a processor and / or a combination thereof.

[0519] The receiving device according to the embodiment may correspond to a point cloud data decoder. The receiving device may include a Fig.37 The receiving device may include a processor configured to perform the operations in the block diagram and a memory configured to store relevant information. Fig.37 The decoder of the block diagram of the operation and the receiver configured to receive the encoded bit stream. The receiving device according to the embodiment may perform Fig.37 Flowchart of the process.

[0520] Fig.37 The operation of the receiving device corresponds to Fig.36 The operation of the sending device in the embodiment of the present invention can be the reverse process.

[0521] The receiver performs entropy decoding, dequantization, and inverse transformation on the geometry and attribute bitstreams sent by the transmitter. Then, for a slice / frame unit, if the corresponding frame is an intra-predicted frame, the receiver performs intra-prediction to reconstruct the point cloud. If the frame is not an intra-predicted frame, the receiver separates the objects into dynamic / static objects and checks whether motion estimation has been performed. If so, the signaled dynamic objects and static point clouds can be separated, and motion vector compensation can be performed for each object. Next, inter-prediction can be performed for prediction between frames. If the dynamic / static object motion estimation method is not used, inter-prediction can be performed. After inter-prediction, the point can be reconstructed to perform attribute encoding and rendering.

[0522] Reference Fig.37 , the receiving device receives Fig.18 bit stream. The receiving device decodes the point cloud data based on the parameters contained in the bit stream. It decodes the point cloud data using a prediction method. It is checked by parameters whether the prediction is intra-frame prediction. If it is intra-frame prediction, the dynamic object enable information is checked by parameters. When dynamic object enable is FALSE, inter-frame prediction is performed without distinguishing dynamic objects. When dynamic object enable is TRUE, static and dynamic objects are classified in the bounding box, and static objects undergo motion estimation and compensation according to static objects. Dynamic objects undergo motion estimation and compensation according to dynamic objects. For motion estimation and compensation according to each object, refer to the description above. After motion estimation and compensation, inter-frame prediction is performed. The inter-frame predicted point cloud data is reconstructed and the reconstructed points are output.

[0523] The method / device according to the embodiment can analyze the relationship between points in the frame, and when the core points form a meaningful set, the corresponding area is separated into dynamic and static areas according to the degree of motion after inter-frame matching, and different motion estimation and compensation can be performed on each area. By avoiding the increase of residuals due to the lack of motion estimation methods and applying different motion vectors for each area, the residuals can be reduced and the accuracy can be increased.

[0524] The process of the transmitting / receiving device according to the embodiment may be described in conjunction with point cloud compression processing.

[0525] The operation according to the embodiment described in this disclosure can be performed by a sending / receiving device including a memory and / or a processor according to the embodiment. The memory can store a program for processing / controlling the operation according to the embodiment, and the processor can control the various operations described in this disclosure. The processor can be referred to as a controller, etc. In an embodiment, the operation can be performed by firmware, software, and / or a combination thereof. Firmware, software, and / or a combination thereof can be stored in a processor or a memory.

[0526] Fig.38 A point cloud data transmission method according to an embodiment is shown.

[0527] Fig.38 Can correspond to Figure 1 The sending device 10000, Figure 1 Point cloud video encoder 10002, Figure 1 Transmitter 10003, Figure 2 Get 20000 / encode 20001 / send 20002, Figure 4 The encoder, Fig.12 The sending device Fig.14 Devices Figures 15 to 17 The encoding, Figures 18 to 27 Bitstream generation, Fig.28 The encoding, Fig.30 and Fig.31 The encoding, Figure 32 to Figure 35 bitstream and parameter generation, Fig.36 Encoding, etc.

[0528] S3800: According to an implementation, the point cloud data sending method may include encoding the point cloud data.

[0529] The encoding operation according to an embodiment may include Figure 1 The sending device 10000, Figure 1 Point cloud video encoder 10002, Figure 1 Transmitter 10003, Figure 2Get 20000 / encode 20001 / send 20002, Figure 4 The encoder, Fig.12 The sending device Fig.14 Devices Figures 15 to 17 The encoding, Figures 18 to 27 Bitstream generation, Fig.28 The encoding, Fig.30 and Fig.31 The encoding, Figure 32 to Figure 35 bitstream and parameter generation and Fig.36 The encoding operation of the encoding.

[0530] S3801: According to the implementation scheme, the point cloud data sending method may further include sending a bit stream containing the point cloud data.

[0531] The sending operation according to an embodiment may include Figure 1 The sending device 10000, Figure 1 Point cloud video encoder 10002, Figure 1 Transmitter 10003, Figure 2 Get 20000 / encode 20001 / send 20002, Figure 4 The encoder, Fig.12 The sending device Fig.14 Devices Figures 15 to 17 The encoding, Figures 18 to 27 Bitstream generation, Fig.28 The encoding, Fig.30 and Fig.31 The encoding, Figure 32 to Figure 35 bitstream and parameter generation and Fig.36 The encoded bit stream sending operation.

[0532] Fig.39 A point cloud data receiving method according to an embodiment is shown.

[0533] Fig.39 Can correspond to Figure 1 The receiving device 10004, Figure 1 Receiver 10005, Figure 1 Point cloud video decoder 10006, Figure 2 Send 20002 / decode 20003 / render 20004, Fig.10 and Fig.11 The decoder, Fig.13 The receiving device Fig.14 Devices Figures 15 to 17 Decoding, Figures 18 to 27 Bitstream parsing, Fig.29 Decoding, Fig.30 and Fig.31 Decoding, Figure 32 to Figure 35Bitstream parsing, Fig.37 Decoding, etc.

[0534] S3900: The point cloud data receiving method according to an embodiment may include receiving a bit stream containing point cloud data.

[0535] The receiving operation according to an embodiment may include Figure 1 The receiving device 10004, Figure 1 Receiver 10005, Figure 1 Point cloud video decoder 10006, Figure 2 Send 20002 / decode 20003 / render 20004, Fig.10 and Fig.11 The decoder, Fig.13 The receiving device Fig.14 Devices Figures 15 to 17 Decoding, Figures 18 to 27 Bitstream parsing, Fig.29 Decoding, Fig.30 and Fig.31 Decoding, Figure 32 to Figure 35 Bitstream parsing, Fig.37 The decoding and other bit stream receiving operations.

[0536] S3901: According to the implementation scheme, the point cloud data receiving method may further include decoding the point cloud data.

[0537] The decoding operation according to an embodiment may include: Figure 1 The receiving device 10004, Figure 1 Receiver 10005, Figure 1 Point cloud video decoder 10006, Figure 2 Send 20002 / decode 20003 / render 20004, Fig.10 and Fig.11 The decoder, Fig.13 The receiving device Fig.14 Devices Figures 15 to 17 Decoding, Figures 18 to 27 Bitstream parsing, Fig.29 Decoding, Fig.30 and Fig.31 Decoding, Figure 32 to Figure 35 Bitstream parsing, Fig.37 The decoding etc. is performed based on the parameters to decode the point cloud data (geometry and attributes).

[0538] Reference Figure 1 According to an embodiment, a point cloud data sending method may include encoding point cloud data and sending a bit stream containing the point cloud data.

[0539] Reference Fig.15, with respect to the basic inter-frame structure, encoding of point cloud data may include predicting a current point in a current frame based on a point in a reference frame of the current frame containing the point cloud data, wherein the point in the reference frame may be searched based on the azimuth and laser ID of a point processed before the current point.

[0540] Reference Fig.16 ,Regarding the compact bounding boxes, the encoding of the point cloud data may include extracting objects from a current frame containing the point cloud data and a reference frame of the current frame, generating a bounding box containing the object, sorting the bounding boxes based on coordinate information associated with the bounding boxes, and generating a list of the bounding boxes.

[0541] Reference Fig.17 , with respect to connecting compact bounding boxes, encoding of point cloud data may include: searching for a bounding box having similar information to a bounding box included in a list of a current frame among bounding boxes included in a list of a reference frame; generating a matching list including the found bounding box; and removing the found bounding box from the list of the reference frame.

[0542] Reference Figure 18 to Figure 27 Regarding parameters related to the compact bounding box, the bitstream may include information indicating whether inter-frame prediction is performed based on an object of point cloud data, information indicating a reference point of a matching list, information indicating a matching criterion of the matching list, and information indicating the matching list.

[0543] Reference Fig.30 and Fig.31 , regarding the classification of dynamic / static objects and dynamic motion vectors of compact bounding boxes, the encoding of point cloud data may include classifying dynamic objects based on a difference in at least one of an azimuth, a radius, or a laser ID between matching bounding boxes from a list of bounding boxes of a reference frame and a list of bounding boxes of a current frame, and generating a motion vector of the dynamic object based on a difference in coordinate values ​​between a bounding box for the dynamic object of the reference frame and a bounding box connected to the dynamic object of the current frame.

[0544] Reference Figures 18 to 35 Regarding dynamic / static object parameters, the bitstream may include information indicating whether to estimate and compensate motion vectors based on dynamic objects of point cloud data, information indicating thresholds of dynamic objects, information indicating coordinate axes of dynamic objects, information indicating the number of dynamic objects, information indicating the positions of dynamic objects, and information indicating motion vectors of static objects.

[0545] The point cloud data transmission method according to the embodiment may be performed by a transmission device. The point cloud data transmission device may include: an encoder configured to encode the point cloud data; and a transmitter configured to transmit a bit stream containing the point cloud data. The point cloud data transmission device may include a memory and a processor connected to the memory. The processor may encode the point cloud data and transmit the bit stream containing the point cloud data.

[0546] The sending method may correspond to the receiving method. The receiving method may perform the inverse process of the sending method. The point cloud data receiving method may include receiving a bit stream containing point cloud data, and decoding the point cloud data.

[0547] Reference Fig.29 , decoding of point cloud data may include predicting a current point in a current frame containing point cloud data based on a point in a reference frame of the current frame, wherein the point in the reference frame may be searched based on an azimuth and a laser ID of a point processed before the current point.

[0548] Reference Fig.29 , decoding of the point cloud data may include extracting objects from a current frame containing the point cloud data and a reference frame of the current frame based on parameter information in the bitstream, and inter-predicting the point cloud data on a block-by-block basis based on a bounding box containing the object and a list of bounding boxes.

[0549] The decoding of point cloud data may include: based on parameter information in the bitstream, searching for a bounding box included in a list of reference frames for a bounding box having similar information to a bounding box included in a list of the current frame, and generating a matching list including the found bounding boxes, or based on the parameter information in the bitstream, performing inter-frame prediction on the point cloud data based on information about the list of bounding boxes.

[0550] The bitstream may include information indicating whether to perform inter-frame prediction based on an object of point cloud data, information indicating a reference point of a matching list, information indicating a matching criterion of the matching list, and information indicating the matching list.

[0551] Decoding of point cloud data may include: classifying dynamic objects based on parameter information in a bitstream, based on a difference in at least one of an azimuth, a radius, or a laser ID between matching bounding boxes from a bounding box list of a reference frame and a bounding box list of a current frame; and generating a motion vector for the dynamic object based on a difference in coordinate values ​​between a bounding box for the dynamic object of the reference frame and a bounding box connected to the dynamic object of the current frame.

[0552] The point cloud data receiving method may be executed by a receiving device. The receiving device may include a memory and a processor connected to the memory. The processor may receive a bit stream containing point cloud data and decode the point cloud data.

[0553] Therefore, the method / device according to the embodiment can increase the performance of predictive geometric coding or predictive attribute coding. Using compact bounding boxes, the relationship between points between frames can be analyzed, and areas containing meaningful points can be matched on a block-by-block basis. When performing inter-frame prediction, a list of compact bounding boxes represented as a list of compact bounding boxes can be generated. The list can represent the matching relationship between similar areas between frames. Using information from the list of matched similar areas between frames, regions and / or frame units can be connected between frames. The embodiment can perform frame prediction between similar areas and predict local motion, thereby achieving accurate local motion estimation, rather than simply sampling data or segmenting data in a structured form. Therefore, the prediction accuracy can be increased, thereby reducing residuals. Since points are not predicted on a separate point-by-point basis, but on the basis of each group of points (i.e., point sets), the encoding and decoding time can be effectively reduced.

[0554] In addition, by performing inter-frame matching on a per-region basis as a point set and predicting whether a region is a dynamic region or a static region based on the degree of motion, motion can be predicted and compensated according to the characteristics of each region. As a result, residual error can be reduced and accuracy can be improved.

[0555] In addition to storing in a mapping list core points with quantized azimuth angles for points and radius processed first for each laser ID in coding order during intra-frame prediction of the predicted geometry, embodiments may separate core information by region, and separate dynamic objects from static objects based on region information matched between regions across frames. Thus, local motion estimation may be performed on a block-by-block basis. In addition, embodiments may search for a box representing a specific core area for each segmentation unit or the entire frame after segmentation, rather than considering only an upper limit of the azimuth angle of a directly encoded point as a target for a prediction value during intra-frame prediction of the predicted geometry. A matching list may be configured based on highly correlated inter-frame box information. Dynamic objects and static objects may be separated, and local motion estimation may be performed on dynamic objects on a block-by-block basis.

[0556] The embodiments have been described from the perspective of the method and / or the apparatus, and the descriptions of the method and the apparatus may be applied to supplement each other.

[0557] Although the drawings are described separately for simplicity, new embodiments can be designed by merging the embodiments shown in the various figures. Designing a recording medium that can be read by a computer and records a program for executing the above-mentioned embodiments according to the needs of those skilled in the art also falls within the scope of the attached claims and their equivalents. The apparatus and method according to the embodiment may not be limited to the configuration and method of the above-mentioned embodiment. Various modifications may be made to the embodiment by selectively combining all or some of the embodiments. Although preferred embodiments are described with reference to the drawings, it will be understood by those skilled in the art that various modifications and changes may be made to the embodiment without departing from the spirit or scope of the present disclosure described in the attached claims. These modifications should not be understood separately from the technical ideas or viewpoints of the embodiments.

[0558] Various elements of the device of the embodiment can be implemented by hardware, software, firmware or a combination thereof. Various elements in the embodiment can be implemented by a single chip (e.g., a single hardware circuit). According to the embodiment, the components according to the embodiment can be implemented as separate chips respectively. According to the embodiment, at least one or more components of the device according to the embodiment may include one or more processors capable of executing one or more programs. One or more programs may execute any one or more operations / methods according to the embodiment or include instructions for executing them. The executable instructions for executing the method / operation of the device according to the embodiment may be stored in a non-transitory CRM or other computer program product configured to be executed by one or more processors, or may be stored in a temporary CRM or other computer program product configured to be executed by one or more processors. In addition, the memory according to the embodiment can be used as not only covering volatile memory (e.g., RAM), but also covering the concepts of non-volatile memory, flash memory and PROM. In addition, it can also be implemented in the form of a carrier wave (e.g., transmission via the Internet). In addition, the processor-readable recording medium can be distributed to a computer system connected via a network so that the processor-readable code can be stored and executed in a distributed manner.

[0559] In the present specification, the terms " / " and "," should be interpreted as indicating "and / or". For example, the expression "A / B" may mean "A and / or B". In addition, "A, B" may mean "A and / or B". In addition, "A / B / C" may mean "at least one of A, B and / or C". In addition, "A / B / C" may mean "at least one of A, B and / or C". In addition, in the present specification, the term "or" should be interpreted as indicating "and / or". For example, the expression "A or B" may mean 1) only A, 2) only B, or 3) both A and B. In other words, the term "or" used in this document should be interpreted as indicating "in addition or alternatively".

[0560] Terms such as first and second can be used to describe various elements of the embodiments. However, various components according to the embodiments should not be limited by the above terms. These terms are only used to distinguish one element from another element. For example, a first user input signal can be referred to as a second user input signal. Similarly, a second user input signal can be referred to as a first user input signal. The use of these terms should not be interpreted without departing from the scope of various embodiments. Both the first user input signal and the second user input signal are user input signals, but unless the context clearly stipulates otherwise, they do not mean the same user input signal.

[0561] The terms used to describe the embodiments are used to describe specific embodiments and are not intended to limit the embodiments. As used in the description of the embodiments and the claims, unless the context clearly stipulates otherwise, the singular form includes plural referents. The expression "and / or" is used to include all possible combinations of terms. Terms such as "including" or "having" are intended to indicate the presence of graphics, quantities, steps, elements and / or components, and should be understood as not excluding the possibility of additional presence of graphics, quantities, steps, elements and / or components. As used herein, conditional expressions such as "if" and "when" are not limited to optional situations, and are intended to be interpreted as performing related operations when specific conditions are met, or interpreting related definitions according to specific conditions.

[0562] The operations according to the embodiments described in this specification may be performed by a transmitting / receiving device including a memory and / or a processor according to the embodiments. The memory may store a program for processing / controlling the operations according to the embodiments, and the processor may control the various operations described in this specification. The processor may be referred to as a controller, etc. In an embodiment, the operations may be performed by firmware, software, and / or a combination thereof. The firmware, software, and / or a combination thereof may be stored in a processor or a memory.

[0563] The operations according to the above-described embodiments may be performed by a transmitting device and / or a receiving device according to the embodiments. The transmitting / receiving device includes: a transmitter / receiver configured to transmit and receive media data; a memory configured to store instructions (program code, algorithm, flow chart and / or data) for the process according to the embodiments; and a processor configured to control the operation of the transmitting / receiving device.

[0564] The processor may be referred to as a controller, etc., and may correspond to, for example, hardware, software, and / or a combination thereof. The operations according to the above-described embodiments may be performed by the processor. In addition, the processor may be implemented as an encoder / decoder for the operations of the above-described embodiments.

[0565] Invention Mode

[0566] As described above, the relevant contents are described in the best mode for carrying out the embodiment.

[0567] Industrial Applicability

[0568] As described above, the embodiments may be applied in whole or in part to a point cloud data transmitting / receiving device and system.

[0569] It will be apparent to those skilled in the art that various changes or modifications may be made to the embodiments within the scope of the embodiments.

[0570] Thus, it is intended that the embodiments cover the modifications and variations of this disclosure provided they come within the scope of the appended claims and their equivalents.

Claims

1. A method for sending point cloud data, the method The following steps are involved: Encoding point cloud data; and A bitstream containing the point cloud data is sent.

2. The method according to claim 1, in, The step of encoding the point cloud data comprises the following steps: predicting a current point in the current frame based on a point in a reference frame for the current frame containing the point cloud data, The point in the reference frame is searched based on the azimuth and laser ID of a point processed before the current point.

3. The method according to claim 1, in, The step of encoding the point cloud data comprises the following steps: Extracting an object from a current frame including the point cloud data and a reference frame for the current frame; generating a bounding box containing the object; and The bounding boxes are sorted based on coordinate information associated with the bounding boxes, and a list for the bounding boxes is generated.

4. The method according to claim 3, in, The step of encoding the point cloud data comprises the following steps: searching, among the bounding boxes included in the list for the reference frame, for a bounding box having similar information to the bounding box included in the list for the current frame; and Generate a list of matches including the found bounding boxes, Therein, the found bounding box is removed from the list for the reference frame.

5. The method according to claim 4, in, The bitstream includes: information indicating whether to perform inter-frame prediction on the point cloud data based on the object; information indicating a reference point for the match list; information indicating matching criteria for the match list; and Information indicating the matching list.

6. The method according to claim 4, in, The step of encoding the point cloud data comprises the following steps: classifying a dynamic object based on a difference in at least one of an azimuth, a radius, or a laser ID between a matched bounding box from among the bounding boxes included in the list for the reference frame and the bounding boxes included in the list for the current frame; and A motion vector for the dynamic object is generated based on a difference in coordinate values ​​between a bounding box for the dynamic object in the reference frame and a bounding box connected to the dynamic object in the current frame.

7. The method according to claim 1, in, The bitstream includes: information indicating whether to estimate and compensate a motion vector based on a dynamic object for the point cloud data; information indicating a threshold value for the dynamic object; information indicating coordinate axes for the dynamic object; information indicating the number of the dynamic objects; information indicating the position of the dynamic object; and Information indicating motion vectors for static objects.

8. A device for transmitting point cloud data, the device include: Memory; as well as a processor connected to the memory, Wherein, the processor is configured to: Encoding the point cloud data; and A bitstream containing the point cloud data is sent.

9. A method for receiving point cloud data, the method The following steps are involved: receiving a bitstream containing point cloud data; as well as The point cloud data is decoded.

10. The method according to claim 9, in, The step of decoding the point cloud data comprises the following steps: predicting a current point in the current frame based on a point in a reference frame for the current frame containing the point cloud data, The point in the reference frame is searched based on the azimuth and laser ID of a point processed before the current point.

11. The method according to claim 9, in, The step of decoding the point cloud data comprises the following steps: Extracting an object from a current frame containing the point cloud data and a reference frame for the current frame based on parameter information in the bitstream; and The point cloud data is inter-predicted on a block-by-block basis based on a bounding box containing the object and a list for the bounding box.

12. The method according to claim 11, in, The step of decoding the point cloud data comprises the following steps: searching, among the bounding boxes included in the list for the reference frame, for a bounding box having similar information to the bounding box included in the list for the current frame based on the parameter information in the bitstream, and generating a matching list including the found bounding box; or Based on the parameter information in the bitstream, the point cloud data is inter-predicted based on information about a list of bounding boxes.

13. The method according to claim 11, in, The bitstream includes: information indicating whether to perform inter-frame prediction on the point cloud data based on the object; information indicating a reference point for matching the list; information indicating matching criteria for the match list; and Information indicating the matching list.

14. The method according to claim 11, in, The step of decoding the point cloud data comprises the following steps: classifying a dynamic object based on a difference in at least one of an azimuth, a radius, or a laser ID between a matching bounding box from among the bounding boxes included in the list for the reference frame and the list for the current frame based on the parameter information in the bitstream; and A motion vector is generated for the dynamic object based on a difference in coordinate values ​​between a bounding box for the dynamic object in the reference frame and a bounding box connected to the dynamic object in the current frame.

15. A device for receiving point cloud data, the device include: Memory; as well as a processor connected to the memory, Wherein, the processor is configured to: receiving a bitstream containing point cloud data; as well as The point cloud data is decoded.

Citation Information

Cited By

  • Methods and devices for encoding and decoding a temporal sequence of multidimensional scenes and corresponding computer program.

    FR3170772A1

  • Methods and devices for coding and decoding a sequence of scenes, and corresponding computer programs

    WO2026139205A1