Point cloud data transmission method, point cloud data transmission device, point cloud data reception method, and point cloud data reception device

Abstract

A method for transmitting point cloud data according to the present invention includes a step of encoding point cloud data, and a step of transmitting a bitstream including the point cloud data. Also, an apparatus for transmitting point cloud data according to the present invention includes an encoder for encoding point cloud data, and a transmitter for transmitting a bitstream including the point cloud data.

Description

[Technical Field]

[0001] The embodiment relates to a method and apparatus for processing point cloud content. [Background technology]

[0002] Point cloud content is content represented using a point cloud, which is a collection of points belonging to a coordinate system that represents three-dimensional space. Point cloud content can represent three-dimensional media and is used to provide various services such as VR (Virtual Reality), AR (Augmented Reality), MR (Mixed Reality), and autonomous driving services. However, representing point cloud content requires tens of thousands to hundreds of thousands of point data points. Therefore, a method for efficiently processing this enormous amount of point data is required. [Overview of the project] [Problems that the invention aims to solve]

[0003] The present invention provides an apparatus and method for efficiently processing point cloud data. The present invention also provides a point cloud data processing method and apparatus for resolving latency and encoding / decoding complexity.

[0004] However, the scope of the present invention is not limited to the technical problems described above, but can be extended to other technical problems that a person skilled in the art can derive from all the content described herein. [Means for solving the problem]

[0005] To achieve the above technical objectives, the point cloud data transmission method according to the present invention includes the steps of: encoding point cloud data; and transmitting a bitstream containing point cloud data. The point cloud data reception method according to the present invention also includes the steps of: receiving a bitstream containing point cloud data; and decoding point cloud data. [Effects of the Invention]

[0006] The apparatus and method according to the present invention can process point cloud data with high efficiency.

[0007] The apparatus and method according to the present invention can provide a high-quality point cloud service.

[0008] The apparatus and method according to the present invention can provide point cloud content for providing general-purpose services such as VR services and autonomous driving services. [Brief explanation of the drawing]

[0009] The accompanying drawings are intended to aid in understanding the present invention (and its embodiments) and illustrate the invention together with the description relating to the invention. For a better understanding of the various aspects of the invention described below, the following description of the invention should be referred to in relation to the following drawings, which include portions corresponding to similar reference numerals in the accompanying drawings.

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

[0010] The desired aspects of the present invention will be described in detail with reference to the accompanying drawings. The following detailed description with reference to the accompanying drawings is intended to describe the desired aspects of the present invention rather than to show only embodiments that can be embodied by the present invention. The following detailed description includes details to provide a thorough understanding of the present invention. However, it will be apparent to those skilled in the art that the present invention can be carried out without such details.

[0011] Most of the terms used in this invention are common and widely used in the relevant field, but some have been arbitrarily selected by the applicant, and their meanings will be explained in detail below as needed. Therefore, this invention should be understood based on the intended meaning of the terms, rather than simply their names or meanings.

[0012] Figure 1 shows an example of a point cloud content provision system according to the present invention.

[0013] The point cloud content provision system shown in Figure 1 includes a transmission device 10000 and a reception device 10004. The transmission device 10000 and the reception device 10004 are capable of wireless communication to send and receive point cloud data.

[0014] The transmitting device 10000 according to the present invention secures, processes, and transmits point cloud video (or point cloud content). In the present invention, the transmitting device 10000 includes a fixed station, a base transceiver system (BTS), a network, AI (Artificial Intelligence) equipment and / or systems, robots, AR / VR / XR equipment and / or servers, etc. In the present invention, the transmitting device 10000 also includes equipment that communicates with base stations and / or other wireless equipment using wireless connection technology (e.g., 5G NR (New RAT), LTE (Long Term Evolution)), robots, vehicles, AR / VR / XR equipment, mobile devices, home appliances, IoT (Internet of Things) equipment, AI equipment / servers, etc.

[0015] The transmitting device 10000 according to the present invention includes a Point Cloud Video Acquisition unit 10001, a Point Cloud Video Encoder 10002, and / or a Transmitter (or communication module) 10003.

[0016] The point cloud video acquisition unit 10001 according to the present invention acquires point cloud video through processing steps such as capture, synthesis, or generation. Point cloud video is point cloud content represented by a point cloud, which is a collection of points located in three-dimensional space, and is called point cloud video data, etc. The point cloud video according to the present invention includes one or more frames. One frame represents a still image / picture. Therefore, point cloud video includes point cloud video / frames / pictures and is referred to as point cloud video, frame, or picture.

[0017] The point cloud video encoder 10002 according to the present invention encodes the acquired point cloud video data. The point cloud video encoder 10002 encodes the point cloud video data based on point cloud compression coding. The point cloud compression coding according to the present invention includes G-PCC (Geometry-based Point Cloud Compression) coding and / or V-PCC (Video-based Point Cloud Compression) coding or next-generation coding. Note that the point cloud compression coding according to the present invention is not limited to the present invention as described above. The point cloud video encoder 10002 outputs a bitstream containing the encoded point cloud video data. The bitstream contains not only the encoded point cloud video data but also signaling information related to the encoding of the point cloud video data.

[0018] The transmitter 10003 according to the present invention transmits a bitstream containing encoded point cloud video data. The bitstream according to the present invention is encapsulated in a file or segment (e.g., a streaming segment) and transmitted over various networks such as broadcast networks and / or broadband networks. Although not shown, the transmitting device 10000 includes an encapsulation unit (or encapsulation module) that performs encapsulation operations. In the present invention, the encapsulation unit is also included in the transmitter 10003. In the present invention, the file or segment is transmitted to the receiving device 10004 via the network or stored in a digital storage medium (e.g., USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc.). The transmitter 10003 according to the present invention can communicate wirelessly with the receiving device 10004 (or receiver 10005) via a network such as 4G, 5G, or 6G. The transmitter 10003 also performs necessary data processing operations via a network system (e.g., a communication network system such as 4G, 5G, or 6G). Furthermore, the transmitting device 10000 can also transmit encapsulated data using an on-demand method.

[0019] The receiving device 10004 according to the present invention includes a receiver 10005, a point cloud video decoder 10006, and / or a renderer 10007. In the present invention, the receiving device 10004 includes devices, robots, vehicles, AR / VR / XR devices, mobile devices, home appliances, IoT (Internet of Things) devices, AI devices / servers, etc., that communicate with base stations and / or other wireless devices using wireless connection technology (e.g., 5G NR (New RAT), LTE (Long Term Evolution)).

[0020] The receiver 10005 according to the present invention receives a bitstream containing point cloud video data or a file / segment in which the bitstream is encapsulated from a network or storage medium. The receiver 10005 performs necessary data processing operations according to the network system (e.g., a communication network system such as 4G, 5G, or 6G). The receiver 10005 according to the present invention decapsulates the received file / segment and outputs a bitstream. In the present invention, the receiver 10005 also includes a decapsulation unit (or decapsulation module) for performing the decapsulation operation. The decapsulation unit is also embodied as an element (or component) separate from the receiver 10005.

[0021] The point cloud video decoder 10006 decodes a bitstream containing point cloud video data. The point cloud video decoder 10006 can decode the point cloud video data using the encoded scheme (for example, the reverse process of the operation of the point cloud video encoder 10002). Therefore, the point cloud video decoder 10006 can decode the point cloud video data by performing point cloud decompression coding, which is the reverse process of point cloud compression. Point cloud decompression coding includes G-PCC coding.

[0022] Renderer 10007 renders the decoded point cloud video data. Renderer 10007 renders not only the point cloud video data but also the audio data to output the point cloud content. In this invention, renderer 10007 includes a display for displaying the point cloud content. In this invention, the display is not included in renderer 10007 but is embodied in a separate device or component.

[0023] In the drawing, the dotted arrows indicate the transmission path of feedback information obtained by the receiving device 10004. The feedback information is information that reflects the interaction with the user consuming the point cloud content and includes user information (e.g., head orientation information, viewport information, etc.). In particular, if the point cloud content is for a service that requires interaction with the user (e.g., an autonomous driving service), the feedback information can be transmitted to the content sender (e.g., the transmitting device 10000) and / or the service provider. In the embodiment, the feedback information can be used not only by the transmitting device 10000 but also by the receiving device 10004, and may or may not be provided.

[0024] The head orientation information according to the present invention is information about the position, direction, angle, and movement of the user's head. The receiving device 10004 according to the present invention calculates viewport information based on the head orientation information. The viewport information is information about the area of ​​the point cloud video that the user is looking at. The viewpoint is the point from which the user is looking at the point cloud video, and it means the center of the viewport area. That is, the viewport is an area centered on the viewpoint, and the size and shape of the area are determined by the FOV (Field of View). Therefore, in addition to the head orientation information, the receiving device 10004 can extract viewport information based on the vertical or horizontal FOV supported by the device. The receiving device 10004 also performs gaze analysis to confirm the user's point cloud consumption method, the point cloud video area that the user gazes at, and the gazing time. In the embodiment, the receiving device 10004 transmits feedback information including the results of the gaze analysis to the transmitting device 10000. The feedback information according to the present invention is obtained during the rendering and / or display process. The feedback information according to the present invention is obtained by one or more sensors included in the receiving device 10004. In this invention, the feedback information is also obtained by the renderer 10007 or another external element (or device, component, etc.). The dotted line shown in Figure 1 shows the transmission process of the feedback information obtained by the renderer 10007. The point cloud content provision system processes (encodes / decodes) the point cloud data based on the feedback information. Therefore, the point cloud video data decoder 10006 can perform decoding operations based on the feedback information. The receiving device 10004 can also transmit the feedback information to the transmitting device 10000. The transmitting device 10000 (or point cloud video data encoder 10002) performs encoding operations based on the feedback information.Therefore, the point cloud content provision system does not process (encode / decode) all point cloud data, but can efficiently process the necessary data (for example, point cloud data corresponding to the user's head position) based on feedback information and provide point cloud content to the user.

[0025] In the present invention, the transmitting device 10000 is referred to as an encoder, transmitting device, transmitter, etc., and the receiving device 10004 is referred to as a decoder, receiving device, receiver, etc.

[0026] The point cloud data processed (processed in a series of steps including acquisition / encoding / transmission / decoding / rendering) by the point cloud content provision system shown in Figure 1 according to the present invention is also called point cloud content data or point cloud video data. In the present invention, point cloud content data is used as a concept that includes metadata or signaling information related to point cloud data.

[0027] The elements of the point cloud content delivery system shown in Figure 1 are embodied in hardware, software, processors, and / or combinations thereof.

[0028] Figure 2 is a block diagram illustrating the operation of point cloud content provision according to the present invention.

[0029] Figure 2 is a block diagram illustrating the operation of the point cloud content delivery system described in Figure 1. As mentioned above, the point cloud content delivery system processes point cloud data based on point cloud compression coding (e.g., G-PCC).

[0030] The point cloud content provision system according to the present invention (for example, a point cloud transmission device 10000 or a point cloud video acquisition unit 10001) acquires a point cloud video (20000). The point cloud video is represented by a point cloud belonging to a coordinate system that represents three-dimensional space. The point cloud video according to the present invention includes a Ply (Polygon File format or the Stanford Triangle format) file. If the point cloud video has one or more frames, the acquired point cloud video includes one or more Ply files. The Ply file includes point cloud data such as the geometry and / or attributes of the points. The geometry includes the position of the points. The position of each point is represented by parameters that indicate a three-dimensional coordinate system (for example, a coordinate system consisting of XYZ axes, etc.) (for example, the values ​​of the X, Y, and Z axes). The attributes include the attributes of the points (for example, the texture information, hue (YCbCr or RGB), reflectance (r), transparency, etc. of each point). A single point has one or more attributes (or attributes). For example, a single point may have one characteristic, hue, or two characteristics, hue and reflectance. In this invention, geometry is also called position, geometry information, geometry data, etc., and characteristics are also called characteristics, characteristic information, characteristic data, etc. Furthermore, a point cloud content provision system (for example, a point cloud transmission device 10000 or a point cloud video acquisition unit 10001) can secure point cloud data from information related to the point cloud video acquisition process (for example, depth information, hue information, etc.).

[0031] The point cloud content provision system according to the present invention (for example, the transmitting device 10000 or the point cloud video encoder 10002) encodes point cloud data (20001). The point cloud content provision system encodes point cloud data based on point cloud compression coding. As described above, the point cloud data includes the geometry and characteristics of points. Therefore, the point cloud content provision system outputs a geometry bitstream by performing geometry encoding to encode the geometry. The point cloud content provision system outputs a characteristics bitstream by performing characteristic encoding to encode the characteristics. In the present invention, the point cloud content provision system performs characteristic encoding based on geometry encoding. The geometry bitstream and characteristic bitstream according to the present invention are multiplexed and output as a single bitstream. The bitstream according to the present invention further includes signaling information related to geometry encoding and characteristic encoding.

[0032] The point cloud content provision system according to the present invention (for example, the transmitting device 10000 or transmitter 10003) transmits encoded point cloud data (20002). As illustrated in Figure 1, the encoded point cloud data is represented by a geometry bitstream and a characteristic bitstream. The encoded point cloud data is also transmitted in bitstream form along with signaling information related to the encoding of the point cloud data (for example, signaling information related to geometry encoding and characteristic encoding). The point cloud content provision system also encapsulates the bitstream that transmits the encoded point cloud data and transmits it in file or segment form.

[0033] The point cloud content provision system according to the present invention (for example, the receiving device 10004 or the receiver 10005) receives a bitstream containing encoded point cloud data. The point cloud content provision system (for example, the receiving device 10004 or the receiver 10005) also demultiplexes the bitstream.

[0034] The point cloud content delivery system (e.g., receiving device 10004 or point cloud video decoder 10005) decodes the encoded point cloud data (e.g., geometry bitstream, characteristic bitstream) transmitted in a bitstream. The point cloud content delivery system (e.g., receiving device 10004 or point cloud video decoder 10005) decodes the point cloud video data based on the signaling information related to the encoding of the point cloud video data contained in the bitstream. The point cloud content delivery system (e.g., receiving device 10004 or point cloud video decoder 10005) decodes the geometry bitstream to restore the point locations (geometry). The point cloud content delivery system decodes the characteristic bitstream based on the restored geometry to restore the point characteristics. The point cloud content delivery system (e.g., receiving device 10004 or point cloud video decoder 10005) restores the point cloud video based on the restored geometry locations and decoded characteristics.

[0035] The point cloud content provision system according to the present invention (e.g., receiving device 10004 or renderer 10007) renders the decoded point cloud data (20004). The point cloud content provision system (e.g., receiving device 10004 or renderer 10007) renders the decoded geometry and characteristics during the decoding process using various rendering methods. The points of the point cloud content are rendered as fixed points with a certain thickness, cubes with a predetermined minimum size centered at the position of the fixed points, or circles centered at the position of the fixed points. All or part of the rendered point cloud content is provided to the user by a display (e.g., a VR / AR display, a general display, etc.).

[0036] The point cloud content provision system according to the present invention (for example, the receiving device 10004) can secure feedback information (20005). The point cloud content provision system encodes and / or decodes point cloud data based on the feedback information. The operation of the feedback information and the point cloud content provision system according to the present invention is the same as the feedback information and operation described in Figure 1, so a detailed explanation is omitted.

[0037] Figure 3 shows an example of the point cloud video capture process according to the present invention.

[0038] Figure 3 shows an example of the point cloud video capture process of the point cloud content provision system described in Figures 1 and 2.

[0039] Point cloud content includes point cloud videos (images and / or footage) that represent objects and / or environments located in various three-dimensional spaces (e.g., a three-dimensional space representing a real environment, a three-dimensional space representing a virtual environment, etc.). Accordingly, the point cloud content provisioning system according to the present invention captures point cloud videos using one or more cameras (e.g., an infrared camera capable of securing depth information, an RGB camera capable of extracting hue information corresponding to depth information), a projector (e.g., an infrared pattern projector for securing depth information), LiDAR, etc., in order to generate point cloud content. The point cloud content provisioning system according to the present invention secures point cloud data by extracting the form of geometry composed of points in three-dimensional space from the depth information and extracting the characteristics of each point from the hue information. Images and / or footage according to the present invention are captured based on either an inward-facing or outward-facing method.

[0040] The left side of Figure 3 shows the inward-facing method. The inward-facing method is a method in which one or more cameras (or camera sensors) positioned around a central object capture the central object. The inward-facing method is used to generate point cloud content (for example, VR / AR content that provides the user with a 360° image of an object (e.g., a core object such as a character, athlete, item, or actor)) that provides the user with a 360° image of the core object.

[0041] The outward-facing method is shown on the right side of Figure 3. In the outward-facing method, one or more cameras (or camera sensors) positioned around a central object capture the environment of a central object that is not the central object itself. The outward-facing method is used to generate point cloud content (for example, content showing the external environment provided to a user of an autonomous vehicle) to provide the surrounding environment from the user's point of view.

[0042] As illustrated, point cloud content is generated based on the capture operation of one or more cameras. In this case, since the coordinate systems of each camera are different, the point cloud content provisioning system performs calibration of one or more cameras to set the global coordinate system before the capture operation. The point cloud content provisioning system also generates point cloud content by combining the images and / or videos captured using the capture method described above with arbitrary images and / or videos. Furthermore, when the point cloud content provisioning system generates point cloud content that represents a virtual space, it does not perform the capture operation described in Figure 3. The point cloud content provisioning system according to the present invention can also perform post-processing on the captured images and / or videos. That is, the point cloud content provisioning system can remove unwanted areas (e.g., background) or recognize the space in which the captured images and / or videos are linked and fill in any spatial holes if they exist.

[0043] Furthermore, the point cloud content provision system can generate a single point cloud content by performing coordinate system transformations on the points in the point cloud video captured from each camera. The point cloud content provision system performs coordinate system transformations on the points based on the position coordinates of each camera. As a result, the point cloud content provision system can generate content that shows a single wide area, or point cloud content with a high density of points.

[0044] Figure 4 shows an example of a point cloud encoder according to the present invention.

[0045] Figure 4 shows an example of the point cloud video encoder 10002 in Figure 1. The point cloud encoder reconstructs and encodes the point cloud data (e.g., point locations and / or characteristics) to adjust the quality of the point cloud content (e.g., lossless, lossy, near-lossless) depending on network conditions or applications. If the overall size of the point cloud content is large (e.g., 60Gbps for 30fps), the point cloud content provider system cannot stream the content in real time. Therefore, the point cloud content provider system can reconstruct the point cloud content based on the maximum target bitrate to provide it according to the network environment.

[0046] As shown in Figures 1 and 2, the point cloud encoder can perform both geometry coding and feature coding. Geometry coding is performed before feature coding.

[0047] The point cloud encoder according to the present invention includes a Transformation Coordinates unit 40000, a Quantize and Remove Points (Voxelize) unit 40001, an Octree Analysis unit 40002, an Analyze Surface Approximation unit 40003, an Arithmetic Encoder 40004, a Reconstruct Geometry unit 40005, a Transform Colors unit 40006, a Transfer Attributes unit 40007, a RAHT transformation unit 40008, a Generated LOD generation unit 40009, a Lifting transformation unit 40010, a Quantize Coefficients unit 40011, and / or an Arithmetic Encoder 40012.

[0048] The coordinate system transformation unit 40000, the quantization unit 40001, the octree analysis unit 40002, the surface approximation analysis unit 40003, the computational encoder 40004, and the geometry reconstruction unit 40005 perform geometry coding. Geometry coding according to the present invention includes octree geometry coding, direct coding, trisoup geometry coding, and entropy coding. Direct coding and trisoup geometry coding are applied selectively or in combination. Note that geometry coding is not limited to the examples given above.

[0049] As illustrated, the coordinate system transformation unit 40000 according to the present invention receives a position and converts it into a coordinate system. For example, the position is converted into position information in a three-dimensional space (for example, a three-dimensional space represented by an XYZ coordinate system). The position information in a three-dimensional space according to the present invention is also called geometric information.

[0050] The quantization unit 40001 according to the present invention quantizes geometry. For example, the quantization unit 40001 quantizes points based on the minimum position value of the total points (e.g., the minimum value on each axis for the X, Y, and Z axes). The quantization unit 40001 multiplies the difference between the minimum position value and the position value of each point by a predetermined quatization scale value, and then performs a quantization operation to find the closest integer value by rounding down or up. Therefore, one or more points can have the same quantized position (or position value). In order to reconstruct the quantized points, the quantization unit 40001 according to the present invention performs voxelization based on the quantized position. Just as the smallest unit containing 2D image / video information is a pixel, the points of the point cloud content (or 3D point cloud video) according to the present invention are contained in one or more voxels. A voxel is a combination of the words "volume" and "pixel," and refers to a three-dimensional cubic space that arises when a three-dimensional space is divided into units (unit = 1.0) based on axes that represent three-dimensional space (e.g., X-axis, Y-axis, Z-axis). The quantization unit 40001 matches groups of points in three-dimensional space using voxels. In this invention, one voxel contains only one point. In this invention, one voxel contains one or more points. Furthermore, in order to represent one voxel with one point, the position of the center point (ceter) of the voxel can be set based on the positions of one or more points contained in the voxel. In this case, the characteristics of all positions contained in the voxel are combined and assigned to the voxel.

[0051] The octree analysis unit 40002 according to the present invention performs octree geometry coding (or octree coding) to represent voxels as an octree structure. The octree structure represents points matched to voxels based on an octave structure.

[0052] The surface approximation analysis unit 40003 according to the present invention analyzes and approximates an octree. The octree analysis and approximation according to the present invention is a process of performing analysis to voxelize a region containing a large number of points in order to efficiently provide an octree and voxelization.

[0053] The arithmetic encoder 40004 according to the present invention entropy encodes an octree and / or an approximated octree. For example, the encoding scheme includes an arithmetic encoding method. A geometry bitstream is generated as a result of the encoding.

[0054] The color conversion unit 40006, the characteristic conversion unit 40007, the RAHT conversion unit 40008, the LOD generation unit 40009, the lift conversion unit 40010, the coefficient quantization unit 40011, and / or the arithmetic encoder 40012 perform characteristic coding. As described above, a single point has one or more characteristics. The characteristic coding according to the present invention is applied equally to the characteristics of a single point. However, if a single characteristic (e.g., hue) includes one or more elements, independent characteristic coding is applied to each element. The characteristic coding according to the present invention includes color conversion coding, characteristic conversion coding, RAHT (Region Adaptive Hierarchical Transform) coding, prediction transformation (Interpolaration-based hierarchical nearest-neighbour prediction-Prediction Transform) coding, and lift transformation (interpolaration-based hierarchical nearest-neighbour prediction with an update / lifting step (Lifting Transform)) coding. Depending on the point cloud content, the RAHT coding, predictive text coding, and lift coding described above may be used selectively, or a combination of one or more coding methods may be used. Furthermore, the characteristic coding according to the present invention is not limited to the examples given above.

[0055] The color conversion unit 40006 according to the present invention performs color conversion coding to convert the color values ​​(or textures) included in the characteristics. For example, the color conversion unit 40006 converts the format of the hue information (for example, from RGB to YCbCr). The operation of the color conversion unit 40006 according to the present invention is applied optionally depending on the color values ​​included in the characteristics.

[0056] The geometry reconstruction unit 40005 according to the present invention reconstructs (restores) an octree and / or an approximated octree. The geometry reconstruction unit 40005 reconstructs the octree / voxels based on the results of analyzing the distribution of points. The reconstructed octree / voxels are also called the reconstructed geometry (or restored geometry).

[0057] The characteristic transformation unit 40007 according to the present invention performs characteristic transformations based on positions where geometry coding has not been performed and / or on reconstructed geometry. As described above, since characteristics are dependent on geometry, the characteristic transformation unit 40007 can transform characteristics based on reconstructed geometry information. For example, the characteristic transformation unit 40007 transforms the characteristics of a point at a given position based on the position value of the point contained in the voxel. As described above, if the position of the center point of a voxel is set based on the positions of one or more points contained in that voxel, the characteristic transformation unit 40007 transforms the characteristics of one or more points. If trisoup geometry coding has been performed, the characteristic transformation unit 40007 transforms characteristics based on trisoup geometry coding.

[0058] The characteristic conversion unit 40007 performs characteristic conversion by calculating the average value of the characteristics or characteristic values ​​(e.g., hue or reflectance of each point) of points adjacent to the center point (or position value) of each voxel within a specific position / radius. When calculating the average value, the characteristic conversion unit 40007 applies a weighted value based on the distance from the center point to each point. Therefore, each voxel has a position and a calculated characteristic (or characteristic value).

[0059] The characteristic transformation unit 40007 searches for adjacent points within a specific position / radius from the position of the center point of each voxel based on a KD tree or Moulton code. The KD tree is a binary search tree that supports a data structure that manages points on a location basis, enabling rapid Nearest Neighbor Search (NNS). The Moulton code represents the coordinate values ​​(e.g., (x,y,z)) indicating the 3D position of all points as bit values, and is generated by mixing the bits. For example, if the coordinate value indicating the position of a point is (5,9,1), the bit values ​​of the coordinate value are (0101, 1001, 0001). Mixing the bit values ​​in the order of z, y, x according to the bit index gives 010001000111. This value in decimal is 1095. That is, the Moulton code value of the point with coordinate values ​​(5,9,1) is 1095. The characteristic conversion unit 40007 aligns points based on Moulton code values ​​and performs shortest neighbor search (NNS) using a depth-first traversal process. After the characteristic conversion operation, if shortest neighbor search (NNS) is required in other conversion processes for characteristic coding, the KD tree or Moulton code is utilized.

[0060] As shown in the diagram, the converted characteristics are input to the RAHT conversion unit 40008 and / or the LOD generation unit 40009.

[0061] The RAHT conversion unit 40008 according to the present invention performs RAHT coding to predict characteristic information based on reconstructed geometry information. For example, the RAHT conversion unit 40008 can predict characteristic information of nodes at higher levels of an octree based on characteristic information associated with nodes at lower levels of the octree.

[0062] The LOD generation unit 40009 according to the present invention generates LOD (Level of Detail) for predictive text coding. The LOD according to the present invention indicates the degree of detail of the point cloud content; a smaller LOD value indicates lower detail of the point cloud content, and a larger LOD value indicates higher detail of the point cloud content. Points can be classified by LOD.

[0063] The lift conversion unit 40010 according to the present invention performs lift conversion coding that converts the characteristics of the point cloud based on weighted values. As described above, the lift conversion coding is applied selectively.

[0064] The coefficient quantization unit 40011 according to the present invention quantizes the feature-coded feature based on the coefficient.

[0065] The arithmetic encoder 40012 according to the present invention encodes quantized properties based on arithmetic coding.

[0066] The point cloud encoder elements in Figure 4, although not shown, are embodied in hardware, software, firmware, or a combination thereof, including one or more processors or integrated circuits configured to communicate with one or more memories included in the point cloud provider. One or more processors can perform any one of the operations and / or functions of the point cloud encoder elements in Figure 4 described above. One or more processors can also operate or execute a set of software programs and / or instructions for performing the operations and / or functions of the point cloud encoder elements in Figure 4. One or more memories according to the present invention include high-speed random-access memory or non-volatile memory (e.g., one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state memory devices).

[0067] Figure 5 shows an example of a voxel according to the present invention.

[0068] Figure 5 shows voxels located in a three-dimensional space represented by a coordinate system consisting of three axes: the X, Y, and Z axes. As shown in Figure 4, a point cloud encoder (e.g., quantization unit 40001) performs voxelization. A voxel represents a three-dimensional cubic space that arises when three-dimensional space is divided into units (unit = 1.0) based on the axes that represent three-dimensional space (e.g., X, Y, Z axes). Figure 5 shows two poles (0,0,0) and (2 d , 2 d , 2 d This shows an example of a voxel generated by an octree structure that recursively subdivides a bounding box defined by ). Each voxel contains at least one point. The spatial coordinates of a voxel can be estimated from its positional relationship to a group of voxels. As mentioned above, voxels have properties (such as hue or reflectivity) similar to pixels in a 2D image / video. A detailed explanation of voxels is provided in Figure 4 and will be omitted here.

[0069] Figure 6 shows an example of an octree and occupancy code according to the present invention.

[0070] As shown in Figures 1 to 4, the point cloud content provision system (point cloud video encoder 10002) or point cloud encoder (e.g., octree analysis unit 40002) performs octree geometry coding (or octree coding) on ​​an octree structure basis in order to efficiently manage the area and / or location of voxels.

[0071] The upper part of FIG. 6 shows an octree structure. The three-dimensional space of the point cloud content according to the present invention is represented by the axes of a coordinate system (e.g., the X-axis, Y-axis, and Z-axis). The octree structure is recursively subdivided from a bounding box (cubical axis-aligned bounding box) defined by two extreme points (0, 0, 0) and (2 d 、2 d 、2 d ). 2d is set to a value that constitutes the smallest bounding box surrounding all the points of the point cloud content (or point cloud video). d indicates the depth of the octree. The d value is determined by the following formula. In the following formula, (x int n 、y int n 、z int n ) indicates the position (or position value) of the quantized point.

[0072]

Equation

[0073] As shown in the upper center of FIG. 6, by the division, the entire three-dimensional space is divided into eight spaces. Each of the divided spaces is represented by a cube having six faces. As shown in the upper right of FIG. 6, the eight spaces are each divided again by the axes of the coordinate system (e.g., the X-axis, Y-axis, and Z-axis). Thus, each space is divided into eight smaller spaces again. The divided smaller spaces are also represented by cubes having six faces. Such a division method is applied until the leaf nodes of the octree become voxels.

[0074] The lower part of Figure 6 shows the occupancy code of an octree. The occupancy code of an octree is generated to indicate whether each of the eight partitioned spaces resulting from the division of a single space contains at least one point. Therefore, one occupancy code is represented by eight child nodes. Each child node indicates the occupancy of the partitioned space, and each child node has a 1-bit value. Thus, the occupancy code is represented by an 8-bit code. That is, if the space corresponding to a child node contains at least one point, the node has a value of 1. If the space corresponding to a node does not contain a point (empty), the node has a value of 0. The occupancy code shown in Figure 6 is 00100001, which indicates that the spaces corresponding to the third and eighth child nodes each contain at least one point. As shown in the diagram, the third and eighth child nodes each have eight child nodes, and each child node is represented by an 8-bit occupancy code. The diagram shows that the occupation code of the third child node is 10000111 and the occupation code of the eighth child node is 01001111. The point cloud encoder according to the present invention (e.g., arithmetic encoder 40004) entropy encodes the occupation codes. In addition, to improve compression efficiency, the point cloud encoder intra / intercodes the occupation codes. The receiving device according to the present invention (e.g., receiving device 10004 or point cloud video decoder 10006) reconstructs the octree based on the occupation codes.

[0075] The point cloud encoder according to the present invention (for example, the point cloud encoder in Figure 4, or the octree analysis unit 40002) performs voxelization and octree coding to store the positions of points. However, since points in 3D space are not always uniformly distributed, there may be specific regions where there are few points. Therefore, performing voxelization on the entire 3D space is inefficient. For example, if there are almost no points in a particular region, there is no need to perform voxelization on that region.

[0076] Therefore, the point cloud encoder according to the present invention does not perform voxelization for the specific region (or nodes excluding leaf nodes of an octree) described above, and performs direct coding, which directly codes the positions of points included in the specific region. The coordinates of the directly coded points according to the present invention are called Direct Coding Mode (DCM). The point cloud encoder according to the present invention can also perform trisoup geometry encoding, which reconstructs the positions of points within a specific region (or node) based on voxels, based on a surface model. Trisoup geometry encoding is a geometry encoding that represents an object as a series of triangle meshes. Therefore, the point cloud decoder can generate a point cloud from a mesh surface. Direct coding and trisoup geometry encoding according to the present invention are performed selectively. Furthermore, direct coding and trisoup geometry encoding according to the present invention can be performed in combination with octree geometry coding (or octree coding).

[0077] For direct coding to be performed, the option to use direct mode for applying direct coding must be activated, and the node to which direct coding is applied must not be a leaf node, but rather a specific node containing points below a certain threshold. Furthermore, the total number of points to be directly coded must not exceed a predetermined threshold. When the above conditions are met, the point cloud encoder (or computation encoder 40004) according to the present invention can entropy code the position (or position value) of the points.

[0078] The point cloud encoder according to the present invention (e.g., surface approximation analysis unit 40003) can define a specific level of an octree (where the level is less than the depth d of the octree) and perform trisoup geometry coding from that level using a surface model to reconstruct the positions of points within the node region based on voxels (trisoup mode). The point cloud encoder according to the present invention can specify the level to which trisoup geometry coding is applied. For example, if the specified level is the same as the depth of the octree, the point cloud encoder does not operate in trisoup mode. That is, the point cloud encoder according to the present invention can only operate in trisoup mode if the specified level is less than the depth value of the octree. The three-dimensional cubic region of a node at a specified level according to the present invention is called a block. A block contains one or more voxels. A block or voxel can also correspond to a brick. Within each block, the geometry is represented as a surface. A surface according to the present invention can intersect each edge of a block at most once.

[0079] Since a block has 12 edges, there are at least 12 intersections within a block. Each intersection is called a vertex. A vertex along an edge is detected if, among all the blocks sharing that edge, there is at least one occupied voxel adjacent to that edge. In this invention, an occupied voxel means a voxel containing a point. The position of a vertex detected along an edge is the average position along the edge of all voxels adjacent to that edge among all the blocks sharing that edge.

[0080] When a vertex is detected, the point cloud encoder according to the present invention entropy codes the starting point of the edge (x, y, z), the direction vector of the edge (Δx, Δy, Δz), and the vertex position value (relative position value within the edge). When trisoup geometry coding is applied, the point cloud encoder according to the present invention (e.g., geometry reconstruction unit 40005) generates the reconstructed geometry (reconstructed geometry) by performing triangle reconstruction, up-sampling, and voxelization processes.

[0081] Vertices located at the edges of a block determine the surface that passes through the block. The surface according to this invention is a non-planar polygon. In the process of triangle reconstruction, the surface represented by triangles is reconstructed based on the starting point of the edge, the direction vector of the edge, and the position value of the vertex. The process of triangle reconstruction is as follows: 1) Circle 1: Calculate the centroid value of each vertex, 2) Circle 2: Subtract the centroid value from the value of each vertex, and 3) Circle 3: Square the result to obtain a value that is the sum of all the results.

[0082]

number

[0083] The minimum value of the added elements is found, and a projection process is performed along an axis where the minimum value is located. For example, if the x-element is the minimum, each vertex is projected onto the x-axis with respect to the center of the block, and then projected onto the (y,z) plane. If the value obtained by projecting onto the (y,z) plane is (ai,bi), the θ value is found using atan2(bi,ai), and the vertices are aligned based on the θ value. The following table shows the combinations of vertices for generating triangles depending on the number of vertices. The vertices are aligned sequentially from 1 to n. The following table shows that for 4 vertices, two triangles are formed by the combination of vertices. The first triangle is composed of the 1st, 2nd, and 3rd vertices from the aligned vertices, and the second triangle is composed of the 3rd, 4th, and 1st vertices from the aligned vertices.

[0084] Triangles formed from vertices ordered 1,…,n

[0085] n triangles

[0086] 3(1,2,3)

[0087] 4(1,2,3), (3,4,1)

[0088] 5(1,2,3), (3,4,5), (5,1,3)

[0089] 6(1,2,3), (3,4,5), (5,6,1), (1,3,5)

[0090] 7(1,2,3), (3,4,5), (5,6,7), (7,1,3), (3,5,7)

[0091] 8(1,2,3), (3,4,5), (5,6,7), (7,8,1), (1,3,5), (5,7,1)

[0092] 9(1,2,3), (3,4,5), (5,6,7), (7,8,9), (9,1,3), (3,5,7), (7,9,3)

[0093] 10(1,2,3), (3,4,5), (5,6,7), (7,8,9), (9,10,1), (1,3,5), (5,7,9), (9,1,5)

[0094] 11(1,2,3), (3,4,5), (5,6,7), (7,8,9), (9,10,11), (11,1,3), (3,5,7), (7,9,11), (11,3,7)

[0095] 12(1,2,3), (3,4,5), (5,6,7), (7,8,9), (9,10,11), (11,12,1), (1,3,5), (5,7,9), (9,11,1), (1,5,9)

[0096] The upsampling process is performed to add intermediate points along the edges of a triangle and voxelize it. Additional points are generated based on the upsampling factor and the width of the block. These additional points are called refined vertices. The point cloud encoder according to the present invention can voxelize refined vertices. The point cloud encoder can also perform feature coding based on the voxelized positions (or position values).

[0097] Figure 7 shows an example of an adjacent node pattern according to the present invention.

[0098] To increase the compression efficiency of point cloud video, the point cloud encoder according to the present invention performs entropy coding based on context adaptive arithmetic coding.

[0099] As explained in Figures 1 to 6, the point cloud content delivery system or point cloud encoder (e.g., point cloud video encoder 10002, point cloud encoder in Figure 4, or arithmetic encoder 40004) immediately entropy codes the occupied code. The point cloud content delivery system or point cloud encoder also performs entropy coding (intra coding) based on the occupied code of the current node and the occupied rate of neighboring nodes, or performs entropy coding (inter coding) based on the occupied code of a previous frame. A frame according to the present invention means a collection of point cloud videos generated at the same time. The compression efficiency of intra coding / inter coding according to the present invention varies depending on the number of neighboring nodes referenced. While it becomes more complex as the number of bits increases, the compression efficiency can be improved by tilting it to one side. For example, with a 3-bit context, coding is performed in 8 ways, which is 2 to the power of 3. The parts that are coded separately affect the complexity of implementation. Therefore, it is necessary to match the appropriate level of compression efficiency with the complexity.

[0100] Figure 7 shows the process of determining the occupancy pattern based on the occupancy rate of neighboring nodes. The point cloud encoder according to the present invention determines the occupancy rate of neighboring nodes for each node in the octvine and obtains a neighbor pattern value. The neighbor pattern is used to infer the occupancy pattern of the node in question. The left side of Figure 7 shows the cube corresponding to the node (the cube located in the middle) and the six cubes (neighboring nodes) that share at least one face with the cube in question. The illustrated nodes are nodes of the same depth. The illustrated numbers indicate the weight values ​​(1, 2, 4, 8, 16, 32, etc.) associated with each of the six nodes. Each weight value is assigned sequentially according to the position of the neighboring node.

[0101] The right side of Figure 7 shows the adjacent node pattern value. The adjacent node pattern value is the sum of the weighted values ​​of the occupied adjacent nodes (adjacent nodes with points). Therefore, the adjacent node pattern value can range from 0 to 63. An adjacent node pattern value of 0 means that none of the adjacent nodes to that node have points (occupied nodes). An adjacent node pattern value of 63 means that all adjacent nodes are occupied nodes. As shown in the figure, adjacent nodes assigned weights of 1, 2, 4, and 8 are occupied nodes, so the adjacent node pattern value is 15, which is the sum of 1, 2, 4, and 8. The point cloud encoder can perform coding based on the adjacent node pattern value (for example, if the adjacent node pattern value is 63, it will perform 64 codings). In this invention, the point cloud encoder can reduce the complexity of coding by changing the adjacent node pattern value (for example, based on a table that changes 64 to 10 or 6).

[0102] Figure 8 shows an example of a point configuration for each LOD according to the present invention.

[0103] As explained in Figures 1 through 7, before feature coding is performed, the coded geometry is reconstructed (restored). When direct coding is applied, the geometry reconstruction operation involves changing the placement of the directly coded points (for example, placing the directly coded points in front of the point cloud data). When trisoup geometry coding is applied, the geometry reconstruction process involves triangulation reconstruction, upsampling, and voxelization, and since features are dependent on geometry, feature coding is performed based on the reconstructed geometry.

[0104] The point cloud encoder (e.g., LOD generation unit 40009) reorganizes the points according to their LODs. The diagram shows the point cloud content corresponding to the LODs. In the diagram, the left side shows the original point cloud content. The second from the left shows the distribution of points with the lowest LOD, and the far right shows the distribution of points with the highest LOD. That is, points with the lowest LOD have a sparse distribution, and points with the highest LOD have a fine distribution. In other words, as the LOD increases along the direction of the arrows shown at the bottom of the diagram, the spacing (or distance) between points becomes shorter.

[0105] Figure 9 shows an example of a point configuration for each LOD according to the present invention.

[0106] As explained in Figures 1 to 8, the point cloud content provision system or point cloud encoder (e.g., point cloud video encoder 10002, point cloud encoder in Figure 4, or LOD generation unit 40009) generates LODs. The LOD is generated by realigning points with a set of refinement levels based on a set LOD distance value (or a set of Euclidean distances). The LOD generation process is performed not only by the point cloud encoder but also by the point cloud decoder.

[0107] The upper part of Figure 9 shows an example of points (P0-P9) in a point cloud content distributed in 3D space. The original order in Figure 9 shows the order of points P0-P9 before LOD generation. The LOD-based order in Figure 9 shows the order of points after LOD generation. Points are rearranged according to LOD. Also, higher LODs include points belonging to lower LODs. As shown in Figure 9, LOD0 includes P0, P5, P4, and P2. LOD1 includes the points of LOD0 and P1, P6, and P3. LOD2 includes the points of LOD0 and LOD1 and P9, P8, and P7.

[0108] As explained in Figure 4, the point cloud encoder according to the present invention can selectively or in combination perform predictive transformation coding, lift transformation coding, and RAHT transformation coding.

[0109] The point cloud encoder according to the present invention generates predictors for points and performs predictive transformation coding to set the predictive characteristics (or predictive characteristic values) for each point. That is, N predictors are generated for N points. The predictors according to the present invention can calculate a weighted value (=1 / distance) based on the LOD value of each point, index information for neighboring points that exist within a distance set for each LOD, and the distance value to the neighboring point.

[0110] The predicted characteristic (or characteristic value) according to the present invention is set as the average value obtained by multiplying the characteristics (or characteristic values, e.g., hue, reflectance, etc.) of adjacent points set in the predictor for each point by a weight (or weight value) calculated based on the distance to each adjacent point. The point cloud encoder according to the present invention (e.g., coefficient quantization unit 40011) can quantize and inverse quantize the residual values ​​(also called residual characteristics, residual characteristic values, characteristic prediction residual values, etc.) obtained by subtracting the predicted characteristic (characteristic value) from the characteristic (characteristic value) of each point. The quantization process is as shown in the table below.

[0111] [Table 1]

[0112] [Table 2]

[0113] The point cloud encoder according to the present invention (e.g., arithmetic encoder 40012) entropy codes the quantized and dequantized residual values ​​as described above, if there are points adjacent to the predictor of each point. If there are no points adjacent to the predictor of each point, the point cloud encoder according to the present invention (e.g., arithmetic encoder 40012) does not perform the above process and entropy codes the characteristics of the point in question.

[0114] The point cloud encoder according to the present invention (for example, the lift conversion unit 40010) generates a predictor for each point, sets the calculated LOD in the predictor and registers adjacent points, sets a weighted value based on the distance to adjacent points and performs lift conversion coding. The lift conversion coding according to the present invention is similar to the measurement conversion coding described above, but differs in that it cumulatively applies a weighted value to the characteristic value. The process of cumulatively applying a weighted value to the characteristic value according to the present invention is as follows.

[0115] 1) Generate an array QW (QuantizationWeight) to store the weight values ​​of each point. The initial value of all elements in QW is 1.0. Add the value obtained by multiplying the QW value of the predictor index of the adjacent node registered in the predictor by the weight value of the current point's predictor.

[0116] 2) Lift prediction process: To calculate the predicted attribute value, subtract the weighted value obtained by multiplying the attribute value of the point by a weighted value from the existing attribute value.

[0117] 3) Create temporary arrays named updateweight and update, and initialize these temporary arrays to 0.

[0118] 4) The weight calculated for all predictors is further multiplied by the weight stored in the QW corresponding to the predictor index, and the resulting weight is accumulated and summed in the update weight array as the index of the adjacent node. The update array is then accumulated and summed by multiplying the characteristic value of the index of the adjacent node by the calculated weight.

[0119] 5) Lift update process: For all predictors, divide the characteristic value of the update array by the weight value of the update weight array of the predictor index, and add the existing characteristic value back to the divided value.

[0120] 6) For all predictors, the predicted characteristic value is calculated by further multiplying the characteristic value updated in the lift update process by the weighted value updated in the lift prediction process (stored in QW). The point cloud encoder according to the present invention (e.g., coefficient quantization unit 40011) quantizes the predicted characteristic value. The point cloud encoder (e.g., calculation encoder 40012) entropy codes the quantized characteristic value.

[0121] The point cloud encoder according to the present invention (e.g., RAHT transformation unit 40008) performs RAHT transformation coding, which predicts the characteristics of higher-level nodes using characteristics associated with lower-level nodes of an octree. RAHT transformation coding is an example of characteristic intracoding by an octree backward scan. The point cloud encoder according to the present invention scans from voxels to the entire area, and each step repeats a merging process up to the root node, aligning the voxels with larger blocks. The merging process according to the present invention is performed only for occupied nodes. The merging process is not performed for empty nodes, but is performed for the node immediately above the empty node.

[0122] The following equation represents the RAHT transformation matrix.

[0123]

number

[0124]

number

[0125] The gDC value, like the high-pass coefficient, is quantized and entropy-coded.

[0126] Figure 10 shows an example of a Point Cloud Decoder according to the present invention.

[0127] The point cloud decoder shown in Figure 10 is an example of the point cloud video decoder 10006 shown in Figure 1, and performs the same or similar operation as the point cloud video decoder 10006 described in Figure 1. As illustrated, the point cloud decoder receives a geometry bitstream and an attribute bitstream contained in one or more bitstreams. The point cloud decoder includes a geometry decoder and an attribute decoder. The geometry decoder performs geometry decoding on the geometry bitstream and outputs the decoded geometry. The attribute decoder performs attribute decoding based on the decoded geometry and attribute bitstream and outputs the decoded attributes. The decoded geometry and decoded attributes are used to decode the point cloud content.

[0128] Figure 11 shows an example of a Point Cloud Decoder according to the present invention.

[0129] The point cloud decoder shown in Figure 11 is an example of the point cloud decoder described in Figure 10, and performs the decoding operation, which is the reverse process of the encoding operation of the point cloud encoder described in Figures 1 to 9.

[0130] As explained in Figures 1 and 10, the point cloud decoder performs geometry decoding and characteristic decoding. Geometry decoding is performed before characteristic decoding.

[0131] The point cloud decoder according to the present invention includes an arithmetic decoder 11000, an octree synthesis unit 11001, a surface approximation synthesis unit 11002, a geometry reconstruction unit 11003, an inverse transform coordinates unit 11004, an arithmetic decoder 11005, an inverse quantization unit 11006, an RAHT transformation unit 11007, an LOD generation unit 11008, an inverse lifting unit 11009, and / or an inverse transform colors unit 11010.

[0132] The computation decoder 11000, octree synthesis unit 11001, surface approximation synthesis unit 11002, geometry reconstruction unit 11003, and coordinate system inverse transformation unit 11004 perform geometry decoding. Geometry decoding according to the present invention includes direct coding and trisoup geometry decoding. Direct coding and trisoup geometry decoding are applied selectively. Furthermore, geometry decoding is not limited to the above examples and is performed in the reverse process of geometry coding as described in Figures 1 to 9.

[0133] The arithmetic decoder 11000 according to the present invention decodes the received geometry bitstream based on arithmetic coding. The operation of the arithmetic decoder 11000 corresponds to the inverse process of the arithmetic encoder 40004.

[0134] The octree synthesis unit 11001 according to the present invention obtains an octree by acquiring an octree code from the decoded geometry bitstream (or the decoding result, information about the allocated geometry). A specific explanation of the octree code is shown in Figures 1 to 9.

[0135] The surface approximation synthesis unit 11002 according to the present invention synthesizes surfaces based on the decoded geometry and / or the generated octree when trisoup geometry coding is applied.

[0136] The geometry reconstruction unit 11003 according to the present invention regenerates geometry based on the surface and / or decoded geometry. As explained in Figures 1 to 9, direct coding and trisoup geometry coding are applied selectively. Therefore, the geometry reconstruction unit 11003 directly retrieves and adds the positional information of points to which direct coding has been applied. Also, when trisoup geometry coding is applied, the geometry reconstruction unit 11003 restores the geometry by performing the reconstruction operations of the geometry reconstruction unit 40005, such as triangle reconstruction, upsampling, and voxelization operations. The specific details are explained in Figure 6 and will be omitted here. The restored geometry includes a point cloud picture or frame that does not contain any features.

[0137] The coordinate system inverse transformation unit 11004 according to the present invention transforms the coordinate system based on the restored geometry to obtain the position of the point.

[0138] The computation decoder 11005, the inverse quantization unit 11006, the RAHT transform unit 11007, the LOD generation unit 11008, the inverse lift unit 11009, and / or the inverse color transform unit 11010 perform characteristic decoding as described in Figure 10. The characteristic decoding according to the present invention includes RAHT (Region Adaptive Hierarchical Transform) decoding, prediction transform (Interpolaration-based hierarchical nearest-neighbour prediction-Prediction Transform) decoding, and lift transform (interpolaration-based hierarchical nearest-neighbour prediction with an update / lifting step (Lifting Transform)) decoding. These three decoding methods may be used selectively, or a combination of one or more decoding methods may be used. Furthermore, the characteristic decoding according to the present invention is not limited to the examples described above.

[0139] The arithmetic decoder 11005 according to the present invention decodes the characteristic bitstream into arithmetic coding.

[0140] The inverse quantization unit 11006 according to the present invention inverse quantizes the decoded characteristic bitstream or the information about the characteristics obtained as a result of decoding, and outputs the inversely quantized characteristics (or characteristic values). Inverse quantization is selectively applied based on the characteristic coding of the point cloud encoder.

[0141] In the present invention, the RAHT conversion unit 11007, the LOD generation unit 11008, and / or the inverse lift unit 11009 process the reconstructed geometry and the inversely quantized properties. As described above, the RAHT conversion unit 11007, the LOD generation unit 11008, and / or the inverse lift unit 11009 selectively perform the corresponding decoding operation by encoding with a point cloud encoder.

[0142] The color inverse conversion unit 11010 according to the present invention performs inverse conversion coding to inversely convert the color values ​​(or textures) contained in the decoded characteristics. The operation of the color inverse conversion unit 11010 is performed selectively based on the operation of the color conversion unit 40006 of the point cloud encoder.

[0143] The elements of the point cloud decoder in Figure 11, although not shown, are embodied in hardware, software, firmware, or a combination thereof, including one or more processors or integrated circuits configured to communicate with one or more memories included in the point cloud provider. One or more processors perform any of the point cloud decoder element operations and / or functions described above in Figure 11. One or more processors also operate or execute a set of software programs and / or instructions for performing the point cloud decoder element operations and / or functions in Figure 11.

[0144] Figure 12 shows an example of a transmitting device according to the present invention.

[0145] The transmitting device shown in Figure 12 is an example of the transmitting device 10000 in Figure 1 (or the point cloud encoder in Figure 4). The transmitting device shown in Figure 12 performs any operation and method that is the same as or similar to the operation and encoding method of the point cloud encoder described in Figures 1 to 9. The transmitting device according to the present invention includes a data input unit 12000, a quantization processing unit 12001, a voxelization processing unit 12002, an octree occupy code generation unit 12003, a surface model processing unit 12004, an intra / intercoding processing unit 12005, an arithmetic coder 12006, a metadata processing unit 12007, a hue conversion processing unit 12008, a characteristic conversion processing unit (or attribute conversion processing unit) 12009, a prediction / lift / RAHT conversion processing unit 12010, an arithmetic coder 12011 and / or a transmitting processing unit 12012.

[0146] The data input unit 12000 according to the present invention receives or acquires point cloud data. The data input unit 12000 performs the same or similar operations and / or acquisition methods as the operation and / or acquisition method of the point cloud video acquisition unit 10001 (or the acquisition process 20000 shown in Figure 2).

[0147] The data input unit 12000, quantization processing unit 12001, voxelization processing unit 12002, octree occupy code generation unit 12003, surface model processing unit 12004, intra / intercoding processing unit 12005, and arithmetic coder 12006 perform geometry coding. The geometry coding according to the present invention is the same as or similar to the geometry coding described in Figures 1 to 9, so a detailed explanation is omitted.

[0148] The quantization processing unit 12001 according to the present invention quantizes geometry (for example, the position value of a point or position value). The operation and / or quantization of the quantization processing unit 12001 is the same as or similar to the operation and / or quantization of the quantization unit 40001 shown in Figure 4. A detailed explanation is provided in Figures 1 to 9.

[0149] The voxelization processing unit 12002 according to the present invention voxels the position values ​​of the quantized points. The voxelization processing unit 120002 performs the same or similar operations and / or processes as the quantization unit 40001 and / or voxelization process shown in Figure 4. A detailed explanation is provided in Figures 1 to 9.

[0150] The octree-occupied code generation unit 12003 according to the present invention performs octree coding based on the octree structure of the voxelized point positions. The octree-occupied code generation unit 12003 generates an occupied code. The octree-occupied code generation unit 12003 performs the same or similar operation and / or method as the point cloud encoder (or octree analysis unit 40002) described in Figures 4 and 6. A detailed explanation is provided in Figures 1 to 9.

[0151] The surface model processing unit 12004 according to the present invention performs trisoup geometry coding to reconstruct the positions of points within a specific region (or node) on a voxel basis based on a surface model. The surface model processing unit 12004 performs the same or similar operations and / or methods as the point cloud encoder (e.g., surface approximation analysis unit 40003) shown in Figure 4. A detailed explanation is provided in Figures 1 to 9.

[0152] The intra / intercoding processing unit 12005 according to the present invention performs intra / intercoding of point cloud data. The intra / intercoding processing unit 12005 performs coding that is the same as or similar to the intra / intercoding described in Figure 7. A specific explanation is as described in Figure 7. In the present invention, the intra / intercoding processing unit 12005 is included in the arithmetic coder 12006.

[0153] The arithmetic coder 12006 according to the present invention entropy encodes an octree and / or an approximated octree of point cloud data. For example, the encoding scheme includes an arithmetic encoding method. The arithmetic coder 12006 performs the same or similar operations and / or methods as the arithmetic encoder 40004.

[0154] The metadata processing unit 12007 according to the present invention processes metadata related to point cloud data, such as setting values, and provides it to necessary processing processes such as geometry coding and / or characteristic coding. The metadata processing unit 12007 according to the present invention also generates and / or processes signaling information related to geometry coding and / or characteristic coding. The signaling information according to the present invention is encoded separately from geometry coding and / or characteristic coding. The signaling information according to the present invention may also be interleaved.

[0155] The hue conversion processing unit 12008, the characteristic conversion processing unit 12009, the prediction / lift / RAHT conversion processing unit 12010, and the arithmetic coder 12011 perform characteristic coding. The characteristic coding according to the present invention is the same as or similar to the characteristic coding described in Figures 1 to 9, so a detailed explanation is omitted.

[0156] The hue conversion processing unit 12008 according to the present invention performs hue conversion coding to convert the hue values ​​included in the characteristics. The hue conversion processing unit 12008 performs hue conversion coding based on the reconstructed geometry. The reconstructed geometry is described in Figures 1 to 9. It also performs the same or similar operation and / or method as the color conversion unit 40006 described in Figure 4. A detailed explanation is omitted.

[0157] The characteristic transformation processing unit 12009 according to the present invention performs characteristic transformation based on positions where geometry coding has not been performed and / or the reconstructed geometry. The characteristic transformation processing unit 12009 performs the same or similar operations and / or methods as the characteristic transformation unit 40007 described in Figure 4. A detailed explanation is omitted. The predictive / lift / RAHT transformation processing unit 12010 according to the present invention codes the transformed characteristics using one or a combination of RAHT coding, predictive transformation coding, and lift transformation coding. The predictive / lift / RAHT transformation processing unit 12010 performs one or a similar operations as the RAHT transformation unit 40008, LOD generation unit 40009, and lift transformation unit 40010 described in Figure 4. Furthermore, the explanations regarding predictive transformation coding, lift transformation coding, and RAHT transformation coding are as described in Figures 1 to 9, so a detailed explanation is omitted.

[0158] The arithmetic coder 12011 according to the present invention encodes the coded characteristics based on the arithmetic coding. The arithmetic coder 12011 performs the same or similar operations and / or methods as the arithmetic encoder 400012.

[0159] The transmission processing unit 12012 according to the present invention transmits each bitstream containing encoded geometry and / or encoded attributes and metadata information, or transmits the encoded geometry and / or encoded attributes and metadata information as a single bitstream. When the encoded geometry and / or encoded attributes and metadata information according to the present invention is composed of a single bitstream, the bitstream contains one or more sub-bitstreams. The bitstream according to the present invention includes signaling information and slice data, including SPS (Sequence Parameter Set) for sequence-level signaling, GPS (Geometry Parameter Set) for geometry information coding signaling, APS (Attribute Parameter Set) for attribute information coding signaling, and TPS (Tile Parameter Set) for tile-level signaling. The slice data contains information about one or more slices. One slice according to the present invention is one geometry bitstream (Geom0 0 ) and one or more attribute bitstreams (Attr0 0 , Attr1 0 ) includes.

[0160] A slice is a series of syntax elements that represent all or part of a coded point cloud frame.

[0161] The TPS according to the present invention includes information about each tile for one or more tiles (e.g., coordinate information and height / size information of the bounding box). The geometry bitstream includes a header and a payload. The header of the geometry bitstream according to the present invention includes identification information of the parameter set included in the GPS (geom_parameter_set_id), a tile identifier (geom_tile_id), a slice identifier (geom_slice_id), and information about the data included in the payload. As described above, the metadata processing unit 12007 according to the present invention can generate and / or process signaling information and transmit it to the transmission processing unit 12012. In the present invention, the elements that perform geometry coding and the elements that perform characteristic coding can share data / information with each other, as shown by the dotted lines. The transmission processing unit 12012 according to the present invention performs the same or similar operation and / or transmission method as the operation and / or transmission method of the transmitter 10003. A detailed explanation is provided in Figures 1 and 2 and will be omitted here.

[0162] Figure 13 shows an example of a receiving device according to the present invention.

[0163] The receiving device shown in Figure 13 is an example of the receiving device 10004 in Figure 1 (or the point cloud decoder in Figures 10 and 11). The receiving device shown in Figure 13 performs one of the same or similar operations and decoding methods as those described for the point cloud decoder in Figures 1 to 11.

[0164] The receiving device according to the present invention includes a receiving unit 13000, a receiving processing unit 13001, an arithmetic decoder 13002, an occupancy code-based octree reconstruction processing unit 13003, a surface model processing unit (triangle reconstruction, upsampling, voxelization) 13004, an inverse quantization processing unit 13005, a metadata parser 13006, an arithmetic decoder 13007, an inverse quantization processing unit 13008, a prediction / lift / RAHT inverse conversion processing unit 13009, a hue inverse conversion processing unit 13010, and / or a renderer 13011. Each component of the decoding according to the present invention performs the inverse process of the component of the encoding according to the present invention.

[0165] The receiving unit 13000 according to the present invention receives point cloud data. The receiving unit 13000 performs the same or similar operation and / or reception method as the receiver 10005 in Figure 1. A detailed explanation is omitted.

[0166] The receiving processing unit 13001 according to the present invention obtains a geometry bitstream and / or characteristic bitstream from the received data. The receiving processing unit 13001 is included in the receiving unit 13000.

[0167] The arithmetic decoder 13002, the octree reconstruction processing unit 13003 of the occupied code base, the surface model processing unit 13004, and the inverse quantization processing unit 13005 perform geometry decoding. The geometry decoding according to the present invention is the same as or similar to the geometry decoding described in Figures 1 to 10, so a detailed explanation is omitted.

[0168] The arithmetic decoder 13002 according to the present invention decodes the geometry bitstream based on arithmetic coding. The arithmetic decoder 13002 performs the same or similar operations and / or coding as the arithmetic decoder 11000.

[0169] The octree reconstruction processing unit 13003 of the occupied code base according to the present invention obtains the occupied code from the decoded geometry bitstream (or the decoding result, information on the allocated geometry) and reconstructs the octree. The octree reconstruction processing unit 13003 of the occupied code base performs the same or similar operations and / or methods as the octree synthesis unit 11001 and / or the octree generation method. When trisoup geometry coding is applied, the surface model processing unit 13004 according to the present invention performs trisoup geometry decoding and related geometry reconstruction (e.g., triangle reconstruction, upsampling, voxelization) based on the surface model scheme. The surface model processing unit 13004 performs the same or similar operations as the surface approximation synthesis unit 11002 and / or the geometry reconstruction unit 11003.

[0170] The inverse quantization processing unit 13005 according to the present invention inversely quantizes the decoded geometry.

[0171] The metadata parser 13006 according to the present invention analyzes metadata contained in the received point cloud data, such as setting values. The metadata parser 13006 transmits the metadata to geometry decoding and / or characteristic decoding. A detailed explanation of the metadata is shown in Figure 12 and will be omitted here.

[0172] The computation decoder 13007, the inverse quantization processing unit 13008, the prediction / lift / RAHT inverse conversion processing unit 13009, and the hue inverse conversion processing unit 13010 perform characteristic decoding. Since the characteristic decoding is the same as or similar to the characteristic decoding described in Figure 1 or Figure 10, a detailed explanation is omitted.

[0173] The arithmetic decoder 13007 according to the present invention decodes the characteristic bitstream into arithmetic coding. The arithmetic decoder 13007 decodes the characteristic bitstream based on the reconstructed geometry. The arithmetic decoder 13007 performs the same or similar operation and / or coding as the arithmetic decoder 11005.

[0174] The inverse quantization processing unit 13008 according to the present invention inversely quantizes the decoded characteristic bitstream. The inverse quantization processing unit 13008 performs the same or similar operations and / or methods as the operations and / or methods of the inverse quantization unit 11006.

[0175] The prediction / lift / RAHT inverse conversion processing unit 13009 according to the present invention processes the reconstructed geometry and inversely quantized features. The prediction / lift / RAHT inverse conversion processing unit 13009 performs any operation and / or decoding that is the same as or similar to the operation and / or decoding of the RAHT conversion unit 11007, the LOD generation unit 11008 and / or the inverse lift unit 11009. The hue inverse conversion processing unit 13010 according to the present invention performs inverse conversion coding to inversely convert the color values ​​(or textures) contained in the decoded features. The hue inverse conversion processing unit 13010 performs any operation and / or inverse conversion coding that is the same as or similar to the operation and / or inverse conversion coding of the color inverse conversion unit 11010. The renderer 13011 according to the present invention renders the point cloud data.

[0176] Figure 14 shows an example of a structure that can be linked to the point cloud data transmission and reception method / device according to the present invention.

[0177] The structure in Figure 14 shows a configuration in which a server 1460, a robot 1410, an autonomous vehicle 1420, an XR device 1430, a smartphone 1440, a home appliance 1450, and / or an HMD 1470 are connected to the cloud network 1410. The robot 1410, autonomous vehicle 1420, XR device 1430, smartphone 1440, or home appliance 1450 are also referred to as devices. The XR device 1430 corresponds to or is linked to a point cloud data (PCC) device according to the present invention.

[0178] Cloud network 1400 refers to a network that constitutes part of the cloud computing infrastructure or resides within the cloud computing infrastructure. Here, cloud network 1400 is configured using a 3G network, 4G or LTE network, or 5G network, etc.

[0179] Server 1460 is connected to any of the robots 1410, autonomous vehicles 1420, XR devices 1430, smartphones 1440, home appliances 1450, and / or HMDs 1470 via a cloud network 1400, and can assist in at least part of the processing of the connected devices 1410 to 1470.

[0180] HMD (Head-Mount Display) 1470 refers to any type of device embodying the XR device and / or PCC device according to the present invention. The HMD type device according to the present invention includes a communications unit, a control unit, a memory unit, an I / O unit, a sensor unit, and a power supply unit, etc.

[0181] The following describes various aspects of the present invention, specifically devices 1410 to 1450 to which the above technology is applied. Here, devices 1410 to 1450 shown in Figure 14 can be linked / coupled with the point cloud data transmission / reception device according to the present invention described above.

[0182] <PCC+XR>

[0183] The XR / PCC device 1430 can be implemented using PCC and / or XR (AR+VR) technology and can be embodied in HMDs (Head-Mount Displays), HUDs (Head-Up Displays) installed in vehicles, TVs, mobile phones, smartphones, computers, wearable devices, home appliances, digital signage, vehicles, stationary robots, and mobile robots.

[0184] The XR / PCC device 1430 can obtain information about the surrounding space or real-world objects by analyzing 3D point cloud data or image data acquired by various sensors or external devices to generate positional data and characteristic data for 3D points, and then render and output an XR object to be output. For example, the XR / PCC device 1430 can output an XR object that includes additional information about the recognized object, corresponding to the recognized object.

[0185] <PCC+XR+Mobile Phone>

[0186] The XR / PCC device 1430 is implemented using PCC technology and is embodied in devices such as the mobile phone 1440.

[0187] The mobile phone 1440 decodes and displays point cloud content based on PCC technology.

[0188] <PCC+Autonomous Driving+XR>

[0189] The autonomous vehicle 1420 will be implemented using PCC technology and XR technology, and will be realized as mobile robots, vehicles, and unmanned aerial vehicles.

[0190] The autonomous vehicle 1420 to which XR / PCC technology is applied refers to an autonomous vehicle equipped with means for providing XR images, or an autonomous vehicle that is the target of control / interaction within the XR images. In particular, the autonomous vehicle 1420 that is the target of control / interaction within the XR images is separate from the XR device 1430 and works in conjunction with it.

[0191] An autonomous vehicle 1420 equipped with means for providing XR / PCC images obtains sensor information from sensors including cameras and outputs XR / PCC images generated based on the obtained sensor information. For example, by providing a HUD and outputting XR / PCC images, the autonomous vehicle 1420 can provide the occupant with XR / PCC objects corresponding to real-world objects or objects on the screen.

[0192] In this case, when the XR / PCC object is output to the HUD, at least a portion of the XR / PCC object is output so as to overlap with the actual object that the occupant is looking at. On the other hand, when the XR / PCC object is output to a display installed in the autonomous vehicle, at least a portion of the XR / PCC object is output so as to overlap with the object on the screen. For example, the autonomous vehicle 1220 can output XR / PCC objects that correspond to objects such as roads, other vehicles, traffic lights, traffic signs, motorcycles, pedestrians, and buildings.

[0193] The VR (Virtual Reality) technology, AR (Augmented Reality) technology, MR (Mixed Reality) technology, and / or PCC (Point Cloud Compression) technology according to the present invention are applicable to a variety of devices.

[0194] In other words, VR technology is a display technology that provides real-world objects and backgrounds using only computer graphics (CG). On the other hand, AR technology is a technology that displays virtual CG images alongside images of real things. MR technology is similar to AR technology in that it mixes virtual objects with the real world. However, in AR technology, the distinction between real objects and virtual objects consisting of CG images is clear, and virtual objects are used in a way that complements real objects. In contrast, MR technology is distinguished from AR technology in that virtual and real objects are considered to have similar characteristics. More specifically, for example, hologram services are an application of the MR technology described above.

[0195] However, recently, rather than clearly distinguishing between VR, AR, and MR technologies, they are also referred to as XR (extended reality) technologies. Therefore, the present invention is applicable to any of VR, AR, MR, and XR technologies. Such technologies are based on PCC, V-PCC, and G-PCC encoding / decoding technologies.

[0196] The PCC method / apparatus according to the present invention can be applied to vehicles that provide autonomous driving services.

[0197] Vehicles providing autonomous driving services are connected to PCC devices via wired and wireless communication.

[0198] The point cloud data (PCC) transceiver according to the present invention, when wirelessly connected to a vehicle, can receive and process AR / VR / PCC service-related content data that can be provided along with autonomous driving services and transmit it to the vehicle. Furthermore, when the point cloud data transceiver is mounted on a vehicle, it can receive and process AR / VR / PCC service-related content data based on user input signals received by a user interface device and provide it to the user. The vehicle or user interface device according to the present invention receives user input signals. These user input signals include signals instructing autonomous driving services.

[0199] As explained in Figures 1 through 14, point cloud data consists of a collection of points, and each point has geometric data (geometric information) and characteristic data (characteristic information). Geometric data is the three-dimensional position information of each point (for example, the coordinate values ​​of the x, y, and z axes). That is, the position of each point is represented by parameters on a coordinate system representing three-dimensional space (for example, the parameters of the three axes representing space, the X, Y, and Z axes (x, y, z)). Characteristic information refers to the hue (RGB, YUV, etc.), reflectance, normal vectors, transparency, etc. of the point. Characteristic information is represented in the form of a scalar or vector.

[0200] According to the present invention, point cloud data is classified into three categories based on the type and acquisition method of the point cloud data: Category 1 for static point cloud data, Category 2 for dynamic point cloud data, and Category 3 for data obtained while moving dynamically. Category 1 consists of a single-frame point cloud with a high point density relative to an object or space. Category 3 data is further divided into frame-based data having multiple frames obtained while moving, and fused data of a single frame in which a point cloud obtained from a LiDAR sensor and a hue image obtained from 2D video are matched for a large space.

[0201] According to the present invention, inter-prediction coding / decoding is used to efficiently compress 3D point cloud data having multiple frames over time, such as frame-based point cloud data with a large number of frames. Inter-prediction coding / decoding can be applied to geometric information and / or characteristic information. Inter-prediction is inter-screen or inter-frame prediction, while intra-prediction is intra-screen prediction.

[0202] The following describes motion estimation methods related to interpretation technology.

[0203] Interpretation uses information from encoded frames that has a high similarity to the current frame to predict all or part of the current frame. The unit used to determine similarity is called a prediction unit (PU). Interpretation is a prediction method that uses the movement between multiple frames obtained over time, and this technique is called motion vector estimation or motion estimation.

[0204] Motion detection is divided into global motion estimation, which rotates, moves, enlarges, and / or shrinks the entire frame, and local motion estimation, which predicts the movement of only a part of the frame.

[0205] Motion detection divides the current frame's space into prediction units (PUs), extracts points within the corresponding regions, extracts points within the corresponding regions of the reference frame, and then calculates the distance and / or vector difference between the extracted points in the current frame and the extracted points in the reference frame. At this time, a vector difference smaller than a specific value is set as the target vector difference.

[0206] Furthermore, for a set of unit vectors consisting of many unit vectors with different directions, it is possible to search for the vector (unit vector x amount) that is closest to (similar to) the target vector difference by multiplying each unit vector by an arbitrary value called amount.

[0207] During this process, the amount gradually decreases from an initial arbitrary value to a predetermined minimum value, while the aforementioned similar vector search process is repeated multiple times. The vector finally found through the repeated search process is set as the best vector, and the cost is calculated using the RDO (Rate Distortion Optimization) method.

[0208] A prediction unit (PU) can be divided into numerous sub-prediction units (sub-PUs), and the best vector search process described above is performed on each of the divided sub-prediction units (sub-PUs). If the sum of the RDO costs calculated for the divided sub-prediction units (sub-PUs) is smaller than the RDO cost value of the prediction unit (PU), the prediction unit (PU) is ultimately divided into sub-prediction units; otherwise, the best vector of the prediction unit (PU) is signaled.

[0209] On the other hand, the present invention disclosed in this specification relates to the encoding and decoding of point cloud data, and is an apparatus / method that improves the transmission and reception efficiency of point cloud data by exploring the relationships between multiple frames and sequentially encoding / decoding the context of the reference frame during encoding / decoding. The point cloud data transmission and reception apparatus / method according to the present invention improves the bitstream compression efficiency using information related to predictions between frames.

[0210] Figure 15 shows an example of a Group of Frames (GoF) of point cloud data according to the present invention. A Group of Frames (GoF) represents a group of frames and is also called a Group of Pictures (GoP). The frames that make up a GoF include intra frames (I frames), predicted frames (P frames), and / or bidirectional predicted frames (B frames). An intra frame (I frame) is a frame that does not refer to other frames. Since the I frame is the first frame in the GoF, there are no previous frames and it does not refer to other frames. A predicted frame (P frame) is a frame predicted by referring to a previous I frame or other P frames. A bidirectional predicted frame (B frame) is a frame predicted by referring to I frames or P frames in both directions. For example, if the second frame among the 0th to 3rd frames all refers to the 1st and 3rd frames, then the second frame is a bidirectional predicted frame. A GoF consists of I frames, P frames, and / or B frames, and a GoF group starting with an I frame can be defined in various forms. Here, the reference frame is the frame referenced for interpretation. A GoF is a group of frames starting with the I-frame and forms the unit of encoding / decoding.

[0211] The point cloud data transmission / receiving device according to the present invention can search for inter-frame relationships and replace signaling with overlapping information during prediction between frames, and can use a context continuity flag using index information of frames having similar geometry / characteristic contexts.

[0212] There are two methods for compressing point cloud data: intra-frame coding and inter-frame coding. Category 3 point cloud data is data that can be applied to inter-frame coding and consists of a set of consecutive frames. Such a set of frames is defined as a Group of Frames (GoF), and depending on the definition method, the GoF can take the form of IPPP, IPPBPP, IBBPBBB, etc., and be encoded or decoded.

[0213] For example, Figure 15 shows a Group of Frames (GoF) in the IPPP configuration. Of the two GoFs shown in Figure 15, the GoF on the left shows that P frames refer to the previous frame. That is, the first P frame refers to the 0th I frame, and the second P frame refers to the first P frame. The GoF shown on the right of Figure 15 shows that the first through third P frames refer to the 0th I frame. That is, P frames can refer to previous frames and can refer to the first I frame. Alternatively, P frames can refer to I frames or specific P frames using frame index information.

[0214] Point cloud data consists of a collection of points, each containing geometric information and characteristic information. The encoding process of point cloud data involves compressing the geometry and then compressing the characteristic information based on the reconstructed geometry (decoded geometry) which is reconstructed using the positional information modified by the compression. Decoding of point cloud data proceeds by transmitting the encoded geometry bitstream and characteristic bitstream, first decoding the geometry, and then decoding the characteristic information based on the geometry reconstructed by the decoding.

[0215] The point cloud data transmission / receiving device according to the present invention proposes a signaling method for relevant information, such as whether or not context continuity is applied, during point cloud data encoding / decoding. The point cloud data transmission / receiving device according to the present invention searches for interrelationships between frames to derive related reference frames and encodes / decodes point cloud data based on the context continuity of the reference frames within the GoF. Furthermore, according to the present invention, a method of signaling the index of the reference frame is presented as an encoding / decoding method between frames using context continuity. The point cloud data transmission / receiving device according to the present invention proposes a GoF frame structure to improve the compression efficiency of point cloud data and proposes searching for interrelationships between frames within the GoF. Furthermore, the present invention proposes encoding / decoding data with context continuity of reference frames applied within the GoF and proposes a method of signaling the index of the reference frame in relation to the application of context continuity. Here, the reference frame indicates a frame that is related to the current frame.

[0216] Applying context continuity to a GoF frame means using context information (context model or arithmetic model, probability interval information) of related reference frames consecutively during encoding / decoding. The point cloud data transmitter / receiver according to the present invention searches for connections between the current frame and other frames during point cloud data encoding / decoding, and can use the context of related frames, i.e., reference frames, consecutively. Whether or not context continuity can be used (whether or not there is inter-frame connection) is defined internally or externally to the GoF structure. The transmitter / receiver according to the present invention can encode / decode the current frame using the context information of the reference frame if there is a reference frame related to the current frame during encoding / decoding of the current frame. That is, it can encode / decode the current frame by applying context continuity.

[0217] The above-mentioned coding will be explained below.

[0218] Arithmetic coding is one of the entropy coding algorithms used for lossless compression. Arithmetic coding can replace the entire message with a single real number n.

[0219] Arithmetic coding allows us to divide the 0.0 to 1.0 interval based on the probability of symbols appearing in a message. For example, if the probability of the symbol "a" appearing in a message is 60%, the probability of the symbol "b" appearing is 20%, and the probability of the symbol "c" appearing is 20%, then in the initial stage, the 0.0 to 1.0 interval can be divided into the 0.0 to 0.6 interval representing the probability of the symbol "a", the 0.6 to 0.8 interval representing the probability of the symbol "b", and the 0.8 to 1.0 interval representing the probability of the symbol "c". In this case, the ratio used to divide probability intervals like 0.0 to 1.0 is called a probability model (or probability mode) or context model.

[0220] For example, when arithmetic coding the symbol "abc", the first symbol to be coded is "a", which falls into the 0.0~0.6 interval. In the next step, the 0.0~0.6 interval corresponding to "a" is again divided into 60%, 20%, and 20% intervals. Therefore, the 0.0~0.6 interval is divided into the 0.0~0.36 interval, the 0.36~0.48 interval, and the 0.48~0.6 interval. Next, the second symbol to be coded is "b", which falls into the 0.36~0.48 interval. Dividing the 0.36~0.48 interval corresponding to "b" again into 60%, 20%, and 20% intervals results in the 0.36~0.432 interval, the 0.432~0.456 interval, and the 0.456~0.48 interval. After that, the third symbol to be coded is "c", which falls into the 0.456~0.48 interval. In other words, arithmetic coding involves dividing the interval corresponding to the symbol to be coded again using a probabilistic model or a context model, and then selecting one of the intervals divided by the symbol to be coded. Arithmetic coding is performed by repeating this process.

[0221] In the example above, the probability interval derived after encoding the message "abc" corresponds to the 0.456 to 0.48 interval. Therefore, the message "abc" is encoded into a real number within that interval. For example, the real number 0.47 represents "abc". The value 0.47 is transmitted to the decoder according to the present invention, and the decoder can decode the message into the symbol corresponding to the probability interval by checking which probability interval the value 0.47 belongs to within the probability intervals divided by the context model.

[0222] The probabilistic model (or probabilistic mode) or context model is either a model initially defined by the arithmetic coding algorithm, or a model set by exploring the probability of symbol occurrence during coding / decoding. The probabilistic model or context model can be updated during the coding / decoding process.

[0223] In the point cloud data transmission / receiving device according to the present invention, applying context continuity during frame encoding / decoding means encoding the current frame consecutively to a probability interval calculated by arithmetic coding of the reference frame. For example, if the probability interval derived at the end of arithmetic coding of the reference frame is 0.456~0.48, the current frame can be encoded consecutively by dividing the 0.456~0.48 interval again using a context model or probability model. In other words, the probability interval or context model is not initialized for each frame, but rather the current frame is encoded / decoded consecutively to the probability interval of the reference frame derived by association search.

[0224] The point cloud data transmission / receiving device according to the present invention uses CABAC (context-adaptive binary arithmetic coding) during encoding / decoding. The context model (or probability mode) used for frame arithmetic coding is initialized for each frame, or the probability mode used in a related reference frame is reused in the current frame. Alternatively, the current frame can be encoded consecutively using probability intervals derived from related reference frames, or the current frame can be decoded using probability interval information. If there are similar characteristics between frames, reusing the probability mode allows for the utilization of overlap between symbols, thus increasing compression efficiency. Using the same probability mode as used in a reference frame is equivalent to using the context model of the reference frame. In addition to CABAC, the probability mode or context model of a reference frame can also be reused based on the search for relevance between frames in Arithmetic coding, Huffman coding, or Range coding schemes.

[0225] The criteria for determining the relationship between frames in the point cloud data transmitting / receiving device according to the present invention are as follows:

[0226] The determination of interrelationships between frames differs depending on the type of frame that makes up the Group of Four (GoF). For example, an I-frame is the first frame in a group and is judged to have no related reference frames and no context model that can be used consecutively. A P-frame that references an I-frame is predicted after motion detection and motion correction from the I-frame, so it is judged to have a low degree of interrelationship with the I-frame. On the other hand, in the case of two different P-frames predicted from the same I-frame, it is judged that there is a high degree of interrelationship between the P-frames.

[0227] Furthermore, the relationship between frames is determined based on the difference in global motion vectors between the current frame and the previous frame. Global motion vectors are motion vectors for the entire point cloud data contained within a frame. By comparing the difference in global motion vectors between frames with a arbitrarily set threshold, the relationship between frames can be determined. Similarly, the relationship between frames is determined based on the difference in local motion vectors between the current frame and the previous frame. Local motion vectors are motion vectors for some objects within a frame. By comparing the sum of the differences in local motion vectors between frames with an arbitrarily set threshold, the relationship between frames can be determined.

[0228] The relationship between frames is determined based on the RDO (rate-distortion optimization) value between the points, geometry, and / or characteristic values ​​of the previous frame and the current frame. The relationship between frames is also determined based on the change in the motion vector values ​​and motion compensated values ​​of roads or other objects between the previous frame and the current frame.

[0229] A previous frame refers to the frame immediately preceding the current frame, or to a frame earlier than that. Information about reference frames highly correlated with the current frame is transmitted to the point cloud data receiving device according to the present invention via an index. When encoding the current frame, if a probabilistic model of reference frames highly correlated with the current frame is used sequentially, index information about the highly correlated reference frames is transmitted to the decoder, and the decoder decodes the current frame based on the transmitted index information.

[0230] The point cloud data transmission / receiving device according to the present invention encodes / decodes the current frame by sharing a context model or probabilistic model of a reference frame associated with the current frame. The encoder encodes the geometric and characteristic information of the current frame sequentially using the context of the associated reference frame. The decoder decodes the current frame using the context used sequentially from the associated reference frame.

[0231] The point cloud data transmission / receiving device according to the present invention transmits / receives index information indicating which frame's context or probabilistic model was used when using contexts or probabilistic models of related reference frames consecutively. The index information for related reference frames is determined based on the distance from the current frame to the reference frame, the distance from the I-frame to the reference frame, or the number of frames from the starting frame.

[0232] Figure 16 shows an example of the point cloud data transmission device 16000 according to the present invention.

[0233] The point cloud data transmission device 16000 according to the present invention (for example, the point cloud encoder or transmission device described in Figures 1, 4, 12, or 18) performs the encoding operation described in Figures 1 to 14. The point cloud data transmission device 16000 according to the present invention further includes one or more components for performing the encoding operation described in Figures 1 to 14, although this is not shown in Figure 16.

[0234] The point cloud data transmission device 16000 according to the present invention (for example, the encoder or transmission device described in Figures 1, 4, 12, or 18) includes a context continuity determination and storage unit. The context continuity determination and storage unit includes a geometry context continuity determination and storage unit 16001 and a characteristic context continuity determination and storage unit 16002.

[0235] The context continuity determination and storage unit determines whether or not to apply context continuity to the current frame to be encoded. At this time, the application of context continuity is determined by whether or not there is a reference frame associated with the current frame. If there is an associated reference frame, the current frame is encoded with context continuity applied. The context continuity determination and storage unit derives a reference frame associated with the current frame by searching for associations between frames. The context continuity determination and storage unit also transmits the index information of the reference frame in which context information (either probability interval information, context model information, or probability model information) is used consecutively, and information regarding the application of context continuity, to the point cloud data receiving device according to the present invention (for example, the decoder or receiving device in Figures 1, 10, 11, and 19).

[0236] The context continuity determination and storage unit signals the indices of the reference frame and the current frame using context continuity. The index information of the reference frame that is related in the geometry information coding process and the index information of the reference frame that is related in the characteristic information coding process are different from each other. The context continuity determination and storage unit also stores the context information (either probability interval information, context model, or probability model information) derived from the arithmetic coding of the reference frame. Therefore, when the current frame is arithmetic coded by the Arithmetic coder according to the present invention, the probability interval information and context model of the reference frame can be used consecutively. That is, context continuity can be used in the Arithmetic coder.

[0237] Figure 17 shows an example of the point cloud data receiving device 17000 according to the present invention.

[0238] The point cloud data receiving device 17000 according to the present invention (for example, the point cloud decoder or receiving device described in Figures 1, 10, 11, or 19) performs the decoding operation described in Figures 1 to 14. The point cloud data receiving device 17000 also performs a decoding operation which corresponds to the reverse process of the encoding operation of the point cloud data transmitting device 16000 described in Figure 16. The point cloud data receiving device 17000 according to the present invention further includes one or more components for performing the decoding operation described in Figures 1 to 14, although this is not shown in Figure 17.

[0239] The point cloud data receiving device 17000 according to the present invention includes a context continuity processing unit. The context continuity processing unit includes a geometry context continuity processing unit 17001 and a characteristic context continuity processing unit 17002.

[0240] The point cloud data receiving device 17000 according to the present invention decodes geometry information or characteristic information using the index information of the current frame and the index information of the reference frame, using context continuity. That is, the receiving device 17000 decodes the point cloud data of the current frame based on the context information (probability interval, context model, or probability model) of the reference frame corresponding to the received index information. The process of restoring geometry information or characteristic information corresponds to the inverse process of the process performed by the geometry context continuity determination and storage unit 16001 and the characteristic context continuity determination and storage unit 16002 in Figure 16.

[0241] The context continuity processing unit according to the present invention determines whether or not context continuity is applied to a frame of point cloud data. The point cloud data receiving device 17000 according to the present invention receives a bitstream containing index information of a reference frame or information regarding whether or not context continuity is applied to a frame. Based on the received information, the context continuity processing unit determines whether or not context continuity is applied to the current frame. Whether or not context continuity is applied may be determined by whether or not there is a reference frame associated with the current frame. If there is index information of a reference frame, the context continuity processing unit determines that context continuity has been applied to the current frame. If context continuity has been applied to the current frame, the receiving device 17000 decodes the current frame based on the context information of the reference frame (probability interval, context model, or probability model). The context continuity processing unit also stores the context information of the frame. The stored context information is used when the receiving device 17000 decodes other frames.

[0242] Figure 18 shows an example of the point cloud data transmission device 18000 according to the present invention.

[0243] The point cloud data transmission device 18000 according to the present invention (for example, the point cloud encoder or transmission device described in Figures 1, 4, 12, or 16) performs the encoding operation described in Figures 1 to 16. The point cloud data transmission device 18000 according to the present invention further includes one or more components for performing the encoding operation described in Figures 1 to 16, although this is not shown in Figure 18.

[0244] The point cloud data transmission device 18000 according to the present invention (for example, the encoder or transmission device described in Figures 1, 4, 12, or 16) includes context continuity determination and storage units 18001 and 18002. The explanation of the context continuity determination and storage units 18001 and 18002 is omitted as it overlaps with the explanation in Figure 16.

[0245] Referring to Figure 18, global motion detection / compensation or local motion detection / compensation is performed from the I frame to predict the P frame or B frame. Context continuity determination and storage units 18001 and 18002 search for inter-frame relationships to derive a reference frame for the current frame and store the context information (probability interval information, context model, or probability model) used during arithmetic coding in the Arithmetic coder. The stored context information is used when encoding frames in the Arithmetic coder. That is, the context of the current frame is sequentially coded based on the stored context information of the reference frame.

[0246] Figure 19 shows an example of the point cloud data receiving device 19000 according to the present invention.

[0247] The point cloud data receiving device 19000 according to the present invention (for example, the point cloud decoder or receiving device described in Figures 1, 10, 11, or 17) performs the decoding operation described in Figures 1 to 17. The point cloud data receiving device 19000 also performs a decoding operation which is the reverse process of the encoding operation of the point cloud data receiving device 17000 described in Figure 17. The point cloud data receiving device 19000 according to the present invention further includes one or more components for performing the decoding operation described in Figures 1 to 17, although this is not shown in Figure 19.

[0248] The point cloud data receiving device 19000 according to the present invention includes context continuity processing units 19001 and 19002. The receiving device 19000 decodes geometry information or characteristic information using the index information of the current frame and the index information of the reference frame. The receiving device 19000 derives context information (probability interval, context model, or probability model) used to decode the reference frame from the index information of the received reference frame. The receiving device 19000 decodes the point cloud data of the current frame based on the context information of the reference frame.

[0249] The context continuity processing units 19001 and 19002 shown in Figure 19 correspond to the context continuity processing units in Figure 17.

[0250] The context continuity processing unit according to the present invention determines whether context continuity is applied to a frame of point cloud data. The point cloud data receiving device 19000 according to the present invention receives a bitstream containing index information of a reference frame or information regarding whether context continuity is applied to a frame. The context continuity processing unit determines whether context continuity is applied to the current frame based on the received information. Whether context continuity is applied may be determined by whether or not there is a reference frame associated with the current frame. If there is index information of a reference frame, the context continuity processing unit determines that context continuity has been applied to the current frame. If context continuity has been applied to the current frame, the receiving device 19000 decodes the current frame based on the context information of the reference frame (probability interval, context model, or probability model). The context continuity processing unit also stores the context information of the frame. The stored context information is used when the receiving device 19000 decodes other frames.

[0251] Figure 20 shows an example of the point cloud data encoding / decoding process according to the present invention. Referring to Figure 20, the point cloud data encoding / decoding process involves determining whether or not context continuity is applied 20011, encoding / decoding the point cloud data based on whether or not context continuity is applied 20013, and then storing the context information 20014.

[0252] The determination of whether context continuity applies checks whether the current frame is a dependent P frame (or dependent frame). The if statement checks for the presence of random_access_period or independent P frame, but if the frame meets either of the two conditions, the frame is not a dependent P frame. random_access_period indicates an I frame, and independent P frame indicates that the P frame is an independent P frame (or independent frame) that has no association with other frames. If either of the two conditions is met, the current frame is not a dependent P frame, and therefore there are no reference frames in previous frames that have an association with the current frame. At this time, the entropy context is initialized (Initialize entropy context, 20012), and the current frame is encoded or decoded using the initialized context information (20013). Initializing the entropy context means encoding or decoding a frame to which context continuity does not apply. The entropy context means a context model, a probability model, or a probability interval.

[0253] From the perspective of the transmitting device according to the present invention (such as the transmitting devices in Figures 16 and 18), whether or not context continuity is applied is determined by judging the relationship between frames and determining whether or not there are related reference frames. If the current frame is a dependent P frame, a related reference frame exists in the previous frame, and context continuity is applied.

[0254] Furthermore, from the perspective of the receiving device according to the present invention (such as the receiving device in Figures 17 and 19), whether or not context continuity is applied is determined based on information generated / received from the transmitting device according to the present invention. That is, the receiving device determines whether or not context continuity is applied to the current frame based on the received information regarding the application of context continuity, the index information of the reference frame, etc.

[0255] If the frame does not fall under random_access_period or independent P frame, the current frame is a dependent P frame. A dependent P frame has a reference frame related to the previous frame. If the current frame is a dependent P frame, the entropy context is not initialized, and the current frame is encoded or decoded. Because the entropy context is not initialized, the current frame is encoded or decoded sequentially using the context of the previous reference frame. That is, the current frame is encoded or decoded based on the context information of the reference frame.

[0256] Next, the entropy context probability is stored (Saving entropy context probability). The entropy context probability represents context information. Context information represents a context model, a probability model, or a probability interval. The stored entropy context probability is used when encoding or decoding other frames. The encoding or decoding process in Figure 20 is repeated for each frame, so the process ends when encoding or decoding has been performed for all frames.

[0257] Figure 21 shows the coding efficiency of point cloud data according to the present invention. Referring to Figure 21, point cloud data coding using context continuity according to the present invention shows a gain of 2.5% at C2-lossy.

[0258] FIG. 22 shows the coding efficiency of the point cloud data according to the present invention. Referring to FIG. 22, the point cloud data coding using context continuity according to the present invention shows a gain of 0.4% at CW-lossless.

[0259] On the other hand, the entropy continuing slice method is used in the low-delay scenario of the intra-frame coding of TMC13 [1]. This document discloses the dependent P frames by the continuity of entropy. This method shows the possibility of improving the compression rate by continuing the entropy state probability for the P frames within a Group of Frames (GoF). As shown in FIGS. 21 and 22, the experimental results of this method show a gain of 2.5% at C2 and 0.4% at CW in the current structure of InterEMv1.0. The entropy continuing slice method is a frame coding or decoding method to which the context continuity according to the present invention is applied.

[0260] This document describes the entropy continuous method of coding between frames. The current InterEMv1.0 supports global and local motion detection / estimation and compensation. After adjusting the motion, the occupancy bits of the current frame are predicted by the reference frame and coded by entropy coding. Since all P frames of the GoF are coded to the motion applied reference frame, the use of the dependent P frames is introduced. Entropy continuity indicates context continuity.

[0261] Intra-frame coding in Cat3 initializes the entropy state in all frames corresponding to a single slice. However, in inter-frame coding, each frame is linked to global / local motion and inter-occupancy prediction. Considering such linked attributes, P-dependent frames that use previous entropy are introduced. The reasons for introducing P-dependent frames are as follows.

[0262] 1.Most of the Cat3 frame datasets are composed of a single slice of a single frame, and in very rare datasets, each frame may be composed of several points.

[0263] 2.Use cases where compression is prioritized over parallelization at the frame level. Parallel decoding is used in GoF, and more connections are required within GoF.

[0264] In the I and P frame structures, the loss of a P frame induces losses in all frames until the next reference frame is represented. From the perspective of supporting random access where frame loss is inevitable, the study of P (dependent / independent entropy states) and B frames is proposed conditional on whether the sharing of entropy states is possible.

[0265] Parallel decoding is supported at the GoF level. However, in the current structure, the flexibility of parallel decoding in P frames is not yet supported. First, it is necessary to define the units that make up GoF. Then, the discussion focuses on considering frame-level parallel decoding. Dependent P frames and independent P frames each have advantages for compression and parallelization. Therefore, the method of introducing P-dependent frames and P-independent frames to take advantage of the benefits between compression and parallelization is discussed.

[0266] Figure 23 shows an example of encoded point cloud data according to the present invention. The point cloud video encoder 10002 according to the present invention encodes point cloud data in the encoding process 20001, and the transmitter 10003 according to the present invention transmits a bitstream containing the encoded point cloud data to the receiver 10004 according to the present invention.

[0267] The encoded point cloud data (bitstream) according to the present invention is generated by hardware, software, firmware, or a combination thereof, including the point cloud video encoder 10002 in Figure 1, the encoder 20001 in Figure 2, the encoder in Figure 4, the transmitter in Figure 12, the XR device 1430 in Figure 14, the transmitters in Figures 16 and 18, and / or one or more processors or integrated circuits configured to communicate with one or more memories.

[0268] Furthermore, the encoded point cloud data (bitstream) according to the present invention is decoded by hardware, software, firmware, or a combination thereof, including the point cloud video decoder 10006 in Figure 1, the decoder 20003 in Figure 2, the decoder in Figure 11, the transmitter in Figure 13, the XR device 1430 in Figure 14, the receivers in Figures 17 and 19, and / or one or more processors or integrated circuits configured to communicate with one or more memories.

[0269] The meanings of the abbreviations shown in Figure 23 are as follows:

[0270] SPS: Sequence Parameter Set

[0271] GPS: Geometry Parameter Set

[0272] APS: Attribute Parameter Set

[0273] TPS: Tile Parameter Set

[0274] Geom: Geometry bitstream = Geometry slice header + Geometry slice data

[0275] Attr: Attribute bitstream = Attribute brick header + Attribute brick data

[0276] Tiles or slice regions are provided to process point cloud data by dividing it into regions. When separating point cloud data into regions, each region has a different level of importance. Therefore, by applying different filters or different filter units according to importance, a filtering method that is complex but produces good results can be used for important regions.

[0277] Furthermore, instead of using a complex filtering method on the entire point cloud data depending on the receiving device's processing capacity, different filtering can be applied to each region (regions divided into tiles or slices). Therefore, improved image quality and appropriate latency can be ensured for important regions. Specifically, if the point cloud data is divided into tiles, a different filter or filter unit can be applied to each tile. Similarly, if the point cloud data is divided into slices, a different filter or filter unit can be applied to each slice.

[0278] Hereinafter, the parameters according to the present invention (metadata, signaling information, etc.) are generated by the processor of the transmitting device according to the present invention and transmitted to the receiving device according to the present invention. For example, the parameters according to the present invention are generated by the metadata processing unit (or metadata generator) of the transmitting device according to the present invention and obtained from the metadata parser of the receiving device according to the present invention.

[0279] The point cloud data transmission device according to the present invention (for example, the encoder or transmission device described in Figures 1, 4, 12, 16, and 18) signals the receiving device to the application information of the context continuity of geometry information and characteristic information, the index information of the current frame, and the index information of the reference frame.

[0280] Figure 24 shows an example of the syntax of the Sequence Parameter Set according to the present invention. Context continuity structure information is added to the SPS and signaled.

[0281] The frameIdx indicates the current frame's position in the sequence as an index.

[0282] The `context_continue_flag` flag indicates whether context continuity is applied to all geometry and characteristic information. True indicates that the current frame uses the context of the previous frame continuously (context continuity is applied), while false indicates that the current frame does not use the context of the previous frame continuously (context continuity is not applied).

[0283] `context_continue_use_frameIdx` indicates the index of the previous frame in which context continuity was used (the reference frame). The reference frame index is determined based on factors such as the distance from the current frame to the reference frame, the distance from the first frame to the reference frame, or the order of consecutive P frames up to the reference frame.

[0284] Figure 25 shows an example of the syntax of the Tile Parameter Set according to the present invention. The structural information of context continuity is added to the TPS and signaled.

[0285] tile_ctr indicates, by index, which tile the current tile is among the entire tiles.

[0286] tile_context_continue_flag indicates whether context continuity is applied on a per-tile basis. It indicates whether context continuity is applied to all of the geometry information and characteristic information. True indicates that the current tile continuously uses the context of the previous tile, and false indicates that the current tile does not continuously use the context of the previous tile.

[0287] tile_context_continue_use_tileIdx indicates the index of the tile (reference tile) for which context continuity was used among the previous tiles. The index of the reference tile is signaled based on the distance from the current tile to the reference tile or the distance from the first tile to the reference tile.

[0288] Figure 26 shows an example of the syntax of the Geometry Parameter Set according to the present invention. The structural information of context continuity is added to the GPS and signaled.

[0289] frameIdx indicates, by index, which frame the current frame is in the sequence.

[0290] geometry_context_continue_flag indicates whether context continuity is applied on a per-geometry basis. True indicates that the geometry of the current frame continuously uses the context of the geometry of the previous frame. False indicates that the geometry of the current frame does not continuously use the context of the geometry of the previous frame.

[0291] `geometry_context_continue_use_frameIdx` indicates the index of the previous frame in which context continuity was used (the reference frame). The reference frame index is signaled based on the distance from the current frame to the reference frame or the distance from the first frame to the reference frame.

[0292] Figure 27 shows an example of the syntax of the Attribute Parameter Set according to the present invention. Structural information of context continuity is added to the APS and signaled.

[0293] The frameIdx indicates the current frame's position in the sequence as an index.

[0294] The `attribute_context_continue_flag` flag indicates whether context continuity is applied to attribute units. True indicates that the attribute of the current frame will continue to use the context of the attribute of the previous frame, while false indicates that the attribute of the current frame will not continue to use the context of the attribute of the previous frame.

[0295] The attribute_context_continue_use_frameIdx indicates the index of the previous frame in which context continuity was used (the reference frame). The reference frame index is signaled based on the distance from the current frame to the reference frame or the distance from the first frame to the reference frame.

[0296] Figure 28 shows an example of the syntax for a geometry bitstream slice header according to the present invention. Context continuity structure information is added to the Geom slice header and signaled.

[0297] gsh_slice_id indicates the slice index, which is the current slice number.

[0298] The `geometry_slice_context_continue_flag` flag indicates whether context continuity is applied on a geometry slice basis. True indicates that the current slice uses the context of the previous slice continuously, while false indicates that the current slice does not use the context of the previous slice continuously.

[0299] `geometry_slice_context_continue_use_frameIdx` indicates the index of the previous slice where context continuity was used (the reference slice). The index of the reference slice is signaled based on the distance from the current slice to the reference slice or the distance from the first slice to the reference slice.

[0300] Figure 29 shows an example of a point cloud data transmission method according to the present invention. The point cloud data transmission method includes a step of encoding point cloud data (S2900) and a step of transmitting a bitstream containing point cloud data (S2910). The step of encoding point cloud data (S2900) also includes a step of encoding the geometry data of the point cloud data and a step of encoding characteristic information of the point cloud data.

[0301] The step of encoding point cloud data (S2900) encodes the point cloud data using hardware, software, firmware, or a combination thereof, including the point cloud video encoder 10002 in Figure 1, the encoder 20001 in Figure 2, the encoder in Figure 4, the transmitter in Figure 12, the XR device 1430 in Figure 14, the transmitter in Figure 16 or Figure 18, and / or one or more processors or integrated circuits configured to communicate with one or more memories.

[0302] The step of transmitting a bitstream containing point cloud data (S2910) involves transmitting the point cloud data using hardware, software, firmware, or a combination thereof, including the transmitter 10003 in Figure 1, the transmission processing unit 12012 in Figure 12, the XR device 1430 in Figure 14, the transmitting device in Figure 16 or Figure 18, and / or one or more processors or integrated circuits configured to communicate with one or more memories.

[0303] In the stage of encoding the point cloud data (S2900), it is determined whether or not context continuity is applied to the point cloud data of the current frame, and the current frame is encoded according to whether or not context continuity is applied, and the context of the current frame is stored.

[0304] The decision of whether or not to apply context continuity to a frame can be made by searching for a reference frame from a previous frame based on the inter-frame relationship. In this case, the search for inter-frame relationships is performed based on the frame type (e.g., I-frame or P-frame). Alternatively, the decision of inter-frame relationships may be made based on the difference in motion vectors between frames, or based on the RDO values ​​of the geometry information or characteristic information between frames. The decision (or search) of inter-frame relationships is performed in the context continuity determination and storage unit in Figure 16 and the context continuity determination and storage unit in Figure 18. The decision of inter-frame relationships has been explained in Figures 16 and 18, so a detailed explanation is omitted here. If there is a reference frame in a previous frame that is related to the current frame, it can be decided to apply context continuity to the point cloud data of the current frame and encode it.

[0305] Frame coding (encoding) with or without the application of context continuity involves the continuous use of context information (probability interval, context model, or probability model) of the reference frame when encoding the current frame, if a reference frame related to the current frame exists in a previous frame. The current frame is either encoded continuously using the probability interval derived from the arithmetic coding of the reference frame, or encoded using the context model or probability model used during the arithmetic coding of the reference frame. In other words, the current frame is encoded based on the context information of the reference frame, and context continuity is applied.

[0306] Context storage for frames currently stores context information (probability interval, context model, or probability model) derived during the encoding of the current frame. Stored context information is used in the encoding process of other frames.

[0307] The frame according to the present invention includes either an I-frame or a P-frame. An I-frame corresponds to the first frame in a GoF (Group of Frames), which is a set of multiple frames, and a P-frame is a frame predicted by referencing an I-frame or another P-frame. The frame according to the present invention may or may not have a reference frame, depending on the association determination. If a current frame has a reference frame, context continuity is applied to it as a dependent frame; if there is no reference frame, it is encoded as an independent (undependent) frame without context continuity being applied.

[0308] At the stage of transmitting a bitstream containing point cloud data (S2910), the bitstream includes index information indicating the reference frame. The bitstream further includes index information for the current frame and information regarding the application of context continuity. The index information indicating the reference frame is based on the distance from the first frame in the Group of Frames (GoF) to the reference frame, or on the distance from the current frame. The index information indicating the reference frame indicates the frame immediately preceding the current frame. The information regarding the application of context continuity indicates whether the current frame is encoded based on the context of the reference frame.

[0309] The determination of whether or not context continuity applies to a frame, and the frame coding and storage of the frame context based on the context of the reference frame in response to the presence or absence of a reference frame associated with the frame, are performed by the point cloud data transmission device, context continuity determination and storage unit, or Arithmetic coder as described in Figure 16 or Figure 18. The generation and transmission of index information of the current frame or reference frame, information regarding whether or not context continuity applies, and context information are performed by the context continuity determination and storage unit as described in Figure 16 or Figure 18.

[0310] Figure 30 shows an example of a point cloud data reception method according to the present invention.

[0311] The point cloud data reception method according to the present invention includes the steps of receiving a bitstream containing point cloud data (S3000), decoding the point cloud data (S3010), and / or rendering the point cloud data. The step of decoding the point cloud data (S3010) includes the steps of decoding the geometry information of the point cloud data and decoding the characteristic information of the point cloud data.

[0312] At the stage of receiving a bitstream containing point cloud data (S3000), the bitstream includes index information indicating the reference frame. The bitstream further includes index information for the current frame and information regarding the application of context continuity. The index information indicating the reference frame is specified based on the distance from the first frame in the GoF (Group of Frames) to the reference frame, or based on the distance from the current frame to the reference frame. Note that the index information indicating the reference frame indicates the frame immediately preceding the current frame.

[0313] In the point cloud data decryption stage (S3010), it is determined whether context continuity applies to the point cloud data frames, and the frames are decrypted based on the context of the reference frames associated with the frames, corresponding to whether context continuity applies. Also in the point cloud data decryption stage (S3010), the context of the decrypted frames is stored.

[0314] Whether or not context continuity is applied to a frame is determined based on the application status information of context continuity or the presence or absence of a reference frame associated with the current frame. The association of a reference frame is determined based on the frame type. The frame type indicates one of I-frames, P-frames, or B-frames. The association between frames is determined based on the types of the current frame and the previous frame. The association is also determined based on the difference in motion vectors between frames, or based on the RDO values ​​of the geometry information or characteristic information between frames. The determination of inter-frame association is explained in Figure 16, so any redundant content will be omitted.

[0315] The frame according to the present invention includes either an I-frame or a P-frame. An I-frame corresponds to the first frame in a GoF (Group of Frames), which is a collection of multiple frames, and a P-frame is a frame predicted by referencing an I-frame or another P-frame. The frame according to the present invention may or may not have a reference frame, depending on the association determination. If the current frame has a reference frame, context continuity is applied as a dependent frame; if there is no reference frame, context continuity is not applied as an independent (undependent) frame, and it is decoded.

[0316] The reference frame associated with the current frame can be determined from the index information of the reference frame included in the bitstream. Furthermore, the application of context continuity to the current frame can be determined from the information indicating whether or not context continuity is applied. If context continuity is applied, or if there is an index for the reference frame, the receiving device according to the present invention can be determined to have a reference frame associated with the current frame.

[0317] In the stage where a frame is decoded based on the context of the reference frame, corresponding to whether or not context continuity is applied, if there is a reference frame in the previous frame, the current frame is decoded using the context of the reference frame. The context of the reference frame includes either a probability interval, a context model, or a probability model. The context information of the reference frame is stored in the context continuity processing unit described in Figure 17 or Figure 19.

[0318] If a reference frame exists, it means that the current frame has been sequentially encoded with the context of the reference frame. Therefore, the receiving device according to the present invention decodes the current frame based on the context information of the reference frame. The receiving device according to the present invention also stores the context information of the current frame. The stored context information can be used when decoding other frames.

[0319] The point cloud data transmission / receiving device, point cloud transmission / receiving method, and signaling method according to the present invention provide the following effects.

[0320] This invention enables efficient construction of a bitstream by removing as much redundant information as possible using contextual information between highly related frames. Therefore, it can improve the compression efficiency of geometry and characteristic information. By signaling the respective scope of application of geometry and characteristics, optimal compression efficiency is achieved, and since a probabilistic model initialized for all frames during encoding / decoding is not used, memory reduction and computing time savings are obtained.

[0321] The present invention enables efficient geometric compression of 3D map data integrated into a single content by assisting in a method of capturing point cloud data with LiDAR equipment and dividing it into slides so that angular mode can be applied.

[0322] When the point cloud dataframes according to the present invention are integrated into a single point cloud content, the efficiency of geometry compression coding / decoding can be increased by providing a method for splitting the dataframes into slides for efficient geometry compression of Geometry-based Point Cloud Compression (G-PCC).

[0323] The point cloud data transmission / reception method / apparatus according to the present invention can more efficiently compress and restore point cloud data based on the operation of dividing point cloud data captured by a LiDAR-equipped device based on a 3D map and related signaling information.

[0324] The present invention is described in terms of a method and / or apparatus, and the description of the method and the description of the apparatus can be applied in a complementary manner.

[0325] For the sake of explanation, each figure has been described separately, but it is also possible to combine the inventions described in each figure to design a new invention. Furthermore, the design of a computer-readable recording medium on which a program for performing the previously described invention is recorded, as required by the needs of an ordinary engineer, also falls within the scope of the rights of this invention. The apparatus and method according to the present invention are not limited to the configurations and methods of the present invention described above, and the present invention can be configured by selectively combining all or part of each invention in a variety of modifiable ways. Although preferred embodiments of the present invention have been shown and described, the present invention is not limited to the specific embodiments described above, and does not deviate from the gist of the invention claimed in the claims, and various modifications can be carried out by a person with ordinary skill in the art to which the invention belongs, and such modifications should not be understood individually from the technical idea or prospects of the present invention.

[0326] The various components of the apparatus according to the present invention consist of hardware, software, firmware, or a combination thereof. The various components of the present invention are embodied in a single chip, for example, a single hardware circuit. In the present invention, each component of the present invention is embodied in an individual chip. In the present invention, any of the components of the apparatus according to the present invention consists of one or more processors capable of executing one or more programs, and one or more programs include instructions to cause or execute one or more of the operations / methods of the present invention.

[0327] The executable instructions for performing the methods / operations of the apparatus according to the present invention may be stored in a non-temporary CRM or other computer program product configured to be executed by one or more processors, or in a temporary CRM or other computer program product configured to be executed by one or more processors.

[0328] Furthermore, the memory according to the present invention is used as a concept that includes not only volatile memory (e.g., RAM) but also non-volatile memory, flash memory, PROM, etc. It also includes being implemented in the form of a carrier wave, such as transmission over the Internet. Moreover, the recording medium read by the processor is distributed across a network of connected computer systems, and the code read by the processor can be stored and executed in a distributed manner.

[0329] In this specification, " / " and "," are interpreted as "and / or". For example, "A / B" is interpreted as "A and / or B", and "A, B" is interpreted as "A and / or B". Furthermore, "A / B / C" means "any of A, B and / or C". Also, "A, B, C" means "any of A, B and / or C". Furthermore, in this document, "or" is interpreted as "and / or". For example, "A or B" means 1) "A" only, 2) "B" only, or 3) "A and B". In other words, in this specification, "or" means "additionally or alternatively".

[0330] Terms such as "first," "second," etc., are used to describe various components of the present invention. However, the interpretation of the various components of the present invention should not be limited by these terms. Such terms are merely used to distinguish one component from another. For example, the first user input signal may be referred to as the second user input signal. Similarly, the second user input signal may be referred to as the first user input signal. The use of such terms does not deviate from the various scopes of the present invention. While both the first user input signal and the second user input signal are user input signals, they do not mean the same user input signal unless explicitly stated in the context.

[0331] The terms used to describe this invention are used to describe a particular invention and are not intended to limit it. Unless otherwise explicitly stated in context, singular includes plural, as used in the description and claims of this invention. “and / or” expressions are used to include all possible combinations between terms. “Includes” describes the presence of features, numbers, stages, elements and / or components, and does not imply the absence of further features, numbers, stages, elements and / or components. Conditional expressions such as “if” and “when” used to describe this invention are not intended to be restrictive, but only if they are optional. When a particular condition is met, the associated behavior or associated definition is intended to be interpreted in accordance with that particular condition.

[0332] Furthermore, the operations of the present invention described in this specification are performed by a transceiver including a memory and / or a processor. The memory stores a program for processing / controlling the operations of the present invention, and the processor controls the various operations described in this specification. The processor is also referred to as a controller. The operations of the present invention are performed by firmware, software, and / or a combination thereof, and the firmware, software, and / or a combination thereof are stored in the processor or in memory.

[0333] On the other hand, the operations according to the present invention described above are performed by a transmitting device and / or receiving device according to the present invention. The transmitting and receiving device includes a transmitting and receiving unit for transmitting and receiving media data, a memory for storing instructions for the process according to the present invention (program code, algorithm, flowchart and / or data), and a processor for controlling the operation of the transmitting and receiving device.

[0334] A processor, also known as a controller, can correspond to, for example, hardware, software, and / or a combination thereof. The operation of the present invention described above is performed by a processor. The processor is also embodied by an encoder / decoder for the operation of the present invention described above.

[0335] [Modes for carrying out the invention] As described above, the best mode for carrying out the present invention will be explained in detail.

[0336] [Industrial applicability] As described above, the present invention can be applied in whole or in part to point cloud data transmission and reception devices and systems. Those skilled in the art can modify and transform the present invention in various ways within its scope. The present invention includes modifications / transformations, which are within the scope of the claims and identical thereof.

Claims

1. A method for encoding point cloud data, The step of encoding the geometric data of point cloud data in a data unit based on a contextual probabilistic model of the geometry of the previous data unit; The step of encoding characteristic data of the point cloud data in the data unit; and The step of transmitting a bitstream containing the aforementioned point cloud data; The aforementioned bitstream is First information specifying that the context probabilistic model relating to the geometry of the aforementioned prior data unit is applied to the data unit, Second information related to the identifier of the data unit, and An encoding method further comprising third information relating to the identifier of the aforementioned prior data unit.

2. The step of encoding the geometry data of the point cloud data includes the step of storing the context of the data unit, The encoding method according to claim 1, wherein the characteristic data in the data unit is encoded based on a contextual probability model relating to the characteristics of the aforementioned prior data unit.

3. The aforementioned prior data unit related to the aforementioned data unit is a reference data unit, The encoding method according to claim 2, wherein the reference data unit is searched based on the type of the previous data unit.

4. The encoding method according to claim 1, wherein the first information indicates whether or not the data unit is encoded, based on the context of the preceding data unit.

5. The aforementioned data unit includes either an I-frame or a P-frame. The aforementioned I-frame is the first frame of a GoF (Group of Frames), which is a collection of multiple frames. The encoding method according to claim 1, wherein the P-frame is a frame predicted by referring to the I-frame or other P-frames.

6. The encoding method according to claim 3, wherein the reference data unit is searched based on the difference in motion vectors between frames.

7. A device for encoding point cloud data, An encoder that encodes point cloud data in a data unit and encodes characteristic data of the point cloud data in the data unit, based on a contextual probabilistic model relating to the geometry of the previous data unit; and A transmitter that transmits a bitstream containing the aforementioned point cloud data; The aforementioned bitstream is First information specifying that the context probabilistic model relating to the geometry of the aforementioned prior data unit is applied to the data unit, Second information related to the identifier of the data unit, and An encoding device further comprising third information relating to the identifier of the aforementioned prior data unit.

8. The encoder stores the context of the data unit, The encoding device according to claim 7, wherein the characteristic data in the data unit is encoded based on a contextual probability model relating to the characteristics of the aforementioned prior data unit.

9. The encoding device according to claim 7, wherein the first information indicates whether or not the data unit is encoded, based on the context of the preceding data unit.

10. A method for decrypting point cloud data, The stage of receiving a bitstream containing point cloud data; A step of decoding the geometric data of the point cloud data in the data unit based on a contextual probabilistic model relating to the geometry of the previous data unit based on the value of the first information; and The step of decoding the characteristic data of the point cloud data in the data unit; The aforementioned bitstream is First information specifying that the context probabilistic model relating to the geometry of the aforementioned prior data unit is applied to the data unit, Second information related to the identifier of the data unit, and A decoding method further comprising third information relating to the identifier of the aforementioned prior data unit.

11. The step of decoding the geometry data of the point cloud data includes the step of storing the context of the data unit, The decoding method according to claim 10, wherein the characteristic data in the data unit is decoded based on a contextual probability model relating to the characteristics of the previous data unit based on the value of the first information.

12. The decoding method according to claim 10, wherein the first information indicates whether or not the data unit is decoded, based on the context of the aforementioned prior data unit.

13. A device for decrypting point cloud data, A receiver that receives a bitstream containing point cloud data; and A decoder is configured to decode the geometry data of the point cloud data in a data unit and the characteristic data of the point cloud data in the data unit, based on a contextual probabilistic model relating to the geometry of a previous data unit based on the value of a first piece of information; The aforementioned bitstream is First information specifying that the context probabilistic model relating to the geometry of the aforementioned prior data unit is applied to the data unit, Second information related to the identifier of the data unit, and A decoding device further comprising third information relating to the identifier of the aforementioned prior data unit.

14. The decoder stores the context of the data unit, The decoding apparatus according to claim 13, wherein the characteristic data in the data unit is decoded based on a contextual probability model relating to the characteristics of the aforementioned prior data unit.

15. The decoding device according to claim 13, wherein the first information indicates whether or not the data unit is encoded, based on the context of the aforementioned prior data unit.

Images