Three dimensional data decoding method and three dimensional data decoding device

Abstract

To improve encoding efficiency or reduce a processing amount.SOLUTION: Obtaining a bitstream including encoded position information in which position information of a three dimensional point is encoded (S13711), generating position information for decoding property information of the three dimensional point by decoding the encoded position information (S13712), reading first flag information from the bitstream, the first flag information indicating whether the position information is represented in polar coordinates or rectangular coordinates, and shifting the position information based on at least the first flag information (S13714); SELECTED DRAWING: Figure 86

Description

[Technical Field]

[0001] The present disclosure relates to a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, and a three-dimensional data decoding device. [Background technology]

[0002] In the future, devices and services that utilize 3D data are expected to become widespread in a wide range of fields, including computer vision for autonomous operation of automobiles or robots, map information, surveillance, infrastructure inspection, video distribution, etc. 3D data can be acquired in a variety of ways, including distance sensors such as range finders, stereo cameras, or a combination of multiple monocular cameras.

[0003] One method of representing three-dimensional data is a point cloud, which represents the shape of a three-dimensional structure using a group of points in three-dimensional space. A point cloud stores the position and color of the points. Point clouds are expected to become the mainstream method of representing three-dimensional data, but point clouds require a very large amount of data. Therefore, when storing or transmitting three-dimensional data, data compression through encoding is essential, just as with two-dimensional video images (examples include MPEG-4 AVC or HEVC standardized by MPEG).

[0004] In addition, compression of point clouds is partially supported by public libraries that perform point cloud-related processing (Point Cloud Library).

[0005] Furthermore, a technique is known in which three-dimensional map data is used to search for and display facilities located around a vehicle (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0006] [Patent Document 1] International Publication No. 2014 / 020663 Summary of the Invention [Problem to be solved by the invention]

[0007] In the encoding process and decoding process of three-dimensional data, it is desirable to be able to improve the encoding efficiency or reduce the amount of processing.

[0008] An object of the present disclosure is to provide a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, or a three-dimensional data decoding device that can improve encoding efficiency or reduce the amount of processing. [Means for solving the problem]

[0009] A three-dimensional data decoding method according to one aspect of the present disclosure acquires a bit stream including encoded position information in which position information of three-dimensional points is encoded, decodes the encoded position information to generate the position information for decoding attribute information of the three-dimensional points, reads first flag information from the bit stream indicating whether the position information is represented in a polar coordinate system or a Cartesian coordinate system, and shifts the position information based on at least the first flag information.

[0010] A three-dimensional data encoding method according to one aspect of the present disclosure generates encoded position information by encoding position information of three-dimensional points, and generates a bit stream including the encoded position information and shift information indicating whether the three-dimensional data decoding device shifts the decoded position information decoded by the three-dimensional data decoding device, and based on the shift information, the three-dimensional data decoding device shifts the decoded position information if the decoded position information is expressed in a Cartesian coordinate system, and does not shift the decoded position information if the decoded position information is expressed in a polar coordinate system. [Effects of the Invention]

[0011] The present disclosure can provide a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, or a three-dimensional data decoding device that can improve encoding efficiency or reduce the amount of processing. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram showing a configuration of a three-dimensional data encoding / decoding system according to the first embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of the configuration of point cloud data according to the first embodiment. [Figure 3] FIG. 3 is a diagram showing an example of the structure of a data file in which point cloud data information according to the first embodiment is described. [Figure 4] FIG. 4 is a diagram showing types of point cloud data according to the first embodiment. [Figure 5] FIG. 5 is a diagram showing a configuration of a first encoding unit according to the first embodiment. [Figure 6] FIG. 6 is a block diagram of a first encoding unit according to the first embodiment. [Figure 7] FIG. 7 is a diagram illustrating a configuration of a first decoding unit according to the first embodiment. [Figure 8] FIG. 8 is a block diagram of a first decoding unit according to the first embodiment. [Figure 9] FIG. 9 is a block diagram of a three-dimensional data encoding device according to the first embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of location information according to the first embodiment. [Figure 11] FIG. 11 is a diagram showing an example of an octree representation of position information according to the first embodiment. [Figure 12] FIG. 12 is a block diagram of a three-dimensional data decoding device according to the first embodiment. [Figure 13] FIG. 13 is a block diagram of the attribute information encoding unit according to the first embodiment. [Figure 14] FIG. 14 is a block diagram of the attribute information decoding unit according to the first embodiment. [Figure 15] FIG. 15 is a block diagram showing a configuration of an attribute information encoding unit according to the first embodiment. [Figure 16] FIG. 16 is a block diagram of the attribute information encoding unit according to the first embodiment. [Figure 17] FIG. 17 is a block diagram showing a configuration of an attribute information decoding unit according to the first embodiment. [Figure 18] FIG. 18 is a block diagram of the attribute information decoding unit according to the first embodiment. [Figure 19] FIG. 19 is a diagram showing a configuration of a second encoding unit according to the first embodiment. [Figure 20] FIG. 20 is a block diagram of a second encoding unit according to the first embodiment. [Figure 21] FIG. 21 is a diagram illustrating a configuration of a second decoding unit according to the first embodiment. [Figure 22] FIG. 22 is a block diagram of a second decoding unit according to the first embodiment. [Figure 23] FIG. 23 is a diagram illustrating a protocol stack related to PCC encoded data according to the first embodiment. [Figure 24] FIG. 24 is a diagram illustrating the configurations of an encoding unit and a multiplexing unit according to the second embodiment. [Figure 25] FIG. 25 is a diagram illustrating an example of a structure of coded data according to the second embodiment. [Figure 26] FIG. 26 is a diagram illustrating an example of the structure of coded data and NAL units according to the second embodiment. [Figure 27] FIG. 27 is a diagram illustrating an example of the semantics of pcc_nal_unit_type according to the second embodiment. [Figure 28] FIG. 28 is a diagram showing an example of a prediction tree used in a three-dimensional data coding method according to the third embodiment. [Figure 29] FIG. 29 is a flowchart showing an example of a three-dimensional data encoding method according to the third embodiment. [Figure 30] FIG. 30 is a flowchart showing an example of a three-dimensional data decoding method according to the third embodiment. [Figure 31] FIG. 31 is a diagram illustrating a method for generating a prediction tree according to the third embodiment. [Figure 32]FIG. 32 is a diagram illustrating a first example of a prediction mode according to the third embodiment. [Figure 33] FIG. 33 is a diagram showing a second example of a table indicating predicted values ​​calculated in each prediction mode according to Embodiment 3. In FIG. [Figure 34] FIG. 34 is a diagram showing a specific example of a second example of a table indicating predicted values ​​calculated in each prediction mode according to Embodiment 3. In FIG. [Figure 35] FIG. 35 is a diagram showing a third example of a table indicating predicted values ​​calculated in each prediction mode according to Embodiment 3. In FIG. [Figure 36] FIG. 36 is a diagram illustrating an example of the syntax of a header of location information according to the third embodiment. [Figure 37] FIG. 37 is a diagram illustrating an example of the syntax of the location information according to the third embodiment. [Figure 38] FIG. 38 is a diagram illustrating another example of the syntax of the location information according to the third embodiment. [Figure 39] FIG. 39 is a block diagram of a three-dimensional data encoding device according to the fourth embodiment. [Figure 40] FIG. 40 is a block diagram of a three-dimensional data decoding device according to the fourth embodiment. [Figure 41] FIG. 41 is a block diagram of a three-dimensional data encoding device according to the fourth embodiment. [Figure 42] FIG. 42 is a block diagram of a three-dimensional data decoding device according to the fourth embodiment. [Figure 43] FIG. 43 is a diagram showing an example of inter prediction according to the fourth embodiment. [Figure 44] FIG. 44 is a diagram illustrating an example of the syntax of an SPS according to the fourth embodiment. [Figure 45] FIG. 45 is a diagram illustrating an example of GPS syntax according to the fourth embodiment. [Figure 46] FIG. 46 is a flowchart of three-dimensional data encoding processing according to the fourth embodiment. [Figure 47]FIG. 47 is a flowchart of three-dimensional data decoding processing according to the fourth embodiment. [Figure 48] FIG. 48 is a block diagram of a three-dimensional data encoding device according to the fifth embodiment. [Figure 49] FIG. 49 is a block diagram of a three-dimensional data decoding device according to the fifth embodiment. [Figure 50] FIG. 50 is a flowchart showing an example of a procedure for encoding each 3D point of the prediction tree according to the fifth embodiment. [Figure 51] FIG. 51 is a flowchart showing an example of a procedure for decoding each 3D point of the prediction tree according to the fifth embodiment. [Figure 52] FIG. 52 is a block diagram of a three-dimensional data encoding device according to a modification of the fifth embodiment. [Figure 53] FIG. 53 is a block diagram of a three-dimensional data decoding device according to a variation of the fifth embodiment. [Figure 54] FIG. 54 is an example of the syntax of the GPS according to the fifth embodiment. [Figure 55] FIG. 55 is an example of the syntax of each 3D point according to the fifth embodiment. [Figure 56] FIG. 56 is a flowchart showing three-dimensional data encoding processing according to the fifth embodiment. [Figure 57] FIG. 57 is a flowchart showing three-dimensional data decoding processing according to the fifth embodiment. [Figure 58] FIG. 58 is a flowchart showing the coordinate system switching process in the encoding process according to the fifth embodiment. [Figure 59] FIG. 59 is a flowchart showing the coordinate system switching process in the decoding process according to the fifth embodiment. [Figure 60] FIG. 60 is a flowchart showing another example of the three-dimensional data encoding process according to the fifth embodiment. [Figure 61] FIG. 61 is a flowchart showing another example of the three-dimensional data decoding process according to the fifth embodiment. [Figure 62]FIG. 62 is a block diagram of a position information encoding unit according to the sixth embodiment. [Figure 63] FIG. 63 is a block diagram of a position information encoding unit according to the sixth embodiment. [Figure 64] FIG. 64 is a block diagram of a position information encoding unit according to the sixth embodiment. [Figure 65] FIG. 65 is a block diagram of a position information decoding unit according to the sixth embodiment. [Figure 66] FIG. 66 is a block diagram of a position information decoding unit according to the sixth embodiment. [Figure 67] FIG. 67 is a block diagram of a position information decoding unit according to the sixth embodiment. [Figure 68] FIG. 68 is a block diagram of a three-dimensional data encoding device according to the sixth embodiment. [Figure 69] FIG. 69 is a block diagram of a three-dimensional data decoding device according to the sixth embodiment. [Figure 70] FIG. 70 is a flowchart of three-dimensional data decoding processing according to the sixth embodiment. [Figure 71] FIG. 71 is a flowchart of three-dimensional data encoding processing according to the sixth embodiment. [Figure 72] FIG. 72 is a flowchart of three-dimensional data decoding processing according to the sixth embodiment. [Figure 73] FIG. 73 is a block diagram of a position information encoding unit that encodes position information in a rectangular coordinate system according to the seventh embodiment. [Figure 74] FIG. 74 is a diagram showing an example of shift processing on position information in a rectangular coordinate system according to the seventh embodiment. [Figure 75] FIG. 75 is a diagram showing an example of shift processing on position information in a rectangular coordinate system according to the seventh embodiment. [Figure 76] FIG. 76 is a block diagram of a position information encoding unit that encodes position information in a polar coordinate system according to the seventh embodiment. [Figure 77] FIG. 77 is a diagram showing an example of processing performed on position information in a polar coordinate system according to the seventh embodiment. [Figure 78] FIG. 78 is a block diagram of a position information decoding unit that decodes position information in an orthogonal coordinate system according to the seventh embodiment. [Figure 79] FIG. 79 is a block diagram of a position information decoding unit that decodes position information in a polar coordinate system according to the seventh embodiment. [Figure 80] FIG. 80 is a block diagram of a position information encoding unit according to the seventh embodiment that encodes position information in a rectangular coordinate system and position information in a polar coordinate system. [Figure 81] FIG. 81 is a block diagram of a position information decoding unit according to the seventh embodiment that decodes position information in a rectangular coordinate system and position information in a polar coordinate system. [Figure 82] FIG. 82 is a block diagram of a position information decoding unit that performs coordinate transformation according to the seventh embodiment. [Figure 83] FIG. 83 is a flowchart of three-dimensional data decoding processing according to the seventh embodiment. [Figure 84] FIG. 84 is a diagram illustrating an example of GPS syntax according to the seventh embodiment. [Figure 85] FIG. 85 is a diagram illustrating an example of the syntax of a GDU header according to the seventh embodiment. [Figure 86] FIG. 86 is a flowchart of three-dimensional data decoding processing according to the seventh embodiment. [Figure 87] FIG. 87 is a flowchart of three-dimensional data encoding processing according to the seventh embodiment. [Figure 88] FIG. 88 is a block diagram of a three-dimensional data creation device according to the eighth embodiment. [Figure 89] FIG. 89 is a flowchart of a three-dimensional data creation method according to the eighth embodiment. [Figure 90] FIG. 90 is a diagram showing a configuration of a system according to the eighth embodiment. [Figure 91] FIG. 91 is a block diagram of a client device according to the eighth embodiment. [Figure 92] FIG. 92 is a block diagram of a server according to the eighth embodiment. [Figure 93]FIG. 93 is a flowchart of three-dimensional data creation processing by the client device according to the eighth embodiment. [Figure 94] FIG. 94 is a flowchart of a sensor information transmission process by a client device according to the eighth embodiment. [Figure 95] FIG. 95 is a flowchart of three-dimensional data creation processing by the server according to the eighth embodiment. [Figure 96] FIG. 96 is a flowchart of a three-dimensional map transmission process by the server according to the eighth embodiment. [Figure 97] FIG. 97 is a diagram showing a configuration of a modified example of the system according to the eighth embodiment. [Figure 98] FIG. 98 is a diagram showing the configurations of a server and a client device according to the eighth embodiment. [Figure 99] FIG. 99 is a diagram showing the configurations of a server and a client device according to the eighth embodiment. [Figure 100] FIG. 100 is a flowchart of processing by the client device according to the eighth embodiment. [Figure 101] FIG. 101 is a diagram illustrating a configuration of a sensor information collection system according to the eighth embodiment. [Figure 102] FIG. 102 is a diagram illustrating an example of a system according to the eighth embodiment. [Figure 103] FIG. 103 is a diagram showing a modification of the system according to the eighth embodiment. [Figure 104] FIG. 104 is a flowchart showing an example of application processing according to the eighth embodiment. [Figure 105] FIG. 105 is a diagram showing the sensor ranges of various sensors according to the eighth embodiment. [Figure 106] FIG. 106 is a diagram illustrating a configuration example of an autonomous driving system according to the eighth embodiment. [Figure 107] FIG. 107 is a diagram showing an example of the structure of a bitstream according to the eighth embodiment. [Figure 108] FIG. 108 is a flowchart of the point group selection process according to the eighth embodiment. [Figure 109] FIG. 109 is a diagram showing an example of a screen for the point group selection process according to the eighth embodiment. [Figure 110] FIG. 110 is a diagram showing an example of a screen for the point group selection process according to the eighth embodiment. [Figure 111] FIG. 111 is a diagram showing an example of a screen for the point group selection process according to the eighth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] A three-dimensional data decoding method according to one aspect of the present disclosure acquires a bit stream including encoded position information in which position information of three-dimensional points is encoded, generates the position information by decoding the encoded position information, and shifts the position information if the position information is expressed in a Cartesian coordinate system, and does not shift the position information if the position information is expressed in a polar coordinate system.

[0014] According to this, the three-dimensional data decoding method can switch whether to perform shift processing on the position information depending on whether the encoded position information is in a Cartesian coordinate system or a polar coordinate system. Here, when the position information is in a Cartesian coordinate system, the encoding efficiency can be improved by performing shift processing in the three-dimensional data encoding device before encoding. When the position information is in a Cartesian coordinate system, the three-dimensional data decoding method can appropriately decode a bitstream with improved encoding efficiency by performing shift processing. Furthermore, when the position information is in a polar coordinate system, the three-dimensional data decoding method can reduce the amount of processing by not performing shift processing.

[0015] For example, the bitstream may include first flag information indicating whether the position information is expressed in the polar coordinate system, and second flag information indicating whether the position information expressed in the polar coordinate system is converted into converted position information expressed in the Cartesian coordinate system.

[0016] For example, if the position information is expressed in the polar coordinate system and the position of the sensor generating the three-dimensional point is the origin of the polar coordinate system, then the shift may not be performed.

[0017] For example, when shift information indicating the amount of shift is obtained from the bit stream and the position information is expressed in the polar coordinate system, the amount indicated by the shift information is zero, and the position information expressed in the polar coordinate system can be shifted based on the amount indicated by the shift information, so that the position information expressed in the polar coordinate system does not need to be shifted.

[0018] According to this, the three-dimensional data decoding method can apply shift processing to position information regardless of whether the position information is in a Cartesian coordinate system or a polar coordinate system, and there is no need to control whether or not to perform shift processing, thereby simplifying processing.

[0019] For example, shift information indicating the amount of shift may be obtained from the bit stream, and if the position information is expressed in the Cartesian coordinate system, the position information may be shifted based on the amount indicated by the shift information, and if the position information is expressed in the polar coordinate system, the amount may be considered to be zero regardless of the amount indicated by the shift information.

[0020] According to this, when the position information is in a polar coordinate system, the three-dimensional data decoding method considers the shift amount to be zero regardless of the shift amount indicated by the shift information. This reduces the amount of processing required by the three-dimensional data encoding device that generates the bitstream. Furthermore, the three-dimensional data decoding method may be able to correctly decode the bitstream even if the bitstream contains an error.

[0021] For example, if the bit stream contains shift information indicating the amount of the shift, it is determined that the position information is represented in the Cartesian coordinate system, and the position information is shifted based on the amount indicated by the shift information; if the bit stream does not contain the shift information, it is determined that the position information is represented in the polar coordinate system, and it is not necessary to shift the position information.

[0022] According to this, when the position information is in polar coordinates, the bit stream does not contain shift information, and therefore the data amount of the bit stream can be reduced.

[0023] For example, the bit stream may include encoding information in which positional information of a plurality of three-dimensional points including the three-dimensional point is encoded, the encoding information is divided into a plurality of division units into which the plurality of three-dimensional points are divided, and the shift information may be included in a parameter set common to the plurality of division units.

[0024] For example, the bit stream may include encoded information in which positional information of a plurality of three-dimensional points including the three-dimensional point is encoded, the encoded information is divided into a plurality of division units into which the plurality of three-dimensional points are divided, and the shift information may be included in a header for each of the plurality of division units.

[0025] For example, when the position information is expressed in the polar coordinate system, the position information may be converted into converted position information expressed in the Cartesian coordinate system, and the converted position information may be shifted.

[0026] According to this, even when the encoded position information is in a polar coordinate system, the three-dimensional data decoding method can convert the position information in the polar coordinate system into position information in a Cartesian coordinate system and output the position information in the Cartesian coordinate system.

[0027] A three-dimensional data encoding method according to one aspect of the present disclosure generates encoded position information by encoding position information of three-dimensional points, and generates a bit stream including the encoded position information and shift information indicating whether the three-dimensional data decoding device shifts the decoded position information decoded by the three-dimensional data decoding device, and based on the shift information, the three-dimensional data decoding device shifts the decoded position information if the decoded position information is expressed in a Cartesian coordinate system, and does not shift the decoded position information if the decoded position information is expressed in a polar coordinate system.

[0028] According to this, the three-dimensional data encoding method can generate a bitstream in which the three-dimensional data decoding device can switch whether to perform shift processing on the position information depending on whether the encoded position information is in a rectangular coordinate system or a polar coordinate system. Here, when the position information is in a rectangular coordinate system, the three-dimensional data encoding device performs shift processing before encoding, thereby improving encoding efficiency. When the position information is in a rectangular coordinate system, the three-dimensional data decoding device performs shift processing, thereby being able to properly decode the bitstream with improved encoding efficiency. Furthermore, when the position information is in a polar coordinate system, the three-dimensional data decoding device does not perform shift processing, thereby reducing the amount of processing.

[0029] For example, the bitstream may include first flag information indicating whether the position information is expressed in the polar coordinate system, and second flag information indicating whether the position information expressed in the polar coordinate system is converted into converted position information expressed in the Cartesian coordinate system.

[0030] For example, if the position information is expressed in the polar coordinate system and the position of the sensor generating the three-dimensional point is the origin of the polar coordinate system, then the shift may not be performed.

[0031] For example, the shift information indicates the amount of shift when shifting the decoded position information, and when the position information is expressed in the polar coordinate system, the amount indicated by the shift information is zero, and the three-dimensional data decoding device shifts the position information expressed in the polar coordinate system based on the amount indicated by the shift information, so that it is not necessary to shift the position information expressed in the polar coordinate system.

[0032] According to this, for example, in a three-dimensional data decoding device, shift processing can be applied to position information regardless of whether the position information is in a Cartesian coordinate system or a polar coordinate system, and there is no need to control whether or not to perform shift processing, thereby simplifying processing in the three-dimensional data decoding device.

[0033] For example, when the position information is expressed in the Cartesian coordinate system, the shift information indicates the amount of shift when shifting the decoded position information, and when the position information is expressed in the polar coordinate system, the shift information does not need to be stored in the bitstream.

[0034] According to this, when the position information is in polar coordinates, the bit stream does not contain shift information, and therefore the data amount of the bit stream can be reduced.

[0035] For example, the bit stream may include encoding information in which positional information of a plurality of three-dimensional points including the three-dimensional point is encoded, the encoding information is divided into a plurality of division units into which the plurality of three-dimensional points are divided, and the shift information may be included in a parameter set common to the plurality of division units.

[0036] For example, the bit stream may include encoded information in which positional information of a plurality of three-dimensional points including the three-dimensional point is encoded, the encoded information is divided into a plurality of division units into which the plurality of three-dimensional points are divided, and the shift information may be included in a header for each of the plurality of division units.

[0037] In addition, a three-dimensional data decoding device according to one aspect of the present disclosure includes a processor and a memory, and the processor uses the memory to obtain a bit stream including encoded position information in which position information of three-dimensional points is encoded, generates position information by decoding the encoded position information, and shifts the position information when the position information is expressed in a Cartesian coordinate system, and does not shift the position information when the position information is expressed in a polar coordinate system.

[0038] According to this, the three-dimensional data decoding device can switch whether to perform shift processing on the position information depending on whether the encoded position information is in a Cartesian coordinate system or a polar coordinate system. Here, when the position information is in a Cartesian coordinate system, the three-dimensional data encoding device performs shift processing before encoding, thereby improving encoding efficiency. When the position information is in a Cartesian coordinate system, the three-dimensional data decoding device performs shift processing, thereby being able to properly decode a bitstream with improved encoding efficiency. Furthermore, when the position information is in a polar coordinate system, the three-dimensional data decoding device does not perform shift processing, thereby reducing the amount of processing.

[0039] In addition, a three-dimensional data encoding device according to one aspect of the present disclosure includes a processor and a memory, and the processor uses the memory to generate encoded position information by encoding position information of three-dimensional points, and generates a bit stream including the encoded position information and shift information indicating whether the three-dimensional data decoding device shifts the decoded position information decoded by the three-dimensional data decoding device, and based on the shift information, the three-dimensional data decoding device shifts the decoded position information if the decoded position information is expressed in a Cartesian coordinate system, and does not shift the decoded position information if the decoded position information is expressed in a polar coordinate system.

[0040] According to this, the three-dimensional data encoding device can generate a bitstream in which the three-dimensional data decoding device can switch whether to perform shift processing on the position information depending on whether the encoded position information is in a rectangular coordinate system or a polar coordinate system. Here, when the position information is in a rectangular coordinate system, the three-dimensional data encoding device performs shift processing before encoding, thereby improving encoding efficiency. When the position information is in a rectangular coordinate system, the three-dimensional data decoding device performs shift processing, thereby being able to properly decode the bitstream with improved encoding efficiency. Furthermore, when the position information is in a polar coordinate system, the three-dimensional data decoding device does not perform shift processing, thereby reducing the amount of processing.

[0041] These comprehensive or specific aspects may be realized as a system, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or may be realized as any combination of a system, a method, an integrated circuit, a computer program, and a recording medium.

[0042] Hereinafter, the embodiments will be described in detail with reference to the drawings. Note that each of the embodiments described below represents a specific example of the present disclosure. The numerical values, shapes, materials, components, component placement and connection configurations, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components not recited in independent claims will be described as optional components.

[0043] (Embodiment 1) When using encoded point cloud data in an actual device or service, it is desirable to transmit and receive the information required for the application in order to reduce network bandwidth. However, until now, such a function has not existed in the encoding structure of 3D data, and no encoding method for this purpose has existed.

[0044] In this embodiment, we will describe a three-dimensional data encoding method and a three-dimensional data encoding device that provide the function of transmitting and receiving information required for the purpose in encoded data of a three-dimensional point cloud, as well as a three-dimensional data decoding method and a three-dimensional data decoding device that decodes the encoded data, a three-dimensional data multiplexing method that multiplexes the encoded data, and a three-dimensional data transmission method that transmits the encoded data.

[0045] In particular, two encoding methods (encoding schemes) are currently being considered as encoding methods (encoding systems) for point cloud data; however, the structure of the encoded data and the method for storing the encoded data in a system format have not been defined, and as things stand, there is a problem that MUX processing (multiplexing) in the encoding unit, or transmission or storage, is not possible.

[0046] Furthermore, there has been no method to date that supports a format in which two codecs, a first encoding method and a second encoding method, are mixed, such as PCC (Point Cloud Compression).

[0047] In this embodiment, a description will be given of the structure of PCC encoded data in which two codecs, a first encoding method and a second encoding method, are mixed, and a method of storing the encoded data in a system format.

[0048] First, the configuration of a three-dimensional data (point cloud data) encoding / decoding system according to this embodiment will be described. Fig. 1 is a diagram showing an example of the configuration of a three-dimensional data encoding / decoding system according to this embodiment. As shown in Fig. 1, the three-dimensional data encoding / decoding system includes a three-dimensional data encoding system 4601, a three-dimensional data decoding system 4602, a sensor terminal 4603, and an external connection unit 4604.

[0049] The three-dimensional data encoding system 4601 generates encoded data or multiplexed data by encoding point cloud data, which is three-dimensional data. Note that the three-dimensional data encoding system 4601 may be a three-dimensional data encoding device realized by a single device, or may be a system realized by multiple devices. Furthermore, the three-dimensional data encoding device may include some of the multiple processing units included in the three-dimensional data encoding system 4601.

[0050] The three-dimensional data encoding system 4601 includes a point cloud data generation system 4611, a presentation unit 4612, an encoding unit 4613, a multiplexing unit 4614, an input / output unit 4615, and a control unit 4616. The point cloud data generation system 4611 includes a sensor information acquisition unit 4617 and a point cloud data generation unit 4618.

[0051] The sensor information acquisition unit 4617 acquires sensor information from the sensor terminal 4603 and outputs the sensor information to the point cloud data generation unit 4618. The point cloud data generation unit 4618 generates point cloud data from the sensor information and outputs the point cloud data to the encoding unit 4613.

[0052] The presentation unit 4612 presents the sensor information or point cloud data to the user. For example, the presentation unit 4612 displays information or an image based on the sensor information or point cloud data.

[0053] The encoding unit 4613 encodes (compresses) the point cloud data, and outputs the obtained encoded data, control information obtained in the encoding process, and other additional information to the multiplexing unit 4614. The additional information includes, for example, sensor information.

[0054] The multiplexing unit 4614 generates multiplexed data by multiplexing the coded data input from the coding unit 4613, the control information, and the additional information. The format of the multiplexed data is, for example, a file format for storage or a packet format for transmission.

[0055] The input / output unit 4615 (e.g., a communication unit or an interface) outputs the multiplexed data to the outside. Alternatively, the multiplexed data is stored in a storage unit such as an internal memory. The control unit 4616 (or application execution unit) controls each processing unit. In other words, the control unit 4616 controls encoding, multiplexing, etc.

[0056] The sensor information may be input to the encoding unit 4613 or the multiplexing unit 4614. The input / output unit 4615 may output the point cloud data or the encoded data directly to the outside.

[0057] The transmission signal (multiplexed data) output from the three-dimensional data encoding system 4601 is input to the three-dimensional data decoding system 4602 via the external connection unit 4604 .

[0058] The three-dimensional data decoding system 4602 generates point cloud data, which is three-dimensional data, by decoding the encoded data or multiplexed data. Note that the three-dimensional data decoding system 4602 may be a three-dimensional data decoding device realized by a single device, or may be a system realized by multiple devices. Furthermore, the three-dimensional data decoding device may include some of the multiple processing units included in the three-dimensional data decoding system 4602.

[0059] The three-dimensional data decoding system 4602 includes a sensor information acquisition unit 4621 , an input / output unit 4622 , a demultiplexing unit 4623 , a decoding unit 4624 , a presentation unit 4625 , a user interface 4626 , and a control unit 4627 .

[0060] The sensor information acquisition unit 4621 acquires sensor information from the sensor terminal 4603 .

[0061] The input / output unit 4622 acquires the transmission signal, decodes the multiplexed data (file format or packets) from the transmission signal, and outputs the multiplexed data to the demultiplexer 4623.

[0062] The demultiplexing unit 4623 obtains the coded data, control information, and additional information from the multiplexed data, and outputs the coded data, control information, and additional information to the decoding unit 4624.

[0063] The decoding unit 4624 reconstructs the point cloud data by decoding the encoded data.

[0064] The presentation unit 4625 presents the point cloud data to the user. For example, the presentation unit 4625 displays information or images based on the point cloud data. The user interface 4626 acquires instructions based on user operations. The control unit 4627 (or the application execution unit) controls each processing unit. In other words, the control unit 4627 controls demultiplexing, decoding, presentation, etc.

[0065] The input / output unit 4622 may acquire point cloud data or encoded data directly from the outside. The presentation unit 4625 may acquire additional information such as sensor information and present information based on the additional information. The presentation unit 4625 may perform presentation based on a user instruction acquired by the user interface 4626.

[0066] The sensor terminal 4603 generates sensor information, which is information obtained by a sensor. The sensor terminal 4603 is a terminal equipped with a sensor or a camera, and examples thereof include a moving body such as an automobile, a flying object such as an airplane, a mobile terminal, or a camera.

[0067] The sensor information that can be acquired by the sensor terminal 4603 includes, for example, (1) the distance between the sensor terminal 4603 and an object, or the reflectance of the object, obtained from a LIDAR, millimeter-wave radar, or infrared sensor, and (2) the distance between a camera and an object, or the reflectance of the object, obtained from multiple monocular camera images or stereo camera images. The sensor information may also include the attitude, direction, gyro (angular velocity), position (GPS information or altitude), speed, acceleration, etc. of the sensor. The sensor information may also include temperature, air pressure, humidity, magnetism, etc.

[0068] The external connection unit 4604 is realized by an integrated circuit (LSI or IC), an external storage unit, communication with a cloud server via the Internet, broadcasting, or the like.

[0069] Next, point cloud data will be described. Fig. 2 is a diagram showing the configuration of point cloud data. Fig. 3 is a diagram showing an example of the configuration of a data file in which information about point cloud data is written.

[0070] Point cloud data includes data on multiple points. The data on each point includes location information (three-dimensional coordinates) and attribute information for that location information. A collection of multiple points is called a point cloud. For example, a point cloud can represent the three-dimensional shape of an object.

[0071] Position information such as three-dimensional coordinates is sometimes called geometry. Data for each point may also include attribute information of multiple attribute types. Attribute types include, for example, color or reflectance.

[0072] One piece of attribute information may be associated with one piece of location information, or multiple pieces of attribute information with different attribute types may be associated with one piece of location information, or multiple pieces of attribute information of the same attribute type may be associated with one piece of location information.

[0073] The configuration example of the data file shown in FIG. 3 is an example in which there is a one-to-one correspondence between position information and attribute information, and shows the position information and attribute information of N points that make up the point cloud data.

[0074] The position information is, for example, information on the three axes x, y, and z. The attribute information is, for example, RGB color information. A typical data file is a ply file.

[0075] Next, the types of point cloud data will be explained. Fig. 4 is a diagram showing the types of point cloud data. As shown in Fig. 4, point cloud data includes static objects and dynamic objects.

[0076] A static object is 3D point cloud data at any time (a certain time). A dynamic object is 3D point cloud data that changes over time. Hereinafter, 3D point cloud data at a certain time will be referred to as a PCC frame, or simply a frame.

[0077] The object may be a point cloud with a certain area restriction, such as ordinary video data, or a large-scale point cloud with no area restriction, such as map information.

[0078] Furthermore, there may be point cloud data of various densities, such as sparse point cloud data and dense point cloud data.

[0079] Each processing unit will be described in detail below. Sensor information is acquired by various methods, such as a distance sensor such as a LIDAR or a range finder, a stereo camera, or a combination of multiple monocular cameras. The point cloud data generation unit 4618 generates point cloud data based on the sensor information acquired by the sensor information acquisition unit 4617. The point cloud data generation unit 4618 generates position information as point cloud data, and adds attribute information for the position information to the position information.

[0080] The point cloud data generation unit 4618 may process the point cloud data when generating position information or adding attribute information. For example, the point cloud data generation unit 4618 may reduce the amount of data by deleting point clouds with overlapping positions. In addition, the point cloud data generation unit 4618 may convert (position shift, rotation, normalization, etc.) the position information or render the attribute information.

[0081] In FIG. 1, the point cloud data generation system 4611 is included in the three-dimensional data encoding system 4601, but it may be provided independently outside the three-dimensional data encoding system 4601.

[0082] The encoding unit 4613 generates encoded data by encoding the point cloud data based on a predetermined encoding method. There are two main types of encoding methods: the first is an encoding method that uses position information, and this encoding method will be referred to as the first encoding method hereinafter; and the second is an encoding method that uses a video codec, and this encoding method will be referred to as the second encoding method hereinafter.

[0083] The decoding unit 4624 decodes the coded data based on a predetermined coding method to decode the point group data.

[0084] The multiplexing unit 4614 generates multiplexed data by multiplexing the encoded data using an existing multiplexing method. The generated multiplexed data is transmitted or stored. In addition to the PCC encoded data, the multiplexing unit 4614 multiplexes other media such as video, audio, subtitles, applications, and files, or reference time information. The multiplexing unit 4614 may also multiplex attribute information related to sensor information or point cloud data.

[0085] Multiplexing methods or file formats include ISOBMFF, MPEG-DASH, which is an ISOBMFF-based transmission method, MMT, MPEG-2 TS Systems, and RMP.

[0086] The demultiplexer 4623 extracts PCC encoded data, other media, time information, and the like from the multiplexed data.

[0087] The input / output unit 4615 transmits the multiplexed data using a method suited to the transmission medium or storage medium, such as broadcasting or communication. The input / output unit 4615 may communicate with other devices via the Internet, or may communicate with a storage unit such as a cloud server.

[0088] The communication protocol used may be http, ftp, TCP, UDP, etc. A PULL type communication method or a PUSH type communication method may be used.

[0089] Either wired or wireless transmission may be used. For wired transmission, Ethernet (registered trademark), USB, RS-232C, HDMI (registered trademark), coaxial cable, etc. are used. For wireless transmission, wireless LAN, Wi-Fi (registered trademark), Bluetooth (registered trademark), millimeter waves, etc. are used.

[0090] As a broadcasting system, for example, DVB-T2, DVB-S2, DVB-C2, ATSC3.0, or ISDB-S3 is used.

[0091] Fig. 5 is a diagram showing the configuration of a first encoding unit 4630, which is an example of the encoding unit 4613 that performs encoding using the first encoding method. Fig. 6 is a block diagram of the first encoding unit 4630. The first encoding unit 4630 generates encoded data (encoded stream) by encoding point cloud data using the first encoding method. This first encoding unit 4630 includes a position information encoding unit 4631, an attribute information encoding unit 4632, an additional information encoding unit 4633, and a multiplexing unit 4634.

[0092] The first encoding unit 4630 is characterized in that it performs encoding taking into consideration a three-dimensional structure. The first encoding unit 4630 is also characterized in that the attribute information encoding unit 4632 performs encoding using information obtained from the position information encoding unit 4631. The first encoding method is also called GPCC (Geometry based PCC).

[0093] The point cloud data is PCC point cloud data such as a PLY file, or PCC point cloud data generated from sensor information, and includes position information (Position), attribute information (Attribute), and other additional information (MetaData). The position information is input to a position information encoder 4631, the attribute information is input to an attribute information encoder 4632, and the additional information is input to an additional information encoder 4633.

[0094] The position information encoding unit 4631 encodes the position information to generate encoded position information (Compressed Geometry), which is encoded data. For example, the position information encoding unit 4631 encodes the position information using an N-ary tree structure such as an octree. Specifically, in an octree, the target space is divided into eight nodes (subspaces), and 8-bit information (occupancy code) indicating whether or not a point cloud is included in each node is generated. Furthermore, the node including the point cloud is further divided into eight nodes, and 8-bit information indicating whether or not a point cloud is included in each of the eight nodes is generated. This process is repeated until the number of point clouds included in a predetermined layer or node falls below a threshold.

[0095] The attribute information encoding unit 4632 generates encoded attribute information (Compressed Attribute) that is encoded data by encoding using the configuration information generated by the position information encoding unit 4631. For example, the attribute information encoding unit 4632 determines a reference point (reference node) to be referenced when encoding a target point (target node) to be processed, based on the octree structure generated by the position information encoding unit 4631. For example, the attribute information encoding unit 4632 references a peripheral node or adjacent node whose parent node in the octree is the same as that of the target node. Note that the method of determining the reference relationship is not limited to this.

[0096] Furthermore, the encoding process of the attribute information may include at least one of a quantization process, a prediction process, and an arithmetic coding process. In this case, the reference means using a reference node to calculate a predicted value of the attribute information, or using the state of the reference node (e.g., occupancy information indicating whether the reference node includes a point group) to determine an encoding parameter. For example, the encoding parameter is a quantization parameter in a quantization process, or a context in an arithmetic coding process.

[0097] The additional information encoding unit 4633 generates encoded additional information (Compressed MetaData) that is encoded data by encoding compressible data from the additional information.

[0098] The multiplexing unit 4634 multiplexes the encoding position information, the encoding attribute information, the encoding additional information, and other additional information to generate a compressed stream, which is encoded data. The generated compressed stream is output to a processing unit in a system layer (not shown).

[0099] Next, a first decoding unit 4640, which is an example of the decoding unit 4624 that performs decoding using the first encoding method, will be described. FIG. 7 is a diagram showing the configuration of the first decoding unit 4640. FIG. 8 is a block diagram of the first decoding unit 4640. The first decoding unit 4640 generates point cloud data by decoding, using the first encoding method, coded data (coded stream) coded using the first coding method. The first decoding unit 4640 includes a demultiplexing unit 4641, a position information decoding unit 4642, an attribute information decoding unit 4643, and an additional information decoding unit 4644.

[0100] A coded stream (compressed stream) that is coded data is input to the first decoding unit 4640 from a processing unit in a system layer (not shown).

[0101] The demultiplexer 4641 separates the encoded position information (Compressed Geometry), the encoded attribute information (Compressed Attribute), the encoded additional information (Compressed MetaData), and other additional information from the encoded data.

[0102] The position information decoding unit 4642 generates position information by decoding the encoded position information. For example, the position information decoding unit 4642 restores the position information of a point group represented by three-dimensional coordinates from the encoded position information represented by an N-ary tree structure such as an octree.

[0103] The attribute information decoding unit 4643 decodes the encoded attribute information based on the configuration information generated by the position information decoding unit 4642. For example, the attribute information decoding unit 4643 determines a reference point (reference node) to be referenced in decoding the target point (target node) to be processed based on the octree structure obtained by the position information decoding unit 4642. For example, the attribute information decoding unit 4643 references a peripheral node or adjacent node whose parent node in the octree is the same as that of the target node. Note that the method of determining the reference relationship is not limited to this.

[0104] Furthermore, the attribute information decoding process may include at least one of an inverse quantization process, a prediction process, and an arithmetic decoding process. In this case, the reference means using a reference node to calculate a predicted value of the attribute information, or using the state of the reference node (e.g., occupancy information indicating whether the reference node includes a point group) to determine a decoding parameter. For example, the decoding parameter is a quantization parameter in an inverse quantization process, or a context in an arithmetic decoding process.

[0105] The additional information decoding unit 4644 generates additional information by decoding the encoded additional information. The first decoding unit 4640 uses the additional information necessary for decoding the position information and attribute information during decoding, and outputs the additional information necessary for the application to the outside.

[0106] Next, an example of the configuration of the position information encoding unit will be described. Fig. 9 is a block diagram of position information encoding unit 2700 according to this embodiment. Position information encoding unit 2700 includes an octree generation unit 2701, a geometric information calculation unit 2702, a coding table selection unit 2703, and an entropy encoding unit 2704.

[0107] The octree generation unit 2701 generates, for example, an occupancy tree from the input position information, and generates an occupancy code for each node in the occupancy tree. The geometric information calculation unit 2702 acquires information indicating whether an adjacent node of the target node is an occupied node. For example, the geometric information calculation unit 2702 calculates the occupancy information of the adjacent node (information indicating whether the adjacent node is an occupied node) from the occupancy code of the parent node to which the target node belongs. The geometric information calculation unit 2702 may also store encoded nodes in a list and search for adjacent nodes from the list. The geometric information calculation unit 2702 may also switch adjacent nodes depending on the position of the target node within the parent node.

[0108] The coding table selection unit 2703 selects a coding table to be used for entropy coding of the target node using the occupancy information of the adjacent nodes calculated by the geometric information calculation unit 2702. For example, the coding table selection unit 2703 may generate a bit string using the occupancy information of the adjacent nodes, and select a coding table for an index number generated from the bit string.

[0109] The entropy coding unit 2704 generates the coding position information and metadata by entropy coding the occupancy code of the target node using the coding table of the selected index number. The entropy coding unit 2704 may add information indicating the selected coding table to the coding position information.

[0110] The octree representation and the scanning order of position information will be explained below. Position information (position data) is converted (octreeized) into an octree structure and then encoded. The octree structure consists of nodes and leaves. Each node has eight nodes or leaves, and each leaf has voxel (VXL) information. Fig. 10 is a diagram showing an example of the structure of position information containing multiple voxels. Fig. 11 is a diagram showing an example of the position information shown in Fig. 10 converted into an octree structure. Here, among the leaves shown in Fig. 11, leaves 1, 2, and 3 represent voxels VXL1, VXL2, and VXL3 shown in Fig. 10, respectively, and represent a VXL containing a point cloud (hereinafter referred to as effective VXL).

[0111] Specifically, node 1 corresponds to the entire space including the position information in Figure 10. The entire space corresponding to node 1 is divided into eight nodes, and of the eight nodes, the node containing a valid VXL is further divided into eight nodes or leaves, and this process is repeated for each level of the tree structure. Here, each node corresponds to a subspace, and has information (occupancy code) as node information indicating at what position the next node or leaf will be located after division. In addition, the block at the bottom level is set as a leaf, and the number of point clouds contained in the leaf, etc., is held as leaf information.

[0112] Next, an example of the configuration of the position information decoding unit will be described. Fig. 12 is a block diagram of the position information decoding unit 2710 according to this embodiment. The position information decoding unit 2710 includes an octree generation unit 2711, a geometric information calculation unit 2712, a coding table selection unit 2713, and an entropy decoding unit 2714.

[0113] The octree generation unit 2711 generates an octree of a certain space (node) using header information or metadata of a bitstream. For example, the octree generation unit 2711 generates a large space (root node) using the sizes of the x-axis, y-axis, and z-axis directions of a certain space added to the header information, and generates an octree by dividing the space into two in the x-axis, y-axis, and z-axis directions to generate eight small spaces A (nodes A0 to A7). In addition, nodes A0 to A7 are set in order as target nodes.

[0114] The geometric information calculation unit 2712 acquires occupancy information indicating whether an adjacent node of the target node is an occupied node. For example, the geometric information calculation unit 2712 calculates the occupancy information of the adjacent node from the occupancy code of the parent node to which the target node belongs. The geometric information calculation unit 2712 may also store decoded nodes in a list and search for adjacent nodes from the list. The geometric information calculation unit 2712 may also switch adjacent nodes depending on the position of the target node within the parent node.

[0115] The coding table selection unit 2713 selects a coding table (decoding table) to be used for entropy decoding of the target node using the occupancy information of the adjacent nodes calculated by the geometric information calculation unit 2712. For example, the coding table selection unit 2713 may generate a bit string using the occupancy information of the adjacent nodes, and select a coding table for an index number generated from the bit string.

[0116] The entropy decoding unit 2714 generates position information by entropy decoding the occupancy code of the target node using the selected coding table. Note that the entropy decoding unit 2714 may obtain information on the selected coding table by decoding it from the bitstream, and entropy decode the occupancy code of the target node using the coding table indicated by the information.

[0117] The configurations of the attribute information encoding unit and the attribute information decoding unit will be described below. Fig. 13 is a block diagram showing an example configuration of the attribute information encoding unit A100. The attribute information encoding unit may include multiple encoding units that execute different encoding methods. For example, the attribute information encoding unit may switch between the following two methods depending on the use case:

[0118] The attribute information encoding unit A100 includes an LoD attribute information encoding unit A101 and a conversion attribute information encoding unit A102. The LoD attribute information encoding unit A101 classifies each 3D point into multiple layers using position information of the 3D points, predicts attribute information of the 3D points belonging to each layer, and encodes the prediction residual. Here, each classified layer is called an LoD (Level of Detail).

[0119] The transformed attribute information encoding unit A102 encodes the attribute information using RAHT (Region Adaptive Hierarchical Transform). Specifically, the transformed attribute information encoding unit A102 applies RAHT or Haar transform to each piece of attribute information based on the position information of the three-dimensional point to generate high-frequency components and low-frequency components for each layer, and encodes these values ​​using quantization, entropy coding, etc.

[0120] 14 is a block diagram showing an example configuration of the attribute information decoding unit A110. The attribute information decoding unit may include multiple decoding units that execute different decoding methods. For example, the attribute information decoding unit may switch between the following two methods for decoding based on information included in the header or metadata:

[0121] The attribute information decoding unit A110 includes an LoD attribute information decoding unit A111 and a converted attribute information decoding unit A112. The LoD attribute information decoding unit A111 classifies each 3D point into multiple layers using the position information of the 3D points, and decodes the attribute values ​​while predicting the attribute information of the 3D points belonging to each layer.

[0122] The transformed attribute information decoding unit A112 decodes the attribute information using RAHT (Region Adaptive Hierarchical Transform). Specifically, the transformed attribute information decoding unit A112 decodes the attribute values ​​by applying inverse RAHT or inverse Haar transform to the high-frequency components and low-frequency components of each attribute value based on the position information of the three-dimensional point.

[0123] FIG. 15 is a block diagram showing the configuration of an attribute information encoding unit 3140, which is an example of the LoD attribute information encoding unit A101.

[0124] The attribute information encoding unit 3140 includes an LoD generation unit 3141, a surrounding search unit 3142, a prediction unit 3143, a prediction residual calculation unit 3144, a quantization unit 3145, an arithmetic encoding unit 3146, an inverse quantization unit 3147, a decoded value generation unit 3148, and a memory 3149.

[0125] The LoD generation unit 3141 generates LoD using the position information of the three-dimensional points.

[0126] The surrounding search unit 3142 searches for nearby 3D points adjacent to each 3D point using the LoD generation result by the LoD generation unit 3141 and distance information indicating the distance between each 3D point.

[0127] The prediction unit 3143 generates a predicted value of the attribute information of the target 3D point to be coded.

[0128] The prediction residual calculation unit 3144 calculates (generates) a prediction residual of the predicted value of the attribute information generated by the prediction unit 3143.

[0129] The quantization unit 3145 quantizes the prediction residual of the attribute information calculated by the prediction residual calculation unit 3144 .

[0130] The arithmetic coding unit 3146 arithmetically codes the prediction residuals after being quantized by the quantization unit 3145. The arithmetic coding unit 3146 outputs a bitstream including the arithmetically coded prediction residuals to, for example, a three-dimensional data decoding device.

[0131] Note that the prediction residual may be binarized by, for example, the quantization unit 3145 before being arithmetically coded by the arithmetic coding unit 3146.

[0132] Furthermore, for example, the arithmetic coding unit 3146 may initialize a coding table used for arithmetic coding before arithmetic coding. The arithmetic coding unit 3146 may initialize a coding table used for arithmetic coding for each layer. Furthermore, the arithmetic coding unit 3146 may output information indicating the position of the layer for which the coding table has been initialized, by including it in the bitstream.

[0133] The inverse quantization unit 3147 inverse quantizes the prediction residuals quantized by the quantization unit 3145 .

[0134] The decoded value generation unit 3148 generates a decoded value by adding the predicted value of the attribute information generated by the prediction unit 3143 and the prediction residual after inverse quantization by the inverse quantization unit 3147.

[0135] The memory 3149 is a memory that stores the decoded values ​​of the attribute information of each 3D point decoded by the decoded value generation unit 3148. For example, when generating a predicted value of a 3D point that has not yet been encoded, the prediction unit 3143 generates the predicted value by using the decoded values ​​of the attribute information of each 3D point stored in the memory 3149.

[0136] 16 is a block diagram of an attribute information encoding unit 6600, which is an example of the transformed attribute information encoding unit A102. The attribute information encoding unit 6600 includes a sorting unit 6601, a Haar transform unit 6602, a quantization unit 6603, an inverse quantization unit 6604, an inverse Haar transform unit 6605, a memory 6606, and an arithmetic encoding unit 6607.

[0137] The sorting unit 6601 generates a Morton code using the position information of the 3D points and sorts the multiple 3D points in Morton code order. The Haar transform unit 6602 generates coding coefficients by applying a Haar transform to the attribute information. The quantization unit 6603 quantizes the coding coefficients of the attribute information.

[0138] The inverse quantization unit 6604 inversely quantizes the quantized coding coefficients. The inverse Haar transform unit 6605 applies an inverse Haar transform to the coding coefficients. The memory 6606 stores values ​​of attribute information of multiple decoded 3D points. For example, the attribute information of the decoded 3D points stored in the memory 6606 may be used for predicting uncoded 3D points.

[0139] The arithmetic coding unit 6607 calculates ZeroCnt from the quantized coding coefficients and arithmetically codes the ZeroCnt. The arithmetic coding unit 6607 also arithmetically codes the quantized non-zero coding coefficients. The arithmetic coding unit 6607 may binarize the coding coefficients before arithmetic coding. The arithmetic coding unit 6607 may also generate and code various header information.

[0140] FIG. 17 is a block diagram showing the configuration of an attribute information decoding unit 3150, which is an example of the LoD attribute information decoding unit A111.

[0141] The attribute information decoding unit 3150 includes an LoD generation unit 3151 , a surrounding search unit 3152 , a prediction unit 3153 , an arithmetic decoding unit 3154 , an inverse quantization unit 3155 , a decoded value generation unit 3156 , and a memory 3157 .

[0142] The LoD generation unit 3151 generates LoD using the position information of the 3D points decoded by the position information decoding unit (not shown in FIG. 17).

[0143] The surrounding search unit 3152 searches for nearby 3D points adjacent to each 3D point using the LoD generation result by the LoD generation unit 3151 and distance information indicating the distance between each 3D point.

[0144] The prediction unit 3153 generates a predicted value of the attribute information of the target 3D point to be decoded.

[0145] The arithmetic decoding unit 3154 arithmetically decodes prediction residuals in the bitstream acquired from the attribute information coding unit 3140 shown in FIG. 15. The arithmetic decoding unit 3154 may initialize a decoding table used for arithmetic decoding. The arithmetic decoding unit 3154 initializes a decoding table used for arithmetic decoding for a layer on which the arithmetic coding unit 3146 shown in FIG. 15 has performed coding processing. The arithmetic decoding unit 3154 may initialize a decoding table used for arithmetic decoding for each layer. The arithmetic decoding unit 3154 may initialize the decoding table based on information included in the bitstream that indicates the position of the layer for which the coding table has been initialized.

[0146] The inverse quantization unit 3155 inverse quantizes the prediction residuals arithmetically decoded by the arithmetic decoding unit 3154 .

[0147] The decoded value generation unit 3156 generates a decoded value by adding the predicted value generated by the prediction unit 3153 and the prediction residual after inverse quantization by the inverse quantization unit 3155. The decoded value generation unit 3156 outputs the decoded attribute information data to another device.

[0148] The memory 3157 is a memory that stores the decoded values ​​of the attribute information of each 3D point decoded by the decoded value generation unit 3156. For example, when generating a predicted value of a 3D point that has not yet been decoded, the prediction unit 3153 generates the predicted value by using the decoded value of the attribute information of each 3D point stored in the memory 3157.

[0149] 18 is a block diagram of an attribute information decoding unit 6610, which is an example of the transformed attribute information decoding unit A112. The attribute information decoding unit 6610 includes an arithmetic decoding unit 6611, an inverse quantization unit 6612, an inverse Haar transform unit 6613, and a memory 6614.

[0150] The arithmetic decoding unit 6611 arithmetically decodes the ZeroCnt and coding coefficients included in the bitstream. Note that the arithmetic decoding unit 6611 may also decode various types of header information.

[0151] The inverse quantization unit 6612 inverse quantizes the arithmetically decoded coding coefficients. The inverse Haar transform unit 6613 applies inverse Haar transform to the inversely quantized coding coefficients. The memory 6614 stores values ​​of attribute information of multiple decoded 3D points. For example, the attribute information of the decoded 3D points stored in the memory 6614 may be used to predict undecoded 3D points.

[0152] Next, a description will be given of second encoding unit 4650, which is an example of encoding unit 4613 that performs encoding using the second encoding method. Fig. 19 is a diagram showing the configuration of second encoding unit 4650. Fig. 20 is a block diagram of second encoding unit 4650.

[0153] The second encoding unit 4650 generates encoded data (encoded stream) by encoding the point cloud data using a second encoding method. The second encoding unit 4650 includes an additional information generation unit 4651, a position image generation unit 4652, an attribute image generation unit 4653, a video encoding unit 4654, an additional information encoding unit 4655, and a multiplexing unit 4656.

[0154] The second encoding unit 4650 has a feature of generating a position image and an attribute image by projecting a three-dimensional structure onto a two-dimensional image, and encoding the generated position image and attribute image using an existing video encoding method. The second encoding method is also called VPCC (Video based PCC).

[0155] The point cloud data is PCC point cloud data such as a PLY file or PCC point cloud data generated from sensor information, and includes position information (Position), attribute information (Attribute), and other additional information (MetaData).

[0156] The additional information generating unit 4651 generates map information of a plurality of two-dimensional images by projecting a three-dimensional structure onto the two-dimensional images.

[0157] The position image generation unit 4652 generates a position image (Geometry Image) based on the position information and the map information generated by the additional information generation unit 4651. This position image is, for example, a distance image in which distance (Depth) is indicated as a pixel value. Note that this distance image may be an image in which multiple point clouds are viewed from a single viewpoint (an image in which multiple point clouds are projected onto a single two-dimensional plane), or may be multiple images in which multiple point clouds are viewed from multiple viewpoints, or may be a single image in which these multiple images are integrated.

[0158] The attribute image generation unit 4653 generates an attribute image based on the attribute information and the map information generated by the additional information generation unit 4651. This attribute image is, for example, an image in which attribute information (for example, color (RGB)) is represented as pixel values. Note that this image may be an image in which multiple point clouds are viewed from one viewpoint (an image in which multiple point clouds are projected onto one two-dimensional plane), or multiple images in which multiple point clouds are viewed from multiple viewpoints, or a single image in which these multiple images are integrated.

[0159] The video encoding unit 4654 generates an encoded position image (Compressed Geometry Image) and an encoded attribute image (Compressed Attribute Image), which are encoded data, by encoding the position image and the attribute image using a video encoding method. Note that any known encoding method may be used as the video encoding method. For example, the video encoding method is AVC, HEVC, or the like.

[0160] The additional information encoding unit 4655 generates encoded additional information (Compressed MetaData) by encoding the additional information, map information, and the like included in the point cloud data.

[0161] The multiplexing unit 4656 multiplexes the encoding position image, the encoding attribute image, the encoding additional information, and other additional information to generate an encoded stream (Compressed Stream) that is encoded data. The generated encoded stream is output to a processing unit in a system layer (not shown).

[0162] Next, a second decoding unit 4660, which is an example of the decoding unit 4624 that performs decoding using the second encoding method, will be described. FIG. 21 is a diagram showing the configuration of the second decoding unit 4660. FIG. 22 is a block diagram of the second decoding unit 4660. The second decoding unit 4660 generates point cloud data by decoding, using the second encoding method, coded data (coded stream) that has been coded using the second coding method. The second decoding unit 4660 includes a demultiplexing unit 4661, a video decoding unit 4662, an additional information decoding unit 4663, a position information generation unit 4664, and an attribute information generation unit 4665.

[0163] A coded stream (compressed stream) that is coded data is input to the second decoding unit 4660 from a processing unit in a system layer (not shown).

[0164] The demultiplexer 4661 separates the encoded position image (Compressed Geometry Image), the encoded attribute image (Compressed Attribute Image), the encoded additional information (Compressed MetaData), and other additional information from the encoded data.

[0165] The video decoding unit 4662 generates a position image and an attribute image by decoding the encoded position image and the encoded attribute image using a video encoding method. Note that any known encoding method may be used as the video encoding method. For example, the video encoding method is AVC or HEVC.

[0166] The additional information decoding unit 4663 decodes the encoded additional information to generate additional information including map information and the like.

[0167] The position information generating unit 4664 generates position information using the position image and map information. The attribute information generating unit 4665 generates attribute information using the attribute image and map information.

[0168] The second decoding unit 4660 uses the additional information necessary for decoding during decoding, and outputs the additional information necessary for the application to the outside.

[0169] The following describes the problems with the PCC encoding method. Fig. 23 is a diagram showing a protocol stack related to PCC encoded data. Fig. 23 shows an example in which other media data such as video (e.g., HEVC) or audio is multiplexed with PCC encoded data and transmitted or stored.

[0170] Multiplexing methods and file formats have the function of multiplexing various coded data and transmitting or storing them. To transmit or store coded data, the coded data must be converted into the format of the multiplexing method. For example, HEVC specifies a technology that stores coded data in a data structure called a NAL unit and stores the NAL unit in ISOBMFF.

[0171] On the other hand, currently, a first encoding method (Codec1) and a second encoding method (Codec2) are being considered as methods for encoding point cloud data, but the structure of the encoded data and the method for storing the encoded data in a system format have not been defined, which poses the problem that, as it stands, it is not possible to perform MUX processing (multiplexing) in the encoding unit, transmission, or storage.

[0172] In the following description, unless a specific encoding method is specified, it refers to either the first encoding method or the second encoding method.

[0173] (Embodiment 2) In this embodiment, the types of coded data (position information (Geometry), attribute information (Attribute), additional information (Metadata)) generated by the first coding unit 4630 or the second coding unit 4650 described above, a method for generating the additional information (Metadata), and multiplexing processing in the multiplexing unit will be described. Note that the additional information (Metadata) may also be referred to as a parameter set or control information.

[0174] In this embodiment, the dynamic object (three-dimensional point cloud data that changes over time) described in Figure 4 will be used as an example, but a similar method may also be used in the case of a static object (three-dimensional point cloud data at any time).

[0175] 24 is a diagram showing the configurations of an encoding unit 4801 and a multiplexing unit 4802 included in the three-dimensional data encoding device according to this embodiment. The encoding unit 4801 corresponds to, for example, the first encoding unit 4630 or the second encoding unit 4650 described above. The multiplexing unit 4802 corresponds to the multiplexing unit 4634 or 4656 described above.

[0176] The encoding unit 4801 encodes point cloud data of multiple PCC (Point Cloud Compression) frames, and generates encoded data (Multiple Compressed Data) of multiple pieces of position information, attribute information, and additional information.

[0177] The multiplexing unit 4802 converts data of multiple data types (position information, attribute information, and additional information) into NAL units, thereby converting the data into a data structure that takes into account data access in the decoding device.

[0178] 25 is a diagram showing an example of the structure of coded data generated by coding unit 4801. Arrows in the diagram indicate dependencies related to the decoding of coded data, with the source of the arrow depending on the data at the end of the arrow. In other words, the decoding device decodes the data at the end of the arrow and uses the decoded data to decode the data at the end of the arrow. In other words, dependency means that the data on which the dependency is based is referenced (used) in the processing (encoding, decoding, etc.) of the data on which the dependency is based.

[0179] First, the process of generating encoded data of position information will be described. The encoding unit 4801 generates encoded position data (compressed geometry data) for each frame by encoding the position information of each frame. The encoded position data is represented by G(i), where i indicates the frame number, the time of the frame, etc.

[0180] The encoding unit 4801 also generates a position parameter set (GPS(i)) corresponding to each frame. The position parameter set includes parameters that can be used to decode the encoded position data. The encoded position data for each frame depends on the corresponding position parameter set.

[0181] Furthermore, encoded position data consisting of multiple frames is defined as a position sequence (Geometry Sequence). The encoding unit 4801 generates a position sequence parameter set (Geometry Sequence PS: also referred to as position SPS) that stores parameters commonly used in decoding processes for multiple frames in the position sequence. The position sequence depends on the position SPS.

[0182] Next, the process of generating coded data of attribute information will be described. The coding unit 4801 generates coded attribute data (Compressed Attribute Data) for each frame by coding the attribute information of each frame. The coded attribute data is represented by A(i). FIG. 25 shows an example in which attribute X and attribute Y exist, and the coded attribute data of attribute X is represented by AX(i) and the coded attribute data of attribute Y is represented by AY(i).

[0183] The encoding unit 4801 also generates an attribute parameter set (APS(i)) corresponding to each frame. The attribute parameter set for attribute X is represented as AXPS(i), and the attribute parameter set for attribute Y is represented as AYPS(i). The attribute parameter set includes parameters that can be used to decode encoded attribute information. The encoded attribute data depends on the corresponding attribute parameter set.

[0184] Furthermore, encoded attribute data consisting of multiple frames is defined as an attribute sequence. The encoding unit 4801 generates an attribute sequence parameter set (Attribute Sequence PS: also referred to as attribute SPS) that stores parameters commonly used in decoding processes for multiple frames in the attribute sequence. The attribute sequence depends on the attribute SPS.

[0185] Furthermore, in the first encoding method, the encoded attribute data depends on the encoded position data.

[0186] 25 shows an example in which two types of attribute information (attribute X and attribute Y) exist. When there are two types of attribute information, for example, two encoding units generate respective data and metadata. Also, for example, an attribute sequence is defined for each type of attribute information, and an attribute SPS is generated for each type of attribute information.

[0187] 25 shows an example in which there is one type of position information and two types of attribute information, but this is not limiting, and there may be one type of attribute information, or three or more types. In this case, encoded data can be generated using a similar method. Furthermore, in the case of point cloud data that does not have attribute information, the attribute information may not be necessary. In this case, the encoding unit 4801 does not need to generate a parameter set related to the attribute information.

[0188] Next, the process of generating additional information (metadata) will be described. The encoding unit 4801 generates a PCC Stream PS (also referred to as stream PS), which is a parameter set for the entire PCC stream. The encoding unit 4801 stores, in the stream PS, parameters that can be used in common for decoding processes for one or more position sequences and one or more attribute sequences. For example, the stream PS includes identification information indicating the codec for the point cloud data, information indicating the algorithm used for encoding, and the like. The position sequence and attribute sequence depend on the stream PS.

[0189] Next, the access unit and GOF will be explained. In this embodiment, the concepts of the access unit (AU) and GOF (Group of Frame) are newly introduced.

[0190] An access unit is a basic unit for accessing data during decoding, and is composed of one or more pieces of data and one or more pieces of metadata. For example, an access unit is composed of position information at the same time and one or more pieces of attribute information. A GOF is a random access unit and is composed of one or more access units.

[0191] The encoding unit 4801 generates an access unit header (AU Header) as identification information indicating the beginning of an access unit. The encoding unit 4801 stores parameters related to the access unit in the access unit header. For example, the access unit header includes the configuration or information of the coded data included in the access unit. The access unit header also includes parameters commonly used for the data included in the access unit, such as parameters related to decoding of the coded data.

[0192] Instead of an access unit header, the encoding unit 4801 may generate an access unit delimiter that does not include parameters related to the access unit. This access unit delimiter is used as identification information indicating the start of the access unit. The decoding device identifies the start of the access unit by detecting the access unit header or the access unit delimiter.

[0193] Next, the generation of identification information for the start of a GOF will be described. The encoding unit 4801 generates a GOF header as identification information that indicates the start of a GOF. The encoding unit 4801 stores parameters related to the GOF in the GOF header. For example, the GOF header includes the configuration or information of the coded data included in the GOF. The GOF header also includes parameters commonly used for the data included in the GOF, such as parameters related to the decoding of the coded data.

[0194] Instead of a GOF header, the encoding unit 4801 may generate a GOF delimiter that does not include parameters related to the GOF. This GOF delimiter is used as identification information that indicates the beginning of the GOF. The decoding device identifies the beginning of the GOF by detecting the GOF header or the GOF delimiter.

[0195] In PCC encoded data, for example, an access unit is defined as a PCC frame unit, and a decoding device accesses a PCC frame based on identification information at the beginning of the access unit.

[0196] Also, for example, GOF is defined as one random access unit. A decoding device accesses the random access unit based on the identification information at the beginning of the GOF. For example, if PCC frames are not dependent on each other and can be decoded independently, the PCC frames may be defined as the random access unit.

[0197] It should be noted that two or more PCC frames may be allocated to one access unit, and multiple random access units may be allocated to one GOF.

[0198] The encoding unit 4801 may also define and generate parameter sets or metadata other than those described above. For example, the encoding unit 4801 may generate SEI (Supplemental Enhancement Information) that stores parameters (optional parameters) that may not necessarily be used during decoding.

[0199] Next, the structure of coded data and the method of storing coded data in NAL units will be described.

[0200] For example, a data format is defined for each type of coded data. Figure 26 shows examples of coded data and NAL units.

[0201] For example, as shown in Fig. 26, the coded data includes a header and a payload. The coded data may include length information indicating the length (amount of data) of the coded data, the header, or the payload. The coded data may not include a header.

[0202] The header includes, for example, identification information for identifying the data, such as the data type or frame number.

[0203] The header includes, for example, identification information indicating a reference relationship. This identification information is stored in the header when, for example, there is a dependency relationship between data, and is information for referencing the reference destination from the reference source. For example, the header of the reference destination includes identification information for identifying the data. The header of the reference source includes identification information indicating the reference destination.

[0204] If the reference destination or the reference source can be identified or derived from other information, the identification information for specifying the data or the identification information indicating the reference relationship may be omitted.

[0205] The multiplexing unit 4802 stores the coded data in the payload of the NAL unit. The NAL unit header includes pcc_nal_unit_type, which is information identifying the coded data. Figure 27 shows an example of the semantics of pcc_nal_unit_type.

[0206] As shown in Fig. 27, when pcc_codec_type is codec 1 (Codec1: first encoding method), values ​​0 to 10 of pcc_nal_unit_type are assigned to the encoded position data (Geometry), encoded attribute X data (AttributeX), encoded attribute Y data (AttributeY), position PS (Geom.PS), attribute XPS (AttrX.PS), attribute YPS (AttrX.PS), position SPS (Geometry Sequence PS), attribute XSPS (AttributeX Sequence PS), attribute YSPS (AttributeY Sequence PS), AU header (AU Header), and GOF header (GOF Header) in codec 1. Values ​​11 and above are assigned as spares for codec 1.

[0207] When pcc_codec_type is Codec2 (Codec2: second encoding method), values ​​0 to 2 of pcc_nal_unit_type are assigned to codec data A (DataA), metadata A (MetaDataA), and metadata B (MetaDataB). Values ​​3 and above are assigned as spares for Codec2.

[0208] (Embodiment 3) In the three-dimensional data encoding method according to the third embodiment, position information of a plurality of three-dimensional points is encoded using a prediction tree generated based on the position information.

[0209] Fig. 28 is a diagram showing an example of a prediction tree used in a three-dimensional data coding method according to Embodiment 3. Fig. 29 is a flowchart showing an example of a three-dimensional data coding method according to Embodiment 3. Fig. 30 is a flowchart showing an example of a three-dimensional data decoding method according to Embodiment 3.

[0210] As shown in Figures 28 and 29, in the three-dimensional data encoding method, a prediction tree is generated using a plurality of three-dimensional points, and then node information included in each node of the prediction tree is encoded. This results in a bitstream including the encoded node information. Each piece of node information is, for example, information about one node of the prediction tree. Each piece of node information includes, for example, position information of the node, an index of the node, the number of child nodes of the node, a prediction mode used to encode the position information of the node, and a prediction residual.

[0211] Also, as shown in Figures 28 and 30, in the three-dimensional data decoding method, each piece of coded node information contained in the bit stream is decoded, and then the position information is decoded while generating a prediction tree.

[0212] Next, a method for generating a prediction tree will be described with reference to FIG.

[0213] FIG. 31 is a diagram illustrating a method for generating a prediction tree according to the third embodiment.

[0214] In the prediction tree generation method, as shown in (a) of Fig. 31, the three-dimensional data encoding device first adds point 0 as the initial point of the prediction tree. The position information of point 0 is represented by coordinates including three elements (x0, y0, z0). The position information of point 0 may be represented by coordinates in a three-axis Cartesian coordinate system or by coordinates in a polar coordinate system.

[0215] The child_count is incremented by 1 each time a child node is added to the node for which the child_count is set. The child_count of each node after the generation of the prediction tree is completed indicates the number of child nodes that each node has, and is added to the bitstream. The pred_mode indicates a prediction mode for predicting the value of the position information of each node. The details of the prediction mode will be described later.

[0216] Next, as shown in (b) of FIG. 31, the three-dimensional data encoding device adds point 1 to the prediction tree. At this time, the three-dimensional data encoding device may search for the nearest point of point 1 from the group of points already added to the prediction tree, and add point 1 as a child node of that nearest point. The position information of point 1 is represented by coordinates including three elements (x1, y1, z1). The position information of point 1 may be represented by coordinates in a three-axis Cartesian coordinate system or by coordinates in a polar coordinate system. In the case of FIG. 31, point 0 is the nearest point of point 1, and point 1 is added as a child node of point 0. Then, the three-dimensional data encoding device increments the value represented by child_count of point 0 by 1.

[0217] Note that the predicted value of the position information of each node may be calculated when a node is added to the prediction tree. For example, in the case of (b) of FIG. 31, the three-dimensional data encoding device may add point 1 as a child node of point 0 and calculate the position information of point 0 as a predicted value. In this case, pred_mode=1 may be set. pred_mode is prediction mode information (prediction mode value) indicating the prediction mode. Furthermore, after calculating the predicted value, the three-dimensional data encoding device may calculate residual_value (prediction residual) of point 1. Here, residual_value is a difference value obtained by subtracting the predicted value calculated in the prediction mode indicated by pred_mode from the position information of each node. In this way, the three-dimensional data encoding method can improve encoding efficiency by encoding the difference value from the predicted value rather than the position information itself.

[0218] Next, as shown in (c) of FIG. 31, the three-dimensional data encoding device adds point 2 to the prediction tree. At this time, the three-dimensional data encoding device may search for the nearest point of point 2 from the point group already added to the prediction tree, and add point 2 as a child node of that nearest point. The position information of point 2 is represented by coordinates including three elements (x2, y2, z2). The position information of point 2 may be represented by coordinates in a three-axis Cartesian coordinate system or by coordinates in a polar coordinate system. In the case of FIG. 31, point 1 becomes the nearest point of point 2, and point 2 is added as a child node of point 1. Then, the three-dimensional data encoding device increments the value represented by child_count of point 1 by 1.

[0219] Next, as shown in (d) of FIG. 31, the three-dimensional data encoding device adds point 3 to the prediction tree. At this time, the three-dimensional data encoding device may search for the nearest point of point 3 from the group of points already added to the prediction tree, and add point 3 as a child node of that nearest point. The position information of point 3 is represented by coordinates including three elements, (x3, y3, z3). The position information of point 3 may be represented by coordinates in a three-axis Cartesian coordinate system, or by coordinates in a polar coordinate system. In the case of FIG. 31, point 0 is the nearest point of point 3, and point 3 is added as a child node of point 0. Then, the three-dimensional data encoding device increments the value represented by child_count of point 0 by 1.

[0220] In this way, the three-dimensional data encoding device adds all points to the prediction tree and completes generation of the prediction tree. When generation of the prediction tree is complete, the node with child_count=0 ultimately becomes a leaf of the prediction tree. After generation of the prediction tree is complete, the three-dimensional data encoding device encodes the child_count, pred_mode, and residual_value of each node selected in depth-first order from the root node. In other words, when selecting a node in depth-first order, the three-dimensional data encoding device selects an unselected child node from one or more child nodes of the selected node as the next node of the selected node. If the selected node does not have a child node, the three-dimensional data encoding device selects another unselected child node of the parent node of the selected node.

[0221] The encoding order is not limited to depth-first order, and may be width-first order, for example. When selecting a node in breadth-first order, the three-dimensional data encoding device selects, as the next node after the selected node, a node that has not yet been selected from one or more nodes at the same depth (layer) as the selected node. If there is no node at the same depth as the selected node, the three-dimensional data encoding device selects a node that has not yet been selected from one or more nodes at the next depth.

[0222] Note that points 0 to 3 are an example of a plurality of three-dimensional points.

[0223] In the above three-dimensional data encoding method, child_count, pred_mode, and residual_value are calculated when each point is added to the prediction tree, but this is not necessarily limited to this. For example, they may be calculated after the generation of the prediction tree is completed.

[0224] The input order of multiple 3D points to the 3D data encoding device may be such that the input 3D points are sorted in ascending or descending order of Morton order and processed in order starting from the first 3D point. This allows the 3D data encoding device to efficiently search for the nearest neighbor of the 3D point to be processed, thereby improving encoding efficiency. Alternatively, the 3D data encoding device may process the 3D points in the order in which they are input without sorting them. For example, the 3D data encoding device may generate a prediction tree without branches in the input order of multiple 3D points. Specifically, the 3D data encoding device may add the 3D point input after the first 3D point in the input order of the multiple 3D points as a child node of a specific 3D point.

[0225] Next, a first example of a prediction mode will be described with reference to Fig. 32. Fig. 32 is a diagram illustrating a first example of a prediction mode according to Embodiment 3. Fig. 32 is a diagram showing a part of a prediction tree.

[0226] Eight prediction modes may be set as shown below. For example, as shown in Fig. 32, a case where a predicted value of point c is calculated will be described as an example. In the prediction tree, it is shown that the parent node of point c is point p0, the grandparent node of point c is point p1, and the great-grandparent node of point c is point p2. Note that point c, point p0, point p1, and point p2 are examples of multiple three-dimensional points.

[0227] A prediction mode having a prediction mode value of 0 (hereinafter referred to as prediction mode 0) may be set to no prediction. That is, in prediction mode 0, the three-dimensional data encoding device may calculate position information of the input point c as a predicted value of the point c.

[0228] Furthermore, a prediction mode having a prediction mode value of 1 (hereinafter referred to as prediction mode 1) may be set to differential prediction with respect to point p0. That is, the three-dimensional data encoding device may calculate position information of point p0, which is the parent node of point c, as a predicted value of point c.

[0229] Furthermore, a prediction mode with a prediction mode value of 2 (hereinafter referred to as prediction mode 2) may be set to linear prediction using points p0 and p1. That is, the three-dimensional data encoding device may calculate, as a predicted value of point c, a prediction result by linear prediction using position information of point p0, which is the parent node of point c, and position information of point p1, which is the grandparent node of point c. Specifically, the three-dimensional data encoding device calculates the predicted value of point c in prediction mode 2 using the following equation T1.

[0230] Predicted value = 2 × p0-p1 (Equation T1) In the formula T1, p0 indicates the position information of the point p0, and p1 indicates the position information of the point p1.

[0231] Furthermore, a prediction mode with a prediction mode value of 3 (hereinafter referred to as prediction mode 3) may be set to Parallelogram prediction using points p0, p1, and p2. That is, the three-dimensional data encoding device may calculate, as a predicted value of point c, a prediction result by Parallelogram prediction using position information of point p0, which is the parent node of point c, position information of point p1, which is the grandparent node of point c, and position information of point p2, which is the great-grandparent node of point c. Specifically, the three-dimensional data encoding device calculates the predicted value of point c in prediction mode 3 using the following equation T2.

[0232] Predicted value = p0 + p1 - p2 (Equation T2)

[0233] In equation T2, p0 indicates the position information of point p0, p1 indicates the position information of point p1, and p2 indicates the position information of point p2.

[0234] Furthermore, a prediction mode having a prediction mode value of 4 (hereinafter referred to as prediction mode 4) may be set to differential prediction with respect to point p1. That is, the three-dimensional data encoding device may calculate position information of point p1, which is the grandparent node of point c, as a predicted value of point c.

[0235] Furthermore, a prediction mode having a prediction mode value of 5 (hereinafter referred to as prediction mode 5) may be set to differential prediction with respect to point p2. That is, the three-dimensional data encoding device may calculate position information of point p2, which is the great-grandfather node of point c, as a predicted value of point c.

[0236] Furthermore, a prediction mode having a prediction mode value of 6 (hereinafter referred to as prediction mode 6) may be set to the average of position information of two or more of points p0, p1, and p2. That is, the three-dimensional data encoding device may calculate, as a predicted value of point c, the average value of two or more of position information of point p0, which is the parent node of point c, position information of point p1, which is the grandparent node of point c, and position information of point p2, which is the great-grandparent node of point c. For example, when the three-dimensional data encoding device uses the position information of point p0 and the position information of point p1 to calculate a predicted value, it calculates the predicted value of point c in prediction mode 6 using the following formula T3.

[0237] Predicted value = (p0 + p1) / 2 (Equation T3)

[0238] In equation T3, p0 indicates the position information of point p0, and p1 indicates the position information of point p1.

[0239] Furthermore, a prediction mode with a prediction mode value of 7 (hereinafter referred to as prediction mode 7) may be set to nonlinear prediction using the distance d0 between points p0 and p1 and the distance d1 between points p2 and p1. In other words, the three-dimensional data encoding device may calculate the prediction result of nonlinear prediction using the distance d0 and the distance d1 as the predicted value of point c.

[0240] Note that the prediction methods assigned to each prediction mode are not limited to the above example. Furthermore, the above eight prediction modes and the above eight prediction methods do not have to be combined as described above, and any combination may be used. For example, when encoding prediction modes using entropy coding such as arithmetic coding, a frequently used prediction method may be assigned to prediction mode 0. This improves encoding efficiency. Furthermore, the three-dimensional data encoding device may improve encoding efficiency by dynamically changing the assignment of prediction modes according to the frequency of use of prediction modes while the encoding process is proceeding. For example, the three-dimensional data encoding device may count the frequency of use of each prediction mode during encoding, and assign a prediction mode indicated by a smaller value to a more frequently used prediction method. This improves encoding efficiency. Note that M is the number of prediction modes indicating the number of prediction modes. In the above example, there are eight prediction modes, prediction modes 0 to 7, so M=8.

[0241] The three-dimensional data encoding device may calculate a predicted value (px, py, pz) of the position information (x, y, z) of the three-dimensional point to be encoded using position information of a three-dimensional point that is close to the three-dimensional point to be encoded among three-dimensional points surrounding the three-dimensional point to be encoded. Furthermore, the three-dimensional data encoding device may add prediction mode information (pred_mode) to each three-dimensional point so that the predicted value calculated according to the prediction mode can be selected.

[0242] For example, in a case where the total number of prediction modes is M, it is conceivable to assign the position information of the nearest three-dimensional point p0 to prediction mode 0, ..., assign the position information of three-dimensional point p2 to prediction mode M-1, and add the prediction mode used for prediction to the bitstream for each three-dimensional point.

[0243] The number of prediction modes M may be added to the bitstream. Alternatively, the number of prediction modes M may not be added to the bitstream, but may have a value defined by a profile, level, or the like of a standard. Alternatively, the number of prediction modes M may use a value calculated from the number of three-dimensional points N used for prediction. For example, the number of prediction modes M may be calculated by M=N+1.

[0244] FIG. 33 is a diagram showing a second example of a table indicating predicted values ​​calculated in each prediction mode according to Embodiment 3. In FIG.

[0245] The table shown in FIG. 33 is an example in which the number of three-dimensional points used for prediction N=4 and the number of prediction modes M=5.

[0246] In a second example, the predicted value of the position information of point c is calculated using the position information of at least one of point p0, point p1, and point p2. A prediction mode is added to each 3D point to be coded. The predicted value is calculated to a value according to the added prediction mode.

[0247] FIG. 34 is a diagram showing a specific example of a second example of a table indicating predicted values ​​calculated in each prediction mode according to Embodiment 3. In FIG.

[0248] The three-dimensional data encoding device may, for example, select prediction mode 1 and encode the position information (x, y, z) of the three-dimensional point to be encoded using predicted values ​​(p0x, p0y, p0z). In this case, a prediction mode value of "1" indicating the selected prediction mode 1 is added to the bitstream.

[0249] In this way, when selecting a prediction mode, the three-dimensional data encoding device may select a prediction mode that is common to the three elements as one prediction mode for calculating predicted values ​​for each of the three elements included in the position information of the three-dimensional point to be encoded.

[0250] FIG. 35 is a diagram showing a third example of a table indicating predicted values ​​calculated in each prediction mode according to Embodiment 3. In FIG.

[0251] The table shown in FIG. 35 is an example in which the number of three-dimensional points used for prediction N=2 and the number of prediction modes M=5.

[0252] In a third example, the predicted value of the position information of point c is calculated using the position information of at least one of point p0 and point p1. A prediction mode is added to each three-dimensional point to be coded. The predicted value is calculated to a value according to the added prediction mode.

[0253] As in the third example, when the number of points around point c (the number of adjacent points) is less than three, a prediction mode to which a predicted value is not assigned may be set to "not available." Furthermore, when a prediction mode set to "not available" occurs, another prediction method may be assigned to the prediction mode. For example, position information of point p2 may be assigned as a predicted value to the prediction mode. Furthermore, a predicted value assigned to another prediction mode may be assigned to the prediction mode. For example, position information of point p1 assigned to prediction mode 4 may be assigned to prediction mode 3 set to "not available." At that time, new position information of point p2 may be assigned to prediction mode 4. In this way, when a prediction mode set to "not available" occurs, coding efficiency can be improved by assigning a new prediction method to the prediction mode.

[0254] Fig. 36 is a diagram illustrating an example of the syntax of the header of location information. NumNeighborPoint, NumPredMode, Thfix, QP, and unique_point_per_leaf in the syntax of Fig. 36 will be described in order.

[0255] NumNeighborPoint indicates the upper limit of the number of surrounding points used to generate the predicted value of the position information of the three-dimensional point. When the number of surrounding points M is less than NumNeighborPoint (M < NumNeighborPoint), in the calculation process of the predicted value, the predicted value may be calculated using M surrounding points.

[0256] NumPredMode indicates the total number M of prediction modes used for predicting the position information. Note that the maximum value MaxM of the possible values of the number of prediction modes may be defined by a standard or the like. The three-dimensional data encoding device may add the value of (MaxM - M) (0 < M <= MaxM) as NumPredMode to the header, and binarize and encode (MaxM - 1) with a truncated unary code. Also, the number of prediction modes NumPredMode may not be added to the bit stream, and may be defined by a profile or level in a standard or the like. Also, the number of prediction modes may be defined by NumNeighborPoint + NumPredMode.

[0257] Thfix is a threshold for determining whether to fix the prediction mode. The distance d0 between the point p1 and the point p0 used for prediction and the distance d1 between the point p2 and the point p1 are calculated, and if the absolute difference distdiff = |d0 - d1| is smaller than the threshold Thfix[i], the prediction mode is fixed to α. α is a prediction mode for calculating the predicted value using linear prediction, and is "2" in the above embodiment. Note that Thfix may not be added to the bit stream, and may be defined by a profile or level in a standard or the like.

[0258] <​​​​unique_point_per_leaf is information that indicates whether or not a duplicated point (a point with the same position information) is included in the bitstream. unique_point_per_leaf=1 indicates that there is no duplicated point in the bitstream. unique_point_per_leaf=0 indicates that there is one or more duplicated points in the bitstream.

[0260] In this embodiment, the determination of whether to fix the prediction mode is performed using the absolute difference between the distance d0 and the distance d1, but this is not necessarily limited to this, and any method may be used for the determination. For example, this determination may be performed by calculating the distance d0 between point p1 and point p0, and if the distance d0 is greater than a threshold, determining that point p1 cannot be used for prediction, and fixing the prediction mode value to "1" (prediction value p0); otherwise, setting the prediction mode. This can reduce overhead and improve coding efficiency.

[0261] The above NumNeighborPoint, NumPredMode, Thfix, and unique_point_per_leaf may be entropy coded and added to the header. For example, each value may be binarized and then computationally coded. Furthermore, each value may be coded at a fixed length to reduce the amount of processing.

[0262] Fig. 37 is a diagram illustrating an example of the syntax of the position information. NumOfPoint, child_count, pred_mode, and residual_value[j] in the syntax of Fig. 37 will be described in order.

[0263] NumOfPoint indicates the total number of 3D points contained in the bitstream.

[0264] child_count indicates the number of child nodes that the i-th 3D point (node[i]) has.

[0265] pred_mode indicates a prediction mode for encoding or decoding position information of the i-th three-dimensional point. pred_mode takes on values ​​ranging from 0 to M-1 (M is the total number of prediction modes). If pred_mode is not present in the bitstream (if the condition distdiff >= Thfix[i] && NumPredMode > 1 is not satisfied), pred_mode may be estimated to a fixed value α. α is a prediction mode for calculating a predicted value using linear prediction, and is set to "2" in the above embodiment. Note that α is not limited to "2" and may be set to any value between 0 and M-1 as an estimated value. Furthermore, if pred_mode is not present in the bitstream, the estimated value may be added to a header or the like. Furthermore, pred_mode may be binarized using a truncated unary code using the number of prediction modes to which the predicted value is assigned, and then arithmetically coded.

[0266] Note that when NumPredMode=1, that is, when the number of prediction modes is 1, the three-dimensional data encoding device may generate a bitstream that does not include a prediction mode value without encoding a prediction mode value that indicates the prediction mode. Furthermore, when the three-dimensional data decoding device acquires a bitstream that does not include a prediction mode value, it may calculate a prediction value of a specific prediction mode when calculating a prediction value. The specific prediction mode is a predetermined prediction mode.

[0267] residual_value[j] indicates coded data of the prediction residual between the predicted value of the position information. residual_value[0] may indicate element x of the position information, residual_value[1] may indicate element y of the position information, and residual_value[2] may indicate element z of the position information.

[0268] Fig. 38 is a diagram illustrating another example of the syntax of the location information. The example in Fig. 38 is a modified example of the example in Fig. 37.

[0269] As shown in Fig. 38, pred_mode may indicate a prediction mode for each of the three elements of position information (x, y, z). That is, pred_mode[0] indicates the prediction mode of element x, pred_mode[1] indicates the prediction mode of element y, and pred_mode[2] indicates the prediction mode of element z. pred_mode[0], pred_mode[1], and pred_mode[2] may be added to a bitstream.

[0270] (Fourth embodiment) In this embodiment, inter prediction for position information (octree) of a point group (point cloud) will be described. Fig. 39 is a block diagram of a three-dimensional data encoding device 12800 according to this embodiment. Note that while Fig. 39 shows a processing unit related to encoding position information (geometry) of a point group, the three-dimensional data encoding device 12800 may also include other processing units, such as a processing unit that encodes attribute information of a point group. In inter prediction, a point group to be encoded is encoded while referring to an already-encoded point group.

[0271] The three-dimensional data encoding device 12800 includes an octree conversion unit 12801, a buffer 12802, an entropy encoding unit 12803, a buffer 12804, a buffer 12805, a point group conversion unit 12806, a buffer 12807, a motion detection and compensation unit 12808, an octree conversion unit 12809, a buffer 12810, and a control unit 12811.

[0272] The octree generator 12801 converts the input target point cloud, which is data of a point cloud to be encoded, into an octree representation, thereby generating a target octree in which the position information of the target point cloud is expressed in an octree. In the input target point cloud, the positions of the point cloud are expressed, for example, by three-dimensional coordinates (e.g., x, y, z). The buffer 12802 stores the generated target octree. The octree is composed of multiple nodes (branching points), and the information of each node includes an 8-bit occupancy code indicating whether each of the node's eight child nodes includes a three-dimensional point. For example, the buffer 12802 may initialize the data stored for each octree (target point cloud).

[0273] The entropy coding unit 12803 generates a bitstream by entropy coding information for each node (for example, occupancy code). In this entropy coding, a probability parameter (also called a coding table or a probability table) is controlled based on information of an encoded node (intra-reference node) in the target point cloud or information of a node (inter-reference node) in the encoded point cloud.

[0274] The buffer 12804 stores information of the target node (for example, occupancy code) as an intra reference node (encoded node). For example, the buffer 12804 may initialize the data stored for each octree (target point group).

[0275] The buffer 12805 stores information about the target node (e.g., occupancy code). The buffer 12805 also stores information about the target node in octree units as an encoded octree. For example, the buffer 12805 may initialize the data stored for each octree (target point group).

[0276] The point group generation unit 12806 generates an inter reference point group (encoded point group) by converting the encoded octree into a point group. The buffer 12807 stores the inter reference point group. That is, the buffer 12807 stores multiple inter reference point groups, which are one or more encoded point groups.

[0277] The motion detection and compensation unit 12808 detects (motion detection) the displacement between the inter reference point group and the target point group, and corrects (motion compensation) the inter reference point group based on the detected displacement to generate a post-alignment point group, which is the inter reference point group after alignment.

[0278] The octree generation unit 12809 converts the aligned point cloud into an octree representation to generate an inter-reference octree in which the position information of the aligned point cloud is expressed as an octree. The buffer 12810 stores the generated inter-reference octree. Furthermore, for example, the buffer 12810 may initialize the data stored for each octree (target point cloud).

[0279] Furthermore, the three-dimensional data encoding device 12800 may perform motion detection and motion compensation on a frame or octree basis, or on a node (point) basis. Furthermore, the three-dimensional data encoding device 12800 may write information related to motion compensation, such as a motion vector, in the header section of the frame or octree, or may entropy-encode the information and then write it in the header section of the node information.

[0280] In addition, the inter-reference point group may be a point group included in an already coded frame different from the frame to be coded, or may be an already coded point group included in the same frame as the frame to be coded.

[0281] The control unit 12811 controls the probability parameters used by the entropy coding unit 12803 for entropy coding (arithmetic coding) of the target node, using an intra reference node stored in the buffer 12804 or an inter reference node included in the inter reference octree stored in the buffer 12810. Whether to use control of the probability parameters using an intra reference node (hereinafter referred to as intra reference) or control of the probability parameters using an inter reference node (hereinafter referred to as inter reference) may be determined in advance for each frame or point group, or may be determined by any method. For example, the actual amount of code may be estimated, and a reference method (intra reference or inter reference) that results in the smallest amount of code may be selected.

[0282] For example, when intra-referencing is used, a probability parameter is selected from a plurality of probability parameters based on the occupancy state (whether or not a node contains a point) of a plurality of adjacent nodes (intra-reference nodes) of the target node. When inter-referencing is used, a probability parameter is selected from a plurality of probability parameters based on the occupancy state of a node (inter-reference node) at the same position as the target node and at least one of the adjacent nodes included in the inter-reference octree. When inter-referencing is selected, the probability parameter may be controlled by combining inter-referencing and intra-referencing. The plurality of probability parameters may include a probability parameter that is updated according to the frequency of occurrence, or may include a fixed value.

[0283] In this way, the three-dimensional data encoding device 12800 controls the probability parameters of entropy encoding based on the information of the inter-reference node as well as the information of the intra-reference node, thereby improving the prediction accuracy of the occurrence probability of the information of the target node, which may improve encoding efficiency.

[0284] Note that the three-dimensional data encoding device 12800 does not need to constantly refer to the inter reference point group, and may encode the target point group based only on the information of the target point group by clearing the buffer 12807 storing the inter reference point group at a predetermined time interval (e.g., every second), a predetermined frame interval (e.g., every 30 frames), or at any timing notified to the three-dimensional data decoding device. This makes it possible for the three-dimensional data decoding device to start interrupt playback from a point group other than the beginning of the bitstream that does not reference the inter reference point group. This may potentially improve the random accessibility and error resistance of the bitstream.

[0285] Fig. 40 is a block diagram of a three-dimensional data decoding device 12820 according to this embodiment. Note that Fig. 40 illustrates a processing unit related to decoding of position information (geometry) of a point cloud, but the three-dimensional data decoding device 12820 may also include other processing units, such as a processing unit that decodes attribute information of the point cloud. The three-dimensional data decoding device 12820 performs inter-prediction decoding to decode a point cloud from an encoded bitstream while referring to the encoded point cloud. For example, the three-dimensional data decoding device 12820 decodes a bitstream generated by the three-dimensional data encoding device 12800 shown in Fig. 39.

[0286] The three-dimensional data decoding device 12820 includes an entropy decoding unit 12821, a buffer 12822, a buffer 12823, a point group generation unit 12824, a buffer 12825, a motion compensation unit 12826, an octree generation unit 12827, a buffer 12828, and a control unit 12829.

[0287] The entropy decoding unit 12821 generates information of the decoded node (for example, occupancy code) by entropy decoding the input bit stream for each branch point (node) of the octree. In this entropy decoding, a probability parameter (also called a coding table or a probability table) is controlled based on information of a decoded node (intra reference node) in the target point cloud or information of a node (inter reference node) in the decoded point cloud.

[0288] The buffer 12822 holds information about the generated decoded nodes as intra reference nodes (decoded nodes). For example, the buffer 12822 may initialize the data held for each octree (decoded point group).

[0289] The buffer 12823 stores information about the decoding nodes (e.g., occupancy codes). The buffer 12823 also stores information about the decoding nodes in units of octrees as decoded octrees. For example, the buffer 12823 may initialize the data stored for each octree (decoded point cloud). The point cloud generation unit 12824 generates a decoded point cloud by converting the decoded octree into a point cloud.

[0290] The buffer 12825 stores the decoded point group as an inter reference point group. The motion compensation unit 12826 corrects (motion compensates) the displacement between the inter reference point group and the point group to be decoded, thereby generating an aligned point group that is the inter reference point group after alignment. For example, the motion compensation unit 12826 obtains information related to motion compensation, such as a motion vector, from the header section of a frame or octree, or the header section of node information, and performs motion compensation using the obtained information.

[0291] The octree generator 12827 converts the aligned point cloud into an octree representation to generate an inter-reference octree in which position information of the aligned point cloud is expressed as an octree. The buffer 12828 stores the generated inter-reference octree. Furthermore, for example, the buffer 12828 may initialize the data stored for each octree (decoded point cloud).

[0292] Furthermore, the three-dimensional data decoding device 12820 may perform motion compensation in units of frames or octrees, or may perform it for each node (point).

[0293] Furthermore, the inter-reference point group may be a point group included in a decoded frame different from the frame to be decoded, or may be a decoded point group included in the same frame as the frame to be decoded.

[0294] The control unit 12829 controls the probability parameters used by the entropy decoding unit 12821 for entropy decoding (arithmetic decoding) of the target node, using the intra reference node stored in the buffer 12822 or the inter reference node included in the inter reference octree stored in the buffer 12828. Whether intra reference or inter reference is used may be determined based on control information included in the bitstream, may be predetermined for each frame or point group, or may be determined by any method, for example.

[0295] For example, when intra-referencing is used, the probability parameter is selected based on the occupancy state (whether or not a node contains a point) of multiple adjacent nodes (intra-reference nodes) of the target node. Also, when inter-referencing is used, the probability parameter is selected based on the occupancy state of a node (inter-reference node) that is in the same position as the target node and at least one of the multiple adjacent nodes included in the inter-reference octree. Note that when inter-referencing is selected, the probability parameter may be controlled by combining inter-referencing and intra-referencing.

[0296] In this way, the three-dimensional data decoding device 12820 can control the probability parameters of entropy decoding based on information on inter reference nodes as well as information on intra reference nodes, thereby decoding a point cloud from an encoded bit stream (for example, a bit stream output from the three-dimensional data encoding device 12800 shown in Figure 39) while referring to the encoded point cloud.

[0297] Note that the three-dimensional data decoding device 12820 does not need to constantly refer to the inter reference point group, and may decode the point group to be decoded based only on the information of the point group to be decoded by clearing the buffer 12825 storing the inter reference point group in accordance with the three-dimensional data encoding device, at a predetermined time interval (for example, every second), a predetermined frame interval (for example, every 30 frames), or at any timing notified by the three-dimensional data encoding device. This allows the three-dimensional data decoding device 12820 to realize interrupt playback start from a point group other than the beginning of the bitstream that does not refer to the inter reference point group.

[0298] 41 is a block diagram of a three-dimensional data encoding device 12800A, which is a modified example of the three-dimensional data encoding device 12800. The three-dimensional data encoding device 12800A shown in FIG. 41 further comprises a motion compensation unit 12812 in addition to the components of the three-dimensional data encoding device 12800 shown in FIG.

[0299] The motion compensation unit 12812 performs motion compensation on the coded point cloud generated by the point cloud generation unit 12806, thereby aligning it with the inter reference point cloud already stored in the buffer 12807. The buffer 12807 updates the inter reference point cloud to be stored by integrating the coded point cloud after motion compensation with the inter reference point cloud already stored. This makes it possible to use a dense point cloud obtained by overlapping point clouds from multiple frames as the inter reference point cloud. Other processing is similar to that of, for example, the three-dimensional data encoding device 12800.

[0300] The inter-reference point group may be a group of points included in an already coded frame different from the frame to be coded, or may be a group of coded points included in the same frame as the frame to be coded.

[0301] In this way, the three-dimensional data encoding device 12800A may be able to improve the point cloud density of the inter-reference point cloud by aligning and integrating the encoded point clouds. This may improve the prediction accuracy of the occurrence probability of the information of the target node, thereby further improving the encoding efficiency.

[0302] The three-dimensional data encoding device 12800A does not need to reference all encoded point groups as inter-reference point groups. Instead, it may encode the target point group based on only the target point group, or on the target point group to be encoded and some of the encoded point groups, by clearing all or part of the buffer 12807 storing the inter-reference point group at a predetermined time interval (e.g., every second), a predetermined frame interval (e.g., every five frames), or at any timing notified to the three-dimensional data decoding device. When encoding is performed based on only the target point group, the three-dimensional data decoding device can jump in and start playback from a point group other than the beginning of the bitstream that does not reference the inter-reference point group. This may improve the random accessibility and error resilience of the bitstream. Furthermore, when encoding is performed based on the target point group and some of the encoded point groups, the capacity of the buffer 12807 storing the inter-reference point group can be reduced, potentially reducing the implementation costs of the three-dimensional data encoding device and the three-dimensional data decoding device.

[0303] Figure 42 is a block diagram of a three-dimensional data decoding device 12820A, which is a modified example of the three-dimensional data decoding device 12820. The three-dimensional data decoding device 12820A shown in Figure 42 further comprises a motion compensation unit 12830 in addition to the components of the three-dimensional data decoding device 12820 shown in Figure 40. For example, the three-dimensional data decoding device 12820A decodes a point group from the bitstream generated by the three-dimensional data encoding device 12800A shown in Figure 41.

[0304] The motion compensation unit 12830 performs motion compensation on the decoded point group to align it with the inter reference point group already stored in the buffer 12825. The buffer 12825 updates the inter reference point group to be stored by integrating the decoded point group after motion compensation with the inter reference point group already stored. This makes it possible to use a dense point group obtained by overlapping point groups from multiple frames as the inter reference point group. Other processing is similar to that of, for example, the 3D data decoding device 12820.

[0305] The inter-reference point group may be a point group included in a decoded frame different from the frame to be decoded, or may be a decoded point group included in the same frame as the frame to be decoded.

[0306] In this way, the three-dimensional data decoding device 12820A has a configuration that aligns and integrates decoded point clouds, and can decode point clouds from a bit stream encoded by a three-dimensional data encoding device having a similar configuration (for example, a bit stream generated by the three-dimensional data encoding device 12800A shown in Figure 41).

[0307] In addition, the three-dimensional data decoding device 12820A does not need to refer to all decoded point groups as inter-reference point groups, and may decode the point group to be decoded based on information on only the point group to be decoded, or on the point group to be decoded and some of the decoded point groups, by clearing all or part of the buffer 12825 storing the inter-reference point group at a predetermined time interval (e.g., every second), a predetermined frame interval (e.g., every five frames), or at any timing notified by the three-dimensional data encoding device.

[0308] When the three-dimensional data decoding device 12820A performs decoding based only on information about the point group to be decoded, it can realize interrupted playback start from a point group other than the beginning of the bitstream that does not reference an inter-reference point group. This may improve the random accessibility and error resistance of the bitstream. Furthermore, when the three-dimensional data decoding device performs decoding based on information about the point group to be decoded and some of the decoded point groups, it may reduce the capacity of the buffer 12825 that holds the inter-reference point group. This may reduce the implementation costs of the three-dimensional data encoding device and the three-dimensional data decoding device.

[0309] Fig. 43 is a diagram showing an example of inter prediction in the 3D data encoding device shown in Fig. 39 and Fig. 41. Note that the inter prediction in the 3D data decoding device shown in Fig. 40 and Fig. 42 is similar.

[0310] As shown in Figure 43, for example, the three-dimensional data encoding device sets a first rectangular parallelepiped containing a target point cloud. The three-dimensional data encoding device sets a second rectangular parallelepiped obtained by translating the first rectangular parallelepiped. The second rectangular parallelepiped is a space containing an encoded point cloud that is referenced in encoding the target point cloud. Furthermore, the three-dimensional data encoding device may write the x, y, and z components of the translation distance between the first rectangular parallelepiped and the second rectangular parallelepiped as motion vector information in the header section of the frame or octree, or may entropy-code the information and then write it in the header section of the node information.

[0311] Note that, although an example of setting a space containing an encoded point group to be referenced in encoding a target point group by parallel translation has been shown here, any other method may be used as long as it can uniquely set a space containing an encoded point group to be referenced.

[0312] Next, an example of header information will be described. Fig. 44 is a diagram showing an example of the syntax of a sequence parameter set (SPS) included in a bitstream. The SPS is control information common to multiple frames, multiple point groups, or multiple slices, and is control information common to attribute information and position information.

[0313] As shown in FIG. 44, the SPS includes sps_inter_prediction_enabled_flag and sps_max_num_ref_frames_minus1.

[0314] Below are example semantics of sequence parameter sets: sps_inter_prediction_enabled_flag equal to 1 specifies that the use of inter prediction is allowed for bitstreams that reference an SPS. sps_inter_prediction_enabled_flag equal to 0 specifies that inter prediction is disabled for bitstreams that reference an SPS.

[0315] sps_max_num_ref_frames_minus1+1 (sps_max_num_ref_frames_minus1 plus 1) specifies the maximum number of reference point cloud frames referenced by a frame. The value of sps_max_num_ref_frames_minus1 must be in the range from 0 to MaxNumRefFrames-1.

[0316] sps_max_num_ref_frames_minus1 is included in the SPS when sps_inter_prediction_enabled_flag is 1, and is not included in the SPS when sps_inter_prediction_enabled_flag is 0.

[0317] 45 is a diagram showing an example of the syntax of a location information parameter set (GPS) included in a bitstream. GPS is control information common to multiple frames, multiple point clouds, or multiple slices, and is control information for location information.

[0318] As shown in FIG. 45, GPS includes gps_inter_prediction_enabled_flag and gps_num_ref_frames_minus1.

[0319] The following are example semantics of the location parameter set: gps_inter_prediction_enabled_flag equal to 1 specifies that the use of inter prediction is allowed in the decoding process of location data units of the bitstream that references GPS. gps_inter_prediction_enabled_flag equal to 0 specifies that inter prediction is disabled in the decoding process of location data units of the bitstream that references GPS. If sps_inter_prediction_enabled_flag is 0, then gps_inter_prediction_enabled_flag is 0.

[0320] gps_num_ref_frames_minus1+1 (gps_num_ref_frames_minus1 plus 1) specifies the number of reference point cloud frames referenced by the GPS reference frame. The value of gps_num_ref_frames_minus1 must be in the range of 0 to sps_max_num_ref_frames_minus1.

[0321] gps_num_ref_frames_minus1 is included in GPS when gps_inter_prediction_enabled_flag is 1, and is not included in GPS when gps_inter_prediction_enabled_flag is 0.

[0322] As shown in these examples, the 3D data encoding device may notify the 3D data decoding device of information indicating whether or not inter-prediction coding is permitted, such as sps_inter_prediction_enabled_flag and gps_inter_prediction_enabled_flag, in the sequence parameter set and the position information parameter set. Furthermore, when notifying the 3D data decoding device of information indicating that inter-prediction coding is permitted, the 3D data encoding device may notify the 3D data decoding device of information regarding the number of frames to reference in inter-prediction coding or its maximum value, such as sps_max_num_ref_frames_minus1 and gps_num_ref_frames_minus1.

[0323] In addition, MaxNumRefFrames is a fixed value that is set as a requirement that the three-dimensional data decoding device must satisfy, and it may be set to a number of frames, such as six frames, but it may also be a larger value as long as both the three-dimensional data encoding device and the three-dimensional data decoding device use the same value for processing.

[0324] By transmitting this information from the three-dimensional data encoding device to the three-dimensional data decoding device, it may be possible to optimize the allocation of memory used for processing in the three-dimensional data decoding device.

[0325] Furthermore, the information indicating whether or not inter-prediction coding is permitted and the information regarding the number of frames to be referenced in inter-prediction coding or its maximum value may be stored in both the SPS and the GPS, or in only one of them. Furthermore, this information may be stored in control information other than the SPS and the GPS.

[0326] 39 to 45 may be implemented in combination with at least a part of other disclosures in the present disclosure. Also, some of the devices, processes, and syntaxes disclosed using Figures 39 to 45 may be implemented in combination with other disclosures.

[0327] Furthermore, not all of the components disclosed using FIGS. 39 to 45 are always necessary, and each device may include only some of the components.

[0328] As described above, the three-dimensional data encoding device according to this embodiment performs the processing shown in FIG. 46. The three-dimensional data encoding device performs motion compensation on a plurality of encoded point groups (S12801). The three-dimensional data encoding device generates a reference point group (for example, the inter-reference point group shown in FIG. 41) by integrating (combining) a plurality of encoded point groups after motion compensation (S12802). The three-dimensional data encoding device generates an N-ary tree structure (N is an integer equal to or greater than 2) of the target point group (for example, the target octree shown in FIG. 41) (S12803). The three-dimensional data encoding device encodes the N-ary tree structure of the target point group using the reference point group (S12804). Note that N is, for example, 8, but may be any power of 2 or any other value.

[0329] According to this, the three-dimensional data encoding device can improve encoding efficiency by encoding the target point group using a reference point group that is an integration of multiple encoded point groups.

[0330] For example, in encoding the N-ary tree structure of the target point group (S12804), the three-dimensional data encoding device performs motion compensation on the reference point group relative to the target point group, generates an N-ary tree structure of the reference point group after motion compensation (for example, the inter-reference octree shown in Figure 41), and encodes the N-ary tree structure of the target point group using the N-ary tree structure of the reference point group.

[0331] For example, in encoding the N-ary tree structure of the target point cloud (S12804), the three-dimensional data encoding device entropy encodes the N-ary tree structure of the target point cloud and controls a probability parameter used in the entropy encoding based on the reference point cloud. For example, the three-dimensional data encoding device selects a probability parameter to use from a plurality of probability parameters based on the reference point cloud.

[0332] For example, a three-dimensional data encoding device generates an encoded target point group (e.g., the encoded point group shown in Figure 41) from the N-ary tree structure of the target point group, performs motion compensation on the encoded target point group relative to the reference point group, and updates the reference point group by integrating the encoded target point group after motion compensation into the reference point group.

[0333] For example, each of the plurality of encoded point groups belongs to a different frame from the target point group, or each of the plurality of encoded point groups belongs to the same frame as the target point group.

[0334] For example, the three-dimensional data encoding device stores first information (e.g., sps_inter_prediction_enabled_flag or gps_inter_prediction_enabled_flag) indicating whether encoding using a reference point group is permitted in control information common to multiple point groups (e.g., SPS or GPS).

[0335] For example, when the first information indicates that encoding using a reference point cloud is permitted, the three-dimensional data encoding device stores second information (e.g., sps_max_num_ref_frames_minus1 or gps_num_ref_frames_minus1) related to the number of multiple encoded point clouds in control information (e.g., SPS or GPS) common to the multiple point clouds. For example, the second information indicates the number or maximum number of encoded point clouds to be integrated.

[0336] For example, the three-dimensional data encoding device includes a processor and a memory, and the processor performs the above-mentioned processing using the memory.

[0337] Furthermore, the three-dimensional data decoding device according to this embodiment performs the processing shown in FIG. 47. The three-dimensional data decoding device performs motion compensation on multiple decoded point groups (S12811). The three-dimensional data decoding device generates a reference point group (for example, the inter-reference point group shown in FIG. 42) by integrating (combining) multiple decoded point groups after motion compensation (S12812). The three-dimensional data decoding device decodes an N-ary tree structure (N is an integer equal to or greater than 2) (for example, the decoded octree shown in FIG. 42) of the target point group using the reference point group (S12813). That is, the three-dimensional data decoding device obtains the N-ary tree structure of the target point group by decoding a bit stream (encoded data) generated by encoding the N-ary tree structure of the target point group. The three-dimensional data decoding device generates a decoded point group of the target point group from the N-ary tree structure of the target point group (S12814). Note that N is, for example, 8, but may be any power of 2 or any other value.

[0338] This allows the three-dimensional data decoding device to decode the target point group using a reference point group that is an integration of multiple decoded point groups.

[0339] For example, in decoding the N-ary tree structure of the target point group (S12813), the three-dimensional data decoding device performs motion compensation on the reference point group relative to the target point group, generates an N-ary tree structure of the reference point group after motion compensation (for example, the inter-reference octree shown in Figure 42), and decodes the N-ary tree structure of the target point group using the N-ary tree structure of the reference point group.

[0340] For example, in decoding the N-ary tree structure of the target point group (S12813), the three-dimensional data decoding device entropy-decodes the N-ary tree structure of the target point group and controls probability parameters used in the entropy decoding based on the reference point group. For example, the three-dimensional data decoding device selects a probability parameter to be used from a plurality of probability parameters based on the reference point group.

[0341] For example, the three-dimensional data decoding device performs motion compensation on the decoded point group of the target point group relative to the reference point group, and integrates the decoded point group after motion compensation with the reference point group, thereby updating the reference point group.

[0342] For example, each of the decoded point groups belongs to a different frame from the target point group. For example, each of the decoded point groups belongs to the same frame as the target point group.

[0343] For example, the three-dimensional data decoding device obtains first information (e.g., sps_inter_prediction_enabled_flag or gps_inter_prediction_enabled_flag) indicating whether decoding using a reference point group is permitted from control information common to multiple point groups (e.g., SPS or GPS).

[0344] For example, when the first information indicates that decoding using a reference point group is permitted, the three-dimensional data decoding device obtains second information (e.g., sps_max_num_ref_frames_minus1 or gps_num_ref_frames_minus1) related to the number of decoded point groups from control information (e.g., SPS or GPS) common to the multiple point groups. For example, the second information indicates the number or maximum number of decoded point groups to be integrated.

[0345] For example, the three-dimensional data decoding device includes a processor and a memory, and the processor performs the above processing using the memory.

[0346] (Embodiment 5) In this embodiment, a case will be described in which inter prediction and intra prediction are switched and performed on position information of a point group (point cloud).

[0347] Fig. 48 is a block diagram of a three-dimensional data encoding device 12900 according to this embodiment. Note that, although Fig. 48 describes a processing unit related to encoding position information (geometry) of a point cloud, the three-dimensional data encoding device 12900 may also include other processing units, such as a processing unit that encodes attribute information of a point cloud. In inter prediction and intra prediction, a point cloud to be encoded is encoded while referring to an already-encoded point cloud.

[0348] The three-dimensional data encoding device 12900 includes a grouping unit 12901, a buffer 12902, a quantization unit 12903, an inverse quantization unit 12904, a buffer 12905, an intra prediction unit 12906, a buffer 12907, a motion estimation and compensation unit 12908, an inter prediction unit 12909, a switching unit 12910, and an entropy encoding unit 12911.

[0349] The grouping unit 12901 extracts a group of points to be used as a prediction tree (Predtree), which is a unit of encoding, from the input target point group, which is data of a group of points to be encoded, and sets the extracted points as one group. In the input target point group, the positions of the points are expressed, for example, by three-dimensional coordinates (e.g., x, y, z). The buffer 12902 stores the generated prediction tree. For example, the buffer 12902 may initialize the data stored for each prediction tree. A process for encoding is sequentially performed on each of the multiple three-dimensional points included in the prediction tree (Predtree) stored in the buffer 12902. The three-dimensional coordinates may be expressed in Cartesian coordinates or polar coordinates. Hereinafter, position information expressed in Cartesian coordinates will be referred to as position information in a Cartesian coordinate system, and position information expressed in polar coordinates will be referred to as position information in a polar coordinate system.

[0350] Then, a difference (first residual signal) between each of the multiple 3D points included in the prediction tree (Predtree) and the selected prediction point is calculated. This first residual signal is also called a prediction residual. The first residual signal is an example of a first residual.

[0351] The quantization unit 12903 quantizes the first residual signal. The entropy coding unit 12911 entropy codes the quantized first residual signal to generate coded data, and outputs (generates) a bitstream including the coded data.

[0352] The inverse quantization unit 12904 inversely quantizes the first residual signal quantized by the quantization unit 12903. The inversely quantized first residual signal is added to a predicted value based on selected prediction points (one or more candidate points) to be decoded as a 3D point (reference point) used for intra prediction and inter prediction. Note that, as described in the above embodiment, the predicted value is calculated based on position information of one or more candidate points. The buffer 12905 holds a group of decoded reference points for intra prediction. For example, the buffer 12905 may initialize the data held for each prediction tree (target point group). Furthermore, the buffer 12907 holds a group of reference points for inter prediction. For example, the buffer 12907 may initialize the data held for each prediction tree (target point group).

[0353] The intra prediction unit 12906 determines the intra prediction point to be used for prediction using a predetermined method, with reference to information in the prediction tree (Predtree) including the 3D point to be coded, such as multiple 3D points (a group of reference points for intra prediction) included in the prediction tree (Predtree). For example, the intra prediction unit 12906 may determine the intra prediction point by extrapolating using two 3D points (decoded points) that have been dequantized immediately before the 3D point to be coded (for example, an ancestor node such as a parent node of the prediction tree).

[0354] The motion detection and compensation unit 12908 reconstructs the encoded point group based on multiple three-dimensional points (multiple decoded points) included in a prediction tree (Predtree) that includes the three-dimensional point to be encoded, detects (motion detection) the displacement between the encoded point group and the point group to be encoded, and corrects (motion compensation) the encoded point group based on the detected displacement, thereby generating an inter-prediction point group, which is a reference point group for inter-prediction after alignment.

[0355] The inter prediction unit 12909 determines an inter prediction point to be used for prediction using a predetermined method based on the motion-compensated inter prediction point group. For example, the inter prediction unit 12909 may select a point closest to the intra prediction point from the inter prediction point group as the inter prediction point, or may select a 3D point (for example, a 3D point closest to the 3D point coded last) near the 3D point coded last (for example, a 3D point corresponding to an ancestor node such as a parent node of a prediction tree) as the inter prediction point without referring to the intra prediction point.

[0356] The switching unit 12910 selects either an intra-prediction point or an inter-prediction point to determine a prediction point to be used for prediction. In this way, the 3D data encoding device 12900 determines position information of one or more candidate points from among a plurality of encoded 3D points as prediction points, and calculates a predicted value based on the prediction point. The intra-prediction unit 12906 and the inter-prediction unit 12909 determine a prediction point (intra-prediction point or inter-prediction point) based on a 3D point encoded immediately before the 3D point to be encoded. In other words, the 3D data encoding device 12900 determines one or more candidate points for calculating a predicted value based on one or more reference points from among a plurality of encoded 3D points. The one or more reference points are 3D points encoded immediately before the 3D point to be encoded, and may be, for example, 3D points corresponding to a parent node (ancestor node) of the 3D point to be encoded in a prediction tree.

[0357] The 3D data encoding device 12900 may select either an intra-prediction point or an inter-prediction point as a prediction point according to the procedure of the flowchart in FIG. 50, which will be described later. Prediction-related information (prediction information) for selecting whether an intra-prediction point or an inter-prediction point is to be used as a prediction point may be entropy-encoded and written in the header of each 3D point, or may be interleaved with each 3D point. Motion compensation information such as a motion vector may be written in the header of a frame or a prediction tree (Predtree), or may be entropy-encoded and written in the header of each 3D point, or may be interleaved with each 3D point. The reference points for inter prediction may be a point group included in a coded frame different from the frame to be coded, or may be a coded point group included in the same frame as the frame to be coded.

[0358] In this way, the three-dimensional data encoding device 12900 can predict the point to be encoded using inter prediction in addition to intra prediction, thereby reducing the amount of information in the first residual signal to be entropy encoded, potentially improving encoding efficiency.

[0359] It should be noted that the 3D data encoding device 12900 does not need to constantly refer to inter-prediction points, and may instead perform encoding based only on information about the point group to be encoded by initializing a buffer 12907 that stores the inter-prediction reference point group at a predetermined time interval (e.g., every second), a predetermined frame interval (e.g., every 30 frames), or at any timing notified to the 3D data decoding device. This makes it possible for a corresponding 3D data decoding device to start interrupt playback from a point group that is not at the beginning of a bitstream that does not reference inter-prediction points, potentially improving the random accessibility or error resistance of the bitstream.

[0360] When the input point group to be encoded has coordinates expressed in Cartesian coordinates as position information, the three-dimensional data encoding device 12900 encodes the position information indicated by the coordinates expressed in Cartesian coordinates.When the input point group to be encoded has coordinates expressed in polar coordinates as position information, the three-dimensional data encoding device 12900 encodes the position information indicated by the coordinates expressed in polar coordinates.

[0361] Fig. 49 is a block diagram of a three-dimensional data decoding device 12920 according to this embodiment. Note that while Fig. 49 describes processing units related to decoding of point cloud position information (geometry), the three-dimensional data decoding device 12920 may also include other processing units, such as a processing unit that decodes point cloud attribute information. The three-dimensional data decoding device 12920 performs inter-prediction decoding to decode a point cloud to be decoded while referencing a point cloud that has already been decoded. For example, the three-dimensional data decoding device 12920 decodes a bitstream generated by the three-dimensional data encoding device 12900 shown in Fig. 48.

[0362] The three-dimensional data decoding device 12920 includes an entropy decoding unit 12921, an inverse quantization unit 12922, a buffer 12923, an intra prediction unit 12924, a buffer 12925, a motion compensation unit 12926, an inter prediction unit 12927, and a switching unit 12928.

[0363] The three-dimensional data decoding device 12920 obtains the bitstream generated by the three-dimensional data encoding device 12900.

[0364] The entropy decoding unit 12921 generates a quantized first residual signal by entropy decoding the input bitstream for each 3D point in the prediction tree (Predtree). The inverse quantization unit 12922 inverse quantizes the quantized first residual signal to reproduce the first residual signal. The first residual signal for each 3D point is added to a predicted value based on the prediction point corresponding to each 3D point, and then generated (output) as a decoded point. In other words, the 3D data decoding device 12920 calculates position information for one 3D point to be decoded by adding the predicted value and the prediction residual.

[0365] The buffer 12923 holds the generated decoding points as a reference point group for intra prediction. For example, the buffer 12923 may initialize the data held for each prediction tree (target point group). Furthermore, the buffer 12925 holds the generated decoding points as a reference point group for inter prediction. For example, the buffer 12925 may initialize the data held for each prediction tree (target point group).

[0366] The intra prediction unit 12924 determines the intra prediction point to be used for prediction using a predetermined method, with reference to information in the prediction tree (Predtree) including the 3D point to be decoded, such as a plurality of 3D points (a group of reference points for intra prediction) included in the prediction tree (Predtree). For example, the intra prediction unit 12924 may determine the intra prediction point by extrapolating using two 3D points (decoded points) that have been dequantized immediately before the 3D point to be decoded (for example, an ancestor node such as a parent node of the prediction tree).

[0367] The motion compensation unit 12926 reconstructs a decoded point group based on multiple three-dimensional points (multiple decoded points) included in a prediction tree (Predtree) that includes the three-dimensional point to be decoded, and corrects (motion compensates) the displacement between the decoded point group and the point group to be decoded, thereby generating an inter-prediction point group that is a reference point group for inter-prediction after alignment.

[0368] The inter prediction unit 12927 determines an inter prediction point to be used for prediction by a predetermined method based on the motion-compensated inter prediction point group. For example, the inter prediction unit 12927 may select a point closest to the intra prediction point from the inter prediction point group and select it as the inter prediction point, or may select a 3D point (for example, a 3D point closest to the 3D point decoded last) near a 3D point decoded last (for example, a 3D point corresponding to an ancestor node such as a parent node of a prediction tree) as the inter prediction point without referring to the intra prediction point.

[0369] The switching unit 12928 selects either an intra-prediction point or an inter-prediction point to determine a prediction point to be used for prediction. In this way, the 3D data decoding device 12920 determines position information of one or more candidate points from among a plurality of decoded 3D points as prediction points, and calculates a predicted value based on the prediction point. The intra prediction unit 12924 and the inter prediction unit 12927 determine a prediction point (intra-prediction point or inter-prediction point) based on a 3D point decoded immediately before the 3D point to be decoded. In other words, the 3D data decoding device 12920 determines one or more candidate points for calculating a predicted value based on one or more reference points from among a plurality of decoded 3D points. The one or more reference points are 3D points coded immediately before the 3D point to be decoded, and may be, for example, 3D points corresponding to a parent node (ancestor node) of the 3D point to be decoded in a prediction tree.

[0370] The 3D data decoding device 12920 may select either an intra-prediction point or an inter-prediction point as a prediction point according to the procedure of the flowchart in FIG. 51, which will be described later. The 3D data decoding device 12920 may also select either an intra-prediction point or an inter-prediction point as a prediction point based on information about prediction (prediction information) for selecting whether an intra-prediction point or an inter-prediction point is to be used as a prediction point. The prediction information may be entropy coded and written in the header of each 3D point, or may be interleaved with each 3D point. Information about motion compensation, such as a motion vector, may be written in the header of a frame or a prediction tree (Predtree), or may be entropy coded and written in the header of each point, or may be interleaved with each 3D point. In this way, the prediction information or information about motion compensation may be notified to the 3D data decoding device 12920 from the corresponding 3D data encoding device 12900. In addition, the reference points for inter prediction may be a group of points included in an already coded frame different from the frame to be coded, or may be a group of coded points included in the same frame as the frame to be coded.

[0371] In this way, the three-dimensional data decoding device 12920 predicts the point to be decoded using inter prediction in addition to intra prediction, thereby being able to decode the point group from the encoded bit stream (for example, the bit stream output from the three-dimensional data encoding device 12900 of Figure 48) while referring to the decoded point group.

[0372] It should be noted that the 3D data decoding device 12920 does not need to constantly refer to inter-prediction points, and may perform decoding based only on information about the point group to be decoded by initializing a buffer 12925 that stores the inter-prediction reference point group at a predetermined time interval (e.g., every second), a predetermined frame interval (e.g., every 30 frames), or at any timing notified by the corresponding 3D data encoding device 12900. This enables the 3D data decoding device 12920 to start interrupt playback from a point group that is not at the beginning of a bitstream that does not refer to inter-prediction points, potentially improving the random accessibility and error resistance of the bitstream.

[0373] When the bitstream contains coded data in which position information indicated by coordinates expressed in Cartesian coordinates is coded, the three-dimensional data decoding device 12920 decodes the position information indicated by coordinates expressed in Cartesian coordinates.When the bitstream contains coded data in which position information indicated by coordinates expressed in polar coordinates is coded, the three-dimensional data decoding device 12920 decodes the position information indicated by coordinates expressed in polar coordinates.

[0374] FIG. 50 is a flowchart showing an example of a procedure for encoding each three-dimensional point of a prediction tree (Predtree) in the three-dimensional data encoding device 12900.

[0375] In this example, the three-dimensional data encoding device 12900 first determines an intra-prediction point from a group of intra-prediction reference points (S12901). The three-dimensional data encoding device 12900 may determine the intra-prediction point using, for example, a method of determining a prediction point using a prediction tree, as disclosed in the previously described embodiments. For example, the three-dimensional data encoding device 12900 may generate a prediction tree using a plurality of encoded three-dimensional points and select one or more candidate points from the plurality of encoded three-dimensional points based on the prediction tree. The three-dimensional data encoding device 12900 may determine, as the intra-prediction point, a prediction point that results in the smallest amount of coding, from among at least one intra-prediction point determined by at least one of the above methods. Furthermore, the three-dimensional data encoding device 12900 may determine, as the intra-prediction point, a prediction point that has the smallest absolute sum (or sum of squares) of coordinate residuals, from among at least one intra-prediction point determined by at least one of the above methods.

[0376] Next, the three-dimensional data encoding device 12900 outputs intra-prediction related parameters (S12902). If there are two or more candidate methods for determining the intra-prediction points determined in step S12901, the three-dimensional data encoding device 12900 may output information indicating the selected candidate method to the bitstream as intra-prediction related parameters.

[0377] Next, the 3D data encoding device 12900 determines an inter-prediction point by referring to at least one candidate point extracted from the inter-prediction point group. For example, the 3D data encoding device 12900 may determine one candidate point as the inter-prediction point, or may determine a prediction point having, as its coordinates, the average values ​​of coordinates of multiple candidate points. Alternatively, the 3D data encoding device 12900 may determine, as the inter-prediction point, a prediction point having, as its coordinates, the average values ​​of coordinates of the intra-prediction point and at least one candidate point.

[0378] Here, the three-dimensional data encoding device 12900 may search for a point located near the intra-prediction point as at least one candidate point (S12903).

[0379] Next, the three-dimensional data encoding device 12900 may assign smaller index values ​​to each of the at least one determined inter-prediction point in order of proximity to the intra-prediction point (S12904).

[0380] Next, the three-dimensional data encoding device 12900 determines whether the search has ended (S12905), and if so (Yes in S12905), proceeds to the next step S12906, and if not (No in S12905), returns to step S12903. The end of the search may be determined when a predetermined number of inter-prediction points have been found, or when all of the point clouds in a predetermined range have been searched, or when either the predetermined number of inter-prediction points have been found or all of the point clouds in a predetermined range have been searched.

[0381] Next, the three-dimensional data encoding device 12900 determines a prediction method (S12906). Specifically, the three-dimensional data encoding device 12900 determines whether to use intra prediction or inter prediction as a method for determining a prediction point. That is, the three-dimensional data encoding device 12900 determines whether to use intra prediction points or inter prediction points as prediction points. For example, the three-dimensional data encoding device 12900 may determine the prediction method of a prediction point that results in the smallest amount of coding, out of the intra prediction points and the inter prediction points. Furthermore, the three-dimensional data encoding device 12900 may determine the prediction method of a prediction point that results in the smallest absolute sum (or sum of squares) of coordinate residuals, out of the intra prediction points and the inter prediction points.

[0382] The three-dimensional data encoding device 12900 determines whether the determined mode of the prediction method is an inter mode indicating that the prediction method is inter prediction, or an intra prediction mode indicating that the prediction method is intra prediction (S12907).

[0383] If the determined prediction method is inter prediction (inter mode in S12907), the three-dimensional data encoding device 12900 outputs identification information (e.g., a flag) indicating that the inter prediction point has been determined as the prediction point to the bitstream (S12908).

[0384] Next, the three-dimensional data encoding device 12900 outputs, as inter prediction related parameters, information on the number of candidate points used to determine the coordinates of the inter prediction point, an index value of each candidate point used, and the like to a bitstream (S12909). The index value may be assigned to one or more candidate points used to determine the prediction value.

[0385] Furthermore, when the determined prediction method is intra prediction (intra mode in S12907), the three-dimensional data encoding device 12900 outputs identification information (for example, a flag) indicating that an intra prediction point has been determined as a prediction point to the bitstream (S12911). Note that the identification information in steps S12908 and S12911 is information indicating whether an inter prediction point or an intra prediction point has been determined as a prediction point.

[0386] After step S12909 or step S12911, the three-dimensional data encoding device 12900 encodes the coordinate information of the three-dimensional point to be encoded by referring to the predicted point obtained by the determined prediction method (S12910).

[0387] In this way, the three-dimensional data encoding device 12900 determines at least one inter-prediction point by referring to the inter-prediction point group and the intra-prediction points, determines a method for obtaining a prediction point from these intra-prediction points and inter-prediction points, and encodes the position information (coordinate information) of the three-dimensional point to be encoded by referring to the prediction point.

[0388] Note that in S12903, an inter-prediction point may be searched for in the vicinity of a 3D point that is uniquely determined without relying on intra-prediction related parameters, such as a 3D point that was encoded immediately before (for example, a 3D point corresponding to an ancestor node such as a parent node of a prediction tree), without referring to an intra-prediction point. In this case, S12902 may be performed immediately after S12911, rather than immediately after S12901.

[0389] Fig. 51 is a flowchart showing an example of a procedure for decoding each 3D point of a prediction tree (Predtree) in the 3D data decoding device 12920. Fig. 51 corresponds to the decoding of a bitstream generated according to the encoding procedure of Fig. 50. That is, the bitstream includes an encoded first residual signal (prediction residual) and an index value assigned to one candidate point used to calculate a predicted value.

[0390] In this example, the three-dimensional data decoding device 12920 first obtains intra-prediction related parameters from the bitstream (S12921).

[0391] Next, the 3D data decoding device 12920 determines intra-prediction points based on the acquired intra-prediction-related parameters (S12922). Specifically, the 3D data decoding device 12920 determines intra-prediction points in the same manner as in step S12901 of Fig. 50. The 3D data decoding device 12920 is notified of the intra-prediction-related parameters from the corresponding 3D data encoding device 12900, and determines intra-prediction points in accordance with the intra-prediction-related parameters. The intra-prediction-related parameters are acquired in step S12921, and include information specifying at least one method for determining intra-prediction points and parameters accompanying this information.

[0392] Next, the three-dimensional data decoding device 12920 obtains, from the bitstream, identification information indicating the mode of the prediction method (S12923).

[0393] Next, the three-dimensional data decoding device 12920 determines whether the acquired identification information indicates an inter mode indicating that the prediction method is inter prediction, or an intra prediction mode indicating that the prediction method is intra prediction (S12924).

[0394] When the prediction method is inter prediction (inter mode in step S12924), the three-dimensional data decoding device 12920 acquires inter prediction related parameters from the bitstream (S12925).

[0395] Next, the 3D data decoding device 12920 executes processing (S12926 to S12929) for determining an inter-prediction point. Specifically, the 3D data decoding device 12920 determines an inter-prediction point using a method similar to steps S12903 to S12905 of FIG. 50. For example, the 3D data decoding device 12920 determines an inter-prediction point by referring to at least one candidate point extracted from an inter-prediction point group. For example, the 3D data decoding device 12920 may determine one candidate point as the inter-prediction point, or may determine a prediction point having, as its coordinates, the average values ​​of coordinates of multiple candidate points as the inter-prediction point. Alternatively, the 3D data decoding device 12920 may determine, as the inter-prediction point, a prediction point having, as its coordinates, the average values ​​of coordinates of the intra-prediction point and at least one candidate point.

[0396] Here, the three-dimensional data decoding device 12920 may search for a point located near the intra-prediction point as at least one candidate point (S12926).

[0397] Next, the 3D data decoding device 12920 may assign smaller index values ​​to each of the at least one determined inter-prediction point in order of proximity to the intra-prediction point (S12927).

[0398] Next, the 3D data decoding device 12920 determines whether the search has ended (S12928), and if the search has ended (Yes in S12928), proceeds to the next step S12929, and if the search has not ended (No in S12928), returns to step S12926. The end of the search may be determined when a predetermined number of inter-prediction points have been found, or when all of the point groups in a predetermined range have been searched, or when either the predetermined number of inter-prediction points have been found or all of the point groups in a predetermined range have been searched.

[0399] Next, the 3D data decoding device 12920 determines an inter-prediction point based on the inter-prediction-related parameters while referring to the inter-prediction point group and the intra-prediction point (S12929). For example, the 3D data decoding device 12920 identifies candidate points to be used in determining the coordinates of the inter-prediction point based on information included in the inter-prediction-related parameters regarding the number of candidate points to be used in determining the coordinates of the inter-prediction point and an index value assigned to each candidate point used, and determines the coordinates of the inter-prediction point using the identified candidate points, thereby determining the inter-prediction point. In other words, the 3D data decoding device 12920 selects one candidate point from multiple decoded 3D points based on the index value included in the bitstream.

[0400] After step S12929, or if intra mode is selected in step S12924, the three-dimensional data decoding device 12920 decodes the position information (coordinate information) of the three-dimensional point to be decoded by referring to the predicted point obtained by the specified prediction method (S12930).

[0401] In this way, when the prediction method is inter-prediction, the three-dimensional data decoding device 12920 decodes the coordinate information of the point to be decoded by referring to the inter-prediction point, and when the prediction method is intra-prediction, it decodes the coordinate information of the point to be decoded by referring to the intra-prediction point.

[0402] Note that in S12926, an inter-prediction point may be searched for in the vicinity of a 3D point that is uniquely determined without reference to intra-prediction related parameters, such as a 3D point decoded immediately before (for example, a 3D point corresponding to an ancestor node such as a parent node of a prediction tree). In this case, S12921 and S12922 may not be performed immediately before S12923, and S12921 and S12922 may be performed when intra mode is determined in S12924.

[0403] FIG. 52 is a block diagram of a three-dimensional data encoding device 12930 according to a modification of this embodiment. Although FIG. 52 illustrates a processing unit related to encoding position information (geometry) of a point cloud, the three-dimensional data encoding device 12930 may also include other processing units, such as a processing unit that encodes attribute information of the point cloud. In inter prediction and intra prediction, a point cloud to be encoded is encoded while referencing an already-encoded point cloud. The three-dimensional data encoding device 12930 differs in its configuration and operation from the three-dimensional data encoding device 12900 of FIG. 48 in that it includes a coordinate conversion unit 12931 that converts a point cloud having position information expressed in Cartesian coordinates into position information expressed in polar coordinates for encoding, does not quantize a prediction residual (first residual signal) of the position information expressed in polar coordinates, and quantizes a second residual signal in Cartesian coordinates that corresponds to an error occurring in the conversion between Cartesian coordinates and polar coordinates. On the other hand, the three-dimensional data encoding device 12930 is the same in configuration and operation as the three-dimensional data encoding device 12900 except for the differences described above.

[0404] The three-dimensional data encoding device 12930 includes a coordinate transformation unit 12931, a grouping unit 12932, a buffer 12933, a buffer 12934, an intra prediction unit 12935, a buffer 12936, a motion estimation and compensation unit 12937, an inter prediction unit 12938, a switching unit 12939, a coordinate transformation unit 12940, a buffer 12941, a quantization unit 12942, and an entropy encoding unit 12943.

[0405] The coordinate conversion unit 12931 converts the coordinate system of the position information of the target point group, which is input data of the point group to be encoded, from a Cartesian coordinate system to a polar coordinate system. That is, the coordinate conversion unit 12931 generates position information in the polar coordinate system by converting the coordinate system of the position information of one three-dimensional point to be encoded in the Cartesian coordinate system. The point group to be encoded converted into polar coordinates is output to the grouping unit 12932.

[0406] The grouping unit 12932 extracts a group of points to be used as a prediction tree (Predtree), which is a unit of encoding, from the target point group, which is a group of points to be encoded that have been converted into polar coordinates, and sets the extracted points as one group. The buffer 12933 stores the generated prediction tree. For example, the buffer 12933 may initialize the data stored for each prediction tree. A process for encoding is executed in order for each of the multiple 3D points included in the prediction tree (Predtree) stored in the buffer 12933.

[0407] A difference (first residual signal) is calculated between each of the multiple 3D points (points to be coded) included in the prediction tree stored in buffer 12933 and a prediction point selected for the point to be coded. This first residual signal is a residual signal of position information expressed in polar coordinates. The first residual signal is also called a prediction residual. This first residual signal is an example of a first residual. Because the position information of the multiple 3D points stored in buffer 12933 has been converted to a polar coordinate system, the first residual is the difference between the position information in the polar coordinate system after the conversion and a prediction value.

[0408] The first residual signal and the predicted point are then added together, and the resulting encoded decoded point is stored in buffers 12934 and 12936. The position information of the decoded point stored in buffers 12934 and 12936 is expressed in polar coordinates. In this respect, the functions of buffers 12934 and 12936 differ from those of buffers 12905 and 12907, but other functions are the same.

[0409] Similarly, the intra prediction unit 12935, motion detection and compensation unit 12937, inter prediction unit 12938, and switching unit 12939 differ in function from the intra prediction unit 12906, motion detection and compensation unit 12908, inter prediction unit 12909, and switching unit 12910 in that the position information of the three-dimensional point to be processed is expressed in polar coordinates, but otherwise their functions are the same.

[0410] The coordinate conversion unit 12940 acquires the same decoding points as those held in the buffers 12934 and 12936, and converts the coordinate system of the position information of the acquired decoding points from the polar coordinate system to the Cartesian coordinate system. In other words, the coordinate conversion unit 12940 generates position information in the Cartesian coordinate system by inversely converting the coordinate system of the position information in the polar coordinate system after conversion by the coordinate conversion unit 12931.

[0411] The buffer 12941 holds the position information of the three-dimensional points expressed in Cartesian coordinates that is input to the three-dimensional data encoding device 12930 .

[0412] Then, a difference (second residual signal) is calculated between the input position information in the Cartesian coordinate system and the position information in the Cartesian coordinate system obtained by converting the coordinate system from a polar coordinate system to a Cartesian coordinate system in the coordinate conversion unit 12940. This second residual signal is an example of a second residual. In other words, the second residual signal is the difference between the position information in the Cartesian coordinate system that has not been subjected to coordinate conversion in the coordinate conversion unit 12931 and the position information that has been converted into polar coordinates and then inversely converted into a Cartesian coordinate system, and is a conversion error that occurs due to the coordinate conversion.

[0413] The quantization unit 12942 quantizes the second residual signal.

[0414] The entropy coding unit 12943 entropy codes the first residual signal and the quantized second residual signal to generate coded data, and outputs a bitstream including the coded data.

[0415] In this way, the three-dimensional data encoding device 12930 converts the coordinate system of the position information of three-dimensional points from a Cartesian coordinate system to a polar coordinate system and encodes the position information in the polar coordinate system. This makes it possible to improve the prediction accuracy of the points to be encoded when encoding a point cloud generated by acquiring the three-dimensional positions of surrounding objects centered on the sensor position, such as LiDAR, and thus potentially improve encoding efficiency.

[0416] Fig. 53 is a block diagram of a three-dimensional data decoding device 12950 according to a variation of this embodiment. Note that Fig. 53 describes a processing unit related to decoding position information (geometry) of a point cloud, but the three-dimensional data decoding device 12950 may also include other processing units, such as a processing unit that decodes attribute information of the point cloud. The three-dimensional data decoding device 12950 performs inter-prediction decoding to decode a point cloud to be decoded while referencing a point cloud that has already been decoded. For example, the three-dimensional data decoding device 12950 decodes a bitstream generated by the three-dimensional data encoding device 12930 shown in Fig. 52. 49, the three-dimensional data decoding device 12950 differs in basic configuration and operation in that it does not perform inverse quantization of the first residual signal (prediction residual), and in that it entropy decodes and inverse quantizes a second residual signal in orthogonal coordinates that corresponds to a conversion error occurring in the conversion between orthogonal coordinates and polar coordinates to reproduce it, adds it to a point converted from the corresponding decoded point in polar coordinates to orthogonal coordinates, and outputs it as a decoded point in orthogonal coordinates.On the other hand, the three-dimensional data decoding device 12950 is the same as the three-dimensional data decoding device 12920 in all respects except for the above differences.

[0417] The three-dimensional data decoding device 12950 includes an entropy decoding unit 12951, a buffer 12952, an intra prediction unit 12953, a buffer 12954, a motion compensation unit 12955, an inter prediction unit 12956, a switching unit 12957, a coordinate transformation unit 12958, and an inverse quantization unit 12959.

[0418] The entropy decoding unit 12951 generates a first residual signal and a quantized second residual signal by entropy decoding the input bitstream for each 3D point of the prediction tree (Predtree). The first residual signal for each 3D point is added to a predicted value based on the prediction point corresponding to each 3D point, and then generated (output) as a decoded point expressed in polar coordinates.

[0419] The buffer 12952 holds the generated decoding points as a reference point group for intra prediction. For example, the buffer 12952 may initialize the data held for each prediction tree (target point group). Furthermore, the buffer 12954 holds the generated decoding points as a reference point group for inter prediction. For example, the buffer 12954 may initialize the data held for each prediction tree (target point group). The position information of the decoding points held in the buffers 12952 and 12954 is expressed in polar coordinates. In this respect, the functions of the buffers 12952 and 12954 differ from those of the buffers 12923 and 12925, but the other functions are the same.

[0420] Similarly, the intra prediction unit 12953, motion compensation unit 12955, inter prediction unit 12956, and switching unit 12957 differ in function from the intra prediction unit 12924, motion compensation unit 12926, inter prediction unit 12927, and switching unit 12928 in that the position information of the three-dimensional point to be processed is expressed in polar coordinates, but otherwise their functions are the same.

[0421] A coordinate conversion unit 12958 acquires the same decoding points as those held in the buffers 12952 and 12954, and converts the coordinate system of the position information of the acquired decoding points from a polar coordinate system to a rectangular coordinate system.

[0422] The inverse quantization unit 12959 inverse quantizes the quantized second residual signal to reproduce the second residual signal.

[0423] The position information in the Cartesian coordinate system obtained by coordinate transformation by the coordinate transformation unit 12958 and the second residual signal reproduced by inverse quantization by the inverse quantization unit 12959 are added together, and then generated (output) as a decoded point including the position information in the Cartesian coordinate system.

[0424] In this way, the three-dimensional data decoding device 12950 includes means for converting the coordinate system of a decoding point having position information in a polar coordinate system from the polar coordinate system to a Cartesian coordinate system, and adding the converted data to a second residual signal in Cartesian coordinates corresponding to an error that occurs in the conversion between the position information in the Cartesian coordinate system and the position information in the polar coordinate system. This allows the three-dimensional data decoding device 12950 to decode a point cloud from an encoded bit stream (for example, a bit stream output from the three-dimensional data encoding device 12930 in FIG. 52) while referring to the point cloud that has already been encoded in polar coordinates.

[0425] 54 shows an example of the syntax of the geometry parameter set (GPS). This syntax is used in the three-dimensional data encoding devices 12900 and 12930 and the three-dimensional data decoding devices 12920 and 12950 described using FIGS.

[0426] As shown in these examples, the GPS may notify information, such as gps_alt_coordinates_flag, indicating whether a coordinate system other than Cartesian coordinates, such as polar coordinates, is used in the decoding process of each point. When the value of gps_alt_coordinates_flag is set to 1 (i.e., when gps_alt_coordinates_flag = 1), it indicates that an alternative coordinate system (e.g., a polar coordinate system) is used in the decoding process of a data unit of position information in a bit stream that references GPS. When the value of gps_alt_coordinates_flag is set to 0 (i.e., when gps_alt_coordinates_flag = 0), it indicates that an alternative coordinate system is not used in the decoding process of a data unit of position information in a bit stream that references GPS. In other words, gps_alt_coordinates_flag may indicate whether the encoded data includes first encoded data calculated in a polar coordinate system. Note that gps_alt_coordinates_flag is an example of first identification information indicating whether the encoded data includes first encoded data calculated in a polar coordinate system.

[0427] Furthermore, when a coordinate system (alternate coordinate system) other than a Cartesian coordinate system, such as a polar coordinate system, is used in the decoding process of each 3D point (for example, when gps_alt_coordinates_flag=1), coordinate transformation information, such as gps_coordinate_trans_enabled_flag, indicating whether or not to perform coordinate transformation of each decoded point (for example, from a polar coordinate system to a Cartesian coordinate system) before outputting each 3D point from the 3D data decoding device may be notified. When gps_alt_coordinates_flag=1 (that is, when the first identification information indicates that the encoded data includes first encoded data), specifically, the position information of one or more candidate points used in calculating a predicted value and the position information of one 3D point to be encoded used in calculating a first residual are position information in a polar coordinate system. In this case, the bitstream includes gps_coordinate_trans_enabled_flag. gps_coordinate_trans_enabled_flag is an example of second identification information indicating whether to output position information in a polar coordinate system or in a Cartesian coordinate system during decoding. Furthermore, when gps_alt_coordinates_flag=1, the three-dimensional data encoding device 12900 quantizes the first residual during encoding and encodes the quantized first residual, encoding position information in the polar coordinate system. Therefore, when gps_alt_coordinates_flag=1 and gps_coordinate_trans_enabled_flag=0, it can be said that position information in the polar coordinate system is encoded, and gps_coordinate_trans_enabled_flag=0 can be said to indicate that position information in the polar coordinate system is output during decoding. Note that whether or not to output position information in the polar coordinate system may be switched by another flag (identification information).

[0428] Note that if gps_alt_coordinates_flag=0 (that is, if the first identification information indicates that the coded data does not include the first coded data), the bitstream does not need to include gps_coordinate_trans_enabled_flag (second identification information).

[0429] When the value of gps_coordinate_trans_enabled_flag is set to 1 (i.e., when gps_coordinate_trans_enabled_flag=1), it indicates that the coordinate system is converted to another coordinate system in the decoding process of the data unit of the position information of the bitstream that refers to GPS. Therefore, when gps_alt_coordinates_flag=1 and gps_coordinate_trans_enabled_flag=0, the position information in the Cartesian coordinate system is decoded, so it can be said that gps_coordinate_trans_enabled_flag=0 indicates that the position information in the Cartesian coordinate system is output in the decoding.

[0430] When the value of gps_coordinate_trans_enabled_flag is set to 0 (i.e., when gps_coordinate_trans_enabled_flag=0), it indicates that the coordinate system is not converted to another coordinate system in the decoding process of the data unit of the position information of the bitstream that refers to GPS. Note that when gps_coordinate_trans_enabled_flag is not indicated, the value of gps_coordinate_trans_enabled_flag may be considered to be set to 0.

[0431] Note that when coordinate transformation of decoded points is not performed before each 3D point is output from the 3D data decoding device (for example, when gps_coordinate_trans_enabled_flag=0), the point group may be encoded and decoded by the 3D data encoding device 12900 shown in Figure 48 and the 3D data decoding device 12920 shown in Figure 49. Also, when coordinate transformation of decoded points is performed before each 3D point is output from the 3D data decoding device (for example, when gps_coordinate_trans_enabled_flag=1), the point group may be encoded and decoded by the 3D data encoding device 12930 shown in Figure 52 and the 3D data decoding device 12950 shown in Figure 53.

[0432] By notifying the three-dimensional data encoding device of gps_alt_coordinates_flag and gps_coordinate_trans_enabled_flag, it becomes possible to switch between the three-dimensional data encoding device 12900 shown in Figure 48 and the three-dimensional data encoding device 12930 shown in Figure 52 depending on the point group to be encoded, even when a coordinate system other than Cartesian coordinates, such as polar coordinates, is used in encoding and decoding each three-dimensional point (for example, when gps_alt_coordinates_flag = 1), which may improve encoding efficiency.

[0433] Note that Figure 54 illustrates an example of GPS syntax, but gps_alt_coordinates_flag and gps_coordinate_trans_enabled_flag may be included in the SPS, the header of the data unit, or other control information as metadata.

[0434] Fig. 55 shows an example of the syntax of each three-dimensional point (a node in a Predtree). This syntax is used in the three-dimensional data encoding devices 12900 and 12930 and the three-dimensional data decoding devices 12920 and 12950 described using Figs. 48 to 54.

[0435] In this example, the three-dimensional data encoding devices 12900, 12930 first notify the three-dimensional data decoding devices 12920, 12950 of identification information (pred_mode) indicating how to determine intra-prediction points at three-dimensional points to be encoded or decoded. In addition, the three-dimensional data encoding devices 12900, 12930 may notify the three-dimensional data decoding devices 12920, 12950 of additional information for determining intra-prediction points according to the identification information (pred_mode).

[0436] Next, when inter prediction is enabled in the GPS referenced by the prediction tree (predtree) being coded (for example, when gps_inter_prediction_enabeled_flag=1), the 3D data coding devices 12900, 12930 may notify the 3D data decoding devices 12920, 12950 of information (intra_pred_flag) indicating whether the prediction method for the 3D point to be coded or decoded is intra prediction (i.e., inter prediction). Note that when gps_inter_prediction_enabeled_flag=0, the value of intra_pred_flag may be set to 1 (intra prediction). When the prediction method for the 3D point to be coded or decoded is inter prediction (for example, intra_pred_flag=0), identification information (inter_pred_mode) indicating how to determine the inter prediction point for the 3D point to be coded or decoded may be notified. Furthermore, the 3D data encoding devices 12900 and 12930 may set the number of candidate points in the inter-prediction point group to be referenced when determining an inter-prediction point in NumRefPoints according to the identification information (inter_pred_mode), and may notify the 3D data decoding devices 12920 and 12950 of NumRefPoints' worth of indices (inter_ref_point_idx) of each candidate point. If multiple candidate points are specified in the inter-prediction point group to be referenced when determining an inter-prediction point, the average value of the coordinates of the specified multiple candidate points may be used as the coordinate of the inter-prediction point. Furthermore, the 3D data encoding devices 12900 and 12930 may provide an inter_pred_mode that omits notification of the candidate point indices and selects a specific candidate point, such as the smallest index. For example, the 3D data encoding devices 12900 and 12930 may omit notification of the candidate point indices by, for example, providing a process for determining whether inter_pred_mode indicates the relevant mode or by setting the value of NumRefPoints to 0.Furthermore, this can be implemented as long as the information necessary for a method of uniquely determining an inter-prediction point is notified. For example, instead of inter_pred_mode, the number of candidate points in the inter-prediction point group to be referenced when determining an inter-prediction point may be notified.

[0437] In addition, when searching for candidate points in an inter-prediction point group in the vicinity of a three-dimensional point that is uniquely determined without depending on identification information (pred_mode) indicating how to obtain an intra-prediction point, such as the three-dimensional point that was most recently encoded or decoded (for example, a three-dimensional point corresponding to an ancestor node such as a parent node of a prediction tree), the identification information (pred_mode) indicating how to obtain an intra-prediction point and additional information for determining the intra-prediction point may be notified to the three-dimensional data decoding devices 12920, 12950 only when the prediction method for the three-dimensional point to be encoded or decoded is intra-prediction (for example, intra_pred_flag=1).

[0438] Next, the three-dimensional data encoding devices 12900 and 12930 may notify a first difference (1st_residual_value) between the position information (coordinate value) of the point to be encoded or decoded and the position information (coordinate value) of the predicted point. When coordinate transformation of the decoded point is performed before outputting each three-dimensional point from the three-dimensional data decoding devices 12920 and 12950 (for example, when gps_coordinate_trans_enabled_flag=1), the decoding result in another coordinate system such as polar coordinates may be coordinate-transformed into the original coordinate system such as Cartesian coordinates, and a second difference (2nd_residual_value) between the original position information (coordinate value) and the position information. Note that, although an example in which this difference information is notified as a single syntax has been shown, it may also be decomposed into multiple syntaxes, such as positive / negative information and absolute value information, and notified.

[0439] This information is communicated from the three-dimensional data encoding devices 12900, 12930 to the three-dimensional data decoding devices 12920, 12950, ​​making it possible for the three-dimensional data encoding devices 12900, 12930 and the three-dimensional data decoding devices 12920, 12950 to perform consistent prediction processing, and the three-dimensional data decoding devices 12920, 12950 can decode the three-dimensional points to be processed without causing any inconsistency with the corresponding three-dimensional data encoding devices 12900, 12930.

[0440] The devices, processes, syntax, etc. disclosed using Figures 48 to 55 may be implemented in combination with at least a part of other embodiments. Also, the devices, processes, parts of syntax, etc. disclosed using Figures 48 to 55 may be implemented in combination with other embodiments. Also, all of the components disclosed using Figures 48 to 55 are not necessarily required, and only some of the components may be provided.

[0441] As described above, the three-dimensional data encoding device according to this embodiment performs the processing shown in Fig. 56. The three-dimensional data encoding device determines a predicted value based on position information of one or more candidate points from among a plurality of encoded three-dimensional points (S12931). The three-dimensional data encoding device calculates a prediction residual, which is the difference between the position information of one three-dimensional point to be encoded from among the plurality of three-dimensional points, and the predicted value (S12932). The three-dimensional data encoding device generates encoded data by encoding the prediction residual (S12933). The three-dimensional data encoding device generates a bitstream including the encoded data (S12934). In step S12931, the three-dimensional data encoding device determines one or more candidate points based on one or more reference points from among the plurality of encoded three-dimensional points.

[0442] For example, the plurality of 3D points constitute a prediction tree, and the one or more reference points include a 3D point corresponding to a parent node of the 3D point to be encoded.

[0443] According to this, one or more candidate points used to calculate the predicted value are determined based on the parent node of one 3D point to be coded in the prediction tree, thereby reducing the prediction residual and improving coding efficiency.

[0444] For example, one or more candidate points may be assigned index values, and the bitstream may further include the index values ​​assigned to the candidate points for use in determining the predicted value.

[0445] This allows the three-dimensional data decoding device to easily identify candidate points based on the index values, thereby reducing the processing load on the three-dimensional data decoding device.

[0446] For example, the three-dimensional data encoding device includes a processor and a memory, and the processor performs the above-mentioned processing using the memory.

[0447] Furthermore, the three-dimensional data decoding device according to this embodiment performs the processing shown in Fig. 57. The three-dimensional data decoding device acquires a bitstream including an encoded prediction residual and an index value assigned to one candidate point used to calculate a predicted value (S12941). The three-dimensional data decoding device determines one candidate point based on one or more reference points from among a plurality of decoded three-dimensional points, based on the index value (S12942). The three-dimensional data decoding device calculates a predicted value based on position information of the determined candidate point (S12943). The three-dimensional data decoding device calculates a prediction residual by decoding the encoded prediction residual (S12944). The three-dimensional data decoding device calculates position information of the one three-dimensional point to be decoded by adding the predicted value and the prediction residual (S12945).

[0448] According to this, one candidate point is determined based on one or more reference points among a plurality of encoded three-dimensional points, and the three-dimensional point to be decoded can be decoded using a predicted value based on the position information of the determined one candidate point.

[0449] For example, the plurality of 3D points constitute a prediction tree, and the one or more reference points include a 3D point corresponding to a parent node of the 3D point to be encoded.

[0450] For example, one or more candidate points may be assigned index values, and the bitstream may further include the index values ​​assigned to the candidate points for use in determining the predicted value.

[0451] This allows the three-dimensional data decoding device to easily identify candidate points based on the index values, thereby reducing the processing load on the three-dimensional data decoding device.

[0452] For example, the three-dimensional data decoding device includes a processor and a memory, and the processor performs the above processing using the memory.

[0453] Figure 58 is a flowchart showing the coordinate system switching process in the encoding process. The flowchart in Figure 58 switches between encoding using the three-dimensional data encoding device 12900 described in Figure 48 and encoding using the three-dimensional data encoding device 12930 described in Figure 52. The flowchart in Figure 58 also switches between encoding position information in a rectangular coordinate system and encoding position information in a polar coordinate system.

[0454] First, the three-dimensional data encoding device checks the coordinate system of the input point group and determines the coordinate system for the encoding process and the decoding process (S13001). That is, the three-dimensional data encoding device determines the coordinate system of the position information that is the target of the encoding process and the decoding process.

[0455] Next, the three-dimensional data encoding device determines whether the coordinate system of the input point group is the same as the coordinate system of the determined encoding process and decoding process (S13002).

[0456] If the three-dimensional data encoding device determines that the coordinate system of the input point cloud is the same as the coordinate system of the determined encoding process and decoding process (Yes in S13002), it sets gps_coordinate_trans_enabled_flag = 0 and decides to encode the position information of the point cloud in the determined coordinate system without performing coordinate conversion (S13003).

[0457] If the three-dimensional data encoding device determines that the coordinate system of the input point cloud is not the same as the coordinate system of the determined encoding process and decoding process (No in S13002), it sets gps_coordinate_trans_enabled_flag=1, performs coordinate transformation, and determines to encode the position information of the point cloud in the determined coordinate system (S13004).

[0458] Note that, in the three-dimensional data encoding device 12930 described in Fig. 52, the coordinate system of the input point cloud is polar coordinates, and the coordinate system of the encoding process and the decoding process is Cartesian coordinates, as an example, but the coordinate system of the input point cloud may be Cartesian coordinates, and the coordinate system of the encoding process and the decoding process may be polar coordinates. In this case, the explanation can be given by reading the polar coordinate system in Fig. 52 as a Cartesian coordinate system, and the Cartesian coordinate system as a polar coordinate system.

[0459] Next, the three-dimensional data encoding device determines whether the encoding coordinate system is a polar coordinate system (or a rectangular coordinate system) (S13005).

[0460] When the three-dimensional data encoding device determines that the encoding coordinate system is the polar coordinate system (Yes in S13005), it sets gps_alt_coordinates_flag=1 and encodes the position information of the point cloud in the polar coordinate system (S13006).

[0461] If the three-dimensional data encoding device determines that the encoding coordinate system is not a polar coordinate system (i.e., a Cartesian coordinate system) (No in S13005), it sets gps_alt_coordinates_flag=0 and encodes the position information of the point cloud in the Cartesian coordinate system (S13007).

[0462] When the syntax is determined by the above process, the gps_coordinate_trans_enabled_flag and gps_alt_coordinates_flag described with reference to Fig. 54 may have a syntax configuration that indicates them independently and without dependency. In other words, the gps_coordinate_trans_enabled_flag may be indicated to the GPS regardless of the presence or absence of the gps_alt_coordinates_flag or the value of the gps_alt_coordinates_flag.

[0463] In addition, in the above processing, the syntax configuration of Figure 54 may be adopted by switching process 1 in steps S13002 to S13004 and process 2 in steps S13005 to S13007 and configuring process 1 to be performed when gps_alt_coordinates_flag = 1 in process 2.

[0464] Fig. 59 is a flowchart showing the coordinate system switching process in the decoding process. In the flowchart of Fig. 59, switching is performed between performing decoding using the three-dimensional data decoding device 12920 described in Fig. 49 and performing decoding using the three-dimensional data decoding device 12950 described in Fig. 53. The flowchart of Fig. 59 also switches between performing decoding of position information in a rectangular coordinate system and decoding of position information in a polar coordinate system.

[0465] First, the 3D data decoding device analyzes the metadata included in the bitstream (S13011). Specifically, the metadata is control information included in the GPS, SPS, header, etc. The 3D data decoding device checks the gps_alt_coordinates_flag and gps_coordinate_trans_enabled_flag included in the metadata.

[0466] Next, the three-dimensional data decoding device determines whether gps_alt_coordinates_flag=1 (S13012).

[0467] If gps_alt_coordinates_flag=1 (Yes in S13012), the three-dimensional data decoding device determines to decode in the polar coordinate system (S13013).

[0468] If gps_alt_coordinates_flag=0 (No in S13012), the three-dimensional data decoding device determines to decode in the orthogonal coordinate system (S13014).

[0469] In this way, the coordinate system of the calculated position information of one three-dimensional point to be decoded is determined according to the value of gps_alt_coordinates_flag, and is therefore determined to be a coordinate system depending on whether or not it includes the first encoded data indicated by the first identification information.

[0470] Next, the three-dimensional data decoding device determines whether gps_coordinate_trans_enabled_flag=1 (S13015).

[0471] If gps_coordinate_trans_enabled_flag=1 (Yes in S13015), the three-dimensional data decoding device decodes the position information in the determined coordinate system without coordinate conversion (S13016). In this case, the position information of the point cloud is decoded using the configuration of the three-dimensional data decoding device 12920. Note that, since gps_coordinate_trans_enabled_flag is indicated when gps_alt_coordinates_flag=1, when gps_coordinate_trans_enabled_flag=1 (that is, when the second identification information indicates that position information in a polar coordinate system is to be output in decoding), the coordinate system of the calculated position information of one three-dimensional point to be decoded is a polar coordinate system.

[0472] If gps_coordinate_trans_enabled_flag=0 (No in S13015), the three-dimensional data decoding device performs coordinate transformation and decodes the position information in the determined coordinate system (S13017). In this case, the position information of the point cloud is decoded using the configuration of the three-dimensional data decoding device 12950. Note that gps_coordinate_trans_enabled_flag is indicated when gps_alt_coordinates_flag=1. Therefore, if gps_coordinate_trans_enabled_flag=0 (that is, if the second identification information indicates that position information in a rectangular coordinate system is to be output in decoding), the coordinate system of the position information in the polar coordinate system obtained by adding the predicted value and the first residual is transformed, and the position information in the rectangular coordinate system obtained by the transformation is calculated as the position information of one three-dimensional point to be decoded. In this case, the encoded data includes a second residual, and the three-dimensional data decoding device calculates the second residual by decoding the encoded second residual, and in calculating the position information of one three-dimensional point to be decoded, adds the position information of the orthogonal coordinate system obtained by converting the coordinate system to the second residual, and calculates the position information obtained by the addition as the position information of one three-dimensional point to be decoded.

[0473] In addition, when metadata is configured with the syntax configuration of Figure 54, if the coordinate system to be decoded is determined to be polar coordinates in process 1 of steps S13012 to S13014, the process proceeds to process 2 of steps S13015 to S13017, and if the coordinate system to be decoded is determined to be Cartesian coordinates in process 1, the determination of process 2 is skipped and decoding may be performed using a three-dimensional data decoding device 12920 with the configuration of Figure 49.

[0474] Note that, although gps_coordinate_trans_enabled_flag is coordinate transformation information indicating whether or not coordinate transformation is performed on decoded points before each point is output from the 3D data decoding device, it may also be information indicating whether or not coordinate transformation was performed when encoding each 3D point, or information indicating whether or not error information (transformation error) generated by coordinate transformation is included in the bit stream. Furthermore, if such information is included, it may be determined whether or not coordinate transformation is performed in the 3D data decoding device or application.

[0475] While the process switching based on the combination of the coordinate system of the input point cloud, the coordinate system of the output point cloud, the coordinate system of the position information to be encoded, and the coordinate system of the position information to be decoded, and the signaling method indicating these coordinate systems have been described using an example of encoding using a predictive tree, a similar method may also be used in the case of octree encoding. Furthermore, if the coordinate system of the input point cloud, the coordinate system of the output point cloud, the coordinate system of the position information to be encoded, and the coordinate system of the position information to be decoded are all one type, the switching process may be omitted, or the process may be extended when there are three or more types of coordinate systems. Furthermore, while the coordinate system for encoding and decoding position information has been described as an example, a similar method may be used to switch the process if the coordinate system for encoding and decoding attribute information corresponds to multiple coordinate systems, or if there is a possibility that the coordinate systems of the input point cloud and the output point cloud will be switched to other coordinate systems. For example, a syntax similar to that of FIG. 54 may be included in attribute_parameter_set.

[0476] Furthermore, the coordinate system for encoding and decoding the position information and the coordinate system for encoding and decoding the attribute information may be controlled to be the same.

[0477] As described above, the three-dimensional data encoding device according to this embodiment performs the processing shown in FIG. 60. The three-dimensional data encoding device encodes a plurality of three-dimensional points in either a Cartesian coordinate system or a polar coordinate system. The three-dimensional data encoding device calculates a predicted value based on position information of one or more candidate points among the plurality of encoded three-dimensional points (S13021). The three-dimensional data encoding device calculates a first residual, which is the difference between the predicted value and position information of one three-dimensional point to be encoded among the plurality of three-dimensional points (S13022). The three-dimensional data encoding device generates encoded data by encoding the first residual (S13023). The three-dimensional data encoding device generates a bitstream including encoded data and first identification information indicating whether the encoded data includes first encoded data calculated in a polar coordinate system (S13024).

[0478] According to this, the bit stream includes first identification information indicating whether the encoded data includes first encoded data calculated in a polar coordinate system, so that the three-dimensional data decoding device can perform appropriate decoding processing based on the first identification information.

[0479] For example, the bitstream further includes second identification information. When the position information of one or more candidate points used to calculate a predicted value and the position information of one three-dimensional point to be coded used to calculate a first residual are position information in a polar coordinate system, the first identification information indicates that the coded data includes first coded data calculated in a polar coordinate system. In this case, the second identification information indicates whether the position information in the polar coordinate system or the position information in the Cartesian coordinate system is to be output in decoding.

[0480] For example, if the first identification information indicates that the coded data does not include the first coded data, the bitstream does not include the second identification information.

[0481] For example, when position information in a polar coordinate system is output in decoding, the second identification information indicates that the position information in a polar coordinate system is output in decoding. In this case, in encoding the first encoded data, the first residual is quantized and the quantized first residual is encoded.

[0482] For example, if the calculation of the first residual further includes generating position information in a polar coordinate system by transforming the coordinate system of position information in a Cartesian coordinate system of one three-dimensional point to be encoded, and the first residual is the difference between the transformed position information in the polar coordinate system and the predicted value, the three-dimensional data encoding device generates position information in a Cartesian coordinate system by inversely transforming the coordinate system of the transformed position information in the polar coordinate system. The three-dimensional data encoding device calculates a second residual, which is the difference between the position information in the Cartesian coordinate system and the position information in the Cartesian coordinate system after the inverse transformation. In generating encoded data, the encoded data is generated by encoding the first residual and the second residual. The first identification information indicates that the encoded data includes the first encoded data. The second identification information indicates that the position information in the Cartesian coordinate system is to be output in decoding.

[0483] For example, the second identification information indicates whether or not the coded data of the second residual is included in the bitstream.

[0484] For example, the three-dimensional data encoding device includes a processor and a memory, and the processor performs the above-mentioned processing using the memory.

[0485] Furthermore, the three-dimensional data decoding device according to this embodiment performs the processing shown in FIG. 61. The three-dimensional data decoding device decodes multiple three-dimensional points in either a rectangular coordinate system or a polar coordinate system. The three-dimensional data decoding device acquires a bitstream including coded data in which a first residual is coded and first identification information indicating whether the coded data includes first coded data calculated in a polar coordinate system (S13031). The three-dimensional data decoding device calculates a predicted value based on position information of one or more candidate points among the multiple decoded three-dimensional points (S13032). The three-dimensional data decoding device calculates a first residual by decoding the coded first residual (S13033). The three-dimensional data decoding device calculates position information of one three-dimensional point to be decoded by adding the predicted value and the first residual. The coordinate system of the calculated position information of one three-dimensional point to be decoded is a coordinate system depending on whether the coded data includes the first coded data indicated by the first identification information.

[0486] This allows the coordinate system of the position information of one three-dimensional point to be decoded to be determined based on the first identification information indicating whether the encoded data includes first encoded data calculated in a polar coordinate system, so that the three-dimensional data decoding device can perform appropriate decoding processing based on the first identification information.

[0487] For example, the bitstream further includes second identification information indicating whether to output position information in a polar coordinate system or position information in a Cartesian coordinate system during decoding. When the first identification information indicates that the encoded data includes first encoded data, the position information of one or more candidate points used to calculate a predicted value and the position information of one three-dimensional point to be encoded used to calculate a first residual are position information in a polar coordinate system.

[0488] For example, if the first identification information indicates that the coded data does not include the first coded data, the bitstream does not include the second identification information.

[0489] For example, when the second identification information indicates that the position information in the polar coordinate system is output in the decoding, the coded data is data obtained by quantizing and coding the first residual. In this case, the coordinate system of the calculated position information of one three-dimensional point to be decoded is the polar coordinate system.

[0490] For example, when the first identification information indicates that the encoded data includes the first encoded data and the second identification information indicates that position information in a Cartesian coordinate system is to be output during decoding, the position information of one three-dimensional point to be decoded is calculated by converting the coordinate system of the position information in a polar coordinate system obtained by adding the predicted value and the first residual, and calculating the position information in the Cartesian coordinate system obtained by the conversion as the position information of one three-dimensional point to be decoded.

[0491] For example, when the first identification information indicates that the coded data includes the first coded data and the second identification information indicates that position information in a Cartesian coordinate system is to be output in decoding, the coded data further includes an encoded second residual. The three-dimensional data coding device calculates the second residual by decoding the encoded second residual. In calculating the position information of one three-dimensional point to be decoded, the device adds the position information in the Cartesian coordinate system obtained by converting the coordinate system and the second residual, and calculates the position information obtained by the addition as the position information of one three-dimensional point to be decoded.

[0492] For example, the second identification information indicates whether or not the coded data of the second residual is included in the bitstream.

[0493] For example, the three-dimensional data decoding device includes a processor and a memory, and the processor performs the above processing using the memory.

[0494] (Embodiment 6) The configuration of a position information encoding unit that encodes position information of three-dimensional points included in point cloud data will be described. Fig. 62 is a block diagram showing the configuration of a position information encoding unit when the coordinate system of the input position information and the coordinate system of the position information used for encoding are both Cartesian coordinate systems. That is, this configuration is the configuration of the position information encoding unit when gps_alt_coordinates_flag=0. Here, in Figs. 62 to 64, (x, y, z) in a block indicates that processing is performed using a Cartesian coordinate system, and (r, q, f) in a block indicates that processing is performed using a polar coordinate system. A signal with (x, y, z) indicates that the Cartesian coordinate system is used, and a signal with (r, q, f) indicates that the polar coordinate system is used.

[0495] For example, gps_alt_coordinates_flag is included in GPS (location information parameter set). GPS is control information (parameter set) for location information included in the bitstream, and is control information common to multiple frames, multiple point clouds, or multiple slices. gps_alt_coordinates_flag is information indicating whether a coordinate system other than Cartesian coordinates, such as polar coordinates, is used in the encoding or decoding process of each point. In other words, gps_alt_coordinates_flag indicates whether the encoded location information data includes first encoded location information data calculated in a polar coordinate system. For example, gps_alt_coordinates_flag=1 indicates that a polar coordinate system is used in the encoding or decoding process of location information referring to GPS. gps_alt_coordinates_flag=0 indicates that a Cartesian coordinate system is used in the encoding or decoding process of location information referring to GPS.

[0496] As shown in Fig. 62, the position information encoding unit includes a prediction unit 13601, a quantization unit 13602, and an encoding unit 13603. The prediction unit 13601 calculates a predicted value of the input position information in an orthogonal coordinate system, and calculates a residual that is the difference between the position information and the predicted value. Note that the predicted value and residual calculated here are expressed in an orthogonal coordinate system.

[0497] The quantization unit 13602 quantizes the residual in the orthogonal coordinate system. The encoding unit 13603 generates a bitstream by encoding (e.g., entropy encoding or arithmetic encoding) the quantized residual. That is, in the position information encoding unit shown in Fig. 62, all processing is performed in the orthogonal coordinate system.

[0498] 63 is a block diagram showing the configuration of a location information encoder when the coordinate system of input location information is a Cartesian coordinate system and the coordinate system of location information used for encoding is a polar coordinate system. That is, this configuration is the configuration of a location information encoder when gps_alt_coordinates_flag=1 and gps_coordinate_trans_enabled_flag=1.

[0499] The gps_coordinate_trans_enabled_flag is included in, for example, GPS. The gps_coordinate_trans_enabled_flag is information indicating whether or not to perform coordinate transformation (e.g., transformation from a polar coordinate system to a Cartesian coordinate system) of each decoded point before outputting the three-dimensional point from the three-dimensional data decoding device. In other words, the gps_coordinate_trans_enabled_flag indicates whether to output position information in a polar coordinate system or in a Cartesian coordinate system during decoding. For example, gps_alt_coordinates_flag=1 indicates that coordinate transformation is performed, and gps_alt_coordinates_flag=0 indicates that coordinate transformation is not performed.

[0500] As shown in FIG. 63, the position information encoding unit includes a transform unit 13611, a prediction unit 13612, an encoding unit 13613, a transform unit 13614, a subtraction unit 13615, a quantization unit 13616, an encoding unit 13617, and a combination unit 13618.

[0501] The conversion unit 13611 converts the input position information in the Cartesian coordinate system into position information in the polar coordinate system. The prediction unit 13612 calculates a predicted value of the position information in the polar coordinate system, and calculates a residual 1 which is the difference between the position information and the predicted value. The predicted value and residual 1 calculated here are expressed in the polar coordinate system. The encoding unit 13613 encodes the residual 1 (for example, entropy encoding or arithmetic encoding).

[0502] The conversion unit 13614 restores the position information in the polar coordinate system by adding the predicted value to the residual 1, and converts the position information into position information in the Cartesian coordinate system. Note that the conversion unit 13614 may convert the position information in the polar coordinate system generated by the conversion unit 13611 into position information in the Cartesian coordinate system. The subtraction unit 13615 calculates the residual 2, which is the difference between the input position information in the Cartesian coordinate system and the position information in the Cartesian coordinate system obtained by the conversion unit 13614. This residual 2 corresponds to the conversion error that occurs when the position information in the Cartesian coordinate system is converted into position information in the polar coordinate system, and the obtained position information in the polar coordinate system is converted back into position information in the Cartesian coordinate system.

[0503] The quantization unit 13616 quantizes the residual 2 in the orthogonal coordinate system. The encoding unit 13617 encodes (e.g., entropy encoding or arithmetic encoding) the quantized residual 2. The combining unit 13618 generates a bitstream including the encoded residual 1 and the encoded residual 2.

[0504] In this way, in the position information encoding unit shown in FIG. 63, the position information in the Cartesian coordinate system is converted into position information in the polar coordinate system, and encoding processing (prediction and encoding) is performed on the position information in the polar coordinate system.

[0505] 64 is a block diagram showing the configuration of a location information encoder when the coordinate system of the input location information and the coordinate system of the location information used for encoding are both polar coordinate systems. In other words, this configuration is the configuration of a location information encoder when gps_alt_coordinates_flag=1 and gps_coordinate_trans_enabled_flag=0.

[0506] As shown in Fig. 64, the position information encoding unit includes a prediction unit 13621, a quantization unit 13622, and an encoding unit 13623. The prediction unit 13621 calculates a predicted value of the input position information in a polar coordinate system, and calculates a residual that is the difference between the position information and the predicted value. Note that the predicted value and residual calculated here are expressed in a polar coordinate system.

[0507] The quantization unit 13622 quantizes the residual in the polar coordinate system. The encoding unit 13623 generates a bitstream by encoding (e.g., entropy encoding or arithmetic encoding) the quantized residual. That is, in the position information encoding unit shown in Fig. 64, all processing is performed in the polar coordinate system.

[0508] As described above, although the configuration shown in Figure 62 is similar to the configuration shown in Figure 64, the coordinate systems of the data handled in the prediction, quantization, and encoding for calculating the residual are different. Note that the data processing content may differ between the configuration shown in Figure 62 and the configuration shown in Figure 64.

[0509] Furthermore, the three-dimensional data encoding device may use gps_alt_coordinates_flag and gps_coordinate_trans_enabled_flag to switch the function of the position information encoding unit to one of those shown in Figures 62 to 64. In other words, the three-dimensional data encoding device may have the three position information encoding units shown in Figures 62 to 64 and switch the position information encoding unit to be used. Alternatively, the three-dimensional data encoding device may use gps_alt_coordinates_flag and gps_coordinate_trans_enabled_flag to switch the presence or absence and content of processing on a processing block basis. This allows common processing blocks among the processing blocks included in the three position information encoding units shown in Figures 62 to 64 to be shared, which may make it possible to realize a position information encoding unit that can achieve the functions of the configurations shown in Figures 62 to 64 with a small configuration.

[0510] As described above, the switching of the processes shown in Figures 62 to 64 is indicated by a combination of gps_alt_coordinates_flag and gps_coordinate_trans_enabled_flag. These pieces of information may be indicated in any form. In other words, the bit stream includes information indicating which of a Cartesian coordinate system and a polar coordinate system is used for the coordinate information output (decoded) by the decoding device. The bit stream also includes information indicating which of a Cartesian coordinate system and a polar coordinate system is used for encoding or decoding. The bit stream also includes information indicating whether coordinate transformation is performed.

[0511] Here, point cloud data acquired by a sensor such as LiDAR is data consisting of, for example, angle and distance information from the sensor and reflectance, and includes position information in a polar coordinate system. To input the polar coordinate system point cloud data output from this sensor to an encoding unit that inputs point cloud data in a Cartesian coordinate system, such as that shown in Figure 62 or Figure 63, the position information of the point cloud data must be converted from the polar coordinate system to a Cartesian coordinate system in advance. This conversion introduces a conversion error into the position information, so some information is lost when the polar coordinate system is converted to a Cartesian coordinate system. For this reason, it is not possible to convert from the polar coordinate system to a Cartesian coordinate system and then perform a reversible inverse conversion.

[0512] On the other hand, by using the configuration shown in Fig. 64 and the method described in this embodiment, it is possible to encode data in a polar coordinate system without conversion errors. In other words, lossless compression is possible when encoding point cloud data output from a sensor and decoding the encoded point cloud data.

[0513] Next, the configuration of the position information decoding unit that decodes the bit stream generated by the position information encoding unit will be described.

[0514] Fig. 65 is a block diagram showing the configuration of a location information decoding unit when the coordinate system of the output location information and the coordinate system of the location information used for decoding are both orthogonal coordinate systems. That is, this configuration is the configuration of the location information decoding unit when gps_alt_coordinates_flag = 0. For example, this location information decoding unit decodes the bit stream generated by the location information encoding unit shown in Fig. 62.

[0515] 65 to 67, (x, y, z) in a block indicates that processing is performed using a Cartesian coordinate system, and (r, q, f) in a block indicates that processing is performed using a polar coordinate system. A signal with (x, y, z) indicates that the Cartesian coordinate system is used, and a signal with (r, q, f) indicates that the polar coordinate system is used.

[0516] As shown in FIG. 65, the position information decoding unit includes a decoding unit 13631, an inverse quantization unit 13632, and a reconstruction unit 13633. The decoding unit 13631 generates a residual (a residual after quantization) by decoding (e.g., entropy decoding or arithmetic decoding) the bitstream. The inverse quantization unit 13632 inverse quantizes the residual obtained by the decoding unit 13631. The reconstruction unit 13633 calculates a predicted value and adds the predicted value to the residual after inverse quantization to generate position information in an orthogonal coordinate system. Note that the predicted value and residual calculated here are represented in an orthogonal coordinate system. As such, in the position information decoding unit shown in FIG. 65, all processing is performed in an orthogonal coordinate system.

[0517] Fig. 66 is a block diagram showing the configuration of a location information decoding unit when the coordinate system of output location information is a Cartesian coordinate system and the coordinate system of location information used for decoding is a polar coordinate system. That is, this configuration is the configuration of the location information decoding unit when gps_alt_coordinates_flag=1 and gps_coordinate_trans_enabled_flag=1. For example, this location information decoding unit decodes the bit stream generated by the location information encoding unit shown in Fig. 63.

[0518] As shown in FIG. 66, the position information decoding unit includes a separating unit 13641, a decoding unit 13642, a reconstructing unit 13643, a transforming unit 13644, a decoding unit 13645, an inverse quantizing unit 13646, and an adding unit 13647.

[0519] The separator 13641 obtains coded data of residual 1 and coded data of residual 2 from the bitstream. The decoder 13642 generates residual 1 by decoding (e.g., entropy decoding or arithmetic decoding) the coded data of residual 1. The reconstructor 13643 calculates a predicted value and adds the predicted value to residual 1 after inverse quantization to generate position information in a polar coordinate system. Note that the calculated predicted value and residual 1 are represented in a polar coordinate system. The converter 13644 converts the position information in the polar coordinate system into position information in a Cartesian coordinate system.

[0520] The decoding unit 13645 generates residual 2 (quantized residual 2) by decoding (e.g., entropy decoding or arithmetic decoding) the coded data of residual 2. The inverse quantization unit 13646 inverse quantizes residual 2 obtained by the decoding unit 13645. The predicted value and residual 2 calculated here are represented in a Cartesian coordinate system.

[0521] The adder 13647 adds the residual 2 to the position information in the orthogonal coordinate system obtained by the converter 13644, thereby generating final position information in the orthogonal coordinate system.

[0522] In this way, the position information decoding unit shown in FIG. 66 performs decoding processing (prediction and decoding) on ​​the position information in the polar coordinate system, and converts the obtained position information in the polar coordinate system into position information in the Cartesian coordinate system.

[0523] Fig. 67 is a block diagram showing the configuration of a location information decoding unit when the coordinate system of the output location information and the coordinate system of the location information used for decoding are both polar coordinate systems. That is, this configuration is the configuration of the location information decoding unit when gps_alt_coordinates_flag=1 and gps_coordinate_trans_enabled_flag=0. For example, this location information decoding unit decodes the bit stream generated by the location information encoding unit shown in Fig. 64.

[0524] As shown in FIG. 67, the position information decoding unit includes a decoding unit 13651, an inverse quantization unit 13652, and a reconstruction unit 13653. The decoding unit 13651 generates a residual (a residual after quantization) by decoding (e.g., entropy decoding or arithmetic decoding) the bitstream. The inverse quantization unit 13652 inverse quantizes the residual obtained by the decoding unit 13651. The reconstruction unit 13653 calculates a predicted value and adds the predicted value to the residual after inverse quantization to generate position information in a polar coordinate system. Note that the predicted value and residual calculated here are represented in a polar coordinate system. As such, in the position information decoding unit shown in FIG. 67, all processing is performed in a polar coordinate system.

[0525] As described above, although the configuration shown in Figure 65 is similar to the configuration shown in Figure 67, the coordinate systems of the data handled in prediction for calculating residuals, inverse quantization, and decoding are different. Note that the data processing content may differ between the configuration shown in Figure 65 and the configuration shown in Figure 67.

[0526] Furthermore, the three-dimensional data decoding device may use gps_alt_coordinates_flag and gps_coordinate_trans_enabled_flag to switch the function of the position information decoding unit to one of those shown in Figures 65 to 67. In other words, the three-dimensional data encoding device may have the three position information decoding units shown in Figures 65 to 67, and switch the position information decoding unit to be used. Alternatively, the three-dimensional data decoding device may use gps_alt_coordinates_flag and gps_coordinate_trans_enabled_flag to switch the presence or absence and content of processing on a processing block basis. This allows common processing blocks among the processing blocks included in the three position information decoding units shown in Figures 65 to 67 to be shared, potentially making it possible to realize a position information decoding unit that can achieve the functions of the configurations shown in Figures 65 to 67 with a small configuration.

[0527] Next, the configuration of a three-dimensional data encoding device will be described. Fig. 68 is a block diagram showing the configuration of a three-dimensional data encoding device that encodes position information and attribute information included in point cloud data. The three-dimensional data encoding device shown in Fig. 68 includes a overlapping point joining unit 13661, a position information encoding unit 13662, an attribute information conversion unit 13663, an attribute information encoding unit 13664, and a joining unit 13665.

[0528] If multiple points have the same position information in the input position information (data A), that is, if there are duplicate points, the duplicate point merger 13661 merges the multiple points with the same position information into one point. If there are duplicate points, this occurs when there are duplicate points originally in the input point group, or when duplicate points are generated by the quantization process.

[0529] The position information encoder 13662 encodes the merged position information to generate encoded position information data. For example, the position information encoder 13662 includes at least one of the three position information encoders shown in FIGS. 62 to 64.

[0530] When overlapping points are merged, the attribute information conversion unit 13663 generates attribute information for one merged point from the attribute information for the multiple overlapping points. The attribute information conversion unit 13663 may generate attribute information for one merged point using any of the multiple overlapping points, or may convert multiple pieces of attribute information for the multiple overlapping points into attribute information for one point using a predetermined method. For example, the attribute information conversion unit 13663 may determine the value of the attribute information by weighting and adding based on the distance between the merged point and the point before the merge, using position information before the merge (data A) and position information after the merge (data B).

[0531] The attribute information encoding unit 13664 generates attribute information encoded data by encoding the attribute information obtained by the attribute information conversion unit 13663 based on the position information (data C). Specifically, when encoding the attribute information using LoD (Level of Detail), the attribute information encoding unit 13664 generates LoD layers using the position information. Here, LoD is a method of classifying each 3D point into multiple layers using the position information of the 3D points, generating predicted values ​​of attribute information of the 3D points based on the set layers, and encoding the prediction residual between the attribute information and the predicted values. In other words, the position information is used, for example, to generate predicted values ​​of the attribute information.

[0532] Alternatively, the attribute information encoding unit 13664 uses position information when encoding attribute information using RAHT (Region Adaptive Hierarchical Transform) or Haar transform. In these methods, the attribute information encoding unit 13664 generates high-frequency components and low-frequency components for each layer and performs quantization and entropy coding on those values. In addition, the position information is used to set two 3D points used as a pair for calculation in the lowest layer.

[0533] For example, if the coordinate system of data A is a Cartesian coordinate system, the position information encoding unit 13662 and the attribute information encoding unit 13664 perform processing using position information in the Cartesian coordinate system. That is, in this case, data B and data C are in Cartesian coordinate systems. In this way, by using a Cartesian coordinate system, it is possible to improve the efficiency of attribute information conversion and attribute information encoding.

[0534] Furthermore, for example, if the coordinate system of data A is a polar coordinate system, the position information encoding unit 13662 and the attribute information encoding unit 13664 perform processing using position information in the polar coordinate system. In other words, data B and data C are in polar coordinate systems. This may improve encoding efficiency. Furthermore, since coordinate conversion processing is not required, the processing amount and processing delay can be reduced. In this way, the three-dimensional data encoding device may be equipped with a configuration that performs processing corresponding to both the Cartesian coordinate system and the polar coordinate system, and may switch between processing in the Cartesian coordinate system and processing in the polar coordinate system based on the coordinate system of the input position information.

[0535] The three-dimensional data encoding device may perform encoding in a coordinate system that is likely to improve encoding efficiency. That is, the three-dimensional data encoding device may convert the coordinate system of the position information. For example, if the input data is arranged in angular order, the three-dimensional data encoding device may perform encoding using a polar coordinate system regardless of the input coordinate system.

[0536] Furthermore, when the three-dimensional data encoding device is equipped with a configuration for processing either a Cartesian coordinate system or a polar coordinate system, if the coordinate system of the input position information differs from the coordinate system of the equipped configuration, the three-dimensional data encoding device may convert the coordinate system of the position information into the coordinate system of the equipped configuration. For example, when the position information encoding unit 13662 and the attribute information encoding unit 13664 support Cartesian coordinate systems, if the coordinate system of data A is a polar coordinate system, the three-dimensional data encoding device converts data B and data C from the polar coordinate system to the Cartesian coordinate system. In this way, when the three-dimensional data encoding device is equipped with a configuration for one of the coordinate systems, it can encode data A in both coordinate systems.

[0537] In other words, the attribute information encoding unit 13664 generates attribute information encoded data by encoding the attribute information of one three-dimensional point to be encoded using the position information of the one three-dimensional point to be encoded in a first coordinate system, which is the coordinate system of the position information output during decoding, of the Cartesian coordinate system and the polar coordinate system.

[0538] The combining unit 13665 generates a bit stream including the coded position information data and the coded attribute information data.

[0539] Next, the configuration of a three-dimensional data decoding device will be described. Fig. 69 is a block diagram showing the configuration of a three-dimensional data decoding device that decodes position information and attribute information included in point cloud data. The three-dimensional data decoding device shown in Fig. 69 includes a separator 13671, a position information decoder 13672, and an attribute information decoder 13673.

[0540] The separator 13671 obtains coded position information data and coded attribute information data from the input bit stream. The position information decoder 13672 generates position information (data D) by decoding the coded position information data. For example, the position information decoder 13672 includes at least one of the three position information decoders shown in FIGS. 65 to 67.

[0541] The attribute information decoding unit 13673 generates attribute information by decoding the attribute information encoded data using the decoded position information (data D). Specifically, when decoding attribute information using LoD, the attribute information decoding unit 13673 generates an LoD layer using the position information. That is, the position information is used, for example, to generate a predicted value of the attribute information. Alternatively, the attribute information decoding unit 13673 uses the position information when encoding attribute information using RAHT or Haar transform.

[0542] Here, the coordinate system of the decoded position information (data D) is either a rectangular coordinate system or a polar coordinate system. For example, when decoding is performed using the configuration of Figure 65 or Figure 66, data D is in a rectangular coordinate system, and when decoding is performed using the configuration of Figure 67, data D is in a polar coordinate system.

[0543] If the coordinate system of data D is a rectangular coordinate system, the attribute information decoding unit 13673 performs processing using position information in the rectangular coordinate system. By using the position information in the rectangular coordinate system, the attribute information decoding unit 13673 may be able to improve the conversion efficiency of the attribute information and the encoding efficiency of the attribute information. Furthermore, if the coordinate system of data D is a polar coordinate system, the attribute information decoding unit 13673 performs processing using position information in the polar coordinate system. By using position information in the polar coordinate system, the attribute information decoding unit 13673 may be able to improve the encoding efficiency. Furthermore, since coordinate conversion processing is not required, the processing amount and processing delay can be reduced. In this way, the three-dimensional data decoding device may be equipped with a configuration that performs processing corresponding to both a rectangular coordinate system and a polar coordinate system, and may switch between processing in the rectangular coordinate system and processing in the polar coordinate system based on the coordinate system of the input encoded position information data.

[0544] Furthermore, when the three-dimensional data decoding device has a configuration for processing either a rectangular coordinate system or a polar coordinate system, if the coordinate system of the position information (data D) from the position information decoding unit 13672 differs from the coordinate system of the device, the three-dimensional data decoding device may convert the coordinate system of the position information into the coordinate system of the device. For example, when the attribute information decoding unit 13673 supports a rectangular coordinate system, if the coordinate system of the position information (data D) is a polar coordinate system, the three-dimensional data decoding device converts the data D from the polar coordinate system to the rectangular coordinate system. As a result, when the three-dimensional data decoding device has a configuration of one of the coordinate systems, it can decode the attribute information based on the data D in the polar coordinate system and the data D in the rectangular coordinate system.

[0545] In this way, the attribute information decoding unit 13673 generates attribute information of one three-dimensional point to be decoded by decoding the attribute information encoded data using the position information of the one three-dimensional point to be decoded in a first coordinate system, which is a coordinate system of the position information of the one three-dimensional point to be decoded, out of a Cartesian coordinate system and a polar coordinate system, to calculate the position information of the one three-dimensional point to be decoded.

[0546] The data D input to the attribute information decoding unit 13673 may be normalized so that the data value falls within a certain range. For example, if the data D is in a Cartesian coordinate system, the three-dimensional data decoding device may derive the maximum value of the position information based on the coordinates (x, y, z) of the points constituting the point cloud or information on the bounding box encompassing the point cloud, and normalize the value of the position information of the data D based on the derived maximum value. Furthermore, if the data D is in a polar coordinate system, the three-dimensional data decoding device may derive the maximum value based on the distance r from the origin to the points constituting the point cloud, and normalize the value of the position information of the data D based on the derived maximum value. For example, the three-dimensional data decoding device may divide the value of the position information of the data D by the derived maximum value.

[0547] Note that the position information decoding unit 13672 may derive the above-mentioned maximum value or bounding box information and output it to the attribute information decoding unit 13673, and the attribute information decoding unit 13673 may then normalize the data D using the maximum value or bounding box information. In this way, by the attribute information decoding unit 13673 performing normalization based on the information output from the position information decoding unit 13672, it may be possible to reduce the processing amount and processing delay of the attribute information decoding unit 13673 in deriving the maximum value.

[0548] Although an example in which normalization is performed in a three-dimensional data decoding device has been described above, similar processing may also be performed on data C in a three-dimensional data encoding device. Furthermore, similar to the position information decoding unit 13672, the position information encoding unit 13662 may derive maximum value or bounding box information and output it to the attribute information encoding unit 13664, and the attribute information encoding unit 13664 may normalize data C using the maximum value or bounding box information.

[0549] In addition, although the coordinate system in which position information is expressed as (x, y, z) was described above as a Cartesian coordinate system, it may also be called a Cartesian coordinate system. Furthermore, although the coordinate system in which position information is expressed as (r, q, f) was described above as a polar coordinate system, it may also be called a spherical coordinate system.

[0550] 70 is a flowchart of three-dimensional data decoding processing according to this embodiment. First, the three-dimensional data decoding device decodes position information from the bit stream (S13601). Next, in process 1 (S13602), the three-dimensional data decoding device determines the coordinate system of the position information to be used for decoding the attribute information. Specifically, if the decoding of the position information has been performed in a polar coordinate system (Yes in S13611), the three-dimensional data decoding device determines to decode the attribute information using position information in the polar coordinate system (S13612). On the other hand, if the decoding of the position information has not been performed in a polar coordinate system (No in S13611), that is, if the decoding of the position information has been performed in a Cartesian coordinate system, the three-dimensional data decoding device determines to decode the attribute information using position information in the Cartesian coordinate system (S13613).

[0551] Next, in process 2 (S13603), the three-dimensional data decoding device normalizes the position information used to decode the attribute information. Specifically, if it is determined that position information in the polar coordinate system is used to decode the attribute information (Yes in S13621), the three-dimensional data decoding device normalizes the position information in the polar coordinate system (S13623). For example, as described above, the three-dimensional data decoding device performs normalization based on the distance r from the origin to the points that make up the point cloud. For example, the three-dimensional data decoding device divides the value of the position information of data D by the maximum value of the distance r.

[0552] On the other hand, if the three-dimensional data decoding device determines that polar coordinate system position information will not be used to decode the attribute information (No in S13621), that is, if it determines that Cartesian coordinate system position information will be used to decode the attribute information, it converts the Cartesian coordinate system position information into polar coordinate system position information (S13622) and normalizes the obtained polar coordinate system position information (S13623).

[0553] Next, the three-dimensional data decoding device encodes the attribute information using the normalized position information. Note that, when decoding the attribute information using position information in a Cartesian coordinate system, the three-dimensional data decoding device may convert the normalized position information into a Cartesian coordinate system and decode the attribute information using the obtained position information. Alternatively, when decoding the attribute information using position information in a Cartesian coordinate system, the three-dimensional data decoding device may normalize the position information in the Cartesian coordinate system without performing the coordinate conversion in step S13622.

[0554] As described above, the three-dimensional data encoding device according to this embodiment performs the processing shown in Fig. 71. The three-dimensional data encoding device encodes a plurality of three-dimensional points. The three-dimensional data encoding device calculates a predicted value based on position information of one or more candidate points among a plurality of encoded three-dimensional points (S13631), calculates a first residual that is the difference between the predicted value and position information of one three-dimensional point to be encoded among the plurality of three-dimensional points (S13632), generates encoded position information data by encoding the first residual (S13633), and generates first identification information (e.g., gps_alt_coordinates_flag) that indicates whether the encoded position information data includes first encoded position information data calculated in a polar coordinate system (S13634).

[0555] The three-dimensional data encoding device generates attribute information encoded data by encoding attribute information of one three-dimensional point to be encoded using position information of the one three-dimensional point to be encoded in a first coordinate system, which is a coordinate system of position information output during decoding, of the Cartesian coordinate system and the polar coordinate system (S13635).The three-dimensional data encoding device generates a bit stream including the position information encoded data, the first identification information, and the attribute information encoded data (S13636).

[0556] According to this, the three-dimensional data encoding device encodes the attribute information using the first coordinates, which are the coordinate system of the position information output during decoding. This improves encoding efficiency. In addition, the amount of processing can be reduced because coordinate conversion processing and the like are no longer necessary.

[0557] For example, when the first coordinate system is a polar coordinate system, the three-dimensional data encoding device normalizes the position information of one three-dimensional point to be encoded in the polar coordinate system, and encodes the attribute information of the one three-dimensional point to be encoded using the position information in the normalized polar coordinate system. This allows the three-dimensional data encoding device to limit the range of values ​​of the position information used to decode the attribute information, thereby reducing the amount of processing.

[0558] For example, in generating position information encoded data, the three-dimensional data encoding device generates information indicating the maximum value of multiple three-dimensional coordinates or a bounding box that encompasses multiple three-dimensional points, normalizes the position information in a first coordinate system of one three-dimensional point to be encoded using the generated information, and encodes the attribute information of the one three-dimensional point to be encoded using the normalized position information in the first coordinate system. This allows the three-dimensional data encoding device to reduce the amount of processing required to decode the attribute information.

[0559] For example, if the position information of one or more candidate points used to calculate the predicted value and the position information of one three-dimensional point to be encoded used to calculate the first residual are position information in a polar coordinate system, the first identification information indicates that the position information encoded data includes the first position information encoded data, and the bit stream further includes second identification information (gps_coordinate_trans_enabled_flag) indicating whether to output position information in a polar coordinate system or position information in a Cartesian coordinate system during decoding.

[0560] For example, if the first identification information indicates that the position information encoded data does not include the first position information encoded data, the bitstream does not include the second identification information.

[0561] For example, in the case where the first residual is quantized in the encoding and the quantized first residual is encoded, the second identification information indicates that position information in a polar coordinate system is to be output in the decoding.

[0562] For example, in calculating the first residual, the three-dimensional data encoding device further generates position information in a polar coordinate system by transforming the coordinate system of the position information in the Cartesian coordinate system of one three-dimensional point to be encoded, and if the first residual is the difference between the position information in the polar coordinate system after the transformation and the predicted value, the three-dimensional data encoding device further generates position information in a Cartesian coordinate system by inversely transforming the coordinate system of the position information in the polar coordinate system after the transformation, and calculates a second residual that is the difference between the position information in the Cartesian coordinate system and the position information in the Cartesian coordinate system after the inverse transformation, and in generating the position information encoded data, generates the position information encoded data by encoding the first residual and the second residual, the first identification information indicates that the position information encoded data includes the first position information encoded data, and the second identification information indicates that the position information in the Cartesian coordinate system is to be output during decoding.

[0563] For example, the three-dimensional data encoding device includes a processor and a memory, and the processor performs the above-mentioned processing using the memory.

[0564] Furthermore, the three-dimensional data decoding device according to this embodiment performs the processing shown in FIG. 72. The three-dimensional data decoding device decodes a plurality of three-dimensional points. The three-dimensional data decoding device acquires a bitstream including position information encoded data obtained by encoding a first residual, first identification information (e.g., gps_alt_coordinates_flag) indicating whether the position information encoded data includes first position information encoded data calculated in a polar coordinate system, and attribute information encoded data (S13641). The three-dimensional data decoding device calculates a predicted value based on position information of one or more candidate points among the plurality of decoded three-dimensional points (S13642), calculates a first residual by decoding the encoded first residual (S13643), and calculates position information of one three-dimensional point to be decoded by adding the predicted value and the first residual (S13644). The first coordinate system, which is the coordinate system of the calculated position information of one three-dimensional point to be decoded, is a coordinate system that corresponds to whether the first position information encoded data indicated by the first identification information is included. The three-dimensional data decoding device generates attribute information of one three-dimensional point to be decoded by decoding the attribute information encoded data using position information of the one three-dimensional point to be decoded in the first coordinate system (S13645).

[0565] According to this, the three-dimensional data decoding device decodes the attribute information using the position information of the first coordinates. This improves coding efficiency. In addition, since coordinate conversion processing and the like are no longer necessary, the amount of processing can be reduced.

[0566] For example, when the first coordinate system is a polar coordinate system, the three-dimensional data decoding device normalizes the position information in the polar coordinate system of one three-dimensional point to be decoded, and decodes the attribute information encoded data using the position information in the normalized polar coordinate system. This allows the three-dimensional data decoding device to limit the range of values ​​of the position information used to decode the attribute information, thereby reducing the amount of processing.

[0567] For example, in calculating the position information of one 3D point to be decoded, the 3D data decoding device generates information indicating the maximum value of multiple 3D coordinates or a bounding box that encompasses multiple 3D points, normalizes the position information of the one 3D point to be decoded in a first coordinate system using the generated information, and decodes the attribute information encoded data using the normalized position information in the first coordinate system. This allows the 3D data decoding device to reduce the amount of processing required to decode the attribute information.

[0568] For example, the bitstream further includes second identification information (gps_coordinate_trans_enabled_flag) indicating whether to output position information in a polar coordinate system or position information in a Cartesian coordinate system during decoding, and when the first identification information indicates that the position information encoded data includes first position information encoded data, the position information of one or more candidate points used to calculate the predicted value and the position information of one three-dimensional point to be encoded used to calculate the first residual are position information in a polar coordinate system.

[0569] For example, if the first identification information indicates that the position information encoded data does not include the first position information encoded data, the bitstream does not include the second identification information.

[0570] For example, when the second identification information indicates that position information in a polar coordinate system is to be output in decoding, the coordinate system of the calculated position information of one three-dimensional point to be decoded is the polar coordinate system.

[0571] For example, in a three-dimensional data decoding device, when the first identification information indicates that the position information encoded data includes first position information encoded data and the second identification information indicates that position information in a Cartesian coordinate system is to be output during decoding, the three-dimensional data decoding device calculates the position information of one three-dimensional point to be decoded by converting the coordinate system of the position information in a polar coordinate system obtained by adding the predicted value and the first residual, and calculating the position information in the Cartesian coordinate system obtained by the conversion as the position information of one three-dimensional point to be decoded.

[0572] For example, if the first identification information indicates that the position information encoded data includes first position information encoded data and the second identification information indicates that decoding outputs position information in a Cartesian coordinate system, the position information encoded data further includes an encoded second residual, and the three-dimensional data decoding device further calculates the second residual by decoding the encoded second residual, and in calculating the position information of one three-dimensional point to be decoded, adds the position information in the Cartesian coordinate system obtained by converting the coordinate system to the second residual, and calculates the position information obtained by the addition as the position information of one three-dimensional point to be decoded.

[0573] For example, the three-dimensional data decoding device includes a processor and a memory, and the processor performs the above processing using the memory.

[0574] (Embodiment 7) In this embodiment, the encoding process and decoding process of the position information when the coordinate system of the position information of the three-dimensional point group is a Cartesian coordinate system and a polar coordinate system will be described. First, the encoding process will be described.

[0575] 73 is a block diagram showing the configuration of a position information encoding unit 13700 when the coordinate system of the position information of an input three-dimensional point group is a Cartesian coordinate system. The position information encoding unit 13700 is included in a three-dimensional data encoding device, and includes a shift unit 13701 and an encoding unit 13702.

[0576] In Figure 73 etc., information marked with (x, y, z) indicates that the Cartesian coordinate system is used, and information marked with (r, θ, φ) indicates that the polar coordinate system is used. Here, r is the distance from the origin to a point that constitutes the point cloud, θ is the horizontal angle relative to the point that constitutes the point cloud on the horizontal plane including the origin, and φ is the elevation angle relative to the origin to the point that constitutes the point cloud.

[0577] The shifting unit 13701 generates shifted position information in the orthogonal coordinate system by shifting the position information in the orthogonal coordinate system. The encoding unit 13702 generates a bit stream by encoding the shifted position information.

[0578] Fig. 74 is a diagram showing an example of shift processing when input position information is in a Cartesian coordinate system and the position information in the Cartesian coordinate system is encoded. As shown in Fig. 74, when input position information is in a Cartesian coordinate system, the shift unit 13701 shifts the coordinates (position information) of the point cloud before encoding the position information. For example, when encoding the position information is performed using a Cartesian coordinate system, the shift unit 13701 shifts (subtracts) the coordinates of the point cloud so that the coordinates of the bounding box of the point cloud (for example, the coordinates of the lower left corner) become the origin. Alternatively, the shift unit 13701 may shift the coordinates of the point cloud so that the center of the coordinates of the bounding box of the point cloud becomes the origin.

[0579] 74 is a diagram of two arbitrary axes, and the shift is not limited to this. That is, a shift is a movement of a position along at least one axis.

[0580] FIG. 75 is a diagram showing an example of shift processing when input position information is in a Cartesian coordinate system and position information in a polar coordinate system is encoded. For example, the example shown in FIG. 75 is a case where position information in the Cartesian coordinate system is converted into position information in a polar coordinate system, and the position information in the polar coordinate system is encoded. When encoding using a polar coordinate system, the shift unit 13701 shifts the coordinates of the Cartesian coordinate system of the point cloud so that the position of a sensor such as a LiDAR that acquires the point cloud becomes the origin. A conversion unit (not shown in FIG. 73) converts the position information in the Cartesian coordinate system into position information in a polar coordinate system. The conversion unit is disposed between the shift unit 13701 and the encoding unit 13702.

[0581] Note that the shifted coordinates are not limited to the above example, and may be set to coordinates that can improve encoding efficiency. For example, by shifting the position information, the value of the position information, i.e., the absolute value, can be reduced, thereby reducing the amount of encoding. In another example, by shifting the position information, it is possible to obtain position information having a value close to that of already encoded position information, thereby improving encoding efficiency. Furthermore, the encoding unit 13702 may include coordinate information regarding the shifted coordinates (e.g., information indicating the origin before the shift or information indicating the shift amount) in the bitstream as metadata. Note that the shift amount may be represented by a single value common to each component in the coordinate system, or may be represented by a value for each component, and the values ​​for each component may be the same or different. Furthermore, the metadata may or may not be encoded.

[0582] 76 is a block diagram showing the configuration of the position information encoding unit 13710 when the coordinate system of the position information of the input three-dimensional point cloud is a polar coordinate system. The position information encoding unit 13710 is included in the three-dimensional data encoding device and includes an encoding unit 13711.

[0583] FIG. 77 is a diagram illustrating an example of processing when input position information is in a polar coordinate system and the position information in the polar coordinate system is encoded. As illustrated in FIG. 77, the position information of the input point cloud is in a polar coordinate system, and the position of a sensor such as a LiDAR that acquires the point cloud is the origin, so the position information encoding unit 13710 does not shift the coordinates of the position information. Note that the position of the sensor such as a LiDAR that acquires the point cloud does not have to be the origin. If the position of the sensor does not coincide with the origin, the position information in the polar coordinate system may be shifted, for example, so that the position of the sensor coincides with the origin. However, the shift of the position information in the polar coordinate system is not limited to this example. The encoding unit 13711 generates a bitstream by encoding the input position information in the polar coordinate system as is.

[0584] Furthermore, the encoding unit 13711 may include sensor position information indicating the coordinates of the sensor position in the bitstream as metadata.

[0585] Furthermore, the three-dimensional data encoding device may include both the position information encoding unit 13700 shown in FIG. 73 and the position information encoding unit 13710 shown in FIG. 76, or may include only one of them.

[0586] Next, a decoding process for decoding the bit stream generated as described above will be described. FIG. 78 is a block diagram showing the configuration of a position information decoding unit 13720 that decodes a bit stream generated by encoding position information in an orthogonal coordinate system. In other words, the position information decoding unit 13720 outputs position information in an orthogonal coordinate system. The position information decoding unit 13720 is included in a three-dimensional data decoding device, and includes a decoding unit 13721 and an inverse shift unit 13722. Note that the example shown in FIG. 78 is a case where the position information in the orthogonal coordinate system is restored by decoding a bit stream generated by encoding position information after shifting in an orthogonal coordinate system, and performing inverse shift processing on the obtained position information based on the coordinate information.

[0587] The decoding unit 13721 obtains post-shift position information in the Cartesian coordinate system by decoding the bitstream. The decoding unit 13721 also obtains coordinate information, which is metadata, from the bitstream (for example, information indicating the origin before the shift or information indicating the shift amount).

[0588] The inverse shift unit 13722 inversely shifts (adds) the decoded shifted position information by the shift amount indicated in the coordinate information, which is metadata. Note that the shift amount is a value calculated based on the difference between the origin coordinates before the shift and the origin coordinates after the shift. This enables the three-dimensional data decoding device to restore the point group input to the three-dimensional data encoding device.

[0589] 79 is a block diagram showing the configuration of a position information decoding unit 13730 that decodes a bit stream generated by encoding position information in a polar coordinate system. In other words, the position information decoding unit 13730 outputs position information in a polar coordinate system. The position information decoding unit 13730 is included in a three-dimensional data decoding device and includes a decoding unit 13731.

[0590] The decoding unit 13731 obtains position information in the polar coordinate system by decoding the bitstream. The decoding unit 13731 also obtains sensor position information, which is metadata and indicates the coordinates of the sensor position, from the bitstream. The position information decoding unit 13730 outputs the decoded position information in the polar coordinate system as is without shifting it.

[0591] The polar coordinate system may be expressed by parameters other than (r, θ, φ). For example, (r, φ, ι) may be used. Here, r is the radius when a point is projected onto the xy plane, and ι is an index representing the height or θ.

[0592] Furthermore, the three-dimensional data decoding device may include both the position information decoding unit 13720 shown in FIG. 78 and the position information decoding unit 13730 shown in FIG. 79, or may include only one of them.

[0593] Modifications of the above configuration will be described below. For example, the shift unit 13701 may be shared between the configuration shown in Fig. 73 and the configuration shown in Fig. 76. In this case, if the input position information is in a polar coordinate system, the three-dimensional data encoding device sets the shift amount in the shift processing of the shift unit 13701 to 0. This makes it possible to reduce implementation costs.

[0594] Furthermore, the three-dimensional data encoding device may switch whether to perform shift processing based on whether the coordinate system of the input position information is a Cartesian coordinate system or a polar coordinate system. Fig. 80 is a block diagram showing the configuration of the position information encoding unit 13740 in this case. The position information encoding unit 13740 is included in the three-dimensional data encoding device, and includes a switching unit 13741, a shift unit 13742, and an encoding unit 13743.

[0595] If the input position information is position information in a Cartesian coordinate system, the switching unit 13741 outputs the position information to the shifting unit 13742. If the input position information is position information in a polar coordinate system, the switching unit 13741 outputs the position information to the encoding unit 13743.

[0596] The shifting unit 13742 generates shifted position information in a Cartesian coordinate system by shifting the position information in the Cartesian coordinate system. The encoding unit 13743 generates a bit stream by encoding the shifted position information. The encoding unit 13743 also generates a bit stream by encoding the position information in a polar coordinate system. The encoding unit 13743 also stores the sensor position information in the bit stream. Note that the encoding unit 13743 may store the sensor position information in the bit stream only when the position information is expressed in a polar coordinate system.

[0597] With this configuration, when the position information is in a polar coordinate system, the position information encoding unit 13740 can reduce the amount of processing by not performing shift processing.

[0598] Alternatively, a configuration may be adopted in which the configuration shown in FIG. 78 and the configuration shown in FIG. 79 are provided, and post-shift position information in a rectangular coordinate system or a polar coordinate system is output from each decoding unit to the inverse shift unit 13722. In this case, if the output position information is in a polar coordinate system, the inverse shift unit 13722 sets the shift amount to 0 regardless of the value of the sensor position information. In other words, the inverse shift unit 13722 does not perform a shift. Alternatively, if the sensor position information is not 0, the three-dimensional data decoding device may determine that the bit stream is a non-compliant stream. This makes it possible to reduce implementation costs.

[0599] Furthermore, the three-dimensional data decoding device may switch whether to perform inverse shift processing based on whether the coordinate system of the position information to be output is a Cartesian coordinate system or a polar coordinate system. Fig. 81 is a block diagram showing the configuration of the position information decoding unit 13750 in this case. The position information decoding unit 13750 is included in the three-dimensional data decoding device, and includes a decoding unit 13751, an inverse shift unit 13752, and a switching unit 13753.

[0600] The decoding unit 13751 generates shifted position information in a rectangular coordinate system or position information in a polar coordinate system by decoding the bit stream. Furthermore, when the decoding unit 13751 decodes position information in a polar coordinate system, it acquires sensor position information from the bit stream.

[0601] Here, the decoding unit 13751 may use the gps_alt_coordinates_flag and gps_coordinate_trans_enabled_flag shown in the sixth embodiment as information indicating the coordinate system of the position information encoded by the three-dimensional data encoding device.

[0602] The gps_alt_coordinates_flag is information indicating whether a coordinate system other than Cartesian coordinates, such as polar coordinates, is used in the encoding or decoding of each point. For example, gps_alt_coordinates_flag=1 indicates that a polar coordinate system is used in the encoding or decoding of position information that references GPS, and gps_alt_coordinates_flag=0 indicates that a Cartesian coordinate system is used in the encoding or decoding of position information that references GPS.

[0603] The gps_coordinate_trans_enabled_flag indicates whether to output location information in polar coordinates or in Cartesian coordinates during decoding. For example, gps_alt_coordinates_flag=1 indicates that coordinate transformation of location information referring to GPS is performed, and gps_alt_coordinates_flag=0 indicates that coordinate transformation of location information referring to GPS is not performed.

[0604] Based on this information, the decoding unit 13751 performs decoding processing, and outputs shifted position information in the orthogonal coordinate system or position information in the polar coordinate system.

[0605] The inverse shift unit 13752 generates position information in the orthogonal coordinate system by inversely shifting the shifted position information in the orthogonal coordinate system.

[0606] The switching unit 13753 outputs the decoded position information when the decoded position information is position information in a rectangular coordinate system, and outputs the decoded position information when the decoded position information is position information in a polar coordinate system. The switching unit 13753 may also perform switching based on information in gps_coordinate_trans_enabled_flag.

[0607] Although the above-described operation examples of the decoding unit 13751 and the switching unit 13753 are based on the gps_alt_coordinates_flag and the gps_coordinate_trans_enabled_flag, the decoding unit 13751 and the switching unit 13753 may operate based on other information. For example, the decoding unit 13751 and the switching unit 13753 may operate based on information (e.g., mode information) included in the bitstream that indicates a combination of an input coordinate system, whether or not the coordinate system is converted, and an output coordinate system.

[0608] With this configuration, the position information decoding unit 13750 can reduce the amount of processing by not performing reverse shift processing when the position information is in a polar coordinate system.

[0609] 80 and 81 are examples where the coordinate system of the position information input to the three-dimensional data encoding device and the coordinate system of the position information output from the three-dimensional data decoding device are the same. For example, if the coordinate system of the position information input to the three-dimensional data encoding device (e.g., the position information shown in Figure 80) is a Cartesian coordinate system, then the coordinate system output from the three-dimensional data decoding device (the position information shown in Figure 81) is also a Cartesian coordinate system. Furthermore, if the coordinate system of the position information input to the three-dimensional data encoding device is a polar coordinate system, then the coordinate system output from the three-dimensional data decoding device is also a polar coordinate system.

[0610] On the other hand, when the decoded position information is in a polar coordinate system, the three-dimensional data decoding device may convert the position information into position information in a rectangular coordinate system and output the converted position information in the rectangular coordinate system. FIG. 82 i...

Claims

1. Obtain a bitstream containing encoded position information in which position information of the three-dimensional point is encoded; generating the position information for decoding the attribute information of the three-dimensional point by decoding the encoded position information; reading, from the bit stream, first flag information indicating whether the position information is expressed in a polar coordinate system or a rectangular coordinate system; Shifting the position information based on at least the first flag information. Three-dimensional data decoding method.

2. shifting the position information when the first flag information indicates that the position information is expressed in the Cartesian coordinate system; 2. The three-dimensional data decoding method according to claim 1.

3. a processor; a memory; The processor uses the memory to: Obtain a bitstream containing encoded position information in which position information of the three-dimensional point is encoded; generating the position information for decoding attribute information of the three-dimensional points by decoding the encoded position information; reading, from the bit stream, first flag information indicating whether the position information is expressed in a polar coordinate system or a rectangular coordinate system; Shifting the position information based on at least the first flag information. Three-dimensional data decoding device.

Images