Three-dimensional data encoding method, three-dimensional data decoding method, three-dimensional data encoding device, and three-dimensional data decoding device

Abstract

This three-dimensional data encoding method comprises: generating a first predicted value of positional information of a first three-dimensional point included in point group data obtained by a sensor, by using the scan angle of the sensor and positional information of a referential three-dimensional point included in the point group data (S10061); calculating a first difference value between the first predicted value and the positional information of the first three-dimensional point (S10062); and generating a bit stream that contains the first difference value and information indicating the scan angle (S10063). For example, the positional information of the first three-dimensional point is represented by xyz coordinates, and the scan angle may be one in an xy plane.

Description

[Technical field]

[0001] The present disclosure relates to a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, and a three-dimensional data decoding device. [Background technology]

[0002] In the future, devices and services that utilize 3D data are expected to become widespread in a wide range of fields, such as computer vision for autonomous operation of automobiles or robots, map information, monitoring, infrastructure inspection, video distribution, etc. 3D data is acquired in various ways, such as distance sensors such as range finders, stereo cameras, or a combination of multiple monocular cameras.

[0003] One method of expressing three-dimensional data is a method called a point cloud, which represents the shape of a three-dimensional structure using a group of points in a three-dimensional space. In a point cloud, the positions and colors of the points are stored. Point clouds are expected to become the mainstream method of expressing three-dimensional data, but point clouds have a very large amount of data. Therefore, when storing or transmitting three-dimensional data, it is essential to compress the amount of data by encoding, just like two-dimensional video images (examples include MPEG-4 AVC or HEVC standardized by MPEG).

[0004] In addition, compression of point clouds is partially supported by public libraries (Point Cloud Library) that perform point cloud-related processing.

[0005] Furthermore, a technique is known that uses three-dimensional map data to search for and display facilities located around a vehicle (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0006] [Patent Document 1] International Publication No. 2014 / 020663 Summary of the Invention [Problem to be solved by the invention]

[0007] In the three-dimensional data encoding process and the three-dimensional data decoding process, it is desirable to be able to improve the encoding efficiency.

[0008] An object of the present disclosure is to provide a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, or a three-dimensional data decoding device that can improve encoding efficiency. [Means for solving the problem]

[0009] A three-dimensional data encoding method according to one aspect of the present disclosure includes: The processor: generating a first predicted value of position information of a first 3D point included in the point cloud data using position information and angle information of a reference 3D point included in the point cloud data; The processor: Calculating a first difference value between the position information of the first 3D point and the first predicted value; The processor generates a second predicted value of position information of a second 3D point included in the point cloud data using the position information of the first 3D point and the angle information, and calculates a second difference value between the position information of the second 3D point and the second predicted value, and the processor: the first difference value; The second difference value; A bit stream including the angle information is generated.

[0010] A three-dimensional data decoding method according to one aspect of the present disclosure includes: The processor: A first difference value between position information of a first 3D point included in the point cloud data and a first predicted value from the bit stream; a second difference value between position information of a second three-dimensional point included in the point cloud data and a second predicted value; Obtain angle information and The processor: generating the first predicted value using position information of a reference 3D point included in the point cloud data and the angle information; The processor: calculating the position information of the first 3D point by adding the first predicted value and the first difference value; The processor generates the second predicted value using the position information of the first 3D point and the angle information, and calculates the position information of the second 3D point by adding the second predicted value and the second difference value. . Effect of the Invention

[0011] The present disclosure can provide a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, or a three-dimensional data decoding device that can improve encoding efficiency. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram showing a configuration of a three-dimensional data encoding / decoding system according to the first embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of a configuration of point cloud data according to the first embodiment. [Diagram 3] FIG. 3 is a diagram showing an example of a configuration of a data file in which point cloud data information according to the first embodiment is described. [Figure 4] FIG. 4 is a diagram showing types of point cloud data according to the first embodiment. [Diagram 5] FIG. 5 is a diagram showing a configuration of a first encoding unit according to the first embodiment. [Figure 6] FIG. 6 is a block diagram of a first encoding unit according to the first embodiment. [Figure 7] FIG. 7 is a diagram showing a configuration of a first decoding unit according to the first embodiment. [Figure 8] FIG. 8 is a block diagram of a first decoding unit according to the first embodiment. [Figure 9] FIG. 9 is a block diagram of a three-dimensional data encoding device according to the first embodiment. [Figure 10] FIG. 10 is a diagram showing an example of location information according to the first embodiment. [Figure 11] FIG. 11 is a diagram showing an example of an octree representation of position information according to the first embodiment. [Figure 12] FIG. 12 is a block diagram of the three-dimensional data decoding device according to the first embodiment. [Figure 13] FIG. 13 is a block diagram of the attribute information encoding unit according to the first embodiment. [Figure 14] FIG. 14 is a block diagram of the attribute information decoding unit according to the first embodiment. [Figure 15] FIG. 15 is a block diagram showing a configuration of an attribute information encoding unit according to the first embodiment. As shown in FIG. [Figure 16] FIG. 16 is a block diagram of the attribute information encoding unit according to the first embodiment. [Figure 17] FIG. 17 is a block diagram showing a configuration of an attribute information decoding unit according to the first embodiment. As shown in FIG. [Figure 18] FIG. 18 is a block diagram of the attribute information decoding unit according to the first embodiment. As shown in FIG. [Figure 19] FIG. 19 is a diagram showing a configuration of a second encoding unit according to the first embodiment. As shown in FIG. [Figure 20] FIG. 20 is a block diagram of a second encoding unit according to the first embodiment. [Figure 21] FIG. 21 is a diagram illustrating a configuration of a second decoding unit according to the first embodiment. As shown in FIG. [Figure 22] FIG. 22 is a block diagram of a second decoding unit according to the first embodiment. [Figure 23] FIG. 23 is a diagram showing a protocol stack related to PCC encoded data according to the first embodiment. [Figure 24] FIG. 24 is a diagram illustrating configurations of an encoding unit and a multiplexing unit according to the second embodiment. In FIG. [Diagram 25] FIG. 25 is a diagram illustrating an example of a structure of encoded data according to the second embodiment. In FIG. [Figure 26] FIG. 26 is a diagram showing an example of the structure of coded data and an NAL unit according to the second embodiment. [Figure 27] FIG. 27 is a diagram illustrating an example of the semantics of pcc_nal_unit_type according to the second embodiment. [Figure 28] FIG. 28 is a diagram showing an example of a transmission order of NAL units according to the second embodiment. [Figure 29] FIG. 29 is a flowchart of processing by the three-dimensional data encoding device according to the second embodiment. [Diagram 30] FIG. 30 is a flowchart of the process performed by the three-dimensional data decoding device according to the second embodiment. [Diagram 31] FIG. 31 is a flowchart of a multiplexing process according to the second embodiment. [Diagram 32] FIG. 32 is a flowchart of a demultiplexing process according to the second embodiment. [Diagram 33]FIG. 33 is a diagram illustrating a configuration of a distribution system according to the third embodiment. [Diagram 34] FIG. 34 is a diagram illustrating an example of a configuration of a bit stream of an encoded 3D map according to the third embodiment. [Diagram 35] FIG. 35 is a diagram for explaining an improvement effect of coding efficiency according to the third embodiment. In FIG. [Diagram 36] FIG. 36 is a flowchart of a process performed by the server according to the third embodiment. [Figure 37] FIG. 37 is a flowchart of processing by a client according to the third embodiment. [Figure 38] FIG. 38 is a diagram illustrating an example of the syntax of a submap according to the third embodiment. [Figure 39] FIG. 39 is a diagram illustrating a coding type switching process according to the third embodiment. In FIG. [Diagram 40] FIG. 40 is a diagram illustrating an example of the syntax of a submap according to the third embodiment. [Diagram 41] FIG. 41 is a flowchart of three-dimensional data encoding processing according to the third embodiment. [Diagram 42] FIG. 42 is a flowchart of three-dimensional data decoding processing according to the third embodiment. [Diagram 43] FIG. 43 is a diagram illustrating an example of an operation of a modification of the coding type switching process according to the third embodiment. In FIG. [Diagram 44] FIG. 44 is a diagram illustrating an example of an operation of a modification of the coding type switching process according to the third embodiment. In FIG. [Diagram 45] FIG. 45 is a diagram illustrating an example of an operation of a modification of the coding type switching process according to the third embodiment. In FIG. [Figure 46] FIG. 46 is a diagram illustrating an example of the operation of a variation of the difference value calculation process according to the third embodiment. In FIG. [Figure 47] FIG. 47 is a diagram illustrating an example of the operation of a variation of the difference value calculation process according to the third embodiment. [Figure 48]FIG. 48 is a diagram illustrating an example of the operation of a variation of the difference value calculation process according to the third embodiment. In FIG. [Figure 49] FIG. 49 is a diagram illustrating an example of the operation of a variation of the difference value calculation process according to the third embodiment. [Figure 50] FIG. 50 is a diagram illustrating an example of the syntax of a volume according to the third embodiment. [Figure 51] FIG. 51 is a diagram showing an example of a prediction tree used in a three-dimensional data encoding method according to the fourth embodiment. [Figure 52] FIG. 52 is a flowchart showing an example of a three-dimensional data encoding method according to the fourth embodiment. [Figure 53] FIG. 53 is a flowchart showing an example of a three-dimensional data decoding method according to the fourth embodiment. [Figure 54] FIG. 54 is a diagram for explaining a method of generating a prediction tree according to the fourth embodiment. [Figure 55] FIG. 55 is a diagram illustrating a first example of a prediction mode according to the fourth embodiment. In FIG. [Figure 56] FIG. 56 is a diagram showing a second example of a table indicating predicted values ​​calculated in each prediction mode according to the fourth embodiment. [Figure 57] FIG. 57 is a diagram showing a specific example of a second example of a table indicating predicted values ​​calculated in each prediction mode according to the fourth embodiment. [Figure 58] FIG. 58 is a diagram showing a third example of a table indicating predicted values ​​calculated in each prediction mode according to the fourth embodiment. [Figure 59] FIG. 59 is a diagram showing a fourth example of a table indicating predicted values ​​calculated in each prediction mode according to the fourth embodiment. [Figure 60] FIG. 60 is a diagram showing a fifth example of a table indicating predicted values ​​calculated in each prediction mode according to the fourth embodiment. [Figure 61] FIG. 61 is a diagram showing a sixth example of a table indicating predicted values ​​calculated in each prediction mode according to the fourth embodiment. [Figure 62]FIG. 62 is a diagram showing a seventh example of a table indicating predicted values ​​calculated in each prediction mode according to the fourth embodiment. In FIG. [Figure 63] FIG. 63 is a diagram showing a first example of a binarization table when binarizing and encoding a prediction mode value according to the fourth embodiment. [Figure 64] FIG. 64 is a diagram showing a second example of a binarization table when binarizing and encoding a prediction mode value according to the fourth embodiment. [Figure 65] FIG. 65 is a diagram showing a third example of a binarization table when binarizing and encoding a prediction mode value according to the fourth embodiment. [Figure 66] FIG. 66 is a diagram for explaining an example of encoding binary data of the binarization table in the case of binarizing and encoding the prediction mode according to the fourth embodiment. In FIG. [Figure 67] FIG. 67 is a flowchart showing an example of encoding of prediction mode values ​​according to the fourth embodiment. [Figure 68] FIG. 68 is a flowchart showing an example of decoding of a prediction mode value according to the fourth embodiment. [Figure 69] FIG. 69 is a diagram showing another example of a table indicating predicted values ​​calculated in each prediction mode according to the fourth embodiment. In FIG. [Figure 70] FIG. 70 is a diagram illustrating an example of encoding binary data of the binarization table in the case where the prediction mode according to the fourth embodiment is binarized and encoded. [Figure 71] FIG. 71 is a flowchart showing another example of encoding of prediction mode values ​​according to the fourth embodiment. [Figure 72] FIG. 72 is a flowchart showing another example of decoding of prediction mode values ​​according to the fourth embodiment. [Figure 73] FIG. 73 is a flowchart showing an example of a process for determining whether or not to fix a prediction mode value in accordance with condition A during encoding according to the fourth embodiment. [Figure 74]FIG. 74 is a flowchart showing an example of a process for determining whether to set the prediction mode value to a fixed value or to decode the prediction mode value in accordance with condition A during decoding according to the fourth embodiment. [Figure 75] FIG. 75 is a diagram illustrating an example of the syntax of a header of position information according to the fourth embodiment. [Figure 76] FIG. 76 is a diagram illustrating an example of the syntax of the position information according to the fourth embodiment. [Figure 77] FIG. 77 is a diagram showing another example of the syntax of the position information according to the fourth embodiment. [Figure 78] FIG. 78 is a diagram showing an example of a prediction tree used in a three-dimensional data encoding method according to the fifth embodiment. [Figure 79] FIG. 79 is a diagram showing another example of the syntax of the position information according to the fifth embodiment. [Figure 80] FIG. 80 is a diagram showing an example of a configuration of a prediction tree commonly used for encoding position information and attribute information according to the fifth embodiment. [Figure 81] FIG. 81 is a flowchart showing an example of a three-dimensional data encoding method according to the modification of the fifth embodiment. [Figure 82] FIG. 82 is a flowchart showing an example of a three-dimensional data decoding method according to the modification of the fifth embodiment. [Figure 83] FIG. 83 is a diagram illustrating an example of the syntax of a header of attribute information according to the fifth embodiment. [Figure 84] FIG. 84 is a diagram showing another example of the syntax of the attribute information according to the fifth embodiment. [Figure 85] FIG. 85 is a diagram showing an example of the syntax of the position information and attribute information according to the fifth embodiment. [Figure 86] FIG. 86 is a flowchart of the process of the three-dimensional data encoding device according to the fourth and fifth embodiments. [Figure 87] FIG. 87 is a flowchart of the process of the three-dimensional data decoding device according to the fourth and fifth embodiments. [Figure 88]FIG. 88 is a diagram illustrating an example of point cloud data obtained by the LiDAR according to the sixth embodiment. [Figure 89] FIG. 89 is a block diagram showing a configuration of a three-dimensional data encoding device according to the sixth embodiment. [Figure 90] FIG. 90 is a block diagram showing a configuration of a three-dimensional data encoding device according to the sixth embodiment. [Figure 91] FIG. 91 is a top view of a point cloud of the LiDAR according to the sixth embodiment projected onto a plane. [Figure 92] FIG. 92 is a diagram showing parent-child relationships among a plurality of points according to the sixth embodiment. In FIG. [Figure 93] FIG. 93 is a top view of a point cloud according to the sixth embodiment. [Figure 94] FIG. 94 is a diagram showing a relationship between point groups according to the sixth embodiment. In FIG. [Figure 95] FIG. 95 is a top view of a point cloud of the LiDAR according to the sixth embodiment projected onto a plane. [Figure 96] FIG. 96 is a diagram showing the relationship between a plurality of points according to the sixth embodiment. [Figure 97] FIG. 97 is a diagram illustrating an example of virtual nodes in point cloud data according to the sixth embodiment. [Figure 98] FIG. 98 is a diagram illustrating an example of virtual nodes and prediction residuals according to the sixth embodiment. In FIG. [Figure 99] FIG. 99 is a diagram showing an example of a configuration of a difference value and a bit stream according to the sixth embodiment. [Figure 100] FIG. 100 is a diagram showing an example of point cloud data according to the sixth embodiment. [Figure 101] FIG. 101 is a diagram showing the reference relationships of the nodes according to the sixth embodiment. [Figure 102] FIG. 102 is a diagram illustrating an example of point cloud data and a prediction relationship according to the sixth embodiment. [Figure 103] FIG. 103 is a flowchart of three-dimensional data encoding processing according to the sixth embodiment. [Figure 104]FIG. 104 is a flowchart of three-dimensional data decoding processing according to the sixth embodiment. [Figure 105] FIG. 105 is a diagram illustrating an example of the syntax of the header 1 according to the sixth embodiment. [Figure 106] FIG. 106 is a diagram illustrating an example of the syntax of the header 2 according to the sixth embodiment. [Figure 107] FIG. 107 is a diagram showing an example of a configuration of a bit stream according to the sixth embodiment. [Figure 108] FIG. 108 is a diagram illustrating an example of the syntax of coded data of position information according to the sixth embodiment. In FIG. [Figure 109] FIG. 109 is a flowchart of a prediction tree generation process according to the sixth embodiment. [Figure 110] FIG. 110 is a flowchart of the prediction tree decoding process according to the sixth embodiment. [Figure 111] FIG. 111 is a diagram showing a reference relationship in the prediction process according to the sixth embodiment. In FIG. [Figure 112] FIG. 112 is a diagram showing examples of prediction modes according to the sixth embodiment. [Figure 113] FIG. 113 is a flowchart of three-dimensional data encoding processing according to the sixth embodiment. [Fig. 114] FIG. 114 is a flowchart of three-dimensional data decoding processing according to the sixth embodiment. [Fig. 115] FIG. 115 is a block diagram of a three-dimensional data creation device according to the seventh embodiment. [Fig. 116] FIG. 116 is a flowchart of a three-dimensional data creating method according to the seventh embodiment. [Fig. 117] FIG. 117 is a diagram showing a configuration of a system according to the seventh embodiment. [Figure 118] FIG. 118 is a block diagram of a client device according to the seventh embodiment. [Figure 119] FIG. 119 is a block diagram of a server according to the seventh embodiment. [Figure 120]FIG. 120 is a flowchart of three-dimensional data creation processing by a client device according to the seventh embodiment. [Figure 121] FIG. 121 is a flowchart of a sensor information transmission process by a client device according to the seventh embodiment. [Figure 122] FIG. 122 is a flowchart of three-dimensional data creation processing by the server according to the seventh embodiment. [Figure 123] FIG. 123 is a flowchart of a 3D map transmission process by the server according to the seventh embodiment. [Figure 124] FIG. 124 is a diagram showing a configuration of a modified example of the system according to the seventh embodiment. In FIG. [Fig. 125] FIG. 125 is a diagram showing configurations of a server and a client device according to the seventh embodiment. In FIG. [Fig. 126] FIG. 126 is a diagram showing configurations of a server and a client device according to the seventh embodiment. In FIG. [Figure 127] FIG. 127 is a flowchart of processing by a client device according to the seventh embodiment. [Figure 128] FIG. 128 is a diagram illustrating a configuration of a sensor information collecting system according to the seventh embodiment. As shown in FIG. [Figure 129] FIG. 129 is a diagram illustrating an example of a system according to the seventh embodiment. [Fig. 130] FIG. 130 is a diagram showing a modification of the system according to the seventh embodiment. [Fig. 131] FIG. 131 is a flowchart showing an example of application processing according to the seventh embodiment. [Fig. 132] FIG. 132 is a diagram showing the sensor ranges of various sensors according to the seventh embodiment. [Fig. 133] FIG. 133 is a diagram illustrating a configuration example of an autonomous driving system according to the seventh embodiment. [Fig. 134] FIG. 134 is a diagram showing an example of a bitstream configuration according to the seventh embodiment. [Fig. 135] FIG. 135 is a flowchart of the point group selection process according to the seventh embodiment. [Fig. 136] FIG. 136 is a diagram showing an example of a screen for the point group selection process according to the seventh embodiment. [Fig. 137] FIG. 137 is a diagram showing an example of a screen for the point group selection process according to the seventh embodiment. [Figure 138] FIG. 138 is a diagram showing an example of a screen for the point group selection process according to the seventh embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] A three-dimensional data encoding method according to one embodiment of the present disclosure generates a first predicted value of position information of a first three-dimensional point included in point cloud data obtained by a sensor using position information of a reference three-dimensional point included in the point cloud data and a scanning angle of the sensor, calculates a first difference value between the position information of the first three-dimensional point and the first predicted value, and generates a bit stream including the first difference value and information indicating the scanning angle.

[0014] According to this, the three-dimensional data encoding method can improve prediction accuracy by generating predicted values ​​using the scanning angle of the sensor that generated the point cloud data, thereby improving encoding efficiency.

[0015] For example, the position information of the first three-dimensional point may be expressed in xyz coordinates, and the scan angle may be a scan angle in an xy plane.

[0016] For example, the sensor may be a radar that performs rotational scanning for each of the scan angles.

[0017] For example, in generating the first predicted value, the first predicted value may be generated by rotating and shifting the position information of the reference three-dimensional point contained in the point cloud data that is immediately before the first three-dimensional point in scanning order by the scanning angle around a reference point corresponding to the position of the sensor.

[0018] For example, a second predicted value may be generated using a value obtained by multiplying a reference predicted value by n (n is an integer greater than or equal to 1), and a second difference value between the position information of the first three-dimensional point and the second predicted value may be calculated, and the bit stream may include the second difference value and information indicating n.

[0019] According to this, the three-dimensional data coding method can improve prediction accuracy, and therefore coding efficiency.

[0020] For example, the reference predicted value may correspond to a difference in position information between two reference 3D points that are included in the point cloud data and that precede the first 3D point in scanning order.

[0021] For example, the reference prediction value may be generated using the scan angle.

[0022] A three-dimensional data decoding method according to one embodiment of the present disclosure obtains from a bit stream a first difference value between position information of a first three-dimensional point included in point cloud data obtained by a sensor and a first predicted value, and information indicating a scanning angle of the sensor, generates the first predicted value using position information of a reference three-dimensional point included in the point cloud data and the scanning angle of the sensor, and calculates the position information of the first three-dimensional point by adding the first predicted value and the first difference value.

[0023] According to this, the three-dimensional data decoding method can improve prediction accuracy by generating a predicted value using the scanning angle of the sensor that generated the point cloud data, and therefore can improve coding efficiency.

[0024] For example, the position information of the first three-dimensional point may be expressed in xyz coordinates, and the scan angle may be a scan angle in an xy plane.

[0025] For example, the sensor may be a radar that performs rotational scanning for each of the scan angles.

[0026] For example, in generating the first predicted value, the first predicted value may be generated by rotating and moving position information of a reference 3D point contained in the point cloud data that is immediately before the first 3D point in scanning order by the scanning angle around a reference point corresponding to the position of the sensor.

[0027] For example, the three-dimensional data decoding method may further include acquiring from the bit stream a second difference value between position information of a second three-dimensional point included in the point cloud data and a second predicted value, and information indicating n (n is an integer greater than or equal to 1), generating the second predicted value using a value obtained by multiplying a reference predicted value by n, and calculating the position information of the second three-dimensional point by adding the second predicted value and the second difference value.

[0028] According to this, the three-dimensional data decoding method can improve prediction accuracy, and therefore coding efficiency.

[0029] For example, the reference predicted value may correspond to a difference in position information between two reference 3D points included in the point cloud data and preceding the second 3D point in scanning order.

[0030] For example, the reference prediction value may be generated using the scan angle.

[0031] In addition, a three-dimensional data encoding device according to one embodiment of the present disclosure includes a processor and a memory, and the processor uses the memory to generate a first predicted value of position information of a first three-dimensional point included in point cloud data obtained by a sensor using position information of a reference three-dimensional point included in the point cloud data and a scanning angle of the sensor, calculates a first difference value between the position information of the first three-dimensional point and the first predicted value, and generates a bit stream including the first difference value and information indicating the scanning angle.

[0032] According to this, the three-dimensional data encoding device can improve prediction accuracy by generating predicted values ​​using the scanning angle of the sensor that generated the point cloud data, thereby improving encoding efficiency.

[0033] In addition, a three-dimensional data decoding device according to one embodiment of the present disclosure includes a processor and a memory, and the processor uses the memory to acquire from a bit stream a first difference value between position information of a first three-dimensional point included in point cloud data obtained by a sensor and a first predicted value, and information indicating a scanning angle of the sensor, generate the first predicted value using position information of a reference three-dimensional point included in the point cloud data and the scanning angle of the sensor, and calculate the position information of the first three-dimensional point by adding the first predicted value and the first difference value.

[0034] According to this, the three-dimensional data decoding device can improve prediction accuracy by generating predicted values ​​using the scanning angle of the sensor that generated the point cloud data, thereby improving encoding efficiency.

[0035] These comprehensive or specific aspects may be realized as a system, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM, or may be realized as any combination of a system, a method, an integrated circuit, a computer program, and a recording medium.

[0036] Hereinafter, the embodiments will be described in detail with reference to the drawings. Note that each of the embodiments described below shows a specific example of the present disclosure. The numerical values, shapes, materials, components, arrangement and connection forms of the components, steps, and order of steps shown in the following embodiments are merely examples and are not intended to limit the present disclosure. In addition, among the components in the following embodiments, components that are not described in the independent claims will be described as optional components.

[0037] (Embodiment 1) When using the encoded data of a point cloud in an actual device or service, it is desirable to transmit and receive the necessary information according to the purpose in order to reduce the network bandwidth. However, until now, such a function has not existed in the encoding structure of three-dimensional data, and there has been no encoding method for this purpose.

[0038] In this embodiment, we describe a three-dimensional data encoding method and a three-dimensional data encoding device for providing the function of transmitting and receiving necessary information depending on the application in encoded data of a three-dimensional point cloud, as well as a three-dimensional data decoding method and a three-dimensional data decoding device for decoding the encoded data, a three-dimensional data multiplexing method for multiplexing the encoded data, and a three-dimensional data transmission method for transmitting the encoded data.

[0039] In particular, the first and second encoding methods are currently being considered as methods (encoding schemes) for encoding point cloud data; however, the structure of the encoded data and the method for storing the encoded data in a system format have not been defined, and as things stand, there is a problem that MUX processing (multiplexing) in the encoding unit, or transmission or storage, is not possible.

[0040] Furthermore, there has been no method to date that supports a format in which two codecs, the first encoding method and the second encoding method, are mixed, such as PCC (Point Cloud Compression).

[0041] In this embodiment, a structure of PCC encoded data in which two codecs, a first encoding method and a second encoding method, are mixed, and a method of storing the encoded data in a system format will be described.

[0042] First, the configuration of a three-dimensional data (point cloud data) encoding / decoding system according to this embodiment will be described. Fig. 1 is a diagram showing an example of the configuration of a three-dimensional data encoding / decoding system according to this embodiment. As shown in Fig. 1, the three-dimensional data encoding / decoding system includes a three-dimensional data encoding system 4601, a three-dimensional data decoding system 4602, a sensor terminal 4603, and an external connection unit 4604.

[0043] The three-dimensional data encoding system 4601 generates encoded data or multiplexed data by encoding point cloud data, which is three-dimensional data. The three-dimensional data encoding system 4601 may be a three-dimensional data encoding device realized by a single device, or may be a system realized by multiple devices. The three-dimensional data encoding device may include some of the multiple processing units included in the three-dimensional data encoding system 4601.

[0044] The three-dimensional data encoding system 4601 includes a point cloud data generation system 4611, a presentation unit 4612, an encoding unit 4613, a multiplexing unit 4614, an input / output unit 4615, and a control unit 4616. The point cloud data generation system 4611 includes a sensor information acquisition unit 4617 and a point cloud data generation unit 4618.

[0045] The sensor information acquisition unit 4617 acquires sensor information from the sensor terminal 4603, and outputs the sensor information to the point cloud data generation unit 4618. The point cloud data generation unit 4618 generates point cloud data from the sensor information, and outputs the point cloud data to the encoding unit 4613.

[0046] The presentation unit 4612 presents the sensor information or the point cloud data to the user. For example, the presentation unit 4612 displays information or an image based on the sensor information or the point cloud data.

[0047] The encoding unit 4613 encodes (compresses) the point cloud data, and outputs the obtained encoded data, control information obtained in the encoding process, and other additional information to the multiplexing unit 4614. The additional information includes, for example, sensor information.

[0048] The multiplexing unit 4614 generates multiplexed data by multiplexing the coded data, control information, and additional information input from the coding unit 4613. The format of the multiplexed data is, for example, a file format for storage, or a packet format for transmission.

[0049] The input / output unit 4615 (e.g., a communication unit or an interface) outputs the multiplexed data to the outside. Alternatively, the multiplexed data is stored in a storage unit such as an internal memory. The control unit 4616 (or an application execution unit) controls each processing unit. In other words, the control unit 4616 controls encoding, multiplexing, etc.

[0050] The sensor information may be input to the encoding unit 4613 or the multiplexing unit 4614. Furthermore, the input / output unit 4615 may output the point cloud data or the encoded data directly to the outside.

[0051] The transmission signal (multiplexed data) output from the three-dimensional data encoding system 4601 is input to the three-dimensional data decoding system 4602 via the external connection unit 4604.

[0052] The three-dimensional data decoding system 4602 generates point cloud data, which is three-dimensional data, by decoding the encoded data or multiplexed data. The three-dimensional data decoding system 4602 may be a three-dimensional data decoding device realized by a single device, or may be a system realized by multiple devices. The three-dimensional data decoding device may also include some of the multiple processing units included in the three-dimensional data decoding system 4602.

[0053] The three-dimensional data decoding system 4602 includes a sensor information acquisition unit 4621 , an input / output unit 4622 , a demultiplexing unit 4623 , a decoding unit 4624 , a presentation unit 4625 , a user interface 4626 , and a control unit 4627 .

[0054] The sensor information acquisition unit 4621 acquires sensor information from the sensor terminal 4603 .

[0055] The input / output unit 4622 acquires a transmission signal, decodes multiplexed data (file format or packets) from the transmission signal, and outputs the multiplexed data to the demultiplexer 4623.

[0056] The demultiplexer 4623 obtains the coded data, control information, and additional information from the multiplexed data, and outputs the coded data, control information, and additional information to the decoder 4624.

[0057] The decoding unit 4624 reconstructs the point cloud data by decoding the encoded data.

[0058] The presentation unit 4625 presents the point cloud data to the user. For example, the presentation unit 4625 displays information or an image based on the point cloud data. The user interface 4626 acquires an instruction based on a user's operation. The control unit 4627 (or the application execution unit) controls each processing unit. That is, the control unit 4627 controls demultiplexing, decoding, presentation, etc.

[0059] The input / output unit 4622 may obtain the point cloud data or the encoded data directly from the outside. The presentation unit 4625 may obtain additional information such as sensor information and present information based on the additional information. The presentation unit 4625 may present information based on a user instruction obtained by the user interface 4626.

[0060] The sensor terminal 4603 generates sensor information obtained by a sensor. The sensor terminal 4603 is a terminal equipped with a sensor or a camera, and examples of the sensor terminal include a moving body such as an automobile, a flying object such as an airplane, a mobile terminal, and a camera.

[0061] The sensor information that can be acquired by the sensor terminal 4603 is, for example, (1) the distance between the sensor terminal 4603 and an object, or the reflectance of the object, obtained from a LIDAR, millimeter wave radar, or an infrared sensor, and (2) the distance between a camera and an object, or the reflectance of the object, obtained from a plurality of monocular camera images or stereo camera images. The sensor information may also include the attitude, direction, gyro (angular velocity), position (GPS information or altitude), speed, acceleration, etc. of the sensor. The sensor information may also include temperature, air pressure, humidity, magnetism, etc.

[0062] The external connection unit 4604 is realized by an integrated circuit (LSI or IC), an external storage unit, communication with a cloud server via the Internet, broadcasting, or the like.

[0063] Next, the point cloud data will be described. Fig. 2 is a diagram showing the configuration of the point cloud data. Fig. 3 is a diagram showing an example of the configuration of a data file in which information on the point cloud data is written.

[0064] Point cloud data includes data on multiple points. The data on each point includes position information (three-dimensional coordinates) and attribute information for that position information. A collection of multiple points is called a point cloud. For example, a point cloud may represent the three-dimensional shape of an object.

[0065] Position information such as three-dimensional coordinates is sometimes called geometry. Data for each point may include attribute information of multiple attribute types. The attribute types may be, for example, color or reflectance.

[0066] One piece of attribute information may be associated with one piece of location information, or multiple pieces of attribute information having different attribute types may be associated with one piece of location information, or multiple pieces of attribute information of the same attribute type may be associated with one piece of location information.

[0067] The configuration example of the data file shown in FIG. 3 is an example in which position information and attribute information correspond one-to-one, and shows the position information and attribute information of N points that make up the point cloud data.

[0068] The position information is, for example, information on three axes, x, y, and z. The attribute information is, for example, RGB color information. A typical data file is a ply file.

[0069] Next, the types of point cloud data will be described. Fig. 4 is a diagram showing the types of point cloud data. As shown in Fig. 4, point cloud data includes static objects and dynamic objects.

[0070] A static object is 3D point cloud data at any time (a certain time). A dynamic object is 3D point cloud data that changes over time. Hereinafter, 3D point cloud data at a certain time will be referred to as a PCC frame, or a frame.

[0071] The object may be a point cloud whose area is restricted to a certain extent, such as ordinary video data, or a large-scale point cloud whose area is not restricted, such as map information.

[0072] Also, there may be point cloud data of various densities, and there may be sparse point cloud data and dense point cloud data.

[0073] The details of each processing unit will be described below. The sensor information is acquired by various methods such as a distance sensor such as a LIDAR or a range finder, a stereo camera, or a combination of multiple monocular cameras. The point cloud data generation unit 4618 generates point cloud data based on the sensor information acquired by the sensor information acquisition unit 4617. The point cloud data generation unit 4618 generates position information as point cloud data, and adds attribute information for the position information to the position information.

[0074] The point cloud data generating unit 4618 may process the point cloud data when generating the position information or adding the attribute information. For example, the point cloud data generating unit 4618 may reduce the amount of data by deleting point clouds with overlapping positions. In addition, the point cloud data generating unit 4618 may convert the position information (position shift, rotation, normalization, etc.) or render the attribute information.

[0075] In FIG. 1, the point cloud data generation system 4611 is included in the three-dimensional data encoding system 4601, but it may be provided independently outside the three-dimensional data encoding system 4601.

[0076] The encoding unit 4613 generates encoded data by encoding the point cloud data based on a predefined encoding method. There are two main types of encoding methods: the first is an encoding method using position information, and this encoding method will be referred to as the first encoding method hereinafter; the second is an encoding method using a video codec, and this encoding method will be referred to as the second encoding method hereinafter.

[0077] The decoding unit 4624 decodes the encoded data based on a predefined encoding method to decode the point group data.

[0078] The multiplexing unit 4614 generates multiplexed data by multiplexing the encoded data using an existing multiplexing method. The generated multiplexed data is transmitted or stored. In addition to the PCC encoded data, the multiplexing unit 4614 multiplexes other media such as video, audio, subtitles, applications, and files, or reference time information. The multiplexing unit 4614 may further multiplex attribute information related to sensor information or point cloud data.

[0079] Multiplexing methods or file formats include ISOBMFF, MPEG-DASH, which is an ISOBMFF-based transmission method, MMT, MPEG-2 TS Systems, and RMP.

[0080] The demultiplexer 4623 extracts the PCC encoded data, other media, time information, and the like from the multiplexed data.

[0081] The input / output unit 4615 transmits the multiplexed data using a method suited to a transmission medium or a storage medium, such as broadcasting or communication. The input / output unit 4615 may communicate with other devices via the Internet, or may communicate with a storage unit such as a cloud server.

[0082] The communication protocol used may be http, ftp, TCP, UDP, etc. A PULL type communication method or a PUSH type communication method may be used.

[0083] Either wired transmission or wireless transmission may be used. For wired transmission, Ethernet (registered trademark), USB, RS-232C, HDMI (registered trademark), coaxial cable, etc. are used. For wireless transmission, wireless LAN, Wi-Fi (registered trademark), Bluetooth (registered trademark), millimeter wave, etc. are used.

[0084] As a broadcasting system, for example, DVB-T2, DVB-S2, DVB-C2, ATSC3.0, or ISDB-S3 is used.

[0085] Fig. 5 is a diagram showing a configuration of a first encoding unit 4630 which is an example of the encoding unit 4613 that performs encoding using the first encoding method. Fig. 6 is a block diagram of the first encoding unit 4630. The first encoding unit 4630 generates encoded data (encoded stream) by encoding point cloud data using the first encoding method. The first encoding unit 4630 includes a position information encoding unit 4631, an attribute information encoding unit 4632, an additional information encoding unit 4633, and a multiplexing unit 4634.

[0086] The first encoding unit 4630 has a feature of performing encoding in consideration of a three-dimensional structure. Also, the first encoding unit 4630 has a feature that the attribute information encoding unit 4632 performs encoding using information obtained from the position information encoding unit 4631. The first encoding method is also called GPCC (Geometry based PCC).

[0087] The point cloud data is PCC point cloud data such as a PLY file, or PCC point cloud data generated from sensor information, and includes position information (Position), attribute information (Attribute), and other additional information (MetaData). The position information is input to a position information encoding unit 4631, the attribute information is input to an attribute information encoding unit 4632, and the additional information is input to an additional information encoding unit 4633.

[0088] The position information encoding unit 4631 generates encoded position information (Compressed Geometry) that is encoded data by encoding the position information. For example, the position information encoding unit 4631 encodes the position information using an N-ary tree structure such as an octet tree. Specifically, in the octet tree, the target space is divided into eight nodes (subspaces), and 8-bit information (occupancy code) indicating whether or not a point cloud is included in each node is generated. In addition, the node that includes the point cloud is further divided into eight nodes, and 8-bit information indicating whether or not a point cloud is included in each of the eight nodes is generated. This process is repeated until the number of point clouds included in a predetermined hierarchy or node is equal to or less than a threshold value.

[0089] The attribute information encoding unit 4632 generates encoded attribute information (Compressed Attribute) that is encoded data by encoding using the configuration information generated by the position information encoding unit 4631. For example, the attribute information encoding unit 4632 determines a reference point (reference node) to be referenced in encoding a target point (target node) to be processed based on the octree structure generated by the position information encoding unit 4631. For example, the attribute information encoding unit 4632 refers to a node, among peripheral nodes or adjacent nodes, whose parent node in the octree is the same as that of the target node. Note that the method of determining the reference relationship is not limited to this.

[0090] Furthermore, the encoding process of the attribute information may include at least one of a quantization process, a prediction process, and an arithmetic coding process. In this case, the reference means using a reference node to calculate a predicted value of the attribute information, or using a state of the reference node (e.g., occupancy information indicating whether or not a point group is included in the reference node) to determine a parameter of the encoding. For example, the parameter of the encoding is a quantization parameter in the quantization process, or a context in the arithmetic coding.

[0091] The additional information encoding unit 4633 generates encoded additional information (Compressed MetaData) that is encoded data by encoding compressible data from the additional information.

[0092] The multiplexing unit 4634 multiplexes the encoding position information, the encoding attribute information, the encoding additional information, and other additional information to generate an encoded stream (Compressed Stream) that is encoded data. The generated encoded stream is output to a processing unit of a system layer (not shown).

[0093] Next, a first decoding unit 4640 which is an example of the decoding unit 4624 that performs decoding of the first encoding method will be described. FIG. 7 is a diagram showing a configuration of the first decoding unit 4640. FIG. 8 is a block diagram of the first decoding unit 4640. The first decoding unit 4640 generates point group data by decoding, by the first encoding method, encoded data (encoded stream) encoded by the first encoding method. The first decoding unit 4640 includes a demultiplexing unit 4641, a position information decoding unit 4642, an attribute information decoding unit 4643, and an additional information decoding unit 4644.

[0094] A coded stream (compressed stream), which is coded data, is input to the first decoding unit 4640 from a processing unit in a system layer (not shown).

[0095] The demultiplexer 4641 separates the encoded position information (Compressed Geometry), the encoded attribute information (Compressed Attribute), the encoded additional information (Compressed MetaData), and other additional information from the encoded data.

[0096] The position information decoding unit 4642 generates position information by decoding the encoded position information. For example, the position information decoding unit 4642 restores the position information of a point group represented by three-dimensional coordinates from the encoded position information represented by an N-ary tree structure such as an octree.

[0097] The attribute information decoding unit 4643 decodes the encoded attribute information based on the configuration information generated by the position information decoding unit 4642. For example, the attribute information decoding unit 4643 determines a reference point (reference node) to be referenced in decoding a target point (target node) to be processed based on the octree structure obtained by the position information decoding unit 4642. For example, the attribute information decoding unit 4643 refers to a node, among peripheral nodes or adjacent nodes, whose parent node in the octree is the same as that of the target node. Note that the method of determining the reference relationship is not limited to this.

[0098] In addition, the decoding process of the attribute information may include at least one of an inverse quantization process, a prediction process, and an arithmetic decoding process. In this case, the reference means using a reference node to calculate a predicted value of the attribute information, or using a state of the reference node (e.g., occupancy information indicating whether or not a point group is included in the reference node) to determine a parameter of the decoding. For example, the parameter of the decoding is a quantization parameter in the inverse quantization process, or a context in the arithmetic decoding.

[0099] The additional information decoding unit 4644 generates additional information by decoding the encoded additional information. The first decoding unit 4640 uses the additional information required for decoding the position information and attribute information during decoding, and outputs the additional information required for an application to the outside.

[0100] Next, a configuration example of the position information encoding unit will be described. Fig. 9 is a block diagram of the position information encoding unit 2700 according to this embodiment. The position information encoding unit 2700 includes an octree generating unit 2701, a geometric information calculating unit 2702, a coding table selecting unit 2703, and an entropy encoding unit 2704.

[0101] The octree generating unit 2701 generates, for example, an octree from the input position information, and generates an occupancy code of each node of the octree. The geometric information calculating unit 2702 obtains information indicating whether an adjacent node of the target node is an occupied node. For example, the geometric information calculating unit 2702 calculates occupancy information of the adjacent node (information indicating whether the adjacent node is an occupied node) from the occupancy code of the parent node to which the target node belongs. The geometric information calculating unit 2702 may also store encoded nodes in a list and search for adjacent nodes from the list. The geometric information calculating unit 2702 may also switch adjacent nodes depending on the position of the target node in the parent node.

[0102] The coding table selection unit 2703 selects a coding table to be used for entropy coding of the target node, using the occupancy information of the adjacent nodes calculated by the geometric information calculation unit 2702. For example, the coding table selection unit 2703 may generate a bit string using the occupancy information of the adjacent nodes, and select a coding table for an index number generated from the bit string.

[0103] The entropy coding unit 2704 generates the coding position information and metadata by performing entropy coding on the occupancy code of the target node using the coding table of the selected index number. The entropy coding unit 2704 may add information indicating the selected coding table to the coding position information.

[0104] The octree representation and the scanning order of position information will be described below. Position information (position data) is converted (octreeized) into an octree structure and then encoded. The octree structure is composed of nodes and leaves. Each node has eight nodes or leaves, and each leaf has voxel (VXL) information. Fig. 10 is a diagram showing an example of the structure of position information including multiple voxels. Fig. 11 is a diagram showing an example of the position information shown in Fig. 10 converted into an octree structure. Here, among the leaves shown in Fig. 11, leaves 1, 2, and 3 respectively represent voxels VXL1, VXL2, and VXL3 shown in Fig. 10, and represent a VXL including a point cloud (hereinafter, effective VXL).

[0105] Specifically, node 1 corresponds to the whole space including the position information in FIG. 10. The whole space corresponding to node 1 is divided into eight nodes, and among the eight nodes, the node including the valid VXL is further divided into eight nodes or leaves, and this process is repeated for the number of levels of the tree structure. Here, each node corresponds to a subspace, and has information (occupancy code) indicating at what position the next node or leaf will be located after division as node information. Also, the block at the bottom level is set as a leaf, and the number of point clouds included in the leaf, etc., is held as leaf information.

[0106] Next, a configuration example of the position information decoding unit will be described. Fig. 12 is a block diagram of the position information decoding unit 2710 according to this embodiment. The position information decoding unit 2710 includes an octree generating unit 2711, a geometric information calculating unit 2712, a coding table selecting unit 2713, and an entropy decoding unit 2714.

[0107] The octree generating unit 2711 generates an octree of a certain space (node) using header information or metadata of a bit stream. For example, the octree generating unit 2711 generates a large space (root node) using the size of the x-axis, y-axis, and z-axis directions of a certain space added to the header information, and generates an octree by dividing the space into two in the x-axis, y-axis, and z-axis directions to generate eight small spaces A (nodes A0 to A7). In addition, nodes A0 to A7 are set in order as target nodes.

[0108] The geometric information calculation unit 2712 obtains occupancy information indicating whether an adjacent node of the target node is an occupancy node. For example, the geometric information calculation unit 2712 calculates the occupancy information of the adjacent node from the occupancy code of the parent node to which the target node belongs. The geometric information calculation unit 2712 may also store decoded nodes in a list and search for adjacent nodes from the list. The geometric information calculation unit 2712 may also switch adjacent nodes depending on the position of the target node in the parent node.

[0109] The coding table selection unit 2713 selects a coding table (decoding table) to be used for entropy decoding of the target node using the occupancy information of the adjacent node calculated by the geometric information calculation unit 2712. For example, the coding table selection unit 2713 may generate a bit string using the occupancy information of the adjacent node, and select a coding table for an index number generated from the bit string.

[0110] The entropy decoding unit 2714 generates position information by entropy decoding the occupancy code of the target node using the selected coding table. Note that the entropy decoding unit 2714 may obtain information on the selected coding table by decoding from the bit stream, and entropy decode the occupancy code of the target node using the coding table indicated by the information.

[0111] The configurations of the attribute information encoding unit and the attribute information decoding unit will be described below. Fig. 13 is a block diagram showing an example of the configuration of the attribute information encoding unit A100. The attribute information encoding unit may include a plurality of encoding units that execute different encoding methods. For example, the attribute information encoding unit may switch between the following two methods depending on the use case.

[0112] The attribute information encoding unit A100 includes an LoD attribute information encoding unit A101 and a conversion attribute information encoding unit A102. The LoD attribute information encoding unit A101 classifies each 3D point into a plurality of layers using the position information of the 3D point, predicts the attribute information of the 3D point belonging to each layer, and encodes the prediction residual. Here, each classified layer is called LoD (Level of Detail).

[0113] The transformed attribute information encoding unit A102 encodes the attribute information using RAHT (Region Adaptive Hierarchical Transform). Specifically, the transformed attribute information encoding unit A102 applies RAHT or Haar transform to each piece of attribute information based on the position information of the three-dimensional point to generate high-frequency components and low-frequency components of each layer, and encodes the values ​​using quantization, entropy coding, or the like.

[0114] 14 is a block diagram showing a configuration example of the attribute information decoding unit A110. The attribute information decoding unit may include multiple decoding units that execute different decoding methods. For example, the attribute information decoding unit may switch between the following two methods for decoding based on information included in the header or metadata.

[0115] The attribute information decoding unit A110 includes a LoD attribute information decoding unit A111 and a converted attribute information decoding unit A112. The LoD attribute information decoding unit A111 classifies each 3D point into a plurality of hierarchical levels using the position information of the 3D point, and decodes the attribute value while predicting the attribute information of the 3D point belonging to each hierarchical level.

[0116] The transformed attribute information decoding unit A112 decodes the attribute information using RAHT (Region Adaptive Hierarchical Transform). Specifically, the transformed attribute information decoding unit A112 decodes the attribute value by applying inverse RAHT or inverse Haar transform to high-frequency components and low-frequency components of each attribute value based on the position information of the three-dimensional point.

[0117] FIG. 15 is a block diagram showing a configuration of an attribute information encoding unit 3140 which is an example of the LoD attribute information encoding unit A101.

[0118] The attribute information encoding unit 3140 includes a LoD generation unit 3141, a surrounding search unit 3142, a prediction unit 3143, a prediction residual calculation unit 3144, a quantization unit 3145, an arithmetic encoding unit 3146, an inverse quantization unit 3147, a decoded value generation unit 3148, and a memory 3149.

[0119] The LoD generation unit 3141 generates LoD using the position information of the three-dimensional points.

[0120] The surrounding search unit 3142 searches for nearby 3D points adjacent to each 3D point, using the LoD generation result by the LoD generation unit 3141 and distance information indicating the distance between each 3D point.

[0121] The prediction unit 3143 generates a predicted value of the attribute information of the target 3D point to be encoded.

[0122] The prediction residual calculation unit 3144 calculates (generates) a prediction residual of the predicted value of the attribute information generated by the prediction unit 3143.

[0123] The quantization unit 3145 quantizes the prediction residual of the attribute information calculated by the prediction residual calculation unit 3144 .

[0124] The arithmetic coding unit 3146 arithmetically codes the prediction residuals after being quantized by the quantization unit 3145. The arithmetic coding unit 3146 outputs a bit stream including the arithmetically coded prediction residuals to, for example, a three-dimensional data decoding device.

[0125] Note that the prediction residual may be binarized by, for example, the quantization unit 3145 before being arithmetically coded by the arithmetic coding unit 3146.

[0126] Also, for example, the arithmetic coding unit 3146 may initialize a coding table used for arithmetic coding before arithmetic coding. The arithmetic coding unit 3146 may initialize a coding table used for arithmetic coding for each layer. Also, the arithmetic coding unit 3146 may output information indicating the position of the layer for which the coding table has been initialized, by including it in the bitstream.

[0127] The inverse quantization unit 3147 inverse quantizes the prediction residual after being quantized by the quantization unit 3145 .

[0128] The decoded value generation unit 3148 generates a decoded value by adding the predicted value of the attribute information generated by the prediction unit 3143 and the prediction residual after inverse quantization by the inverse quantization unit 3147.

[0129] The memory 3149 is a memory that stores the decoded values ​​of the attribute information of each 3D point decoded by the decoded value generation unit 3148. For example, when generating a predicted value of a 3D point that has not yet been encoded, the prediction unit 3143 generates the predicted value by using the decoded value of the attribute information of each 3D point stored in the memory 3149.

[0130] 16 is a block diagram of an attribute information encoding unit 6600 which is an example of the transformed attribute information encoding unit A102. The attribute information encoding unit 6600 includes a sorting unit 6601, a Haar transform unit 6602, a quantization unit 6603, an inverse quantization unit 6604, an inverse Haar transform unit 6605, a memory 6606, and an arithmetic encoding unit 6607.

[0131] The sorting unit 6601 generates a Morton code using the position information of the 3D points, and sorts the multiple 3D points in Morton code order. The Haar transform unit 6602 generates coding coefficients by applying a Haar transform to the attribute information. The quantization unit 6603 quantizes the coding coefficients of the attribute information.

[0132] The inverse quantization unit 6604 inversely quantizes the quantized coding coefficients. The inverse Haar transform unit 6605 applies an inverse Haar transform to the coding coefficients. The memory 6606 stores values ​​of attribute information of multiple decoded 3D points. For example, the attribute information of the decoded 3D points stored in the memory 6606 may be used for predicting uncoded 3D points.

[0133] The arithmetic coding unit 6607 calculates ZeroCnt from the coding coefficients after quantization, and arithmetically codes the ZeroCnt. Furthermore, the arithmetic coding unit 6607 arithmetically codes the non-zero coding coefficients after quantization. The arithmetic coding unit 6607 may binarize the coding coefficients before arithmetic coding. Furthermore, the arithmetic coding unit 6607 may generate and code various header information.

[0134] FIG. 17 is a block diagram showing a configuration of an attribute information decoding unit 3150 which is an example of the LoD attribute information decoding unit A111.

[0135] The attribute information decoding unit 3150 includes an LoD generation unit 3151 , a surrounding search unit 3152 , a prediction unit 3153 , an arithmetic decoding unit 3154 , an inverse quantization unit 3155 , a decoded value generation unit 3156 , and a memory 3157 .

[0136] The LoD generation unit 3151 generates LoD using position information of the 3D points decoded by a position information decoding unit (not shown in FIG. 17).

[0137] The surrounding search unit 3152 searches for nearby 3D points adjacent to each 3D point, using the LoD generation result by the LoD generation unit 3151 and distance information indicating the distance between each 3D point.

[0138] The prediction unit 3153 generates a predicted value of the attribute information of the target 3D point to be decoded.

[0139] The arithmetic decoding unit 3154 arithmetically decodes the prediction residual in the bitstream acquired from the attribute information encoding unit 3140 shown in FIG. 15. The arithmetic decoding unit 3154 may initialize a decoding table used for arithmetic decoding. The arithmetic decoding unit 3154 initializes a decoding table used for arithmetic decoding for a layer on which the arithmetic encoding unit 3146 shown in FIG. 15 has performed an encoding process. The arithmetic decoding unit 3154 may initialize a decoding table used for arithmetic decoding for each layer. The arithmetic decoding unit 3154 may initialize the decoding table based on information included in the bitstream and indicating the position of the layer for which the encoding table has been initialized.

[0140] The inverse quantization unit 3155 inverse quantizes the prediction residual arithmetically decoded by the arithmetic decoding unit 3154 .

[0141] The decoded value generation unit 3156 generates a decoded value by adding the predicted value generated by the prediction unit 3153 and the prediction residual after inverse quantization by the inverse quantization unit 3155. The decoded value generation unit 3156 outputs the decoded attribute information data to another device.

[0142] The memory 3157 is a memory that stores the decoded values ​​of the attribute information of each 3D point decoded by the decoded value generation unit 3156. For example, when generating a predicted value of a 3D point that has not yet been decoded, the prediction unit 3153 generates the predicted value by using the decoded value of the attribute information of each 3D point stored in the memory 3157.

[0143] 18 is a block diagram of an attribute information decoding unit 6610 which is an example of the transformed attribute information decoding unit A112. The attribute information decoding unit 6610 includes an arithmetic decoding unit 6611, an inverse quantization unit 6612, an inverse Haar transform unit 6613, and a memory 6614.

[0144] The arithmetic decoding unit 6611 arithmetically decodes the ZeroCnt and the coding coefficients included in the bit stream. Note that the arithmetic decoding unit 6611 may also decode various types of header information.

[0145] The inverse quantization unit 6612 inverse quantizes the arithmetically decoded coding coefficients. The inverse Haar transform unit 6613 applies an inverse Haar transform to the coding coefficients after the inverse quantization. The memory 6614 stores values ​​of attribute information of multiple decoded 3D points. For example, the attribute information of the decoded 3D points stored in the memory 6614 may be used to predict undecoded 3D points.

[0146] Next, a second encoding unit 4650, which is an example of the encoding unit 4613 that performs encoding by the second encoding method, will be described. Fig. 19 is a diagram showing a configuration of the second encoding unit 4650. Fig. 20 is a block diagram of the second encoding unit 4650.

[0147] The second encoding unit 4650 generates encoded data (encoded stream) by encoding the point cloud data by a second encoding method. The second encoding unit 4650 includes an additional information generating unit 4651, a position image generating unit 4652, an attribute image generating unit 4653, a video encoding unit 4654, an additional information encoding unit 4655, and a multiplexing unit 4656.

[0148] The second encoding unit 4650 has a feature of generating a position image and an attribute image by projecting a three-dimensional structure onto a two-dimensional image, and encoding the generated position image and attribute image using an existing video encoding method. The second encoding method is also called VPCC (Video based PCC).

[0149] The point cloud data is PCC point cloud data such as a PLY file, or PCC point cloud data generated from sensor information, and includes position information (Position), attribute information (Attribute), and other additional information (MetaData).

[0150] The additional information generating unit 4651 generates map information of a plurality of two-dimensional images by projecting a three-dimensional structure onto the two-dimensional images.

[0151] The position image generating unit 4652 generates a position image (Geometry Image) based on the position information and the map information generated by the additional information generating unit 4651. This position image is, for example, a distance image in which distance (Depth) is indicated as a pixel value. Note that this distance image may be an image in which a plurality of point clouds are viewed from one viewpoint (an image in which a plurality of point clouds are projected onto one two-dimensional plane), a plurality of images in which a plurality of point clouds are viewed from multiple viewpoints, or a single image in which these multiple images are integrated.

[0152] The attribute image generating unit 4653 generates an attribute image based on the attribute information and the map information generated by the additional information generating unit 4651. This attribute image is, for example, an image in which the attribute information (for example, color (RGB)) is indicated as pixel values. Note that this image may be an image in which multiple point clouds are viewed from one viewpoint (an image in which multiple point clouds are projected onto one two-dimensional plane), or multiple images in which multiple point clouds are viewed from multiple viewpoints, or a single image in which these multiple images are integrated.

[0153] The video encoding unit 4654 generates an encoded position image (Compressed Geometry Image) and an encoded attribute image (Compressed Attribute Image), which are encoded data, by encoding the position image and the attribute image using a video encoding method. Note that any known encoding method may be used as the video encoding method. For example, the video encoding method is AVC, HEVC, or the like.

[0154] The additional information encoding unit 4655 generates encoded additional information (Compressed MetaData) by encoding the additional information and map information included in the point cloud data.

[0155] The multiplexing unit 4656 multiplexes the encoding position image, the encoding attribute image, the encoding additional information, and other additional information to generate an encoded stream (Compressed Stream) that is encoded data. The generated encoded stream is output to a processing unit of a system layer (not shown).

[0156] Next, a second decoding unit 4660 which is an example of the decoding unit 4624 that performs decoding of the second encoding method will be described. FIG. 21 is a diagram showing a configuration of the second decoding unit 4660. FIG. 22 is a block diagram of the second decoding unit 4660. The second decoding unit 4660 generates point group data by decoding, by the second encoding method, encoded data (encoded stream) encoded by the second encoding method. The second decoding unit 4660 includes a demultiplexing unit 4661, a video decoding unit 4662, an additional information decoding unit 4663, a position information generating unit 4664, and an attribute information generating unit 4665.

[0157] A compressed stream, which is encoded data, is input to the second decoding unit 4660 from a processing unit in a system layer (not shown).

[0158] The demultiplexer 4661 separates the encoded position image (Compressed Geometry Image), the encoded attribute image (Compressed Attribute Image), the encoded additional information (Compressed MetaData), and other additional information from the encoded data.

[0159] The video decoding unit 4662 generates a position image and an attribute image by decoding the encoded position image and the encoded attribute image using a video encoding method. Note that any known encoding method may be used as the video encoding method. For example, the video encoding method is AVC or HEVC.

[0160] The additional information decoding unit 4663 decodes the encoded additional information to generate additional information including map information and the like.

[0161] The position information generating unit 4664 generates position information using the position image and map information. The attribute information generating unit 4665 generates attribute information using the attribute image and map information.

[0162] The second decoding unit 4660 uses the additional information necessary for decoding during decoding, and outputs the additional information necessary for an application to the outside.

[0163] Problems with the PCC encoding method will be described below. Fig. 23 is a diagram showing a protocol stack related to PCC encoded data. Fig. 23 shows an example in which other media data such as video (e.g., HEVC) or audio is multiplexed with PCC encoded data and transmitted or stored.

[0164] The multiplexing method and file format have the function of multiplexing various encoded data and transmitting or storing them. In order to transmit or store the encoded data, the encoded data must be converted into the format of the multiplexing method. For example, HEVC specifies a technology for storing encoded data in a data structure called a NAL unit and storing the NAL unit in ISOBMFF.

[0165] Meanwhile, a first encoding method (Codec1) and a second encoding method (Codec2) are currently being considered as methods for encoding point cloud data. However, the structure of the encoded data and the method for storing the encoded data in a system format have not been defined, which poses the problem that MUX processing (multiplexing) in the encoding unit, transmission, and storage cannot be performed as is.

[0166] In the following description, unless a specific encoding method is specified, it refers to either the first encoding method or the second encoding method.

[0167] (Embodiment 2) In this embodiment, the types of encoded data (position information (Geometry), attribute information (Attribute), additional information (Metadata)) generated by the above-mentioned first encoding unit 4630 or second encoding unit 4650, a generation method of the additional information (metadata), and multiplexing processing in the multiplexing unit will be described. Note that the additional information (metadata) may also be referred to as a parameter set or control information.

[0168] In this embodiment, the dynamic object (three-dimensional point cloud data that changes over time) described in Figure 4 will be used as an example, but a similar method may also be used in the case of a static object (three-dimensional point cloud data at any time).

[0169] 24 is a diagram showing configurations of an encoding unit 4801 and a multiplexing unit 4802 included in the three-dimensional data encoding device according to this embodiment. The encoding unit 4801 corresponds to, for example, the first encoding unit 4630 or the second encoding unit 4650 described above. The multiplexing unit 4802 corresponds to the multiplexing unit 4634 or 4656 described above.

[0170] The encoding unit 4801 encodes point cloud data of multiple PCC (Point Cloud Compression) frames, and generates encoded data (Multiple Compressed Data) of multiple pieces of position information, attribute information, and additional information.

[0171] The multiplexing unit 4802 converts data of multiple data types (position information, attribute information, and additional information) into NAL units, thereby converting the data into a data structure that takes into consideration data access in the decoding device.

[0172] 25 is a diagram showing an example of the structure of coded data generated by the coding unit 4801. The arrows in the diagram indicate dependencies related to the decoding of coded data, with the source of the arrow depending on the data at the tip of the arrow. In other words, the decoding device decodes the data at the tip of the arrow, and uses the decoded data to decode the data at the tip of the arrow. In other words, dependency means that the data at the dependency destination is referenced (used) in the processing (encoding, decoding, etc.) of the data at the dependency destination.

[0173] First, the process of generating encoded data of position information will be described. The encoding unit 4801 generates encoded position data (compressed geometry data) for each frame by encoding the position information of each frame. The encoded position data is represented as G(i), where i indicates the frame number, the time of the frame, etc.

[0174] The encoding unit 4801 also generates a position parameter set (GPS(i)) corresponding to each frame. The position parameter set includes parameters that can be used to decode the encoded position data. The encoded position data for each frame depends on the corresponding position parameter set.

[0175] Moreover, encoded position data consisting of multiple frames is defined as a position sequence (Geometry Sequence). The encoding unit 4801 generates a position sequence parameter set (Geometry Sequence PS: also written as position SPS) that stores parameters commonly used in decoding processes for multiple frames in the position sequence. The position sequence depends on the position SPS.

[0176] Next, a process for generating encoded data of attribute information will be described. Encoding unit 4801 generates encoded attribute data (Compressed Attribute Data) for each frame by encoding the attribute information of each frame. The encoded attribute data is represented as A(i). FIG. 25 shows an example in which attribute X and attribute Y exist, and the encoded attribute data of attribute X is represented as AX(i) and the encoded attribute data of attribute Y is represented as AY(i).

[0177] The encoding unit 4801 also generates an attribute parameter set (APS(i)) corresponding to each frame. The attribute parameter set for attribute X is represented as AXPS(i), and the attribute parameter set for attribute Y is represented as AYPS(i). The attribute parameter set includes parameters that can be used to decode the encoded attribute information. The encoded attribute data depends on the corresponding attribute parameter set.

[0178] Moreover, encoded attribute data consisting of multiple frames is defined as an attribute sequence. The encoding unit 4801 generates an attribute sequence parameter set (Attribute Sequence PS: also referred to as attribute SPS) that stores parameters commonly used in decoding processes for multiple frames in the attribute sequence. The attribute sequence depends on the attribute SPS.

[0179] Furthermore, in the first encoding method, the encoded attribute data depends on the encoded position data.

[0180] 25 shows an example in which two types of attribute information (attribute X and attribute Y) exist. When there are two types of attribute information, for example, two encoding units generate respective data and metadata. Also, for example, an attribute sequence is defined for each type of attribute information, and an attribute SPS is generated for each type of attribute information.

[0181] In addition, while FIG. 25 shows an example in which there is one type of position information and two types of attribute information, the present invention is not limited to this, and there may be one type of attribute information, or three or more types. In this case, the encoded data can be generated in a similar manner. In addition, in the case of point cloud data that does not have attribute information, the attribute information may not be necessary. In this case, the encoding unit 4801 does not need to generate a parameter set related to the attribute information.

[0182] Next, a process of generating additional information (metadata) will be described. The encoding unit 4801 generates a PCC stream PS (also written as stream PS), which is a parameter set for the entire PCC stream. The encoding unit 4801 stores parameters that can be commonly used in decoding processes for one or more position sequences and one or more attribute sequences in the stream PS. For example, the stream PS includes identification information indicating the codec of the point cloud data, information indicating the algorithm used for encoding, and the like. The position sequence and the attribute sequence depend on the stream PS.

[0183] Next, an access unit and a GOF will be described. In this embodiment, the concept of an access unit (AU) and a group of frames (GOF) are newly introduced.

[0184] An access unit is a basic unit for accessing data during decoding, and is composed of one or more pieces of data and one or more pieces of metadata. For example, an access unit is composed of position information at the same time and one or more pieces of attribute information. A GOF is a random access unit, and is composed of one or more access units.

[0185] The encoding unit 4801 generates an access unit header (AU Header) as identification information indicating the beginning of an access unit. The encoding unit 4801 stores parameters related to the access unit in the access unit header. For example, the access unit header includes the configuration or information of the encoded data included in the access unit. The access unit header also includes parameters commonly used for data included in the access unit, such as parameters related to the decoding of the encoded data.

[0186] Alternatively, instead of an access unit header, the encoding unit 4801 may generate an access unit delimiter that does not include parameters related to the access unit. This access unit delimiter is used as identification information indicating the beginning of the access unit. The decoding device identifies the beginning of the access unit by detecting the access unit header or the access unit delimiter.

[0187] Next, generation of identification information for the start of a GOF will be described. The encoding unit 4801 generates a GOF header as identification information indicating the start of a GOF. The encoding unit 4801 stores parameters related to the GOF in the GOF header. For example, the GOF header includes the configuration or information of the encoded data included in the GOF. The GOF header also includes parameters commonly used for data included in the GOF, such as parameters related to the decoding of the encoded data.

[0188] Instead of a GOF header, the encoding unit 4801 may generate a GOF delimiter that does not include parameters related to the GOF. This GOF delimiter is used as identification information indicating the beginning of the GOF. The decoding device identifies the beginning of the GOF by detecting the GOF header or the GOF delimiter.

[0189] In PCC encoded data, for example, an access unit is defined as a PCC frame unit, and a decoding device accesses a PCC frame based on identification information at the beginning of the access unit.

[0190] Also, for example, GOF is defined as one random access unit. The decoding device accesses the random access unit based on the identification information at the beginning of the GOF. For example, if PCC frames are not dependent on each other and can be decoded independently, the PCC frames may be defined as the random access unit.

[0191] Note that two or more PCC frames may be assigned to one access unit, and multiple random access units may be assigned to one GOF.

[0192] Furthermore, the encoding unit 4801 may define and generate parameter sets or metadata other than those described above. For example, the encoding unit 4801 may generate SEI (Supplemental Enhancement Information) that stores parameters (optional parameters) that may not necessarily be used during decoding.

[0193] Next, the structure of the coded data and the method of storing the coded data in the NAL unit will be described.

[0194] For example, a data format is defined for each type of encoded data. Figure 26 shows an example of encoded data and NAL units.

[0195] For example, as shown in Fig. 26, the encoded data includes a header and a payload. The encoded data may include length information indicating the length (amount of data) of the encoded data, the header, or the payload. The encoded data may not include a header.

[0196] The header includes, for example, identification information for identifying the data, which indicates, for example, the data type or the frame number.

[0197] The header includes, for example, identification information indicating a reference relationship. This identification information is stored in the header when, for example, there is a dependency between data, and is information for referring to the reference destination from the reference source. For example, the header of the reference destination includes identification information for identifying the data. The header of the reference source includes identification information indicating the reference destination.

[0198] In addition, when the reference destination or the reference source can be identified or derived from other information, the identification information for specifying the data or the identification information indicating the reference relationship may be omitted.

[0199] The multiplexing unit 4802 stores the encoded data in the payload of the NAL unit. The NAL unit header includes pcc_nal_unit_type, which is identification information of the encoded data. Figure 27 shows an example of the semantics of pcc_nal_unit_type.

[0200] 27, when pcc_codec_type is codec 1 (Codec1: first encoding method), values ​​0 to 10 of pcc_nal_unit_type are assigned to the encoded position data (Geometry), encoded attribute X data (AttributeX), encoded attribute Y data (AttributeY), position PS (Geom.PS), attribute XPS (AttrX.PS), attribute YPS (AttrX.PS), position SPS (Geometry Sequence PS), attribute XSPS (AttributeX Sequence PS), attribute YSPS (AttributeY Sequence PS), AU header (AU Header), and GOF header (GOF Header) in codec 1. Values ​​11 and above are assigned as spares for codec 1.

[0201] When pcc_codec_type is Codec2 (Codec2: second encoding method), values ​​0 to 2 of pcc_nal_unit_type are assigned to codec data A (DataA), metadata A (MetaDataA), and metadata B (MetaDataB). Values ​​3 and above are assigned as spares for Codec2.

[0202] Next, the data transmission order will be described. The following describes the constraints on the transmission order of NAL units.

[0203] The multiplexing unit 4802 collectively transmits NAL units in units of GOF or AU. The multiplexing unit 4802 places a GOF header at the beginning of a GOF, and places an AU header at the beginning of an AU.

[0204] The multiplexing unit 4802 may allocate a sequence parameter set (SPS) for each AU so that the decoding device can decode from the next AU even if data is lost due to packet loss or the like.

[0205] If there is a dependency relationship related to decoding in the encoded data, the decoding device decodes the reference data after decoding the reference data. In order to enable the decoding device to decode the data in the order in which it was received without rearranging the data, the multiplexing unit 4802 sends out the reference data first.

[0206] Fig. 28 is a diagram showing examples of the transmission order of NAL units, showing three examples: position information priority, parameter priority, and data integration.

[0207] The position information priority transmission order is an example in which information related to position information and information related to attribute information are transmitted together. In this transmission order, the transmission of information related to position information is completed earlier than the transmission of information related to attribute information.

[0208] For example, by using this transmission order, a decoding device that does not decode attribute information may be able to set a time during which it does not process the attribute information by ignoring the decoding of the attribute information. Also, for example, in the case of a decoding device that wishes to decode position information quickly, it may be able to decode the position information more quickly by obtaining the encoded data of the position information quickly.

[0209] In FIG. 28, the attributes XSPS and YSPS are integrated and written as the attribute SPS, but the attributes XSPS and YSPS may be arranged separately.

[0210] In a parameter set priority sending order, the parameter sets are sent first and the data is sent later.

[0211] As described above, as long as the constraints on the NAL unit transmission order are met, the multiplexing unit 4802 may transmit the NAL units in any order. For example, order identification information may be defined, and the multiplexing unit 4802 may have a function for transmitting the NAL units in a number of different orders. For example, the NAL unit order identification information may be stored in the stream PS.

[0212] The three-dimensional data decoding device may perform decoding based on the order identification information. The three-dimensional data decoding device may instruct the three-dimensional data encoding device of a desired transmission order, and the three-dimensional data encoding device (multiplexing unit 4802) may control the transmission order according to the instructed transmission order.

[0213] The multiplexing unit 4802 may generate coded data in which multiple functions are merged, as long as the data is transmitted in a certain order, such as in the transmission order of the integrated data. For example, as shown in FIG. 28, the GOF header and the AU header may be integrated, or the AXPS and the AYPS may be integrated. In this case, an identifier indicating that the data has multiple functions is defined in the pcc_nal_unit_type.

[0214] A modified example of this embodiment will be described below. If the PS has levels, such as a frame-level PS, a sequence-level PS, and a PCC sequence-level PS, and the PCC sequence level is the higher level and the frame level is the lower level, the following method may be used to store parameters.

[0215] The default PS value is indicated in the higher PS. Also, if the value of the lower PS differs from the value of the higher PS, the PS value is indicated in the lower PS. Or, the PS value is not written in the higher PS, and is written in the lower PS. Or, information on whether the PS value is to be written in the lower PS, the higher PS, or both is indicated in either or both of the lower PS and the higher PS. Or, the lower PS may be merged into the higher PS. Or, if the lower PS and the higher PS overlap, the multiplexing unit 4802 may omit sending one of them.

[0216] The encoding unit 4801 or the multiplexing unit 4802 may divide data into slices or tiles, and transmit the divided data. The divided data includes information for identifying the divided data, and parameters used for decoding the divided data are included in the parameter set. In this case, an identifier indicating that the data stores data or parameters related to tiles or slices is defined in pcc_nal_unit_type.

[0217] The processing related to the order identification information will be explained below. Fig. 29 is a flowchart of processing by the three-dimensional data encoding device (the encoding unit 4801 and the multiplexing unit 4802) related to the transmission order of NAL units.

[0218] First, the three-dimensional data encoding device determines the transmission order (positional information priority or parameter set priority) of the NAL units (S4801). For example, the three-dimensional data encoding device determines the transmission order based on a specification from a user or an external device (e.g., a three-dimensional data decoding device).

[0219] If the determined transmission order is position information priority (position information priority in S4802), the three-dimensional data encoding device sets the order identification information included in the stream PS to position information priority (S4803). That is, in this case, the order identification information indicates that the NAL units are to be transmitted in a position information priority order. Then, the three-dimensional data encoding device transmits the NAL units in the position information priority order (S4804).

[0220] On the other hand, if the determined transmission order is parameter set priority (parameter set priority in S4802), the three-dimensional data encoding device sets the order identification information included in the stream PS to parameter set priority (S4805). That is, in this case, the order identification information indicates that the NAL units are to be transmitted in parameter set priority order. Then, the three-dimensional data encoding device transmits the NAL units in parameter set parameter set priority order (S4806).

[0221] 30 is a flowchart of a process performed by the three-dimensional data decoding device relating to the transmission order of NAL units. First, the three-dimensional data decoding device analyzes the order identification information included in the stream PS (S4811).

[0222] If the transmission order indicated by the order identification information has priority over positional information (positional information priority in S4812), the three-dimensional data decoding device assumes that the transmission order of the NAL units has priority over positional information and decodes the NAL units (S4813).

[0223] On the other hand, if the transmission order indicated by the order identification information is parameter set priority (parameter set priority in S4812), the three-dimensional data decoding device decodes the NAL units assuming that the transmission order of the NAL units is parameter set priority (S4814).

[0224] For example, if the three-dimensional data decoding device does not decode attribute information, in step S4813, it may obtain NAL units related to position information without obtaining all NAL units, and decode the position information from the obtained NAL units.

[0225] Next, a process related to generation of AUs and GOFs will be described. Fig. 31 is a flowchart of a process by the three-dimensional data encoding device (the multiplexing unit 4802) related to generation of AUs and GOFs in multiplexing of NAL units.

[0226] First, the three-dimensional data encoding device determines the type of encoded data (S4821). Specifically, the three-dimensional data encoding device determines whether the encoded data to be processed is data at the beginning of an AU, data at the beginning of a GOF, or other data.

[0227] If the encoded data is data at the start of a GOF (GOF start in S4822), the three-dimensional data encoding device places a GOF header and an AU header at the start of the encoded data belonging to the GOF to generate a NAL unit (S4823).

[0228] If the encoded data is data at the beginning of an AU (AU beginning in S4822), the three-dimensional data encoding device places an AU header at the beginning of the encoded data belonging to the AU and generates a NAL unit (S4824).

[0229] If the encoded data is not at the beginning of a GOF or an AU (other than the beginning of a GOF or an AU in S4822), the three-dimensional data encoding device places the encoded data after the AU header of the AU to which the encoded data belongs and generates a NAL unit (S4825).

[0230] Next, a process related to access to AUs and GOFs will be described. Fig. 32 is a flowchart of a process of a three-dimensional data decoding device related to access to AUs and GOFs in demultiplexing of NAL units.

[0231] First, the three-dimensional data decoding device analyzes the nal_unit_type included in the NAL unit to determine the type of coded data included in the NAL unit (S4831). Specifically, the three-dimensional data decoding device determines whether the coded data included in the NAL unit is data at the beginning of an AU, data at the beginning of a GOF, or other data.

[0232] If the encoded data contained in the NAL unit is data at the start of the GOF (start of GOF in S4832), the three-dimensional data decoding device determines that the NAL unit is the start position of random access, accesses the NAL unit, and starts the decoding process (S4833).

[0233] On the other hand, if the encoded data contained in the NAL unit is data at the beginning of an AU (AU beginning in S4832), the three-dimensional data decoding device determines that the NAL unit is the beginning of an AU, accesses the data contained in the NAL unit, and decodes the AU (S4834).

[0234] On the other hand, if the coded data included in the NAL unit is neither the start of a GOF nor the start of an AU (other than the start of a GOF or an AU in S4832), the three-dimensional data decoding device does not process the NAL unit.

[0235] (Embodiment 3) In this embodiment, a method for expressing three-dimensional points (point cloud) in encoding three-dimensional data will be described.

[0236] 33 is a block diagram showing the configuration of a three-dimensional data delivery system according to this embodiment. The delivery system shown in FIG.

[0237] The server 1501 includes a storage unit 1511 and a control unit 1512. The storage unit 1511 stores an encoded three-dimensional map 1513 that is encoded three-dimensional data.

[0238] FIG. 34 is a diagram showing an example of the bit stream of the encoded 3D map 1513. The 3D map is divided into multiple sub-maps, and each sub-map is encoded. A random access header (RA) including sub-coordinate information is added to each sub-map. The sub-coordinate information is used to improve the encoding efficiency of the sub-map. This sub-coordinate information indicates the sub-coordinate of the sub-map. The sub-coordinate is the coordinate of the sub-map based on a reference coordinate. A 3D map including multiple sub-maps is called an overall map. Furthermore, a coordinate (e.g., an origin) that is a reference in the overall map is called a reference coordinate. That is, the sub-coordinate is the coordinate of the sub-map in the coordinate system of the overall map. In other words, the sub-coordinate indicates an offset between the coordinate system of the overall map and the coordinate system of the sub-map. Furthermore, a coordinate in the coordinate system of the overall map based on the reference coordinate is called an overall coordinate. A coordinate in the coordinate system of the sub-map based on the sub-coordinate is called a differential coordinate.

[0239] The client 1502 transmits a message to the server 1501. This message includes the position information of the client 1502. The control unit 1512 included in the server 1501 obtains a bit stream of a submap located closest to the position of the client 1502 based on the position information included in the received message. The bit stream of the submap includes sub-coordinate information and is transmitted to the client 1502. The decoder 1521 included in the client 1502 uses this sub-coordinate information to obtain the overall coordinates of the submap based on the reference coordinates. The application 1522 included in the client 1502 executes an application related to its own position using the obtained overall coordinates of the submap.

[0240] Moreover, a submap indicates a partial area of ​​the overall map. Subcoordinates are coordinates at which the submap is located in the reference coordinate space of the overall map. For example, assume that submap A of AA and submap B of AB exist in the overall map of A. When a vehicle wants to refer to the map of AA, it starts decoding from submap A, and when a vehicle wants to refer to the map of AB, it starts decoding from submap B. Here, the submap is a random access point. Specifically, A is Osaka Prefecture, AA is Osaka City, AB is Takatsuki City, etc.

[0241] Each submap is transmitted to the client together with sub-coordinate information, which is included in the header information of each submap, or in a transmission packet.

[0242] The reference coordinates that serve as the reference coordinates for the sub-coordinate information of each sub-map may be added to the header information of a space higher than the sub-map, such as the header information of the overall map.

[0243] A submap may consist of one space (SPC), or it may consist of multiple SPCs.

[0244] A submap may also include a GOS (Group of Space). A submap may also consist of a world. For example, if there are multiple objects in a submap, the submap will consist of multiple SPCs if the multiple objects are assigned to different SPCs. Also, if the multiple objects are assigned to one SPC, the submap will consist of one SPC.

[0245] Next, the effect of improving the coding efficiency when using the sub-coordinate information will be described. FIG. 35 is a diagram for explaining this effect. For example, in order to code a three-dimensional point A located far from the reference coordinates shown in FIG. 35, a large number of bits is required. Here, the distance between the sub-coordinates and the three-dimensional point A is shorter than the distance between the reference coordinates and the three-dimensional point A. Therefore, coding the coordinates of the three-dimensional point A based on the sub-coordinates can improve coding efficiency more than coding the coordinates of the three-dimensional point A based on the reference coordinates. In addition, the bit stream of the submap includes the sub-coordinate information. By sending the bit stream of the submap and the reference coordinates to the decoding side (client), the overall coordinates of the submap can be restored on the decoding side.

[0246] FIG. 36 is a flowchart of the process performed by server 1501, which is the sender of the submap.

[0247] First, the server 1501 receives a message including the location information of the client 1502 from the client 1502 (S1501). The control unit 1512 acquires the coded bit stream of the submap based on the client's location information from the storage unit 1511 (S1502). Then, the server 1501 transmits the coded bit stream of the submap and the reference coordinates to the client 1502 (S1503).

[0248] FIG. 37 is a flowchart of the process performed by the client 1502 which is the recipient of the submap.

[0249] First, the client 1502 receives the encoded bit stream and the reference coordinates of the submap transmitted from the server 1501 (S1511). Next, the client 1502 obtains the submap and the subcoordinate information by decoding the encoded bit stream (S1512). Next, the client 1502 restores the difference coordinates in the submap to the global coordinates using the reference coordinates and the subcoordinates (S1513).

[0250] Next, an example of the syntax of information related to a submap will be described. In encoding a submap, a three-dimensional data encoding device calculates differential coordinates by subtracting sub-coordinates from the coordinates of each point cloud (three-dimensional point). Then, the three-dimensional data encoding device encodes the differential coordinates into a bit stream as the value of each point cloud. In addition, the encoding device encodes sub-coordinate information indicating the sub-coordinates as header information of the bit stream. This allows a three-dimensional data decoding device to obtain the overall coordinates of each point cloud. For example, the three-dimensional data encoding device is included in the server 1501, and the three-dimensional data decoding device is included in the client 1502.

[0251] Fig. 38 is a diagram showing an example of the syntax of a submap. NumOfPoint shown in Fig. 38 indicates the number of point clouds included in the submap. sub_coordinate_x, sub_coordinate_y, and sub_coordinate_z are sub-coordinate information. sub_coordinate_x indicates the x-coordinate of the sub-coordinate. sub_coordinate_y indicates the y-coordinate of the sub-coordinate. sub_coordinate_z indicates the z-coordinate of the sub-coordinate.

[0252] Also, diff_x[i], diff_y[i], and diff_z[i] are the difference coordinates of the i-th point cloud in the submap. diff_x[i] indicates the difference value between the x coordinate of the i-th point cloud in the submap and the x coordinate of the subcoordinate. diff_y[i] indicates the difference value between the y coordinate of the i-th point cloud in the submap and the y coordinate of the subcoordinate. diff_z[i] indicates the difference value between the z coordinate of the i-th point cloud in the submap and the z coordinate of the subcoordinate.

[0253] The three-dimensional data decoding device decodes point_cloud[i]_x, point_cloud[i]_y, and point_cloud[i]_z, which are the global coordinates of the i-th point cloud, using the following equations: point_cloud[i]_x is the x coordinate of the global coordinates of the i-th point cloud. point_cloud[i]_y is the y coordinate of the global coordinates of the i-th point cloud. point_cloud[i]_z is the z coordinate of the global coordinates of the i-th point cloud.

[0254] point_cloud[i]_x=sub_coordinate_x+diff_x[i] point_cloud[i]_y=sub_coordinate_y+diff_y[i] point_cloud[i]_z=sub_coordinate_z+diff_z[i]

[0255] Next, a process of switching the application of octree coding will be described. When coding a submap, the three-dimensional data coding device selects whether to code each point cloud in an octree representation (hereinafter referred to as octree coding) or to code a difference value from a sub-coordinate (hereinafter referred to as non-octree coding). FIG. 39 is a diagram showing this operation in a schematic manner. For example, when the number of point clouds in a submap is equal to or greater than a predetermined threshold, the three-dimensional data coding device applies octree coding to the submap. When the number of point clouds in a submap is less than the threshold, the three-dimensional data coding device applies non-octree coding to the submap. This allows the three-dimensional data coding device to appropriately select whether to use octree coding or non-octree coding according to the shape and density of objects included in the submap, thereby improving coding efficiency.

[0256] Furthermore, the three-dimensional data encoding device adds information indicating whether octree encoding or non-octree encoding has been applied to the submap (hereinafter referred to as octree encoding application information) to the header of the submap, etc. This allows the three-dimensional data decoding device to determine whether the bitstream is a bitstream obtained by subjecting the submap to octree encoding or a bitstream obtained by subjecting the submap to non-octree encoding.

[0257] In addition, the three-dimensional data encoding device may calculate the encoding efficiency when octree encoding and non-octree encoding are applied to the same point cloud, and apply the encoding method with the best encoding efficiency to the sub-map.

[0258] Figure 40 is a diagram showing an example of the syntax of a submap when this switching is performed. coding_type shown in Figure 40 is information indicating the coding type, and is the above-mentioned octree coding application information. coding_type=00 indicates that octree coding has been applied. coding_type=01 indicates that non-octree coding has been applied. coding_type=10 or 11 indicates that a coding method other than the above has been applied.

[0259] If the coding type is non_octree coding, the submap includes NumOfPoint and sub-coordinate information (sub_coordinate_x, sub_coordinate_y, and sub_coordinate_z).

[0260] If the coding type is octree coding, the submap includes octree_info, which is information necessary for octree coding, such as depth information.

[0261] If the encoding type is non_octree, the submap contains difference coordinates (diff_x[i], diff_y[i], and diff_z[i]).

[0262] If the coding type is octree coding, the submap contains octree_data, which is coding data for the octree coding.

[0263] Although an example in which the xyz coordinate system is used as the coordinate system of the point cloud has been shown here, a polar coordinate system may also be used.

[0264] 41 is a flowchart of a three-dimensional data encoding process performed by a three-dimensional data encoding device. First, the three-dimensional data encoding device calculates the number of point clouds in a target submap, which is a submap to be processed (S1521). Next, the three-dimensional data encoding device determines whether the calculated number of point clouds is equal to or greater than a predetermined threshold (S1522).

[0265] If the number of point clouds is equal to or greater than the threshold (Yes in S1522), the 3D data encoding device applies octree encoding to the target submap (S1523). In addition, the 3D point data encoding device adds octree encoding application information indicating that octree encoding has been applied to the target submap to the header of the bitstream (S1525).

[0266] On the other hand, if the number of point clouds is less than the threshold (No in S1522), the 3D data encoding device applies non-octree encoding to the target submap (S1524). Also, the 3D point data encoding device adds octree encoding application information indicating that non-octree encoding has been applied to the target submap to the header of the bitstream (S1525).

[0267] 42 is a flowchart of a three-dimensional data decoding process by a three-dimensional data decoding device. First, the three-dimensional data decoding device decodes octree coding application information from the header of the bit stream (S1531). Next, the three-dimensional data decoding device determines whether the coding type applied to the target submap is octree coding or not based on the decoded octree coding application information (S1532).

[0268] If the coding type indicated by the octree coding application information is octree coding (Yes in S1532), the three-dimensional data decoding device decodes the target submap by octree decoding (S1533). On the other hand, if the coding type indicated by the octree coding application information is non-octree coding (No in S1532), the three-dimensional data decoding device decodes the target submap by non-octree decoding (S1534).

[0269] A modification of this embodiment will be described below. Figures 43 to 45 are diagrams showing the operation of a modification of the coding type switching process.

[0270] As shown in Fig. 43, the three-dimensional data encoding device may select whether to apply octree encoding or non-octree encoding for each space. In this case, the three-dimensional data encoding device adds octree encoding application information to the header of the space. This allows the three-dimensional data decoding device to determine whether octree encoding has been applied for each space. Also, in this case, the three-dimensional data encoding device sets sub-coordinates for each space, and encodes the difference values ​​obtained by subtracting the values ​​of the sub-coordinates from the coordinates of each point cloud in the space.

[0271] This allows the three-dimensional data encoding device to appropriately switch whether or not to apply octree encoding depending on the shape of the object in the space or the number of point clouds, thereby improving encoding efficiency.

[0272] Also, as shown in Fig. 44, the three-dimensional data encoding device may select whether to apply octree encoding or non-octree encoding for each volume. In this case, the three-dimensional data encoding device adds octree encoding application information to the header of the volume. This allows the three-dimensional data decoding device to determine whether octree encoding has been applied for each volume. Also, in this case, the three-dimensional data encoding device sets sub-coordinates for each volume, and encodes the difference values ​​obtained by subtracting the values ​​of the sub-coordinates from the coordinates of each point cloud in the volume.

[0273] This allows the three-dimensional data encoding device to appropriately switch whether or not to apply octree encoding depending on the shape of the object in the volume or the number of point clouds, thereby improving encoding efficiency.

[0274] In the above description, an example of encoding the difference obtained by subtracting the sub-coordinates from the coordinates of each point cloud as non-octree encoding has been shown, but the encoding is not necessarily limited to this, and any encoding method other than octree encoding may be used. For example, as shown in Fig. 45, the three-dimensional data encoding device may use a method (hereinafter referred to as original coordinate encoding) for encoding the value of the point cloud itself in a submap, space, or volume, instead of the difference from the sub-coordinates, as non-octree encoding.

[0275] In this case, the three-dimensional data encoding device stores information in the header indicating that the original coordinate encoding has been applied to the target space (submap, space, or volume), which allows the three-dimensional data decoding device to determine whether the original coordinate encoding has been applied to the target space.

[0276] Furthermore, when applying original coordinate coding, the three-dimensional data coding device may perform coding without applying quantization and arithmetic coding to the original coordinates. Furthermore, the three-dimensional data coding device may code the original coordinates with a predetermined fixed bit length. This allows the three-dimensional data coding device to generate a stream with a constant bit length at a certain timing.

[0277] In the above description, an example has been given in which the difference obtained by subtracting the sub-coordinates from the coordinates of each point cloud is encoded as non-octree encoding, but the present invention is not necessarily limited to this.

[0278] For example, the three-dimensional data encoding device may sequentially encode the difference values ​​between the coordinates of each point cloud. FIG. 46 is a diagram for explaining the operation in this case. For example, in the example shown in FIG. 46, when encoding the point cloud PA, the three-dimensional data encoding device uses the sub-coordinates as predicted coordinates and encodes the difference values ​​between the coordinates of the point cloud PA and the predicted coordinates. When encoding the point cloud PB, the three-dimensional data encoding device uses the coordinates of the point cloud PA as predicted coordinates and encodes the difference values ​​between the point cloud PB and the predicted coordinates. When encoding the point cloud PC, the three-dimensional data encoding device uses the point cloud PB as predicted coordinates and encodes the difference values ​​between the point cloud PB and the predicted coordinates. In this way, the three-dimensional data encoding device may set a scan order for a plurality of point clouds and encode the difference values ​​between the coordinates of the target point cloud to be processed and the coordinates of the point cloud immediately preceding the target point cloud in the scan order.

[0279] In the above description, the sub-coordinate is the coordinate of the lower left front corner of the sub-map, but the position of the sub-coordinate is not limited to this. FIGS. 47 to 49 are diagrams showing other examples of the position of the sub-coordinate. The setting position of the sub-coordinate may be set to any coordinate in the target space (sub-map, space, or volume). That is, the sub-coordinate may be the coordinate of the lower left front corner of the target space, as described above. As shown in FIG. 47, the sub-coordinate may be the coordinate of the center of the target space. As shown in FIG. 48, the sub-coordinate may be the coordinate of the upper right back corner of the target space. Also, the sub-coordinate is not limited to the coordinate of the lower left front or upper right back corner of the target space, but may be the coordinate of any corner of the target space.

[0280] In addition, the setting position of the sub-coordinates may be the same as the coordinates of a certain point cloud in the target space (submap, space, or volume). For example, in the example shown in FIG. 49, the coordinates of the sub-coordinates match the coordinates of the point cloud PD.

[0281] In addition, in the present embodiment, an example of switching between applying octree coding and applying non-octree coding has been shown, but this is not necessarily limited to this. For example, the three-dimensional data coding device may switch between applying a different tree structure other than an octree and applying a non-tree structure other than the tree structure. For example, the different tree structure may be a kd tree that performs division using a plane perpendicular to one of the coordinate axes. Note that any method may be used as the different tree structure.

[0282] In addition, in the present embodiment, an example of encoding coordinate information of a point cloud has been shown, but this is not necessarily limited to this. The three-dimensional data encoding device may also encode, for example, color information, three-dimensional feature values, or visible light feature values ​​in the same manner as the coordinate information. For example, the three-dimensional data encoding device may set the average value of color information of each point cloud in a sub-map to sub-color information (sub-color), and encode the difference between the color information of each point cloud and the sub-color information.

[0283] In addition, in the present embodiment, an example has been shown in which a coding method (octree coding or non-octree coding) with good coding efficiency is selected according to the number of point clouds, etc., but this is not necessarily limited to this. For example, the three-dimensional data coding device on the server side may hold a bit stream of a point cloud coded by octree coding, a bit stream of a point cloud coded by non-octree coding, and a bit stream of a point cloud coded by both, and switch the bit stream to be transmitted to the three-dimensional data decoding device according to the communication environment or the processing capacity of the three-dimensional data decoding device.

[0284] Fig. 50 is a diagram showing an example of the syntax of a volume when switching the application of octree coding. The syntax shown in Fig. 50 is basically the same as the syntax shown in Fig. 40, but differs in that each piece of information is information on a volume basis. Specifically, NumOfPoint indicates the number of point clouds included in the volume. sub_coordinate_x, sub_coordinate_y, and sub_coordinate_z are sub-coordinate information of the volume.

[0285] Also, diff_x[i], diff_y[i], and diff_z[i] are the difference coordinates of the i-th point cloud in the volume. diff_x[i] indicates the difference value between the x coordinate of the i-th point cloud in the volume and the x coordinate of the sub-coordinate. diff_y[i] indicates the difference value between the y coordinate of the i-th point cloud in the volume and the y coordinate of the sub-coordinate. diff_z[i] indicates the difference value between the z coordinate of the i-th point cloud in the volume and the z coordinate of the sub-coordinate.

[0286] Note that if the relative position of a volume in space can be calculated, the three-dimensional data encoding device does not need to include sub-coordinate information in the header of the volume. In other words, the three-dimensional data encoding device may calculate the relative position of a volume in space without including the sub-coordinate information in the header, and use the calculated position as the sub-coordinate of each volume.

[0287] As described above, the three-dimensional data encoding device according to this embodiment determines whether or not to encode the target spatial unit among a plurality of spatial units (e.g., submap, space, or volume) included in the three-dimensional data with an octree structure (e.g., S1522 in FIG. 41). For example, if the number of three-dimensional points included in the target spatial unit is greater than a predetermined threshold, the three-dimensional data encoding device determines to encode the target spatial unit with an octree structure. Also, if the number of three-dimensional points included in the target spatial unit is equal to or less than the threshold, the three-dimensional data encoding device determines not to encode the target spatial unit with an octree structure.

[0288] If it is determined that the target space unit is to be coded with an octree structure (Yes in S1522), the three-dimensional data coding device codes the target space unit using the octree structure (S1523). If it is determined that the target space unit is not to be coded with an octree structure (No in S1522), the three-dimensional data coding device codes the target space unit with a method different from the octree structure (S1524). For example, in the different method, the three-dimensional data coding device codes the coordinates of a three-dimensional point included in the target space unit. Specifically, in the different method, the three-dimensional data coding device codes the difference between the reference coordinates of the target space unit and the coordinates of a three-dimensional point included in the target space unit.

[0289] Next, the three-dimensional data encoding device adds information indicating whether the target spatial unit has been encoded in an octree structure to the bit stream (S1525).

[0290] According to this, the three-dimensional data encoding device can reduce the amount of data in the encoded signal, thereby improving encoding efficiency.

[0291] For example, the three-dimensional data encoding device includes a processor and a memory, and the processor uses the memory to perform the above-mentioned processing.

[0292] Furthermore, the three-dimensional data decoding device according to this embodiment decodes information indicating whether or not a target spatial unit among a plurality of target spatial units (e.g., submaps, spaces, or volumes) included in the three-dimensional data is to be decoded in an octree structure from the bitstream (e.g., S1531 in FIG. 42). If the above information indicates that the target spatial unit is to be decoded in an octree structure (Yes in S1532), the three-dimensional data decoding device decodes the target spatial unit using the octree structure (S1533).

[0293] If the above information indicates that the target space unit is not to be decoded in an octree structure (No in S1532), the three-dimensional data decoding device decodes the target space unit in a method different from the octree structure (S1534). For example, in the different method, the three-dimensional data decoding device decodes the coordinates of a three-dimensional point included in the target space unit. Specifically, in the different method, the three-dimensional data decoding device decodes the difference between the reference coordinates of the target space unit and the coordinates of a three-dimensional point included in the target space unit.

[0294] According to this, the three-dimensional data decoding device can reduce the amount of data of the encoded signal, thereby improving the encoding efficiency.

[0295] For example, the three-dimensional data decoding device includes a processor and a memory, and the processor uses the memory to perform the above-mentioned processing.

[0296] (Embodiment 4) In the three-dimensional data encoding method according to the fourth embodiment, position information of a plurality of three-dimensional points is encoded using a prediction tree generated based on the position information.

[0297] Fig. 51 is a diagram showing an example of a prediction tree used in a three-dimensional data encoding method according to embodiment 4. Fig. 52 is a flowchart showing an example of a three-dimensional data encoding method according to embodiment 4. Fig. 53 is a flowchart showing an example of a three-dimensional data decoding method according to embodiment 4.

[0298] As shown in Fig. 51 and Fig. 52, in the three-dimensional data encoding method, a prediction tree is generated using a plurality of three-dimensional points, and then node information included in each node of the prediction tree is encoded. This results in a bit stream including encoded node information. Each node information is, for example, information about one node of the prediction tree. Each node information includes, for example, position information of one node, an index of the one node, the number of child nodes that the one node has, a prediction mode used to encode the position information of the one node, and a prediction residual.

[0299] Also, as shown in Figures 51 and 53, in the three-dimensional data decoding method, each piece of coded node information contained in the bit stream is decoded, and then the position information is decoded while generating a prediction tree.

[0300] Next, a method for generating a prediction tree will be described with reference to FIG.

[0301] FIG. 54 is a diagram for explaining a method of generating a prediction tree according to the fourth embodiment.

[0302] In the prediction tree generation method, as shown in (a) of Fig. 54, the three-dimensional data encoding device first adds point 0 as an initial point of the prediction tree. The position information of point 0 is represented by coordinates including three elements (x0, y0, z0). The position information of point 0 may be represented by coordinates of a three-axis orthogonal coordinate system or by coordinates of a polar coordinate system.

[0303] The child_count is incremented by 1 each time a child node is added to the node to which the child_count is set. The child_count of each node after the generation of the prediction tree is completed indicates the number of child nodes that each node has, and is added to the bit stream. The pred_mode indicates a prediction mode for predicting the value of the position information of each node. The details of the prediction mode will be described later.

[0304] Next, as shown in (b) of FIG. 54, the three-dimensional data encoding device adds point 1 to the prediction tree. At this time, the three-dimensional data encoding device may search for the nearest point of point 1 from the point group already added to the prediction tree, and add point 1 as a child node of the nearest point. The position information of point 1 is indicated by coordinates including three elements (x1, y1, z1). The position information of point 1 may be indicated by coordinates of a three-axis orthogonal coordinate system, or may be indicated by coordinates of a polar coordinate system. In the case of FIG. 54, point 0 becomes the nearest point of point 1, and point 1 is added as a child node of point 0. Then, the three-dimensional data encoding device increments the value indicated by child_count of point 0 by 1.

[0305] In addition, the predicted value of the position information of each node may be calculated when the node is added to the prediction tree. For example, in the case of (b) of FIG. 54, the three-dimensional data encoding device may add point 1 as a child node of point 0 and calculate the position information of point 0 as a predicted value. In that case, pred_mode=1 may be set. pred_mode is prediction mode information (prediction mode value) indicating a prediction mode. In addition, the three-dimensional data encoding device may calculate a residual_value (prediction residual) of point 1 after calculating the predicted value. Here, the residual_value is a difference value obtained by subtracting the predicted value calculated in the prediction mode indicated by pred_mode from the position information of each node. In this way, in the three-dimensional data encoding method, the encoding efficiency can be improved by encoding the difference value from the predicted value instead of the position information itself.

[0306] Next, as shown in (c) of FIG. 54, the three-dimensional data encoding device adds point 2 to the prediction tree. At this time, the three-dimensional data encoding device may search for the nearest point of point 2 from the point group already added to the prediction tree, and add point 2 as a child node of the nearest point. The position information of point 2 is indicated by coordinates including three elements (x2, y2, z2). The position information of point 2 may be indicated by coordinates of a three-axis orthogonal coordinate system, or may be indicated by coordinates of a polar coordinate system. In the case of FIG. 54, point 1 becomes the nearest point of point 2, and point 2 is added as a child node of point 1. Then, the three-dimensional data encoding device increments the value indicated by child_count of point 1 by 1.

[0307] Next, as shown in (d) of FIG. 54, the three-dimensional data encoding device adds point 3 to the prediction tree. At this time, the three-dimensional data encoding device may search for the nearest point of point 3 from the point group already added to the prediction tree, and add point 3 as a child node of the nearest point. The position information of point 3 is indicated by coordinates including three elements (x3, y3, z3). The position information of point 3 may be indicated by coordinates of a three-axis orthogonal coordinate system, or may be indicated by coordinates of a polar coordinate system. In the case of FIG. 54, point 0 becomes the nearest point of point 3, and point 3 is added as a child node of point 0. Then, the three-dimensional data encoding device increments the value indicated by child_count of point 0 by 1.

[0308] In this way, the three-dimensional data encoding device adds all points to the prediction tree and completes the generation of the prediction tree. When the generation of the prediction tree is completed, the node having child_count=0 finally becomes the leaf of the prediction tree. After the generation of the prediction tree is completed, the three-dimensional data encoding device encodes the child_count, pred_mode, and residual_value of each node selected in depth-priority order from the root node. In other words, when the three-dimensional data encoding device selects a node in depth-priority order, it selects a child node that has not yet been selected among one or more child nodes of the selected node as the next node of the selected node. When the selected node does not have a child node, the three-dimensional data encoding device selects another unselected child node of the parent node of the selected node.

[0309] The encoding order is not limited to depth-first order, and may be, for example, width-first order. When selecting a node in breadth-first order, the three-dimensional data encoding device selects, as the next node after the selected node, a node that has not yet been selected among one or more nodes at the same depth (layer) as the selected node. If there is no node at the same depth as the selected node, the three-dimensional data encoding device selects a node that has not yet been selected among one or more nodes at the next depth.

[0310] It should be noted that points 0 to 3 are an example of a plurality of three-dimensional points.

[0311] In the above three-dimensional data encoding method, child_count, pred_mode, and residual_value are calculated when each point is added to the prediction tree, but this is not necessarily limited to the above, and they may be calculated, for example, after the generation of the prediction tree is completed.

[0312] The input order of the multiple three-dimensional points to the three-dimensional data encoding device may be such that the input three-dimensional points are rearranged in ascending or descending order of the Morton order, and the first three-dimensional point is processed first. This allows the three-dimensional data encoding device to efficiently search for the nearest point of the target three-dimensional point, thereby improving encoding efficiency. The three-dimensional data encoding device may also process the three-dimensional points in the input order without rearranging them. For example, the three-dimensional data encoding device may generate a prediction tree without branches in the input order of the multiple three-dimensional points. Specifically, the three-dimensional data encoding device may add the three-dimensional point input next to the first three-dimensional point in the input order of the multiple three-dimensional points as a child node of a specific three-dimensional point.

[0313] Next, a first example of a prediction mode will be described with reference to Fig. 55. Fig. 55 is a diagram for explaining a first example of a prediction mode according to Embodiment 4. Fig. 55 is a diagram showing a part of a prediction tree.

[0314] As shown below, eight prediction modes may be set. For example, as shown in FIG. 55, a case where a predicted value of a point c is calculated will be described as an example. In the prediction tree, it is shown that the parent node of the point c is the point p0, the grandfather node of the point c is the point p1, and the great-grandfather node of the point c is the point p2. Note that the points c, p0, p1, and p2 are examples of a plurality of three-dimensional points.

[0315] A prediction mode having a prediction mode value of 0 (hereinafter, referred to as prediction mode 0) may be set without prediction. That is, in prediction mode 0, the three-dimensional data encoding device may calculate position information of the input point c as a predicted value of the point c.

[0316] Furthermore, a prediction mode with a prediction mode value of 1 (hereinafter, referred to as prediction mode 1) may be set to differential prediction with respect to point p0. In other words, the three-dimensional data encoding device may calculate position information of point p0, which is a parent node of point c, as a predicted value of point c.

[0317] Furthermore, a prediction mode with a prediction mode value of 2 (hereinafter, referred to as prediction mode 2) may be set to linear prediction using points p0 and p1. That is, the three-dimensional data encoding device may calculate a prediction result by linear prediction using position information of point p0, which is the parent node of point c, and position information of point p1, which is the grandparent node of point c, as a predicted value of point c. Specifically, the three-dimensional data encoding device calculates a predicted value of point c in prediction mode 2 using the following formula T1.

[0318] Prediction value = 2 × p0-p1 (Formula T1)

[0319] In the formula T1, p0 indicates the position information of the point p0, and p1 indicates the position information of the point p1.

[0320] Furthermore, a prediction mode with a prediction mode value of 3 (hereinafter, referred to as prediction mode 3) may be set to Parallelogram prediction using points p0, p1, and p2. That is, the three-dimensional data encoding device may calculate a prediction result by Parallelogram prediction using position information of point p0, which is the parent node of point c, position information of point p1, which is the grandfather node of point c, and position information of point p2, which is the great-grandfather node of point c, as a predicted value of point c. Specifically, the three-dimensional data encoding device calculates a predicted value of point c in prediction mode 3 using the following formula T2.

[0321] Predicted value = p0 + p1 - p2 (Formula T2)

[0322] In formula T2, p0 indicates position information of point p0, p1 indicates position information of point p1, and p2 indicates position information of point p2.

[0323] Furthermore, a prediction mode with a prediction mode value of 4 (hereinafter, referred to as prediction mode 4) may be set to differential prediction with respect to point p1. In other words, the three-dimensional data encoding device may calculate position information of point p1, which is a grandparent node of point c, as a predicted value of point c.

[0324] Furthermore, a prediction mode with a prediction mode value of 5 (hereinafter, referred to as prediction mode 5) may be set to differential prediction with respect to point p2. In other words, the three-dimensional data encoding device may calculate position information of point p2, which is the great-grandfather node of point c, as a predicted value of point c.

[0325] In addition, a prediction mode with a prediction mode value of 6 (hereinafter, referred to as prediction mode 6) may be set to the average of two or more of the position information of points p0, p1, and p2. That is, the three-dimensional data encoding device may calculate the average value of two or more of the position information of point p0, which is the parent node of point c, the position information of point p1, which is the grandparent node of point c, and the position information of point p2, which is the great-grandparent node of point c, as the predicted value of point c. For example, when the three-dimensional data encoding device uses the position information of point p0 and the position information of point p1 to calculate the predicted value, it calculates the predicted value of point c in prediction mode 6 using the following formula T3.

[0326] Predicted value = (p0 + p1) / 2 (Formula T3)

[0327] In formula T3, p0 indicates the position information of point p0, and p1 indicates the position information of point p1.

[0328] Furthermore, a prediction mode with a prediction mode value of 7 (hereinafter, referred to as prediction mode 7) may be set to nonlinear prediction using a distance d0 between points p0 and p1 and a distance d1 between points p2 and p1. In other words, the three-dimensional data encoding device may calculate a prediction result by nonlinear prediction using the distance d0 and the distance d1 as a predicted value of point c.

[0329] The prediction method to be assigned to each prediction mode is not limited to the above example. The above eight prediction modes and the above eight prediction methods may be any combination other than the above combination. For example, when the prediction modes are encoded using entropy encoding such as arithmetic encoding, a prediction method with a high usage frequency may be assigned to prediction mode 0. This can improve the encoding efficiency. The three-dimensional data encoding device may improve the encoding efficiency by dynamically changing the assignment of the prediction modes according to the usage frequency of the prediction modes while proceeding with the encoding process. The three-dimensional data encoding device may, for example, count the usage frequency of each prediction mode during encoding, and assign a prediction mode indicated by a smaller value to a prediction method with a higher usage frequency. This can improve the encoding efficiency. Note that M is the number of prediction modes indicating the number of prediction modes, and in the above example, there are eight prediction modes, prediction modes 0 to 7, so M=8.

[0330] The three-dimensional data encoding device may calculate a predicted value (px, py, pz) of the position information (x, y, z) of the three-dimensional point to be encoded, using position information of a three-dimensional point that is close to the three-dimensional point to be encoded among three-dimensional points surrounding the three-dimensional point to be encoded, to calculate a predicted value to be used in calculating the position information of the three-dimensional point to be encoded. Furthermore, the three-dimensional data encoding device may add prediction mode information (pred_mode) to each three-dimensional point, so that the predicted value calculated according to the prediction mode can be selected.

[0331] For example, in a case where the total number of prediction modes is M, it is conceivable to assign the position information of the nearest 3D point p0 to prediction mode 0, ..., the position information of 3D point p2 to prediction mode M-1, and add the prediction mode used for prediction to the bitstream for each 3D point.

[0332] The number of prediction modes M may be added to the bit stream. The number of prediction modes M may not be added to the bit stream, and may be specified by a profile, level, or the like of a standard. The number of prediction modes M may be calculated from the number of three-dimensional points N used for prediction. For example, the number of prediction modes M may be calculated by M=N+1.

[0333] FIG. 56 is a diagram showing a second example of a table indicating predicted values ​​calculated in each prediction mode according to the fourth embodiment.

[0334] The table shown in FIG. 56 is an example in which the number of three-dimensional points used for prediction N=4 and the number of prediction modes M=5.

[0335] In a second example, the predicted value of the position information of the point c is calculated using at least one of the position information of the points p0, p1, and p2. A prediction mode is added to each of the three-dimensional points to be encoded. The predicted value is calculated to a value according to the added prediction mode.

[0336] FIG. 57 is a diagram showing a specific example of a second example of a table indicating predicted values ​​calculated in each prediction mode according to the fourth embodiment.

[0337] The three-dimensional data encoding device may, for example, select prediction mode 1 and encode the position information (x, y, z) of the three-dimensional point to be encoded using the predicted values ​​(p0x, p0y, p0z). In this case, a prediction mode value "1" indicating the selected prediction mode 1 is added to the bitstream.

[0338] In this way, when selecting a prediction mode, the three-dimensional data encoding device may select a prediction mode that is common to the three elements as one prediction mode for calculating predicted values ​​of each of the three elements contained in the position information of the three-dimensional point to be encoded.

[0339] FIG. 58 is a diagram showing a third example of a table indicating predicted values ​​calculated in each prediction mode according to the fourth embodiment.

[0340] The table shown in FIG. 58 is an example in which the number of three-dimensional points used for prediction N=2 and the number of prediction modes M=5.

[0341] In a third example, the predicted value of the position information of the point c is calculated using at least one of the position information of the point p0 and the point p1. A prediction mode is added to each three-dimensional point to be encoded. The predicted value is calculated to a value according to the added prediction mode.

[0342] In addition, as in the third example, when the number of points (adjacent points) around the point c is less than three, the prediction mode to which the predicted value is not assigned may be set to "not available." In addition, when a prediction mode in which "not available" is set occurs, another prediction method may be assigned to the prediction mode. For example, the position information of the point p2 may be assigned as a predicted value to the prediction mode. In addition, a predicted value assigned to another prediction mode may be assigned to the prediction mode. For example, the position information of the point p1 assigned to the prediction mode 4 may be assigned to the prediction mode 3 in which "not available" is set. At that time, the position information of the point p2 may be newly assigned to the prediction mode 4. In this way, when a prediction mode in which "not available" is set occurs, the coding efficiency can be improved by assigning a new prediction method to the prediction mode.

[0343] In addition, when the position information has three elements such as a three-axis orthogonal coordinate system or a polar coordinate system, the predicted value may be calculated in a prediction mode divided for each of the three elements. For example, when the three elements are expressed by x, y, and z of the coordinates (x, y, z) of a three-axis orthogonal coordinate system, each of the predicted values ​​of the three elements may be calculated in a prediction mode selected for each element. For example, a prediction mode value may be selected in each of a prediction mode pred_mode_x for calculating a predicted value of element x (i.e., x coordinate), a prediction mode pred_mode_y for calculating a predicted value of element y (i.e., y coordinate), and a prediction mode pred_mode_z for calculating a predicted value of element z (i.e., z coordinate). In this case, the values ​​of the tables in FIG. 59 to FIG. 61 described later are used as the prediction mode value indicating the prediction mode of each element, and these prediction mode values ​​may be added to the bit stream. In addition, although the above describes the coordinates of a three-axis orthogonal coordinate system as an example of position information, the same can be applied to the coordinates of a polar coordinate system.

[0344] In this way, when selecting a prediction mode, the three-dimensional data encoding device may select an independent prediction mode for each of the three elements as one prediction mode for calculating a predicted value of each of the three elements contained in the position information of the three-dimensional point to be encoded.

[0345] In addition, a predicted value including two or more elements of the multiple elements of the position information may be calculated in a common prediction mode. For example, when three elements are expressed by x, y, and z of coordinates (x, y, z) of a three-axis orthogonal coordinate system, a prediction mode value may be selected in each of a prediction mode pred_mode_x for calculating a predicted value using the element x and a prediction mode pred_mode_yz for calculating a predicted value using the element y and the element z. In this case, the values ​​of the tables of Figures 59 and 62 described later are used as the prediction mode value indicating the prediction mode of each component, and these prediction mode values ​​may be added to the bit stream.

[0346] In this way, when selecting a prediction mode, the three-dimensional data encoding device may select a common prediction mode for two of the three elements as one prediction mode for calculating predicted values ​​of each of the three elements contained in the position information of the three-dimensional point to be encoded, and may select a prediction mode independent of the above two elements for the remaining element.

[0347] 59 is a diagram showing a fourth example of a table showing predicted values ​​calculated in each prediction mode. Specifically, the fourth example is an example in which position information used for the predicted value is the value of element x of position information of surrounding three-dimensional points.

[0348] As shown in FIG. 59, the predicted value calculated in the prediction mode pred_mode_x indicated by the prediction mode value "0" is 0. The predicted value calculated in the prediction mode pred_mode_x indicated by the prediction mode value "1" is the x-coordinate of the point p0, which is p0x. The predicted value calculated in the prediction mode pred_mode_x indicated by the prediction mode value "2" is the prediction result of the linear prediction by the x-coordinate of the point p0 and the x-coordinate of the point p1, which is (2×p0x-p1x). The predicted value calculated in the prediction mode pred_mode_x indicated by the prediction mode value "3" is the prediction result of the Parallelogram prediction by the x-coordinate of the point p0, the x-coordinate of the point p1, and the x-coordinate of the point p2, which is (p0x+p1x-p2x). The predicted value calculated in the prediction mode pred_mode_x indicated by the prediction mode value "4" is the x-coordinate of the point p1, which is p1x.

[0349] For example, when a prediction mode pred_mode_x indicated by a prediction mode value of "1" in the table of FIG. 59 is selected, the x coordinate of the position information of the 3D point to be encoded may be encoded using the predicted value p0x. In this case, "1" as the prediction mode value is added to the bit stream.

[0350] 60 is a diagram showing a fifth example of a table showing predicted values ​​calculated in each prediction mode. Specifically, the fifth example is an example in which position information used for the predicted value is the value of element y of the position information of surrounding three-dimensional points.

[0351] As shown in FIG. 60, the predicted value calculated in the prediction mode pred_mode_y indicated by the prediction mode value "0" is 0. The predicted value calculated in the prediction mode pred_mode_y indicated by the prediction mode value "1" is the y coordinate of the point p0, which is p0y. The predicted value calculated in the prediction mode pred_mode_y indicated by the prediction mode value "2" is the prediction result of the linear prediction by the y coordinate of the point p0 and the y coordinate of the point p1, which is (2×p0y-p1y). The predicted value calculated in the prediction mode pred_mode_y indicated by the prediction mode value "3" is the prediction result of the Parallelogram prediction by the y coordinate of the point p0, the y coordinate of the point p1, and the y coordinate of the point p2, which is (p0y+p1y-p2y). The predicted value calculated in the prediction mode pred_mode_y indicated by the prediction mode value "4" is the y coordinate of the point p1, which is p1y.

[0352] For example, when a prediction mode pred_mode_y whose prediction mode value is "1" in the table of FIG. 60 is selected, the y coordinate of the position information of the 3D point to be encoded may be encoded using the predicted value p0y. In this case, "1" as the prediction mode value is added to the bit stream.

[0353] 61 is a diagram showing a sixth example of a table showing predicted values ​​calculated in each prediction mode. Specifically, the sixth example is an example in which position information used for the predicted value is the value of element z of the position information of surrounding three-dimensional points.

[0354] As shown in FIG. 61, the predicted value calculated in the prediction mode pred_mode_z indicated by the prediction mode value "0" is 0. The predicted value calculated in the prediction mode pred_mode_z indicated by the prediction mode value "1" is the z coordinate of the point p0, which is p0z. The predicted value calculated in the prediction mode pred_mode_z indicated by the prediction mode value "2" is the prediction result of the linear prediction by the z coordinate of the point p0 and the z coordinate of the point p1, which is (2×p0z-p1z). The predicted value calculated in the prediction mode pred_mode_z indicated by the prediction mode value "3" is the prediction result of the Parallelogram prediction by the z coordinate of the point p0, the z coordinate of the point p1, and the z coordinate of the point p2, which is (p0z+p1z-p2z). The predicted value calculated in the prediction mode pred_mode_z indicated by the prediction mode value "4" is the z coordinate of the point p1, which is p1z.

[0355] For example, when a prediction mode pred_mode_z whose prediction mode value is "1" in the table of Fig. 61 is selected, the z coordinate of the position information of the 3D point to be encoded may be encoded using the predicted value p0z. In this case, "1" as the prediction mode value is added to the bit stream.

[0356] 62 is a diagram showing a seventh example of a table showing predicted values ​​calculated in each prediction mode. Specifically, the seventh example is an example in which position information used for the predicted value is the values ​​of elements y and z of position information of surrounding three-dimensional points.

[0357] As shown in FIG. 62, the predicted value calculated in the prediction mode pred_mode_yz indicated by the prediction mode value "0" is 0. The predicted value calculated in the prediction mode pred_mode_yz indicated by the prediction mode value "1" is the y coordinate and z coordinate of point p0, which is (p0y, p0z). The predicted value calculated in the prediction mode pred_mode_yz indicated by the prediction mode value "2" is the prediction result of linear prediction using the y coordinate and z coordinate of point p0 and the y coordinate and z coordinate of point p1, which is (2×p0y-p1y, 2×p0z-p1z). In addition, the predicted value calculated in the prediction mode pred_mode_yz indicated by the prediction mode value "3" is the prediction result of the Parallelogram prediction by the y coordinate and z coordinate of the point p0, the y coordinate and z coordinate of the point p1, and the y coordinate and z coordinate of the point p2, and is (p0y+p1y-p2y, p0z+p1z-p2z). In addition, the predicted value calculated in the prediction mode pred_mode_yz indicated by the prediction mode value "4" is the y coordinate and z coordinate of the point p1, and is (p1y, p1z).

[0358] For example, when a prediction mode pred_mode_yz indicated by a prediction mode value of "1" in the table of FIG. 62 is selected, the y coordinate and z coordinate of the position information of the 3D point to be encoded may be encoded using the predicted value (p0y, p0z). In this case, "1" as the prediction mode value is added to the bit stream.

[0359] In the tables in the fourth to seventh examples, the correspondence relationship between the prediction mode and the prediction method of the calculated predicted value is similar to the above-mentioned correspondence relationship in the table in the second example.

[0360] The prediction mode during encoding may be selected by RD optimization. For example, it is possible to calculate the cost (P) when a certain prediction mode P is selected, and select the prediction mode P with the minimum cost (P). The cost (P) may be calculated by the formula D1 using the prediction residual residual_value (P) when the prediction value of the prediction mode P is used, the number of bits (P) required to encode the prediction mode P, and the adjustment parameter λ value.

[0361] cost(P)=abs(residual(P))+λ×bit(P)...(Formula D1)

[0362] abs(x) denotes the absolute value of x.

[0363] The squared value of x may be used instead of abs(x).

[0364] By using the above formula D1, it is possible to select a prediction mode that takes into account the balance between the magnitude of the prediction residual and the number of bits required to encode the prediction mode. The adjustment parameter λ may be set to a different value depending on the value of the quantization scale. For example, when the quantization scale is small (at high bit rate), the λ value may be reduced to select a prediction mode that reduces the prediction residual residual_value(P) and improve prediction accuracy as much as possible, and when the quantization scale is large (at low bit rate), the λ value may be increased to select an appropriate prediction mode while taking into account the number of bits bit(P) required to encode the prediction mode P.

[0365] In addition, a case where the quantization scale is small is, for example, a case where the quantization scale is smaller than the first quantization scale. A case where the quantization scale is large is, for example, a case where the quantization scale is larger than a second quantization scale that is equal to or larger than the first quantization scale. In addition, the smaller the quantization scale is, the smaller the λ value may be set to.

[0366] The prediction residual residual_value(P) is calculated by subtracting the predicted value of the prediction mode P from the position information of the 3D point to be coded. Note that instead of the prediction residual residual_value(P) used when calculating the cost, the prediction residual residual_value(P) may be quantized and dequantized, and added to the predicted value to obtain a decoded value, and the difference (encoding error) between the position information of the original 3D point and the decoded value when using the prediction mode P may be reflected in the cost value. This makes it possible to select a prediction mode with a small coding error.

[0367] For the number of bits (P) required to code a prediction mode P, when the prediction mode is binarized and coded, the number of bits after binarization may be used.

[0368] For example, in the case of the number of prediction modes M=5, as shown in FIG. 63, the prediction mode value indicating the prediction mode may be binarized by a truncated unary code with a maximum value of 5 using the number of prediction modes M. In this case, 1 bit is used for the prediction mode value "0", 2 bits for the prediction mode value "1", 3 bits for the prediction mode value "2", and 4 bits for the prediction mode value "3" and "4" as the number of bits (P) required for encoding each prediction mode value. By using the truncated unary code, the smaller the value of the prediction mode, the smaller the number of bits. For this reason, it is possible to reduce the amount of code of the prediction mode value indicating the prediction mode that calculates a prediction value that is likely to be selected, for example, a prediction value that is likely to be minimized, such as 0 calculated as a predicted value when the prediction mode value is "0" or the position information of the three-dimensional point p0 calculated as a predicted value when the prediction mode value is "1", that is, the position information of the three-dimensional point close to the three-dimensional point to be encoded.

[0369] In this way, the three-dimensional data encoding device may encode the prediction mode value indicating the selected prediction mode by using the number of prediction modes. Specifically, the three-dimensional data encoding device may encode the prediction mode value by a truncated unary code in which the number of prediction modes is set to the maximum value.

[0370] Furthermore, when the maximum value of the number of prediction modes has not been determined, the prediction mode value indicating the prediction mode may be binarized with a unary code as shown in Figure 64. Furthermore, when the occurrence probability of each prediction mode is close, the prediction mode value indicating the prediction mode may be binarized with a fixed code as shown in Figure 65 to reduce the amount of code.

[0371] Note that the number of bits (P) required to encode a prediction mode value indicating a prediction mode P may be determined by arithmetically encoding binary data of the prediction mode value indicating the prediction mode P, and the amount of code after arithmetically encoding may be set as the value of bit(P). This allows the cost to be calculated using a more accurate number of required bits (P), making it possible to select a more appropriate prediction mode.

[0372] 63 is a diagram showing a first example of a binarization table when binarizing and encoding prediction mode values ​​according to Embodiment 4. Specifically, the first example is an example in which prediction mode values ​​are binarized by truncated unary code when the number of prediction modes M=5.

[0373] 64 is a diagram showing a second example of a binarization table when binarizing and encoding prediction mode values ​​according to embodiment 4. Specifically, the second example is an example in which prediction mode values ​​are binarized by unary code when the number of prediction modes M=5.

[0374] 65 is a diagram showing a third example of a binarization table when binarizing and encoding prediction mode values ​​according to Embodiment 4. Specifically, the third example is an example in which prediction mode values ​​are binarized by fixed code when the number of prediction modes M=5.

[0375] The prediction mode value indicating the prediction mode (pred_mode) may be binarized, arithmetically coded, and added to the bit stream. As described above, the prediction mode value may be binarized by a truncated unary code using the value of the number of prediction modes M, for example. In this case, the maximum number of bits of the prediction mode value after binarization is M-1.

[0376] In addition, the binary data after binarization may be arithmetically coded using a coding table. In this case, for example, coding efficiency may be improved by switching coding tables for each bit of the binary data. In addition, in order to suppress the number of coding tables, the first bit of the binary data may be coded using a coding table A for one bit, and each bit of the remaining bits may be coded using a coding table B for the remaining bits. For example, when coding binary data "1110" with a prediction mode value of "3" shown in FIG. 66, the first bit "1" may be coded using the coding table A, and each bit of the remaining bits "110" may be coded using the coding table B.

[0377] 66 is a diagram for explaining an example of encoding binary data of a binarization table when binarizing and encoding a prediction mode according to embodiment 4. The binarization table in FIG. 66 is an example of binarizing a prediction mode value by a truncated unary code when the number of prediction modes M=5.

[0378] This makes it possible to improve the coding efficiency by switching the coding table according to the bit position of the binary data while suppressing the number of coding tables. Note that when coding the remaining bits, the coding table may be further switched for each bit to perform arithmetic coding, or the coding table may be switched according to the result of arithmetic coding to perform decoding.

[0379] When binarizing and encoding a prediction mode value using a truncated unary code using the number of prediction modes M, the number of prediction modes M used in the truncated unary code may be added to a header of a bit stream or the like so that the prediction mode can be specified from the binary data decoded on the decoding side. The header of the bit stream may be, for example, a sequence parameter set (SPS), a position parameter set (GPS), a slice header, or the like. In addition, a possible value MaxM of the number of prediction modes may be specified by a standard or the like, and a value of MaxM-M (M<=MaxM) may be added to the header. In addition, the number of prediction modes M may be specified by a profile or level of a standard or the like without being added to the stream.

[0380] It is considered that the prediction mode value binarized using the truncated unary code is arithmetically coded by switching the coding table between the one bit part and the remaining part as described above. The occurrence probability of 0 and 1 in each coding table may be updated according to the value of the binary data that has actually occurred. Also, the occurrence probability of 0 and 1 in either coding table may be fixed. This may reduce the number of updates of the occurrence probability and reduce the amount of processing. For example, the occurrence probability of the one bit part may be updated, and the occurrence probability of the remaining bit part may be fixed.

[0381] 67 is a flowchart showing an example of encoding of prediction mode values ​​according to Embodiment 4. FIG 68 is a flowchart showing an example of decoding of prediction mode values ​​according to Embodiment 4.

[0382] As shown in FIG. 67, in encoding a prediction mode value, first, the prediction mode value is binarized by a truncated unary code using the number of prediction modes M (S9701).

[0383] Next, the binary data of the truncated unary code is arithmetically coded (S9702), so that the binary data is included as a prediction mode in the bit stream.

[0384] Furthermore, as shown in FIG. 68, in decoding the prediction mode value, first, the bit stream is arithmetically decoded using the number of prediction modes M to generate binary data of a truncated unary code (S9711).

[0385] Next, a prediction mode value is calculated from the binary data of the truncated unary code (S9712).

[0386] As a method of binarizing a prediction mode value indicating a prediction mode (pred_mode), an example of binarizing with a truncated unary code using the value of the prediction mode number M has been shown, but this is not necessarily limited to this. For example, the prediction mode value may be binarized with a truncated unary code using the number L (L<=M) of prediction values ​​assigned to the prediction mode. For example, when the prediction mode number M=5, when there is one surrounding three-dimensional point available for prediction of a certain encoding target three-dimensional point, there are cases where two prediction modes are available as shown in FIG. 69, and the remaining three prediction modes are not available. For example, when the prediction mode number M=5 as shown in FIG. 69, there are cases where there is one surrounding three-dimensional point available for prediction of a certain encoding target three-dimensional point, and no prediction value is assigned to the prediction mode indicating the prediction mode values ​​"2", "3" and "4".

[0387] In this case, as shown in Fig. 70, by binarizing the prediction mode value with a truncated unary code with the value L assigned to the prediction mode as the maximum value, it is possible to reduce the number of bits after binarization compared to the case of using truncated unary code with the number of prediction modes M. For example, since L=3 in this case, the number of bits can be reduced by binarizing with a truncated unary code with a maximum value of 3. In this way, the number of bits after binarization of the prediction mode value may be reduced by binarizing with a truncated unary code with the number L assigned to the prediction value as the maximum value.

[0388] The binary data after binarization may be arithmetically coded using a coding table. In this case, for example, coding efficiency may be improved by switching coding tables for each bit of the binary data. In addition, in order to suppress the number of coding tables, the first bit of the binary data may be coded using coding table A for one bit, and each bit of the remaining bits may be coded using coding table B for the remaining bits. For example, when coding binary data "1" with a prediction mode value of "1" shown in FIG. 70, the first bit "1" is coded using coding table A. There are no remaining bits, so coding is not necessary. If there are remaining bits, the remaining bits may be coded using coding table B.

[0389] 70 is a diagram for explaining an example of encoding binary data of a binarization table when binarizing and encoding a prediction mode according to embodiment 4. The binarization table in FIG. 70 is an example of binarizing a prediction mode value by a truncated unary code when the number L of predicted values ​​assigned to a prediction mode is 2.

[0390] This makes it possible to improve the coding efficiency by switching the coding table according to the bit position of the binary data while suppressing the number of coding tables. Note that when coding the remaining bits, the coding table may be further switched for each bit to perform arithmetic coding, or the coding table may be switched according to the result of arithmetic coding to perform decoding.

[0391] When the prediction mode value is binarized and encoded using a truncated unary code using a number L to which a prediction value is assigned, the number L may be calculated by assigning a prediction value to the prediction mode in the same manner as during encoding, and the prediction mode may be decoded using the calculated L so that the prediction mode can be identified from the binary data decoded on the decoding side.

[0392] It is considered that the prediction mode value binarized using the truncated unary code is arithmetically coded by switching the coding table between the one bit part and the remaining part as described above. The occurrence probability of 0 and 1 in each coding table may be updated according to the value of the binary data that has actually occurred. Also, the occurrence probability of 0 and 1 in either coding table may be fixed. This may reduce the number of updates of the occurrence probability and reduce the amount of processing. For example, the occurrence probability of the one bit part may be updated, and the occurrence probability of the remaining bit part may be fixed.

[0393] 71 is a flowchart showing another example of encoding of prediction mode values ​​according to Embodiment 4. FIG 72 is a flowchart showing another example of decoding of prediction mode values ​​according to Embodiment 4.

[0394] As shown in FIG. 71, in encoding a prediction mode value, first, the number L of prediction values ​​assigned to the prediction modes is calculated (S9721).

[0395] Next, the prediction mode value is binarized by a truncated unary code using the number L (S9722).

[0396] Next, the binary data of the truncated unary code is arithmetically coded (S9723).

[0397] As shown in FIG. 72, in decoding a prediction mode value, first, the number L of predicted values ​​assigned to the prediction modes is calculated (S9731).

[0398] Next, the bit stream is arithmetically decoded using the number L to generate binary data of a truncated unary code (S9732).

[0399] Next, a prediction mode value is calculated from the binary data of the truncated unary code (S9733).

[0400] The prediction mode value does not have to be added to every piece of position information. For example, if a certain condition is met, the prediction mode may be fixed and the prediction mode value may not be added to the bit stream, and if a certain condition is not met, the prediction mode may be selected and the prediction mode value may be added to the bit stream. For example, if condition A is met, the prediction mode value may be fixed to "2" and a predicted value may be calculated from the linear prediction of the surrounding three-dimensional points, and if condition A is not met, one prediction mode may be selected from a plurality of prediction modes and a prediction mode value indicating the selected prediction mode may be added to the bit stream.

[0401] As a certain condition A, for example, the distance d0 between point p1 and point P0 and the distance d1 between point p2 and point p1 are calculated, and the absolute difference distdiff=|d0-d1| is less than the threshold value Thfix. When the absolute difference value is less than the threshold value Thfix, the three-dimensional data encoding device determines that the difference between the predicted value by linear prediction and the position information of the point to be processed is small, and fixes the prediction mode value to "2" and does not encode the prediction mode value, thereby making it possible to generate an appropriate predicted value while reducing the amount of code for encoding the prediction mode. Note that, when the absolute difference value is equal to or greater than the threshold value Thfix, the three-dimensional data encoding device may select a prediction mode and encode a prediction mode value indicating the selected prediction mode.

[0402] The threshold value Thfix may be added to the header of the bit stream, and the encoder may change the value of the threshold value Thfix for encoding. For example, the encoder may add the threshold value Thfix to the header at a lower value than that at a lower bit rate when encoding at a high bit rate, and may increase the number of cases in which the prediction mode is selected and encoded, so that the prediction residual is reduced as much as possible. In addition, the encoder may add the threshold value Thfix to the header at a higher value than that at a high bit rate when encoding at a low bit rate, and may perform encoding with the prediction mode fixed. In this way, by increasing the number of cases in which the prediction mode is fixed and encoded at a low bit rate, it is possible to improve the encoding efficiency while suppressing the amount of bits for encoding the prediction mode. In addition, the threshold value Thfix may not be added to the bit stream, but may be specified in the profile or level of the standard.

[0403] The N 3D points around the 3D point to be coded that are used for prediction are the N coded and decoded 3D points whose distance from the 3D point to be coded is less than a threshold THd. The maximum value of N may be added to the bitstream as NumNeighborPoint. The value of N does not always have to match the value of NumNeighborPoint, such as when the number of coded and decoded 3D points around is less than the value of NumNeighborPoint.

[0404] Although an example has been shown in which the prediction mode value is fixed to "2" if the absolute difference value distdiff used in prediction is smaller than the threshold value Thfix[i], the present invention is not limited to this, and the prediction mode value may be fixed to any one of "0" to "M-1". In addition, the fixed prediction mode value may be added to the bit stream.

[0405] Fig. 73 is a flowchart showing an example of a process for determining whether to fix a prediction mode value in accordance with condition A during encoding according to embodiment 4. Fig. 74 is a flowchart showing an example of a process for determining whether to fix a prediction mode value or decode a prediction mode value in accordance with condition A during decoding according to embodiment 4.

[0406] As shown in FIG. 73, first, the three-dimensional data encoding device calculates the distance d0 between points p1 and p0, and the distance d1 between points p2 and p1, and calculates the absolute difference distdiff=|d0-d1| (S9741).

[0407] Next, the three-dimensional data encoding device determines whether the difference absolute value distdiff is less than a threshold value Thfix (S9742). Note that the threshold value Thfix may be encoded and added to the header of the stream, etc.

[0408] If the difference absolute value distdiff is less than the threshold value Thfix (Yes in S9742), the three-dimensional data encoding device determines the prediction mode value to be "2" (S9743).

[0409] On the other hand, if the difference absolute value distdiff is equal to or greater than the threshold value Thfix (No in S9742), the three-dimensional data encoding device sets one prediction mode from among the multiple prediction modes (S9744).

[0410] Then, the three-dimensional data encoding device arithmetically encodes the prediction mode value indicating the set prediction mode (S9745). Specifically, the three-dimensional data encoding device arithmetically encodes the prediction mode value by executing steps S9701 and S9702 described in FIG. 67. Note that the three-dimensional data encoding device may arithmetically encode the prediction mode pred_mode by binarizing it with a truncated unary code using the number of prediction modes to which the prediction value is assigned. That is, the three-dimensional data encoding device may arithmetically encode the prediction mode value by executing steps S9721 to S9723 described in FIG. 71.

[0411] The three-dimensional data encoding device calculates a predicted value of the prediction mode determined in step S9743 or the prediction mode set in step S9745, and outputs the calculated predicted value (S9746). When using the prediction mode value determined in step S9743, the three-dimensional data encoding device calculates a predicted value of the prediction mode indicated by a prediction mode value of "2" by linear prediction of position information of the surrounding N three-dimensional points.

[0412] Also, as shown in FIG. 74, first, the three-dimensional data decoding device calculates the distance d0 between points p1 and p0, and the distance d1 between points p2 and p1, and calculates the absolute difference distdiff=|d0-d1| (S9751).

[0413] Next, the three-dimensional data decoding device judges whether or not the absolute difference value distdiff is less than a threshold value Thfix (S9752). Note that the threshold value Thfix may be set by decoding the header of the stream, etc.

[0414] If the difference absolute value distdiff is less than the threshold value Thfix (Yes in S9752), the three-dimensional data decoding device determines the prediction mode value to be "2" (S9753).

[0415] On the other hand, if the difference absolute value distdiff is equal to or greater than the threshold value Thfix (No in S9752), the three-dimensional data decoding device decodes the prediction mode value from the bitstream (S9754).

[0416] The three-dimensional data decoding device calculates a predicted value of the prediction mode indicated by the prediction mode value determined in step S9753 or the prediction mode value decoded in step S9754, and outputs the calculated predicted value (S9755). When using the prediction mode value determined in step S9753, the three-dimensional data decoding device calculates a predicted value of the prediction mode indicated by a prediction mode value of "2" by linear prediction of position information of the surrounding N three-dimensional points.

[0417] FIG. 75 is a diagram showing an example of the syntax of the header of the position information. The NumNeighborPoint, NumPredMode, Thfix, QP, and unique_point_per_leaf in the syntax of FIG. 75 will be described in order.

[0418] NumNeighborPoint indicates the upper limit value of the number of surrounding points used to generate the predicted value of the position information of the three-dimensional points. When the number of surrounding points M is less than NumNeighborPoint (M < NumNeighborPoint), in the calculation process of the predicted value, the predicted value may be calculated using M surrounding points.

[0419] NumPredMode indicates the total number M of prediction modes used for predicting the position information. Note that the maximum value MaxM of the possible values of the number of prediction modes may be specified by a standard or the like. The three-dimensional data encoding device may add the value of (MaxM - M) (0 < M <= MaxM) as NumPredMode to the header, and binarize and encode (MaxM - 1) with a truncated unary code. Also, the number of prediction modes NumPredMode may not be added to the bit stream, and may be specified by a profile or level of a standard or the like. Also, the number of prediction modes may be defined by NumNeighborPoint + NumPredMode.

[0420] Thfix is a threshold value for determining whether to fix the prediction mode. The distance d0 between the points p1 and p0 used for prediction and the distance d1 between the points p2 and p1 are calculated, and if the absolute value of the difference distdiff = |d0 - d1| is smaller than the threshold value Thfix[i], the prediction mode is fixed to α. α is a prediction mode for calculating a predicted value using linear prediction, and is "2" in the above embodiment. Note that Thfix may not be added to the bit stream, and may be specified by a profile or level of a standard or the like.

[0421] QP indicates a quantization parameter used when quantizing the position information. The three-dimensional data encoding device may calculate a quantization step from the quantization parameter, and quantize the position information using the calculated quantization step.

[0422] "unique_point_per_leaf" is information indicating whether or not a duplicated point (a point with the same position information) is included in the bitstream. "unique_point_per_leaf=1" indicates that there is no duplicated point in the bitstream. "unique_point_per_leaf=0" indicates that there is one or more duplicated points in the bitstream.

[0423] In this embodiment, the judgment of whether to fix the prediction mode is performed using the absolute difference between the distance d0 and the distance d1, but it is not limited to this and any method may be used. For example, this judgment may be performed by calculating the distance d0 between the point p1 and the point p0, and if the distance d0 is greater than a threshold, it is determined that the point p1 cannot be used for prediction, and the prediction mode value is fixed to "1" (prediction value p0), and if not, the prediction mode may be set. This can improve the coding efficiency while suppressing overhead.

[0424] The above NumNeighborPoint, NumPredMode, Thfix, and unique_point_per_leaf may be entropy coded and added to the header. For example, each value may be binarized and computationally coded. Each value may also be coded at a fixed length to reduce the amount of processing.

[0425] Fig. 76 is a diagram illustrating an example of the syntax of position information. NumOfPoint, child_count, pred_mode, and residual_value[j] in the syntax of Fig. 76 will be described in order.

[0426] NumOfPoint indicates the total number of 3D points contained in the bitstream.

[0427] child_count indicates the number of child nodes that the i-th 3D point (node[i]) has.

[0428] pred_mode indicates a prediction mode for encoding or decoding the position information of the i-th three-dimensional point. pred_mode takes a value from 0 to M-1 (M is the total number of prediction modes). If pred_mode is not in the bitstream (if the condition distdiff>=Thfix[i] &&NumPredMode>1 is not satisfied), pred_mode may be estimated to a fixed value α. α is a prediction mode for calculating a predicted value using linear prediction, and is "2" in the above embodiment. Note that α is not limited to "2" and may be set to any value from 0 to M-1 as an estimated value. In addition, the estimated value when pred_mode is not in the bitstream may be added to a header or the like. In addition, pred_mode may be binarized and arithmetically coded using the number of prediction modes to which the predicted value is assigned, using a truncated unary code.

[0429] In addition, when NumPredMode=1, that is, when the number of prediction modes is 1, the three-dimensional data encoding device may generate a bit stream that does not include a prediction mode value without encoding a prediction mode value indicating the prediction mode. In addition, when the three-dimensional data decoding device acquires a bit stream that does not include a prediction mode value, the three-dimensional data decoding device may calculate a prediction value of a specific prediction mode in the calculation of the prediction value. The specific prediction mode is a predetermined prediction mode.

[0430] residual_value[j] indicates coded data of the prediction residual between the predicted value of the position information. residual_value[0] may indicate element x of the position information, residual_value[1] may indicate element y of the position information, and residual_value[2] may indicate element z of the position information.

[0431] Fig. 77 is a diagram showing another example of the syntax of the position information. The example of Fig. 77 is a modified example of the example of Fig. 76.

[0432] As shown in Fig. 77, pred_mode may indicate a prediction mode for each of the three elements of position information (x, y, z). That is, pred_mode[0] indicates the prediction mode of element x, pred_mode[1] indicates the prediction mode of element y, and pred_mode[2] indicates the prediction mode of element z. pred_mode[0], pred_mode[1], and pred_mode[2] may be added to the bitstream.

[0433] (Embodiment 5) FIG. 78 is a diagram showing an example of a prediction tree used in a three-dimensional data encoding method according to the fifth embodiment.

[0434] In comparison with the fourth embodiment, the fifth embodiment may calculate the depth of each node when generating a prediction tree in the method for generating a prediction tree.

[0435] For example, the root of the prediction tree may be set to depth=0, the child node of the root may be set to depth=1, and the child node may be set to depth=2. In this case, the possible values ​​of pred_mode may be changed according to the value of depth. That is, in setting the prediction mode, the three-dimensional data encoding device may set a prediction mode for predicting each three-dimensional point based on the depth of the hierarchical structure of the three-dimensional point. For example, pred_mode may be limited to a value equal to or less than the value of depth. That is, the set prediction mode value may be set to a value equal to or less than the depth of the hierarchical structure of each three-dimensional point.

[0436] Also, when pred_mode is binarized with a truncated unary code and arithmetic coded according to the number of prediction modes, it may be binarized with a truncated unary code with the number of prediction modes = min(depth, number of prediction modes M). Thereby, when depth < M, the bit length of the binary data of pred_mode can be reduced, and the coding efficiency can be improved.

[0437] In the method for generating a prediction tree, when adding the three-dimensional point A to the prediction tree, an example was shown in which the nearest neighbor point B is searched and the three-dimensional point A is added to the child node of the three-dimensional point B. Here, any method may be used for the method of searching for the nearest neighbor point. For example, the nearest neighbor point may be searched using the kd-tree method. Thereby, the nearest neighbor point can be searched efficiently, and the coding efficiency can be improved.

[0438] Also, the nearest neighbor point may be searched using the nearest neighbor method. Thereby, the nearest neighbor point can be searched while suppressing the processing load, and the balance between the processing amount and the coding efficiency can be achieved. Also, when searching for the nearest neighbor point using the nearest neighbor method, a search range may be set. Thereby, the processing amount can be reduced.

[0439] Also, the three-dimensional data encoding device may quantize and encode the prediction residual residual_value. For example, the three-dimensional data encoding device may add a quantization parameter QP to a header such as a slice, quantize the residual_value using a Qstep calculated from the QP, and binarize the quantized value to perform arithmetic encoding. In this case, the three-dimensional data decoding device may decode the position information by applying inverse quantization to the quantized value of the residual_value using the same Qstep and adding it to the predicted value. In this case, the decoded position information may be added to the prediction tree. As a result, even when quantization is applied, the three-dimensional data encoding device or the three-dimensional data decoding device can calculate the predicted value using the decoded position information, so that the three-dimensional data encoding device can generate a bit stream that can be correctly decoded by the three-dimensional data decoding device. In addition, an example has been shown in which the nearest point of a three-dimensional point is searched for and added to the prediction tree when generating the prediction tree, but this is not necessarily limited to this, and the prediction tree may be generated in any method or order. For example, if the input 3D points are data acquired by LIDAR, a prediction tree may be generated by adding the 3D points in the order they were scanned by LIDAR, which improves prediction accuracy and coding efficiency.

[0440] Fig. 79 is a diagram showing another example of the syntax of the position information. Residual_is_zero, residual_sign, residual_bitcount_minus1, and residual_bit[k] in the syntax of Fig. 79 will be described in order.

[0441] Residual_is_zero is information indicating whether residual_value is 0 or not. For example, residual_is_zero=1 indicates that residual_value is 0, and residual_is_zero=0 indicates that residual_value is not 0. Note that, in the case of pred_mode=0 (no prediction, predicted value 0), residual_value is unlikely to be 0, so residual_is_zero does not need to be coded and added to the bitstream. In the case of pred_mode=0, the three-dimensional data decoding device may estimate that residual_is_zero=0 without decoding residual_is_zero from the bitstream.

[0442] The residual_sign is positive / negative information (sign bit) indicating whether the residual_value is positive or negative. For example, residual_sign=1 indicates that the residual_value is negative, and residual_sign=0 indicates that the residual_value is positive.

[0443] In addition, in the case of pred_mode=0, the predicted value is 0, so the residual_value is always positive or 0. Therefore, the three-dimensional data encoding device may not encode the residual_sign and add it to the bitstream. In other words, when the three-dimensional data encoding device is set to a prediction mode in which the predicted value is calculated to 0, the three-dimensional data encoding device may generate a bitstream that does not include positive / negative information without encoding positive / negative information indicating whether the prediction residual is positive or negative. In the case of pred_mode=0, the three-dimensional data decoding device may not decode the residual_sign from the bitstream and may estimate that residual_sign=0. In other words, when the three-dimensional data decoding device acquires a bitstream that does not include positive / negative information indicating whether the prediction residual is positive or negative, the three-dimensional data decoding device may treat the prediction residual as 0 or a positive number.

[0444] residual_bitcount_minus1 indicates the number of bits in residual_bit minus 1. In other words, residual_bitcount is equal to residual_bitcount_minus1 plus 1.

[0445] residual_bit[k] indicates the kth bit information when the absolute value of residual_value is binarized to a fixed length according to the value of residual_bitcount.

[0446] In addition, when condition A is defined as "unique_point_per_leaf=1 (no duplicated points) when the position information of any one of points p0, p1, and p2 is directly used as the predicted value, as in prediction mode 1," residual_is_zero[0] of element x, residual_is_zero[1] of element y, and residual_is_zero[2] of element z will never all be 0 at the same time, so it is not necessary to add residual_is_zero of any one of the elements to the bitstream.

[0447] For example, the three-dimensional data encoding device may not add residual_is_zero[2] to the bitstream if condition A is true and residual_is_zero[0] and residual_is_zero[1] are 0. In this case, the three-dimensional data decoding device may infer that residual_is_zero[2]=1, which was not added to the bitstream.

[0448] (Modification) In this embodiment, an example is shown in which a prediction tree is generated using position information (x, y, z) of a three-dimensional point, and the position information is encoded and decoded, but this is not necessarily limited to this. For example, predictive encoding using a prediction tree may be applied to encoding attribute information (color, reflectance, etc.) of a three-dimensional point. In addition, the prediction tree generated in encoding the position information may also be used when encoding the attribute information. This eliminates the need to generate a prediction tree when encoding the attribute information, and the amount of processing can be reduced.

[0449] FIG. 80 is a diagram showing an example of a prediction tree configuration commonly used for encoding position information and attribute information.

[0450] As shown in Fig. 80, each node of the prediction tree includes child_count, g_pred_mode, g_residual_value, a_pred_mode, and a_residual_value. g_pred_mode indicates the prediction mode of the position information. g_residual_value indicates the prediction residual of the position information. a_pred_mode indicates the prediction mode of the attribute information. a_residual_value indicates the prediction mode of the attribute information.

[0451] Here, child_count may be shared between the position information and the attribute information, which can reduce overhead and improve coding efficiency.

[0452] In addition, child_count may be added to the position information and the attribute information independently. This allows the three-dimensional data decoding device to decode the position information and the attribute information independently. For example, the three-dimensional data decoding device can decode only the attribute information.

[0453] The three-dimensional data encoding device may generate separate prediction trees for the position information and the attribute information. This allows the three-dimensional data encoding device to generate prediction trees suitable for each of the position information and the attribute information, thereby improving encoding efficiency. In this case, the three-dimensional data encoding device may add information (such as child_count) required for the three-dimensional data decoding device to reconstruct the prediction trees for each of the position information and the attribute information, to the bit stream. The three-dimensional data encoding device may add identification information indicating whether or not the prediction tree is shared between the position information and the attribute information, to a header or the like. This allows adaptive switching between sharing the prediction tree between the position information and the attribute information, thereby controlling the balance between encoding efficiency and low processing volume.

[0454] FIG. 81 is a flowchart showing an example of a three-dimensional data encoding method according to the modification of the fifth embodiment.

[0455] The three-dimensional data encoding device generates a prediction tree using position information of a plurality of three-dimensional points (S9761).

[0456] Next, the three-dimensional data encoding device encodes the node information and the prediction residual of the position information included in each node of the prediction tree (S9762). Specifically, the three-dimensional data encoding device calculates a prediction value for predicting the position information of each node, calculates a prediction residual that is the difference between the calculated prediction value and the position information of the node, and encodes the prediction residual of the node information and the position information.

[0457] Next, the three-dimensional data encoding device encodes the node information and the prediction residual of the attribute information included in each node of the prediction tree (S9763). Specifically, the three-dimensional data encoding device calculates a prediction value for predicting the attribute information of each node, calculates a prediction residual that is the difference between the calculated prediction value and the attribute information of the node, and encodes the prediction residual of the node information and the attribute information.

[0458] FIG. 82 is a flowchart showing an example of a three-dimensional data decoding method according to the modification of the fifth embodiment.

[0459] The three-dimensional data decoding device decodes node information to reconstruct the prediction tree (S9771).

[0460] Next, the three-dimensional data decoding device decodes the position information of the nodes (S9772). Specifically, the three-dimensional data decoding device calculates the predicted value of the position information of each node, and adds the calculated predicted value and the obtained prediction residual to decode the position information.

[0461] Next, the three-dimensional data decoding device decodes the attribute information of the nodes (S9773). Specifically, the three-dimensional data decoding device calculates the predicted value of the attribute information of each node, and adds the calculated predicted value and the obtained prediction residual to decode the position information.

[0462] Next, the three-dimensional data decoding device determines whether the decoding of all nodes is completed (S9774). If the decoding of all nodes is completed, the three-dimensional data decoding method ends. If the decoding of all nodes is not completed, steps S9771 to S9773 are executed for the unprocessed nodes.

[0463] FIG. 83 is a diagram showing an example of the syntax of the header of the attribute information. The NumNeighborPoint, NumPredMode, Thfix, QP, and unique_point_per_leaf in the syntax of FIG. 83 will be described in order.

[0464] NumNeighborPoint indicates the upper limit value of the number of surrounding points used for generating the predicted value of the attribute information of the three-dimensional points. When the number of surrounding points M is less than NumNeighborPoint (M < NumNeighborPoint), in the calculation process of the predicted value, the predicted value may be calculated using M surrounding points.

[0465] NumPredMode indicates the total number M of prediction modes used for predicting attribute information. Note that the maximum value MaxM that the number of prediction modes can take may be specified by a standard or the like. The three-dimensional data encoding device may add the value of (MaxM - M) (0 < M <= MaxM) as NumPredMode to the header, and binarize and encode (MaxM - 1) with a truncated unary code. Also, the number of prediction modes NumPredMode may not be added to the bitstream, and may be specified by a profile or level in a standard or the like. Also, the number of prediction modes may be defined by NumNeighborPoint + NumPredMode.

[0466] Thfix is a threshold for determining whether to fix the prediction mode. The distance d0 between the points p1 and p0 used for prediction and the distance d1 between the points p2 and p1 are calculated, and if the absolute difference distdiff = |d0 - d1| is smaller than the threshold Thfix[i], the prediction mode is fixed to α. α is a prediction mode for calculating a predicted value using linear prediction, and is "2" in the above embodiment. Note that Thfix may not be added to the bitstream, and may be specified by a profile or level in a standard or the like.

[0467] QP indicates the quantization parameter used when quantizing attribute information. The three-dimensional data encoding device may calculate a quantization step from the quantization parameter and quantize the attribute information using the calculated quantization step.

[0468] unique_point_per_leaf is information indicating whether the bitstream contains a duplicated point (a point with the same position information). unique_point_per_leaf = 1 indicates that there is no duplicated point in the bitstream. unique_point_per_leaf = 0 indicates that there is one or more duplicated points in the bitstream.

[0469] In this embodiment, the judgment of whether to fix the prediction mode is performed using the absolute difference between the distance d0 and the distance d1, but it is not limited to this and any method may be used. For example, this judgment may be performed by calculating the distance d0 between the point p1 and the point p0, and if the distance d0 is greater than a threshold, it is determined that the point p1 cannot be used for prediction, and the prediction mode value is fixed to "1" (prediction value p0), and if not, the prediction mode may be set. This can improve the coding efficiency while suppressing overhead.

[0470] The above NumNeighborPoint, NumPredMode, Thfix, or unique_point_per_leaf can be shared with the location information and do not need to be added to the attribute_header. This can reduce overhead.

[0471] The above NumNeighborPoint, NumPredMode, Thfix, and unique_point_per_leaf may be entropy coded and added to the header. For example, each value may be binarized and computationally coded. Each value may also be coded at a fixed length to reduce the amount of processing.

[0472] Fig. 84 is a diagram showing another example of the syntax of attribute information. NumOfPoint, child_count, pred_mode, dimension, residual_is_zero, residual_sign, residual_bitcount_minus1, and residual_bit[k] in the syntax of Fig. 84 will be described in order.

[0473] NumOfPoint indicates the total number of 3D points included in the bit stream. NumOfPoint may be common to NumOfPoint of position information.

[0474] The child_count indicates the number of child nodes of the i-th 3D point (node[i]). The child_count may be shared with the child_count of the position information. If the child_count is shared with the child_count of the position information, the child_count does not need to be added to the attribute_data. This can reduce overhead.

[0475] pred_mode indicates a prediction mode for encoding or decoding the position information of the i-th three-dimensional point. pred_mode takes a value from 0 to M-1 (M is the total number of prediction modes). If pred_mode is not in the bitstream (if the condition distdiff>=Thfix[i] &&NumPredMode>1 is not satisfied), pred_mode may be estimated to a fixed value α. α is a prediction mode for calculating a predicted value using linear prediction, and is "2" in the above embodiment. Note that α is not limited to "2" and may be set to any value from 0 to M-1 as an estimated value. In addition, the estimated value when pred_mode is not in the bitstream may be added to a header or the like. In addition, pred_mode may be binarized and arithmetically coded using the number of prediction modes to which the predicted value is assigned, using a truncated unary code.

[0476] The dimension is information indicating the dimension of the attribute information. The dimension may be added to a header such as an SPS. For example, if the attribute information is color, the dimension may be set to "3", and if the attribute information is reflectance, the dimension may be set to "1".

[0477] Residual_is_zero is information indicating whether residual_value is 0 or not. For example, residual_is_zero=1 indicates that residual_value is 0, and residual_is_zero=0 indicates that residual_value is not 0. Note that, in the case of pred_mode=0 (no prediction, predicted value 0), residual_value is unlikely to be 0, so residual_is_zero does not need to be coded and added to the bitstream. In the case of pred_mode=0, the three-dimensional data decoding device may estimate that residual_is_zero=0 without decoding residual_is_zero from the bitstream.

[0478] The residual_sign is positive / negative information (sign bit) indicating whether the residual_value is positive or negative. For example, residual_sign=1 indicates that the residual_value is negative, and residual_sign=0 indicates that the residual_value is positive.

[0479] In addition, in the case of pred_mode=0 (no prediction, predicted value 0), since the residual_value is positive, the three-dimensional data encoding device may not encode the residual_sign and add it to the bit stream. That is, when the prediction residual is positive, the three-dimensional data encoding device may generate a bit stream that does not include positive / negative information without encoding positive / negative information indicating whether the prediction residual is positive or negative, and may generate a bit stream that includes positive / negative information when the prediction residual is negative. In the case of pred_mode=0, the three-dimensional data decoding device may not decode the residual_sign from the bit stream and may estimate that residual_sign=0. That is, when the three-dimensional data decoding device acquires a bit stream that does not include positive / negative information indicating whether the prediction residual is positive or negative, the three-dimensional data decoding device may treat the prediction residual as a positive number, and when the three-dimensional data decoding device acquires a bit stream that includes positive / negative information, the prediction residual may be treated as a negative number.

[0480] residual_bitcount_minus1 indicates the number of bits in residual_bit minus 1. In other words, residual_bitcount is equal to residual_bitcount_minus1 plus 1.

[0481] residual_bit[k] indicates the kth bit information when the absolute value of residual_value is binarized to a fixed length according to the value of residual_bitcount.

[0482] In addition, when condition A is defined as "unique_point_per_leaf=1 (no duplicated points) when the attribute information of any one of points p0, p1, and p2 is directly used as the predicted value, as in prediction mode 1," residual_is_zero[0] of element x, residual_is_zero[1] of element y, and residual_is_zero[2] of element z will never all be 0 at the same time, so it is not necessary to add residual_is_zero of any one of the elements to the bitstream.

[0483] For example, the three-dimensional data encoding device may not add residual_is_zero[2] to the bitstream if condition A is true and residual_is_zero[0] and residual_is_zero[1] are 0. In this case, the three-dimensional data decoding device may infer that residual_is_zero[2]=1, which was not added to the bitstream.

[0484] FIG. 85 is a diagram showing an example of the syntax of the position information and attribute information.

[0485] As shown in Figure 85, encoded information of position information and attribute information may be stored in one data unit. Here, g_* indicates encoded information related to geometry, and a_* indicates encoded information related to attribute information. This allows the position information and attribute information to be decoded simultaneously.

[0486] As described above, the three-dimensional data encoding device according to one aspect of the present embodiment performs the process shown in FIG. 86. The three-dimensional data encoding device executes a three-dimensional data encoding method for encoding a plurality of three-dimensional points having a hierarchical structure. The three-dimensional data encoding device sets one prediction mode out of two or more prediction modes for calculating a predicted value of first position information of a first three-dimensional point using second position information of one or more second three-dimensional points around the first three-dimensional point (S9781). Next, the three-dimensional data encoding device calculates a predicted value of the set prediction mode (S9782). Next, the three-dimensional data encoding device calculates a prediction residual which is a difference between the first position information and the calculated predicted value (S9783). Next, the three-dimensional data encoding device generates a first bitstream including the set prediction mode and the prediction residual (S9784). In the setting (S9781), the prediction mode is set based on the depth of the hierarchical structure of the first three-dimensional point.

[0487] According to this, the position information can be encoded using a predicted value of one prediction mode out of two or more prediction modes, which is set based on the depth of the hierarchical structure, thereby improving the encoding efficiency of the position information.

[0488] For example, in the setting (S9784), the 3D data encoding device sets a prediction mode value that is equal to or less than the depth value of the hierarchical structure of the first 3D point. The prediction mode value indicates the prediction mode.

[0489] For example, the first bitstream further includes a prediction mode number indicating the number of the two or more prediction modes.

[0490] For example, in the generating (S9784), the 3D data encoding device encodes a prediction mode value indicating the set prediction mode using the number of prediction modes. The first bitstream includes the encoded prediction mode value as the set prediction mode.

[0491] For example, in the generation (S9784), the prediction mode value is coded using a truncated unary code in which the number of prediction modes is set to a maximum value, thereby reducing the amount of coding for the prediction mode value.

[0492] For example, each of the first position information and the second position information includes three elements. In the setting (S9781), the three-dimensional data encoding device sets a common prediction mode for the three elements as the one prediction mode for calculating a predicted value of each of the three elements included in the first position information. Therefore, the amount of coding of the prediction mode value can be reduced.

[0493] For example, each of the first position information and the second position information includes three elements. In the setting, the three-dimensional data encoding device sets an independent prediction mode for each of the three elements included in the first position information as the one prediction mode for calculating a predicted value of each of the three elements. Therefore, the three-dimensional data decoding device can decode each element independently.

[0494] For example, each of the first position information and the second position information includes three elements. In the setting, the three-dimensional data encoding device sets a common prediction mode for two of the three elements as the one prediction mode for calculating a predicted value of each of the three elements included in the first position information, and sets a prediction mode independent of the two elements for the remaining element. Therefore, it is possible to reduce the amount of coding of the prediction mode values ​​for the two elements. In addition, the three-dimensional data decoding device can independently decode the remaining element.

[0495] For example, in the generation, when the number of prediction modes is 1, the three-dimensional data encoding device does not encode a prediction mode value indicating the prediction mode, and generates a second bitstream that does not include the prediction mode value. This makes it possible to reduce the amount of code in the bitstream.

[0496] For example, in the generation, when a prediction mode in which the prediction value calculated in the calculation is 0 is set, the three-dimensional data encoding device does not encode positive / negative information indicating whether the prediction residual is positive or negative, and generates a third bitstream that does not include the positive / negative information. This makes it possible to reduce the amount of code in the bitstream.

[0497] For example, the three-dimensional data encoding device includes a processor and a memory, and the processor uses the memory to perform the above-mentioned processing.

[0498] Moreover, the three-dimensional data decoding device according to one aspect of the present embodiment performs the process shown in FIG. 87. The three-dimensional data decoding device executes a three-dimensional data decoding method for decoding a plurality of three-dimensional points having a hierarchical structure. The three-dimensional data decoding device acquires a first bit stream including a prediction mode of a first three-dimensional point among the plurality of encoded three-dimensional points and an encoded prediction residual (S9791). Next, the three-dimensional data decoding device decodes a prediction mode value indicating the encoded prediction mode and the encoded prediction residual (S9792). Next, the three-dimensional data decoding device calculates a prediction value of a prediction mode indicated by the prediction mode value obtained by decoding (S9793). Next, the three-dimensional data decoding device calculates first position information of the first three-dimensional point by adding the prediction value and the prediction residual obtained by decoding (S9794). The encoded prediction mode included in the first bit stream is a prediction mode set based on the depth of the hierarchical structure of the first three-dimensional point.

[0499] This makes it possible to appropriately decode the position information that is coded using a predicted value of one prediction mode that is set based on the depth of the hierarchical structure among two or more prediction modes.

[0500] For example, a prediction mode value indicating an encoded prediction mode included in the first bitstream is less than or equal to a depth value of the hierarchical structure of the first 3D point.

[0501] For example, the first bitstream includes a prediction mode number indicating the number of the two or more prediction modes.

[0502] For example, in the decoding (S9792), the three-dimensional data decoding device decodes the encoded prediction mode value using a truncated unary code in which the number of prediction modes is set to a maximum value.

[0503] For example, each of the first position information and second position information of one or more second 3D points around the first 3D point includes three elements, and the prediction mode is used to calculate a predicted value of each of the three elements included in the first position information and is set commonly for the three elements.

[0504] For example, each of the first position information and the second position information of one or more second 3D points around the first 3D point includes three elements, and the prediction mode is used to calculate a predicted value of each of the three elements included in the first position information and is set independently for each of the three elements.

[0505] For example, each of the first position information and second position information of one or more second 3D points around the first 3D point includes three elements, and the prediction mode is used to calculate a predicted value of each of the three elements included in the first position information, and is set commonly for two of the three elements, and is set independently of the two elements for the remaining element.

[0506] For example, if the three-dimensional data decoding device acquires, in the acquisition (S9791), a second bitstream that does not include the prediction mode value, in the calculation of the prediction value, a prediction value of a specific prediction mode is calculated.

[0507] For example, if the three-dimensional data decoding device acquires a third bitstream in the acquisition (S9791) that does not include positive / negative information indicating whether the prediction residual is positive or negative, in calculating the first position information (S9794), the prediction residual is treated as 0 or a positive number.

[0508] For example, the three-dimensional data decoding device includes a processor and a memory, and the processor uses the memory to perform the above-mentioned processing.

[0509] (Embodiment 6) FIG. 88 is a diagram showing an example of point cloud data obtained by LiDAR. A point cloud obtained by scanning with LiDAR is usually sparse and has specific characteristics depending on the specifications of the LiDAR, such as 16 layers, 32 layers, 64 layers, and 128 layers. The three-dimensional data encoding device compresses the position (geometry) information of the point cloud using point cloud compression with octree. The three-dimensional data encoding device uses point cloud compression with prediction for the sparse point cloud. The three-dimensional data encoding device also uses the characteristics of the sparse point cloud (such as information obtained from LiDAR) to build a prediction tree.

[0510] 89 is a block diagram showing the configuration of a three-dimensional data encoding device according to this embodiment. The three-dimensional data encoding device includes a rearrangement unit 10001, a prediction tree generation unit 10002, a prediction conversion unit 10003, a quantization unit 10004, and an entropy encoding unit 10005.

[0511] The sorting unit 10001 sorts multiple pieces of position information of the input point cloud based on a predetermined criterion. The prediction tree generation unit 10002 generates a prediction tree for multiple pieces of position information. Here, the prediction tree indicates a reference relationship in the prediction of multiple pieces of position information. Specifically, the prediction tree generation unit 10002 generates the prediction tree using a configuration file. The configuration file is information on the hardware specifications of the LiDAR or the like that generated the point cloud, and indicates, for example, horizontal and vertical angle information (angular resolution), the number of horizontal scans, and the total number of nodes.

[0512] The prediction conversion unit 10003 uses a prediction tree to generate a prediction value of a target point, which is a 3D point to be processed, and calculates a prediction residual, which is a difference between the position information of the target point and the predicted value. For example, the prediction conversion unit 10003 generates a prediction value using a parent node, a grandfather node, a great-grandfather node, or a combination thereof, which are indicated in the prediction tree.

[0513] The quantization unit 10004 quantizes the prediction residual generated by the prediction conversion unit 10003. The entropy coding unit 10005 generates a coded stream by entropy coding (arithmetic coding) the quantized prediction residual.

[0514] The 3D data encoder builds a prediction tree with good correlation with other neighboring point clouds, allowing for efficient encoding of sparse point cloud location information obtained by LiDAR and other sensors with high compression gain.

[0515] The sparse point cloud is generated via a rotating laser such as a LiDAR. The three-dimensional data encoding device generates a prediction tree using information indicated in a configuration file or product specifications, such as the angular resolution of the vertical and horizontal axes, and the range (e.g., the number of horizontal layers). The three-dimensional data encoding device may also use the field of view, the range, and the relative distance between points. The three-dimensional data encoding device generates a prediction tree for the point cloud using this information.

[0516] The three-dimensional data encoding device may perform a transform process on the prediction residual and quantize the transformed prediction residual. This transform may be a discrete cosine transform (DCT) or a Haar transform. This transform may be performed using other methods such as a method of referring to a transform table. A prediction method selected from a plurality of prediction methods (prediction modes) may be used to generate a predicted value. For example, the plurality of prediction methods differ in the nodes to be referred to, the number of nodes, or the calculation method.

[0517] The setting file is generated by the user, for example, and a bitstream is generated according to the user's specifications.

[0518] Fig. 90 is a block diagram showing another example of the configuration of a three-dimensional data encoding device. The three-dimensional data encoding device shown in Fig. 90 includes a sensor detection unit 10006 in addition to the configuration of the three-dimensional data encoding device shown in Fig. 89.

[0519] The sensor detection unit 10006 determines the characteristics of the LiDAR hardware by comparing adjacent point cloud data and obtaining information such as the number of layers, resolution, and total number of point clouds per frame.

[0520] If the sensor detection unit 10006 knows that the input point cloud data is a three-dimensional point cloud obtained with a specific LiDAR, or if the user runs a test to confirm that it is a three-dimensional point cloud obtained with a specific LiDAR, it can start preprocessing after the sorting process.

[0521] Here, the sorting process is a process for arranging nodes based on a specific arrangement format. For example, the x and y coordinates are arranged in order from smallest to largest for each layer of the rotational scanning layer of the LiDAR based on the z-axis or horizontal layer value. As another method, the absolute value of the distance from a certain point to each point may be calculated, and the point cloud may be sorted using the obtained absolute value of the distance.

[0522] Alternatively, the sorting process may be performed using known LiDAR hardware operating characteristics. In the case of using LiDAR characteristics, a normalization process may be applied to improve the accuracy of determining the horizontal or vertical angle information. The extracted information is then used to generate a prediction tree.

[0523] When the input point cloud is a point cloud obtained by scanning with a plurality of layers of LiDAR, the three-dimensional data encoding device may individually scan the point clouds of each LiDAR to form a prediction tree. For example, when the input point cloud includes point clouds generated by LiDAR1 and LiDAR2, a prediction tree 1 for point cloud 1 acquired by LiDAR1 and a prediction tree 2 for point cloud 2 acquired by LiDAR2 are generated. In addition, the three-dimensional data encoding device may individually encode point cloud 1 and point cloud 2. Similarly, the three-dimensional data decoding device may individually decode point cloud 1 and point cloud 2. As a result, the three-dimensional data encoding device can generate a prediction tree appropriate for each point cloud, thereby improving the encoding efficiency.

[0524] The three-dimensional data encoding device may also store the encoded data encoded using prediction tree 1 and the encoded data encoded using prediction tree 2 in one bit stream. In this case, the three-dimensional data encoding device may add at least one of the start position of the encoded data related to prediction tree 1 in the bit stream and the start position of the encoded data related to prediction tree 2 to the header of the bit stream. This allows the three-dimensional data decoding device to decode the encoded data of prediction tree 1 and the encoded data of prediction tree 2 in parallel, thereby reducing the decoding processing time. The three-dimensional data decoding device can also selectively decode, for example, the encoded data related to prediction tree 2 by reading the start position of the encoded data of each prediction tree from the header.

[0525] A concrete example of a prediction tree will be described below. A sparse point cloud can be projected onto a specific view (plane). FIG. 91 is a top view of a LiDAR point cloud projected onto a plane. The three-dimensional data encoding device generates a prediction tree using the characteristics of the projected point cloud. In the following, each three-dimensional point included in the point cloud is also called a node.

[0526] FIG. 92 is a diagram showing the parent-child relationships of multiple points. The three-dimensional data encoding device generates a prediction tree using the parent-child relationships. Here, the child node (p0) is the three-dimensional point to be processed. The parent node (p1) and the grandparent node (p2) are three-dimensional points that have already been encoded. The relationships between the child node, the parent node, and the grandparent node are also shown in the prediction tree.

[0527] The three-dimensional data encoding device calculates a predicted value from position information of a parent node or a grandparent node. For example, the predicted value may be calculated using a difference (p1-p2) between the position information of the parent node and the position information of the grandparent node. The three-dimensional data encoding device calculates a prediction residual, which is the difference between the position information of a child node and the predicted value. The smaller the prediction residual, the more the encoding efficiency can be improved.

[0528] Next, a method for generating a prediction tree using angle information will be described. For example, a prediction model can be generated on a plane by using the characteristics of a rotating laser of a LiDAR. FIG. 93 is a top view of a point cloud. FIG. 94 is a diagram showing the relationship between point clouds. As shown in FIG. 94, the coordinates of the grandfather node are (x, y), the coordinates of the parent node are (x', y'), and the coordinates of the child node are (x'', y''). In addition, the coordinates of the center position corresponding to the sensor position (LiDAR position) are (Cx, Cy). Note that these coordinates are two-dimensional coordinates in the xy plane. In addition, for example, the angle information θ indicating the angular resolution in the rotational scanning of the LiDAR is acquired, for example, from a setting file. In addition, the vector from the center to the grandfather node is r, the vector from the center to the father node is r', and the vector from the center to the child node is r''.

[0529] The relationship between the direction of r and the direction of r' is determined by θ. The relationship between the direction of r' and the direction of r'' is determined by θ. The magnitude of r' (the distance between the center and the father node) is expressed as the sum of the magnitude of r (the distance between the center and the grandfather node) and the residual. The magnitude of r'' (the distance between the center and the child node) is expressed as the sum of the magnitude of r' (the distance between the center and the father node) and the residual. Here, the residual is the error between the predicted value and the actual value, and the higher the prediction accuracy, the smaller the residual will be.

[0530] Here, the following relationships (Equation U1) to (Equation U3) hold.

[0531] x 2 +y 2 =r 2 ...(Formula U1) x'=Cx+r×sinθ...(Formula U2) y'=Cy+r×cosθ (equation U3)

[0532] Furthermore, when Cx and Cy are 0, the following (Equation U4) and (Equation U5) hold.

[0533] x''=x'×cosθ-y'×sinθ...(Formula U4) y''=x'×sinθ+y'×cosθ...(Formula U5)

[0534] In this way, the three-dimensional data encoding device can generate a predicted value of a child node from the position information of the parent node and the grandparent node by using the cos function, the sin function, and the angle information θ. Note that although the magnitudes of r, r', and r'' may differ, each node exists in the direction of the corresponding vector.

[0535] The residual is a positive or negative value, and this residual may be encoded. The angle information θ is stored in an SPS (Sequence Parameter Set), a GPS (Geometry Parameter Set), a slice header, or the like.

[0536] Also, here, an example of calculating the predicted value using (Equation U1) to (Equation U5) has been shown, but the calculation method of the predicted value is not necessarily limited to this. Any equation may be used as long as it is a method of generating a predicted value using the angle information θ of the LiDAR. For example, as in (Equation U4) and (Equation U5), a predicted value may be generated using θ as a rotation angle. In other words, the three-dimensional data encoding device may calculate a predicted value of a child node by rotating the position information of the previous point (e.g., a parent node) by an angle θ with a radius r' around the sensor position.

[0537] Note that the angle information θ is, for example, a value based on the specifications of the LiDAR, and the angle between two actual consecutive points (for example, a parent node and a child node) does not necessarily match θ because it includes variations.

[0538] The three-dimensional data encoding device may select a prediction method to be used from a plurality of prediction methods (prediction modes) including a prediction method using the angle information θ of the LiDAR described here. Here, the plurality of prediction methods may include any known method. For example, the plurality of prediction methods include a mode in which the position information of any one of the parent node, the grandfather node, and the great-grandfather node is used as a predicted value as is, and a method in which a predicted value is calculated using position information of at least two of the parent node, the grandfather node, and the great-grandfather node. The three-dimensional data encoding device may select a prediction method that minimizes the amount of generated code from the plurality of prediction methods. This can improve the encoding efficiency.

[0539] In addition, the above describes a method for generating a predicted value using angle information θ, but the three-dimensional data encoding device may use a similar method for horizontal angle information α from the base layer to the upper layer. That is, in the above, an example using angle information in the xy plane was described, but a similar method may be used for the z direction. FIG. 95 is a top view of the LiDAR point cloud in this case projected onto a plane. FIG. 96 is a diagram showing the relationship between multiple points.

[0540] As shown in FIG. 96, the coordinates of the grandfather node are (x, y), the coordinates of the parent node are (x', y'), and the coordinates of the child node are (x'', y''). The coordinates of the center position corresponding to the sensor position (LiDAR position) are (Cx, Cy). For example, the angle information α is obtained from a configuration file. The vector from the center to the grandfather node is r, the vector from the center to the father node is r', and the vector from the center to the child node is r''.

[0541] The relationship between the direction of r and the direction of r' is determined by α. The relationship between the direction of r' and the direction of r'' is determined by α. The size of r' (the distance between the center and the father node) is expressed as the sum of the size of r (the distance between the center and the grandfather node) and the residual. The size of r'' (the distance between the center and the child node) is expressed as the sum of the size of r' (the distance between the center and the father node) and the residual.

[0542] Here, the following relationships (Equation U6) to (Equation U8) hold.

[0543] x 2 +y 2 =r 2 ...(Formula U6) x'=Cx+r×sinα...(Formula U7) y'=Cy+r×cosα (equation U8)

[0544] Furthermore, when Cx and Cy are 0, the following (Equation U9) and (Equation U10) hold.

[0545] x''=x'×cosα-y'×sinα (Formula U9) y''=x'×sinα+y'×cosα (Formula U10)

[0546] In this way, the three-dimensional data encoding device can generate a predicted value of a child node from the position information of a parent node and a grandparent node by using the cos function and the sin function together with the angle information α. ​​Note that prediction within the same plane may be combined with prediction on the horizontal plane. This can improve the prediction accuracy.

[0547] The residual may be a positive or negative value, and this residual may be encoded. The angle information α is stored in a sequence parameter set (SPS), a position information parameter set (GPS), a slice header, or the like.

[0548] Also, here, an example of calculating the predicted value using (Equation U6) to (Equation U10) has been shown, but the calculation method of the predicted value is not necessarily limited to this. Any equation may be used as long as it is a method of generating a predicted value using horizontal angle information α of the LiDAR. For example, as in (Equation U9) and (Equation U10), a predicted value may be generated using α as a rotation angle. In other words, the three-dimensional data encoding device may calculate a predicted value of a child node by rotating the position information of the previous point (e.g., a parent node) by an angle α with a radius r' around the sensor position.

[0549] The three-dimensional data encoding device may select a prediction method to be used from a plurality of prediction methods (prediction modes) including a prediction method using the horizontal angle information α of the LiDAR described herein. Here, the plurality of prediction methods may include any known method. For example, the plurality of prediction methods include a mode in which the position information of any one of the parent node, the grandfather node, and the great-grandfather node is used as a predicted value as is, and a method in which a predicted value is calculated using position information of at least two of the parent node, the grandfather node, and the great-grandfather node. The three-dimensional data encoding device may select a prediction method that minimizes the amount of generated code from the plurality of prediction methods. This can improve the encoding efficiency.

[0550] Although the example in which the point cloud data is obtained by LiDAR has been described here, the point cloud data may be obtained by a rotary laser other than LiDAR. Also, the point cloud data may be obtained by a sensor other than a rotary laser (for example, a TOF (Time Of Flight) sensor). In this case, the angle information θ is, for example, the scanning angle of the sensor.

[0551] Next, we will explain the prediction method using virtual nodes. In a sparse 3D point cloud or in certain cases, the location information of a node may be far from its neighbors.

[0552] A virtual node is a virtual node that is inserted to improve correlation in a prediction tree. Fig. 97 is a diagram showing an example of a virtual node in point cloud data. Fig. 98 is a diagram showing an example of a virtual node and a prediction residual. As shown in Fig. 98, when no virtual node is used, the prediction residual is 142. In other words, the prediction residual has a large value.

[0553] In this example, two virtual nodes are added. In addition, two split indicators are added, so that two virtual nodes are generated. In addition, the virtual nodes are empty nodes and do not have a prediction residual. By adding the virtual nodes, the prediction residual of the target node p0 becomes 2, so that the prediction residual can be reduced.

[0554] In addition, the three-dimensional data encoding device determines whether to add a virtual node using a threshold value VRth. Specifically, the three-dimensional data encoding device adds a virtual node when the prediction residual is greater than the threshold value VRth. The threshold value VRth may be set based on a configuration file or may be calculated during the evaluation process of the LiDAR characteristics.

[0555] The three-dimensional data encoding device may also add information indicating this threshold value VRth to the bit stream. Note that the bit stream does not necessarily need to include information indicating the threshold value VRth. The three-dimensional data decoding device does not necessarily need information indicating the threshold value VRth for the decoding process of the position information, but may use this information for reference or for post-processing purposes.

[0556] An example of this determination process is shown below. First, the three-dimensional data encoding device determines whether p0-p1>VRth holds. That is, the three-dimensional data encoding device determines whether the difference between the position information of the target node p0 and the position information of the parent node p1 is greater than the threshold value VRth. If p0-p1>VRth holds, the three-dimensional data encoding device calculates p1-p2=a. Then, the three-dimensional data encoding device calculates n such that p0-p1=n×a, where n is the number of virtual nodes to be inserted.

[0557] For example, when VRth=60 and p0-p1=142, p0-p1=142 is greater than VRth=60. Also, the difference value a=p1-p2=70. Therefore, the number of virtual nodes to be inserted is n=142 / 70=2. That is, n is the quotient obtained by dividing the original prediction residual (p0-p1) by the difference value a. Note that n may be determined so that the absolute value of the difference between (p0-p1) and n×a is the smallest. In this case, the prediction residual of node p0 is p0-p1-2a=2.

[0558] Since the virtual node does not have a prediction residual, the prediction residual of the virtual node is not included in the bit stream. The three-dimensional data encoding device may add information indicating the number of inserted virtual nodes to the bit stream for each three-dimensional point. For example, in the example shown in FIG. 98, the three-dimensional data encoding device adds information indicating that the number of virtual nodes of p1 to p5 is 0 and the number of virtual nodes of p0 is 2 to the bit stream. This allows the three-dimensional data decoding device to generate an appropriate prediction value by correcting the prediction value calculated from the prediction mode according to the number of virtual nodes added to each three-dimensional point. This allows the three-dimensional data decoding device to appropriately decode the bit stream encoded using the virtual nodes.

[0559] In the present embodiment, an example is shown in which the predicted value of p0 is corrected using the number of virtual nodes and the difference value a between the position information of p2 and the position information of p1, but this is not necessarily limited to this. The three-dimensional data encoding device may use any method as long as it is a method for correcting a predicted value according to the number of virtual nodes. For example, the three-dimensional data encoding device may correct the predicted value of p0 using a value obtained by multiplying the difference value between the position information of p3 and the position information of p2 by the number of virtual nodes. In addition, when generating a predicted value of p0, the three-dimensional data encoding device may determine whether or not a virtual node exists between p0 and p1, and if a virtual node exists, may correct the value of the predicted value calculated using at least one or more of p1 to p5 according to the number of virtual nodes. In addition, if a virtual node does not exist, the three-dimensional data encoding device may use a predicted value calculated using at least one or more of p1 to p5 as the predicted value of p0.

[0560] For example, if the difference between the position information of p0 and the position information of p1 is greater than a threshold VRth, the three-dimensional data encoding device may determine that a virtual node exists between p0 and p1, and may calculate the number of virtual nodes, for example, by the above-mentioned method. For example, when two virtual nodes exist between p0 and p1, in a prediction mode in which p1 is the predicted value of p0, the three-dimensional data encoding device corrects the predicted value p1 by adding n times the difference value a between p2 and p1 (n is the number of virtual nodes between p0 and p1) to the predicted value p1. Specifically, the predicted value is p1+a×2. As a result, the prediction residual of p0 is p0-(p1+a×2). Therefore, as shown in Fig. 98, in the case where p2 = 10, p1 = 80, and p0 = 222, if no virtual node is used, the prediction residual of p0 is p0 - p1 = 142, and if the prediction value is corrected using the virtual node, the prediction residual is p0 - (p1 + a x 2) = 222 - (80 + 70 x 2) = 2. Therefore, the prediction residual is reduced and the coding efficiency is improved.

[0561] Although an example of generating a predicted value using the difference value a of position information has been described here, a predicted value may be generated using angle information θ. For example, a first virtual node may be generated by rotating the sensor position at a radius r' (r' is the distance between the center and p1) by an angle θ, and a second virtual node may be generated by rotating the sensor position at a radius r' by an angle 2θ.

[0562] In addition, the three-dimensional data encoding device x , a y , a z 99 is a diagram showing an example of the configuration of the difference values ​​and the bit stream in this case.

[0563] For example, the three-dimensional data encoding device acquires the immediately preceding difference value ax and threshold value VRthx from a setting file, an SPS, a GPS, or the like.

[0564] For example, another difference value a y is the difference between the location information of the parent node and the grandparent node. The threshold VRthy is a y For example, VRthy = 0.5 × a y In addition, the difference value a z To calculate the threshold value VRthz, an equation or function different from the above may be used.

[0565] In this case, the three-dimensional data encoding device uses a plurality of difference values ​​and virtual node thresholds. Also, different division indicators included in the bit stream use identifiers such as 001 or 002. For example, the value of the prediction residual is a value between 0 and 255.

[0566] Also, the above split indicator may be added to the SPS, GPS or slice header to distinguish the difference value to be used.

[0567] In addition, Rate Distortion Optimization (RDO) may be used to verify whether the best prediction has been selected and to compare the use of virtual nodes with the direct coding of the prediction residual from p1 to p0 without using virtual nodes.

[0568] Next, direct encoding will be described. In sparse 3D point clouds or in certain cases, the location information of a node may be far from its neighbors.

[0569] If a suitable prediction value cannot be generated, the three-dimensional data encoding device encodes the target node as is without using a prediction value using direct encoding, and starts generating a new prediction tree starting from the target node.

[0570] Fig. 100 is a diagram showing an example of point cloud data in this case. Fig. 101 is a diagram showing a reference relationship of each node. For example, the three-dimensional data encoding device uses a threshold DCth to determine whether or not to use direct encoding. Specifically, when the prediction residual is larger than the threshold DCth, the three-dimensional data encoding device determines to use direct encoding.

[0571] For example, if p0-p1>DCth is satisfied, the three-dimensional data encoding device uses direct encoding. Also, the three-dimensional data encoding device may evaluate the effectiveness of direct encoding using the RDO, in the same way as in the case of using virtual nodes.

[0572] Next, we explain how to generate a prediction tree using ray tracing. One common way to build a prediction tree is to use the location information of the nearest neighbors as the prediction value. Under normal circumstances, this prediction method produces a good correlation between the locations of the 3D point cloud in most cases.

[0573] However, in some cases, an undesirable prediction tree may be generated. FIG. 102 is a diagram showing an example of point cloud data and a prediction relationship. As shown in FIG. 102, when position information of a nearby point is used, an inappropriate prediction may be performed. For example, when a prediction tree in a horizontal LiDAR scan intersects with another prediction tree in the vertical direction, a point with a low correlation may be used for prediction. This reduces the coding efficiency.

[0574] Here, the horizontal prediction tree and the vertical prediction tree are approximately straight lines. Therefore, by selecting a predicted value in the horizontal or vertical direction, it is possible to prevent the prediction tree from being generated in a zigzag manner, thereby improving the coding efficiency.

[0575] Next, the operation of the three-dimensional data encoding device will be described with reference to Fig. 103, which is a flowchart showing the three-dimensional data encoding process performed by the three-dimensional data encoding device.

[0576] The three-dimensional data encoding device starts predictive tree position information encoding using the configuration file (S10001). Next, the three-dimensional data encoding device determines whether hardware information can be acquired from the configuration file (S10002). The hardware information includes angle information (horizontal / vertical scanning angle information), resolution, range, etc.

[0577] If the hardware information cannot be acquired from the setting file (No in S10002), the three-dimensional data encoding device determines whether the characteristic evaluation of the input point cloud data was successful (S10003).

[0578] If the characteristic evaluation of the input point cloud data is not successful (No in S10003), that is, if angle information, etc. cannot be obtained from either the setting file or the characteristic evaluation, the three-dimensional data encoding device generates a normal prediction tree (not using angle information) (S10004). In addition, the three-dimensional data encoding device encodes the point cloud data using the generated prediction tree (S10005).

[0579] On the other hand, if angle information can be acquired from the setting file or characteristic evaluation (Yes in S10002 or Yes in S10003), the three-dimensional data encoding device performs angle prediction (FIGS. 93 and 94, etc.) using the angle information (S10006). In this case, the three-dimensional data encoding device stores information related to angle prediction, such as the angle information, in the SPS or GPS.

[0580] Furthermore, if a virtual node is available (Yes in S10007), the three-dimensional data encoding device performs virtual node prediction (FIGS. 97 and 98, etc.) (S10008). Furthermore, if direct encoding is available (Yes in S10009), the three-dimensional data encoding device performs direct encoding (FIGS. 100 and 101, etc.) (S10010).

[0581] In addition, the three-dimensional data encoding device may determine which prediction mode to use by using RDO (Rate Distortion Optimization). For example, the three-dimensional data encoding device encodes the position information or attribute information of the three-dimensional point in all prediction modes, and selects the prediction mode that minimizes the amount of generated bits.

[0582] Furthermore, data according to the prediction mode used is stored in the bitstream. Specifically, the bitstream includes information indicating the prediction mode, information on the virtual node, a division indicator, and the like.

[0583] Next, the operation of the three-dimensional data decoding device will be described with reference to Fig. 104, which is a flowchart of the three-dimensional data decoding process performed by the three-dimensional data decoding device.

[0584] First, the three-dimensional data decoding device decodes a bit stream generated by the process shown in Fig. 103, for example, and acquires various information (S10021). Specifically, the three-dimensional data decoding device acquires angle information and the like from an SPS or a GPS.

[0585] Next, the three-dimensional data decoding device starts prediction tree position information decoding (S10022). The three-dimensional data decoding device generates a prediction tree (S10023). Furthermore, if the hardware information and angle information are available (Yes in S10024), the three-dimensional data decoding device performs angle prediction using the angle information (S10025). Furthermore, if the virtual node is available (Yes in S10026), the three-dimensional data decoding device performs virtual node prediction (S10027). Furthermore, if direct encoding (direct decoding) is available (Yes in S10028), the three-dimensional data decoding device performs direct encoding (S10029). For example, the three-dimensional data decoding device determines whether each prediction mode is available or not by using information included in the bit stream.

[0586] Next, the syntax structure of the bitstream will be described. Fig. 105 is a diagram showing an example of the syntax of Header 1 included in the bitstream. Here, Header 1 is, for example, an SPS.

[0587] Header 1 includes an angle information available flag (hardware_specification_available_flag), horizontal angle information (horizontal_angular, θ), and vertical angle information (vertical_angular, α) based on the hardware specifications of the LiDAR. The horizontal angle information and vertical angle information correspond to the angle information θ and α described above. The angle information available flag is information indicating whether angle prediction using angle information is available or not, and when the angle information available flag is on (angle prediction is available), header 1 includes horizontal angle information and vertical angle information.

[0588] This allows the three-dimensional data encoding device and the three-dimensional data decoding device to generate angle prediction values ​​that have the same value. Furthermore, by storing this information in the SPS, which is control information for each sequence (multiple frames), it is used commonly for the entire sequence. Note that this information is used to encode position information as described above, but may also be used to encode attribute information.

[0589] 106 is a diagram showing an example of the syntax of Header 2 included in the bit stream, where Header 2 is, for example, GPS or APS (Attribute Parameter Set).

[0590] The header 2 includes a virtual node enable flag (pt_virtual_node_enable_flag), a direct coding enable flag (pt_direct_code_enable_flag), and a line following enable flag (pt_line_follow_enable_flag).

[0591] The virtual node enable flag (pt_virtual_node_enable_flag) is information indicating whether or not virtual node prediction can be used (whether or not it has been used). The direct coding enable flag (pt_direct_code_enable_flag) is information indicating whether or not direct coding can be used (whether or not it has been used). The line following enable flag (pt_line_follow_enable_flag) is information indicating whether or not a prediction tree has been generated using line addition as shown in FIG. 102 etc.

[0592] Furthermore, when the virtual node permission flag is on (virtual node prediction can be used), the header 2 includes a virtual node threshold (pt_virtual_node_threshold). The virtual node threshold indicates the above-mentioned threshold VRth. When the direct coding permission flag is on (direct coding can be used), the header 2 includes a direct coding threshold (pt_direct_code_threshold). The direct coding threshold indicates the above-mentioned threshold DCth.

[0593] At least one of pt_virtual_node_enable_flag and pt_direct_code_enable_flag may be stored in the SPS. In this case, the same value is used throughout the sequence. This can reduce the amount of code for GPS or APS.

[0594] Fig. 107 is a diagram showing an example of the configuration of a bit stream. As shown in Fig. 107, the bit stream includes an SPS, a GPS, an APS, and slice data (or brick data). The slice data (or brick data) is coded data in slice units (or brick units), and includes coded data of position information (Geom) and coded data of attribute information (Attr(0) and Attr(1)).

[0595] Fig. 108 is a diagram illustrating an example of the syntax of encoded data (geometry_data) of position information. As illustrated in Fig. 108, the encoded data of position information includes the number of virtual nodes (num_virtual_node) and a prediction residual (residual_value[j]).

[0596] The number of virtual nodes (num_virtual_node) indicates the number of virtual nodes at the i-th three-dimensional point. The three-dimensional data encoding device and the three-dimensional data decoding device may correct the predicted value calculated in the prediction mode specified by pred_mode using the value of num_virtual_node. When pred_mode=0 (no prediction), the three-dimensional data encoding device may not need to add num_virtual_node to the bitstream. This allows the amount of code to be reduced when pred_mode=0. Furthermore, when num_virtual_node is not added to the bitstream, the three-dimensional data decoding device may estimate the value to be 0. This allows the three-dimensional data decoding device to properly decode the bitstream.

[0597] The three-dimensional data encoding device may entropy-encode num_virtual_node and then add it to the header. For example, the three-dimensional data encoding device may binarize each value and perform calculation encoding. The three-dimensional data encoding device may also use fixed-length encoding to reduce the amount of processing.

[0598] FIG. 109 is a flowchart of a prediction tree generation process performed by a three-dimensional data encoding device.

[0599] First, the three-dimensional data encoding device calculates the number of virtual nodes of the target three-dimensional point to be encoded (S10041). Note that the three-dimensional data encoding device may omit this process if it does not use virtual nodes (pt_virtual_node_enable_flag=0).

[0600] Next, the three-dimensional data encoding device determines a prediction mode (pred_mode) of the target three-dimensional point (S10042). For example, the three-dimensional data encoding device may determine the prediction mode using RDO. After determining the prediction mode, the three-dimensional data encoding device may generate a predicted value using the determined prediction mode and calculate a prediction residual that is the difference between the original value of the position information or attribute information of the target three-dimensional point and the predicted value.

[0601] Next, the three-dimensional data encoding device adds the target three-dimensional point to the prediction tree (S10043). The three-dimensional data encoding device repeats this series of processes (S10041 to S10043) until all three-dimensional points have been added to the prediction tree (S10044).

[0602] 110 is a flowchart of a prediction tree decoding process by a three-dimensional data decoding device. First, the three-dimensional data decoding device decodes (obtains) the number of virtual nodes from the bit stream (S10051). Note that if a virtual node is not used (pt_virtual_node_enable_flag=0), the three-dimensional data decoding device may omit this process.

[0603] Next, the 3D data decoding device decodes (obtains) the prediction mode (pred_mode) of the target 3D point to be decoded from the bit stream (S10052). Next, the 3D data decoding device calculates a prediction value using the prediction mode indicated by pred_mode, and adds the prediction value to the decoded prediction residual to decode the position information or attribute information of the 3D point (S10053). The 3D data decoding device repeats this series of processes (S10051 to S10053) until the processing of all 3D points is completed (S10054).

[0604] Fig. 111 is a diagram showing a reference relationship in a prediction process. Fig. 112 is a diagram showing an example of a prediction mode. As shown in Fig. 112, the multiple prediction modes may include no prediction, linear prediction, parallelogram prediction, and the like, in addition to the above-mentioned angle prediction and virtual node prediction.

[0605] As described above, the three-dimensional data encoding device according to this embodiment performs the process shown in FIG. 113. First, the three-dimensional data encoding device generates a first predicted value of position information of a first three-dimensional point included in point cloud data obtained by a sensor, using position information of a reference three-dimensional point included in the point cloud data and a scanning angle (e.g., θ) of the sensor (S10061). The three-dimensional data encoding device calculates a first difference value between the position information of the first three-dimensional point and the first predicted value (S10062), and generates a bit stream including the first difference value (e.g., residual_value[j]) and information indicating the scanning angle (e.g., horizontal_angular, θ, or vertical_angular, α) (S10063).

[0606] According to this, the three-dimensional data encoding device can improve prediction accuracy by generating predicted values ​​using the scanning angle of the sensor that generated the point cloud data, thereby improving encoding efficiency.

[0607] For example, the position information of the first three-dimensional point is represented by xyz coordinates, and the scan angle is a scan angle (e.g., θ) in the xy plane. For example, the sensor is a radar (e.g., LiDAR) that performs rotational scanning for each scan angle.

[0608] For example, in generating a first predicted value, a three-dimensional data encoding device generates the first predicted value by rotating and shifting the position information of a reference three-dimensional point (e.g., a parent node) that is immediately before the first three-dimensional point in scanning order, which is included in the point cloud data, by a scanning angle around a reference point that corresponds to the position of the sensor.

[0609] For example, when the first difference value is greater than a predetermined threshold, the three-dimensional data encoding device generates a second predicted value using a value obtained by multiplying a reference predicted value (e.g., a) by n (n is an integer equal to or greater than 1). For example, the three-dimensional data encoding device generates a second predicted value by adding a value obtained by multiplying a reference predicted value by n to the first predicted value (e.g., position information of a parent node). A second difference value between the position information of the first three-dimensional point and the second predicted value is calculated, and the bit stream includes the second difference value (e.g., residual_value[j]) and information indicating n (e.g., num_virtual_node).

[0610] This enables the three-dimensional data encoding device to improve prediction accuracy and therefore improve encoding efficiency.

[0611] For example, the reference predicted value corresponds to a difference (e.g., a) between position information of two reference 3D points that are included in the point cloud data and that are located before the first 3D point in the scanning order. For example, the reference predicted value is generated using a scanning angle (e.g., θ).

[0612] For example, the three-dimensional data encoding device includes a processor and a memory, and the processor uses the memory to perform the above-mentioned processing.

[0613] Moreover, the three-dimensional data decoding device according to this embodiment performs the process shown in FIG. 114. First, the three-dimensional data decoding device acquires from the bit stream a first difference value (e.g., residual_value[j]) between position information of a first three-dimensional point included in point cloud data obtained by a sensor and a first predicted value, and information indicating a scanning angle of the sensor (e.g., horizontal_angular, θ, or vertical_angular, α) (S10071). The three-dimensional data decoding device generates the first predicted value using position information of a reference three-dimensional point included in the point cloud data and the scanning angle of the sensor (S10072), and calculates position information of the first three-dimensional point by adding the first predicted value and the first difference value (S10073).

[0614] According to this, the three-dimensional data decoding device can improve prediction accuracy by generating predicted values ​​using the scanning angle of the sensor that generated the point cloud data, thereby improving encoding efficiency.

[0615] For example, the position information of the first three-dimensional point is represented by xyz coordinates, and the scan angle is a scan angle (e.g., θ) in the xy plane. For example, the sensor is a radar (e.g., LiDAR) that performs rotational scanning for each scan angle.

[0616] For example, in generating a first predicted value, the three-dimensional data decoding device generates the first predicted value by rotating and shifting the position information of a reference three-dimensional point (e.g., a parent node) that is contained in the point cloud data and is located immediately before the first three-dimensional point in scanning order by a scanning angle around a reference point that corresponds to the position of the sensor.

[0617] For example, the three-dimensional data decoding device further acquires from the bit stream a second differential value (e.g., residual_value[j]) between the position information of a second three-dimensional point included in the point cloud data and the second predicted value, and information indicating n (n is an integer greater than or equal to 1) (e.g., num_virtual_node), generates the second predicted value using a value obtained by multiplying a reference predicted value (e.g., a) by n, and calculates the position information of the second three-dimensional point by adding the second predicted value and the second differential value.

[0618] This enables the three-dimensional data decoding device to improve prediction accuracy and therefore improve coding efficiency.

[0619] For example, the reference predicted value corresponds to a difference (e.g., a) between position information of two reference 3D points that are included in the point cloud data and that are located before the first 3D point in the scanning order. For example, the reference predicted value is generated using a scanning angle (e.g., θ).

[0620] For example, the three-dimensional data decoding device includes a processor and a memory, and the processor uses the memory to perform the above-mentioned processing.

[0621] (Embodiment 7) Next, a configuration of a three-dimensional data creation device 810 according to this embodiment will be described. Fig. 115 is a block diagram showing an example of the configuration of a three-dimensional data creation device 810 according to this embodiment. This three-dimensional data creation device 810 is mounted on a vehicle, for example. The three-dimensional data creation device 810 transmits and receives three-dimensional data to and from an external traffic monitoring cloud, a leading vehicle, or a following vehicle, and creates and stores three-dimensional data.

[0622] The three-dimensional data creation device 810 includes a data receiving unit 811, a communication unit 812, a receiving control unit 813, a format conversion unit 814, multiple sensors 815, a three-dimensional data creation unit 816, a three-dimensional data synthesis unit 817, a three-dimensional data storage unit 818, a communication unit 819, a transmission control unit 820, a format conversion unit 821, and a data transmission unit 822.

[0623] The data receiving unit 811 receives three-dimensional data 831 from a traffic monitoring cloud or a preceding vehicle. The three-dimensional data 831 includes information such as a point cloud, a visible light image, depth information, sensor position information, or speed information, including an area that cannot be detected by the sensor 815 of the vehicle itself.

[0624] The communication unit 812 communicates with the traffic monitoring cloud or the vehicle ahead, and transmits data transmission requests, etc. to the traffic monitoring cloud or the vehicle ahead.

[0625] The reception control unit 813 exchanges information such as compatible formats with the communication destination via the communication unit 812, and establishes communication with the communication destination.

[0626] The format conversion unit 814 generates three-dimensional data 832 by performing format conversion or the like on the three-dimensional data 831 received by the data receiving unit 811. Furthermore, when the three-dimensional data 831 is compressed or encoded, the format conversion unit 814 performs decompression or decoding processing.

[0627] The multiple sensors 815 are a group of sensors such as LiDAR, a visible light camera, or an infrared camera that acquires information about the outside of the vehicle, and generate sensor information 833. For example, when the sensor 815 is a laser sensor such as LiDAR, the sensor information 833 is three-dimensional data such as a point cloud (point cloud data). Note that the number of sensors 815 does not need to be multiple.

[0628] The three-dimensional data creation unit 816 generates three-dimensional data 834 from the sensor information 833. The three-dimensional data 834 includes information such as a point cloud, a visible light image, depth information, sensor position information, or velocity information.

[0629] The three-dimensional data synthesis unit 817 synthesizes three-dimensional data 834 created based on the host vehicle's sensor information 833 with three-dimensional data 832 created by a traffic monitoring cloud or a preceding vehicle, etc., to construct three-dimensional data 835 that includes the space in front of the preceding vehicle that cannot be detected by the host vehicle's sensor 815.

[0630] The three-dimensional data storage unit 818 stores the generated three-dimensional data 835 and the like.

[0631] The communication unit 819 communicates with the traffic monitoring cloud or the following vehicle, and transmits data transmission requests, etc. to the traffic monitoring cloud or the following vehicle.

[0632] The transmission control unit 820 exchanges information such as compatible formats with the communication destination and establishes communication with the communication destination via the communication unit 819. In addition, the transmission control unit 820 determines a transmission region, which is the space of the three-dimensional data to be transmitted, based on three-dimensional data construction information of the three-dimensional data 832 generated by the three-dimensional data synthesis unit 817 and a data transmission request from the communication destination.

[0633] Specifically, the transmission control unit 820 determines a transmission region including a space in front of the vehicle that cannot be detected by the sensor of the following vehicle in response to a data transmission request from the traffic monitoring cloud or the following vehicle. The transmission control unit 820 also determines the transmission region by judging whether or not a transmittable space or a transmitted space has been updated based on the three-dimensional data construction information. For example, the transmission control unit 820 determines the region specified in the data transmission request and in which the corresponding three-dimensional data 835 exists as the transmission region. Then, the transmission control unit 820 notifies the format conversion unit 821 of the format supported by the communication destination and the transmission region.

[0634] The format conversion unit 821 converts three-dimensional data 836 of the transmission region, among the three-dimensional data 835 stored in the three-dimensional data storage unit 818, into a format supported by the receiving side, thereby generating three-dimensional data 837. Note that the format conversion unit 821 may reduce the amount of data by compressing or encoding the three-dimensional data 837.

[0635] The data transmission unit 822 transmits the three-dimensional data 837 to the traffic monitoring cloud or the following vehicle. The three-dimensional data 837 includes information such as a point cloud, visible light image, depth information, or sensor position information in front of the vehicle, including blind spots of the following vehicle.

[0636] In the above description, the format conversion units 814 and 821 perform format conversion, but the format conversion does not necessarily have to be performed.

[0637] With this configuration, the three-dimensional data creation device 810 externally acquires three-dimensional data 831 of an area that cannot be detected by the sensor 815 of the host vehicle, and generates three-dimensional data 835 by synthesizing the three-dimensional data 831 with three-dimensional data 834 based on sensor information 833 detected by the sensor 815 of the host vehicle. In this way, the three-dimensional data creation device 810 can generate three-dimensional data of an area that cannot be detected by the sensor 815 of the host vehicle.

[0638] In addition, in response to a data transmission request from a traffic monitoring cloud or a following vehicle, the three-dimensional data creation device 810 can transmit three-dimensional data including the space in front of the vehicle that cannot be detected by the sensors of the following vehicle to the traffic monitoring cloud or the following vehicle, etc.

[0639] Next, there will be described a procedure for transmitting three-dimensional data to a following vehicle in the three-dimensional data creation device 810. Fig. 116 is a flowchart showing an example of a procedure for transmitting three-dimensional data to a traffic monitoring cloud or a following vehicle by the three-dimensional data creation device 810.

[0640] First, the three-dimensional data creation device 810 generates and updates three-dimensional data 835 of a space including the space on the road ahead of the vehicle (S801). Specifically, the three-dimensional data creation device 810 constructs three-dimensional data 835 including the space ahead of the vehicle ahead that cannot be detected by the sensor 815 of the vehicle ahead, for example, by combining three-dimensional data 834 created based on the sensor information 833 of the vehicle ahead with three-dimensional data 831 created by the traffic monitoring cloud or the vehicle ahead.

[0641] Next, the three-dimensional data creation device 810 determines whether the three-dimensional data 835 contained in the transmitted space has changed (S802).

[0642] If a change occurs in the three-dimensional data 835 contained in the space that has already been transmitted, such as when a vehicle or person enters the space from outside (Yes in S802), the three-dimensional data creation device 810 transmits three-dimensional data including the three-dimensional data 835 of the space where the change has occurred to the traffic monitoring cloud or a following vehicle (S803).

[0643] The three-dimensional data creation device 810 may transmit the three-dimensional data of the space where a change has occurred in accordance with the transmission timing of the three-dimensional data transmitted at a predetermined interval, or may transmit the three-dimensional data immediately after detecting the change. In other words, the three-dimensional data creation device 810 may transmit the three-dimensional data of the space where a change has occurred in priority over the three-dimensional data transmitted at a predetermined interval.

[0644] In addition, the three-dimensional data creation device 810 may transmit all of the three-dimensional data of the space in which a change has occurred as the three-dimensional data of the space in which a change has occurred, or may transmit only the differences in the three-dimensional data (for example, information on three-dimensional points that have appeared or disappeared, or displacement information of three-dimensional points, etc.).

[0645] Furthermore, the three-dimensional data creation device 810 may transmit metadata regarding the danger avoidance operation of the vehicle, such as a sudden braking warning, to the following vehicle prior to the three-dimensional data of the space where a change has occurred. This allows the following vehicle to quickly recognize the sudden braking of the vehicle ahead, and to start danger avoidance operations, such as deceleration, earlier.

[0646] If there is no change in the three-dimensional data 835 contained in the transmitted space (No in S802), or after step S803, the three-dimensional data creation device 810 transmits the three-dimensional data contained in a space of a specified shape at a distance L ahead of the vehicle to the traffic monitoring cloud or a following vehicle (S804).

[0647] Furthermore, for example, the processes in steps S801 to S804 are repeatedly performed at predetermined time intervals.

[0648] Furthermore, when there is no difference between the three-dimensional data 835 of the space currently being transmitted and the three-dimensional map, the three-dimensional data creation device 810 does not need to transmit the three-dimensional data 837 of the space.

[0649] In this embodiment, the client device transmits sensor information obtained by the sensor to a server or another client device.

[0650] First, the configuration of the system according to this embodiment will be described. Fig. 117 is a diagram showing the configuration of a transmission / reception system for a 3D map and sensor information according to this embodiment. This system includes a server 901 and client devices 902A and 902B. When there is no particular need to distinguish between the client devices 902A and 902B, they are also referred to as client devices 902.

[0651] The client device 902 is, for example, an in-vehicle device mounted on a moving object such as a vehicle. The server 901 is, for example, a traffic monitoring cloud or the like, and is capable of communicating with a plurality of client devices 902.

[0652] The server 901 transmits a three-dimensional map composed of a point cloud to the client device 902. Note that the composition of the three-dimensional map is not limited to a point cloud, and may represent other three-dimensional data such as a mesh structure.

[0653] The client device 902 transmits sensor information acquired by the client device 902 to the server 901. The sensor information includes, for example, at least one of LiDAR acquisition information, a visible light image, an infrared image, a depth image, sensor position information, and speed information.

[0654] Data transmitted between the server 901 and the client device 902 may be compressed to reduce data, or may be left uncompressed to maintain data accuracy. When compressing data, a three-dimensional compression method based on an octree structure, for example, can be used for the point cloud. Also, a two-dimensional image compression method can be used for the visible light image, the infrared image, and the depth image. The two-dimensional image compression method is, for example, MPEG-4 AVC or HEVC standardized by MPEG.

[0655] Furthermore, the server 901 transmits the three-dimensional map managed by the server 901 to the client device 902 in response to a transmission request for the three-dimensional map from the client device 902. Note that the server 901 may transmit the three-dimensional map without waiting for a transmission request for the three-dimensional map from the client device 902. For example, the server 901 may broadcast the three-dimensional map to one or more client devices 902 in a predetermined space. Furthermore, the server 901 may transmit a three-dimensional map suitable for the position of the client device 902 at regular intervals to the client device 902 that has received a transmission request once. Furthermore, the server 901 may transmit the three-dimensional map to the client device 902 every time the three-dimensional map managed by the server 901 is updated.

[0656] The client device 902 issues a request for transmitting a three-dimensional map to the server 901. For example, when the client device 902 wants to estimate its own position while driving, the client device 902 transmits a request for transmitting a three-dimensional map to the server 901.

[0657] In the following cases, the client device 902 may issue a transmission request for a three-dimensional map to the server 901. If the three-dimensional map held by the client device 902 is old, the client device 902 may issue a transmission request for a three-dimensional map to the server 901. For example, if a certain period of time has passed since the client device 902 acquired the three-dimensional map, the client device 902 may issue a transmission request for a three-dimensional map to the server 901.

[0658] The client device 902 may issue a request to send a three-dimensional map to the server 901 a certain time before the client device 902 leaves the space shown in the three-dimensional map held by the client device 902. For example, when the client device 902 is within a predetermined distance from the boundary of the space shown in the three-dimensional map held by the client device 902, the client device 902 may issue a request to send a three-dimensional map to the server 901. In addition, when the movement route and movement speed of the client device 902 are known, the time when the client device 902 will leave the space shown in the three-dimensional map held by the client device 902 may be predicted based on these.

[0659] If the error in aligning the three-dimensional data created by the client device 902 from sensor information with the three-dimensional map is equal to or greater than a certain level, the client device 902 may issue a request to the server 901 to transmit the three-dimensional map.

[0660] The client device 902 transmits the sensor information to the server 901 in response to a request to transmit the sensor information transmitted from the server 901. The client device 902 may transmit the sensor information to the server 901 without waiting for a request to transmit the sensor information from the server 901. For example, once the client device 902 receives a request to transmit the sensor information from the server 901, the client device 902 may transmit the sensor information to the server 901 periodically for a certain period of time. In addition, when an error occurs in aligning three-dimensional data created by the client device 902 based on the sensor information with the three-dimensional map obtained from the server 901, the client device 902 may determine that a change may have occurred in the three-dimensional map around the client device 902 and transmit a message to that effect along with the sensor information to the server 901.

[0661] The server 901 issues a transmission request for sensor information to the client device 902. For example, the server 901 receives location information of the client device 902, such as a GPS, from the client device 902. When the server 901 determines, based on the location information of the client device 902, that the client device 902 is approaching a space with little information in the three-dimensional map managed by the server 901, it issues a transmission request for sensor information to the client device 902 in order to generate a new three-dimensional map. The server 901 may also issue a transmission request for sensor information when it is desired to update the three-dimensional map, when it is desired to check road conditions such as during snowfall or a disaster, when it is desired to check traffic congestion conditions, or when it is desired to check incident and accident conditions, etc.

[0662] Furthermore, the client device 902 may set the amount of data of the sensor information to be transmitted to the server 901 depending on the communication state or bandwidth at the time of receiving a request to transmit the sensor information from the server 901. Setting the amount of data of the sensor information to be transmitted to the server 901 means, for example, increasing or decreasing the amount of the data itself or appropriately selecting a compression method.

[0663] 118 is a block diagram showing an example of the configuration of a client device 902. The client device 902 receives a three-dimensional map composed of a point cloud or the like from the server 901, and estimates the self-position of the client device 902 from three-dimensional data created based on sensor information of the client device 902. In addition, the client device 902 transmits the acquired sensor information to the server 901.

[0664] The client device 902 includes a data receiving unit 1011, a communication unit 1012, a reception control unit 1013, a format conversion unit 1014, multiple sensors 1015, a three-dimensional data creation unit 1016, a three-dimensional image processing unit 1017, a three-dimensional data storage unit 1018, a format conversion unit 1019, a communication unit 1020, a transmission control unit 1021, and a data transmission unit 1022.

[0665] The data receiving unit 1011 receives a three-dimensional map 1031 from the server 901. The three-dimensional map 1031 is data including a point cloud such as a WLD or a SWLD. The three-dimensional map 1031 may include either compressed data or uncompressed data.

[0666] The communication unit 1012 communicates with the server 901, and transmits data transmission requests (for example, requests to transmit a three-dimensional map) to the server 901.

[0667] The reception control unit 1013 exchanges information such as compatible formats with the communication destination via the communication unit 1012, and establishes communication with the communication destination.

[0668] The format conversion unit 1014 generates a three-dimensional map 1032 by performing format conversion and the like on the three-dimensional map 1031 received by the data receiving unit 1011. Furthermore, if the three-dimensional map 1031 is compressed or encoded, the format conversion unit 1014 performs decompression or decoding processing. Note that if the three-dimensional map 1031 is uncompressed data, the format conversion unit 1014 does not perform decompression or decoding processing.

[0669] The multiple sensors 1015 are a group of sensors, such as LiDAR, a visible light camera, an infrared camera, or a depth sensor, that acquire information about the outside of the vehicle in which the client device 902 is mounted, and generate sensor information 1033. For example, when the sensor 1015 is a laser sensor such as LiDAR, the sensor information 1033 is three-dimensional data such as a point cloud (point cloud data). Note that the number of sensors 1015 does not need to be multiple.

[0670] The three-dimensional data creation unit 1016 creates three-dimensional data 1034 of the surroundings of the vehicle based on the sensor information 1033. For example, the three-dimensional data creation unit 1016 creates point cloud data with color information of the surroundings of the vehicle using information acquired by the LiDAR and a visible light image acquired by a visible light camera.

[0671] The three-dimensional image processing unit 1017 performs self-position estimation processing of the vehicle using a three-dimensional map 1032 such as a received point cloud and three-dimensional data 1034 of the surroundings of the vehicle generated from the sensor information 1033. The three-dimensional image processing unit 1017 may generate three-dimensional data 1035 of the surroundings of the vehicle by combining the three-dimensional map 1032 and the three-dimensional data 1034, and perform self-position estimation processing using the generated three-dimensional data 1035.

[0672] The three-dimensional data storage unit 1018 stores a three-dimensional map 1032, three-dimensional data 1034, and three-dimensional data 1035, and the like.

[0673] The format conversion unit 1019 generates sensor information 1037 by converting the sensor information 1033 into a format supported by the receiving side. The format conversion unit 1019 may reduce the amount of data by compressing or encoding the sensor information 1037. Furthermore, the format conversion unit 1019 may omit processing when format conversion is not necessary. Furthermore, the format conversion unit 1019 may control the amount of data to be transmitted in accordance with a designated transmission range.

[0674] The communication unit 1020 communicates with the server 901, and receives data transmission requests (sensor information transmission requests) and the like from the server 901.

[0675] The transmission control unit 1021 exchanges information such as compatible formats with the communication destination via the communication unit 1020, and establishes communication.

[0676] The data transmission unit 1022 transmits the sensor information 1037 to the server 901. The sensor information 1037 includes information acquired by a plurality of sensors 1015, such as information acquired by a LiDAR, a luminance image acquired by a visible light camera, an infrared image acquired by an infrared camera, a depth image acquired by a depth sensor, sensor position information, and speed information.

[0677] Next, the configuration of the server 901 will be described. Fig. 119 is a block diagram showing an example of the configuration of the server 901. The server 901 receives sensor information transmitted from the client device 902, and creates three-dimensional data based on the received sensor information. The server 901 uses the created three-dimensional data to update a three-dimensional map managed by the server 901. In addition, the server 901 transmits the updated three-dimensional map to the client device 902 in response to a transmission request for the three-dimensional map from the client device 902.

[0678] The server 901 includes a data receiving unit 1111, a communication unit 1112, a receiving control unit 1113, a format conversion unit 1114, a three-dimensional data creation unit 1116, a three-dimensional data synthesis unit 1117, a three-dimensional data storage unit 1118, a format conversion unit 1119, a communication unit 1120, a transmission control unit 1121, and a data transmission unit 1122.

[0679] The data receiving unit 1111 receives sensor information 1037 from the client device 902. The sensor information 1037 includes, for example, information acquired by a LiDAR, a luminance image acquired by a visible light camera, an infrared image acquired by an infrared camera, a depth image acquired by a depth sensor, sensor position information, and speed information.

[0680] The communication unit 1112 communicates with the client device 902 and transmits a data transmission request (for example, a request to transmit sensor information) to the client device 902 .

[0681] The reception control unit 1113 exchanges information such as compatible formats with the communication destination via the communication unit 1112, and establishes communication.

[0682] When the received sensor information 1037 is compressed or encoded, the format conversion unit 1114 performs decompression or decoding processing to generate the sensor information 1132. Note that, when the sensor information 1037 is non-compressed data, the format conversion unit 1114 does not perform decompression or decoding processing.

[0683] The three-dimensional data creation unit 1116 creates three-dimensional data 1134 of the periphery of the client device 902 based on the sensor information 1132. For example, the three-dimensional data creation unit 1116 creates point cloud data with color information of the periphery of the client device 902 using information acquired by the LiDAR and visible light images acquired by a visible light camera.

[0684] A three-dimensional data synthesis unit 1117 synthesizes three-dimensional data 1134 created based on the sensor information 1132 with a three-dimensional map 1135 managed by the server 901, thereby updating the three-dimensional map 1135.

[0685] The three-dimensional data storage unit 1118 stores a three-dimensional map 1135 and the like.

[0686] The format conversion unit 1119 generates the three-dimensional map 1031 by converting the three-dimensional map 1135 into a format supported by the receiving side. The format conversion unit 1119 may reduce the amount of data by compressing or encoding the three-dimensional map 1135. Furthermore, the format conversion unit 1119 may omit processing when format conversion is not necessary. Furthermore, the format conversion unit 1119 may control the amount of data to be transmitted in accordance with a designated transmission range.

[0687] The communication unit 1120 communicates with the client device 902 and receives a data transmission request (a request to transmit a three-dimensional map) or the like from the client device 902 .

[0688] A transmission control unit 1121 exchanges information such as compatible formats with a communication destination via a communication unit 1120, and establishes communication.

[0689] The data transmission unit 1122 transmits the three-dimensional map 1031 to the client device 902. The three-dimensional map 1031 is data including a point cloud such as a WLD or a SWLD. The three-dimensional map 1031 may include either compressed data or uncompressed data.

[0690] Next, a description will be given of the operation flow of the client device 902. Fig. 120 is a flowchart showing the operation of the client device 902 when acquiring a three-dimensional map.

[0691] First, the client device 902 requests the server 901 to transmit a three-dimensional map (such as a point cloud) (S1001). At this time, the client device 902 may also transmit position information of the client device 902 obtained by GPS or the like, thereby requesting the server 901 to transmit a three-dimensional map related to the position information.

[0692] Next, the client device 902 receives the three-dimensional map from the server 901 (S1002). If the received three-dimensional map is compressed data, the client device 902 decodes the received three-dimensional map to generate an uncompressed three-dimensional map (S1003).

[0693] Next, the client device 902 creates three-dimensional data 1034 of the periphery of the client device 902 from sensor information 1033 obtained by the multiple sensors 1015 (S1004). Next, the client device 902 estimates its own position using the three-dimensional map 1032 received from the server 901 and the three-dimensional data 1034 created from the sensor information 1033 (S1005).

[0694] 121 is a flowchart showing an operation of the client device 902 when transmitting sensor information. First, the client device 902 receives a request to transmit sensor information from the server 901 (S1011). The client device 902 that has received the transmission request transmits sensor information 1037 to the server 901 (S1012). Note that, when the sensor information 1033 includes a plurality of pieces of information obtained by a plurality of sensors 1015, the client device 902 may generate the sensor information 1037 by compressing each piece of information using a compression method suitable for each piece of information.

[0695] Next, the operation flow of the server 901 will be described. Fig. 122 is a flowchart showing the operation when the server 901 acquires sensor information. First, the server 901 requests the client device 902 to transmit sensor information (S1021). Next, the server 901 receives the sensor information 1037 transmitted from the client device 902 in response to the request (S1022). Next, the server 901 creates three-dimensional data 1134 using the received sensor information 1037 (S1023). Next, the server 901 reflects the created three-dimensional data 1134 in the three-dimensional map 1135 (S1024).

[0696] Fig. 123 is a...

Claims

1. A method for generating a first predicted value of position information of a first three-dimensional point included in point cloud data using position information and angle information of a reference three-dimensional point included in the point cloud data, Calculating a first difference value between the position information of the first 3D point and the first predicted value by the processor; generating, by the processor, a second predicted value of position information of a second 3D point included in the point cloud data, using the position information of the first 3D point and the angle information; Calculating a second difference value between the position information of the second 3D point and the second predicted value by the processor; generating, by the processor, a bit stream including the first difference value, the second difference value, and the angle information; Three-dimensional data encoding method.

2. The position information of the first 3D point is expressed in xyz coordinates, and the angle information includes an angle in an xy plane.

2. The three-dimensional data encoding method according to claim 1.

3. the angle information includes a scanning angle of a sensor for acquiring the point cloud data; The sensor is a radar that performs rotational scanning for each of the scan angles.

3. The three-dimensional data encoding method according to claim 1 or 2.

4. the angle information includes a scanning angle of a sensor for acquiring the point cloud data; In generating the first predicted value, the processor rotates and moves the position information of the reference 3D point, which is included in the point cloud data and is located immediately before the first 3D point in scanning order, by the scanning angle around a reference point corresponding to the position of the sensor, to generate the first predicted value.

3. The three-dimensional data encoding method according to claim 1 or 2.

5. The position information of the first 3D point is expressed in polar coordinates.

2. The three-dimensional data encoding method according to claim 1.

6. The angle information includes information regarding an angle between the first 3D point and the reference 3D point.

3. The three-dimensional data encoding method according to claim 1 or 2.

7. The angle information includes information regarding angles between a plurality of three-dimensional points included in the point cloud data.

3. The three-dimensional data encoding method according to claim 1 or 2.

8. A processor obtains, from the bit stream, a first difference value between position information of a first three-dimensional point included in the point cloud data and a first predicted value, a second difference value between position information of a second three-dimensional point included in the point cloud data and a second predicted value, and angle information; The processor generates the first predicted value using position information of a reference 3D point included in the point cloud data and the angle information; The processor calculates the position information of the first 3D point by adding the first predicted value and the first difference value; generating, by the processor, the second predicted value using position information of the first 3D point and the angle information; The processor calculates the position information of the second 3D point by adding the second predicted value and the second difference value. A method for decoding three-dimensional data.

9. The position information of the first 3D point is expressed in xyz coordinates, and the angle information includes an angle in an xy plane.

9. The three-dimensional data decoding method according to claim 8.

10. the angle information includes a scanning angle of a sensor for acquiring the point cloud data; The sensor is a radar that performs rotational scanning for each of the scan angles.

10. The three-dimensional data decoding method according to claim 8 or 9.

11. the angle information includes a scanning angle of a sensor for acquiring the point cloud data; In generating the first predicted value, the processor generates the first predicted value by rotating and moving position information of a reference 3D point that is included in the point cloud data and is located immediately before the first 3D point in scanning order by the scanning angle around a reference point that corresponds to the position of the sensor.

10. The three-dimensional data decoding method according to claim 8 or 9.

12. The position information of the first 3D point is expressed in polar coordinates.

9. The three-dimensional data decoding method according to claim 8.

13. The angle information includes information regarding an angle between the first 3D point and the reference 3D point.

10. The three-dimensional data decoding method according to claim 8 or 9.

14. The angle information includes information regarding angles between a plurality of three-dimensional points included in the point cloud data.

10. The three-dimensional data decoding method according to claim 8 or 9.

15. A processor; A memory. The processor uses the memory to: generating a first predicted value of position information of a first 3D point included in the point cloud data using position information and angle information of a reference 3D point included in the point cloud data; Calculating a first difference value between the position information of the first 3D point and the first predicted value; generating a second predicted value of position information of a second 3D point included in the point cloud data using the position information of the first 3D point and the angle information; Calculating a second difference value between the position information of the second 3D point and the second predicted value; generating a bit stream including the first difference value, the second difference value, and the angle information; Three-dimensional data encoding device.

16. A processor; A memory. The processor uses the memory to: obtain, from the bit stream, a first difference value between position information of a first three-dimensional point included in the point cloud data and a first predicted value, a second difference value between position information of a second three-dimensional point included in the point cloud data and a second predicted value, and angle information; generating the first predicted value using position information of a reference 3D point included in the point cloud data and the angle information; Calculating the position information of the first 3D point by adding the first predicted value and the first difference value; generating the second predicted value using position information of the first 3D point and the angle information; The position information of the second 3D point is calculated by adding the second predicted value and the second difference value. Three-dimensional data decoding device.

Images