Encoding method, decoding method, encoding device, and decoding device

Abstract

To provide an encoding method, etc., capable of improving encoding efficiency.SOLUTION: An encoding method is implemented by an encoding device comprising a processor and a memory. The encoding method includes: generating encoded data by encoding a first coefficient value in a plurality of coefficient values using a first parameter and a second parameter; and generating a bitstream including the encoded data, the first parameter and the second parameter. The first coefficient value belongs to a first layer in one or more layers. The first parameter is used for encoding the coefficient value belonging to the first layer. The first coefficient value belongs to a first region in a space, and the second parameter is used for encoding the coefficient value belonging to the first region.SELECTED DRAWING: Figure 152

Description

[Technical Field]

[0001] The present disclosure relates to an encoding method, a decoding method, an encoding device, and a decoding device. [Background technology]

[0002] In the future, devices and services that utilize 3D data are expected to become widespread in a wide range of fields, including computer vision for autonomous operation of automobiles or robots, map information, surveillance, infrastructure inspection, video distribution, etc. 3D data can be acquired in a variety of ways, including distance sensors such as range finders, stereo cameras, or a combination of multiple monocular cameras.

[0003] One method of representing three-dimensional data is a point cloud, which represents the shape of a three-dimensional structure using a group of points in three-dimensional space. A point cloud stores the position and color of the points. Point clouds are expected to become the mainstream method of representing three-dimensional data, but point clouds require a very large amount of data. Therefore, when storing or transmitting three-dimensional data, data compression through encoding is essential, just as with two-dimensional video images (examples include MPEG-4 AVC or HEVC standardized by MPEG).

[0004] In addition, compression of point clouds is partially supported by public libraries that perform point cloud-related processing (Point Cloud Library).

[0005] Furthermore, a technique is known in which three-dimensional map data is used to search for and display facilities located around a vehicle (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0006] [Patent Document 1] International Publication No. 2014 / 020663 Summary of the Invention [Problem to be solved by the invention]

[0007] In the encoding and decoding of three-dimensional data, it is desirable to be able to improve the encoding efficiency.

[0008] An object of the present disclosure is to provide an encoding method, a decoding method, an encoding device, or a decoding device that can improve encoding efficiency. [Means for solving the problem]

[0009] An encoding method according to one embodiment of the present disclosure is an encoding method implemented by an encoding device having a processor and a memory, which encodes a first coefficient value among a plurality of coefficient values ​​using a first parameter and a second parameter to generate encoded data, and generates a bitstream including the encoded data, the first parameter, and the second parameter, wherein the first coefficient value belongs to a first layer among one or more layers, the first parameter is used to encode the coefficient value belonging to the first layer, the first coefficient value belongs to a first region in space, and the second parameter is used to encode the coefficient value belonging to the first region.

[0010] A decoding method according to one embodiment of the present disclosure is a decoding method implemented by a decoding device having a processor and a memory, which obtains coded data, a first parameter, and a second parameter from a bitstream, decodes multiple coded data using the first parameter and the second parameter to generate first coefficient values, the first coefficient values ​​belonging to a first hierarchical layer among one or more hierarchical layers, the first parameter being used in a decoding process to generate coefficient values ​​belonging to the first hierarchical layer, the first coefficient values ​​belonging to a first region in space, and the second parameter being used in a decoding process to generate coefficient values ​​belonging to the first region.

[0011] A three-dimensional data encoding method according to one embodiment of the present disclosure encodes multiple pieces of attribute information of multiple three-dimensional points using parameters, and generates a bitstream including the encoded multiple pieces of attribute information and the parameters, wherein each of the multiple pieces of attribute information belongs to one or more hierarchies, and each of the multiple three-dimensional points belongs to one or more regions, and the parameters are determined according to the hierarchies to which the attribute information to be encoded belongs and the regions to which the three-dimensional points having the attribute information to be encoded belong, and the parameters included in the bitstream include a predetermined reference value, a first difference value determined for each hierarchies, and a second difference value determined for each region.

[0012] A three-dimensional data decoding method according to one embodiment of the present disclosure acquires a bitstream to acquire multiple pieces of encoded attribute information and parameters, and decodes the multiple pieces of encoded attribute information using the parameters, wherein the multiple pieces of attribute information are attribute information of multiple three-dimensional points, each of the multiple pieces of attribute information belonging to one or more hierarchies, and each of the multiple three-dimensional points belonging to one or more regions, and the parameters are determined according to the hierarchies to which the attribute information to be encoded belongs and the regions to which the three-dimensional points having the attribute information to be encoded belong, and the parameters included in the bitstream include a predetermined reference value, a first difference value determined for each of the hierarchies, and a second difference value determined for each of the regions.

[0013] These general or specific aspects may be realized as a system, device, integrated circuit, computer program, or computer-readable recording medium such as a CD-ROM, or may be realized as any combination of a system, device, integrated circuit, computer program, and recording medium. [Effects of the Invention]

[0014] The present disclosure can provide an encoding method, a decoding method, an encoding device, or a decoding device that can improve encoding efficiency. [Brief explanation of the drawings]

[0015] [Figure 1] FIG. 1 is a diagram showing the structure of encoded three-dimensional data according to the first embodiment. [Figure 2] FIG. 2 is a diagram showing an example of a prediction structure between SPCs belonging to the lowest layer of a GOS according to the first embodiment. [Figure 3] FIG. 3 is a diagram showing an example of an inter-layer prediction structure according to the first embodiment. [Figure 4] FIG. 4 is a diagram showing an example of the coding order of the GOS according to the first embodiment. [Figure 5] FIG. 5 is a diagram showing an example of the coding order of GOS according to the first embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of meta information according to the first embodiment. [Figure 7] FIG. 7 is a schematic diagram showing how three-dimensional data is transmitted and received between vehicles according to the second embodiment. [Figure 8] FIG. 8 is a diagram illustrating an example of three-dimensional data transmitted between vehicles according to the second embodiment. [Figure 9] FIG. 9 is a diagram for explaining the transmission process of three-dimensional data according to the third embodiment. [Figure 10] FIG. 10 is a diagram showing a configuration of a system according to the fourth embodiment. [Figure 11] FIG. 11 is a block diagram of a client device according to the fourth embodiment. [Figure 12] FIG. 12 is a block diagram of a server according to the fourth embodiment. [Figure 13] FIG. 13 is a flowchart of a three-dimensional data creation process performed by a client device according to the fourth embodiment. [Figure 14] FIG. 14 is a flowchart of a sensor information transmission process performed by a client device according to the fourth embodiment. [Figure 15] FIG. 15 is a flowchart of three-dimensional data creation processing by the server according to the fourth embodiment. [Figure 16]FIG. 16 is a flowchart of a three-dimensional map transmission process performed by the server according to the fourth embodiment. [Figure 17] FIG. 17 is a diagram showing a configuration of a modified example of the system according to the fourth embodiment. [Figure 18] FIG. 18 is a diagram illustrating the configurations of a server and a client device according to the fourth embodiment. [Figure 19] FIG. 19 is a diagram illustrating the configurations of a server and a client device according to the fifth embodiment. [Figure 20] FIG. 20 is a flowchart of processing by the client device according to the fifth embodiment. [Figure 21] FIG. 21 is a diagram illustrating a configuration of a sensor information collection system according to the fifth embodiment. [Figure 22] FIG. 22 is a diagram illustrating an example of a volume according to the sixth embodiment. [Figure 23] FIG. 23 is a diagram showing an example of an octree representation of a volume according to the sixth embodiment. [Figure 24] FIG. 24 is a diagram illustrating an example of a bit string of a volume according to the sixth embodiment. [Figure 25] FIG. 25 is a diagram showing an example of an octree representation of a volume according to the sixth embodiment. [Figure 26] FIG. 26 is a diagram illustrating an example of a volume according to the sixth embodiment. [Figure 27] FIG. 27 is a diagram showing an example of three-dimensional points according to the seventh embodiment. [Figure 28] FIG. 28 is a diagram illustrating an example of setting LoD according to the seventh embodiment. [Figure 29] FIG. 29 is a diagram illustrating an example of threshold values ​​used for setting the LoD according to the seventh embodiment. [Figure 30] FIG. 30 is a diagram showing an example of attribute information used for a predicted value according to the seventh embodiment. [Figure 31] FIG. 31 is a diagram illustrating an example of an exponential-Golomb code according to the seventh embodiment. [Figure 32] FIG. 32 is a diagram illustrating processing for exponential-Golomb coding according to the seventh embodiment. [Figure 33] FIG. 33 is a diagram illustrating an example of the syntax of an attribute header according to the seventh embodiment. [Figure 34] FIG. 34 is a diagram illustrating an example of the syntax of attribute data according to the seventh embodiment. [Figure 35] FIG. 35 is a flowchart of three-dimensional data encoding processing according to the seventh embodiment. [Figure 36] FIG. 36 is a flowchart of the attribute information encoding process according to the seventh embodiment. [Figure 37] FIG. 37 is a diagram illustrating processing for exponential-Golomb coding according to the seventh embodiment. [Figure 38] FIG. 38 is a diagram showing an example of a reverse lookup table showing the relationship between the remaining codes and their values ​​according to the seventh embodiment. [Figure 39] FIG. 39 is a flowchart of three-dimensional data decoding processing according to the seventh embodiment. [Figure 40] FIG. 40 is a flowchart of the attribute information decoding process according to the seventh embodiment. [Figure 41] FIG. 41 is a block diagram of a three-dimensional data encoding device according to the seventh embodiment. [Figure 42] FIG. 42 is a block diagram of a three-dimensional data decoding device according to the seventh embodiment. [Figure 43] FIG. 43 is a diagram for explaining encoding of attribute information using RAHT according to the eighth embodiment. [Figure 44] FIG. 44 is a diagram showing an example of setting a quantization scale for each layer according to the eighth embodiment. [Figure 45] FIG. 45 shows examples of the first code string and the second code string according to the eighth embodiment. [Figure 46] FIG. 46 is a diagram illustrating an example of a truncated unary code according to the eighth embodiment. [Figure 47] FIG. 47 is a diagram for explaining the inverse Haar transform according to the eighth embodiment. [Figure 48] FIG. 48 is a diagram illustrating an example of the syntax of attribute information according to the eighth embodiment. [Figure 49] FIG. 49 is a diagram showing an example of coding coefficients and ZeroCnt according to the eighth embodiment. [Figure 50] FIG. 50 is a flowchart of three-dimensional data encoding processing according to the eighth embodiment. [Figure 51] FIG. 51 is a flowchart of the attribute information encoding process according to the eighth embodiment. [Figure 52] FIG. 52 is a flowchart of the coding coefficient coding process according to the eighth embodiment. [Figure 53] FIG. 53 is a flowchart of three-dimensional data decoding processing according to the eighth embodiment. [Figure 54] FIG. 54 is a flowchart of the attribute information decoding process according to the eighth embodiment. [Figure 55] FIG. 55 is a flowchart of the coding coefficient decoding process according to the eighth embodiment. [Figure 56] FIG. 56 is a block diagram of an attribute information encoding unit according to the eighth embodiment. [Figure 57] FIG. 57 is a block diagram of an attribute information decoding unit according to the eighth embodiment. [Figure 58] FIG. 58 shows examples of a first code string and a second code string according to a modification of the eighth embodiment. [Figure 59] FIG. 59 is a diagram illustrating an example of the syntax of attribute information according to a modification of the eighth embodiment. [Figure 60] FIG. 60 shows examples of coding coefficients, ZeroCnt, and TotalZeroCnt according to a modification of the eighth embodiment. [Figure 61] FIG. 61 is a flowchart of a coding coefficient coding process according to a modification of the eighth embodiment. [Figure 62] FIG. 62 is a flowchart of a coding coefficient decoding process according to a modification of the eighth embodiment. [Figure 63] FIG. 63 is a diagram illustrating an example of the syntax of attribute information according to a modification of the eighth embodiment. [Figure 64]FIG. 64 is a block diagram showing a configuration of a three-dimensional data encoding device according to the ninth embodiment. [Figure 65] FIG. 65 is a block diagram showing a configuration of a three-dimensional data decoding device according to the ninth embodiment. [Figure 66] FIG. 66 is a diagram showing an example of setting LoD according to the ninth embodiment. [Figure 67] FIG. 67 is a diagram showing an example of the hierarchical structure of the RAHT according to the ninth embodiment. [Figure 68] FIG. 68 is a block diagram of a three-dimensional data encoding device according to the ninth embodiment. [Figure 69] FIG. 69 is a block diagram of a division unit according to the ninth embodiment. [Figure 70] FIG. 70 is a block diagram of an attribute information encoding unit according to the ninth embodiment. [Figure 71] FIG. 71 is a block diagram of a three-dimensional data decoding device according to the ninth embodiment. [Figure 72] FIG. 72 is a block diagram of an attribute information decoding unit according to the ninth embodiment. [Figure 73] FIG. 73 is a diagram showing an example of setting quantization parameters in tile and slice division according to the ninth embodiment. [Figure 74] FIG. 74 is a diagram showing an example of setting the quantization parameter according to the ninth embodiment. [Figure 75] FIG. 75 is a diagram showing an example of setting the quantization parameter according to the ninth embodiment. [Figure 76] FIG. 76 is a diagram illustrating an example of the syntax of the attribute information header according to the ninth embodiment. [Figure 77] FIG. 77 is a diagram illustrating an example of the syntax of the attribute information header according to the ninth embodiment. [Figure 78] FIG. 78 is a diagram showing an example of setting the quantization parameter according to the ninth embodiment. [Figure 79] FIG. 79 is a diagram illustrating an example of the syntax of an attribute information header according to the ninth embodiment. [Figure 80]FIG. 80 is a diagram illustrating an example of the syntax of the attribute information header according to the ninth embodiment. [Figure 81] FIG. 81 is a flowchart of three-dimensional data encoding processing according to the ninth embodiment. [Figure 82] FIG. 82 is a flowchart of the attribute information encoding process according to the ninth embodiment. [Figure 83] FIG. 83 is a flowchart of the ΔQP determination process according to the ninth embodiment. [Figure 84] FIG. 84 is a flowchart of three-dimensional data decoding processing according to the ninth embodiment. [Figure 85] FIG. 85 is a flowchart of the attribute information decoding process according to the ninth embodiment. [Figure 86] FIG. 86 is a block diagram of an attribute information encoding unit according to the ninth embodiment. [Figure 87] FIG. 87 is a block diagram of an attribute information decoding unit according to the ninth embodiment. [Figure 88] FIG. 88 is a diagram showing an example of setting the quantization parameter according to the ninth embodiment. [Figure 89] FIG. 89 is a diagram illustrating an example of the syntax of the attribute information header according to the ninth embodiment. [Figure 90] FIG. 90 is a diagram illustrating an example of the syntax of the attribute information header according to the ninth embodiment. [Figure 91] FIG. 91 is a flowchart of three-dimensional data encoding processing according to the ninth embodiment. [Figure 92] FIG. 92 is a flowchart of the attribute information encoding process according to the ninth embodiment. [Figure 93] FIG. 93 is a flowchart of three-dimensional data decoding processing according to the ninth embodiment. [Figure 94] FIG. 94 is a flowchart of the attribute information decoding process according to the ninth embodiment. [Figure 95] FIG. 95 is a block diagram of an attribute information encoding unit according to the ninth embodiment. [Figure 96]FIG. 96 is a block diagram of an attribute information decoding unit according to the ninth embodiment. [Figure 97] FIG. 97 is a diagram illustrating an example of the syntax of the attribute information header according to the ninth embodiment. [Figure 98] FIG. 98 is a graph showing the relationship between the bit rate and time for encoding a bitstream according to the tenth embodiment. [Figure 99] FIG. 99 is a diagram showing the hierarchical structure of a 3D point group according to the tenth embodiment and the number of 3D points belonging to each layer. [Figure 100] FIG. 100 is a diagram showing a first example of classifying a 3D point cloud of one layer into sub-layers with a specified number of 3D points according to the tenth embodiment. [Figure 101] FIG. 101 is a diagram showing a second example of classifying a 3D point cloud of one layer into sub-layers with a fixed number of 3D points according to the tenth embodiment. [Figure 102] FIG. 102 shows an example of the syntax of the header of the attribute information in the second example according to the tenth embodiment. [Figure 103] FIG. 103 shows another example of the syntax of the attribute information in the second example according to the tenth embodiment. [Figure 104] FIG. 104 is a diagram showing a third example of classifying a 3D point cloud of one layer into sub-layers of a number different from a predetermined number according to the tenth embodiment. [Figure 105] FIG. 105 illustrates an example of the syntax of the header of the attribute information in the third example according to the tenth embodiment. [Figure 106] FIG. 106 shows another example of the syntax of the header of the attribute information in the third example according to the tenth embodiment. [Figure 107] FIG. 107 is a diagram showing a fourth example of classifying a 3D point cloud of one layer into sub-layers according to the tenth embodiment, using the number of 3D points specified as a ratio (percentage). [Figure 108] FIG. 108 shows an example of the syntax of the header of the attribute information in the fourth example according to the tenth embodiment. [Figure 109]FIG. 109 is a diagram showing a fifth example of classifying a 3D point cloud of one layer into sub-layers using Morton's index according to the tenth embodiment. [Figure 110] FIG. 110 shows an example of the syntax of the header of the attribute information in the fifth example according to the tenth embodiment. [Figure 111] FIG. 111 is a diagram showing a sixth example of classifying a 3D point cloud of one layer into sub-layers using Morton's index according to the tenth embodiment. [Figure 112] FIG. 112 is a diagram showing a sixth example of classifying a 3D point cloud of one layer into sub-layers using Morton's index according to the tenth embodiment. [Figure 113] FIG. 113 is a diagram showing a seventh example of classifying a 3D point cloud of one layer into sub-layers using residuals or delta values ​​according to the tenth embodiment. [Figure 114] FIG. 114 is a diagram showing an arrangement of three-dimensional points when arranged in two-dimensional Morton order according to the tenth embodiment. [Figure 115] FIG. 115 shows an example of the syntax of the header of the attribute information in the seventh example according to the tenth embodiment. [Figure 116] FIG. 116 shows an example of the syntax of the residual bitstream according to the tenth embodiment. [Figure 117] FIG. 117 shows a formula for calculating the encoding cost according to the tenth embodiment. [Figure 118] FIG. 118 is a graph showing the relationship between BPP (bits per point) and time according to the tenth embodiment. [Figure 119] FIG. 119 is a diagram showing that the QP value applied to encoding of the attribute information according to the tenth embodiment is set for each sub-layer. [Figure 120] FIG. 120 is a diagram showing an eighth example of classifying a 3D point cloud into sub-hierarchies using Morton codes according to the tenth embodiment. [Figure 121] FIG. 121 shows an example of the syntax of the header of the attribute information in the eighth example according to the tenth embodiment. [Figure 122] FIG. 122 is a flowchart of three-dimensional data encoding processing according to the tenth embodiment. [Figure 123] FIG. 123 is a flowchart of three-dimensional data decoding processing according to the tenth embodiment. [Figure 124] FIG. 124 is a diagram illustrating an example of a three-dimensional space to which a three-dimensional point group of a three-dimensional model according to the eleventh embodiment belongs. [Figure 125] FIG. 125 shows an example of the syntax of the attribute information header according to the eleventh embodiment. [Figure 126] FIG. 126 is a diagram illustrating another example of a three-dimensional space to which a three-dimensional point group according to the eleventh embodiment belongs. [Figure 127] FIG. 127 is a diagram illustrating another example of a three-dimensional space to which a three-dimensional point group according to the eleventh embodiment belongs. [Figure 128] FIG. 128 illustrates an example of the syntax of the attribute information header when a cylindrical three-dimensional space according to the eleventh embodiment is set. [Figure 129] FIG. 129 is a flowchart showing an example of processing for applying different quantization parameters to each three-dimensional space according to the eleventh embodiment. [Figure 130] FIG. 130 is a flowchart showing another example of processing for applying different quantization parameters to each three-dimensional space according to the eleventh embodiment. [Figure 131] FIG. 131 is a diagram illustrating a first example of encoding of attribute information using RAHT according to the eleventh embodiment. [Figure 132] FIG. 132 is a diagram illustrating a second example of encoding of attribute information using RAHT according to the eleventh embodiment. [Figure 133] FIG. 133 is a diagram illustrating a third example of encoding of attribute information using RAHT according to the eleventh embodiment. [Figure 134] FIG. 134 is a diagram illustrating a fourth example of encoding of attribute information using RAHT according to the eleventh embodiment. [Figure 135]FIG. 135 is a diagram showing an example of a hierarchical structure (tree structure) of a RAHT according to the eleventh embodiment. [Figure 136] FIG. 136 is a diagram illustrating a fifth example of encoding of attribute information using RAHT according to the eleventh embodiment. [Figure 137] FIG. 137 is a diagram illustrating a sixth example of encoding of attribute information using RAHT according to the eleventh embodiment. [Figure 138] FIG. 138 is a block diagram illustrating an example of a processing unit that executes encoding processing of attribute information according to the eleventh embodiment. [Figure 139] FIG. 139 is a diagram showing the relationship between attribute information and QP values ​​in the hierarchical structure of LoD according to the eleventh embodiment. [Figure 140] FIG. 140 is a block diagram illustrating an example of a processing unit that executes the decoding process of the attribute information according to the eleventh embodiment. [Figure 141] FIG. 141 is a flowchart of three-dimensional data encoding processing according to the eleventh embodiment. [Figure 142] FIG. 142 is a flowchart of three-dimensional data decoding processing according to the eleventh embodiment. [Figure 143] FIG. 143 is a diagram illustrating a method of transmitting a quantization parameter used for encoding attribute information according to the twelfth embodiment. [Figure 144] FIG. 144 is an example of a flowchart for deriving a QP value to be applied to attribute information to be processed according to the twelfth embodiment. [Figure 145] FIG. 145 is a flowchart of a method for determining a QP value for each 3D point according to the twelfth embodiment. [Figure 146] FIG. 146 is an example of a flowchart of an encoding method performed by a three-dimensional data encoding device according to the twelfth embodiment. [Figure 147] FIG. 147 is an example of a flowchart of a decoding method performed by the three-dimensional data decoding device according to the twelfth embodiment. [Figure 148] FIG. 148 is a diagram illustrating an example of the relationship between slices and regions according to the twelfth embodiment. [Figure 149] FIG. 149 is a diagram illustrating another example of the relationship between slices and regions according to the twelfth embodiment. [Figure 150] FIG. 150 is a diagram illustrating an example of setting regions after data division according to the twelfth embodiment. [Figure 151] FIG. 151 is a diagram illustrating an example of dividing data after setting an area according to the twelfth embodiment. [Figure 152] FIG. 152 is a flowchart showing another example of the three-dimensional data encoding process according to the twelfth embodiment. [Figure 153] FIG. 153 is a flowchart showing another example of the three-dimensional data encoding process according to the twelfth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0016] A three-dimensional data encoding method according to one embodiment of the present disclosure encodes multiple pieces of attribute information of multiple three-dimensional points using parameters, and generates a bitstream including the encoded multiple pieces of attribute information and the parameters, wherein each of the multiple pieces of attribute information belongs to one or more hierarchies, and each of the multiple three-dimensional points belongs to one or more regions, and the parameters are determined according to the hierarchies to which the attribute information to be encoded belongs and the regions to which the three-dimensional points having the attribute information to be encoded belong, and the parameters included in the bitstream include a predetermined reference value, a first difference value determined for each hierarchies, and a second difference value determined for each region.

[0017] According to this, the three-dimensional data encoding method can improve encoding efficiency by representing parameters in the bitstream using a predetermined reference value, a first differential value determined for each layer, and a second differential value determined for each region.

[0018] For example, each of the multiple pieces of attribute information may further belong to one or more slices, the parameters may be determined according to the slice to which the attribute information to be encoded belongs, and the parameters included in the bitstream may further include a third differential value determined for each slice.

[0019] According to this, the three-dimensional data encoding method can improve encoding efficiency by representing parameters in the bitstream using a predetermined reference value, a first differential value determined for each layer, a second differential value determined for each region, and a third differential value determined for each slice.

[0020] For example, each of the plurality of attribute information may further belong to one or more frames, the parameters may be determined according to the frame to which the attribute information to be encoded belongs, and the parameters included in the bitstream may further include a fourth differential value determined for each of the frames.

[0021] According to this, the three-dimensional data encoding method can improve encoding efficiency by representing parameters in the bitstream using a predetermined reference value, a first differential value determined for each layer, a second differential value determined for each region, and the fourth differential value determined for each frame.

[0022] For example, a first region among the one or more regions may belong to a first slice among the one or more slices, and the origin coordinates of the first region may be coordinates based on the origin coordinates of the first slice.

[0023] Therefore, the origin coordinates of the first region are represented by the origin coordinates of the first slice and the difference value thereof, which can improve coding efficiency.

[0024] Furthermore, a three-dimensional data decoding method according to one embodiment of the present disclosure acquires a bitstream to acquire multiple pieces of encoded attribute information and parameters, and decodes the multiple pieces of encoded attribute information using the parameters, wherein the multiple pieces of attribute information are attribute information of multiple three-dimensional points, each of the multiple pieces of attribute information belonging to one or more hierarchies, and each of the multiple three-dimensional points belonging to one or more regions, the parameters are determined according to the hierarchies to which the attribute information to be encoded belongs and the regions to which the three-dimensional points having the attribute information to be encoded belong, and the parameters included in the bitstream include a predetermined reference value, a first difference value determined for each hierarchies, and a second difference value determined for each region.

[0025] Therefore, the three-dimensional data decoding method can correctly decode the attribute information of the three-dimensional points.

[0026] For example, each of the multiple pieces of attribute information may further belong to one or more slices, the parameters may be determined according to the slice to which the attribute information to be decoded belongs, and the parameters included in the bitstream may further include a third differential value determined for each slice.

[0027] Therefore, the three-dimensional data decoding method can correctly decode the attribute information of the three-dimensional points.

[0028] For example, each of the plurality of attribute information may further belong to one or more frames, the parameters may be determined according to the frame to which the attribute information to be decoded belongs, and the parameters included in the bitstream may further include a fourth difference value determined for each of the frames.

[0029] Therefore, the three-dimensional data decoding method can correctly decode the attribute information of the three-dimensional points.

[0030] For example, a first region among the one or more regions may belong to a first slice among the one or more slices, and the origin coordinates of the first region may be coordinates based on the origin coordinates of the first slice.

[0031] Therefore, the three-dimensional data decoding method can correctly decode the origin coordinates of the first region.

[0032] These comprehensive or specific aspects may be realized as a system, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or may be realized as any combination of a system, a method, an integrated circuit, a computer program, and a recording medium.

[0033] Hereinafter, the embodiments will be described in detail with reference to the drawings. Note that each of the embodiments described below represents a specific example of the present disclosure. The numerical values, shapes, materials, components, component placement and connection configurations, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components not recited in independent claims will be described as optional components.

[0034] (Embodiment 1) First, the data structure of encoded three-dimensional data (hereinafter also referred to as encoded data) according to this embodiment will be described. Fig. 1 is a diagram showing the structure of encoded three-dimensional data according to this embodiment.

[0035] In this embodiment, a three-dimensional space is divided into spaces (SPCs) corresponding to pictures in video encoding, and three-dimensional data is encoded using the spaces as units. The spaces are further divided into volumes (VLMs) corresponding to macroblocks or the like in video encoding, and prediction and conversion are performed using the VLMs as units. A volume includes a plurality of voxels (VXLs), which are the smallest units to which position coordinates can be associated. Note that prediction, like prediction performed for two-dimensional images, refers to generating predicted three-dimensional data similar to the processing unit to be processed by referring to other processing units, and encoding the difference between the predicted three-dimensional data and the processing unit to be processed. Furthermore, this prediction includes not only spatial prediction that refers to other prediction units at the same time, but also temporal prediction that refers to a prediction unit at a different time.

[0036] For example, when a three-dimensional data encoding device (hereinafter also referred to as an encoding device) encodes a three-dimensional space represented by point cloud data such as a point cloud, it encodes each point of the point cloud or multiple points contained in a voxel collectively according to the size of the voxel. By subdividing the voxels, the three-dimensional shape of the point cloud can be expressed with high precision, and by increasing the voxel size, the three-dimensional shape of the point cloud can be expressed roughly.

[0037] In the following, an example will be described in which the three-dimensional data is a point cloud, but the three-dimensional data is not limited to a point cloud and may be three-dimensional data in any format.

[0038] Alternatively, voxels with a hierarchical structure may be used. In this case, the nth layer may indicate in order whether a sample point exists in the n-1th layer or lower (a layer below the nth layer). For example, when decoding only the nth layer, if a sample point exists in the n-1th layer or lower, the sample point can be decoded by assuming that the sample point exists at the center of the voxel in the nth layer.

[0039] The encoding device also acquires point cloud data using a distance sensor, a stereo camera, a monocular camera, a gyro, an inertial sensor, or the like.

[0040] Similar to video coding, spaces are classified into at least three prediction structures, including independently decodable intra-space (I-SPC), predictive space (P-SPC), and bidirectional space (B-SPC). Spaces also have two types of time information: decoding time and display time.

[0041] As shown in Figure 1, there is a random access unit called a Group Of Space (GOS), which is a processing unit that includes multiple spaces. There is also a World (WLD), which is a processing unit that includes multiple GOS.

[0042] The spatial region occupied by the world is associated with an absolute position on Earth using GPS or latitude and longitude information. This position information is stored as meta information. Note that the meta information may be included in the encoded data or may be transmitted separately from the encoded data.

[0043] Furthermore, within a GOS, all SPCs may be three-dimensionally adjacent, or there may be SPCs that are not three-dimensionally adjacent to other SPCs.

[0044] In the following, the process of encoding, decoding, referencing, etc. of three-dimensional data included in a processing unit such as a GOS, SPC, or VLM will also be simply referred to as encoding, decoding, or referencing the processing unit, etc. The three-dimensional data included in the processing unit includes, for example, at least one pair of a spatial position such as three-dimensional coordinates and a characteristic value such as color information.

[0045] Next, the prediction structure of SPCs in a GOS will be explained. Multiple SPCs in the same GOS or multiple VLMs in the same SPC occupy different spaces, but have the same time information (decoding time and display time).

[0046] Furthermore, the first SPC in a GOS in decoding order is the I-SPC. There are two types of GOS: closed GOS and open GOS. A closed GOS is a GOS that can decode all SPCs in the GOS when decoding starts from the first I-SPC. In an open GOS, some SPCs that appear earlier in the GOS than the first I-SPC refer to a different GOS, and cannot be decoded using only that GOS.

[0047] In addition, in coded data such as map information, WLDs are sometimes decoded in the reverse order of coding, and if there is dependency between GOSs, reverse playback is difficult. Therefore, in such cases, closed GOSs are generally used.

[0048] Furthermore, the GOS has a layer structure in the height direction, and encoding or decoding is performed in order from the SPC in the lower layer.

[0049] Fig. 2 is a diagram showing an example of a prediction structure between SPCs belonging to the lowest layer of a GOS, and Fig. 3 is a diagram showing an example of a prediction structure between layers.

[0050] A GOS contains one or more I-SPCs. Objects such as people, animals, cars, bicycles, traffic lights, and landmark buildings exist in three-dimensional space, and it is particularly effective to encode small objects as I-SPCs. For example, a three-dimensional data decoding device (hereinafter also referred to as a decoding device) decodes only the I-SPCs in the GOS when decoding the GOS with low processing load or at high speed.

[0051] The encoding device may also switch the encoding interval or frequency of occurrence of I-SPC depending on the density of objects in the WLD.

[0052] 3, the encoding device or decoding device encodes or decodes multiple layers in order from the lowest layer (layer 1). This allows, for example, an autonomous vehicle to prioritize data near the ground, which contains more information.

[0053] In addition, encoded data used by drones, etc. may be encoded or decoded in order from the SPC of the highest layer in the height direction within the GOS.

[0054] Alternatively, the encoding or decoding device may encode or decode multiple layers so that the decoding device can roughly grasp the GOS and gradually increase the resolution. For example, the encoding or decoding device may encode or decode layers 3, 8, 1, 9, etc. in that order.

[0055] Next, we will explain how to handle static and dynamic objects.

[0056] In a three-dimensional space, there exist static objects or scenes such as buildings or roads (hereinafter collectively referred to as static objects), and dynamic objects such as cars or people (hereinafter referred to as dynamic objects). Object detection is performed separately by extracting feature points from point cloud data or camera images such as a stereo camera. Here, an example of a method for encoding dynamic objects will be described.

[0057] The first method is to encode static objects without distinguishing between static and dynamic objects, and the second method is to distinguish between static and dynamic objects using identification information.

[0058] For example, GOS is used as the identification unit. In this case, GOS including SPCs that constitute static objects and GOS including SPCs that constitute dynamic objects are distinguished by identification information stored within the coded data or separately from the coded data.

[0059] Alternatively, the SPC may be used as the identification unit, in which case the SPC including the VLM that constitutes a static object and the SPC including the VLM that constitutes a dynamic object are distinguished by the above-mentioned identification information.

[0060] Alternatively, the VLM or VXL may be used as the identification unit, in which case the VLM or VXL containing static objects and the VLM or VXL containing dynamic objects are distinguished by the above-mentioned identification information.

[0061] The encoding device may also encode a dynamic object as one or more VLMs or SPCs, and encode a VLM or SPC containing a static object and an SPC containing a dynamic object as different GOSs. If the size of the GOS varies depending on the size of the dynamic object, the encoding device stores the size of the GOS separately as meta information.

[0062] The encoding device may also encode static objects and dynamic objects independently of each other, and overlay the dynamic objects on a world made up of static objects. In this case, the dynamic object is made up of one or more SPCs, and each SPC corresponds to one or more SPCs that make up the static object on which the SPC is overlaid. Note that the dynamic object may be represented by one or more VLMs or VXLs instead of SPCs.

[0063] The encoding device may also encode static objects and dynamic objects as different streams.

[0064] The encoding device may also generate a GOS that includes one or more SPCs that make up a dynamic object. Furthermore, the encoding device may set the GOS (GOS_M) that includes the dynamic object and the GOS of the static object that corresponds to the spatial region of GOS_M to the same size (occupy the same spatial region). This allows superimposition processing to be performed on a GOS-by-GOS basis.

[0065] A P-SPC or B-SPC that configures a dynamic object may refer to an SPC included in a different GOS that has already been coded. In cases where the position of a dynamic object changes over time and the same dynamic object is coded as a GOS at different times, referencing across GOSs is effective from the viewpoint of compression ratio.

[0066] The encoding device may switch between the first and second methods depending on the intended use of the encoded data. For example, when the encoded three-dimensional data is used as a map, it is desirable to be able to separate dynamic objects, so the encoding device uses the second method. On the other hand, when encoding three-dimensional data of an event such as a concert or sporting event, the encoding device uses the first method if there is no need to separate dynamic objects.

[0067] The decode time and display time of a GOS or SPC can be stored in the coded data or as meta information. The time information of all static objects may be the same. In this case, the actual decode time and display time may be determined by the decoding device. Alternatively, a different value may be assigned as the decode time for each GOS or SPC, and the same value may be assigned as the display time for all. Furthermore, a model may be introduced that ensures that the decoder has a buffer of a predetermined size and can decode without failure if it reads a bitstream at a predetermined bit rate according to the decode time, as in a decoder model used in video coding, such as the HEVC HRD (Hypothetical Reference Decoder).

[0068] Next, we will explain the arrangement of GOS within a world. The coordinates of the three-dimensional space in a world are expressed by three mutually orthogonal coordinate axes (x-axis, y-axis, and z-axis). By establishing a predetermined rule for the encoding order of GOS, encoding can be performed so that spatially adjacent GOS are continuous within the encoded data. For example, in the example shown in Figure 4, GOS within the xz plane are encoded continuously. After encoding of all GOS within a certain xz plane is completed, the value of the y-axis is updated. In other words, as encoding progresses, the world extends in the y-axis direction. Furthermore, the index numbers of GOS are set in the encoding order.

[0069] Here, the three-dimensional space of the world is associated one-to-one with absolute geographical coordinates such as GPS or latitude and longitude. Alternatively, the three-dimensional space may be expressed by relative positions from a preset reference position. The directions of the x-, y-, and z-axes of the three-dimensional space are expressed as direction vectors determined based on the latitude and longitude, and the direction vectors are stored as meta information together with the encoded data.

[0070] The size of the GOS is fixed, and the encoding device stores the size as meta information. The size of the GOS may be changed depending on, for example, whether the location is an urban area or whether the location is indoors or outdoors. That is, the size of the GOS may be changed depending on the quantity or nature of objects that have information value. Alternatively, the encoding device may adaptively change the size of the GOS or the spacing between I-SPCs within the GOS depending on, for example, the density of objects within the same world. For example, the higher the object density, the smaller the GOS size and the shorter the spacing between I-SPCs within the GOS.

[0071] In the example shown in Figure 5, the third to tenth GOS regions have a high density of objects, so the GOS are subdivided to allow finer granularity for random access. Note that the seventh to tenth GOS regions are located behind the third to sixth GOS regions, respectively.

[0072] (Embodiment 2) In this embodiment, a method for transmitting and receiving three-dimensional data between vehicles will be described.

[0073] FIG. 7 is a schematic diagram showing how three-dimensional data 607 is transmitted and received between a vehicle 600 and a nearby vehicle 601. As shown in FIG.

[0074] When three-dimensional data is acquired using a sensor (such as a distance sensor such as a range finder, a stereo camera, or a combination of multiple monocular cameras) mounted on the host vehicle 600, an area (hereinafter referred to as an occlusion area 604) where three-dimensional data cannot be created occurs due to obstacles such as surrounding vehicles 601, even though it is within the sensor detection range 602 of the host vehicle 600. Furthermore, the accuracy of autonomous operation increases as the space for acquiring three-dimensional data increases, but the sensor detection range of the host vehicle 600 alone is limited.

[0075] The sensor detection range 602 of the host vehicle 600 includes an area 603 from which three-dimensional data can be acquired and an occlusion area 604. The area from which the host vehicle 600 wishes to acquire three-dimensional data includes the sensor detection range 602 of the host vehicle 600 and other areas. In addition, the sensor detection range 605 of the surrounding vehicle 601 includes the occlusion area 604 and an area 606 that is not included in the sensor detection range 602 of the host vehicle 600.

[0076] The surrounding vehicles 601 transmit information detected by the surrounding vehicles 601 to the host vehicle 600. By acquiring information detected by the surrounding vehicles 601, such as a vehicle ahead, the host vehicle 600 can acquire three-dimensional data 607 of an occlusion region 604 and a region 606 outside the sensor detection range 602 of the host vehicle 600. The host vehicle 600 uses the information acquired by the surrounding vehicles 601 to complement the three-dimensional data of the occlusion region 604 and the region 606 outside the sensor detection range.

[0077] The three-dimensional data used in the autonomous operation of a vehicle or robot is used for self-location estimation, detection of surrounding conditions, or both. For example, for self-location estimation, three-dimensional data generated by the host vehicle 600 based on sensor information of the host vehicle 600 is used. For detection of surrounding conditions, in addition to the three-dimensional data generated by the host vehicle 600, three-dimensional data acquired from a nearby vehicle 601 is also used.

[0078] The nearby vehicle 601 that transmits the three-dimensional data 607 to the host vehicle 600 may be determined according to the state of the host vehicle 600. For example, the nearby vehicle 601 is a vehicle ahead when the host vehicle 600 is traveling straight, an oncoming vehicle when the host vehicle 600 is turning right, and a vehicle behind when the host vehicle 600 is reversing. Also, the driver of the host vehicle 600 may directly specify the nearby vehicle 601 that transmits the three-dimensional data 607 to the host vehicle 600.

[0079] Furthermore, the vehicle 600 may search for a nearby vehicle 601 that possesses three-dimensional data of an area that is included in the space where the vehicle 600 wishes to acquire three-dimensional data 607 but cannot be acquired by the vehicle 600. The area that the vehicle 600 cannot acquire is an occlusion area 604 or an area 606 outside the sensor detection range 602, etc.

[0080] Furthermore, the host vehicle 600 may identify the occlusion region 604 based on sensor information of the host vehicle 600. For example, the host vehicle 600 identifies, as the occlusion region 604, a region included in the sensor detection range 602 of the host vehicle 600 and for which three-dimensional data cannot be created.

[0081] An example of operation will be described below when it is the vehicle ahead that transmits the three-dimensional data 607. Fig. 8 is a diagram showing an example of the three-dimensional data transmitted in this case.

[0082] As shown in Fig. 8, the three-dimensional data 607 transmitted from the vehicle in front is, for example, a sparse world of point cloud (SWLD). That is, the vehicle in front creates three-dimensional data (point cloud) of WLD from information detected by the sensor of the vehicle in front, and then creates three-dimensional data of SWLD (point cloud) by extracting data whose feature amount is equal to or greater than a threshold from the three-dimensional data of WLD. The vehicle in front then transmits the created three-dimensional data of SWLD to the host vehicle 600.

[0083] The host vehicle 600 receives the SWLD and merges the received SWLD with the point cloud created by the host vehicle 600.

[0084] The transmitted SWLD has information on absolute coordinates (the position of the SWLD in the coordinate system of the three-dimensional map). The vehicle 600 can realize the merging process by overwriting the point cloud generated by the vehicle 600 based on these absolute coordinates.

[0085] The SWLD transmitted from the nearby vehicle 601 may be the SWLD of an area 606 that is outside the sensor detection range 602 of the host vehicle 600 but within the sensor detection range 605 of the nearby vehicle 601, or the SWLD of an occlusion area 604 for the host vehicle 600, or both. In addition, the transmitted SWLD may be the SWLD of an area among the above SWLDs that the nearby vehicle 601 is using to detect the surrounding situation.

[0086] Furthermore, the surrounding vehicle 601 may change the density of the transmitted point cloud depending on the available communication time based on the speed difference between the vehicle itself 600 and the surrounding vehicle 601. For example, when the speed difference is large and the available communication time is short, the surrounding vehicle 601 may reduce the density (amount of data) of the point cloud by extracting three-dimensional points with large feature amounts from the SWLD.

[0087] Furthermore, detecting the surrounding conditions means determining whether or not there are people, vehicles, road construction equipment, etc., identifying their types, and detecting their positions, movement directions, movement speeds, etc.

[0088] Furthermore, the host vehicle 600 may acquire braking information of the surrounding vehicle 601 instead of or in addition to the three-dimensional data 607 generated by the surrounding vehicle 601. Here, the braking information of the surrounding vehicle 601 is, for example, information indicating whether the accelerator or brake of the surrounding vehicle 601 has been depressed or the degree to which it has been depressed.

[0089] In addition, in the point clouds generated by each vehicle, the three-dimensional space is subdivided into random access units in consideration of low-latency communication between vehicles. On the other hand, in the case of three-dimensional maps, which are map data downloaded from a server, the three-dimensional space is divided into larger random access units compared to the case of vehicle-to-vehicle communication.

[0090] Data for areas that are likely to become occlusion areas, such as the area in front of a leading vehicle or the area behind a trailing vehicle, is divided into small random access units as data for low latency.

[0091] Since the front becomes more important when driving at high speeds, each vehicle creates SWLDs with a narrower field of view in small random access units when driving at high speeds.

[0092] If the SWLD created for transmission by the vehicle in front includes an area from which the vehicle 600 can acquire point clouds, the vehicle in front may reduce the amount of transmission by removing the point clouds from that area.

[0093] (Embodiment 3) In this embodiment, a method of transmitting three-dimensional data to a following vehicle will be described, etc. Fig. 9 is a diagram showing an example of a target space of three-dimensional data to be transmitted to a following vehicle, etc.

[0094] Vehicle 801 transmits three-dimensional data such as a point cloud contained in a rectangular space 802 of width W, height H, and depth D at a distance L from vehicle 801 ahead of vehicle 801 at time intervals of Δt to a traffic monitoring cloud that monitors road conditions or a following vehicle.

[0095] If a change occurs in the three-dimensional data contained in the space 802 that has been previously transmitted, such as when a vehicle or person enters the space 802 from outside, the vehicle 801 also transmits the three-dimensional data of the space where the change occurred.

[0096] Although Figure 9 shows an example in which the shape of space 802 is a rectangular parallelepiped, space 802 does not necessarily have to be a rectangular parallelepiped as long as it includes the space on the road ahead that is in a blind spot for following vehicles.

[0097] It is desirable to set the distance L to a distance that allows the following vehicle, having received the three-dimensional data, to safely stop. For example, the distance L is set to the sum of the distance the following vehicle travels while it takes to receive the three-dimensional data, the distance the following vehicle travels before it starts to decelerate in response to the received data, and the distance the following vehicle requires to safely stop after starting to decelerate. Since these distances change depending on the speed, the distance L may change depending on the speed V of the vehicle, as in L = a × V + b (a and b are constants).

[0098] The width W is set to a value at least larger than the width of the lane in which the vehicle 801 is traveling. More preferably, the width W is set to a size that includes adjacent spaces such as left and right lanes or shoulder strips.

[0099] The depth D may be a fixed value, or may vary according to the vehicle speed V, such as D=c×V+d (c and d are constants). Furthermore, by setting D so that D>V×Δt, the space to be transmitted can overlap with previously transmitted spaces. This allows the vehicle 801 to more reliably transmit the space on the road to following vehicles and the like without omission.

[0100] In this way, by limiting the three-dimensional data transmitted by vehicle 801 to spaces that are useful to following vehicles, the volume of three-dimensional data transmitted can be effectively reduced, thereby achieving low communication latency and low costs.

[0101] (Fourth embodiment) In the third embodiment, an example has been described in which a client device such as a vehicle transmits three-dimensional data to another vehicle or a server such as a traffic monitoring cloud. In this embodiment, the client device transmits sensor information obtained by a sensor to the server or another client device.

[0102] First, the configuration of a system according to this embodiment will be described. Fig. 10 is a diagram showing the configuration of a system for transmitting and receiving 3D maps and sensor information according to this embodiment. This system includes a server 901 and client devices 902A and 902B. When there is no need to distinguish between the client devices 902A and 902B, they will also be referred to as client device 902.

[0103] The client device 902 is, for example, an in-vehicle device mounted on a mobile object such as a vehicle. The server 901 is, for example, a traffic monitoring cloud or the like, and is capable of communicating with a plurality of client devices 902.

[0104] The server 901 transmits a three-dimensional map composed of a point cloud to the client device 902. Note that the composition of the three-dimensional map is not limited to a point cloud, and may represent other three-dimensional data such as a mesh structure.

[0105] The client device 902 transmits sensor information acquired by the client device 902 to the server 901. The sensor information includes, for example, at least one of LiDAR acquisition information, a visible light image, an infrared image, a depth image, sensor position information, and speed information.

[0106] Data transmitted between the server 901 and the client device 902 may be compressed to reduce data size, or may remain uncompressed to maintain data accuracy. When compressing data, a three-dimensional compression method based on an octree structure, for example, can be used for point clouds. Also, a two-dimensional image compression method can be used for visible light images, infrared images, and depth images. Examples of two-dimensional image compression methods include MPEG-4 AVC or HEVC standardized by MPEG.

[0107] Furthermore, the server 901 transmits the three-dimensional map managed by the server 901 to the client device 902 in response to a three-dimensional map transmission request from the client device 902. Note that the server 901 may transmit the three-dimensional map without waiting for a three-dimensional map transmission request from the client device 902. For example, the server 901 may broadcast the three-dimensional map to one or more client devices 902 that are in a predetermined space. Furthermore, the server 901 may transmit a three-dimensional map appropriate for the position of the client device 902 to the client device 902 that has received a transmission request once, at regular intervals. Furthermore, the server 901 may transmit the three-dimensional map to the client device 902 every time the three-dimensional map managed by the server 901 is updated.

[0108] The client device 902 issues a request to send a three-dimensional map to the server 901. For example, when the client device 902 wants to estimate its own position while driving, the client device 902 sends a request to send a three-dimensional map to the server 901.

[0109] In the following cases, the client device 902 may issue a request to the server 901 to transmit a three-dimensional map. If the three-dimensional map held by the client device 902 is old, the client device 902 may issue a request to the server 901 to transmit a three-dimensional map. For example, if a certain period of time has passed since the client device 902 obtained the three-dimensional map, the client device 902 may issue a request to the server 901 to transmit a three-dimensional map.

[0110] The client device 902 may issue a request to the server 901 to transmit a three-dimensional map a certain time before the client device 902 leaves the space shown in the three-dimensional map held by the client device 902. For example, when the client device 902 is located within a predetermined distance from the boundary of the space shown in the three-dimensional map held by the client device 902, the client device 902 may issue a request to the server 901 to transmit a three-dimensional map. Furthermore, when the movement route and movement speed of the client device 902 are known, the time when the client device 902 will leave the space shown in the three-dimensional map held by the client device 902 may be predicted based on these.

[0111] If the error in aligning the three-dimensional data created by the client device 902 from sensor information with the three-dimensional map is equal to or greater than a certain level, the client device 902 may issue a request to the server 901 to send the three-dimensional map.

[0112] The client device 902 transmits sensor information to the server 901 in response to a request to transmit sensor information transmitted from the server 901. Note that the client device 902 may transmit sensor information to the server 901 without waiting for a request to transmit sensor information from the server 901. For example, once the client device 902 receives a request to transmit sensor information from the server 901, the client device 902 may periodically transmit the sensor information to the server 901 for a certain period of time. Furthermore, if the error in aligning the three-dimensional data created by the client device 902 based on the sensor information with the three-dimensional map obtained from the server 901 is equal to or greater than a certain level, the client device 902 may determine that a change may have occurred in the three-dimensional map around the client device 902, and may transmit this information along with the sensor information to the server 901.

[0113] The server 901 issues a request to transmit sensor information to the client device 902. For example, the server 901 receives location information of the client device 902, such as GPS, from the client device 902. When the server 901 determines, based on the location information of the client device 902, that the client device 902 is approaching a space with little information in the three-dimensional map managed by the server 901, the server 901 issues a request to transmit sensor information to the client device 902 in order to generate a new three-dimensional map. The server 901 may also issue a request to transmit sensor information when it wants to update the three-dimensional map, when it wants to check road conditions during snowfall or a disaster, when it wants to check traffic congestion, or when it wants to check the status of incidents and accidents, etc.

[0114] Furthermore, the client device 902 may set the amount of data of the sensor information to be transmitted to the server 901 depending on the communication state or bandwidth at the time of receiving a request to transmit the sensor information from the server 901. Setting the amount of data of the sensor information to be transmitted to the server 901 means, for example, increasing or decreasing the amount of the data itself or appropriately selecting a compression method.

[0115] 11 is a block diagram showing an example configuration of a client device 902. The client device 902 receives a three-dimensional map composed of a point cloud or the like from the server 901, and estimates the self-position of the client device 902 from three-dimensional data created based on sensor information of the client device 902. The client device 902 also transmits the acquired sensor information to the server 901.

[0116] The client device 902 includes a data receiving unit 1011, a communication unit 1012, a reception control unit 1013, a format conversion unit 1014, multiple sensors 1015, a three-dimensional data creation unit 1016, a three-dimensional image processing unit 1017, a three-dimensional data storage unit 1018, a format conversion unit 1019, a communication unit 1020, a transmission control unit 1021, and a data transmission unit 1022.

[0117] The data receiving unit 1011 receives a three-dimensional map 1031 from the server 901. The three-dimensional map 1031 is data including a point cloud such as a WLD or SWLD. The three-dimensional map 1031 may include either compressed data or uncompressed data.

[0118] The communication unit 1012 communicates with the server 901 and transmits data transmission requests (for example, requests to transmit a three-dimensional map) to the server 901.

[0119] The reception control unit 1013 exchanges information such as compatible formats with the communication destination via the communication unit 1012, and establishes communication with the communication destination.

[0120] The format conversion unit 1014 generates a three-dimensional map 1032 by performing format conversion and the like on the three-dimensional map 1031 received by the data receiving unit 1011. Furthermore, if the three-dimensional map 1031 is compressed or encoded, the format conversion unit 1014 performs decompression or decoding processing. Note that if the three-dimensional map 1031 is uncompressed data, the format conversion unit 1014 does not perform decompression or decoding processing.

[0121] The multiple sensors 1015 are a group of sensors, such as LiDAR, a visible light camera, an infrared camera, or a depth sensor, that acquire information about the outside of the vehicle on which the client device 902 is mounted, and generate sensor information 1033. For example, if the sensor 1015 is a laser sensor such as LiDAR, the sensor information 1033 is three-dimensional data such as a point cloud (point cloud data). Note that the number of sensors 1015 does not need to be multiple.

[0122] The three-dimensional data creation unit 1016 creates three-dimensional data 1034 of the surroundings of the vehicle based on the sensor information 1033. For example, the three-dimensional data creation unit 1016 creates point cloud data with color information of the surroundings of the vehicle using information acquired by LiDAR and visible light images acquired by a visible light camera.

[0123] The three-dimensional image processing unit 1017 performs a process of estimating the vehicle's own position using a three-dimensional map 1032 such as a received point cloud and three-dimensional data 1034 of the surroundings of the vehicle generated from the sensor information 1033. The three-dimensional image processing unit 1017 may generate three-dimensional data 1035 of the surroundings of the vehicle by combining the three-dimensional map 1032 and the three-dimensional data 1034, and perform a process of estimating the vehicle's own position using the generated three-dimensional data 1035.

[0124] The three-dimensional data storage unit 1018 stores a three-dimensional map 1032, three-dimensional data 1034, three-dimensional data 1035, and the like.

[0125] The format conversion unit 1019 generates sensor information 1037 by converting the sensor information 1033 into a format supported by the receiving side. The format conversion unit 1019 may reduce the amount of data by compressing or encoding the sensor information 1037. The format conversion unit 1019 may omit the process if format conversion is not necessary. The format conversion unit 1019 may also control the amount of data to be transmitted in accordance with a specified transmission range.

[0126] The communication unit 1020 communicates with the server 901 and receives data transmission requests (sensor information transmission requests) and the like from the server 901.

[0127] The transmission control unit 1021 exchanges information such as compatible formats with the communication destination via the communication unit 1020, and establishes communication.

[0128] The data transmission unit 1022 transmits the sensor information 1037 to the server 901. The sensor information 1037 includes information acquired by a plurality of sensors 1015, such as information acquired by a LiDAR, a luminance image acquired by a visible light camera, an infrared image acquired by an infrared camera, a depth image acquired by a depth sensor, sensor position information, and speed information.

[0129] Next, the configuration of the server 901 will be described. Fig. 12 is a block diagram showing an example of the configuration of the server 901. The server 901 receives sensor information transmitted from the client device 902 and creates three-dimensional data based on the received sensor information. The server 901 uses the created three-dimensional data to update the three-dimensional map managed by the server 901. In addition, in response to a request from the client device 902 to transmit the three-dimensional map, the server 901 transmits the updated three-dimensional map to the client device 902.

[0130] The server 901 includes a data receiving unit 1111, a communication unit 1112, a receiving control unit 1113, a format conversion unit 1114, a three-dimensional data creation unit 1116, a three-dimensional data synthesis unit 1117, a three-dimensional data storage unit 1118, a format conversion unit 1119, a communication unit 1120, a transmission control unit 1121, and a data transmission unit 1122.

[0131] The data receiving unit 1111 receives sensor information 1037 from the client device 902. The sensor information 1037 includes, for example, information acquired by a LiDAR, a luminance image acquired by a visible light camera, an infrared image acquired by an infrared camera, a depth image acquired by a depth sensor, sensor position information, and speed information.

[0132] The communication unit 1112 communicates with the client device 902 and transmits a data transmission request (for example, a request to transmit sensor information) to the client device 902 .

[0133] The reception control unit 1113 exchanges information such as compatible formats with the communication destination via the communication unit 1112, and establishes communication.

[0134] If the received sensor information 1037 is compressed or encoded, the format conversion unit 1114 performs decompression or decoding processing to generate the sensor information 1132. Note that if the sensor information 1037 is uncompressed data, the format conversion unit 1114 does not perform decompression or decoding processing.

[0135] The three-dimensional data creation unit 1116 creates three-dimensional data 1134 of the periphery of the client device 902 based on the sensor information 1132. For example, the three-dimensional data creation unit 1116 creates point cloud data with color information of the periphery of the client device 902 using information acquired by the LiDAR and visible light images acquired by the visible light camera.

[0136] A three-dimensional data synthesis unit 1117 synthesizes three-dimensional data 1134 created based on sensor information 1132 with a three-dimensional map 1135 managed by the server 901, thereby updating the three-dimensional map 1135.

[0137] The three-dimensional data storage unit 1118 stores a three-dimensional map 1135 and the like.

[0138] The format conversion unit 1119 generates the three-dimensional map 1031 by converting the three-dimensional map 1135 into a format supported by the receiving side. The format conversion unit 1119 may reduce the amount of data by compressing or encoding the three-dimensional map 1135. The format conversion unit 1119 may also omit processing if format conversion is not necessary. The format conversion unit 1119 may also control the amount of data to be transmitted in accordance with the designation of the transmission range.

[0139] The communication unit 1120 communicates with the client device 902 and receives a data transmission request (a request to transmit a three-dimensional map) or the like from the client device 902 .

[0140] The transmission control unit 1121 exchanges information such as compatible formats with the communication destination via the communication unit 1120, and establishes communication.

[0141] The data transmission unit 1122 transmits the three-dimensional map 1031 to the client device 902. The three-dimensional map 1031 is data including a point cloud such as a WLD or SWLD. The three-dimensional map 1031 may include either compressed data or uncompressed data.

[0142] Next, a description will be given of the operational flow of the client device 902. Fig. 13 is a flowchart showing the operation of the client device 902 when acquiring a three-dimensional map.

[0143] First, the client device 902 requests the server 901 to transmit a three-dimensional map (such as a point cloud) (S1001). At this time, the client device 902 may also transmit location information of the client device 902 obtained by GPS or the like, thereby requesting the server 901 to transmit a three-dimensional map related to the location information.

[0144] Next, the client device 902 receives the three-dimensional map from the server 901 (S1002). If the received three-dimensional map is compressed data, the client device 902 decodes the received three-dimensional map to generate an uncompressed three-dimensional map (S1003).

[0145] Next, the client device 902 creates three-dimensional data 1034 of the surroundings of the client device 902 from sensor information 1033 obtained by the multiple sensors 1015 (S1004). Next, the client device 902 estimates its own position using the three-dimensional map 1032 received from the server 901 and the three-dimensional data 1034 created from the sensor information 1033 (S1005).

[0146] 14 is a flowchart showing the operation of the client device 902 when transmitting sensor information. First, the client device 902 receives a request to transmit sensor information from the server 901 (S1011). Upon receiving the transmission request, the client device 902 transmits sensor information 1037 to the server 901 (S1012). Note that, when the sensor information 1033 includes multiple pieces of information obtained by multiple sensors 1015, the client device 902 may generate the sensor information 1037 by compressing each piece of information using a compression method suitable for that piece of information.

[0147] Next, the operation flow of the server 901 will be described. Fig. 15 is a flowchart showing the operation when the server 901 acquires sensor information. First, the server 901 requests the client device 902 to transmit sensor information (S1021). Next, the server 901 receives the sensor information 1037 transmitted from the client device 902 in response to the request (S1022). Next, the server 901 creates three-dimensional data 1134 using the received sensor information 1037 (S1023). Next, the server 901 reflects the created three-dimensional data 1134 in the three-dimensional map 1135 (S1024).

[0148] 16 is a flowchart showing the operation of the server 901 when transmitting a three-dimensional map. First, the server 901 receives a request to transmit a three-dimensional map from the client device 902 (S1031). Having received the request to transmit the three-dimensional map, the server 901 transmits a three-dimensional map 1031 to the client device 902 (S1032). At this time, the server 901 may extract a three-dimensional map of the vicinity based on the location information of the client device 902 and transmit the extracted three-dimensional map. The server 901 may also compress the three-dimensional map made up of a point cloud using, for example, a compression method with an octree structure, and transmit the compressed three-dimensional map.

[0149] A modification of this embodiment will now be described.

[0150] The server 901 uses the sensor information 1037 received from the client device 902 to create three-dimensional data 1134 of the vicinity of the position of the client device 902. Next, the server 901 matches the created three-dimensional data 1134 with a three-dimensional map 1135 of the same area managed by the server 901, thereby calculating the difference between the three-dimensional data 1134 and the three-dimensional map 1135. If the difference is equal to or greater than a predetermined threshold, the server 901 determines that some abnormality has occurred in the vicinity of the client device 902. For example, when ground subsidence occurs due to a natural disaster such as an earthquake, a large difference may occur between the three-dimensional map 1135 managed by the server 901 and the three-dimensional data 1134 created based on the sensor information 1037.

[0151] The sensor information 1037 may include information indicating at least one of the sensor type, sensor performance, and sensor model number. A class ID or the like according to the sensor performance may also be added to the sensor information 1037. For example, if the sensor information 1037 is information acquired by a LiDAR, an identifier may be assigned to the sensor performance, such as Class 1 for a sensor capable of acquiring information with an accuracy of several millimeters, Class 2 for a sensor capable of acquiring information with an accuracy of several centimeters, and Class 3 for a sensor capable of acquiring information with an accuracy of several meters. The server 901 may also estimate the sensor performance information and the like from the model number of the client device 902. For example, if the client device 902 is installed in a vehicle, the server 901 may determine the sensor specification information from the vehicle model. In this case, the server 901 may acquire vehicle model information in advance, or the information may be included in the sensor information. The server 901 may also use the acquired sensor information 1037 to switch the degree of correction for the three-dimensional data 1134 created using the sensor information 1037. For example, if the sensor performance is high accuracy (class 1), the server 901 does not perform correction on the three-dimensional data 1134. If the sensor performance is low accuracy (class 3), the server 901 applies correction according to the accuracy of the sensor to the three-dimensional data 1134. For example, the server 901 increases the degree (strength) of correction as the accuracy of the sensor decreases.

[0152] The server 901 may simultaneously issue requests to send sensor information to multiple client devices 902 in a certain space. When the server 901 receives multiple pieces of sensor information from multiple client devices 902, the server 901 does not need to use all of the sensor information to create the three-dimensional data 1134, and may select the sensor information to use, for example, depending on the performance of the sensor. For example, when updating the three-dimensional map 1135, the server 901 may select high-precision sensor information (Class 1) from the multiple pieces of sensor information it has received, and create the three-dimensional data 1134 using the selected sensor information.

[0153] The server 901 is not limited to a server such as a traffic monitoring cloud, but may be another client device (mounted in a vehicle). Fig. 17 is a diagram showing the system configuration in this case.

[0154] For example, client device 902C issues a request to transmit sensor information to nearby client device 902A and acquires the sensor information from client device 902A. Client device 902C then creates three-dimensional data using the acquired sensor information from client device 902A and updates the three-dimensional map of client device 902C. This allows client device 902C to generate a three-dimensional map of the space that can be acquired from client device 902A by taking advantage of the performance of client device 902C. For example, such a case is likely to occur when client device 902C has high performance.

[0155] In this case, the client device 902A that provided the sensor information is granted the right to obtain the high-precision 3D map generated by the client device 902C. The client device 902A receives the high-precision 3D map from the client device 902C in accordance with the right.

[0156] In addition, client device 902C may issue requests to send sensor information to multiple nearby client devices 902 (client device 902A and client device 902B). If the sensor of client device 902A or client device 902B is high performance, client device 902C can create three-dimensional data using the sensor information obtained by this high performance sensor.

[0157] 18 is a block diagram showing the functional configuration of the server 901 and the client device 902. The server 901 includes, for example, a 3D map compression / decoding processing unit 1201 that compresses and decodes 3D maps, and a sensor information compression / decoding processing unit 1202 that compresses and decodes sensor information.

[0158] The client device 902 includes a three-dimensional map decoding processor 1211 and a sensor information compression processor 1212. The three-dimensional map decoding processor 1211 receives encoded data of the compressed three-dimensional map and decodes the encoded data to acquire the three-dimensional map. The sensor information compression processor 1212 compresses the sensor information itself instead of three-dimensional data created from the acquired sensor information, and transmits the encoded data of the compressed sensor information to the server 901. With this configuration, the client device 902 only needs to internally include a processing unit (device or LSI) that performs processing to decode the three-dimensional map (point cloud, etc.), and does not need to internally include a processing unit that performs processing to compress the three-dimensional data of the three-dimensional map (point cloud, etc.). This allows the cost and power consumption of the client device 902 to be reduced.

[0159] As described above, the client device 902 according to this embodiment is mounted on a mobile body and generates three-dimensional data 1034 of the surroundings of the mobile body from sensor information 1033 indicating the surrounding conditions of the mobile body, which is obtained by the sensor 1015 mounted on the mobile body. The client device 902 estimates the self-position of the mobile body using the generated three-dimensional data 1034. The client device 902 transmits the acquired sensor information 1033 to the server 901 or another mobile body 902.

[0160] According to this, the client device 902 transmits the sensor information 1033 to the server 901 or the like. This may reduce the amount of data to be transmitted compared to when transmitting three-dimensional data. Furthermore, since the client device 902 does not need to perform processing such as compression or encoding of the three-dimensional data, the amount of processing by the client device 902 can be reduced. Therefore, the client device 902 can reduce the amount of data to be transmitted or simplify the device configuration.

[0161] Furthermore, the client device 902 further transmits a request to the server 901 to send a three-dimensional map, and receives a three-dimensional map 1031 from the server 901. The client device 902 estimates its own location using the three-dimensional data 1034 and the three-dimensional map 1032.

[0162] The sensor information 1033 includes at least one of information obtained by a laser sensor, a luminance image, an infrared image, a depth image, sensor position information, and sensor speed information.

[0163] The sensor information 1033 also includes information indicating the performance of the sensor.

[0164] Furthermore, the client device 902 encodes or compresses the sensor information 1033, and transmits the encoded or compressed sensor information 1037 to the server 901 or another mobile body 902. This allows the client device 902 to reduce the amount of data to be transmitted.

[0165] For example, the client device 902 includes a processor and a memory, and the processor uses the memory to perform the above-described processing.

[0166] Furthermore, server 901 according to this embodiment is capable of communicating with client device 902 mounted on the mobile object, and receives sensor information 1037 indicating the surrounding conditions of the mobile object, obtained by sensor 1015 mounted on the mobile object, from client device 902. Server 901 creates three-dimensional data 1134 of the surroundings of the mobile object from the received sensor information 1037.

[0167] According to this, the server 901 creates three-dimensional data 1134 using the sensor information 1037 transmitted from the client device 902. This may reduce the amount of data to be transmitted compared to when the client device 902 transmits the three-dimensional data. Furthermore, since the client device 902 does not need to perform processing such as compression or encoding of the three-dimensional data, the amount of processing by the client device 902 can be reduced. Therefore, the server 901 can reduce the amount of data to be transmitted or simplify the device configuration.

[0168] Furthermore, the server 901 further transmits a request to the client device 902 to transmit the sensor information.

[0169] The server 901 also updates a three-dimensional map 1135 using the created three-dimensional data 1134 and transmits the three-dimensional map 1135 to the client device 902 in response to a request from the client device 902 to transmit the three-dimensional map 1135.

[0170] The sensor information 1037 includes at least one of information obtained by a laser sensor, a luminance image, an infrared image, a depth image, sensor position information, and sensor speed information.

[0171] The sensor information 1037 also includes information indicating the performance of the sensor.

[0172] Furthermore, the server 901 further corrects the three-dimensional data in accordance with the performance of the sensor, thereby enabling the three-dimensional data creation method to improve the quality of the three-dimensional data.

[0173] Furthermore, when receiving sensor information, the server 901 receives a plurality of pieces of sensor information 1037 from a plurality of client devices 902, and selects the sensor information 1037 to be used to create the three-dimensional data 1134 based on a plurality of pieces of information indicating the performance of the sensors included in the plurality of pieces of sensor information 1037. This allows the server 901 to improve the quality of the three-dimensional data 1134.

[0174] Furthermore, the server 901 decodes or decompresses the received sensor information 1037, and creates three-dimensional data 1134 from the decoded or decompressed sensor information 1132. This allows the server 901 to reduce the amount of data to be transmitted.

[0175] For example, the server 901 includes a processor and a memory, and the processor uses the memory to perform the above-mentioned processing.

[0176] (Embodiment 5) In this embodiment, a modification of the above-mentioned embodiment 4 will be described. Fig. 19 is a diagram showing the configuration of a system according to this embodiment. The system shown in Fig. 19 includes a server 2001, a client device 2002A, and a client device 2002B.

[0177] Client device 2002A and client device 2002B are mounted on a moving body such as a vehicle, and transmit sensor information to server 2001. Server 2001 transmits a three-dimensional map (point cloud) to client device 2002A and client device 2002B.

[0178] The client device 2002A includes a sensor information acquisition unit 2011, a storage unit 2012, and a data transmission permission determination unit 2013. The client device 2002B has a similar configuration. In the following description, when there is no need to distinguish between the client device 2002A and the client device 2002B, they will also be referred to as the client device 2002.

[0179] FIG. 20 is a flowchart showing the operation of client device 2002 according to this embodiment.

[0180] The sensor information acquisition unit 2011 acquires various types of sensor information using a sensor (sensor group) mounted on the mobile object. That is, the sensor information acquisition unit 2011 acquires sensor information indicating the surrounding conditions of the mobile object, obtained by the sensor (sensor group) mounted on the mobile object. The sensor information acquisition unit 2011 also stores the acquired sensor information in the storage unit 2012. This sensor information includes at least one of LiDAR acquisition information, visible light images, infrared images, and depth images. The sensor information may also include at least one of sensor position information, speed information, acquisition time information, and acquisition location information. The sensor position information indicates the position of the sensor that acquired the sensor information. The speed information indicates the speed of the mobile object when the sensor acquired the sensor information. The acquisition time information indicates the time when the sensor information was acquired by the sensor. The acquisition location information indicates the position of the mobile object or the sensor when the sensor information was acquired by the sensor.

[0181] Next, the data transmission possibility determination unit 2013 determines whether the mobile object (client device 2002) is in an environment where it can transmit sensor information to the server 2001 (S2002). For example, the data transmission possibility determination unit 2013 may use information such as GPS to identify the location and time of the client device 2002 and determine whether data transmission is possible. Alternatively, the data transmission possibility determination unit 2013 may determine whether data transmission is possible based on whether connection to a specific access point is possible.

[0182] When the client device 2002 determines that the mobile object is in an environment where it can transmit sensor information to the server 2001 (Yes in S2002), the client device 2002 transmits the sensor information to the server 2001 (S2003). In other words, when the client device 2002 is in a situation where it can transmit the sensor information to the server 2001, the client device 2002 transmits the sensor information it holds to the server 2001. For example, a millimeter wave access point capable of high-speed communication is installed at an intersection or the like. When the client device 2002 enters the intersection, it uses millimeter wave communication to transmit the sensor information it holds to the server 2001 at high speed.

[0183] Next, the client device 2002 deletes the sensor information that has been transmitted to the server 2001 from the storage unit 2012 (S2004). Note that the client device 2002 may delete the sensor information that has not been transmitted to the server 2001 if the sensor information satisfies a predetermined condition. For example, the client device 2002 may delete the sensor information from the storage unit 2012 when the acquisition time of the stored sensor information becomes older than a predetermined time from the current time. That is, the client device 2002 may delete the sensor information from the storage unit 2012 when the difference between the time when the sensor information was acquired by the sensor and the current time exceeds a predetermined time. Furthermore, the client device 2002 may delete the sensor information from the storage unit 2012 when the acquisition location of the stored sensor information becomes more than a predetermined distance away from the current location. That is, the client device 2002 may delete the sensor information from the storage unit 2012 when the difference between the position of the mobile object or sensor when the sensor information was acquired by the sensor and the current position of the mobile object or sensor exceeds a predetermined distance. This makes it possible to reduce the capacity of the storage unit 2012 of the client device 2002 .

[0184] If the client device 2002 has not yet completed acquiring the sensor information (No in S2005), the client device 2002 repeats the processes from step S2001 onwards. If the client device 2002 has completed acquiring the sensor information (Yes in S2005), the client device 2002 ends the process.

[0185] Furthermore, the client device 2002 may select the sensor information to be transmitted to the server 2001 in accordance with the communication conditions. For example, when high-speed communication is possible, the client device 2002 transmits sensor information (e.g., LiDAR acquired information) having a large size stored in the storage unit 2012 with priority. When high-speed communication is difficult, the client device 2002 transmits sensor information (e.g., visible light images) having a small size stored in the storage unit 2012 with a high priority. This allows the client device 2002 to efficiently transmit the sensor information stored in the storage unit 2012 to the server 2001 in accordance with the network conditions.

[0186] Furthermore, the client device 2002 may acquire time information indicating the current time and location information indicating the current location from the server 2001. Furthermore, the client device 2002 may determine the acquisition time and acquisition location of the sensor information based on the acquired time information and location information. That is, the client device 2002 may acquire time information from the server 2001 and generate acquisition time information using the acquired time information. Furthermore, the client device 2002 may acquire location information from the server 2001 and generate acquisition location information using the acquired location information.

[0187] For example, with regard to time information, the server 2001 and the client device 2002 synchronize their time using a mechanism such as NTP (Network Time Protocol) or PTP (Precision Time Protocol). This allows the client device 2002 to acquire accurate time information. Furthermore, since time can be synchronized between the server 2001 and multiple client devices, the time in the sensor information acquired by different client devices 2002 can be synchronized. Therefore, the server 2001 can handle sensor information that indicates synchronized time. Note that the time synchronization mechanism may be any method other than NTP or PTP. Furthermore, GPS information may be used as the time information and location information.

[0188] The server 2001 may acquire sensor information from multiple client devices 2002 by specifying a time or a location. For example, when an accident occurs, the server 2001 broadcasts a sensor information transmission request to multiple client devices 2002 by specifying the time and location of the accident to search for clients who were nearby. Then, the client devices 2002 that have sensor information for the corresponding time and location transmit the sensor information to the server 2001. That is, the client device 2002 receives a sensor information transmission request from the server 2001, including designation information that designates the location and time. When the client device 2002 determines that the sensor information obtained at the location and time indicated by the designation information is stored in the memory unit 2012 and that a mobile object is in an environment where it can transmit sensor information to the server 2001, the client device 2002 transmits the sensor information obtained at the location and time indicated by the designation information to the server 2001. In this way, the server 2001 can acquire sensor information related to the occurrence of an accident from multiple client devices 2002 and use it for accident analysis, etc.

[0189] Note that the client device 2002 may refuse to transmit the sensor information when receiving a sensor information transmission request from the server 2001. Also, the client device 2002 may set in advance which of the multiple pieces of sensor information are transmittable. Alternatively, the server 2001 may inquire of the client device 2002 each time whether or not it is possible to transmit the sensor information.

[0190] In addition, points may be awarded to client device 2002 that transmits sensor information to server 2001. These points can be used to pay for, for example, gasoline purchases, EV (Electric Vehicle) charging fees, highway tolls, or rental car fees. After acquiring the sensor information, server 2001 may delete information for identifying client device 2002 that transmitted the sensor information. For example, this information may be information such as the network address of client device 2002. This allows the sensor information to be anonymized, so that the user of client device 2002 can transmit the sensor information from client device 2002 to server 2001 with peace of mind. In addition, server 2001 may be composed of multiple servers. For example, by sharing sensor information among multiple servers, even if one server fails, the other servers can communicate with client device 2002. This makes it possible to avoid service interruptions due to server failures.

[0191] Furthermore, the specified location specified in the sensor information transmission request indicates the location where an accident occurred, etc., and may differ from the location of the client device 2002 at the specified time specified in the sensor information transmission request. Therefore, the server 2001 can specify a range, such as within XX meters, as the specified location and request information acquisition from the client device 2002 that is located within that range. Similarly, for the specified time, the server 2001 may specify a range, such as within N seconds before and after a certain time. This allows the server 2001 to acquire sensor information from the client device 2002 that was located "at a location: within XX meters of absolute position S, from time: tN to t+N." When transmitting three-dimensional data such as LiDAR data, the client device 2002 may transmit data generated immediately after time t.

[0192] Furthermore, the server 2001 may separately specify, as the specified location, information indicating the location of the client device 2002 from which sensor information is to be acquired and the location from which the sensor information is desired. For example, the server 2001 specifies that sensor information covering at least a range of YYm from the absolute position S is to be acquired from the client device 2002 that is located within XXm of the absolute position S. When selecting three-dimensional data to transmit, the client device 2002 selects one or more randomly accessible units of three-dimensional data that include at least the sensor information in the specified range. Furthermore, when transmitting visible light images, the client device 2002 may transmit multiple temporally consecutive image data that include at least a frame immediately before or after time t.

[0193] When the client device 2002 can use multiple physical networks, such as 5G or WiFi, or multiple modes in 5G, to transmit sensor information, the client device 2002 may select a network to use according to the priority order notified by the server 2001. Alternatively, the client device 2002 itself may select a network that can ensure an appropriate bandwidth based on the size of the data to be transmitted. Alternatively, the client device 2002 may select a network to use based on the cost of transmitting data, etc. Furthermore, the transmission request from the server 2001 may include information indicating a transmission deadline, such as a request that the client device 2002 transmit if it can start transmission by time T. If the server 2001 is unable to acquire sufficient sensor information within the deadline, the server 2001 may issue a transmission request again.

[0194] The sensor information may include header information indicating characteristics of the sensor data along with compressed or uncompressed sensor data. The client device 2002 may transmit the header information to the server 2001 via a physical network or communication protocol different from that for transmitting the sensor data. For example, the client device 2002 transmits the header information to the server 2001 prior to transmitting the sensor data. The server 2001 determines whether to acquire the sensor data of the client device 2002 based on an analysis result of the header information. For example, the header information may include information indicating the point cloud acquisition density, elevation angle, or frame rate of the LiDAR, or the resolution, signal-to-noise ratio, or frame rate of the visible light image. This allows the server 2001 to acquire sensor information from the client device 2002 having sensor data of the determined quality.

[0195] As described above, client device 2002 is mounted on a mobile object, acquires sensor information indicating the surrounding conditions of the mobile object obtained by a sensor mounted on the mobile object, and stores the sensor information in storage unit 2012. Client device 2002 determines whether the mobile object is in an environment where it can transmit the sensor information to server 2001, and if it determines that the mobile object is in an environment where it can transmit the sensor information to the server, transmits the sensor information to server 2001.

[0196] Furthermore, the client device 2002 creates three-dimensional data of the surroundings of the mobile object from the sensor information, and estimates the mobile object's own position using the created three-dimensional data.

[0197] Furthermore, the client device 2002 further transmits a request to the server 2001 to send a three-dimensional map, and receives the three-dimensional map from the server 2001. The client device 2002 estimates its own location using the three-dimensional data and the three-dimensional map.

[0198] The above processing by the client device 2002 may be realized as an information transmission method in the client device 2002.

[0199] The client device 2002 may also include a processor and a memory, and the processor may use the memory to perform the above processing.

[0200] Next, a sensor information collection system according to this embodiment will be described. Fig. 21 is a diagram showing the configuration of the sensor information collection system according to this embodiment. As shown in Fig. 21, the sensor information collection system according to this embodiment includes a terminal 2021A, a terminal 2021B, a communication device 2022A, a communication device 2022B, a network 2023, a data collection server 2024, a map server 2025, and a client device 2026. When there is no particular need to distinguish between the terminal 2021A and the terminal 2021B, they are also referred to as terminals 2021. When there is no particular need to distinguish between the communication devices 2022A and 2022B, they are also referred to as communication devices 2022.

[0201] The data collection server 2024 collects data such as sensor data obtained by a sensor provided in the terminal 2021 as position-related data associated with a position in three-dimensional space.

[0202] The sensor data is, for example, data acquired by using a sensor provided in the terminal 2021, such as the state of the surroundings of the terminal 2021 or the state of the inside of the terminal 2021. The terminal 2021 transmits to the data collection server 2024 sensor data collected from one or more sensor devices that are located in positions that can communicate directly with the terminal 2021 or that can communicate via one or more relay devices using the same communication method.

[0203] The data included in the location-related data may include, for example, information indicating the operating status of the terminal itself or a device included in the terminal, an operation log, a service usage status, etc. Furthermore, the data included in the location-related data may include information associating an identifier of the terminal 2021 with the location or movement route of the terminal 2021, etc.

[0204] The information indicating a position included in the position-related data is associated with information indicating a position in three-dimensional data such as three-dimensional map data, etc. The information indicating a position will be described in detail later.

[0205] The position-related data may include, in addition to position information indicating a position, at least one of the time information described above and information indicating the attributes of the data included in the position-related data or the type of sensor that generated the data (e.g., model number). The position information and time information may be stored in a header field of the position-related data or in a header field of a frame that stores the position-related data. Furthermore, the position information and time information may be transmitted and / or stored separately from the position-related data as metadata associated with the position-related data.

[0206] The map server 2025 is connected to, for example, the network 2023, and transmits three-dimensional data such as three-dimensional map data in response to requests from other devices such as the terminal 2021. As described in the above-mentioned embodiments, the map server 2025 may also have a function of updating the three-dimensional data using sensor information transmitted from the terminal 2021.

[0207] The data collection server 2024 is connected to the network 2023, for example, and collects location-related data from other devices such as the terminal 2021, and stores the collected location-related data in a storage device internally or in another server. The data collection server 2024 also transmits the collected location-related data or metadata of three-dimensional map data generated based on the location-related data to the terminal 2021 in response to a request from the terminal 2021.

[0208] The network 2023 is a communication network such as the Internet. The terminal 2021 is connected to the network 2023 via a communication device 2022. The communication device 2022 communicates with the terminal 2021 by using one communication method or by switching between multiple communication methods. The communication device 2022 is, for example, (1) a base station such as LTE (Long Term Evolution), (2) an access point (AP) such as WiFi or millimeter wave communication, (3) a gateway of an LPWA (Low Power Wide Area) network such as SIGFOX, LoRaWAN, or Wi-SUN, or (4) a communication satellite that communicates using a satellite communication method such as DVB-S2.

[0209] The base station may communicate with the terminal 2021 using a method classified as LPWA, such as NB-IoT (Narrow Band-IoT) or LTE-M, or may communicate with the terminal 2021 by switching between these methods.

[0210] Here, an example is given in which the terminal 2021 has a function for communicating with a communication device 2022 that uses two types of communication methods, and communicates with the map server 2025 or the data collection server 2024 using one of these communication methods, or by switching between these multiple communication methods and the communication device 2022 that is the direct communication partner; however, the configuration of the sensor information collection system and the terminal 2021 is not limited to this. For example, the terminal 2021 may not have a communication function for multiple communication methods, but may have a function for communicating using any one of the communication methods. Furthermore, the terminal 2021 may support three or more communication methods. Furthermore, each terminal 2021 may support a different communication method.

[0211] The terminal 2021 has, for example, the configuration of the client device 902 shown in Fig. 11. The terminal 2021 performs position estimation such as its own position using the received three-dimensional data. The terminal 2021 also generates position-related data by associating the sensor data acquired from the sensor with the position information obtained by the position estimation process.

[0212] The location information added to the location-related data indicates, for example, a location in a coordinate system used in three-dimensional data. For example, the location information is a coordinate value expressed as latitude and longitude values. In this case, the terminal 2021 may include, in the location information, information indicating the coordinate system that serves as the basis for the coordinate value, and the three-dimensional data used for location estimation, along with the coordinate value. The coordinate value may also include altitude information.

[0213] Furthermore, the position information may be associated with a data unit or a spatial unit that can be used to encode the three-dimensional data. Examples of such units include WLD, GOS, SPC, VLM, and VXL. In this case, the position information is expressed by an identifier for identifying a data unit, such as an SPC, that corresponds to the position-related data. The position information may also include, in addition to an identifier for identifying a data unit, such as an SPC, information indicating three-dimensional data obtained by encoding a three-dimensional space that includes the data unit, such as the SPC, or information indicating a detailed position within the SPC. The information indicating the three-dimensional data may be, for example, the file name of the three-dimensional data.

[0214] In this way, by generating location-related data associated with location information based on location estimation using three-dimensional data, the system can assign location information to the sensor information with higher accuracy than when location information based on the self-location of the client device (terminal 2021) acquired using GPS is added to the sensor information. As a result, even when the location-related data is used by another device for another service, it may be possible to more accurately identify the location corresponding to the location-related data in real space by performing location estimation based on the same three-dimensional data.

[0215] In the present embodiment, the data transmitted from the terminal 2021 is location-related data, but the data transmitted from the terminal 2021 may be data that is not associated with location information. That is, the transmission and reception of the three-dimensional data or sensor data described in other embodiments may be performed via the network 2023 described in the present embodiment.

[0216] Next, different examples of location information indicating a position in a three-dimensional or two-dimensional real space or map space will be described. The location information added to the location-related data may be information indicating a relative position with respect to a feature point in the three-dimensional data. Here, the feature point serving as the reference for the location information is, for example, a feature point encoded as SWLD and notified to the terminal 2021 as three-dimensional data.

[0217] The information indicating the relative position with respect to the feature point may be expressed as a vector from the feature point to the point indicated by the position information, and may indicate the direction and distance from the feature point to the point indicated by the position information. Alternatively, the information indicating the relative position with respect to the feature point may be information indicating the amount of displacement along each of the X-axis, Y-axis, and Z-axis from the feature point to the point indicated by the position information. Furthermore, the information indicating the relative position with respect to the feature point may be information indicating the distance from each of three or more feature points to the point indicated by the position information. Note that the relative position may not be the relative position of the point indicated by the position information expressed with each feature point as the reference, but may be the relative position of each feature point expressed with the point indicated by the position information as the reference. An example of the position information based on the relative position with respect to the feature point includes information for identifying the reference feature point and information indicating the relative position of the point indicated by the position information with respect to the feature point. Furthermore, when the information indicating the relative position with respect to the feature point is provided separately from the three-dimensional data, the information indicating the relative position with respect to the feature point may include the coordinate axes used to derive the relative position, information indicating the type of three-dimensional data, and / or information indicating the size per unit amount (e.g., scale) of the value of the information indicating the relative position.

[0218] Furthermore, the position information may include information indicating the relative positions of multiple feature points with respect to each feature point. When the position information is expressed as relative positions with respect to multiple feature points, the terminal 2021 attempting to identify the position indicated by the position information in real space may calculate candidate points for the position indicated by the position information from the positions of each feature point estimated from sensor data, and determine that the point obtained by averaging the calculated candidate points is the point indicated by the position information. This configuration reduces the influence of errors when estimating the positions of feature points from sensor data, thereby improving the estimation accuracy of the point indicated by the position information in real space. Furthermore, when the position information includes information indicating the relative positions with respect to multiple feature points, even if there is a feature point that cannot be detected due to limitations such as the type or performance of the sensor equipped in the terminal 2021, it is possible to estimate the value of the point indicated by the position information as long as any one of the multiple feature points can be detected.

[0219] Points that can be identified from sensor data can be used as feature points. Points that can be identified from sensor data are points or points within an area that satisfy a predetermined condition for feature point detection, such as the above-mentioned three-dimensional feature amount or feature amount of visible light data being equal to or greater than a threshold.

[0220] Markers placed in real space may also be used as feature points. In this case, the markers may be detected and their positions identified from data acquired using sensors such as LiDAR or cameras. For example, the markers may be represented by changes in color or brightness (reflectance), or by three-dimensional shapes (such as unevenness). Alternatively, coordinate values ​​indicating the position of the marker, or a two-dimensional code or barcode generated from the identifier of the marker, may be used.

[0221] Furthermore, a light source that transmits an optical signal may be used as a marker. When a light source of an optical signal is used as a marker, not only information for acquiring a position, such as coordinate values ​​or an identifier, but also other data may be transmitted by the optical signal. For example, the optical signal may include information indicating the content of a service corresponding to the position of the marker, an address such as a URL for acquiring the content, or an identifier of a wireless communication device for receiving the service, and a wireless communication method for connecting to the wireless communication device. Using an optical communication device (light source) as a marker facilitates the transmission of data other than information indicating a position, and enables dynamic switching of the data.

[0222] The terminal 2021 grasps the correspondence between feature points between different data by using, for example, an identifier commonly used between the data, or information or a table indicating the correspondence between feature points between the data. Furthermore, if there is no information indicating the correspondence between feature points, the terminal 2021 may determine that the feature points that are closest when the coordinates of a feature point in one of the three-dimensional data are converted to positions in the three-dimensional data space of the other are corresponding feature points.

[0223] When using the position information based on the relative positions described above, even between terminals 2021 or services that use different three-dimensional data, it is possible to identify or estimate the position indicated by the position information based on common feature points included in or associated with each piece of three-dimensional data. As a result, it is possible to identify or estimate the same position with higher accuracy between terminals 2021 or services that use different three-dimensional data.

[0224] Furthermore, even when using map data or three-dimensional data expressed using different coordinate systems, the effect of errors associated with coordinate system conversion can be reduced, enabling the integration of services based on more accurate location information.

[0225] Below, an example of a function provided by the data collection server 2024 will be described. The data collection server 2024 may transfer the received location-related data to another data server. If there are multiple data servers, the data collection server 2024 determines to which data server the received location-related data should be transferred, and transfers the location-related data to the data server determined as the transfer destination.

[0226] The data collection server 2024 determines the destination of transfer, for example, based on a determination rule for the destination server that is set in advance in the data collection server 2024. The determination rule for the destination server is set, for example, in a transfer destination table that associates an identifier associated with each terminal 2021 with a destination data server.

[0227] The terminal 2021 adds an identifier associated with the terminal 2021 to the location-related data to be transmitted, and transmits the data to the data collection server 2024. The data collection server 2024 identifies a destination data server corresponding to the identifier added to the location-related data based on a destination server determination rule using a destination table or the like, and transmits the location-related data to the identified data server. The destination server determination rule may be specified by a determination condition using the time or place at which the location-related data was acquired. Here, the identifier associated with the above-mentioned transmission source terminal 2021 is, for example, an identifier unique to each terminal 2021, or an identifier indicating a group to which the terminal 2021 belongs.

[0228] Furthermore, the transfer destination table does not necessarily have to directly associate an identifier associated with a source terminal with a destination data server. For example, the data collection server 2024 holds a management table storing tag information assigned to each unique identifier of the terminal 2021, and a transfer destination table associating the tag information with a destination data server. The data collection server 2024 may determine a destination data server based on the tag information using the management table and the transfer destination table. Here, the tag information is, for example, control information for management or control information for service provision assigned to the type, model number, owner, group to which the terminal 2021 corresponding to the identifier, or other identifier. Furthermore, the transfer destination table may use an identifier unique to each sensor instead of the identifier associated with the source terminal 2021. Furthermore, a rule for determining the destination server may be set from the client device 2026.

[0229] The data collection server 2024 may determine multiple data servers as transfer destinations and transfer the received location-related data to the multiple data servers. With this configuration, for example, when automatically backing up location-related data or when it is necessary to send location-related data to data servers that provide different services in order to share the location-related data, the intended data transfer can be achieved by changing the settings for the data collection server 2024. As a result, the number of steps required to build and change the system can be reduced compared to when the destination of location-related data is set in each individual terminal 2021.

[0230] In response to a transfer request signal received from a data server, the data collection server 2024 may register the data server specified in the transfer request signal as a new transfer destination, and transfer subsequently received location-related data to that data server.

[0231] The data collection server 2024 may store the location-related data received from the terminal 2021 in a recording device, and in response to a transmission request signal received from the terminal 2021 or the data server, may transmit the location-related data specified in the transmission request signal to the requesting terminal 2021 or data server.

[0232] The data collection server 2024 may determine whether or not it is possible to provide the location-related data to the requesting data server or terminal 2021, and if it is determined that it is possible to provide the location-related data, it may transfer or transmit the location-related data to the requesting data server or terminal 2021.

[0233] When a request for current location-related data is received from client device 2026, even if it is not the timing for terminal 2021 to transmit the location-related data, data collection server 2024 may request terminal 2021 to transmit the location-related data, and terminal 2021 may transmit the location-related data in response to the transmission request.

[0234] In the above explanation, it is assumed that the terminal 2021 transmits location information data to the data collection server 2024, but the data collection server 2024 may also have functions necessary for collecting location-related data from the terminal 2021, such as a function for managing the terminal 2021, or functions used when collecting location-related data from the terminal 2021.

[0235] The data collection server 2024 may have a function of transmitting a data request signal to the terminal 2021 to request the transmission of location information data, and collecting location-related data.

[0236] Management information such as an address for communicating with the terminal 2021 from which data is to be collected or an identifier unique to the terminal 2021 is registered in advance in the data collection server 2024. The data collection server 2024 collects location-related data from the terminal 2021 based on the registered management information. The management information may include information such as the type of sensor included in the terminal 2021, the number of sensors included in the terminal 2021, and the communication method supported by the terminal 2021.

[0237] The data collection server 2024 may collect information from the terminal 2021, such as the operating status or current location of the terminal 2021.

[0238] The management information may be registered from the client device 2026, or the registration process may be initiated by the terminal 2021 sending a registration request to the data collection server 2024. The data collection server 2024 may have a function of controlling communication with the terminal 2021.

[0239] The communication between the data collection server 2024 and the terminal 2021 may be a dedicated line provided by a service provider such as an MNO (Mobile Network Operator) or an MVNO (Mobile Virtual Network Operator), or a virtual dedicated line configured by a VPN (Virtual Private Network). With this configuration, the communication between the terminal 2021 and the data collection server 2024 can be performed safely.

[0240] The data collection server 2024 may have a function of authenticating the terminal 2021 or a function of encrypting data transmitted and received between the terminal 2021. Here, the authentication process of the terminal 2021 or the encryption process of the data is performed using an identifier unique to the terminal 2021 or an identifier unique to a terminal group including multiple terminals 2021, which is shared in advance between the data collection server 2024 and the terminal 2021. This identifier is, for example, an International Mobile Subscriber Identity (IMSI), which is a unique number stored in a Subscriber Identity Module (SIM) card. The identifier used in the authentication process and the identifier used in the data encryption process may be the same or different.

[0241] The authentication or data encryption process between the data collection server 2024 and the terminal 2021 can be provided as long as both the data collection server 2024 and the terminal 2021 have the function of performing the process, and is not dependent on the communication method used by the relay communication device 2022. Therefore, a common authentication or encryption process can be used regardless of the communication method used by the terminal 2021, improving the convenience of system construction for users. However, being independent of the communication method used by the relay communication device 2022 means that it is not necessary to change the process depending on the communication method. In other words, for the purpose of improving transmission efficiency or ensuring security, the authentication or data encryption process between the data collection server 2024 and the terminal 2021 may be switched depending on the communication method used by the relay device.

[0242] The data collection server 2024 may provide the client device 2026 with a UI for managing data collection rules, such as the type of location-related data to be collected from the terminal 2021 and the data collection schedule. This allows the user to specify the terminal 2021 from which data is to be collected using the client device 2026, as well as the time and frequency of data collection. The data collection server 2024 may also specify an area on a map from which data is to be collected, and collect location-related data from the terminal 2021 included in that area.

[0243] When managing data collection rules for each terminal 2021, the client device 2026 presents, for example, a list of the terminals 2021 or sensors to be managed on a screen. The user sets whether or not data collection is necessary, the collection schedule, etc. for each item in the list.

[0244] When specifying an area on a map from which data is to be collected, the client device 2026 presents, for example, a two-dimensional or three-dimensional map of the area to be managed on the screen. The user selects the area from which data is to be collected on the displayed map. The area selected on the map may be a circular or rectangular area centered on a specified point on the map, or a circular or rectangular area that can be specified by dragging. The client device 2026 may also select the area in a predetermined unit, such as a city, an area within a city, a block, or a major road. Instead of specifying the area using a map, the area may be set by inputting latitude and longitude values, or the area may be selected from a list of candidate areas derived based on input text information. The text information may be, for example, the name of a region, city, or landmark.

[0245] Furthermore, the user may specify one or more terminals 2021 and set conditions such as a range of 100 meters around the terminal 2021, so that data may be collected while dynamically changing the specified area.

[0246] Furthermore, if the client device 2026 is equipped with a sensor such as a camera, an area on a map may be designated based on the position of the client device 2026 in real space obtained from sensor data. For example, the client device 2026 may estimate its own location using the sensor data and designate an area within a predetermined distance or a user-specified distance from a point on the map corresponding to the estimated location as an area from which data is to be collected. The client device 2026 may also designate the sensing area of ​​the sensor, i.e., an area corresponding to the acquired sensor data, as an area from which data is to be collected. Alternatively, the client device 2026 may designate an area based on a position corresponding to the user-specified sensor data as an area from which data is to be collected. The area on the map or the location corresponding to the sensor data may be estimated by the client device 2026 or the data collection server 2024.

[0247] When designating an area on a map, the data collection server 2024 may identify the terminals 2021 within the designated area by collecting current location information of each terminal 2021, and may request the identified terminals 2021 to transmit location-related data. Alternatively, instead of the data collection server 2024 identifying the terminals 2021 within the area, the data collection server 2024 may transmit information indicating the designated area to the terminal 2021, and the terminal 2021 may determine whether or not it is within the designated area, and transmit the location-related data if it is determined that it is within the designated area.

[0248] The data collection server 2024 transmits data such as a list or a map for providing the above-mentioned UI (User Interface) in an application executed by the client device 2026 to the client device 2026. The data collection server 2024 may transmit not only data such as a list or a map but also an application program to the client device 2026. The above-mentioned UI may be provided as content created in HTML or the like that can be displayed in a browser. Note that some data, such as map data, may be provided from a server other than the data collection server 2024, such as a map server 2025.

[0249] When the user inputs information to notify completion of input, such as by pressing a setting button, the client device 2026 transmits the input information as setting information to the data collection server 2024. Based on the setting information received from the client device 2026, the data collection server 2024 transmits a signal to each terminal 2021 requesting location-related data or notifying the rules for collecting location-related data, and collects the location-related data.

[0250] Next, an example will be described in which the operation of the terminal 2021 is controlled based on additional information added to three-dimensional or two-dimensional map data.

[0251] In this configuration, object information indicating the position of a power supply unit such as a wireless power supply antenna or power supply coil buried in a road or parking lot is included in or associated with the three-dimensional data and provided to a terminal 2021, such as a car or a drone.

[0252] When a vehicle or drone acquires the object information to charge, it automatically moves its own position so that the position of its charging unit, such as a charging antenna or charging coil, faces the area indicated by the object information, and begins charging. In the case of a vehicle or drone without an autonomous driving function, the direction to move or the operation to be performed is displayed on the screen or audio is used to inform the driver or pilot. When it is determined that the position of the charging unit calculated based on the estimated self-position is within the area indicated by the object information or within a predetermined distance from that area, the displayed image or audio is switched to one instructing the driver or pilot to stop driving or piloting, and charging begins.

[0253] Furthermore, the object information may not be information indicating the position of the power supply unit, but may be information indicating an area in which a charging efficiency equal to or greater than a predetermined threshold can be obtained when a charging unit is placed within the area. The position of the object information may be represented by a point at the center of the area indicated by the object information, or may be represented by an area or line in a two-dimensional plane, or an area, line, or plane in three-dimensional space.

[0254] This configuration makes it possible to grasp the position of the power feeding antenna, which cannot be grasped from the LiDAR sensing data or the video captured by the camera, and therefore it is possible to align the wireless charging antenna provided in the terminal 2021 such as a car with the wireless power feeding antenna buried in the road, etc. with higher accuracy. As a result, it is possible to shorten the charging speed during wireless charging and improve the charging efficiency.

[0255] The object information may be an object other than the power supply antenna. For example, the three-dimensional data includes the position of a millimeter-wave wireless communication AP as object information. This allows the terminal 2021 to know the position of the AP in advance, and therefore can start communication by directing the beam direction in the direction of the object information. As a result, it is possible to improve communication quality by improving transmission speed, shortening the time until communication starts, and extending the period during which communication is possible.

[0256] The object information may include information indicating the type of object corresponding to the object information. The object information may also include information indicating a process to be performed by the terminal 2021 when the terminal 2021 is within an area in real space corresponding to the position of the object information in the three-dimensional data, or within a predetermined distance from the area.

[0257] The object information may be provided from a server different from the server that provides the three-dimensional data. When the object information is provided separately from the three-dimensional data, object groups that store object information used in the same service may be provided as separate data depending on the type of target service or target device.

[0258] The three-dimensional data used in combination with the object information may be point cloud data of the WLD or feature point data of the SWLD.

[0259] (Embodiment 6) The octree representation and the voxel scanning order will be explained below. A volume is converted into an octree structure (octreeing) and then encoded. The octree structure consists of nodes and leaves. Each node has eight nodes or leaves, and each leaf has voxel (VXL) information. Fig. 22 is a diagram showing an example of the structure of a volume including multiple voxels. Fig. 23 is a diagram showing an example of the volume shown in Fig. 22 converted into an octree structure. Here, among the leaves shown in Fig. 23, leaves 1, 2, and 3 represent voxels VXL1, VXL2, and VXL3 shown in Fig. 22, respectively, and represent a VXL including a point cloud (hereinafter referred to as an effective VXL).

[0260] An octree is represented by a binary sequence of, for example, 0 and 1. For example, if a node or a valid VXL is set to value 1 and the rest to value 0, then the binary sequence shown in FIG. 23 is assigned to each node and leaf. Then, this binary sequence is scanned according to the scan order of breadth-first or depth-first. For example, when scanned breadth-first, the binary sequence shown in A of FIG. 24 is obtained. When scanned depth-first, the binary sequence shown in B of FIG. 24 is obtained. The binary sequence obtained by this scan is coded by entropy coding to reduce the amount of information.

[0261] Next, we will explain depth information in octree representation. Depth in octree representation is used to control the granularity of the point cloud information contained in the volume to be retained. Setting a larger depth allows the point cloud information to be reproduced at a finer level, but the amount of data required to represent nodes and leaves increases. Conversely, setting a smaller depth reduces the amount of data, but since multiple point cloud information with different positions and colors is considered to be in the same position and with the same color, the information contained in the original point cloud information will be lost.

[0262] For example, FIG. 25 is a diagram showing an example in which the octree with depth=2 shown in FIG. 23 is expressed as an octree with depth=1. The octree shown in FIG. 25 has a smaller amount of data than the octree shown in FIG. 23. In other words, the octree shown in FIG. 25 has a smaller number of bits after binarization than the octree shown in FIG. 25. Here, leaf 1 and leaf 2 shown in FIG. 23 are expressed as leaf 1 shown in FIG. 24. In other words, the information that leaf 1 and leaf 2 shown in FIG. 23 were in different positions is lost.

[0263] FIG. 26 is a diagram showing volumes corresponding to the octree shown in FIG. 25. VXL1 and VXL2 shown in FIG. 22 correspond to VXL12 shown in FIG. 26. In this case, the three-dimensional data encoding device generates color information for VXL12 shown in FIG. 26 from the color information for VXL1 and VXL2 shown in FIG. 22. For example, the three-dimensional data encoding device calculates the average value, median value, or weighted average value of the color information for VXL1 and VXL2 as the color information for VXL12. In this way, the three-dimensional data encoding device may control the reduction of data amount by changing the depth of the octree.

[0264] The three-dimensional data encoding device may set the depth information of the octree in any unit of world, space, or volume. In this case, the three-dimensional data encoding device may add the depth information to the header information of the world, the header information of the space, or the header information of the volume. Furthermore, the same value may be used as the depth information for all worlds, spaces, and volumes at different times. In this case, the three-dimensional data encoding device may add the depth information to the header information that manages the worlds of all times.

[0265] (Embodiment 7) The information of a 3D point cloud includes position information (geometry) and attribute information (attribute). The position information includes coordinates (x-coordinate, y-coordinate, z-coordinate) based on a certain point. When encoding the position information, instead of directly encoding the coordinates of each 3D point, a method is used in which the position of each 3D point is expressed in an octree representation and the octree information is encoded to reduce the amount of code.

[0266] On the other hand, the attribute information includes information indicating color information (RGB, YUV, etc.) of each three-dimensional point, reflectance, normal vector, etc. For example, the three-dimensional data encoding device can encode the attribute information using an encoding method different from that for the position information.

[0267] In this embodiment, a method for encoding attribute information will be described. Note that in this embodiment, the attribute information values ​​will be described using integer values. For example, if each color component of RGB or YUV color information has 8-bit precision, each color component will take an integer value between 0 and 255. If the reflectance value has 10-bit precision, the reflectance value will take an integer value between 0 and 1023. Note that if the bit precision of the attribute information is decimal precision, the three-dimensional data encoding device may multiply the value by a scale value and then round it to an integer value so that the value of the attribute information becomes an integer value. Note that the three-dimensional data encoding device may add this scale value to the header of the bitstream, etc.

[0268] One possible method for encoding attribute information of a three-dimensional point is to calculate a predicted value of the attribute information of the three-dimensional point and encode the difference (prediction residual) between the value of the original attribute information and the predicted value. For example, if the value of the attribute information of a three-dimensional point p is Ap and the predicted value is Pp, the three-dimensional data encoding device encodes the absolute difference Diffp = |Ap - Pp|. In this case, if the predicted value Pp can be generated with high accuracy, the value of the absolute difference Diffp will be small. Therefore, for example, the amount of code can be reduced by entropy encoding the absolute difference Diffp using a coding table in which the smaller the value, the fewer bits are generated.

[0269] One method for generating predicted values ​​of attribute information is to use attribute information of a reference 3D point, which is another 3D point surrounding the target 3D point to be encoded. Here, the reference 3D point is a 3D point within a predetermined distance range from the target 3D point. For example, if there are target 3D point p = (x1, y1, z1) and 3D point q = (x2, y2, z2), the 3D data encoding device calculates the Euclidean distance d(p, q) between 3D point p and 3D point q as shown in (Equation A1).

[0270]

number

[0271] If the Euclidean distance d(p, q) is smaller than a predetermined threshold THd, the three-dimensional data encoding device determines that the position of the three-dimensional point q is close to the position of the target three-dimensional point p, and determines to use the value of the attribute information of the three-dimensional point q to generate a predicted value of the attribute information of the target three-dimensional point p. Note that other distance calculation methods may be used, such as Mahalanobis distance. The three-dimensional data encoding device may also determine that three-dimensional points outside a predetermined distance range from the target three-dimensional point are not used in the prediction process. For example, if a three-dimensional point r exists and the distance d(p, r) between the target three-dimensional point p and the three-dimensional point r is equal to or greater than a threshold THd, the three-dimensional data encoding device may determine that the three-dimensional point r is not used for prediction. Note that the three-dimensional data encoding device may add information indicating the threshold THd to a header of the bitstream, for example.

[0272] 27 is a diagram showing an example of a 3D point. In this example, the distance d(p, q) between the target 3D point p and the 3D point q is smaller than the threshold value THd. Therefore, the 3D data encoding device determines that the 3D point q is the reference 3D point of the target 3D point p, and determines that the value of the attribute information Aq of the 3D point q will be used to generate the predicted value Pp of the attribute information Ap of the target 3D point p.

[0273] On the other hand, the distance d(p, r) between the target 3D point p and the 3D point r is equal to or greater than the threshold value THd. Therefore, the 3D data encoding device determines that the 3D point r is not a reference 3D point of the target 3D point p, and determines not to use the value of the attribute information Ar of the 3D point r to generate the predicted value Pp of the attribute information Ap of the target 3D point p.

[0274] Furthermore, when the 3D data encoding device encodes attribute information of a target 3D point using a predicted value, it uses a 3D point whose attribute information has already been encoded and decoded as a reference 3D point. Similarly, when the 3D data decoding device decodes attribute information of a target 3D point to be decoded using a predicted value, it uses a 3D point whose attribute information has already been decoded as a reference 3D point. This allows the same predicted value to be generated during encoding and decoding, so that the bit stream of 3D points generated during encoding can be correctly decoded on the decoding side.

[0275] Furthermore, when encoding attribute information of 3D points, it is possible to classify each 3D point into multiple layers using the position information of the 3D points and then encode them. Here, each classified layer is called LoD (Level of Detail). The method of generating LoD will be explained using FIG. 28.

[0276] First, the three-dimensional data encoding device selects an initial point a0 and assigns it to LoD0. Next, the three-dimensional data encoding device extracts point a1 whose distance from point a0 is greater than the LoD0 threshold Thres_LoD[0] and assigns it to LoD0. Next, the three-dimensional data encoding device extracts point a2 whose distance from point a1 is greater than the LoD0 threshold Thres_LoD[0] and assigns it to LoD0. In this way, the three-dimensional data encoding device constructs LoD0 so that the distance between each point within LoD0 is greater than the threshold Thres_LoD[0].

[0277] Next, the three-dimensional data encoding device selects point b0, which has not yet been assigned an LoD, and assigns it to LoD1. Next, the three-dimensional data encoding device extracts point b1, whose distance from point b0 is greater than the LoD1 threshold Thres_LoD[1] and whose LoD has not been assigned, and assigns it to LoD1. Next, the three-dimensional data encoding device extracts point b2, whose distance from point b1 is greater than the LoD1 threshold Thres_LoD[1] and whose LoD has not been assigned, and assigns it to LoD1. In this way, the three-dimensional data encoding device configures LoD1 so that the distance between each point within LoD1 is greater than the threshold Thres_LoD[1].

[0278] Next, the three-dimensional data encoding device selects point c0, which has not yet been assigned an LoD, and assigns it to LoD2. Next, the three-dimensional data encoding device extracts point c1, which has not yet been assigned an LoD and whose distance from point c0 is greater than the LoD2 threshold Thres_LoD[2], and assigns it to LoD2. Next, the three-dimensional data encoding device extracts point c2, which has not yet been assigned an LoD and whose distance from point c1 is greater than the LoD2 threshold Thres_LoD[2], and assigns it to LoD2. In this way, the three-dimensional data encoding device configures LoD2 so that the distance between each point within LoD2 is greater than the threshold Thres_LoD[2]. For example, as shown in FIG. 29, thresholds Thres_LoD[0], Thres_LoD[1], and Thres_LoD[2] for each LoD are set.

[0279] Furthermore, the three-dimensional data encoding device may add information indicating the threshold values ​​of each LoD to the header of the bitstream, etc. For example, in the example shown in Fig. 29, the three-dimensional data encoding device may add threshold values ​​Thres_LoD[0], Thres_LoD[1], and Thres_LoD[2] to the header.

[0280] Alternatively, the three-dimensional data encoding device may assign all three-dimensional points to which no LoD has been assigned to the lowest layer of the LoD. In this case, the three-dimensional data encoding device can reduce the amount of header coding by not adding the threshold value for the lowest layer of the LoD to the header. For example, in the example shown in FIG. 29, the three-dimensional data encoding device adds thresholds Thres_LoD[0] and Thres_LoD[1] to the header, but does not add Thres_LoD[2] to the header. In this case, the three-dimensional data decoding device may estimate the value of Thres_LoD[2] to be 0. The three-dimensional data encoding device may also add the number of layers of the LoD to the header. This allows the three-dimensional data decoding device to determine the LoD of the lowest layer using the number of layers of the LoD.

[0281] In addition, by setting the threshold value of each LoD layer to be larger for higher layers as shown in Figure 29, the higher the layer (layer closer to LoD0), the sparser the point cloud becomes, with 3D points spaced farther apart, and the lower the layer, the denser the point cloud becomes, with 3D points spaced closer together. In the example shown in Figure 29, LoD0 is the top layer.

[0282] Furthermore, the method for selecting the initial 3D point when setting each LoD may depend on the encoding order used when encoding the position information. For example, the 3D data encoding device may select the 3D point that was encoded first when encoding the position information as the initial point a0 of LoD0, and then select points a1 and a2 using the initial point a0 as the base point to construct LoD0. The 3D data encoding device may then select, as the initial point b0 of LoD1, the 3D point whose position information is encoded earliest among the 3D points that do not belong to LoD0. In other words, the 3D data encoding device may select, as the initial point n0 of LoDn, the 3D point whose position information is encoded earliest among the 3D points that do not belong to the upper layers (LoD0 to LoDn-1) of LoDn. This allows the 3D data decoding device to use the same initial point selection method during decoding to construct the same LoD as during encoding, thereby enabling appropriate decoding of the bitstream. Specifically, the 3D data decoding device selects, as the initial point n0 of LoDn, the 3D point whose position information is decoded earliest among the 3D points that do not belong to the upper layers of LoDn.

[0283] A method for generating predicted values ​​of attribute information of 3D points using LoD information will be described below. For example, when encoding 3D points included in LoD0 in order, a 3D data encoding device generates a target 3D point included in LoD1 using coded and decoded attribute information (hereinafter simply referred to as "coded") included in LoD0 and LoD1. In this way, the 3D data encoding device generates predicted values ​​of attribute information of 3D points included in LoDn using coded attribute information included in LoDn' (n'<=n). In other words, the 3D data encoding device does not use attribute information of 3D points included in a layer below LoDn when calculating predicted values ​​of attribute information of 3D points included in LoDn.

[0284] For example, the three-dimensional data encoding device generates a predicted value of attribute information of a three-dimensional point by calculating the average of attribute values ​​of N or fewer three-dimensional points among encoded three-dimensional points surrounding the target three-dimensional point to be encoded. The three-dimensional data encoding device may also add the value of N to a bitstream header or the like. The three-dimensional data encoding device may also change the value of N for each three-dimensional point and add a value of N to each three-dimensional point. This allows an appropriate N to be selected for each three-dimensional point, thereby improving the accuracy of the predicted value. This reduces the prediction residual. The three-dimensional data encoding device may also add the value of N to the bitstream header and fix the value of N within the bitstream. This eliminates the need to encode or decode the value of N for each three-dimensional point, thereby reducing the amount of processing. The three-dimensional data encoding device may also encode the value of N separately for each LoD. This allows an appropriate N to be selected for each LoD, thereby improving coding efficiency.

[0285] Alternatively, the three-dimensional data encoding device may calculate a predicted value of the attribute information of a three-dimensional point as a weighted average value of the attribute information of N surrounding encoded three-dimensional points. For example, the three-dimensional data encoding device calculates the weight using distance information between the target three-dimensional point and each of the N surrounding three-dimensional points.

[0286] When a 3D data encoding device encodes the value of N separately for each LoD, it sets the value of N larger for higher LoD layers and smaller for lower LoD layers. Because the distance between 3D points in higher LoD layers is greater, it may be possible to improve prediction accuracy by setting the value of N larger and selecting and averaging multiple surrounding 3D points. Also, because the distance between 3D points in lower LoD layers is closer, it is possible to set the value of N smaller, reducing the amount of averaging processing and enabling efficient prediction.

[0287] Fig. 30 is a diagram showing an example of attribute information used for predicted values. As described above, the predicted value of point P included in LoDN is generated using the coded surrounding points P' included in LoDN' (N'<=N). Here, the surrounding points P' are selected based on the distance from point P. For example, the predicted value of the attribute information of point b2 shown in Fig. 30 is generated using the attribute information of points a0, a1, a2, b0, and b1.

[0288] The surrounding points selected vary depending on the value of N. For example, when N=5, a0, a1, a2, b0, and b1 are selected as surrounding points of point b2. When N=4, points a0, a1, a2, and b1 are selected based on distance information.

[0289] The predicted value is calculated by a distance-dependent weighted average. For example, in the example shown in FIG. 30, the predicted value a2p of point a2 is calculated by a weighted average of the attribute information of points a0 and a1, as shown in (Equation A2) and (Equation A3). i is the value of the attribute information of point ai.

[0290]

number

[0291] Furthermore, the predicted value b2p of point b2 is calculated by the weighted average of the attribute information of points a0, a1, a2, b0, and b1, as shown in (Equation A4) to (Equation A6). i is the value of the attribute information of point bi.

[0292]

number

[0293] Furthermore, the three-dimensional data encoding device may calculate a difference value (prediction residual) between the value of the attribute information of the three-dimensional point and a predicted value generated from the surrounding points, and quantize the calculated prediction residual. For example, the three-dimensional data encoding device performs quantization by dividing the prediction residual by a quantization scale (also called a quantization step). In this case, the smaller the quantization scale, the smaller the error (quantization error) that may occur due to quantization. Conversely, the larger the quantization scale, the larger the quantization error.

[0294] The three-dimensional data encoding device may change the quantization scale used for each LoD. For example, the three-dimensional data encoding device may use a smaller quantization scale for higher layers and a larger quantization scale for lower layers. Since the values ​​of attribute information of three-dimensional points belonging to higher layers may be used as predicted values ​​of attribute information of three-dimensional points belonging to lower layers, the encoding efficiency can be improved by reducing the quantization scale for higher layers to suppress quantization errors that may occur in higher layers and increasing the accuracy of predicted values. The three-dimensional data encoding device may add the quantization scale used for each LoD to a header or the like. This allows the three-dimensional data decoding device to correctly decode the quantization scale and therefore appropriately decode the bitstream.

[0295] Furthermore, the three-dimensional data encoding device may convert signed integer values ​​(signed quantized values), which are prediction residuals after quantization, into unsigned integer values ​​(unsigned quantized values). This eliminates the need to consider the occurrence of negative integers when entropy encoding the prediction residuals. Note that the three-dimensional data encoding device does not necessarily need to convert signed integer values ​​into unsigned integer values; for example, the sign bit may be separately entropy encoded.

[0296] The prediction residual is calculated by subtracting a predicted value from an original value. For example, the prediction residual a2r of point a2 is calculated by subtracting the predicted value a2p of point a2 from the attribute information value A2 of point a2, as shown in (Equation A7). The prediction residual b2r of point b2 is calculated by subtracting the predicted value b2p of point b2 from the attribute information value B2 of point b2, as shown in (Equation A8).

[0297] a2r=A2-a2p (Equation A7)

[0298] b2r=B2-b2p (Equation A8)

[0299] Furthermore, the prediction residual is quantized by dividing it by QS (Quantization Step). For example, the quantized value a2q of point a2 is calculated by (Equation A9). The quantized value b2q of point b2 is calculated by (Equation A10). Here, QS_LoD0 is the QS for LoD0, and QS_LoD1 is the QS for LoD1. That is, the QS may be changed depending on the LoD.

[0300] a2q=a2r / QS_LoD0 (Equation A9)

[0301] b2q=b2r / QS_LoD1 (Formula A10)

[0302] Furthermore, the three-dimensional data encoding device converts the signed integer value, which is the quantized value, into an unsigned integer value as follows: If the signed integer value a2q is less than 0, the three-dimensional data encoding device sets the unsigned integer value a2u to -1-(2×a2q). If the signed integer value a2q is 0 or greater, the three-dimensional data encoding device sets the unsigned integer value a2u to 2×a2q.

[0303] Similarly, the three-dimensional data encoding device sets the unsigned integer value b2u to -1-(2×b2q) if the signed integer value b2q is less than 0. The three-dimensional data encoding device sets the unsigned integer value b2u to 2×b2q if the signed integer value b2q is greater than or equal to 0.

[0304] Furthermore, the three-dimensional data encoding device may encode the quantized prediction residuals (unsigned integer values) by entropy encoding. For example, the unsigned integer values ​​may be binarized and then subjected to binary arithmetic encoding.

[0305] In this case, the three-dimensional data encoding device may switch the binarization method depending on the value of the prediction residual. For example, when the prediction residual pu is smaller than a threshold R_TH, the three-dimensional data encoding device binarizes the prediction residual pu using a fixed number of bits required to express the threshold R_TH. When the prediction residual pu is equal to or greater than the threshold R_TH, the three-dimensional data encoding device binarizes the binarized data of the threshold R_TH and the value (pu-R_TH) using Exponential-Golomb or the like.

[0306] For example, if the threshold value R_TH is 63 and the prediction residual pu is smaller than 63, the three-dimensional data encoding device binarizes the prediction residual pu using 6 bits. Furthermore, if the prediction residual pu is 63 or greater, the three-dimensional data encoding device performs arithmetic encoding by binarizing the binary data of the threshold value R_TH (111111) and (pu-63) using Exponential Golomb algorithm.

[0307] In a more specific example, when the prediction residual pu is 32, the three-dimensional data encoding device generates 6-bit binary data (100000) and arithmetically encodes this bit string. Also, when the prediction residual pu is 66, the three-dimensional data encoding device generates binary data (111111) of the threshold R_TH and a bit string (00100) that expresses the value 3 (66-63) in Exponential Golomb notation, and arithmetically encodes this bit string (111111+00100).

[0308] In this way, the 3D data encoding device switches the binarization method depending on the size of the prediction residual, thereby enabling encoding while suppressing a sudden increase in the number of binarization bits when the prediction residual becomes large. Note that the 3D data encoding device may add the threshold value R_TH to the header of the bitstream, etc.

[0309] For example, when encoding is performed at a high bit rate, that is, when the quantization scale is small, the quantization error is small and the prediction accuracy is high, and as a result, the prediction residual may not be large. Therefore, in this case, the three-dimensional data encoding device sets the threshold R_TH to a large value. This reduces the possibility of encoding binarized data of the threshold R_TH, and improves encoding efficiency. Conversely, when encoding is performed at a low bit rate, that is, when the quantization scale is large, the quantization error is large and the prediction accuracy is poor, and as a result, the prediction residual may be large. Therefore, in this case, the three-dimensional data encoding device sets the threshold R_TH to a small value. This prevents a sudden increase in the bit length of the binarized data.

[0310] Furthermore, the three-dimensional data encoding device may switch the threshold R_TH for each LoD and add the threshold R_TH for each LoD to a header or the like. That is, the three-dimensional data encoding device may switch the binarization method for each LoD. For example, in higher layers, the distance between three-dimensional points is large, which may result in poor prediction accuracy and large prediction residuals. Therefore, the three-dimensional data encoding device prevents a sudden increase in the bit length of the binarized data by setting a small threshold R_TH for higher layers. Furthermore, in lower layers, the distance between three-dimensional points is small, which may result in high prediction accuracy and small prediction residuals. Therefore, the three-dimensional data encoding device improves encoding efficiency by setting a large threshold R_TH for each layer.

[0311] 31 is a diagram showing an example of an exponential-Golomb code, and is a diagram showing the relationship between values ​​(multiple values) before binarization and bits (codes) after binarization. Note that 0 and 1 shown in FIG. 31 may be reversed.

[0312] Furthermore, the three-dimensional data encoding device applies arithmetic coding to the binary data of the prediction residual, thereby improving encoding efficiency. Note that when applying arithmetic coding, the tendency of the occurrence probability of 0 and 1 for each bit may differ between the n-bit code, which is the portion of the binary data binarized with n bits, and the remaining code, which is the portion binarized using the Exponential Golomb algorithm. Therefore, the three-dimensional data encoding device may switch the application method of arithmetic coding between the n-bit code and the remaining code.

[0313] For example, a three-dimensional data encoder performs arithmetic coding on an n-bit code using a different coding table (probability table) for each bit. In this case, the three-dimensional data encoder may change the number of coding tables used for each bit. For example, the three-dimensional data encoder performs arithmetic coding on the first bit b0 of an n-bit code using one coding table. The three-dimensional data encoder also uses two coding tables for the next bit b1. The three-dimensional data encoder also switches the coding table used for arithmetic coding of bit b1 depending on the value of b0 (0 or 1). Similarly, the three-dimensional data encoder uses four coding tables for the next bit b2. The three-dimensional data encoder also switches the coding table used for arithmetic coding of bit b2 depending on the values ​​of b0 and b1 (0 to 3).

[0314] In this way, the three-dimensional data encoding device performs arithmetic encoding on each bit bn-1 of the n-bit code. n-1 The three-dimensional data encoding device uses coding tables. In addition, the three-dimensional data encoding device switches the coding table to be used depending on the value (occurrence pattern) of the bits before bn-1. This allows the three-dimensional data encoding device to use an appropriate coding table for each bit, thereby improving coding efficiency.

[0315] Note that the three-dimensional data encoding device may reduce the number of encoding tables used for each bit. For example, when arithmetic-encoding each bit bn-1, the three-dimensional data encoding device may switch between two encoding tables according to the values (occurrence patterns) of the m bits (m < n-1) before bn-1. This can improve the encoding efficiency while suppressing the number of encoding tables used for each bit. Note that the three-dimensional data encoding device may update the occurrence probabilities of 0 and 1 in each encoding table according to the value of the actually generated binarized data. Also, the three-dimensional data encoding device may fix the occurrence probabilities of 0 and 1 in the encoding tables of some bits. This can suppress the number of updates of the occurrence probabilities and thus reduce the processing amount. m Moreover, the three-dimensional data encoding device may switch between two encoding tables according to the values (occurrence patterns) of the m bits (m < n-1) before bn-1 when arithmetic-encoding each bit bn-1. This can improve the encoding efficiency while suppressing the number of encoding tables used for each bit. Note that the three-dimensional data encoding device may update the occurrence probabilities of 0 and 1 in each encoding table according to the value of the actually generated binarized data. Also, the three-dimensional data encoding device may fix the occurrence probabilities of 0 and 1 in the encoding tables of some bits. This can suppress the number of updates of the occurrence probabilities and thus reduce the processing amount.

[0316] For example, when the n-bit code is b0b1b2…bn-1, the encoding table for b0 is one (CTb0). The encoding tables for b1 are two (CTb10, CTb11). Also, the encoding table to be used is switched according to the value (0 to 1) of b0. The encoding tables for b2 are four (CTb20, CTb21, CTb22, CTb23). Also, the encoding table to be used is switched according to the values (0 to 3) of b0 and b1. The encoding tables for bn-1 are two n-1 (CTbn0, CTbn1, …, CTbn(2 n-1 -1)). Also, the encoding table to be used is switched according to the value (0 to 2 n-1 -1) of b0b1…bn-2.

[0317] Note that the three-dimensional data encoding device may apply m-ary arithmetic encoding (m = 2 n ) that sets values from 0 to 2 n -1 without binarization to the n-bit code. Also, when the three-dimensional data encoding device arithmetic-encodes the n-bit code in m-ary, the three-dimensional data decoding device may also restore the n-bit code by m-ary arithmetic decoding.

[0318] 32 is a diagram for explaining processing when the residual code is an exponential-Golomb code, for example. The residual code, which is the portion binarized using the exponential-Golomb code, includes a prefix portion and a suffix portion as shown in FIG. 32. For example, the three-dimensional data encoding device switches encoding tables between the prefix portion and the suffix portion. That is, the three-dimensional data encoding device arithmetically encodes each bit included in the prefix portion using the encoding table for the prefix, and arithmetically encodes each bit included in the suffix portion using the encoding table for the suffix.

[0319] The three-dimensional data encoding device may update the occurrence probabilities of 0 and 1 in each encoding table according to the value of the binary data that actually occurs. Alternatively, the three-dimensional data encoding device may fix the occurrence probabilities of 0 and 1 in one of the encoding tables. This reduces the number of updates to the occurrence probabilities, thereby reducing the amount of processing. For example, the three-dimensional data encoding device may update the occurrence probabilities for the prefix part and fix the occurrence probabilities for the suffix part.

[0320] Furthermore, the three-dimensional data encoding device decodes the quantized prediction residual by inverse quantization and reconstruction, and uses the decoded value, which is the decoded prediction residual, for prediction of the three-dimensional point to be encoded and beyond. Specifically, the three-dimensional data encoding device calculates an inverse quantized value by multiplying the quantized prediction residual (quantized value) by the quantization scale, and obtains a decoded value (reconstructed value) by adding the inverse quantized value and the predicted value.

[0321] For example, the inverse quantization value a2iq of point a2 is calculated by (Equation A11) using the quantization value a2q of point a2. The inverse quantization value b2iq of point b2 is calculated by (Equation A12) using the quantization value b2q of point b2. Here, QS_LoD0 is the QS for LoD0, and QS_LoD1 is the QS for LoD1. In other words, the QS may change depending on the LoD.

[0322] a2iq=a2q×QS_LoD0 (Formula A11)

[0323] b2iq=b2q×QS_LoD1 (Formula A12)

[0324] For example, the decoded value a2rec of point a2 is calculated by adding the predicted value a2p of point a2 to the inverse quantized value a2iq of point a2 as shown in (Equation A13). The decoded value b2rec of point b2 is calculated by adding the predicted value b2p of point b2 to the inverse quantized value b2iq of point b2 as shown in (Equation A14).

[0325] a2rec=a2iq+a2p (formula A13)

[0326] b2rec=b2iq+b2p (formula A14)

[0327] An example of the syntax of a bitstream according to this embodiment will be described below. Fig. 33 is a diagram showing an example of the syntax of an attribute header (attribute_header) according to this embodiment. The attribute header is header information of attribute information. As shown in Fig. 33, the attribute header includes hierarchical number information (NumLoD), three-dimensional point number information (NumOfPoint[i]), hierarchical threshold (Thres_Lod[i]), neighboring point number information (NumNeighorPoint[i]), predicted threshold (THd[i]), quantization scale (QS[i]), and binarization threshold (R_TH[i]).

[0328] The number of layers information (NumLoD) indicates the number of layers of the LoD to be used.

[0329] The three-dimensional point number information (NumOfPoint[i]) indicates the number of three-dimensional points belonging to layer i. The three-dimensional data encoding device may add three-dimensional point total number information (AllNumOfPoint), indicating the total number of three-dimensional points, to a separate header. In this case, the three-dimensional data encoding device does not need to add NumOfPoint[NumLoD-1], indicating the number of three-dimensional points belonging to the lowest layer, to the header. In this case, the three-dimensional data decoding device can calculate NumOfPoint[NumLoD-1] using (Equation A15). This allows the amount of coding in the header to be reduced.

[0330]

Number

[0331] The hierarchical threshold (Thres_Lod[i]) is the threshold used for the setting of layer i. The three-dimensional data encoding device and the three-dimensional data decoding device configure LoDi such that the distance between each point in LoDi is greater than the threshold Thres_LoD[i]. Also, the three-dimensional data encoding device may not add the value of Thres_Lod[NumLoD - 1] (the lowest layer) to the header. In this case, the three-dimensional data decoding device estimates the value of Thres_Lod[NumLoD - 1] as 0. Thereby, the amount of code for the header can be reduced.

[0332] The number of surrounding points information (NumNeighorPoint[i]) indicates the upper limit value of the number of surrounding points used for generating the predicted value of the three-dimensional points belonging to layer i. When the number of surrounding points M is less than NumNeighorPoint[i] (M < NumNeighorPoint[i]), the three-dimensional data encoding device may calculate the predicted value using M surrounding points. Also, when the three-dimensional data encoding device does not need to divide the value of NumNeighorPoint[i] for each LoD, it may add one piece of surrounding points information (NumNeighorPoint) used for all LoDs to the header.

[0333] The prediction threshold (THd[i]) indicates the upper limit value of the distance between the surrounding three-dimensional points and the target three-dimensional point used for predicting the target three-dimensional point to be encoded or decoded at layer i. The three-dimensional data encoding device and the three-dimensional data decoding device do not use the three-dimensional points whose distance from the target three-dimensional point is farther than THd[i] for prediction. Note that when the three-dimensional data encoding device does not need to divide the value of THd[i] for each LoD, it may add one prediction threshold (THd) used for all LoDs to the header.

[0334] The quantization scale (QS[i]) indicates the quantization scale used for quantization and inverse quantization of layer i.

[0335] The binarization threshold (R_TH[i]) is a threshold for switching the binarization method for the prediction residual of a 3D point belonging to layer i. For example, if the prediction residual is smaller than the threshold R_TH, the 3D data encoding device binarizes the prediction residual pu with a fixed number of bits, and if the prediction residual is equal to or greater than the threshold R_TH, the 3D data encoding device binarizes the binarized data of the threshold R_TH and the value (pu-R_TH) using the Exponential Golomb algorithm. Note that if it is not necessary to switch the value of R_TH[i] for each LoD, the 3D data encoding device may add a single binarization threshold (R_TH) used for all LoDs to the header.

[0336] Note that R_TH[i] may be the maximum value that can be represented in n bits. For example, R_TH is 63 in 6 bits, and R_TH is 255 in 8 bits. The three-dimensional data encoding device may also encode the number of bits as the binarization threshold instead of encoding the maximum value that can be represented in n bits. For example, the three-dimensional data encoding device may add a value of 6 to the header when R_TH[i]=63, or a value of 8 to the header when R_TH[i]=255. The three-dimensional data encoding device may also define a minimum value (minimum number of bits) for the number of bits representing R_TH[i], and add the relative number of bits from the minimum value to the header. For example, the three-dimensional data encoding device may add a value of 0 to the header when R_TH[i]=63 and the minimum number of bits is 6, or add a value of 2 to the header when R_TH[i]=255 and the minimum number of bits is 6.

[0337] The three-dimensional data encoding device may also entropy-encode at least one of NumLod, Thres_Lod[i], NumNeighborPoint[i], THd[i], QS[i], and R_TH[i] and add the result to the header. For example, the three-dimensional data encoding device may binarize each value and arithmetically encode it. The three-dimensional data encoding device may also encode each value at a fixed length to reduce the amount of processing.

[0338] Furthermore, the three-dimensional data encoding device may not need to add at least one of NumLod, Thres_Lod[i], NumNeighborPoint[i], THd[i], QS[i], and R_TH[i] to the header. For example, the value of at least one of these may be specified by a profile or level of a standard, etc. This allows the number of bits in the header to be reduced.

[0339] Fig. 34 is a diagram showing an example of the syntax of attribute data (attribute_data) according to this embodiment. This attribute data includes coded data of attribute information of multiple three-dimensional points. As shown in Fig. 34, the attribute data includes an n-bit code and a remaining code.

[0340] An n-bit code is the coded data of the prediction residual of the attribute information value or a part thereof. The bit length of the n-bit code depends on the value of R_TH[i]. For example, if the value indicated by R_TH[i] is 63, the n-bit code is 6 bits, and if the value indicated by R_TH[i] is 255, the n-bit code is 8 bits.

[0341] The remaining code is the coded data of the prediction residual of the attribute information value that is coded using the exponential Golomb coding method. This remaining code is coded or decoded when the n-bit code is the same as R_TH[i]. The three-dimensional data decoding device also decodes the prediction residual by adding the value of the n-bit code and the value of the remaining code. Note that if the n-bit code is not the same value as R_TH[i], the remaining code does not need to be coded or decoded.

[0342] The flow of processing in the three-dimensional data encoding device will be explained below. Fig. 35 is a flowchart of three-dimensional data encoding processing by the three-dimensional data encoding device.

[0343] First, the three-dimensional data encoding device encodes position information (geometry) (S3001). For example, the three-dimensional data is encoded using an octree representation.

[0344] If the position of a three-dimensional point changes due to quantization or the like after encoding the position information, the three-dimensional data encoding device reallocates attribute information of the original three-dimensional point to the changed three-dimensional point (S3002). For example, the three-dimensional data encoding device performs the reallocation by interpolating the value of the attribute information according to the amount of change in position. For example, the three-dimensional data encoding device detects N pre-change three-dimensional points that are close to the changed three-dimensional position, and calculates a weighted average of the attribute information values ​​of the N three-dimensional points. For example, the three-dimensional data encoding device determines a weight in the weighted average based on the distance from the changed three-dimensional position to each of the N three-dimensional points. Then, the three-dimensional data encoding device determines the value obtained by the weighted average as the value of the attribute information of the changed three-dimensional point. Furthermore, if two or more three-dimensional points change to the same three-dimensional position due to quantization or the like, the three-dimensional data encoding device may assign the average value of the attribute information of the two or more pre-change three-dimensional points as the value of the attribute information of the changed three-dimensional point.

[0345] Next, the three-dimensional data encoding device encodes the reallocated attribute information (Attribute) (S3003). For example, when encoding multiple types of attribute information, the three-dimensional data encoding device may encode the multiple types of attribute information in order. For example, when encoding color and reflectance as attribute information, the three-dimensional data encoding device may generate a bitstream in which the encoding result of reflectance is added after the encoding result of color. Note that the order in which the encoding results of the multiple attribute information are added to the bitstream is not limited to this order and may be any order.

[0346] The three-dimensional data encoding device may also add information indicating the start location of the encoded data for each piece of attribute information in the bitstream to a header or the like. This allows the three-dimensional data decoding device to selectively decode attribute information that needs to be decoded, thereby omitting the decoding process for attribute information that does not need to be decoded. This reduces the amount of processing required by the three-dimensional data decoding device. The three-dimensional data encoding device may also encode multiple types of attribute information in parallel and combine the encoding results into a single bitstream. This allows the three-dimensional data encoding device to encode multiple types of attribute information at high speed.

[0347] 36 is a flowchart of the attribute information encoding process (S3003). First, the three-dimensional data encoding device sets the LoD (S3011). That is, the three-dimensional data encoding device assigns each three-dimensional point to one of a plurality of LoDs.

[0348] Next, the three-dimensional data encoding device starts a loop for each LoD (S3012). That is, the three-dimensional data encoding device repeatedly performs the processes of steps S3013 to S3021 for each LoD.

[0349] Next, the three-dimensional data encoding device starts a loop for each three-dimensional point (S3013). That is, the three-dimensional data encoding device repeatedly performs the processes of steps S3014 to S3020 for each three-dimensional point.

[0350] First, the three-dimensional data encoding device searches for a plurality of surrounding points, which are three-dimensional points existing around the target three-dimensional point to be processed, to be used in calculating a predicted value of the target three-dimensional point (S3014). Next, the three-dimensional data encoding device calculates a weighted average of the values ​​of the attribute information of the plurality of surrounding points, and sets the obtained value as the predicted value P (S3015). Next, the three-dimensional data encoding device calculates a prediction residual, which is the difference between the attribute information of the target three-dimensional point and the predicted value (S3016). Next, the three-dimensional data encoding device calculates a quantized value by quantizing the prediction residual (S3017). Next, the three-dimensional data encoding device arithmetically encodes the quantized value (S3018).

[0351] The three-dimensional data encoding device also calculates inverse quantized values ​​by inverse quantizing the quantized values ​​(S3019). Next, the three-dimensional data encoding device generates decoded values ​​by adding predicted values ​​to the inverse quantized values ​​(S3020). Next, the three-dimensional data encoding device ends the loop in units of three-dimensional points (S3021). Also, the three-dimensional data encoding device ends the loop in units of LoDs (S3022).

[0352] Hereinafter, a three-dimensional data decoding process in a three-dimensional data decoding device that decodes a bit stream generated by the above three-dimensional data encoding device will be described.

[0353] The three-dimensional data decoding device generates decoded binary data by arithmetically decoding the binary data of the attribute information in the bit stream generated by the three-dimensional data encoding device using the same method as the three-dimensional data encoding device. Note that, in the three-dimensional data encoding device, if the application method of arithmetic coding is switched between the part binarized with n bits (n-bit code) and the part binarized using Exponential Golomb (remaining code), the three-dimensional data decoding device performs decoding accordingly when applying arithmetic decoding.

[0354] For example, in an arithmetic decoding method for an n-bit code, a three-dimensional data decoding device performs arithmetic decoding using a different coding table (decoding table) for each bit. At this time, the three-dimensional data decoding device may change the number of coding tables used for each bit. For example, arithmetic decoding is performed using one coding table for the first bit b0 of an n-bit code. The three-dimensional data decoding device also uses two coding tables for the next bit b1. The three-dimensional data decoding device also switches the coding table used for arithmetic decoding of bit b1 depending on the value of b0 (0 or 1). Similarly, the three-dimensional data decoding device uses four coding tables for the next bit b2. The three-dimensional data decoding device also switches the coding table used for arithmetic decoding of bit b2 depending on the values ​​of b0 and b1 (0 to 3).

[0355] Thus, when the three-dimensional data decoder arithmetically decodes each bit bn-1 of the n-bit code, it uses n-1 two encoding tables. Also, the three-dimensional data decoder switches the encoding table to be used according to the values (occurrence patterns) of the bits before bn-1. Thereby, the three-dimensional data decoder can appropriately decode a bit stream with improved encoding efficiency by using an appropriate encoding table for each bit.

[0356] Note that the three-dimensional data decoder may reduce the number of encoding tables used for each bit. For example, when the three-dimensional data decoder arithmetically decodes each bit bn-1, it may switch two m encoding tables according to the values (occurrence patterns) of m bits (m < n-1) before bn-1. Thereby, the three-dimensional data decoder can appropriately decode a bit stream with improved encoding efficiency while suppressing the number of encoding tables used for each bit. Note that the three-dimensional data decoder may update the occurrence probabilities of 0 and 1 in each encoding table according to the values of the actually-occurred binarized data. Also, the three-dimensional data decoder may fix the occurrence probabilities of 0 and 1 in the encoding tables of some bits. Thereby, the number of update times of the occurrence probabilities can be suppressed, so the processing amount can be reduced.

[0357] For example, when the n-bit code is b0b1b2…bn-1, the encoding table for b0 is one (CTb0). The encoding tables for b1 are two (CTb10, CTb11). Also, the encoding table is switched according to the value (0 to 1) of b0. The encoding tables for b2 are four (CTb20, CTb21, CTb22, CTb23). Also, the encoding table is switched according to the values (0 to 3) of b0 and b1. The encoding tables for bn-1 are n-1 two (CTbn0, CTbn1, …, CTbn(2 n-1 -1)). Also, the encoding table is switched according to the values (0 to 2 n-1 -1) of b0b1…bn-2.

[0358] Fig. 37 is a diagram for explaining processing when, for example, the residual code is an exponential-Golomb code. The portion (residual code) that the three-dimensional data encoding device binarizes and encodes using the exponential-Golomb code includes a prefix portion and a suffix portion, as shown in Fig. 37. For example, the three-dimensional data decoding device switches the encoding table between the prefix portion and the suffix portion. That is, the three-dimensional data decoding device arithmetically decodes each bit included in the prefix portion using the encoding table for the prefix, and arithmetically decodes each bit included in the suffix portion using the encoding table for the suffix.

[0359] The three-dimensional data decoding device may update the occurrence probabilities of 0 and 1 in each encoding table according to the value of the binarized data generated during decoding. Alternatively, the three-dimensional data decoding device may fix the occurrence probabilities of 0 and 1 in one of the encoding tables. This reduces the number of updates to the occurrence probabilities, thereby reducing the amount of processing. For example, the three-dimensional data decoding device may update the occurrence probabilities for the prefix part and fix the occurrence probabilities for the suffix part.

[0360] Furthermore, the three-dimensional data decoding device decodes the quantized prediction residual (unsigned integer value) by multi-valuing the binary data of the arithmetically decoded prediction residual in accordance with the encoding method used in the three-dimensional data encoding device. The three-dimensional data decoding device first calculates the value of the decoded n-bit code by arithmetically decoding the binary data of the n-bit code. Next, the three-dimensional data decoding device compares the value of the n-bit code with the value of R_TH.

[0361] When the value of the n-bit code matches the value of R_TH, the three-dimensional data decoding device determines that a bit coded using the exponential-Golomb method exists next, and arithmetically decodes the remaining code, which is binary data coded using the exponential-Golomb method.The three-dimensional data decoding device then calculates the value of the remaining code from the decoded remaining code using a reverse lookup table showing the relationship between the remaining code and its value.FIG. 38 is a diagram showing an example of a reverse lookup table showing the relationship between the remaining code and its value.Next, the three-dimensional data decoding device adds the value of the obtained remaining code to R_TH to obtain a multi-valued quantized prediction residual.

[0362] On the other hand, if the value of the n-bit code does not match the value of R_TH (the value is smaller than R_TH), the three-dimensional data decoding device determines the value of the n-bit code as the multi-valued post-quantization prediction residual. This allows the three-dimensional data decoding device to properly decode the bitstream generated by the three-dimensional data encoding device by switching the binarization method according to the value of the prediction residual.

[0363] If the threshold value R_TH is added to the header or the like of the bitstream, the three-dimensional data decoding device may decode the value of the threshold value R_TH from the header and switch the decoding method using the decoded threshold value R_TH. Also, if the threshold value R_TH is added to the header or the like for each LoD, the three-dimensional data decoding device switches the decoding method using the decoded threshold value R_TH for each LoD.

[0364] For example, if the threshold R_TH is 63 and the value of the decoded n-bit code is 63, the three-dimensional data decoding device obtains the value of the residual code by decoding the residual code using the Exponential Golomb method. For example, in the example shown in FIG. 38, the residual code is 00100, and the value of the residual code is obtained as 3. Next, the three-dimensional data decoding device obtains the value of the prediction residual as 66 by adding the value of the threshold R_TH, 63, and the value of the residual code, 3.

[0365] Furthermore, if the value of the decoded n-bit code is 32, the three-dimensional data decoding device sets the value of the n-bit code, 32, as the value of the prediction residual.

[0366] Furthermore, the three-dimensional data decoding device converts the decoded quantized prediction residuals from unsigned integer values ​​to signed integer values, for example, by a process reverse to that performed by the three-dimensional data encoding device. This allows the three-dimensional data decoding device to appropriately decode the generated bitstream when entropy encoding the prediction residuals, without taking into account the occurrence of negative integers. Note that the three-dimensional data decoding device does not necessarily need to convert unsigned integer values ​​to signed integer values; for example, when decoding a bitstream generated by separately entropy encoding the sign bits, the sign bits may be decoded.

[0367] The three-dimensional data decoding device generates decoded values ​​by decoding the quantized prediction residuals converted into signed integer values ​​through inverse quantization and reconstruction. The three-dimensional data decoding device also uses the generated decoded values ​​for prediction of the three-dimensional point to be decoded and thereafter. Specifically, the three-dimensional data decoding device calculates an inverse quantization value by multiplying the quantized prediction residuals by the decoded quantization scale, and obtains a decoded value by adding the inverse quantization value and the prediction value.

[0368] The decoded unsigned integer value (unsigned quantized value) is converted into a signed integer value by the following process. If the LSB (least significant bit) of the decoded unsigned integer value a2u is 1, the three-dimensional data decoding device sets the signed integer value a2q to -((a2u+1)>>1). If the LSB of the unsigned integer value a2u is not 1, the three-dimensional data decoding device sets the signed integer value a2q to (a2u>>1).

[0369] Similarly, if the LSB of the decoded unsigned integer value b2u is 1, the three-dimensional data decoding device sets the signed integer value b2q to -((b2u+1)>>1). If the LSB of the unsigned integer value n2u is not 1, the three-dimensional data decoding device sets the signed integer value b2q to (b2u>>1).

[0370] Furthermore, the details of the inverse quantization and reconstruction processing by the three-dimensional data decoding device are the same as those of the inverse quantization and reconstruction processing by the three-dimensional data encoding device.

[0371] The flow of processing in the three-dimensional data decoding device will be explained below. Fig. 39 is a flowchart of three-dimensional data decoding processing by the three-dimensional data decoding device. First, the three-dimensional data decoding device decodes position information (geometry) from the bitstream (S3031). For example, the three-dimensional data decoding device performs decoding using an octree representation.

[0372] Next, the three-dimensional data decoding device decodes attribute information (Attribute) from the bitstream (S3032). For example, when decoding multiple types of attribute information, the three-dimensional data decoding device may decode the multiple types of attribute information in order. For example, when decoding color and reflectance as attribute information, the three-dimensional data decoding device decodes the encoding result of color and the encoding result of reflectance in the order in which they are added to the bitstream. For example, when the encoding result of reflectance is added after the encoding result of color in the bitstream, the three-dimensional data decoding device decodes the encoding result of color, and then decodes the encoding result of reflectance. Note that the three-dimensional data decoding device may decode the encoding results of attribute information added to the bitstream in any order.

[0373] Furthermore, the three-dimensional data decoding device may obtain information indicating the start location of the encoded data for each piece of attribute information in the bitstream by decoding a header or the like. This allows the three-dimensional data decoding device to selectively decode attribute information that requires decoding, thereby omitting the decoding process for attribute information that does not require decoding. This reduces the amount of processing required by the three-dimensional data decoding device. Furthermore, the three-dimensional data decoding device may decode multiple types of attribute information in parallel and integrate the decoding results into a single three-dimensional point cloud. This allows the three-dimensional data decoding device to decode multiple types of attribute information at high speed.

[0374] 40 is a flowchart of the attribute information decoding process (S3032). First, the three-dimensional data decoding device sets the LoD (S3041). That is, the three-dimensional data decoding device assigns each of the multiple three-dimensional points having decoded position information to one of the multiple LoDs. For example, this assignment method is the same as the assignment method used in the three-dimensional data encoding device.

[0375] Next, the three-dimensional data decoding device starts a loop for each LoD (S3042). That is, the three-dimensional data decoding device repeatedly performs the processes of steps S3043 to S3049 for each LoD.

[0376] Next, the three-dimensional data decoding device starts a loop for each three-dimensional point (S3043). That is, the three-dimensional data decoding device repeatedly performs the processes of steps S3044 to S3048 for each three-dimensional point.

[0377] First, the three-dimensional data decoding device searches for a plurality of surrounding points, which are three-dimensional points existing around the target three-dimensional point to be processed, and are used to calculate a predicted value of the target three-dimensional point (S3044). Next, the three-dimensional data decoding device calculates a weighted average of the values ​​of the attribute information of the plurality of surrounding points, and sets the obtained value as the predicted value P (S3045). Note that these processes are similar to those in the three-dimensional data encoding device.

[0378] Next, the three-dimensional data decoding device arithmetically decodes the quantized value from the bitstream (S3046). The three-dimensional data decoding device also calculates an inverse quantized value by inverse quantizing the decoded quantized value (S3047). Next, the three-dimensional data decoding device generates a decoded value by adding a predicted value to the inverse quantized value (S3048). Next, the three-dimensional data decoding device ends the loop in three-dimensional point units (S3049). The three-dimensional data decoding device also ends the loop in LoD units (S3050).

[0379] Next, the configurations of a three-dimensional data encoding device and a three-dimensional data decoding device according to this embodiment will be described. Fig. 41 is a block diagram showing the configuration of a three-dimensional data encoding device 3000 according to this embodiment. This three-dimensional data encoding device 3000 includes a position information encoding unit 3001, an attribute information reallocation unit 3002, and an attribute information encoding unit 3003.

[0380] The attribute information encoding unit 3003 encodes position information (geometry) of multiple 3D points included in the input point cloud. The attribute information reallocation unit 3002 reallocates values ​​of attribute information of multiple 3D points included in the input point cloud using the results of encoding and decoding the position information. The attribute information encoding unit 3003 encodes the reallocated attribute information. In addition, the 3D data encoding device 3000 generates a bitstream including the encoded position information and the encoded attribute information.

[0381] 42 is a block diagram showing the configuration of a three-dimensional data decoding device 3010 according to this embodiment. This three-dimensional data decoding device 3010 includes a position information decoding unit 3011 and an attribute information decoding unit 3012.

[0382] The position information decoding unit 3011 decodes position information (geometry) of multiple 3D points from the bitstream. The attribute information decoding unit 3012 decodes attribute information (attribute) of multiple 3D points from the bitstream. The 3D data decoding device 3010 generates an output point cloud by combining the decoded position information and the decoded attribute information.

[0383] (Embodiment 8) Hereinafter, a method using RAHT (Region Adaptive Hierarchical Transform) will be described as another method for encoding attribute information of three-dimensional points. Fig. 43 is a diagram for explaining encoding of attribute information using RAHT.

[0384] First, the three-dimensional data encoding device generates a Morton code based on the position information of the three-dimensional points, and sorts the attribute information of the three-dimensional points in Morton code order. For example, the three-dimensional data encoding device may sort in ascending order of the Morton code. Note that the sorting order is not limited to the Morton code order, and other orders may be used.

[0385] Next, the three-dimensional data encoding device applies a Haar transform to the attribute information of two adjacent three-dimensional points in Morton code order to generate high-frequency and low-frequency components of layer L. For example, the three-dimensional data encoding device may use a 2x2 matrix Haar transform. The generated high-frequency components are included in the encoding coefficients as high-frequency components of layer L, and the generated low-frequency components are used as input values ​​for layer L+1, which is higher than layer L.

[0386] After generating high-frequency components for layer L using the attribute information for layer L, the three-dimensional data encoding device continues processing for layer L+1. In processing layer L+1, the three-dimensional data encoding device generates high-frequency and low-frequency components for layer L+1 by applying a Haar transform to two low-frequency components obtained by Haar transforming the attribute information for layer L. The generated high-frequency components are included in the coding coefficients as high-frequency components for layer L+1, and the generated low-frequency components are used as input values ​​for layer L+2, which is higher than layer L+1.

[0387] The three-dimensional data encoding device repeats this hierarchical processing until it determines that it has reached the highest hierarchical level Lmax when only one low-frequency component is input to the hierarchical level. The three-dimensional data encoding device includes the low-frequency component of hierarchical level Lmax-1 input to hierarchical level Lmax in the encoding coefficients. The values ​​of the low-frequency or high-frequency components included in the encoding coefficients are then quantized and encoded using entropy encoding or the like.

[0388] In addition, when applying the Haar transform, if there is only one three-dimensional point as two adjacent three-dimensional points, the three-dimensional data encoding device may use the value of the attribute information of the one existing three-dimensional point as the input value of the higher hierarchy.

[0389] In this way, the three-dimensional data encoding device applies a hierarchical Haar transform to the input attribute information to generate high-frequency and low-frequency components of the attribute information, and then performs encoding by applying quantization, etc., which will be described later, thereby improving encoding efficiency.

[0390] When the attribute information is N-dimensional, the three-dimensional data encoding device may apply a Haar transform to each dimension independently to calculate the respective encoding coefficients. For example, when the attribute information is color information (RGB, YUV, etc.), the three-dimensional data encoding device applies a Haar transform to each component to calculate the respective encoding coefficients.

[0391] The three-dimensional data encoding device may apply the Haar transform in the order of layer L, L+1, ..., layer Lmax. The closer to layer Lmax, the more encoding coefficients are generated that include low-frequency components of the input attribute information.

[0392] w0 and w1 shown in Fig. 43 are weights assigned to each three-dimensional point. For example, the three-dimensional data encoding device may calculate weights based on distance information between two adjacent three-dimensional points to which the Haar transform is applied. For example, the three-dimensional data encoding device may improve encoding efficiency by increasing the weight as the distance becomes shorter. Note that the three-dimensional data encoding device may calculate these weights using a different method, or may not use weights at all.

[0393] In the example shown in Figure 43, the input attribute information is a0, a1, a2, a3, a4, and a5. Of the coding coefficients after Haar transform, Ta1, Ta5, Tb1, Tb3, Tc1, and d0 are coded. The other coding coefficients (b0, b2, c0, etc.) are intermediate values ​​and are not coded.

[0394] 43, a high-frequency component Ta1 and a low-frequency component b0 are generated by performing a Haar transform on a0 and a1. Here, when the weights w0 and w1 are equal, the low-frequency component b0 is the average value of a0 and a1, and the high-frequency component Ta1 is the difference between a0 and a1.

[0395] Since there is no attribute information paired with a2, a2 ​​is used as b1. Similarly, there is no attribute information paired with a3, so a3 is used as b2. Furthermore, by performing a Haar transform on a4 and a5, a high-frequency component Ta5 and a low-frequency component b3 are generated.

[0396] At layer L+1, a Haar transform is performed on b0 and b1 to generate a high-frequency component Tb1 and a low-frequency component c0. Similarly, a Haar transform is performed on b2 and b3 to generate a high-frequency component Tb3 and a low-frequency component c1.

[0397] At the layer Lmax-1, a Haar transform is performed on c0 and c1 to generate a high frequency component Tc1 and a low frequency component d0.

[0398] The three-dimensional data encoding device may quantize the encoding coefficients after applying the Haar transform before encoding. For example, the three-dimensional data encoding device performs quantization by dividing the encoding coefficients by a quantization scale (also called a quantization step (QS)). In this case, the smaller the quantization scale, the smaller the error (quantization error) that may occur due to quantization. Conversely, the larger the quantization scale, the larger the quantization error.

[0399] The three-dimensional data encoding device may change the value of the quantization scale for each layer. FIG. 44 is a diagram showing an example of setting the quantization scale for each layer. For example, the three-dimensional data encoding device may set a smaller quantization scale for higher layers and a larger quantization scale for lower layers. The coding coefficients of three-dimensional points belonging to higher layers contain more low-frequency components than lower layers, and are therefore likely to be important components in terms of human visual characteristics, etc. Therefore, by reducing the quantization scale for higher layers and suppressing quantization errors that may occur in the higher layers, visual degradation can be suppressed and encoding efficiency can be improved.

[0400] The three-dimensional data encoding device may add the quantization scale for each layer to a header, etc. This allows the three-dimensional data decoding device to correctly decode the quantization scale and appropriately decode the bitstream.

[0401] Furthermore, the three-dimensional data encoding device may adaptively switch the value of the quantization scale according to the importance of the target three-dimensional point to be encoded. For example, the three-dimensional data encoding device may use a small quantization scale for three-dimensional points of high importance and a large quantization scale for three-dimensional points of low importance. For example, the three-dimensional data encoding device may calculate the importance from weights used in the Haar transform. For example, the three-dimensional data encoding device may calculate the quantization scale using the sum of w0 and w1. By reducing the quantization scale for three-dimensional points of high importance in this way, the quantization error is reduced, thereby improving encoding efficiency.

[0402] Alternatively, the QS value may be decreased in higher layers, which increases the QW value in higher layers and improves prediction efficiency by reducing the quantization error of the 3D points.

[0403] Here, the coding coefficient Ta1q after quantization of the coding coefficient Ta1 of the attribute information a1 is expressed as Ta1 / QS_L. Note that QS may be the same value in all layers or in some layers.

[0404] QW (Quantization Weight) is a value that represents the importance of the 3D point to be coded. For example, the sum of the above-mentioned w0 and w1 may be used as QW. This makes the QW value larger in higher layers, and reduces the quantization error of the 3D point, thereby improving prediction efficiency.

[0405] For example, the three-dimensional data encoding device may initially initialize the QW values ​​of all three-dimensional points to 1, and then update the QW values ​​of each three-dimensional point using the values ​​of w0 and w1 during the Haar transform. Alternatively, the three-dimensional data encoding device may change the initial value according to the layer, rather than initializing the QW values ​​of all three-dimensional points to the value 1. For example, the quantization scale of a higher layer is reduced by setting a larger initial value for the QW. This reduces prediction errors in the higher layers, thereby improving prediction accuracy in the lower layers and encoding efficiency. Note that the three-dimensional data encoding device does not necessarily have to use the QW.

[0406] When QW is used, the quantized value Ta1q of Ta1 is calculated by (Equation K1) and (Equation K2).

[0407]

number

[0408] Furthermore, the three-dimensional data encoding device scans and encodes the quantized encoding coefficients (unsigned integer values) in a certain order. For example, the three-dimensional data encoding device encodes a plurality of three-dimensional points in order from the three-dimensional points included in the upper layer to the lower layer.

[0409] For example, in the example shown in Figure 43, the three-dimensional data encoding device encodes multiple three-dimensional points included in the upper layer Lmax in the order of d0q to Tc1q, Tb1q, Tb3q, Ta1q, and Ta5q. Here, the lower the layer L, the more likely the coding coefficients after quantization are to become 0. The following factors can be cited as reasons for this.

[0410] The coding coefficients of the lower layer L have higher frequency components than the upper layers, and therefore tend to be 0 depending on the target 3D point. Also, by switching the quantization scale according to the importance, etc., as described above, the quantization scale becomes larger in the lower layers, and the coding coefficients after quantization tend to be 0.

[0411] In this way, the lower the layer, the more likely the coding coefficient after quantization is to become 0, and the more likely it is that consecutive 0s will occur in the first code string. Fig. 45 is a diagram showing examples of the first code string and the second code string.

[0412] The three-dimensional data encoding device counts the number of times the value 0 occurs in the first code string, and encodes the number of times the value 0 occurs in succession instead of consecutive values ​​of 0. In other words, the three-dimensional data encoding device generates a second code string by replacing consecutive coding coefficients of the value 0 in the first code string with the number of consecutive 0s (ZeroCnt). This makes it possible to improve encoding efficiency by encoding the number of consecutive 0s rather than encoding a large number of 0s when consecutive values ​​of 0 occur in coding coefficients after quantization.

[0413] Furthermore, the three-dimensional data encoding device may entropy-encode the value of ZeroCnt. For example, the three-dimensional data encoding device binarizes the value of ZeroCnt using a truncated unary code with a total number T of encoding three-dimensional points, and arithmetically encodes each bit after binarization. FIG. 46 is a diagram showing an example of a truncated unary code when the total number of encoding three-dimensional points is T. In this case, the three-dimensional data encoding device may improve encoding efficiency by using a different encoding table for each bit. For example, the three-dimensional data encoding device uses encoding table 1 for the first bit, encoding table 2 for the second bit, and encoding table 3 for the bits thereafter. In this way, the three-dimensional data encoding device can improve encoding efficiency by switching the encoding table for each bit.

[0414] Furthermore, the three-dimensional data encoding device may perform arithmetic coding after binarizing ZeroCnt using Exponential-Golomb. This can improve efficiency compared to binarized arithmetic coding using truncated unary codes when the value of ZeroCnt is likely to be large. The three-dimensional data encoding device may add a flag to the header for switching between using truncated unary codes and using Exponential-Golomb. This allows the three-dimensional data encoding device to improve encoding efficiency by selecting the optimal binarization method. Furthermore, the three-dimensional data decoding device can switch the binarization method by referring to the flag included in the header, and correctly decode the bitstream.

[0415] The three-dimensional data decoding device may convert the decoded quantized coding coefficients from unsigned integer values ​​to signed integer values ​​using a method reverse to that used by the three-dimensional data encoding device. This allows the three-dimensional data decoding device to properly decode the generated bitstream without considering the occurrence of negative integers when the coding coefficients are entropy coded. Note that the three-dimensional data decoding device does not necessarily have to convert the coding coefficients from unsigned integer values ​​to signed integer values. For example, when decoding a bitstream containing coded bits that have been separately entropy coded, the three-dimensional data decoding device may decode the coded bits.

[0416] The three-dimensional data decoding device decodes the quantized coding coefficients converted into signed integer values ​​by inverse quantization and inverse Haar transform. The three-dimensional data decoding device also uses the decoded coding coefficients for prediction of the three-dimensional point to be decoded and beyond. Specifically, the three-dimensional data decoding device calculates inverse quantized values ​​by multiplying the quantized coding coefficients by the decoded quantization scale. Next, the three-dimensional data decoding device obtains decoded values ​​by applying the inverse Haar transform described below to the inverse quantized values.

[0417] For example, the three-dimensional data decoding device converts the decoded unsigned integer value into a signed integer value in the following manner: If the LSB (least significant bit) of the decoded unsigned integer value a2u is 1, the signed integer value Ta1q is set to -((a2u+1)>>1). If the LSB of the decoded unsigned integer value a2u is not 1 (is 0), the signed integer value Ta1q is set to (a2u>>1).

[0418] The inverse quantization value of Ta1 is expressed as Ta1q×QS_L, where Ta1q is the quantization value of Ta1 and QS_L is the quantization step of layer L.

[0419] Furthermore, the QS may be the same value for all layers or for some layers. Furthermore, the three-dimensional data encoding device may add information indicating the QS to a header, etc. This allows the three-dimensional data decoding device to perform correct inverse quantization using the same QS as that used by the three-dimensional data encoding device.

[0420] Next, the inverse Haar transform will be described. Fig. 47 is a diagram for explaining the inverse Haar transform. The three-dimensional data decoding device decodes the attribute values ​​of the three-dimensional points by applying the inverse Haar transform to the coding coefficients after dequantization.

[0421] First, the three-dimensional data decoding device generates a Morton code based on the position information of the three-dimensional points, and sorts the three-dimensional points in Morton code order. For example, the three-dimensional data decoding device may sort the three-dimensional points in ascending order of the Morton codes. Note that the sorting order is not limited to the Morton code order, and other orders may be used.

[0422] Next, the three-dimensional data decoding device applies an inverse Haar transform to the coding coefficients containing low-frequency components of layer L+1 and the coding coefficients containing high-frequency components of layer L to restore attribute information of adjacent three-dimensional points in Morton code order in layer L. For example, the three-dimensional data decoding device may use an inverse Haar transform of a 2×2 matrix. The restored attribute information of layer L is used as input values ​​for the lower layer L−1.

[0423] The three-dimensional data decoding device repeats this hierarchical processing, and ends the processing when all the attribute information in the lowest layer has been decoded. Note that if only one three-dimensional point exists as two adjacent three-dimensional points in layer L-1 when applying the inverse Haar transform, the three-dimensional data decoding device may substitute the value of the coded component in layer L for the attribute value of the one existing three-dimensional point. In this way, the three-dimensional data decoding device can apply the Haar transform to all values ​​of the input attribute information and correctly decode a bitstream with improved coding efficiency.

[0424] When the attribute information is N-dimensional, the three-dimensional data decoding device may apply an inverse Haar transform to each dimension independently to decode each encoding coefficient. For example, when the attribute information is color information (RGB, YUV, etc.), the three-dimensional data decoding device applies an inverse Haar transform to the encoding coefficient for each component to decode each attribute value.

[0425] The three-dimensional data decoding device may apply the inverse Haar transform in the order of layers Lmax, L+1, ..., layer L. Furthermore, w0 and w1 shown in FIG. 47 are weights assigned to each three-dimensional point. For example, the three-dimensional data decoding device may calculate a weight based on distance information between two adjacent three-dimensional points to which the inverse Haar transform is applied. For example, the three-dimensional data encoding device may decode a bitstream with improved encoding efficiency by increasing the weight as the distance becomes shorter.

[0426] In the example shown in FIG. 47, the coding coefficients after dequantization are Ta1, Ta5, Tb1, Tb3, Tc1, and d0, and the decoded values ​​obtained are a0, a1, a2, a3, a4, and a5.

[0427] 48 is a diagram illustrating an example of the syntax of attribute information (attribute_data). The attribute information (attribute_data) includes the number of consecutive zeros (ZeroCnt), the number of attribute dimensions (attribute_dimension), and the coding coefficient (value[j][i]).

[0428] The number of consecutive zeros (ZeroCnt) indicates the number of consecutive 0s in the coding coefficient after quantization. Note that the three-dimensional data coding device may binarize ZeroCnt and then perform arithmetic coding.

[0429] 48, the three-dimensional data encoding device may determine whether the layer L (layer L) to which the encoding coefficient belongs is equal to or greater than a predetermined threshold TH_layer, and may switch the information to be added to the bitstream depending on the determination result. For example, if the determination result is true, the three-dimensional data encoding device may add all the encoding coefficients of the attribute information to the bitstream. Alternatively, if the determination result is false, the three-dimensional data encoding device may add only some of the encoding coefficients to the bitstream.

[0430] Specifically, if the determination result is true, the three-dimensional data encoding device adds the encoding result of the RGB or YUV three-dimensional color information to the bitstream. If the determination result is false, the three-dimensional data encoding device adds some of the color information, such as G or Y, to the bitstream, and does not need to add other components to the bitstream. In this way, the three-dimensional data encoding device can improve encoding efficiency by not adding some of the encoding coefficients of layers (layers lower than the TH_layer) that include encoding coefficients indicating high-frequency components whose degradation is less noticeable visually to the bitstream.

[0431] The attribute dimension number (attribute_dimension) indicates the number of dimensions of the attribute information. For example, if the attribute information is color information (RGB or YUV, etc.) of a three-dimensional point, the color information is three-dimensional, so the attribute dimension number is set to a value of 3. If the attribute information is reflectance, the reflectance is one-dimensional, so the attribute dimension number is set to a value of 1. The attribute dimension number may be added to the header of the attribute information of the bitstream, etc.

[0432] The coding coefficient (value[j][i]) indicates the coding coefficient of the j-th dimensional attribute information of the i-th 3D point after quantization. For example, if the attribute information is color information, value

[99] [1] indicates the coding coefficient of the second dimensional information (e.g., G value) of the 100th 3D point. Also, if the attribute information is reflectance information, value

[0119] [0] indicates the coding coefficient of the first dimensional information (e.g., reflectance) of the 120th 3D point.

[0433] If the following condition is satisfied, the three-dimensional data encoding device may subtract the value 1 from value[j][i] and entropy encode the resulting value. In this case, the three-dimensional data decoding device restores the encoding coefficient by adding the value 1 to value[j][i] after entropy decoding.

[0434] The above conditions are (1) when attribute_dimension = 1, or (2) when attribute_dimension is 1 or greater and all dimension values ​​are equal. For example, if the attribute information is reflectance, attribute_dimension = 1, so the three-dimensional data encoder subtracts the value 1 from the encoding coefficient to calculate the value and encodes the calculated value. The three-dimensional data decoder adds the value 1 to the decoded value to calculate the encoding coefficient.

[0435] More specifically, for example, if the encoding coefficient for reflectance is 10, the three-dimensional data encoding device encodes the value 9 obtained by subtracting the value 1 from the encoding coefficient value 10. The three-dimensional data decoding device adds the value 1 to the decoded value 9 to calculate the encoding coefficient value 10.

[0436] Furthermore, since attribute_dimension=3 when the attribute information is color, for example, if the quantized coding coefficients of each R, G, and B component are the same, the three-dimensional data coding device subtracts a value of 1 from each coding coefficient and encodes the obtained value. The three-dimensional data decoding device adds a value of 1 to the decoded value. More specifically, for example, if the coding coefficients of R, G, and B=(1, 1, 1), the three-dimensional data coding device encodes (0, 0, 0). The three-dimensional data decoding device adds 1 to each component of (0, 0, 0) to calculate (1, 1, 1). Furthermore, if the coding coefficients of R, G, and B=(2, 1, 2), the three-dimensional data coding device encodes (2, 1, 2) as is. The three-dimensional data decoding device uses the decoded (2, 1, 2) as is as the coding coefficient.

[0437] In this way, by providing ZeroCnt, a pattern in which all dimensions of value are 0 is not generated, so that the value obtained by subtracting 1 from the value of value can be coded. This improves coding efficiency.

[0438] Also, value[0][i] shown in Fig. 48 indicates the coding coefficient after quantization of the first-dimensional attribute information of the i-th three-dimensional point. As shown in Fig. 48, if the layer L (layerL) to which the coding coefficient belongs is smaller than the threshold TH_layer, the first-dimensional attribute information may be added to the bitstream (attribute information from the second dimension onwards is not added to the bitstream), thereby reducing the amount of code.

[0439] The three-dimensional data encoding device may switch the calculation method of the ZeroCnt value depending on the value of attribute_dimension. For example, when attribute_dimension=3, the three-dimensional data encoding device may count the number of times the values ​​of the encoding coefficients of all components (dimensions) are 0. FIG. 49 is a diagram showing an example of the encoding coefficients and ZeroCnt in this case. For example, in the case of the color information shown in FIG. 49, the three-dimensional data encoding device counts the number of consecutive encoding coefficients in which the R, G, and B components are all 0, and adds the counted number to the bitstream as ZeroCnt. This eliminates the need to encode ZeroCnt for each component, thereby reducing overhead. This improves encoding efficiency. Note that the three-dimensional data encoding device may calculate ZeroCnt for each dimension even when attribute_dimension is 2 or more, and add the calculated ZeroCnt to the bitstream.

[0440] 50 is a flowchart of three-dimensional data encoding processing according to this embodiment. First, the three-dimensional data encoding device encodes position information (geometry) (S6601). For example, the three-dimensional data encoding device performs encoding using an octree representation.

[0441] Next, the three-dimensional data encoding device converts the attribute information (S6602). For example, if the position of a three-dimensional point changes due to quantization or the like after encoding the position information, the three-dimensional data encoding device reallocates the attribute information of the original three-dimensional point to the changed three-dimensional point. The three-dimensional data encoding device may perform the reallocation by interpolating the value of the attribute information according to the amount of change in position. For example, the three-dimensional data encoding device detects N pre-change three-dimensional points that are close to the changed three-dimensional position, weights the values ​​of the attribute information of the N three-dimensional points based on the distance from the changed three-dimensional position to each of the N three-dimensional points, and sets the obtained value as the value of the attribute information of the changed three-dimensional point. Furthermore, if two or more three-dimensional points change to the same three-dimensional position due to quantization or the like, the three-dimensional data encoding device may assign the average value of the attribute information of the two or more pre-change three-dimensional points as the value of the attribute information of the changed three-dimensional point.

[0442] Next, the three-dimensional data encoding device encodes the attribute information (S6603). For example, when encoding multiple pieces of attribute information, the three-dimensional data encoding device may encode the multiple pieces of attribute information in order. For example, when encoding color and reflectance as attribute information, the three-dimensional data encoding device generates a bitstream in which the encoding result of reflectance is added after the encoding result of color. Note that the encoding results of the multiple pieces of attribute information added to the bitstream may be in any order.

[0443] The three-dimensional data encoding device may also add information indicating the start location of the encoded data for each piece of attribute information in the bitstream to a header or the like. This allows the three-dimensional data decoding device to selectively decode attribute information that needs to be decoded, thereby omitting the decoding process for attribute information that does not need to be decoded. This reduces the amount of processing required by the three-dimensional data decoding device. The three-dimensional data encoding device may also encode multiple pieces of attribute information in parallel and combine the encoding results into a single bitstream. This allows the three-dimensional data encoding device to encode multiple pieces of attribute information at high speed.

[0444] 51 is a flowchart of the attribute information encoding process (S6603). First, the three-dimensional data encoding device generates encoding coefficients from the attribute information using Haar transform (S6611). Next, the three-dimensional data encoding device applies quantization to the encoding coefficients (S6612). Next, the three-dimensional data encoding device generates encoded attribute information (bit stream) by encoding the quantized encoding coefficients (S6613).

[0445] Furthermore, the three-dimensional data encoding device applies inverse quantization to the quantized coding coefficients (S6614). Next, the three-dimensional data decoding device decodes attribute information by applying an inverse Haar transform to the inverse quantized coding coefficients (S6615). For example, the decoded attribute information is referenced in subsequent encoding.

[0446] FIG. 52 is a flowchart of the coding coefficient coding process (S6613). First, the three-dimensional data coding device converts the coding coefficient from a signed integer value to an unsigned integer value (S6621). For example, the three-dimensional data coding device converts the signed integer value to an unsigned integer value as follows: If the signed integer value Ta1q is smaller than 0, the unsigned integer value is set to -1-(2×Ta1q). If the signed integer value Ta1q is 0 or greater, the unsigned integer value is set to 2×Ta1q. Note that if the coding coefficient does not become a negative value, the three-dimensional data coding device may code the coding coefficient as an unsigned integer value as is.

[0447] If all coding coefficients have not been processed (No in S6622), the three-dimensional data coding device determines whether the value of the coding coefficient to be processed is zero (S6623). If the value of the coding coefficient to be processed is zero (Yes in S6623), the three-dimensional data coding device increments ZeroCnt by 1 (S6624) and returns to step S6622.

[0448] If the value of the coding coefficient to be processed is not zero (No in S6623), the three-dimensional data coding device encodes ZeroCnt and resets ZeroCnt to 0 (S6625). The three-dimensional data coding device also arithmetically codes the coding coefficient to be processed (S6626) and returns to step S6622. For example, the three-dimensional data coding device performs binary arithmetic coding. The three-dimensional data coding device may also subtract a value of 1 from the coding coefficient and code the resulting value.

[0449] Furthermore, the processes of steps S6623 to S6626 are repeated for each coding coefficient. Furthermore, if all coding coefficients have been processed (Yes in S6622), the three-dimensional data coding device ends the process.

[0450] 53 is a flowchart of three-dimensional data decoding processing according to this embodiment. First, the three-dimensional data decoding device decodes position information (geometry) from the bitstream (S6631). For example, the three-dimensional data decoding device performs decoding using an octree representation.

[0451] Next, the three-dimensional data decoding device decodes the attribute information from the bitstream (S6632). For example, when decoding multiple pieces of attribute information, the three-dimensional data decoding device may decode the multiple pieces of attribute information in order. For example, when decoding color and reflectance as attribute information, the three-dimensional data decoding device decodes the encoding result of color and the encoding result of reflectance in the order in which they are added to the bitstream. For example, when the encoding result of reflectance is added after the encoding result of color in the bitstream, the three-dimensional data decoding device decodes the encoding result of color, and then decodes the encoding result of reflectance. Note that the three-dimensional data decoding device may decode the encoding results of the attribute information added to the bitstream in any order.

[0452] Furthermore, the three-dimensional data decoding device may obtain information indicating the start location of the coded data for each piece of attribute information in the bitstream by decoding a header or the like. This allows the three-dimensional data decoding device to selectively decode attribute information that needs to be decoded, thereby omitting the decoding process for attribute information that does not need to be decoded. This reduces the amount of processing required by the three-dimensional data decoding device. Furthermore, the three-dimensional data decoding device may decode multiple pieces of attribute information in parallel and integrate the decoding results into a single three-dimensional point cloud. This allows the three-dimensional data decoding device to decode multiple pieces of attribute information at high speed.

[0453] 54 is a flowchart of the attribute information decoding process (S6632). First, the three-dimensional data decoding device decodes the coding coefficients from the bitstream (S6641). Next, the three-dimensional data decoding device applies inverse quantization to the coding coefficients (S6642). Next, the three-dimensional data decoding device decodes the attribute information by applying an inverse Haar transform to the coding coefficients after the inverse quantization (S6643).

[0454] 55 is a flowchart of the coding coefficient decoding process (S6641). First, the three-dimensional data decoding device decodes ZeroCnt from the bitstream (S6651). If all coding coefficients have not been processed (No in S6652), the three-dimensional data decoding device determines whether ZeroCnt is greater than 0 (S6653).

[0455] If ZeroCnt is greater than zero (Yes in S6653), the three-dimensional data decoding device sets the coding coefficient to be processed to 0 (S6654). Next, the three-dimensional data decoding device subtracts 1 from ZeroCnt (S6655) and returns to step S6652.

[0456] If ZeroCnt is zero (No in S6653), the three-dimensional data decoding device decodes the coding coefficient to be processed (S6656). For example, the three-dimensional data decoding device uses binary arithmetic decoding. The three-dimensional data decoding device may also add a value of 1 to the decoded coding coefficient.

[0457] Next, the three-dimensional data decoding device decodes ZeroCnt, sets the obtained value to ZeroCnt (S6657), and returns to step S6652.

[0458] Furthermore, the processes of steps S6653 to S6657 are repeated for each coding coefficient. Furthermore, if all coding coefficients have been processed (Yes in S6652), the three-dimensional data decoding device converts the decoded coding coefficients from unsigned integer values ​​to signed integer values ​​(S6658). For example, the three-dimensional data decoding device may convert the decoded coding coefficients from unsigned integer values ​​to signed integer values ​​as follows: If the LSB (least significant bit) of the decoded unsigned integer value Ta1u is 1, the signed integer value Ta1q is set to -((Ta1u+1)>>1). If the LSB of the decoded unsigned integer value Ta1u is not 1 (is 0), the signed integer value Ta1q is set to (Ta1u>>1). Note that if the coding coefficient does not become a negative value, the three-dimensional data decoding device may use the decoded coding coefficient as a signed integer value as is.

[0459] 56 is a block diagram of an attribute information encoding unit 6600 included in the three-dimensional data encoding device. The attribute information encoding unit 6600 includes a sorting unit 6601, a Haar transform unit 6602, a quantization unit 6603, an inverse quantization unit 6604, an inverse Haar transform unit 6605, a memory 6606, and an arithmetic encoding unit 6607.

[0460] The sorting unit 6601 generates a Morton code using the position information of the 3D points and sorts the multiple 3D points in Morton code order. The Haar transform unit 6602 generates coding coefficients by applying a Haar transform to the attribute information. The quantization unit 6603 quantizes the coding coefficients of the attribute information.

[0461] The inverse quantization unit 6604 inversely quantizes the quantized coding coefficients. The inverse Haar transform unit 6605 applies an inverse Haar transform to the coding coefficients. The memory 6606 stores values ​​of attribute information of multiple decoded 3D points. For example, the attribute information of the decoded 3D points stored in the memory 6606 may be used for predicting uncoded 3D points.

[0462] The arithmetic coding unit 6607 calculates ZeroCnt from the quantized coding coefficients and arithmetically codes the ZeroCnt. The arithmetic coding unit 6607 also arithmetically codes the quantized non-zero coding coefficients. The arithmetic coding unit 6607 may binarize the coding coefficients before arithmetic coding. The arithmetic coding unit 6607 may also generate and code various header information.

[0463] 57 is a block diagram of the attribute information decoding unit 6610 included in the three-dimensional data decoding device. The attribute information decoding unit 6610 includes an arithmetic decoding unit 6611, an inverse quantization unit 6612, an inverse Haar transform unit 6613, and a memory 6614.

[0464] The arithmetic decoding unit 6611 arithmetically decodes the ZeroCnt and coding coefficients included in the bitstream. Note that the arithmetic decoding unit 6611 may also decode various types of header information.

[0465] The inverse quantization unit 6612 inverse quantizes the arithmetically decoded coding coefficients. The inverse Haar transform unit 6613 applies inverse Haar transform to the inversely quantized coding coefficients. The memory 6614 stores values ​​of attribute information of multiple decoded 3D points. For example, the attribute information of the decoded 3D points stored in the memory 6614 may be used to predict undecoded 3D points.

[0466] In the above embodiment, an example is shown in which the three-dimensional points are coded in the order from the lower layer to the upper layer as the coding order, but this is not necessarily limited to this. For example, a method of scanning the coding coefficients after Haar transform in the order from the upper layer to the lower layer may be used. In this case, the three-dimensional data coding device may also code the number of consecutive zeros as ZeroCnt.

[0467] Furthermore, the three-dimensional data encoding device may switch whether to use the encoding method using ZeroCnt described in this embodiment on a WLD, SPC, or volume basis. In this case, the three-dimensional data encoding device may add information indicating whether the encoding method using ZeroCnt has been applied to the header information. This allows the three-dimensional data decoding device to perform decoding appropriately. As an example of the switching method, the three-dimensional data encoding device counts the number of occurrences of coding coefficients with a value of 0 for one volume. If the count value exceeds a predetermined threshold, the three-dimensional data encoding device applies the method using ZeroCnt to the next volume. If the count value is equal to or less than the threshold, the three-dimensional data encoding device does not apply the method using ZeroCnt to the next volume. This allows the three-dimensional data encoding device to appropriately switch whether to apply the encoding method using ZeroCnt depending on the characteristics of the three-dimensional point to be encoded, thereby improving encoding efficiency.

[0468] Another technique (variant) of this embodiment will be described below. The three-dimensional data encoding device scans and encodes quantized encoding coefficients (unsigned integer values) in a certain order. For example, the three-dimensional data encoding device encodes multiple three-dimensional points in order from three-dimensional points included in a lower layer toward higher layers.

[0469] Figure 58 is a diagram showing examples of the first code string and the second code string when this technique is used for the attribute information shown in Figure 43. In this example, the three-dimensional data encoding device encodes multiple coding coefficients included in the lower layer L in the order Ta1q to Ta5q, Tb1q, Tb3q, Tc1q, and d0q. Here, the lower the layer, the more likely the coding coefficients after quantization are to be 0. The following factors can be cited as reasons for this.

[0470] The coding coefficients of the lower layer L have higher frequency components than the upper layers, and therefore tend to be 0 depending on the 3D point to be coded. Also, by switching the quantization scale according to the importance, etc., as described above, the quantization scale becomes larger in the lower layers, and the coding coefficients after quantization tend to be 0.

[0471] In this way, the lower the layer, the more likely the coding coefficient after quantization is to become 0, and the more likely consecutive 0s are to occur in the first code string. The three-dimensional data coding device counts the number of times that 0 occurs in the first code string, and encodes the number of times that 0 occurs in succession (ZeroCnt) instead of consecutive 0s. This makes it possible to improve coding efficiency when consecutive 0s occur in coding coefficients after quantization, by encoding the number of consecutive 0s rather than encoding a large number of 0s.

[0472] Furthermore, the three-dimensional data encoding device may encode information indicating the total number of occurrences of the value 0. This reduces the overhead of encoding ZeroCnt and improves encoding efficiency.

[0473] For example, the three-dimensional data encoding device encodes the total number of coding coefficients with a value of 0 as TotalZeroCnt. As a result, in the example shown in FIG. 58, when the three-dimensional data decoding device decodes the second ZeroCnt (value 1) included in the second code string, the total number of decoded ZeroCnt becomes N+1 (=TotalZeroCnt). Therefore, the three-dimensional data decoding device can determine that no more zeros will occur. Therefore, the three-dimensional data encoding device does not need to encode ZeroCnt for each value thereafter, and the amount of code can be reduced.

[0474] Furthermore, the three-dimensional data encoding device may perform entropy encoding on TotalZeroCnt. For example, the three-dimensional data encoding device may binarize the value of TotalZeroCnt using a truncated unary code with a total number T of encoded three-dimensional points, and then arithmetically encode each bit after binarization. In this case, the three-dimensional data encoding device may improve encoding efficiency by using a different encoding table for each bit. For example, the three-dimensional data encoding device may use encoding table 1 for the first bit, encoding table 2 for the second bit, and encoding table 3 for the bits thereafter. In this way, the three-dimensional data encoding device can improve encoding efficiency by switching the encoding table for each bit.

[0475] The three-dimensional data encoding device may also perform arithmetic coding after binarizing TotalZeroCnt using exponential Golomb coding. This can improve efficiency compared to binarized arithmetic coding using truncated unary codes when the value of TotalZeroCnt is likely to be large. The three-dimensional data encoding device may also add a flag to the header to switch between using truncated unary codes and exponential Golomb coding. This allows the three-dimensional data encoding device to improve encoding efficiency by selecting the optimal binarization method. The three-dimensional data decoding device can also switch binarization methods by referring to the flag included in the header, thereby correctly decoding the bitstream.

[0476] Fig. 59 is a diagram showing an example of the syntax of attribute information (attribute_data) in this modification. The attribute information (attribute_data) shown in Fig. 59 further includes a total number of zeros (TotalZeroCnt) in addition to the attribute information shown in Fig. 48. Note that other information is the same as in Fig. 48. The total number of zeros (TotalZeroCnt) indicates the total number of coding coefficients with a value of 0 after quantization.

[0477] Furthermore, the three-dimensional data encoding device may switch the calculation method for the values ​​of TotalZereCnt and ZeroCnt depending on the value of attribute_dimension. For example, when attribute_dimension=3, the three-dimensional data encoding device may count the number of times the values ​​of the encoding coefficients of all components (dimensions) are 0. FIG. 60 is a diagram showing examples of the encoding coefficients, ZeroCnt, and TotalZeroCnt in this case. For example, in the case of the color information shown in FIG. 60, the three-dimensional data encoding device counts the number of consecutive encoding coefficients in which the R, G, and B components are all 0, and adds the counted numbers to the bitstream as TotalZeroCnt and ZeroCnt. This eliminates the need to encode TotalZeroCnt and ZeroCnt for each component, thereby reducing overhead. This improves encoding efficiency. Note that the three-dimensional data encoding device may calculate TotalZeroCnt and ZeroCnt for each dimension even when attribute_dimension is 2 or more, and add the calculated TotalZeoCnt and ZeroCnt to the bitstream.

[0478] 61 is a flowchart of the coding coefficient coding process (S6613) in this modification. First, the three-dimensional data coding device converts the coding coefficients from signed integer values ​​to unsigned integer values ​​(S6661). Next, the three-dimensional data coding device codes TotalZeroCnt (S6662).

[0479] If all coding coefficients have not been processed (No in S6663), the three-dimensional data coding device determines whether the value of the coding coefficient to be processed is zero (S6664). If the value of the coding coefficient to be processed is zero (Yes in S6664), the three-dimensional data coding device increments ZeroCnt by 1 (S6665) and returns to step S6663.

[0480] If the value of the coding coefficient to be processed is not zero (No in S6664), the three-dimensional data coding device determines whether TotalZeroCnt is greater than 0 (S6666). If TotalZeroCnt is greater than 0 (Yes in S6666), the three-dimensional data coding device encodes ZeroCnt and sets TotalZeroCnt to TotalZeroCnt-ZeroCnt (S6667).

[0481] After step S6667, or if TotalZeroCnt is 0 (No in S6666), the three-dimensional data encoding device encodes the encoding coefficients, resets ZeroCnt to 0 (S6668), and returns to step S6663. For example, the three-dimensional data encoding device performs binary arithmetic encoding. Alternatively, the three-dimensional data encoding device may subtract a value of 1 from the encoding coefficients and encode the obtained values.

[0482] Furthermore, the processes of steps S6664 to S6668 are repeated for each coding coefficient. Furthermore, if all coding coefficients have been processed (Yes in S6663), the three-dimensional data coding device ends the process.

[0483] 62 is a flowchart of the coding coefficient decoding process (S6641) in this modification. First, the three-dimensional data decoding device decodes TotalZeroCnt from the bitstream (S6671). Next, the three-dimensional data decoding device decodes ZeroCnt from the bitstream and sets TotalZeroCnt to TotalZeroCnt-ZeroCnt (S6672).

[0484] If all the coding coefficients have not been processed (No in S6673), the three-dimensional data coding device determines whether ZeroCnt is greater than 0 (S6674).

[0485] If ZeroCnt is greater than zero (Yes in S6674), the three-dimensional data decoding apparatus sets the coding coefficient to be processed to 0 (S6675). Next, the three-dimensional data decoding apparatus subtracts 1 from ZeroCnt (S6676) and returns to step S6673.

[0486] If ZeroCnt is zero (No in S6674), the three-dimensional data decoding device decodes the coding coefficient to be processed (S6677). For example, the three-dimensional data decoding device uses binary arithmetic decoding. The three-dimensional data decoding device may also add a value of 1 to the decoded coding coefficient.

[0487] Next, the three-dimensional data decoding device determines whether TotalZeroCnt is greater than 0 (S6678). If TotalZeroCnt is greater than 0 (Yes in S6678), the three-dimensional data decoding device decodes ZeroCnt, sets the obtained value to ZeroCnt, sets TotalZeroCnt to TotalZeroCnt-ZeroCnt (S6679), and returns to step S6673. If TotalZeroCnt is 0 (No in S6678), the three-dimensional data decoding device returns to step S6673.

[0488] Furthermore, the processes of steps S6674 to S6679 are repeated for each coding coefficient. Furthermore, if all coding coefficients have been processed (Yes in S6673), the three-dimensional data coding device converts the decoded coding coefficients from unsigned integer values ​​to signed integer values ​​(S6680).

[0489] Figure 63 is a diagram showing another syntax example of attribute information (attribute_data). The attribute information (attribute_data) shown in Figure 63 includes value[j][i]_greater_zero_flag, value[j][i]_greater_one_flag, and value[j][i] instead of the coding coefficients (value[j][i]) shown in Figure 48. Note that other information is the same as in Figure 48.

[0490] The value[j][i]_greater_zero_flag indicates whether the value of the coding coefficient (value[j][i]) is greater than 0 (is greater than or equal to 1). In other words, the value[j][i]_greater_zero_flag indicates whether the value of the coding coefficient (value[j][i]) is 0.

[0491] For example, if the value of the coding coefficient is greater than 0, value[j][i]_greater_zero_flag is set to the value 1, and if the value of the coding coefficient is 0, value[j][i]_greater_zero_flag is set to the value 0. If the value of value[j][i]_greater_zero_flag is 0, the three-dimensional data coding device may not add value[j][i] to the bitstream. In this case, the three-dimensional data decoding device may determine that the value of value[j][i] is 0. This makes it possible to reduce the amount of coding.

[0492] value[j][i]_greater_one_flag indicates whether the value of the coding coefficient (value[j][i]) is greater than 1 (2 or more). In other words, value[j][i]_greater_one_flag indicates whether the value of the coding coefficient (value[j][i]) is 1.

[0493] For example, if the value of the coding coefficient is greater than 1, value[j][i]_greater_one_flag is set to the value 1. Otherwise (if the value of the coding coefficient is 1 or less), value[j][i]_greater_one_flag is set to the value 0. If the value of value[j][i]_greater_one_flag is 0, the three-dimensional data coding device may not add value[j][i] to the bitstream. In this case, the three-dimensional data decoding device may determine that the value of value[j][i] is 1.

[0494] Value[j][i] indicates the quantized coding coefficient of the j-th attribute information of the i-th 3D point. For example, if the attribute information is color information, value

[99] [1] indicates the coding coefficient of the second dimension (e.g., G value) of the 100th 3D point. Also, if the attribute information is reflectance information, value

[0119] [0] indicates the coding coefficient of the first dimension (e.g., reflectance) of the 120th 3D point.

[0495] The three-dimensional data encoding device may add value[j][i] to the bitstream when value[j][i]_greater_zero_flag=1 and value[j][i]_greater_one_flag=1. The three-dimensional data encoding device may also add a value obtained by subtracting 2 from value[j][i] to the bitstream. In this case, the three-dimensional data decoding device calculates the encoding coefficient by adding the value 2 to the decoded value[j][i].

[0496] The three-dimensional data encoding device may entropy-encode value[j][i]_greater_zero_flag and value[j][i]_greater_one_flag. For example, binary arithmetic encoding and binary arithmetic decoding may be used. This can improve encoding efficiency.

[0497] (Embodiment 9) To achieve high compression, attribute information included in PCC (Point Cloud Compression) data is transformed using multiple methods, such as Lifting, RAHT (Region Adaptive Hierarchical Transform), or other transformation methods. Here, Lifting is a transformation method using LoD (Level of Detail).

[0498] Since important signal information tends to be contained in low-frequency components, quantizing high-frequency components reduces the amount of coding required. This means that the transform process has strong energy compaction properties. However, quantization also results in a loss of precision depending on the size of the quantization parameter.

[0499] 64 is a block diagram showing the configuration of a three-dimensional data encoding device according to this embodiment. This three-dimensional data encoding device includes a subtraction unit 7001, a transformation unit 7002, a transformation matrix storage unit 7003, a quantization unit 7004, a quantization control unit 7005, and an entropy encoding unit 7006.

[0500] The subtraction unit 7001 calculates a coefficient value that is the difference between input data and reference data. For example, the input data is attribute information included in the point cloud data, and is a predicted value of the attribute information with the reference data.

[0501] The conversion unit 7002 performs a conversion process on the coefficient values. For example, this conversion process is a process of classifying multiple pieces of attribute information into LoDs. This conversion process may be a Haar transform or the like. The conversion matrix holding unit 7003 holds a conversion matrix used in the conversion process by the conversion unit 7002. For example, this conversion matrix is ​​a Haar transform matrix. Note that, although an example is shown in which the three-dimensional data encoding device has the function of performing both a conversion process using LoD and a conversion process such as a Haar transform, it may have the function of performing only one of them. Furthermore, the three-dimensional data encoding device may selectively use these two types of conversion processes. Furthermore, the three-dimensional data encoding device may switch the conversion process to be used for each predetermined processing unit.

[0502] The quantization unit 7004 generates quantized values ​​by quantizing coefficient values. The quantization control unit 7005 controls the quantization parameters used by the quantization unit 7004 for quantization. For example, the quantization control unit 7005 may switch the quantization parameters (or quantization steps) according to the hierarchical structure of the encoding. This allows the amount of generated code to be controlled for each hierarchical structure by selecting an appropriate quantization parameter for each hierarchical structure. Furthermore, the quantization control unit 7005 may set the quantization parameters for a certain hierarchical level or lower, which includes frequency components that have little effect on subjective image quality, to the maximum value, and set the quantization coefficients for that hierarchical level or lower to zero. This allows the amount of generated code to be reduced while suppressing degradation of subjective image quality. Furthermore, the quantization control unit 7005 can more precisely control the subjective image quality and the amount of generated code. Here, the hierarchical level refers to a level (depth in a tree structure) in LoD or RAHT (Haar Transform).

[0503] The entropy coding unit 7006 generates a bitstream by entropy coding (for example, arithmetic coding) the quantization coefficients. The entropy coding unit 7006 also codes the quantization parameters for each layer set by the quantization control unit 7005.

[0504] 65 is a block diagram showing the configuration of a three-dimensional data decoding device according to this embodiment. This three-dimensional data decoding device includes an entropy decoding unit 7011, an inverse quantization unit 7012, a quantization control unit 7013, an inverse transform unit 7014, a transformation matrix holding unit 7015, and an adder 7016.

[0505] The entropy decoding unit 7011 decodes quantized coefficients and quantization parameters for each layer from the bitstream. The inverse quantization unit 7012 generates coefficient values ​​by inverse quantizing the quantized coefficients. The quantization control unit 7013 controls the quantization parameters used by the inverse quantization unit 7012 for inverse quantization, based on the quantization parameters for the layer codes obtained by the entropy decoding unit 7011.

[0506] The inverse transform unit 7014 performs an inverse transform on the coefficient values. For example, the inverse transform unit 7014 performs an inverse Haar transform on the coefficient values. The transform matrix holding unit 7015 holds a transform matrix used in the inverse transform process by the inverse transform unit 7014. For example, this transform matrix is ​​an inverse Haar transform matrix.

[0507] The adder 7016 generates output data by adding the reference data to the coefficient values. For example, the output data is attribute information included in the point cloud data, and is a predicted value of the attribute information with the reference data.

[0508] Next, an example of setting a quantization parameter for each layer will be described. When encoding attribute information such as Predicting / Lifting, different quantization parameters are applied based on the LoD layer. For example, the quantization parameter for the lower layer is reduced to increase the accuracy of the lower layer. This can improve the prediction accuracy of the higher layer. Furthermore, by increasing the quantization parameter for the higher layer, the amount of data can be reduced. In this way, the quantization tree value (Qt) can be set individually for each LoD according to the user's usage policy. Here, the quantization tree value is, for example, the quantization parameter.

[0509] Fig. 66 is a diagram showing an example of setting LoD. For example, as shown in Fig. 66, Qt0 to Qt2 are set independently for LoD0 to LoD2.

[0510] Furthermore, when encoding attribute information using RAHT, different quantization parameters are applied depending on the depth of the tree structure. For example, the quantization parameters for lower layers are reduced to increase the accuracy of the lower layers. This improves the prediction accuracy of higher layers. In addition, by increasing the quantization parameters for higher layers, the amount of data can be reduced. In this way, the quantization tree value (Qt) can be set individually for each depth of the tree structure according to the user's usage policy.

[0511] Fig. 67 is a diagram showing an example of a hierarchical structure (tree structure) of RAHT. For example, as shown in Fig. 67, independent Qt0 to Qt2 are set for each depth of the tree structure.

[0512] The configuration of a three-dimensional data encoding device according to this embodiment will be described below. Fig. 68 is a block diagram showing the configuration of a three-dimensional data encoding device 7020 according to this embodiment. The three-dimensional data encoding device 7020 generates encoded data (encoded stream) by encoding point group data (point cloud). This three-dimensional data encoding device 7020 includes a dividing unit 7021, a plurality of position information encoding units 7022, a plurality of attribute information encoding units 7023, an additional information encoding unit 7024, and a multiplexing unit 7025.

[0513] The dividing unit 7021 generates a plurality of pieces of divided data by dividing the point cloud data. Specifically, the dividing unit 7021 generates a plurality of pieces of divided data by dividing the space of the point cloud data into a plurality of subspaces. Here, a subspace is one of a tile and a slice, or a combination of a tile and a slice. More specifically, the point cloud data includes position information, attribute information (color, reflectance, etc.), and additional information. The dividing unit 7021 generates a plurality of pieces of divided position information by dividing the position information, and generates a plurality of pieces of divided attribute information by dividing the attribute information. The dividing unit 7021 also generates additional information related to the division.

[0514] For example, the dividing unit 7021 first divides the point cloud into tiles, and then further divides the resulting tiles into slices.

[0515] The position information encoding units 7022 encode the divided position information to generate multiple pieces of encoded position information. For example, the position information encoding unit 7022 encodes the divided position information using an N-ary tree structure such as an octree. Specifically, in an octree, the target space is divided into eight nodes (subspaces), and 8-bit information (occupancy code) indicating whether a point cloud is included in each node is generated. Furthermore, the node including the point cloud is further divided into eight nodes, and 8-bit information indicating whether a point cloud is included in each of the eight nodes is generated. This process is repeated until the number of point clouds included in a predetermined layer or node falls below a threshold. For example, the position information encoding units 7022 process the divided position information in parallel.

[0516] The attribute information encoding unit 7023 generates encoded attribute information, which is encoded data, by encoding the attribute information using the configuration information generated by the position information encoding unit 7022. For example, the attribute information encoding unit 7023 determines a reference point (reference node) to be referenced when encoding a target point (target node) to be processed, based on the octree structure generated by the position information encoding unit 7022. For example, the attribute information encoding unit 7023 references a peripheral node or adjacent node whose parent node in the octree is the same as that of the target node. Note that the method of determining the reference relationship is not limited to this.

[0517] Furthermore, the encoding process of the position information or attribute information may include at least one of a quantization process, a prediction process, and an arithmetic coding process. In this case, the reference means using a reference node to calculate a predicted value of the attribute information, or using the state of the reference node (e.g., occupancy information indicating whether the reference node includes a point group) to determine an encoding parameter. For example, the encoding parameter is a quantization parameter in a quantization process, or a context in an arithmetic coding process.

[0518] The additional information encoding unit 7024 generates encoded additional information by encoding the additional information included in the point cloud data and the additional information related to the data division generated by the division unit 7021 at the time of division.

[0519] The multiplexing unit 7025 multiplexes a plurality of pieces of encoding position information, a plurality of pieces of encoding attribute information, and encoding additional information to generate encoded data (encoded stream), and transmits the generated encoded data. The encoded additional information is also used during decoding.

[0520] 69 is a block diagram of the dividing unit 7021. The dividing unit 7021 includes a tile dividing unit 7031 and a slice dividing unit 7032.

[0521] The tile dividing unit 7031 generates multiple pieces of tile position information by dividing position information (Position (Geometry)) into tiles. The tile dividing unit 7031 also generates multiple pieces of tile attribute information by dividing attribute information (Attribute) into tiles. The tile dividing unit 7031 also outputs tile additional information (Tile MetaData) including information related to the tile division and information generated during the tile division.

[0522] The slice division unit 7032 generates a plurality of pieces of division position information (a plurality of pieces of slice position information) by dividing a plurality of pieces of tile position information into slices. The slice division unit 7032 also generates a plurality of pieces of division attribute information (a plurality of pieces of slice attribute information) by dividing a plurality of pieces of tile attribute information into slices. The slice division unit 7032 also outputs slice additional information (Slice MetaData) including information related to the slice division and information generated in the slice division.

[0523] Furthermore, the tile dividing unit 7031 and the slice dividing unit 7032 determine quantization tree values ​​(quantization parameters) based on the generated additional information.

[0524] 70 is a block diagram of the attribute information encoding unit 7023. The attribute information encoding unit 7023 includes a transform unit 7035, a quantization unit 7036, and an entropy encoding unit 7037.

[0525] The conversion unit 7035 classifies the split attribute information into hierarchical levels such as LoD, and calculates the difference between the split attribute information and the predicted value to generate a coefficient value (difference value). Note that the conversion unit 7035 may generate the coefficient value by performing a Haar transform on the split attribute information.

[0526] The quantization unit 7036 quantizes the coefficient values ​​to generate quantized values. Specifically, the quantization unit 7036 divides the coefficients by a quantization step based on the quantization parameter. The entropy coding unit 7037 entropy codes the quantized values ​​to generate coding attribute information.

[0527] The configuration of a three-dimensional data decoding device according to this embodiment will be described below. Fig. 71 is a block diagram showing the configuration of a three-dimensional data decoding device 7040. The three-dimensional data decoding device 7040 restores point cloud data by decoding coded data (coded stream) generated by encoding point cloud data. This three-dimensional data decoding device 7040 includes a demultiplexing unit 7041, a plurality of position information decoding units 7042, a plurality of attribute information decoding units 7043, an additional information decoding unit 7044, and a combining unit 7045.

[0528] The demultiplexing unit 7041 demultiplexes the coded data (coded stream) to generate a plurality of pieces of coding position information, a plurality of pieces of coding attribute information, and coded additional information.

[0529] The position information decoding units 7042 generate a plurality of pieces of divided position information by decoding the plurality of pieces of encoded position information. For example, the position information decoding units 7042 process the plurality of pieces of encoded position information in parallel.

[0530] The attribute information decoding units 7043 generate a plurality of pieces of divided attribute information by decoding the plurality of pieces of encoded attribute information. For example, the attribute information decoding units 7043 process the plurality of pieces of encoded attribute information in parallel.

[0531] The plurality of additional information decoders 7044 generate additional information by decoding the coded additional information.

[0532] The combining unit 7045 generates position information by combining a plurality of pieces of divided position information using the additional information. The combining unit 7045 generates attribute information by combining a plurality of pieces of divided attribute information using the additional information.

[0533] FIG. 72 is a block diagram of the attribute information decoding unit 7043. The attribute information decoding unit 7043 includes an entropy decoding unit 7051, an inverse quantization unit 7052, and an inverse transform unit 7053. The entropy decoding unit 7051 generates quantized values ​​by entropy decoding the encoded attribute information. The inverse quantization unit 7052 generates coefficient values ​​by inverse quantizing the quantized values. Specifically, the coefficient values ​​are multiplied by a quantization step based on a quantization tree value (quantization parameter) acquired from the bitstream. The inverse transform unit 7053 generates divided attribute information by inversely transforming the coefficient values. Here, the inverse transform is, for example, a process of adding a predicted value to a coefficient value. Alternatively, the inverse transform is an inverse Haar transform.

[0534] An example of a method for determining a quantization parameter will be described below. Figure 73 is a diagram showing an example of setting a quantization parameter in tile and slice division.

[0535] A small quantization parameter value makes it easier to maintain the original information. For example, the default value of the quantization parameter is 1. For example, in an encoding process using tiles of PCC data, the quantization parameter for tiles of main roads is set to a small value to maintain data quality. On the other hand, the quantization parameter for tiles of surrounding areas is set to a large value. This reduces the data quality of the surrounding areas, but improves encoding efficiency.

[0536] Similarly, in the coding process using slices of PCC data, sidewalks, trees, and buildings are important in localization and mapping, so the quantization parameters for the slices of sidewalks, trees, and buildings are set to small values, while the importance of moving objects and other data is low, so the quantization parameters for the slices of moving objects and other data are set to high values.

[0537] Furthermore, when using ΔQP (DeltaQP), which will be described later, the three-dimensional data encoding device may perform encoding so as to reduce the quantization error by setting a negative value for ΔQP to reduce the quantization parameter when encoding three-dimensional points that belong to important areas, such as major roads. This allows the decoded attribute values ​​of three-dimensional points that belong to important areas to approach the values ​​before encoding. Furthermore, when encoding three-dimensional points that belong to unimportant areas, such as peripheral areas, the three-dimensional data encoding device may set a positive value for ΔQP to increase the quantization parameter, thereby reducing the amount of information. This allows the overall amount of code to be reduced while maintaining the amount of information in important areas.

[0538] An example of information indicating quantization parameters for each layer is described below. When quantizing and encoding attribute information of three-dimensional points, in addition to the quantization parameter QPbase for a frame, slice, or tile, a mechanism for controlling the quantization parameter in finer units is introduced. For example, when encoding attribute information using LoD, a three-dimensional data encoding device provides a Delta_Layer for each LoD and performs encoding while changing the value of the quantization parameter by adding the Delta_Layer to the QPbase value for each LoD. The three-dimensional data encoding device also adds the Delta_Layer used for encoding to the header of the bitstream, etc. This allows the three-dimensional data encoding device to encode the attribute information of three-dimensional points while changing the quantization parameter for each LoD according to, for example, the target code amount and the generated code amount, thereby ultimately generating a bitstream with a code amount close to the target code amount. The three-dimensional data decoding device can properly decode the bitstream by decoding the QPbase and Delta_Layer included in the header to generate the quantization parameter used by the three-dimensional data encoding device.

[0539] FIG. 74 is a diagram showing an example of encoding attribute information of all 3D points using a quantization parameter QPbase. FIG. 75 is a diagram showing an example of encoding by switching the quantization parameter for each LoD layer. In the example shown in FIG. 75, the quantization parameter of the first LoD is calculated by adding the Delta_Layer of the first LoD to QPbase. For the second and subsequent LoDs, the quantization parameter of the LoD to be processed is calculated by adding the Delta_Layer of the LoD to the quantization parameter of the immediately preceding LoD. For example, the first quantization parameter QP3 of LoD3 is calculated as QP3=QP2+Delta_Layer[3].

[0540] Note that Delta_Layer[i] of each LoD may indicate a difference value from QPbase. That is, the quantization parameter QPi of the i-th LoD is expressed as QPi=QPbase+Delta_Layer[i]. For example, QP1 is expressed as QPbase+Delta_Layer[1], and QP2 is expressed as QPbase+Delta_Layer[2].

[0541] Fig. 76 is a diagram showing an example of the syntax of an attribute information header. Here, the attribute information header is, for example, a header for each frame, slice, or tile, and is a header of attribute information. As shown in Fig. 76, the attribute information header includes QPbase (base quantization parameter), NumLayer (number of layers), and Delta_Layer[i] (differential quantization parameter).

[0542] QPbase indicates the value of a reference quantization parameter in a frame, slice, tile, etc. NumLayer indicates the number of layers of LoD or RAHT. In other words, NumLayer indicates the number of Delta_Layer[i] included in the attribute information header.

[0543] Delta_Layer[i] indicates the value of ΔQP for layer i. Here, ΔQP is the value obtained by subtracting the quantization parameter for layer i from the quantization parameter for layer i-1. Note that ΔQP may also be the value obtained by subtracting the quantization parameter for layer i from QPbase. ΔQP can also be a positive or negative value. Note that Delta_Layer[0] does not need to be added to the header. In this case, the quantization parameter for layer 0 is equal to QPbase. This reduces the header coding amount.

[0544] Fig. 77 is a diagram showing another example of the syntax of the attribute header information. The attribute header shown in Fig. 77 further includes delta_Layer_present_flag in addition to the attribute header shown in Fig. 76.

[0545] The delta_Layer_present_flag is a flag indicating whether or not the Delta_Layer is included in the bitstream. For example, a value of 1 indicates that the Delta_Layer is included in the bitstream, and a value of 0 indicates that the Delta_Layer is not included in the bitstream. When the delta_Layer_present_flag is 0, the three-dimensional data decoding device performs subsequent decoding processing, for example, with the Delta_Layer set to 0.

[0546] Although an example in which the quantization parameter is represented by QPbase and Delta_Layer has been described above, the quantization step may also be represented by QPbase and Delta_Layer. The quantization step is calculated from the quantization parameter using a predetermined formula, table, or the like. The three-dimensional data encoding device divides coefficient values ​​by the quantization step in the quantization process. The three-dimensional data decoding device restores coefficient values ​​by multiplying the quantized value by the quantization step in the inverse quantization process.

[0547] Next, an example of controlling the quantization parameter in an even finer unit will be described. Fig. 78 is a diagram showing an example of controlling the quantization parameter in an even finer unit than the LoD.

[0548] For example, when a three-dimensional data encoding device encodes attribute information using LoD, in addition to Delta_Layer for each LoD layer, it defines ADelta_QP and NumPointADelta, which represents the position information of the three-dimensional point to which ADelta_QP is added. The three-dimensional data encoding device performs encoding while changing the value of the quantization parameter based on Delta_Layer, ADelta_QP, and NumPointADelta.

[0549] Furthermore, the three-dimensional data encoding device may add the ADelta and NumPointADelta used in encoding to the header of the bitstream, etc. This allows the three-dimensional data encoding device to encode attribute information of multiple three-dimensional points while changing the quantization parameter for each three-dimensional point according to the target code amount and the generated code amount, for example. This allows the three-dimensional data encoding device to ultimately generate a bitstream with a code amount close to the target code amount. Furthermore, the three-dimensional data decoding device can properly decode the bitstream by decoding the QPbase, Delta_Layer, and ADelta included in the header to generate the quantization parameter used by the three-dimensional data encoding device.

[0550] For example, as shown in FIG. 78, the quantized value QP4 of the N0th attribute information is calculated as QP4=QP3+ADelta_QP[0].

[0551] Alternatively, the encoding / decoding order may be reversed from that shown in Fig. 78. For example, encoding / decoding may be performed in the order of LoD3, LoD2, LoD1, and LoD0.

[0552] Fig. 79 is a diagram showing an example of the syntax of the attribute header information when using the example shown in Fig. 78. The attribute information header shown in Fig. 79 further includes NumADelta, NumPointADelta[i], and ADelta_QP[i] in addition to the attribute information header shown in Fig. 76.

[0553] NumADelta indicates the number of ADelta_QPs included in the bitstream. NumPointADelta[i] indicates the identification number of 3D point A to which ADelta_QP[i] is applied. For example, NumPointADelta[i] indicates the number of 3D points from the first 3D point to 3D point A in encoding / decoding order. NumPointADelta[i] may also indicate the number of 3D points from the first 3D point to 3D point A in the LoD to which 3D point A belongs.

[0554] Alternatively, NumPointADelta[i] may indicate the difference between the identification number of the 3D point indicated by NumPointADelta[i-1] and the identification number of 3D point A. This allows the value of NumPointADelta[i] to be small, thereby reducing the amount of coding.

[0555] ADelta_QP[i] indicates the value of ΔQP of the 3D point indicated by NumPointADelta[i]. That is, ADelta_QP[i] indicates the difference between the quantization parameter of the 3D point indicated by NumPointADelta[i] and the quantization parameter of the 3D point immediately preceding the 3D point.

[0556] Fig. 80 is a diagram showing another syntax example of the attribute header information when using the example shown in Fig. 78. The attribute information header shown in Fig. 80 further includes delta_Layer_present_flag and additional_delta_QP_present_flag in addition to the attribute information header shown in Fig. 79, and includes NumADelta_minus1 instead of NumADelta.

[0557] The delta_Layer_present_flag is the same as that already explained using FIG.

[0558] "additional_delta_QP_present_flag" is a flag indicating whether ADelta_QP is included in the bitstream. For example, a value of 1 indicates that ADelta_QP is included in the bitstream, and a value of 0 indicates that ADelta_QP is not included in the bitstream. When "additional_delta_QP_present_flag" is 0, the three-dimensional data decoding device performs subsequent decoding processing, for example, with ADelta_QP set to 0.

[0559] NumADelta_minus1 indicates the number of ADelta_QP included in the bitstream minus 1. In this way, by adding a value obtained by subtracting 1 from the number of ADelta_QP to the header, the amount of header coding can be reduced. For example, the three-dimensional data decoding device calculates NumADelta=NumADelta_minus1+1. ADelta_QP[i] indicates the value of the i-th ADelta_QP. Note that ADelta_QP[i] may be set to not only positive values ​​but also negative values.

[0560] 81 is a flowchart of three-dimensional data encoding processing according to this embodiment. First, the three-dimensional data encoding device encodes position information (geometry) (S7001). For example, the three-dimensional data encoding device performs encoding using an octree representation.

[0561] Next, the three-dimensional data encoding device converts the attribute information (S7002). For example, if the position of a three-dimensional point changes due to quantization or the like after encoding the position information, the three-dimensional data encoding device reassigns the attribute information of the original three-dimensional point to the changed three-dimensional point. The three-dimensional data encoding device may perform the reassignment by interpolating the value of the attribute information according to the amount of change in position. For example, the three-dimensional data encoding device detects N pre-change three-dimensional points that are close to the changed three-dimensional position, weights the values ​​of the attribute information of the N three-dimensional points based on the distance from the changed three-dimensional position to each of the N three-dimensional points, and determines the obtained value as the value of the attribute information of the changed three-dimensional point. Furthermore, if two or more three-dimensional points change to the same three-dimensional position due to quantization or the like, the three-dimensional data encoding device may assign the average value of the attribute information of the two or more pre-change three-dimensional points as the value of the attribute information of the changed three-dimensional point.

[0562] Next, the three-dimensional data encoding device encodes the attribute information (S7003). For example, when encoding multiple pieces of attribute information, the three-dimensional data encoding device may encode the multiple pieces of attribute information in order. For example, when encoding color and reflectance as attribute information, the three-dimensional data encoding device generates a bitstream in which the encoding result of reflectance is added after the encoding result of color. Note that the encoding results of the multiple pieces of attribute information added to the bitstream may be in any order.

[0563] The three-dimensional data encoding device may also add information indicating the start location of the encoded data for each piece of attribute information in the bitstream to a header or the like. This allows the three-dimensional data decoding device to selectively decode attribute information that needs to be decoded, thereby omitting the decoding process for attribute information that does not need to be decoded. This reduces the amount of processing required by the three-dimensional data decoding device. The three-dimensional data encoding device may also encode multiple pieces of attribute information in parallel and combine the encoding results into a single bitstream. This allows the three-dimensional data encoding device to encode multiple pieces of attribute information at high speed.

[0564] 82 is a flowchart of the attribute information encoding process (S7003). First, the three-dimensional data encoding device sets the LoD (S7011). That is, the three-dimensional data encoding device assigns each three-dimensional point to one of a plurality of LoDs.

[0565] Next, the three-dimensional data encoding device starts a loop for each LoD (S7012). That is, the three-dimensional data encoding device repeatedly performs the processes of steps S7013 to S7021 for each LoD.

[0566] Next, the three-dimensional data encoding device starts a loop for each three-dimensional point (S7013). That is, the three-dimensional data encoding device repeatedly performs the processes of steps S7014 to S7020 for each three-dimensional point.

[0567] First, the three-dimensional data encoding device searches for multiple surrounding points, which are three-dimensional points that exist around the target three-dimensional point to be processed, and is used to calculate a predicted value of the target three-dimensional point (S7014). Next, the three-dimensional data encoding device calculates a weighted average of the attribute information values ​​of the multiple surrounding points, and sets the obtained value as the predicted value P (S7015). Next, the three-dimensional data encoding device calculates a prediction residual, which is the difference between the attribute information of the target three-dimensional point and the predicted value (S7016). Next, the three-dimensional data encoding device calculates a quantized value by quantizing the prediction residual (S7017). Next, the three-dimensional data encoding device arithmetically encodes the quantized value (S7018). Next, the three-dimensional data encoding device determines a ΔQP (S7019). The determined ΔQP is used to determine a quantization parameter to be used in quantizing the subsequent prediction residual.

[0568] The three-dimensional data encoding device also calculates inverse quantized values ​​by inverse quantizing the quantized values ​​(S7020). Next, the three-dimensional data encoding device generates decoded values ​​by adding predicted values ​​to the inverse quantized values ​​(S7021). Next, the three-dimensional data encoding device ends the loop in units of three-dimensional points (S7022). Also, the three-dimensional data encoding device ends the loop in units of LoDs (S7023).

[0569] 83 is a flowchart of the ΔQP determination process (S7019). First, the 3D data encoding device calculates the layer i to which the target 3D point A to be next encoded belongs and the encoding order N (S7031). Layer i represents, for example, an LoD layer or a RAHT layer.

[0570] Next, the three-dimensional data encoding device adds the generated code amount to the cumulative code amount (S7032). Here, the cumulative code amount is the cumulative code amount for one frame, one slice, or one tile of the target three-dimensional point. Note that the cumulative code amount may be the cumulative code amount obtained by adding up the code amounts for multiple frames, multiple slices, or multiple tiles. Furthermore, the cumulative code amount for attribute information may be used, or the cumulative code amount obtained by adding up both position information and attribute information may be used.

[0571] Next, the three-dimensional data encoding device determines whether the cumulative code amount is greater than the target code amount x TH1 (S7033). Here, the target code amount is the target code amount for one frame, one slice, or one tile of the target three-dimensional point. Note that the target code amount may be the target code amount obtained by adding together multiple frames, multiple slices, or multiple tiles. Furthermore, the target code amount for attribute information may be used, or the target code amount obtained by adding together both position information and attribute information may be used.

[0572] If the cumulative code amount is equal to or less than the target code amount×TH1 (No in S7033), the three-dimensional data encoding device determines whether or not the cumulative code amount is greater than the target code amount×TH2 (S7036).

[0573] Here, the thresholds TH1 and TH2 are set to values ​​between 0.0 and 1.0, for example. TH1>TH2. For example, if the cumulative code amount exceeds the value of target code amount×TH1 (Yes in S7033), the three-dimensional data encoding device determines that it is necessary to immediately reduce the code amount, and sets ADelta_QP to the value α to increase the quantization parameter for the next three-dimensional point N. The three-dimensional data encoding device also sets NumPointADelta to the value N and increments j by 1 (S7034). Next, the three-dimensional data encoding device adds ADelta_QP=α and NumPointADelta=N to the header (S7035). The value α may be a fixed value or a variable value. For example, the three-dimensional data encoding device may determine the value of α based on the magnitude of the difference between the cumulative code amount and target code amount×TH1. For example, the three-dimensional data encoding device sets the value of α to a larger value as the difference between the cumulative code amount and target code amount×TH1 increases. This allows the three-dimensional data encoding device to control the quantization parameter so that the cumulative code amount does not exceed the target code amount.

[0574] Furthermore, if the cumulative code amount exceeds the value of target code amount × TH2 (Yes in S7036), the three-dimensional data encoding device sets Delta_Layer to the value β in order to increase the quantization parameter of layer i to which the target three-dimensional point A belongs or the next layer i+1 (S7037). For example, if the target three-dimensional point A is at the top of layer i, the three-dimensional data encoding device sets Delta_Layer[i] of layer i to the value β, and if the target three-dimensional point A is not at the top of layer i, it sets Delta_Layer[i+1] of layer i+1 to the value β.

[0575] The three-dimensional data encoding device also adds Delta_Layer=β for layer i or layer i+1 to the header (S7038). The value β may be a fixed value or a variable value. For example, the three-dimensional data encoding device may determine the value of β based on the magnitude of the difference between the accumulated code amount and the target code amount × TH2. For example, the three-dimensional data encoding device sets the value of β to a larger value the greater the difference between the accumulated code amount and the target code amount × TH2. This allows the three-dimensional data encoding device to control the quantization parameter so that the accumulated code amount does not exceed the target code amount.

[0576] Furthermore, when the cumulative code amount exceeds or is likely to exceed the target code amount, the three-dimensional data encoding device may set the value of ADelta_QP or Delta_Layer so that the quantization parameter becomes the maximum value supported by the standard, etc. In this way, the three-dimensional data encoding device can suppress an increase in the generated code amount by setting the quantization coefficients from three-dimensional point A onwards or from layer i onwards to 0, thereby controlling the cumulative code amount so that it does not exceed the target code amount.

[0577] Furthermore, if the cumulative code amount is smaller than the target code amount × TH3, the three-dimensional data encoding device may lower the quantization parameter so that the generated code amount increases. For example, the three-dimensional data encoding device may lower the quantization parameter by setting a negative value for Delta_Layer or Adelta_QP according to the difference between the cumulative code amount and the target code amount. This allows the three-dimensional data encoding device to generate a bitstream close to the target code amount.

[0578] 84 is a flowchart of three-dimensional data decoding processing according to this embodiment. First, the three-dimensional data decoding device decodes position information (geometry) from the bit stream (S7005). For example, the three-dimensional data decoding device performs decoding using an octree representation.

[0579] Next, the three-dimensional data decoding device decodes the attribute information from the bitstream (S7006). For example, when decoding multiple pieces of attribute information, the three-dimensional data decoding device may decode the multiple pieces of attribute information in order. For example, when decoding color and reflectance as attribute information, the three-dimensional data decoding device decodes the encoding result of color and the encoding result of reflectance in the order in which they are added to the bitstream. For example, when the encoding result of reflectance is added after the encoding result of color in the bitstream, the three-dimensional data decoding device decodes the encoding result of color, and then decodes the encoding result of reflectance. Note that the three-dimensional data decoding device may decode the encoding results of the attribute information added to the bitstream in any order.

[0580] Furthermore, the three-dimensional data decoding device may obtain information ...

Claims

1. An encoding method implemented by an encoding device having a processor and a memory, comprising: encoding a first coefficient value of the plurality of coefficient values ​​using the first parameter and the second parameter to generate encoded data; generating a bitstream including the encoded data, the first parameter, and the second parameter; the first coefficient value belongs to a first layer of one or more layers, and the first parameter is used to encode the coefficient value belonging to the first layer; the first coefficient values ​​belong to a first region in space, and the second parameters are used to encode the coefficient values ​​belonging to the first region. Encoding method.

2. the generation of the encoded data is performed using a third parameter in addition to the first parameter and the second parameter; the first coefficient value belongs to a first data unit of one or more data units, and the third parameter is used to encode the coefficient value belonging to the first data unit. The encoding method according to claim 1 .

3. the generation of the encoded data is performed using a fourth parameter in addition to the first parameter and the second parameter; the first coefficient value belongs to a first frame of one or more frames, and the fourth parameter is used to encode the coefficient value belonging to the first frame. The encoding method according to claim 1 .

4. the first region belongs to a first data unit; The position of the first area is indicated by coordinate values ​​based on the origin coordinates of the first data unit. The encoding method according to claim 2.

5. A decoding method implemented by a decoding device having a processor and a memory, comprising: Obtaining encoded data, a first parameter, and a second parameter from the bitstream; Decoding the plurality of encoded data using the first and second parameters to generate first coefficient values; the first coefficient value belongs to a first layer of one or more layers, and the first parameter is used in a decoding process for generating coefficient values ​​belonging to the first layer; the first coefficient values ​​belong to a first region in space, and the second parameters are used in a decoding process to generate coefficient values ​​belonging to the first region. Decryption method.

6. the generation of the first coefficient value is performed using a third parameter in addition to the first parameter and the second parameter; the first coefficient value belongs to a first data unit of one or more data units, and the third parameter is used in a decoding process to generate the coefficient value belonging to the first data unit. The decoding method according to claim 5.

7. the generation of the first coefficient value is performed using a fourth parameter in addition to the first and second parameters; the first coefficient value belongs to a first frame of one or more frames, and the fourth parameter is used in a decoding process to generate the coefficient value belonging to the first frame. The decoding method according to claim 5.

8. the first region belongs to a first data unit; The position of the first area is indicated by coordinate values ​​based on the origin coordinates of the first data unit. The decoding method according to claim 6.

9. A processor; A memory, The processor uses the memory to: encoding a first coefficient value of the plurality of coefficient values ​​using the first parameter and the second parameter to generate encoded data; generating a bitstream including the encoded data, the first parameter, and the second parameter; the first coefficient value belongs to a first layer of one or more layers, and the first parameter is used to encode the coefficient value belonging to the first layer; the first coefficient values ​​belong to a first region in space, and the second parameters are used to encode the coefficient values ​​belonging to the first region. Encoding device.

10. A processor; A memory, The processor uses the memory to: Obtaining encoded data, a first parameter, and a second parameter from the bitstream; Decoding the plurality of encoded data using the first and second parameters to generate first coefficient values; the first coefficient value belongs to a first layer of one or more layers, and the first parameter is used in a decoding process for generating coefficient values ​​belonging to the first layer; the first coefficient values ​​belong to a first region in space, and the second parameters are used in a decoding process to generate coefficient values ​​belonging to the first region. Decryption device.

Images