Three-dimensional data encoding method, three-dimensional data decoding method, three-dimensional data encoding device, and three-dimensional data decoding device

Abstract

The invention relates to a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device and a three-dimensional data decoding device. This three-dimensional data encoding method encodes position information of first three-dimensional data and position information of second three-dimensional data by adding first information indicating a space of the first three-dimensional data to a header of the first three-dimensional data and adding second information indicating a space of the second three-dimensional data to a header of the second three-dimensional data.

Description

[0001] This application is a divisional application of the invention patent application with the application date of April 18, 2019, application number 201980026225.5, and invention name “Three-dimensional data encoding method, three-dimensional data decoding method, three-dimensional data encoding device, and three-dimensional data decoding device”. Technical Field

[0002] The present disclosure relates to a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, and a three-dimensional data decoding device. Background Art

[0003] Devices and services that utilize three-dimensional data will become increasingly common in a wide range of fields, including computer vision for autonomous vehicles and robots, mapping, surveillance, infrastructure inspection, and image distribution. Three-dimensional data is acquired using various methods, including distance sensors such as rangefinders, stereo cameras, and combinations of multiple single-lens reflex cameras.

[0004] One method of representing three-dimensional data is a point cloud, which represents the shape of a three-dimensional structure using a group of points in a three-dimensional space (for example, see Non-Patent Document 1). The point cloud stores the position and color of the point group. Although point clouds are expected to become the mainstream method of representing three-dimensional data, the amount of point group data is very large. Therefore, in the storage or transmission of three-dimensional data, as with two-dimensional dynamic images (for example, MPEG-4AVC or HEVC standardized by MPEG), data compression through encoding is required.

[0005] Furthermore, compression of point clouds is partially supported by a public library (PointCloud Library) that performs point cloud association processing.

[0006] Furthermore, there is a known technique for searching for facilities around a vehicle using three-dimensional map data and displaying the facilities (for example, see Patent Document 1).

[0007] Prior art literature

[0008] Patent Literature

[0009] Patent Document 1 International Publication No. 2014 / 020663 Summary of the Invention

[0010] Problems to be solved by the invention

[0011] It is desirable to reduce processing time in encoding or decoding three-dimensional data.

[0012] An object of the present disclosure is to provide a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, or a three-dimensional data decoding device that can shorten processing time.

[0013] Means used to solve problems

[0014] A three-dimensional data encoding method according to one embodiment of the present invention attaches first information representing the space of first three-dimensional data to the header of the first three-dimensional data, attaches second information representing the space of second three-dimensional data to the header of the second three-dimensional data, and encodes the position information of the first three-dimensional data and the position information of the second three-dimensional data.

[0015] A three-dimensional data decoding method according to one embodiment of the present invention obtains first information representing the space of the first three-dimensional data from a header of the first three-dimensional data, obtains second information representing the space of the second three-dimensional data from a header of the second three-dimensional data, and decodes first encoded data generated by encoding the position information of the first three-dimensional data and second encoded data generated by encoding the position information of the second three-dimensional data.

[0016] Effects of the Invention

[0017] The present disclosure can provide a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, or a three-dimensional data decoding device that can shorten processing time. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 The structure of the encoded three-dimensional data in the first embodiment is shown.

[0019] Figure 2 An example of the prediction structure between SPCs belonging to the lowest layer of the GOS in the first embodiment is shown.

[0020] Figure 3 An example of the inter-layer prediction structure in the first embodiment is shown.

[0021] Figure 4 An example of the encoding order of the GOS in the first embodiment is shown.

[0022] Figure 5 An example of the encoding order of the GOS in the first embodiment is shown.

[0023] Figure 6 This is a block diagram of the three-dimensional data encoding device according to the first embodiment.

[0024] Figure 7 This is a flowchart of the encoding process in implementation mode 1.

[0025] Figure 8This is a block diagram of the three-dimensional data decoding device according to the first embodiment.

[0026] Figure 9 This is a flowchart of the decoding process in the first embodiment.

[0027] Figure 10 An example of meta-information in the first embodiment is shown.

[0028] Figure 11 A configuration example of a SWLD according to the second embodiment is shown.

[0029] Figure 12 An operation example of the server and client according to the second embodiment is shown.

[0030] Figure 13 An operation example of the server and client according to the second embodiment is shown.

[0031] Figure 14 An operation example of the server and client according to the second embodiment is shown.

[0032] Figure 15 An operation example of the server and client according to the second embodiment is shown.

[0033] Figure 16 This is a block diagram of a three-dimensional data encoding device according to Embodiment 2.

[0034] Figure 17 This is a flowchart of the encoding process in implementation mode 2.

[0035] Figure 18 This is a block diagram of a three-dimensional data decoding device according to Embodiment 2.

[0036] Figure 19 This is a flowchart of the decoding process of implementation mode 2.

[0037] Figure 20 A configuration example of a WLD according to the second embodiment is shown.

[0038] Figure 21 An example of the octree structure of the WLD according to the second embodiment is shown.

[0039] Figure 22 A configuration example of a SWLD according to the second embodiment is shown.

[0040] Figure 23 An example of the octree structure of the SWLD according to the second embodiment is shown.

[0041] Figure 24 This is a block diagram of a three-dimensional data creation device according to a third embodiment.

[0042] Figure 25This is a block diagram of a three-dimensional data transmitting device according to a third embodiment.

[0043] Figure 26 This is a block diagram of a three-dimensional information processing device according to a fourth embodiment.

[0044] Figure 27 This is a block diagram of a three-dimensional data creation device according to a fifth embodiment.

[0045] Figure 28 The configuration of the system according to the sixth embodiment is shown.

[0046] Figure 29 This is a block diagram of a client device according to a sixth embodiment.

[0047] Figure 30 This is a block diagram of a server in implementation mode 6.

[0048] Figure 31 This is a flowchart of the three-dimensional data creation process performed by the client device in the sixth embodiment. Figure 32 This is a flowchart of the sensor information transmission process performed by the client device in the sixth embodiment. Figure 33 This is a flowchart of the three-dimensional data creation process performed by the server in the sixth embodiment.

[0049] Figure 34 This is a flowchart of the three-dimensional map transmission processing performed by the server in the sixth embodiment.

[0050] Figure 35 The configuration of a modified example of the system of the sixth embodiment is shown.

[0051] Figure 36 The configuration of the server and client device according to the sixth embodiment is shown.

[0052] Figure 37 This is a block diagram of a three-dimensional data encoding device according to embodiment 7.

[0053] Figure 38 An example of the prediction residual in the seventh embodiment is shown.

[0054] Figure 39 An example of volume in embodiment 7 is shown.

[0055] Figure 40 An example of octree representation of volume in embodiment 7 is shown.

[0056] Figure 41 An example of a bit string of volume in embodiment 7 is shown.

[0057] Figure 42 An example of octree representation of volume in embodiment 7 is shown.

[0058] Figure 43 An example of volume in embodiment 7 is shown.

[0059] Figure 44 This is a diagram for explaining the intra-frame prediction processing of embodiment 7.

[0060] Figure 45 This is a diagram used to illustrate the rotation and translation processing of embodiment 7.

[0061] Figure 46 An example of the syntax of the RT application flag and RT information in the seventh embodiment is shown.

[0062] Figure 47 This is a diagram used to illustrate the inter-frame prediction processing of embodiment 7.

[0063] Figure 48 This is a block diagram of a three-dimensional data decoding device according to a seventh embodiment.

[0064] Figure 49 This is a flowchart of a three-dimensional data encoding process performed by the three-dimensional data encoding device according to the seventh embodiment. Figure 50 This is a flowchart of a three-dimensional data decoding process performed by the three-dimensional data decoding device according to the seventh embodiment. Figure 51 The configuration of the distribution system according to the eighth embodiment is shown.

[0065] Figure 52 An example of the structure of a bit stream encoding a three-dimensional map according to the eighth embodiment is shown.

[0066] Figure 53 This is a diagram used to illustrate the improvement effect of coding efficiency in embodiment 8.

[0067] Figure 54 This is a flowchart of the processing performed by the server in implementation mode 8.

[0068] Figure 55 This is a flowchart of the processing performed by the client in implementation mode 8.

[0069] Figure 56 A syntax example of a submap in the eighth embodiment is shown.

[0070] Figure 57 The coding type switching process of the eighth embodiment is schematically shown.

[0071] Figure 58 A syntax example of a submap in the eighth embodiment is shown.

[0072] Figure 59 This is a flowchart of the three-dimensional data encoding process in the eighth embodiment.

[0073] Figure 60 This is a flowchart of three-dimensional data decoding processing in the eighth embodiment.

[0074] Figure 61 The operation of a modified example of the coding type switching process of the eighth embodiment is schematically shown.

[0075] Figure 62 The operation of a modified example of the coding type switching process of the eighth embodiment is schematically shown.

[0076] Figure 63 The operation of a modified example of the coding type switching process of the eighth embodiment is schematically shown.

[0077] Figure 64 The operation of a modified example of the difference value calculation process of the eighth embodiment is schematically shown.

[0078] Figure 65 The operation of a modified example of the difference value calculation process of the eighth embodiment is schematically shown.

[0079] Figure 66 The operation of a modified example of the difference value calculation process of the eighth embodiment is schematically shown.

[0080] Figure 67 The operation of a modified example of the difference value calculation process of the eighth embodiment is schematically shown.

[0081] Figure 68 A syntax example of volume in embodiment 8 is shown.

[0082] Figure 69 This is a diagram showing an example of an important area in Implementation Example 9.

[0083] Figure 70 This is a diagram showing an example of an occupancy code according to the ninth embodiment.

[0084] Figure 71 This is a diagram showing an example of a quadtree structure according to the ninth embodiment.

[0085] Figure 72 This is a diagram showing an example of an occupancy code and a position code according to the ninth embodiment.

[0086] Figure 73 This is a diagram showing an example of three-dimensional points obtained by LiDAR in the ninth embodiment.

[0087] Figure 74 This is a diagram showing an example of an octree structure according to the ninth embodiment.

[0088] Figure 75 This is a diagram showing an example of mixed coding according to the ninth embodiment.

[0089] Figure 76 This is a diagram used to illustrate the switching method between position coding and occupancy coding in embodiment 9.

[0090] Figure 77 This is a diagram showing an example of a position-coded bit stream according to the ninth embodiment.

[0091] Figure 78 This is a diagram showing an example of a mixed coded bit stream according to the ninth embodiment.

[0092] Figure 79 This is a diagram showing the tree structure of occupancy codes of important three-dimensional points in implementation mode 9.

[0093] Figure 80 This is a diagram showing the tree structure of occupancy codes of non-important three-dimensional points in implementation mode 9.

[0094] Figure 81 This is a diagram showing an example of a mixed coded bit stream according to the ninth embodiment.

[0095] Figure 82 This is a diagram showing an example of a bit stream including coding mode information according to the ninth embodiment.

[0096] Figure 83 This is a diagram showing a syntax example of implementation mode 9.

[0097] Figure 84 This is a flowchart of the encoding process of implementation mode 9.

[0098] Figure 85 This is a flowchart of the node encoding process of implementation mode 9.

[0099] Figure 86 This is a flowchart of the decoding process of implementation mode 9.

[0100] Figure 87 This is a flowchart of the node decoding process of implementation mode 9.

[0101] Figure 88 This is a diagram showing an example of a tree structure related to embodiment 10.

[0102] Figure 89 This is a diagram showing an example of the number of valid leaf nodes that each branch has according to the tenth embodiment. Figure 90 This is a diagram showing an application example of the encoding method related to embodiment 10.

[0103] Figure 91 This is a diagram showing an example of a dense branch area according to the tenth embodiment.

[0104] Figure 92 This is a diagram showing an example of a dense three-dimensional point group related to embodiment 10.

[0105] Figure 93 This is a diagram showing an example of a sparse three-dimensional point group related to embodiment 10.

[0106] Figure 94 This is a flowchart of the encoding process related to implementation mode 10.

[0107] Figure 95 This is a flowchart of the decoding process related to implementation mode 10.

[0108] Figure 96 This is a flowchart of the encoding process related to implementation mode 10.

[0109] Figure 97 This is a flowchart of the decoding process related to implementation mode 10.

[0110] Figure 98 This is a flowchart of the encoding process related to implementation mode 10.

[0111] Figure 99 This is a flowchart of the decoding process related to implementation mode 10.

[0112] Figure 100 This is a flowchart showing the three-dimensional point separation process according to the tenth embodiment.

[0113] Figure 101 This is a diagram showing a syntax example related to implementation mode 10.

[0114] Figure 102 This is a diagram showing an example of dense branching related to embodiment 10.

[0115] Figure 103 This is a diagram showing an example of sparse branching according to the tenth embodiment.

[0116] Figure 104 This is a flowchart of the encoding process of a variant example of implementation mode 10.

[0117] Figure 105 This is a flowchart of the decoding process of a variant example of implementation mode 10.

[0118] Figure 106 This is a flowchart of the three-dimensional point separation process according to a variation of the tenth embodiment.

[0119] Figure 107 This is a diagram showing a syntax example of a variation of the tenth embodiment.

[0120] Figure 108This is a flowchart of the encoding process related to implementation mode 10.

[0121] Figure 109 This is a flowchart of the decoding process related to implementation mode 10.

[0122] Figure 110 This is a diagram showing an example of a tree structure related to implementation example 11.

[0123] Figure 111 This is a diagram showing an example of an occupancy code related to implementation example 11.

[0124] Figure 112 This is a diagram schematically showing the operation of the three-dimensional data encoding device related to embodiment 11. Figure 113 This is a diagram showing an example of geometric information related to implementation example 11.

[0125] Figure 114 This is a diagram showing an example of selecting a coding table using geometric information related to embodiment 11. Figure 115 This is a diagram showing an example of selecting a coding table using structural information related to embodiment 11. Figure 116 This is a diagram showing an example of selecting a coding table for usage attribute information related to Implementation 11. Figure 117 This is a diagram showing an example of selecting a coding table for usage attribute information related to Implementation 11. Figure 118 This is a diagram showing a structural example of a bit stream related to embodiment 11.

[0126] Figure 119 This is a diagram showing an example of a coding table related to embodiment 11.

[0127] Figure 120 This is a diagram showing an example of a coding table related to embodiment 11.

[0128] Figure 121 This is a diagram showing a structural example of a bit stream related to embodiment 11.

[0129] Figure 122 This is a diagram showing an example of a coding table related to embodiment 11.

[0130] Figure 123 This is a diagram showing an example of a coding table related to embodiment 11.

[0131] Figure 124 This is a diagram showing an example of the bit number of the occupancy code related to embodiment 11.

[0132] Figure 125 This is a flowchart of the encoding process using geometric information related to implementation mode 11.

[0133] Figure 126This is a flowchart of the decoding process using geometric information related to implementation mode 11.

[0134] Figure 127 This is a flowchart of the encoding process using structural information related to implementation mode 11.

[0135] Figure 128 This is a flowchart of the decoding process of the usage structure information related to implementation mode 11.

[0136] Figure 129 This is a flowchart of the encoding process of usage attribute information related to implementation mode 11.

[0137] Figure 130 This is a flowchart of the decoding process of usage attribute information related to implementation mode 11.

[0138] Figure 131 This is a flowchart of the coding table selection process using geometric information related to implementation mode 11.

[0139] Figure 132 This is a flowchart of the coding table selection process using structural information related to implementation mode 11.

[0140] Figure 133 This is a flowchart of the coding table selection process for using attribute information related to implementation mode 11.

[0141] Figure 134 This is a block diagram of a three-dimensional data encoding device related to embodiment 11.

[0142] Figure 135 This is a block diagram of a three-dimensional data decoding device according to embodiment 11.

[0143] Figure 136 This is a diagram showing the reference relationship in the octree structure of embodiment 12.

[0144] Figure 137 This is a diagram showing the reference relationship in the spatial area of ​​Implementation Example 12.

[0145] Figure 138 This is a diagram showing an example of adjacent reference nodes in implementation mode 12.

[0146] Figure 139 This is a diagram showing the relationship between the parent node and the node in implementation mode 12.

[0147] Figure 140 This is a diagram showing an example of an occupancy code of a parent node in implementation mode 12.

[0148] Figure 141 This is a block diagram showing a three-dimensional data encoding device according to embodiment 12.

[0149] Figure 142 This is a block diagram showing a three-dimensional data decoding device according to a twelfth embodiment.

[0150] Figure 143 This is a flowchart showing the three-dimensional data encoding process of embodiment 12.

[0151] Figure 144 This is a flowchart showing the three-dimensional data encoding process of embodiment 12.

[0152] Figure 145 This is a diagram showing an example of switching the coding table in embodiment 12.

[0153] Figure 146 This is a diagram showing the reference relationship in the spatial region of Modification 1 of Implementation 12.

[0154] Figure 147 This is a diagram showing a syntax example of header information according to variant example 1 of implementation example 12.

[0155] Figure 148 This is a diagram showing a syntax example of header information according to variant example 1 of implementation example 12.

[0156] Figure 149 This is a diagram showing an example of adjacent reference nodes in variant example 2 of implementation example 12.

[0157] Figure 150 This is a diagram showing an example of a target node and adjacent nodes in Variation 2 of Implementation Example 12. Figure 151 This is a diagram of the reference relationship in the octree structure of variant example 3 of implementation example 12.

[0158] Figure 152 This is a diagram showing the reference relationship in the spatial area of ​​variant example 3 of implementation example 12.

[0159] Figure 153 This is a diagram showing a syntax example of header information in implementation mode 13.

[0160] Figure 154 This is a diagram showing an example of the structure of an octree when the mode information is 1 in implementation mode 13. Figure 155 This is a diagram showing an example of the structure of an octree when the mode information of Implementation 13 is 0. Figure 156 This is a diagram showing a syntax example of node information according to the thirteenth embodiment.

[0161] Figure 157 This is a flowchart of the three-dimensional data encoding process of embodiment 13.

[0162] Figure 158 This is a flowchart of three-dimensional data decoding processing in embodiment 13.

[0163] Figure 159 This is a block diagram of a three-dimensional data encoding device according to embodiment 13.

[0164] Figure 160 This is a block diagram of a three-dimensional data decoding device according to embodiment 13.

[0165] Figure 161 This is a flowchart of the three-dimensional data encoding process of embodiment 13.

[0166] Figure 162 This is a flowchart of three-dimensional data decoding processing in embodiment 13.

[0167] Figure 163 This is a diagram showing an example of the 1-bit occupied position and the remaining bits in Implementation 14. Figure 164 This is a diagram for explaining the determination process of whether occupied position coding is applied in the fourteenth embodiment. Figure 165 This is a diagram showing a syntax example of node information according to the fourteenth embodiment.

[0168] Figure 166 This is a flowchart of the three-dimensional data encoding process in the fourteenth embodiment.

[0169] Figure 167 This is a flowchart of the occupied position coding process in the fourteenth embodiment.

[0170] Figure 168 This is a flowchart of three-dimensional data decoding processing in the fourteenth embodiment.

[0171] Figure 169 This is a flowchart of the occupied position decoding process in the fourteenth embodiment.

[0172] Figure 170 This is a block diagram of a three-dimensional data encoding device according to a fourteenth embodiment.

[0173] Figure 171 This is a block diagram of a three-dimensional data decoding device according to a fourteenth embodiment.

[0174] Figure 172 This is a flowchart of the three-dimensional data encoding process in the fourteenth embodiment.

[0175] Figure 173 This is a flowchart of three-dimensional data decoding processing in the fourteenth embodiment.

[0176] Figure 174 This is a diagram used to illustrate the repetitive points of embodiment 15.

[0177] Figure 175 This is a diagram for explaining the processing of duplicate points in embodiment 15.

[0178] Figure 176 This is a diagram showing a syntax example of header information in implementation mode 15.

[0179] Figure 177 This is a diagram showing a syntax example of node information according to the fifteenth embodiment.

[0180] Figure 178 This is a flowchart of the three-dimensional data encoding process of embodiment 15.

[0181] Figure 179 This is a flowchart of the three-dimensional data encoding process of embodiment 15.

[0182] Figure 180 This is a flowchart of three-dimensional data decoding processing in embodiment 15.

[0183] Figure 181 This is a block diagram of a three-dimensional data encoding device according to embodiment 15.

[0184] Figure 182 This is a block diagram of a three-dimensional data decoding device according to a fifteenth embodiment.

[0185] Figure 183 This is a flowchart of a modified example of the three-dimensional data encoding process of the fifteenth embodiment.

[0186] Figure 184 This is a diagram for explaining the processing of duplicate points in embodiment 15.

[0187] Figure 185 This is a diagram showing an example of adjacent nodes in implementation mode 16.

[0188] Figure 186 This is a diagram showing an example of a node that becomes a search target in the sixteenth embodiment.

[0189] Figure 187 This is a diagram used to illustrate the adjacent node search process of implementation mode 16.

[0190] Figure 188 This is a diagram used to illustrate the update process of adjacent information in implementation mode 16.

[0191] Figure 189 This is a diagram used to illustrate the update process of adjacent information in implementation mode 16.

[0192] Figure 190 This is a diagram for explaining the search process in which the search threshold value of the sixteenth embodiment is set.

[0193] Figure 191 This is a diagram showing an example of an index using a Morton code according to the sixteenth embodiment.

[0194] Figure 192 This is a diagram showing an example of a queue using Morton code according to the sixteenth embodiment.

[0195] Figure 193 This is a block diagram of a three-dimensional data encoding device according to embodiment 16.

[0196] Figure 194 This is a block diagram of a three-dimensional data decoding device according to a sixteenth embodiment.

[0197] Figure 195 This is a flowchart of the three-dimensional data encoding process of the sixteenth embodiment.

[0198] Figure 196 This is a flowchart of three-dimensional data decoding processing in the sixteenth embodiment.

[0199] Figure 197 This is a diagram showing a syntax example of header information in implementation mode 16.

[0200] Figure 198 This is a diagram showing a syntax example of node information in the sixteenth embodiment.

[0201] Figure 199 This is a flowchart of the three-dimensional data encoding process of the sixteenth embodiment.

[0202] Figure 200 This is a flowchart of three-dimensional data decoding processing in the sixteenth embodiment.

[0203] Figure 201 This is a flowchart of the three-dimensional data encoding process of the sixteenth embodiment.

[0204] Figure 202 This is a flowchart of three-dimensional data decoding processing in the sixteenth embodiment.

[0205] Figure 203 This is a diagram showing an example of a tree structure according to the seventeenth embodiment.

[0206] Figure 204 This is a diagram showing an example of a sub-three-dimensional point group according to embodiment 17.

[0207] Figure 205 This is a diagram showing an example of a sub-three-dimensional point group according to embodiment 17.

[0208] Figure 206 This is a diagram showing an example of the structure of a bit stream according to embodiment 17.

[0209] Figure 207 This is a diagram showing an example of a tree structure according to the seventeenth embodiment.

[0210] Figure 208 This is a diagram used to illustrate the fully parallel coding of embodiment 17, and is a diagram showing an example of a tree structure.

[0211] Figure 209 This is a diagram spatially representing the sub-three-dimensional point group processed in parallel in embodiment 17.

[0212] Figure 210 This is a diagram showing an example of the structure of a bit stream according to embodiment 17.

[0213] Figure 211 This is a diagram used to illustrate the parallel decoding processing of embodiment 17.

[0214] Figure 212 This is a diagram schematically showing the process of full parallel encoding processing in embodiment 17.

[0215] Figure 213 This is a diagram schematically showing the process of fully parallel decoding processing in embodiment 17.

[0216] Figure 214 This is a diagram used to illustrate progressive parallel coding in embodiment 17, and is a diagram showing an example of a tree structure. Figure 215 This is a diagram showing the operation of the core in progressive parallel coding according to the seventeenth embodiment.

[0217] Figure 216 This is a diagram schematically showing the process of progressive parallel encoding processing in embodiment 17.

[0218] Figure 217 This is a flowchart of the three-dimensional data encoding process of embodiment 17.

[0219] Figure 218 This is a flowchart of three-dimensional data decoding processing in embodiment 17. DETAILED DESCRIPTION

[0220] A three-dimensional data encoding method according to one embodiment of the present invention divides a plurality of three-dimensional points contained in three-dimensional data into a plurality of sub-three-dimensional point groups including a first sub-three-dimensional point group and a second sub-three-dimensional point group, appends first information representing the space of the first sub-three-dimensional point group to a header of the first sub-three-dimensional point group, appends second information representing the space of the second sub-three-dimensional point group to a header of the second sub-three-dimensional point group, and encodes each of the first sub-three-dimensional point group and the second sub-three-dimensional point group so as to be independently decodable.

[0221] Thus, this 3D data encoding method can generate encoded data that can independently decode the first and second sub-3D point groups. This allows a 3D data decoding device to process this encoded data in parallel. Alternatively, the 3D data decoding device can selectively decode either the first or second sub-3D point group. This reduces processing time in the 3D data decoding device.

[0222] For example, in the segmentation, the N (N is an integer greater than 2) fork tree structure of the multiple three-dimensional points may be segmented into multiple branches, and the multiple branches may include a first branch corresponding to the first sub-three-dimensional point group and a second branch corresponding to the second three-dimensional point.

[0223] For example, information indicating the layer to which the root of the first branch belongs and the layer to which the root of the second branch belongs may be encoded.

[0224] For example, the layer to which the root of the first branch belongs and the layer to which the root of the second branch belongs may be the same layer.

[0225] For example, different coding tables may be used to perform entropy coding on the first sub-three-dimensional point group and the second sub-three-dimensional point group.

[0226] For example, the coding table may be initialized after entropy coding is performed on the first three-dimensional point group sub-group and before entropy coding is performed on the second three-dimensional point group sub-group.

[0227] For example, it may be possible that, in encoding of the first sub-three-dimensional point group, reference to the second sub-three-dimensional point group is prohibited, and in encoding of the second sub-three-dimensional point group, reference to the first sub-three-dimensional point group is prohibited.

[0228] For example, the position information of the multiple first three-dimensional points included in the first sub-three-dimensional point group and the position information of the multiple second three-dimensional points included in the second sub-three-dimensional point group can be encoded separately so as to be independently decodable, and the attribute information of the multiple first three-dimensional points and the attribute information of the multiple second three-dimensional points can be encoded separately so as to be independently decodable.

[0229] For example, it may be that (1) each of the position information of the multiple first three-dimensional points included in the first sub-three-dimensional point group and the position information of the multiple second three-dimensional points included in the second sub-three-dimensional point group, and (2) one of the attribute information of the multiple first three-dimensional points and the attribute information of the multiple second three-dimensional points are encoded so as to be independently decodable, and the other of (1) each of the position information of the multiple first three-dimensional points and the position information of the multiple second three-dimensional points and (2) the attribute information of the multiple first three-dimensional points and the attribute information of the multiple second three-dimensional points are encoded so as to have a dependency relationship.

[0230] For example, a flag indicating whether the first sub-three-dimensional point group and the second sub-three-dimensional point group are each encoded so as to be independently decodable may be encoded.

[0231] For example, the first information may indicate the maximum coordinates of the first sub-three-dimensional point group in the space, and the second information may indicate the maximum coordinates of the second sub-three-dimensional point group in the space.

[0232] A three-dimensional data decoding method according to one embodiment of the present invention obtains first encoded data and second encoded data, wherein the first encoded data and the second encoded data are generated by encoding a first sub-three-dimensional point group and a second sub-three-dimensional point group contained in a plurality of sub-three-dimensional point groups so as to be independently decodable, wherein the plurality of sub-three-dimensional point groups are generated by segmenting a plurality of three-dimensional points contained in the three-dimensional data, first information representing the space of the first sub-three-dimensional point group is obtained from a head of the first sub-three-dimensional point group, and second information representing the space of the second sub-three-dimensional point group is obtained from a head of the second sub-three-dimensional point group, and the first sub-three-dimensional point group and the second sub-three-dimensional point group are restored by decoding the first encoded data and the second encoded data.

[0233] Thus, the 3D data decoding method can process the first and second encoded data in parallel. Alternatively, the 3D data decoding device can selectively decode either the first or second sub-3D point group. This reduces processing time in the 3D data decoding device.

[0234] For example, the first encoded data and the second encoded data may be generated by encoding the first branch and the second branch contained in the N (N is an integer greater than 2) fork tree structure of the multiple three-dimensional points so that they can be independently decoded, and the first branch corresponds to the first sub-three-dimensional point group, and the second branch corresponds to the second sub-three-dimensional point group.

[0235] For example, information indicating the layer to which the root of the first branch belongs and the layer to which the root of the second branch belongs may be decoded.

[0236] For example, the layer to which the root of the first branch belongs and the layer to which the root of the second branch belongs may be the same layer.

[0237] For example, the first sub-three-dimensional point group and the second sub-three-dimensional point group may be entropy decoded using different coding tables.

[0238] For example, the coding table may be initialized after entropy decoding is performed on the first sub-three-dimensional point group and before entropy decoding is performed on the second sub-three-dimensional point group.

[0239] For example, the second sub-3D point group may not be referenced in decoding the first sub-3D point group, and the first sub-3D point group may not be referenced in decoding the second sub-3D point group.

[0240] For example, it may also be that the first encoded data includes first encoded position data and first encoded attribute data, the first encoded position data is generated by encoding the position information of multiple first three-dimensional points included in the first sub-three-dimensional point group, and the first encoded attribute data is generated by encoding the attribute information of the multiple first three-dimensional points; the second encoded data includes second encoded position data and second encoded attribute data, the second encoded position data is generated by encoding the position information of multiple second three-dimensional points included in the second sub-three-dimensional point group, and the second encoded attribute data is generated by encoding the attribute information of the multiple second three-dimensional points; the first encoded position data and the second encoded position data are generated to be independently decodable, and the second encoded attribute data and the second encoded attribute data are generated to be independently decodable.

[0241] For example, it may also be that the first encoded data and the second encoded data are generated by encoding one of the position information and attribute information of the multiple first three-dimensional points included in the first sub-three-dimensional point group and the multiple second three-dimensional points included in the second sub-three-dimensional point group so as to be independently decodable, and by decoding the first encoded data and the second encoded data respectively, the position information and the one of the attribute information of the multiple first three-dimensional points and the multiple second three-dimensional points are restored respectively, and the three-dimensional data decoding method further obtains third encoded data and fourth encoded data, and the third encoded data and the fourth encoded data are generated by encoding the other of the position information and the attribute information of the multiple first three-dimensional points and the multiple second three-dimensional points so as to have a dependency relationship, and by decoding the third encoded data and the fourth encoded data respectively, the position information and the other of the attribute information of the multiple first three-dimensional points and the multiple second three-dimensional points are restored respectively.

[0242] For example, a flag indicating whether each of the first sub-three-dimensional point group and the second sub-three-dimensional point group is encoded so as to be independently decodable may be decoded.

[0243] For example, the first information may indicate the maximum coordinates of the first sub-three-dimensional point group in the space, and the second information may indicate the maximum coordinates of the second sub-three-dimensional point group in the space.

[0244] In addition, a three-dimensional data encoding device of one embodiment of the present invention includes a processor and a memory, and the processor uses the memory to divide multiple three-dimensional points contained in the three-dimensional data into multiple sub-three-dimensional point groups including a first sub-three-dimensional point group and a second sub-three-dimensional point group, attaches first information representing the space of the first sub-three-dimensional point group to the header of the first sub-three-dimensional point group, attaches second information representing the space of the second sub-three-dimensional point group to the header of the second sub-three-dimensional point group, and encodes the first sub-three-dimensional point group and the second sub-three-dimensional point group respectively so that they can be independently decoded.

[0245] Thus, the 3D data encoding device can generate encoded data that can be independently decoded for the first and second sub-3D point groups. This allows the 3D data decoding device to process this encoded data in parallel. Alternatively, the 3D data decoding device can selectively decode either the first or second sub-3D point group. This reduces processing time in the 3D data decoding device.

[0246] In addition, a three-dimensional data decoding device involved in one embodiment of the present invention includes a processor and a memory, and the processor uses the memory to obtain first encoded data and second encoded data, the first encoded data and the second encoded data are generated by encoding the first sub-3D point group and the second sub-3D point group contained in multiple sub-3D point groups so that they can be independently decoded, the multiple sub-3D point groups are generated by dividing multiple three-dimensional points contained in the three-dimensional data, obtaining first information representing the space of the first sub-3D point group from the head of the first sub-3D point group, and obtaining second information representing the space of the second sub-3D point group from the head of the second sub-3D point group, and restoring the first sub-3D point group and the second sub-3D point group by decoding the first encoded data and the second encoded data.

[0247] This allows the 3D data decoding device to process the first and second encoded data in parallel. Alternatively, the 3D data decoding device can selectively decode either the first or second sub-3D point group. This reduces processing time within the 3D data decoding device.

[0248] In addition, these general or specific forms can be implemented by systems, methods, integrated circuits, computer programs or computer-readable recording media such as CD-ROMs, and can be implemented by any combination of systems, methods, integrated circuits, computer programs and recording media.

[0249] The following detailed description of the embodiments is given with reference to the accompanying drawings. In addition, the embodiments to be described below are all specific examples of the present disclosure. The numerical values, shapes, materials, components, configuration positions of components, connection forms, steps, order of steps, etc. shown in the following embodiments are all examples, and their purpose is not to limit the present disclosure. Moreover, among the components of the following embodiments, the components that are not recorded in the technical solution showing the highest concept are described as arbitrary components.

[0250] (Implementation Method 1)

[0251] First, the data structure of encoded three-dimensional data (hereinafter also referred to as encoded data) according to this embodiment will be described. Figure 1 The structure of the encoded three-dimensional data according to this embodiment is shown.

[0252] In this embodiment, the three-dimensional space is divided into spaces (SPC) equivalent to the spaces of pictures in the encoding of dynamic images, and the three-dimensional data is encoded in units of space. The space is further divided into volumes (VLM) equivalent to macroblocks in dynamic image encoding, and prediction and conversion are performed in units of VLM. The volume includes a plurality of voxels (VXL), which are the smallest units corresponding to position coordinates. In addition, prediction means generating predicted three-dimensional data similar to the processing unit of the processing object with reference to other processing units, and encoding the difference between the predicted three-dimensional data and the processing unit of the processing object, similar to the prediction performed in the two-dimensional image. Furthermore, the prediction includes not only spatial prediction with reference to other prediction units at the same time, but also temporal prediction with reference to prediction units at different times.

[0253] For example, when encoding a three-dimensional space represented by point cloud data, such as a point cloud, a three-dimensional data encoding device (hereinafter referred to as an encoding device) encodes each point in the point cloud or multiple points contained within a voxel, all at once, according to the size of the voxel. Subdividing the voxels allows for a highly accurate representation of the point cloud's three-dimensional shape, while increasing the voxel size allows for a coarser representation of the point cloud's three-dimensional shape.

[0254] In addition, although the following description uses the case where the three-dimensional data is a point cloud as an example, the three-dimensional data is not limited to point clouds and can also be three-dimensional data in any form.

[0255] Furthermore, voxels with a hierarchical structure can be utilized. In this case, within the nth level, it is possible to sequentially indicate whether a sampling point exists in the n-1th level or lower levels (the levels below the nth level). For example, when decoding only the nth level, if a sampling point exists in the n-1th level or lower levels, decoding can be performed as if the sampling point exists at the center of the voxel in the nth level.

[0256] Furthermore, the encoding device obtains point group data through a distance sensor, a stereo camera, a monocular camera, a gyroscope, or an inertial sensor.

[0257] Similar to video encoding, spaces are classified into at least one of the following three prediction structures: independently decodable intra-frame space (I-SPC), unidirectionally referenced prediction space (P-SPC), and bidirectionally referenced prediction space (B-SPC). Spaces also contain two types of time information: decoding time and display time.

[0258] And, as Figure 1 As shown, as a processing unit including a plurality of spaces, there is a GOS (Group of Space) which is a random access unit. Furthermore, as a processing unit including a plurality of GOS, there is a world space (WLD).

[0259] The spatial area occupied by world space is associated with an absolute position on Earth using GPS or latitude and longitude information. This location information is stored as metadata. Metadata can be included in the encoded data or transmitted separately.

[0260] Furthermore, within the GOS, all SPCs may be three-dimensionally adjacent, or there may be SPCs that are not three-dimensionally adjacent to other SPCs.

[0261] In addition, below, the encoding, decoding, or referencing of three-dimensional data contained in a processing unit such as a GOS, SPC, or VLM will also be referred to simply as encoding, decoding, or referencing the processing unit. Furthermore, the three-dimensional data contained in the processing unit includes, for example, at least one pair of spatial positions such as three-dimensional coordinates and characteristic values ​​such as color information.

[0262] Next, the prediction structure of the SPC in the GOS will be described. Although multiple SPCs in the same GOS or multiple VLMs in the same SPC occupy different spaces, they have the same time information (decoding time and display time).

[0263] Furthermore, within a GOS, the first SPC in decoding order is the I-SPC. Furthermore, there are two types of GOS: closed GOS and open GOS. A closed GOS is one that can decode all SPCs within the GOS when decoding starts from the first I-SPC. In an open GOS, some SPCs within the GOS that are earlier than the display time of the first I-SPC refer to a different GOS and can only be decoded in that GOS.

[0264] In addition, in coded data such as map information, the WLD may be decoded in the reverse order of the coding order. If there is a dependency between GOS, it will be difficult to reproduce the data in the reverse order. Therefore, in this case, a closed GOS is basically used.

[0265] Furthermore, the GOS has a layer structure in the height direction, and encoding or decoding is performed sequentially starting from the SPC of the bottom layer.

[0266] Figure 2 An example of the prediction structure between SPCs belonging to the lowest layer of the GOS is shown. Figure 3 An example of an inter-layer prediction structure is shown.

[0267] There are one or more I-SPCs within a GOS. While objects such as people, animals, cars, bicycles, traffic lights, and buildings serving as land landmarks exist within a three-dimensional space, encoding small objects as I-SPCs is particularly effective. For example, a three-dimensional data decoding device (hereinafter also referred to as a decoding device) decodes only the I-SPCs within the GOS when decoding the GOS with low throughput or high speed.

[0268] Furthermore, the encoding device may switch the encoding interval or the appearance frequency of the I-SPC according to the density of objects within the WLD.

[0269] And, in Figure 3 In the configuration shown, the encoding device or decoding device encodes or decodes multiple layers sequentially starting from the lower layer (layer 1). This allows, for example, autonomous vehicles to prioritize data near the ground, which contains a large amount of information.

[0270] In addition, in the coded data used by drones, etc., coding or decoding can be performed sequentially starting from the SPC of the upper layer in the height direction within the GOS.

[0271] Furthermore, the encoding device or decoding device may encode or decode multiple layers in such a manner that the decoding device can roughly grasp the GOS and gradually increase the resolution. For example, the encoding device or decoding device may encode or decode layers in the order of 3, 8, 1, 9, etc.

[0272] Next, the corresponding method of static objects and dynamic objects is explained.

[0273] In three-dimensional space, there are static objects or scenes such as buildings and roads (hereinafter referred to as static objects), as well as dynamic objects such as vehicles and people (hereinafter referred to as dynamic objects). Object detection can also be performed by extracting feature points from point cloud data or images captured by stereo cameras. Here, an example of a method for encoding dynamic objects is described.

[0274] The first method is a method of encoding without distinguishing between static objects and dynamic objects. The second method is a method of distinguishing between static objects and dynamic objects using identification information.

[0275] For example, GOS is used as the identification unit. In this case, GOS including SPCs constituting static objects and GOS including SPCs constituting dynamic objects are distinguished within the coded data or by identification information stored separately from the coded data.

[0276] Alternatively, SPC is used as the identification unit. In this case, the SPC including only the VLM constituting the static object and the SPC including the VLM constituting the dynamic object are distinguished by the above-mentioned identification information.

[0277] Alternatively, VLM or VXL may be used as the identification unit. In this case, a VLM or VXL including a static object and a VLM or VXL including a dynamic object are distinguished by the above-mentioned identification information.

[0278] Furthermore, the encoding device may encode dynamic objects as one or more VLMs or SPCs, and may encode the VLMs or SPCs containing static objects and the SPCs containing dynamic objects as separate GOSs. Furthermore, if the size of the GOS is variable depending on the size of the dynamic object, the encoding device may store the size of the GOS separately as meta-information.

[0279] Furthermore, the encoding device encodes static objects and dynamic objects independently, allowing dynamic objects to be superimposed on the world space composed of static objects. In this case, a dynamic object is composed of one or more SPCs, each of which corresponds to one or more SPCs that constitute the static object with which it is superimposed. Furthermore, dynamic objects can be represented not by SPCs but by one or more VLMs or VXLs.

[0280] Furthermore, the encoding device may encode static objects and dynamic objects as different streams.

[0281] Furthermore, the encoding device may generate a GOS that includes one or more SPCs constituting a dynamic object. Furthermore, the encoding device may set the GOS (GOS_M) including the dynamic object and the GOS of the static object corresponding to the spatial region of the GOS_M to be of the same size (occupying the same spatial region). This enables overlapping processing on a GOS basis.

[0282] The P-SPC or B-SPC that constitutes a dynamic object can also refer to the SPC contained in a different coded GOS. When the position of a dynamic object changes over time and the same dynamic object is coded as a GOS at different times, cross-GOS reference is effective from the perspective of compression rate.

[0283] Furthermore, the first and second methods can be switched depending on the intended use of the encoded data. For example, when the encoded three-dimensional data is used as a map, separation from dynamic objects is desirable, so the encoding device uses the second method. Alternatively, when encoding three-dimensional data of events such as concerts or sports, where separation of dynamic objects is not necessary, the encoding device uses the first method.

[0284] Furthermore, the decoding time and display time of the GOS or SPC can be stored in the encoded data or as meta-information. Furthermore, the time information of static objects can all be the same. In this case, the actual decoding time and display time can be determined by the decoding device. Alternatively, different values ​​can be assigned to each GOS or SPC as the decoding time, and the same value can be assigned to all as the display time. Moreover, as shown in the decoder mode in dynamic image encoding such as HEVC's HRD (Hypothetical Reference Decoder), the decoder has a buffer of a specified size. As long as the bit stream is read at a specified bit rate according to the decoding time, a model that will not be destroyed and is guaranteed to be decodable can be imported.

[0285] Next, the configuration of GOS in the world space is explained. The coordinates of the three-dimensional space in the world space are represented by three mutually orthogonal coordinate axes (x-axis, y-axis, z-axis). By setting a prescribed rule in the encoding order of GOS, spatially adjacent GOS can be encoded continuously in the encoded data. For example, Figure 4 In the example shown, the GOS within the xz plane are encoded continuously. After encoding all GOS within a single xz plane, the y-axis value is updated. In other words, as encoding continues, world space extends in the y-axis direction. Furthermore, the GOS index numbers are set to the encoding order.

[0286] Here, the three-dimensional world space corresponds one-to-one to absolute geographic coordinates such as GPS or latitude and longitude. Alternatively, the three-dimensional space can be represented by relative positions relative to a predetermined reference position. The directions of the x-axis, y-axis, and z-axis of the three-dimensional space are represented as direction vectors determined based on latitude and longitude, and these direction vectors are stored as metadata along with the encoded data.

[0287] Furthermore, the size of the GOS is set to be fixed, and the encoding device stores this size as meta-information. Furthermore, the size of the GOS can be switched, for example, depending on whether the location is urban, indoors, or outdoors. In other words, the size of the GOS can be switched based on the quantity or nature of objects with informational value. Alternatively, within the same world space, the encoding device can appropriately switch the size of the GOS or the spacing of I-SPCs within the GOS based on, for example, the density of objects. For example, the encoding device can set the GOS size to be smaller and the spacing of I-SPCs within the GOS to be shorter when the density of objects is higher.

[0288] exist Figure 5 In the example, in the area from the 3rd to the 10th GOS, the density of objects is high, so the GOS is subdivided to achieve fine-grained random access. In addition, the 7th to the 10th GOS exist on the back of the 3rd to the 6th GOS, respectively.

[0289] Next, the configuration and operation flow of the three-dimensional data encoding device according to this embodiment will be described. Figure 6 This is a block diagram of the three-dimensional data encoding device 100 according to this embodiment. Figure 7 3D data encoding apparatus 100 is a flowchart showing an example of its operation.

[0290] Figure 6 The three-dimensional data encoding device 100 shown generates encoded three-dimensional data 112 by encoding three-dimensional data 111. The three-dimensional data encoding device 100 includes an acquisition unit 101, an encoding region determination unit 102, a division unit 103, and an encoding unit 104.

[0291] like Figure 7 As shown, first, the acquisition unit 101 acquires three-dimensional data 111 as point cloud data ( S101 ).

[0292] Next, the coding region determination unit 102 determines a coding target region from the spatial region corresponding to the obtained point cloud data (S102). For example, the coding region determination unit 102 determines a spatial region around the position of the user or vehicle as the coding target region.

[0293] Next, the segmentation unit 103 segments the point cloud data contained in the encoding target region into processing units. Here, the processing units are the aforementioned GOS and SPCs, for example. Furthermore, the encoding target region corresponds to the aforementioned world space, for example. Specifically, the segmentation unit 103 segments the point cloud data into processing units based on the size of the pre-set GOS and the presence or size of dynamic objects (S103). Furthermore, the segmentation unit 103 determines the starting position of the SPC, which is the first in the encoding order, within each GOS.

[0294] Next, the encoding unit 104 generates the encoded three-dimensional data 112 by sequentially encoding the plurality of SPCs in each GOS ( S104 ).

[0295] In addition, although an example is shown here in which each GOS is encoded after the encoding target area is divided into GOS and SPC, the processing order is not limited to the above. For example, the order may be such that the composition of a GOS is determined, the GOS is encoded, and then the composition of the GOS is determined.

[0296] In this manner, the 3D data encoding device 100 generates encoded 3D data 112 by encoding the 3D data 111. Specifically, the 3D data encoding device 100 divides the 3D data into random access units, namely, first processing units (GOS) corresponding to respective 3D coordinates. The first processing units (GOS) are divided into a plurality of second processing units (SPCs), and the second processing units (SPCs) are divided into a plurality of third processing units (VLMs). Furthermore, each third processing unit (VLM) includes one or more voxels (VXLs), which are the smallest units corresponding to positional information.

[0297] Next, the 3D data encoding device 100 encodes each of the plurality of first processing units (GOS) to generate the encoded 3D data 112. Specifically, the 3D data encoding device 100 encodes each of the plurality of second processing units (SPC) in each first processing unit (GOS). Furthermore, the 3D data encoding device 100 encodes each of the plurality of third processing units (VLM) in each second processing unit (SPC).

[0298] For example, when the first processing unit (GOS) to be processed is a closed GOS, the three-dimensional data encoding device 100 encodes the second processing unit (SPC) to be processed included in the first processing unit (GOS) to be processed by referring to other second processing units (SPC) included in the first processing unit (GOS) to be processed. In other words, the three-dimensional data encoding device 100 does not refer to the second processing unit (SPC) included in the first processing unit (GOS) different from the first processing unit (GOS) to be processed.

[0299] Furthermore, when the first processing unit (GOS) of the processing object is an open GOS, the second processing unit (SPC) of the processing object included in the first processing unit (GOS) of the processing object is encoded with reference to other second processing units (SPC) included in the first processing unit (GOS) of the processing object, or a second processing unit (SPC) included in a first processing unit (GOS) different from the first processing unit (GOS) of the processing object.

[0300] Furthermore, the three-dimensional data encoding device 100 selects one as the type of the second processing unit (SPC) of the processing object from a first type (I-SPC) that does not refer to other second processing units (SPCs), a second type (P-SPC) that refers to one other second processing unit (SPC), and a third type that refers to two other second processing units (SPCs), and encodes the second processing unit (SPC) of the processing object according to the selected type.

[0301] Next, the configuration and operation flow of the three-dimensional data decoding device according to this embodiment will be described. Figure 8 This is a block diagram of a three-dimensional data decoding device 200 according to this embodiment. Figure 9 3D data decoding apparatus 200 is a flowchart showing an example of its operation.

[0302] Figure 8 The illustrated three-dimensional data decoding device 200 generates decoded three-dimensional data 212 by decoding encoded three-dimensional data 211. Here, the encoded three-dimensional data 211 is, for example, the encoded three-dimensional data 112 generated by the three-dimensional data encoding device 100. The three-dimensional data decoding device 200 includes an acquisition unit 201, a decoding start GOS determination unit 202, a decoding SPC determination unit 203, and a decoding unit 204.

[0303] First, the acquisition unit 201 acquires the encoded 3D data 211 (S201). Next, the decoding start GOS determination unit 202 determines the GOS to be decoded (S202). Specifically, the decoding start GOS determination unit 202 refers to metadata stored within or separately from the encoded 3D data 211 and determines the GOS that includes the SPC corresponding to the spatial position, object, or time at which decoding should start as the decoding start GOS.

[0304] Next, the decoded SPC determination unit 203 determines the type (I, P, B) of the SPC to be decoded within the GOS (S203). For example, the decoded SPC determination unit 203 determines (1) whether to decode only the I-SPC, (2) whether to decode both the I-SPC and the P-SPC, or (3) whether to decode all types. Alternatively, this step may be omitted if the type of SPC to be decoded is predetermined, such as when all SPCs are to be decoded.

[0305] Next, the decoding unit 204 obtains the first SPC in the decoding order (the same as the encoding order) within the GOS, and the address position at the beginning of the encoded 3D data 211. From this address position, it obtains the encoded data of the first SPC and decodes each SPC sequentially from the first SPC (S204). The address position is stored in meta-information, etc.

[0306] In this manner, the 3D data decoding device 200 decodes the decoded 3D data 212. Specifically, the 3D data decoding device 200 decodes each of the encoded 3D data 211 of the first processing unit (GOS) corresponding to the three-dimensional coordinates, thereby generating the decoded 3D data 212 of the first processing unit (GOS), which is a random access unit. More specifically, the 3D data decoding device 200 decodes each of the plurality of second processing units (SPC) for each first processing unit (GOS). Furthermore, the 3D data decoding device 200 decodes each of the plurality of third processing units (VLM) for each second processing unit (SPC).

[0307] The following describes the meta-information for random access. This meta-information is generated by the three-dimensional data encoding device 100 and included in the encoded three-dimensional data 112 (211).

[0308] In conventional random access of two-dimensional moving images, decoding starts from the first frame of the random access unit near a specified time. However, in world space, random access to (coordinates, objects, etc.) is also possible in addition to time.

[0309] Therefore, in order to realize random access to at least the three elements of coordinates, objects, and time, a table is prepared that associates each element with the index number of the GOS. Furthermore, the index number of the GOS is associated with the address of the I-SPC at the beginning of the GOS. Figure 10 An example of a table included in the meta information is shown. Figure 10 Of all the tables shown, at least one table may be used.

[0310] The following describes random access using coordinates as the starting point, as an example. When accessing the coordinates (x2, y2, z2), the coordinate-GOS table is first referenced, and it is determined that the location with coordinates (x2, y2, z2) is included in the second GOS. Next, the GOS address table is referenced, and since the address of the first I-SPC in the second GOS is known to be addr(2), the decoding unit 204 obtains data from this address and begins decoding.

[0311] In addition, the address can be an address in a logical format or a physical address of an HDD or memory. Furthermore, information identifying a file segment can be used instead of an address. For example, a file segment is a unit obtained by segmenting one or more GOSs.

[0312] Furthermore, if an object spans multiple GOSs, the object GOS table may also indicate the GOSs to which the multiple objects belong. If the multiple GOSs are closed, the encoding and decoding devices can perform encoding and decoding in parallel. Furthermore, if the multiple GOSs are open, the GOSs can reference each other, further improving compression efficiency.

[0313] Examples of objects include people, animals, cars, bicycles, traffic lights, and buildings that serve as landmarks on land. For example, when encoding in world space, the three-dimensional data encoding device 100 extracts feature points unique to an object from a three-dimensional point cloud, detects the object based on the feature points, and can set the detected object as a random access point.

[0314] In this manner, the three-dimensional data encoding device 100 generates first information indicating a plurality of first processing units (GOS) and the three-dimensional coordinates corresponding to each of the plurality of first processing units (GOS). The encoded three-dimensional data 112 (211) includes the first information. Furthermore, the first information further indicates at least one of an object, a time, and a data storage destination corresponding to each of the plurality of first processing units (GOS).

[0315] The three-dimensional data decoding device 200 obtains first information from the encoded three-dimensional data 211 , uses the first information to identify the first processing unit of encoded three-dimensional data 211 corresponding to the specified three-dimensional coordinates, object or time, and decodes the encoded three-dimensional data 211 .

[0316] Other examples of meta-information are described below. In addition to the meta-information for random access, the 3D data encoding device 100 can also generate and store the following meta-information. The 3D data decoding device 200 can also use this meta-information during decoding.

[0317] When using 3D data as map information, profiles are defined based on the intended use, and information indicating the profiles can be included in the meta-information. For example, profiles for urban areas or suburban areas, or for flying objects, can be defined, with the maximum and minimum sizes of the world space, SPC, or VLM defined for each. For example, in an urban area profile, more detailed information is required than in a suburban area, so the minimum size of the VLM is set smaller.

[0318] Meta-information may also include a tag value indicating the type of object. This tag value corresponds to the VLM, SPC, or GOS that constitutes the object. Tag values ​​can be set based on the object type, for example, a tag value of "0" representing a "person," a tag value of "1" representing a "car," and a tag value of "2" representing a "traffic light." Alternatively, when the object type is difficult or unnecessary to determine, a tag value indicating properties such as size or whether the object is dynamic or static can be used.

[0319] Furthermore, the meta-information may include information indicating the range of the spatial region occupied by the world space.

[0320] Furthermore, the meta-information may be used as header information common to the entire stream of coded data or a plurality of SPCs such as an SPC within a GOS, and may store the size of the SPC or VXL.

[0321] Furthermore, the meta-information may include identification information of a distance sensor, a camera, or the like used in generating the point cloud, or information indicating the positional accuracy of a point group within the point cloud.

[0322] Also, the meta information may include information showing whether the world space is composed of only static objects or contains dynamic objects.

[0323] Modifications of this embodiment will be described below.

[0324] The encoding device or decoding device can encode or decode two or more different SPCs or GOSs in parallel. The GOSs to be encoded or decoded in parallel can be determined based on meta-information indicating the spatial position of the GOSs.

[0325] When three-dimensional data is used as a spatial map for a moving vehicle or flying object, or when such a spatial map is generated, the encoding device or decoding device can encode or decode the GOS or SPC contained in a space determined based on GPS, path information, or zoom factor, etc.

[0326] Furthermore, the decoding device may also decode sequentially starting from spaces closest to its own position or path. The encoding device or decoding device may also prioritize spaces farther from its own position or path, lowering the priority of spaces closer to it, and perform encoding or decoding accordingly. Here, lowering the priority means reducing the processing order, reducing the resolution (post-filtering processing), or reducing image quality (to improve coding efficiency, such as increasing the quantization step size).

[0327] Furthermore, when decoding coded data that has been spatially hierarchically coded, the decoding device may decode only the lower layer.

[0328] Furthermore, the decoding device may start decoding from a lower layer according to the zoom ratio or purpose of the map.

[0329] Furthermore, in applications such as estimating the self-position of a car or robot during autonomous driving or identifying objects, the encoding device or decoding device may also reduce the resolution of areas outside the area within a specified height from the road surface (the area for identification) for encoding or decoding.

[0330] Furthermore, the encoding device may independently encode the point clouds representing indoor and outdoor spatial shapes. For example, by separating the GOS representing indoor spaces (indoor GOS) from the GOS representing outdoor spaces (outdoor GOS), the decoding device can select the GOS to decode based on the viewpoint position when using the encoded data.

[0331] Furthermore, the encoding device can encode indoor and outdoor GOSs that are close to each other in the coded stream. For example, the encoding device can associate identifiers between the two and store information indicating the corresponding identifiers within the coded stream or in separately stored meta-information. This allows the decoding device to identify the indoor and outdoor GOSs that are close to each other by referring to the information in the meta-information.

[0332] Furthermore, the encoding device may switch the size of the GOS or SPC between the indoor GOS and the outdoor GOS. For example, the encoding device may set the size of the GOS to be smaller indoors than outdoors. Furthermore, the encoding device may change the accuracy of feature point extraction from the point cloud or the accuracy of object detection between the indoor GOS and the outdoor GOS.

[0333] Furthermore, the encoding device can add information to the encoded data that allows the decoding device to distinguish between dynamic and static objects. This allows the decoding device to display dynamic objects in combination with a red frame or explanatory text. Alternatively, the decoding device can replace dynamic objects with a red frame or explanatory text alone. Furthermore, the decoding device can display more detailed object categories. For example, a car can be displayed with a red frame, while a person can be displayed with a yellow frame.

[0334] Furthermore, the encoding or decoding device may determine whether to encode or decode dynamic objects and static objects as different SPCs or GOSs based on the frequency of occurrence of dynamic objects or the ratio of static objects to dynamic objects. For example, if the frequency or ratio of dynamic objects exceeds a threshold, an SPC or GOS containing a mixture of dynamic and static objects is permitted. If the frequency or ratio of dynamic objects does not exceed the threshold, an SPC or GOS containing a mixture of dynamic and static objects is not permitted.

[0335] When dynamic objects are detected from 2D camera image information rather than point clouds, the encoding device can obtain information identifying the detection result (box or text, etc.) and the object position separately, encoding this information as part of the 3D encoded data. In this case, the decoding device displays auxiliary information representing the dynamic object (box or text) overlaid on the decoded result of the static object.

[0336] Furthermore, the encoding device may change the density of VXL or VLM based on factors such as the complexity of the shape of the static object. For example, the encoding device may set VXL or VLM to be denser when the shape of the static object is more complex. Furthermore, the encoding device may determine the quantization step size when quantizing spatial position or color information based on the density of VXL or VLM. For example, the encoding device may set the quantization step size to be smaller when the VXL or VLM is denser.

[0337] As described above, the encoding device or decoding device according to the present embodiment performs spatial encoding or decoding in spatial units having coordinate information.

[0338] Furthermore, the encoding device and the decoding device perform encoding or decoding in units of volume in space. The volume includes voxels, which are the minimum units corresponding to the position information.

[0339] Furthermore, the encoding device and decoding device perform encoding or decoding by creating a table that associates various elements of spatial information, including coordinates, objects, and time, with GOPs, or by creating a table that associates various elements, thereby establishing a correspondence between arbitrary elements. Furthermore, the decoding device uses the values ​​of the selected elements to determine coordinates, identifies a volume, voxel, or space based on the coordinates, and decodes the space including the volume or voxel, or the identified space.

[0340] Furthermore, the encoding device determines a volume, voxel, or space that can be selected by an element through feature point extraction or object recognition, and encodes it as a volume, voxel, or space that can be randomly accessed.

[0341] Spaces are divided into three types: I-SPC, which can be encoded or decoded by the space itself; P-SPC, which can be encoded or decoded with reference to any one processed space; and B-SPC, which can be encoded or decoded with reference to any two processed spaces.

[0342] One or more volumes correspond to static objects or dynamic objects. The space containing static objects and the space containing dynamic objects are encoded or decoded as different GOSs. That is, the SPC containing static objects and the SPC containing dynamic objects are assigned to different GOSs.

[0343] Dynamic objects are encoded or decoded for each object and correspond to one or more spaces containing only static objects. In other words, multiple dynamic objects are encoded separately, and the resulting encoded data of multiple dynamic objects corresponds to an SPC containing only static objects.

[0344] The encoding and decoding devices increase the priority of the I-SPC within the GOS to perform encoding or decoding. For example, the encoding device performs encoding to minimize degradation of the I-SPC (enabling more faithful reproduction of the original 3D data after decoding). Furthermore, the decoding device, for example, only decodes the I-SPC.

[0345] The encoding device can change the frequency of using I-SPCs to perform encoding according to the density or number of objects in world space. Specifically, the encoding device changes the frequency of selecting I-SPCs according to the number or density of objects contained in the three-dimensional data. For example, the encoding device may increase the frequency of using I-space as the density of objects in world space increases.

[0346] Furthermore, the encoding device sets a random access point in units of GOS, and stores information indicating a spatial region corresponding to the GOS in the header information.

[0347] The encoding device, for example, uses a default value as the spatial size of the GOS. Alternatively, the encoding device may change the size of the GOS based on the number (quantity) or density of objects or dynamic objects. For example, the encoding device may set the spatial size of the GOS to be smaller when the objects or dynamic objects are denser or more numerous.

[0348] Furthermore, the space or volume includes a cluster of feature points derived from information obtained by sensors such as depth sensors, gyroscopes, or cameras. The coordinates of the feature points are set to the center of the voxel. Furthermore, by subdividing the voxels, it is possible to achieve higher accuracy in positional information.

[0349] The feature point cluster is derived using multiple pictures. The multiple pictures have at least two types of time information: actual time information and spatially corresponding time information common to multiple pictures (for example, encoding time used for rate control, etc.).

[0350] Furthermore, encoding or decoding is performed in units of GOS including one or more spaces.

[0351] The encoding device and the decoding device refer to the space in the processed GOS to predict the P space or B space in the GOS to be processed.

[0352] Alternatively, the encoding device and the decoding device predict the P space or B space in the GOS to be processed by using the processed space in the GOS to be processed without referring to different GOSs.

[0353] Furthermore, the encoding device and the decoding device transmit or receive the encoded stream in units of a world space including one or more GOSs.

[0354] Furthermore, the GOS has a layer structure in at least one direction within world space, and the encoding and decoding devices perform encoding or decoding starting from the lowest layer. For example, the randomly accessible GOS belongs to the lowest layer. The GOS belonging to a higher layer only references the GOS belonging to the layers below it. In other words, the GOS is spatially divided in a predetermined direction and includes multiple layers, each having one or more SPCs. The encoding and decoding devices perform encoding or decoding for each SPC by referencing SPCs contained in layers in the same layer or layers below it.

[0355] Furthermore, the encoding device and decoding device continuously encode or decode the GOS within a world space unit that includes multiple GOS. The encoding device and decoding device write or read information indicating the order (direction) of encoding or decoding as metadata. In other words, the encoded data includes information indicating the order in which the multiple GOS were encoded.

[0356] Furthermore, the encoding device and the decoding device encode or decode two or more different spaces or GOSs in parallel.

[0357] Furthermore, the encoding device and the decoding device encode or decode the space information (coordinates, size, etc.) of the space or the GOS.

[0358] Furthermore, the encoding device and the decoding device encode or decode the space or GOS included in the specific space specified based on external information related to the own position and / or area size, such as GPS, route information, or magnification.

[0359] The encoding device or decoding device performs encoding or decoding by giving a lower priority to a space far from the own position than to a space close to the own position.

[0360] The encoding device sets a direction in the world space according to a magnification or application, and encodes a GOS having a layer structure in that direction. Furthermore, the decoding device decodes the GOS having a layer structure in the world space direction set according to the magnification or application, preferentially starting from the lower layer.

[0361] The encoding device varies the accuracy of feature point extraction and object recognition, the size of the spatial area, etc. in indoor and outdoor spaces. However, the encoding device and decoding device encode or decode indoor GOS and outdoor GOS with close coordinates adjacent to each other in world space, and also encode or decode these identifiers in correspondence.

[0362] (Implementation Method 2)

[0363] When using point cloud encoded data in actual devices or services, it is desirable to transmit and receive only the required information according to the application in order to reduce network bandwidth. However, existing 3D data encoding structures do not have this capability, and therefore no corresponding encoding method exists.

[0364] What will be described in this embodiment is a three-dimensional data encoding method and a three-dimensional data encoding device for providing the function of sending and receiving required information according to the purpose in the encoded data of a three-dimensional point cloud, as well as a three-dimensional data decoding method and a three-dimensional data decoding device for decoding the encoded data.

[0365] A voxel (VXL) having a certain feature value or more is defined as a feature voxel (FVXL), and a world space (WLD) formed by the FVXL is defined as a sparse world space (SWLD). Figure 11This section shows an example of a sparse world space and its configuration. SWLD includes: FGOS, a GOS constructed from FVXL; FSPC, an SPC constructed from FVXL; and FVLM, a VLM constructed from FVXL. The data structures and prediction structures of FGOS, FSPC, and FVLM can be the same as those of GOS, SPC, and VLM.

[0366] Feature quantities represent the three-dimensional positional information of the VXL or visible light information at the VXL position. In particular, feature quantities are more likely to be detected at corners and edges of three-dimensional objects. Specifically, these feature quantities may be three-dimensional feature quantities or visible light feature quantities as described below, but any feature quantity may be used as long as it represents the position, brightness, or color information of the VXL.

[0367] As the three-dimensional feature, a SHOT feature (Signature of Histograms of Orientations), a PFH feature (Point Feature Histograms), or a PPF feature (Point Pair Feature) is used.

[0368] SHOT features are obtained by segmenting the area around the VXL, calculating the inner product between the reference point and the normal vector of the segmented area, and then histizing the result. These features are characterized by high dimensionality and high expressiveness.

[0369] PFH features are obtained by selecting multiple pairs of points near VXL, calculating normal vectors and other parameters from these two points, and then histogramming them. Since these PFH features are histogram features, they are robust to small amounts of noise and have high expressiveness.

[0370] The PPF feature is calculated based on the VXL of two points using the normal vector, etc. Since all VXLs are used in this PPF feature, it is insensitive to occlusion.

[0371] Furthermore, as feature quantities of visible light, SIFT (Scale-Invariant Feature Transform), SURF (Speeded Up Robust Features), or HOG (Histogram of Oriented Gradients) that utilize information such as brightness gradient information of an image can be used.

[0372] SWLD is generated by calculating the above-mentioned feature quantity from each VXL of WLD and extracting FVXL. Here, SWLD can be updated every time WLD is updated, or it can be updated regularly after a certain period of time regardless of the update timing of WLD.

[0373] SWLDs can be generated for each feature. For example, as shown in SWLD1 based on SHOT features and SWLD2 based on SIFT features, SWLDs can be generated for each feature and used according to the application. Furthermore, the calculated feature values ​​for each FVXL can be stored as feature value information in each FVXL.

[0374] Next, the method of using the sparse world space (SWLD) will be described. Since the SWLD only contains feature voxels (FVXL), the data size is generally smaller than that of the WLD which includes all VXLs.

[0375] In applications that utilize feature quantities to achieve a specific purpose, using SWLD information instead of WLD can reduce the time required to read data from the hard disk and reduce the bandwidth and transmission time during network transmission. For example, by pre-storing WLD and SWLD as map information on a server, switching the transmitted map information to WLD or SWLD based on client requests can reduce network bandwidth and transmission time. A specific example is shown below.

[0376] Figure 12 as well as Figure 13 The following shows the use of SWLD and WLD. Figure 12 As shown, when client 1, which is an in-vehicle device, needs map information for determining its own position, client 1 sends a request to the server to obtain map data for estimating its own position (S301). The server sends SWLD to client 1 in accordance with the acquisition request (S302). Client 1 uses the received SWLD to determine its own position (S303). At this time, client 1 obtains VXL information of the surrounding area of ​​client 1 through various methods such as distance sensors such as rangefinders, stereo cameras, or a combination of multiple monocular cameras, and estimates its own position information based on the obtained VXL information and SWLD. Here, the own position information includes the three-dimensional position information and orientation of client 1.

[0377] like Figure 13As shown, when client 2, an in-vehicle device, requires map information for mapping purposes such as three-dimensional maps, client 2 sends a request to the server to obtain map data for mapping (S311). In response to this request, the server sends a WLD to client 2 (S312). Client 2 uses the received WLD to perform map rendering (S313). At this point, client 2 creates a conceptual image using, for example, images captured by a visible light camera and the WLD received from the server, and then renders the created image on a screen such as a car navigation system.

[0378] As described above, the server sends SWLD to the client for applications that primarily require the characteristic values ​​of each VXL, such as estimating its own position, and sends WLD to the client when detailed VXL information is required, such as for map drawing. This enables efficient transmission and reception of map data.

[0379] In addition, the client can determine whether it needs SWLD or WLD and request the server to send SWLD or WLD. Moreover, the server can determine whether to send SWLD or WLD based on the status of the client or the network.

[0380] Next, a method for switching between the transmission and reception of the sparse world space (SWLD) and the world space (WLD) will be described.

[0381] The reception of WLD or SWLD can be switched according to the network bandwidth. Figure 14 An example of operation in this case is shown. For example, when a low-speed network with sufficient network bandwidth, such as in an LTE (Long Term Evolution) environment, is used, the client accesses the server via the low-speed network (S321) and obtains the SWLD (map information) from the server (S322). Alternatively, when a high-speed network with sufficient network bandwidth, such as in a WiFi environment, is used, the client accesses the server via the high-speed network (S323) and obtains the WLD from the server (S324). This allows the client to obtain appropriate map information based on the client's network bandwidth.

[0382] Specifically, the client receives SWLD via LTE outdoors, and acquires WLD via WiFi when entering a facility or other indoor location. This allows the client to acquire more detailed map information indoors.

[0383] In this way, the client can request WLD or SWLD from the server according to the frequency band of the network it is using. Alternatively, the client can send information indicating the frequency band of the network it is using to the server, and the server can send appropriate data (WLD or SWLD) to the client based on this information. Alternatively, the server can determine the network bandwidth of the client and send appropriate data (WLD or SWLD) to the client.

[0384] Furthermore, the reception of WLD or SWLD can be switched according to the moving speed. Figure 15 An example of operation in this case is shown. For example, when the client is moving at high speed (S331), the client receives SWLD from the server (S332). In addition, when the client is moving at low speed (S333), the client receives WLD from the server (S334). Accordingly, the client can suppress the network bandwidth and obtain map information according to the speed. Specifically, when the client is driving on a highway, by receiving SWLD with a small amount of data, the map information can be updated at a roughly appropriate speed. In addition, when the client is driving on a general road, by receiving WLD, more detailed map information can be obtained.

[0385] In this way, the client can request a WLD or SWLD from the server based on its own moving speed. Alternatively, the client can send information indicating its own moving speed to the server, and the server can send appropriate data (WLD or SWLD) to the client based on this information. Alternatively, the server can determine the client's moving speed and send appropriate data (WLD or SWLD) to the client.

[0386] Alternatively, the client can first obtain the SWLD from the server and then obtain the WLD of important areas within it. For example, when acquiring map data, the client can first use the SWLD to obtain general map information, filter out areas with a high incidence of features such as buildings, signs, or people, and then obtain the WLD of these filtered areas. This allows the client to reduce the amount of data received from the server while still obtaining detailed information about the desired area.

[0387] Alternatively, the server can create separate SWLDs for each object based on the WLD, and the client can receive them separately based on their intended use. This can reduce network bandwidth. For example, the server can pre-identify people and cars from the WLD and create SWLDs for people and cars. If the client wants to obtain information about people around it, it will receive the SWLDs for people; if it wants to obtain information about cars, it will receive the SWLDs for cars. Furthermore, the type of SWLD can be distinguished based on information attached to the header (such as a flag or type).

[0388] Next, the configuration and operation flow of the three-dimensional data encoding device (for example, a server) according to this embodiment will be described. Figure 16 This is a block diagram of a three-dimensional data encoding device 400 according to this embodiment. Figure 17 This is a flowchart of the three-dimensional data encoding process performed by the three-dimensional data encoding device 400.

[0389] Figure 16 The illustrated three-dimensional data encoding device 400 encodes input three-dimensional data 411 to generate encoded three-dimensional data 413 and 414 as encoded streams. Encoded three-dimensional data 413 corresponds to the WLD, and encoded three-dimensional data 414 corresponds to the SWLD. The three-dimensional data encoding device 400 includes an acquisition unit 401, a coding region determination unit 402, an SWLD extraction unit 403, a WLD encoding unit 404, and a SWLD encoding unit 405.

[0390] like Figure 17 As shown, first, the acquisition unit 401 acquires input three-dimensional data 411 which is point cloud data in a three-dimensional space ( S401 ).

[0391] Next, the coding region determination unit 402 determines a spatial region to be coded based on the spatial region where the point cloud data exists ( S402 ).

[0392] Next, the SWLD extraction unit 403 defines the spatial region to be encoded as a WLD and calculates features from each VXL contained in the WLD. Furthermore, the SWLD extraction unit 403 extracts VXLs whose features exceed a predetermined threshold, defines these VXLs as FVXLs, and appends these FVXLs to the SWLD to generate extracted three-dimensional data 412 (S403). Specifically, extracted three-dimensional data 412 with features exceeding the threshold is extracted from the input three-dimensional data 411.

[0393] Next, the WLD encoder 404 encodes the input 3D data 411 corresponding to the WLD to generate encoded 3D data 413 corresponding to the WLD (S404). At this time, the WLD encoder 404 appends information to the header of the encoded 3D data 413 that identifies the encoded 3D data 413 as a stream containing the WLD.

[0394] Then, the SWLD encoding unit 405 encodes the extracted 3D data 412 corresponding to the SWLD to generate encoded 3D data 414 corresponding to the SWLD (S405). At this time, the SWLD encoding unit 405 adds information to the header of the encoded 3D data 414 to distinguish that the encoded 3D data 414 is a stream containing the SWLD.

[0395] Furthermore, the order of the process of generating the encoded three-dimensional data 413 and the process of generating the encoded three-dimensional data 414 may be reversed. Furthermore, part or all of the above processes may be executed in parallel.

[0396] The information assigned to the header of the encoded three-dimensional data 413 and 414 is defined as a parameter such as "world_type". In the case of world_type = 0, it indicates that the stream contains WLD, and in the case of world_type = 1, it indicates that the stream contains SWLD. When more categories are defined, the assigned value can be increased, such as world_type = 2. In addition, a specific flag can be included in one of the encoded three-dimensional data 413 and 414. For example, the encoded three-dimensional data 414 can be assigned a flag indicating that the stream contains SWLD. In this case, the decoding device can determine whether it is a stream containing WLD or a stream containing SWLD based on the presence or absence of the flag.

[0397] Furthermore, the encoding method used by the WLD encoding unit 404 when encoding the WLD may be different from the encoding method used by the SWLD encoding unit 405 when encoding the SWLD.

[0398] For example, since data is decimated in SWLD, the correlation with surrounding data may be lower than that in WLD. Therefore, in the encoding method for SWLD, inter-frame prediction is prioritized over intra-frame prediction and inter-frame prediction in comparison with the encoding method for WLD.

[0399] Furthermore, the encoding method used for SWLD and the encoding method used for WLD may differ in the representation of the three-dimensional position. For example, the three-dimensional position of FVXL may be represented by three-dimensional coordinates in FWLD, while the three-dimensional position may be represented by an octree (described later) in WLD, or vice versa.

[0400] Furthermore, the SWLD encoding unit 405 encodes the SWLD encoded three-dimensional data 414 so that the data size is smaller than the WLD encoded three-dimensional data 413. For example, as described above, the correlation between data in SWLD and WLD may be lower. This may reduce the encoding efficiency, and the data size of the encoded three-dimensional data 414 may be larger than the data size of the WLD encoded three-dimensional data 413. Therefore, if the data size of the obtained encoded three-dimensional data 414 is larger than the data size of the WLD encoded three-dimensional data 413, the SWLD encoding unit 405 re-encodes the data to generate the encoded three-dimensional data 414 with a reduced data size.

[0401] For example, the SWLD extraction unit 403 generates extracted three-dimensional data 412 again with a reduced number of extracted feature points, and the SWLD encoding unit 405 encodes the extracted three-dimensional data 412. Alternatively, the degree of quantization in the SWLD encoding unit 405 may be coarsened. For example, in the octree structure described later, the degree of quantization can be coarsened by rounding the data in the lowest layer.

[0402] Furthermore, if the data size of the SWLD encoded three-dimensional data 414 cannot be made smaller than the data size of the WLD encoded three-dimensional data 413, the SWLD encoding unit 405 may not generate the SWLD encoded three-dimensional data 414. Alternatively, the WLD encoded three-dimensional data 413 may be copied to the SWLD encoded three-dimensional data 414. In other words, the WLD encoded three-dimensional data 413 may be used as the SWLD encoded three-dimensional data 414.

[0403] Next, the configuration and operation flow of the three-dimensional data decoding device (eg, client) according to this embodiment will be described. Figure 18 This is a block diagram of a three-dimensional data decoding device 500 according to this embodiment. Figure 19 3D data decoding processing performed by the 3D data decoding apparatus 500 is shown in FIG.

[0404] Figure 18 The illustrated three-dimensional data decoding apparatus 500 generates decoded three-dimensional data 512 or 513 by decoding encoded three-dimensional data 511. Here, the encoded three-dimensional data 511 is, for example, the encoded three-dimensional data 413 or 414 generated by the three-dimensional data encoding apparatus 400.

[0405] The three-dimensional data decoding device 500 includes an acquisition unit 501 , a header analysis unit 502 , a WLD decoding unit 503 , and a SWLD decoding unit 504 .

[0406] like Figure 19 As shown, first, the acquisition unit 501 obtains the encoded 3D data 511 (S501). Next, the header analysis unit 502 analyzes the header of the encoded 3D data 511 to determine whether the encoded 3D data 511 is a stream containing WLD or a stream containing SWLD (S502). For example, this determination is made by referring to the world_type parameter described above.

[0407] If the encoded 3D data 511 is a stream containing a WLD ("Yes" in S503), the WLD decoding unit 503 decodes the encoded 3D data 511 to generate decoded 3D WLD data 512 (S504). If the encoded 3D data 511 is a stream containing a SWLD ("No" in S503), the SWLD decoding unit 504 decodes the encoded 3D data 511 to generate decoded 3D SWLD data 513 (S505).

[0408] Furthermore, similar to the encoding device, the decoding method used by the WLD decoding unit 503 when decoding the WLD may be different from the decoding method used by the SWLD decoding unit 504 when decoding the SWLD. For example, in the decoding method for the SWLD, priority may be given to the inter-frame prediction of intra-frame prediction and inter-frame prediction over the decoding method for the WLD.

[0409] Furthermore, the three-dimensional position representation method can be different between the decoding method used for SWLD and the decoding method used for WLD. For example, the three-dimensional position of FVXL can be represented by three-dimensional coordinates in SWLD, while the three-dimensional position can be represented by an octree (described later) in WLD, or vice versa.

[0410] Next, octree representation as a method of representing three-dimensional positions will be described. VXL data included in three-dimensional data is converted into an octree structure and then encoded. Figure 20 An example of VXL of WLD is shown. Figure 21 Shown Figure 20 The octree structure of WLD is shown in Figure 2. Figure 20 In the example shown, there are three VXLs 1 to 3 that are VXLs (hereinafter referred to as valid VXLs) that include a point group. Figure 21 As shown, the octree structure consists of nodes and leaf nodes. Each node has a maximum of 8 nodes or leaf nodes. Each leaf node has VXL information. Here, Figure 21 Among the leaf nodes shown, leaf nodes 1, 2, and 3 represent Figure 20 VXL1, VXL2, and VXL3 shown.

[0411] Specifically, each node and leaf node corresponds to a three-dimensional position. Figure 20 The block corresponding to node 1 is divided into 8 blocks. Among the 8 blocks, the blocks containing valid VXLs are set as nodes, and the remaining blocks are set as leaf nodes. The block corresponding to the node is further divided into 8 nodes or leaf nodes. This process is repeated as many times as the number of levels in the tree structure. Finally, all the blocks at the bottom level are set as leaf nodes.

[0412] and, Figure 22 Shown from Figure 20 An example of SWLD generated by WLD is shown. Figure 20 The feature extraction results of VXL1 and VXL2 shown are determined to be FVXL1 and FVXL2 and are included in SWLD. In addition, VXL3 is not determined to be FVXL and is therefore not included in SWLD. Figure 23 Shown Figure 22 The octree structure of SWLD is shown in Figure 1. Figure 23 In the octree structure shown, Figure 21 The leaf node 3 shown, which is equivalent to VXL3, is deleted. Figure 21 Node 3 shown does not have a valid VXL and is changed to a leaf node. Generally speaking, the number of leaf nodes of SWLD is smaller than that of WLD, and the coded three-dimensional data of SWLD is also smaller than that of WLD.

[0413] Modifications of this embodiment will be described below.

[0414] For example, when a client such as a vehicle-mounted device estimates its own position, it receives SWLD from the server, uses SWLD to estimate its own position, and performs obstacle detection. It then uses various methods such as distance sensors such as rangefinders, stereo cameras, or a combination of multiple monocular cameras to perform obstacle detection based on the three-dimensional information of the surrounding area obtained by itself.

[0415] Furthermore, it's generally difficult to include VXL data for flat areas in SWLD. Therefore, the server maintains a subsampled world space (SubWLD) that downsamples the WLD for stationary obstacle detection and can send both the SWLD and SubWLD to the client. This reduces network bandwidth while enabling client-side position estimation and obstacle detection.

[0416] Furthermore, when clients need to quickly render 3D map data, a grid-like structure is often more convenient. Therefore, the server can generate a grid based on the WLD and store it in advance as a grid world space (MWLD). For example, if the client requires a coarse 3D rendering, it receives the MWLD; if it requires a detailed 3D rendering, it receives the WLD. This helps reduce network bandwidth.

[0417] Furthermore, while the server sets the VXL with a feature value above a threshold value from each VXL as the FVXL, the FVXL can also be calculated using different methods. For example, if the server determines that the VXL, VLM, SPC, or GOS that constitute a signal or intersection are necessary for self-position estimation, driving assistance, or autonomous driving, they can be included in the SWLD as FVXL, FVLM, FSPC, or FGOS. Furthermore, this determination can be made manually. In addition, the FVXL, etc. obtained by the above method can be added to the FVXL, etc. set based on the feature value. That is, the SWLD extraction unit 403 can further extract data corresponding to objects with predetermined attributes from the input three-dimensional data 411 as extracted three-dimensional data 412.

[0418] Furthermore, labels different from feature values ​​can be assigned to situations required for these purposes. The server can separately maintain FVXL, which is required for estimating its own position, such as signals or intersections, driving assistance, or autonomous driving, as a higher layer of SWLD (e.g., lane world space).

[0419] Furthermore, the server can also add attributes to the VXL within the WLD in random access units or specified units. Attributes include, for example, information indicating whether a location is required or not for estimating the location, or information indicating whether traffic information such as signals or intersections is important. Attributes can also include the correspondence between lane information (GDF: Geographic Data Files, etc.) and features (such as intersections or roads).

[0420] Furthermore, as a method of updating WLD or SWLD, the following method can be adopted.

[0421] Update information showing changes in people, construction, or street trees (trajectory orientation) is uploaded to the server as a point group or metadata. The server updates the WLD based on this upload, and then updates the SWLD using the updated WLD.

[0422] Furthermore, if the client detects a mismatch between the 3D information generated by itself when estimating its own position and the 3D information received from the server, it can send the generated 3D information to the server along with an update notification. In this case, the server uses the WLD to update the SWLD. If the SWLD is not updated, the server determines that the WLD itself is outdated.

[0423] Furthermore, while information distinguishing between WLD and SWLD is added to the coded stream header, if there are multiple world spaces, such as a grid world space or a lane world space, information distinguishing between them can be added to the header. Furthermore, if there are multiple SWLDs with different feature values, information distinguishing between them can also be added to the header.

[0424] Furthermore, while the SWLD consists of FVXLs, it can also include VXLs that are not identified as FVXLs. For example, the SWLD can include adjacent VXLs used to calculate the characteristics of the FVXLs. This allows the client to calculate the characteristics of the FVXLs upon receiving the SWLD, even if the FVXLs in the SWLD do not have feature information attached. Furthermore, the SWLD can include information to distinguish whether each VXL is an FVXL or a VXL.

[0425] As described above, the three-dimensional data encoding device 400 extracts extracted three-dimensional data 412 (second three-dimensional data) whose feature value is above a threshold value from the input three-dimensional data 411 (first three-dimensional data), and generates encoded three-dimensional data 414 (first encoded three-dimensional data) by encoding the extracted three-dimensional data 412.

[0426] Based on this, the three-dimensional data encoding device 400 generates encoded three-dimensional data 414 by encoding data with a feature value greater than or equal to a threshold value. This reduces the amount of data compared to directly encoding the input three-dimensional data 411. Therefore, the three-dimensional data encoding device 400 can reduce the amount of data required for transmission.

[0427] Furthermore, the three-dimensional data encoding device 400 further encodes the input three-dimensional data 411 to generate encoded three-dimensional data 413 (second encoded three-dimensional data).

[0428] With this, the three-dimensional data encoding device 400 can selectively transmit the encoded three-dimensional data 413 and the encoded three-dimensional data 414 according to the intended use, for example.

[0429] Furthermore, the extracted three-dimensional data 412 is encoded using a first encoding method, and the input three-dimensional data 411 is encoded using a second encoding method that is different from the first encoding method.

[0430] Accordingly, the three-dimensional data encoding device 400 can adopt appropriate encoding methods for the input three-dimensional data 411 and the extracted three-dimensional data 412 respectively.

[0431] Furthermore, in the first encoding method, compared with the second encoding method, priority is given to inter-frame prediction among intra-frame prediction and inter-frame prediction.

[0432] According to this, the three-dimensional data encoding apparatus 400 can increase the priority of inter-frame prediction for the extracted three-dimensional data 412 where the correlation between adjacent data tends to be low.

[0433] Furthermore, the first encoding method and the second encoding method use different methods to represent three-dimensional positions. For example, the second encoding method represents three-dimensional positions using an octree, while the first encoding method represents three-dimensional positions using three-dimensional coordinates.

[0434] With this, the three-dimensional data encoding device 400 can adopt a more appropriate three-dimensional position expression method for three-dimensional data having different data amounts (number of VXLs or FVXLs).

[0435] Furthermore, at least one of the coded 3D data 413 and 414 includes an identifier indicating whether the coded 3D data is obtained by encoding the input 3D data 411 or by encoding a portion of the input 3D data 411. In other words, the identifier indicates whether the coded 3D data is the coded 3D data 413 of the WLD or the coded 3D data 414 of the SWLD.

[0436] Based on this, the decoding device can easily determine whether the acquired encoded three-dimensional data is the encoded three-dimensional data 413 or the encoded three-dimensional data 414.

[0437] Furthermore, the three-dimensional data encoding device 400 encodes the extracted three-dimensional data 412 so that the data amount of the encoded three-dimensional data 414 is smaller than the data amount of the encoded three-dimensional data 413 .

[0438] As a result, the three-dimensional data encoding device 400 can make the data amount of the encoded three-dimensional data 414 smaller than the data amount of the encoded three-dimensional data 413 .

[0439] Furthermore, the three-dimensional data encoding device 400 extracts data corresponding to objects having predetermined attributes from the input three-dimensional data 411 as extracted three-dimensional data 412. For example, the objects having predetermined attributes may be objects required for self-position estimation, driving assistance, or autonomous driving, such as signals or intersections.

[0440] In this way, the three-dimensional data encoding device 400 can generate the encoded three-dimensional data 414 including the data required by the decoding device.

[0441] Furthermore, the three-dimensional data encoding device 400 (server) further transmits one of the encoded three-dimensional data 413 and 414 to the client according to the status of the client.

[0442] Thus, the three-dimensional data encoding device 400 can send appropriate data according to the status of the client.

[0443] Furthermore, the client status includes the client's communication status (eg, network bandwidth) or the client's moving speed.

[0444] Furthermore, the three-dimensional data encoding device 400 further transmits one of the encoded three-dimensional data 413 and 414 to the client according to the client's request.

[0445] Thus, the three-dimensional data encoding device 400 can send appropriate data according to the client's request.

[0446] Furthermore, the three-dimensional data decoding device 500 according to this embodiment decodes the encoded three-dimensional data 413 or 414 generated by the three-dimensional data encoding device 400 .

[0447] Specifically, the three-dimensional data decoding apparatus 500 decodes the encoded three-dimensional data 414 obtained by encoding the extracted three-dimensional data 412 whose feature quantity extracted from the input three-dimensional data 411 is greater than or equal to a threshold value using a first decoding method. Furthermore, the three-dimensional data decoding apparatus 500 decodes the encoded three-dimensional data 413 obtained by encoding the input three-dimensional data 411 using a second decoding method that is different from the first decoding method.

[0448] Thus, the 3D data decoding device 500 can selectively receive the encoded 3D data 414 and 413, which are obtained by encoding data with feature quantities exceeding a threshold value, according to, for example, the intended use. This allows the 3D data decoding device 500 to reduce the amount of data transmitted. Furthermore, the 3D data decoding device 500 can employ appropriate decoding methods for each of the input 3D data 411 and the extracted 3D data 412.

[0449] Furthermore, in the first decoding method, compared with the second decoding method, priority is given to inter prediction between intra prediction and inter prediction.

[0450] With this, the three-dimensional data decoding apparatus 500 can increase the priority of inter-frame prediction for extracting three-dimensional data in which the correlation between adjacent data tends to be low.

[0451] Furthermore, the first decoding method and the second decoding method use different methods to represent the three-dimensional position. For example, the second decoding method represents the three-dimensional position using an octree, while the first decoding method represents the three-dimensional position using three-dimensional coordinates.

[0452] Thus, the three-dimensional data decoding apparatus 500 can adopt a more appropriate three-dimensional position expression method for three-dimensional data having different data amounts (number of VXLs or FVXLs).

[0453] Furthermore, at least one of the encoded 3D data 413 and 414 includes an identifier indicating whether the encoded 3D data is obtained by encoding the input 3D data 411 or by encoding a portion of the input 3D data 411. The 3D data decoding apparatus 500 identifies the encoded 3D data 413 and 414 by referring to the identifier.

[0454] Based on this, the three-dimensional data decoding apparatus 500 can easily determine whether the obtained encoded three-dimensional data is the encoded three-dimensional data 413 or the encoded three-dimensional data 414 .

[0455] Furthermore, the 3D data decoding device 500 notifies the server of the status of the client (the 3D data decoding device 500 ) and receives one of the encoded 3D data 413 and 414 transmitted from the server according to the status of the client.

[0456] Thus, the three-dimensional data decoding apparatus 500 can receive appropriate data according to the status of the client.

[0457] Furthermore, the client status includes the client's communication status (eg, network bandwidth) or the client's moving speed.

[0458] Furthermore, the three-dimensional data decoding apparatus 500 further requests one of the encoded three-dimensional data 413 and 414 from the server, and receives one of the encoded three-dimensional data 413 and 414 transmitted from the server in accordance with the request.

[0459] This allows the three-dimensional data decoding apparatus 500 to receive appropriate data corresponding to the intended use.

[0460] (Implementation 3)

[0461] In this embodiment, a method for transmitting and receiving three-dimensional data between vehicles will be described. For example, three-dimensional data is transmitted and received between a vehicle and surrounding vehicles.

[0462] Figure 24 This is a block diagram of a three-dimensional data creation device 620 according to this embodiment. This three-dimensional data creation device 620 is included in, for example, a vehicle and creates denser third three-dimensional data 636 by combining received second three-dimensional data 635 with first three-dimensional data 632 created by the three-dimensional data creation device 620.

[0463] The three-dimensional data creation device 620 includes a three-dimensional data creation unit 621 , a request range determination unit 622 , a search unit 623 , a reception unit 624 , a decoding unit 625 , and a synthesis unit 626 .

[0464] First, the three-dimensional data creation unit 621 uses sensor information 631 detected by sensors in the vehicle to create first three-dimensional data 632. Next, the request range determination unit 622 determines a request range, which is a three-dimensional space range where the data in the created first three-dimensional data 632 is insufficient.

[0465] Next, the search unit 623 searches for surrounding vehicles that hold three-dimensional data for the requested range and sends request range information 633 indicating the requested range to the surrounding vehicles identified by the search. Next, the receiving unit 624 receives the encoded three-dimensional data 634 as a coded stream for the requested range from the surrounding vehicles (S624). Alternatively, the search unit 623 can issue a request indiscriminately to all vehicles within the identified range and receive the encoded three-dimensional data 634 from any responding vehicles. Furthermore, the search unit 623 is not limited to vehicles and can also issue a request to objects such as traffic lights or signs and receive the encoded three-dimensional data 634 from those objects.

[0466] Next, the decoder 625 decodes the received encoded three-dimensional data 634 to obtain second three-dimensional data 635. Next, the synthesizer 626 synthesizes the first three-dimensional data 632 and the second three-dimensional data 635 to create denser third three-dimensional data 636.

[0467] Next, the configuration and operation of the three-dimensional data transmitting device 640 according to this embodiment will be described. Figure 25 It is a block diagram of the three-dimensional data transmitting device 640.

[0468] The three-dimensional data sending device 640 is included in the above-mentioned surrounding vehicles, for example, and processes the fifth three-dimensional data 652 produced by the surrounding vehicles into the sixth three-dimensional data 654 requested by the own vehicle, and generates encoded three-dimensional data 634 by encoding the sixth three-dimensional data 654, and sends the encoded three-dimensional data 634 to the own vehicle.

[0469] The three-dimensional data transmitting device 640 includes a three-dimensional data creating unit 641 , a receiving unit 642 , an extracting unit 643 , an encoding unit 644 , and a transmitting unit 645 .

[0470] First, the three-dimensional data creation unit 641 uses sensor information 651 detected by sensors included in surrounding vehicles to create fifth three-dimensional data 652. Next, the receiving unit 642 receives the requested range information 633 transmitted from the own vehicle.

[0471] Next, the extraction unit 643 extracts the three-dimensional data within the requested range indicated by the requested range information 633 from the fifth three-dimensional data 652, and processes the fifth three-dimensional data 652 into sixth three-dimensional data 654. The encoding unit 644 then encodes the sixth three-dimensional data 654, generating encoded three-dimensional data 634 as an encoded stream. The transmission unit 645 then transmits the encoded three-dimensional data 634 to the vehicle itself.

[0472] In addition, although the example in which the own vehicle has the three-dimensional data creation device 620 and the surrounding vehicles have the three-dimensional data transmission device 640 is described here, each vehicle may also have the functions of the three-dimensional data creation device 620 and the three-dimensional data transmission device 640.

[0473] (Implementation 4)

[0474] In this embodiment, an operation related to abnormal situations in self-position estimation based on a three-dimensional map will be described.

[0475] Applications such as autonomous driving of vehicles and autonomous movement of mobile objects such as robots and drones are expected to expand in the future. One example of a method for achieving such autonomous movement is a method in which a mobile object estimates its own position within a three-dimensional map (self-position estimation) and drives according to the map.

[0476] Self-position estimation is achieved by matching the three-dimensional map with the three-dimensional information around the own vehicle obtained by sensors such as the rangefinder (LIDAR, etc.) or stereo camera installed on the own vehicle (hereinafter referred to as the own vehicle detection three-dimensional data), and estimating the own vehicle position within the three-dimensional map.

[0477] 3D maps, such as those offered by HERE's HD maps, are not just 3D point clouds; they also include 2D map data such as road and intersection shape information, as well as real-time information such as traffic jams and accidents. 3D maps are constructed from multiple layers of data, including 3D and 2D data and metadata that changes over time. Devices can retrieve only the data they need, or they can reference only the data they need.

[0478] The point cloud data may be the aforementioned SWLD, or may include point group data that is not a feature point. Furthermore, the transmission and reception of the point cloud data is basically performed in one or more random access units.

[0479] The following methods can be used to match a 3D map with the 3D data detected by the own vehicle. For example, the device compares the shapes of the point clusters in the respective point clouds and determines areas with high similarity between feature points as being at the same location. Furthermore, if the 3D map is constructed using SWLD, the device compares and matches the feature points that make up the SWLD with the 3D feature points extracted from the 3D data detected by the own vehicle.

[0480] To accurately estimate the vehicle's position, the following conditions (A) and (B) must be met: (A) a 3D map and 3D vehicle detection data must be available, and (B) their accuracy must meet a predetermined standard. However, in the following exceptional circumstances, either (A) or (B) may not be met.

[0481] (1) A three-dimensional map cannot be obtained through the communication path.

[0482] (2) There is no three-dimensional map, or the obtained three-dimensional map is damaged.

[0483] (3) The sensor of the own vehicle fails, or due to bad weather, the accuracy of the three-dimensional data generated by the own vehicle is insufficient.

[0484] The following describes the operations for dealing with these abnormal situations. Although the following operation is described using a vehicle as an example, the following method can also be applied to all moving objects that move autonomously, such as robots and drones.

[0485] The following describes the configuration and operation of the three-dimensional information processing device according to this embodiment for detecting abnormalities in three-dimensional data corresponding to a three-dimensional map or a vehicle. Figure 26 This is a block diagram showing a configuration example of a three-dimensional information processing device 700 according to this embodiment.

[0486] The three-dimensional information processing device 700 is mounted on a mobile object such as a motor vehicle. Figure 26 As shown, the three-dimensional information processing device 700 includes a three-dimensional map acquisition unit 701 , a vehicle detection data acquisition unit 702 , an abnormality determination unit 703 , a countermeasure operation determination unit 704 , and an operation control unit 705 .

[0487] The 3D information processing device 700 may also include a camera that captures 2D images, or a sensor (not shown) that uses ultrasonic or laser-based 1D data to detect structures or moving objects around the vehicle. Furthermore, the 3D information processing device 700 may include a communication unit (not shown) that generates a 3D map via a mobile communication network such as 4G or 5G, or between vehicles or between roads and vehicles.

[0488] The three-dimensional map acquisition unit 701 acquires a three-dimensional map 711 near the driving route. For example, the three-dimensional map acquisition unit 701 acquires the three-dimensional map 711 through a mobile communication network, vehicle-to-vehicle communication, or road-to-vehicle communication.

[0489] Next, the vehicle detection data acquisition unit 702 acquires the vehicle detection three-dimensional data 712 based on the sensor information. For example, the vehicle detection data acquisition unit 702 generates the vehicle detection three-dimensional data 712 based on the sensor information acquired by the sensors included in the vehicle.

[0490] Next, abnormality determination unit 703 detects abnormalities by performing predetermined checks on at least one of the obtained three-dimensional map 711 and the vehicle's own three-dimensional detection data 712. Specifically, abnormality determination unit 703 determines whether at least one of the obtained three-dimensional map 711 and the vehicle's own three-dimensional detection data 712 is abnormal.

[0491] When an abnormality is detected, the countermeasure decision unit 704 decides a countermeasure for the abnormality. Next, the task control unit 705 controls the operations of various processing units required for executing the countermeasure, such as the three-dimensional map acquisition unit 701.

[0492] If no abnormality is detected, the three-dimensional information processing apparatus 700 ends the processing.

[0493] The 3D information processing device 700 estimates the vehicle's own position using the 3D map 711 and the own vehicle detection 3D data 712. The 3D information processing device 700 then uses the result of the own position estimation to enable the vehicle to autonomously drive.

[0494] Based on this, the three-dimensional information processing device 700 obtains map data (three-dimensional map 711) including the first three-dimensional position information via the channel. For example, the first three-dimensional position information is encoded in units of subspaces having three-dimensional coordinate information. The first three-dimensional position information includes multiple random access units, each of which is a collection of one or more subspaces and can be independently decoded. For example, the first three-dimensional position information is data (SWLD) encoded at feature points where the three-dimensional feature quantity exceeds a predetermined threshold.

[0495] The three-dimensional information processing device 700 generates second three-dimensional position information (vehicle detection three-dimensional data 712) based on the information detected by the sensor. The three-dimensional information processing device 700 then performs abnormality determination processing on the first three-dimensional position information or the second three-dimensional position information to determine whether the first three-dimensional position information or the second three-dimensional position information is abnormal.

[0496] When the three-dimensional information processing device 700 determines that the first three-dimensional position information or the second three-dimensional position information is abnormal, it determines a countermeasure for the abnormality and then executes control required for executing the countermeasure.

[0497] With this, the three-dimensional information processing device 700 can detect abnormality in the first three-dimensional position information or the second three-dimensional position information and perform a corresponding operation.

[0498] (Implementation 5)

[0499] In this embodiment, a method of transmitting three-dimensional data to a following vehicle will be described.

[0500] Figure 27 This is a block diagram illustrating an example configuration of a three-dimensional data creation device 810 according to this embodiment. This three-dimensional data creation device 810 is, for example, mounted on a vehicle. It transmits and receives three-dimensional data with external traffic cloud monitoring, preceding vehicles, and following vehicles, while also creating and storing the three-dimensional data.

[0501] The three-dimensional data production device 810 includes: a data receiving unit 811, a communication unit 812, a receiving control unit 813, a format conversion unit 814, multiple sensors 815, a three-dimensional data production unit 816, a three-dimensional data synthesis unit 817, a three-dimensional data storage unit 818, a communication unit 819, a sending control unit 820, a format conversion unit 821, and a data sending unit 822.

[0502] The data receiving unit 811 receives 3D data 831 from traffic cloud monitoring or the preceding vehicle. The 3D data 831 includes, for example, a point cloud containing information about areas that the vehicle's sensors 815 cannot detect, visible light images, depth information, sensor position information, or speed information.

[0503] The communication unit 812 communicates with the traffic cloud monitoring or the vehicle ahead, and sends a data transmission request or the like to the traffic cloud monitoring or the vehicle ahead.

[0504] The reception control unit 813 exchanges information such as a corresponding format with the communication partner via the communication unit 812 to establish communication with the communication partner.

[0505] The format conversion unit 814 generates three-dimensional data 832 by performing format conversion on the three-dimensional data 831 received by the data reception unit 811. Furthermore, if the three-dimensional data 831 is compressed or encoded, the format conversion unit 814 performs decompression or decoding processing.

[0506] Multiple sensors 815 are a group of sensors such as LiDAR, visible light cameras, or infrared cameras that obtain information about the exterior of the vehicle and generate sensor information 833. For example, if sensor 815 is a laser sensor such as LiDAR, sensor information 833 is three-dimensional data such as a point cloud (point cloud data). Furthermore, the number of sensors 815 does not need to be multiple.

[0507] The three-dimensional data generating unit 816 generates three-dimensional data 834 based on the sensor information 833. The three-dimensional data 834 includes, for example, point clouds, visible light images, depth information, sensor position information, or speed information.

[0508] The three-dimensional data synthesis unit 817 synthesizes the three-dimensional data 832 produced by traffic cloud monitoring or the vehicle in front, etc., into the three-dimensional data 834 produced based on the sensor information 833 of the own vehicle, thereby constructing three-dimensional data 835 that also includes the space in front of the vehicle in front that cannot be detected by the sensor 815 of the own vehicle.

[0509] The three-dimensional data accumulation unit 818 accumulates the generated three-dimensional data 835 and the like.

[0510] The communication unit 819 communicates with the traffic cloud monitoring system or the vehicle behind, and sends a data transmission request or the like to the traffic cloud monitoring system or the vehicle behind.

[0511] The transmission control unit 820 exchanges information such as the corresponding format with the communication partner via the communication unit 819 to establish communication with the communication partner. Furthermore, the transmission control unit 820 determines the spatial transmission area of ​​the three-dimensional data to be transmitted based on the three-dimensional data construction information of the three-dimensional data 832 generated by the three-dimensional data synthesis unit 817 and the data transmission request from the communication partner.

[0512] Specifically, based on data transmission requests from traffic cloud monitoring or following vehicles, the transmission control unit 820 determines a transmission area, including the space in front of the vehicle itself, which is not detectable by the following vehicle's sensors. Furthermore, the transmission control unit 820 determines the transmission area by determining, based on the three-dimensional data construction information, whether updates to the available transmission space or the already transmitted space have occurred. For example, the transmission control unit 820 determines as the transmission area an area that is both specified in the data transmission request and contains the corresponding three-dimensional data 835. Furthermore, the transmission control unit 820 notifies the format conversion unit 821 of the communication partner's supported format and the transmission area.

[0513] Format conversion unit 821 converts three-dimensional data 836 in the transmission area of ​​three-dimensional data 835 stored in three-dimensional data storage unit 818 into a format compatible with the receiving end, thereby generating three-dimensional data 837. Format conversion unit 821 may also compress or encode three-dimensional data 837 to reduce the data volume.

[0514] The data transmission unit 822 transmits the three-dimensional data 837 to the traffic cloud monitoring or the following vehicle. The three-dimensional data 837 includes, for example, a point cloud in front of the vehicle including information on areas that are blind spots for the following vehicle, visible light images, depth information, or sensor position information.

[0515] In addition, although the format conversion units 814 and 821 perform format conversion, the format conversion may not be performed.

[0516] With this configuration, three-dimensional data creation device 810 obtains three-dimensional data 831 from an external source for an area that is undetectable by the vehicle's sensors 815, and generates three-dimensional data 835 by combining three-dimensional data 831 with three-dimensional data 834 based on sensor information 833 detected by the vehicle's sensors 815. This allows three-dimensional data creation device 810 to generate three-dimensional data for an area that is undetectable by the vehicle's sensors 815.

[0517] In addition, the three-dimensional data production device 810 can send three-dimensional data of the space in front of its own vehicle that cannot be detected by the sensors of the rear vehicle to the traffic cloud monitoring or the rear vehicle, etc. according to the data sending request from the traffic cloud monitoring or the rear vehicle.

[0518] (Implementation 6)

[0519] In the fifth embodiment, a client device such as a vehicle transmits three-dimensional data to another vehicle or a server such as a traffic cloud monitoring system. In this embodiment, the client device transmits sensor information obtained by a sensor to a server or another client device.

[0520] First, the configuration of the system according to this embodiment will be described. Figure 28 The configuration of a system for transmitting and receiving three-dimensional maps and sensor information according to this embodiment is shown. The system includes a server 901 and client devices 902A and 902B. In addition, when client devices 902A and 902B are not specifically distinguished, they are also referred to as client device 902.

[0521] The client device 902 is, for example, an in-vehicle device mounted on a mobile object such as a vehicle. The server 901 is, for example, a traffic cloud monitoring system, and is capable of communicating with a plurality of client devices 902 .

[0522] The server 901 transmits a three-dimensional map composed of point clouds to the client device 902. In addition, the composition of the three-dimensional map is not limited to point clouds, and can also be expressed by other three-dimensional data such as a mesh structure.

[0523] The client device 902 sends sensor information obtained by the client device 902 to the server 901. The sensor information includes, for example, at least one of LiDAR information, visible light images, infrared images, depth images, sensor position information, and speed information.

[0524] The data sent and received between server 901 and client device 902 can be compressed to reduce data size, or not compressed to maintain data accuracy. When data compression is used, for example, a three-dimensional compression method based on an octree can be used for point clouds. Furthermore, two-dimensional image compression methods can be used for visible light images, infrared images, and depth images. Examples of two-dimensional image compression methods include MPEG-4 AVC and HEVC standardized by MPEG.

[0525] Furthermore, in response to a request from client device 902 to transmit the three-dimensional map, server 901 transmits the three-dimensional map managed by server 901 to client device 902. Alternatively, server 901 may transmit the three-dimensional map without waiting for a request from client device 902 to transmit the three-dimensional map. For example, server 901 may broadcast the three-dimensional map to one or more client devices 902 in a predetermined space. Furthermore, server 901 may transmit a three-dimensional map appropriate for the location of client device 902 at regular intervals to a client device 902 that has received a transmission request. Furthermore, server 901 may transmit the three-dimensional map to client device 902 whenever the three-dimensional map managed by server 901 is updated.

[0526] The client device 902 sends a three-dimensional map transmission request to the server 901. For example, when the client device 902 wants to estimate its own position while driving, the client device 902 sends the three-dimensional map transmission request to the server 901.

[0527] In addition, the client device 902 may also issue a request to transmit a three-dimensional map to the server 901 in the following situations. If the three-dimensional map held by the client device 902 is relatively old, the client device 902 may also issue a request to transmit the three-dimensional map to the server 901. For example, if a certain period of time has passed since the client device 902 acquired the three-dimensional map, the client device 902 may also issue a request to transmit the three-dimensional map to the server 901.

[0528] Alternatively, client device 902 may issue a three-dimensional map transmission request to server 901 a certain time before client device 902 is about to exit the space shown on the three-dimensional map held by client device 902. For example, client device 902 may issue a three-dimensional map transmission request to server 901 when client device 902 is within a predetermined distance from the boundary of the space shown on the three-dimensional map held by client device 902. Furthermore, if the movement path and speed of client device 902 are known, the time at which client device 902 will exit the space shown on the three-dimensional map held by client device 902 can be predicted based on the known movement path and speed.

[0529] When the error between the three-dimensional data generated by the client device 902 based on the sensor information and the position of the three-dimensional map is greater than a certain range, the client device 902 may send a request to the server 901 to send the three-dimensional map.

[0530] Client device 902 transmits sensor information to server 901 in response to a sensor information transmission request received from server 901. Alternatively, client device 902 may transmit sensor information to server 901 without waiting for a sensor information transmission request from server 901. For example, after receiving a sensor information transmission request from server 901, client device 902 may periodically transmit sensor information to server 901 over a predetermined period. Furthermore, if the position error between three-dimensional data generated by client device 902 based on sensor information and a three-dimensional map received from server 901 is within a predetermined range, client device 902 may determine that the three-dimensional map surrounding client device 902 may have changed, and transmit this determination result along with the sensor information to server 901.

[0531] Server 901 issues a sensor information transmission request to client device 902. For example, server 901 receives location information of client device 902, such as GPS, from client device 902. Based on the location information of client device 902, if server 901 determines that client device 902 is approaching a space with less information on the three-dimensional map managed by server 901, server 901 issues a sensor information transmission request to client device 902 in order to regenerate the three-dimensional map. Alternatively, server 901 may issue a sensor information transmission request to update the three-dimensional map, to check road conditions such as during snow accumulation or disasters, or to check traffic conditions or incidents.

[0532] Furthermore, the client device 902 may set the data volume of the sensor information to be transmitted to the server 901 according to the communication state or bandwidth at the time of receiving the sensor information transmission request received from the server 901. Setting the data volume of the sensor information to be transmitted to the server 901 may include, for example, increasing or decreasing the data itself or selecting an appropriate compression method.

[0533] Figure 29 This is a block diagram illustrating an example configuration of client device 902. Client device 902 receives a three-dimensional map composed of a point cloud or the like from server 901 and estimates its own position based on the three-dimensional data generated based on sensor information from client device 902. Client device 902 then transmits the obtained sensor information to server 901.

[0534] The client device 902 includes: a data receiving unit 1011, a communication unit 1012, a receiving control unit 1013, a format conversion unit 1014, multiple sensors 1015, a three-dimensional data production unit 1016, a three-dimensional image processing unit 1017, a three-dimensional data storage unit 1018, a format conversion unit 1019, a communication unit 1020, a sending control unit 1021, and a data sending unit 1022.

[0535] The data receiving unit 1011 receives a three-dimensional map 1031 from the server 901. The three-dimensional map 1031 is data including a point cloud such as WLD or SWLD. The three-dimensional map 1031 may include either compressed data or uncompressed data.

[0536] The communication unit 1012 communicates with the server 901 and transmits a data transmission request (for example, a transmission request for a three-dimensional map) and the like to the server 901 .

[0537] The reception control unit 1013 exchanges information such as the corresponding format with the communication partner via the communication unit 1012 and establishes communication with the communication partner.

[0538] The format conversion unit 1014 generates a three-dimensional map 1032 by performing format conversion on the three-dimensional map 1031 received by the data receiving unit 1011. Furthermore, if the three-dimensional map 1031 is compressed or encoded, the format conversion unit 1014 performs decompression or decoding. If the three-dimensional map 1031 is uncompressed data, the format conversion unit 1014 does not perform decompression or decoding.

[0539] Multiple sensors 1015 are a group of sensors mounted on client device 902, such as LiDAR, visible light cameras, infrared cameras, or depth sensors, for obtaining information outside the vehicle and generating sensor information 1033. For example, if sensor 1015 is a laser sensor such as LiDAR, sensor information 1033 is three-dimensional data such as a point cloud (point cloud data). Furthermore, there may not be multiple sensors 1015.

[0540] The three-dimensional data generation unit 1016 generates three-dimensional data 1034 of the surrounding area of ​​the own vehicle based on the sensor information 1033. For example, the three-dimensional data generation unit 1016 uses information obtained by LiDAR and visible light images obtained by a visible light camera to generate point cloud data with color information of the surrounding area of ​​the own vehicle.

[0541] The three-dimensional image processing unit 1017 uses the received three-dimensional map 1032, such as a point cloud, and three-dimensional data 1034 of the vehicle's surroundings generated based on the sensor information 1033 to perform processing such as estimating the vehicle's own position. Alternatively, the three-dimensional image processing unit 1017 may synthesize the three-dimensional map 1032 and the three-dimensional data 1034 to create three-dimensional data 1035 of the vehicle's surroundings, and use the created three-dimensional data 1035 to perform processing for estimating the vehicle's own position.

[0542] The three-dimensional data storage unit 1018 stores the three-dimensional map 1032 , the three-dimensional data 1034 , and the three-dimensional data 1035 .

[0543] The format conversion unit 1019 converts the sensor information 1033 into a format compatible with the receiving end to generate sensor information 1037. Furthermore, the format conversion unit 1019 can reduce the data volume by compressing or encoding the sensor information 1037. Furthermore, if format conversion is not necessary, the format conversion unit 1019 can omit the processing. Furthermore, the format conversion unit 1019 can control the amount of data transmitted according to the specified transmission range.

[0544] The communication unit 1020 communicates with the server 901 and receives a data transmission request (a sensor information transmission request) and the like from the server 901 .

[0545] The transmission control unit 1021 exchanges information such as the corresponding format with the communication partner via the communication unit 1020 to establish communication.

[0546] The data transmission unit 1022 transmits the sensor information 1037 to the server 901. The sensor information 1037 includes information obtained by the plurality of sensors 1015, such as information obtained by LiDAR, a brightness image (visible light image) obtained by a visible light camera, an infrared image obtained by an infrared camera, a depth image obtained by a depth sensor, sensor position information, and speed information.

[0547] Next, the configuration of the server 901 will be described. Figure 30 This is a block diagram illustrating an example configuration of server 901. Server 901 receives sensor information transmitted from client device 902 and generates three-dimensional data based on the received sensor information. Server 901 uses the generated three-dimensional data to update the three-dimensional map managed by server 901. Furthermore, server 901 transmits the updated three-dimensional map to client device 902 in response to a request for transmission of the three-dimensional map from client device 902.

[0548] The server 901 includes: a data receiving unit 1111, a communication unit 1112, a receiving control unit 1113, a format conversion unit 1114, a three-dimensional data production unit 1116, a three-dimensional data synthesis unit 1117, a three-dimensional data storage unit 1118, a format conversion unit 1119, a communication unit 1120, a sending control unit 1121, and a data sending unit 1122.

[0549] The data receiving unit 1111 receives sensor information 1037 from the client device 902. The sensor information 1037 includes, for example, information obtained by LiDAR, a brightness image (visible light image) obtained by a visible light camera, an infrared image obtained by an infrared camera, a depth image obtained by a depth sensor, sensor position information, and speed information.

[0550] The communication unit 1112 communicates with the client device 902 and transmits a data transmission request (for example, a sensor information transmission request) and the like to the client device 902 .

[0551] The reception control unit 1113 exchanges information such as the corresponding format with the communication partner via the communication unit 1112 to establish communication.

[0552] If the received sensor information 1037 is compressed or encoded, the format conversion unit 1114 performs decompression or decoding processing to generate the sensor information 1132. If the sensor information 1037 is uncompressed data, the format conversion unit 1114 does not perform decompression or decoding processing.

[0553] The three-dimensional data generation unit 1116 generates three-dimensional data 1134 of the surroundings of the client device 902 based on the sensor information 1132. For example, the three-dimensional data generation unit 1116 uses information obtained by LiDAR and visible light images obtained by a visible light camera to generate point cloud data with color information of the surroundings of the client device 902.

[0554] The three-dimensional data synthesis unit 1117 synthesizes the three-dimensional data 1134 generated based on the sensor information 1132 with the three-dimensional map 1135 managed by the server 901 , thereby updating the three-dimensional map 1135 .

[0555] The three-dimensional data storage unit 1118 stores the three-dimensional map 1135 and the like.

[0556] Format conversion unit 1119 generates three-dimensional map 1031 by converting three-dimensional map 1135 into a format compatible with the receiving end. Furthermore, format conversion unit 1119 can reduce the data volume by compressing or encoding three-dimensional map 1135. Furthermore, format conversion unit 1119 can omit this processing if format conversion is not necessary. Furthermore, format conversion unit 1119 can control the amount of data transmitted according to the specified transmission range.

[0557] The communication unit 1120 communicates with the client device 902 and receives a data transmission request (a request to transmit a three-dimensional map) and the like from the client device 902 .

[0558] The transmission control unit 1121 exchanges information such as the corresponding format with the communication partner via the communication unit 1120 to establish communication.

[0559] The data transmission unit 1122 transmits the three-dimensional map 1031 to the client device 902. The three-dimensional map 1031 is data including a point cloud such as WLD or SWLD. The three-dimensional map 1031 may include either compressed data or uncompressed data.

[0560] Next, the workflow of the client device 902 will be described. Figure 31 This is a flowchart showing the operation performed by the client device 902 when obtaining a three-dimensional map.

[0561] First, the client device 902 requests the server 901 to transmit a three-dimensional map (such as a point cloud) (S1001). At this time, the client device 902 also transmits its location information obtained by GPS, etc., and thus requests the server 901 to transmit a three-dimensional map related to the location information.

[0562] Next, the client device 902 receives a 3D map from the server 901 ( S1002 ). If the received 3D map is compressed data, the client device 902 decodes the received 3D map to generate an uncompressed 3D map ( S1003 ).

[0563] Next, the client device 902 creates three-dimensional data 1034 of the client device 902's surroundings based on the sensor information 1033 obtained from the plurality of sensors 1015 (S1004). Next, the client device 902 estimates its own position using the three-dimensional map 1032 received from the server 901 and the three-dimensional data 1034 created based on the sensor information 1033 (S1005).

[0564] Figure 32 1037 is a flowchart illustrating the operation of client device 902 when transmitting sensor information. First, client device 902 receives a request to transmit sensor information from server 901 (S1011). Upon receiving the request, client device 902 transmits sensor information 1037 to server 901 (S1012). If sensor information 1033 includes multiple pieces of information obtained by multiple sensors 1015, client device 902 compresses each piece of information using a compression method appropriate for that piece of information, thereby generating sensor information 1037.

[0565] Next, the workflow of the server 901 will be described. Figure 33 This is a flowchart illustrating the operation of server 901 when acquiring sensor information. First, server 901 requests client device 902 to transmit sensor information (S1021). Next, server 901 receives sensor information 1037 transmitted from client device 902 in response to the request (S1022). Next, server 901 uses received sensor information 1037 to create three-dimensional data 1134 (S1023). Next, server 901 reflects the created three-dimensional data 1134 on three-dimensional map 1135 (S1024).

[0566] Figure 341 is a flowchart showing the operation of the server 901 when transmitting a three-dimensional map. First, the server 901 receives a request to transmit a three-dimensional map from the client device 902 (S1031). The server 901, having received the request to transmit the three-dimensional map, transmits the three-dimensional map 1031 to the client device 902 (S1032). At this time, the server 901 can extract a three-dimensional map in the vicinity corresponding to the location information of the client device 902 and transmit the extracted three-dimensional map. Furthermore, the server 901 can compress the three-dimensional map composed of the point cloud, for example, using an octree compression method, and transmit the compressed three-dimensional map.

[0567] Hereinafter, modifications of this embodiment will be described.

[0568] Server 901 uses sensor information 1037 received from client device 902 to create three-dimensional data 1134 near the location of client device 902. Next, server 901 matches the created three-dimensional data 1134 with a three-dimensional map 1135 of the same area managed by server 901, calculating the difference between the three-dimensional data 1134 and the three-dimensional map 1135. If the difference exceeds a predetermined threshold, server 901 determines that some abnormality has occurred near client device 902. For example, if ground subsidence occurs due to a natural disaster such as an earthquake, a significant difference may occur between the three-dimensional map 1135 managed by server 901 and the three-dimensional data 1134 created based on the sensor information 1037.

[0569] Sensor information 1037 may also include at least one of the sensor type, sensor performance, and sensor model. Furthermore, a category ID corresponding to the sensor's performance may be added to sensor information 1037. For example, if sensor information 1037 is obtained by LiDAR, an identifier may be assigned based on sensor performance. For example, a sensor capable of obtaining information with an accuracy of several millimeters may be assigned category 1, a sensor capable of obtaining information with an accuracy of several centimeters may be assigned category 2, and a sensor capable of obtaining information with an accuracy of several meters may be assigned category 3. Furthermore, server 901 may estimate sensor performance information based on the model of client device 902. For example, if client device 902 is mounted in a vehicle, server 901 may determine sensor specifications based on the vehicle's model. In this case, server 901 may obtain the vehicle's model information in advance or include this information in the sensor information. Furthermore, server 901 may use the obtained sensor information 1037 to switch the degree of correction applied to three-dimensional data 1134 generated using sensor information 1037. For example, when the sensor performance is high accuracy (category 1), the server 901 does not perform correction on the three-dimensional data 1134. When the sensor performance is low accuracy (category 3), the server 901 applies correction appropriate to the sensor accuracy to the three-dimensional data 1134. For example, the lower the sensor accuracy, the greater the degree (intensity) of correction.

[0570] The server 901 can also simultaneously request sensor information to multiple client devices 902 located in a given space. When the server 901 receives multiple sensor information items from multiple client devices 902, it is not necessary to utilize all of the sensor information in creating the three-dimensional data 1134. For example, the server 901 can select sensor information to be used based on sensor performance. For example, when updating the three-dimensional map 1135, the server 901 can select highly accurate sensor information (category 1) from the multiple sensor information items received and use this information to create the three-dimensional data 1134.

[0571] The server 901 is not limited to servers such as traffic cloud monitoring, and can also be other client devices (vehicle-mounted). Figure 35 The system configuration in this case is shown.

[0572] For example, client device 902C requests sensor information from nearby client device 902A and receives the sensor information from client device 902A. Client device 902C then uses the sensor information received from client device 902A to create three-dimensional data and update the three-dimensional map on client device 902C. In this way, client device 902C can leverage the performance of client device 902C to generate a three-dimensional map of the space available from client device 902A. This is possible, for example, when client device 902C has high performance.

[0573] In this case, client device 902A, which has provided the sensor information, is granted the right to obtain the highly accurate three-dimensional map generated by client device 902C. Client device 902A receives the highly accurate three-dimensional map from client device 902C in accordance with this right.

[0574] Alternatively, client device 902C may request the transmission of sensor information to multiple nearby client devices 902 (client device 902A and client device 902B). If the sensor of client device 902A or client device 902B is high-performance, client device 902C can create three-dimensional data using the sensor information obtained by the high-performance sensor.

[0575] Figure 36 This is a block diagram showing the functional configuration of the server 901 and the client device 902. The server 901 includes, for example, a 3D map compression / decoding processing unit 1201 that compresses and decodes a 3D map, and a sensor information compression / decoding processing unit 1202 that compresses and decodes sensor information.

[0576] The client device 902 includes a three-dimensional map decoding processing unit 1211 and a sensor information compression processing unit 1212. The three-dimensional map decoding processing unit 1211 receives the compressed encoded data of the three-dimensional map, decodes the encoded data, and obtains the three-dimensional map. The sensor information compression processing unit 1212 does not compress the three-dimensional data generated by the obtained sensor information, but instead compresses the sensor information itself and transmits the compressed encoded data of the sensor information to the server 901. With this configuration, the client device 902 can internally retain the processing unit (device or LSI) for decoding the three-dimensional map (point cloud, etc.), without having to internally retain the processing unit for compressing the three-dimensional data of the three-dimensional map (point cloud, etc.). This can reduce the cost and power consumption of the client device 902.

[0577] As described above, the client device 902 according to this embodiment is mounted on a mobile object and generates three-dimensional data 1034 of the mobile object's surroundings based on sensor information 1033 indicating the surrounding conditions of the mobile object obtained by the sensor 1015 mounted on the mobile object. The client device 902 estimates the mobile object's own position using the generated three-dimensional data 1034. The client device 902 transmits the obtained sensor information 1033 to the server 901 or another mobile object 902.

[0578] Based on this, the client device 902 transmits the sensor information 1033 to the server 901, etc. This can potentially reduce the amount of data transmitted compared to when transmitting three-dimensional data. Furthermore, since the client device 902 does not need to perform processing such as compression or encoding of the three-dimensional data, the amount of processing required by the client device 902 can be reduced. Consequently, the client device 902 can reduce the amount of data transmitted and simplify the device configuration.

[0579] Furthermore, the client device 902 sends a request to send a three-dimensional map to the server 901, and receives the three-dimensional map 1031 from the server 901. The client device 902 estimates its own position using the three-dimensional data 1034 and the three-dimensional map 1032.

[0580] Furthermore, the sensor information 1033 includes at least one of information obtained by a laser sensor, a brightness image (visible light image), an infrared image, a depth image, position information of the sensor, and speed information of the sensor.

[0581] Also, the sensor information 1033 includes information showing the performance of the sensor.

[0582] Furthermore, the client device 902 encodes or compresses the sensor information 1033, and transmits the encoded or compressed sensor information 1037 to the server 901 or another mobile object 902. Thus, the client device 902 can reduce the amount of transmitted data.

[0583] For example, the client device 902 includes a processor and a memory, and the processor performs the above-described processing using the memory.

[0584] Furthermore, the server 901 according to this embodiment can communicate with a client device 902 mounted on a mobile object, and receives sensor information 1037 indicating the surrounding conditions of the mobile object obtained by a sensor 1015 mounted on the mobile object from the client device 902. Based on the received sensor information 1037, the server 901 creates three-dimensional data 1134 of the surroundings of the mobile object.

[0585] Based on this, server 901 uses sensor information 1037 transmitted from client device 902 to create three-dimensional data 1134. This potentially reduces the amount of data transmitted compared to when client device 902 transmits the three-dimensional data. Furthermore, since client device 902 does not need to perform processing such as compression or encoding of the three-dimensional data, the processing load on client device 902 can be reduced. This allows server 901 to reduce the amount of data transmitted and simplify the device configuration.

[0586] Furthermore, the server 901 further transmits a request to transmit sensor information to the client device 902 .

[0587] Furthermore, the server 901 further updates the three-dimensional map 1135 using the generated three-dimensional data 1134 , and transmits the three-dimensional map 1135 to the client device 902 in response to a transmission request for the three-dimensional map 1135 from the client device 902 .

[0588] Furthermore, the sensor information 1037 includes at least one of information obtained by a laser sensor, a brightness image (visible light image), an infrared image, a depth image, position information of the sensor, and speed information of the sensor.

[0589] Also, the sensor information 1037 includes information showing the performance of the sensor.

[0590] Furthermore, the server 901 further calibrates the three-dimensional data according to the performance of the sensor.

[0591] Furthermore, when receiving sensor information, server 901 receives multiple sensor information 1037 from multiple client devices 902 and selects sensor information 1037 for use in creating three-dimensional data 1134 based on multiple pieces of information indicating sensor performance included in the multiple sensor information 1037. This allows server 901 to improve the quality of three-dimensional data 1134.

[0592] Furthermore, the server 901 decodes or decompresses the received sensor information 1037 and creates three-dimensional data 1134 based on the decoded or decompressed sensor information 1132. In this way, the server 901 can reduce the amount of data to be transmitted.

[0593] For example, the server 901 includes a processor and a memory, and the processor performs the above-mentioned processing using the memory.

[0594] (Implementation 7)

[0595] In this embodiment, a method for encoding and a method for decoding three-dimensional data using inter-frame prediction processing will be described.

[0596] Figure 37 3D data encoding device 1300 according to this embodiment is a block diagram. The 3D data encoding device 1300 encodes 3D data to generate an encoded bit stream (hereinafter also referred to as a bit stream) as an encoded signal. Figure 37 As shown, the three-dimensional data encoding device 1300 includes: a segmentation unit 1301, a subtraction unit 1302, a transformation unit 1303, a quantization unit 1304, an inverse quantization unit 1305, an inverse transformation unit 1306, an addition unit 1307, a reference volume memory 1308, an intra-frame prediction unit 1309, a reference space memory 1310, an inter-frame prediction unit 1311, a prediction control unit 1312, and an entropy coding unit 1313.

[0597] The segmentation unit 1301 segments each space (SPC) contained in the three-dimensional data into a plurality of volumes (VLMs) serving as coding units. Furthermore, the segmentation unit 1301 performs an octree representation (octree conversion) on the voxels within each volume. Alternatively, the segmentation unit 1301 may make the space and the volume the same size and perform an octree representation on the space. Furthermore, the segmentation unit 1301 may append information required for octree conversion (such as depth information) to the header of the bitstream.

[0598] The subtraction unit 1302 calculates the difference between the volume (coding target volume) output from the division unit 1301 and the prediction volume generated by intra prediction or inter prediction described later, and outputs the calculated difference as a prediction residual to the transformation unit 1303 . Figure 38 An example of calculating the prediction residual is shown. The bit strings of the encoding target volume and the predicted volume shown here are, for example, position information indicating the positions of three-dimensional points (eg, point clouds) included in the volume.

[0599] The following describes octree representation and the voxel scanning order. The volume is converted to an octree structure (octreeing) and then encoded. The octree structure consists of nodes and leaf nodes. Each node has eight nodes or leaves, and each leaf node contains voxel (VXL) information. Figure 39 An example of the configuration of a volume including a plurality of voxels is shown. Figure 40 Shows the Figure 39 The volume shown is transformed into an example of an octree structure. Figure 40 The leaf nodes 1, 2, and 3 in the leaf nodes shown represent Figure 39 The voxels VXL1, VXL2, and VXL3 shown express the VXL (hereinafter referred to as effective VXL) including the point group.

[0600] For example, an octree is represented by a binary sequence of 0 and 1. For example, when a node or a valid VXL is set to 1 and the others are set to 0, a node or a leaf node is assigned Figure 40 The binary sequence shown in FIG. Then, the binary sequence is scanned in a width-first or depth-first scanning order. For example, in the case of a width-first scan, Figure 41 The binary sequence shown in A. When scanning in depth first, we get Figure 41 The binary sequence obtained by this scanning is encoded by entropy coding, thereby reducing the amount of information.

[0601] Next, we will explain the depth information in the octree representation. The depth in the octree representation is used to control the granularity of the point cloud information contained in the volume. If the depth is set to a large value, the point cloud information can be reproduced at a finer level, but the amount of data used to represent the nodes and leaf nodes will increase. Conversely, if the depth is set to a small value, the amount of data can be reduced, but multiple point cloud information with different positions and colors will be treated as the same position and the same color, thus losing the original information of the point cloud information.

[0602] For example, Figure 42 Shows the Figure 40 The example of an octree with depth = 2 shown is expressed as an octree with depth = 1. Figure 42 The octree shown is Figure 40 The octree shown has a small amount of data. Figure 42 The octree shown is the same as Figure 42 Compared to the octree shown, the number of bits after binary serialization is less. Figure 40 The leaf nodes 1 and 2 shown in the figure become Figure 41 The leaf node 1 shown is shown. That is, Figure 40 The leaf node 1 and the leaf node 2 are shown as information in different positions.

[0603] Figure 43 Shown with Figure 42 The volume corresponding to the octree shown. Figure 39 VXL1 and VXL2 shown with Figure 43 In this case, the three-dimensional data encoding device 1300 corresponds to the VXL12 shown. Figure 39 The color information of VXL1 and VXL2 shown here generates Figure 43For example, the three-dimensional data encoding device 1300 calculates the color information of VXL12 using the average, median, or weighted average of the color information of VXL1 and VXL2 as the color information of VXL12. In this way, the three-dimensional data encoding device 1300 can control the reduction of data volume by changing the depth of the octree.

[0604] The three-dimensional data encoding device 1300 may also set the depth information of the octree using any of world space units, space units, and volume units. Furthermore, in this case, the three-dimensional data encoding device 1300 may also attach the depth information to the world space header information, the space header information, or the volume header information. Furthermore, the same value may be used as depth information in all world spaces, spaces, and volumes at different times. In this case, the three-dimensional data encoding device 1300 may also attach the depth information to the header information that manages the world space at all times.

[0605] When the voxels contain color information, the transformation unit 1303 applies a frequency transformation, such as an orthogonal transformation, to the prediction residuals of the color information of the voxels within the volume. For example, the transformation unit 1303 scans the prediction residuals in a certain scanning order to produce a one-dimensional array. Thereafter, the transformation unit 1303 applies a one-dimensional orthogonal transformation to the produced one-dimensional array, thereby transforming the one-dimensional array into the frequency domain. Consequently, when the values ​​of the prediction residuals within the volume are close, the values ​​of the frequency components in the low-frequency band become larger, while the values ​​of the frequency components in the high-frequency band become smaller. Therefore, the quantization unit 1304 can more effectively reduce the amount of code.

[0606] Furthermore, the transform unit 1303 may utilize an orthogonal transform of two or more dimensions instead of a one-dimensional orthogonal transform. For example, the transform unit 1303 maps the prediction residual into a two-dimensional arrangement in a certain scanning order, and applies a two-dimensional orthogonal transform to the obtained two-dimensional arrangement. Furthermore, the transform unit 1303 may select the orthogonal transform method to be used from a plurality of orthogonal transform methods. In this case, the three-dimensional data encoding device 1300 appends information indicating which orthogonal transform method is used to the bitstream. Furthermore, the transform unit 1303 may select the orthogonal transform method to be used from a plurality of orthogonal transform methods with different dimensions. In this case, the three-dimensional data encoding device 1300 appends information indicating which dimensional orthogonal transform method is used to the bitstream.

[0607] For example, the transformation unit 1303 matches the scanning order of the prediction residuals with the scanning order (width-first or depth-first, etc.) in the octree within the volume. Accordingly, since it is not necessary to append information indicating the scanning order of the prediction residuals to the bitstream, it is possible to reduce additional overhead. Furthermore, the transformation unit 1303 may also apply a scanning order different from the scanning order of the octree. In this case, the three-dimensional data encoding device 1300 appends information indicating the scanning order of the prediction residuals to the bitstream. Accordingly, the three-dimensional data encoding device 1300 can efficiently encode the prediction residuals. Furthermore, the three-dimensional data encoding device 1300 may append information indicating whether the scanning order of the octree is applied (a flag, etc.) to the bitstream, and in the case where the scanning order of the octree is not applied, append information indicating the scanning order of the prediction residuals to the bitstream.

[0608] The conversion unit 1303 may convert not only the prediction residual of color information but also other attribute information of the voxel. For example, the conversion unit 1303 may convert and encode information such as reflectivity obtained when a point cloud is acquired by LiDAR or the like.

[0609] The conversion unit 1303 may skip processing when the space does not have attribute information such as color information. Furthermore, the three-dimensional data encoding device 1300 may add information (flag) indicating whether to skip processing by the conversion unit 1303 to the bitstream.

[0610] The quantization unit 1304 quantizes the frequency components of the prediction residual generated by the transformation unit 1303 using the quantization control parameters to generate quantization coefficients. This reduces the amount of information. The generated quantization coefficients are output to the entropy coding unit 1313. The quantization unit 1304 can control the quantization control parameters according to world space units, space units, or volume units. In this case, the three-dimensional data encoding device 1300 attaches the quantization control parameters to the respective header information, etc. In addition, the quantization unit 1304 can also change the weight according to the frequency components of each prediction residual to perform quantization control. For example, the quantization unit 1304 can perform fine quantization on the low-frequency components and coarse quantization on the high-frequency components. In this case, the three-dimensional data encoding device 1300 can attach parameters representing the weights of each frequency component to the header.

[0611] The quantization unit 1304 may skip the process if the space does not have attribute information such as color information. Furthermore, the three-dimensional data encoding apparatus 1300 may add information (flag) indicating whether the process of the quantization unit 1304 is skipped to the bitstream.

[0612] The inverse quantization unit 1305 inversely quantizes the quantization coefficients generated by the quantization unit 1304 using the quantization control parameters, thereby generating inverse quantization coefficients of the prediction residual, and outputs the generated inverse quantization coefficients to the inverse transformation unit 1306 .

[0613] The inverse transform unit 1306 applies inverse transform to the inverse quantized coefficients generated by the inverse quantization unit 1305 to generate a post-inverse transform prediction residual. Since this post-inverse transform prediction residual is a prediction residual generated after quantization, it may not completely match the prediction residual output by the transform unit 1303.

[0614] The adder 1307 adds the inverse-transformed prediction residual generated by the inverse-transformer 1306 to the prediction volume generated by intra-frame prediction or inter-frame prediction, which is used to generate the pre-quantized prediction residual, to generate a reconstructed volume. This reconstructed volume is stored in the reference volume memory 1308 or the reference space memory 1310.

[0615] The intra prediction unit 1309 generates a predicted volume of the encoding target volume using the attribute information of the adjacent volumes stored in the reference volume memory 1308. The attribute information includes the color information or reflectivity of the voxels. The intra prediction unit 1309 generates a predicted value of the color information or reflectivity of the encoding target volume.

[0616] Figure 44 1309 is a diagram for explaining the operation of the intra prediction unit 1309. For example, Figure 44 As shown, the intra-frame prediction unit 1309 generates a predicted volume of the encoding target volume (volume idx = 3) based on the adjacent volume (volume idx = 0). Here, volume idx is the identifier information added to the volume in the space, and different values ​​are assigned to each volume. The order of assigning volume idx can be the same as the encoding order or different from the encoding order. For example, as Figure 44 The intra-frame prediction unit 1309 uses the average value of the color information of the voxels contained in volume idx=0, which is the adjacent volume, as the predicted value of the color information of the encoding target volume. In this case, the prediction residual is generated by subtracting the predicted value of the color information from the color information of each voxel contained in the encoding target volume. The processing subsequent to the transformation unit 1303 is performed on this prediction residual. In this case, the three-dimensional data encoding device 1300 also adds the adjacent volume information and prediction mode information to the bitstream. Here, the adjacent volume information is information indicating the adjacent volume used in the prediction, such as the volume idx indicating the adjacent volume used in the prediction. The prediction mode information indicates the mode used to generate the prediction volume. The mode may be, for example, an average value mode that generates a prediction value based on the average value of the voxels in the adjacent volume, or an average value mode that generates a prediction value based on the average value of the voxels in the adjacent volume.

[0617] The intra prediction unit 1309 can also generate a prediction volume based on multiple adjacent volumes. Figure 44 In the illustrated configuration, the intra-frame prediction unit 1309 generates prediction volume 0 based on volume idx = 0, and generates prediction volume 1 based on volume idx = 1. The intra-frame prediction unit 1309 then generates the final prediction volume by averaging the prediction volumes 0 and 1. In this case, the three-dimensional data encoding apparatus 1300 may also append multiple volume idx values ​​used in generating the prediction volumes to the bitstream.

[0618] Figure 45 The inter-frame prediction process involved in this embodiment is schematically shown. The inter-frame prediction unit 1311 performs encoding (inter-frame prediction) on the space (SPC) at a certain time T_Cur using the coded space at a different time T_LX. In this case, the inter-frame prediction unit 1311 applies rotation and translation processing to the coded space at different times T_LX to perform encoding.

[0619] Furthermore, the 3D data encoding apparatus 1300 appends RT information related to the spatial rotation and translation processing applied at a different time T_LX to the bitstream. For example, the different time T_LX is a time T_L0 prior to the certain time T_Cur. In this case, the 3D data encoding apparatus 1300 may also append RT information RT_L0 related to the spatial rotation and translation processing applied at time T_L0 to the bitstream.

[0620] Alternatively, the different time T_LX is, for example, time T_L1 after the certain time T_Cur. In this case, the three-dimensional data encoding apparatus 1300 may add RT information RT_L1 regarding the spatial rotation and translation processing applied at time T_L1 to the bitstream.

[0621] Alternatively, the inter-frame prediction unit 1311 performs encoding by referring to spaces at different times T_L0 and T_L1 (bi-prediction). In this case, the 3D data encoding apparatus 1300 may add RT information RT_L0 and RT_L1 regarding spatial rotation and translation, respectively, to the bitstream.

[0622] In addition, although T_L0 is set as the time before T_Cur and T_L1 is set as the time after T_Cur, the present invention is not limited to this. For example, T_L0 and T_L1 can both be the time before T_Cur. Alternatively, T_L0 and T_L1 can both be the time after T_Cur.

[0623] Furthermore, when encoding with reference to multiple spaces at different times, the three-dimensional data encoding device 1300 may append RT information related to the rotation and translation applied to each space to the bitstream. For example, the three-dimensional data encoding device 1300 manages the multiple referenced encoded spaces using two reference lists (L0 list and L1 list). When the first reference space in the L0 list is set to L0R0, the second reference space in the L0 list is set to L0R1, the first reference space in the L1 list is set to L1R0, and the second reference space in the L1 list is set to L1R1, the three-dimensional data encoding device 1300 appends RT information RT_L0R0 for L0R0, RT information RT_L0R1 for L0R1, RT information RT_L1R0 for L1R0, and RT information RT_L1R1 for L1R1 to the bitstream. For example, the three-dimensional data encoding device 1300 appends this RT information to the header of the bitstream.

[0624] Alternatively, when encoding with reference to multiple reference spaces at different times, the three-dimensional data encoding device 1300 may determine whether rotation and translation are applicable for each reference space. In this case, the three-dimensional data encoding device 1300 may append information indicating whether rotation and translation are applicable for each reference space (such as an RT applicable flag) to header information of the bitstream. For example, the three-dimensional data encoding device 1300 calculates RT information and an ICP error value using an ICP (Interactive Closest Point) algorithm for each reference space to be referenced based on the encoding target space. If the ICP error value is below a predetermined value, the three-dimensional data encoding device 1300 determines that rotation and translation are not necessary and sets the RT applicable flag to OFF (invalid). Furthermore, if the ICP error value is greater than the predetermined value, the three-dimensional data encoding device 1300 sets the RT applicable flag to ON (valid) and appends the RT information to the bitstream.

[0625] Figure 46 This section shows an example of a syntax for appending RT information and an RT applicable flag to a header. Furthermore, the number of bits allocated to each syntax can be determined based on the range of values ​​that the syntax can take. For example, if the number of reference spaces included in reference list L0 is 8, 3 bits can be allocated to MaxRefSpc_l0. The number of allocated bits can be varied based on the values ​​that each syntax can take, or can be fixed regardless of the values ​​that can be taken. When the number of allocated bits is fixed, the three-dimensional data encoding device 1300 can append this fixed number of bits to other header information.

[0626] Here, Figure 46MaxRefSpc_10 indicates the number of reference spaces included in reference list L0. RT_flag_10[i] is the RT application flag for reference space i in reference list L0. When RT_flag_10[i] is 1, rotation and translation are applied to reference space i. When RT_flag_10[i] is 0, rotation and translation are not applied to reference space i.

[0627] R_l0[i] and T_l0[i] are the RT information for reference space i within reference list L0. R_l0[i] is the rotation information for reference space i within reference list L0. The rotation information indicates the content of the applied rotation processing, such as a rotation matrix or quaternion. T_l0[i] is the translation information for reference space i within reference list L0. The translation information indicates the content of the applied translation processing, such as a translation vector.

[0628] MaxRefSpc_l1 indicates the number of reference spaces included in reference list L1. RT_flag_l1[i] is the RT application flag for reference space i in reference list L1. When RT_flag_l1[i] is 1, rotation and translation are applied to reference space i. When RT_flag_l1[i] is 0, rotation and translation are not applied to reference space i.

[0629] R_l1[i] and T_l1[i] are the RT information for reference space i within reference list L1. R_l1[i] is the rotation information for reference space i within reference list L1. The rotation information indicates the content of the applied rotation processing, such as a rotation matrix or quaternion. T_l1[i] is the translation information for reference space i within reference list L1. The translation information indicates the content of the applied translation processing, such as a translation vector.

[0630] The inter-frame prediction unit 1311 generates a predicted volume for the target volume using information about the already-encoded reference space stored in the reference space memory 1310. As described above, before generating the predicted volume for the target volume, the inter-frame prediction unit 1311 uses the ICP (Interactive Closest Point) algorithm to determine RT information in the target space and the reference space to approximate the overall positional relationship between the target space and the reference space. The inter-frame prediction unit 1311 then uses the determined RT information to rotate and translate the reference space, thereby obtaining reference space B. Subsequently, the inter-frame prediction unit 1311 generates a predicted volume for the target volume within the target space using information within reference space B. The three-dimensional data encoding device 1300 appends the RT information used to obtain reference space B to, for example, header information of the target space.

[0631] In this way, the inter-frame prediction unit 1311 applies rotation and translation processing to the reference space, thereby making the overall positional relationship between the encoding target space and the reference space close, and then uses the information in the reference space to generate the prediction volume. This can improve the accuracy of the prediction volume. In addition, since the prediction residual can be suppressed, the amount of code can be reduced. In addition, although an example of performing ICP using the encoding target space and the reference space is shown here, it is not limited to this. For example, in order to reduce the processing amount, the inter-frame prediction unit 1311 can also perform ICP using at least one of the encoding target space from which the number of voxels or point clouds is extracted, and the reference space from which the number of voxels or point clouds is extracted, to obtain RT information.

[0632] Furthermore, if the ICP error value obtained from the ICP result is less than a predetermined first threshold, that is, for example, if the positional relationship between the encoding target space and the reference space is close, the inter-frame prediction unit 1311 may determine that rotation and translation are unnecessary and not perform rotation and translation. In this case, the three-dimensional data encoding device 1300 does not need to add RT information to the bitstream, thereby reducing overhead.

[0633] Furthermore, if the ICP error value exceeds a predetermined second threshold, the inter-frame prediction unit 1311 determines that the spatial shape change is significant and may apply intra-frame prediction to all volumes in the encoding target space. Hereinafter, the space to which intra-frame prediction is applied is referred to as the intra-frame space. Furthermore, the second threshold is a value greater than the first threshold described above. Furthermore, the method is not limited to ICP; any method that can determine RT information from two voxel sets or two point cloud sets may be applied.

[0634] Furthermore, when the three-dimensional data contains attribute information such as shape or color, the inter-frame prediction unit 1311 searches, for example, for a volume in the reference space that is closest to the shape or color attribute information of the target volume as a prediction volume for the target volume within the target space. Furthermore, this reference space is, for example, a reference space that has been subjected to the aforementioned rotation and translation processing. The inter-frame prediction unit 1311 generates a prediction volume based on the volume (reference volume) obtained through the search. Figure 47 is a diagram for explaining the generation of the prediction volume. Figure 47When encoding the target volume (volume idx = 0) using inter-frame prediction, the reference volumes within the reference space are sequentially scanned, searching for the volume with the smallest prediction residual, the difference between the target volume and the reference volume. The inter-frame prediction unit 1311 selects the volume with the smallest prediction residual as the prediction volume. The prediction residual between the target volume and the prediction volume is encoded by the transformation unit 1303 and subsequent processing. Here, the prediction residual refers to the difference between the attribute information of the target volume and the attribute information of the prediction volume. Furthermore, the 3D data encoding device 1300 appends the volume idx of the reference volume within the reference space used as the prediction volume to the header of the bitstream.

[0635] exist Figure 47 In the example shown, the reference volume idx=4 in the reference space L0R0 is selected as the prediction volume of the encoding target volume. Then, the prediction residual between the encoding target volume and the reference volume and the reference volume idx=4 are encoded and added to the bitstream.

[0636] In addition, although the description here is given of an example of generating a predicted volume of attribute information, the same processing can be performed on a predicted volume of position information.

[0637] The prediction control unit 1312 controls whether intra-frame prediction or inter-frame prediction is used to encode the target volume. A mode that includes both intra-frame prediction and inter-frame prediction is referred to as a prediction mode. For example, the prediction control unit 1312 calculates an evaluation value using the prediction residual when the target volume is predicted using intra-frame prediction and the prediction residual when the target volume is predicted using inter-frame prediction, and selects the prediction mode with the lower evaluation value. Alternatively, the prediction control unit 1312 may apply orthogonal transform, quantization, and entropy coding to the prediction residual for intra-frame prediction and the prediction residual for inter-frame prediction, respectively, to calculate the actual code amount and use the calculated code amount as the evaluation value to select the prediction mode. Furthermore, overhead information other than the prediction residual (such as reference volume IDX information) may be added to the evaluation value. Furthermore, if the target space is predetermined to be encoded in intra-frame space, the prediction control unit 1312 may normally select intra-frame prediction.

[0638] The entropy coding unit 1313 generates a coded signal (coded bit stream) by performing variable-length coding on the quantized coefficients input from the quantization unit 1304. Specifically, the entropy coding unit 1313 binarizes the quantized coefficients and performs arithmetic coding on the resulting binary signal.

[0639] Next, a three-dimensional data decoding device that decodes the encoded signal generated by the three-dimensional data encoding device 1300 will be described. Figure 48This is a block diagram of a three-dimensional data decoding device 1400 according to this embodiment. The three-dimensional data decoding device 1400 includes an entropy decoding unit 1401, an inverse quantization unit 1402, an inverse transform unit 1403, an addition unit 1404, a reference volume memory 1405, an intra-frame prediction unit 1406, a reference space memory 1407, an inter-frame prediction unit 1408, and a prediction control unit 1409.

[0640] The entropy decoding unit 1401 performs variable-length decoding on the coded signal (coded bit stream). For example, the entropy decoding unit 1401 performs arithmetic decoding on the coded signal to generate a binary signal, and generates quantization coefficients based on the generated binary signal.

[0641] The inverse quantization unit 1402 performs inverse quantization on the quantized coefficients input from the entropy decoding unit 1401 using a quantization parameter added to a bit stream or the like, thereby generating inverse quantized coefficients.

[0642] The inverse transform unit 1403 generates a prediction residual by performing an inverse transform on the inverse quantized coefficients input from the inverse quantization unit 1402. For example, the inverse transform unit 1403 generates a prediction residual by performing an inverse orthogonal transform on the inverse quantized coefficients based on information added to the bitstream.

[0643] The adder 1404 adds the prediction residual generated by the inverse transform unit 1403 to the prediction volume generated by intra-frame prediction or inter-frame prediction to generate a reconstructed volume. This reconstructed volume is output as decoded 3D data and stored in the reference volume memory 1405 or the reference space memory 1407.

[0644] The intra prediction unit 1406 generates a prediction volume through intra prediction using the reference volume in the reference volume memory 1405 and information appended to the bitstream. Specifically, the intra prediction unit 1406 obtains prediction mode information and neighboring volume information (e.g., volume idx) appended to the bitstream, and generates a prediction volume using the neighboring volumes indicated by the neighboring volume information using the mode indicated by the prediction mode information. The details of this processing are the same as those of the intra prediction unit 1309 described above, except that the information appended to the bitstream is used.

[0645] The inter-frame prediction unit 1408 generates a prediction volume through inter-frame prediction using the reference space in the reference space memory 1407 and the information attached to the bitstream. Specifically, the inter-frame prediction unit 1408 uses the RT information of each reference space attached to the bitstream to apply rotation and translation processing to the reference space, and generates a prediction volume using the applied reference space. In addition, when the RT application flag of each reference space is present in the bitstream, the inter-frame prediction unit 1408 applies rotation and translation processing to the reference space according to the RT application flag. In addition, the details of the above-mentioned processing are the same as those of the above-mentioned inter-frame prediction unit 1311, except that the information attached to the bitstream is used.

[0646] Whether the decoding target volume is decoded using intra-frame prediction or inter-frame prediction is controlled by the prediction control unit 1409. For example, the prediction control unit 1409 selects intra-frame prediction or inter-frame prediction based on information appended to the bitstream indicating the prediction mode to be used. Alternatively, if the decoding target space is predetermined to be decoded as an intra-frame space, the prediction control unit 1409 may normally select intra-frame prediction.

[0647] The following describes variations of this embodiment. While this embodiment describes the application of rotation and translation using spatial units as an example, rotation and translation can also be applied using finer units. For example, the three-dimensional data encoding device 1300 can divide the space into subspaces and apply rotation and translation in subspace units. In this case, the three-dimensional data encoding device 1300 generates RT information for each subspace and appends the generated RT information to the header of the bitstream. Alternatively, the three-dimensional data encoding device 1300 can apply rotation and translation using volume units as encoding units. In this case, the three-dimensional data encoding device 1300 generates RT information in encoded volume units and appends the generated RT information to the header of the bitstream. Furthermore, these methods can be combined. That is, the three-dimensional data encoding device 1300 can apply rotation and translation in large units and then apply rotation and translation in finer units. For example, the three-dimensional data encoding device 1300 can apply rotation and translation in spatial units and then apply different rotations and translations to each of the multiple volumes contained in the resulting space.

[0648] Furthermore, while this embodiment illustrates the application of rotation and translation to the reference space as an example, this is not limiting. For example, the three-dimensional data encoding device 1300 may apply scaling to change the size of the three-dimensional data. Furthermore, the three-dimensional data encoding device 1300 may also apply any one or both of rotation, translation, and scaling. Furthermore, as described above, when processing is applied in multiple stages using different units, the type of processing applied to each unit may be different. For example, rotation and translation may be applied to spatial units, while translation may be applied to volume units.

[0649] Note that these modifications are also applicable to the three-dimensional data decoding apparatus 1400 .

[0650] As described above, the three-dimensional data encoding device 1300 according to this embodiment performs the following processing. Figure 48 This is a flowchart of the inter-frame prediction process performed by the three-dimensional data encoding device 1300.

[0651] First, the 3D data encoding device 1300 generates predicted position information (e.g., predicted volume) using position information of 3D points included in target 3D data (e.g., encoding target space) and reference 3D data at different times (e.g., reference space) (S1301). Specifically, the 3D data encoding device 1300 generates predicted position information by applying rotation and translation processing to the position information of 3D points included in the reference 3D data.

[0652] Furthermore, the 3D data encoding device 1300 performs rotation and translation processing using a first unit (e.g., space) and generates predicted position information using a second unit (e.g., volume) that is smaller than the first unit. For example, the 3D data encoding device 1300 may search for a volume in the reference space after rotation and translation that minimizes the difference between the encoding target volume and the position information, and use the resulting volume as the predicted volume. Alternatively, the 3D data encoding device 1300 may perform rotation and translation processing and generation of predicted position information using the same unit.

[0653] Furthermore, the three-dimensional data encoding device 1300 may apply a first rotation and translation process to the position information of the three-dimensional points contained in the reference three-dimensional data using a first unit (e.g., space), and apply a second rotation and translation process to the position information of the three-dimensional points obtained by the first rotation and translation process using a second unit (e.g., volume) that is smaller than the first unit, thereby generating predicted position information.

[0654] Here, the position information of the three-dimensional point and the predicted position information are as follows: Figure 41As shown, the octree structure is used for representation. For example, the 3D point position information and predicted position information are represented in a depth-first scan order of the octree structure. Alternatively, the 3D point position information and predicted position information are represented in a depth-first scan order of the octree structure.

[0655] And, as Figure 46 As shown, the three-dimensional data encoding device 1300 encodes an RT applicable flag indicating whether rotation and translation processing is applied to the position information of the three-dimensional point included in the reference three-dimensional data. That is, the three-dimensional data encoding device 1300 generates a coded signal (coded bit stream) including the RT applicable flag. Furthermore, the three-dimensional data encoding device 1300 encodes RT information indicating the content of the rotation and translation processing. That is, the three-dimensional data encoding device 1300 generates a coded signal (coded bit stream) including RT information. Alternatively, the three-dimensional data encoding device 1300 may encode the RT information when the RT applicable flag indicates that rotation and translation processing is applied, and may not encode the RT information when the RT applicable flag indicates that rotation and translation processing is not applied.

[0656] The 3D data includes, for example, position information of 3D points and attribute information (color information, etc.) of each 3D point. The 3D data encoding device 1300 generates predicted attribute information by using the attribute information of the 3D points included in the reference 3D data (S1302).

[0657] Next, the three-dimensional data encoding device 1300 uses the predicted position information to encode the position information of the three-dimensional points included in the object three-dimensional data. Figure 38 As shown, differential position information, which is a difference between the position information of the three-dimensional point included in the target three-dimensional data and the predicted position information, is calculated (S1303).

[0658] The 3D data encoding device 1300 then uses the predicted attribute information to encode the attribute information of the 3D points included in the target 3D data. For example, the 3D data encoding device 1300 calculates differential attribute information, which is the difference between the attribute information of the 3D points included in the target 3D data and the predicted attribute information (S1304). The 3D data encoding device 1300 then transforms and quantizes the calculated differential attribute information (S1305).

[0659] Finally, the 3D data encoding device 1300 encodes (eg, entropy encodes) the differential position information and the quantized differential attribute information ( S1306 ). That is, the 3D data encoding device 1300 generates an encoded signal (encoded bitstream) including the differential position information and the differential attribute information.

[0660] If the 3D data does not contain attribute information, the 3D data encoding device 1300 may not perform steps S1302, S1304, and S1305. Furthermore, the 3D data encoding device 1300 may only encode the 3D point position information or the 3D point attribute information.

[0661] and, Figure 49 The order of processing shown is merely an example and is not intended to be limiting. For example, since the processing of location information (S1301, S1303) and the processing of attribute information (S1302, S1304, S1305) are independent of each other, they can be executed in any order, or some of them can be processed in parallel.

[0662] As described above, in this embodiment, the 3D data encoding device 1300 generates predicted position information using the position information of 3D points contained in the target 3D data and reference 3D data at different times. The device then encodes the difference between the position information of the 3D points contained in the target 3D data and the predicted position information, i.e., the differential position information. This reduces the amount of data in the encoded signal, thereby improving encoding efficiency.

[0663] Furthermore, in this embodiment, the 3D data encoding device 1300 generates predicted attribute information by using attribute information of 3D points contained in the reference 3D data. The device then encodes the difference between the attribute information of the 3D points contained in the target 3D data and the predicted attribute information, i.e., the difference attribute information. This reduces the amount of data in the encoded signal, thereby improving encoding efficiency.

[0664] For example, the three-dimensional data encoding device 1300 includes a processor and a memory, and the processor performs the above-mentioned processing using the memory.

[0665] Figure 48 This is a flowchart of the inter-frame prediction process performed by the three-dimensional data decoding apparatus 1400 .

[0666] First, the three-dimensional data decoding apparatus 1400 decodes (eg, performs entropy decoding) the differential position information and the differential attribute information based on the encoded signal (encoded bit stream) ( S1401 ).

[0667] Furthermore, the 3D data decoding apparatus 1400 decodes, based on the coded signal, an RT application flag indicating whether rotation and translation processing is applied to the position information of the 3D points included in the reference 3D data. Furthermore, the 3D data decoding apparatus 1400 decodes RT information indicating the content of the rotation and translation processing. Furthermore, the 3D data decoding apparatus 1400 decodes the RT information if the RT application flag indicates that rotation and translation processing is applied, and does not decode the RT information if the RT application flag indicates that rotation and translation processing is not applied.

[0668] Next, the three-dimensional data decoding apparatus 1400 performs inverse quantization and inverse transformation on the decoded differential attribute information ( S1402 ).

[0669] Next, the 3D data decoding device 1400 generates predicted position information (e.g., predicted volume) using the position information of 3D points included in the target 3D data (e.g., the decoding target space) and the reference 3D data at different times (e.g., the reference space) (S1403). Specifically, the 3D data decoding device 1400 generates the predicted position information by applying rotation and translation processing to the position information of 3D points included in the reference 3D data.

[0670] More specifically, when the RT application flag indicates that rotation and translation processing is applicable, the three-dimensional data decoding apparatus 1400 applies the rotation and translation processing to the position information of the three-dimensional points included in the reference three-dimensional data indicated by the RT information. Furthermore, when the RT application flag indicates that rotation and translation processing is not applicable, the three-dimensional data decoding apparatus 1400 does not apply the rotation and translation processing to the position information of the three-dimensional points included in the reference three-dimensional data.

[0671] Furthermore, the three-dimensional data decoding apparatus 1400 may perform rotation and translation processing in a first unit (e.g., space), and may generate predicted position information in a second unit (e.g., volume) that is smaller than the first unit. Furthermore, the three-dimensional data decoding apparatus 1400 may perform rotation and translation processing and generation of predicted position information in the same unit.

[0672] Furthermore, the three-dimensional data decoding device 1400 may apply a first rotation and translation process using a first unit (e.g., space) to the position information of the three-dimensional points contained in the reference three-dimensional data, and may apply a second rotation and translation process using a second unit (e.g., volume) that is smaller than the first unit to the position information of the three-dimensional points obtained by the first rotation and translation process, thereby generating predicted position information.

[0673] Here, the position information of the three-dimensional point and the predicted position information are as follows: Figure 41As shown, the octree structure is used for representation. For example, the 3D point position information and predicted position information are represented in a depth-first scan order between depth and width in the octree structure. Alternatively, the 3D point position information and predicted position information are represented in a depth-first scan order between depth and width in the octree structure.

[0674] The three-dimensional data decoding apparatus 1400 generates predicted attribute information using attribute information of three-dimensional points included in the reference three-dimensional data ( S1404 ).

[0675] Next, the 3D data decoding device 1400 decodes the encoded position information contained in the encoded signal using the predicted position information, thereby restoring the position information of the 3D points contained in the target 3D data. Here, the encoded position information is, for example, differential position information, and the 3D data decoding device 1400 restores the position information of the 3D points contained in the target 3D data by adding the differential position information to the predicted position information (S1405).

[0676] The 3D data decoding device 1400 then decodes the coded attribute information contained in the coded signal using the predicted attribute information, thereby restoring the attribute information of the 3D points contained in the target 3D data. Here, the coded attribute information is, for example, differential attribute information, and the 3D data decoding device 1400 restores the attribute information of the 3D points contained in the target 3D data by adding the differential attribute information to the predicted attribute information (S1406).

[0677] Alternatively, if the 3D data does not include attribute information, the 3D data decoding apparatus 1400 may not execute steps S1402, S1404, and S1406. Furthermore, the 3D data decoding apparatus 1400 may only decode the 3D point position information or the 3D point attribute information.

[0678] and, Figure 50 The order of processing shown is an example and is not limiting. For example, since the processing of location information (S1403, S1405) and the processing of attribute information (S1402, S1404, S1406) are independent of each other, they can be performed in any order, and some of them can be processed in parallel.

[0679] (Implementation 8)

[0680] In this embodiment, a method of expressing three-dimensional points (point clouds) in encoding three-dimensional data will be described.

[0681] Figure 51 This is a block diagram showing the configuration of a three-dimensional data distribution system according to this embodiment. Figure 51 The distribution system shown includes a server 1501 and multiple clients 1502 .

[0682] The server 1501 includes a storage unit 1511 and a control unit 1512. The storage unit 1511 stores a coded three-dimensional map 1513 which is coded three-dimensional data.

[0683] Figure 52 An example of the structure of a bit stream encoding a three-dimensional map 1513 is shown. The three-dimensional map is divided into multiple sub-maps, and each sub-map is encoded. A random access header (RA) including sub-coordinate information is added to each sub-map. The sub-coordinate information is used to improve the encoding efficiency of the sub-map. The sub-coordinate information indicates the sub-coordinate of the sub-map. The sub-coordinate is the coordinate of the sub-map based on the reference coordinate. In addition, a three-dimensional map including multiple sub-maps is called a whole map. And, in the whole map, the coordinates that serve as the reference (for example, the origin) are called reference coordinates. That is, the sub-coordinate is the coordinate of the sub-map in the coordinate system of the whole map. In other words, the sub-coordinate indicates the deviation between the coordinate system of the whole map and the coordinate system of the sub-map. And, the coordinate in the coordinate system of the whole map based on the reference coordinate is called the whole coordinate. The coordinate in the coordinate system of the sub-map based on the sub-coordinate is called the differential coordinate.

[0684] Client 1502 sends a message to server 1501. This message includes the location information of client 1502. Based on the location information included in the received message, control unit 1512 in server 1501 obtains a bitstream for the submap at the location closest to client 1502. The submap bitstream includes submap coordinate information and is sent to client 1502. Decoder 1521 in client 1502 uses this submap coordinate information to obtain the overall coordinates of the submap relative to the reference coordinates. Application 1522 in client 1502 uses the obtained overall coordinates of the submap to execute an application related to its own location.

[0685] Furthermore, a submap represents a portion of the overall map. Subcoordinates are the coordinates of the submap's location within the overall map's base coordinate space. For example, consider a submap A for AA and a submap B for AB within the overall map of A. If a vehicle wishes to refer to the map of AA, it begins decoding from submap A; if it wishes to refer to the map of AB, it begins decoding from submap B. Here, a submap is a random access point. Specifically, A represents Osaka Prefecture, AA represents Osaka City, and AB represents Takatsuki City.

[0686] Each submap is sent to the client along with its subcoordinate information, which is included in the header of each submap or in the data packet sent.

[0687] The reference coordinates that serve as the reference coordinates for the sub-coordinate information of each sub-map may be added to header information of a space higher than the sub-map, such as header information of the entire map.

[0688] A submap can be composed of one space (SPC). In addition, a submap can be composed of multiple SPCs.

[0689] Furthermore, a submap can also include a GOS (Group of Space). Furthermore, a submap can also consist of world space. For example, if a submap contains multiple objects, if the multiple objects are assigned to different SPCs, the submap will consist of multiple SPCs. Alternatively, if multiple objects are assigned to a single SPC, the submap will consist of a single SPC.

[0690] Next, the effect of improving the coding efficiency when the sub-coordinate information is used will be described. Figure 53 is a diagram used to illustrate this effect. For example, to Figure 53 To encode a 3D point A that is far from the reference coordinates shown in FIG. , a larger number of bits are required. Here, the distance between the sub-coordinates and the 3D point A is shorter than the distance between the reference coordinates and the 3D point A. Therefore, compared to encoding the coordinates of the 3D point A based on the reference coordinates, encoding the coordinates of the 3D point A based on the sub-coordinates can improve coding efficiency. Furthermore, the bitstream of the submap includes sub-coordinate information. By sending the bitstream of the submap and the reference coordinates to the decoding side (client), the entire coordinates of the submap can be restored on the decoding side.

[0691] Figure 54 This is a flowchart of the processing performed by the server 1501 on the side transmitting the submap.

[0692] First, server 1501 receives a message from client 1502 that includes the client's location information (S1501). Control unit 1512 obtains a submap encoded bitstream based on the client's location information from storage unit 1511 (S1502). Server 1501 then transmits the submap encoded bitstream and reference coordinates to client 1502 (S1503).

[0693] Figure 55 This is a flowchart of the processing performed by the client 1502 on the receiving side of the submap.

[0694] First, client 1502 receives the encoded bitstream and reference coordinates of the submap sent from server 1501 (S1511). Next, client 1502 decodes the encoded bitstream to obtain the submap and subcoordinate information (S1512). Next, client 1502 uses the reference coordinates and subcoordinates to restore the differential coordinates within the submap to the global coordinates (S1513).

[0695] Next, an example of the syntax of information related to submaps is described. In encoding the submap, the three-dimensional data encoding device calculates the differential coordinates by subtracting the sub-coordinates from the coordinates of each point cloud (3D point). The three-dimensional data encoding device then encodes the differential coordinates into a bit stream as the value of each point cloud. Furthermore, the encoding device encodes the sub-coordinate information indicating the sub-coordinates as header information of the bit stream. Based on this, the three-dimensional data decoding device can obtain the overall coordinates of each point cloud. For example, the three-dimensional data encoding device is included in the server 1501, and the three-dimensional data decoding device is included in the client 1502.

[0696] Figure 56 An example of the syntax of a submap is shown. Figure 56 The NumOfPoint field indicates the number of point clouds contained in the submap. sub_coordinate_x, sub_coordinate_y, and sub_coordinate_z are sub-coordinate information. sub_coordinate_x indicates the x-coordinate of the submap. sub_coordinate_y indicates the y-coordinate of the submap. sub_coordinate_z indicates the z-coordinate of the submap.

[0697] Furthermore, diff_x[i], diff_y[i], and diff_z[i] are the differential coordinates of the i-th point cloud within the submap. diff_x[i] represents the difference between the x-coordinate of the i-th point cloud within the submap and the x-coordinate of the submap. diff_y[i] represents the difference between the y-coordinate of the i-th point cloud within the submap and the y-coordinate of the submap. diff_z[i] represents the difference between the z-coordinate of the i-th point cloud within the submap and the z-coordinate of the submap.

[0698] The three-dimensional data decoding device decodes point_cloud[i]_x, point_cloud[i]_y, and point_cloud[i]_z, which are the global coordinates of the i-th point cloud, using the following equations: point_cloud[i]_x is the x-coordinate of the global coordinates of the i-th point cloud. point_cloud[i]_y is the y-coordinate of the global coordinates of the i-th point cloud. point_cloud[i]_z is the z-coordinate of the global coordinates of the i-th point cloud.

[0699] point_cloud[i]_x=sub_coordinate_x + diff_x[i]

[0700] point_cloud[i]_y=sub_coordinate_y + diff_y[i]

[0701] point_cloud[i]_z=sub_coordinate_z + diff_z[i]

[0702] Next, the switching process for octree encoding will be explained. When encoding a submap, the 3D data encoding device can either select an octree representation to encode each point cloud (hereinafter referred to as octree encoding) or encode the difference between the sub-coordinates (hereinafter referred to as non-octree encoding). Figure 57 This operation is schematically illustrated. For example, when the number of point clouds within a submap exceeds a predetermined threshold, the 3D data encoding device applies octree encoding to the submap. When the number of point clouds within a submap is less than the threshold, the 3D data encoding device applies non-octree encoding to the submap. Thus, the 3D data encoding device appropriately selects whether to use octree or non-octree encoding based on the shape and density of objects within the submap, thereby improving encoding efficiency.

[0703] Furthermore, the 3D data encoding device appends information indicating whether octree encoding or non-octree encoding is applied to the submap (hereinafter referred to as octree encoding application information) to the submap header. This allows the 3D data decoding device to distinguish whether the bitstream is a bitstream obtained by octree encoding the submap or a bitstream obtained by non-octree encoding the submap.

[0704] Furthermore, the three-dimensional data encoding device can calculate the encoding efficiency when applying octree encoding and non-octree encoding to the same point cloud, and apply the encoding method with higher encoding efficiency to the sub-map.

[0705] Figure 58 An example of the syntax of the submap when such switching is performed is shown. Figure 58The coding_type shown is information indicating the coding type, which is the octree coding application information mentioned above. coding_type = 00 indicates that octree coding is applied. coding_type = 01 indicates that non-octree coding is applied. coding_type = 10 or 11 indicates that other coding methods other than the above are applied.

[0706] In the case where the encoding type is non-octree encoding (non_octree), the submap includes NumOfPoint and sub-coordinate information (sub_coordinate_x, sub_coordinate_y, and sub_coordinate_z).

[0707] When the encoding type is octree encoding, the submap includes octree_info. Octree_info is information required for octree encoding, such as depth information.

[0708] In the case where the encoding type is non-octree encoding (non_octree), the submap includes differential coordinates (diff_x[i], diff_y[i], and diff_z[i]).

[0709] When the encoding type is octree encoding (octree), the submap includes encoding data related to the octree encoding, namely, octree_data.

[0710] In addition, although an example of using the xyz coordinate system as the coordinate system of the point cloud is shown here, a polar coordinate system may also be used.

[0711] Figure 59 This is a flowchart of the 3D data encoding process performed by the 3D data encoding device. First, the 3D data encoding device calculates the number of point clouds within the target submap (S1521). Next, the 3D data encoding device determines whether the calculated number of point clouds exceeds a predetermined threshold (S1522).

[0712] If the number of point clouds is greater than the threshold ("Yes" in S1522), the 3D data encoding device applies octree encoding to the target submap (S1523). Furthermore, the 3D point data encoding device appends octree encoding application information indicating that octree encoding has been applied to the target submap to the header of the bitstream (S1525).

[0713] If the number of point clouds is lower than the threshold (No in S1522), the 3D data encoding device applies non-octree encoding to the object submap (S1524). Furthermore, the 3D point data encoding device appends octree encoding application information indicating that non-octree encoding has been applied to the object submap to the header of the bitstream (S1525).

[0714] Figure 60 This is a flowchart of the 3D data decoding process performed by the 3D data decoding device. First, the 3D data decoding device decodes octree coding application information from the bitstream header (S1531). Next, based on the decoded octree coding application information, the 3D data decoding device determines whether the coding type applied to the target submap is octree coding (S1532).

[0715] If the encoding type indicated by the octree encoding application information is octree encoding ("Yes" in S1532), the three-dimensional data decoding device decodes the object submap using octree decoding (S1533). If the encoding type indicated by the octree encoding application information is non-octree encoding ("No" in S1532), the three-dimensional data decoding device decodes the object submap using non-octree decoding (S1534).

[0716] Modifications of this embodiment will be described below. Figures 61 to 63 The operation of a modified example of the switching process of the encoding type is schematically shown.

[0717] like Figure 61 As shown, the 3D data encoding device can select whether to apply octree encoding or non-octree encoding for each space. In this case, the 3D data encoding device appends octree encoding application information to the space header. This allows the 3D data decoding device to determine whether to apply octree encoding for each space. Furthermore, in this case, the 3D data encoding device sets sub-coordinates for each space and encodes the difference between the sub-coordinates and the coordinates of each point cloud within the space.

[0718] According to this, the three-dimensional data encoding device can appropriately switch whether to apply octree encoding according to the shape of the object in space or the number of point clouds, thereby improving encoding efficiency.

[0719] And, as Figure 62As shown, the 3D data encoding device can select whether to apply octree encoding or non-octree encoding for each volume. In this case, the 3D data encoding device appends octree encoding application information to the volume header. This allows the 3D data decoding device to determine whether to apply octree encoding for each volume. Furthermore, in this case, the 3D data encoding device sets sub-coordinates for each volume and encodes the difference between the sub-coordinates and the coordinates of each point cloud within the volume.

[0720] According to this, the three-dimensional data encoding device can appropriately switch whether to apply octree encoding according to the shape of the object or the number of point clouds within the volume, thereby improving encoding efficiency.

[0721] Furthermore, in the above description, an example of encoding the difference after subtracting the sub-coordinates from the coordinates of each point cloud is shown as a non-octree encoding, but the present invention is not limited to this, and any encoding method other than octree encoding can also be used for encoding. Figure 63 As shown, the three-dimensional data encoding device may not use the difference with the sub-coordinates, but may use a method of encoding the value of the point cloud within the sub-map, space, or volume itself (hereinafter referred to as original coordinate encoding) as a non-octree encoding.

[0722] In this case, the 3D data encoding device stores information indicating that native coordinate encoding is applied to the object space (submap, space, or volume) in the header, thereby enabling the 3D data decoding device to determine whether native coordinate encoding is applied to the object space.

[0723] Furthermore, when encoding the original coordinates, the 3D data encoding device can encode the original coordinates without applying quantization or arithmetic coding. Furthermore, the 3D data encoding device can encode the original coordinates using a predetermined fixed bit length. This allows the 3D data encoding device to generate a stream with a constant bit length at a specific timing.

[0724] Furthermore, in the above description, an example of encoding the difference obtained by subtracting the sub-coordinates from the coordinates of each point cloud is shown as non-octree encoding, but the present invention is not limited to this.

[0725] For example, the three-dimensional data encoding device may sequentially encode the difference values ​​between the coordinates of each point cloud. Figure 64 is a diagram used to illustrate the operation of this case. Figure 64In the example shown, when encoding point cloud PA, the 3D data encoding device uses the sub-coordinates as predicted coordinates and encodes the difference between the coordinates of point cloud PA and the predicted coordinates. Furthermore, when encoding point cloud PB, the 3D data encoding device uses the coordinates of point cloud PA as predicted coordinates and encodes the difference between point cloud PB and the predicted coordinates. Furthermore, when encoding point cloud PC, the 3D data encoding device uses point cloud PB as predicted coordinates and encodes the difference between point cloud PB and the predicted coordinates. In this way, the 3D data encoding device can set a scanning order for multiple point clouds and encode the difference between the coordinates of a target point cloud to be processed and the coordinates of the point cloud immediately preceding the target point cloud in the scanning order.

[0726] Furthermore, in the above description, the sub-coordinates are the coordinates of the lower left front corner of the sub-map, but the position of the sub-coordinates is not limited thereto. Figures 65 to 67 1 shows another example of the position of the sub-coordinate. The sub-coordinate setting position can be set to any coordinate in the object space (sub-map, space, or volume). That is, as described above, the sub-coordinate can be the coordinate of the lower left front corner. Figure 65 As shown in , the sub-coordinates can also be the coordinates of the center of the object space. Figure 66 As shown, the sub-coordinates may also be the coordinates of the upper right rear corner of the object space. Furthermore, the sub-coordinates are not limited to the coordinates of the lower left front or upper right rear corner of the object space, but may be the coordinates of any corner in the object space.

[0727] Furthermore, the sub-coordinates can also be set to the same position as the coordinates of a point cloud in the object space (sub-map, space, or volume). Figure 67 In the example shown, the coordinates of the sub-coordinates coincide with the coordinates of the point cloud PD.

[0728] Furthermore, while this embodiment illustrates an example of switching between octree encoding and non-octree encoding, the present invention is not limited thereto. For example, the three-dimensional data encoding device may also switch between using a tree structure other than an octree, or a non-tree structure other than an octree. For example, another tree structure may include a kd-tree that is partitioned using a plane perpendicular to one of the coordinate axes. Furthermore, any other tree structure may be employed.

[0729] Furthermore, while this embodiment illustrates encoding the coordinate information of a point cloud, the present invention is not limited to this embodiment. The 3D data encoding device may also encode color information, 3D feature quantities, or visible light feature quantities using the same method as coordinate information. For example, the 3D data encoding device may set the average value of the color information of each point cloud within a submap as sub-color information and encode the difference between the color information of each point cloud and the sub-color information.

[0730] Furthermore, while this embodiment illustrates an example of selecting an encoding method (octree encoding or non-octree encoding) with higher encoding efficiency based on the number of point clouds, the present invention is not limited to this. For example, a 3D data encoding device on the server side may store bit streams of point clouds encoded using octree encoding, bit streams of point clouds encoded using non-octree encoding, and bit streams of point clouds encoded using both methods, and switch the bit streams sent to the 3D data decoding device based on the communication environment or the processing capabilities of the 3D data decoding device.

[0731] Figure 68 An example of the syntax of volume when switching the application of octree coding is shown. Figure 68 The syntax shown is the same as Figure 58 The syntax shown is basically the same, except that each piece of information is expressed in units of volume. Specifically, NumOfPoint indicates the number of point clouds contained in the volume. sub_coordinate_x, sub_coordinate_y, and sub_coordinate_z are sub-coordinate information of the volume.

[0732] Furthermore, diff_x[i], diff_y[i], and diff_z[i] are the differential coordinates of the i-th point cloud within the volume. diff_x[i] represents the difference between the x-coordinate of the i-th point cloud within the volume and the x-coordinate of the sub-coordinate. diff_y[i] represents the difference between the y-coordinate of the i-th point cloud within the volume and the y-coordinate of the sub-coordinate. diff_z[i] represents the difference between the z-coordinate of the i-th point cloud within the volume and the z-coordinate of the sub-coordinate.

[0733] Furthermore, if the relative positions of volumes in space can be calculated, the 3D data encoding device may not include the sub-coordinate information in the volume header. Specifically, the 3D data encoding device may calculate the relative positions of volumes in space without including the sub-coordinate information in the header, and use the calculated positions as the sub-coordinates of each volume.

[0734] As described above, the three-dimensional data encoding device according to this embodiment determines whether to encode the object space unit among the multiple space units (e.g., submaps, spaces, or volumes) included in the three-dimensional data using an octree structure (e.g., Figure 59 For example, if the number of three-dimensional points included in the target space unit is greater than a predetermined threshold, the three-dimensional data encoding device determines to encode the target space unit using the octree structure. Furthermore, if the number of three-dimensional points included in the target space unit is less than the threshold, the three-dimensional data encoding device determines not to encode the target space unit using the octree structure.

[0735] If it is determined that the target space unit is encoded using an octree structure ("Yes" in S1522), the three-dimensional data encoding device encodes the target space unit using the octree structure (S1523). Furthermore, if it is determined that the target space unit is not encoded using an octree structure ("No" in S1522), the three-dimensional data encoding device encodes the target space unit using a method different from the octree structure (S1524). For example, as a different method, the three-dimensional data encoding device encodes the coordinates of the three-dimensional points included in the target space unit. Specifically, as a different method, the three-dimensional data encoding device encodes the difference between the base coordinates of the target space unit and the coordinates of the three-dimensional points included in the target space unit.

[0736] Next, the three-dimensional data encoding apparatus adds information indicating whether the target space unit is encoded in the octree structure to the bit stream ( S1525 ).

[0737] Accordingly, the three-dimensional data encoding device can reduce the data amount of the encoded signal, thereby improving the encoding efficiency.

[0738] For example, the three-dimensional data encoding device includes a processor and a memory, and the processor performs the above-mentioned processing using the memory.

[0739] Furthermore, the three-dimensional data decoding apparatus according to the present embodiment decodes information indicating whether to decode an object space unit among a plurality of object space units (e.g., a submap, a space, or a volume) included in the three-dimensional data in an octree structure from a bit stream (e.g., Figure 60 If the information indicates that the target space unit is to be decoded in the octree structure (Yes in S1532 ), the three-dimensional data decoding apparatus decodes the target space unit in the octree structure ( S1533 ).

[0740] If the information indicates that the object space unit is not to be decoded using the octree structure ("No" in S1532), the 3D data decoding device decodes the object space unit using a method different from the octree structure (S1534). For example, the 3D data decoding device decodes the coordinates of the 3D points included in the object space unit using a different method. Specifically, the 3D data decoding device decodes the difference between the reference coordinates of the object space unit and the coordinates of the 3D points included in the object space unit using a different method.

[0741] Therefore, the three-dimensional data decoding device can reduce the data amount of the encoded signal, thereby improving the encoding efficiency.

[0742] For example, the three-dimensional data decoding device includes a processor and a memory, and the processor performs the above-mentioned processing using the memory.

[0743] (Implementation 9)

[0744] In this embodiment, a coding method for a tree structure such as an octree structure is described.

[0745] By identifying important areas and giving priority to decoding the three-dimensional data of the important areas, efficiency can be improved.

[0746] Figure 69 This figure shows an example of important areas in a 3D map. Important areas are, for example, areas containing a certain number or more of 3D points with large feature values ​​among the 3D points in the 3D map. Alternatively, important areas can be areas containing a certain number or more of 3D points required for a client, such as one mounted on a vehicle, to estimate its own position. Alternatively, important areas can be areas of a face in a 3D model of a person. In this way, important areas can be defined for each application, and can also be switched depending on the application.

[0747] In this embodiment, occupancy coding and location coding are used as methods for expressing the octree structure, etc. The bit string obtained by occupancy coding is called an occupancy code. The bit string obtained by location coding is called a location code.

[0748] Figure 70 This is a diagram showing an example of an occupancy code. Figure 70 Example of occupancy code representing a quadtree structure. Figure 70In the example, an occupancy code is assigned to each node. Each occupancy code indicates whether a three-dimensional point is contained in the child nodes or leaf nodes of each node. For example, in the case of a quadtree, the information indicating whether the four child nodes or leaf nodes of each node respectively contain three-dimensional points is represented by a 4-bit occupancy code. In addition, in the case of an octree, the information indicating whether the eight child nodes or leaf nodes of each node respectively contain three-dimensional points is represented by an 8-bit occupancy code. In addition, here, for the sake of simplicity, the quadtree structure is used as an example, but the same is applicable to the octree structure. For example, Figure 70 As shown, the occupancy code is based on Figure 40 In the occupancy code, the information of multiple three-dimensional points is decoded in a fixed order, so it is not possible to give priority to decoding the information of any three-dimensional point. In addition, the occupancy code can also be based on Figure 40 The bit string of the depth-first scan of nodes and leaf nodes described in

[15] .

[0749] The following describes position coding. Using position codes allows for direct decoding of important parts of the octree structure. Furthermore, it allows for efficient encoding of important 3D points deep within the structure.

[0750] Figure 71 This is a diagram for explaining position coding, and is a diagram showing an example of a quadtree structure. Figure 71 In the example shown, three-dimensional points A to I are represented by a quadtree structure. 3D points A and C are important three-dimensional points included in the important area.

[0751] Figure 72 Yes Figure 71 The diagram shows the occupancy codes and position codes of important three-dimensional points A and C in a quadtree structure.

[0752] In position coding, in a tree structure, the indexes of the nodes and leaf nodes that exist in the path to the leaf node to which the target 3D point to be coded belongs are coded. Here, the index is a numerical value assigned to each node and leaf node. In other words, the index is an identifier for identifying multiple child nodes of the target node. Figure 71 As shown, in the case of a quadtree, the index represents any one of 0 to 3.

[0753] For example, in Figure 71In the quadtree structure shown, when leaf node A is the object three-dimensional point, leaf node A is expressed as 0→2→1→0→1→2→1. Here, in the case of the right figure, the maximum value of each index is 4 (which can be expressed in 2 bits), so the number of bits required for the position code of leaf node A is 7×2bit=14bit. When leaf node C is the encoding object, the number of bits required is also 14 bits. In addition, in the case of an octree, since the maximum value of each index is 8 (which can be expressed in 3 bits), the required number of bits can be calculated as 3bit×the depth of the leaf node. In addition, the three-dimensional data encoding device can also reduce the amount of data by performing entropy encoding after binarizing each index.

[0754] In addition, if Figure 72 As shown in FIG, in the occupancy code, in order to decode the leaf nodes A and C, it is necessary to decode all the nodes in the upper layer. On the other hand, in the position code, only the data of the leaf nodes A and C can be decoded. Figure 72 As shown, by using the position code, the number of bits can be reduced compared to the occupancy code.

[0755] In addition, if Figure 72 As shown, the code amount can be further reduced by performing dictionary compression such as LZ77 on part or all of the position code.

[0756] Next, an example of applying position coding to three-dimensional points (point cloud) obtained by LiDAR will be described. Figure 73 This figure shows an example of 3D points obtained by LiDAR. 3D points obtained by LiDAR are sparse. That is, when representing these 3D points with occupancy codes, the number of zero values ​​increases. Furthermore, high 3D accuracy is required for these 3D points. This means that the octree structure becomes deeper.

[0757] Figure 74 is a diagram showing an example of such a sparse deep octree structure. Figure 74 The occupancy code in the octree structure shown is 136 bits (8 bits x 17 nodes). Furthermore, since the depth is 6 and there are 6 three-dimensional points, the position code is 3 bits x 6 x 6 = 108 bits. This means that the position code can reduce the code size by 20% compared to the occupancy code. Thus, applying position coding to a sparse, deep octree structure can reduce the code size.

[0758] The following describes the code size of the occupancy code and the position code. When the depth of the octree structure is 10, the maximum number of three-dimensional points is 8. 10 = 1073741824. In addition, the number of bits of the occupancy code of the octree structure is L o It is represented by the following.

[0759] L o =8+8 2 +…+8 10 =127133512 bits

[0760] Therefore, the number of bits per three-dimensional point is 1.143 bits. In addition, in the occupancy code, this number of bits does not change even if the number of three-dimensional points included in the octree structure changes.

[0761] On the other hand, in the position code, the number of bits of each 3D point directly affects the depth of the octree structure. Specifically, the number of bits of the position code of each 3D point is 3 bits×depth 10=30 bits.

[0762] Therefore, the number of bits of the position code of the octree structure is L l It is represented by the following.

[0763] L l =30×N

[0764] Here, N is the number of three-dimensional points included in the octree structure.

[0765] Therefore, when N<L o When / 30=40904450.4, that is, when the number of three-dimensional points is less than 40904450, the code amount of the position code becomes smaller than the code amount of the occupancy code (Ll<Lo).

[0766] In this way, when there are few three-dimensional points, the code amount of the position code is smaller than the code amount of the occupancy code, and when there are many three-dimensional points, the code amount of the position code is larger than the code amount of the occupancy code.

[0767] Therefore, the 3D data encoding device may switch between position coding and occupancy coding based on the number of input 3D points. In this case, the 3D data encoding device may also attach information indicating whether position coding or occupancy coding was used to the header information of the bitstream, etc.

[0768] Hybrid coding that combines position coding and occupancy coding is described below. Hybrid coding that combines position coding and occupancy coding is effective when coding densely populated important areas. Figure 75 is a diagram showing this example. Figure 75In the example shown, important 3D points are densely arranged. In this case, the 3D data encoding device performs position encoding on shallow upper layers and occupancy encoding on lower layers. Specifically, position encoding is used up to the deepest shared node, and occupancy encoding is used for layers deeper than the deepest shared node. Here, the deepest shared node refers to the deepest node among the nodes that are common ancestors of multiple important 3D points.

[0769] Next, hybrid coding that prioritizes compression efficiency will be described. The three-dimensional data coding device may switch between position coding and occupancy coding according to a rule predetermined in octree coding.

[0770] Figure 76 The following diagram illustrates an example of this rule. First, the 3D data encoding device determines the proportion of nodes containing 3D points at each level (depth). If this proportion exceeds a predetermined threshold, the 3D data encoding device performs occupancy coding on several nodes above the object level. For example, the 3D data encoding device applies occupancy coding to the levels from the object level to the deepest shared node.

[0771] For example, in Figure 76 In the example shown, the proportion of nodes containing three-dimensional points in the third level is higher than the threshold. Therefore, the three-dimensional data encoding device applies occupancy coding to the second and third levels from the third level to the deepest shared node, and applies position coding to the remaining first and fourth levels.

[0772] The calculation method for the above threshold is explained below. In one layer of the octree structure, there is one root node and eight child nodes. Therefore, occupancy coding requires eight bits to encode one layer of the octree structure. On the other hand, position coding requires three bits for each child node containing a 3D point. Therefore, when the number of nodes containing 3D points exceeds two, occupancy coding is more efficient than position coding. In this case, the threshold is 2.

[0773] Hereinafter, a configuration example of a bit stream generated by the above-mentioned position coding, occupancy coding, or hybrid coding will be described.

[0774] Figure 77 is a diagram showing an example of a bit stream generated by position coding. Figure 77 As shown in FIG, the bit stream generated by position coding includes a header and multiple position codes. Each position code is for a 3D point.

[0775] With this configuration, the three-dimensional data decoding device can decode a plurality of three-dimensional points with high accuracy. Figure 77This shows an example of a bitstream in the case of a quadtree structure. In the case of an octree structure, each index can take a value from 0 to 7.

[0776] Alternatively, the 3D data encoding device may binarize the column (string) of indices representing a 3D point and then perform entropy encoding. For example, if the column of indices is 0121, the 3D data encoding device may binarize 0121 into 00011001 and perform arithmetic encoding on the bit string.

[0777] Figure 78 is a diagram showing an example of a bit stream generated by hybrid coding in the case of including important three-dimensional points. Figure 78 As shown, the upper layer position code, the lower layer occupancy code of the important 3D points, and the lower layer occupancy code of the non-important 3D points other than the important 3D points are sequentially configured. Figure 78 The position code length shown indicates the code size of the position code that follows, and the occupancy code size indicates the code size of the occupancy code that follows.

[0778] With this configuration, the three-dimensional data decoding device can select different decoding plans according to the application.

[0779] The encoded data of important three-dimensional points is stored near the beginning of the bitstream, and the encoded data of unimportant three-dimensional points not included in the important area is stored after the encoded data of the important three-dimensional points.

[0780] Figure 79 It means by Figure 78 A diagram showing a tree structure of occupancy codes representing important three-dimensional points is shown. Figure 80 It means by Figure 78 The tree structure of the occupancy code representation of non-significant 3D points is shown in FIG. Figure 79 As shown, the occupancy code of the important 3D points excludes information related to non-important 3D points. Specifically, since nodes 0 and 3 at depth 5 do not contain important 3D points, the value 0 indicating that no 3D points are contained is assigned to nodes 0 and 3.

[0781] On the other hand, Figure 80 As shown, the occupancy code of the non-important 3D points excludes information related to the important 3D points. Specifically, since the node 1 at depth 5 does not contain any non-important 3D points, the value 0 indicating that no 3D points are contained is assigned to the node 1.

[0782] In this way, the 3D data encoding device divides the original tree structure into a first tree structure containing important 3D points and a second tree structure containing unimportant 3D points, and independently performs occupancy encoding on the first and second tree structures. This allows the 3D data decoding device to prioritize decoding of important 3D points.

[0783] Next, a configuration example of a bit stream generated by hybrid coding with an emphasis on efficiency will be described. Figure 81 This is a diagram showing an example of the structure of a bit stream generated by hybrid coding that prioritizes efficiency. Figure 81 As shown, for each subtree, the subtree root node position, occupancy code amount and occupancy code are configured in sequence. Figure 81 The subtree position shown is the position code of the root node of the subtree.

[0784] In the above-described configuration, when only one of position coding and occupancy coding is applied to the octree structure, the following holds true.

[0785] If the length of the position code of the root node of a subtree is equal to the depth of the octree structure, the subtree has no child nodes. In other words, the position code is applied to the entire tree structure.

[0786] When the root node of a subtree is equal to the root node of the octree structure, occupancy encoding is applied to the entire tree structure.

[0787] For example, based on the above-mentioned rules, the three-dimensional data decoding device can determine whether the bit stream contains the position code or the occupancy code.

[0788] Furthermore, the bitstream may include coding mode information indicating whether position coding, occupancy coding, or hybrid coding is used. Figure 82 is a diagram showing an example of a bit stream in this case. Figure 82 As shown, 2-bit encoding mode information indicating the encoding mode is added to the bit stream.

[0789] In addition, the "number of 3D points" in (1) position coding indicates the number of subsequent 3D points. In addition, the "occupancy code amount" in (2) occupancy coding indicates the code amount of the subsequent occupancy code. In addition, the "number of important subtrees" in (3) hybrid coding (important 3D points) indicates the number of subtrees containing important 3D points. In addition, the "number of occupancy subtrees" in (4) hybrid coding (efficiency-oriented) indicates the number of subtrees after occupancy coding.

[0790] Next, an example of syntax used to switch between application of occupancy coding and position coding will be described. Figure 83 It is a diagram showing the sentence example.

[0791] Figure 83 The isleaf shown indicates whether the object node is a leaf node. isleaf = 1 indicates that the object node is a leaf node, and isleaf = 0 indicates that the object node is not a leaf node but a node.

[0792] When the object node is a leaf node, a point_flag is appended to the bitstream. point_flag is a flag indicating whether the object node (leaf node) contains three-dimensional points. point_flag = 1 indicates that the object node contains three-dimensional points, and point_flag = 0 indicates that the object node does not contain three-dimensional points.

[0793] When the object node is not a leaf node, a coding_type is appended to the bitstream. coding_type is coding type information indicating the applicable coding type. coding_type = 00 indicates that position coding is applied, coding_type = 01 indicates that occupancy coding is applied, and coding_type = 10 or 11 indicates that other coding methods are applied, etc.

[0794] When the coding type is position coding, numPoint, num_idx[i], and idx[i][j] are appended to the bitstream.

[0795] numPoint represents the number of three-dimensional points for which position coding is performed. num_idx[i] represents the number (depth) of indices from the object node to the three-dimensional point i. When all the three-dimensional points for which position coding is performed are at the same depth, num_idx[i] are all the same value. Therefore, it can also be that, before the Figure 83 shown for loop (for(i = 0; i < numPoint; i++) {), num_idx is defined as a common value.

[0796] idx[i][j] represents the value of the j-th index in the indices from the object node to the three-dimensional point i. In the case of an octree, the number of bits of idx[i][j] is 3 bits.

[0797] In addition, as described above, an index is an identifier used to identify multiple child nodes of an object node. In the case of an octree, idx[i][j] represents any one of 0 to 7. In addition, in the case of an octree, there are 8 child nodes, and each child node corresponds to each of the 8 sub-blocks obtained by spatially dividing the object block corresponding to the object node into 8 parts. Therefore, idx[i][j] can also be information representing the three-dimensional position of the sub-block corresponding to the child node. For example, idx[i][j] can also be 3-bit information in total containing 1 bit each representing the positions of x, y, and z of the sub-block.

[0798] When the coding type is occupancy coding, occupancy_code is added to the bitstream. Occupancy_code is the occupancy code of the target node. In the case of an octree, occupancy_code is an 8-bit bit string, such as the bit string "00101000".

[0799] When the value of the (i+1)th bit of occupancy_code is 1, the process moves to the child node. That is, the child node is set as the next target node, and a bit string is recursively generated.

[0800] In this embodiment, an example of indicating the end of an octree by appending leaf node information (isleaf, point_flag) to the bitstream is shown, but the present invention is not necessarily limited to this. For example, the three-dimensional data encoding device may append the maximum depth (depth) from the start node (root node) of the occupancy code to the end (leaf node) where the three-dimensional point exists to the header of the start node. The three-dimensional data encoding device may then recursively bit-string the child node information while increasing the depth from the start node, and determine that a leaf node has been reached when the depth reaches the maximum depth. In addition, the three-dimensional data encoding device may append information indicating the maximum depth to the first node where coding_type becomes occupancy coding, or to the start node (root node) of the octree.

[0801] As described above, the three-dimensional data encoding device may also add information for switching between occupancy coding and position coding to the bit stream as header information of each node.

[0802] Furthermore, the three-dimensional data encoding device may perform entropy coding on the coding_type, numPoint, num_idx, idx, and occupancy_code of each node generated by the above method. For example, the three-dimensional data encoding device may perform arithmetic coding after binarizing each value.

[0803] In addition, the above syntax illustrates the use of a depth-first bitstream in an octree structure as an occupancy code, but the present invention is not necessarily limited to this. The three-dimensional data encoding device may also use a width-first bitstream in an octree structure as an occupancy code. When using a width-first bitstream, the three-dimensional data encoding device may also add information for switching between occupancy coding and position coding to the bitstream as header information for each node.

[0804] In this embodiment, an octree structure is used as an example, but the present invention is not necessarily limited thereto. The above method can also be applied to N-ary trees (N is an integer greater than or equal to 2) such as quadtrees and hexadecimal trees, or other tree structures.

[0805] An example of a flow of encoding processing for switching between application of occupancy coding and position coding will be described below. Figure 84 This is a flowchart of the encoding process in this embodiment.

[0806] First, the 3D data encoding device represents multiple 3D points contained in the 3D data using an octree structure (S1601). Next, the 3D data encoding device sets the root node of the octree structure as the target node (S1602). Next, the 3D data encoding device performs node encoding on the target node to generate a bitstream in the octree structure (S1603). Next, the 3D data encoding device performs entropy encoding on the generated bitstream to generate a bitstream (S1604).

[0807] Figure 85 This is a flowchart of the node encoding process (S1603). First, the three-dimensional data encoding device determines whether the target node is a leaf node (S1611). If the target node is not a leaf node ("No" in S1611), the three-dimensional data encoding device sets the leaf node flag (isleaf) to 0 and adds the leaf node flag to the bit string (S1612).

[0808] Next, the three-dimensional data encoding device determines whether the number of child nodes including the three-dimensional point is greater than a predetermined threshold value (S1613). Alternatively, the three-dimensional data encoding device may add the threshold value to the bit string.

[0809] When the number of child nodes including three-dimensional points is greater than a predetermined threshold (Yes in S1613 ), the three-dimensional data encoding device sets the encoding type (coding_type) to occupancy coding and adds the encoding type to the bit string ( S1614 ).

[0810] Next, the three-dimensional data encoding device sets occupancy coding information and adds the occupancy coding information to the bit string. Specifically, the three-dimensional data encoding device generates an occupancy code for the target node and adds the occupancy code to the bit string (S1615).

[0811] Next, the three-dimensional data encoding device sets the next target node according to the occupancy code (S1616). Specifically, the three-dimensional data encoding device sets the unprocessed child node with the occupancy code "1" as the next target node.

[0812] Next, the three-dimensional data encoding device performs node encoding processing on the newly set object node (S1617). Figure 85 The processing shown.

[0813] If all child nodes have not been processed (No in S1618), the process from step S1616 onward is repeated. On the other hand, if all child nodes have been processed (Yes in S1618), the three-dimensional data encoding device ends the node encoding process.

[0814] In addition, in step S1613, when the number of child nodes containing three-dimensional points is below a predetermined threshold ("No" in S1613), the three-dimensional data encoding device sets the encoding type to position encoding and appends the encoding type to the bit string (S1619).

[0815] Next, the 3D data encoding device sets position coding information and appends the position coding information to the bit string. Specifically, the 3D data encoding device generates a position code and appends the position code to the bit string (S1620). The position code includes numPoint, num_idx, and idx.

[0816] Furthermore, in step S1611, if the target node is a leaf node ("Yes" in S1611), the three-dimensional data encoding device sets a leaf node flag to 1 and appends the leaf node flag to the bit string (S1621). Furthermore, the three-dimensional data encoding device sets a point flag (point_flag), which indicates whether the leaf node contains a three-dimensional point, and appends the point flag to the bit string (S1622).

[0817] Next, an example of a decoding process flow for switching between application of occupancy coding and position coding will be described. Figure 85 This is a flowchart of the decoding process of this embodiment.

[0818] The 3D data decoding device generates a bit string by performing entropy decoding on the bit stream (S1631). Next, the 3D data decoding device performs node decoding on the resulting bit string to restore the octree structure (S1632). Next, the 3D data decoding device generates 3D points based on the restored octree structure (S1633).

[0819] Figure 87 This is a flowchart of the node decoding process (S1632). First, the three-dimensional data decoding device obtains (decodes) the leaf node flag (isleaf) from the bit string (S1641). Next, the three-dimensional data decoding device determines whether the target node is a leaf node based on the leaf node flag (S1642).

[0820] If the target node is not a leaf node (No in S1642), the three-dimensional data decoding apparatus obtains the coding type (coding_type) from the bit string (S1643) and determines whether the coding type is occupancy coding (S1644).

[0821] If the encoding type is occupancy encoding (Yes in S1644 ), the three-dimensional data decoding apparatus obtains occupancy encoding information from the bit string. Specifically, the three-dimensional data decoding apparatus obtains an occupancy code from the bit string ( S1645 ).

[0822] Next, the 3D data decoding apparatus sets the next target node according to the occupancy code (S1646). Specifically, the 3D data decoding apparatus sets the unprocessed child node with the occupancy code "1" as the next target node.

[0823] Next, the three-dimensional data decoding device performs node decoding processing on the newly set target node (S1647). Figure 87 The processing shown.

[0824] If processing of all child nodes is not completed (No in S1648), the process from step S1646 onward is performed again. On the other hand, if processing of all child nodes is completed (Yes in S1648), the three-dimensional data decoding apparatus ends the node decoding process.

[0825] If the encoding type is position encoding in step S1644 (No in S1644), the 3D data decoding device obtains position encoding information from the bit string. Specifically, the 3D data decoding device obtains a position code from the bit string (S1649). The position code includes numPoint, num_idx, and idx.

[0826] If the target node is a leaf node in step S1642 (Yes in S1642 ), the three-dimensional data decoding apparatus obtains a point flag (point_flag), which is information indicating whether the leaf node includes a three-dimensional point, from the bit string ( S1650 ).

[0827] In addition, this embodiment shows an example of switching the encoding type for each node, but this is not necessarily limited to this. The encoding type can also be fixed in volume, space, or world space units. In this case, the three-dimensional data encoding device can also attach the encoding type information to the volume, space, or world space header information.

[0828] As described above, the 3D data encoding device of this embodiment generates first information representing an N-ary tree structure (N is an integer greater than or equal to 2) of multiple 3D points contained in 3D data using a first method (position encoding), and generates a bitstream containing the first information. The first information includes 3D point information (position code) corresponding to each of the multiple 3D points. Each piece of 3D point information includes an index (idx) corresponding to each of the multiple layers in the N-ary tree structure. Each index indicates the sub-block to which the corresponding 3D point belongs, among the N sub-blocks belonging to the corresponding layer.

[0829] In other words, each piece of 3D point information represents a path to the corresponding 3D point in an N-ary tree structure, and each index represents a child node included in the path among the N child nodes belonging to the corresponding layer (node).

[0830] Thus, the three-dimensional data encoding method can generate a bit stream capable of selectively decoding three-dimensional points.

[0831] For example, the 3D point information (position code) includes information (num_idx) indicating the number of indices included in the 3D point information. In other words, this information indicates the depth (number of layers) to the corresponding 3D point in the N-ary tree structure.

[0832] For example, the first information includes information (numPoint) indicating the number of three-dimensional point information included in the first information. In other words, this information indicates the number of three-dimensional points included in the N-ary tree structure.

[0833] For example, N is 8 and the index is 3 bits.

[0834] For example, a three-dimensional data encoding device has a first encoding mode for generating first information, and a second encoding mode for generating second information (occupancy code) representing an N-ary tree structure using a second method (occupancy coding), and generates a bitstream including the second information. The second information includes a plurality of one-bit information corresponding to each of a plurality of sub-blocks belonging to a plurality of layers in the N-ary tree structure and indicating whether a three-dimensional point exists in the corresponding sub-block.

[0835] For example, the 3D data encoding device uses a first encoding mode when the number of 3D points is below a predetermined threshold, and uses a second encoding mode when the number of 3D points exceeds the threshold. This reduces the bitstream code size.

[0836] For example, the first information and the second information include information (coding mode information) indicating whether the information represents the N-ary tree structure in the first manner or the second manner.

[0837] For example, Figure 75As shown in FIG. 1 , the three-dimensional data encoding device uses the first encoding mode in a part of the N-ary tree structure and uses the second encoding mode in another part of the N-ary tree structure.

[0838] For example, the three-dimensional data encoding device includes a processor and a memory, and the processor performs the above-mentioned processing using the memory.

[0839] Furthermore, the 3D data decoding device of this embodiment obtains first information (position codes) from a bitstream that represents an N-ary tree structure (N is an integer greater than or equal to 2) of multiple 3D points contained in the 3D data using a first method (position coding). The first information includes 3D point information (position codes) corresponding to each of the multiple 3D points. Each piece of 3D point information includes an index (idx) corresponding to each of the multiple layers in the N-ary tree structure. Each index indicates the sub-block to which the corresponding 3D point belongs, among the N sub-blocks belonging to the corresponding layer.

[0840] In other words, each piece of 3D point information represents a path to the corresponding 3D point in the N-ary tree structure, and each index represents a child node included in the path among the N child nodes belonging to the corresponding layer (node).

[0841] The three-dimensional data decoding device further uses the three-dimensional point information to restore the three-dimensional point corresponding to the three-dimensional point information.

[0842] Thus, the three-dimensional data decoding device can selectively decode three-dimensional points from a bit stream.

[0843] For example, the 3D point information (position code) includes information (num_idx) indicating the number of indices included in the 3D point information. In other words, this information indicates the depth (number of layers) to the corresponding 3D point in the N-ary tree structure.

[0844] For example, the first information includes information (numPoint) indicating the number of three-dimensional point information included in the first information. In other words, this information indicates the number of three-dimensional points included in the N-ary tree structure.

[0845] For example, N is 8 and the index is 3 bits.

[0846] For example, the 3D data decoding device further obtains second information (occupancy code) from the bitstream that represents the N-ary tree structure using a second method (occupancy coding). The 3D data decoding device uses the second information to reconstruct the plurality of 3D points. The second information includes a plurality of 1-bit information corresponding to each of the plurality of sub-blocks belonging to the plurality of layers in the N-ary tree structure and indicating whether a 3D point exists in the corresponding sub-block.

[0847] For example, the first information and the second information include information (coding mode information) indicating whether the information represents the N-ary tree structure in the first manner or the second manner.

[0848] For example, Figure 75 As shown in FIG. 1 , a portion of the N-ary tree structure is represented by the first method, and another portion of the N-ary tree structure is represented by the second method.

[0849] For example, the three-dimensional data decoding device includes a processor and a memory, and the processor performs the above-mentioned processing using the memory.

[0850] (Implementation 10)

[0851] In this embodiment, another example of a method for encoding a tree structure such as an octree structure will be described. Figure 88 This is a diagram showing an example of a tree structure related to this embodiment. Figure 88 An example showing a quadtree structure.

[0852] Leaf nodes that contain 3D points are called valid leaf nodes, and leaf nodes that do not contain 3D points are called invalid leaf nodes. Branches with a number of valid leaf nodes greater than a threshold are called dense branches. Branches with a number of valid leaf nodes less than the threshold are called sparse branches.

[0853] The three-dimensional data encoding device calculates the number of three-dimensional points included in each branch (ie, the number of valid leaf nodes) in a certain layer of the tree structure. Figure 88 This example shows a case where the threshold is 5. In this example, there are two branches in layer 1. Since the left branch contains seven 3D points, it is determined to be a dense branch. Since the right branch contains two 3D points, it is determined to be a sparse branch.

[0854] Figure 89 For example, this is a diagram showing an example of the number of valid leaf nodes (3D points) included in each branch of layer 5. Figure 89 The horizontal axis represents the identification number or index of the branch of layer 5. Figure 89 As shown in Figure 3, certain branches contain significantly more 3D points than other branches. In such dense branches, occupancy encoding is more effective than in sparse branches.

[0855] The following describes how to apply occupancy coding and position coding. Figure 90 This is a diagram showing the relationship between the number of three-dimensional points (the number of valid leaf nodes) included in each branch of layer 5 and the applied encoding method. Figure 90As shown, the three-dimensional data encoding device applies occupancy coding to dense branches and position coding to sparse branches, thereby improving encoding efficiency.

[0856] Figure 91 is a diagram showing an example of a dense branch region in LiDAR data. Figure 91 As shown, the density of three-dimensional points calculated based on the number of three-dimensional points included in each branch differs depending on the region.

[0857] Furthermore, separating dense 3D points (branches) from sparse 3D points (branches) offers the following advantages. The closer to the LiDAR sensor, the higher the 3D point density. Therefore, by separating branches according to sparseness, it is possible to perform range-based segmentation. This segmentation is effective in specific applications. Furthermore, for sparse branches, using methods other than occupancy coding is also effective.

[0858] In this embodiment, the three-dimensional data encoding device separates an input three-dimensional point group into two or more sub-three-dimensional point groups and applies a different encoding method to each sub-three-dimensional point group.

[0859] For example, the three-dimensional data encoding device separates the input three-dimensional point group into a sub-three-dimensional point group A including dense branches (dense cloud) and a sub-three-dimensional point group B including sparse branches (sparse cloud). Figure 92 It means from Figure 88 FIG. 1 is a diagram showing an example of a sub-three-dimensional point group A (a dense three-dimensional point group) including dense branches separated by a tree structure. Figure 93 It means from Figure 88 FIG. 1 is a diagram showing an example of a sub-three-dimensional point group B (sparse three-dimensional point group) including sparse branches separated by the tree structure.

[0860] Next, the three-dimensional data encoding device encodes the sub-three-dimensional point group A through occupancy encoding and encodes the sub-three-dimensional point group B through position encoding.

[0861] In addition, an example of applying different encoding methods (occupancy encoding and position encoding) is shown here, but for example, a three-dimensional data encoding device can also use the same encoding method for sub-three-dimensional point group A and sub-three-dimensional point group B, and make the parameters used in encoding different between sub-three-dimensional point group A and sub-three-dimensional point group B.

[0862] The following describes the flow of three-dimensional data encoding processing performed by the three-dimensional data encoding device. Figure 94 This is a flowchart of a three-dimensional data encoding process performed by the three-dimensional data encoding device according to this embodiment.

[0863] First, the three-dimensional data encoding device separates the input three-dimensional point group into sub-three-dimensional point groups (S1701). The three-dimensional data encoding device can perform the separation automatically or based on information input by the user. For example, the range of the sub-three-dimensional point group can be specified by the user. In addition, as an example of automatic operation, for example, when the input data is LiDAR data, the three-dimensional data encoding device uses the distance information to each point group for separation. Specifically, the three-dimensional data encoding device separates the point group within a certain range from the measurement location from the point group outside the range. In addition, the three-dimensional data encoding device can also use information about important areas and unimportant areas for separation.

[0864] Next, the three-dimensional data encoding device encodes the sub-three-dimensional point group A using method A to generate encoded data (encoded bit stream) (S1702). In addition, the three-dimensional data encoding device encodes the sub-three-dimensional point group B using method B to generate encoded data (S1703). In addition, the three-dimensional data encoding device may also encode the sub-three-dimensional point group B using method A. In this case, the three-dimensional data encoding device encodes the sub-three-dimensional point group B using a parameter different from the encoding parameter used in the encoding of the sub-three-dimensional point group A. For example, the parameter may also be a quantization parameter. For example, the three-dimensional data encoding device encodes the sub-three-dimensional point group B using a quantization parameter larger than the quantization parameter used in the encoding of the sub-three-dimensional point group A. In this case, the three-dimensional data encoding device may also add information indicating the quantization parameter used in the encoding of the sub-three-dimensional point group to the header of the encoded data of each sub-three-dimensional point group.

[0865] Next, the three-dimensional data encoding device generates a bit stream by combining the encoded data obtained in step S1702 and the encoded data obtained in step S1703 ( S1704 ).

[0866] Furthermore, the three-dimensional data encoding device may encode information for decoding each sub-three-dimensional point group as header information of the bit stream. For example, the three-dimensional data encoding device may encode the following information.

[0867] The header information may also include information indicating the number of sub-3D points to be encoded. In this example, the information indicates 2.

[0868] The header information may also include information indicating the number of 3D points contained in each sub-3D point group and the encoding method. In this example, the information indicates the number of 3D points contained in sub-3D point group A, the encoding method applied to sub-3D point group A (method A), the number of 3D points contained in sub-3D point group B, and the encoding method applied to sub-3D point group B (method B).

[0869] The header information may include information for identifying the start position or the end position of the encoded data of each sub-three-dimensional point group.

[0870] Furthermore, the three-dimensional data encoding device may encode the three-dimensional point group sub-group A and the three-dimensional point group sub-group B in parallel. Alternatively, the three-dimensional data encoding device may encode the three-dimensional point group sub-group A and the three-dimensional point group sub-group B sequentially.

[0871] Furthermore, the method for separating the three-dimensional point groups into subgroups is not limited to the above. For example, the three-dimensional data encoding device may vary the separation method, perform encoding using each of multiple separation methods, and calculate the coding efficiency of the encoded data obtained using each separation method. Furthermore, the three-dimensional data encoding device may select the separation method with the highest coding efficiency. For example, the three-dimensional data encoding device may separate the three-dimensional point group in each of multiple layers, calculate the coding efficiency in each case, select the separation method with the highest coding efficiency (i.e., the layer to be separated), and generate subgroups of the three-dimensional point group using the selected separation method for encoding.

[0872] Furthermore, when combining the encoded data, the 3D data encoding device may also arrange the encoded information of more important sub-3D point groups closer to the beginning of the bitstream. This allows the 3D data decoding device to obtain important information by decoding only the beginning bitstream, thereby obtaining the important information earlier.

[0873] Next, the flow of the three-dimensional data decoding process performed by the three-dimensional data decoding device will be described. Figure 95 This is a flowchart of a three-dimensional data decoding process performed by the three-dimensional data decoding device according to this embodiment.

[0874] First, the 3D data decoding device obtains a bitstream generated by the 3D data encoding device described above. Next, the 3D data decoding device separates the encoded data for sub-3D point group A from the encoded data for sub-3D point group B from the obtained bitstream (S1711). Specifically, the 3D data decoding device decodes information used to decode each sub-3D point group from the header information in the bitstream and uses this information to separate the encoded data for each sub-3D point group.

[0875] Next, the 3D data decoding device decodes the encoded data of sub-3D point group A using technique A to obtain sub-3D point group A (S1712). Furthermore, the 3D data decoding device decodes the encoded data of sub-3D point group B using technique B to obtain sub-3D point group B (S1713). Next, the 3D data decoding device combines sub-3D point group A with sub-3D point group B (S1714).

[0876] Furthermore, the three-dimensional data decoding apparatus may decode the three-dimensional point group sub-group A and the three-dimensional point group sub-group B in parallel. Alternatively, the three-dimensional data decoding apparatus may decode the three-dimensional point group sub-group A and the three-dimensional point group sub-group B sequentially.

[0877] Furthermore, the 3D data decoding device can also decode a required sub-3D point group. For example, the 3D data decoding device can decode sub-3D point group A but not sub-3D point group B. For example, if sub-3D point group A is included in a significant region of the LiDAR data, the 3D data decoding device will decode the 3D point group in that significant region. This significant region's 3D point group can be used to estimate the vehicle's ego position, for example.

[0878] Next, a specific example of the encoding process according to this embodiment will be described. Figure 96 This is a flowchart of a three-dimensional data encoding process performed by the three-dimensional data encoding device according to this embodiment.

[0879] First, the three-dimensional data encoding device separates the input three-dimensional points into a sparse three-dimensional point group and a dense three-dimensional point group (S1721). Specifically, the three-dimensional data encoding device counts the number of valid leaf nodes of the branches of a certain layer of the octree structure. The three-dimensional data encoding device sets each branch as a dense branch or a sparse branch based on the number of valid leaf nodes of each branch. Furthermore, the three-dimensional data encoding device generates a sub-three-dimensional point group (dense three-dimensional point group) that aggregates the dense branches and a sub-three-dimensional point group (sparse three-dimensional point group) that aggregates the sparse branches.

[0880] Next, the three-dimensional data encoding device generates encoded data by encoding the sparse three-dimensional point group (S1722). For example, the three-dimensional data encoding device encodes the sparse three-dimensional point group using position coding.

[0881] Furthermore, the three-dimensional data encoding device generates encoded data by encoding the dense three-dimensional point group (S1723). For example, the three-dimensional data encoding device encodes the dense three-dimensional point group using occupancy coding.

[0882] Next, the three-dimensional data encoding apparatus generates a bit stream by combining the encoded data of the sparse three-dimensional point group obtained in step S1722 and the encoded data of the dense three-dimensional point group obtained in step S1723 ( S1724 ).

[0883] Furthermore, the 3D data encoding device may encode information for decoding a sparse 3D point group and a dense 3D point group as header information of the bit stream. For example, the 3D data encoding device may encode the following information.

[0884] The header information may also include information indicating the number of sub-three-dimensional point groups to be encoded. In this example, the information indicates 2.

[0885] The header information may also include information indicating the number of 3D points contained in each sub-3D point group and the encoding method. In this example, the information indicates the number of 3D points contained in the sparse 3D point group and the encoding method applied to the sparse 3D point group (position encoding), and the number of 3D points contained in the dense 3D point group and the encoding method applied to the dense 3D point group (occupancy encoding).

[0886] The header information may also include information identifying the starting position or ending position of the encoded data for each sub-3D point group. In this example, the information indicates at least one of the starting position and ending position of the encoded data for the sparse 3D point group and the starting position and ending position of the encoded data for the dense 3D point group.

[0887] Furthermore, the 3D data encoding device may encode the sparse 3D point group and the dense 3D point group in parallel, or may encode the sparse 3D point group and the dense 3D point group sequentially.

[0888] Next, a specific example of three-dimensional data decoding processing will be described. Figure 97 This is a flowchart of a three-dimensional data decoding process performed by the three-dimensional data decoding device according to this embodiment.

[0889] First, a 3D data decoding device, for example, obtains a bitstream generated by the 3D data encoding device described above. Next, the 3D data decoding device separates the obtained bitstream into encoded data for a sparse 3D point group and encoded data for a dense 3D point group (S1731). Specifically, the 3D data decoding device decodes information used to decode each sub-3D point group from the header information in the bitstream and uses this information to separate the encoded data for each sub-3D point group. In this example, the 3D data decoding device uses the header information to separate the encoded data for the sparse 3D point group and the dense 3D point group from the bitstream.

[0890] Next, the 3D data decoding apparatus decodes the coded data of the sparse 3D point group to obtain a sparse 3D point group (S1732). For example, the 3D data decoding apparatus decodes the sparse 3D point group using position decoding for decoding the position-coded coded data.

[0891] Furthermore, the 3D data decoding apparatus decodes the dense 3D point group encoded data to obtain a dense 3D point group (S1733). For example, the 3D data decoding apparatus decodes the dense 3D point group using occupancy decoding for decoding the occupancy-encoded encoded data.

[0892] Next, the three-dimensional data decoding apparatus combines the sparse three-dimensional point group obtained in step S1732 with the dense three-dimensional point group obtained in step S1733 ( S1734 ).

[0893] Furthermore, the 3D data decoding apparatus may decode the sparse 3D point group and the dense 3D point group in parallel, or may decode the sparse 3D point group and the dense 3D point group sequentially.

[0894] Furthermore, the 3D data decoding device can also decode a portion of a required sub-3D point group. For example, the 3D data decoding device can decode a dense 3D point group instead of decoding sparse 3D data. For example, if the dense 3D point group is contained in a significant region of LiDAR data, the 3D data decoding device decodes the 3D point group in that significant region. This significant region's 3D point group can be used to estimate the vehicle's ego position, for example.

[0895] Figure 98 First, the three-dimensional data encoding device generates a sparse three-dimensional point group and a dense three-dimensional point group by separating the input three-dimensional point group into a sparse three-dimensional point group and a dense three-dimensional point group (S1741).

[0896] Next, the 3D data encoding device generates encoded data by encoding the dense 3D point group (S1742). Furthermore, the 3D data encoding device generates encoded data by encoding the sparse 3D point group (S1743). Finally, the 3D data encoding device generates a bitstream by combining the encoded data of the sparse 3D point group obtained in step S1742 with the encoded data of the dense 3D point group obtained in step S1743 (S1744).

[0897] Figure 99 This is a flowchart of the decoding process related to the present embodiment. First, the three-dimensional data decoding device extracts the encoded data of the dense three-dimensional point group and the encoded data of the sparse three-dimensional point group from the bit stream (S1751). Then, the three-dimensional data decoding device obtains the decoded data of the dense three-dimensional point group by decoding the encoded data of the dense three-dimensional point group (S1752). In addition, the three-dimensional data decoding device obtains the decoded data of the sparse three-dimensional point group by decoding the encoded data of the sparse three-dimensional point group (S1753). Then, the three-dimensional data decoding device generates a three-dimensional point group by combining the decoded data of the dense three-dimensional point group obtained in step S1752 with the decoded data of the sparse three-dimensional point group obtained in step S1753 (S1754).

[0898] Furthermore, the 3D data encoding device and the 3D data decoding device may encode or decode the dense 3D point group or the sparse 3D point group first. Furthermore, the encoding process or the decoding process may be performed in parallel by multiple processors.

[0899] In addition, the three-dimensional data encoding device can also encode one of a dense three-dimensional point group and a sparse three-dimensional point group. For example, when the dense three-dimensional point group contains important information, the three-dimensional data encoding device extracts the dense three-dimensional point group and the sparse three-dimensional point group from the input three-dimensional point group, encodes the dense three-dimensional point group, and does not encode the sparse three-dimensional point group. In this way, the three-dimensional data encoding device can attach important information to the stream while suppressing the bit amount. For example, between a server and a client, when there is a request from the client to send three-dimensional point group information around the client to the server, the server encodes the important information around the client as a dense three-dimensional point group and sends it to the client. In this way, the server can send the information requested by the client while suppressing the network bandwidth.

[0900] Furthermore, the 3D data decoding device can decode either a dense 3D point group or a sparse 3D point group. For example, if the dense 3D point group contains important information, the 3D data decoding device can decode the dense 3D point group instead of the sparse 3D point group. This allows the 3D data decoding device to obtain the necessary information while reducing the processing load of the decoding process.

[0901] Figure 100 yes Figure 98 Flowchart of the 3D point separation process (S1741) shown in FIG. First, the 3D data encoding device sets the layer L and threshold TH (S1761). Alternatively, the 3D data encoding device may append information indicating the set layer L and threshold TH to the bitstream. In other words, the 3D data encoding device may generate a bitstream containing information indicating the set layer L and threshold TH.

[0902] Next, the three-dimensional data encoding device moves the processing target position from the root node of the octree to the first branch of layer L. That is, the three-dimensional data encoding device selects the first branch of layer L as the processing target branch (S1762).

[0903] Next, the three-dimensional data encoding device counts the number of valid leaf nodes of the target branch of layer L (S1763). If the number of valid leaf nodes of the target branch is greater than the threshold value TH ("Yes" in S1764), the three-dimensional data encoding device registers the target branch as a dense branch with the dense three-dimensional point cloud (S1765). On the other hand, if the number of valid leaf nodes of the target branch is less than the threshold value TH ("No" in S1764), the thr...

Claims

1. A three-dimensional data encoding method, wherein: adding first information indicating a space of first three-dimensional data to a header of the first three-dimensional data, adding second information indicating a space of the second three-dimensional data to a header of the second three-dimensional data, The position information of the first three-dimensional data and the position information of the second three-dimensional data are encoded.

2. The three-dimensional data encoding method according to claim 1, wherein: In the encoding, entropy encoding is performed on the position information of the first three-dimensional data using a first context, and entropy encoding is performed on the position information of the second three-dimensional data using a second context.

3. The three-dimensional data encoding method according to claim 1, wherein: The three-dimensional data is divided into the first three-dimensional data and the second three-dimensional data.

4. The three-dimensional data encoding method according to claim 1, wherein: The second three-dimensional data is not referred to in the encoding of the first three-dimensional data.

5. The three-dimensional data encoding method according to claim 4, wherein: The first three-dimensional data is not referred to in the encoding of the second three-dimensional data.

6. A three-dimensional data decoding method, wherein: acquiring first information indicating a space of the first three-dimensional data from a header of the first three-dimensional data, acquiring second information indicating a space of the second three-dimensional data from a header of the second three-dimensional data, The first encoded data generated by encoding the position information of the first three-dimensional data and the second encoded data generated by encoding the position information of the second three-dimensional data are decoded.

7. The three-dimensional data decoding method according to claim 6, wherein: In the decoding, the first coded data is entropy decoded using the first context, and the second coded data is entropy decoded using the second context.

8. The three-dimensional data decoding method according to claim 6, wherein: The first three-dimensional data and the second three-dimensional data are data obtained by segmenting three-dimensional data.

9. The three-dimensional data decoding method according to claim 6, wherein: The second coded data is not referred to in the decoding of the first coded data.

10. The three-dimensional data decoding method according to claim 9, wherein: The first coded data is not referred to in the decoding of the second coded data.

11. A three-dimensional data encoding device, wherein: have: Processor; and Memory, The processor uses the memory, adding first information indicating a space of first three-dimensional data to a header of the first three-dimensional data, adding second information indicating a space of the second three-dimensional data to a header of the second three-dimensional data, The position information of the first three-dimensional data and the position information of the second three-dimensional data are encoded.

12. A three-dimensional data decoding device, wherein: have: Processor; and Memory, The processor uses the memory, acquiring first information indicating a space of the first three-dimensional data from a header of the first three-dimensional data, acquiring, from a header of the second three-dimensional data, second information indicating a space of the second three-dimensional data, The first encoded data generated by encoding the position information of the first three-dimensional data and the second encoded data generated by encoding the position information of the second three-dimensional data are decoded.

Images