Three-dimensional data encoding method, three-dimensional data decoding method, three-dimensional data encoding device, and three-dimensional data decoding device

Abstract

The invention relates to a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, and a three-dimensional data decoding device. This three-dimensional data encoding method encodes information of a target node included in an N (N is an integer of 2 or more) fork tree structure of a plurality of three-dimensional points included in three-dimensional data using only information of a referenceable node among a plurality of adjacent nodes spatially adjacent to the target node, encodes a parameter, and encodes the encoded parameter using only information of a referenceable node among a plurality of adjacent nodes spatially adjacent to the target node, and encodes the encoded parameter using only information of a referenceable node among a plurality of adjacent nodes spatially adjacent to the target node. And generating a bitstream containing the encoded target node information and the parameter, and if the parameter contains a first value, the referenceable node is only a node that is a sibling node of the target node, and if the parameter contains a second value, generating a bitstream containing the encoded target node information and the parameter, the referenceable node being only a node that is a sibling node of the target node.

Description

[0001] This application is a divisional application of the invention patent application filed on February 7, 2019, with application number 201980023416.6 and entitled "Three-dimensional data encoding method, three-dimensional data decoding method, three-dimensional data encoding device, and three-dimensional data decoding device". Technical Field

[0002] This disclosure relates to a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, and a three-dimensional data decoding device. Background Technology

[0003] In major fields such as computer vision, mapping, surveillance, infrastructure inspection, and image distribution for autonomous operation of automobiles or robots, devices and services that flexibly utilize 3D data will become increasingly common in the future. 3D data is obtained through various methods, including distance sensors such as rangefinders, stereo cameras, or combinations of multiple monocular cameras.

[0004] One method of representing 3D data is the point cloud method, which uses groups of points in 3D space to represent the shape of 3D structures (see, for example, Non-Patent Document 1). The point cloud stores the positions and colors of the point groups. While point clouds are expected to become the mainstream method of representing 3D data, the data volume of point groups is extremely large. Therefore, in the accumulation or transmission of 3D data, similar to 2D dynamic images (for example, MPEG-4 AVC or HEVC standardized by MPEG), data compression through encoding is necessary.

[0005] Furthermore, point cloud compression is partly supported by publicly available libraries (such as the PointCloud Library) that perform point cloud association processing.

[0006] Furthermore, there are well-known techniques for using three-dimensional map data to retrieve and display facilities around a vehicle (for example, see Patent Document 1).

[0007] Existing technical documents Patent documents Patent Document 1 International Publication No. 2014 / 020663 Summary of the Invention

[0008] The problem the invention aims to solve The goal is to improve encoding efficiency and reduce processing volume in the encoding of 3D data.

[0009] The purpose of this disclosure is to provide a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, or a three-dimensional data decoding device that can improve encoding efficiency and reduce processing volume.

[0010] Methods used to solve problems One form of the three-dimensional data encoding method disclosed herein encodes the information of object nodes contained in an N (N is an integer greater than 2)-ary tree structure of multiple three-dimensional points contained in three-dimensional data using only the information of reference nodes among multiple neighboring nodes spatially adjacent to the object node, encoding parameters, and generating a bit stream containing the encoded information of the object node and the parameters. When the parameters contain a first value, the reference node is only a sibling node of the object node.

[0011] One form of the three-dimensional data decoding method disclosed herein obtains parameters from a bitstream and decodes the information of object nodes contained in an N (N is an integer greater than 2)-ary tree structure of multiple three-dimensional points contained in the three-dimensional data using only the information of reference nodes among multiple neighboring nodes that are spatially adjacent to the object node. When the parameter contains a first value, the reference node is only a sibling node of the object node.

[0012] One aspect of the three-dimensional data encoding apparatus disclosed herein includes a processor and a memory. The processor uses the memory to encode information of object nodes contained in an N (N is an integer greater than 2)-ary tree structure of multiple three-dimensional points contained in three-dimensional data, using only the information of reference nodes among multiple adjacent nodes spatially adjacent to the object node, encoding parameters, and generating a bit stream containing the encoded information of the object node and the parameters. When the parameters contain a first value, the reference node is only a sibling node of the object node.

[0013] One aspect of the three-dimensional data decoding apparatus disclosed herein includes a processor and a memory. The processor uses the memory to obtain parameters from a bitstream and decodes information of object nodes contained in an N (N is an integer greater than 2)-ary tree structure of multiple three-dimensional points contained in the three-dimensional data, using only the information of reference nodes among multiple neighboring nodes that are spatially adjacent to the object node. When the parameters include a first value, the reference node is only a sibling node of the object node.

[0014] Invention Effects This disclosure provides a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, or a three-dimensional data decoding device that can improve encoding efficiency and reduce processing volume. Attached Figure Description

[0015] Figure 1 The structure of the encoded three-dimensional data involved in Implementation 1 is shown.

[0016] Figure 2 An example of the prediction structure between SPCs at the lowest level of the GOS involved in Implementation 1 is shown.

[0017] Figure 3 An example of the interlayer prediction structure involved in Implementation 1 is shown.

[0018] Figure 4 An example of the encoding order of the GOS involved in Implementation 1 is shown.

[0019] Figure 5 An example of the encoding order of the GOS involved in Implementation 1 is shown.

[0020] Figure 6 This is a block diagram of the three-dimensional data encoding device according to Embodiment 1.

[0021] Figure 7 This is a flowchart of the encoding process involved in Implementation Method 1.

[0022] Figure 8 This is a block diagram of the three-dimensional data decoding device according to Embodiment 1.

[0023] Figure 9 This is a flowchart of the decoding process involved in Implementation Method 1.

[0024] Figure 10 An example of the metadata involved in Implementation 1 is shown.

[0025] Figure 11 An example of the configuration of the SWLD involved in Embodiment 2 is shown.

[0026] Figure 12 An example of the operation of the server and client involved in Implementation Method 2 is shown.

[0027] Figure 13 An example of the operation of the server and client involved in Implementation Method 2 is shown.

[0028] Figure 14 An example of the operation of the server and client involved in Implementation Method 2 is shown.

[0029] Figure 15 An example of the operation of the server and client involved in Implementation Method 2 is shown.

[0030] Figure 16 This is a block diagram of the three-dimensional data encoding device involved in Embodiment 2.

[0031] Figure 17 This is a flowchart of the encoding process involved in Implementation Method 2.

[0032] Figure 18 This is a block diagram of the three-dimensional data decoding device involved in Embodiment 2.

[0033] Figure 19 This is a flowchart of the decoding process involved in Implementation Method 2.

[0034] Figure 20 An example of the configuration of the WLD involved in Embodiment 2 is shown.

[0035] Figure 21 An example of the octree structure of the WLD involved in Embodiment 2 is shown.

[0036] Figure 22 An example of the configuration of the SWLD involved in Embodiment 2 is shown.

[0037] Figure 23 An example of the octree structure of the SWLD involved in Implementation 2 is shown.

[0038] Figure 24 This is a block diagram of the three-dimensional data production apparatus according to Embodiment 3.

[0039] Figure 25 This is a block diagram of the three-dimensional data transmission device according to Embodiment 3.

[0040] Figure 26 This is a block diagram of the three-dimensional information processing device involved in Embodiment 4.

[0041] Figure 27 This is a block diagram of the three-dimensional data production apparatus according to Embodiment 5.

[0042] Figure 28 The configuration of the system involved in Implementation 6 is shown.

[0043] Figure 29 This is a block diagram of the client device according to Embodiment 6.

[0044] Figure 30 This is a block diagram of the server involved in implementation method 6.

[0045] Figure 31 This is a flowchart of the three-dimensional data creation process performed by the client device involved in Implementation 6.

[0046] Figure 32 This is a flowchart of the sensor information transmission process performed by the client device according to Embodiment 6.

[0047] Figure 33 This is a flowchart of the three-dimensional data creation process performed by the server involved in Implementation Method 6.

[0048] Figure 34 This is a flowchart of the three-dimensional map transmission process performed by the server involved in Implementation Method 6.

[0049] Figure 35 The configuration of a modified example of the system according to Embodiment 6 is shown.

[0050] Figure 36 The configuration of the server and client device involved in Embodiment 6 is shown.

[0051] Figure 37 This is a block diagram of the three-dimensional data encoding device according to Embodiment 7.

[0052] Figure 38 An example of the predicted residuals involved in Implementation 7 is shown.

[0053] Figure 39 An example of the volume involved in Embodiment 7 is shown.

[0054] Figure 40 An example of the octree representation of the volume involved in Implementation 7 is shown.

[0055] Figure 41 An example of a bit string of the volume involved in Implementation 7 is shown.

[0056] Figure 42 An example of the octree representation of the volume involved in Implementation 7 is shown.

[0057] Figure 43 An example of the volume involved in Embodiment 7 is shown.

[0058] Figure 44 This is a diagram used to illustrate the intra-frame prediction processing involved in Implementation 7.

[0059] Figure 45 This is a diagram used to illustrate the rotation and translation processes involved in Embodiment 7.

[0060] Figure 46 Examples of RT application flags and RT information syntax involved in Implementation 7 are shown.

[0061] Figure 47 This is a diagram used to illustrate the inter-frame prediction processing involved in Implementation 7.

[0062] Figure 48 This is a block diagram of the three-dimensional data decoding device according to Embodiment 7.

[0063] Figure 49 This is a flowchart of the three-dimensional data encoding process performed by the three-dimensional data encoding apparatus according to Embodiment 7.

[0064] Figure 50 This is a flowchart of the three-dimensional data decoding process performed by the three-dimensional data decoding apparatus according to Embodiment 7.

[0065] Figure 51 The configuration of the distribution system according to Embodiment 8 is shown.

[0066] Figure 52 An example of the configuration of the bitstream of the encoded three-dimensional map according to Embodiment 8 is shown.

[0067] Figure 53 This is a diagram used to illustrate the improvement in coding efficiency involved in Implementation Method 8.

[0068] Figure 54 This is a flowchart of the processing performed by the server according to Implementation Method 8.

[0069] Figure 55 This is a flowchart of the processing performed by the client according to Implementation Method 8.

[0070] Figure 56 Syntactic examples of the submaps involved in Implementation 8 are shown.

[0071] Figure 57 The switching process of the encoding type involved in Implementation 8 is shown in the pattern.

[0072] Figure 58 Syntactic examples of the submaps involved in Implementation 8 are shown.

[0073] Figure 59 This is a flowchart of the three-dimensional data encoding process involved in Implementation Method 8.

[0074] Figure 60 This is a flowchart of the three-dimensional data decoding process involved in Implementation Method 8.

[0075] Figure 61 The working model illustrates a variation of the encoding type switching process involved in Implementation 8.

[0076] Figure 62 The working model illustrates a variation of the encoding type switching process involved in Implementation 8.

[0077] Figure 63 The working model illustrates a variation of the encoding type switching process involved in Implementation 8.

[0078] Figure 64 The working model shows a modified example of the differential value calculation process involved in Implementation 8.

[0079] Figure 65 The working model shows a modified example of the differential value calculation process involved in Implementation 8.

[0080] Figure 66 The working model shows a modified example of the differential value calculation process involved in Implementation 8.

[0081] Figure 67 The working model shows a modified example of the differential value calculation process involved in Implementation 8.

[0082] Figure 68 Syntactic examples of the volume involved in Implementation 8 are shown.

[0083] Figure 69 This is a diagram showing an example of an important area involved in Implementation Method 9.

[0084] Figure 70 This is a diagram illustrating an example of the occupancy code involved in Implementation Method 9.

[0085] Figure 71 This is a diagram illustrating an example of the quadtree structure involved in Implementation 9.

[0086] Figure 72 This is a diagram illustrating an example of the occupancy rate code and location code involved in Implementation Method 9.

[0087] Figure 73 This is a diagram showing an example of three-dimensional points obtained in LiDAR according to Embodiment 9.

[0088] Figure 74 This is a diagram illustrating an example of the octree structure involved in Implementation 9.

[0089] Figure 75 This is a diagram illustrating an example of the hybrid encoding involved in Implementation 9.

[0090] Figure 76 This is a diagram illustrating the method for switching between position coding and occupancy coding according to Implementation Method 9.

[0091] Figure 77 This is a diagram illustrating an example of a position-encoded bitstream according to Implementation Method 9.

[0092] Figure 78 This is a diagram illustrating an example of a hybrid encoded bitstream according to Implementation Method 9.

[0093] Figure 79 It is a tree structure diagram representing the occupancy codes of the important three-dimensional points involved in Implementation Method 9.

[0094] Figure 80 It is a tree structure diagram representing the occupancy codes of non-critical three-dimensional points involved in Implementation 9.

[0095] Figure 81 This is a diagram illustrating an example of a hybrid encoded bitstream according to Implementation Method 9.

[0096] Figure 82 This is a diagram illustrating an example of a bitstream containing encoding pattern information according to Implementation Method 9.

[0097] Figure 83 This is a diagram illustrating the syntax examples involved in Implementation Method 9.

[0098] Figure 84 This is a flowchart of the encoding process involved in Implementation Method 9.

[0099] Figure 85 This is a flowchart of the node encoding process involved in Implementation Method 9.

[0100] Figure 86 This is a flowchart of the decoding process involved in Implementation Method 9.

[0101] Figure 87 This is a flowchart of the node decoding process involved in Implementation Method 9.

[0102] Figure 88 This is a diagram illustrating an example of the tree structure involved in Implementation 10.

[0103] Figure 89 This is a diagram illustrating an example of the number of effective leaf nodes in each branch involved in Implementation 10.

[0104] Figure 90 This is a diagram illustrating an application example of the encoding method involved in Implementation Method 10.

[0105] Figure 91 This is a diagram illustrating an example of a dense branching region involved in Implementation 10.

[0106] Figure 92 This is a diagram illustrating an example of a dense three-dimensional point group involved in Implementation 10.

[0107] Figure 93 This is a diagram illustrating an example of a sparse three-dimensional point group involved in Implementation 10.

[0108] Figure 94 This is a flowchart of the encoding process involved in Implementation 10.

[0109] Figure 95 This is a flowchart of the decoding process involved in Implementation Method 10.

[0110] Figure 96 This is a flowchart of the encoding process involved in Implementation 10.

[0111] Figure 97 This is a flowchart of the decoding process involved in Implementation Method 10.

[0112] Figure 98 This is a flowchart of the encoding process involved in Implementation 10.

[0113] Figure 99 This is a flowchart of the decoding process involved in Implementation Method 10.

[0114] Figure 100 This is a flowchart illustrating the separation process of three-dimensional points involved in Implementation 10.

[0115] Figure 101 This is a diagram illustrating the syntax examples involved in implementation method 10.

[0116] Figure 102 This is a diagram illustrating an example of dense branches involved in Implementation 10.

[0117] Figure 103 This is a diagram illustrating an example of sparse branches involved in implementation 10.

[0118] Figure 104 This is a flowchart of the encoding process of the variant example involved in Implementation 10.

[0119] Figure 105 This is a flowchart of the decoding process of the variant example involved in Implementation 10.

[0120] Figure 106 This is a flowchart of the three-dimensional point separation process in the modified example of Implementation 10.

[0121] Figure 107 This is a diagram illustrating the syntax of a variant example involved in Implementation 10.

[0122] Figure 108 This is a flowchart of the encoding process involved in Implementation 10.

[0123] Figure 109 This is a flowchart of the decoding process involved in Implementation Method 10.

[0124] Figure 110 This is a diagram illustrating an example of the tree structure involved in Implementation 11.

[0125] Figure 111 This is a diagram illustrating an example of the occupancy code involved in Implementation 11.

[0126] Figure 112 This is a diagram schematically illustrating the operation of the three-dimensional data encoding device according to Embodiment 11.

[0127] Figure 113 This is a diagram illustrating an example of the geometric information involved in Implementation 11.

[0128] Figure 114 This is a diagram illustrating an example of selecting an encoding table using geometric information according to Implementation Method 11.

[0129] Figure 115 This is a diagram illustrating an example of selecting an encoding table using structural information according to Implementation Method 11.

[0130] Figure 116 This is a diagram illustrating an example of selecting an encoding table for using attribute information according to Implementation Method 11.

[0131] Figure 117 This is a diagram illustrating an example of selecting an encoding table for using attribute information according to Implementation Method 11.

[0132] Figure 118 This is a diagram illustrating a structural example of the bitstream involved in Implementation 11.

[0133] Figure 119 This is a diagram illustrating an example of the encoding table involved in Implementation 11.

[0134] Figure 120 This is a diagram illustrating an example of the encoding table involved in Implementation 11.

[0135] Figure 121 This is a diagram illustrating a structural example of the bitstream involved in Implementation 11.

[0136] Figure 122 This is a diagram illustrating an example of the encoding table involved in Implementation 11.

[0137] Figure 123 This is a diagram illustrating an example of the encoding table involved in Implementation 11.

[0138] Figure 124 This is a diagram showing an example of the bit number of the occupancy code involved in Implementation 11.

[0139] Figure 125 This is a flowchart of the encoding process using geometric information involved in Implementation 11.

[0140] Figure 126 This is a flowchart of the decoding process using geometric information involved in Implementation 11.

[0141] Figure 127This is a flowchart of the encoding process using structural information involved in Implementation Method 11.

[0142] Figure 128 This is a flowchart of the decoding process using structural information involved in Implementation 11.

[0143] Figure 129 This is a flowchart of the encoding process using attribute information involved in Implementation Method 11.

[0144] Figure 130 This is a flowchart of the decoding process using attribute information involved in Implementation Method 11.

[0145] Figure 131 This is a flowchart of the encoding table selection process using geometric information involved in Implementation Method 11.

[0146] Figure 132 This is a flowchart of the encoding table selection process using structural information involved in Implementation Method 11.

[0147] Figure 133 This is a flowchart of the encoding table selection process using attribute information involved in Implementation Method 11.

[0148] Figure 134 This is a block diagram of the three-dimensional data encoding apparatus according to Embodiment 11.

[0149] Figure 135 This is a block diagram of the three-dimensional data decoding apparatus according to Embodiment 11.

[0150] Figure 136 This is a diagram showing the reference relationships in the octree structure involved in Implementation 12.

[0151] Figure 137 It is a diagram showing the reference relationships in the spatial regions involved in Implementation 12.

[0152] Figure 138 This is a diagram showing an example of adjacent reference nodes involved in Implementation 12.

[0153] Figure 139 This is a diagram showing the relationship between the parent node and the node involved in implementation method 12.

[0154] Figure 140 This is a diagram illustrating an example of the occupancy code of the parent node involved in Implementation 12.

[0155] Figure 141 This is a block diagram illustrating the three-dimensional data encoding apparatus according to Embodiment 12.

[0156] Figure 142This is a block diagram illustrating the three-dimensional data decoding apparatus according to Embodiment 12.

[0157] Figure 143 This is a flowchart illustrating the three-dimensional data encoding process involved in Implementation Method 12.

[0158] Figure 144 This is a flowchart illustrating the three-dimensional data decoding process involved in Implementation Method 12.

[0159] Figure 145 This is a diagram illustrating an example of switching the encoding table according to Implementation Method 12.

[0160] Figure 146 This is a diagram showing the reference relationships in the spatial region involved in the variation 1 of embodiment 12.

[0161] Figure 147 This is a diagram illustrating the syntax of the header information involved in a variation of implementation 12.

[0162] Figure 148 This is a diagram illustrating the syntax of the header information involved in a variation of implementation 12.

[0163] Figure 149 This is a diagram illustrating an example of adjacent reference nodes involved in a variation 2 of implementation 12.

[0164] Figure 150 This is a diagram illustrating an example of the object node and adjacent nodes involved in a variation of embodiment 12, namely, Example 2.

[0165] Figure 151 This is a diagram showing the reference relationships in the octree structure involved in Variation 3 of Implementation Method 12.

[0166] Figure 152 This is a diagram showing the reference relationships in the spatial region involved in the variation 3 of embodiment 12. Detailed Implementation

[0167] One aspect of the three-dimensional data encoding method disclosed herein is to encode information of object nodes contained in an N (N is an integer greater than 2)-ary tree structure of multiple three-dimensional points contained in three-dimensional data. In the encoding, reference is permitted to the information of a first node among multiple spatially adjacent nodes that is the same as the parent node of the object node, while reference is prohibited to the information of a second node whose parent node is different from the parent node of the object node.

[0168] Therefore, this 3D data encoding method can improve encoding efficiency by referencing information of the first node among multiple spatially adjacent nodes whose parent node is the same as the parent node of the object node. Furthermore, since this 3D data encoding method does not refer to information of the second node among multiple adjacent nodes whose parent node is different from the parent node of the object node, it can reduce the processing load. Thus, this 3D data encoding method can improve encoding efficiency and reduce processing load.

[0169] For example, the three-dimensional data encoding method further determines whether to prohibit referencing the information of the second node. In the encoding, based on the result of the determination, a switch is made to prohibit or permit referencing the information of the second node. The three-dimensional data encoding method further generates a bit stream containing prohibition switching information, where the prohibition switching information is the result of the determination and indicates whether referencing the information of the second node is prohibited.

[0170] Therefore, this 3D data encoding method can switch between prohibiting reference to the information of the second node. Furthermore, the 3D data decoding device can appropriately perform decoding processing using the prohibition switching information.

[0171] For example, the information of the object node may be information indicating whether there are three-dimensional points in each of the child nodes belonging to the object node, the information of the first node may be information indicating whether there are three-dimensional points in the first node, and the information of the second node may be information indicating whether there are three-dimensional points in the second node.

[0172] For example, in the encoding, an encoding table can be selected based on whether a three-dimensional point exists in the first node, and the selected encoding table can be used to entropy encode the information of the object node.

[0173] For example, in the encoding, it is permissible to refer to the information of the child nodes of the first node among the plurality of adjacent nodes.

[0174] Therefore, this three-dimensional data encoding method can refer to more detailed information from adjacent nodes, thus improving encoding efficiency.

[0175] For example, in the encoding, the referenced neighboring node among the multiple neighboring nodes can be switched according to the spatial position within the parent node of the object node.

[0176] Therefore, this three-dimensional data encoding method can refer to appropriate adjacent nodes based on the spatial position of the object node within its parent node.

[0177] One aspect of the three-dimensional data decoding method disclosed herein is to decode information of object nodes contained in an N (N is an integer greater than 2)-ary tree structure of multiple three-dimensional points contained in three-dimensional data. In the decoding, it is permissible to refer to the information of a first node among multiple adjacent nodes that are spatially adjacent to the object node, whose parent node is the same as the parent node of the object node. It is prohibited to refer to the information of a second node whose parent node is different from the parent node of the object node.

[0178] Therefore, this 3D data decoding method can improve encoding efficiency by referring to the information of the first node among multiple spatially adjacent nodes whose parent node is the same as the parent node of the object node. Furthermore, since this 3D data decoding method does not refer to the information of the second node among multiple adjacent nodes whose parent node is different from the parent node of the object node, it can reduce the processing load. Thus, this 3D data decoding method can improve encoding efficiency and reduce processing load.

[0179] For example, the three-dimensional data decoding method may further obtain prohibition switching information from the bitstream, indicating whether referencing the information of the second node is prohibited. In the decoding, based on the prohibition switching information, the method determines whether referencing the information of the second node is prohibited or permitted.

[0180] Therefore, this three-dimensional data decoding method can appropriately perform decoding processing using the prohibition switching information.

[0181] For example, the information of the object node may be information indicating whether there are three-dimensional points in each of the child nodes belonging to the object node, the information of the first node may be information indicating whether there are three-dimensional points in the first node, and the information of the second node may be information indicating whether there are three-dimensional points in the second node.

[0182] For example, in the decoding, an encoding table can be selected based on whether a three-dimensional point exists in the first node, and the selected encoding table can be used to perform entropy decoding on the information of the object node.

[0183] For example, in the decoding, it is permissible to refer to the information of the child nodes of the first node among the plurality of adjacent nodes.

[0184] Therefore, this three-dimensional data decoding method can refer to more detailed information from adjacent nodes, thus improving the efficiency of reference coding.

[0185] For example, in the decoding process, the referenced neighboring node among the multiple neighboring nodes can be switched according to the spatial position within the parent node of the object node.

[0186] Therefore, this 3D data decoding method can refer to appropriate adjacent nodes based on the spatial position of the object node within its parent node.

[0187] Additionally, one aspect of the three-dimensional data encoding apparatus disclosed herein includes a processor and a memory. The processor uses the memory to encode information of object nodes contained in an N (N is an integer greater than 2)-ary tree structure of multiple three-dimensional points contained in three-dimensional data. In this encoding, reference is permitted to the information of a first node among multiple spatially adjacent nodes that is the same as the parent node of the object node, while reference is prohibited to the information of a second node whose parent node is different from the parent node of the object node.

[0188] Therefore, this 3D data encoding device improves encoding efficiency by referencing information of the first node among multiple spatially adjacent nodes whose parent node is the same as the parent node of the object node. Furthermore, by not referencing information of the second node among multiple adjacent nodes whose parent node is different from the parent node of the object node, this 3D data encoding device reduces processing load. Thus, this 3D data encoding device can improve encoding efficiency and reduce processing load.

[0189] Additionally, one aspect of the three-dimensional data decoding apparatus disclosed herein includes a processor and a memory. The processor uses the memory to decode information of object nodes contained in an N (N is an integer greater than 2)-ary tree structure of multiple three-dimensional points contained in the three-dimensional data. In the decoding, reference is permitted to the information of a first node among multiple spatially adjacent nodes that is the same as the parent node of the object node, while reference is prohibited to the information of a second node whose parent node is different from the parent node of the object node.

[0190] Therefore, this 3D data decoding device improves encoding efficiency by referencing information of the first node among multiple spatially adjacent nodes whose parent node is the same as the parent node of the object node. Furthermore, by not referencing information of the second node among multiple adjacent nodes whose parent node is different from the parent node of the object node, the device reduces processing load. Thus, this 3D data decoding device improves encoding efficiency and reduces processing load.

[0191] Furthermore, these general or specific forms can be realized by systems, methods, integrated circuits, computer programs, or computer-readable recording media such as CD-ROMs, and can be realized by any combination of systems, methods, integrated circuits, computer programs, and recording media.

[0192] The embodiments will now be described in detail with reference to the accompanying drawings. Furthermore, the embodiments described below are all specific examples illustrating this disclosure. The numerical values, shapes, materials, constituent elements, arrangement positions of constituent elements, connection methods, steps, and order of steps shown in the following embodiments are all examples and are not intended to limit this disclosure. Moreover, constituent elements not described in the technical solution illustrating the highest-level concept among the constituent elements of the following embodiments are described as arbitrary constituent elements.

[0193] (Implementation Method 1) First, the data structure of the encoded three-dimensional data (hereinafter also referred to as encoded data) involved in this embodiment will be explained. Figure 1 The structure of the encoded three-dimensional data involved in this embodiment is shown.

[0194] In this embodiment, the three-dimensional space is divided into spatial units (SPCs) equivalent to image space in motion image coding, and the three-dimensional data is encoded on a spatial basis. The space is further divided into volumes (VLMs) equivalent to macroblocks in motion image coding, and prediction and transformation are performed on a VLM basis. A volume includes multiple voxels (VXLs), the smallest unit corresponding to position coordinates. Furthermore, prediction refers to generating predicted three-dimensional data similar to the processing unit of the object being processed, referencing other processing units, similar to the prediction performed in two-dimensional images, and encoding the differences between the predicted three-dimensional data and the processing unit of the object being processed. Moreover, this prediction includes not only spatial prediction referencing other prediction units at the same time, but also temporal prediction referencing prediction units at different times.

[0195] For example, when encoding a three-dimensional space represented by point cloud or other point group data, a three-dimensional data encoding device (hereinafter also referred to as an encoding device) encodes each point in the point group or multiple points contained within a voxel according to the size of the voxel. If the voxels are subdivided, the three-dimensional shape of the point group can be represented with high precision; if the size of the voxels is increased, the three-dimensional shape of the point group can be represented with coarse precision.

[0196] Furthermore, although the following explanation uses the case of 3D data as point clouds as an example, 3D data is not limited to point clouds and can be any form of 3D data.

[0197] Furthermore, voxels with hierarchical structures can be utilized. In this case, within an nth-order hierarchy, the presence or absence of sampling points in the (n-1)th-order and lower-order hierarchy (the lower layers of the nth-order hierarchy) can be sequentially shown. For example, when decoding only the nth-order hierarchy, if sampling points exist in the (n-1)th-order and lower-order hierarchy, the decoding can be performed by considering the center of the voxel in the nth-order hierarchy as the location of the sampling point.

[0198] Furthermore, the encoding device obtains point group data through distance sensors, stereo cameras, SLR cameras, gyroscopes, or inertial sensors.

[0199] Regarding spatial representation, similar to the coding of moving images, it is classified into at least one of the following three prediction structures: intra-frame spatial representation (I-SPC), which can be decoded independently; prediction spatial representation (P-SPC), which can only be referenced in one direction; and bidirectional spatial representation (B-SPC), which can be referenced in both directions. Furthermore, the spatial representation contains both decoding and display time information.

[0200] And, as Figure 1 As shown, as a processing unit comprising multiple spaces, there is a Group of Spaces (GOS) which serves as a random access unit. Furthermore, as a processing unit comprising multiple GOS, there exists a World Space (WLD).

[0201] The spatial regions occupied by the world are mapped to absolute locations on Earth using GPS, latitude, and longitude information. This location information is stored as metadata. Furthermore, metadata can be included within coded data or transmitted separately from it.

[0202] Furthermore, within GOS, all SPCs can be three-dimensionally adjacent, or they can exist in a way that is not three-dimensionally adjacent to other SPCs.

[0203] Furthermore, the encoding, decoding, or referencing processes corresponding to the 3D data contained in processing units such as GOS, SPC, or VLM will also be simply referred to as encoding, decoding, or referencing the processing unit. The 3D data contained in the processing unit includes, for example, at least one set of characteristic values ​​such as spatial position (3D coordinates) and color information.

[0204] Next, the prediction structure of SPC in GOS will be explained. Multiple SPCs within the same GOS, or multiple VLMs within the same SPC, although occupying different spaces, hold the same timing information (decoding timing and display timing).

[0205] Furthermore, within the GOS, the SPC that begins the decoding order is the I-SPC. There are also two types of GOS: closed GOS and open GOS. A closed GOS is a GOS that can decode all SPCs within the GOS starting from the first I-SPC. In an open GOS, a subset of SPCs whose display time is earlier than the first I-SPC references a different GOS and can only be decoded within that GOS.

[0206] Furthermore, in encoded data such as map information, there are cases where WLD is decoded in the reverse direction of the encoding order. If there is a dependency between GOS, it is difficult to perform reverse regeneration. Therefore, in such cases, a closed GOS is generally used.

[0207] Furthermore, GOS has a layered structure in the height direction, and encoding or decoding is performed sequentially starting from the SPC of the bottom layer.

[0208] Figure 2 An example of the prediction structure between SPCs in the lowest layer of GOS is shown. Figure 3 An example of the predicted structure between layers is shown.

[0209] There is more than one I-SPC within a GOS. Although objects such as people, animals, cars, bicycles, traffic lights, or buildings that serve as land landmarks exist in three-dimensional space, encoding small objects as I-SPCs is particularly effective. For example, a 3D data decoding device (hereinafter also referred to as the decoding device) decodes only the I-SPCs within the GOS when decoding the GOS with low processing power or high speed.

[0210] Furthermore, the encoding device can switch the encoding interval or occurrence frequency of I-SPC according to the density of objects within the WLD.

[0211] Furthermore, in Figure 3 In the configuration shown, the encoding or decoding device encodes or decodes multiple layers sequentially, starting from the lower layer (layer 1). Accordingly, for example, for autonomous vehicles, the priority of data near the ground, which contains a large amount of information, can be increased.

[0212] In addition, in the encoded data used by drones, etc., within GOS, encoding or decoding can be performed sequentially starting from the SPC of the upper layer in the height direction.

[0213] Furthermore, the encoding or decoding device can encode or decode multiple layers in a manner that allows the decoding device to roughly grasp the GOS and gradually increase the resolution. For example, the encoding or decoding device can encode or decode in the order of layers 3, 8, 1, 9...

[0214] Next, the corresponding methods for static and dynamic objects will be explained.

[0215] In three-dimensional space, there exist static objects or scenes such as buildings or roads (hereinafter referred to as static objects) and dynamic objects such as vehicles or people (hereinafter referred to as dynamic objects). Object detection can be performed separately by extracting feature points from point cloud data or images captured by stereo cameras. Here, an example of an encoding method for dynamic objects is illustrated.

[0216] The first method is to encode objects without distinguishing between static and dynamic objects. The second method is to distinguish between static and dynamic objects by identifying information.

[0217] For example, GOS is used as the identification unit. In this case, the GOS that constitutes the SPC of a static object and the GOS that constitutes the SPC of a dynamic object are distinguished either within the encoded data or by identification information stored separately from the encoded data.

[0218] Alternatively, the SPC can be used as the identification unit. In this case, the SPC that includes only the VLM constituting a static object and the SPC that includes the VLM constituting a dynamic object are distinguished by the identification information described above.

[0219] Alternatively, VLM or VXL can be used as the identification unit. In this case, VLM or VXL including static objects and VLM or VXL including dynamic objects are distinguished by the identification information described above.

[0220] Furthermore, the encoding device can encode a dynamic object as one or more VLMs or SPCs, and encode VLMs or SPCs including static objects and SPCs including dynamic objects as different GOSs. Moreover, when the size of the GOS becomes variable according to the size of the dynamic object, the encoding device stores the size of the GOS separately as metadata.

[0221] Furthermore, the encoding device encodes static and dynamic objects independently, allowing dynamic objects to overlap within a world space composed of static objects. In this case, a dynamic object is composed of one or more SPCs, and each SPC corresponds to one or more SPCs of the static object that overlaps with it. Alternatively, a dynamic object may not be represented by an SPC, but rather by one or more VLMs or VXLs.

[0222] Furthermore, the encoding device can encode static objects and dynamic objects as distinct streams.

[0223] Furthermore, the encoding device can also generate GOS that includes one or more SPCs constituting a dynamic object. Moreover, the encoding device can set the GOS (GOS_M) including the dynamic object and the GOS of the static object corresponding to the spatial region of GOS_M to be of the same size (occupying the same spatial region). In this way, overlapping processing can be performed on a GOS-by-GOS basis.

[0224] The P-SPC or B-SPC that constitutes a dynamic object can also refer to the SPCs contained in different encoded GOS. Since the position of a dynamic object changes over time, and the same dynamic object is encoded as a GOS at different times, referencing across GOSs is effective from a compression ratio perspective.

[0225] Furthermore, the first and second methods described above can be switched depending on the intended use of the encoded data. For example, when encoding 3D data for use as a map, the encoding device uses the second method because it is desirable to separate it from dynamic objects. Conversely, when encoding 3D data for events such as concerts or sporting events, the encoding device uses the first method if it is not necessary to separate dynamic objects.

[0226] Furthermore, the decoding and display times of GOS or SPC can be stored within the encoded data or as metadata. Also, the timing information for static objects can all be identical. In this case, the actual decoding and display times can be determined by the decoding device. Alternatively, different values ​​can be assigned to each GOS or SPC as the decoding time, while the same value can be assigned to all display times. Moreover, as shown in the decoder modes of motion graphics coding such as HEVC's HRD (Hypothetical Reference Decoder), the decoder has a buffer of a specified size. As long as the bitstream is read at a specified bit rate according to the decoding time, a model that will not be corrupted and can be decoded can be imported.

[0227] Next, the configuration of GOS within world space will be explained. The three-dimensional coordinates in world space are represented by three mutually orthogonal coordinate axes (x-axis, y-axis, and z-axis). By setting prescribed rules in the encoding order of GOS, spatially adjacent GOS can be encoded consecutively within the encoded data. For example, in... Figure 4 In the example shown, the World Space (GOS) within the xz plane is encoded sequentially. After encoding all GOS within an xz plane, the y-axis value is updated. That is, as encoding continues, the world space extends towards the y-axis. Furthermore, the index number of the GOS is set as the encoding order.

[0228] Here, the three-dimensional space of the world corresponds one-to-one with absolute geographical coordinates such as GPS, latitude, and longitude. Alternatively, the three-dimensional space can be represented by relative positions relative to a pre-defined reference position. The directions of the x, y, and z axes of the three-dimensional space are represented as direction vectors determined based on latitude and longitude, and these direction vectors are stored as metadata along with the encoded data.

[0229] Furthermore, the size of the GOS is set to a fixed value, and the encoding device stores this size as metadata. The size of the GOS can be switched, for example, depending on whether it is indoors or outdoors, or whether it is in a city. That is, the size of the GOS can be switched according to the quantity or nature of objects with informational value. Alternatively, the encoding device can appropriately switch the size of the GOS or the interval of the I-SPCs within the GOS according to the density of objects within the same world space. For example, the higher the object density, the smaller the size of the GOS and the shorter the interval of the I-SPCs within the GOS.

[0230] exist Figure 5 In the example, in the region from the 3rd to the 10th GOS, due to the high density of objects, the GOS is subdivided to achieve fine-grained random access. Furthermore, the 7th to 10th GOS are located on the back sides of the 3rd to 6th GOS, respectively.

[0231] Next, the structure and operation flow of the three-dimensional data encoding device involved in this embodiment will be explained. Figure 6 This is a block diagram of the three-dimensional data encoding device 100 according to this embodiment. Figure 7 This is a flowchart illustrating an example of the operation of the three-dimensional data encoding device 100.

[0232] Figure 6 The 3D data encoding apparatus 100 shown generates encoded 3D data 112 by encoding 3D data 111. The 3D data encoding apparatus 100 includes: an acquisition unit 101, an encoding region determination unit 102, a segmentation unit 103, and an encoding unit 104.

[0233] like Figure 7 As shown, firstly, the acquisition unit 101 acquires three-dimensional data 111 as point group data (S101).

[0234] Next, the encoding region determination unit 102 determines the region of the encoding object from the spatial region corresponding to the obtained point group data (S102). For example, the encoding region determination unit 102 determines the spatial region surrounding the user or vehicle's location as the region of the encoding object.

[0235] Next, the segmentation unit 103 divides the point group data contained in the region of the encoding object into individual processing units. Here, the processing units are the aforementioned GOS and SPC, etc. Furthermore, the region of the encoding object corresponds, for example, to the aforementioned world space. Specifically, the segmentation unit 103 divides the point group data into processing units based on a pre-set GOS size, the presence or size of dynamic objects (S103). Furthermore, the segmentation unit 103 determines the starting position of the SPC that will be the first in the encoding sequence within each GOS.

[0236] Next, the encoding unit 104 generates encoded three-dimensional data 112 by sequentially encoding multiple SPCs within each GOS (S104).

[0237] Furthermore, although an example of encoding each GOS is shown here after dividing the region of the encoded object into GOS and SPC, the processing order is not limited to the above. For example, a GOS can be encoded after its structure is determined, and then the order of GOS structure can be determined afterward.

[0238] In this way, the 3D data encoding device 100 generates encoded 3D data 112 by encoding the 3D data 111. Specifically, the 3D data encoding device 100 divides the 3D data into random access units, that is, into first processing units (GOS) corresponding to 3D coordinates, then divides the first processing units (GOS) into multiple second processing units (SPC), and then divides the second processing units (SPC) into multiple third processing units (VLM). Furthermore, each third processing unit (VLM) includes one or more voxels (VXL), where a voxel (VXL) is the smallest unit corresponding to position information.

[0239] Next, the 3D data encoding device 100 generates encoded 3D data 112 by encoding each of the plurality of first processing units (GOS). Specifically, the 3D data encoding device 100 encodes each of the plurality of second processing units (SPC) in each of the first processing units (GOS). Furthermore, the 3D data encoding device 100 encodes each of the plurality of third processing units (VLM) in each of the second processing units (SPC).

[0240] For example, when the first processing unit (GOS) of the object being processed is a closed GOS, the 3D data encoding apparatus 100 encodes the second processing unit (SPC) of the object being processed, which is contained within the first processing unit (GOS), by referring to other second processing units (SPCs) contained within the first processing unit (GOS). That is, the 3D data encoding apparatus 100 does not refer to second processing units (SPCs) contained within a first processing unit (GOS) that is different from the first processing unit (GOS) of the object being processed.

[0241] Furthermore, when the first processing unit (GOS) of the processing object is an open GOS, the second processing unit (SPC) of the processing object contained in the first processing unit (GOS) of the processing object is encoded with reference to other second processing units (SPCs) contained in the first processing unit (GOS) of the processing object, or second processing units (SPCs) contained in a first processing unit (GOS) different from the first processing unit (GOS) of the processing object.

[0242] Furthermore, the three-dimensional data encoding device 100 selects one of the following as the type of the second processing unit (SPC) of the processing object: a first type (I-SPC) that does not refer to other second processing units (SPCs), a second type (P-SPC) that refers to one other second processing unit (SPC), and a third type that refers to two other second processing units (SPCs), and encodes the second processing unit (SPC) of the processing object according to the selected type.

[0243] Next, the configuration and operation flow of the three-dimensional data decoding device involved in this embodiment will be described. Figure 8 This is a block diagram of the three-dimensional data decoding device 200 involved in this embodiment. Figure 9 This is a flowchart illustrating an example of the operation of the three-dimensional data decoding device 200.

[0244] Figure 8 The 3D data decoding apparatus 200 shown generates decoded 3D data 212 by decoding encoded 3D data 211. Here, encoded 3D data 211 is, for example, encoded 3D data 112 generated by the 3D data encoding apparatus 100. The 3D data decoding apparatus 200 includes: an acquisition unit 201, a decoding start GOS determination unit 202, a decoding SPC determination unit 203, and a decoding unit 204.

[0245] First, the acquisition unit 201 acquires the encoded three-dimensional data 211 (S201). Next, the decoding start GOS determination unit 202 determines the GOS of the decoding object (S202). Specifically, the decoding start GOS determination unit 202 refers to the metadata stored in or separately from the encoded three-dimensional data 211, and determines the GOS of the decoding object, including the spatial position of the start of decoding, the object, or the SPC corresponding to the time.

[0246] Next, the SPC decoding decision unit 203 determines the type (I, P, B) of the SPC to be decoded within the GOS (S203). For example, the SPC decoding decision unit 203 determines (1) whether to decode only I-SPC, (2) whether to decode both I-SPC and P-SPC, and (3) whether to decode all types. Alternatively, if the type of SPC to be decoded is predetermined, such as when decoding all SPCs, this step may be omitted.

[0247] Next, the decoding unit 204 obtains the address position of the SPC that starts in the decoding order (same as the encoding order) within the GOS, starting in the encoded three-dimensional data 211, obtains the encoded data of the starting SPC from that address position, and decodes each SPC sequentially from that starting SPC (S204). Furthermore, the aforementioned address position is stored in metadata, etc.

[0248] Thus, the 3D data decoding device 200 decodes the decoded 3D data 212. Specifically, the 3D data decoding device 200 generates decoded 3D data 212, which serves as a random access unit, by decoding each of the encoded 3D data 211 of the first processing unit (GOS) corresponding to the 3D coordinates. More specifically, the 3D data decoding device 200 decodes each of the plurality of second processing units (SPCs) in each first processing unit (GOS). Furthermore, the 3D data decoding device 200 decodes each of the plurality of third processing units (VLMs) in each second processing unit (SPC).

[0249] The metadata used for random access is described below. This metadata is generated by the three-dimensional data encoding device 100 and is contained in the encoded three-dimensional data 112 (211).

[0250] In conventional two-dimensional random access to moving images, decoding begins with the first frame of a random access unit near a specified time. However, in world space, random access based on coordinates or objects is envisioned in addition to time.

[0251] Therefore, in order to achieve random access to at least three elements—coordinates, objects, and time—a table was prepared that corresponds to the index number of each element and the GOS. Furthermore, a correspondence was established between the index number of the GOS and the address of the I-SPC that begins with the GOS. Figure 10 An example of a table included in the metadata is shown. Additionally, there is no need to use... Figure 10 Of all the tables shown, at least one table is required.

[0252] The following example illustrates random access starting from coordinates. When accessing coordinates (x2, y2, z2), the coordinate-GOS table is consulted first. It is known that the location with coordinates (x2, y2, z2) is included in the second GOS. Next, the GOS address table is consulted. Since the address of the first I-SPC in the second GOS is known to be addr(2), the decoding unit 204 obtains the data from this address and begins decoding.

[0253] Furthermore, the address can be a logical address or a physical address of the HDD or memory. Also, information defining file segments can be used instead of addresses. For example, a file segment is a unit of data after dividing one or more GOS (Gateway Operating System) units.

[0254] Furthermore, when an object spans multiple GOS, the GOS to which multiple objects belong can be displayed in the object GOS table. If these multiple GOS are closed GOS, the encoding and decoding devices can perform encoding or decoding in parallel. Additionally, if these multiple GOS are open GOS, the compression efficiency can be further improved by referencing each other.

[0255] Examples of objects include people, animals, cars, bicycles, traffic lights, or buildings that serve as landmarks on land. For example, when encoding in world space, the 3D data encoding device 100 extracts feature points unique to the object from 3D point clouds, detects the object based on these feature points, and can set the detected object as a random access point.

[0256] Thus, the three-dimensional data encoding device 100 generates first information showing a plurality of first processing units (GOS) and three-dimensional coordinates corresponding to each of the plurality of first processing units (GOS). Furthermore, the encoded three-dimensional data 112 (211) includes this first information. The first information further shows at least one of the object, time, and data storage destination corresponding to each of the plurality of first processing units (GOS).

[0257] The three-dimensional data decoding device 200 obtains first information from the encoded three-dimensional data 211, uses the first information to determine the encoded three-dimensional data 211 of the first processing unit corresponding to the specified three-dimensional coordinates, object or time, and decodes the encoded three-dimensional data 211.

[0258] Examples of other metadata are described below. In addition to metadata for random access, the 3D data encoding device 100 can also generate and store the following metadata. Furthermore, the 3D data decoding device 200 can utilize this metadata during decoding.

[0259] When using 3D data as map information, profiles are defined according to their purpose, and the information for that profile can be included in the metadata. For example, profiles may be defined for urban or suburban areas, or for flying objects, and the maximum or minimum size of world space, SPC, or VLM may be defined respectively. For example, in an urban-oriented profile, more detailed information is needed than in a suburban area, so the minimum size of the VLM is set to be smaller.

[0260] Meta-information can also include label values ​​indicating the type of object. These label values ​​correspond to the VLM, SPC, or GOS that constitute the object. Label values ​​can be set according to the type of object, for example, label value "0" represents "person," label value "1" represents "car," and label value "2" represents "traffic light." Alternatively, when the type of object is difficult to determine or does not need to be determined, label values ​​representing properties such as size, or whether the object is dynamic or static, can be used.

[0261] Furthermore, metadata can also include information showing the extent of the spatial region occupied by the world space.

[0262] Furthermore, metadata can also be used as header information shared by the entire stream of encoded data or multiple SPCs such as SPCs within GOS to store the size of an SPC or VXL.

[0263] Furthermore, the metadata may also include identification information such as distance sensors or cameras used in the generation of the point cloud, or information showing the positional accuracy of point groups within the point cloud.

[0264] Furthermore, meta-information can include information indicating whether the world space consists only of static objects or contains dynamic objects.

[0265] The following describes variations of this embodiment.

[0266] The encoding or decoding device can encode or decode two or more SPCs or GOSs that are different from each other in parallel. The GOSs that are encoded or decoded in parallel can be determined based on metadata indicating the spatial location of the GOSs.

[0267] In cases where three-dimensional data is used as a spatial map of moving vehicles or flying objects, or in the generation of such spatial maps, the encoding or decoding device can encode or decode the GOS or SPC contained in the space determined based on GPS, path information, or zoom level.

[0268] Furthermore, the decoding device can also start decoding sequentially from the spaces closest to its own position or path. The encoding or decoding device can also prioritize spaces farther from its own position or path over closer spaces when encoding or decoding. Here, lowering priority means reducing the processing order, reducing resolution (post-processing), or reducing image quality (to improve encoding efficiency, such as by increasing the quantization step size).

[0269] Furthermore, when decoding encoded data that is hierarchically encoded in space, the decoding device can also decode only the lower levels.

[0270] Furthermore, the decoding device can also start decoding from the lower levels, depending on the map's zoom level or purpose.

[0271] Furthermore, in applications such as self-position estimation or object recognition during the autonomous movement of cars or robots, the encoding or decoding device can reduce the resolution of the area outside the area within a specified height of the road surface (the area to be identified) for encoding or decoding.

[0272] Furthermore, the encoding device can also encode point clouds representing indoor and outdoor spatial shapes independently. For example, by separating the GOS representing the interior (indoor GOS) from the GOS representing the exterior (outdoor GOS), the decoding device can select the GOS to be decoded according to the viewpoint position when using the encoded data.

[0273] Furthermore, the encoding device can encode indoor and outdoor GOS with close coordinates adjacently in the encoding stream. For example, the encoding device maps the identifiers of the two together and stores information showing that the corresponding identifiers have been established in the encoding stream or in separately stored metadata. Accordingly, the decoding device can identify indoor and outdoor GOS with close coordinates by referring to the information in the metadata.

[0274] Furthermore, the encoding device can switch the size of GOS or SPC between indoor and outdoor GOS. For example, the encoding device can set a smaller GOS size indoors compared to outdoors. Additionally, the encoding device can also change the accuracy of feature point extraction from the point cloud or the accuracy of object detection between indoor and outdoor GOS.

[0275] Furthermore, the encoding device can append information used by the decoding device to distinguish between dynamic and static objects to the encoded data. Accordingly, the decoding device can combine dynamic objects with red boxes or explanatory text to represent them. Alternatively, the decoding device can replace dynamic objects with only red boxes or explanatory text. Moreover, the decoding device can represent more detailed object categories. For example, a car can be represented with a red box, and a person with a yellow box.

[0276] Furthermore, the encoding or decoding device can determine whether to encode or decode by classifying dynamic objects and static objects as different SPCs or GOSs based on factors such as the frequency of occurrence of dynamic objects or the ratio of static objects to dynamic objects. For example, if the frequency or ratio of dynamic objects exceeds a threshold, an SPC or GOS in which dynamic and static objects are mixed is allowed; if the frequency or ratio of dynamic objects does not exceed the threshold, an SPC or GOS in which dynamic and static objects are mixed is not allowed.

[0277] When a dynamic object is detected not from a point cloud but from two-dimensional image information from a camera, the encoding device can obtain the information (boxes or text, etc.) used to identify the detection result and the object's position separately, and encode this information as part of the three-dimensional encoded data. In this case, the decoding device overlays the auxiliary information (boxes or text) representing the dynamic object onto the decoding result of the static object.

[0278] Furthermore, the encoding device can adjust the density of VXL or VLM according to the complexity of the static object's shape. For example, the more complex the shape of the static object, the denser the VXL or VLM will be. Moreover, the encoding device can determine the quantization step size when quantizing spatial position or color information based on the density of VXL or VLM. For example, the denser the VXL or VLM, the smaller the quantization step size will be.

[0279] As shown above, the encoding or decoding device involved in this embodiment uses spatial units with coordinate information to encode or decode space.

[0280] Furthermore, the encoding and decoding devices perform encoding or decoding in space using volume units. Volume includes the smallest unit corresponding to positional information, namely a voxel.

[0281] Furthermore, the encoding and decoding devices establish correspondences between arbitrary elements by creating tables that correspond to each element, including spatial information such as coordinates, objects, and time, with the Group of Pictures (GOP), or tables that correspond between elements. The decoding device uses the value of the selected element to determine the coordinates, and determines the volume, voxel, or space based on the coordinates, then decodes the space including that volume or voxel, or the determined space.

[0282] Furthermore, the encoding device determines the volume, voxel, or space that can be selected by the elements through feature point extraction or object recognition, and encodes it as a volume, voxel, or space that can be randomly accessed.

[0283] The space is divided into three types: I-SPC, which can be encoded or decoded by a single space unit; P-SPC, which is encoded or decoded with reference to any one processed space; and B-SPC, which is encoded or decoded with reference to any two processed spaces.

[0284] More than one volume corresponds to either a static object or a dynamic object. The space containing static objects and the space containing dynamic objects are encoded or decoded as different GOS. That is, the SPC containing static objects and the SPC containing dynamic objects are assigned to different GOS.

[0285] Dynamic objects are encoded or decoded individually, corresponding to one or more spaces containing only static objects. That is, multiple dynamic objects are encoded separately, and the resulting encoded data of multiple dynamic objects corresponds to an SPC containing only static objects.

[0286] The encoding and decoding devices prioritize I-SPCs within the GOS for encoding or decoding. For example, the encoding device encodes in a way that reduces I-SPC degradation (so that the original 3D data can be reproduced more faithfully after decoding). The decoding device, for example, decodes only the I-SPCs.

[0287] The encoding device can adjust the frequency of I-SPC utilization based on the density or quantity of objects in world space. In other words, the encoding device changes the frequency of I-SPC selection according to the number or density of objects contained in the 3D data. For example, the higher the density of objects in world space, the more frequently the encoding device uses I-space.

[0288] Furthermore, the encoding device sets random access points in units of GOS and stores information showing the spatial region corresponding to the GOS in the header information.

[0289] The encoding device may use a default value as the size of the GOS. Alternatively, the encoding device may change the size of the GOS according to the number or density of objects or dynamic objects. For example, the encoding device will set the size of the GOS smaller when the objects or dynamic objects are denser or more numerous.

[0290] Furthermore, the space or volume includes a group of feature points derived from information obtained using sensors such as depth sensors, gyroscopes, or cameras. The coordinates of the feature points are set to the center position of the voxels. Moreover, through voxel subdivision, high precision of positional information can be achieved.

[0291] Feature point groups are derived using multiple images. The multiple images have at least two types of temporal information: the actual temporal information and the spatially corresponding temporal information of the same time in the multiple images (e.g., the encoded temporal for rate control, etc.).

[0292] Furthermore, encoding or decoding is performed in units of GOS that include more than one space.

[0293] The encoding and decoding devices, with reference to the space within the processed GOS, predict the P space or B space within the GOS of the object being processed.

[0294] Alternatively, the encoding and decoding devices do not refer to different GOS, but use the processed space within the GOS of the object being processed to predict the P space or B space within the GOS of the object being processed.

[0295] Furthermore, the encoding and decoding devices transmit or receive encoded streams in units of world space comprising one or more GOS.

[0296] Furthermore, the GOS has a layered structure in at least one direction within world space, and the encoding and decoding devices encode or decode starting from the lower layer. For example, a GOS capable of random access belongs to the lowest layer. A GOS belonging to a higher layer only refers to GOS belonging to layers below the same layer. That is, the GOS is spatially divided in a predetermined direction, including multiple layers, each with more than one SPC. The encoding and decoding devices encode or decode for each SPC by referring to SPCs contained in layers that are in the same layer as or lower than that SPC.

[0297] Furthermore, the encoding and decoding devices continuously encode or decode GOS within a world space unit comprising multiple GOS. The encoding and decoding devices write or read information indicating the order (direction) of encoding or decoding as metadata. That is, the encoded data includes information indicating the encoding order of multiple GOS.

[0298] Furthermore, the encoding and decoding devices encode or decode two or more different spaces or GOS in parallel.

[0299] Furthermore, the encoding and decoding devices encode or decode the spatial information (coordinates, size, etc.) of the space or GOS.

[0300] Furthermore, the encoding and decoding devices encode or decode the space or GOS contained in a specific space determined based on external information related to their own location and / or area size, such as GPS, path information, or magnification.

[0301] Encoding or decoding devices prioritize spaces farther away from themselves over spaces closer to themselves when performing encoding or decoding.

[0302] The encoding device sets a direction in world space according to a magnification or purpose, and encodes GOS with a layered structure in that direction. The decoding device, for a GOS with a layered structure in one direction of world space set according to a magnification or purpose, preferentially decodes it starting from the lower layer.

[0303] The encoding device alters the accuracy of feature point extraction, object recognition, and spatial area size contained in indoor and outdoor spaces. However, the encoding and decoding devices encode or decode indoor and outdoor GOS that are close in coordinates, placing them adjacent in world space, and also map these identifiers together for encoding or decoding.

[0304] (Implementation Method 2) When using encoded point cloud data for practical devices or services, it is desirable to transmit and receive the required information according to its intended purpose in order to conserve network bandwidth. However, current encoding structures for 3D data do not possess this functionality, and therefore, there is no corresponding encoding method.

[0305] This embodiment will describe a three-dimensional data encoding method and a three-dimensional data encoding apparatus for providing the function of sending and receiving required information according to the purpose in the encoded data of a three-dimensional point cloud, as well as a three-dimensional data decoding method and a three-dimensional data decoding apparatus for decoding the encoded data.

[0306] A voxel with a certain number of characteristics (VXL) is defined as a characteristic voxel (FVXL), and the world space (WLD) composed of FVXL is defined as a sparse world space (SWLD). Figure 11This illustrates a sparse world space and examples of its composition. SWLD includes: FGOS, a GOS constructed from FVXL; FSPC, an SPC constructed from FVXL; and FVLM, a VLM constructed from FVXL. The data structures and prediction structures of FGOS, FSPC, and FVLM can be the same as those of GOS, SPC, and VLM.

[0307] Feature quantities refer to the characteristic quantities that represent the three-dimensional position information of VXL, or the visible light information of VXL position, especially the corners and edges of three-dimensional objects where more feature quantities can be detected. Specifically, although the feature quantity is the three-dimensional feature quantity or visible light feature quantity described below, it can be any feature quantity that represents the position, brightness, or color information of VXL.

[0308] As three-dimensional feature quantities, SHOT (Signature of Histograms of Orientations), PFH (Point Feature Histograms), or PPF (Point Pair Feature) features are used.

[0309] The SHOT feature is obtained by segmenting the area around the VXL region, calculating the inner product of the reference point and the normal vector of the segmented region, and then performing histogram transformation. This SHOT feature is characterized by high dimensionality and high feature expressiveness.

[0310] The PFH feature is obtained by selecting multiple pairs of points near VXL, calculating the normal vector from these points, and then performing histogram transformation. Because it is a histogram feature, the PFH feature is robust against a small amount of interference and exhibits high feature expressiveness.

[0311] The PPF feature is a feature calculated using the VXL of a 2-point vector and the normal vector, etc. Because all VXLs are used, the PPF feature is robust for occlusion.

[0312] Furthermore, as a feature quantity of visible light, it is possible to use SIFT (Scale-Invariant Feature Transform), SURF (Speeded Up Robust Features), or HOG (Histogram of Oriented Gradients), which incorporate information such as the brightness gradient of the image.

[0313] SWLD is generated by calculating the aforementioned feature quantities from each VXL of WLD and extracting FVXL. Here, SWLD can be updated every time WLD is updated, or it can be updated periodically after a certain period of time, regardless of the update timing of WLD.

[0314] SWLDs can be generated for each feature. For example, SWLD1 based on SHOT features and SWLD2 based on SIFT features can be generated separately for each feature, and the SWLDs can be used according to their intended purpose. Furthermore, the calculated features of each FVXL can also be stored as feature information in each FVXL.

[0315] Next, the utilization method of Sparse World Space (SWLD) will be explained. Since SWLD only contains feature voxels (FVXL), its data size is generally smaller compared to WLD, which includes all VXLs.

[0316] In applications that utilize feature quantities to achieve a certain purpose, using SWLD information instead of WLD can suppress hard drive read time and reduce network transmission bandwidth and transmission time. For example, as map information, by pre-storing WLD and SWLD on the server and switching the sent map information to WLD or SWLD according to the client's request, network bandwidth and transmission time can be reduced. Specific examples are shown below.

[0317] Figure 12 as well as Figure 13 Examples of the use of SWLD and WLD are shown. For example... Figure 12 As shown, when client 1, acting as a vehicle-mounted device, needs map information for determining its own location, client 1 sends a request to the server for map data for location estimation (S301). The server sends the SWLD (Site Layout Data) to client 1 according to the request (S302). Client 1 uses the received SWLD to determine its own location (S303). At this time, client 1 uses various methods such as a rangefinder, a stereo camera, or a combination of multiple monocular cameras to acquire VXL (Very Large Scale) information of its surroundings, and estimates its own location information based on the obtained VXL information and SWLD. Here, the location information includes client 1's three-dimensional location information and orientation, etc.

[0318] like Figure 13As shown, when client 2, acting as a vehicle-mounted device, needs map information for map drawing purposes such as 3D maps, client 2 sends a request to the server for map data acquisition (S311). The server, in accordance with this request, sends a WLD (Wide Image Data) to client 2 (S312). Client 2 uses the received WLD to perform map drawing (S313). At this time, client 2, for example, uses images captured by itself with a visible light camera, and the WLD acquired from the server, to create a conceptual image, which is then displayed on a screen such as a car navigation system.

[0319] As shown above, the server sends the SWLD to the client when it primarily needs VXL feature quantities such as its own location estimation, similar to map drawing. When more detailed VXL information is required, the server sends the WLD to the client. This enables efficient sending and receiving of map data.

[0320] Additionally, the client can determine whether it needs a SWLD or a WLD and request the server to send either one. Furthermore, the server can determine which SWLD or WLD to send based on the client's or network conditions.

[0321] Next, we will explain the method for switching between sending and receiving Sparse World Space (SWLD) and World Space (WLD).

[0322] The reception of WLD or SWLD can be switched according to the network bandwidth. Figure 14 An example of this operation is shown. For instance, when a low-speed network with sufficient bandwidth, such as an LTE (Long Term Evolution) network, is used, the client accesses the server via the low-speed network (S321) and obtains the SWLD (Wide Layout Map) as map information from the server (S322). Conversely, when a high-speed network with ample bandwidth, such as a WiFi network, is used, the client accesses the server via the high-speed network (S323) and obtains the SWLD from the server (S324). Accordingly, the client can obtain appropriate map information based on its network bandwidth.

[0323] Specifically, the client receives SWLD via LTE outdoors, and obtains WLD via WiFi when entering indoor facilities. Based on this, the client can obtain more detailed indoor map information.

[0324] In this way, the client can request WLD or SWLD from the server according to the frequency band of its network. Alternatively, the client can send information indicating the frequency band of its network to the server, and the server can send the appropriate data (WLD or SWLD) to the client based on that information. Or, the server can determine the client's network bandwidth and send the appropriate data (WLD or SWLD) to the client.

[0325] Furthermore, the reception of WLD or SWLD can be switched according to the movement speed. Figure 15 An example of operation in this scenario is shown. For instance, when the client is moving at high speed (S331), the client receives a SWLD from the server (S332). Conversely, when the client is moving at low speed (S333), the client receives a WLD from the server (S334). Accordingly, the client can both conserve network bandwidth and obtain map information according to speed. Specifically, when the client is traveling on a highway, by receiving a small amount of SWLD, it can update the map information at a roughly appropriate speed. Furthermore, when the client is traveling on a regular road, by receiving a WLD, it can obtain more detailed map information.

[0326] In this way, the client can request WLD or SWLD from the server based on its own movement speed. Alternatively, the client can send information indicating its own movement speed to the server, and the server can send the appropriate data (WLD or SWLD) to the client based on that information. Or, the server can determine the client's movement speed and send the appropriate data (WLD or SWLD) to the client.

[0327] Alternatively, the client can first obtain the SWLD from the server, and then obtain the WLD for the more important areas within it. For example, when acquiring map data, the client can first obtain a general map information from the SWLD, filter out areas with a high frequency of features such as buildings, signs, or people, and then obtain the WLD for the filtered areas. In this way, the client can both control the amount of data received from the server and obtain the detailed information for the required areas.

[0328] Furthermore, the server can create separate SWLDs for each object based on the WLD, and the client can receive them according to their intended use. This can reduce network bandwidth usage. For example, the server can pre-identify people or vehicles from the WLD and create separate SWLDs for people and vehicles. The client receives the person SWLD when it wants to obtain information about people around it, and the vehicle SWLD when it wants to obtain information about vehicles. Moreover, the types of SWLDs can be distinguished based on information attached to the header (such as logos or types).

[0329] Next, the configuration and operation flow of the three-dimensional data encoding device (e.g., a server) involved in this embodiment will be described. Figure 16 This is a block diagram of the three-dimensional data encoding device 400 involved in this embodiment. Figure 17 This is a flowchart of the three-dimensional data encoding process performed by the three-dimensional data encoding device 400.

[0330] Figure 16 The illustrated 3D data encoding apparatus 400 generates encoded 3D data 413 and 414 as encoded streams by encoding input 3D data 411. Here, encoded 3D data 413 is encoded 3D data corresponding to WLD (Wide Area Decoding), and encoded 3D data 414 is encoded 3D data corresponding to SWLD (Simplified Swing Data Decoding). The 3D data encoding apparatus 400 includes: an acquisition unit 401, an encoding region determination unit 402, an SWLD extraction unit 403, a WLD encoding unit 404, and an SWLD encoding unit 405.

[0331] like Figure 17 As shown, firstly, the acquisition unit 401 acquires input three-dimensional data 411 as point group data in three-dimensional space (S401).

[0332] Next, the encoding region determination unit 402 determines the spatial region of the encoding object based on the spatial region where the point group data exists (S402).

[0333] Next, the SWLD extraction unit 403 defines the spatial region of the encoded object as WLD and calculates the feature quantity based on each VXL contained in the WLD. Furthermore, the SWLD extraction unit 403 extracts VXLs whose feature quantity is above a predetermined threshold, defines the extracted VXLs as FVXLs, and appends these FVXLs to the SWLD to generate extracted three-dimensional data 412 (S403). That is, extracted three-dimensional data 412 with feature quantities above the threshold is extracted from the input three-dimensional data 411.

[0334] Next, the WLD encoding unit 404 generates encoded three-dimensional data 413 corresponding to the WLD by encoding the input three-dimensional data 411 corresponding to the WLD (S404). At this time, the WLD encoding unit 404 appends information used to distinguish whether the encoded three-dimensional data 413 is a stream containing the WLD to the header of the encoded three-dimensional data 413.

[0335] Furthermore, the SWLD encoding unit 405 generates encoded three-dimensional data 414 corresponding to the SWLD by encoding the extracted three-dimensional data 412 corresponding to the SWLD (S405). At this time, the SWLD encoding unit 405 appends information used to distinguish whether the encoded three-dimensional data 414 is a stream containing the SWLD to the header of the encoded three-dimensional data 414.

[0336] Furthermore, the processing order of generating coded 3D data 413 and generating coded 3D data 414 can also be reversed as described above. Additionally, some or all of the above processes can be executed in parallel.

[0337] Information assigned to the headers of encoded 3D data 413 and 414 is, for example, defined as a parameter such as "world_type". When world_type=0, it indicates that the stream contains WLD; when world_type=1, it indicates that the stream contains SWLD. When defining other categories, the assigned value can be increased, such as world_type=2. Furthermore, specific flags can be included in either encoded 3D data 413 or 414. For example, encoded 3D data 414 can be assigned a flag indicating that the stream contains SWLD. In this case, the decoding device can determine whether the stream contains WLD or SWLD based on the presence or absence of the flag.

[0338] Furthermore, the encoding method used by the WLD encoding unit 404 when encoding WLD can be different from the encoding method used by the SWLD encoding unit 405 when encoding SWLD.

[0339] For example, because SWLD data is sampled, its correlation with surrounding data may be lower compared to WLD. Therefore, in coding methods used for SWLD, inter-frame prediction is prioritized over intra-frame prediction and inter-frame prediction compared to coding methods used for WLD.

[0340] Furthermore, the representation of 3D position can differ between the encoding methods used for SWLD and WLD. For example, in FWLD, the 3D position of FVXL can be represented by 3D coordinates, while in WLD, the 3D position can be represented by an octree (described later), or vice versa.

[0341] Furthermore, the SWLD encoding unit 405 encodes the SWLD encoded three-dimensional data 414 in a manner where the data size of the SWLD encoded three-dimensional data 413 is smaller than the data size of the WLD encoded three-dimensional data 413. For example, as described above, the correlation between data in SWLD may be reduced compared to WLD. Consequently, the encoding efficiency decreases, and the data size of the encoded three-dimensional data 414 may be larger than the data size of the WLD encoded three-dimensional data 413. Therefore, when the obtained encoded three-dimensional data 414 has a larger data size than the WLD encoded three-dimensional data 413, the SWLD encoding unit 405 re-encodes it to generate encoded three-dimensional data 414 with a reduced data size.

[0342] For example, the SWLD extraction unit 403 regenerates extracted 3D data 412 with a reduced number of extracted feature points, and the SWLD encoding unit 405 encodes the extracted 3D data 412. Alternatively, the quantization level in the SWLD encoding unit 405 can be made coarser. For example, in the octree structure described later, the quantization level can be made coarser by rounding the data at the lowest level.

[0343] Furthermore, if the SWLD encoding unit 405 cannot make the data size of the SWLD encoded three-dimensional data 414 smaller than the data size of the WLD encoded three-dimensional data 413, it may choose not to generate the SWLD encoded three-dimensional data 414. Alternatively, the WLD encoded three-dimensional data 413 may be copied to the SWLD encoded three-dimensional data 414. That is, the WLD encoded three-dimensional data 413 can be directly used as the SWLD encoded three-dimensional data 414.

[0344] Next, the configuration and operation flow of the three-dimensional data decoding device (e.g., client) involved in this embodiment will be described. Figure 18 This is a block diagram of the three-dimensional data decoding device 500 involved in this embodiment. Figure 19 This is a flowchart of the three-dimensional data decoding process performed by the three-dimensional data decoding device 500.

[0345] Figure 18 The illustrated 3D data decoding apparatus 500 generates decoded 3D data 512 or 513 by decoding encoded 3D data 511. Here, encoded 3D data 511 is, for example, encoded 3D data 413 or 414 generated by the 3D data encoding apparatus 400.

[0346] The 3D data decoding device 500 includes: an acquisition unit 501, a head parsing unit 502, a WLD decoding unit 503, and a SWLD decoding unit 504.

[0347] like Figure 19 As shown, firstly, the acquisition unit 501 acquires the encoded three-dimensional data 511 (S501). Next, the header parsing unit 502 parses the header of the encoded three-dimensional data 511 and determines whether the encoded three-dimensional data 511 is a stream containing WLD or a stream containing SWLD (S502). For example, the determination is made by referring to the world_type parameter mentioned above.

[0348] If the encoded 3D data 511 is a stream containing WLD (S503 "Yes"), the WLD decoding unit 503 decodes the encoded 3D data 511 to generate decoded 3D data 512 of WLD (S504). Alternatively, if the encoded 3D data 511 is a stream containing SWLD (S503 "No"), the SWLD decoding unit 504 decodes the encoded 3D data 511 to generate decoded 3D data 513 of SWLD (S505).

[0349] Furthermore, similar to the encoding apparatus, the decoding method used by the WLD decoding unit 503 when decoding the WLD can be different from the decoding method used by the SWLD decoding unit 504 when decoding the SWLD. For example, in the decoding method for the SWLD, compared with the decoding method for the WLD, inter-frame prediction in intra-frame prediction and inter-frame prediction can be prioritized.

[0350] Furthermore, the methods used to represent the three-dimensional position can differ between the decoding methods used for SWLD and WLD. For example, SWLD can represent the three-dimensional position of FVXL using three-dimensional coordinates, while WLD can represent the three-dimensional position using an octree (described later), and vice versa.

[0351] Next, the octree representation as a method of representing three-dimensional location will be explained. The VXL data contained in the three-dimensional data is converted into an octree structure and then encoded. Figure 20 An example of VXL for WLD is shown. Figure 21 It shows Figure 20 The octree structure of WLD is shown. Figure 20 In the example shown, there are three VXL1 to VXL3 that constitute the VXL (hereinafter, valid VXL) of the point group. Figure 21 As shown, the octree structure consists of nodes and leaf nodes. Each node has a maximum of 8 nodes or leaf nodes. Each leaf node contains VXL information. Here, Figure 21 Among the leaf nodes shown, leaf nodes 1, 2, and 3 respectively represent Figure 20 VXL1, VXL2, and VXL3 are shown.

[0352] Specifically, each node and leaf node corresponds to a 3D position. Node 1 and... Figure 20 All the blocks shown correspond to each other. The block corresponding to node 1 is divided into 8 blocks. Among these 8 blocks, the block with valid VXL is set as a node, and the rest are set as leaf nodes. The block corresponding to a node is further divided into 8 nodes or leaf nodes, and this process is repeated the same number of times as the level in the tree structure. Furthermore, all the blocks at the bottom level are set as leaf nodes.

[0353] and,Figure 22 It shows from Figure 20 The example shown is a SWLD generated from a WLD. Figure 20 The feature extraction results of VXL1 and VXL2 shown are identified as FVXL1 and FVXL2 and added to SWLD. VXL3, however, is not identified as FVXL and therefore is not included in SWLD. Figure 23 It shows Figure 22 The octree structure of SWLD is shown. Figure 23 In the octree structure shown, Figure 21 Leaf node 3, which corresponds to VXL3, is deleted. Accordingly, Figure 21 Node 3 shown does not have a valid VXL and has been changed to a leaf node. Thus, generally speaking, the number of leaf nodes in SWLD is less than that in WLD, and the encoded 3D data of SWLD is also smaller than that of WLD.

[0354] The following describes variations of this embodiment.

[0355] For example, when a vehicle-mounted device or other client is estimating its own position, it receives a SWLD from the server, uses the SWLD to estimate its own position, and performs obstacle detection. Then, it uses various methods such as distance sensors such as rangefinders, stereo cameras, or combinations of multiple monocular cameras to perform obstacle detection based on the three-dimensional information of the surrounding environment it has obtained.

[0356] Furthermore, it is generally difficult to include VXL data for flat areas in a SWLD. Therefore, the server maintains a downsampled world space (SubWLD) that is downsampled from the WLD for detecting stationary obstacles, and can send both the SWLD and SubWLD to the client. This allows for both network bandwidth management and client-side position estimation and obstacle detection.

[0357] Furthermore, a grid-structured map is advantageous when the client rapidly draws 3D map data. Therefore, the server can generate a grid based on the World Layout (WLD) and maintain it beforehand as a Grid World Space (MWLD). For example, the client receives the MWLD when a rough 3D drawing is needed, and the WLD when a detailed 3D drawing is required. This helps to control network bandwidth usage.

[0358] Furthermore, although the server sets VXLs with feature values ​​above a threshold as FVXLs from each VXL, FVXLs can also be calculated using different methods. For example, if the server determines that VXLs, VLMs, SPCs, or GOSs constituting signals or intersections are needed in self-position estimation, driving assistance, or autonomous driving, they can be included in the SWLD as FVXLs, FVLMs, FSPCs, or FGOSs. Moreover, the above determination can be performed manually. In addition, FVXLs obtained by the above method can be added to FVXLs, etc., set based on feature values. That is, the SWLD extraction unit 403 can further extract data corresponding to objects with predefined attributes from the input 3D data 411 as extracted 3D data 412.

[0359] Furthermore, different labels can be assigned to the feature values ​​depending on the situation requiring these uses. The server can maintain the FVXL required for self-position estimation of signals or intersections, driver assistance, or autonomous driving as a higher layer of SWLD (e.g., lane world space).

[0360] Furthermore, the server can also append attributes to the VXL within the WLD according to random access units or specified units. Attributes may include, for example, information indicating whether they are needed or not in the self-location estimation, or information indicating whether they are important as traffic information such as signals or intersections. Additionally, attributes may include the correspondence between lane information (GDF: Geographic Data Files, etc.) and features (intersections or roads, etc.).

[0361] Furthermore, the following methods can be used as an update method for WLD or SWLD.

[0362] Updated information such as changes in people, construction, or street trees (track-oriented) is loaded onto the server as point clusters or metadata. The server updates the WLD based on this loading, and then uses the updated WLD to update the SWLD.

[0363] Furthermore, if the client detects a mismatch between the 3D information it generates when estimating its own position and the 3D information received from the server, it can send its generated 3D information along with an update notification to the server. In this case, the server uses the WLD to update the SWLD. If the SWLD has not been updated, the server determines that the WLD itself is outdated.

[0364] Furthermore, as header information of the encoded stream, information for distinguishing between WLD and SWLD is appended. For example, in cases where multiple world spaces exist, such as grid world space or lane world space, information for differentiating them can be appended to the header information. Also, when multiple SWLDs with different feature values ​​exist, information for distinguishing them individually can be appended to the header information.

[0365] Furthermore, although the SWLD is composed of FVXLs, it can also include VXLs that are not identified as FVXLs. For example, the SWLD can include adjacent VXLs used when calculating the feature values ​​of the FVXLs. Accordingly, even if each FVXL in the SWLD does not have additional feature value information, the client can calculate the feature values ​​of the FVXLs when receiving the SWLD. In addition, in this case, the SWLD can include information for distinguishing whether each VXL is an FVXL or a VXL.

[0366] As described above, the three-dimensional data encoding device 400 extracts three-dimensional data 412 (second three-dimensional data) from the input three-dimensional data 411 (first three-dimensional data) with a feature quantity of more than a threshold, and generates encoded three-dimensional data 414 (first encoded three-dimensional data) by encoding the extracted three-dimensional data 412.

[0367] Accordingly, the 3D data encoding device 400 generates encoded 3D data 414 by encoding data whose feature values ​​are above a threshold. This reduces the amount of data compared to directly encoding the input 3D data 411. Therefore, the 3D data encoding device 400 can reduce the amount of data transmitted.

[0368] Furthermore, the three-dimensional data encoding device 400 generates encoded three-dimensional data 413 (second encoded three-dimensional data) by encoding the input three-dimensional data 411.

[0369] Accordingly, the three-dimensional data encoding device 400 can selectively transmit encoded three-dimensional data 413 and encoded three-dimensional data 414, for example, according to its intended use.

[0370] Furthermore, the extracted 3D data 412 is encoded by the first encoding method, and the input 3D data 411 is encoded by the second encoding method, which is different from the first encoding method.

[0371] Accordingly, the three-dimensional data encoding device 400 can employ appropriate encoding methods for the input three-dimensional data 411 and the extracted three-dimensional data 412 respectively.

[0372] Furthermore, in the first coding method, compared to the second coding method, inter-frame prediction is prioritized in both intra-frame prediction and inter-frame prediction.

[0373] Accordingly, the 3D data encoding device 400 can extract 3D data 412 for adjacent data where the correlation between them is likely to decrease, thereby increasing the priority of inter-frame prediction.

[0374] Furthermore, the representation of 3D position differs between the first and second encoding methods. For example, the second encoding method uses an octree to represent 3D position, while the first encoding method uses 3D coordinates.

[0375] Accordingly, the three-dimensional data encoding device 400 can adopt a more appropriate three-dimensional position representation method for three-dimensional data with different data numbers (number of VXL or FVXL).

[0376] Furthermore, at least one of the encoded three-dimensional data 413 and 414 includes an identifier indicating whether the encoded three-dimensional data is obtained by encoding the input three-dimensional data 411 or by encoding a portion of the input three-dimensional data 411. That is, the identifier indicates whether the encoded three-dimensional data is WLD encoded three-dimensional data 413 or SWLD encoded three-dimensional data 414.

[0377] Accordingly, the decoding device can easily determine whether the acquired encoded three-dimensional data is encoded three-dimensional data 413 or encoded three-dimensional data 414.

[0378] Furthermore, the three-dimensional data encoding device 400 encodes the extracted three-dimensional data 412 in such a way that the amount of data encoded in the three-dimensional data 414 is less than the amount of data encoded in the three-dimensional data 413.

[0379] Accordingly, the three-dimensional data encoding device 400 can encode three-dimensional data 414 in a smaller amount than the amount of data encodes three-dimensional data 413.

[0380] Furthermore, the 3D data encoding device 400 further extracts data corresponding to objects with predefined attributes from the input 3D data 411 as extracted 3D data 412. For example, objects with predefined attributes refer to objects needed in self-position estimation, driving assistance, or autonomous driving, such as signals or intersections.

[0381] Accordingly, the three-dimensional data encoding device 400 is able to generate encoded three-dimensional data 414, including the data required by the decoding device.

[0382] Furthermore, the three-dimensional data encoding device 400 (server) further sends one of the encoded three-dimensional data 413 and 414 to the client according to the client's state.

[0383] Accordingly, the three-dimensional data encoding device 400 can send appropriate data according to the client's status.

[0384] Furthermore, the client's status includes the client's communication status (such as network bandwidth) or the client's movement speed.

[0385] Furthermore, the three-dimensional data encoding device 400 further sends one of the encoded three-dimensional data 413 and 414 to the client according to the client's request.

[0386] Accordingly, the three-dimensional data encoding device 400 can send appropriate data according to the client's request.

[0387] Furthermore, the three-dimensional data decoding device 500 according to this embodiment decodes the encoded three-dimensional data 413 or 414 generated by the three-dimensional data encoding device 400.

[0388] That is, the 3D data decoding device 500 decodes the encoded 3D data 414 obtained by encoding the extracted 3D data 412, in which the feature quantity extracted from the input 3D data 411 is above a threshold, using the first decoding method. Furthermore, the 3D data decoding device 500 decodes the encoded 3D data 413 obtained by encoding the input 3D data 411 using a second decoding method different from the first decoding method.

[0389] Accordingly, the 3D data decoding device 500 can selectively receive encoded 3D data 414 and encoded 3D data 413 obtained by encoding data with feature values ​​above a threshold, for example, according to their intended use. This reduces the amount of data transmitted. Furthermore, the 3D data decoding device 500 can employ appropriate decoding methods for the input 3D data 411 and the extracted 3D data 412 respectively.

[0390] Furthermore, in the first decoding method, compared to the second decoding method, intra-frame prediction and inter-frame prediction are given priority.

[0391] Accordingly, the 3D data decoding device 500 can improve the priority of inter-frame prediction by extracting 3D data where the correlation between adjacent data is easily reduced.

[0392] Furthermore, the methods used to represent 3D position differ between the first and second decoding methods. For example, the second decoding method uses an octree to represent 3D position, while the first decoding method uses 3D coordinates.

[0393] Accordingly, the 3D data decoding device 500 can employ more appropriate 3D position representation methods for 3D data with different data numbers (number of VXL or FVXL).

[0394] Furthermore, at least one of the encoded three-dimensional data 413 and 414 includes an identifier indicating whether the encoded three-dimensional data is obtained by encoding the input three-dimensional data 411 or by encoding a portion of the input three-dimensional data 411. The three-dimensional data decoding device 500 refers to this identifier to identify the encoded three-dimensional data 413 and 414.

[0395] Accordingly, the three-dimensional data decoding device 500 can easily determine whether the obtained encoded three-dimensional data is encoded three-dimensional data 413 or encoded three-dimensional data 414.

[0396] Furthermore, the 3D data decoding device 500 notifies the server of the client's (3D data decoding device 500) status. The 3D data decoding device 500 receives encoded 3D data 413 and 414 sent from the server according to the client's status.

[0397] Accordingly, the 3D data decoding device 500 can receive appropriate data according to the client's status.

[0398] Furthermore, the client's status includes the client's communication status (such as network bandwidth) or the client's movement speed.

[0399] Furthermore, the 3D data decoding device 500 further requests the server to encode one of the 3D data 413 and 414, and receives the encoded 3D data 413 and 414 sent from the server in accordance with the request.

[0400] Accordingly, the 3D data decoding device 500 is able to receive appropriate data corresponding to its purpose.

[0401] (Implementation Method 3) In this embodiment, a method for transmitting and receiving three-dimensional data between vehicles will be described. For example, transmitting and receiving three-dimensional data between the vehicle itself and surrounding vehicles.

[0402] Figure 24 This is a block diagram of the three-dimensional data creation apparatus 620 according to this embodiment. The three-dimensional data creation apparatus 620 is included, for example, in its own vehicle, and creates a denser third three-dimensional data 636 by combining the received second three-dimensional data 635 with the first three-dimensional data 632 created by the three-dimensional data creation apparatus 620.

[0403] The 3D data production apparatus 620 includes: a 3D data production unit 621, a request range determination unit 622, a search unit 623, a receiving unit 624, a decoding unit 625, and a synthesis unit 626.

[0404] First, the 3D data production unit 621 uses sensor information 631 detected by sensors on its own vehicle to produce first 3D data 632. Next, the request range determination unit 622 determines the request range, which refers to the 3D spatial range where the data in the produced first 3D data 632 is insufficient.

[0405] Next, the search unit 623 searches for surrounding vehicles holding 3D data of the requested range and sends request range information 633, showing the requested range, to the surrounding vehicles identified through the search. Then, the receiving unit 624 receives the encoded 3D data 634, which is the encoded stream of the requested range, from the surrounding vehicles (S624). Furthermore, the search unit 623 can issue requests indiscriminately to all vehicles present within the determined range and receive the encoded 3D data 634 from those that respond. Moreover, the search unit 623 is not limited to vehicles; it can also issue requests to objects such as traffic lights or signs and receive the encoded 3D data 634 from those objects.

[0406] Next, the received encoded three-dimensional data 634 is decoded by the decoding unit 625 to obtain the second three-dimensional data 635. Then, the synthesis unit 626 synthesizes the first three-dimensional data 632 and the second three-dimensional data 635 to create a denser third three-dimensional data 636.

[0407] Next, the configuration and operation of the three-dimensional data transmission device 640 according to this embodiment will be described. Figure 25 This is a block diagram of the three-dimensional data transmission device 640. The three-dimensional data transmitting device 640, for example, is included in the surrounding vehicles described above. It processes the fifth three-dimensional data 652 generated by the surrounding vehicles into the sixth three-dimensional data 654 requested by its own vehicle, and generates encoded three-dimensional data 634 by encoding the sixth three-dimensional data 654, and transmits the encoded three-dimensional data 634 to its own vehicle.

[0408] The three-dimensional data transmitting device 640 includes: a three-dimensional data production unit 641, a receiving unit 642, an extraction unit 643, an encoding unit 644, and a transmitting unit 645.

[0409] First, the 3D data production unit 641 uses sensor information 651 detected by sensors on surrounding vehicles to produce fifth 3D data 652. Next, the receiving unit 642 receives request range information 633 sent from its own vehicle.

[0410] Next, the extraction unit 643 extracts the three-dimensional data representing the requested range indicated by the requested range information 633 from the fifth three-dimensional data 652, and processes the fifth three-dimensional data 652 into sixth three-dimensional data 654. Then, the encoding unit 644 encodes the sixth three-dimensional data 654 to generate encoded three-dimensional data 634 as an encoded stream. Finally, the transmission unit 645 transmits the encoded three-dimensional data 634 to its own vehicle.

[0411] Furthermore, although the example described here is that the vehicle itself has a 3D data creation device 620 and the surrounding vehicles have a 3D data transmission device 640, each vehicle may also have the functions of a 3D data creation device 620 and a 3D data transmission device 640.

[0412] (Implementation Method 4) In this embodiment, the work on anomalies in self-position estimation based on a 3D map is explained.

[0413] Applications such as autonomous driving of motor vehicles, autonomous movement of robots, and autonomous movement of flying objects such as drones will expand in the future. One example of a method to achieve this autonomous movement is a method in which the moving body estimates its own position in a three-dimensional map (self-position estimation) and moves according to the map.

[0414] Self-position estimation is achieved by matching a 3D map with 3D information about the vehicle's surroundings obtained by sensors such as rangefinders (LIDAR, etc.) or stereo cameras mounted on the vehicle (hereinafter referred to as self-vehicle detection 3D data), and then estimating the vehicle's position within the 3D map.

[0415] 3D maps, such as the HD maps proposed by HERE, are not only three-dimensional point clouds, but can also include two-dimensional map data such as the shape information of roads and intersections, or information that changes in real time, such as congestion and accidents. A 3D map is composed of multiple layers, including 3D data, 2D data, and metadata that changes in real time. The device can acquire only the data it needs, or it can refer to the data it needs.

[0416] Point cloud data can be the aforementioned SWLD, or it can include point cluster data that are not feature points. Furthermore, the transmission and reception of point cloud data are generally performed using one or more random access units.

[0417] As a method for matching 3D maps with the 3D data of vehicle detection, the following methods can be used. For example, the device compares the shapes of point groups in each other's point clouds and determines the locations with high similarity between feature points as the same location. Furthermore, when the 3D map is composed of SWLD (Signal-Side Layout Map), the device compares and matches the feature points constituting the SWLD with the 3D feature points extracted from the 3D data of vehicle detection.

[0418] To achieve high-precision self-position estimation, the following conditions (A) and (B) must be met: (A) a 3D map and 3D data of the vehicle's own detection are available; and (B) their accuracy meets a predetermined benchmark. However, in the following exceptional case, (A) or (B) cannot be met.

[0419] (1) A 3D map cannot be obtained through the communication path.

[0420] (2) There is no 3D map, or the obtained 3D map is damaged.

[0421] (3) The vehicle's own sensors malfunctioned, or the accuracy of the generated three-dimensional data of the vehicle was insufficient due to bad weather.

[0422] The following describes how to handle these abnormal situations. Although the following explanation uses a vehicle as an example, the methods described can also be applied to all moving objects that perform autonomous movement, such as robots or drones.

[0423] The following will describe the configuration and operation of the three-dimensional information processing device involved in this embodiment for detecting anomalies in three-dimensional data corresponding to three-dimensional maps or the vehicle itself. Figure 26 This is a block diagram illustrating a configuration example of the three-dimensional information processing apparatus 700 according to this embodiment. The three-dimensional information processing device 700 is, for example, mounted on a moving object such as a motor vehicle. Figure 26 As shown, the three-dimensional information processing device 700 includes: a three-dimensional map acquisition unit 701, a vehicle detection data acquisition unit 702, an abnormal situation judgment unit 703, a response work decision unit 704, and a work control unit 705.

[0424] Additionally, the 3D information processing device 700 may also include a camera for acquiring 2D images, or a sensor (not shown) that uses ultrasonic or laser-based 1D data sensors for detecting structural or moving objects around the vehicle. Furthermore, the 3D information processing device 700 may also include a communication unit (not shown) for acquiring 3D maps via mobile communication networks such as 4G or 5G, vehicle-to-vehicle communication, or road-to-vehicle communication.

[0425] The 3D map acquisition unit 701 acquires a 3D map 711 near the driving path. For example, the 3D map acquisition unit 701 acquires the 3D map 711 through a mobile communication network, or vehicle-to-vehicle communication, or road-to-vehicle communication.

[0426] Next, the vehicle detection data acquisition unit 702 acquires three-dimensional detection data 712 of the vehicle itself based on the sensor information. For example, the vehicle detection data acquisition unit 702 generates three-dimensional detection data 712 of the vehicle itself based on sensor information acquired by the sensors equipped on the vehicle.

[0427] Next, the anomaly detection unit 703 detects anomalies by performing a predetermined check on at least one of the acquired 3D map 711 and its own vehicle detection 3D data 712. That is, the anomaly detection unit 703 determines whether at least one of the acquired 3D map 711 and its own vehicle detection 3D data 712 is abnormal.

[0428] If an anomaly is detected, the response decision-making unit 704 decides on the response measures. Next, the work control unit 705 controls the work of the various processing units, such as the 3D map acquisition unit 701, required for the implementation of the response measures.

[0429] Furthermore, if no abnormalities are detected, the 3D information processing device 700 terminates the processing.

[0430] Furthermore, the 3D information processing device 700 uses the 3D map 711 and its own vehicle detection 3D data 712 to estimate the vehicle's own position. Then, the 3D information processing device 700 uses the result of its own position estimation to enable the vehicle to perform autonomous driving.

[0431] Accordingly, the three-dimensional information processing device 700 obtains map data (three-dimensional map 711) including first three-dimensional location information via a channel. For example, the first three-dimensional location information is encoded in units of partial space having three-dimensional coordinate information. The first three-dimensional location information includes multiple random access units, each of which is a collection of more than one partial space and can be decoded independently. For example, the first three-dimensional location information is data (SWLD) encoded by feature points whose three-dimensional feature values ​​are above a predetermined threshold.

[0432] Furthermore, the three-dimensional information processing device 700 generates second three-dimensional position information (three-dimensional data 712 detected by the vehicle itself) based on the information detected by the sensors. Next, the three-dimensional information processing device 700 determines whether the first three-dimensional position information or the second three-dimensional position information is abnormal by performing anomaly detection processing on the first three-dimensional position information or the second three-dimensional position information.

[0433] When the 3D information processing device 700 determines that either the first or second 3D position information is abnormal, it decides on a response to the abnormality. Then, the 3D information processing device 700 executes the controls necessary for implementing the response.

[0434] Accordingly, the three-dimensional information processing device 700 can detect anomalies in the first or second three-dimensional position information and take appropriate action.

[0435] (Implementation Method 5) In this embodiment, the method for sending three-dimensional data to vehicles behind will be described.

[0436] Figure 27 This is a block diagram illustrating a configuration example of the three-dimensional data creation apparatus 810 according to this embodiment. The three-dimensional data creation apparatus 810 is, for example, mounted on a vehicle. The three-dimensional data creation apparatus 810 transmits and receives three-dimensional data from external traffic cloud monitoring systems, vehicles ahead, or vehicles behind, while simultaneously creating and accumulating the three-dimensional data.

[0437] The 3D data production apparatus 810 includes: a data receiving unit 811, a communication unit 812, a receiving control unit 813, a format conversion unit 814, multiple sensors 815, a 3D data production unit 816, a 3D data synthesis unit 817, a 3D data storage unit 818, a communication unit 819, a transmission control unit 820, a format conversion unit 821, and a data transmission unit 822.

[0438] The data receiving unit 811 receives 3D data 831 from traffic cloud monitoring or vehicles ahead. The 3D data 831 includes, for example, point clouds containing information about areas that the vehicle's own sensors 815 cannot detect, visible light images, depth information, sensor position information, or speed information.

[0439] The 812 communication unit communicates with the traffic cloud monitoring system or the vehicle ahead, sending data transmission requests and other information to the traffic cloud monitoring system or the vehicle ahead.

[0440] The receiving control unit 813 exchanges the corresponding format and other information with the communication counterpart via the communication unit 812 to establish communication with the communication counterpart.

[0441] The format conversion unit 814 generates three-dimensional data 832 by performing format conversion on the three-dimensional data 831 received by the data receiving unit 811. Furthermore, the format conversion unit 814 performs decompression or decoding processing when the three-dimensional data 831 is compressed or encoded.

[0442] Multiple sensors 815 are a group of sensors, such as LiDAR, visible light cameras, or infrared cameras, that acquire information about the exterior of the vehicle, generating sensor information 833. For example, if the sensors 815 are laser sensors such as LiDAR, the sensor information 833 is three-dimensional data such as point clouds (point group data). Alternatively, there may be more than one sensor 815.

[0443] The 3D data production unit 816 generates 3D data 834 based on sensor information 833. 3D data 834 may include, for example, point clouds, visible light images, depth information, sensor position information, or velocity information.

[0444] The 3D data synthesis unit 817 synthesizes 3D data 832 generated by traffic cloud monitoring or vehicles ahead into 3D data 834 generated based on the sensor information 833 of its own vehicle, thereby constructing 3D data 835 that includes the space in front of the vehicle ahead, which cannot be detected by the sensors 815 of its own vehicle.

[0445] The three-dimensional data storage unit 818 stores the generated three-dimensional data 835, etc.

[0446] The communication unit 819 communicates with the traffic cloud monitoring system or vehicles behind it, sending data transmission requests and other information to the traffic cloud monitoring system or vehicles behind it.

[0447] The transmission control unit 820 exchanges corresponding format and other information with the communication counterpart via the communication unit 819 to establish communication with the communication counterpart. Furthermore, the transmission control unit 820 determines the transmission area of ​​the spatial dimension of the dimension data to be transmitted based on the dimension data construction information of the dimension data 832 generated in the dimension data synthesis unit 817 and the data transmission request from the communication counterpart.

[0448] Specifically, the transmission control unit 820 determines the transmission area, including the space in front of its own vehicle that cannot be detected by the sensors of the rear vehicle, based on data transmission requests from traffic cloud monitoring or rear vehicles. Furthermore, the transmission control unit 820 determines the transmission area by judging whether there are updates to the space that can be transmitted or the space that has already been transmitted, based on three-dimensional data construction information. For example, the transmission control unit 820 determines the transmission area as the area that is both specified by the data transmission request and where the corresponding three-dimensional data 835 exists. The transmission control unit 820 also notifies the format conversion unit 821 of the format corresponding to the communication counterpart and the transmission area.

[0449] The format conversion unit 821 generates three-dimensional data 837 by converting the three-dimensional data 836 of the transmission area stored in the three-dimensional data 835 of the three-dimensional data storage unit 818 into a format corresponding to the receiving side. Alternatively, the format conversion unit 821 can compress or encode the three-dimensional data 837 to reduce the data volume.

[0450] The data transmission unit 822 sends the three-dimensional data 837 to the traffic cloud monitoring or the vehicle behind. The three-dimensional data 837 may include, for example, point clouds of the front of the vehicle containing information about areas that would become blind spots for the vehicle behind, visible light images, depth information, or sensor position information.

[0451] Furthermore, although the format conversion units 814 and 821 are used as examples for format conversion, it is also possible to omit format conversion.

[0452] With this configuration, the 3D data generation apparatus 810 obtains 3D data 831 from an external source in an area that cannot be detected by the vehicle's own sensors 815, and generates 3D data 835 by combining the 3D data 831 with 3D data 834 based on sensor information 833 detected by the vehicle's own sensors 815. Accordingly, the 3D data generation apparatus 810 is capable of generating 3D data for a range that cannot be detected by the vehicle's own sensors 815.

[0453] Furthermore, the 3D data generation device 810 can send 3D data, including the space in front of its own vehicle that cannot be detected by the sensors of the rear vehicle, to the traffic cloud monitoring or rear vehicle according to data transmission requests from traffic cloud monitoring or rear vehicles.

[0454] (Implementation Method 6) In an example to be described in Implementation 5, a client device such as a vehicle sends 3D data to other vehicle or traffic cloud monitoring servers. In this implementation, the client device sends sensor information obtained from sensors to a server or other client devices.

[0455] First, the configuration of the system involved in this embodiment will be explained. Figure 28 The configuration of the three-dimensional map and sensor information transceiver system according to this embodiment is shown. The system includes a server 901, client devices 902A and 902B. Unless otherwise specified, client devices 902A and 902B are also referred to as client device 902.

[0456] The client device 902 is, for example, an in-vehicle device mounted on a moving vehicle. The server 901 is, for example, a traffic cloud monitoring system, capable of communicating with multiple client devices 902.

[0457] Server 901 sends a 3D map composed of point clouds to client device 902. Furthermore, the composition of the 3D map is not limited to point clouds; it can also be represented using other 3D data such as grid structures.

[0458] The client device 902 sends sensor information obtained by the client device 902 to the server 901. The sensor information includes, for example, at least one of the following: LiDAR-acquired information, visible light image, infrared image, depth image, sensor position information, and velocity information.

[0459] Regarding the data transmitted and received between server 901 and client device 902, compression can be performed to reduce data volume, while compression can be omitted to maintain data accuracy. When compressing data, for example, an octree-based 3D compression method can be used for point clouds. Furthermore, 2D image compression methods can be used for visible light images, infrared images, and depth images. 2D image compression methods include, for example, MPEG-4 AVC or HEVC standardized by MPEG.

[0460] Furthermore, server 901 sends the 3D map managed by server 901 to client device 902 according to the 3D map sending request from client device 902. Alternatively, server 901 may send the 3D map without waiting for a sending request from client device 902. For example, server 901 may broadcast the 3D map to more than one client device 902 in a predefined space. Server 901 may also send a 3D map appropriate to the location of client device 902 at regular intervals to client devices 902 that have received a sending request once. Furthermore, server 901 may send the 3D map to client device 902 whenever the 3D map managed by server 901 is updated.

[0461] Client device 902 sends a request to server 901 to send a 3D map. For example, if client device 902 wants to estimate its own position while driving, it sends the request to server 901.

[0462] Additionally, client device 902 may also send a request to server 901 to transmit a 3D map in the following situations: If the 3D map held by client device 902 is relatively old, client device 902 may also send a request to server 901 to transmit the 3D map. For example, if client device 902 has obtained the 3D map and a certain period of time has passed, client device 902 may also send a request to server 901 to transmit the 3D map.

[0463] Alternatively, the client device 902 may send a request to the server 901 to transmit the 3D map a certain time before it is about to leave the space shown on the 3D map held by the client device 902. For example, the client device 902 may send the request to the server 901 when it is within a pre-defined distance from the boundary of the space shown on the 3D map held by the client device 902. Furthermore, knowing the movement path and speed of the client device 902, the time when the client device 902 will leave the space shown on the 3D map held by the client device 902 can be predicted based on the known movement path and speed.

[0464] When the error between the three-dimensional data generated by the client device 902 based on sensor information and the position of the three-dimensional map exceeds a certain range, the client device 902 can send a request to the server 901 to send the three-dimensional map.

[0465] Client device 902 sends sensor information to server 901 according to the sensor information transmission request sent from server 901. Alternatively, client device 902 may send sensor information to server 901 without waiting for a transmission request from server 901. For example, if client device 902 has received a sensor information transmission request from server 901 once, it may periodically send sensor information to server 901 within a certain period. Furthermore, if the error between the 3D data generated by client device 902 based on sensor information and the 3D map obtained from server 901 exceeds a certain range, client device 902 determines that the 3D map around client device 902 may have changed, and sends this determination along with the sensor information to server 901.

[0466] Server 901 sends a sensor information transmission request to client device 902. For example, server 901 receives location information of client device 902, such as GPS data, from client device 902. Based on the location information of client device 902, if server 901 determines that client device 902 is approaching an area with limited information in the 3D map managed by server 901, it sends a sensor information transmission request to client device 902 in order to regenerate the 3D map. Alternatively, server 901 may also send sensor information transmission requests when it wants to update the 3D map, check road conditions such as snow accumulation or disasters, or check traffic congestion or accident situations.

[0467] Alternatively, the client device 902 can set the amount of sensor information to be sent to the server 901 based on the communication state or frequency band at the time of receiving the sensor information transmission request from the server 901. Setting the amount of sensor information to be sent to the server 901 can mean, for example, increasing or decreasing the amount of data itself, or selecting an appropriate compression method.

[0468] Figure 29 This is a block diagram illustrating an example of the configuration of client device 902. Client device 902 receives a 3D map, such as a point cloud, from server 901, and estimates its own position based on 3D data generated from sensor information of client device 902. Furthermore, client device 902 sends the obtained sensor information to server 901.

[0469] The client device 902 includes: a data receiving unit 1011, a communication unit 1012, a receiving control unit 1013, a format conversion unit 1014, multiple sensors 1015, a 3D data production unit 1016, a 3D image processing unit 1017, a 3D data storage unit 1018, a format conversion unit 1019, a communication unit 1020, a transmission control unit 1021, and a data transmission unit 1022.

[0470] The data receiving unit 1011 receives the 3D map 1031 from the server 901. The 3D map 1031 includes point cloud data such as WLD or SWLD. The 3D map 1031 may also include either compressed data or uncompressed data.

[0471] The communication unit 1012 communicates with the server 901 and sends data transmission requests (e.g., a 3D map transmission request) to the server 901.

[0472] The receiving control unit 1013 exchanges information such as the corresponding format with the communication counterpart via the communication unit 1012 to establish communication with the communication counterpart.

[0473] The format conversion unit 1014 generates a 3D map 1032 by performing format conversion and other operations on the 3D map 1031 received by the data receiving unit 1011. Furthermore, the format conversion unit 1014 performs decompression or decoding processing when the 3D map 1031 is compressed or encoded. Conversely, the format conversion unit 1014 does not perform decompression or decoding processing when the 3D map 1031 is uncompressed data.

[0474] Multiple sensors 1015 are a group of sensors mounted on the client device 902, such as LiDAR, visible light cameras, infrared cameras, or depth sensors, used to obtain information about the exterior of the vehicle, generating sensor information 1033. For example, if the sensor 1015 is a laser sensor such as LiDAR, the sensor information 1033 is three-dimensional data such as point cloud (point group data). Alternatively, there may be more than one sensor 1015.

[0475] The 3D data production unit 1016 produces 3D data 1034 of the vehicle's surroundings based on sensor information 1033. For example, the 3D data production unit 1016 uses information obtained by LiDAR and visible light images obtained by a visible light camera to produce point cloud data with color information of the vehicle's surroundings.

[0476] The 3D image processing unit 1017 uses the received point cloud and other 3D map 1032, as well as the 3D surrounding data 1034 generated based on sensor information 1033, to perform self-position estimation processing for the vehicle. Alternatively, the 3D image processing unit 1017 can synthesize the 3D map 1032 and the 3D data 1034 to create 3D surrounding data 1035 for the vehicle, and then use the created 3D data 1035 to perform self-position estimation processing.

[0477] The three-dimensional data storage unit 1018 stores three-dimensional maps 1032, three-dimensional data 1034, and three-dimensional data 1035.

[0478] The format conversion unit 1019 generates sensor information 1037 by converting sensor information 1033 into a format corresponding to the receiving side. Furthermore, the format conversion unit 1019 can reduce the data volume by compressing or encoding the sensor information 1037. Moreover, the format conversion unit 1019 can omit processing when format conversion is not required. Furthermore, the format conversion unit 1019 can control the amount of data transmitted according to a specified transmission range.

[0479] The communication unit 1020 communicates with the server 901 and receives data transmission requests (sensor information transmission requests) from the server 901.

[0480] The transmission control unit 1021 exchanges information such as the corresponding format with the communication counterpart via the communication unit 1020, thereby establishing communication.

[0481] The data transmission unit 1022 sends the sensor information 1037 to the server 901. The sensor information 1037 includes, for example, information obtained by LiDAR, brightness images (visible light images) obtained by a visible light camera, infrared images obtained by an infrared camera, depth images obtained by a depth sensor, sensor position information, and velocity information, etc., obtained by multiple sensors 1015.

[0482] Next, the structure of server 901 will be explained. Figure 30 This is a block diagram illustrating an example of the configuration of server 901. Server 901 receives sensor information sent from client device 902 and creates 3D data based on the received sensor information. Server 901 uses the created 3D data to update the 3D map managed by server 901. Furthermore, server 901 sends the updated 3D map to client device 902 according to a 3D map sending request from client device 902.

[0483] Server 901 includes: a data receiving unit 1111, a communication unit 1112, a receiving control unit 1113, a format conversion unit 1114, a 3D data production unit 1116, a 3D data synthesis unit 1117, a 3D data storage unit 1118, a format conversion unit 1119, a communication unit 1120, a transmission control unit 1121, and a data transmission unit 1122.

[0484] The data receiving unit 1111 receives sensor information 1037 from the client device 902. The sensor information 1037 includes, for example, information obtained by LiDAR, brightness images (visible light images) obtained by a visible light camera, infrared images obtained by an infrared camera, depth images obtained by a depth sensor, sensor position information, and speed information.

[0485] The communication unit 1112 communicates with the client device 902 and sends data transmission requests (e.g., sensor information transmission requests) to the client device 902.

[0486] The receiving control unit 1113 exchanges information such as the corresponding format with the communication counterpart via the communication unit 1112, thereby establishing communication.

[0487] When the received sensor information 1037 is compressed or encoded, the format conversion unit 1114 generates sensor information 1132 by performing decompression or decoding processing. Conversely, when the sensor information 1037 is uncompressed data, the format conversion unit 1114 does not perform decompression or decoding processing.

[0488] The 3D data production unit 1116 produces 3D data 1134 of the surrounding area of ​​the client device 902 based on sensor information 1132. For example, the 3D data production unit 1116 uses information obtained by LiDAR and visible light images obtained by a visible light camera to produce point cloud data with color information of the surrounding area of ​​the client device 902.

[0489] The 3D data synthesis unit 1117 synthesizes the 3D data 1134 produced based on sensor information 1132 with the 3D map 1135 managed by the server 901, thereby updating the 3D map 1135.

[0490] The three-dimensional data storage unit 1118 stores three-dimensional maps such as 1135.

[0491] The format conversion unit 1119 generates the 3D map 1031 by converting the 3D map 1135 into a format corresponding to the receiving side. Alternatively, the format conversion unit 1119 can reduce the data volume by compressing or encoding the 3D map 1135. Furthermore, the format conversion unit 1119 can omit this process when format conversion is not required. The format conversion unit 1119 can also control the amount of data transmitted according to a specified transmission range.

[0492] The communication unit 1120 communicates with the client device 902 and receives data transmission requests (such as 3D map transmission requests) from the client device 902.

[0493] The transmission control unit 1121 exchanges information such as the corresponding format with the communication counterpart via the communication unit 1120, thereby establishing communication.

[0494] The data transmission unit 1122 sends the 3D map 1031 to the client device 902. The 3D map 1031 includes point cloud data such as WLD or SWLD. The 3D map 1031 may also include either compressed data or uncompressed data.

[0495] Next, the workflow of the client device 902 will be explained. Figure 31 This is a flowchart illustrating the process of acquiring a 3D map using the client device 902.

[0496] First, the client device 902 requests the server 901 to send a 3D map (point cloud, etc.) (S1001). At this time, the client device 902 also sends its location information obtained through GPS, etc., and can then request the server 901 to send a 3D map related to that location information.

[0497] Next, the client device 902 receives a 3D map from the server 901 (S1002). If the received 3D map is compressed data, the client device 902 decodes the received 3D map to generate an uncompressed 3D map (S1003).

[0498] Next, the client device 902 generates 3D data 1034 of its surroundings based on sensor information 1033 obtained from multiple sensors 1015 (S1004). Then, the client device 902 estimates its own position using the 3D map 1032 received from the server 901 and the 3D data 1034 generated from the sensor information 1033 (S1005).

[0499] Figure 32 This is a flowchart illustrating the operation of the client device 902 when transmitting sensor information. First, the client device 902 receives a sensor information transmission request from the server 901 (S1011). Having received the transmission request, the client device 902 sends the sensor information 1037 to the server 901 (S1012). Furthermore, if the sensor information 1033 includes multiple pieces of information obtained from multiple sensors 1015, the client device 902 compresses each piece of information using a compression method suitable for that information, thereby generating the sensor information 1037.

[0500] Next, the workflow of server 901 will be explained. Figure 33 This is a flowchart illustrating the operation of server 901 when acquiring sensor information. First, server 901 requests the transmission of sensor information from client device 902 (S1021). Next, server 901 receives sensor information 1037 transmitted from client device 902 according to the request (S1022). Then, server 901 uses the received sensor information 1037 to create 3D data 1134 (S1023). Finally, server 901 reflects the created 3D data 1134 onto a 3D map 1135 (S1024).

[0501] Figure 34This is a flowchart illustrating the operation of server 901 when sending a 3D map. First, server 901 receives a 3D map transmission request from client device 902 (S1031). Having received the 3D map transmission request, server 901 sends the 3D map 1031 to client device 902 (S1032). At this time, server 901 can extract a nearby 3D map corresponding to the location information of client device 902 and send the extracted 3D map. Alternatively, server 901 can compress the 3D map composed of point clouds, for example, using an octree compression method, and send the compressed 3D map.

[0502] Hereinafter, variations of this embodiment will be described.

[0503] Server 901 uses sensor information 1037 received from client device 902 to create 3D data 1134 of the vicinity of client device 902's location. Next, server 901 matches the created 3D data 1134 with a 3D map 1135 of the same area managed by server 901, calculating the difference between the 3D data 1134 and the 3D map 1135. If the difference exceeds a predetermined threshold, server 901 determines that an anomaly has occurred in the vicinity of client device 902. For example, in the event of ground subsidence due to natural disasters such as earthquakes, a significant difference can be considered between the 3D map 1135 managed by server 901 and the 3D data 1134 created based on sensor information 1037.

[0504] Sensor information 1037 may include at least one of the following: sensor type, sensor performance, and sensor model. It may also include a category ID corresponding to the sensor's performance. For example, if sensor information 1037 is information obtained from LiDAR, it's possible to assign identifiers based on sensor performance; for instance, assigning category 1 to sensors capable of obtaining information with an accuracy of a few millimeters, category 2 to sensors capable of obtaining information with an accuracy of a few centimeters, and category 3 to sensors capable of obtaining information with an accuracy of a few meters. Furthermore, server 901 may estimate sensor performance information from the model of client device 902. For example, if client device 902 is mounted on a vehicle, server 901 can determine the sensor specifications based on the vehicle's model. In this case, server 901 can obtain the vehicle model information beforehand and include that information in the sensor information. It's also possible that server 901 uses the obtained sensor information 1037 to switch the degree of correction for the 3D data 1134 created using sensor information 1037. For example, when the sensor performance is high accuracy (Category 1), server 901 does not perform calibration for the 3D data 1134. When the sensor performance is low accuracy (Category 3), server 901 applies calibration appropriate to the sensor's accuracy to the 3D data 1134. For example, the lower the sensor's accuracy, the greater the degree (intensity) of the calibration.

[0505] Server 901 can also simultaneously send sensor information transmission requests to multiple client devices 902 existing in a certain space. When server 901 receives multiple sensor information from multiple client devices 902, it is not necessary to utilize all the sensor information in the creation of 3D data 1134. For example, it can select the sensor information to be used based on the sensor's performance. For instance, when updating the 3D map 1135, server 901 can select high-precision sensor information (category 1) from the received sensor information and use the selected sensor information to create 3D data 1134.

[0506] Server 901 is not limited to servers such as traffic cloud monitoring; it can also be other client devices (vehicle-mounted). Figure 35 The system configuration in this case is shown.

[0507] For example, client device 902C sends a sensor information transmission request to a nearby client device 902A and obtains the sensor information from client device 902A. Then, client device 902C uses the obtained sensor information from client device 902A to create 3D data and update its own 3D map. In this way, client device 902C can utilize its performance to generate a 3D map of the space obtained from client device 902A. This scenario is possible, for example, when client device 902C has high performance.

[0508] Furthermore, in this scenario, the client device 902A, which provides sensor information, is granted the right to obtain a high-precision 3D map generated by the client device 902C. The client device 902A receives the high-precision 3D map from the client device 902C in accordance with this right.

[0509] Alternatively, client device 902C may send sensor information transmission requests to multiple nearby client devices 902 (client device 902A and client device 902B). If the sensor of client device 902A or client device 902B is high-performance, client device 902C can use the sensor information obtained through the high-performance sensor to create three-dimensional data.

[0510] Figure 36 This is a block diagram illustrating the functional configuration of server 901 and client device 902. Server 901 includes, for example, a 3D map compression / decoding processing unit 1201 for compressing and decoding 3D maps, and a sensor information compression / decoding processing unit 1202 for compressing and decoding sensor information.

[0511] The client device 902 includes a 3D map decoding processing unit 1211 and a sensor information compression processing unit 1212. The 3D map decoding processing unit 1211 receives compressed coded data of a 3D map, decodes the coded data, and obtains a 3D map. The sensor information compression processing unit 1212 does not compress the 3D data created from the obtained sensor information; instead, it compresses the sensor information itself and sends the compressed coded data of the sensor information to the server 901. With this configuration, the client device 902 can internally house the processing unit (device or LSI) used for decoding 3D maps (point clouds, etc.), without needing to internally house the processing unit used for compressing 3D data (point clouds, etc.). This reduces the cost and power consumption of the client device 902.

[0512] As described above, the client device 902 of this embodiment is mounted on a mobile body. Based on sensor information 1033 obtained by the sensor 1015 mounted on the mobile body, which indicates the surrounding conditions of the mobile body, three-dimensional data 1034 of the mobile body's surroundings is generated. The client device 902 uses the generated three-dimensional data 1034 to estimate the mobile body's own position. The client device 902 sends the obtained sensor information 1033 to the server 901 or another mobile body 902.

[0513] Accordingly, the client device 902 sends the sensor information 1033 to the server 901, etc. This potentially reduces the amount of data transmitted compared to transmitting 3D data. Furthermore, since it is unnecessary to perform 3D data compression or encoding on the client device 902, the processing load on the client device 902 can be reduced. Therefore, the client device 902 can achieve a reduction in the amount of data transmitted or a simplification of its configuration.

[0514] Furthermore, the client device 902 sends a request to the server 901 to send a 3D map, and receives the 3D map 1031 from the server 901. In estimating its own position, the client device 902 uses the 3D data 1034 and the 3D map 1032 to estimate its own position.

[0515] Furthermore, the sensor information 1033 includes at least one of the following: information obtained through the laser sensor, brightness image (visible light image), infrared image, depth image, sensor position information, and sensor speed information.

[0516] Furthermore, sensor information 1033 includes information indicating the performance of the sensor.

[0517] Furthermore, the client device 902 encodes or compresses the sensor information 1033, and during the transmission of sensor information, sends the encoded or compressed sensor information 1037 to the server 901 or other mobile device 902. Accordingly, the client device 902 can reduce the amount of data transmitted.

[0518] For example, the client device 902 has a processor and a memory, and the processor uses the memory to perform the above-mentioned processing.

[0519] Furthermore, the server 901 according to this embodiment can communicate with the client device 902 mounted on the mobile body, and receive sensor information 1037 from the client device 902, which is obtained by the sensor 1015 mounted on the mobile body and shows the surrounding conditions of the mobile body. The server 901 generates three-dimensional data 1134 of the surroundings of the mobile body based on the received sensor information 1037.

[0520] Accordingly, server 901 uses sensor information 1037 sent from client device 902 to create 3D data 1134. This potentially reduces the amount of data transmitted compared to when client device 902 sends 3D data directly. Furthermore, since 3D data compression or encoding is unnecessary on client device 902, its processing load can be reduced. Thus, server 901 can achieve a reduction in the amount of data transmitted or a simplification of the device configuration.

[0521] Furthermore, server 901 sends a request to the client device 902 to transmit sensor information.

[0522] Furthermore, server 901 uses the generated 3D data 1134 to update 3D map 1135, and sends 3D map 1135 to client device 902 according to the sending request of 3D map 1135 from client device 902.

[0523] Furthermore, the sensor information 1037 includes at least one of the following: information obtained through the laser sensor, brightness image (visible light image), infrared image, depth image, sensor position information, and sensor speed information.

[0524] Furthermore, sensor information 1037 includes information indicating the performance of the sensor.

[0525] Furthermore, server 901 further corrects the 3D data according to the sensor's performance. Therefore, this 3D data creation method can improve the quality of 3D data.

[0526] Furthermore, during the reception of sensor information, server 901 receives multiple sensor information 1037 from multiple client devices 902, and selects the sensor information 1037 to be used in the production of 3D data 1134 based on multiple pieces of information including sensor performance included in the multiple sensor information 1037. Accordingly, server 901 can improve the quality of 3D data 1134.

[0527] Furthermore, server 901 decodes or decompresses the received sensor information 1037, and generates 3D data 1134 based on the decoded or decompressed sensor information 1132. In this way, server 901 can reduce the amount of data transmitted.

[0528] For example, server 901 has a processor and memory, and the processor uses the memory to perform the above-mentioned processing.

[0529] (Implementation Method 7) In this embodiment, the encoding and decoding methods for three-dimensional data utilizing inter-frame prediction processing are described.

[0530] Figure 37 This is a block diagram of the three-dimensional data encoding apparatus 1300 according to this embodiment. This three-dimensional data encoding apparatus 1300 generates an encoded bitstream (hereinafter simply referred to as a bitstream) as an encoded signal by encoding three-dimensional data. Figure 37 As shown, the three-dimensional data encoding device 1300 includes: a segmentation unit 1301, a subtraction unit 1302, a transformation unit 1303, a quantization unit 1304, an inverse quantization unit 1305, an inverse transformation unit 1306, an addition unit 1307, a reference volume memory 1308, an intra-frame prediction unit 1309, a reference space memory 1310, an inter-frame prediction unit 1311, a prediction control unit 1312, and an entropy encoding unit 1313.

[0531] The segmentation unit 1301 divides each spatial component (SPC) contained in the 3D data into multiple volumes (VLMs) as encoding units. Furthermore, the segmentation unit 1301 performs octree representation (octreeification) on the voxels within each volume. Alternatively, the segmentation unit 1301 can make the space and volume the same size and perform octree representation on the space. The segmentation unit 1301 can also append information required for octreeification (depth information, etc.) to the header of the bitstream.

[0532] The subtraction unit 1302 calculates the difference between the volume (encoded object volume) output from the segmentation unit 1301 and the prediction volume generated by intra-frame prediction or inter-frame prediction as described later, and outputs the calculated difference as the prediction residual to the transform unit 1303. Figure 38 An example of calculating the prediction residual is shown. Additionally, the bit strings of the encoded object volume and the prediction volume shown here, for example, indicate the location information of the three-dimensional points (e.g., point clouds) contained within the volume.

[0533] The following explains the octree representation and the voxel scanning order. The volume is transformed into an octree structure (octreeification) and then encoded. The octree structure consists of nodes and leaf nodes. Each node has 8 nodes or leaf nodes, and each leaf node contains voxel (VXL) information. Figure 39 An example of the composition of a volume including multiple voxels is shown. Figure 40 It shows that Figure 39 The volume transformation shown is an example of an octree structure. Here, Figure 40 Leaf nodes 1, 2, and 3 in the leaf nodes shown represent, respectively Figure 39 The voxels VXL1, VXL2, and VXL3 shown represent VXL including point groups (hereinafter referred to as effective VXL).

[0534] An octree can be represented as a binary sequence of 0s and 1s. For example, when a node or valid VXL is set to a value of 1, and all others are set to a value of 0, the nodes and leaf nodes are assigned values. Figure 40 The binary sequence is shown. Therefore, this binary sequence is scanned according to either width-first or depth-first scanning order. For example, in the case of a width-first scan, we obtain... Figure 41 The binary sequence A is shown. After a depth-first scan, the following is obtained: Figure 41 The binary sequence shown in B. The binary sequence obtained through this scan is encoded by entropy coding, thereby reducing the amount of information.

[0535] Next, we will explain the depth information in the octree representation. The depth in the octree representation controls the granularity at which the point cloud information contained within a volume is preserved. Setting a large depth allows for the reproduction of point cloud information at a finer level, but this increases the amount of data used to represent nodes and leaf nodes. Conversely, setting a small depth reduces the amount of data, but multiple point cloud information points at different locations and with different colors will be treated as the same location and color, thus losing the original information inherent in the point cloud.

[0536] For example, Figure 42 It shows that Figure 40 The example shown is an octree with a depth of 2, represented as an octree with a depth of 1. Figure 42 The octree shown is more Figure 40 The octree shown has a small amount of data. That is, Figure 42 The octree shown Figure 42 Compared to the octree shown, the binary serialization has fewer bits. Here, Figure 40 Leaf node 1 and leaf node 2 shown become, in order to Figure 41 This is represented by leaf node 1. That is, it has lost... Figure 40 The information shown is that leaf node 1 and leaf node 2 are in different positions.

[0537] Figure 43 It shows the relationship with Figure 42 The volume corresponding to the octree shown. Figure 39 The VXL1 and VXL2 shown are... Figure 43 The VXL12 shown corresponds to this. In this case, the three-dimensional data encoding device 1300 is based on... Figure 39 The color information of VXL1 and VXL2 shown is used to generate... Figure 43The color information of VXL12 is shown. For example, the three-dimensional data encoding device 1300 calculates the color information of VXL12 using the average value, median value, or weighted average value of the color information of VXL1 and VXL2. In this way, the three-dimensional data encoding device 1300 can control the reduction of data volume by changing the depth of the octree.

[0538] The 3D data encoding device 1300 can also use any one of the world space units, spatial units, and volume units to set the depth information of the octree. Furthermore, the 3D data encoding device 1300 can also append the depth information to the header information of the world space, the header information of the spatial space, or the header information of the volume. Moreover, the same value can be used as depth information in all world spaces, spatial spaces, and volumes at different times. In this case, the 3D data encoding device 1300 can also append the depth information to the header information that manages the world space at all times.

[0539] When voxels contain color information, the transformation unit 1303 applies frequency transformations, such as orthogonal transformations, to the prediction residuals of the color information of the voxels within the volume. For example, the transformation unit 1303 scans the prediction residuals in a certain scanning order to create a one-dimensional arrangement. Afterward, the transformation unit 1303 transforms the created one-dimensional arrangement into the frequency domain by applying a one-dimensional orthogonal transformation. Accordingly, when the values ​​of the prediction residuals within the volume are similar, the values ​​of the low-frequency components increase, and the values ​​of the high-frequency components decrease. Therefore, the quantization unit 1304 can more effectively reduce the amount of coding.

[0540] Furthermore, the transformation unit 1303 can utilize orthogonal transformations of two dimensions or higher, instead of one-dimensional orthogonal transformations. For example, the transformation unit 1303 maps the prediction residuals to a two-dimensional arrangement in a certain scanning order, and applies a two-dimensional orthogonal transformation to the resulting two-dimensional arrangement. The transformation unit 1303 can also select the orthogonal transformation method to be used from multiple orthogonal transformation methods. In this case, the three-dimensional data encoding device 1300 appends information indicating which orthogonal transformation method was used to the bitstream. Alternatively, the transformation unit 1303 can select the orthogonal transformation method to be used from multiple orthogonal transformation methods with different dimensions. In this case, the three-dimensional data encoding device 1300 appends information indicating which dimension of orthogonal transformation method was used to the bitstream.

[0541] For example, the transformation unit 1303 matches the scan order of the predicted residual with the scan order (width-first or depth-first, etc.) in the octree within the volume. Therefore, since it is not necessary to append information indicating the scan order of the predicted residual to the bitstream, additional overhead can be reduced. Furthermore, the transformation unit 1303 can also apply a scan order different from the octree scan order. In this case, the three-dimensional data encoding device 1300 appends information indicating the scan order of the predicted residual to the bitstream. Therefore, the three-dimensional data encoding device 1300 can efficiently encode the predicted residual. Alternatively, the three-dimensional data encoding device 1300 may append information indicating whether the octree scan order is applicable (a flag, etc.) to the bitstream; if the octree scan order is not applicable, it may append information indicating the scan order of the predicted residual to the bitstream.

[0542] The transformation unit 1303 can transform not only the prediction residual of color information, but also other attribute information of voxels. For example, the transformation unit 1303 can transform and encode information such as reflectance obtained when acquiring point clouds through LiDAR or the like.

[0543] If the transformation unit 1303 does not have attribute information such as color information in the space, it can skip processing. Furthermore, the three-dimensional data encoding device 1300 can append information (flags) indicating whether to skip the processing of the transformation unit 1303 to the bit stream.

[0544] The quantization unit 1304 quantizes the frequency components of the prediction residuals generated in the transformation unit 1303 using quantization control parameters, thereby generating quantization coefficients. This reduces the amount of information. The generated quantization coefficients are output to the entropy encoding unit 1313. The quantization unit 1304 can control the quantization control parameters according to world space units, spatial units, or volume units. At this time, the three-dimensional data encoding device 1300 attaches the quantization control parameters to the respective header information, etc. Furthermore, the quantization unit 1304 can also perform quantization control by changing the weights according to the frequency components of each prediction residual. For example, the quantization unit 1304 can perform fine quantization on low-frequency components and coarse quantization on high-frequency components. In this case, the three-dimensional data encoding device 1300 can attach parameters representing the weights of each frequency component to the header.

[0545] The quantization unit 1304 can skip processing if the space does not have attribute information such as color information. Furthermore, the three-dimensional data encoding device 1300 can also append information (flags) indicating whether to skip the processing of the quantization unit 1304 to the bitstream.

[0546] The inverse quantization unit 1305 uses quantization control parameters to inverse quantize the quantization coefficients generated by the quantization unit 1304, thereby generating inverse quantization coefficients of the prediction residuals, and outputting the generated inverse quantization coefficients to the inverse transform unit 1306.

[0547] The inverse transform unit 1306 applies an inverse transform to the inverse quantization coefficients generated in the inverse quantization unit 1305, thereby generating a prediction residual after the inverse transform is applied. Since this prediction residual after the inverse transform is a prediction residual generated after quantization, it may not be completely consistent with the prediction residual output by the transform unit 1303.

[0548] The addition unit 1307 adds the prediction residual after the inverse transform generated by the inverse transform unit 1306 to the prediction volume generated by intra-frame prediction or inter-frame prediction used in the generation of the prediction residual before quantization, to generate a reconstruction volume. This reconstruction volume is stored in the reference volume memory 1308 or the reference space memory 1310.

[0549] The intra-prediction unit 1309 generates a predicted volume of the encoded object volume using attribute information of adjacent volumes stored in the reference volume memory 1308. The attribute information includes voxel color information or reflectance. The intra-prediction unit 1309 generates predicted values ​​for the color information or reflectance of the encoded object volume.

[0550] Figure 44 This is a diagram used to illustrate the operation of the intra-frame prediction unit 1309. For example, Figure 44 As shown, the intra-frame prediction unit 1309 generates a predicted volume for the encoded object volume (volume idx=3) based on adjacent volumes (volume idx=0). Here, volume idx is identifier information added to volumes within the space, and different values ​​are assigned to each volume. The order in which volume idx is assigned can be the same as or different from the encoding order. For example, as... Figure 44 The intra-frame prediction unit 1309 uses the average value of the color information of the voxels contained in the adjacent volume idx=0 as the predicted value of the color information of the encoded object volume. In this case, a prediction residual is generated by subtracting the predicted value of the color information from the color information of each voxel contained in the encoded object volume. The subsequent processing of the transformation unit 1303 is performed on this prediction residual. Furthermore, in this case, the 3D data encoding apparatus 1300 appends adjacent volume information and prediction mode information to the bitstream. Here, the adjacent volume information shows the information of the adjacent volumes used in the prediction, such as showing the volume idx of the adjacent volumes used in the prediction. And the prediction mode information shows the mode used in the generation of the prediction volume. The mode is, for example, an average value mode that generates the prediction value based on the average value of the voxels in the adjacent volumes, or an intermediate value mode that generates the prediction value based on the median value of the voxels in the adjacent volumes, etc.

[0551] The intra-frame prediction unit 1309 can also generate a prediction volume based on multiple adjacent volumes. For example, in Figure 44 In the configuration shown, the intra-prediction unit 1309 generates prediction volume 0 based on the volume where volume idx = 0, and generates prediction volume 1 based on the volume where volume idx = 1. Then, the intra-prediction unit 1309 generates the final prediction volume by averaging prediction volume 0 and prediction volume 1. In this case, the 3D data encoding apparatus 1300 can also append multiple volume idx values ​​of the multiple volumes used in generating the prediction volume to the bitstream.

[0552] Figure 45 The inter-frame prediction process involved in this embodiment is illustrated in the diagram. The inter-frame prediction unit 1311 performs encoding (inter-frame prediction) on the space (SPC) of a certain time T_Cur using the encoded space of different times T_LX. In this case, the inter-frame prediction unit 1311 performs encoding processing by applying rotation and translation processing to the encoded space of different times T_LX.

[0553] Furthermore, the three-dimensional data encoding apparatus 1300 appends RT information related to the rotation and translation processing of the space applied at different times T_LX to the bitstream. Different times T_LX are, for example, times T_L0 preceding a certain time T_Cur. In this case, the three-dimensional data encoding apparatus 1300 may also append RT information RT_L0 related to the rotation and translation processing of the space applied at time T_L0 to the bitstream.

[0554] Alternatively, different times T_LX may be, for example, times T_L1 following a certain time T_Cur. In this case, the three-dimensional data encoding device 1300 may append RT information RT_L1, which relates to the rotation and translation processing of the space to which time T_L1 applies, to the bitstream.

[0555] Alternatively, the inter-frame prediction unit 1311 may perform encoding (double prediction) with reference to the spaces of different times T_L0 and T_L1. In this case, the three-dimensional data encoding apparatus 1300 may append the RT information RT_L0 and RT_L1, which are respectively related to the rotation and translation of the space, to the bitstream.

[0556] Furthermore, although the above sets T_L0 to a time before T_Cur and T_L1 to a time after T_Cur, this is not a limitation. For example, both T_L0 and T_L1 can be times before T_Cur. Or, both T_L0 and T_L1 can be times after T_Cur.

[0557] Furthermore, when the 3D data encoding apparatus 1300 encodes data by referencing multiple spaces at different times, it may append RT information related to the rotation and translation applied to each space to the bitstream. For example, the 3D data encoding apparatus 1300 manages the multiple encoded spaces referenced through two reference lists (L0 list and L1 list). When the first reference space in the L0 list is set to L0R0, the second reference space in the L0 list is set to L0R1, the first reference space in the L1 list is set to L1R0, and the second reference space in the L1 list is set to L1R1, the 3D data encoding apparatus 1300 appends the RT information RT_L0R0 of L0R0, the RT information RT_L0R1 of L0R1, the RT information RT_L1R0 of L1R0, and the RT information RT_L1R1 of L1R1 to the bitstream. For example, the 3D data encoding apparatus 1300 appends this RT information to the beginning of the bitstream.

[0558] Furthermore, when encoding with reference to multiple reference spaces at different times, the 3D data encoding device 1300 determines whether rotation and translation are applicable to each reference space. In this case, the 3D data encoding device 1300 can append information indicating whether rotation and translation are applicable to each reference space (such as an RT application flag) to the header information of the bitstream. For example, based on the encoding target space, the 3D data encoding device 1300 calculates RT information and ICP error values ​​using the ICP (Interactive Closest Point) algorithm for each reference space to be referenced. If the ICP error value is below a predetermined certain value, the 3D data encoding device 1300 determines that rotation and translation are not required and sets the RT application flag to OFF (invalid). Conversely, if the ICP error value is greater than the aforementioned certain value, the 3D data encoding device 1300 sets the RT application flag to ON (valid) and appends the RT information to the bitstream.

[0559] Figure 46 A syntax example of appending RT information and RT applicability flags to the header is shown. Furthermore, the number of bits allocated to each syntax can be determined based on the range of values ​​that syntax can take. For example, if the reference list L0 contains 8 reference spaces, 3 bits can be allocated in MaxRefSpc_l0. The number of allocated bits can be changed according to the values ​​that each syntax can take, or the number of allocated bits can be fixed regardless of the possible values. When the number of allocated bits is fixed, the 3D data encoding device 1300 can append this fixed number of bits to other header information.

[0560] Here, Figure 46The MaxRefSpc_l0 shown indicates the number of reference spaces contained in the reference list L0. RT_flag_l0[i] is the RT applicability flag for reference space i within the reference list L0. When RT_flag_l0[i] is 1, rotation and translation are applied to reference space i. When RT_flag_l0[i] is 0, rotation and translation are not applied to reference space i.

[0561] R_l0[i] and T_l0[i] are the RT information of reference space i within reference list L0. R_l0[i] is the rotation information of reference space i within reference list L0. The rotation information indicates the content of the applicable rotation process, such as a rotation matrix or quaternion. T_l0[i] is the translation information of reference space i within reference list L0. The translation information indicates the content of the applicable translation process, such as a translation vector.

[0562] MaxRefSpc_l1 indicates the number of reference spaces contained in reference list L1. RT_flag_l1[i] is the RT applicability flag for reference space i in reference list L1. When RT_flag_l1[i] is 1, rotation and translation are applied to reference space i. When RT_flag_l1[i] is 0, rotation and translation are not applied to reference space i.

[0563] R_l1[i] and T_l1[i] are the RT information of reference space i within reference list L1. R_l1[i] is the rotation information of reference space i within reference list L1. The rotation information indicates the content of the applicable rotation process, such as a rotation matrix or quaternion. T_l1[i] is the translation information of reference space i within reference list L1. The translation information indicates the content of the applicable translation process, such as a translation vector.

[0564] The inter-frame prediction unit 1311 uses information from the encoded reference space stored in the reference space memory 1310 to generate a predicted volume of the encoded object volume. As described above, before generating the predicted volume of the encoded object volume, the inter-frame prediction unit 1311 uses the ICP (Interactive Closest Point) algorithm to obtain RT information in the encoded object space and the reference space to make the positional relationship between them approximate. Then, the inter-frame prediction unit 1311 uses the obtained RT information to apply rotation and translation processing to the reference space to obtain the reference space B. After that, the inter-frame prediction unit 1311 uses information in the reference space B to generate a predicted volume of the encoded object volume in the encoded object space. Here, the 3D data encoding apparatus 1300 appends the RT information used to obtain the reference space B to the header information of the encoded object space, etc.

[0565] In this way, the inter-frame prediction unit 1311 applies rotation and translation processing to the reference space, thereby making the overall positional relationship between the coding object space and the reference space approximate, and then uses the information of the reference space to generate the prediction volume, thus improving the accuracy of the prediction volume. Furthermore, since prediction residuals can be suppressed, the amount of coding can be reduced. Although an example of using the coding object space and the reference space for ICP has been shown here, it is not limited to this. For example, to reduce processing load, the inter-frame prediction unit 1311 may also use at least one of the coding object space with voxel or point cloud data extracted, and the reference space with voxel or point cloud data extracted, to perform ICP and obtain RT information.

[0566] Furthermore, if the ICP error value obtained from the ICP result is smaller than a predetermined first threshold, i.e., if the positional relationship between the encoded object space and the reference space is close, the inter-frame prediction unit 1311 can determine that rotation and translation processing are not required, and therefore does not perform rotation and translation. In this case, the three-dimensional data encoding apparatus 1300 can avoid appending RT information to the bitstream, thereby suppressing additional overhead.

[0567] Furthermore, if the ICP error value is greater than a predefined second threshold, the inter-frame prediction unit 1311 determines that the shape change in space is large, and can apply intra-frame prediction to all volumes of the encoded object space. Hereinafter, the space for which intra-frame prediction is applied will be referred to as the intra-frame space. The second threshold is a value larger than the first threshold mentioned above. Moreover, it is not limited to ICP; any method can be applied as long as it is used to obtain RT information from two voxel sets or two point cloud sets.

[0568] Furthermore, when the 3D data contains attribute information such as shape or color, the inter-frame prediction unit 1311 searches for, for example, the volume in the reference space that is closest to the shape or color attribute information of the encoded object volume, serving as the predicted volume of the encoded object volume within the encoded object space. This reference space is, for example, a reference space that has undergone the aforementioned rotation and translation processing. The inter-frame prediction unit 1311 generates the predicted volume based on the volume obtained through the search (reference volume). Figure 47 This diagram illustrates the process of generating the prediction volume. The inter-frame prediction unit 1311, in response to... Figure 47The volume of the encoded object (volume idx=0) shown is encoded using inter-frame prediction. While sequentially scanning reference volumes in the reference space, the volume with the smallest prediction residual (the difference between the encoded object volume and the reference volume) is searched. The inter-frame prediction unit 1311 selects the volume with the smallest prediction residual as the prediction volume. The prediction residual between the encoded object volume and the prediction volume is encoded by processing after the transform unit 1303. Here, the prediction residual refers to the difference between the attribute information of the encoded object volume and the attribute information of the prediction volume. Furthermore, the three-dimensional data encoding apparatus 1300 appends the volume idx of the reference volume in the reference space, which is used as the prediction volume, to the header of the bitstream, etc.

[0569] exist Figure 47 In the example shown, the reference volume idx=4 of the reference space L0R0 is selected as the predicted volume of the encoded object volume. Therefore, the prediction residual between the encoded object volume and the reference volume, as well as the reference volume idx=4, are encoded and appended to the bitstream.

[0570] Furthermore, although the example given here is the predicted volume for generating attribute information, the same processing can be performed on the predicted volume for location information.

[0571] The prediction control unit 1312 controls whether to use intra-frame prediction or inter-frame prediction to encode the coding object volume. Here, the mode including intra-frame prediction and inter-frame prediction is referred to as the prediction mode. For example, the prediction control unit 1312 calculates the prediction residual when the coding object volume is predicted via intra-frame prediction and the prediction residual when it is predicted via inter-frame prediction, using these as evaluation values, and selects the prediction mode with the smaller evaluation value. Alternatively, the prediction control unit 1312 may apply orthogonal transformation, quantization, and entropy coding to the prediction residuals of intra-frame prediction and inter-frame prediction, respectively, to calculate the actual coding volume, and use the calculated coding volume as the evaluation value to select the prediction mode. Furthermore, additional overhead information other than the prediction residual (such as reference volume idx information) may be added to the evaluation value. Moreover, if the coding object space is predetermined to be encoded in the intra-frame space, the prediction control unit 1312 may typically select intra-frame prediction.

[0572] The entropy coding unit 1313 generates a coded signal (coded bitstream) by performing variable-length coding on the input from the quantization unit 1304, namely the quantization coefficients. Specifically, the entropy coding unit 1313 performs arithmetic coding on the quantization coefficients, for example, by binarizing the quantization coefficients.

[0573] Next, the three-dimensional data decoding device for decoding the encoded signal generated by the three-dimensional data encoding device 1300 will be described. Figure 48This is a block diagram of the three-dimensional data decoding apparatus 1400 according to this embodiment. The three-dimensional data decoding apparatus 1400 includes: an entropy decoding unit 1401, an inverse quantization unit 1402, an inverse transform unit 1403, an adder unit 1404, a reference volume memory 1405, an intra-frame prediction unit 1406, a reference space memory 1407, an inter-frame prediction unit 1408, and a prediction control unit 1409.

[0574] The entropy decoding unit 1401 performs variable-length decoding on the encoded signal (encoded bitstream). For example, the entropy decoding unit 1401 performs arithmetic decoding on the encoded signal to generate a binary signal, and generates quantization coefficients based on the generated binary signal.

[0575] The inverse quantization unit 1402 performs inverse quantization on the quantization coefficients input from the entropy decoding unit 1401 using quantization parameters attached to the bit stream, thereby generating inverse quantization coefficients.

[0576] The inverse transform unit 1403 performs an inverse transform on the inverse quantization coefficients input from the inverse quantization unit 1402 to generate a prediction residual. For example, the inverse transform unit 1403 performs an inverse orthogonal transform on the inverse quantization coefficients based on information appended to the bitstream to generate a prediction residual.

[0577] The addition unit 1404 adds the prediction residual generated by the inverse transform unit 1403 to the prediction volume generated by intra-frame prediction or inter-frame prediction to generate a reconstructed volume. This reconstructed volume is output as decoded 3D data and stored in the reference volume memory 1405 or the reference space memory 1407.

[0578] The intra-prediction unit 1406 generates a prediction volume by using a reference volume in the reference volume memory 1405 and information appended to the bitstream through intra-prediction. Specifically, the intra-prediction unit 1406 obtains prediction mode information and adjacent volume information (e.g., volume idx) appended to the bitstream, and generates a prediction volume using the adjacent volumes shown in the adjacent volume information and the pattern shown in the prediction mode information. Furthermore, the details of these processes are the same as those of the intra-prediction unit 1309 described above, except that the information appended to the bitstream is used.

[0579] The inter-frame prediction unit 1408 generates a prediction volume by using reference spaces in the reference space memory 1407 and information appended to the bitstream through inter-frame prediction. Specifically, the inter-frame prediction unit 1408 uses the RT information of each reference space appended to the bitstream, applies rotation and translation processing to the reference spaces, and generates the prediction volume using the applied reference spaces. Furthermore, if an RT application flag exists in the bitstream for each reference space, the inter-frame prediction unit 1408 applies rotation and translation processing to the reference spaces according to the RT application flag. The details of the above processing are the same as those of the inter-frame prediction unit 1311, except for the use of information appended to the bitstream.

[0580] Whether to use intra-frame prediction or inter-frame prediction to decode the decoded object space is controlled by the prediction control unit 1409. For example, the prediction control unit 1409 selects intra-frame prediction or inter-frame prediction based on information appended to the bitstream that indicates the prediction mode to be used. Alternatively, if it is predetermined that the decoded object space will be decoded using intra-frame prediction, the prediction control unit 1409 may generally select intra-frame prediction.

[0581] The following describes variations of this embodiment. While this embodiment uses spatial units for rotation and translation as an example, smaller units can also be used. For example, the three-dimensional data encoding apparatus 1300 can divide space into subspaces and apply rotation and translation on a subspace unit basis. In this case, the three-dimensional data encoding apparatus 1300 generates RT information for each subspace and appends the generated RT information to the header of the bitstream. Furthermore, the three-dimensional data encoding apparatus 1300 can use volume units as encoding units for rotation and translation. In this case, the three-dimensional data encoding apparatus 1300 generates RT information using encoding volume units and appends the generated RT information to the header of the bitstream. Moreover, the above can be combined. That is, the three-dimensional data encoding apparatus 1300 can apply rotation and translation on a large unit first, and then apply rotation and translation on a smaller unit. For example, the three-dimensional data encoding apparatus 1300 can apply rotation and translation on a spatial unit, applying different rotations and translations to each of the multiple volumes contained in the obtained space.

[0582] Furthermore, although this embodiment uses rotation and translation applied to a reference space as examples, it is not limited to this. For example, the 3D data encoding device 1300 may apply scaling processing to change the size of the 3D data. The 3D data encoding device 1300 may also apply any one or two of rotation, translation, and scaling. Moreover, as described above, when processing is applied in multiple stages using different units, the types of processing applied in each unit may be different. For example, rotation and translation may be applied in spatial units, while translation may be applied in volume units.

[0583] Furthermore, these variations are also applicable to the three-dimensional data decoding device 1400.

[0584] As described above, the three-dimensional data encoding device 1300 according to this embodiment performs the following processing. Figure 48 This is a flowchart of the inter-frame prediction processing performed by the three-dimensional data encoding device 1300. First, the three-dimensional data encoding device 1300 generates predicted position information (e.g., predicted volume) using the position information of three-dimensional points contained in the object's three-dimensional data (e.g., the object space) and reference three-dimensional data (e.g., the reference space) at different times (S1301). Specifically, the three-dimensional data encoding device 1300 generates predicted position information by applying rotation and translation processing to the position information of three-dimensional points contained in the reference three-dimensional data.

[0585] Furthermore, the 3D data encoding apparatus 1300 performs rotation and translation processing in a first unit (e.g., space), and generates predicted position information in a second unit (e.g., volume), which is smaller than the first unit. For example, the 3D data encoding apparatus 1300 may search among multiple volumes contained in the reference space after rotation and translation processing for the volume containing the encoded object in the encoded object space where the difference between the volume and the position information is the smallest, and use the obtained volume as the predicted volume. Alternatively, the 3D data encoding apparatus 1300 may perform rotation and translation processing and the generation of predicted position information in the same unit.

[0586] Alternatively, the three-dimensional data encoding device 1300 may apply a first rotation and translation process to the position information of three-dimensional points contained in the reference three-dimensional data in a first unit (e.g., space), and apply a second rotation and translation process to the position information of three-dimensional points obtained by the first rotation and translation process in a second unit (e.g., volume) smaller than the first unit, thereby generating predicted position information.

[0587] Here, the position information and predicted position information of the three-dimensional points are as follows: Figure 41As shown, this is represented using an octree structure. For example, the positional information and predicted positional information of the 3D points are represented in a width-first scan order within the depth and width of the octree structure. Alternatively, the positional information and predicted positional information of the 3D points are represented in a depth-first scan order within the depth and width of the octree structure.

[0588] And, as Figure 46 As shown, the 3D data encoding device 1300 encodes a RT applicability flag indicating whether rotation and translation processing are applicable to the position information of 3D points contained in the reference 3D data. That is, the 3D data encoding device 1300 generates an encoded signal (encoded bitstream) including the RT applicability flag. Furthermore, the 3D data encoding device 1300 encodes RT information indicating the content of rotation and translation processing. That is, the 3D data encoding device 1300 generates an encoded signal (encoded bitstream) including RT information. Alternatively, the 3D data encoding device 1300 may encode the RT information when the RT applicability flag indicates that rotation and translation processing are applicable, and not encode the RT information when the RT applicability flag indicates that rotation and translation processing are not applicable.

[0589] Furthermore, the three-dimensional data includes, for example, the positional information of three-dimensional points and the attribute information (color information, etc.) of each three-dimensional point. The three-dimensional data encoding device 1300 uses the attribute information of the three-dimensional points contained in the reference three-dimensional data to generate predicted attribute information (S1302). Next, the 3D data encoding device 1300 uses the predicted position information to encode the position information of the 3D points contained in the 3D data of the object. For example, the 3D data encoding device 1300... Figure 38 As shown, the difference between the position information of the three-dimensional points contained in the object's three-dimensional data and the predicted position information is calculated, which is the differential position information (S1303). Furthermore, the 3D data encoding device 1300 uses the predicted attribute information to encode the attribute information of the 3D points contained in the 3D data of the object. For example, the 3D data encoding device 1300 calculates the difference between the attribute information of the 3D points contained in the 3D data of the object and the predicted attribute information, i.e., the differential attribute information (S1304). Next, the 3D data encoding device 1300 transforms and quantizes the calculated differential attribute information (S1305). Finally, the three-dimensional data encoding device 1300 encodes the differential position information and the quantized differential attribute information (e.g., entropy encoding) (S1306). That is, the three-dimensional data encoding device 1300 generates an encoded signal (encoded bit stream) that includes differential position information and differential attribute information.

[0590] Furthermore, if the 3D data does not include attribute information, the 3D data encoding device 1300 may omit steps S1302, S1304, and S1305. Also, the 3D data encoding device 1300 may only encode either the position information of the 3D points or the attribute information of the 3D points.

[0591] and, Figure 49 The processing order shown is merely an example and is not a limitation. For example, since the processing of location information (S1301, S1303) and the processing of attribute information (S1302, S1304, S1305) are independent of each other, they can be executed in any order, or some of them can be processed in parallel.

[0592] As described above, in this embodiment, the three-dimensional data encoding device 1300 uses the position information of three-dimensional points contained in the object's three-dimensional data and reference three-dimensional data at different times to generate predicted position information, and encodes the difference between the position information of the three-dimensional points contained in the object's three-dimensional data and the predicted position information, i.e., differential position information. Accordingly, since the amount of data in the encoded signal can be reduced, the encoding efficiency can be improved.

[0593] Furthermore, in this embodiment, the three-dimensional data encoding apparatus 1300 generates predicted attribute information by utilizing the attribute information of three-dimensional points contained in the reference three-dimensional data, and encodes the difference between the attribute information of three-dimensional points contained in the object's three-dimensional data and the predicted attribute information, i.e., differential attribute information. Accordingly, since the amount of data in the encoded signal can be reduced, the encoding efficiency can be improved.

[0594] For example, the three-dimensional data encoding device 1300 includes a processor and a memory, with the processor using the memory to perform the aforementioned processing.

[0595] Figure 48 This is a flowchart of the inter-frame prediction processing performed by the 3D data decoding device 1400. First, the three-dimensional data decoding device 1400 decodes the differential position information and differential attribute information (e.g., entropy decoding) based on the encoded signal (encoded bit stream) (S1401). Furthermore, the 3D data decoding device 1400 decodes the RT applicability flag, which indicates whether rotation and translation processing are applicable to the position information of 3D points contained in the reference 3D data, based on the encoded signal. The 3D data decoding device 1400 also decodes the RT information indicating the content of rotation and translation processing. Additionally, the 3D data decoding device 1400 decodes the RT information when the RT applicability flag indicates that rotation and translation processing are applicable, and does not decode the RT information when the RT applicability flag indicates that rotation and translation processing are not applicable.

[0596] Next, the three-dimensional data decoding device 1400 performs inverse quantization and inverse transformation on the decoded differential attribute information (S1402). Next, the 3D data decoding device 1400 uses the position information of 3D points contained in the object's 3D data (e.g., the decoded object space) and reference 3D data at different times (e.g., the reference space) to generate predicted position information (e.g., predicted volume) (S1403). Specifically, the 3D data decoding device 1400 generates predicted position information by applying rotation and translation processing to the position information of 3D points contained in the reference 3D data.

[0597] More specifically, when the RT application flag indicates that rotation and translation processing is applicable, the 3D data decoding device 1400 applies rotation and translation processing to the position information of the 3D points contained in the reference 3D data shown in the RT information. Furthermore, when the RT application flag indicates that rotation and translation processing is not applicable, the 3D data decoding device 1400 does not apply rotation and translation processing to the position information of the 3D points contained in the reference 3D data.

[0598] Furthermore, the 3D data decoding device 1400 can perform rotation and translation processing in a first unit (e.g., space), and can generate predicted position information in a second unit (e.g., volume) smaller than the first unit. Alternatively, the 3D data decoding device 1400 can also perform rotation and translation processing and the generation of predicted position information in the same unit.

[0599] Alternatively, the 3D data decoding device 1400 may apply a first rotation and translation process to the position information of 3D points contained in the reference 3D data in a first unit (e.g., space), and apply a second rotation and translation process to the position information of 3D points obtained by the first rotation and translation process in a second unit (e.g., volume) smaller than the first unit, thereby generating predicted position information.

[0600] Here, the position information of the three-dimensional points and the predicted position information are, for example... Figure 41 As shown, this is represented using an octree structure. For example, the positional information and predicted positional information of the 3D points are represented in a width-first scan order between depth and width in the octree structure. Alternatively, the positional information and predicted positional information of the 3D points are represented in a depth-first scan order between depth and width in the octree structure.

[0601] The three-dimensional data decoding device 1400 uses the attribute information of the three-dimensional points contained in the reference three-dimensional data to generate predicted attribute information (S1404). Next, the 3D data decoding device 1400 decodes the encoded position information contained in the encoded signal using the predicted position information, thereby restoring the position information of the 3D points contained in the object's 3D data. Here, the encoded position information is, for example, differential position information, and the 3D data decoding device 1400 restores the position information of the 3D points contained in the object's 3D data by adding the differential position information to the predicted position information (S1405). Furthermore, the 3D data decoding device 1400 decodes the encoded attribute information contained in the encoded signal using the predicted attribute information, thereby restoring the attribute information of the 3D points contained in the object's 3D data. Here, the encoded attribute information is, for example, differential attribute information. The 3D data decoding device 1400 restores the attribute information of the 3D points contained in the object's 3D data by adding the differential attribute information to the predicted attribute information (S1406). Alternatively, if the 3D data does not contain attribute information, the 3D data decoding device 1400 may not execute steps S1402, S1404, and S1406. Furthermore, the 3D data decoding device 1400 may only decode either the position information of the 3D points or the attribute information of the 3D points.

[0602] and, Figure 50 The processing order shown is an example and is not limited to it. For example, since the processing of location information (S1403, S1405) and the processing of attribute information (S1402, S1404, S1406) are independent of each other, they can be performed in any order, and some of them can be processed in parallel.

[0603] (Implementation Method 8) In this embodiment, the method for representing three-dimensional points (point clouds) in the encoding of three-dimensional data is described.

[0604] Figure 51 This is a block diagram illustrating the configuration of the three-dimensional data distribution system according to this embodiment. Figure 51 The distribution system shown includes a server 1501 and multiple clients 1502.

[0605] Server 1501 includes a storage unit 1511 and a control unit 1512. The storage unit 1511 stores the encoded 3D data, i.e., the encoded 3D map 1513.

[0606] Figure 52An example of the bitstream configuration for encoding a 3D map 1513 is shown. The 3D map is divided into multiple sub-maps, each of which is encoded. A random access header (RA) including sub-coordinate information is appended to each sub-map. This sub-coordinate information is used to improve the encoding efficiency of the sub-maps. This sub-coordinate information shows the sub-coordinates of the sub-map. Sub-coordinates are the coordinates of the sub-map relative to a reference coordinate. Furthermore, the 3D map comprising multiple sub-maps is called the overall map. And, in the overall map, the coordinates that will serve as the reference (e.g., the origin) are called the reference coordinates. That is, sub-coordinates are the coordinates of the sub-maps within the coordinate system of the overall map. In other words, sub-coordinates show the deviation between the coordinate system of the overall map and the coordinate system of the sub-maps. And, the coordinates in the coordinate system of the overall map relative to the reference coordinates are called the overall coordinates. The coordinates in the coordinate system of the sub-map relative to the sub-coordinates are called the differential coordinates.

[0607] Client 1502 sends a message to server 1501. This message includes the location information of client 1502. Control unit 1512, included in server 1501, obtains a bitstream of a sub-map representing the location closest to client 1502 based on the location information received in the message. This sub-map bitstream, including sub-coordinate information, is sent to client 1502. Decoder 1521, included in client 1502, uses this sub-coordinate information to obtain the total coordinates of the sub-map based on reference coordinates. Application 1522, included in client 1502, uses the obtained total coordinates of the sub-map to execute an application related to its own location.

[0608] Furthermore, a submap represents a portion of the overall map. Sub-coordinates are the coordinates of the submap's location within the base coordinate space of the overall map. For example, consider a submap A of AA and a submap B of AB within the overall map of map A. When a vehicle wants to reference map AA, it starts decoding from submap A; when it wants to reference map AB, it starts decoding from submap B. Here, the submap is a random access point. Specifically, A represents Osaka Prefecture, AA represents Osaka City, AB represents Takagi City, etc.

[0609] Each sub-map, along with its sub-coordinate information, is sent to the client. The sub-coordinate information is included in the header information of each sub-map or in the data packet sent.

[0610] The reference coordinates of the sub-coordinate information of each sub-map can also be attached to the header information of the entire map, which is spatial header information above the sub-map.

[0611] A submap can consist of a single spatial configuration (SPC). Furthermore, a submap can also consist of multiple SPCs.

[0612] Furthermore, a submap can also include a Group of Space (GOS). A submap can also be composed of world space. For example, if a submap contains multiple objects and these objects are assigned to different Space Processing Units (SPCs), then the submap consists of multiple SPCs. Conversely, if multiple objects are assigned to a single SPC, the submap consists of a single SPC.

[0613] Next, the improvement in coding efficiency when using sub-coordinate information will be explained. Figure 53 This is a diagram used to illustrate the effect. For example, to... Figure 53 Encoding a 3D point A located far from the reference coordinates requires a larger number of bits. Here, the distance between the sub-coordinates and 3D point A is shorter than the distance between the reference coordinates and 3D point A. Therefore, encoding the coordinates of 3D point A based on the sub-coordinates improves encoding efficiency compared to encoding the coordinates of 3D point A based on the reference coordinates. Furthermore, the submap's bitstream includes sub-coordinate information. By sending the submap's bitstream and the reference coordinates to the decoding side (client), the coordinates of the entire submap can be reconstructed at the decoding side.

[0614] Figure 54 This is a flowchart of the processing performed by the sending side of the sub-map, namely server 1501. First, server 1501 receives a message from client 1502 including the location information of client 1502 (S1501). Control unit 1512 obtains the encoded bit stream of the sub-map based on the client's location information from storage unit 1511 (S1502). Then, server 1501 sends the encoded bit stream of the sub-map and the reference coordinates to client 1502 (S1503).

[0615] Figure 55 This is a flowchart of the processing performed by the receiving side of the sub-map, i.e., client 1502. First, client 1502 receives the encoded bitstream of the sub-map and reference coordinates sent from server 1501 (S1511). Next, client 1502 obtains the sub-map and sub-coordinate information by decoding the encoded bitstream (S1512). Then, client 1502 uses the reference coordinates and sub-coordinates to restore the differential coordinates within the sub-map to the overall coordinates (S1513). Next, a syntactic example of information related to the submap will be explained. In the encoding of the submap, the 3D data encoding device calculates the differential coordinates by subtracting the subcoordinates from the coordinates of each point cloud (3D point). The 3D data encoding device then encodes the differential coordinates into a bitstream as the value for each point cloud. Furthermore, the encoding device encodes the subcoordinate information, which indicates the subcoordinates, as header information of the bitstream. Based on this, the 3D data decoding device can obtain the total coordinates of each point cloud. For example, the 3D data encoding device is included in server 1501, and the 3D data decoding device is included in client 1502.

[0616] Figure 56 Syntactic examples of submaps are shown. Figure 56 The NumOfPoint shown represents the number of point clouds contained in the sub-map. sub_coordinate_x, sub_coordinate_y, and sub_coordinate_z are sub-coordinate information. sub_coordinate_x represents the x-coordinate of the sub-coordinate. sub_coordinate_y represents the y-coordinate of the sub-coordinate. sub_coordinate_z represents the z-coordinate of the sub-coordinate.

[0617] Furthermore, diff_x[i], diff_y[i], and diff_z[i] are the difference coordinates of the i-th point cloud within the sub-map. diff_x[i] represents the difference between the x-coordinate of the i-th point cloud within the sub-map and the x-coordinate of the sub-map. diff_y[i] represents the difference between the y-coordinate of the i-th point cloud within the sub-map and the y-coordinate of the sub-map. diff_z[i] represents the difference between the z-coordinate of the i-th point cloud within the sub-map and the z-coordinate of the sub-map.

[0618] The 3D data decoding device decodes the coordinates of point_cloud[i]_x, point_cloud[i]_y, and point_cloud[i]_z, which are the coordinates of the i-th point cloud, using the following formula: point_cloud[i]_x is the x-coordinate of the i-th point cloud; point_cloud[i]_y is the y-coordinate of the i-th point cloud; and point_cloud[i]_z is the z-coordinate of the i-th point cloud.

[0619] point_cloud[i]_x=sub_coordinate_x+diff_x[i] point_cloud[i]_y=sub_coordinate_y+diff_y[i] point_cloud[i]_z=sub_coordinate_z+diff_z[i] Next, the applicable switching process for octree encoding will be explained. When encoding sub-maps, the 3D data encoding device may choose to encode each point cloud using an octree representation (hereinafter referred to as octree encoding) or to encode the difference values ​​with the sub-coordinates (hereinafter referred to as non-octree encoding). Figure 57 The working principle is illustrated in the diagram. For example, when the number of point clouds in a submap exceeds a predetermined threshold, the 3D data encoding device applies octree encoding to the submap. When the number of point clouds in a submap is less than the threshold, the 3D data encoding device applies non-octree encoding to the submap. Accordingly, the 3D data encoding device appropriately selects whether to use octree encoding or non-octree encoding based on the shape and density of the objects contained in the submap, thereby improving encoding efficiency.

[0620] Furthermore, the 3D data encoding device will append information such as which encoding method (octree encoding or non-octree encoding) is applicable to the submap (hereinafter referred to as octree encoding application information) to the header of the submap. Based on this, the 3D data decoding device can determine whether the bitstream is obtained by octree encoding the submap or by non-octree encoding the submap.

[0621] Furthermore, the 3D data encoding device can calculate the encoding efficiency when applying octree encoding and non-octree encoding to the same point cloud, and apply the encoding method with higher encoding efficiency to the sub-map.

[0622] Figure 58 Syntactic examples of submaps are shown for situations where such switching occurs. Figure 58 The `coding_type` shown indicates the encoding type, specifically the octree encoding information mentioned above. `coding_type=00` indicates octree encoding is used. `coding_type=01` indicates non-octree encoding is used. `coding_type=10` or `11` indicates other encoding methods besides those mentioned above are used.

[0623] When the encoding type is non-octree encoding, the sub-map includes NumOfPoint and sub-coordinate information (sub_coordinate_x, sub_coordinate_y, and sub_coordinate_z).

[0624] When the encoding type is octree encoding, the submap includes octree_info. octree_info contains the information needed for octree encoding, such as depth information.

[0625] When the encoding type is non-octree encoding, the submap includes differential coordinates (diff_x[i], diff_y[i], and diff_z[i]).

[0626] When the encoding type is octree encoding, the submap includes the encoded data associated with the octree encoding, namely octree_data.

[0627] Furthermore, although an example of using the xyz coordinate system is shown here as the coordinate system for the point cloud, the polar coordinate system can also be used.

[0628] Figure 59 This is a flowchart of the 3D data encoding process performed by the 3D data encoding device. First, the 3D data encoding device calculates the number of point clouds within the sub-map of the object being processed (S1521). Next, the 3D data encoding device determines whether the calculated number of point clouds is above a predetermined threshold (S1522). If the number of point clouds is above a threshold ("Yes" in S1522), the 3D data encoding device applies octree encoding to the object sub-map (S1523). Furthermore, the 3D point data encoding device appends octree encoding application information indicating that octree encoding has been applied to the object sub-map to the header of the bitstream (S1525).

[0629] Furthermore, if the number of point clouds is below a threshold ("No" in S1522), the 3D data encoding device applies non-octree encoding to the object sub-map (S1524). The 3D point data encoding device then appends information indicating that non-octree encoding was applied to the object sub-map to the header of the bitstream (S1525).

[0630] Figure 60 This is a flowchart of the 3D data decoding process performed by the 3D data decoding device. First, the 3D data decoding device decodes the octree encoding application information from the header of the bitstream (S1531). Next, the 3D data decoding device determines whether the encoding type of the object submap to which the decoded octree encoding application information is applied is octree encoding based on the decoded octree encoding application information (S1532).

[0631] If the encoding type indicated in the octree encoding application information is octree encoding ("Yes" in S1532), the 3D data decoding device uses octree decoding to decode the object submap (S1533). Alternatively, if the encoding type indicated in the octree encoding application information is non-octree encoding ("No" in S1532), the 3D data decoding device uses non-octree decoding to decode the object submap (S1534).

[0632] The following describes variations of this embodiment. Figures 61 to 63 The working example of a variation of the encoding type switching process is shown in the pattern.

[0633] like Figure 61 As shown, the 3D data encoding device can select whether to use octree encoding or non-octree encoding for each space. In this case, the 3D data encoding device appends octree encoding application information to the header of the space. Accordingly, the 3D data decoding device can determine whether octree encoding is applicable for each space. Furthermore, in this case, the 3D data encoding device sets sub-coordinates for each space and encodes the difference value obtained by subtracting the sub-coordinate value from the coordinates of each point cloud in the space.

[0634] Therefore, since the 3D data encoding device can appropriately switch between octree encoding and other methods based on the shape of the object or the number of point clouds in space, the encoding efficiency can be improved.

[0635] And, as Figure 62 As shown, the 3D data encoding device can select whether to use octree encoding or non-octree encoding for each volume. In this case, the 3D data encoding device appends octree encoding application information to the header of the volume. Accordingly, the 3D data decoding device can determine whether octree encoding is applicable for each volume. Furthermore, in this case, the 3D data encoding device sets sub-coordinates for each volume and encodes the difference value obtained by subtracting the sub-coordinates from the coordinates of each point cloud within the volume.

[0636] Therefore, since the 3D data encoding device can appropriately switch between octree encoding and other methods based on the shape of the object or the number of point clouds within the volume, the encoding efficiency can be improved.

[0637] Furthermore, the above description, as a non-octree coding example, shows encoding the difference after subtracting the sub-coordinates from the coordinates of each point cloud. However, this is not a limitation, and any coding method other than octree coding can be used. For example... Figure 63As shown, the 3D data encoding device can also use a method of encoding the values ​​of the point cloud within the sub-map, space, or volume (hereinafter referred to as original coordinate encoding) instead of sub-coordinate difference, as a non-octree encoding.

[0638] In this scenario, the 3D data encoding device stores information showing that original coordinate encoding has been applied to the object space (submap, space, or volume) in the header. Based on this, the 3D data decoding device can determine whether original coordinate encoding has been applied to the object space.

[0639] Furthermore, when using original coordinate encoding, the 3D data encoding device can encode the original coordinates without applying quantization and arithmetic encoding. Moreover, the 3D data encoding device can encode the original coordinates with a pre-defined fixed bit length. Accordingly, the 3D data encoding device can generate a stream with a certain bit length at a specific timing.

[0640] Furthermore, although the above description shows an example of encoding the difference between the coordinates of each point cloud and the sub-coordinates as a non-octree encoding, it is not limited to this.

[0641] For example, a 3D data encoding device can sequentially encode the difference values ​​between the coordinates of each point cloud. Figure 64 This is a diagram used to illustrate the work in this situation. For example, in Figures 65 to 67 In the example shown, when encoding point cloud PA, the 3D data encoding device uses the sub-coordinates as the predicted coordinates and encodes the difference between the coordinates of point cloud PA and the predicted coordinates. Similarly, when encoding point cloud PB, the 3D data encoding device uses the coordinates of point cloud PA as the predicted coordinates and encodes the difference between the coordinates of point cloud PB and the predicted coordinates. Furthermore, when encoding point cloud PC, the 3D data encoding device uses point cloud PB as the predicted coordinates and encodes the difference between the coordinates of point cloud PB and the predicted coordinates. In this way, the 3D data encoding device can set a scanning order for multiple point clouds and encode the coordinates of the object point cloud and the difference between the object point cloud's coordinates and the coordinates of the preceding point cloud in the scanning order.

[0642] Furthermore, although the sub-coordinates mentioned above refer to the coordinates of the lower left and front corner of the sub-map, the position of the sub-coordinates is not limited to this. Figure 65 Other examples of sub-coordinate positions are shown. The position of a sub-coordinate can be set to any coordinate within the object space (submap, space, or volume). That is, as described above, a sub-coordinate could be the coordinate of the lower left front corner. Figure 66 As shown, sub-coordinates can also be the coordinates of the center of the object space. For example... Figure 67As shown, sub-coordinates can also be the coordinates of the upper right rear corner of the object space. Furthermore, sub-coordinates are not limited to the coordinates of the lower left front or upper right rear corner of the object space; they can be the coordinates of any corner in the object space.

[0643] Furthermore, the sub-coordinates can be set to the same location as the coordinates of a point cloud within the object space (submap, space, or volume). For example, in Figure 68 In the example shown, the coordinates of the sub-coordinates are consistent with the coordinates of the point cloud PD.

[0644] Furthermore, while this embodiment shows an example of switching between octree encoding and non-octree encoding, it is not limited to this. For example, the 3D data encoding device can also switch between using a tree structure other than an octree and a non-tree structure other than that tree structure. For example, other tree structures refer to kd-trees, etc., which are segmented using a plane perpendicular to one of the coordinate axes. In addition, any other tree structure can be used.

[0645] Furthermore, while this embodiment shows an example of encoding coordinate information of point clouds, it is not limited to this. The 3D data encoding device may also encode color information, 3D feature quantities, or visible light feature quantities in the same way as coordinate information. For example, the 3D data encoding device may also set the average value of the color information of each point cloud within a sub-map as sub-color information and encode the difference between the color information of each point cloud and the sub-color information.

[0646] Furthermore, while this embodiment illustrates an example of selecting an efficient encoding method (octree encoding or non-octree encoding) based on factors such as the number of point clouds, it is not limited to this approach. For instance, a server-side 3D data encoding device can pre-store bitstreams of point clouds encoded using octree encoding, bitstreams of point clouds encoded using non-octree encoding, and bitstreams of point clouds encoded using both methods, and switch the bitstreams sent to the 3D data decoding device according to the communication environment or the processing capability of the 3D data decoding device.

[0647] Figure 68 Syntactic examples of volume are shown when switching the application of octree encoding. Figure 58 The syntax shown is Figure 59 The syntax shown is basically the same, but the details where the information is in units of volume differ. Specifically, NumOfPoint indicates the number of point clouds contained in the volume. sub_coordinate_x, sub_coordinate_y, and sub_coordinate_z are the sub-coordinate information of the volume.

[0648] Furthermore, diff_x[i], diff_y[i], and diff_z[i] are the difference coordinates of the i-th point cloud within the volume. diff_x[i] represents the difference between the x-coordinate of the i-th point cloud within the volume and the x-coordinate of the sub-coordinate. diff_y[i] represents the difference between the y-coordinate of the i-th point cloud within the volume and the y-coordinate of the sub-coordinate. diff_z[i] represents the difference between the z-coordinate of the i-th point cloud within the volume and the z-coordinate of the sub-coordinate.

[0649] Furthermore, if the relative positions of volumes in space can be calculated, the 3D data encoding device may not include sub-coordinate information in the volume header. That is, the 3D data encoding device may calculate the relative positions of volumes in space without including sub-coordinate information in the header, and use the calculated positions as the sub-coordinates of each volume.

[0650] As described above, the three-dimensional data encoding device involved in this embodiment determines whether to encode the object spatial units (e.g., sub-maps, spaces, or volumes) in the multiple spatial units (e.g., volumes, spaces, spaces, or volumes) contained in the three-dimensional data using an octree structure (e.g., Figure 60 (S1522). For example, if the number of 3D points contained in an object space unit is greater than a predetermined threshold, the 3D data encoding device determines that the object space unit should be encoded using an octree structure. Furthermore, if the number of 3D points contained in an object space unit is less than the aforementioned threshold, the 3D data encoding device determines that the object space unit should not be encoded using an octree structure.

[0651] If it is determined that the object space unit is encoded using an octree structure (S1522 "Yes"), the 3D data encoding device encodes the object space unit using an octree structure (S1523). Furthermore, if it is determined that the object space unit is not encoded using an octree structure (S1522 "No"), the 3D data encoding device encodes the object space unit in a manner different from the octree structure (S1524). For example, as a different method, the 3D data encoding device encodes the coordinates of the 3D points contained in the object space unit. Specifically, as a different method, the 3D data encoding device encodes the reference coordinates of the object space unit and the difference between the reference coordinates and the coordinates of the 3D points contained in the object space unit.

[0652] Next, the three-dimensional data encoding device appends information indicating whether to encode the spatial units of the object in an octree structure to the bit stream (S1525).

[0653] Accordingly, the three-dimensional data encoding device can reduce the amount of data in the encoded signal, thereby improving encoding efficiency.

[0654] For example, a three-dimensional data encoding device includes a processor and a memory, and the processor uses the memory to perform the above-mentioned processing.

[0655] Furthermore, the three-dimensional data decoding apparatus according to this embodiment decodes from the bitstream information indicating whether to decode the object spatial units (e.g., sub-maps, spaces, or volumes) contained in the three-dimensional data using an octree structure. Figure 69 (S1531). When the above information shows that the object space unit is decoded in an octree structure ("Yes" in S1532), the three-dimensional data decoding device decodes the object space unit in an octree structure (S1533).

[0656] If the above information indicates that the object space unit is not decoded using an octree structure ("No" in S1532), the three-dimensional data decoding device decodes the object space unit in a manner different from the octree structure (S1534). For example, the three-dimensional data decoding device decodes the coordinates of the three-dimensional points contained in the object space unit in a different manner. Specifically, the three-dimensional data decoding device decodes the difference between the reference coordinates of the object space unit and the coordinates of the three-dimensional points contained in the object space unit in a different manner.

[0657] Therefore, this three-dimensional data decoding device can reduce the amount of data in the encoded signal, thus improving encoding efficiency.

[0658] For example, a 3D data decoding device has a processor and a memory, with the processor using the memory to perform the aforementioned processing.

[0659] (Implementation Method 9) In this embodiment, the encoding method for tree structures such as octree structures is explained.

[0660] By identifying important areas and prioritizing the decoding of their 3D data, efficiency can be improved.

[0661] Figure 70 This is a diagram illustrating an example of an important region in a 3D map. An important region is, for example, an area containing a certain number of 3D points in the map whose feature values ​​are large. Alternatively, an important region could be an area containing a certain number of 3D points needed for a client such as a vehicle to estimate its own position. Or, an important region could be the face region in a 3D model of a person. Thus, important regions can be defined for each application, or they can be switched depending on the application.

[0662] In this embodiment, occupancy coding and location coding are used to represent octree structures, etc. The bit string obtained through occupancy coding is called the occupancy code, and the bit string obtained through location coding is called the location code.

[0663] Figure 70 This is a diagram representing an example of a occupancy code. Figure 70 An example of code representing the occupancy of a quadtree structure. Figure 70 In this example, each node is assigned an occupancy code. Each occupancy code indicates whether a node's child nodes or leaf nodes contain a 3D point. For example, in a quadtree, a 4-bit occupancy code indicates whether each node's four child nodes or leaf nodes each contain a 3D point. Similarly, in an octree, an 8-bit occupancy code indicates whether each node's eight child nodes or leaf nodes each contain a 3D point. For simplicity, a quadtree structure is used here, but the same principle applies to octree structures. For example, as... Figure 40 As shown, the occupancy code is... Figure 40 The example described above illustrates a width-first approach that scans both nodes and leaf nodes. In occupancy code, because information from multiple 3D points is decoded in a fixed order, it's not possible to prioritize the decoding of information from arbitrary 3D points. Additionally, occupancy code can also be... Figure 71 The depth-first approach described in the document scans the bit string of nodes and leaf nodes.

[0664] The following explains positional encoding. By using positional codes, important parts of the octree structure can be directly decoded. Furthermore, it enables efficient encoding of important 3D points located deep within the structure.

[0665] Figure 71 This is a diagram used to illustrate positional encoding; it is an example diagram representing a quadtree structure. In Figure 72 In the example shown, three-dimensional points A to I are represented using a quadtree structure. Furthermore, three-dimensional points A and C are important three-dimensional points contained within important regions.

[0666] Figure 71 It means Figure 71 The diagram shows the occupancy codes and position codes of important 3D points A and C in the quadtree structure shown.

[0667] In positional encoding, within a tree structure, the indices of nodes along the path leading to the leaf node to which the 3D point of the object being encoded belongs, and the indices of the leaf nodes themselves, are encoded. Here, the index is a numerical value assigned to each node and leaf node. In other words, the index refers to an identifier used to identify the multiple child nodes of the object node. For example... Figure 71 As shown, in the case of a quadtree, the index represents any one of 0 to 3.

[0668] For example, in Figure 72 In the quadtree structure shown, when leaf node A is a 3D point of the object, leaf node A is represented as 0→2→1→0→1→2→1. Here, the maximum value of each index is 4 in the case shown in the right figure (which can be represented by 2 bits), so the number of bits required for the position code of leaf node A is 7 × 2 bits = 14 bits. When leaf node C is the object being encoded, the number of bits required is also 14 bits. In the case of an octree, since the maximum value of each index is 8 (which can be represented by 3 bits), the number of bits required can be calculated as 3 bits × the depth of the leaf node. In addition, the 3D data encoding device can also reduce the amount of data by performing entropy encoding after binarying each index.

[0669] In addition, such as Figure 72 As shown, in the occupancy code, to decode leaf nodes A and C, all nodes above them need to be decoded. On the other hand, in the position code, only the data of leaf nodes A and C can be decoded. Therefore, as... Figure 72 As shown, by using position codes, the number of bits can be reduced compared to occupancy codes.

[0670] In addition, such as Figure 73 As shown, by applying dictionary-based compression such as LZ77 to part or all of the location code, the amount of code can be further reduced.

[0671] Next, an example of applying position encoding to three-dimensional points (point clouds) obtained through LiDAR will be explained. Figure 74 This is a diagram representing an example of 3D points obtained through LiDAR. 3D points obtained through LiDAR are sparse; that is, when representing the 3D point using occupancy codes, the number of zero values ​​increases. Additionally, high 3D precision is required for these 3D points, meaning the octree structure becomes more hierarchical.

[0672] Figure 74 This is a diagram illustrating an example of such a sparse deep octree structure. Figure 75The occupancy code of the octree structure shown is 136 bits (= 8 bits × 17 nodes). Additionally, since the depth is 6, there are 6 three-dimensional points, so the position code is 3 bits × 6 × 6 = 108 bits. That is, the position code can reduce the code size by 20% compared to the occupancy code. Thus, by applying position encoding to sparse deep octree structures, the code size can be reduced.

[0673] The following explains the code size for occupancy and location codes. With an octree structure depth of 10, the maximum number of 3D points is 8. 10 =1073741824. Additionally, the number of bits L in the octet code of the octet structure. o It is represented by the following.

[0674] L o =8+8 2 +…+8 10 =127133512 bits Therefore, each 3D point has 1.143 bits. Furthermore, in the occupancy code, this number of bits remains unchanged even if the number of 3D points in the octree structure varies.

[0675] On the other hand, in the position code, the number of bits for each 3D point directly affects the depth of the octree structure. Specifically, the number of bits for the position code of each 3D point is 3 bits × depth 10 = 30 bits.

[0676] Therefore, the number of bits L of the position code in the octree structure l It is represented by the following.

[0677] L l =30×N Here, N is the number of three-dimensional points contained in the octree structure.

[0678] Therefore, when N < L o When / 30=40904450.4, that is, when the number of 3D points is less than 40904450, the code size of the location code becomes smaller than that of the occupancy code (L). l <L o ).

[0679] Thus, when there are few 3D points, the code size of the position code is smaller than that of the occupancy code, while when there are many 3D points, the code size of the position code is larger than that of the occupancy code.

[0680] Therefore, the 3D data encoding device can switch between position encoding and occupancy rate encoding depending on the number of input 3D points. In this case, the 3D data encoding device can also append information indicating which encoding method (position encoding or occupancy rate encoding) was used to the header information of the bitstream.

[0681] The following explains hybrid coding that combines location coding and occupancy coding. Hybrid coding, which combines location coding and occupancy coding, is effective when encoding dense, important areas. Figure 75 This is a diagram illustrating this example. In Figure 76 In the example shown, important 3D points are densely arranged. In this case, the 3D data encoding device performs positional encoding on the shallower upper layers and uses occupancy encoding on the lower layers. Specifically, positional encoding is used up to the deepest shared node, and occupancy encoding is used at positions deeper than the deepest shared node. Here, the deepest shared node refers to the deepest node among the nodes that are the common ancestor of multiple important 3D points.

[0682] Next, the hybrid encoding prioritizing compression efficiency will be explained. The three-dimensional data encoding device can also switch between position encoding and occupancy encoding according to predefined rules in the octree encoding.

[0683] Figure 76 This is a diagram illustrating an example of this rule. First, the 3D data encoding device determines the proportion of nodes containing 3D points at each level (depth). If this proportion is higher than a predefined threshold, the 3D data encoding device performs occupancy encoding on several nodes at the upper level of the object level. For example, the 3D data encoding device applies occupancy encoding to the level from the object level to the deepest shared node.

[0684] For example, in Figure 77 In the example shown, the proportion of nodes containing 3D points in level 3 is higher than the threshold. Therefore, the 3D data encoding device applies occupancy encoding to levels 2 and 3 from level 3 to the deepest shared node, and position encoding to levels 1 and 4, which are otherwise different.

[0685] The calculation method for the aforementioned threshold is explained. In one level of an octree structure, there is one root node and eight child nodes. Therefore, in occupancy coding, 8 bits are needed to encode one level of the octree structure. On the other hand, in position coding, 3 bits are needed for each child node containing a 3D point. Therefore, when the number of nodes containing 3D points is greater than 2, occupancy coding is more efficient than position coding. That is, in this case, the threshold is 2.

[0686] The following describes examples of bitstreams generated by the aforementioned position encoding, occupancy rate encoding, or mixed encoding.

[0687] Figure 77 This is a diagram representing an example of a bitstream generated through positional encoding. For example... Figure 77 As shown, the bitstream generated by positional encoding contains a header and multiple positional codes. Each positional code is for a three-dimensional point.

[0688] With this configuration, the 3D data decoding device can decode multiple 3D points with high precision. Furthermore, Figure 78 This example shows a bitstream in the case of a quadtree structure. In the case of an octree structure, each index can take values ​​from 0 to 7.

[0689] Alternatively, the three-dimensional data encoding device can perform entropy encoding after binarying the column (string) representing the index of a three-dimensional point. For example, if the column of the index is 0121, the three-dimensional data encoding device can binary-code 0121 into 00011001 and perform arithmetic encoding on the bit string.

[0690] Figure 78 This is a diagram illustrating an example of a bitstream generated through hybrid encoding that includes important three-dimensional points. (See diagram for example.) Figure 78 As shown, the position codes for the upper layer, the occupancy codes for important 3D points in the lower layer, and the occupancy codes for non-important 3D points in the lower layer are configured sequentially. Additionally, Figure 79 The position code length shown indicates the amount of code at the next position. Additionally, the occupancy code length indicates the amount of code at the next occupancy position.

[0691] With this configuration, the 3D data decoding device can select different decoding schemes according to the application.

[0692] In addition, the encoded data of important 3D points is stored near the beginning of the bit stream, while the encoded data of non-important 3D points not included in the important regions is stored after the encoded data of the important 3D points.

[0693] Figure 78 It means by Figure 80 The occupancy codes of the important 3D points shown are represented by a tree structure diagram. Figure 78 It means by ​ The diagram shows a tree structure representing the occupancy codes of non-critical 3D points. (Example:) Figure 79 As shown, in the occupancy code for important 3D points, information related to non-important 3D points is excluded. Specifically, nodes 0 and 3 at depth 5 do not contain important 3D points, therefore nodes 0 and 3 are assigned the value 0, representing the absence of 3D points.

[0694] On the other hand, such as Figure 80 As shown, information related to important 3D points is excluded from the occupancy code of non-important 3D points. Specifically, node 1 at depth 5 does not contain non-important 3D points, so node 1 is assigned the value 0, representing that it does not contain any 3D points.

[0695] In this way, the 3D data encoding device divides the original tree structure into a first tree structure containing important 3D points and a second tree structure containing non-important 3D points, and encodes the occupancy rate of the first and second tree structures independently. As a result, the 3D data decoding device can prioritize decoding important 3D points.

[0696] Next, we will explain an example of how a bitstream is generated through efficient hybrid coding. Figure 81 This is a diagram illustrating an example of a bitstream generated through efficient hybrid coding. For example... Figure 81 As shown, for each subtree, the subtree root node position, occupancy code amount, and occupancy code are configured sequentially. Figure 81 The subtree position shown is the location code of the root node of the subtree.

[0697] In the above configuration, the following holds true when only positional encoding and occupancy encoding are applied to the octree structure.

[0698] If the length of the positional encoding of the root node of a subtree is equal to the depth of the octree structure, the subtree has no child nodes. That is, positional encoding applies to the entire tree structure.

[0699] Encode the overall usable occupancy rate of the tree structure when the root node of the subtree is equal to the root node of the octree structure.

[0700] For example, based on the above rules, a 3D data decoding device can determine whether a bitstream contains position codes or occupancy codes.

[0701] In addition, the bitstream may also contain encoding mode information indicating which of the following is used: position coding, occupancy coding, or mixed coding. Figure 82 This is a diagram representing an example of a bitstream in this situation. For example, as... Figure 82 As shown, 2 bits of encoding mode information, representing the encoding mode, are appended to the bit stream.

[0702] In addition, the "number of 3D points" in the (1) position encoding represents the number of subsequent 3D points. In addition, the "amount of occupancy code" in the (2) occupancy encoding represents the amount of subsequent occupancy codes. In addition, the "number of important subtrees" in the (3) hybrid encoding (important 3D points) represents the number of subtrees containing important 3D points. Furthermore, the "number of occupancy subtrees" in the (4) hybrid encoding (efficiency emphasis) represents the number of subtrees after occupancy encoding.

[0703] Next, a syntactic example used to switch the application of occupancy encoding and position encoding will be described. Figure 83 It is a diagram showing this syntactic example.

[0704] Figure 83 The shown isleaf is a flag indicating whether the object node is a leaf node. isleaf = 1 indicates that the object node is a leaf node, and isleaf = 0 indicates that the object node is not a leaf node but a node.

[0705] When the object node is a leaf node, point_flag is appended to the bitstream. point_flag is a flag indicating whether the object node (leaf node) contains 3D points. point_flag = 1 indicates that the object node contains 3D points, and point_flag = 0 indicates that the object node does not contain 3D points.

[0706] When the object node is not a leaf node, coding_type is appended to the bitstream. coding_type is the coding type information indicating the applicable coding type. coding_type = 00 indicates that position encoding is applied, coding_type = 01 indicates that occupancy encoding is applied, and coding_type = 10 or 11 indicates that other coding methods are applied, etc.

[0707] When the coding type is position encoding, numPoint, num_idx[i], and idx[i][j] are appended to the bitstream.

[0708] numPoint represents the number of 3D points for which position encoding is performed. num_idx[i] represents the number (depth) of indices from the object node to the 3D point i. When all the 3D points for which position encoding is performed are at the same depth, num_idx[i] are all the same value. Therefore, it can also be that, before the Figure 83 shown for loop (for(i = 0; i < numPoint; i++) {), num_idx is defined as a common value.

[0709] idx[i][j] represents the value of the j-th index in the index from the object node to the 3D point i. In the case of an octree, idx[i][j] has 3 bits.

[0710] Furthermore, as mentioned above, an index refers to an identifier used to identify multiple child nodes of an object node. In the case of an octree, idx[i][j] represents any one of 0 to 7. Additionally, in the case of an octree, there are 8 child nodes, each corresponding to one of the 8 sub-blocks obtained by spatially partitioning the object block corresponding to the object node into 8 parts. Therefore, idx[i][j] can also represent information about the three-dimensional position of the sub-block corresponding to the child node. For example, idx[i][j] can also be a total of 3 bits containing 1 bit each representing the position of each of the x, y, and z axes of the sub-block.

[0711] When the encoding type is occupancy encoding, an occupancy_code is appended to the bitstream. The occupancy_code is the occupancy code of the object node. In the case of an octree, the occupancy_code is, for example, an 8-bit bit string such as "00101000".

[0712] If the (i+1)th bit of occupancy_code is 1, the processing is moved to the child node. That is, the child node is set as the next object node, and the bit string is generated recursively.

[0713] This embodiment illustrates an example of representing the end of an octree by appending leaf node information (isleaf, point_flag) to the bitstream, but it is not necessarily limited to this. For example, a 3D data encoding device can append the maximum depth from the start node (root node) of the occupancy code to the end (leaf node) where the 3D point exists to the head of the start node. Then, the 3D data encoding device can also recursively bit-stringify the information of the child nodes while increasing the depth from the start node, and determine that a leaf node has been reached when the depth reaches the maximum depth. Furthermore, the 3D data encoding device can append the information representing the maximum depth to the initial node where coding_type becomes occupancy encoding, or it can append it to the start node (root node) of the octree.

[0714] As mentioned above, the three-dimensional data encoding device can also append information for switching occupancy rate encoding and position encoding to the bitstream as header information for each node.

[0715] Furthermore, the 3D data encoding device can also perform entropy encoding on the coding_type, numPoint, num_idx, idx, and occupancy_code of each node generated by the above method. For example, the 3D data encoding device performs arithmetic encoding after binarizing each value.

[0716] Furthermore, the syntax described above illustrates the use of a depth-first bit string with an octree structure as the occupancy code, but it is not necessarily limited to this. A 3D data encoding device can also use a width-first bit string with an octree structure as the occupancy code. When using a width-first bit string, the 3D data encoding device can also append information for switching between occupancy code and position code to the bit stream as header information for each node.

[0717] In this embodiment, an octree structure is used as an example, but it is not limited to this. The above method can also be applied to N-ary trees (where N is an integer greater than 2) such as quadtrees and hexadecimals, or other tree structures.

[0718] The following is an example of the encoding process for switching between occupancy rate encoding and location encoding. Figure 84 This is a flowchart of the encoding process in this embodiment.

[0719] First, the 3D data encoding device represents multiple 3D points contained in the 3D data using an octree structure (S1601). Next, the 3D data encoding device sets the root node of the octree structure as the object node (S1602). Then, the 3D data encoding device generates a bit string of the octree structure by performing node encoding processing on the object node (S1603). Finally, the 3D data encoding device generates a bit stream by performing entropy encoding on the generated bit string (S1604).

[0720] Figure 85 This is a flowchart of the node encoding process (S1603). First, the three-dimensional data encoding device determines whether the object node is a leaf node (S1611). If the object node is not a leaf node ("No" in S1611), the three-dimensional data encoding device sets the leaf node flag (isleaf) to 0 and appends the leaf node flag to the bit string (S1612).

[0721] Next, the three-dimensional data encoding device determines whether the number of child nodes containing three-dimensional points is greater than a predetermined threshold (S1613). Alternatively, the three-dimensional data encoding device may also append this threshold to the bit string.

[0722] If the number of child nodes containing three-dimensional points is greater than a predetermined threshold ("Yes" in S1613), the three-dimensional data encoding device sets the encoding type (coding_type) to occupancy encoding and appends the encoding type to the bit string (S1614).

[0723] Next, the 3D data encoding device sets occupancy rate encoding information and appends this occupancy rate encoding information to the bit string. Specifically, the 3D data encoding device generates the occupancy rate code of the object node and appends the occupancy rate code to the bit string (S1615).

[0724] Next, the 3D data encoding device sets the next object node according to the occupancy code (S1616). Specifically, the 3D data encoding device sets the unprocessed child node with an occupancy code of "1" as the next object node.

[0725] Next, the 3D data encoding device performs node encoding processing on the newly defined object nodes (S1617). That is, it performs node encoding processing on the newly defined object nodes. Figure 85 The processing is shown.

[0726] If the processing of all child nodes is not completed (S1618 "No"), the processing after step S1616 is performed again. On the other hand, if the processing of all child nodes is completed (S1618 "Yes"), the three-dimensional data encoding device ends the node encoding process.

[0727] Additionally, in step S1613, if the number of child nodes containing three-dimensional points is below a predetermined threshold ("No" in S1613), the three-dimensional data encoding device sets the encoding type to position encoding and appends the encoding type to the bit string (S1619).

[0728] Next, the three-dimensional data encoding device sets position encoding information and appends it to the bit string. Specifically, the three-dimensional data encoding device generates a position code and appends this position code to the bit string (S1620). The position code includes numPoint, num_idx, and idx.

[0729] Additionally, in step S1611, if the object node is a leaf node (S1611 "Yes"), the 3D data encoding device sets the leaf node flag to 1 and appends the leaf node flag to the bit string (S1621). Furthermore, the 3D data encoding device sets a point flag (point_flag) indicating whether the leaf node contains 3D points and appends the point flag to the bit string (S1622).

[0730] Next, an example of the decoding process for switching between occupancy rate encoding and position encoding will be explained. Figure 85 This is a flowchart of the decoding process in this embodiment.

[0731] The three-dimensional data decoding device generates a bit string by performing entropy decoding on the bit stream (S1631). Next, the three-dimensional data decoding device reconstructs the octree structure by performing node decoding processing on the obtained bit string (S1632). Then, the three-dimensional data decoding device generates three-dimensional points based on the reconstructed octree structure (S1633).

[0732] Figure 87 This is a flowchart of the node decoding process (S1632). First, the 3D data decoding device obtains (decodes) the leaf node flag (isleaf) from the bit string (S1641). Next, the 3D data decoding device determines whether the object node is a leaf node based on the leaf node flag (S1642).

[0733] If the object node is not a leaf node ("No" in S1642), the 3D data decoding device obtains the encoding type (coding_type) from the bit string (S1643). The 3D data decoding device determines whether the encoding type is occupancy-rate encoding (S1644).

[0734] When the encoding type is occupancy encoding ("Yes" in S1644), the 3D data decoding device obtains occupancy encoding information from the bit string. Specifically, the 3D data decoding device obtains the occupancy code from the bit string (S1645).

[0735] Next, the 3D data decoding device sets the next object node according to the occupancy code (S1646). Specifically, the 3D data decoding device sets the unprocessed child node with an occupancy code of "1" as the next object node.

[0736] Next, the 3D data decoding device performs node decoding processing on the newly defined object nodes (S1647). That is, it performs node decoding processing on the newly defined object nodes. Figure 87 The processing is shown.

[0737] If the processing of all child nodes is not completed (S1648 "No"), the processing after step S1646 is performed again. On the other hand, if the processing of all child nodes is completed (S1648 "Yes"), the 3D data decoding device ends the node decoding process.

[0738] Furthermore, if the encoding type in step S1644 is position encoding (S1644 is "No"), the 3D data decoding device obtains position encoding information from the bit string. Specifically, the 3D data decoding device obtains the position code from the bit string (S1649). The position code includes numPoint, num_idx, and idx.

[0739] Additionally, if the object node is a leaf node in step S1642 (S1642 is "yes"), the three-dimensional data decoding device obtains information from the bit string indicating whether the leaf node contains a three-dimensional point, namely the point flag (S1650).

[0740] Furthermore, this embodiment illustrates an example of switching the encoding type per node, but it is not necessarily limited to this. The encoding type can also be fixed in volume, space, or world space units. In this case, the 3D data encoding device can also append encoding type information to the header information of volume, space, or world space.

[0741] As described above, the three-dimensional data encoding apparatus of this embodiment generates first information representing an N-ary (N is an integer greater than 2) tree structure of multiple three-dimensional points contained in the three-dimensional data in a first manner (position encoding), and generates a bit stream containing the first information. The first information includes three-dimensional point information (position code) corresponding to each of the multiple three-dimensional points. Each three-dimensional point information includes an index (idx) corresponding to each of the multiple layers in the N-ary tree structure. Each index represents the sub-block to which the corresponding three-dimensional point belongs among the N sub-blocks of the corresponding layer.

[0742] In other words, each 3D point represents the path to the corresponding 3D point in the N-ary tree structure. Each index represents a child node included in the above path among the N child nodes of the corresponding layer (node).

[0743] Therefore, this three-dimensional data encoding method can generate a bitstream that can selectively decode three-dimensional points.

[0744] For example, the 3D point information (location code) contains information (num_idx) indicating the number of indices contained in that 3D point. In other words, this information represents the depth (number of levels) in the N-ary tree structure up to the corresponding 3D point.

[0745] For example, the first piece of information contains information (numPoint) indicating the number of 3D points contained in that first piece of information. In other words, this information indicates the number of 3D points contained in the N-ary tree structure.

[0746] For example, N is 8, and the index is 3 bits.

[0747] For example, a three-dimensional data encoding device has: a first encoding mode for generating first information; and a second encoding mode for generating second information (occupancy code) representing an N-ary tree structure in a second manner (occupancy code), and generating a bit stream containing the second information. The second information contains a plurality of correspondences to each of a plurality of sub-blocks belonging to a plurality of layers in the N-ary tree structure, and indicates 1 bit of information indicating whether a three-dimensional point exists in the corresponding sub-block.

[0748] For example, the 3D data encoding device uses a first encoding mode when the number of multiple 3D points is below a predetermined threshold, and a second encoding mode when the number of multiple 3D points is greater than the threshold. Thus, the 3D data encoding device can reduce the code size of the bitstream.

[0749] For example, the first and second information include information indicating whether the information represents the N-ary tree structure in the first manner or in the second manner (encoding mode information).

[0750] For example, such as Figure 75 As shown, the three-dimensional data encoding device uses the first encoding mode in one part of the N-ary tree structure and the second encoding mode in another part of the N-ary tree structure.

[0751] For example, a three-dimensional data encoding device has a processor and a memory, and the processor uses the memory to perform the above-mentioned processing.

[0752] Furthermore, the three-dimensional data decoding apparatus of this embodiment obtains first information (position code) from the bitstream representing an N-ary tree structure (N being an integer greater than 2) of multiple three-dimensional points contained in the three-dimensional data in a first manner (position encoding). The first information includes three-dimensional point information (position code) corresponding to each of the multiple three-dimensional points. Each three-dimensional point information includes an index (idx) corresponding to each of the multiple layers in the N-ary tree structure. Each index represents the sub-block to which the corresponding three-dimensional point belongs among the N sub-blocks of the corresponding layer.

[0753] In other words, each 3D point represents the path to the corresponding 3D point in the N-ary tree structure. Each index represents a child node included in the above path among the N child nodes of the corresponding layer (node).

[0754] The 3D data decoding device further uses the 3D point information to reconstruct the 3D point corresponding to the 3D point information.

[0755] Therefore, this three-dimensional data decoding device is able to selectively decode three-dimensional points from the bit stream.

[0756] For example, the 3D point information (location code) contains information (num_idx) indicating the number of indices contained in that 3D point. In other words, this information represents the depth (number of levels) in the N-ary tree structure up to the corresponding 3D point.

[0757] For example, the first piece of information contains information (numPoint) indicating the number of 3D points contained in that first piece of information. In other words, this information indicates the number of 3D points contained in the N-ary tree structure.

[0758] For example, N is 8, and the index is 3 bits.

[0759] For example, the 3D data decoding device further obtains second information (occupancy code) from the bitstream, representing the N-ary tree structure in a second manner (occupancy encoding). The 3D data decoding device uses the second information to reconstruct multiple 3D points. The second information contains multiple correspondences to each of multiple sub-blocks belonging to multiple layers in the N-ary tree structure, and indicates 1 bit of information indicating whether a 3D point exists in the corresponding sub-block.

[0760] For example, the first and second information include information indicating whether the information represents the N-ary tree structure in the first manner or in the second manner (encoding mode information).

[0761] For example, such as Figure 75 As shown, a part of the N-ary tree structure is represented by the first method, and the other part of the N-ary tree structure is represented by the second method.

[0762] For example, a 3D data decoding device has a processor and a memory, and the processor uses the memory to perform the above-mentioned processing.

[0763] (Implementation Method 10) In this embodiment, another example of a tree structure encoding method, such as an octree structure, will be described. Figure 88 This is a diagram illustrating an example of the tree structure described in this embodiment. Additionally, Figure 88 Examples representing quadtree structures.

[0764] Leaf nodes containing 3D points are called valid leaf nodes, and leaf nodes not containing 3D points are called invalid leaf nodes. Branches with more than a threshold number of valid leaf nodes are called dense branches. Branches with fewer than the threshold number of valid leaf nodes are called sparse branches.

[0765] The 3D data encoding device calculates the number of 3D points (i.e., the number of effective leaf nodes) contained in each branch in a certain layer of the tree structure. Figure 88This example illustrates the case where the threshold is 5. In this example, there are two branches in layer 1. Since the left branch contains 7 3D points, it is classified as a dense branch. Since the right branch contains two 3D points, it is classified as a sparse branch.

[0766] Figure 89 For example, a diagram showing the number of valid leaf nodes (3D points) in each branch of layer 5. Figure 89 The horizontal axis represents the identification number, or index, of the branch in layer 5. For example... Figure 89 As shown, a specific branch contains significantly more 3D points than other branches. In such dense branches, occupancy coding is more efficient compared to sparse branches.

[0767] The following explains the application methods of occupancy rate coding and location coding. Figure 90 This is a graph showing the relationship between the number of 3D points (the number of effective leaf nodes) in each branch of layer 5 and the applied coding method. For example... Figure 90 As shown, the 3D data encoding device applies occupancy-based encoding to dense branches and position-based encoding to sparse branches. This improves encoding efficiency.

[0768] Figure 91 This is a diagram illustrating an example of a densely branched region in LiDAR data. For example... Figure 91 As shown, the density of 3D points calculated based on the number of 3D points contained in each branch varies depending on the region.

[0769] Furthermore, separating dense 3D points (branches) from sparse 3D points (branches) offers the following advantages: The closer to the LiDAR sensor, the higher the density of 3D points. Therefore, by separating the branches according to their density, it is possible to perform range-direction partitioning. Such partitioning is effective in certain applications. Moreover, for sparse branches, methods other than occupancy rate coding are effective.

[0770] In this embodiment, the three-dimensional data encoding device separates the input three-dimensional point group into two or more sub-three-dimensional point groups and applies different encoding methods to each sub-three-dimensional point group.

[0771] For example, a 3D data encoding device can separate an input 3D point group into a sub-3D point group A (dense cloud) that includes dense branches and a sub-3D point group B (sparse cloud) that includes sparse branches. Figure 92 It means from Figure 88 The diagram shows an example of a subgroup of three-dimensional points A (dense three-dimensional point group) separated from a tree structure, including dense branches. Figure 93 It means from Figure 88 The diagram shows an example of a subgroup of three-dimensional points B (a sparse three-dimensional point group) separated from the tree structure, including sparse branches.

[0772] Next, the 3D data encoding device encodes sub-3D point group A using occupancy rate encoding and sub-3D point group B using position encoding.

[0773] Furthermore, this example illustrates the application of different encoding methods (occupancy rate encoding and position encoding) as different encoding approaches. However, for example, a three-dimensional data encoding device can also use the same encoding method for sub-three-dimensional point group A and sub-three-dimensional point group B, and make the parameters used in encoding different between sub-three-dimensional point group A and sub-three-dimensional point group B.

[0774] The following describes the process of three-dimensional data encoding by a three-dimensional data encoding device. Figure 94 This is a flowchart of the three-dimensional data encoding process performed by the three-dimensional data encoding apparatus of this embodiment.

[0775] First, the 3D data encoding device separates the input 3D point group into sub-3D point groups (S1701). This separation can be performed automatically or based on information input by the user. For example, the user can specify the range of the sub-3D point groups. Furthermore, as an example of automatic separation, when the input data is LiDAR data, the 3D data encoding device uses distance information to each point group for separation. Specifically, the 3D data encoding device separates point groups within a certain range from the measurement location from point groups outside that range. Additionally, the 3D data encoding device can also use information about important and unimportant areas for separation.

[0776] Next, the 3D data encoding device encodes sub-3D point group A using method A to generate encoded data (encoded bitstream) (S1702). Furthermore, the 3D data encoding device encodes sub-3D point group B using method B to generate encoded data (S1703). Alternatively, the 3D data encoding device can also encode sub-3D point group B using method A. In this case, the 3D data encoding device encodes sub-3D point group B using parameters different from the encoding parameters used in the encoding of sub-3D point group A. For example, this parameter could be a quantization parameter. For example, the 3D data encoding device encodes sub-3D point group B using a quantization parameter larger than the quantization parameter used in the encoding of sub-3D point group A. In this case, the 3D data encoding device can also append information indicating the quantization parameters used in the encoding of each sub-3D point group to the header of the encoded data.

[0777] Next, the three-dimensional data encoding device generates a bit stream by combining the encoded data obtained in step S1702 with the encoded data obtained in step S1703 (S1704).

[0778] Furthermore, the 3D data encoding device can also encode information used to decode each sub-3D point group, which is also part of the bitstream header information. For example, the 3D data encoding device can also encode information such as the following.

[0779] The header information can also include information indicating the number of coded sub-3D points. In this example, the information indicates 2.

[0780] The header information may also include information indicating the number of 3D points contained in each sub-3D point group and the encoding method. In this example, the information indicates the number of 3D points contained in sub-3D point group A, the encoding method applied to sub-3D point group A (method A), the number of 3D points contained in sub-3D point group B, and the encoding method applied to sub-3D point group B (method B).

[0781] The header information may also include information used to identify the start or end positions of the coded data for each sub-3D point group.

[0782] Furthermore, the 3D data encoding device can also encode sub-3D point group A and sub-3D point group B in parallel. Alternatively, the 3D data encoding device can also encode sub-3D point group A and sub-3D point group B sequentially.

[0783] Furthermore, the method for separating sub-3D point groups is not limited to the above. For example, the 3D data encoding device changes the separation method, encodes using each of multiple separation methods, and calculates the encoding efficiency of the encoded data obtained using each separation method. The 3D data encoding device then selects the separation method with the highest encoding efficiency. For example, the 3D data encoding device may also separate the 3D point group in each of multiple layers, calculate the encoding efficiency in each case, select the separation method with the highest encoding efficiency (i.e., the layer where separation is performed), and generate sub-3D point groups using the selected separation method for encoding.

[0784] Furthermore, when combining encoded data, the 3D data encoding device can position the encoded information of more important sub-3D point groups closer to the beginning of the bitstream. Therefore, the 3D data decoding device can obtain important information simply by decoding the initial bitstream, thus enabling it to acquire crucial information earlier.

[0785] Next, the process of three-dimensional data decoding by the three-dimensional data decoding device will be explained. Figure 95 This is a flowchart of the three-dimensional data decoding process performed by the three-dimensional data decoding apparatus of this embodiment.

[0786] First, the three-dimensional data decoding device obtains, for example, the bitstream generated by the three-dimensional data encoding device described above. Next, the three-dimensional data decoding device separates the encoded data of sub-three-dimensional point group A from the obtained bitstream (S1711). Specifically, the three-dimensional data decoding device decodes the information used to decode each sub-three-dimensional point group from the header information of the bitstream, and uses this information to separate the encoded data of each sub-three-dimensional point group.

[0787] Next, the 3D data decoding device obtains sub-3D point group A by decoding the encoded data of sub-3D point group A using method A (S1712). Furthermore, the 3D data decoding device obtains sub-3D point group B by decoding the encoded data of sub-3D point group B using method B (S1713). Then, the 3D data decoding device combines sub-3D point group A and sub-3D point group B (S1714).

[0788] Alternatively, the 3D data decoding device can decode sub-3D point group A and sub-3D point group B in parallel. Or, the 3D data decoding device can decode sub-3D point group A and sub-3D point group B sequentially.

[0789] Furthermore, the 3D data decoding device can also decode the required sub-3D point groups. For example, the 3D data decoding device can decode sub-3D point group A but not sub-3D point group B. For example, if sub-3D point group A is a 3D point group contained in an important region of LiDAR data, the 3D data decoding device decodes the 3D point group of that important region. The 3D point group of the important region is then used for self-position estimation of vehicles, etc.

[0790] Next, a specific example of the encoding process in this embodiment will be described. Figure 96 This is a flowchart of the three-dimensional data encoding process performed by the three-dimensional data encoding apparatus of this embodiment.

[0791] First, the 3D data encoding device separates the input 3D points into sparse and dense 3D point groups (S1721). Specifically, the 3D data encoding device counts the number of valid leaf nodes in a branch of a certain layer of the octree structure. Based on the number of valid leaf nodes in each branch, the 3D data encoding device classifies each branch as a dense or sparse branch. Furthermore, the 3D data encoding device generates sub-3D point groups (dense 3D point groups) that aggregate dense branches and sub-3D point groups (sparse 3D point groups) that aggregate sparse branches.

[0792] Next, the three-dimensional data encoding device generates encoded data by encoding the sparse three-dimensional point group (S1722). For example, the three-dimensional data encoding device uses position encoding to encode the sparse three-dimensional point group.

[0793] Furthermore, the three-dimensional data encoding device generates encoded data by encoding dense three-dimensional point groups (S1723). For example, the three-dimensional data encoding device uses occupancy rate encoding to encode dense three-dimensional point groups.

[0794] Next, the three-dimensional data encoding device generates a bit stream by combining the encoded data of the sparse three-dimensional point group obtained in step S1722 with the encoded data of the dense three-dimensional point group obtained in step S1723 (S1724).

[0795] Furthermore, the 3D data encoding device can also encode information used for decoding sparse and dense 3D point groups as header information of the bitstream. For example, the 3D data encoding device can also encode information as described below.

[0796] The header information can also include information indicating the number of coded subgroups of 3D points. In this example, the information indicates 2.

[0797] The header information may also include information indicating the number of 3D points contained in each sub-3D point group and the encoding method. In this example, the information indicates the number of 3D points contained in the sparse 3D point group, the encoding method applied to the sparse 3D point group (positional encoding), the number of 3D points contained in the dense 3D point group, and the encoding method applied to the dense 3D point group (occupancy encoding).

[0798] The header information may also include information used to identify the start or end positions of the encoded data for each sub-3D point group. In this example, the information represents at least one of the start and end positions of the encoded data for a sparse 3D point group and the start and end positions of the encoded data for a dense 3D point group.

[0799] Furthermore, the 3D data encoding device can also encode sparse and dense 3D point groups in parallel. Alternatively, the 3D data encoding device can also encode sparse and dense 3D point groups sequentially.

[0800] Next, a specific example of 3D data decoding processing will be explained. Figure 97 This is a flowchart of the three-dimensional data decoding process performed by the three-dimensional data decoding apparatus of this embodiment.

[0801] First, the 3D data decoding apparatus obtains, for example, the bitstream generated by the aforementioned 3D data encoding apparatus. Next, the 3D data decoding apparatus separates the obtained bitstream into coded data of sparse 3D point groups and coded data of dense 3D point groups (S1731). Specifically, the 3D data decoding apparatus decodes the header information of the bitstream, which is used to decode the information of each sub-3D point group, and uses this information to separate the coded data of each sub-3D point group. In this example, the 3D data decoding apparatus uses the header information to separate the coded data of sparse and dense 3D point groups from the bitstream.

[0802] Next, the 3D data decoding device obtains the sparse 3D point group by decoding the encoded data of the sparse 3D point group (S1732). For example, the 3D data decoding device decodes the sparse 3D point group using position decoding, which is used to decode the encoded data that will be encoded at the position.

[0803] Furthermore, the three-dimensional data decoding device obtains a dense three-dimensional point group by decoding the encoded data of the dense three-dimensional point group (S1733). For example, the three-dimensional data decoding device decodes the dense three-dimensional point group using occupancy rate decoding, which is used to decode the encoded data encoded by the occupancy rate.

[0804] Next, the three-dimensional data decoding device combines the sparse three-dimensional point group obtained in step S1732 with the dense three-dimensional point group obtained in step S1733 (S1734).

[0805] Alternatively, the 3D data decoding device can decode sparse and dense 3D point groups in parallel. Or, it can decode sparse and dense 3D point groups sequentially.

[0806] Furthermore, the 3D data decoding device can also decode a portion of the required sub-3D point groups. For example, the 3D data decoding device can decode dense 3D point groups without decoding sparse 3D data. For instance, if the dense 3D point group is contained within an important region of LiDAR data, the 3D data decoding device decodes the 3D point group of that important region. The 3D point group of the important region can then be used for tasks such as vehicle self-position estimation.

[0807] Figure 98 This is a flowchart of the encoding process in this embodiment. First, the three-dimensional data encoding device generates sparse three-dimensional point groups and dense three-dimensional point groups by separating the input three-dimensional point group into sparse three-dimensional point groups and dense three-dimensional point groups (S1741).

[0808] Next, the three-dimensional data encoding device generates encoded data by encoding dense three-dimensional point groups (S1742). Furthermore, the three-dimensional data encoding device generates encoded data by encoding sparse three-dimensional point groups (S1743). Finally, the three-dimensional data encoding device generates a bitstream by combining the encoded data of the sparse three-dimensional point groups obtained in step S1742 with the encoded data of the dense three-dimensional point groups obtained in step S1743 (S1744).

[0809] Figure 99 This is a flowchart of the decoding process according to this embodiment. First, the three-dimensional data decoding device extracts encoded data of dense three-dimensional point groups and encoded data of sparse three-dimensional point groups from the bitstream (S1751). Next, the three-dimensional data decoding device obtains decoded data of dense three-dimensional point groups by decoding the encoded data of dense three-dimensional point groups (S1752). Furthermore, the three-dimensional data decoding device obtains decoded data of sparse three-dimensional point groups by decoding the encoded data of sparse three-dimensional point groups (S1753). Next, the three-dimensional data decoding device generates three-dimensional point groups by combining the decoded data of dense three-dimensional point groups obtained in step S1752 with the decoded data of sparse three-dimensional point groups obtained in step S1753 (S1754).

[0810] Furthermore, the 3D data encoding and decoding devices can choose to encode or decode either dense or sparse 3D point groups first. Additionally, encoding or decoding can be performed in parallel by multiple processors.

[0811] Furthermore, the 3D data encoding device can also encode either dense or sparse 3D point groups. For example, when important information is contained within a dense 3D point group, the 3D data encoding device extracts both the dense and sparse 3D point groups from the input 3D point group, encoding the dense group but not the sparse group. Thus, the 3D data encoding device can append important information to the stream while minimizing bit load. For instance, in a server-client communication where the client requests information about the 3D point groups around the client, the server encodes the important information about the client's surroundings as a dense 3D point group and sends it to the client. This allows the server to send the requested information while minimizing network bandwidth usage.

[0812] Furthermore, the 3D data decoding device can decode either dense or sparse 3D point groups. For example, if important information is contained in a dense 3D point group, the 3D data decoding device will decode the dense point group but not the sparse point group. Thus, the 3D data decoding device can obtain the required information while suppressing the processing load of the decoding process.

[0813] Figure 100 yes Figure 98 The flowchart shown illustrates the separation process of three-dimensional points (S1741). First, the three-dimensional data encoding device sets the layer L and the threshold TH (S1761). Alternatively, the three-dimensional data encoding device can append information representing the set layer L and threshold TH to the bitstream. That is, the three-dimensional data encoding device can also generate a bitstream containing information representing the set layer L and threshold TH.

[0814] Next, the 3D data encoding device moves the position of the object to be processed from the root node of the octree to the beginning branch of layer L. That is, the 3D data encoding device selects the beginning branch of layer L as the branch of the object to be processed (S1762).

[0815] Next, the 3D data encoding device counts the number of valid leaf nodes of the branch of the processing object in layer L (S1763). If the number of valid leaf nodes of the branch of the processing object is greater than the threshold TH ("Yes" in S1764), the 3D data encoding device registers the branch of the processing object as a dense branch and adds it to the dense 3D point group (S1765). On the other hand, if the number of valid leaf nodes of the branch of the processing object is less than or equal to the threshold TH ("No" in S1764), the 3D data encoding device registers the branch of the processing object as a sparse branch and adds it to the sparse 3D point group (S1766).

[0816] If the processing of all branches of layer L has not been completed (No in S1767), the 3D data encoding device moves the position of the processing object to the next branch of layer L. That is, the 3D data encoding device selects the next branch of layer L as the branch of the processing object (S1768). Furthermore, the 3D data encoding device performs the processing after step S1763 on the selected next branch of the processing object.

[0817] Repeat the above process until all branches of layer L have been processed (S1767 "Yes").

[0818] Furthermore, while the layer L and threshold TH are predefined in the above description, this is not a limitation. For example, the 3D data encoding device sets multiple patterns for the groups of layer L and threshold TH, generates dense and sparse 3D point groups using each group, and encodes them separately. The 3D data encoding device ultimately encodes the dense and sparse 3D point groups using the group of layer L and threshold TH that has the highest encoding efficiency among the multiple groups. This improves encoding efficiency. Additionally, the 3D data encoding device can also calculate layer L and threshold TH. For example, the 3D data encoding device can set layer L to half the maximum value of the layers contained in the tree structure. Furthermore, the 3D data encoding device can set the threshold TH to half the total number of multiple 3D points contained in the tree structure.

[0819] Furthermore, the above description illustrates examples of classifying input 3D point groups into dense and sparse 3D point groups. However, the 3D data encoding device can also classify input 3D point groups into three or more types. For example, if the number of effective leaf nodes of a branch of the processing object is a threshold TH1 or higher, the 3D data encoding device classifies the branch of the processing object into a first dense 3D point group; if the number of effective leaf nodes of the branch of the processing object is less than the first threshold TH1 but is a second threshold TH2 or higher, the 3D data encoding device classifies the branch of the processing object into a first sparse 3D point group; if the number of effective leaf nodes of the branch of the processing object is less than the second threshold TH2 but is a third threshold TH3 or higher, the 3D data encoding device classifies the branch of the processing object into a second sparse 3D point group; if the number of effective leaf nodes of the branch of the processing object is less than the threshold TH3, the 3D data encoding device classifies the branch of the processing object into a second sparse 3D point group.

[0820] The following describes a syntax example of the encoding data for the three-dimensional point group in this embodiment. Figure 101 This is a graph representing the syntax example. `pc_header()`, for example, contains header information for multiple 3D points that are input.

[0821] Figure 101 The given num_sub_pc represents the number of sub-3D point groups. numPoint[i] represents the number of 3D points contained in the i-th sub-3D point group. coding_type[i] is the encoding type information indicating the encoding type (encoding method) applied to the i-th sub-3D point group. For example, coding_type=00 indicates that positional encoding is applied. coding_type=01 indicates that occupancy encoding is applied. coding_type=10 or 11 indicates that other encoding methods are applied.

[0822] `data_sub_cloud()` represents the encoded data of the i-th sub-3D point group. `coding_type_00_data` represents the encoded data using the encoding type `coding_type` of 00, such as encoded data using position encoding. `coding_type_01_data` represents the encoded data using the encoding type `coding_type` of 01, such as encoded data using occupancy encoding.

[0823] `end_of_data` represents end-of-data information. For example, for this `end_of_data`, there is no fixed bit string allocated for the encoded data. Therefore, a 3D data decoding device can skip decoding of encoded data that does not need to be decoded by searching for the bit string of `end_of_data` in the bit stream.

[0824] Alternatively, the 3D data encoding device can also perform entropy encoding on the encoded data generated by the above method. For example, the 3D data encoding device performs encoding calculations after binarizing each value.

[0825] Furthermore, this embodiment illustrates examples of quadtree or octree structures, but it is not necessarily limited to these. The above method can also be applied to binary trees, 16-trees, N-trees (where N is an integer greater than 2), or other tree structures.

[0826] [Variation Example] In the above explanation, such as Figure 93 and Figure 94 As shown, tree structures including dense branches and their upper levels (tree structures from the root node of the overall tree structure to the root nodes of the dense branches) are encoded, as are tree structures including sparse branches and their upper levels (tree structures from the root node of the overall tree structure to the root nodes of the sparse branches). In this variation, the 3D data encoding device separates dense branches from sparse branches and encodes both. That is, the encoded tree structure does not include the upper-level tree structure. For example, the 3D data encoding device applies occupancy encoding to dense branches and position encoding to sparse branches.

[0827] Figure 102 It means from Figure 88 The diagram shows an example of a densely branched tree structure. Figure 103 It means from Figure 88 The diagram shows an example of a tree structure with sparsely branched separation. This variation will... Figure 102 and Figure 103 The tree structures shown are encoded respectively.

[0828] Furthermore, instead of encoding the upper-level tree structure, the three-dimensional data encoding device encodes information representing the location of branches. For example, this information represents the location of the root node of a branch.

[0829] For example, a 3D data encoding device encodes layer information representing a layer with dense branches and branch information indicating which branch of that layer the dense branch belongs to as encoded data for that dense branch. Thus, a 3D data decoding device can decode the layer and branch information from the bitstream and use this information to determine which layer and which branch of the 3D point group the decoded dense branch belongs to. Similarly, a 3D data encoding device encodes layer information representing a layer with sparse branches and branch information indicating which branch of that layer the sparse branch belongs to as encoded data for that sparse branch.

[0830] Therefore, the 3D data decoding device can decode layer information and branch information from the bitstream, and use this layer information and branch information to determine which layer and which branch of the 3D point group the decoded sparse branch belongs to. This reduces the overhead of encoding information from layers higher than dense and sparse branches, thus improving encoding efficiency.

[0831] Alternatively, branch information can represent the values ​​assigned to each branch within a layer represented by layer information. Furthermore, branch information can also represent the values ​​assigned to each node starting from the root node of the octree. In this case, layer information may not need to be encoded. Additionally, the 3D data encoding device can generate multiple dense branches and multiple sparse branches separately.

[0832] Figure 104 This is a flowchart of the encoding process in this variant example. First, the three-dimensional data encoding device generates one or more sparse branches and one or more dense branches based on the input three-dimensional point group (S1771).

[0833] Next, the three-dimensional data encoding device generates encoded data by encoding the dense branches (S1772). Then, the three-dimensional data encoding device determines whether the encoding of all the dense branches generated in step S1771 has been completed (S1773).

[0834] If the encoding of all dense branches is not completed ("No" in S1773), the three-dimensional data encoding device selects the next dense branch (S1774) and generates encoded data by encoding the selected dense branch (S1772).

[0835] On the other hand, if the encoding of all dense branches is completed ("Yes" in S1773), the three-dimensional data encoding device generates encoded data by encoding the sparse branches (S1775). Next, the three-dimensional data encoding device determines whether the encoding of all sparse branches generated in step S1771 has been completed (S1776).

[0836] If the encoding of all sparse branches is not completed ("No" in S1776), the three-dimensional data encoding device selects the next sparse branch (S1777) and generates encoded data by encoding the selected sparse branch (S1775).

[0837] On the other hand, when the encoding of all sparse branches is completed ("Yes" in S1776), the three-dimensional data encoding device combines the encoded data generated in steps S1772 and S1775 to generate a bit stream (S1778).

[0838] Figure 104 This is a flowchart of the decoding process in this variant example. First, the three-dimensional data decoding device extracts one or more encoded data of dense branches and one or more encoded data of sparse branches from the bitstream (S1781). Next, the three-dimensional data decoding device decodes the encoded data of dense branches to obtain the decoded data of dense branches (S1782).

[0839] Next, the three-dimensional data decoding device determines whether the decoding of the encoded data of all dense branches extracted in step S1781 has been completed (S1783). If the decoding of the encoded data of all dense branches has not been completed ("No" in S1783), the three-dimensional data decoding device selects the encoded data of the next dense branch (S1784) and obtains the decoded data of the dense branch by decoding the encoded data of the selected dense branch (S1782).

[0840] On the other hand, when the decoding of the encoded data of all dense branches is completed (S1783 "Yes"), the three-dimensional data decoding device obtains the decoded data of sparse branches by decoding the encoded data of sparse branches (S1785).

[0841] Next, the 3D data decoding device determines whether the decoding of the encoded data of all sparse branches extracted in step S1781 has been completed (S1786). If the decoding of the encoded data of all sparse branches has not been completed ("No" in S1786), the 3D data decoding device selects the encoded data of the next sparse branch (S1787), and obtains the decoded data of the sparse branch by decoding the encoded data of the selected sparse branch (S1785).

[0842] On the other hand, when the decoding of the encoded data of all sparse branches has been completed ("Yes" in S1786), the three-dimensional data decoding device generates a three-dimensional point group by combining the decoded data obtained in steps S1782 and S1785 (S1788).

[0843] Furthermore, the 3D data encoding and decoding devices can choose to encode or decode either dense or sparse branches first. Additionally, encoding or decoding can be performed in parallel by multiple processors.

[0844] Furthermore, the 3D data encoding device can encode either dense or sparse branches. It can also encode a portion of multiple dense branches. For example, if important information is contained in a specific dense branch, the 3D data encoding device extracts the dense and sparse branches from the input 3D point set, encodes the dense branch containing the important information, and does not encode other dense or sparse branches. Thus, the 3D data encoding device can append important information to the stream while suppressing bit volume. For example, in a request between a server and a client to send information about the 3D point set around the client, the server encodes the important information around the client as a dense branch and sends it to the client. Thus, the server can send the client's requested information while suppressing network bandwidth.

[0845] Furthermore, the 3D data decoding device can decode either dense or sparse branches. It can also decode a portion of multiple dense branches. For example, if a particular dense branch contains important information, the 3D data decoding device decodes that specific dense branch without decoding other dense or sparse branches. Thus, the 3D data decoding device can obtain the necessary information while suppressing the processing load of the decoding process.

[0846] Figure 106 yes Figure 104 The flowchart shown illustrates the separation process of three-dimensional points (S1771). First, the three-dimensional data encoding device sets the layer L and the threshold TH (S1761). Alternatively, the three-dimensional data encoding device can append information representing the set layer L and threshold TH to the bitstream.

[0847] Next, the 3D data encoding device selects the first branch of layer L as the branch to be processed (S1762). Then, the 3D data encoding device counts the number of valid leaf nodes of the branch to be processed in layer L (S1763). If the number of valid leaf nodes of the branch to be processed is greater than the threshold TH ("Yes" in S1764), the 3D data encoding device sets the branch to be processed as a dense branch and adds layer information and branch information to the bitstream (S1765A). On the other hand, if the number of valid leaf nodes of the branch to be processed is less than or equal to the threshold TH ("No" in S1764), the 3D data encoding device sets the branch to be processed as a sparse branch and adds layer information and branch information to the bitstream (S1766A).

[0848] If the processing of all branches of layer L has not been completed (S1767 "No"), the 3D data encoding device selects the next branch of layer L as the branch to be processed (S1768). Then, the 3D data encoding device performs the processing steps after S1763 on the selected next branch to be processed. This process is repeated until the processing of all branches of layer L is completed (S1767 "Yes").

[0849] Furthermore, while the layer L and threshold TH are predefined in the above description, this is not a limitation. For example, the 3D data encoding apparatus sets multiple groups of layers L and threshold TH for different patterns, uses each group to generate dense and sparse branches, and encodes them separately. The 3D data encoding apparatus ultimately encodes the dense and sparse branches using the group of layers L and threshold TH that generates the most efficient encoded data. This improves encoding efficiency. Additionally, the 3D data encoding apparatus can also calculate the layer L and threshold TH. For example, the 3D data encoding apparatus can set the layer L as half the maximum value of the layers contained in the tree structure. Furthermore, the 3D data encoding apparatus can set the threshold TH as half the total number of multiple 3D points contained in the tree structure.

[0850] The following is a syntactic example of the encoding data for the three-dimensional point group in this variant. Figure 107 This is a diagram representing the syntax example. In Figure 107 In the syntactic examples shown, relative to Figure 101 The syntax example shown appends layer_id[i] as layer information...

Claims

1. A three-dimensional data encoding method, wherein, For the information of object nodes contained in an N-ary tree structure of multiple 3D points in 3D data, only the information of referable nodes among multiple spatially adjacent neighboring nodes is used for encoding, where N is an integer greater than 2. Encoding parameters, Generate a bitstream containing the encoded information of the object node and the parameters. When the parameter includes the first value, the reference node is only a sibling node of the object node.

2. The three-dimensional data encoding method according to claim 1, wherein, The value of the parameter indicates the reference node.

3. The three-dimensional data encoding method according to claim 2, wherein, The object node and the sibling node have the same parent node.

4. The three-dimensional data encoding method according to claim 3, wherein, If the parameter includes a second value, the reference node includes the sibling node and other nodes.

5. The three-dimensional data encoding method according to claim 1, wherein, When the parameter includes the second value, the referable node includes the sibling node of the object node and nodes other than the sibling node.

6. The three-dimensional data encoding method according to claim 1, wherein, The parameters are included in the header information of the bitstream.

7. A three-dimensional data decoding method, wherein, Obtain parameters from the bitstream. For the information of object nodes contained in an N-ary tree structure of multiple 3D points in 3D data, decoding is performed using only the information of referential nodes among multiple spatially adjacent neighboring nodes of the object node, where N is an integer greater than 2. When the parameter includes the first value, the reference node is only a sibling node of the object node.

8. The three-dimensional data decoding method according to claim 7, wherein, The value of the parameter indicates the reference node.

9. The three-dimensional data decoding method according to claim 8, wherein, The object node and the sibling node have the same parent node.

10. The three-dimensional data decoding method according to claim 9, wherein, If the parameter includes a second value, the reference node includes the sibling node and other nodes.

11. The three-dimensional data decoding method according to claim 7, wherein, When the parameter includes the second value, the referable node includes the sibling node of the object node and nodes other than the sibling node.

12. The three-dimensional data decoding method according to claim 7, wherein, The parameters are included in the header of the bitstream.

13. A three-dimensional data encoding device, wherein, have: Processor; and memory, The processor uses the memory. For the information of object nodes contained in an N-ary tree structure of multiple 3D points in 3D data, only the information of referable nodes among multiple spatially adjacent neighboring nodes is used for encoding, where N is an integer greater than 2. Encoding parameters, Generate a bitstream containing the encoded information of the object node and the parameters. When the parameter includes the first value, the reference node is only a sibling node of the object node.

14. A three-dimensional data decoding device, wherein, have: Processor; and memory, The processor uses the memory. Obtain parameters from the bitstream. For the information of object nodes contained in an N-ary tree structure of multiple 3D points in 3D data, decoding is performed using only the information of referential nodes among multiple spatially adjacent neighboring nodes of the object node, where N is an integer greater than 2. When the parameter includes the first value, the reference node is only a sibling node of the object node.

Images