Three-dimensional data encoding method, three-dimensional data decoding method, three-dimensional data encoding device, and three-dimensional data decoding device

Abstract

This three-dimensional data encoding method sets, in a plurality of items of position information for a plurality of three-dimensional points included in point cloud data, a hierarchy structure having a plurality of depths and a plurality of layers each including at least one depth (S8881), generates a plurality of items of first encoding data for each depth by encoding the plurality of items of position information for each depth (S8882), and generates a bitstream containing a plurality of items of second encoding data, the second encoding data being the encoding data for each layer and containing one or more items of first encoding data for one or more depths included in the corresponding layer (S8883). The bitstream contains first information indicating the data length of each of the plurality of items of second encoding data.

Description

[Technical field]

[0001] The present disclosure relates to a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, and a three-dimensional data decoding device. [Background technology]

[0002] In the future, devices and services that utilize 3D data are expected to become widespread in a wide range of fields, such as computer vision for autonomous operation of automobiles or robots, map information, monitoring, infrastructure inspection, video distribution, etc. 3D data is acquired in various ways, such as distance sensors such as range finders, stereo cameras, or a combination of multiple monocular cameras.

[0003] One method of expressing three-dimensional data is a method called a point cloud, which represents the shape of a three-dimensional structure using a group of points in a three-dimensional space. In a point cloud, the positions and colors of the points are stored. Point clouds are expected to become the mainstream method of expressing three-dimensional data, but point clouds have a very large amount of data. Therefore, when storing or transmitting three-dimensional data, it is essential to compress the amount of data by encoding, just like two-dimensional video images (examples include MPEG-4 AVC or HEVC standardized by MPEG).

[0004] In addition, compression of point clouds is partially supported by public libraries (Point Cloud Library) that perform point cloud-related processing.

[0005] Furthermore, a technique is known that uses three-dimensional map data to search for and display facilities located around a vehicle (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0006] [Patent Document 1] International Publication No. 2014 / 020663 Summary of the Invention [Problem to be solved by the invention]

[0007] In the encoding process and the decoding process of three-dimensional data, it is desirable to reduce the amount of processing.

[0008] An object of the present disclosure is to provide a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, or a three-dimensional data decoding device that can reduce the amount of processing. [Means for solving the problem]

[0009] A three-dimensional data encoding method according to one aspect of the present disclosure includes the steps of: generating, by a processor, at least one of a plurality of sub-encoded data constituting encoded data in which a plurality of three-dimensional points are occupancy-encoded using a hierarchical structure having a plurality of hierarchical levels; Among a plurality of pieces of first information each corresponding to the plurality of sub-encoded data, First information corresponding to the generated sub-encoded data is generated by the processor, and each of the multiple sub-encoded data corresponds to a group hierarchy in which one or more hierarchies from the multiple hierarchies are grouped, and the first information includes information indicating the number of hierarchies included in the group hierarchy corresponding to the generated sub-encoded data.

[0010] A three-dimensional data decoding method according to one aspect of the present disclosure includes: acquiring, by a processor, at least one of a plurality of sub-encoded data constituting encoded data in which a plurality of three-dimensional points are occupancy-encoded using a hierarchical structure having a plurality of layers; Among a plurality of pieces of first information each corresponding to the plurality of sub-encoded data, First information corresponding to the acquired sub-encoded data is acquired by the processor, and the acquired sub-encoded data is decoded by the processor based on the first information, wherein each of the multiple sub-encoded data corresponds to a group hierarchy in which one or more hierarchies from the multiple hierarchies are grouped, and the first information includes information indicating the number of hierarchies included in the group hierarchy corresponding to the acquired sub-encoded data. Effect of the Invention

[0011] The present disclosure can provide a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, or a three-dimensional data decoding device that can reduce the amount of processing. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram showing a configuration of a three-dimensional data encoding / decoding system according to the first embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of a configuration of point cloud data according to the first embodiment. [Diagram 3] FIG. 3 is a diagram showing an example of a configuration of a data file in which point cloud data information according to the first embodiment is described. [Figure 4] FIG. 4 is a diagram showing types of point cloud data according to the first embodiment. [Diagram 5] FIG. 5 is a diagram showing a configuration of a first encoding unit according to the first embodiment. [Figure 6] FIG. 6 is a block diagram of a first encoding unit according to the first embodiment. [Figure 7] FIG. 7 is a diagram showing a configuration of a first decoding unit according to the first embodiment. [Figure 8] FIG. 8 is a block diagram of a first decoding unit according to the first embodiment. [Figure 9] FIG. 9 is a block diagram of a three-dimensional data encoding device according to the first embodiment. [Figure 10] FIG. 10 is a diagram showing an example of location information according to the first embodiment. [Figure 11] FIG. 11 is a diagram showing an example of an octree representation of position information according to the first embodiment. [Figure 12] FIG. 12 is a block diagram of the three-dimensional data decoding device according to the first embodiment. [Figure 13] FIG. 13 is a block diagram of the attribute information encoding unit according to the first embodiment. [Figure 14] FIG. 14 is a block diagram of the attribute information decoding unit according to the first embodiment. [Figure 15]FIG. 15 is a block diagram showing a configuration of an attribute information encoding unit according to the first embodiment. As shown in FIG. [Figure 16] FIG. 16 is a block diagram of the attribute information encoding unit according to the first embodiment. [Figure 17] FIG. 17 is a block diagram showing a configuration of an attribute information decoding unit according to the first embodiment. As shown in FIG. [Figure 18] FIG. 18 is a block diagram of the attribute information decoding unit according to the first embodiment. As shown in FIG. [Figure 19] FIG. 19 is a diagram showing a configuration of a second encoding unit according to the first embodiment. As shown in FIG. [Figure 20] FIG. 20 is a block diagram of a second encoding unit according to the first embodiment. [Figure 21] FIG. 21 is a diagram illustrating a configuration of a second decoding unit according to the first embodiment. As shown in FIG. [Figure 22] FIG. 22 is a block diagram of a second decoding unit according to the first embodiment. [Diagram 23] FIG. 23 is a diagram showing a protocol stack related to PCC encoded data according to the first embodiment. [Figure 24] FIG. 24 is a diagram illustrating configurations of an encoding unit and a multiplexing unit according to the second embodiment. In FIG. [Diagram 25] FIG. 25 is a diagram illustrating an example of a structure of encoded data according to the second embodiment. In FIG. [Figure 26] FIG. 26 is a diagram showing an example of the structure of coded data and an NAL unit according to the second embodiment. [Figure 27] FIG. 27 is a diagram illustrating an example of the semantics of pcc_nal_unit_type according to the second embodiment. [Figure 28] FIG. 28 is a diagram showing an example of a transmission order of NAL units according to the fourth embodiment. [Figure 29] FIG. 29 is a block diagram of a first encoding unit according to the third embodiment. As shown in FIG. [Diagram 30] FIG. 30 is a block diagram of a first decoding unit according to the third embodiment. [Diagram 31] FIG. 31 is a block diagram of a division unit according to the third embodiment. [Diagram 32] FIG. 32 is a diagram showing an example of division into slices and tiles according to the third embodiment. [Diagram 33] FIG. 33 is a diagram showing an example of a division pattern of slices and tiles according to the third embodiment. [Diagram 34] FIG. 34 is a diagram illustrating an example of a dependency relationship according to the third embodiment. [Diagram 35] FIG. 35 is a diagram showing an example of a data decoding order according to the third embodiment. [Diagram 36] FIG. 36 is a flowchart of the encoding process according to the third embodiment. [Figure 37] FIG. 37 is a block diagram of a coupling unit according to the third embodiment. [Figure 38] FIG. 38 is a diagram showing an example of the structure of coded data and an NAL unit according to the third embodiment. [Figure 39] FIG. 39 is a flowchart of the encoding process according to the third embodiment. [Diagram 40] FIG. 40 is a flowchart of the decoding process according to the third embodiment. [Diagram 41] FIG. 41 is a diagram illustrating an example of the syntax of the tile additional information according to the fourth embodiment. [Diagram 42] FIG. 42 is a block diagram of a coding / decoding system according to the fourth embodiment. [Diagram 43] FIG. 43 is a diagram showing an example of the syntax of slice additional information according to the fourth embodiment. [Diagram 44] FIG. 44 is a flowchart of the encoding process according to the fourth embodiment. [Diagram 45] FIG. 45 is a flowchart of the decoding process according to the fourth embodiment. [Figure 46] FIG. 46 is a diagram showing an example of a tree structure according to the fifth embodiment. [Figure 47] FIG. 47 is a diagram showing an example of a data structure of coded data having an octree structure according to the fifth embodiment. [Figure 48]FIG. 48 is a diagram illustrating an example of the syntax of the payload according to the fifth embodiment. [Figure 49] FIG. 49 is a flowchart of the decoding process according to the fifth embodiment. [Figure 50] FIG. 50 is a diagram showing the relationship between levels and decoded data according to the fifth embodiment. In FIG. [Figure 51] FIG. 51 is a schematic diagram showing levels according to the fifth embodiment. [Figure 52] FIG. 52 is a diagram illustrating an example of the syntax of a header according to the fifth embodiment. [Diagram 53] FIG. 53 is a diagram illustrating an example of the syntax of a payload according to the fifth embodiment. [Figure 54] FIG. 54 is a diagram showing a structure of the entire coded data according to the fifth embodiment. In FIG. [Figure 55] FIG. 55 is a diagram showing a structure of the entire coded data according to the fifth embodiment. In FIG. [Figure 56] FIG. 56 is a diagram illustrating an example of the syntax of the depth information according to the fifth embodiment. [Figure 57] FIG. 57 is a diagram showing an example of the syntax of hierarchical structure metadata according to the fifth embodiment. [Figure 58] FIG. 58 is a diagram illustrating an example of the syntax of a header according to the fifth embodiment. [Figure 59] FIG. 59 is a diagram showing a structure of the entire coded data according to the fifth embodiment. In FIG. [Figure 60] FIG. 60 is a diagram illustrating an example of the syntax of hierarchy information according to the fifth embodiment. [Figure 61] FIG. 61 is a diagram showing an example of the syntax of hierarchical structure metadata according to the fifth embodiment. [Figure 62] FIG. 62 is a diagram illustrating an example of the syntax of the header of the entire coded data according to the fifth embodiment. [Figure 63] FIG. 63 is a diagram showing an example of a configuration of a bit stream according to the fifth embodiment. [Figure 64]FIG. 64 is a diagram showing an example of the syntax of hierarchical structure metadata according to the fifth embodiment. [Figure 65] FIG. 65 is a diagram showing a reference relationship between position information and attribute information according to the fifth embodiment. [Figure 66] FIG. 66 is a diagram showing a reference relationship between position information and attribute information according to the fifth embodiment. In FIG. [Figure 67] FIG. 67 is a diagram showing a reference relationship between position information and attribute information according to the fifth embodiment. [Figure 68] FIG. 68 is a diagram showing an example of a configuration of a bit stream according to the fifth embodiment. [Figure 69] FIG. 69 is a diagram illustrating an example of the configuration of a three-dimensional data encoding device according to the fifth embodiment. [Figure 70] FIG. 70 is a diagram illustrating an example of a configuration of a three-dimensional data decoding device according to the fifth embodiment. [Figure 71] FIG. 71 is a diagram showing a basic structure of an ISOBMFF according to the fifth embodiment. [Figure 72] FIG. 72 is a protocol stack diagram for storing a NAL unit common to the PCC codec in the fifth embodiment in an ISOBMFF. [Figure 73] FIG. 73 is a diagram showing a conversion process from a bit stream to a file format according to the fifth embodiment. [Figure 74] FIG. 74 is a flowchart of a format conversion process according to the fifth embodiment. [Figure 75] FIG. 75 is a flowchart of the decoding process according to the fifth embodiment. [Figure 76] FIG. 76 is a diagram showing a process of converting a bit stream into a file format according to the fifth embodiment. [Figure 77] FIG. 77 is a diagram showing an example of the syntax of hierarchical structure metadata according to the fifth embodiment. [Figure 78] FIG. 78 is a diagram illustrating a division process according to the fifth embodiment. [Figure 79]FIG. 79 is a flowchart of a conversion process using hierarchical information according to the fifth embodiment. [Figure 80] FIG. 80 is a flowchart of the conversion process not using hierarchical information according to the fifth embodiment. [Figure 81] FIG. 81 is a flowchart of a decoding process of hierarchical data sample data according to the fifth embodiment. [Figure 82] FIG. 82 is a diagram showing an example of a structure of the entire coded data according to the fifth embodiment. In FIG. [Figure 83] FIG. 83 is a diagram showing an example of a structure of the entire coded data according to the fifth embodiment. In FIG. [Figure 84] FIG. 84 is a diagram showing a conversion process from a bit stream to a file format according to the fifth embodiment. [Figure 85] FIG. 85 is a flowchart of a format conversion process according to the fifth embodiment. [Figure 86] FIG. 86 is a flowchart of the decoding process according to the fifth embodiment. [Figure 87] FIG. 87 is a diagram illustrating an example of the syntax of the depth information according to the fifth embodiment. [Figure 88] FIG. 88 is a diagram illustrating an example of the syntax of the sample size box according to the fifth embodiment. [Figure 89] FIG. 89 is a diagram illustrating an example of the syntax of hierarchy information according to the fifth embodiment. [Figure 90] FIG. 90 is a diagram illustrating an example of the syntax of PCCLayerStructureBox according to the fifth embodiment. [Figure 91] FIG. 91 is a diagram illustrating an extraction operation according to the fifth embodiment. [Figure 92] FIG. 92 is a diagram showing an example of a file format according to the fifth embodiment. [Figure 93] FIG. 93 is a diagram showing an example of an extracted bitstream according to the fifth embodiment. In FIG. [Figure 94] FIG. 94 is a diagram showing an example of an extracted bitstream according to the fifth embodiment. In FIG. [Figure 95] FIG. 95 is a diagram showing an example of an extracted bitstream according to the fifth embodiment. In FIG. [Figure 96] FIG. 96 is a diagram showing an example of the direct mode according to the fifth embodiment. In FIG. [Figure 97] FIG. 97 is a flowchart of three-dimensional data encoding processing according to the fifth embodiment. [Figure 98] FIG. 98 is a flowchart of three-dimensional data decoding processing according to the fifth embodiment. [Figure 99] FIG. 99 is a block diagram of a three-dimensional data creation device according to the sixth embodiment. [Figure 100] FIG. 100 is a flowchart of a three-dimensional data creating method according to the sixth embodiment. [Figure 101] FIG. 101 is a diagram showing a configuration of a system according to the sixth embodiment. [Figure 102] FIG. 102 is a block diagram of a client device according to the sixth embodiment. [Figure 103] FIG. 103 is a block diagram of a server according to the sixth embodiment. [Figure 104] FIG. 104 is a flowchart of three-dimensional data creation processing by a client device according to the sixth embodiment. [Figure 105] FIG. 105 is a flowchart of a sensor information transmission process by a client device according to the sixth embodiment. [Figure 106] FIG. 106 is a flowchart of three-dimensional data creation processing by the server according to the sixth embodiment. [Figure 107] FIG. 107 is a flowchart of a 3D map transmission process by the server according to the sixth embodiment. [Figure 108] FIG. 108 is a diagram showing a configuration of a modified example of the system according to the sixth embodiment. In FIG. [Fig. 109] FIG. 109 is a diagram showing configurations of a server and a client device according to the sixth embodiment. In FIG. [Figure 110] FIG. 110 is a diagram showing the configurations of a server and a client device according to the sixth embodiment. In FIG. [Figure 111] FIG. 111 is a flowchart of processing by the client device according to the sixth embodiment. [Figure 112] FIG. 112 is a diagram illustrating a configuration of a sensor information collecting system according to the sixth embodiment. As shown in FIG. [Figure 113] FIG. 113 is a diagram illustrating an example of a system according to the sixth embodiment. [Fig. 114] FIG. 114 is a diagram showing a modification of the system according to the sixth embodiment. [Figure 115] FIG. 115 is a flowchart showing an example of application processing according to the sixth embodiment. [Fig. 116] FIG. 116 is a diagram showing the sensor ranges of various sensors according to the sixth embodiment. [Figure 117] FIG. 117 is a diagram illustrating a configuration example of an autonomous driving system according to the sixth embodiment. [Fig. 118] FIG. 118 is a diagram showing an example of a configuration of a bit stream according to the sixth embodiment. [Figure 119] FIG. 119 is a flowchart of the point group selection process according to the sixth embodiment. [Figure 120] FIG. 120 is a diagram showing an example of a screen for the point group selection process according to the sixth embodiment. [Figure 121] FIG. 121 is a diagram showing an example of a screen for the point group selection process according to the sixth embodiment. [Figure 122] FIG. 122 is a diagram showing an example of a screen for the point group selection process according to the sixth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] A three-dimensional data encoding method according to one embodiment of the present disclosure sets a hierarchical structure having multiple depths and multiple hierarchies each including one or more depths for multiple positional information of multiple three-dimensional points included in point cloud data, generates multiple first encoded data for each depth by encoding the multiple positional information for each depth, and generates a bit stream including multiple second encoded data for each hierarchical level, the encoded data for each hierarchical level including one or more first encoded data for one or more depths included in the corresponding hierarchical level, and the bit stream includes first information indicating the data length of each of the multiple second encoded data.

[0014] According to this, a three-dimensional data decoding device that decodes a bit stream can easily access data of an arbitrary layer by using the first information. Therefore, the three-dimensional data encoding method can reduce the amount of processing of the three-dimensional data decoding device.

[0015] For example, the first information may include second information indicating the number of depths included in each of the plurality of hierarchical layers, and third information indicating a data length of each of the plurality of first encoded data.

[0016] For example, the bit stream may include a first header common to the plurality of second encoded data, and the first header may include the first information.

[0017] For example, the bit stream may include a plurality of second headers for each second encoded data, the first information may correspond to any one of the plurality of second encoded data and include a plurality of fourth information indicating a data length of the corresponding second encoded data, and each of the plurality of second headers may include fourth information indicating the data length of the second encoded data corresponding to the second header.

[0018] For example, the bit stream may include a plurality of third headers for each of the plurality of first encoded data, the first information may include second information indicating the number of depths included in each of the plurality of hierarchical layers, and fifth information corresponding to each of the plurality of first encoded data and indicating a data length of the corresponding first encoded data, and each of the plurality of third headers may include fifth information indicating the data length of the first encoded data corresponding to the third header.

[0019] For example, the three-dimensional data encoding method may further generate multiple third encoded data for each depth by encoding multiple attribute information possessed by the multiple three-dimensional points for each depth, and the bit stream may include multiple fourth encoded data, which are encoded data for each layer and include one or more third encoded data for one or more depths included in the corresponding layer, and the bit stream may include sixth information indicating a data length of each of the multiple fourth encoded data.

[0020] A three-dimensional data decoding method according to one embodiment of the present disclosure includes: acquiring the first information from a bit stream including a plurality of second encoded data and first information indicating a data length of each of the plurality of second encoded data; acquiring at least one second encoded data from the plurality of second encoded data using the first information; and decoding the acquired at least one second encoded data, wherein the bit stream includes a plurality of first encoded data for each depth generated by encoding a plurality of positional information for a plurality of three-dimensional points included in point cloud data, the positional information being set in a hierarchical structure having a plurality of depths and a plurality of hierarchies each including one or more depths, each of the plurality of second encoded data corresponding to one of the plurality of hierarchies, and including one or more first encoded data included in the hierarchical layer corresponding to the second encoded data among the plurality of first encoded data.

[0021] According to this, the three-dimensional data decoding method can easily access data at any layer using the first information, and therefore the three-dimensional data decoding method can reduce the amount of processing.

[0022] For example, the first information may include second information indicating the number of depths included in each of the plurality of hierarchical layers, and third information indicating a data length of each of the plurality of first encoded data.

[0023] For example, the bit stream may include a first header common to the plurality of second encoded data, and the first header may include the first information.

[0024] For example, the bit stream may include a plurality of second headers for each second encoded data, the first information may correspond to any one of the plurality of second encoded data and include a plurality of fourth information indicating a data length of the corresponding second encoded data, and each of the plurality of second headers may include fourth information indicating the data length of the second encoded data corresponding to the second header.

[0025] For example, the bit stream may include a plurality of third headers for each of the plurality of first encoded data, the first information may include second information indicating the number of depths included in each of the plurality of hierarchical layers, and fifth information corresponding to each of the plurality of first encoded data and indicating a data length of the corresponding first encoded data, and each of the plurality of third headers may include fifth information indicating the data length of the first encoded data corresponding to the third header.

[0026] For example, the bit stream includes a plurality of third encoded data for each depth generated by encoding a plurality of attribute information possessed by the plurality of three-dimensional points for each depth, the bit stream includes a plurality of fourth encoded data which is encoded data for each layer and includes one or more third encoded data for one or more depths included in a corresponding layer, the bit stream includes sixth information indicating a data length of each of the plurality of fourth encoded data, and the three-dimensional data decoding method may further include acquiring the sixth information from the bit stream, acquiring at least one fourth encoded data among the plurality of fourth encoded data using the sixth information, and decoding the acquired at least one fourth encoded data.

[0027] Moreover, a three-dimensional data encoding device according to one embodiment of the present disclosure includes a processor and a memory, and the processor uses the memory to set a hierarchical structure having multiple depths and multiple hierarchies each including one or more depths for multiple positional information of multiple three-dimensional points included in point cloud data, generates multiple first encoded data for each depth by encoding the multiple positional information for each depth, and generates a bit stream including multiple second encoded data for each hierarchical level, the encoded data for each hierarchical level including one or more first encoded data of one or more depths included in the corresponding hierarchical level, and the bit stream includes first information indicating a data length of each of the multiple second encoded data.

[0028] According to this, a three-dimensional data decoding device that decodes a bit stream can easily access data of an arbitrary layer by using the first information. Therefore, the three-dimensional data encoding device can reduce the amount of processing of the three-dimensional data decoding device.

[0029] In addition, a three-dimensional data decoding device according to one embodiment of the present disclosure includes a processor and a memory, and the processor uses the memory to acquire the first information from a bit stream including a plurality of second encoded data and first information indicating a data length of each of the plurality of second encoded data, and acquires at least one of the plurality of second encoded data using the first information, and decodes the acquired at least one second encoded data, wherein the bit stream includes a plurality of first encoded data for each depth generated by encoding a plurality of positional information of a plurality of three-dimensional points included in point cloud data, the positional information being set in a hierarchical structure having a plurality of depths and a plurality of hierarchies each including one or more depths, and each of the plurality of second encoded data corresponds to one of the plurality of hierarchies, and includes one or more first encoded data included in the hierarchical layer corresponding to the second encoded data among the plurality of first encoded data.

[0030] This allows the three-dimensional data decoding device to easily access data at any layer using the first information, thereby reducing the amount of processing.

[0031] These comprehensive or specific aspects may be realized as a system, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM, or may be realized as any combination of a system, a method, an integrated circuit, a computer program, and a recording medium.

[0032] Hereinafter, the embodiments will be described in detail with reference to the drawings. Note that each of the embodiments described below shows a specific example of the present disclosure. The numerical values, shapes, materials, components, arrangement and connection forms of the components, steps, and order of steps shown in the following embodiments are merely examples and are not intended to limit the present disclosure. In addition, among the components in the following embodiments, components that are not described in the independent claims will be described as optional components.

[0033] (Embodiment 1) When using the encoded data of a point cloud in an actual device or service, it is desirable to transmit and receive the necessary information according to the purpose in order to reduce the network bandwidth. However, until now, such a function has not existed in the encoding structure of three-dimensional data, and there has been no encoding method for this purpose.

[0034] In this embodiment, we describe a three-dimensional data encoding method and a three-dimensional data encoding device for providing the function of transmitting and receiving necessary information depending on the application in encoded data of a three-dimensional point cloud, as well as a three-dimensional data decoding method and a three-dimensional data decoding device for decoding the encoded data, a three-dimensional data multiplexing method for multiplexing the encoded data, and a three-dimensional data transmission method for transmitting the encoded data.

[0035] In particular, the first and second encoding methods are currently being considered as methods (encoding schemes) for encoding point cloud data; however, the structure of the encoded data and the method for storing the encoded data in a system format have not been defined, and as things stand, there is a problem that MUX processing (multiplexing) in the encoding unit, or transmission or storage, is not possible.

[0036] Furthermore, there has been no method to date that supports a format in which two codecs, the first encoding method and the second encoding method, are mixed, such as PCC (Point Cloud Compression).

[0037] In this embodiment, a structure of PCC encoded data in which two codecs, a first encoding method and a second encoding method, are mixed, and a method of storing the encoded data in a system format will be described.

[0038] First, the configuration of a three-dimensional data (point cloud data) encoding / decoding system according to this embodiment will be described. Fig. 1 is a diagram showing an example of the configuration of a three-dimensional data encoding / decoding system according to this embodiment. As shown in Fig. 1, the three-dimensional data encoding / decoding system includes a three-dimensional data encoding system 4601, a three-dimensional data decoding system 4602, a sensor terminal 4603, and an external connection unit 4604.

[0039] The three-dimensional data encoding system 4601 generates encoded data or multiplexed data by encoding point cloud data, which is three-dimensional data. The three-dimensional data encoding system 4601 may be a three-dimensional data encoding device realized by a single device, or may be a system realized by multiple devices. The three-dimensional data encoding device may include some of the multiple processing units included in the three-dimensional data encoding system 4601.

[0040] The three-dimensional data encoding system 4601 includes a point cloud data generation system 4611, a presentation unit 4612, an encoding unit 4613, a multiplexing unit 4614, an input / output unit 4615, and a control unit 4616. The point cloud data generation system 4611 includes a sensor information acquisition unit 4617 and a point cloud data generation unit 4618.

[0041] The sensor information acquisition unit 4617 acquires sensor information from the sensor terminal 4603, and outputs the sensor information to the point cloud data generation unit 4618. The point cloud data generation unit 4618 generates point cloud data from the sensor information, and outputs the point cloud data to the encoding unit 4613.

[0042] The presentation unit 4612 presents the sensor information or the point cloud data to the user. For example, the presentation unit 4612 displays information or an image based on the sensor information or the point cloud data.

[0043] The encoding unit 4613 encodes (compresses) the point cloud data, and outputs the obtained encoded data, control information obtained in the encoding process, and other additional information to the multiplexing unit 4614. The additional information includes, for example, sensor information.

[0044] The multiplexing unit 4614 generates multiplexed data by multiplexing the coded data, control information, and additional information input from the coding unit 4613. The format of the multiplexed data is, for example, a file format for storage, or a packet format for transmission.

[0045] The input / output unit 4615 (e.g., a communication unit or an interface) outputs the multiplexed data to the outside. Alternatively, the multiplexed data is stored in a storage unit such as an internal memory. The control unit 4616 (or an application execution unit) controls each processing unit. In other words, the control unit 4616 controls encoding, multiplexing, etc.

[0046] The sensor information may be input to the encoding unit 4613 or the multiplexing unit 4614. Furthermore, the input / output unit 4615 may output the point cloud data or the encoded data directly to the outside.

[0047] The transmission signal (multiplexed data) output from the three-dimensional data encoding system 4601 is input to the three-dimensional data decoding system 4602 via the external connection unit 4604.

[0048] The three-dimensional data decoding system 4602 generates point cloud data, which is three-dimensional data, by decoding the encoded data or multiplexed data. The three-dimensional data decoding system 4602 may be a three-dimensional data decoding device realized by a single device, or may be a system realized by multiple devices. The three-dimensional data decoding device may also include some of the multiple processing units included in the three-dimensional data decoding system 4602.

[0049] The three-dimensional data decoding system 4602 includes a sensor information acquisition unit 4621 , an input / output unit 4622 , a demultiplexing unit 4623 , a decoding unit 4624 , a presentation unit 4625 , a user interface 4626 , and a control unit 4627 .

[0050] The sensor information acquisition unit 4621 acquires sensor information from the sensor terminal 4603 .

[0051] The input / output unit 4622 acquires a transmission signal, decodes multiplexed data (file format or packets) from the transmission signal, and outputs the multiplexed data to the demultiplexer 4623.

[0052] The demultiplexer 4623 obtains the coded data, control information, and additional information from the multiplexed data, and outputs the coded data, control information, and additional information to the decoder 4624.

[0053] The decoding unit 4624 reconstructs the point cloud data by decoding the encoded data.

[0054] The presentation unit 4625 presents the point cloud data to the user. For example, the presentation unit 4625 displays information or an image based on the point cloud data. The user interface 4626 acquires an instruction based on a user's operation. The control unit 4627 (or the application execution unit) controls each processing unit. That is, the control unit 4627 controls demultiplexing, decoding, presentation, etc.

[0055] The input / output unit 4622 may obtain the point cloud data or the encoded data directly from the outside. The presentation unit 4625 may obtain additional information such as sensor information and present information based on the additional information. The presentation unit 4625 may present information based on a user instruction obtained by the user interface 4626.

[0056] The sensor terminal 4603 generates sensor information obtained by a sensor. The sensor terminal 4603 is a terminal equipped with a sensor or a camera, and examples of the sensor terminal include a moving body such as an automobile, a flying object such as an airplane, a mobile terminal, and a camera.

[0057] The sensor information that can be acquired by the sensor terminal 4603 is, for example, (1) the distance between the sensor terminal 4603 and an object, or the reflectance of the object, obtained from a LIDAR, millimeter wave radar, or an infrared sensor, and (2) the distance between a camera and an object, or the reflectance of the object, obtained from a plurality of monocular camera images or stereo camera images. The sensor information may also include the attitude, direction, gyro (angular velocity), position (GPS information or altitude), speed, acceleration, etc. of the sensor. The sensor information may also include temperature, air pressure, humidity, magnetism, etc.

[0058] The external connection unit 4604 is realized by an integrated circuit (LSI or IC), an external storage unit, communication with a cloud server via the Internet, broadcasting, or the like.

[0059] Next, the point cloud data will be described. Fig. 2 is a diagram showing the configuration of the point cloud data. Fig. 3 is a diagram showing an example of the configuration of a data file in which information on the point cloud data is written.

[0060] Point cloud data includes data on multiple points. The data on each point includes position information (three-dimensional coordinates) and attribute information for that position information. A collection of multiple points is called a point cloud. For example, a point cloud may represent the three-dimensional shape of an object.

[0061] Position information such as three-dimensional coordinates is sometimes called geometry. Data for each point may include attribute information of multiple attribute types. The attribute types may be, for example, color or reflectance.

[0062] One piece of attribute information may be associated with one piece of location information, or multiple pieces of attribute information having different attribute types may be associated with one piece of location information, or multiple pieces of attribute information of the same attribute type may be associated with one piece of location information.

[0063] The configuration example of the data file shown in FIG. 3 is an example in which position information and attribute information correspond one-to-one, and shows the position information and attribute information of N points that make up the point cloud data.

[0064] The position information is, for example, information on three axes, x, y, and z. The attribute information is, for example, RGB color information. A typical data file is a ply file.

[0065] Next, the types of point cloud data will be described. Fig. 4 is a diagram showing the types of point cloud data. As shown in Fig. 4, point cloud data includes static objects and dynamic objects.

[0066] A static object is 3D point cloud data at any time (a certain time). A dynamic object is 3D point cloud data that changes over time. Hereinafter, 3D point cloud data at a certain time will be referred to as a PCC frame, or a frame.

[0067] The object may be a point cloud whose area is restricted to a certain extent, such as ordinary video data, or a large-scale point cloud whose area is not restricted, such as map information.

[0068] Also, there may be point cloud data of various densities, and there may be sparse point cloud data and dense point cloud data.

[0069] The details of each processing unit will be described below. The sensor information is acquired by various methods such as a distance sensor such as a LIDAR or a range finder, a stereo camera, or a combination of multiple monocular cameras. The point cloud data generation unit 4618 generates point cloud data based on the sensor information acquired by the sensor information acquisition unit 4617. The point cloud data generation unit 4618 generates position information as point cloud data, and adds attribute information for the position information to the position information.

[0070] The point cloud data generating unit 4618 may process the point cloud data when generating the position information or adding the attribute information. For example, the point cloud data generating unit 4618 may reduce the amount of data by deleting point clouds with overlapping positions. In addition, the point cloud data generating unit 4618 may convert the position information (position shift, rotation, normalization, etc.) or render the attribute information.

[0071] In FIG. 1, the point cloud data generation system 4611 is included in the three-dimensional data encoding system 4601, but it may be provided independently outside the three-dimensional data encoding system 4601.

[0072] The encoding unit 4613 generates encoded data by encoding the point cloud data based on a predefined encoding method. There are two main types of encoding methods: the first is an encoding method using position information, and this encoding method will be referred to as the first encoding method hereinafter; the second is an encoding method using a video codec, and this encoding method will be referred to as the second encoding method hereinafter.

[0073] The decoding unit 4624 decodes the encoded data based on a predefined encoding method to decode the point group data.

[0074] The multiplexing unit 4614 generates multiplexed data by multiplexing the encoded data using an existing multiplexing method. The generated multiplexed data is transmitted or stored. In addition to the PCC encoded data, the multiplexing unit 4614 multiplexes other media such as video, audio, subtitles, applications, and files, or reference time information. The multiplexing unit 4614 may further multiplex attribute information related to sensor information or point cloud data.

[0075] Multiplexing methods or file formats include ISOBMFF, MPEG-DASH, which is an ISOBMFF-based transmission method, MMT, MPEG-2 TS Systems, and RMP.

[0076] The demultiplexer 4623 extracts the PCC encoded data, other media, time information, and the like from the multiplexed data.

[0077] The input / output unit 4615 transmits the multiplexed data using a method suited to a transmission medium or a storage medium, such as broadcasting or communication. The input / output unit 4615 may communicate with other devices via the Internet, or may communicate with a storage unit such as a cloud server.

[0078] The communication protocol used may be http, ftp, TCP, UDP, etc. A PULL type communication method or a PUSH type communication method may be used.

[0079] Either wired transmission or wireless transmission may be used. For wired transmission, Ethernet (registered trademark), USB, RS-232C, HDMI (registered trademark), coaxial cable, etc. are used. For wireless transmission, wireless LAN, Wi-Fi (registered trademark), Bluetooth (registered trademark), millimeter wave, etc. are used.

[0080] As a broadcasting system, for example, DVB-T2, DVB-S2, DVB-C2, ATSC3.0, or ISDB-S3 is used.

[0081] Fig. 5 is a diagram showing a configuration of a first encoding unit 4630 which is an example of the encoding unit 4613 that performs encoding using the first encoding method. Fig. 6 is a block diagram of the first encoding unit 4630. The first encoding unit 4630 generates encoded data (encoded stream) by encoding point cloud data using the first encoding method. The first encoding unit 4630 includes a position information encoding unit 4631, an attribute information encoding unit 4632, an additional information encoding unit 4633, and a multiplexing unit 4634.

[0082] The first encoding unit 4630 has a feature of performing encoding in consideration of a three-dimensional structure. Also, the first encoding unit 4630 has a feature that the attribute information encoding unit 4632 performs encoding using information obtained from the position information encoding unit 4631. The first encoding method is also called GPCC (Geometry based PCC).

[0083] The point cloud data is PCC point cloud data such as a PLY file, or PCC point cloud data generated from sensor information, and includes position information (Position), attribute information (Attribute), and other additional information (MetaData). The position information is input to a position information encoding unit 4631, the attribute information is input to an attribute information encoding unit 4632, and the additional information is input to an additional information encoding unit 4633.

[0084] The position information encoding unit 4631 generates encoded position information (Compressed Geometry) that is encoded data by encoding the position information. For example, the position information encoding unit 4631 encodes the position information using an N-ary tree structure such as an octet tree. Specifically, in the octet tree, the target space is divided into eight nodes (subspaces), and 8-bit information (occupancy code) indicating whether or not a point cloud is included in each node is generated. In addition, the node that includes the point cloud is further divided into eight nodes, and 8-bit information indicating whether or not a point cloud is included in each of the eight nodes is generated. This process is repeated until the number of point clouds included in a predetermined hierarchy or node is equal to or less than a threshold value.

[0085] The attribute information encoding unit 4632 generates encoded attribute information (Compressed Attribute) that is encoded data by encoding using the configuration information generated by the position information encoding unit 4631. For example, the attribute information encoding unit 4632 determines a reference point (reference node) to be referenced in encoding a target point (target node) to be processed based on the octree structure generated by the position information encoding unit 4631. For example, the attribute information encoding unit 4632 refers to a node, among peripheral nodes or adjacent nodes, whose parent node in the octree is the same as that of the target node. Note that the method of determining the reference relationship is not limited to this.

[0086] Furthermore, the encoding process of the attribute information may include at least one of a quantization process, a prediction process, and an arithmetic coding process. In this case, the reference means using a reference node to calculate a predicted value of the attribute information, or using a state of the reference node (e.g., occupancy information indicating whether or not a point group is included in the reference node) to determine a parameter of the encoding. For example, the parameter of the encoding is a quantization parameter in the quantization process, or a context in the arithmetic coding.

[0087] The additional information encoding unit 4633 generates encoded additional information (Compressed MetaData) that is encoded data by encoding compressible data from the additional information.

[0088] The multiplexing unit 4634 multiplexes the encoding position information, the encoding attribute information, the encoding additional information, and other additional information to generate an encoded stream (Compressed Stream) that is encoded data. The generated encoded stream is output to a processing unit of a system layer (not shown).

[0089] Next, a first decoding unit 4640 which is an example of the decoding unit 4624 that performs decoding of the first encoding method will be described. FIG. 7 is a diagram showing a configuration of the first decoding unit 4640. FIG. 8 is a block diagram of the first decoding unit 4640. The first decoding unit 4640 generates point group data by decoding, by the first encoding method, encoded data (encoded stream) encoded by the first encoding method. The first decoding unit 4640 includes a demultiplexing unit 4641, a position information decoding unit 4642, an attribute information decoding unit 4643, and an additional information decoding unit 4644.

[0090] A coded stream (compressed stream), which is coded data, is input to the first decoding unit 4640 from a processing unit in a system layer (not shown).

[0091] The demultiplexer 4641 separates the encoded position information (Compressed Geometry), the encoded attribute information (Compressed Attribute), the encoded additional information (Compressed MetaData), and other additional information from the encoded data.

[0092] The position information decoding unit 4642 generates position information by decoding the encoded position information. For example, the position information decoding unit 4642 restores the position information of a point group represented by three-dimensional coordinates from the encoded position information represented by an N-ary tree structure such as an octree.

[0093] The attribute information decoding unit 4643 decodes the encoded attribute information based on the configuration information generated by the position information decoding unit 4642. For example, the attribute information decoding unit 4643 determines a reference point (reference node) to be referenced in decoding a target point (target node) to be processed based on the octree structure obtained by the position information decoding unit 4642. For example, the attribute information decoding unit 4643 refers to a node, among peripheral nodes or adjacent nodes, whose parent node in the octree is the same as that of the target node. Note that the method of determining the reference relationship is not limited to this.

[0094] In addition, the decoding process of the attribute information may include at least one of an inverse quantization process, a prediction process, and an arithmetic decoding process. In this case, the reference means using a reference node to calculate a predicted value of the attribute information, or using a state of the reference node (e.g., occupancy information indicating whether or not a point group is included in the reference node) to determine a parameter of the decoding. For example, the parameter of the decoding is a quantization parameter in the inverse quantization process, or a context in the arithmetic decoding.

[0095] The additional information decoding unit 4644 generates additional information by decoding the encoded additional information. The first decoding unit 4640 uses the additional information required for decoding the position information and attribute information during decoding, and outputs the additional information required for an application to the outside.

[0096] Next, a configuration example of the position information encoding unit will be described. Fig. 9 is a block diagram of the position information encoding unit 2700 according to this embodiment. The position information encoding unit 2700 includes an octree generating unit 2701, a geometric information calculating unit 2702, a coding table selecting unit 2703, and an entropy encoding unit 2704.

[0097] The octree generating unit 2701 generates, for example, an octree from the input position information, and generates an occupancy code of each node of the octree. The geometric information calculating unit 2702 obtains information indicating whether an adjacent node of the target node is an occupied node. For example, the geometric information calculating unit 2702 calculates occupancy information of the adjacent node (information indicating whether the adjacent node is an occupied node) from the occupancy code of the parent node to which the target node belongs. The geometric information calculating unit 2702 may also store encoded nodes in a list and search for adjacent nodes from the list. The geometric information calculating unit 2702 may also switch adjacent nodes depending on the position of the target node in the parent node.

[0098] The coding table selection unit 2703 selects a coding table to be used for entropy coding of the target node, using the occupancy information of the adjacent nodes calculated by the geometric information calculation unit 2702. For example, the coding table selection unit 2703 may generate a bit string using the occupancy information of the adjacent nodes, and select a coding table for an index number generated from the bit string.

[0099] The entropy coding unit 2704 generates the coding position information and metadata by performing entropy coding on the occupancy code of the target node using the coding table of the selected index number. The entropy coding unit 2704 may add information indicating the selected coding table to the coding position information.

[0100] The octree representation and the scanning order of position information will be described below. Position information (position data) is converted (octreeized) into an octree structure and then encoded. The octree structure is composed of nodes and leaves. Each node has eight nodes or leaves, and each leaf has voxel (VXL) information. Fig. 10 is a diagram showing an example of the structure of position information including multiple voxels. Fig. 11 is a diagram showing an example of the position information shown in Fig. 10 converted into an octree structure. Here, among the leaves shown in Fig. 11, leaves 1, 2, and 3 respectively represent voxels VXL1, VXL2, and VXL3 shown in Fig. 10, and represent a VXL including a point cloud (hereinafter, effective VXL).

[0101] Specifically, node 1 corresponds to the whole space including the position information in FIG. 10. The whole space corresponding to node 1 is divided into eight nodes, and among the eight nodes, the node including the valid VXL is further divided into eight nodes or leaves, and this process is repeated for the number of levels of the tree structure. Here, each node corresponds to a subspace, and has information (occupancy code) indicating at what position the next node or leaf will be located after division as node information. Also, the block at the bottom level is set as a leaf, and the number of point clouds included in the leaf, etc., is held as leaf information.

[0102] Next, a configuration example of the position information decoding unit will be described. Fig. 12 is a block diagram of the position information decoding unit 2710 according to this embodiment. The position information decoding unit 2710 includes an octree generating unit 2711, a geometric information calculating unit 2712, a coding table selecting unit 2713, and an entropy decoding unit 2714.

[0103] The octree generating unit 2711 generates an octree of a certain space (node) using header information or metadata of a bit stream. For example, the octree generating unit 2711 generates a large space (root node) using the size of the x-axis, y-axis, and z-axis directions of a certain space added to the header information, and generates an octree by dividing the space into two in the x-axis, y-axis, and z-axis directions to generate eight small spaces A (nodes A0 to A7). In addition, nodes A0 to A7 are set in order as target nodes.

[0104] The geometric information calculation unit 2712 obtains occupancy information indicating whether an adjacent node of the target node is an occupancy node. For example, the geometric information calculation unit 2712 calculates the occupancy information of the adjacent node from the occupancy code of the parent node to which the target node belongs. The geometric information calculation unit 2712 may also store decoded nodes in a list and search for adjacent nodes from the list. The geometric information calculation unit 2712 may also switch adjacent nodes depending on the position of the target node in the parent node.

[0105] The coding table selection unit 2713 selects a coding table (decoding table) to be used for entropy decoding of the target node using the occupancy information of the adjacent node calculated by the geometric information calculation unit 2712. For example, the coding table selection unit 2713 may generate a bit string using the occupancy information of the adjacent node, and select a coding table for an index number generated from the bit string.

[0106] The entropy decoding unit 2714 generates position information by entropy decoding the occupancy code of the target node using the selected coding table. Note that the entropy decoding unit 2714 may obtain information on the selected coding table by decoding from the bit stream, and entropy decode the occupancy code of the target node using the coding table indicated by the information.

[0107] The configurations of the attribute information encoding unit and the attribute information decoding unit will be described below. Fig. 13 is a block diagram showing an example of the configuration of the attribute information encoding unit A100. The attribute information encoding unit may include a plurality of encoding units that execute different encoding methods. For example, the attribute information encoding unit may switch between the following two methods depending on the use case.

[0108] The attribute information encoding unit A100 includes an LoD attribute information encoding unit A101 and a conversion attribute information encoding unit A102. The LoD attribute information encoding unit A101 classifies each 3D point into a plurality of layers using the position information of the 3D point, predicts the attribute information of the 3D point belonging to each layer, and encodes the prediction residual. Here, each classified layer is called LoD (Level of Detail).

[0109] The transformed attribute information encoding unit A102 encodes the attribute information using RAHT (Region Adaptive Hierarchical Transform). Specifically, the transformed attribute information encoding unit A102 applies RAHT or Haar transform to each piece of attribute information based on the position information of the three-dimensional point to generate high-frequency components and low-frequency components of each layer, and encodes the values ​​using quantization, entropy coding, or the like.

[0110] 14 is a block diagram showing a configuration example of the attribute information decoding unit A110. The attribute information decoding unit may include multiple decoding units that execute different decoding methods. For example, the attribute information decoding unit may switch between the following two methods for decoding based on information included in the header or metadata.

[0111] The attribute information decoding unit A110 includes a LoD attribute information decoding unit A111 and a converted attribute information decoding unit A112. The LoD attribute information decoding unit A111 classifies each 3D point into a plurality of hierarchical levels using the position information of the 3D point, and decodes the attribute value while predicting the attribute information of the 3D point belonging to each hierarchical level.

[0112] The transformed attribute information decoding unit A112 decodes the attribute information using RAHT (Region Adaptive Hierarchical Transform). Specifically, the transformed attribute information decoding unit A112 decodes the attribute value by applying inverse RAHT or inverse Haar transform to high-frequency components and low-frequency components of each attribute value based on the position information of the three-dimensional point.

[0113] FIG. 15 is a block diagram showing a configuration of an attribute information encoding unit 3140 which is an example of the LoD attribute information encoding unit A101.

[0114] The attribute information encoding unit 3140 includes a LoD generation unit 3141, a surrounding search unit 3142, a prediction unit 3143, a prediction residual calculation unit 3144, a quantization unit 3145, an arithmetic encoding unit 3146, an inverse quantization unit 3147, a decoded value generation unit 3148, and a memory 3149.

[0115] The LoD generation unit 3141 generates LoD using the position information of the three-dimensional points.

[0116] The surrounding search unit 3142 searches for nearby 3D points adjacent to each 3D point, using the LoD generation result by the LoD generation unit 3141 and distance information indicating the distance between each 3D point.

[0117] The prediction unit 3143 generates a predicted value of the attribute information of the target 3D point to be encoded.

[0118] The prediction residual calculation unit 3144 calculates (generates) a prediction residual of the predicted value of the attribute information generated by the prediction unit 3143.

[0119] The quantization unit 3145 quantizes the prediction residual of the attribute information calculated by the prediction residual calculation unit 3144 .

[0120] The arithmetic coding unit 3146 arithmetically codes the prediction residuals after being quantized by the quantization unit 3145. The arithmetic coding unit 3146 outputs a bit stream including the arithmetically coded prediction residuals to, for example, a three-dimensional data decoding device.

[0121] Note that the prediction residual may be binarized by, for example, the quantization unit 3145 before being arithmetically coded by the arithmetic coding unit 3146.

[0122] Also, for example, the arithmetic coding unit 3146 may initialize a coding table used for arithmetic coding before arithmetic coding. The arithmetic coding unit 3146 may initialize a coding table used for arithmetic coding for each layer. Also, the arithmetic coding unit 3146 may output information indicating the position of the layer for which the coding table has been initialized, by including it in the bitstream.

[0123] The inverse quantization unit 3147 inverse quantizes the prediction residual after being quantized by the quantization unit 3145 .

[0124] The decoded value generation unit 3148 generates a decoded value by adding the predicted value of the attribute information generated by the prediction unit 3143 and the prediction residual after inverse quantization by the inverse quantization unit 3147.

[0125] The memory 3149 is a memory that stores the decoded values ​​of the attribute information of each 3D point decoded by the decoded value generation unit 3148. For example, when generating a predicted value of a 3D point that has not yet been encoded, the prediction unit 3143 generates the predicted value by using the decoded value of the attribute information of each 3D point stored in the memory 3149.

[0126] 16 is a block diagram of an attribute information encoding unit 6600 which is an example of the transformed attribute information encoding unit A102. The attribute information encoding unit 6600 includes a sorting unit 6601, a Haar transform unit 6602, a quantization unit 6603, an inverse quantization unit 6604, an inverse Haar transform unit 6605, a memory 6606, and an arithmetic encoding unit 6607.

[0127] The sorting unit 6601 generates a Morton code using the position information of the 3D points, and sorts the multiple 3D points in Morton code order. The Haar transform unit 6602 generates coding coefficients by applying a Haar transform to the attribute information. The quantization unit 6603 quantizes the coding coefficients of the attribute information.

[0128] The inverse quantization unit 6604 inversely quantizes the quantized coding coefficients. The inverse Haar transform unit 6605 applies an inverse Haar transform to the coding coefficients. The memory 6606 stores values ​​of attribute information of multiple decoded 3D points. For example, the attribute information of the decoded 3D points stored in the memory 6606 may be used for predicting uncoded 3D points.

[0129] The arithmetic coding unit 6607 calculates ZeroCnt from the coding coefficients after quantization, and arithmetically codes the ZeroCnt. Furthermore, the arithmetic coding unit 6607 arithmetically codes the non-zero coding coefficients after quantization. The arithmetic coding unit 6607 may binarize the coding coefficients before arithmetic coding. Furthermore, the arithmetic coding unit 6607 may generate and code various header information.

[0130] FIG. 17 is a block diagram showing a configuration of an attribute information decoding unit 3150 which is an example of the LoD attribute information decoding unit A111.

[0131] The attribute information decoding unit 3150 includes an LoD generation unit 3151 , a surrounding search unit 3152 , a prediction unit 3153 , an arithmetic decoding unit 3154 , an inverse quantization unit 3155 , a decoded value generation unit 3156 , and a memory 3157 .

[0132] The LoD generation unit 3151 generates LoD using position information of the 3D points decoded by a position information decoding unit (not shown in FIG. 17).

[0133] The surrounding search unit 3152 searches for nearby 3D points adjacent to each 3D point, using the LoD generation result by the LoD generation unit 3151 and distance information indicating the distance between each 3D point.

[0134] The prediction unit 3153 generates a predicted value of the attribute information of the target 3D point to be decoded.

[0135] The arithmetic decoding unit 3154 arithmetically decodes the prediction residual in the bitstream acquired from the attribute information encoding unit 3140 shown in FIG. 15. The arithmetic decoding unit 3154 may initialize a decoding table used for arithmetic decoding. The arithmetic decoding unit 3154 initializes a decoding table used for arithmetic decoding for a layer on which the arithmetic encoding unit 3146 shown in FIG. 15 has performed an encoding process. The arithmetic decoding unit 3154 may initialize a decoding table used for arithmetic decoding for each layer. The arithmetic decoding unit 3154 may initialize the decoding table based on information included in the bitstream and indicating the position of the layer for which the encoding table has been initialized.

[0136] The inverse quantization unit 3155 inverse quantizes the prediction residual arithmetically decoded by the arithmetic decoding unit 3154 .

[0137] The decoded value generation unit 3156 generates a decoded value by adding the predicted value generated by the prediction unit 3153 and the prediction residual after inverse quantization by the inverse quantization unit 3155. The decoded value generation unit 3156 outputs the decoded attribute information data to another device.

[0138] The memory 3157 is a memory that stores the decoded values ​​of the attribute information of each 3D point decoded by the decoded value generation unit 3156. For example, when generating a predicted value of a 3D point that has not yet been decoded, the prediction unit 3153 generates the predicted value by using the decoded value of the attribute information of each 3D point stored in the memory 3157.

[0139] 18 is a block diagram of an attribute information decoding unit 6610 which is an example of the transformed attribute information decoding unit A112. The attribute information decoding unit 6610 includes an arithmetic decoding unit 6611, an inverse quantization unit 6612, an inverse Haar transform unit 6613, and a memory 6614.

[0140] The arithmetic decoding unit 6611 arithmetically decodes the ZeroCnt and the coding coefficients included in the bit stream. Note that the arithmetic decoding unit 6611 may also decode various types of header information.

[0141] The inverse quantization unit 6612 inverse quantizes the arithmetically decoded coding coefficients. The inverse Haar transform unit 6613 applies an inverse Haar transform to the coding coefficients after the inverse quantization. The memory 6614 stores values ​​of attribute information of multiple decoded 3D points. For example, the attribute information of the decoded 3D points stored in the memory 6614 may be used to predict undecoded 3D points.

[0142] Next, a second encoding unit 4650, which is an example of the encoding unit 4613 that performs encoding by the second encoding method, will be described. Fig. 19 is a diagram showing a configuration of the second encoding unit 4650. Fig. 20 is a block diagram of the second encoding unit 4650.

[0143] The second encoding unit 4650 generates encoded data (encoded stream) by encoding the point cloud data by a second encoding method. The second encoding unit 4650 includes an additional information generating unit 4651, a position image generating unit 4652, an attribute image generating unit 4653, a video encoding unit 4654, an additional information encoding unit 4655, and a multiplexing unit 4656.

[0144] The second encoding unit 4650 has a feature of generating a position image and an attribute image by projecting a three-dimensional structure onto a two-dimensional image, and encoding the generated position image and attribute image using an existing video encoding method. The second encoding method is also called VPCC (Video based PCC).

[0145] The point cloud data is PCC point cloud data such as a PLY file, or PCC point cloud data generated from sensor information, and includes position information (Position), attribute information (Attribute), and other additional information (MetaData).

[0146] The additional information generating unit 4651 generates map information of a plurality of two-dimensional images by projecting a three-dimensional structure onto the two-dimensional images.

[0147] The position image generating unit 4652 generates a position image (Geometry Image) based on the position information and the map information generated by the additional information generating unit 4651. This position image is, for example, a distance image in which distance (Depth) is indicated as a pixel value. Note that this distance image may be an image in which a plurality of point clouds are viewed from one viewpoint (an image in which a plurality of point clouds are projected onto one two-dimensional plane), a plurality of images in which a plurality of point clouds are viewed from multiple viewpoints, or a single image in which these multiple images are integrated.

[0148] The attribute image generating unit 4653 generates an attribute image based on the attribute information and the map information generated by the additional information generating unit 4651. This attribute image is, for example, an image in which the attribute information (for example, color (RGB)) is indicated as pixel values. Note that this image may be an image in which multiple point clouds are viewed from one viewpoint (an image in which multiple point clouds are projected onto one two-dimensional plane), or multiple images in which multiple point clouds are viewed from multiple viewpoints, or a single image in which these multiple images are integrated.

[0149] The video encoding unit 4654 generates an encoded position image (Compressed Geometry Image) and an encoded attribute image (Compressed Attribute Image), which are encoded data, by encoding the position image and the attribute image using a video encoding method. Note that any known encoding method may be used as the video encoding method. For example, the video encoding method is AVC, HEVC, or the like.

[0150] The additional information encoding unit 4655 generates encoded additional information (Compressed MetaData) by encoding the additional information and map information included in the point cloud data.

[0151] The multiplexing unit 4656 generates a compressed stream, which is encoded data, by multiplexing the encoded position image, the encoded attribute image, the encoded additional information, and other additional information. The generated compressed stream is output to a processing unit of a system layer (not shown).

[0152] Next, a second decoding unit 4660 which is an example of the decoding unit 4624 that performs decoding of the second encoding method will be described. FIG. 21 is a diagram showing a configuration of the second decoding unit 4660. FIG. 22 is a block diagram of the second decoding unit 4660. The second decoding unit 4660 generates point group data by decoding, by the second encoding method, encoded data (encoded stream) encoded by the second encoding method. The second decoding unit 4660 includes a demultiplexing unit 4661, a video decoding unit 4662, an additional information decoding unit 4663, a position information generating unit 4664, and an attribute information generating unit 4665.

[0153] A compressed stream, which is encoded data, is input to the second decoding unit 4660 from a processing unit in a system layer (not shown).

[0154] The demultiplexer 4661 separates the encoded position image (Compressed Geometry Image), the encoded attribute image (Compressed Attribute Image), the encoded additional information (Compressed MetaData), and other additional information from the encoded data.

[0155] The video decoding unit 4662 generates a position image and an attribute image by decoding the encoded position image and the encoded attribute image using a video encoding method. Note that any known encoding method may be used as the video encoding method. For example, the video encoding method is AVC or HEVC.

[0156] The additional information decoding unit 4663 decodes the encoded additional information to generate additional information including map information and the like.

[0157] The position information generating unit 4664 generates position information using the position image and map information. The attribute information generating unit 4665 generates attribute information using the attribute image and map information.

[0158] The second decoding unit 4660 uses the additional information necessary for decoding during decoding, and outputs the additional information necessary for an application to the outside.

[0159] Problems with the PCC encoding method will be described below. Fig. 23 is a diagram showing a protocol stack related to PCC encoded data. Fig. 23 shows an example in which other media data such as video (e.g., HEVC) or audio is multiplexed with PCC encoded data and transmitted or stored.

[0160] The multiplexing method and file format have the function of multiplexing various encoded data and transmitting or storing them. In order to transmit or store the encoded data, the encoded data must be converted into the format of the multiplexing method. For example, HEVC specifies a technology for storing encoded data in a data structure called a NAL unit and storing the NAL unit in ISOBMFF.

[0161] Meanwhile, a first encoding method (Codec1) and a second encoding method (Codec2) are currently being considered as methods for encoding point cloud data. However, the structure of the encoded data and the method for storing the encoded data in a system format have not been defined, which poses the problem that MUX processing (multiplexing) in the encoding unit, transmission, and storage cannot be performed as is.

[0162] In the following description, unless a specific encoding method is specified, it refers to either the first encoding method or the second encoding method.

[0163] (Embodiment 2) In this embodiment, the types of encoded data (position information (Geometry), attribute information (Attribute), additional information (Metadata)) generated by the above-mentioned first encoding unit 4630 or second encoding unit 4650, a generation method of the additional information (metadata), and multiplexing processing in the multiplexing unit will be described. Note that the additional information (metadata) may also be referred to as a parameter set or control information.

[0164] In this embodiment, the dynamic object (three-dimensional point cloud data that changes over time) described in Figure 4 will be used as an example, but a similar method may also be used in the case of a static object (three-dimensional point cloud data at any time).

[0165] 24 is a diagram showing configurations of an encoding unit 4801 and a multiplexing unit 4802 included in the three-dimensional data encoding device according to this embodiment. The encoding unit 4801 corresponds to, for example, the first encoding unit 4630 or the second encoding unit 4650 described above. The multiplexing unit 4802 corresponds to the multiplexing unit 4634 or 4656 described above.

[0166] The encoding unit 4801 encodes point cloud data of multiple PCC (Point Cloud Compression) frames, and generates encoded data (Multiple Compressed Data) of multiple pieces of position information, attribute information, and additional information.

[0167] The multiplexing unit 4802 converts data of multiple data types (position information, attribute information, and additional information) into NAL units, thereby converting the data into a data structure that takes into consideration data access in the decoding device.

[0168] 25 is a diagram showing an example of the structure of coded data generated by the coding unit 4801. The arrows in the diagram indicate dependencies related to the decoding of coded data, with the source of the arrow depending on the data at the tip of the arrow. In other words, the decoding device decodes the data at the tip of the arrow, and uses the decoded data to decode the data at the tip of the arrow. In other words, dependency means that the data at the dependency destination is referenced (used) in the processing (encoding, decoding, etc.) of the data at the dependency destination.

[0169] First, the process of generating encoded data of position information will be described. The encoding unit 4801 generates encoded position data (compressed geometry data) for each frame by encoding the position information of each frame. The encoded position data is represented as G(i), where i indicates the frame number, the time of the frame, etc.

[0170] The encoding unit 4801 also generates a position parameter set (GPS(i)) corresponding to each frame. The position parameter set includes parameters that can be used to decode the encoded position data. The encoded position data for each frame depends on the corresponding position parameter set.

[0171] Moreover, encoded position data consisting of multiple frames is defined as a position sequence (Geometry Sequence). The encoding unit 4801 generates a position sequence parameter set (Geometry Sequence PS: also written as position SPS) that stores parameters commonly used in decoding processes for multiple frames in the position sequence. The position sequence depends on the position SPS.

[0172] Next, a process for generating encoded data of attribute information will be described. Encoding unit 4801 generates encoded attribute data (Compressed Attribute Data) for each frame by encoding the attribute information of each frame. The encoded attribute data is represented as A(i). FIG. 25 shows an example in which attribute X and attribute Y exist, and the encoded attribute data of attribute X is represented as AX(i) and the encoded attribute data of attribute Y is represented as AY(i).

[0173] The encoding unit 4801 also generates an attribute parameter set (APS(i)) corresponding to each frame. The attribute parameter set for attribute X is represented as AXPS(i), and the attribute parameter set for attribute Y is represented as AYPS(i). The attribute parameter set includes parameters that can be used to decode the encoded attribute information. The encoded attribute data depends on the corresponding attribute parameter set.

[0174] Moreover, encoded attribute data consisting of multiple frames is defined as an attribute sequence. The encoding unit 4801 generates an attribute sequence parameter set (Attribute Sequence PS: also referred to as attribute SPS) that stores parameters commonly used in decoding processes for multiple frames in the attribute sequence. The attribute sequence depends on the attribute SPS.

[0175] Furthermore, in the first encoding method, the encoded attribute data depends on the encoded position data.

[0176] 25 shows an example in which two types of attribute information (attribute X and attribute Y) exist. When there are two types of attribute information, for example, two encoding units generate respective data and metadata. Also, for example, an attribute sequence is defined for each type of attribute information, and an attribute SPS is generated for each type of attribute information.

[0177] In addition, while FIG. 25 shows an example in which there is one type of position information and two types of attribute information, the present invention is not limited to this, and there may be one type of attribute information, or three or more types. In this case, the encoded data can be generated in a similar manner. In addition, in the case of point cloud data that does not have attribute information, the attribute information may not be necessary. In this case, the encoding unit 4801 does not need to generate a parameter set related to the attribute information.

[0178] Next, a process of generating additional information (metadata) will be described. The encoding unit 4801 generates a PCC stream PS (also written as stream PS), which is a parameter set for the entire PCC stream. The encoding unit 4801 stores parameters that can be commonly used in decoding processes for one or more position sequences and one or more attribute sequences in the stream PS. For example, the stream PS includes identification information indicating the codec of the point cloud data, information indicating the algorithm used for encoding, and the like. The position sequence and the attribute sequence depend on the stream PS.

[0179] Next, an access unit and a GOF will be described. In this embodiment, the concept of an access unit (AU) and a group of frames (GOF) are newly introduced.

[0180] An access unit is a basic unit for accessing data during decoding, and is composed of one or more pieces of data and one or more pieces of metadata. For example, an access unit is composed of position information at the same time and one or more pieces of attribute information. A GOF is a random access unit, and is composed of one or more access units.

[0181] The encoding unit 4801 generates an access unit header (AU Header) as identification information indicating the beginning of an access unit. The encoding unit 4801 stores parameters related to the access unit in the access unit header. For example, the access unit header includes the configuration or information of the encoded data included in the access unit. The access unit header also includes parameters commonly used for data included in the access unit, such as parameters related to the decoding of the encoded data.

[0182] Alternatively, instead of an access unit header, the encoding unit 4801 may generate an access unit delimiter that does not include parameters related to the access unit. This access unit delimiter is used as identification information indicating the beginning of the access unit. The decoding device identifies the beginning of the access unit by detecting the access unit header or the access unit delimiter.

[0183] Next, generation of identification information for the start of a GOF will be described. The encoding unit 4801 generates a GOF header as identification information indicating the start of a GOF. The encoding unit 4801 stores parameters related to the GOF in the GOF header. For example, the GOF header includes the configuration or information of the encoded data included in the GOF. The GOF header also includes parameters commonly used for data included in the GOF, such as parameters related to the decoding of the encoded data.

[0184] Instead of a GOF header, the encoding unit 4801 may generate a GOF delimiter that does not include parameters related to the GOF. This GOF delimiter is used as identification information indicating the beginning of the GOF. The decoding device identifies the beginning of the GOF by detecting the GOF header or the GOF delimiter.

[0185] In PCC encoded data, for example, an access unit is defined as a PCC frame unit, and a decoding device accesses a PCC frame based on identification information at the beginning of the access unit.

[0186] Also, for example, GOF is defined as one random access unit. The decoding device accesses the random access unit based on the identification information at the beginning of the GOF. For example, if PCC frames are not dependent on each other and can be decoded independently, the PCC frames may be defined as the random access unit.

[0187] Note that two or more PCC frames may be assigned to one access unit, and multiple random access units may be assigned to one GOF.

[0188] Furthermore, the encoding unit 4801 may define and generate parameter sets or metadata other than those described above. For example, the encoding unit 4801 may generate SEI (Supplemental Enhancement Information) that stores parameters (optional parameters) that may not necessarily be used during decoding.

[0189] Next, the structure of the coded data and the method of storing the coded data in the NAL unit will be described.

[0190] For example, a data format is defined for each type of encoded data. Figure 26 shows an example of encoded data and NAL units.

[0191] For example, as shown in Fig. 26, the encoded data includes a header and a payload. The encoded data may include length information indicating the length (amount of data) of the encoded data, the header, or the payload. The encoded data may not include a header.

[0192] The header includes, for example, identification information for identifying the data, which indicates, for example, the data type or the frame number.

[0193] The header includes, for example, identification information indicating a reference relationship. This identification information is stored in the header when, for example, there is a dependency between data, and is information for referring to the reference destination from the reference source. For example, the header of the reference destination includes identification information for identifying the data. The header of the reference source includes identification information indicating the reference destination.

[0194] In addition, when the reference destination or the reference source can be identified or derived from other information, the identification information for specifying the data or the identification information indicating the reference relationship may be omitted.

[0195] The multiplexing unit 4802 stores the encoded data in the payload of the NAL unit. The NAL unit header includes pcc_nal_unit_type, which is identification information of the encoded data. Figure 27 shows an example of the semantics of pcc_nal_unit_type.

[0196] 27, when pcc_codec_type is codec 1 (Codec1: first encoding method), values ​​0 to 10 of pcc_nal_unit_type are assigned to the encoded position data (Geometry), encoded attribute X data (AttributeX), encoded attribute Y data (AttributeY), position PS (Geom.PS), attribute XPS (AttrX.PS), attribute YPS (AttrX.PS), position SPS (Geometry Sequence PS), attribute XSPS (AttributeX Sequence PS), attribute YSPS (AttributeY Sequence PS), AU header (AU Header), and GOF header (GOF Header) in codec 1. Values ​​11 and above are assigned as spares for codec 1.

[0197] When pcc_codec_type is Codec2 (Codec2: second encoding method), values ​​0 to 2 of pcc_nal_unit_type are assigned to codec data A (DataA), metadata A (MetaDataA), and metadata B (MetaDataB). Values ​​3 and above are assigned as spares for Codec2.

[0198] Next, the data transmission order will be described. The following describes the constraints on the transmission order of NAL units.

[0199] The multiplexing unit 4802 collectively transmits NAL units in units of GOF or AU. The multiplexing unit 4802 places a GOF header at the beginning of a GOF, and places an AU header at the beginning of an AU.

[0200] The multiplexing unit 4802 may allocate a sequence parameter set (SPS) for each AU so that the decoding device can decode from the next AU even if data is lost due to packet loss or the like.

[0201] If there is a dependency relationship related to decoding in the encoded data, the decoding device decodes the reference data after decoding the reference data. In order to enable the decoding device to decode the data in the order in which it was received without rearranging the data, the multiplexing unit 4802 sends out the reference data first.

[0202] Fig. 28 is a diagram showing examples of the transmission order of NAL units, showing three examples: position information priority, parameter priority, and data integration.

[0203] The position information priority transmission order is an example in which information related to position information and information related to attribute information are transmitted together. In this transmission order, the transmission of information related to position information is completed earlier than the transmission of information related to attribute information.

[0204] For example, by using this transmission order, a decoding device that does not decode attribute information may be able to set a time during which it does not process the attribute information by ignoring the decoding of the attribute information. Also, for example, in the case of a decoding device that wishes to decode position information quickly, it may be able to decode the position information more quickly by obtaining the encoded data of the position information quickly.

[0205] In FIG. 28, the attributes XSPS and YSPS are integrated and written as the attribute SPS, but the attributes XSPS and YSPS may be arranged separately.

[0206] In a parameter set priority sending order, the parameter sets are sent first and the data is sent later.

[0207] As described above, as long as the constraints on the NAL unit transmission order are met, the multiplexing unit 4802 may transmit the NAL units in any order. For example, order identification information may be defined, and the multiplexing unit 4802 may have a function for transmitting the NAL units in a number of different orders. For example, the NAL unit order identification information may be stored in the stream PS.

[0208] The three-dimensional data decoding device may perform decoding based on the order identification information. The three-dimensional data decoding device may instruct the three-dimensional data encoding device of a desired transmission order, and the three-dimensional data encoding device (multiplexing unit 4802) may control the transmission order according to the instructed transmission order.

[0209] The multiplexing unit 4802 may generate coded data in which multiple functions are merged, as long as the data is transmitted in a certain order, such as in the transmission order of the integrated data. For example, as shown in FIG. 28, the GOF header and the AU header may be integrated, or the AXPS and the AYPS may be integrated. In this case, an identifier indicating that the data has multiple functions is defined in the pcc_nal_unit_type.

[0210] A modified example of this embodiment will be described below. If the PS has levels, such as a frame-level PS, a sequence-level PS, and a PCC sequence-level PS, and the PCC sequence level is the higher level and the frame level is the lower level, the following method may be used to store parameters.

[0211] The default PS value is indicated in the higher PS. Also, if the value of the lower PS differs from the value of the higher PS, the PS value is indicated in the lower PS. Or, the PS value is not written in the higher PS, and is written in the lower PS. Or, information on whether the PS value is to be written in the lower PS, the higher PS, or both is indicated in either or both of the lower PS and the higher PS. Or, the lower PS may be merged into the higher PS. Or, if the lower PS and the higher PS overlap, the multiplexing unit 4802 may omit sending one of them.

[0212] The encoding unit 4801 or the multiplexing unit 4802 may divide data into slices or tiles, and transmit the divided data. The divided data includes information for identifying the divided data, and parameters used for decoding the divided data are included in the parameter set. In this case, an identifier indicating that the data stores data or parameters related to tiles or slices is defined in pcc_nal_unit_type.

[0213] (Embodiment 3) HEVC encoding has data division tools such as slicing or tiling to enable parallel processing in the decoding device, but PCC (Point Cloud Compression) encoding does not yet have such tools.

[0214] In PCC, various data division methods can be considered depending on parallel processing, compression efficiency, and compression algorithms. This section describes the definitions of slices and tiles, the data structure, and the transmission and reception methods.

[0215] 29 is a block diagram showing a configuration of a first encoding unit 4910 included in the three-dimensional data encoding device according to this embodiment. The first encoding unit 4910 generates encoded data (encoded stream) by encoding point cloud data using a first encoding method (GPCC (Geometry based PCC)). The first encoding unit 4910 includes a division unit 4911, a plurality of position information encoding units 4912, a plurality of attribute information encoding units 4913, an additional information encoding unit 4914, and a multiplexing unit 4915.

[0216] The division unit 4911 generates a plurality of divided data by dividing the point cloud data. Specifically, the division unit 4911 generates a plurality of divided data by dividing the space of the point cloud data into a plurality of subspaces. Here, the subspace is one of a tile and a slice, or a combination of a tile and a slice. More specifically, the point cloud data includes position information, attribute information, and additional information. The division unit 4911 divides the position information into a plurality of divided position information, and divides the attribute information into a plurality of divided attribute information. In addition, the division unit 4911 generates additional information related to the division.

[0217] The position information encoding units 4912 generate a plurality of pieces of encoded position information by encoding the plurality of pieces of divided position information. For example, the position information encoding units 4912 process the plurality of pieces of divided position information in parallel.

[0218] The attribute information encoding units 4913 generate a plurality of pieces of encoded attribute information by encoding the plurality of pieces of divided attribute information. For example, the attribute information encoding units 4913 process the plurality of pieces of divided attribute information in parallel.

[0219] The additional information encoding unit 4914 generates encoded additional information by encoding the additional information included in the point cloud data and the additional information related to the data division generated by the division unit 4911 at the time of division.

[0220] The multiplexing unit 4915 multiplexes a plurality of pieces of encoding position information, a plurality of pieces of encoding attribute information, and encoding additional information to generate encoded data (encoded stream), and transmits the generated encoded data. In addition, the encoded additional information is used during decoding.

[0221] 29 shows an example in which there are two position information encoders 4912 and two attribute information encoders 4913, but the number of position information encoders 4912 and two attribute information encoders 4913 may be one, or three or more. Furthermore, multiple pieces of divided data may be processed in parallel within the same chip, such as multiple cores in a CPU, or may be processed in parallel by cores on multiple chips, or may be processed in parallel by multiple cores on multiple chips.

[0222] 30 is a block diagram showing a configuration of the first decoding unit 4920. The first decoding unit 4920 restores the point cloud data by decoding the encoded data (encoded stream) generated by encoding the point cloud data by the first encoding method (GPCC). The first decoding unit 4920 includes a demultiplexing unit 4921, a plurality of position information decoding units 4922, a plurality of attribute information decoding units 4923, an additional information decoding unit 4924, and a combining unit 4925.

[0223] The demultiplexer 4921 demultiplexes the coded data (coded stream) to generate a plurality of pieces of coding position information, a plurality of pieces of coding attribute information, and coded additional information.

[0224] The position information decoders 4922 generate a plurality of pieces of divided position information by decoding the plurality of pieces of encoded position information. For example, the position information decoders 4922 process the plurality of pieces of encoded position information in parallel.

[0225] The multiple attribute information decoding unit 4923 generates multiple pieces of divided attribute information by decoding the multiple pieces of encoded attribute information. For example, the multiple attribute information decoding unit 4923 processes the multiple pieces of encoded attribute information in parallel.

[0226] The multiple additional information decoders 4924 generate additional information by decoding the encoded additional information.

[0227] The combining unit 4925 generates position information by combining a plurality of pieces of divided position information using the additional information The combining unit 4925 generates attribute information by combining a plurality of pieces of divided attribute information using the additional information.

[0228] 30 shows an example in which the number of each of the position information decoding units 4922 and the attribute information decoding units 4923 is two, but the number of each of the position information decoding units 4922 and the attribute information decoding units 4923 may be one, or three or more. Furthermore, multiple pieces of divided data may be processed in parallel within the same chip, such as multiple cores within a CPU, or may be processed in parallel by cores on multiple chips, or may be processed in parallel by multiple cores on multiple chips.

[0229] Next, it is described how the dividing unit 4911 has a configuration. Fig. 31 is a block diagram of the dividing unit 4911. The dividing unit 4911 includes a slice dividing unit 4931, a position information tile dividing unit 4932, and an attribute information tile dividing unit 4933.

[0230] The slice division unit 4931 generates a plurality of slice position information by dividing position information (Position (Geometry)) into slices. The slice division unit 4931 also generates a plurality of slice attribute information by dividing attribute information (Attribute) into slices. The slice division unit 4931 also outputs slice additional information (SliceMetaData) including information related to slice division and information generated in the slice division.

[0231] The position information tile division unit 4932 divides a plurality of slice position information pieces into tiles to generate a plurality of pieces of divided position information pieces (a plurality of pieces of tile position information pieces). In addition, the position information tile division unit 4932 outputs position tile additional information (Geometry Tile MetaData) including information related to the tile division of the position information pieces and information generated in the tile division of the position information pieces.

[0232] The attribute information tile division unit 4933 divides a plurality of slice attribute information pieces into tiles to generate a plurality of pieces of divided attribute information (a plurality of pieces of tile attribute information). In addition, the attribute information tile division unit 4933 outputs attribute tile additional information (Attribute Tile MetaData) including information related to the tile division of the attribute information and information generated in the tile division of the attribute information.

[0233] The number of slices or tiles to be divided is equal to or greater than 1. In other words, it is not necessary to divide the slices or tiles.

[0234] Although an example in which tile division is performed after slice division has been shown here, slice division may be performed after tile division. Furthermore, a new division type may be defined in addition to slices and tiles, and division may be performed using three or more division types.

[0235] A method for dividing point cloud data will be described below. Fig. 32 is a diagram showing an example of division into slices and tiles.

[0236] First, a method of dividing into slices will be described. The dividing unit 4911 divides the three-dimensional point cloud data into arbitrary point clouds in slice units. In the slice division, the dividing unit 4911 does not divide the position information and attribute information constituting a point, but divides the position information and attribute information together. That is, the dividing unit 4911 divides into slices so that the position information and attribute information at an arbitrary point belong to the same slice. In addition, according to these, the number of divisions and the division method may be any method. In addition, the minimum unit of division is a point. For example, the number of divisions of the position information and the attribute information are the same. For example, the three-dimensional point corresponding to the position information after the slice division and the three-dimensional point corresponding to the attribute information are included in the same slice.

[0237] Furthermore, the division unit 4911 generates slice additional information, which is additional information related to the number of divisions and the division method when dividing a slice. The slice additional information is the same as the position information and the attribute information. For example, the slice additional information includes information indicating the reference coordinate position, size, or side length of the bounding box after division. Furthermore, the slice additional information includes information indicating the number of divisions, the division type, etc.

[0238] Next, a tile division method will be described. The division unit 4911 divides the data divided into slices into slice position information (G slices) and slice attribute information (A slices), and divides each of the slice position information and the slice attribute information into tiles.

[0239] Although FIG. 32 shows an example of division using an octree structure, the number of divisions and the division method may be any method.

[0240] Furthermore, the division unit 4911 may divide the position information and the attribute information using different division methods or the same division method. Furthermore, the division unit 4911 may divide a plurality of slices into tiles using different division methods or the same division method.

[0241] Furthermore, the division unit 4911 generates tile additional information related to the number of divisions and the division method when dividing tiles. The tile additional information (position tile additional information and attribute tile additional information) is independent of position information and attribute information. For example, the tile additional information includes information indicating the reference coordinate position, size, or side length of the bounding box after division. The tile additional information also includes information indicating the number of divisions, the division type, etc.

[0242] Next, an example of a method for dividing the point cloud data into slices or tiles will be described. The dividing unit 4911 may use a predetermined method as a method for dividing the point cloud data into slices or tiles, or may adaptively switch the method to be used depending on the point cloud data.

[0243] When dividing into slices, the dividing unit 4911 divides the three-dimensional space collectively based on the position information and the attribute information. For example, the dividing unit 4911 determines the shape of an object and divides the three-dimensional space into slices according to the shape of the object. For example, the dividing unit 4911 extracts objects such as trees or buildings, and divides the space on an object-by-object basis. For example, the dividing unit 4911 divides the space into slices such that one or more objects are entirely included in one slice. Alternatively, the dividing unit 4911 divides one object into multiple slices.

[0244] In this case, the encoding device may change the encoding method for each slice, for example. For example, the encoding device may use a high-quality compression method for a specific object or a specific part of an object. In this case, the encoding device may store information indicating the encoding method for each slice in additional information (metadata).

[0245] Furthermore, the dividing unit 4911 may divide the image into slices based on map information or location information so that each slice corresponds to a predetermined coordinate space.

[0246] When dividing into tiles, the dividing unit 4911 divides the position information and the attribute information independently. For example, the dividing unit 4911 divides a slice into tiles according to the amount of data or the amount of processing. For example, the dividing unit 4911 determines whether the amount of data of a slice (for example, the number of three-dimensional points included in a slice) is greater than a predetermined threshold. If the amount of data of a slice is greater than the threshold, the dividing unit 4911 divides the slice into tiles. If the amount of data of a slice is less than the threshold, the dividing unit 4911 does not divide the slice into tiles.

[0247] For example, the division unit 4911 divides the slice into tiles so that the processing amount or processing time in the decoding device is within a certain range (a predetermined value or less). This makes the processing amount per tile in the decoding device constant, facilitating distributed processing in the decoding device.

[0248] Furthermore, when the processing amount differs between the position information and the attribute information, for example when the processing amount of the position information is greater than the processing amount of the attribute information, dividing section 4911 divides the position information into a number greater than the number of divisions of the attribute information.

[0249] Also, for example, in the case where the decoding device may decode and display the position information quickly and the attribute information may be decoded and displayed slowly later depending on the content, the division unit 4911 may divide the position information into a larger number of parts than the attribute information. This allows the decoding device to process the position information in parallel in a larger number, so that the position information can be processed faster than the attribute information.

[0250] Note that the decoding device does not necessarily need to process sliced ​​or tiled data in parallel, and may determine whether to process the data in parallel depending on the number or capabilities of the decoding processing units.

[0251] By dividing the image in the above manner, it is possible to realize adaptive encoding according to the content or object. In addition, it is possible to realize parallel processing in the decoding process. This improves the flexibility of the point cloud encoding system or the point cloud decoding system.

[0252] Fig. 33 is a diagram showing an example of a division pattern of slices and tiles. DU in the diagram is a data unit (DataUnit) and indicates data of a tile or slice. Each DU includes a slice index (SliceIndex) and a tile index (TileIndex). The numerical value in the upper right of the DU in the diagram indicates the slice index, and the numerical value in the lower left of the DU indicates the tile index.

[0253] In pattern 1, the number of divisions and the division method are the same for G slices and A slices in slice division. In tile division, the number of divisions and the division method for G slices are different from the number of divisions and the division method for A slices. Furthermore, the same number of divisions and the division method are used among multiple G slices. The same number of divisions and the division method are used among multiple A slices.

[0254] In pattern 2, the number of divisions and the division method are the same for G slices and A slices in slice division. In tile division, the number of divisions and the division method for G slices are different from the number of divisions and the division method for A slices. In addition, the number of divisions and the division method are different between multiple G slices. The number of divisions and the division method are different between multiple A slices.

[0255] Next, a method for encoding divided data will be described. A three-dimensional data encoding device (first encoding unit 4910) encodes each of the divided data. When encoding attribute information, the three-dimensional data encoding device generates dependency information indicating which configuration information (position information, additional information, or other attribute information) was used for encoding as additional information. That is, the dependency information indicates, for example, the configuration information of the reference destination (dependency destination). In this case, the three-dimensional data encoding device generates dependency information based on configuration information corresponding to the division shape of the attribute information. Note that the three-dimensional data encoding device may generate dependency information based on configuration information corresponding to a plurality of division shapes.

[0256] The dependency information may be generated by the three-dimensional data encoding device, and the generated dependency information may be sent to the three-dimensional data decoding device. Alternatively, the three-dimensional data decoding device may generate the dependency information, and the three-dimensional data encoding device may not send the dependency information. Also, the dependency relationships used by the three-dimensional data encoding device may be determined in advance, and the three-dimensional data encoding device may not send the dependency information.

[0257] Fig. 34 is a diagram showing an example of the dependency relationship of each data. The tip of the arrow in the diagram indicates the dependency destination, and the start of the arrow indicates the dependency source. The three-dimensional data decoding device decodes data in the order from the dependency destination to the dependency source. Also, data shown by solid lines in the diagram is data that is actually sent, and data shown by dotted lines is data that is not sent.

[0258] In the figure, G indicates location information, and A indicates attribute information. s1 indicates the position information of slice number 1, and G s2 indicates the position information of slice number 2. s1t1 indicates the position information of slice number 1 and tile number 1, and G s1t2 indicates the position information of slice number 1 and tile number 2, and G s2t1 indicates the position information of slice number 2 and tile number 1, and G s2t2 indicates the position information of slice number 2 and tile number 2. Similarly, A s1 indicates the attribute information of slice number 1, and A s2 indicates the attribute information of slice number 2. s1t1 indicates the attribute information of slice number 1 and tile number 1, and A s1t2 indicates the attribute information of slice number 1 and tile number 2, and A s2t1 indicates the attribute information of slice number 2 and tile number 1, and A s2t2 indicates attribute information of slice number 2 and tile number 2.

[0259] Mslice indicates slice additional information, MGtile indicates position tile additional information, and MAtile indicates attribute tile additional information. s1t1 is attribute information As1t1 D s2t1 is attribute information A s2t1 This shows dependency information for

[0260] Furthermore, the three-dimensional data encoding device may rearrange the data in the decoding order so that there is no need to rearrange the data in the three-dimensional data decoding device. Note that the data may be rearranged in the three-dimensional data decoding device, or the data may be rearranged in both the three-dimensional data encoding device and the three-dimensional data decoding device.

[0261] Fig. 35 is a diagram showing an example of the data decoding order. In the example of Fig. 35, decoding is performed in order from the left data. When data has a dependency relationship, the three-dimensional data decoding device decodes the dependent data first. For example, the three-dimensional data encoding device rearranges the data in advance so as to achieve this order, and transmits the data. Any order may be used as long as the dependent data comes first. The three-dimensional data encoding device may also transmit additional information and dependency information before the data.

[0262] Fig. 36 is a flowchart showing the flow of processing by the three-dimensional data encoding device. First, the three-dimensional data encoding device encodes data of a plurality of slices or tiles as described above (S4901). Next, the three-dimensional data encoding device rearranges the data so that the dependent data comes first (S4902), as shown in Fig. 35. Next, the three-dimensional data encoding device multiplexes (converts into NAL units) the rearranged data (S4903).

[0263] Next, a description will be given of the configuration of the combining unit 4925 included in the first decoding unit 4920. Fig. 37 is a block diagram showing the configuration of the combining unit 4925. The combining unit 4925 includes a position information tile combining unit 4941 (geometry tile combiner), an attribute information tile combining unit 4942 (attribute tile combiner), and a slice combining unit (slice combiner).

[0264] The position information tile combining unit 4941 generates a plurality of slice position information pieces by combining a plurality of pieces of divided position information pieces using the position tile additional information. The attribute information tile combining unit 4942 generates a plurality of slice attribute information pieces by combining a plurality of pieces of divided attribute information pieces using the attribute tile additional information.

[0265] The slice combining unit 4943 generates position information by combining a plurality of slice position information pieces using the slice additional information. Also, the slice combining unit 4943 generates attribute information by combining a plurality of slice attribute information pieces using the slice additional information.

[0266] The number of slices or tiles to be divided is equal to or greater than 1. In other words, the division into slices or tiles does not necessarily have to be performed.

[0267] Although an example in which tile division is performed after slice division has been shown here, slice division may be performed after tile division. Furthermore, a new division type may be defined in addition to slices and tiles, and division may be performed using three or more division types.

[0268] Next, a structure of coded data divided into slices or tiles and a method of storing the coded data in NAL units (multiplexing method) will be described. Fig. 38 is a diagram showing a structure of coded data and a method of storing the coded data in NAL units.

[0269] The encoded data (partition position information and partition attribute information) is stored in the payload of the NAL unit.

[0270] The encoded data includes a header and a payload. The header includes identification information for identifying data included in the payload. This identification information includes, for example, a type of slice division or tile division (slice_type, tile_type), index information for identifying a slice or tile (slice_idx, tile_idx), position information of data (slice or tile), or an address of data (address). The index information for identifying a slice is also referred to as a slice index (SliceIndex). The index information for identifying a tile is also referred to as a tile index (TileIndex). The type of division is, for example, a method based on an object shape as described above, a method based on map information or position information, or a method based on a data amount or a processing amount.

[0271] In addition, all or part of the above information may be stored in one of the headers of the divided position information and the divided attribute information, and may not be stored in the other. For example, when the same division method is used for the position information and the attribute information, the division type (slice_type, tile_type) and index information (slice_idx, tile_idx) are the same for the position information and the attribute information. Therefore, these information may be included in the header of one of the position information and the attribute information. For example, when the attribute information depends on the position information, the position information is processed first. Therefore, these information may be included in the header of the position information, and may not be included in the header of the attribute information. In this case, the three-dimensional data decoding device determines that the dependent attribute information belongs to the same slice or tile as the slice or tile of the dependent position information, for example.

[0272] In addition, additional information related to slice division or tile division (slice additional information, position tile additional information, or attribute tile additional information), and dependency information indicating dependency, etc. may be stored in an existing parameter set (GPS, APS, position SPS, attribute SPS, etc.) and transmitted. When the division method changes for each frame, information indicating the division method may be stored in a parameter set for each frame (GPS or APS, etc.). When the division method does not change within a sequence, information indicating the division method may be stored in a parameter set for each sequence (position SPS or attribute SPS). Furthermore, when the same division method is used for position information and attribute information, information indicating the division method may be stored in a parameter set of a PCC stream (stream PS).

[0273] The above information may be stored in any one of the above parameter sets, or in multiple parameter sets. A parameter set for tile division or slice division may be defined, and the above information may be stored in the parameter set. The information may be stored in a header of the encoded data.

[0274] Furthermore, the header of the encoded data includes identification information indicating a dependency relationship. That is, when there is a dependency relationship between data, the header includes identification information for referring to the dependency from the dependency source. For example, the header of the dependency destination data includes identification information for identifying the data. The header of the dependency source data includes identification information indicating the dependency. Note that, when the identification information for identifying the data, the additional information related to the slice division or tile division, and the identification information indicating the dependency relationship can be identified or derived from other information, these pieces of information may be omitted.

[0275] Next, a flow of the encoding process and the decoding process of the point cloud data according to this embodiment will be described. Fig. 39 is a flowchart of the encoding process of the point cloud data according to this embodiment.

[0276] First, the three-dimensional data encoding device determines the division method to be used (S4911). The division method includes whether or not to perform slice division and whether or not to perform tile division. The division method may also include the number of divisions when performing slice division or tile division, and the type of division. The type of division may be a method based on the object shape as described above, a method based on map information or position information, or a method based on the amount of data or the amount of processing. The division method may be determined in advance.

[0277] When slice division is performed (Yes in S4912), the three-dimensional data encoding device generates a plurality of slice position information and a plurality of slice attribute information by dividing the position information and the attribute information together (S4913). In addition, the three-dimensional data encoding device generates slice additional information related to the slice division. Note that the three-dimensional data encoding device may divide the position information and the attribute information independently.

[0278] If tile division is performed (Yes in S4914), the three-dimensional data encoding device generates a plurality of division position information and a plurality of division attribute information by independently dividing a plurality of slice position information and a plurality of slice attribute information (or position information and attribute information) (S4915). In addition, the three-dimensional data encoding device generates position tile additional information and attribute tile additional information related to the tile division. Note that the three-dimensional data encoding device may divide the slice position information and slice attribute information together.

[0279] Next, the three-dimensional data encoding device generates a plurality of pieces of encoding position information and a plurality of pieces of encoding attribute information by encoding each of the plurality of pieces of division position information and the plurality of pieces of division attribute information (S4916). Also, the three-dimensional data encoding device generates dependency relationship information.

[0280] Next, the three-dimensional data encoding device generates encoded data (encoded stream) by forming (multiplexing) the plurality of pieces of encoding position information, the plurality of pieces of encoding attribute information, and the additional information into NAL units (S4917). In addition, the three-dimensional data encoding device transmits the generated encoded data.

[0281] 40 is a flowchart of a decoding process of point cloud data according to this embodiment. First, the three-dimensional data decoding device analyzes additional information (slice additional information, position tile additional information, and attribute tile additional information) related to the division method included in the encoded data (encoded stream) to determine the division method (S4921). This division method includes whether or not to perform slice division and whether or not to perform tile division. In addition, the division method may include the number of divisions when performing slice division or tile division, the type of division, etc.

[0282] Next, the three-dimensional data decoding device generates split position information and split attribute information by decoding the multiple pieces of encoded position information and multiple pieces of encoded attribute information contained in the encoded data using dependency information contained in the encoded data (S4922).

[0283] When the additional information indicates that tile division has been performed (Yes in S4923), the three-dimensional data decoding device generates a plurality of slice position information and a plurality of slice attribute information by combining a plurality of pieces of division position information and a plurality of pieces of division attribute information by respective methods based on the position tile additional information and the attribute tile additional information (S4924). Note that the three-dimensional data decoding device may combine a plurality of pieces of division position information and a plurality of pieces of division attribute information by the same method.

[0284] When the additional information indicates that slice division has been performed (Yes in S4925), the three-dimensional data decoding device generates position information and attribute information by combining multiple slice position information and multiple slice attribute information (multiple division position information and multiple division attribute information) in the same manner based on the slice additional information (S4926). Note that the three-dimensional data decoding device may combine multiple slice position information and multiple slice attribute information in different manners.

[0285] In addition, attribute information of tiles or slices (identifiers, area information, address information, position information, etc.) may be stored in other control information, not limited to SEI. For example, the attribute information may be stored in control information indicating the configuration of the entire PCC data, or may be stored in control information for each tile or slice.

[0286] Furthermore, when transmitting PCC data to another device, the three-dimensional data encoding device (three-dimensional data transmission device) may convert control information such as SEI into control information specific to the protocol of that system and display it.

[0287] For example, when the three-dimensional data encoding device converts PCC data including attribute information into ISOBMFF (ISO Base Media File Format), the SEI may be stored in an "mdat box" together with the PCC data, or in a "track box" that describes control information related to the stream. In other words, the three-dimensional data encoding device may store the control information in a table for random access. Also, when the three-dimensional data encoding device packets and transmits the PCC data, the SEI may be stored in a packet header. In this way, by making it possible to acquire attribute information at a system layer, access to the attribute information and the tile data or slice data becomes easier, and the access speed can be improved.

[0288] In addition, in the configuration of the three-dimensional data decoding device, the memory management unit may determine in advance whether or not the information necessary for the decoding process is in the memory, and if the information necessary for the decoding process is not present, the information may be obtained from storage or a network.

[0289] When the three-dimensional data decoding device acquires PCC data from a storage or a network using Pull in a protocol such as MPEG-DASH, the memory management unit may identify attribute information of data necessary for decoding processing based on information from a localization unit or the like, request tiles or slices including the identified attribute information, and acquire the necessary data (PCC stream). Identification of tiles or slices including attribute information may be performed on the storage or network side, or may be performed by the memory management unit. For example, the memory management unit may acquire SEI of all PCC data in advance, and identify tiles or slices based on that information.

[0290] When all PCC data is transmitted from the storage or network using Push in a UDP protocol or the like, the memory management unit may identify attribute information of the data required for the decoding process and tiles or slices based on information from the localization unit or the like, and obtain the desired data by filtering the desired tiles or slices from the transmitted PCC data.

[0291] Furthermore, when acquiring data, the three-dimensional data encoding device may determine whether or not desired data is available, whether or not real-time processing is possible based on the data size, etc., or the communication state, etc. If the three-dimensional data encoding device determines that data acquisition is difficult based on the result of this determination, it may select and acquire another slice or tile with a different priority or amount of data.

[0292] In addition, the three-dimensional data decoding device may transmit information from a localization unit or the like to a cloud server, and the cloud server may determine the necessary information based on that information.

[0293] (Embodiment 4) Next, the tile additional information will be described. The three-dimensional data encoding device generates tile additional information, which is metadata related to a method of dividing tiles, and transmits the generated tile additional information to the three-dimensional data decoding device.

[0294] Fig. 41 is a diagram showing an example of the syntax of tile additional information (TileMetaData). As shown in Fig. 41, for example, the tile additional information includes division method information (type_of_divide), shape information (topview_shape), overlap flag (tile_overlap_flag), overlap information (type_of_overlap), height information (tile_height), number of tiles (tile_number), and tile position information (global_position, relative_position).

[0295] The division method information (type_of_divide) indicates a division method of tiles. For example, the division method information indicates whether the division method of tiles is based on map information, that is, division based on a top view (top_view), or other (other).

[0296] The shape information (topview_shape) is included in the tile additional information, for example, when the tile division method is division based on a top view. The shape information indicates the shape of the tile when viewed from above. For example, this shape includes a square and a circle. Note that this shape may include an ellipse, a rectangle, or a polygon other than a quadrangle, or may include other shapes. Note that the shape information is not limited to the shape of the tile when viewed from above, and may also indicate the three-dimensional shape of the tile (for example, a cube, a cylinder, etc.).

[0297] The overlap flag (tile_overlap_flag) indicates whether tiles overlap. For example, the overlap flag is included in the tile additional information when the tile division method is division based on a top view. In this case, the overlap flag indicates whether tiles overlap in a top view. Note that the overlap flag may also indicate whether tiles overlap in a three-dimensional space.

[0298] The overlap information (type_of_overlap) is included in the tile additional information when tiles overlap, for example. The overlap information indicates how the tiles overlap, etc. For example, the overlap information indicates the size of the overlapping area, etc.

[0299] Height information (tile_height) indicates the height of a tile. The height information may include information indicating the shape of the tile. For example, if the shape of the tile in top view is rectangular, the information may indicate the lengths of the sides of the rectangle (the vertical length and the horizontal length). Furthermore, if the shape of the tile in top view is circular, the information may indicate the diameter or radius of the circle.

[0300] The height information may indicate the height of each tile, or may indicate a common height for multiple tiles. Multiple height types, such as roads and intersections, may be set in advance, and the height information may indicate the height of each height type and the height type of each tile. Alternatively, the height of each height type may be defined in advance, and the height information may indicate the height type of each tile. In other words, the height of each height type does not have to be indicated by the height information.

[0301] The tile number (tile_number) indicates the number of tiles. The tile additional information may include information indicating the spacing between tiles.

[0302] The tile position information (global_position, relative_position) is information for identifying the position of each tile. For example, the tile position information indicates the absolute coordinates or relative coordinates of each tile.

[0303] Note that some or all of the above information may be provided for each tile, or for each set of multiple tiles (for example, for each frame or for each set of multiple frames).

[0304] The three-dimensional data encoding device may include the tile additional information in SEI (Supplemental Enhancement Information) and transmit it. Alternatively, the three-dimensional data encoding device may store the tile additional information in an existing parameter set (such as PPS, GPS, or APS) and transmit it.

[0305] For example, if the tile additional information changes for each frame, the tile additional information may be stored in a parameter set for each frame (GPS or APS, etc.). If the tile additional information does not change within a sequence, the tile additional information may be stored in a parameter set for each sequence (position SPS or attribute SPS). Furthermore, if the same tile division information is used for position information and attribute information, the tile additional information may be stored in a parameter set of a PCC stream (stream PS).

[0306] Furthermore, the tile additional information may be stored in any one of the above parameter sets, or in multiple parameter sets. Furthermore, the tile additional information may be stored in a header of the encoded data. Furthermore, the tile additional information may be stored in a header of a NAL unit.

[0307] In addition, all or part of the tile additional information may be stored in one of the headers of the divided position information and the divided attribute information, and not in the other. For example, when the same tile additional information is used in the position information and the attribute information, the tile additional information may be included in the header of either the position information or the attribute information. For example, when the attribute information depends on the position information, the position information is processed first. Therefore, the header of the position information may include these tile additional information, and the header of the attribute information may not include the tile additional information. In this case, the three-dimensional data decoding device determines, for example, that the dependent attribute information belongs to the same tile as the tile of the dependent position information.

[0308] The three-dimensional data decoding device reconstructs the point cloud data divided into tiles based on the tile additional information. If there is overlapping point cloud data, the three-dimensional data decoding device identifies the overlapping multiple point cloud data and selects one of them or merges the multiple point cloud data.

[0309] The three-dimensional data decoding device may also perform decoding using tile additional information. For example, when multiple tiles overlap, the three-dimensional data decoding device may perform decoding for each tile, and perform processing (e.g., smoothing or filtering) using the multiple decoded data to generate point cloud data. This may enable highly accurate decoding.

[0310] 42 is a diagram showing an example of the configuration of a system including a three-dimensional data encoding device and a three-dimensional data decoding device. A tile dividing unit 5051 divides point cloud data including position information and attribute information into a first tile and a second tile. The tile dividing unit 5051 also sends tile additional information related to the tile division to a decoding unit 5053 and a tile combining unit 5054.

[0311] The encoding unit 5052 generates encoded data by encoding the first tile and the second tile.

[0312] The decoding unit 5053 restores the first tile and the second tile by decoding the coded data generated by the coding unit 5052. The tile combining unit 5054 restores the point cloud data (position information and attribute information) by combining the first tile and the second tile using the tile additional information.

[0313] Next, slice additional information will be described. The three-dimensional data encoding device generates slice additional information, which is metadata related to a method of dividing a slice, and transmits the generated slice additional information to the three-dimensional data decoding device.

[0314] Fig. 43 is a diagram showing an example of the syntax of slice additional information (SliceMetaData). As shown in Fig. 43, for example, the slice additional information includes division method information (type_of_divide), an overlap flag (slice_overlap_flag), overlap information (type_of_overlap), the number of slices (slice_number), slice position information (global_position, relative_position), and slice size information (slice_bounding_box_size).

[0315] The division method information (type_of_divide) indicates a division method of a slice. For example, the division method information indicates whether the division method of a slice is division based on object information as shown in FIG. 60 (object). The slice additional information may include information indicating a method of object division. For example, this information indicates whether one object is divided into multiple slices or assigned to one slice. This information may also indicate the number of divisions when one object is divided into multiple slices.

[0316] The overlap flag (slice_overlap_flag) indicates whether the slices overlap. The overlap information (type_of_overlap) is included in the slice additional information, for example, when the slices overlap. The overlap information indicates how the slices overlap, etc. For example, the overlap information indicates the size of the overlapping area, etc.

[0317] The slice number (slice_number) indicates the number of slices.

[0318] Slice position information (global_position, relative_position) and slice size information (slice_bounding_box_size) are information related to the area of ​​a slice. Slice position information is information for identifying the position of each slice. For example, slice position information indicates absolute coordinates or relative coordinates of each slice. Slice size information (slice_bounding_box_size) indicates the size of each slice. For example, slice size information indicates the size of the bounding box of each slice.

[0319] The three-dimensional data encoding device may include the slice additional information in the SEI and transmit it. Alternatively, the three-dimensional data encoding device may store the slice additional information in an existing parameter set (such as PPS, GPS, or APS) and transmit it.

[0320] For example, when slice additional information changes for each frame, the slice additional information may be stored in a parameter set for each frame (GPS or APS, etc.). When slice additional information does not change within a sequence, the slice additional information may be stored in a parameter set for each sequence (position SPS or attribute SPS). Furthermore, when the same slice division information is used for position information and attribute information, the slice additional information may be stored in a parameter set of a PCC stream (stream PS).

[0321] Furthermore, the slice additional information may be stored in any one of the above parameter sets, or in a plurality of parameter sets. Furthermore, the slice additional information may be stored in a header of the encoded data. Furthermore, the slice additional information may be stored in a header of the NAL unit.

[0322] In addition, all or a part of the slice additional information may be stored in one of the headers of the division position information and the header of the division attribute information, and may not be stored in the other. For example, when the same slice additional information is used in the position information and the attribute information, the slice additional information may be included in the header of one of the position information and the attribute information. For example, when the attribute information depends on the position information, the position information is processed first. Therefore, the header of the position information may include these slice additional information, and the header of the attribute information may not include the slice additional information. In this case, the three-dimensional data decoding device determines that the dependent attribute information belongs to the same slice as the slice of the dependent position information, for example.

[0323] The three-dimensional data decoding device reconstructs the point cloud data divided into slices based on the slice additional information. If there is overlapping point cloud data, the three-dimensional data decoding device identifies the overlapping multiple point cloud data and selects one of them or merges the multiple point cloud data.

[0324] The three-dimensional data decoding device may also perform decoding using slice additional information. For example, when multiple slices overlap, the three-dimensional data decoding device may perform decoding for each slice, and perform processing (e.g., smoothing or filtering) using the multiple decoded data to generate point cloud data. This may enable highly accurate decoding.

[0325] FIG. 44 is a flowchart of three-dimensional data encoding processing, including processing for generating tile additional information, performed by the three-dimensional data encoding device according to this embodiment.

[0326] First, the three-dimensional data encoding device determines a tile division method (S5031). Specifically, the three-dimensional data encoding device determines whether to use a division method based on a top view (top_view) or another method (other) as a tile division method. The three-dimensional data encoding device also determines the shape of the tile when using a division method based on a top view. The three-dimensional data encoding device also determines whether the tile overlaps with other tiles.

[0327] If the tile division method determined in step S5031 is a division method based on a top view (Yes in S5032), the three-dimensional data encoding device describes in the tile additional information that the tile division method is a division method based on a top view (top_view) (S5033).

[0328] On the other hand, if the tile division method determined in step S5031 is other than a division method based on a top view (No in S5032), the three-dimensional data encoding device describes in the tile additional information that the tile division method is other than a division method based on a top view (top_view) (S5034).

[0329] Furthermore, if the shape of the tile viewed from above determined in step S5031 is a square (square in S5035), the three-dimensional data encoding device records in the tile additional information that the shape of the tile viewed from above is a square (S5036).On the other hand, if the shape of the tile viewed from above determined in step S5031 is a circle (circle in S5035), the three-dimensional data encoding device records in the tile additional information that the shape of the tile viewed from above is a circle (S5037).

[0330] Next, the three-dimensional data encoding device determines whether the tile overlaps with other tiles (S5038). If the tile overlaps with other tiles (Yes in S5038), the three-dimensional data encoding device records in the tile additional information that the tile overlaps (S5039). On the other hand, if the tile does not overlap with other tiles (No in S5038), the three-dimensional data encoding device records in the tile additional information that the tile does not overlap (S5040).

[0331] Next, the three-dimensional data encoding device divides the tiles based on the tile division method determined in step S5031, encodes each tile, and transmits the generated encoded data and tile additional information (S5041).

[0332] FIG. 45 is a flowchart of three-dimensional data decoding processing using tile additional information by the three-dimensional data decoding device according to this embodiment.

[0333] First, the 3D data decoding device analyzes the tile additional information included in the bitstream (S5051).

[0334] If the tile additional information indicates that the tile does not overlap with other tiles (No in S5052), the 3D data decoding device generates point cloud data for each tile by decoding each tile (S5053). Next, the 3D data decoding device reconstructs point cloud data from the point cloud data for each tile based on the tile division method and tile shape indicated in the tile additional information (S5054).

[0335] On the other hand, if the tile additional information indicates that the tile overlaps with other tiles (Yes in S5052), the three-dimensional data decoding device generates point cloud data for each tile by decoding each tile. The three-dimensional data decoding device also identifies overlapping parts of the tiles based on the tile additional information (S5055). Note that the three-dimensional data decoding device may perform decoding processing for the overlapping parts using multiple pieces of overlapping information. Next, the three-dimensional data decoding device reconstructs point cloud data from the point cloud data of each tile based on the tile division method, tile shape, and overlap information indicated in the tile additional information (S5056).

[0336] Modifications and the like regarding slices will be described below. The three-dimensional data encoding device may transmit information indicating the type (road, building, tree, etc.) or attribute (dynamic information, static information, etc.) of an object as additional information. Alternatively, encoding parameters may be predefined according to the object, and the three-dimensional data encoding device may notify the three-dimensional data decoding device of the encoding parameters by sending the type or attribute of the object.

[0337] The following methods may be used for the coding order and transmission order of slice data. For example, the three-dimensional data encoding device may encode slice data in the order of data that is easiest to recognize or cluster. Alternatively, the three-dimensional data encoding device may encode slice data in the order of slice data that has been clustered first. Furthermore, the three-dimensional data encoding device may transmit the encoded slice data in the order of decreasing priority for decoding in an application. For example, when the priority of decoding dynamic information is high, the three-dimensional data encoding device may transmit the slice data in the order of decreasing priority for decoding dynamic information.

[0338] Furthermore, when the order of the encoded data differs from the order of the decoding priority, the three-dimensional data encoding device may rearrange the encoded data before transmitting it. Furthermore, when storing the encoded data, the three-dimensional data encoding device may store the encoded data after rearranging it.

[0339] An application (a three-dimensional data decoding device) requests a server (a three-dimensional data encoding device) to transmit slices containing desired data. The server transmits slice data required by the application, but does not need to transmit unnecessary slice data.

[0340] An application requests the server to send tiles containing desired data. The server sends the tile data required by the application, but does not need to send unnecessary tile data.

[0341] (Embodiment 5) The encoding of position information (geometry) will be explained. In encoding position information, the three-dimensional data encoding device divides an area including point cloud data using an octree, and converts it into a set of point occupancy information for each node. The occupancy information is 8-bit information that indicates whether or not each child node has a point, and whether or not each child node contains a point is indicated by 0 or 1.

[0342] The order of division into an octree can be determined by using a breadth-first search method, dividing the nodes in order from the smallest depth (depth value), or by using a depth-first search method, searching for points to the lowest depth, then returning to the upper depth and searching again. In encoding, the occupancy information is coded in the above order.

[0343] Fig. 46 is a diagram showing a tree structure when dividing a point cloud having a depth of up to depth=6 into an octree. Fig. 47 is a diagram showing an example of a data structure of encoded data having an octree structure using a breadth-first search method. The encoded data of a point cloud includes a header and a payload. In the payload, information for each depth (depth#0 to #6) is arranged in order. Fig. 48 is a diagram showing an example of the syntax of the payload. The payload includes an occupancy code for each depth.

[0344] Next, the hierarchical structure will be described. For example, as shown in FIG. 46, a plurality of levels (also called hierarchical levels) are defined. Level 2 is a point cloud expressed using point cloud data that has been subjected to octree division from depth=0 to the final depth (depth=6), level 1 is a point cloud expressed using point cloud data that has been subjected to octree division from depth=0 to depth=5, and level 0 is a point cloud expressed using point cloud data that has been subjected to octree division from depth=0 to depth=4. That is, the resolution of the point cloud increases in the order of level 0, level 1, and level 2. Conversely, the resolution of the point cloud decreases in the order of level 2, level 1, and level 0. This can also be said to be quantized by 1 / 2 each time the level decreases. In this way, by using the levels, it is possible to express the entire point cloud using all data up to the lowest layer, or to express low-resolution point cloud data using data from depth=0 to some upper layers. Note that the levels can be adaptively set in various combinations as necessary according to the resolution or amount of data to be handled.

[0345] 49 is a flowchart of a decoding process for decoding coded data in which position information is coded at a target resolution. First, the three-dimensional data decoding device determines a level (resolution) to be decoded and a depth corresponding to that level (S8801).

[0346] Next, the three-dimensional data decoding device decodes the encoded data of the determined depth. Specifically, the three-dimensional data decoding device decodes the first depth (depth 0 (depth=0)) (S8802). If the decoding of all depths to be decoded has not been completed (No in S8803), the three-dimensional data decoding device decodes the next depth (S8804). Note that the three-dimensional data decoding device may decode the depth to be decoded using data of the previous level (or depth). If the decoding of all depths to be decoded has been completed (Yes in S8803), the three-dimensional data decoding device displays the obtained point cloud (S8805).

[0347] Furthermore, the three-dimensional data decoding device may decode the coded data of the determined depth, and not decode the coded data of the remaining depths. Fig. 50 is a diagram showing the relationship between the level and the data to be decoded. Fig. 51 is a schematic diagram showing the levels. For example, in Fig. 50, in order to decode the point group of level 0, it is necessary to decode depth#0 to depth#4. Therefore, the three-dimensional data decoding device decodes up to the coded data of A.

[0348] Moreover, to decode the point group at level 1, decoding from depth#0 to depth#5 is necessary. Therefore, the three-dimensional data decoding device decodes up to the encoded data of B. Moreover, to decode the point group at level 2 (all), decoding from depth#0 to depth#6 is necessary. Therefore, the three-dimensional data decoding device decodes up to the encoded data of C.

[0349] By the above process, the three-dimensional data decoding device can decode low-resolution data. Therefore, when a high-resolution point cloud is not required, the three-dimensional data decoding device can reduce the amount of data or reduce the amount of processing by skipping the decoding process.

[0350] Alternatively, the three-dimensional data decoding device can display the low-resolution data after decoding the low-resolution data without waiting for the remaining data to be decoded, and display the remaining data after decoding the high-resolution data, thereby shortening the initial delay in decoding and display.

[0351] Here, in order to decode intermediate data, the three-dimensional data decoding device needs to determine that data has been acquired up to point A or B, that is, to determine the boundary between depths or depth information in the encoded data. Fig. 52 is a diagram showing an example of the syntax of the header. Fig. 53 is a diagram showing an example of the syntax of the payload.

[0352] For example, a three-dimensional data decoding device may use the number of points (numPoint) in the point cloud indicated in the header, the number of depths (depth), and the encoded data (occupancy_code) for each node or leaf stored for each depth indicated in the payload, decode the encoded data from the beginning, and analyze the information of the decoded occupancy code to determine the boundary between depths or depth information.

[0353] In this embodiment, a data structure that facilitates partial decoding of such depth data, as well as data division and joining is described. Here, a new data structure that takes into account a hierarchical structure is defined. By using this data structure, data division and joining of hierarchical data units becomes possible. The three-dimensional data encoding device or the three-dimensional data decoding device can reduce the amount of data required for transmission by extracting specific necessary hierarchical data. Furthermore, the functionality of the three-dimensional data encoding device or the three-dimensional data decoding device is improved by being able to perform data division or joining without decoding the encoded data.

[0354] FIG. 54 is a diagram showing the structure of coded data having all position information from depth 0 to depth 6 (depth#0 to depth#6). This coded data is also called the entire coded data or a bit stream (coded bit stream). In this example, the entire coded data does not include information indicating an explicit boundary in the data between depths. The three-dimensional data decoding device can obtain the boundary between depths or the depth information by analyzing the occupancy code (occupancy_code). Here, the number of points (numPoint) included in the header indicates the number of all points included in the entire coded data. Also, the depth number indicating the number of depths is "7" in this example.

[0355] Fig. 55 is a diagram showing the structure of the entire encoded data. The entire encoded data shown in Fig. 55 includes hierarchical structure metadata, which is metadata indicating a hierarchical structure, in addition to the structure shown in Fig. 54. Fig. 56 is a diagram showing an example of the syntax of depth information (depth_info). Fig. 57 is a diagram showing an example in which depth information is stored in hierarchical structure metadata (layer_metadata).

[0356] The depth information includes a depth number and length information indicating the length of data for each depth. The length information indicates, for example, the difference between the start position and the end position of the encoded data (also referred to as depth data) of the corresponding depth in the number of bytes or bits.

[0357] The hierarchical structure metadata including the length information may be sent before or after the encoded data. The length information may also be stored in the header of the entire encoded data. Figure 58 is a diagram showing an example of the header syntax in this case. The header includes the number of points (numPoint) and depth information (depth_info).

[0358] In this way, for example, the data boundaries between depths are clearly indicated in the hierarchical structure metadata or the header. Note that the three-dimensional data decoding device does not need to use the hierarchical structure metadata for decoding. In this case, the three-dimensional data decoding device uses the hierarchical structure metadata when dividing or reconstructing the hierarchical data.

[0359] In the configurations shown in Figures 54 and 55, all of the depth data is continuous, and the context information used for entropy coding is also continuous and not initialized.

[0360] By using this structure, the three-dimensional data decoding device can easily divide the entire encoded data into data for each depth, thereby reducing the amount of processing. In addition, the divided data can be transmitted, thereby reducing the amount of transmission.

[0361] Next, a case where the concept of a hierarchy (layer) is introduced will be described. FIG. 59 is a diagram showing an example of the configuration of the entire encoded data in this case. In the figure, depth 0 to depth 4 (depth#0 to depth#4) are defined as layer 0, depth 5 (depth#5) is defined as layer 1, and depth 6 (depth#6) is defined as layer 2. That is, compared with the above-mentioned levels (hierarchical levels), a level is defined at a plurality of depths from depth 0 to a desired depth, whereas a plurality of hierarchical levels are defined at depths that do not overlap with each other. For example, layer 0 includes the same depth as level 0, layer 1 corresponds to a depth of the difference between level 1 and level 0, and layer 2 corresponds to the depth of the difference between level 2 and level 1.

[0362] In this case, layer information (layer_info) is added to the entire encoded data. FIG. 60 is a diagram showing an example of the syntax of the layer information. The layer information includes a layer number (layer) indicating the number of layers, and a layer depth number (num_depth) indicating the number of depths included in each layer. FIG. 61 is a diagram showing an example of the syntax of the hierarchical structure metadata. For example, the hierarchical structure metadata includes layer information (layer_info). FIG. 62 is a diagram showing an example of the syntax of the header of the entire encoded data. For example, as shown in FIG. 62, the header includes depth information (depth_info). Note that both the layer information and the depth information may be included in the header or the hierarchical structure metadata. Note that the syntax structure shown here is an example, and is not limited to this. The information included in the entire encoded data may be information that allows the three-dimensional data decoding device to obtain the depth number, the layer number, the layer depth number, and the length information. For example, the entire encoded data may include information indicating the length of each layer.

[0363] Furthermore, if the layer structure is common to a plurality of processing units (eg, frames, etc.), some or all of this information may be included in higher level metadata such as a sequence level parameter set (eg, SPS).

[0364] In addition, a flag indicating whether the entire encoded data includes layer information (layer_info) and depth information (depth_info) may be included in the entire encoded data. When the flag is on (e.g., value 1), the layer information and depth information may be included in the entire encoded data. In addition, the flag may be provided separately for the layer information and the depth information.

[0365] By using this structure, it is easy to divide the data into data for each depth or to divide the data into data for each layer, so the amount of processing can be reduced. In addition, it is possible to transmit the divided data, so the amount of transmission can be reduced.

[0366] Next, the overall configuration of the point cloud data will be described. In addition to position information (geometry), the point cloud data may have one or more pieces of attribute information (Attribute), such as color or reflectance. The attribute information may also have a hierarchical structure, similar to the position information.

[0367] When the attribute information has a hierarchical structure similar to that of the position information, the three-dimensional data encoding device stores the hierarchical information in the hierarchical metadata or data header similar to that of the position information. For example, the three-dimensional data encoding device stores the hierarchical information in each of the attribute information header and the position information header. When storing the hierarchical information in the hierarchical metadata, the three-dimensional data encoding device may store the hierarchical metadata in separate parameter sets for the position information such as GPS or APS and the attribute information, or may store the hierarchical metadata in a common parameter set such as SPS. In addition, the three-dimensional data encoding device may store the hierarchical metadata in the SEI or other metadata.

[0368] FIG. 63 is a diagram showing an example of the configuration of a bit stream in the case where the position information and attribute information each have a hierarchical structure, and the hierarchical structure information is stored in hierarchical structure metadata. By using this structure, it becomes easy to divide the data into data for each layer, so that the amount of processing can be reduced. In addition, since it becomes possible to transmit the divided data, the amount of transmission can be reduced. In addition, it becomes possible to divide the position information and attribute information for each layer in the same way.

[0369] 64 is a diagram showing an example of the syntax of hierarchical structure metadata, which shows an example of a case where common hierarchical structure metadata is applied to position information and attribute information.

[0370] The hierarchical structure metadata includes layer information (layer_info), the number of components (component), and depth information (depth_info) for each component. The number of components indicates the number of components such as position information and attribute information. For example, if point cloud data has color and reflectance in addition to position information, the number of components is 3. Note that the number of components of attribute information may be indicated on the assumption that position information is always present. Also, if the number of components of attribute information is indicated in the SPS, this information may be omitted. This reduces the amount of data.

[0371] Layer information (layer_info) indicates the number of layers and the number of depths contained in each layer. For example, layer information is common to all components.

[0372] The depth information (depth_info) indicates the depth number and the data length (length information) of each depth data. The depth information is set for each component, for example. Note that some or all of the depth information may be common to all components.

[0373] Furthermore, when the hierarchical structure is made independent for each component, the hierarchical information may be generated for each component.

[0374] Although the method of layering based on depth has been described here, layering may also be performed based on time information or space information. When layering is performed, hierarchical structure metadata is indicated by the above method. Also, a configuration without a hierarchical structure may be generated. Information indicating whether or not the encoded data has a hierarchical structure may be included in the header or metadata. This allows data with a hierarchical structure and data without a hierarchical structure to be mixed. For example, the position information may have a hierarchical structure, and the attribute information may not have a hierarchical structure. Also, information indicating this may be indicated in the header, etc.

[0375] Next, the data reference relationship and dependency relationship between the position information and the attribute information will be described. Figures 65, 66 and 67 are diagrams showing the reference relationship between the position information and the attribute information.

[0376] When the encoded data has a hierarchical structure, layer 0 is a base layer and is data that can be decoded independently. On the other hand, layer 1 cannot be decoded independently, but is decoded after being integrated with data of layer 0. Also, layer 2 cannot be decoded independently, but is decoded after being integrated with data of layers 0 and 1. Also, when an octree-based encoding method is used, as shown in FIG. 65, layer 0 of the attribute information is decoded with reference to layer 0 of the position information. For example, as shown in FIG. 66, layer 1 of the attribute information is decoded with reference to layer 0 of the attribute information and layers 0 and 1 of the position information. When there is a reference relationship or a dependency relationship in the decoding, the referenced data is sent first. By transmitting the referenced data first, it becomes possible to decode the data in the order in which it is acquired in the three-dimensional data decoding device, and efficient decoding is possible, such as reducing the capacity of the receiving buffer.

[0377] A method of storing encoded data in a file format such as ISOBMFF will be described below. Fig. 68 is a diagram showing an example of the configuration of a bitstream. Fig. 69 is a diagram showing an example of the configuration of a three-dimensional data encoding device. The three-dimensional data encoding device includes an encoding unit 8801 and a file conversion unit 8802. The encoding unit 8801 generates a bitstream including encoded data and control information by encoding point cloud data. The file conversion unit 8802 converts the bitstream into a file format.

[0378] 70 is a diagram showing an example of the configuration of a three-dimensional data decoding device. The three-dimensional data decoding device includes a file inverse conversion unit 8811 and a decoding unit 8812. The file inverse conversion unit 8811 converts a file format into a bit stream including encoded data and control information. The decoding unit 8812 generates point cloud data by decoding the bit stream.

[0379] Fig. 71 is a diagram showing the basic structure of ISOBMFF. Fig. 72 is a protocol stack diagram when NAL units common to PCC codecs are stored in ISOBMFF. Here, the NAL units stored in ISOBMFF are PCC codec NAL units.

[0380] NAL units include NAL units for data and NAL units for metadata. NAL units for data include geometry slice data and attribute slice data. NAL units for metadata include SPS, GPS, APS, and SEI.

[0381] ISOBMFF (ISO based media file format) is a file format standard defined in ISO / IEC14496-12. It specifies a format that can store multiplexed data of various media such as video, audio, and text, and is a media-independent standard.

[0382] The basic unit in ISOBMFF is a box. A box consists of type, length, and data, and a collection of boxes of various types is a file. A file mainly consists of boxes such as ftyp, which indicates the brand of the file using 4CC, moov, which stores metadata such as control information, and mdat, which stores data.

[0383] For example, the method of storing AVC video and HEVC video is specified in ISO / IEC 14496-15. In addition, it is possible to extend the functionality of ISOBMFF to store and transmit PCC encoded data.

[0384] When storing a metadata NAL unit in ISOBMFF, the SEI may be stored in an "mdat box" together with PCC data, or in a "track box" that describes control information related to the stream. When data is packetized and transmitted, the SEI may be stored in a packet header. By indicating the SEI to the system layer, access to attribute information, tiles, and slice data becomes easier, thereby improving the access speed.

[0385] Next, a first example of format conversion of PCC hierarchical data will be described. An encoding method is a technology for compressing data. On the other hand, additional functions are provided in a system format and have a different role from the encoding method. Such additional functions are regulated by a standard different from that of the encoding method. In order to create an optimal format for providing the additional functions, the three-dimensional data encoding device converts the data. In doing so, the three-dimensional data encoding device stores information that makes the conversion easier in advance in the encoded data. This makes it possible to reduce the amount of processing involved in the conversion.

[0386] A method for converting data for each layer and hierarchical structure metadata including hierarchical information into a file format will be described below. Slice data such as position information and attribute information are stored in samples of the file format. The samples are stored in mdat. In addition, in order to access sample data, information indicating the data structure, offset information indicating the position of the data, and data length information are stored in metadata such as a random access table. Note that these pieces of information may be stored in a table different from the random access table.

[0387] The case where slice data is stored in a sample will be described below. FIG. 73 is a diagram showing a conversion process from a bit stream to a file format. The three-dimensional data encoding device stores position information slices and attribute information slices in a one-to-one correspondence with samples. Here, a slice includes information (hierarchical data) of all layers.

[0388] The position information sample belongs to a position information track (Geometry Track), and the attribute information sample belongs to an attribute information track (Attribute Track). The hierarchical information is stored in metadata for each frame belonging to a metadata track (Meta Data Track). In addition, information indicating that the position information sample and the attribute information sample belong to the same frame, information indicating whether the attribute information sample refers to the position information sample in the case of an octree-based encoding method, time stamp information common to the position information sample and the attribute information sample, etc. may be stored in the metadata sample.

[0389] A frame unit that operates with a common timestamp may be called an access unit. The layer information (layer_information) may be stored in the moov. Here, the layer information includes, for example, the layer information (layer_info) described above. The layer information may include at least a part of the above-mentioned hierarchical structure metadata such as depth information (depth_info) or other information included in the header.

[0390] When using this method, slices can be stored directly as samples, making processing easy.

[0391] 74 is a flowchart of the format conversion process. First, the three-dimensional data encoding device starts format conversion of encoded data (S8811). Next, the three-dimensional data encoding device stores one slice including multiple layers in one sample (S8812). In addition, the three-dimensional data encoding device stores hierarchical information in metadata (S8813). The three-dimensional data encoding device configures a frame (AU: access unit) (S8814).

[0392] Next, a partial decoding method using a file format will be described. Fig. 75 is a flowchart of this decoding process. First, the three-dimensional data decoding device extracts a desired sample by random access (S8821). Specifically, the three-dimensional data decoding device identifies the position of the desired sample using metadata included in the moov, a random access table, and the like, and extracts the data of the sample.

[0393] The three-dimensional data decoding device analyzes the hierarchical information metadata (S8822) and extracts layer boundary information within the sample (S8823). Specifically, the three-dimensional data decoding device acquires layer boundary information from the number of depths included in each layer within the sample and the data length of each depth from the hierarchical information metadata. For example, the three-dimensional data decoding device calculates the data length of each layer from the number of depths included in each layer and the data length of each depth, and determines the data boundary of the layer based on the calculated data length of each layer.

[0394] Next, the three-dimensional data decoding device divides the layers using the layer boundary information and decodes the desired data (S8824). For example, the three-dimensional data decoding device extracts a specific layer component from the sample.

[0395] In this way, by including hierarchical information in the metadata, a three-dimensional data decoding device can use the hierarchical information to extract specific information without decoding the encoded data.

[0396] A second example of format conversion of PCC hierarchical data will be described below. When storing hierarchical data in samples, the three-dimensional data encoding device may store one piece of hierarchical data as one sample or one sub-sample.

[0397] 76 is a diagram showing a conversion process from a bit stream to a file format. The three-dimensional data encoding device stores data for each layer among the position information slices and attribute information slices in one-to-one correspondence with samples.

[0398] The position information samples belong to the position information track (Geometry Track), and the attribute information samples belong to the attribute information track (Attribute Track). The hierarchical information is stored in the metadata for each frame that belongs to the metadata track (Meta Data Track). In addition, there is a track for each layer, and there are samples that belong to each track. By providing a track for each layer, it becomes easy to handle data for each layer.

[0399] Furthermore, since slice data in the bit stream of encoded data includes all hierarchical data, the three-dimensional data encoding device divides the data and stores it in samples. If hierarchical information is indicated in the bit stream, the three-dimensional data encoding device divides the slice data using information indicating the data length of each hierarchical data, etc. If hierarchical information is not indicated in the bit stream, the three-dimensional data encoding device calculates the hierarchical information while decoding the encoded data. The three-dimensional data encoding device re-encodes and divides the data based on the obtained hierarchical information.

[0400] This process allows information for each layer to be stored in the tracks and samples for each layer, and therefore allows data for each layer to be extracted in a three-dimensional data decoding device, making it easy to handle data for each layer.

[0401] 77 is a diagram illustrating an example of the syntax of hierarchical structure metadata. The hierarchical structure metadata includes layer information (layer_info), the number of components (component), and depth information (depth_info) of each component.

[0402] When dividing a slice into hierarchical data, the three-dimensional data encoding device may copy and add the slice header to all divided data if header information is to be included in the sample. FIG. 78 is a diagram showing this division process in schematic form. The three-dimensional data encoding device may store the slice header in the metadata instead of including it in the sample. Copying the header information can reduce the header creation process.

[0403] The three-dimensional data encoding device may also add an identifier to the file format indicating whether the data stored in the sample is hierarchical or not. If the data is hierarchical, the three-dimensional data encoding device may also add an identifier to the file format indicating whether the data includes all hierarchical data or whether the data stored in the sample is hierarchical data. The three-dimensional data encoding device may also indicate this information by a media type or a box type such as 4CC. This makes it easier to identify the media.

[0404] FIG. 79 is a flowchart of a conversion process using hierarchical information. First, the three-dimensional data encoding device starts format conversion of encoded data (S8831). Next, the three-dimensional data encoding device divides a slice into information for each layer using hierarchical information metadata (S8832). Next, the three-dimensional data encoding device stores each of the divided multiple pieces of hierarchical data in one sample (S8833). Next, the three-dimensional data encoding device stores the hierarchical information in metadata (S8834). Next, the three-dimensional data encoding device configures a frame (AU) (S8835).

[0405] FIG. 80 is a flowchart of a conversion process that does not use hierarchical information. First, the three-dimensional data encoding device starts format conversion of encoded data (S8841). Next, the three-dimensional data encoding device decodes the data and determines the boundary of the hierarchical data (S8842). Next, the three-dimensional data encoding device re-encodes and divides the data (S8843). Next, the three-dimensional data encoding device stores each of the divided multiple hierarchical data in one sample (S8844). Next, the three-dimensional data encoding device stores the hierarchical information in metadata (S8845). Next, the three-dimensional data encoding device configures a frame (AU) (S8846).

[0406] 81 is a flowchart of the decoding process of hierarchical data sample data. First, the three-dimensional data decoding device extracts a desired sample by random access (S8851). Next, the three-dimensional data decoding device decodes the data included in the extracted sample (S8852).

[0407] Next, another example of a hierarchical data structure will be described. Figures 82 and 83 are diagrams showing an example of the structure of the entire coded data (PCC data). The hierarchical structure in the example shown in Figures 82 and 83 is the same as that in Figure 59.

[0408] FIG. 82 shows a case where one piece of depth data is used as one piece of slice data, and a slice header is added to each piece of depth data. The slice header includes a depthId that identifies a layer of the depth data, a layerId that indicates a layer to which the depth belongs, and a length that indicates the length of the depth data. The slice header may also include a groupId that indicates that the data belongs to the same frame. That is, the groupId indicates the frame (time) to which the data belongs.

[0409] If these pieces of information are included in the slice header, the entire encoded data does not need to have hierarchical structure metadata. Also, the three-dimensional data encoding device may store parameters common to all depths in the header of the slice that transmits the first depth, or may store them in a common header and place them before the data of depth#0. Note that the three-dimensional data encoding device may store depthId and groupId in the slice header, and store the number of depths, and layerId and length for each depth in the hierarchical structure metadata or common header.

[0410] Also, depth#0 can be decoded independently, but depths other than depth#0 cannot be decoded independently and depend on other data. The three-dimensional data decoding device determines that data other than depth#0 cannot be decoded independently, and decodes the depth data to be decoded together with depth data that has the same groupId as the depth data to be decoded and a depthId smaller than the depthId of the depth data to be decoded.

[0411] FIG. 83 shows a case where one hierarchical data is used as one slice data, and a slice header is added to each hierarchical data. The slice header includes a layerId, the number of depths included in the layer (num_depth), and the length of the depth data (length). The slice header may also include a groupId indicating that the hierarchical data belongs to the same frame. The slice header includes a layerId and a groupId, and the number of layers, the number of depths included in each layer, and depth length information (length) may be included in the hierarchical structure metadata.

[0412] By using this structure, it becomes easy to divide the data into data for each layer, so the amount of processing required for division can be reduced. In addition, it becomes possible to transmit the divided data, so the amount of transmission can be reduced. In addition, it becomes possible to divide the location information and attribute information for each layer in the same way.

[0413] Next, a third example of format conversion of PCC hierarchical data will be described. When storing hierarchical data in samples, the three-dimensional data encoding device may store one piece of depth data as one sample, or may store one piece of depth data as one sub-sample.

[0414] 84 is a diagram showing a conversion process from a bit stream to a file format. The three-dimensional data encoding device stores data for each layer among the position information slices and attribute information slices in one-to-one correspondence with samples.

[0415] The position information samples belong to the position information track (Geometry Track), and the attribute information samples belong to the attribute information track (Attribute Track). The hierarchical information is stored in the metadata for each frame that belongs to the metadata track (Meta Data Track). There is a track for each layer, and there are samples that belong to each track. Providing a track for each layer makes it easy to handle data for each layer.

[0416] In addition, since slice data is configured for each hierarchical data in the bit stream of the encoded data, the three-dimensional data encoding device can directly store the data in the sample as it is. Therefore, the amount of processing can be reduced compared to a case where slice data is not configured for each hierarchical data. Moreover, the hierarchical information is stored in the metadata.

[0417] 85 is a flowchart of the format conversion process. First, the three-dimensional data encoding device starts format conversion of encoded data (S8861). Next, the three-dimensional data encoding device stores slice data for each layer in one sample (S8862). Next, the three-dimensional data encoding device stores layer information in metadata (S8863). Next, the three-dimensional data encoding device composes a frame (AU) (S8864).

[0418] Fig. 86 is a flowchart of the decoding process. First, the three-dimensional data decoding device analyzes the metadata to access specific hierarchical data and obtains the depth number belonging to the layer (S8871). Next, the three-dimensional data decoding device uses the obtained information to calculate the start position of the depth data at the beginning of the hierarchical data and the entire size of the layer (S8872). Next, the three-dimensional data decoding device decodes the hierarchical data (S8873).

[0419] Fig. 87 is a diagram showing an example of the syntax of depth information. Fig. 88 is a diagram showing an example of the syntax of a sample size box (sample_size_box:stsz). The three-dimensional data encoding device may store the size (entry_size) of each hierarchical data in the sample size box that stores size information for each sample.

[0420] Fig. 89 is a diagram showing an example of the syntax of layer information (layer_info). Fig. 90 is a diagram showing an example of the syntax of PCCLayerStructureBox. For example, as shown in Figs. 89 and 90, the three-dimensional data encoding device stores the number of layers (layer) and the number of depths included in the layers (num_depth) in PCCLayerStructureBox. Note that the three-dimensional data encoding device may store these pieces of information in the same box or in separate boxes.

[0421] Next, a process of extracting partial data from a file format will be described. The three-dimensional data decoding device randomly accesses partially decoded data from a file using the data structure and hierarchical structure metadata described in this embodiment, and extracts data. The three-dimensional data decoding device can access data and extract desired data based on the information on the frame, layer, and data length of each of the position information and attribute information included in the metadata, and the number of depths included in the layers.

[0422] FIG. 91 is a diagram showing a schematic diagram of this extraction operation. The sending unit 8821 has a file (file format) of complete data having layers 0 and 1, and the receiving unit 8823 has a data file of layer 0. In this state, when the receiving unit 8823 wants to acquire data of layer 1, it requests the sending unit 8821 to send the file of layer 1. The extracting unit 8822 included in the sending unit 8821 extracts the file of layer 1 from the file of complete data (file format) and provides the file of layer 1 (bit stream) to the receiving unit 8823. The receiving unit 8823 integrates the file of layer 0 and the file of layer 1 to generate a file of complete data.

[0423] FIG. 92 is a diagram showing an example of a complete data file (file format). FIG. 93, FIG. 94, and FIG. 95 are diagrams showing examples of bit streams extracted by the extraction unit 8822. For example, as shown in FIG. 93, the extraction unit 8822 may extract all data from the file format. Alternatively, as shown in FIG. 94, the extraction unit 8822 may extract position information but not attribute information. Alternatively, as shown in FIG. 95, the extraction unit 8822 may extract layer 0 but not layer 1. Alternatively, the extraction unit 8822 may transmit data in a rearranged state, for example (not shown).

[0424] By using the data structure and hierarchical structure metadata described in this embodiment, it becomes easy to divide hierarchical data, so it is possible to realize a function of acquiring necessary data and not acquiring unnecessary data. This makes it possible to reduce the transmission band and transmission delay, and improve the functionality of data transmission.

[0425] Next, partial decoding processing of direct mode data will be explained. Direct mode is a method for encoding using an octree, where the octree encoding is stopped for a certain node, and the coordinates of the point of the leaf node are directly encoded. For example, direct mode is used when the points belonging to a node are sparse. By using direct mode, the amount of data can be reduced.

[0426] FIG. 96 is a diagram showing an example of direct mode. There are two leaf points (sparse) in the area surrounded by the dotted line in FIG. 96. For example, in the A node at depth=1, if it is determined that the points are sparse, the coordinates of these two points are directly written in the data area of ​​depth1. The coordinates of the two points are coordinates from the A node, which are coordinates with a resolution of depth=4. In other words, the data of depth=1 includes the occupancy code (occpancy_code) of depth=1 and the coordinate (area B) data in the direct mode of depth=4 surrounded by the dotted line.

[0427] Next, a method for decoding such data will be described. When decoding from depth0 to depth4, the three-dimensional data decoding device performs decoding processing using a normal decoding method. On the other hand, when partially extracting and decoding depth0 to depth1, if it is necessary to make all decoded data have the same resolution, the three-dimensional data decoding device may perform decoding without using the coordinates of depth4 included in depth1. Furthermore, if it is acceptable to mix in higher resolutions, the three-dimensional data decoding device may perform decoding using the coordinates of depth4 included in depth1.

[0428] Next, data division when the direct mode is used will be described. When extracting and transmitting data from depth0 to depth1, the transmitting device may or may not include information on the direct mode (depth4) in depth1.

[0429] Furthermore, the transmitting device may determine whether to include direct mode information depending on whether the receiving device requires the information, and if the receiving device requires the information, include the information in the bit stream, and if the information is not required, do not include the information in the bit stream. This allows the amount of data to be reduced. For example, the transmitting device may determine that direct mode information is necessary when data with resolutions of depth2 to depth4 may be decoded together with depth0 to depth1, and may determine that direct mode information is not necessary when depth2 to depth4 is not decoded.

[0430] As described above, the three-dimensional data encoding device according to this embodiment performs the process shown in FIG. 97. The three-dimensional data encoding device sets a hierarchical structure having a plurality of depths and a plurality of hierarchical layers each including one or more depths for a plurality of pieces of position information of a plurality of three-dimensional points included in the point cloud data (S8881). The three-dimensional data encoding device generates a plurality of first encoded data (e.g., depth data) for each depth by encoding the plurality of pieces of position information for each depth (S8882). The three-dimensional data encoding device generates a bit stream including a plurality of second encoded data, which is encoded data for each layer and includes one or more pieces of first encoded data for one or more depths included in the corresponding layer (S8883). The bit stream includes first information indicating the data length of each of the plurality of second encoded data.

[0431] According to this, a three-dimensional data decoding device that decodes a bit stream can easily access data of an arbitrary layer by using the first information. Therefore, the three-dimensional data encoding device can reduce the amount of processing of the three-dimensional data decoding device.

[0432] For example, the first information includes second information (eg, num_depth) indicating the number of depths included in each of the multiple layers, and third information (eg, length) indicating the data length of each of the multiple first encoded data.

[0433] For example, the bit stream includes a first header (for example, the hierarchical structure metadata or header shown in FIG. 59) that is common to a plurality of second encoded data, and the first header includes the first information.

[0434] For example, the bit stream includes a plurality of second headers (e.g., a slice header shown in FIG. 83) for each second encoded data, the first information corresponds to one of the plurality of second encoded data and includes a plurality of fourth information indicating the data length of the corresponding second encoded data, and each of the plurality of second headers includes fourth information indicating the data length of the second encoded data corresponding to the second header.

[0435] For example, the bit stream includes multiple third headers (e.g., slice headers shown in FIG. 82) for each of multiple first encoded data, the first information includes second information (e.g., num_depth) indicating the number of depths included in each of the multiple hierarchical layers, and fifth information (e.g., length) corresponding to each of the multiple first encoded data and indicating the data length of the corresponding first encoded data, and each of the multiple third headers includes the fifth information indicating the data length of the first encoded data corresponding to the third header.

[0436] For example, the three-dimensional data encoding device further generates multiple third encoded data for each depth by encoding multiple attribute information possessed by multiple three-dimensional points for each depth, and the bit stream includes multiple fourth encoded data, which is encoded data for each layer and includes one or more third encoded data for one or more depths included in the corresponding layer, and the bit stream includes sixth information indicating the data length of each of the multiple fourth encoded data.

[0437] For example, the three-dimensional data encoding device includes a processor and a memory, and the processor uses the memory to perform the above-mentioned processing.

[0438] Moreover, the three-dimensional data decoding device according to this embodiment performs the process shown in FIG. 98. The three-dimensional data decoding device acquires the first information from a bit stream including a plurality of second encoded data and first information indicating the data length of each of the plurality of second encoded data (S8886). The three-dimensional data decoding device acquires at least one of the plurality of second encoded data using the first information (S8887). The three-dimensional data decoding device decodes the acquired at least one of the second encoded data (S8888). The bit stream includes a plurality of first encoded data for each depth that is generated by encoding, for each depth, a plurality of position information of a plurality of three-dimensional points included in the point cloud data, in which a hierarchical structure having a plurality of depths and a plurality of hierarchies each including one or more depths is set. Each of the plurality of second encoded data corresponds to one of the plurality of hierarchies, and includes one or more first encoded data included in the hierarchies corresponding to the second encoded data among the plurality of first encoded data.

[0439] This allows the three-dimensional data decoding device to easily access data at any layer using the first information, thereby reducing the amount of processing.

[0440] For example, the first information includes second information (e.g., num_depth) indicating the number of depths included in each of the multiple layers, and third information (e.g., length) indicating the data length of each of the multiple first encoded data. For example, the three-dimensional data decoding device calculates the data length of the second encoded data using the second information and the third information.

[0441] For example, the bit stream includes a first header (for example, the hierarchical structure metadata or header shown in FIG. 59) that is common to a plurality of second encoded data, and the first header includes the first information.

[0442] For example, the bit stream includes a plurality of second headers (e.g., a slice header shown in FIG. 83) for each second encoded data, the first information corresponds to one of the plurality of second encoded data and includes a plurality of fourth information indicating the data length of the corresponding second encoded data, and each of the plurality of second headers includes fourth information indicating the data length of the second encoded data corresponding to the second header.

[0443] For example, the bit stream includes multiple third headers (e.g., slice headers shown in FIG. 82) for each of multiple first encoded data, the first information includes second information (e.g., num_depth) indicating the number of depths included in each of the multiple hierarchical layers, and fifth information (e.g., length) corresponding to each of the multiple first encoded data and indicating the data length of the corresponding first encoded data, and each of the multiple third headers includes the fifth information indicating the data length of the first encoded data corresponding to the third header.

[0444] For example, the bit stream includes a plurality of third encoded data for each depth generated by encoding a plurality of attribute information possessed by a plurality of three-dimensional points for each depth. The bit stream includes a plurality of fourth encoded data, which is encoded data for each layer and includes one or more third encoded data for one or more depths included in a corresponding layer. The bit stream includes sixth information indicating a data length of each of the plurality of fourth encoded data. The three-dimensional data decoding device further obtains the sixth information from the bit stream, obtains at least one fourth encoded data from the plurality of fourth encoded data using the sixth information, and decodes the obtained at least one fourth encoded data.

[0445] For example, the three-dimensional data decoding device includes a processor and a memory, and the processor uses the memory to perform the above-mentioned processing.

[0446] (Embodiment 6) Next, a configuration of a three-dimensional data creation device 810 according to this embodiment will be described. Fig. 99 is a block diagram showing an example of the configuration of a three-dimensional data creation device 810 according to this embodiment. This three-dimensional data creation device 810 is mounted on a vehicle, for example. The three-dimensional data creation device 810 transmits and receives three-dimensional data to and from an external traffic monitoring cloud, a leading vehicle, or a following vehicle, and creates and stores three-dimensional data.

[0447] The three-dimensional data creation device 810 includes a data receiving unit 811, a communication unit 812, a receiving control unit 813, a format conversion unit 814, multiple sensors 815, a three-dimensional data creation unit 816, a three-dimensional data synthesis unit 817, a three-dimensional data storage unit 818, a communication unit 819, a transmission control unit 820, a format conversion unit 821, and a data transmission unit 822.

[0448] The data receiving unit 811 receives three-dimensional data 831 from a traffic monitoring cloud or a preceding vehicle. The three-dimensional data 831 includes information such as a point cloud, a visible light image, depth information, sensor position information, or speed information, including an area that cannot be detected by the sensor 815 of the vehicle itself.

[0449] The communication unit 812 communicates with the traffic monitoring cloud or the vehicle ahead, and transmits data transmission requests, etc. to the traffic monitoring cloud or the vehicle ahead.

[0450] The reception control unit 813 exchanges information such as compatible formats with the communication destination via the communication unit 812, and establishes communication with the communication destination.

[0451] The format conversion unit 814 generates three-dimensional data 832 by performing format conversion or the like on the three-dimensional data 831 received by the data receiving unit 811. Furthermore, when the three-dimensional data 831 is compressed or encoded, the format conversion unit 814 performs decompression or decoding processing.

[0452] The multiple sensors 815 are a group of sensors such as LiDAR, a visible light camera, or an infrared camera that acquires information about the outside of the vehicle, and generate sensor information 833. For example, when the sensor 815 is a laser sensor such as LiDAR, the sensor information 833 is three-dimensional data such as a point cloud (point cloud data). Note that the number of sensors 815 does not need to be multiple.

[0453] The three-dimensional data creation unit 816 generates three-dimensional data 834 from the sensor information 833. The three-dimensional data 834 includes information such as a point cloud, a visible light image, depth information, sensor position information, or velocity information.

[0454] The three-dimensional data synthesis unit 817 synthesizes three-dimensional data 834 created based on the host vehicle's sensor information 833 with three-dimensional data 832 created by a traffic monitoring cloud or a preceding vehicle, etc., to construct three-dimensional data 835 that includes the space in front of the preceding vehicle that cannot be detected by the host vehicle's sensor 815.

[0455] The three-dimensional data storage unit 818 stores the generated three-dimensional data 835 and the like.

[0456] The communication unit 819 communicates with the traffic monitoring cloud or the following vehicle, and transmits data transmission requests, etc. to the traffic monitoring cloud or the following vehicle.

[0457] The transmission control unit 820 exchanges information such as compatible formats with the communication destination and establishes communication with the communication destination via the communication unit 819. In addition, the transmission control unit 820 determines a transmission region, which is the space of the three-dimensional data to be transmitted, based on three-dimensional data construction information of the three-dimensional data 832 generated by the three-dimensional data synthesis unit 817 and a data transmission request from the communication destination.

[0458] Specifically, the transmission control unit 820 determines a transmission region including a space in front of the vehicle that cannot be detected by the sensor of the following vehicle in response to a data transmission request from the traffic monitoring cloud or the following vehicle. The transmission control unit 820 also determines the transmission region by judging whether or not a transmittable space or a transmitted space has been updated based on the three-dimensional data construction information. For example, the transmission control unit 820 determines the region specified in the data transmission request and in which the corresponding three-dimensional data 835 exists as the transmission region. Then, the transmission control unit 820 notifies the format conversion unit 821 of the format supported by the communication destination and the transmission region.

[0459] The format conversion unit 821 converts three-dimensional data 836 of the transmission region, among the three-dimensional data 835 stored in the three-dimensional data storage unit 818, into a format supported by the receiving side, thereby generating three-dimensional data 837. Note that the format conversion unit 821 may reduce the amount of data by compressing or encoding the three-dimensional data 837.

[0460] The data transmission unit 822 transmits the three-dimensional data 837 to the traffic monitoring cloud or the following vehicle. The three-dimensional data 837 includes information such as a point cloud, visible light image, depth information, or sensor position information in front of the vehicle, including blind spots of the following vehicle.

[0461] In the above description, the format conversion units 814 and 821 perform format conversion, but the format conversion does not necessarily have to be performed.

[0462] With this configuration, the three-dimensional data creation device 810 externally acquires three-dimensional data 831 of an area that cannot be detected by the sensor 815 of the host vehicle, and generates three-dimensional data 835 by synthesizing the three-dimensional data 831 with three-dimensional data 834 based on sensor information 833 detected by the sensor 815 of the host vehicle. In this way, the three-dimensional data creation device 810 can generate three-dimensional data of an area that cannot be detected by the sensor 815 of the host vehicle.

[0463] In addition, in response to a data transmission request from a traffic monitoring cloud or a following vehicle, the three-dimensional data creation device 810 can transmit three-dimensional data including the space in front of the vehicle that cannot be detected by the sensors of the following vehicle to the traffic monitoring cloud or the following vehicle, etc.

[0464] Next, there will be described a procedure for transmitting three-dimensional data to a following vehicle in the three-dimensional data creation device 810. Fig. 100 is a flowchart showing an example of a procedure for transmitting three-dimensional data to a traffic monitoring cloud or a following vehicle by the three-dimensional data creation device 810.

[0465] First, the three-dimensional data creation device 810 generates and updates three-dimensional data 835 of a space including the space on the road ahead of the vehicle (S801). Specifically, the three-dimensional data creation device 810 constructs three-dimensional data 835 including the space ahead of the vehicle ahead that cannot be detected by the sensor 815 of the vehicle ahead, for example, by combining three-dimensional data 834 created based on the sensor information 833 of the vehicle ahead with three-dimensional data 831 created by the traffic monitoring cloud or the vehicle ahead.

[0466] Next, the three-dimensional data creation device 810 determines whether the three-dimensional data 835 included in the transmitted space has changed (S802).

[0467] If a change occurs in the three-dimensional data 835 contained in the space that has already been transmitted, such as when a vehicle or person enters the space from outside (Yes in S802), the three-dimensional data creation device 810 transmits three-dimensional data including the three-dimensional data 835 of the space where the change has occurred to the traffic monitoring cloud or a following vehicle (S803).

[0468] The three-dimensional data creation device 810 may transmit the three-dimensional data of the space where a change has occurred in accordance with the transmission timing of the three-dimensional data transmitted at a predetermined interval, or may transmit the three-dimensional data immediately after detecting the change. In other words, the three-dimensional data creation device 810 may transmit the three-dimensional data of the space where a change has occurred in priority over the three-dimensional data transmitted at a predetermined interval.

[0469] In addition, the three-dimensional data creation device 810 may transmit all of the three-dimensional data of the space in which a change has occurred as the three-dimensional data of the space in which a change has occurred, or may transmit only the differences in the three-dimensional data (for example, information on three-dimensional points that have appeared or disappeared, or displacement information of three-dimensional points, etc.).

[0470] Furthermore, the three-dimensional data creation device 810 may transmit metadata regarding the danger avoidance operation of the vehicle, such as a sudden braking warning, to the following vehicle prior to the three-dimensional data of the space where a change has occurred. This allows the following vehicle to quickly recognize the sudden braking of the vehicle ahead, and to start danger avoidance operations, such as deceleration, earlier.

[0471] If there is no change in the three-dimensional data 835 contained in the transmitted space (No in S802), or after step S803, the three-dimensional data creation device 810 transmits the three-dimensional data contained in a space of a specified shape at a distance L ahead of the vehicle to the traffic monitoring cloud or a following vehicle (S804).

[0472] Furthermore, for example, the processes in steps S801 to S804 are repeatedly performed at predetermined time intervals.

[0473] Furthermore, when there is no difference between the three-dimensional data 835 of the space currently being transmitted and the three-dimensional map, the three-dimensional data creation device 810 does not need to transmit the three-dimensional data 837 of the space.

[0474] In this embodiment, the client device transmits sensor information obtained by the sensor to a server or another client device.

[0475] First, the configuration of the system according to this embodiment will be described. Fig. 101 is a diagram showing the configuration of a transmission / reception system for a 3D map and sensor information according to this embodiment. This system includes a server 901 and client devices 902A and 902B. When there is no particular need to distinguish between the client devices 902A and 902B, they are also referred to as client devices 902.

[0476] The client device 902 is, for example, an in-vehicle device mounted on a moving object such as a vehicle. The server 901 is, for example, a traffic monitoring cloud or the like, and is capable of communicating with a plurality of client devices 902.

[0477] The server 901 transmits a three-dimensional map composed of a point cloud to the client device 902. Note that the composition of the three-dimensional map is not limited to a point cloud, and may represent other three-dimensional data such as a mesh structure.

[0478] The client device 902 transmits sensor information acquired by the client device 902 to the server 901. The sensor information includes, for example, at least one of LiDAR acquisition information, a visible light image, an infrared image, a depth image, sensor position information, and speed information.

[0479] Data transmitted between the server 901 and the client device 902 may be compressed to reduce data, or may be left uncompressed to maintain data accuracy. When compressing data, a three-dimensional compression method based on an octree structure, for example, can be used for the point cloud. Also, a two-dimensional image compression method can be used for the visible light image, the infrared image, and the depth image. The two-dimensional image compression method is, for example, MPEG-4 AVC or HEVC standardized by MPEG.

[0480] Furthermore, the server 901 transmits the three-dimensional map managed by the server 901 to the client device 902 in response to a transmission request for the three-dimensional map from the client device 902. Note that the server 901 may transmit the three-dimensional map without waiting for a transmission request for the three-dimensional map from the client device 902. For example, the server 901 may broadcast the three-dimensional map to one or more client devices 902 in a predetermined space. Furthermore, the server 901 may transmit a three-dimensional map suitable for the position of the client device 902 at regular intervals to the client device 902 that has received a transmission request once. Furthermore, the server 901 may transmit the three-dimensional map to the client device 902 every time the three-dimensional map managed by the server 901 is updated.

[0481] The client device 902 issues a request for transmitting a three-dimensional map to the server 901. For example, when the client device 902 wants to estimate its own position while driving, the client device 902 transmits a request for transmitting a three-dimensional map to the server 901.

[0482] In the following cases, the client device 902 may issue a transmission request for a three-dimensional map to the server 901. If the three-dimensional map held by the client device 902 is old, the client device 902 may issue a transmission request for a three-dimensional map to the server 901. For example, if a certain period of time has passed since the client device 902 acquired the three-dimensional map, the client device 902 may issue a transmission request for a three-dimensional map to the server 901.

[0483] The client device 902 may issue a request to send a three-dimensional map to the server 901 a certain time before the client device 902 leaves the space shown in the three-dimensional map held by the client device 902. For example, when the client device 902 is within a predetermined distance from the boundary of the space shown in the three-dimensional map held by the client device 902, the client device 902 may issue a request to send a three-dimensional map to the server 901. In addition, when the movement route and movement speed of the client device 902 are known, the time when the client device 902 will leave the space shown in the three-dimensional map held by the client device 902 may be predicted based on these.

[0484] If the error in aligning the three-dimensional data created by the client device 902 from sensor information with the three-dimensional map is equal to or greater than a certain level, the client device 902 may issue a request to the server 901 to transmit the three-dimensional map.

[0485] The client device 902 transmits the sensor information to the server 901 in response to a request to transmit the sensor information transmitted from the server 901. The client device 902 may transmit the sensor information to the server 901 without waiting for a request to transmit the sensor information from the server 901. For example, once the client device 902 receives a request to transmit the sensor information from the server 901, the client device 902 may transmit the sensor information to the server 901 periodically for a certain period of time. In addition, when an error occurs in aligning three-dimensional data created by the client device 902 based on the sensor information with the three-dimensional map obtained from the server 901, the client device 902 may determine that a change may have occurred in the three-dimensional map around the client device 902 and transmit a message to that effect along with the sensor information to the server 901.

[0486] The server 901 issues a transmission request for sensor information to the client device 902. For example, the server 901 receives location information of the client device 902, such as a GPS, from the client device 902. When the server 901 determines, based on the location information of the client device 902, that the client device 902 is approaching a space with little information in the three-dimensional map managed by the server 901, it issues a transmission request for sensor information to the client device 902 in order to generate a new three-dimensional map. The server 901 may also issue a transmission request for sensor information when it is desired to update the three-dimensional map, when it is desired to check road conditions such as during snowfall or a disaster, when it is desired to check traffic congestion conditions, or when it is desired to check incident and accident conditions, etc.

[0487] Furthermore, the client device 902 may set the amount of data of the sensor information to be transmitted to the server 901 depending on the communication state or bandwidth at the time of receiving a request to transmit the sensor information from the server 901. Setting the amount of data of the sensor information to be transmitted to the server 901 means, for example, increasing or decreasing the amount of the data itself or appropriately selecting a compression method.

[0488] 102 is a block diagram showing an example of the configuration of a client device 902. The client device 902 receives a three-dimensional map composed of a point cloud or the like from the server 901, and estimates the self-position of the client device 902 from three-dimensional data created based on sensor information of the client device 902. In addition, the client device 902 transmits the acquired sensor information to the server 901.

[0489] The client device 902 includes a data receiving unit 1011, a communication unit 1012, a reception control unit 1013, a format conversion unit 1014, multiple sensors 1015, a three-dimensional data creation unit 1016, a three-dimensional image processing unit 1017, a three-dimensional data storage unit 1018, a format conversion unit 1019, a communication unit 1020, a transmission control unit 1021, and a data transmission unit 1022.

[0490] The data receiving unit 1011 receives a three-dimensional map 1031 from the server 901. The three-dimensional map 1031 is data including a point cloud such as a WLD or a SWLD. The three-dimensional map 1031 may include either compressed data or uncompressed data.

[0491] The communication unit 1012 communicates with the server 901, and transmits data transmission requests (for example, requests to transmit a three-dimensional map) to the server 901.

[0492] The reception control unit 1013 exchanges information such as compatible formats with the communication destination via the communication unit 1012, and establishes communication with the communication destination.

[0493] The format conversion unit 1014 generates a three-dimensional map 1032 by performing format conversion and the like on the three-dimensional map 1031 received by the data receiving unit 1011. Furthermore, if the three-dimensional map 1031 is compressed or encoded, the format conversion unit 1014 performs decompression or decoding processing. Note that if the three-dimensional map 1031 is uncompressed data, the format conversion unit 1014 does not perform decompression or decoding processing.

[0494] The multiple sensors 1015 are a group of sensors, such as LiDAR, a visible light camera, an infrared camera, or a depth sensor, that acquire information about the outside of the vehicle in which the client device 902 is mounted, and generate sensor information 1033. For example, when the sensor 1015 is a laser sensor such as LiDAR, the sensor information 1033 is three-dimensional data such as a point cloud (point cloud data). Note that the number of sensors 1015 does not need to be multiple.

[0495] The three-dimensional data creation unit 1016 creates three-dimensional data 1034 of the surroundings of the vehicle based on the sensor information 1033. For example, the three-dimensional data creation unit 1016 creates point cloud data with color information of the surroundings of the vehicle using information acquired by the LiDAR and a visible light image acquired by a visible light camera.

[0496] The three-dimensional image processing unit 1017 performs self-position estimation processing of the vehicle using a three-dimensional map 1032 such as a received point cloud and three-dimensional data 1034 of the surroundings of the vehicle generated from the sensor information 1033. The three-dimensional image processing unit 1017 may generate three-dimensional data 1035 of the surroundings of the vehicle by combining the three-dimensional map 1032 and the three-dimensional data 1034, and perform self-position estimation processing using the generated three-dimensional data 1035.

[0497] The three-dimensional data storage unit 1018 stores a three-dimensional map 1032, three-dimensional data 1034, and three-dimensional data 1035, and the like.

[0498] The format conversion unit 1019 generates sensor information 1037 by converting the sensor information 1033 into a format supported by the receiving side. The format conversion unit 1019 may reduce the amount of data by compressing or encoding the sensor information 1037. Furthermore, the format conversion unit 1019 may omit processing when format conversion is not necessary. Furthermore, the format conversion unit 1019 may control the amount of data to be transmitted in accordance with a designated transmission range.

[0499] The communication unit 1020 communicates with the server 901, and receives data transmission requests (sensor information transmission requests) and the like from the server 901.

[0500] The transmission control unit 1021 exchanges information such as compatible formats with the communication destination via the communication unit 1020, and establishes communication.

[0501] The data transmission unit 1022 transmits the sensor information 1037 to the server 901. The sensor information 1037 includes information acquired by a plurality of sensors 1015, such as information acquired by a LiDAR, a luminance image acquired by a visible light camera, an infrared image acquired by an infrared camera, a depth image acquired by a depth sensor, sensor position information, and speed information.

[0502] Next, the configuration of the server 901 will be described. Fig. 103 is a block diagram showing an example of the configuration of the server 901. The server 901 receives sensor information transmitted from the client device 902, and creates three-dimensional data based on the received sensor information. The server 901 uses the created three-dimensional data to update a three-dimensional map managed by the server 901. In addition, the server 901 transmits the updated three-dimensional map to the client device 902 in response to a transmission request for the three-dimensional map from the client device 902.

[0503] The server 901 includes a data receiving unit 1111, a communication unit 1112, a receiving control unit 1113, a format conversion unit 1114, a three-dimensional data creation unit 1116, a three-dimensional data synthesis unit 1117, a three-dimensional data storage unit 1118, a format conversion unit 1119, a communication unit 1120, a transmission control unit 1121, and a data transmission unit 1122.

[0504] The data receiving unit 1111 receives sensor information 1037 from the client device 902. The sensor information 1037 includes, for example, information acquired by a LiDAR, a luminance image acquired by a visible light camera, an infrared image acquired by an infrared camera, a depth image acquired by a depth sensor, sensor position information, and speed information.

[0505] The communication unit 1112 communicates with the client device 902 and transmits a data transmission request (for example, a request to transmit sensor information) to the client device 902 .

[0506] The reception control unit 1113 exchanges information such as compatible formats with the communication destination via the communication unit 1112, and establishes communication.

[0507] When the received sensor information 1037 is compressed or encoded, the format conversion unit 1114 performs decompression or decoding processing to generate the sensor information 1132. Note that, when the sensor information 1037 is non-compressed data, the format conversion unit 1114 does not perform decompression or decoding processing.

[0508] The three-dimensional data creation unit 1116 creates three-dimensional data 1134 of the periphery of the client device 902 based on the sensor information 1132. For example, the three-dimensional data creation unit 1116 creates point cloud data with color information of the periphery of the client device 902 using information acquired by the LiDAR and visible light images acquired by a visible light camera.

[0509] A three-dimensional data synthesis unit 1117 synthesizes three-dimensional data 1134 created based on the sensor information 1132 with a three-dimensional map 1135 managed by the server 901, thereby updating the three-dimensional map 1135.

[0510] The three-dimensional data storage unit 1118 stores a three-dimensional map 1135 and the like.

[0511] The format conversion unit 1119 generates the three-dimensional map 1031 by converting the three-dimensional map 1135 into a format supported by the receiving side. The format conversion unit 1119 may reduce the amount of data by compressing or encoding the three-dimensional map 1135. Furthermore, the format conversion unit 1119 may omit processing when format conversion is not necessary. Furthermore, the format conversion unit 1119 may control the amount of data to be transmitted in accordance with a designated transmission range.

[0512] The communication unit 1120 communicates with the client device 902 and receives a data transmission request (a request to transmit a three-dimensional map) or the like from the client device 902 .

[0513] A transmission control unit 1121 exchanges information such as compatible formats with a communication destination via a communication unit 1120, and establishes communication.

[0514] The data transmission unit 1122 transmits the three-dimensional map 1031 to the client device 902. The three-dimensional map 1031 is data including a point cloud such as a WLD or a SWLD. The three-dimensional map 1031 may include either compressed data or uncompressed data.

[0515] Next, a description will be given of the operation flow of the client device 902. Fig. 104 is a flowchart showing the operation of the client device 902 when acquiring a three-dimensional map.

[0516] First, the client device 902 requests the server 901 to transmit a three-dimensional map (such as a point cloud) (S1001). At this time, the client device 902 may also transmit position information of the client device 902 obtained by GPS or the like, thereby requesting the server 901 to transmit a three-dimensional map related to the position information.

[0517] Next, the client device 902 receives the three-dimensional map from the server 901 (S1002). If the received three-dimensional map is compressed data, the client device 902 decodes the received three-dimensional map to generate an uncompressed three-dimensional map (S1003).

[0518] Next, the client device 902 creates three-dimensional data 1034 of the periphery of the client device 902 from sensor information 1033 obtained by the multiple sensors 1015 (S1004). Next, the client device 902 estimates its own position using the three-dimensional map 1032 received from the server 901 and the three-dimensional data 1034 created from the sensor information 1033 (S1005).

[0519] 105 is a flowchart showing an operation of the client device 902 when transmitting sensor information. First, the client device 902 receives a request to transmit sensor information from the server 901 (S1011). The client device 902 that has received the transmission request transmits sensor information 1037 to the server 901 (S1012). Note that, when the sensor information 1033 includes a plurality of pieces of information obtained by a plurality of sensors 1015, the client device 902 may generate the sensor information 1037 by compressing each piece of information using a compression method suitable for each piece of information.

[0520] Next, the operation flow of the server 901 will be described. Fig. 106 is a flowchart showing the operation when the server 901 acquires sensor information. First, the server 901 requests the client device 902 to transmit sensor information (S1021). Next, the server 901 receives the sensor information 1037 transmitted from the client device 902 in response to the request (S1022). Next, the server 901 creates three-dimensional data 1134 using the received sensor information 1037 (S1023). Next, the server 901 reflects the created three-dimensional data 1134 in the three-dimensional map 1135 (S1024).

[0521] Fig. 107 is a flowchart showing the operation of the server 901 when transmitting a three-dimensional map. First, the server 901 receives a request to transmit a three-dimensional map from the client device 902 (S1031). The server 901 that has received the request to transmit the three-dimensional map transmits a three-dimensional map 1031 to the client device 902 (S1032). At this time, the server 901 may extract a three-dimensional map of the vicinity according to the position information of the client device 902, and transmit the extracted three-dimensional map. The server 901 may also compress the three-dimensional map composed of a point cloud using, for example, a compression method based on an octet structure, and transmit the compressed three-dimensional map.

[0522] A modification of this embodiment will now be described.

[0523] The server 901 creates three-dimensional data 1134 of the vicinity of the position of the client device 902 using the sensor information 1037 received from the client device 902. Next, the server 901 calculates the difference between the three-dimensional data 1134 and the three-dimensional map 1135 by matching the created three-dimensional data 1134 with a three-dimensional map 1135 of the same area managed by the server 901. If the difference is equal to or greater than a predetermined threshold, the server 901 determines that some abnormality has occurred in the vicinity of the client device 902. For example, when ground subsidence or the like occurs due to a natural disaster such as an earthquake, it is considered that a large difference occurs between the three-dimensional map 1135 managed by the server 901 and the three-dimensional data 1134 created based on the sensor information 1037.

[0524] The sensor information 1037 may include information indicating at least one of the type of sensor, the performance of the sensor, and the model number of the sensor. In addition, a class ID or the like according to the performance of the sensor may be added to the sensor information 1037. For example, when the sensor information 1037 is information acquired by LiDAR, it is possible to assign an identifier to the performance of the sensor, such as class 1 for a sensor that can acquire information with an accuracy of several millimeters, class 2 for a sensor that can acquire information with an accuracy of several centimeters, and class 3 for a sensor that can acquire information with an accuracy of several meters. In addition, the server 901 may estimate the performance information of the sensor from the model number of the client device 902. For example, when the client device 902 is mounted on a vehicle, the server 901 may determine the spec information of the sensor from the model of the vehicle. In this case, the server 901 may acquire information on the model of the vehicle in advance, or the information may be included in the sensor information. In addition, the server 901 may use the acquired sensor information 1037 to switch the degree of correction for the three-dimensional data 1134 created using the sensor information 1037. For example, when the sensor performance is high accuracy (class 1), the server 901 does not perform correction on the three-dimensional data 1134. When the sensor performance is low accuracy (class 3), the server 901 applies correction according to the accuracy of the sensor to the three-dimensional data 1134. For example, the server 901 increases the degree (strength) of correction as the accuracy of the sensor decreases.

[0525] The server 901 may simultaneously issue requests to send sensor information to multiple client devices 902 in a certain space. When the server 901 receives multiple pieces of sensor information from multiple client devices 902, the server 901 does not need to use all of the sensor information to create the three-dimensional data 1134, and may select the sensor information to be used according to the performance of the sensor, for example. For example, when updating the three-dimensional map 1135, the server 901 may select high-precision sensor information (class 1) from the multiple pieces of sensor information received, and create the three-dimensional data 1134 using the selected sensor information.

[0526] The server 901 is not limited to a server such as a traffic monitoring cloud, but may be another client device (mounted in a vehicle). Figure 108 is a diagram showing the system configuration in this case.

[0527] For example, client device 902C issues a request to send sensor information to nearby client device 902A and acquires the sensor information from client device 902A. Client device 902C then creates three-dimensional data using the acquired sensor information of client device 902A and updates the three-dimensional map of client device 902C. This allows client device 902C to generate a three-dimensional map of the space that can be acquired from client device 902A by taking advantage of the performance of client device 902C. For example, it is considered that such a case occurs when client device 902C has high performance.

[0528] In this case, the client device 902A that provided the sensor information is given the right to obtain the high-precision 3D map generated by the client device 902C. The client device 902A receives the high-precision 3D map from the client device 902C in accordance with the right.

[0529] In addition, the client device 902C may issue a request to transmit sensor information to multiple nearby client devices 902 (client device 902A and client device 902B). If the sensor of the client device 902A or the client device 902B is high performance, the client device 902C can create three-dimensional data using the sensor information obtained by the high performance sensor.

[0530] 109 is a block diagram showing the functional configuration of a server 901 and a client device 902. The server 901 includes, for example, a 3D map compression / decoding processing unit 1201 that compresses and decodes a 3D map, and a sensor information compression / decoding processing unit 1202 that compresses and decodes sensor information.

[0531] The client device 902 includes a three-dimensional map decoding processing unit 1211 and a sensor information compression processing unit 1212. The three-dimensional map decoding processing unit 1211 receives encoded data of the compressed three-dimensional map and decodes the encoded data to obtain the three-dimensional map. The sensor information compression processing unit 1212 compresses the sensor information itself instead of the three-dimensional data created from the acquired sensor information, and transmits the encoded data of the compressed sensor information to the server 901. With this configuration, the client device 902 only needs to internally hold a processing unit (device or LSI) that performs processing to decode the three-dimensional map (point cloud, etc.), and does not need to internally hold a processing unit that performs processing to compress the three-dimensional data of the three-dimensional map (point cloud, etc.). This makes it possible to reduce the cost and power consumption of the client device 902.

[0532] As described above, the client device 902 according to this embodiment is mounted on a moving object, and creates three-dimensional data 1034 of the surroundings of the moving object from sensor information 1033 indicating the surrounding conditions of the moving object, obtained by a sensor 1015 mounted on the moving object. The client device 902 estimates the self-position of the moving object using the created three-dimensional data 1034. The client device 902 transmits the acquired sensor information 1033 to the server 901 or another client device 902.

[0533] According to this, the client device 902 transmits the sensor information 1033 to the server 901 or the like. This may reduce the amount of data to be transmitted compared to the case of transmitting three-dimensional data. Also, since the client device 902 does not need to perform processing such as compression or encoding of the three-dimensional data, the amount of processing by the client device 902 can be reduced. Therefore, the client device 902 can reduce the amount of data to be transmitted or simplify the device configuration.

[0534] Moreover, the client device 902 further transmits a request to the server 901 to transmit a three-dimensional map, and receives a three-dimensional map 1031 from the server 901. The client device 902 estimates its own location using the three-dimensional data 1034 and the three-dimensional map 1032.

[0535] The sensor information 1033 includes at least one of information obtained by a laser sensor, a luminance image, an infrared image, a depth image, sensor position information, and sensor speed information.

[0536] Furthermore, the sensor information 1033 includes information indicating the performance of the sensor.

[0537] Furthermore, the client device 902 encodes or compresses the sensor information 1033, and transmits the encoded or compressed sensor information 1037 to the server 901 or another client device 902. This allows the client device 902 to reduce the amount of data transmitted.

[0538] For example, the client device 902 includes a processor and a memory, and the processor uses the memory to perform the above-mentioned processing.

[0539] Furthermore, the server 901 according to this embodiment is capable of communicating with a client device 902 mounted on the mobile object, and receives sensor information 1037 indicating the surrounding conditions of the mobile object, obtained by a sensor 1015 mounted on the mobile object, from the client device 902. The server 901 creates three-dimensional data 1134 of the surroundings of the mobile object from the received sensor information 1037.

[0540] According to this, the server 901 creates three-dimensional data 1134 using the sensor information 1037 transmitted from the client device 902. This may reduce the amount of data transmitted compared to when the client device 902 transmits three-dimensional data. In addition, since the client device 902 does not need to perform processing such as compression or encoding of the three-dimensional data, the amount of processing by the client device 902 can be reduced. Therefore, the server 901 can reduce the amount of data transmitted or simplify the device configuration.

[0541] Moreover, the server 901 further transmits a request to the client device 902 to transmit the sensor information.

[0542] Furthermore, the server 901 further updates a three-dimensional map 1135 using the created three-dimensional data 1134 , and transmits the three-dimensional map 1135 to the client device 902 in response to a transmission request for the three-dimensional map 1135 from the client device 902 .

[0543] The sensor information 1037 includes at least one of information obtained by a laser sensor, a luminance image, an infrared image, a depth image, sensor position information, and sensor speed information.

[0544] Additionally, the sensor information 1037 includes information indicating the performance of the sensor.

[0545] Moreover, the server 901 further corrects the three-dimensional data in accordance with the performance of the sensor, thereby enabling the three-dimensional data creation method to improve the quality of the three-dimensional data.

[0546] Furthermore, in receiving the sensor information, the server 901 receives a plurality of pieces of sensor information 1037 from a plurality of client devices 902, and selects the sensor information 1037 to be used for creating the three-dimensional data 1134 based on a plurality of pieces of information indicating the performance of the sensors included in the plurality of pieces of sensor information 1037. This allows the server 901 to improve the quality of the three-dimensional data 1134.

[0547] Furthermore, the server 901 decodes or decompresses the received sensor information 1037, and creates three-dimensional data 1134 from the decoded or decompressed sensor information 1132. This allows the server 901 to reduce the amount of data to be transmitted.

[0548] For example, the server 901 includes a processor and a memory, and the processor uses the memory to perform the above-mentioned processing.

[0549] The following describes the modified example. Fig. 110 is a diagram showing the configuration of a system according to this embodiment. The system shown in Fig. 110 includes a server 2001, a client device 2002A, and a client device 2002B.

[0550] Client device 2002A and client device 2002B are mounted on a moving object such as a vehicle, and transmit sensor information to server 2001. Server 2001 transmits a three-dimensional map (point cloud) to client device 2002A and client device 2002B.

[0551] The client device 2002A includes a sensor information acquisition unit 2011, a storage unit 2012, and a data transmission permission determination unit 2013. The client device 2002B has a similar configuration. In the following description, the client device 2002A and the client device 2002B will also be referred to as the client device 2002 when there is no need to distinguish between them.

[0552] FIG. 111 is a flowchart showing the operation of client device 2002 according to the present embodiment.

[0553] The sensor information acquisition unit 2011 acquires various sensor information using a sensor (sensor group) mounted on the moving object. That is, the sensor information acquisition unit 2011 acquires sensor information indicating the surrounding situation of the moving object, acquired by a sensor (sensor group) mounted on the moving object. The sensor information acquisition unit 2011 also stores the acquired sensor information in the storage unit 2012. This sensor information includes at least one of LiDAR acquisition information, a visible light image, an infrared image, and a depth image. The sensor information may also include at least one of sensor position information, speed information, acquisition time information, and acquisition location information. The sensor position information indicates the position of the sensor that acquired the sensor information. The speed information indicates the speed of the moving object when the sensor acquired the sensor information. The acquisition time information indicates the time when the sensor information was acquired by the sensor. The acquisition location information indicates the position of the moving object or the sensor when the sensor information was acquired by the sensor.

[0554] Next, the data transmission possibility determination unit 2013 determines whether the mobile object (client device 2002) is in an environment where it can transmit sensor information to the server 2001 (S2002). For example, the data transmission possibility determination unit 2013 may use information such as GPS to identify the location and time of the client device 2002 and determine whether data can be transmitted. The data transmission possibility determination unit 2013 may also determine whether data can be transmitted based on whether it can connect to a specific access point.

[0555] When the client device 2002 determines that the moving object is in an environment in which it can transmit sensor information to the server 2001 (Yes in S2002), it transmits the sensor information to the server 2001 (S2003). That is, when the client device 2002 is in a situation in which it can transmit the sensor information to the server 2001, the client device 2002 transmits the sensor information it holds to the server 2001. For example, a millimeter wave access point capable of high-speed communication is installed at an intersection or the like. When the client device 2002 enters the intersection, it transmits the sensor information it holds to the server 2001 at high speed using millimeter wave communication.

[0556] Next, the client device 2002 deletes the sensor information already transmitted to the server 2001 from the storage unit 2012 (S2004). The client device 2002 may delete the sensor information when the sensor information not transmitted to the server 2001 satisfies a predetermined condition. For example, the client device 2002 may delete the sensor information from the storage unit 2012 when the acquisition time of the sensor information held by the client device 2002 becomes older than a certain time before the current time. That is, the client device 2002 may delete the sensor information from the storage unit 2012 when the difference between the time when the sensor information was acquired by the sensor and the current time exceeds a predetermined time. The client device 2002 may also delete the sensor information from the storage unit 2012 when the acquisition location of the sensor information held by the client device 2002 becomes more distant from the current location by a certain distance. That is, the client device 2002 may delete the sensor information from the storage unit 2012 when the difference between the position of the moving object or the sensor when the sensor information was acquired by the sensor and the current position of the moving object or the sensor exceeds a predetermined distance. This makes it possible to reduce the capacity of the storage unit 2012 of the client device 2002 .

[0557] If the client device 2002 has not finished acquiring the sensor information (No in S2005), the client device 2002 performs the processes from step S2001 onward again. If the client device 2002 has finished acquiring the sensor information (Yes in S2005), the client device 2002 ends the process.

[0558] Furthermore, the client device 2002 may select the sensor information to be transmitted to the server 2001 in accordance with the communication conditions. For example, when high-speed communication is possible, the client device 2002 transmits sensor information having a large size stored in the storage unit 2012 (e.g., LiDAR acquisition information, etc.) with priority. When high-speed communication is difficult, the client device 2002 transmits sensor information having a small size stored in the storage unit 2012 with high priority (e.g., visible light image). This allows the client device 2002 to efficiently transmit the sensor information stored in the storage unit 2012 to the server 2001 in accordance with the network conditions.

[0559] Furthermore, the client device 2002 may acquire time information indicating the current time and location information indicating the current location from the server 2001. Furthermore, the client device 2002 may determine the acquisition time and acquisition location of the sensor information based on the acquired time information and location information. That is, the client device 2002 may acquire time information from the server 2001 and generate acquisition time information using the acquired time information. Furthermore, the client device 2002 may acquire location information from the server 2001 and generate acquisition location information using the acquired location information.

[0560] For example, regarding time information, the server 2001 and the client device 2002 perform time synchronization using a mechanism such as NTP (Network Time Protocol) or PTP (Precision Time Protocol). This allows the client device 2002 to obtain accurate time information. In addition, since time can be synchronized between the server 2001 and multiple client devices, the time in the sensor information obtained by the different client devices 2002 can be synchronized. Therefore, the server 2001 can handle the sensor information indicating the synchronized time. Note that the time synchronization mechanism may be any method other than NTP or PTP. Furthermore, GPS information may be used as the above-mentioned time information and location information.

[0561] The server 2001 may acquire sensor information from a plurality of client devices 2002 by specifying a time or a location. For example, when an accident occurs, the server 2001 broadcasts a sensor information transmission request to a plurality of client devices 2002 by specifying the time and location of the accident in order to search for a client who was in the vicinity. Then, the client device 2002 having the sensor information of the corresponding time and location transmits the sensor information to the server 2001. That is, the client device 2002 receives a sensor information transmission request including designation information designating a location and time from the server 2001. When the client device 2002 determines that the sensor information obtained at the location and time indicated by the designation information is stored in the storage unit 2012 and that the mobile object is in an environment in which the sensor information can be transmitted to the server 2001, the client device 2002 transmits the sensor information obtained at the location and time indicated by the designation information to the server 2001. In this way, the server 2001 can acquire sensor information related to the occurrence of an accident from a plurality of client devices 2002 and use it for accident analysis, etc.

[0562] Note that the client device 2002 may refuse to transmit the sensor information when receiving a sensor information transmission request from the server 2001. Also, the client device 2002 may set in advance which of a plurality of pieces of sensor information is transmittable. Alternatively, the server 2001 may inquire of the client device 2002 each time whether or not the sensor information can be transmitted.

[0563] Also, points may be given to the client device 2002 that transmits the sensor information to the server 2001. The points may be used to pay for, for example, gasoline purchase costs, EV (Electric Vehicle) charging costs, highway tolls, rental car costs, and the like. Also, after acquiring the sensor information, the server 2001 may delete information for identifying the client device 2002 that transmitted the sensor information. For example, this information is information such as the network address of the client device 2002. This allows the sensor information to be anonymized, so that the user of the client device 2002 can transmit the sensor information from the client device 2002 to the server 2001 with peace of mind. Also, the server 2001 may be composed of multiple servers. For example, by sharing the sensor information among multiple servers, even if one server breaks down, the other servers can communicate with the client device 2002. This makes it possible to avoid a service interruption due to a server failure.

[0564] Furthermore, the designated location specified in the sensor information transmission request indicates the location where an accident occurred, etc., and may differ from the location of the client device 2002 at the designated time specified in the sensor information transmission request. Therefore, the server 2001 can request information acquisition from the client device 2002 that exists within the designated location by specifying a range, such as within XX m of the surrounding area. Similarly, for the designated time, the server 2001 may specify a range, such as within N seconds before and after a certain time. This allows the server 2001 to acquire sensor information from the client device 2002 that exists in "location: within XX m from absolute position S at time: tN to t+N." When transmitting three-dimensional data such as LiDAR data, the client device 2002 may transmit data generated immediately after time t.

[0565] Furthermore, the server 2001 may separately specify, as the specified location, information indicating the location of the client device 2002 from which sensor information is to be acquired and a location from which sensor information is desired. For example, the server 2001 specifies that sensor information including at least a range of YYm from the absolute position S is to be acquired from the client device 2002 that exists within XXm from the absolute position S. When selecting three-dimensional data to be transmitted, the client device 2002 selects one or more randomly accessible units of three-dimensional data so as to include at least the sensor information in the specified range. Furthermore, when transmitting a visible light image, the client device 2002 may transmit a plurality of temporally consecutive image data including at least a frame immediately before or after time t.

[0566] When the client device 2002 can use multiple physical networks, such as 5G or WiFi, or multiple modes in 5G, for transmitting sensor information, the client device 2002 may select a network to be used according to the priority order notified by the server 2001. Alternatively, the client device 2002 itself may select a network that can secure an appropriate bandwidth based on the size of the transmission data. Alternatively, the client device 2002 may select a network to be used based on the cost of data transmission, etc. Also, the transmission request from the server 2001 may include information indicating a transmission deadline, such as "transmission will be performed if the client device 2002 can start transmission by time T." The server 2001 may issue a transmission request again if sufficient sensor information cannot be acquired within the deadline.

[0567] The sensor information may include header information indicating characteristics of the sensor data along with compressed or uncompressed sensor data. The client device 2002 may transmit the header information to the server 2001 via a physical network or communication protocol different from that of the sensor data. For example, the client device 2002 transmits the header information to the server 2001 prior to transmitting the sensor data. The server 2001 determines whether to acquire the sensor data of the client device 2002 based on an analysis result of the header information. For example, the header information may include information indicating the point cloud acquisition density, elevation angle, or frame rate of the LiDAR, or the resolution, signal-to-noise ratio, or frame rate of the visible light image. This allows the server 2001 to acquire the sensor information from the client device 2002 having the sensor data of the determined quality.

[0568] As described above, the client device 2002 is mounted on a moving object, acquires sensor information indicating the surrounding conditions of the moving object obtained by a sensor mounted on the moving object, and stores the sensor information in the storage unit 2012. The client device 2002 determines whether the moving object is in an environment where the sensor information can be transmitted to the server 2001, and transmits the sensor information to the server 2001 when it is determined that the moving object is in an environment where the sensor information can be transmitted to the server.

[0569] Furthermore, the client device 2002 further creates three-dimensional data of the surroundings of the moving object from the sensor information, and estimates the self-position of the moving object using the created three-dimensional data.

[0570] Moreover, the client device 2002 further transmits a request to the server 2001 to transmit a three-dimensional map, and receives the three-dimensional map from the server 2001. The client device 2002 estimates its own location using the three-dimensional data and the three-dimensional map.

[0571] The above-described processing by client device 2002 may be realized as an information transmission method in client device 2002.

[0572] The client device 2002 may also include a processor and a memory, and the processor may use the memory to perform the above-mentioned processes.

[0573] Next, a sensor information collection system according to the present embodiment will be described. FIG. 112 is a diagram showing the configuration of the sensor information collection system according to the present embodiment. As shown in FIG. 112, the sensor information collection system according to the present embodiment includes a terminal 2021A, a terminal 2021B, a communication device 2022A, a communication device 2022B, a network 2023, a data collection server 2024, a map server 2025, and a client device 2026. When the terminal 2021A and the terminal 2021B are not particularly distinguished from each other, they are also referred to as the terminal 2021. When the communication device 2022A and the communication device 2022B are not particularly distinguished from each other, they are also referred to as the communication device 2022.

[0574] The data collection server 2024 collects data such as sensor data obtained by a sensor included in the terminal 2021 as position-related data associated with a position in a three-dimensional space.

[0575] The sensor data is, for example, data acquired by using a sensor provided in the terminal 2021, such as the state of the surroundings of the terminal 2021 or the state of the inside of the terminal 2021. The terminal 2021 transmits to the data collection server 2024 sensor data collected from one or more sensor devices that are located in positions that can directly communicate with the terminal 2021 or can communicate with the terminal 2021 via one or more relay devices using the same communication method.

[0576] The data included in the location-related data may include, for example, information indicating the operating state of the terminal itself or a device included in the terminal, an operating log, a service usage status, etc. Furthermore, the data included in the location-related data may include information associating an identifier of the terminal 2021 with the location or a movement route of the terminal 2021, etc.

[0577] The information indicating a position included in the position-related data is associated with information indicating a position in three-dimensional data such as three-dimensional map data, etc. The information indicating a position will be described in detail later.

[0578] The position related data may include at least one of the above-mentioned time information and information indicating an attribute of the data included in the position related data or the type of the sensor that generated the data (e.g., model number, etc.) in addition to the position information that indicates a position. The position information and the time information may be stored in a header area of ​​the position related data or a header area of ​​a frame that stores the position related data. In addition, the position information and the time information may be transmitted and / or stored separately from the position related data as metadata associated with the position related data.

[0579] The map server 2025 is connected to the network 2023, for example, and transmits three-dimensional data such as three-dimensional map data in response to a request from another device such as the terminal 2021. As described in the above-mentioned embodiments, the map server 2025 may also have a function of updating the three-dimensional data using sensor information transmitted from the terminal 2021.

[0580] The data collection server 2024 is connected to the network 2023, for example, and collects location-related data from other devices such as the terminal 2021, and stores the collected location-related data in a storage device inside the server or in another server. The data collection server 2024 also transmits the collected location-related data or metadata of three-dimensional map data generated based on the location-related data to the terminal 2021 in response to a request from the terminal 2021.

[0581] The network 2023 is a communication network such as the Internet. The terminal 2021 is connected to the network 2023 via a communication device 2022. The communication device 2022 communicates with the terminal 2021 by switching one communication method or a plurality of communication methods. The communication device 2022 is, for example, (1) a base station such as LTE (Long Term Evolution), (2) an access point (AP) such as WiFi or millimeter wave communication, (3) a gateway of an LPWA (Low Power Wide Area) Network such as SIGFOX, LoRaWAN, or Wi-SUN, or (4) a communication satellite that communicates by using a satellite communication method such as DVB-S2.

[0582] In addition, the base station may communicate with the terminal 2021 using a method classified as LPWA, such as NB-IoT (Narrow Band-IoT) or LTE-M, or may communicate with the terminal 2021 by switching between these methods.

[0583] Here, the terminal 2021 has a function of communicating with a communication device 2022 using two types of communication methods, and communicates with the map server 2025 or the data collection server 2024 using one of these communication methods, or by switching between these multiple communication methods and the communication device 2022 that is the direct communication partner, but the configuration of the sensor information collection system and the terminal 2021 is not limited to this. For example, the terminal 2021 may not have a communication function for multiple communication methods, but may have a function of communicating using any one of the communication methods. The terminal 2021 may also support three or more communication methods. Each terminal 2021 may support a different communication method.

[0584] The terminal 2021 has, for example, the configuration of the client device 902 shown in Fig. 102. The terminal 2021 performs position estimation such as its own position using the received three-dimensional data. The terminal 2021 also generates position-related data by associating the sensor data acquired from the sensor with position information obtained by position estimation processing.

[0585] The location information added to the location-related data indicates, for example, a location in a coordinate system used in three-dimensional data. For example, the location information is a coordinate value expressed by latitude and longitude values. At this time, the terminal 2021 may include, in the location information, information indicating the coordinate system that is the reference for the coordinate value together with the coordinate value and the three-dimensional data used for the location estimation. The coordinate value may also include altitude information.

[0586] The position information may also be associated with a data unit or a space unit that can be used to encode the above-mentioned three-dimensional data. The unit may be, for example, WLD, GOS, SPC, VLM, or VXL. In this case, the position information is expressed by an identifier for identifying a data unit such as an SPC corresponding to the position-related data. Note that the position information may include, in addition to an identifier for identifying a data unit such as an SPC, information indicating three-dimensional data obtained by encoding a three-dimensional space including the data unit such as the SPC, or information indicating a detailed position within the SPC. The information indicating the three-dimensional data may be, for example, a file name of the three-dimensional data.

[0587] In this way, the system generates location-related data associated with location information based on location estimation using three-dimensional data, and thereby can add location information to the sensor information with higher accuracy than when location information based on the self-location of the client device (terminal 2021) acquired using GPS is added to the sensor information. As a result, even when the location-related data is used by another device for another service, it may be possible to more accurately identify the location corresponding to the location-related data in real space by performing location estimation based on the same three-dimensional data.

[0588] In the present embodiment, the data transmitted from the terminal 2021 is location-related data, but the data transmitted from the terminal 2021 may be data that is not associated with location information. That is, the three-dimensional data or sensor data described in the other embodiments may be transmitted and received via the network 2023 described in the present embodiment.

[0589] Next, different examples of location information indicating a position in a three-dimensional or two-dimensional real space or map space will be described. The location information added to the location-related data may be information indicating a relative position with respect to a feature point in the three-dimensional data. Here, the feature point serving as the reference of the location information is, for example, a feature point encoded as SWLD and notified to the terminal 2021 as three-dimensional data.

[0590] The information indicating the relative position with respect to the feature point may be, for example, represented by a vector from the feature point to the point indicated by the position information, and may be information indicating the direction and distance from the feature point to the point indicated by the position information. Alternatively, the information indicating the relative position with respect to the feature point may be information indicating the amount of displacement on each of the X-axis, Y-axis, and Z-axis from the feature point to the point indicated by the position information. Furthermore, the information indicating the relative position with respect to the feature point may be information indicating the distance from each of three or more feature points to the point indicated by the position information. Note that the relative position may not be the relative position of the point indicated by the position information expressed with each feature point as a reference, but may be the relative position of each feature point expressed with the point indicated by the position information as a reference. An example of the position information based on the relative position with respect to the feature point includes information for identifying the reference feature point and information indicating the relative position of the point indicated by the position information with respect to the feature point. Furthermore, when the information indicating the relative position with respect to the feature point is provided separately from the three-dimensional data, the information indicating the relative position with respect to the feature point may include the coordinate axis used to derive the relative position, information indicating the type of three-dimensional data, and / or information indicating the size per unit amount (such as scale) of the value of the information indicating the relative position.

[0591] Furthermore, the position information may include information indicating a relative position with respect to each of a plurality of feature points. When the position information is expressed as a relative position with respect to a plurality of feature points, the terminal 2021 that attempts to identify a position indicated by the position information in the real space may calculate a candidate point of the position indicated by the position information from the position of the feature point estimated from the sensor data for each feature point, and may determine that a point obtained by averaging the calculated candidate points is the point indicated by the position information. According to this configuration, the influence of an error when estimating the position of the feature point from the sensor data can be reduced, so that the estimation accuracy of the point indicated by the position information in the real space can be improved. Furthermore, when the position information includes information indicating a relative position with respect to a plurality of feature points, even if there is a feature point that cannot be detected due to restrictions such as the type or performance of the sensor provided in the terminal 2021, it is possible to estimate the value of the point indicated by the position information as long as any one of the plurality of feature points can be detected.

[0592] A point that can be identified from sensor data can be used as a feature point. An example of a point that can be identified from sensor data is a point or a point within an area that satisfies a predetermined condition for detecting a feature point, such as the above-mentioned three-dimensional feature amount or the feature amount of visible light data being equal to or greater than a threshold value.

[0593] Also, markers installed in the real space may be used as feature points. In this case, the markers may be detected and their positions specified from data acquired using a sensor such as LiDAR or a camera. For example, the markers are expressed by changes in color or brightness (reflectance), or three-dimensional shapes (such as unevenness). Also, coordinate values ​​indicating the position of the marker, or two-dimensional codes or barcodes generated from the identifiers of the markers may be used.

[0594] Also, a light source that transmits an optical signal may be used as a marker. When a light source of an optical signal is used as a marker, not only information for acquiring a position such as a coordinate value or an identifier, but also other data may be transmitted by the optical signal. For example, the optical signal may include information indicating the content of a service corresponding to the position of the marker, an address such as a URL for acquiring the content, or an identifier of a wireless communication device for receiving the service, and a wireless communication method for connecting to the wireless communication device. By using an optical communication device (light source) as a marker, it becomes easier to transmit data other than information indicating a position, and it becomes possible to dynamically switch the data.

[0595] The terminal 2021 grasps the correspondence between feature points between different data by using, for example, an identifier commonly used between the data, or information or a table indicating the correspondence between feature points between the data. Furthermore, when there is no information indicating the correspondence between feature points, the terminal 2021 may determine that the feature points that are closest when the coordinates of a feature point in one three-dimensional data are converted to a position in the three-dimensional data space of the other are corresponding feature points.

[0596] When using the position information based on the relative positions described above, even between terminals 2021 or services using different three-dimensional data, it is possible to identify or estimate the position indicated by the position information based on common feature points included in or associated with each three-dimensional data. As a result, it is possible to identify or estimate the same position with higher accuracy between terminals 2021 or services using different three-dimensional data.

[0597] Furthermore, even when using map data or three-dimensional data expressed using different coordinate systems, the effect of errors associated with coordinate system conversion can be reduced, making it possible to link services based on more accurate location information.

[0598] Below, an example of a function provided by the data collection server 2024 will be described. The data collection server 2024 may transfer the received location-related data to another data server. When there are multiple data servers, the data collection server 2024 determines to which data server the received location-related data should be transferred, and transfers the location-related data to the data server determined as the transfer destination.

[0599] The data collection server 2024 determines the destination of transfer, for example, based on a determination rule for the destination server that is set in advance in the data collection server 2024. The determination rule for the destination server is set, for example, in a transfer destination table that associates an identifier associated with each terminal 2021 with a destination data server.

[0600] The terminal 2021 adds an identifier associated with the terminal 2021 to the location related data to be transmitted, and transmits the data to the data collection server 2024. The data collection server 2024 identifies a destination data server corresponding to the identifier added to the location related data based on a destination server determination rule using a destination table or the like, and transmits the location related data to the identified data server. The destination server determination rule may be specified by a determination condition using the time or place at which the location related data was acquired. Here, the identifier associated with the above-mentioned transmission source terminal 2021 is, for example, an identifier unique to each terminal 2021, or an identifier indicating a group to which the terminal 2021 belongs.

[0601] In addition, the destination table does not have to directly associate the identifier associated with the terminal of the source with the data server of the destination. For example, the data collection server 2024 holds a management table storing tag information given to each unique identifier of the terminal 2021, and a destination table in which the tag information is associated with the data server of the destination. The data collection server 2024 may determine the data server of the destination based on the tag information using the management table and the destination table. Here, the tag information is, for example, control information for management or control information for providing services given to the type, model number, owner, group to which the terminal 2021 corresponding to the identifier, or other identifiers. In addition, the destination table may use an identifier unique to each sensor instead of the identifier associated with the terminal 2021 of the source. In addition, the determination rule of the destination server may be set from the client device 2026.

[0602] The data collection server 2024 may determine a plurality of data servers as transfer destinations and transfer the received location-related data to the plurality of data servers. With this configuration, for example, when the location-related data is automatically backed up, or when the location-related data needs to be sent to a data server for providing each service in order to use the location-related data in common for different services, the data can be transferred as intended by changing the settings for the data collection server 2024. As a result, compared to the case where the destination of the location-related data is set in each individual terminal 2021, the number of steps required for building and modifying the system can be reduced.

[0603] In response to a transfer request signal received from a data server, the data collection server 2024 may register the data server specified in the transfer request signal as a new transfer destination, and transfer subsequently received location-related data to that data server.

[0604] The data collection server 2024 may store the location-related data received from the terminal 2021 in a recording device, and in response to a transmission request signal received from the terminal 2021 or the data server, may transmit the location-related data specified in the transmission request signal to the requesting terminal 2021 or data server.

[0605] The data collection server 2024 may determine whether or not to provide the location-related data to the requesting data server or terminal 2021, and if it is determined that the location-related data can be provided, may transfer or transmit the location-related data to the requesting data server or terminal 2021.

[0606] When a request for current location-related data is received from client device 2026, even if it is not the timing for terminal 2021 to transmit location-related data, data collection server 2024 may request terminal 2021 to transmit location-related data, and terminal 2021 may transmit the location-related data in response to the transmission request.

[0607] In the above explanation, it is described that the terminal 2021 transmits location information data to the data collection server 2024. However, the data collection server 2024 may be provided with a function necessary for collecting location-related data from the terminal 2021, such as a function for managing the terminal 2021, or a function used when collecting location-related data from the terminal 2021.

[0608] The data collection server 2024 may have a function of transmitting a data request signal to the terminal 2021 to request the transmission of location information data, and collecting location-related data.

[0609] Management information such as an address for communicating with the terminal 2021 from which data is to be collected or an identifier unique to the terminal 2021 is registered in advance in the data collection server 2024. The data collection server 2024 collects location-related data from the terminal 2021 based on the registered management information. The management information may include information such as the type of sensor included in the terminal 2021, the number of sensors included in the terminal 2021, and a communication method supported by the terminal 2021.

[0610] The data collection server 2024 may collect information from the terminal 2021, such as the operating status or current location of the terminal 2021.

[0611] The management information may be registered from the client device 2026, or the terminal 2021 may transmit a registration request to the data collection server 2024 to start the registration process. The data collection server 2024 may have a function of controlling communication with the terminal 2021.

[0612] The communication between the data collection server 2024 and the terminal 2021 may be a dedicated line provided by a service provider such as an MNO (Mobile Network Operator) or an MVNO (Mobile Virtual Network Operator), or a virtual dedicated line configured by a VPN (Virtual Private Network). With this configuration, the communication between the terminal 2021 and the data collection server 2024 can be performed safely.

[0613] The data collection server 2024 may have a function of authenticating the terminal 2021 or a function of encrypting data transmitted and received between the terminal 2021. Here, the authentication process of the terminal 2021 or the encryption process of the data is performed using an identifier unique to the terminal 2021 or an identifier unique to a terminal group including a plurality of terminals 2021, which is shared in advance between the data collection server 2024 and the terminal 2021. This identifier is, for example, an International Mobile Subscriber Identity (IMSI), which is a unique number stored in a Subscriber Identity Module (SIM) card. The identifier used in the authentication process and the identifier used in the data encryption process may be the same or different.

[0614] The authentication or data encryption process between the data collection server 2024 and the terminal 2021 can be provided if both the data collection server 2024 and the terminal 2021 have a function for performing the process, and does not depend on the communication method used by the relay communication device 2022. Therefore, a common authentication or encryption process can be used without considering the communication method used by the terminal 2021, improving the convenience of the user's system construction. However, not depending on the communication method used by the relay communication device 2022 means that it is not necessary to change depending on the communication method. In other words, for the purpose of improving transmission efficiency or ensuring safety, the authentication or data encryption process between the data collection server 2024 and the terminal 2021 may be switched depending on the communication method used by the relay device.

[0615] The data collection server 2024 may provide the client device 2026 with a UI for managing data collection rules such as the type of location-related data to be collected from the terminal 2021 and the schedule for data collection. This allows the user to specify the terminal 2021 from which data is to be collected using the client device 2026, as well as the time and frequency of data collection. The data collection server 2024 may also specify an area on a map from which data is to be collected, and collect location-related data from the terminal 2021 included in that area.

[0616] When managing data collection rules for each terminal 2021, the client device 2026 presents, for example, a list of the terminals 2021 or sensors to be managed on a screen. The user sets the necessity of data collection or a collection schedule for each item in the list.

[0617] When specifying an area on a map from which data is to be collected, the client device 2026 presents, for example, a two-dimensional or three-dimensional map of the area to be managed on a screen. The user selects an area from which data is to be collected on the displayed map. The area selected on the map may be a circular or rectangular area centered on a point specified on the map, or may be a circular or rectangular area that can be specified by a dragging operation. The client device 2026 may also select an area in a preset unit such as a city, an area within a city, a block, or a major road. Instead of specifying an area using a map, an area may be set by inputting numerical values ​​of latitude and longitude, or an area may be selected from a list of candidate areas derived based on input text information. The text information is, for example, the name of a region, a city, or a landmark.

[0618] Furthermore, the user may specify one or more terminals 2021 and set conditions such as within a range of 100 meters around the terminal 2021, so that data collection may be performed while dynamically changing the specified area.

[0619] Also, when the client device 2026 is equipped with a sensor such as a camera, an area on a map may be specified based on the position of the client device 2026 in real space obtained from the sensor data. For example, the client device 2026 may estimate its own position using the sensor data, and specify an area within a predetermined distance from a point on the map corresponding to the estimated position or a distance specified by the user as an area for collecting data. Also, the client device 2026 may specify a sensing area of ​​the sensor, i.e., an area corresponding to the acquired sensor data, as an area for collecting data. Alternatively, the client device 2026 may specify an area based on a position corresponding to the sensor data specified by the user as an area for collecting data. The area on the map or the position corresponding to the sensor data may be estimated by the client device 2026 or the data collection server 2024.

[0620] When designating an area on a map, the data collection server 2024 may identify the terminal 2021 in the designated area by collecting current position information of each terminal 2021, and may request the identified terminal 2021 to transmit position-related data. Alternatively, instead of the data collection server 2024 identifying the terminal 2021 in the area, the data collection server 2024 may transmit information indicating the designated area to the terminal 2021, and the terminal 2021 may determine whether or not it is within the designated area, and transmit the position-related data if it is determined that it is within the designated area.

[0621] The data collection server 2024 transmits data such as a list or a map for providing the above-mentioned UI (User Interface) in an application executed by the client device 2026 to the client device 2026. The data collection server 2024 may transmit not only data such as a list or a map but also an application program to the client device 2026. The above-mentioned UI may be provided as content created in HTML or the like that can be displayed in a browser. Note that some data such as map data may be provided from a server other than the data collection server 2024, such as a map server 2025.

[0622] When the user presses a setting button or performs other input to notify completion of input, the client device 2026 transmits the input information as setting information to the data collection server 2024. The data collection server 2024 transmits a signal to each terminal 2021, based on the setting information received from the client device 2026, requesting location-related data or notifying a collection rule for location-related data, and collects location-related data.

[0623] Next, an example of controlling the operation of the terminal 2021 based on additional information added to three-dimensional or two-dimensional map data will be described.

[0624] In this configuration, object information indicating the position of a power supply unit such as a wireless power supply antenna or a power supply coil buried in a road or parking lot is included in or associated with the three-dimensional data and provided to a terminal 2021, such as a car or a drone.

[0625] The vehicle or drone that has acquired the object information for charging moves the vehicle's own position by autonomous driving so that the position of the charging part, such as a charging antenna or charging coil, equipped on the vehicle faces the area indicated by the object information, and starts charging. In the case of a vehicle or drone that does not have an autonomous driving function, the direction to move or the operation to be performed is presented to the driver or operator using images or audio displayed on the screen. Then, when it is determined that the position of the charging part calculated based on the estimated self-position is within the area indicated by the object information or within a predetermined distance from the area, the image or audio presented is switched to one that stops driving or operation, and charging starts.

[0626] In addition, the object information may not be information indicating the position of the power supply unit, but may be information indicating an area in which a charging efficiency equal to or higher than a predetermined threshold can be obtained when the charging unit is placed in the area. The position of the object information may be represented by a point at the center of the area indicated by the object information, or may be represented by an area or line in a two-dimensional plane, or an area, line, or plane in a three-dimensional space.

[0627] According to this configuration, the position of the power supply antenna, which cannot be grasped by the sensing data of the LiDER or the image captured by the camera, can be grasped, so that the position of the wireless charging antenna equipped in the terminal 2021 such as a car and the wireless power supply antenna buried in the road or the like can be aligned with high accuracy. As a result, the charging speed during wireless charging can be shortened and the charging efficiency can be improved.

[0628] The object information may be an object other than the power supply antenna. For example, the three-dimensional data includes the position of an AP for millimeter wave wireless communication as object information. This allows the terminal 2021 to know the position of the AP in advance, and therefore the terminal 2021 can start communication by directing the beam direction in the direction of the object information. As a result, it is possible to improve communication quality, such as improving the transmission speed, shortening the time until communication starts, and extending the period during which communication is possible.

[0629] The object information may include information indicating a type of object corresponding to the object information. The object information may also include information indicating a process to be performed by the terminal 2021 when the terminal 2021 is included in an area in real space corresponding to the position of the object information in the three-dimensional data, or within a predetermined distance from the area.

[0630] The object information may be provided from a server different from the server providing the three-dimensional data. When the object information is provided separately from the three-dimensional data, object groups in which object information used in the same service is stored may be provided as separate data according to the type of target service or target device.

[0631] The three-dimensional data used in combination with the object information may be point cloud data of the WLD or feature point data of the SWLD.

[0632] In a three-dimensional data encoding device, when the attribute information of a target three-dimensional point, which is a three-dimensional point to be encoded, is hierarchically encoded using LoD (Level of Detail), the three-dimensional data decoding device may decode the attribute information up to the LoD level required by the three-dimensional data decoding device and not decode the attribute information of levels that are not necessary. For example, when the total number of LoDs of the attribute information in the bit stream encoded by the three-dimensional data encoding device is N, the three-dimensional data decoding device may decode M (M < N) LoDs from LoD0 at the highest layer to LoD(M - 1), and not decode the remaining LoDs up to LoD(N - 1). Thereby, the three-dimensional data decoding device can decode the attribute information from LoD0 to LoD(M - 1) required by the three-dimensional data decoding device while suppressing the processing load.

[0633] FIG. 113 is a diagram showing the above use case. In the example of FIG. 113, the server holds a three-dimensional map obtained by encoding three-dimensional position information and attribute information. The server (three-dimensional data encoding device) broadcasts and transmits the three-dimensional map to a client device (three-dimensional data decoding device: for example, a vehicle or a drone, etc.) in the area managed by the server, and the client device uses the three-dimensional map received from the server to identify its own position, or performs a process of displaying map information to a user who operates the client device, etc.

[0634] Next, an operation example in this example will be described. First, the server encodes the position information of the three-dimensional map using a quadtree structure or the like. Then, the server hierarchically encodes the attribute information of the three-dimensional map using N LoDs constructed based on the position information. The server saves the bit stream of the three-dimensional map obtained by the hierarchical encoding.

[0635] Next, in response to a transmission request for map information transmitted from a client device in the area managed by the server, the server transmits the bit stream of the encoded three-dimensional map to the client device.

[0636] The client device receives the bitstream of the three-dimensional map transmitted from the server, and decodes the position information and attribute information of the three-dimensional map according to the use of the client device. For example, when the client device performs high-precision self-position estimation using the position information and the attribute information of N LoDs, the client device determines that the decoding result up to the dense three-dimensional points is required as the attribute information, and decodes all the information in the bitstream.

[0637] Also, when the client device displays the information of the three-dimensional map to the user or the like, the client device determines that the decoding result up to the sparse three-dimensional points is required as the attribute information, and decodes the position information and the attribute information of M LoDs (M < N) from LoD0, which is the upper layer of LoD.

[0638] By switching the LoD of the attribute information to be decoded according to the use of the client device in this way, the processing load of the client device can be reduced.

[0639] In the example shown in FIG. 113, for example, the three-dimensional point map includes position information and attribute information. The position information is encoded in a quadtree. The attribute information is encoded in N LoDs.

[0640] The client device A performs high-precision self-position estimation. In this case, the client device A determines that all the position information and attribute information are required, and decodes all the position information and the attribute information composed of N LoDs in the bitstream.

[0641] The client device B displays the three-dimensional map to the user. In this case, the client device B determines that the position information and the attribute information of M LoDs (M < N) are required, and decodes the position information and the attribute information composed of M LoDs in the bitstream.

[0642] Note that the server may broadcast and transmit the three-dimensional map to the client device, or may perform multicast transmission or unicast transmission.

[0643] Next, a modification example of the system according to this embodiment will be described. In the three-dimensional data encoding device, when hierarchically encoding the attribute information of the target three-dimensional point, which is the three-dimensional point to be encoded, using LoD, the three-dimensional data encoding device may encode the attribute information up to the LoD level required by the three-dimensional data decoding device and not encode the attribute information of the levels that are not necessary. For example, when the total number of LoDs is N, the three-dimensional data encoding device may generate a bitstream by encoding M (M < N) LoDs from the topmost layer LoD0 to LoD(M - 1) and not encoding the remaining LoDs up to LoD(N - 1). Thereby, the three-dimensional data encoding device can provide a bitstream encoding the attribute information from LoD0 to LoD(M - 1) required by the three-dimensional data decoding device according to the request from the three-dimensional data decoding device.

[0644] FIG. 114 is a diagram showing the above use case. In the example shown in FIG. 114, the server holds a three-dimensional map obtained by encoding three-dimensional position information and attribute information. The server (three-dimensional data encoding device) unicast-transmits the three-dimensional map to a client device (three-dimensional data decoding device: for example, a vehicle or a drone, etc.) in the area managed by the server according to the request of the client device, and the client device uses the three-dimensional map received from the server to specify its own position, or performs a process of displaying the map information to a user who operates the client device, etc.

[0645] Next, an operation example in this example will be described. Fir...

Claims

1. Obtaining at least one of a plurality of sub-encoded data constituting encoded data in which a plurality of three-dimensional points are occupancy-encoded using a hierarchical structure having a plurality of layers by a processor; obtaining, by the processor, a piece of first information corresponding to the obtained sub-encoded data among a plurality of pieces of first information each corresponding to the plurality of sub-encoded data; Decoding the obtained sub-encoded data by the processor based on the first information; each of the plurality of sub-encoded data corresponds to a group hierarchy obtained by grouping one or more hierarchies from among the plurality of hierarchies; The first information includes information indicating the number of hierarchical layers included in the group hierarchy corresponding to the acquired sub-encoded data. A method for decoding three-dimensional data.

2. acquiring, by the processor, at least one of a plurality of sub-encoded attribute data constituting encoded data in which attribute information of the plurality of three-dimensional points is encoded using the hierarchical structure having the plurality of hierarchical levels; The hierarchical structure used for the plurality of sub-encoded data is the same as the hierarchical structure used for the plurality of sub-encoded attribute data.

2. The three-dimensional data decoding method according to claim 1.

3. The plurality of sub-encoded data include first sub-encoded data that is independently decodable and second sub-encoded data that is not independently decodable.

2. The three-dimensional data decoding method according to claim 1.

4. The second sub-encoded data is decoded by the processor after the first sub-encoded data is decoded.

4. The three-dimensional data decoding method according to claim 3.

5. obtaining second information from a header of the obtained sub-encoded data by the processor; The second information includes identification information of the acquired sub-encoded data.

2. The three-dimensional data decoding method according to claim 1.

6. generating at least one of a plurality of sub-encoded data constituting encoded data in which a plurality of three-dimensional points are occupancy-encoded using a hierarchical structure having a plurality of layers by a processor; generating, by the processor, first information corresponding to the generated sub-encoded data among a plurality of first information pieces each corresponding to the plurality of sub-encoded data pieces; each of the plurality of sub-encoded data corresponds to a group hierarchy obtained by grouping one or more hierarchies from among the plurality of hierarchies; The first information includes information indicating the number of hierarchical layers included in the group hierarchy corresponding to the generated sub-encoded data. Three-dimensional data encoding method.

7. A processor; A memory. The processor uses the memory to: obtaining at least one of a plurality of sub-encoded data constituting encoded data in which a plurality of three-dimensional points are occupancy-encoded using a hierarchical structure having a plurality of layers; obtaining first information corresponding to the obtained sub-encoded data from among a plurality of first information pieces each corresponding to the plurality of sub-encoded data pieces; Decoding the acquired sub-encoded data based on the first information; each of the plurality of sub-encoded data corresponds to a group hierarchy obtained by grouping one or more hierarchies from among the plurality of hierarchies; The first information includes information indicating the number of hierarchical layers included in the group hierarchy corresponding to the acquired sub-encoded data. Three-dimensional data decoding device.

8. A processor; A memory. The processor uses the memory to: generating at least one of a plurality of sub-encoded data constituting encoded data in which a plurality of three-dimensional points are occupancy-encoded using a hierarchical structure having a plurality of layers; generating first information corresponding to the generated sub-encoded data among a plurality of first information pieces each corresponding to the plurality of sub-encoded data pieces; each of the plurality of sub-encoded data corresponds to a group hierarchy obtained by grouping one or more hierarchies from among the plurality of hierarchies; The first information includes information indicating the number of hierarchical layers included in the group hierarchy corresponding to the generated sub-encoded data. Three-dimensional data encoding device.

Images