Three-dimensional data encoding method, three-dimensional data decoding method, three-dimensional data encoding device, and three-dimensional data decoding device

Abstract

To provide a three-dimensional data encoding method, and the like, for properly decoding point cloud data.SOLUTION: A three-dimensional data encoding method to be executed by a processor includes: encoding point cloud data including a plurality of three-dimensional points to generate encoded data; and generating a bitstream including the encoded data and a conformance index and satisfying conformance information corresponding to the conformance index, the conformance information including at least one of the maximum bit count of a slice corresponding to part of the encoded data and the maximum bit count of the encoded data.SELECTED DRAWING: Figure 112

Description

[Technical Field]

[0001] This disclosure relates to a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, and a three-dimensional data decoding device. [Background technology]

[0002] In the future, devices and services utilizing three-dimensional data are expected to become widespread in a wide range of fields, including computer vision for autonomous operation of automobiles or robots, map information, monitoring, infrastructure inspection, and video distribution. Three-dimensional data can be acquired in various ways, such as using distance sensors like rangefinders, stereo cameras, or combinations of multiple monocular cameras.

[0003] One method of representing three-dimensional data is called a point cloud, which represents the shape of a three-dimensional structure using a cloud of points in three-dimensional space. In a point cloud, the position and color of the points are stored. Point clouds are expected to become the mainstream method of representing three-dimensional data, but point clouds are extremely large in size. Therefore, in the storage or transmission of three-dimensional data, data compression through encoding is essential, just as with two-dimensional moving images (for example, MPEG-4 AVC or HEVC, which are standardized by MPEG).

[0004] Furthermore, point cloud compression is partially supported by publicly available libraries (such as the Point Cloud Library) that handle point cloud-related processing.

[0005] Furthermore, there is a known technique for searching for and displaying facilities located around a vehicle using three-dimensional map data (see, for example, Patent Document 1). [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] International Publication No. 2014 / 020663 [Overview of the Initiative] [Problems that the invention aims to solve]

[0007] In the encoding and decoding of three-dimensional data, it is desirable to be able to properly decode point cloud data.

[0008] The purpose of this disclosure is to provide a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, or a three-dimensional data decoding device that can appropriately decode point cloud data. [Means for solving the problem]

[0009] A three-dimensional data encoding method according to one aspect of the present disclosure is a three-dimensional data encoding method performed by a processor, which generates encoded data by encoding point cloud data including a plurality of three-dimensional points, generates a bitstream that includes the encoded data and a conformance index and satisfies conformance information corresponding to the conformance index, wherein the conformance information includes at least one of the maximum number of bits of a slice corresponding to a part of the encoded data and the maximum number of bits of the encoded data.

[0010] A three-dimensional data decoding method according to one aspect of the present disclosure includes encoded data and conformance information obtained by encoding point cloud data including a plurality of three-dimensional points, and a bitstream that satisfies conformance information corresponding to the conformance index, decodes the encoded data, and the conformance information includes at least one of the maximum number of bits of a slice corresponding to a part of the encoded data and the maximum number of bits of the encoded data.

[0011] A three-dimensional data encoding method according to one aspect of the present disclosure involves determining a first maximum number of bits for encoded data after encoding at least one of the divided data units when point cloud data representing a three-dimensional point cloud is divided into multiple parts, and the point cloud data unit before division, and generating a bitstream by encoding the multiple divided data obtained from the divided point cloud data, or the point cloud data before division, to satisfy the determined first maximum number of bits, wherein the bitstream includes first bit number information indicating the first maximum number of bits.

[0012] A three-dimensional data decoding method according to one aspect of the present disclosure acquires a bitstream that includes encoded data which is the encoded data of at least one of the divided data units when point cloud data representing a three-dimensional point cloud is divided into multiple parts, and first bit number information which indicates the first maximum number of bits of the encoded data, determines whether the acquired bitstream satisfies the first maximum number of bits indicated by the first bit number information, and decodes the encoded data if it is determined that the bitstream satisfies the first maximum number of bits.

[0013] These general or specific embodiments may be implemented as a system, device, integrated circuit, computer program, or recording medium such as a computer-readable CD-ROM, or as any combination of a system, device, integrated circuit, computer program, and recording medium. [Effects of the Invention]

[0014] This disclosure provides a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, or a three-dimensional data decoding device that can appropriately decode point cloud data. [Brief explanation of the drawing]

[0015] [Figure 1] Figure 1 shows the configuration of a three-dimensional data encoding and decoding system according to Embodiment 1. [Figure 2]FIG. 2 is a diagram showing a configuration example of point cloud data according to Embodiment 1. [Figure 3] FIG. 3 is a diagram showing a configuration example of a data file in which point cloud data information according to Embodiment 1 is described. [Figure 4] FIG. 4 is a diagram showing types of point cloud data according to Embodiment 1. [Figure 5] FIG. 5 is a diagram showing a configuration of a first encoding unit according to Embodiment 1. [Figure 6] FIG. 6 is a block diagram of a first encoding unit according to Embodiment 1. [Figure 7] FIG. 7 is a diagram showing a configuration of a first decoding unit according to Embodiment 1. [Figure 8] FIG. 8 is a block diagram of a first decoding unit according to Embodiment 1. [Figure 9] FIG. 9 is a diagram showing a configuration of a second encoding unit according to Embodiment 1. [Figure 10] FIG. 10 is a block diagram of a second encoding unit according to Embodiment 1. [Figure 11] FIG. 11 is a diagram showing a configuration of a second decoding unit according to Embodiment 1. [Figure 12] FIG. 12 is a block diagram of a second decoding unit according to Embodiment 1. [Figure 13] FIG. 13 is a diagram showing a protocol stack related to PCC encoded data according to Embodiment 1. [Figure 14] FIG. 14 is a diagram showing a configuration of an encoding unit and a multiplexing unit according to Embodiment 2. [Figure 15] FIG. 15 is a diagram showing a configuration example of encoded data according to Embodiment 2. [Figure 16] FIG. 16 is a diagram showing a configuration example of encoded data and a NAL unit according to Embodiment 2. [Figure 17] FIG. 17 is a diagram showing a semantics example of pcc_nal_unit_type according to Embodiment 2. [Figure 18] FIG. 18 is a diagram showing an example of a transmission order of NAL units according to Embodiment 2. [Figure 19] Figure 19 is a block diagram of the first encoding unit according to Embodiment 3. [Figure 20] Figure 20 is a block diagram of the first decoding unit according to Embodiment 3. [Figure 21] Figure 21 is a block diagram of the divided section according to Embodiment 3. [Figure 22] Figure 22 shows an example of slice and tile division according to Embodiment 3. [Figure 23] Figure 23 shows an example of a slice and tile division pattern according to Embodiment 3. [Figure 24] Figure 24 is a diagram showing an example of a dependency relationship according to Embodiment 3. [Figure 25] Figure 25 shows an example of the data decoding order according to Embodiment 3. [Figure 26] Figure 26 is a flowchart of the encoding process according to Embodiment 3. [Figure 27] Figure 27 is a block diagram of the joint according to Embodiment 3. [Figure 28] Figure 28 shows an example of the configuration of encoded data and a NAL unit according to Embodiment 3. [Figure 29] Figure 29 is a flowchart of the encoding process according to Embodiment 3. [Figure 30] Figure 30 is a flowchart of the decoding process according to Embodiment 3. [Figure 31] Figure 31 shows an example of the syntax for tile addition information according to Embodiment 4. [Figure 32] Figure 32 is a block diagram of the coding and decoding system according to Embodiment 4. [Figure 33] Figure 33 shows an example of the syntax for slice addition information according to Embodiment 4. [Figure 34] Figure 34 is a flowchart of the encoding process according to Embodiment 4. [Figure 35] Figure 35 is a flowchart of the decoding process according to Embodiment 4. [Figure 36] Figure 36 shows an example of a tree structure according to Embodiment 5. [Figure 37] Figure 37 shows an example of the data structure of encoded data in an octave tree according to Embodiment 5. [Figure 38] Figure 38 shows an example of payload syntax according to Embodiment 5. [Figure 39] Figure 39 is a flowchart of the decoding process according to Embodiment 5. [Figure 40] Figure 40 is a diagram showing the relationship between the level and the data to be decoded according to Embodiment 5. [Figure 41] Figure 41 is a schematic diagram showing the level according to Embodiment 5. [Figure 42] Figure 42 shows an example of the header syntax according to Embodiment 5. [Figure 43] Figure 43 shows an example of payload syntax according to Embodiment 5. [Figure 44] Figure 44 is a diagram showing the configuration of the overall encoded data according to Embodiment 5. [Figure 45] Figure 45 is a diagram showing the configuration of the overall encoded data according to Embodiment 5. [Figure 46] Figure 46 shows an example of the syntax for depth information according to Embodiment 5. [Figure 47] Figure 47 shows an example of the syntax of hierarchical metadata according to Embodiment 5. [Figure 48] Figure 48 shows an example of the header syntax according to Embodiment 5. [Figure 49] Figure 49 is a diagram showing the configuration of the overall encoded data according to Embodiment 5. [Figure 50] Figure 50 shows an example of the syntax of hierarchical information according to Embodiment 5. [Figure 51] Figure 51 shows an example of the syntax of hierarchical metadata according to Embodiment 5. [Figure 52]Figure 52 shows an example of the syntax of the header of the overall encoded data according to Embodiment 5. [Figure 53] Figure 53 shows an example of the bitstream configuration according to Embodiment 5. [Figure 54] Figure 54 shows an example of the syntax of hierarchical metadata according to Embodiment 5. [Figure 55] Figure 55 is a diagram showing the reference relationship between location information and attribute information according to Embodiment 5. [Figure 56] Figure 56 is a diagram showing the reference relationship between location information and attribute information according to Embodiment 5. [Figure 57] Figure 57 is a diagram showing the reference relationship between location information and attribute information according to Embodiment 5. [Figure 58] Figure 58 shows an example of the bitstream configuration according to Embodiment 5. [Figure 59] Figure 59 shows an example of the configuration of a three-dimensional data encoding device according to Embodiment 5. [Figure 60] Figure 60 shows an example of the configuration of a three-dimensional data decoding device according to Embodiment 5. [Figure 61] Figure 61 shows the basic structure of ISOBMFF according to Embodiment 5. [Figure 62] Figure 62 is a protocol stack diagram showing the case where the NAL unit common to the PCC codec according to Embodiment 5 is stored in ISOBMFF. [Figure 63] Figure 63 is a diagram showing the bitstream to file format conversion process according to Embodiment 5. [Figure 64] Figure 64 is a flowchart of the format conversion process according to Embodiment 5. [Figure 65] Figure 65 is a flowchart of the decoding process according to Embodiment 5. [Figure 66] Figure 66 is a diagram showing the bitstream to file format conversion process according to Embodiment 5. [Figure 67]Figure 67 shows an example of the syntax of hierarchical metadata according to Embodiment 5. [Figure 68] Figure 68 is a schematic diagram showing the division process according to Embodiment 5. [Figure 69] Figure 69 is a flowchart of the conversion process using hierarchical information according to Embodiment 5. [Figure 70] Figure 70 is a flowchart of the conversion process that does not use hierarchical information according to Embodiment 5. [Figure 71] Figure 71 is a flowchart of the decoding process for hierarchical data sample data according to Embodiment 5. [Figure 72] Figure 72 shows an example of the overall encoded data configuration according to Embodiment 5. [Figure 73] Figure 73 shows an example of the overall encoded data configuration according to Embodiment 5. [Figure 74] Figure 74 is a diagram showing the bitstream to file format conversion process according to Embodiment 5. [Figure 75] Figure 75 is a flowchart of the format conversion process according to Embodiment 5. [Figure 76] Figure 76 is a flowchart of the decoding process according to Embodiment 5. [Figure 77] Figure 77 shows an example of the syntax for depth information according to Embodiment 5. [Figure 78] Figure 78 shows an example of the syntax for a sample size box according to Embodiment 5. [Figure 79] Figure 79 shows an example of the syntax for hierarchical information according to Embodiment 5. [Figure 80] Figure 80 shows an example of the syntax of PCCLayerStructureBox according to Embodiment 5. [Figure 81] Figure 81 is a schematic diagram showing the extraction operation according to Embodiment 5. [Figure 82] Figure 82 shows an example of a file format according to Embodiment 5. [Figure 83] Figure 83 shows an example of an extracted bitstream according to Embodiment 5. [Figure 84] Figure 84 shows an example of an extracted bitstream according to Embodiment 5. [Figure 85] Figure 85 shows an example of an extracted bitstream according to Embodiment 5. [Figure 86] Figure 86 shows an example of the direct mode according to Embodiment 5. [Figure 87] Figure 87 is a flowchart of the three-dimensional data encoding process according to Embodiment 5. [Figure 88] Figure 88 is a flowchart of the three-dimensional data decoding process according to Embodiment 5. [Figure 89] Figure 89 is a block diagram showing an example of the configuration of a three-dimensional data encoding device according to Embodiment 6. [Figure 90] Figure 90 is a flowchart showing a first example of a three-dimensional data encoding method according to Embodiment 6. [Figure 91] Figure 91 is a block diagram showing an example of the configuration of a three-dimensional data decoding device according to Embodiment 6. [Figure 92] Figure 92 is a flowchart showing an example of a three-dimensional data decoding method according to Embodiment 6. [Figure 93] Figure 93 is a flowchart showing a second example of a three-dimensional data encoding method according to Embodiment 6. [Figure 94] Figure 94 shows an example of a bounding box according to Embodiment 6. [Figure 95] Figure 95 is a flowchart showing another example of the three-dimensional data decoding method according to Embodiment 6. [Figure 96] Figure 96 is a flowchart showing a third example of a three-dimensional data encoding method according to Embodiment 6. [Figure 97] Figure 97 shows an example of the process for reducing the number of bits according to Embodiment 6. [Figure 98]Figure 98 shows another example of the bit reduction process according to Embodiment 6. [Figure 99] Figure 99 is a flowchart showing a fourth example of a three-dimensional data encoding method according to Embodiment 6. [Figure 100] Figure 100 shows an example of the process for increasing the number of bits according to Embodiment 6. [Figure 101] Figure 101 shows another example of the process for increasing the number of bits according to Embodiment 6. [Figure 102] Figure 102 is a flowchart showing a fifth example of the three-dimensional data encoding method according to Embodiment 6. [Figure 103] Figure 103 shows an example of a conformance combination according to Embodiment 6. [Figure 104] Figure 104 is a flowchart showing a sixth example of the three-dimensional data encoding method according to Embodiment 6. [Figure 105] Figure 105 shows another example of conformance combinations according to Embodiment 6. [Figure 106] Figure 106 shows another example of conformance combinations according to Embodiment 6. [Figure 107] Figure 107 shows an example (Example 1) of an SPS (Sequence Parameter Set) according to Embodiment 6. [Figure 108] Figure 108 shows an example of an SPS according to Embodiment 6 (Example 2). [Figure 109] Figure 109 shows an example (Example 3) of a GPS (Geometry Parameter Set) according to Embodiment 6. [Figure 110] Figure 110 is a diagram showing the configuration of the bitstream according to Embodiment 6. [Figure 111] Figure 111 is a diagram illustrating an example of switching conformances according to the location of a three-dimensional point cloud, according to Embodiment 6. [Figure 112]Figure 112 is a flowchart showing another example of the three-dimensional data encoding process according to Embodiment 6. [Figure 113] Figure 113 is a flowchart showing another example of the three-dimensional data decoding process according to Embodiment 6. [Figure 114] Figure 114 is a block diagram of a three-dimensional data creation device according to Embodiment 7. [Figure 115] Figure 115 is a flowchart of the three-dimensional data creation method according to Embodiment 7. [Figure 116] Figure 116 is a diagram showing the configuration of the system according to Embodiment 7. [Figure 117] Figure 117 is a block diagram of the client device according to Embodiment 7. [Figure 118] Figure 118 is a block diagram of the server according to Embodiment 7. [Figure 119] Figure 119 is a flowchart of the three-dimensional data creation process by the client device according to Embodiment 7. [Figure 120] Figure 120 is a flowchart of the sensor information transmission process by the client device according to Embodiment 7. [Figure 121] Figure 121 is a flowchart of the three-dimensional data creation process performed by the server according to Embodiment 7. [Figure 122] Figure 122 is a flowchart of the three-dimensional map transmission process by the server according to Embodiment 7. [Figure 123] Figure 123 shows a modified configuration of the system according to Embodiment 7. [Figure 124] Figure 124 is a diagram showing the configuration of the server and client device according to Embodiment 7. [Figure 125] Figure 125 is a diagram showing the configuration of the server and client device according to Embodiment 7. [Figure 126] Figure 126 is a flowchart of the processing performed by the client device according to Embodiment 7. [Figure 127]Figure 127 is a diagram showing the configuration of the sensor information collection system according to Embodiment 7. [Figure 128] Figure 128 shows an example of a system according to Embodiment 7. [Figure 129] Figure 129 shows a modified example of the system according to Embodiment 7. [Figure 130] Figure 130 is a flowchart showing an example of application processing according to Embodiment 7. [Figure 131] Figure 131 is a diagram showing the sensor ranges of various sensors according to Embodiment 7. [Figure 132] Figure 132 is a diagram showing an example configuration of an automated driving system according to Embodiment 7. [Figure 133] Figure 133 shows an example of the bitstream configuration according to Embodiment 7. [Figure 134] Figure 134 is a flowchart of the point cloud selection process according to Embodiment 7. [Figure 135] Figure 135 shows an example of the screen for point cloud selection processing according to Embodiment 7. [Figure 136] Figure 136 shows an example of the point cloud selection process screen according to Embodiment 7. [Figure 137] Figure 137 shows an example of the screen for point cloud selection processing according to Embodiment 7. [Modes for carrying out the invention]

[0016] A three-dimensional data encoding method according to one aspect of the present disclosure involves determining a first maximum number of bits for encoded data after encoding at least one of the divided data units when point cloud data representing a three-dimensional point cloud is divided into multiple parts, and the point cloud data unit before division, and generating a bitstream by encoding the multiple divided data obtained from the divided point cloud data, or the point cloud data before division, to satisfy the determined first maximum number of bits, wherein the bitstream includes first bit number information indicating the first maximum number of bits.

[0017] According to this, the three-dimensional data encoding method generates a bitstream containing first bit number information indicating the first maximum number of bits in the encoded data after encoding. Therefore, the three-dimensional data decoding device can determine whether it can properly decode the encoded data without analyzing the bitstream. This reduces the processing load on the three-dimensional data decoding device.

[0018] For example, the point cloud data includes position information for each three-dimensional point in the three-dimensional point cloud, and the first maximum number of bits refers to the number of bits after encoding the position information. In the generation process, the bitstream may be generated by encoding the multiple divided data obtained by dividing the point cloud data, or the position information of the point cloud data before division, to satisfy the determined first maximum number of bits.

[0019] According to this, the three-dimensional data encoding method generates a bitstream that includes first bit information indicating the first maximum number of bits of the encoded position information. Therefore, the three-dimensional data decoding device can determine whether it can properly decode the position information without analyzing the bitstream.

[0020] For example, the range of the number of three-dimensional points included in at least one of the divided data unit and the point cloud data unit is determined, and in the generation, the bitstream is generated by encoding the multiple divided data obtained by dividing the point cloud data, or the point cloud data before division, so as to satisfy the determined first maximum number of bits and the range of the number, and the bitstream may further include range information indicating the range of the number. This reduces the processing load on the three-dimensional data decoding device.

[0021] According to this, the three-dimensional data encoding method generates a bitstream that includes range information indicating the range of the number of three-dimensional points in the encoded data. Therefore, the three-dimensional data decoding device can determine whether it can properly decode the position information without analyzing the bitstream. This reduces the processing load on the three-dimensional data decoding device.

[0022] For example, the point cloud data further includes attribute information for each three-dimensional point in the three-dimensional point cloud, and the three-dimensional data encoding method further determines a second maximum number of bits after encoding the attribute information of at least one of the three-dimensional point clouds in the divided data unit and the point cloud data unit, and in the generation, (i) the position information of the divided data obtained from the point cloud data, or the point cloud data before division, is encoded to satisfy the first maximum number of bits determined, and (ii) the attribute information of the divided data obtained from the point cloud data, or the point cloud data before division, is encoded to satisfy the second maximum number of bits determined, thereby generating the bitstream, and the bitstream may further include second bit number information indicating the second maximum number of bits.

[0023] According to this, the three-dimensional data encoding method generates a bitstream that includes a second bit number information indicating the second maximum number of bits in the encoded attribute information. Therefore, the three-dimensional data decoding device can determine whether it can properly decode the attribute information without analyzing the bitstream. This reduces the processing load on the three-dimensional data decoding device.

[0024] A three-dimensional data decoding method according to one aspect of the present disclosure acquires a bitstream that includes encoded data which is the encoded data of at least one of the divided data units when point cloud data representing a three-dimensional point cloud is divided into multiple parts, and first bit number information which indicates the first maximum number of bits of the encoded data, determines whether the acquired bitstream satisfies the first maximum number of bits indicated by the first bit number information, and decodes the encoded data if it is determined that the bitstream satisfies the first maximum number of bits.

[0025] According to this, the three-dimensional data decoding method obtains first bit number information, which indicates the first maximum number of bits in the encoded data after encoding, from the bitstream, and can appropriately decode the point cloud data based on the obtained first bit number information.

[0026] For example, in the decoding process, if it is determined that the bitstream does not satisfy the first maximum number of bits, the encoded data does not need to be decoded.

[0027] According to this method, the decoding process for encoded data of bitstreams that cannot be properly decoded is not performed, thus reducing the processing load.

[0028] For example, the point cloud data includes positional information for each three-dimensional point in the three-dimensional point cloud, and the first maximum number of bits may also be the number of bits after encoding the positional information.

[0029] According to this, the three-dimensional data decoding method obtains first bit number information indicating the first maximum number of bits of encoded position information from the bitstream, and can appropriately decode the point cloud data based on the obtained first bit number information.

[0030] For example, the bitstream further includes range information indicating the range of the number of three-dimensional points included in at least one of the divided data unit and the point cloud data unit, and the determination further determines whether the bitstream satisfies the range of the number indicated by the range information, and in the decoding, if it is determined that the bitstream satisfies the first maximum number of bits and satisfies the range of the number, the encoded data is decoded, and if it is determined that the bitstream does not satisfy the first maximum number of bits or does not satisfy the range of the number, the encoded data does not need to be decoded.

[0031] According to this, the three-dimensional data decoding method obtains range information from the bitstream that indicates the range of the number of three-dimensional points in the encoded data, and can appropriately decode the point cloud data based on the obtained range information.

[0032] For example, the point cloud data further includes attribute information for each three-dimensional point in the three-dimensional point cloud, and the bitstream further includes second bit number information indicating the second maximum number of bits after encoding of the attribute information of at least one of the three-dimensional point clouds in the divided data unit and the point cloud data unit. The determination further determines whether the bitstream satisfies the second maximum number of bits indicated by the second bit number information. In the decoding, if it is determined that the bitstream satisfies both the first maximum number of bits and the second maximum number of bits, the encoded data is decoded. If it is determined that the bitstream does not satisfy either the first maximum number of bits or the second maximum number of bits, the encoded data does not need to be decoded.

[0033] According to this, the three-dimensional data decoding method obtains second bit number information, which indicates the second maximum number of bits of the encoded attribute information, from the bitstream, and can appropriately decode the point cloud data based on the obtained second bit number information.

[0034] (Embodiment 1) When using encoded point cloud data in actual devices or services, it is desirable to send and receive necessary information depending on the application in order to reduce network bandwidth. However, until now, such functionality has not existed in the encoded structure of three-dimensional data, nor has there been an encoding method for that purpose.

[0035] This embodiment describes a three-dimensional data encoding method and a three-dimensional data encoding device for providing a function to send and receive information necessary for use in encoded data of a three-dimensional point cloud, a three-dimensional data decoding method and a three-dimensional data decoding device for decoding the encoded data, a three-dimensional data multiplexing method for multiplexing the encoded data, and a three-dimensional data transmission method for transmitting the encoded data.

[0036] In particular, while two encoding methods (encoding schemes) for point cloud data are currently being considered, the structure of the encoded data and the method for storing the encoded data in a system format have not been defined. As a result, there is a problem in that MUX processing (multiplexing), transmission, or storage cannot be performed in the encoding unit.

[0037] Furthermore, there has been no existing method to support formats like PCC (Point Cloud Compression) that use a mixture of two codecs, a first encoding method and a second encoding method.

[0038] This embodiment describes the structure of PCC encoded data in which two codecs, a first encoding method and a second encoding method, coexist, and a method for storing the encoded data in a system format.

[0039] First, the configuration of the three-dimensional data (point cloud data) encoding and decoding system according to this embodiment will be described. Figure 1 is a diagram showing an example of the configuration of the three-dimensional data encoding and decoding system according to this embodiment. As shown in Figure 1, the three-dimensional data encoding and decoding system includes a three-dimensional data encoding system 4601, a three-dimensional data decoding system 4602, a sensor terminal 4603, and an external connection unit 4604.

[0040] The three-dimensional data encoding system 4601 generates encoded data or multiplexed data by encoding point cloud data, which is three-dimensional data. The three-dimensional data encoding system 4601 may be a three-dimensional data encoding device implemented by a single device, or it may be a system implemented by multiple devices. Furthermore, the three-dimensional data encoding device may include some of the multiple processing units included in the three-dimensional data encoding system 4601.

[0041] The three-dimensional data encoding system 4601 includes a point cloud data generation system 4611, a presentation unit 4612, an encoding unit 4613, a multiplexing unit 4614, an input / output unit 4615, and a control unit 4616. The point cloud data generation system 4611 includes a sensor information acquisition unit 4617 and a point cloud data generation unit 4618.

[0042] The sensor information acquisition unit 4617 acquires sensor information from the sensor terminal 4603 and outputs the sensor information to the point cloud data generation unit 4618. The point cloud data generation unit 4618 generates point cloud data from the sensor information and outputs the point cloud data to the encoding unit 4613.

[0043] The display unit 4612 presents sensor information or point cloud data to the user. For example, the display unit 4612 displays information or images based on sensor information or point cloud data.

[0044] The encoding unit 4613 encodes (compresses) the point cloud data and outputs the resulting encoded data, control information obtained during the encoding process, and other additional information to the multiplexing unit 4614. The additional information includes, for example, sensor information.

[0045] The multiplexing unit 4614 generates multiplexed data by multiplexing the encoded data input from the encoding unit 4613, control information, and additional information. The format of the multiplexed data is, for example, a file format for storage or a packet format for transmission.

[0046] The input / output unit 4615 (for example, the communication unit or interface) outputs the multiplexed data to the outside. Alternatively, the multiplexed data is stored in a storage unit such as internal memory. The control unit 4616 (or application execution unit) controls each processing unit. In other words, the control unit 4616 performs control such as encoding and multiplexing.

[0047] The sensor information may also be input to the encoding unit 4613 or the multiplexing unit 4614. Furthermore, the input / output unit 4615 may output the point cloud data or encoded data directly to the outside.

[0048] The transmission signal (multiplexed data) output from the three-dimensional data encoding system 4601 is input to the three-dimensional data decoding system 4602 via the external connection unit 4604.

[0049] The three-dimensional data decoding system 4602 generates point cloud data, which is three-dimensional data, by decoding encoded data or multiplexed data. The three-dimensional data decoding system 4602 may be a three-dimensional data decoding device implemented by a single device, or it may be a system implemented by multiple devices. Furthermore, the three-dimensional data decoding device may include some of the multiple processing units included in the three-dimensional data decoding system 4602.

[0050] The three-dimensional data decoding system 4602 includes a sensor information acquisition unit 4621, an input / output unit 4622, a demultiplexing unit 4623, a decoding unit 4624, a presentation unit 4625, a user interface 4626, and a control unit 4627.

[0051] The sensor information acquisition unit 4621 acquires sensor information from the sensor terminal 4603.

[0052] The input / output unit 4622 acquires the transmission signal, decodes the multiplexed data (file format or packet) from the transmission signal, and outputs the multiplexed data to the demultiplexing unit 4623.

[0053] The demultiplexing unit 4623 acquires encoded data, control information, and additional information from the multiplexed data, and outputs the encoded data, control information, and additional information to the decoding unit 4624.

[0054] The decoding unit 4624 reconstructs the point cloud data by decoding the encoded data.

[0055] The presentation unit 4625 presents point cloud data to the user. For example, the presentation unit 4625 displays information or images based on the point cloud data. The user interface 4626 acquires instructions based on user operations. The control unit 4627 (or application execution unit) controls each processing unit. In other words, the control unit 4627 performs control such as demultiplexing, decoding, and presentation.

[0056] The input / output unit 4622 may acquire point cloud data or encoded data directly from an external source. The presentation unit 4625 may acquire additional information such as sensor information and present information based on that additional information. The presentation unit 4625 may also make presentations based on user instructions acquired through the user interface 4626.

[0057] The sensor terminal 4603 generates sensor information, which is information obtained from the sensor. The sensor terminal 4603 is a terminal equipped with a sensor or camera, and may be, for example, a mobile object such as an automobile, an aerial object such as an airplane, a mobile terminal, or a camera.

[0058] The sensor information that can be acquired by the sensor terminal 4603 includes, for example, (1) the distance between the sensor terminal 4603 and the object, or the reflectivity of the object, obtained from a LiDAR, millimeter-wave radar, or infrared sensor, and (2) the distance between the camera and the object, or the reflectivity of the object, obtained from multiple monocular camera images or stereo camera images. The sensor information may also include the sensor's attitude, orientation, gyroscope (angular velocity), position (GPS information or altitude), speed, or acceleration. The sensor information may also include temperature, atmospheric pressure, humidity, or magnetism.

[0059] The external connection unit 4604 is implemented by an integrated circuit (LSI or IC), an external storage unit, communication with a cloud server via the internet, or broadcasting, etc.

[0060] Next, we will explain point cloud data. Figure 2 shows the structure of point cloud data. Figure 3 shows an example of the structure of a data file containing information about point cloud data.

[0061] Point cloud data contains data for multiple points. Each point's data includes location information (three-dimensional coordinates) and attribute information related to that location. A collection of these points is called a point cloud. For example, a point cloud represents the three-dimensional shape of an object.

[0062] Position information, such as three-dimensional coordinates, is sometimes referred to as geometry. Furthermore, the data for each point may include attribute information of multiple attribute types. Attribute types include, for example, color or reflectance.

[0063] One location information may be associated with one attribute information, or multiple attribute information of different attribute types may be associated with one location information. Furthermore, multiple attribute information of the same attribute type may be associated with one location information.

[0064] The example data file structure shown in Figure 3 represents a case where location information and attribute information correspond one-to-one, and it shows the location information and attribute information of the N points that make up the point cloud data.

[0065] Location information includes, for example, information for the three axes: x, y, and z. Attribute information includes, for example, RGB color information. A typical data file is a ply file.

[0066] Next, we will explain the types of point cloud data. Figure 4 is a diagram illustrating the types of point cloud data. As shown in Figure 4, point cloud data includes static objects and dynamic objects.

[0067] A static object is three-dimensional point cloud data for any given time (a specific moment). A dynamic object is three-dimensional point cloud data that changes over time. Hereafter, three-dimensional point cloud data for a given time will be referred to as a PCC frame, or simply a frame.

[0068] The object can be a point cloud with a somewhat limited area, like regular video data, or it can be a large-scale point cloud with no area limitations, like map information.

[0069] Furthermore, point cloud data of various densities may exist, including both sparse and dense point cloud data.

[0070] The details of each processing unit are described below. Sensor information is acquired by various methods, such as distance sensors like LIDAR or rangefinders, stereo cameras, or combinations of multiple monocular cameras. The point cloud data generation unit 4618 generates point cloud data based on the sensor information obtained by the sensor information acquisition unit 4617. The point cloud data generation unit 4618 generates position information as point cloud data and adds attribute information to the position information.

[0071] The point cloud data generation unit 4618 may process the point cloud data when generating position information or adding attribute information. For example, the point cloud data generation unit 4618 may reduce the amount of data by deleting point clouds with overlapping positions. The point cloud data generation unit 4618 may also transform the position information (such as position shifting, rotation, or normalization) or render the attribute information.

[0072] In Figure 1, the point cloud data generation system 4611 is included in the three-dimensional data encoding system 4601, but it may also be provided independently outside of the three-dimensional data encoding system 4601.

[0073] The encoding unit 4613 generates encoded data by encoding the point cloud data based on a predetermined encoding method. There are two main types of encoding methods. The first is an encoding method using positional information, which will be referred to as the first encoding method hereafter. The second is an encoding method using a video codec, which will be referred to as the second encoding method hereafter.

[0074] The decoding unit 4624 decodes the point cloud data by decoding the encoded data based on a predetermined encoding method.

[0075] The multiplexing unit 4614 generates multiplexed data by multiplexing the encoded data using an existing multiplexing method. The generated multiplexed data is transmitted or stored. In addition to PCC encoded data, the multiplexing unit 4614 multiplexes other media such as video, audio, subtitles, applications, files, or reference time information. Furthermore, the multiplexing unit 4614 may also multiplex attribute information related to sensor information or point cloud data.

[0076] Multiplexing methods or file formats include ISOBMFF, ISOBMFF-based transmission methods such as MPEG-DASH, MMT, MPEG-2 TS Systems, and RMP.

[0077] The demultiplexing unit 4623 extracts PCC encoded data, other media, and time information from the multiplexed data.

[0078] The input / output unit 4615 transmits the multiplexed data using a method appropriate to the transmission medium or storage medium, such as broadcasting or communication. The input / output unit 4615 may communicate with other devices via the internet or with storage units such as cloud servers.

[0079] Communication protocols such as HTTP, FTP, TCP, or UDP can be used. Either a pull-type or push-type communication method may be employed.

[0080] Either wired or wireless transmission may be used. Wired transmission methods include Ethernet®, USB, RS-232C, HDMI®, or coaxial cable. Wireless transmission methods include wireless LAN, Wi-Fi®, Bluetooth®, or millimeter wave.

[0081] Furthermore, broadcasting formats such as DVB-T2, DVB-S2, DVB-C2, ATSC3.0, or ISDB-S3 may be used.

[0082] Figure 5 shows the configuration of a first encoding unit 4630, which is an example of an encoding unit 4613 that performs encoding using the first encoding method. Figure 6 is a block diagram of the first encoding unit 4630. The first encoding unit 4630 generates encoded data (encoded stream) by encoding point cloud data using the first encoding method. This first encoding unit 4630 includes a location information encoding unit 4631, an attribute information encoding unit 4632, an additional information encoding unit 4633, and a multiplexing unit 4634.

[0083] The first encoding unit 4630 is characterized by performing encoding while being aware of the three-dimensional structure. Furthermore, the first encoding unit 4630 is characterized by the attribute information encoding unit 4632 performing encoding using information obtained from the location information encoding unit 4631. The first encoding method is also called GPCC (Geometry-based PCC).

[0084] The point cloud data is PCC point cloud data such as a PLY file, or PCC point cloud data generated from sensor information, and includes position information, attribute information, and other additional information (metadata). The position information is input to the position information encoding unit 4631, the attribute information is input to the attribute information encoding unit 4632, and the additional information is input to the additional information encoding unit 4633.

[0085] The location information encoding unit 4631 generates encoded location information (Compressed Geometry), which is encoded data, by encoding location information. For example, the location information encoding unit 4631 encodes location information using an N-tree structure such as an octree. Specifically, in an octree, the target space is divided into 8 nodes (subspaces), and 8 bits of information (occupancy code) are generated to indicate whether or not a point cloud is contained in each node. Furthermore, nodes containing point clouds are further divided into 8 nodes, and 8 bits of information are generated to indicate whether or not a point cloud is contained in each of these 8 nodes. This process is repeated until the number of point clouds contained in a predetermined hierarchy or node falls below a threshold.

[0086] The attribute information encoding unit 4632 generates encoded attribute information (Compressed Attribute), which is encoded data, by encoding it using the configuration information generated by the location information encoding unit 4631. For example, the attribute information encoding unit 4632 determines the reference point (reference node) to be referenced in encoding the target point (target node) to be processed, based on the octave tree structure generated by the location information encoding unit 4631. For example, the attribute information encoding unit 4632 references a surrounding node or adjacent node whose parent node in the octave tree is the same as the target node. Note that the method for determining the reference relationship is not limited to this.

[0087] Furthermore, the attribute information encoding process may include at least one of the following: quantization, prediction, and arithmetic encoding. In this case, a reference means using a reference node to calculate the predicted value of the attribute information, or using the state of a reference node (for example, occupancy information indicating whether or not the reference node contains a point cloud) to determine the encoding parameters. For example, encoding parameters may be quantization parameters in the quantization process, or context in arithmetic encoding.

[0088] The additional information encoding unit 4633 generates encoded data, or compressed additional information (Compressed MetaData), by encoding the compressible data from the additional information.

[0089] The multiplexing unit 4634 generates a compressed stream, which is encoded data, by multiplexing encoded position information, encoded attribute information, encoded additional information, and other additional information. The generated compressed stream is output to a processing unit of the system layer (not shown).

[0090] Next, we will describe a first decoding unit 4640, which is an example of a decoding unit 4624 that performs decoding of the first encoding method. Figure 7 is a diagram showing the configuration of the first decoding unit 4640. Figure 8 is a block diagram of the first decoding unit 4640. The first decoding unit 4640 generates point cloud data by decoding the encoded data (encoded stream) encoded by the first encoding method using the first encoding method. This first decoding unit 4640 includes a demultiplexing unit 4641, a location information decoding unit 4642, an attribute information decoding unit 4643, and an additional information decoding unit 4644.

[0091] A compressed stream, which is encoded data, is input to the first decoding unit 4640 from a processing unit of the system layer (not shown).

[0092] The demultiplexing unit 4641 separates encoded location information (Compressed Geometry), encoded attribute information (Compressed Attribute), encoded additional information (Compressed MetaData), and other additional information from the encoded data.

[0093] The location information decoding unit 4642 generates location information by decoding the encoded location information. For example, the location information decoding unit 4642 reconstructs the location information of a point cloud represented by three-dimensional coordinates from encoded location information represented by an N-tree structure such as an octree.

[0094] The attribute information decoding unit 4643 decodes the encoded attribute information based on the configuration information generated by the location information decoding unit 4642. For example, the attribute information decoding unit 4643 determines the reference point (reference node) to be referenced in the decoding of the target point (target node) to be processed, based on the octave tree structure obtained by the location information decoding unit 4642. For example, the attribute information decoding unit 4643 references a surrounding node or adjacent node whose parent node in the octave tree is the same as the target node. Note that the method for determining the reference relationship is not limited to this.

[0095] Furthermore, the attribute information decoding process may include at least one of the following: inverse quantization, prediction, and arithmetic decoding. In this case, "reference" means using a reference node to calculate the predicted value of the attribute information, or using the state of the reference node (for example, occupancy information indicating whether or not the reference node contains a point cloud) to determine the decoding parameters. For example, decoding parameters may be quantization parameters in the inverse quantization process, or context in arithmetic decoding.

[0096] The additional information decoding unit 4644 generates additional information by decoding the encoded additional information. The first decoding unit 4640 uses the additional information necessary for decoding location information and attribute information during decoding and outputs the additional information necessary for the application to the outside.

[0097] Next, we will describe a second encoding unit 4650, which is an example of an encoding unit 4613 that performs encoding using the second encoding method. Figure 9 is a diagram showing the configuration of the second encoding unit 4650. Figure 10 is a block diagram of the second encoding unit 4650.

[0098] The second encoding unit 4650 generates encoded data (encoded stream) by encoding the point cloud data using a second encoding method. This second encoding unit 4650 includes an additional information generation unit 4651, a position image generation unit 4652, an attribute image generation unit 4653, a video encoding unit 4654, an additional information encoding unit 4655, and a multiplexing unit 4656.

[0099] The second encoding unit 4650 generates a position image and an attribute image by projecting a three-dimensional structure onto a two-dimensional image, and then encodes the generated position image and attribute image using an existing video encoding scheme. The second encoding method is also called VPCC (Video based PCC).

[0100] The point cloud data is PCC point cloud data such as a PLY file, or PCC point cloud data generated from sensor information, and includes position information, attribute information, and other additional information (metadata).

[0101] The additional information generation unit 4651 generates map information for multiple two-dimensional images by projecting a three-dimensional structure onto a two-dimensional image.

[0102] The position image generation unit 4652 generates a position image (geometry image) based on position information and map information generated by the additional information generation unit 4651. This position image is, for example, a depth image in which the distance is indicated as a pixel value. This depth image may be an image of multiple point clouds viewed from one viewpoint (an image of multiple point clouds projected onto a single two-dimensional plane), or multiple images of multiple point clouds viewed from multiple viewpoints, or a single image formed by integrating these multiple images.

[0103] The attribute image generation unit 4653 generates an attribute image based on attribute information and map information generated by the additional information generation unit 4651. This attribute image is, for example, an image in which attribute information (e.g., color (RGB)) is shown as pixel values. This image may be an image of multiple point clouds viewed from one viewpoint (an image of multiple point clouds projected onto a single two-dimensional plane), or multiple images of multiple point clouds viewed from multiple viewpoints, or a single image formed by integrating these multiple images.

[0104] The video encoding unit 4654 generates encoded data, namely a compressed geometric image and a compressed attribute image, by encoding the position image and attribute image using a video encoding scheme. Any known encoding scheme may be used as the video encoding scheme. For example, the video encoding scheme may be AVC or HEVC.

[0105] The additional information encoding unit 4655 generates encoded additional information (Compressed MetaData) by encoding additional information and map information included in the point cloud data.

[0106] The multiplexing unit 4656 generates a compressed stream, which is encoded data, by multiplexing the encoded position image, encoded attribute image, encoded additional information, and other additional information. The generated compressed stream is output to a processing unit of the system layer (not shown).

[0107] Next, we will describe a second decoding unit 4660, which is an example of a decoding unit 4624 that performs decoding of the second encoding method. Figure 11 is a diagram showing the configuration of the second decoding unit 4660. Figure 12 is a block diagram of the second decoding unit 4660. The second decoding unit 4660 generates point cloud data by decoding the encoded data (encoded stream) encoded by the second encoding method using the second encoding method. This second decoding unit 4660 includes a demultiplexing unit 4661, a video decoding unit 4662, an additional information decoding unit 4663, a location information generation unit 4664, and an attribute information generation unit 4665.

[0108] A compressed stream, which is encoded data, is input to the second decoding unit 4660 from a processing unit of the system layer (not shown).

[0109] The demultiplexing unit 4661 separates the encoded location image (Compressed Geometry Image), encoded attribute image (Compressed Attribute Image), encoded additional information (Compressed MetaData), and other additional information from the encoded data.

[0110] The video decoding unit 4662 generates a position image and an attribute image by decoding the encoded position image and the encoded attribute image using a video encoding scheme. Any known encoding scheme may be used as the video encoding scheme. For example, the video encoding scheme may be AVC or HEVC.

[0111] The additional information decoding unit 4663 generates additional information, including map information, by decoding the encoded additional information.

[0112] The location information generation unit 4664 generates location information using the location image and map information. The attribute information generation unit 4665 generates attribute information using the attribute image and map information.

[0113] The second decoding unit 4660 uses the additional information necessary for decoding during the decoding process and outputs the additional information necessary for the application to the outside.

[0114] The following describes the challenges in the PCC encoding scheme. Figure 13 is a diagram showing the protocol stack involved in PCC encoded data. Figure 13 shows an example in which data from other media, such as video (e.g., HEVC) or audio, is multiplexed onto PCC encoded data and then transmitted or stored.

[0115] Multiplexing schemes and file formats have the function of multiplexing, transmitting, or storing various encoded data. In order to transmit or store encoded data, the encoded data must be converted into the format of the multiplexing scheme. For example, HEVC specifies a technique in which encoded data is stored in a data structure called a NAL unit, and the NAL unit is stored in ISOBMFF.

[0116] On the other hand, while two encoding methods are currently being considered for encoding point cloud data, the structure of the encoded data and the method for storing the encoded data in a system format have not been defined. As a result, there is a problem in that MUX processing (multiplexing), transmission, and storage cannot be performed in the encoding unit.

[0117] In the following text, unless a specific encoding method is mentioned, either the first encoding method or the second encoding method will be referred to.

[0118] (Embodiment 2) This embodiment describes the types of encoded data (geometry, attribute, and metadata) generated by the first encoding unit 4630 or the second encoding unit 4650 described above, the method for generating metadata, and the multiplexing process in the multiplexing unit. Note that metadata may also be referred to as parameter sets or control information.

[0119] In this embodiment, we will explain using the dynamic object (three-dimensional point cloud data that changes over time) described in Figure 4 as an example, but the same method may be used for static objects (three-dimensional point cloud data at any given time).

[0120] Figure 14 shows the configuration of the encoding unit 4801 and the multiplexing unit 4802 included in the three-dimensional data encoding device according to this embodiment. The encoding unit 4801 corresponds, for example, to the first encoding unit 4630 or the second encoding unit 4650 described above. The multiplexing unit 4802 corresponds to the multiplexing unit 4634 or 4656 described above.

[0121] The encoding unit 4801 encodes point cloud data from multiple PCC (Point Cloud Compression) frames and generates encoded data (Multiple Compressed Data) containing multiple location information, attribute information, and additional information.

[0122] The multiplexing unit 4802 converts data of multiple data types (location information, attribute information, and additional information) into NAL units, thereby transforming the data into a data configuration that takes into account data access by the decoding device.

[0123] Figure 15 shows an example of the structure of encoded data generated by the encoding unit 4801. The arrows in the figure indicate dependencies related to the decoding of encoded data, with the source of the arrow depending on the data at the end of the arrow. In other words, the decoding device decodes the data at the end of the arrow and uses that decoded data to decode the source of the arrow. To put it another way, dependency means that the dependent data is referenced (used) in the processing of the dependent data (encoding or decoding, etc.).

[0124] First, let's explain the process of generating encoded location data. The encoding unit 4801 generates compressed location data (Compressed Geometry Data) for each frame by encoding the location information of each frame. The encoded location data is represented by G(i), where i represents the frame number or the time of the frame.

[0125] Furthermore, the encoding unit 4801 generates a position parameter set (GPS(i)) corresponding to each frame. The position parameter set includes parameters that can be used to decode the encoded position data. Also, the encoded position data for each frame depends on the corresponding position parameter set.

[0126] Furthermore, encoded position data consisting of multiple frames is defined as a position sequence (Geometry Sequence). The encoding unit 4801 generates a position sequence parameter set (Geometry Sequence PS: also written as Position SPS) that stores parameters commonly used for decoding multiple frames within the position sequence. The position sequence depends on the Position SPS.

[0127] Next, the process for generating encoded attribute data will be explained. The encoding unit 4801 generates compressed attribute data for each frame by encoding the attribute information of each frame. The compressed attribute data is represented by A(i). Figure 15 shows an example where attribute X and attribute Y exist, with the compressed attribute data for attribute X represented by AX(i) and the compressed attribute data for attribute Y represented by AY(i).

[0128] Furthermore, the encoding unit 4801 generates an attribute parameter set (APS(i)) corresponding to each frame. The attribute parameter set for attribute X is represented by AXPS(i), and the attribute parameter set for attribute Y is represented by AYPS(i). The attribute parameter set includes parameters that can be used to decode the encoded attribute information. The encoded attribute data depends on the corresponding attribute parameter set.

[0129] Furthermore, encoded attribute data consisting of multiple frames is defined as an attribute sequence. The encoding unit 4801 generates an attribute sequence parameter set (Attribute Sequence PS, also written as attribute SPS) that stores parameters commonly used for decoding multiple frames within the attribute sequence. The attribute sequence depends on the attribute SPS.

[0130] Furthermore, in the first encoding method, the encoded attribute data depends on the encoded position data.

[0131] Figure 15 also shows an example where there are two types of attribute information (attribute X and attribute Y). When there are two types of attribute information, for example, two encoding units generate the respective data and metadata. Also, for example, an attribute sequence is defined for each type of attribute information, and an attribute SPS is generated for each type of attribute information.

[0132] Note that Figure 15 shows an example where there is one type of positional information and two types of attribute information, but the example is not limited to this; there may be one type of attribute information or three or more types. In this case as well, encoded data can be generated in the same way. Furthermore, in the case of point cloud data that does not have attribute information, attribute information is not required. In that case, the encoding unit 4801 does not need to generate a parameter set related to attribute information.

[0133] Next, the process of generating additional information (metadata) will be described. The encoding unit 4801 generates a PCC stream PS (also written as Stream PS), which is a parameter set for the entire PCC stream. The encoding unit 4801 stores in Stream PS parameters that can be used in common for decoding one or more location sequences and one or more attribute sequences. For example, Stream PS includes identification information indicating the codec of the point cloud data, and information indicating the algorithm used for encoding. The location sequences and attribute sequences depend on Stream PS.

[0134] Next, we will explain the Access Unit and GOF. In this embodiment, we introduce the new concepts of Access Unit (AU) and GOF (Group of Frame).

[0135] An access unit is the basic unit for accessing data during decryption, and consists of one or more data points and one or more metadata points. For example, an access unit consists of location information at the same time and one or more attribute information points. A GOF (Group of Four) is a random access unit and consists of one or more access units.

[0136] The encoding unit 4801 generates an access unit header (AU Header) as identification information indicating the beginning of an access unit. The encoding unit 4801 stores parameters related to the access unit in the access unit header. For example, the access unit header includes the structure or information of the encoded data contained in the access unit. The access unit header also includes parameters commonly used in the data contained in the access unit, such as parameters related to the decoding of the encoded data.

[0137] The encoding unit 4801 may generate an access unit delimiter that does not include parameters related to the access unit, instead of an access unit header. This access unit delimiter is used as identification information to indicate the beginning of the access unit. The decoding device identifies the beginning of the access unit by detecting the access unit header or the access unit delimiter.

[0138] Next, the generation of identification information at the beginning of the GOF will be explained. The encoding unit 4801 generates a GOF header as identification information indicating the beginning of the GOF. The encoding unit 4801 stores parameters related to the GOF in the GOF header. For example, the GOF header includes the structure or information of the encoded data included in the GOF. The GOF header also includes parameters commonly used in the data included in the GOF, such as parameters related to decoding the encoded data.

[0139] The encoding unit 4801 may generate a GOF delimiter that does not include parameters related to the GOF, instead of a GOF header. This GOF delimiter is used as identification information to indicate the beginning of the GOF. The decoding device identifies the beginning of the GOF by detecting either the GOF header or the GOF delimiter.

[0140] In PCC encoded data, for example, an access unit is defined as a PCC frame. The decoder accesses the PCC frame based on the identification information at the beginning of the access unit.

[0141] Furthermore, for example, a GOF (Group of Frames) is defined as a single random access unit. The decryption device accesses the random access unit based on the identification information at the beginning of the GOF. For example, if PCC frames are independent of each other and can be decrypted individually, then a PCC frame may be defined as a random access unit.

[0142] Furthermore, two or more PCC frames may be assigned to a single access unit, and multiple random access units may be assigned to a single GOF.

[0143] Furthermore, the encoding unit 4801 may define and generate parameter sets or metadata other than those described above. For example, the encoding unit 4801 may generate SEI (Supplemental Enhancement Information) that stores parameters that may not necessarily be used during decoding (optional parameters).

[0144] Next, we will explain the structure of the encoded data and how to store the encoded data in the NAL unit.

[0145] For example, a data format is defined for each type of encoded data. Figure 16 shows an example of encoded data and a NAL unit.

[0146] For example, as shown in Figure 16, encoded data includes a header and a payload. The encoded data may also include length information indicating the length (data volume) of the encoded data, header, or payload. Furthermore, the encoded data does not necessarily have to include a header.

[0147] The header includes, for example, identification information to identify the data. This identification information may indicate, for example, the data type or frame number.

[0148] The header contains, for example, identification information indicating a reference relationship. This identification information is stored in the header when there is a dependency between data, and it is information used to reference the referenced data from the source. For example, the header of the referenced data contains identification information to identify that data. The header of the referenced data contains identification information indicating the referenced data.

[0149] Furthermore, if the referenced or source can be identified or derived from other information, the identifying information for identifying the data or identifying information indicating the reference relationship may be omitted.

[0150] The multiplexing unit 4802 stores the encoded data in the payload of the NAL unit. The NAL unit header contains pcc_nal_unit_type, which is identification information for the encoded data. Figure 17 shows an example of the semantics of pcc_nal_unit_type.

[0151] As shown in Figure 17, when pcc_codec_type is Codec 1 (Codec1: First encoding method), values ​​of pcc_nal_unit_type from 0 to 10 are assigned to the encoded position data (Geometry), encoded attribute X data (AttributeX), encoded attribute Y data (AttributeY), position PS (Geom.PS), attribute XPS (AttrX.PS), attribute YPS (AttrX.PS), position SPS (Geometry Sequence PS), attribute XSPS (AttributeX Sequence PS), attribute YSPS (AttributeY Sequence PS), AU header (AU Header), and GOF header (GOF Header) in Codec 1. Values ​​11 and above are assigned to the reserves of Codec 1.

[0152] If pcc_codec_type is Codec 2 (the second encoding method), values ​​of pcc_nal_unit_type from 0 to 2 are assigned to the codec's Data A, Metadata A, and Metadata B. Values ​​3 and above are assigned to the backup of Codec 2.

[0153] Next, we will explain the data transmission order. Below, we will explain the constraints on the transmission order of the NAL unit.

[0154] The multiplexing unit 4802 sends out NAL units in groups of GOF or AU units. The multiplexing unit 4802 places a GOF header at the beginning of a GOF and an AU header at the beginning of an AU.

[0155] The multiplexing unit 4802 may provide a sequence parameter set (SPS) for each AU so that the decoding device can decode from the next AU even if data is lost due to packet loss or other reasons.

[0156] If there are dependencies in the encoded data related to decoding, the decoding device decodes the referenced data first, and then decodes the source data. In order for the decoding device to decode the data in the order it was received without rearranging it, the multiplexing unit 4802 sends the referenced data first.

[0157] Figure 18 shows an example of the transmission order of the NAL unit. Figure 18 shows three examples: location information priority, parameter priority, and data integration.

[0158] The location-prioritized transmission order is an example where location information and attribute information are transmitted together. In this transmission order, the transmission of location information is completed earlier than the transmission of attribute information.

[0159] For example, by using this transmission order, a decoding device that does not decode attribute information may be able to create a period of time where it does not process attribute information by ignoring the decoding of attribute information. Also, for example, a decoding device that wants to decode location information quickly may be able to decode the location information faster by obtaining the encoded location information data earlier.

[0160] Note that in Figure 18, the attributes XSPS and YSPS are combined and labeled as attribute SPS, but it is also acceptable to place attributes XSPS and YSPS separately.

[0161] In a parameter set priority transmission order, the parameter set is sent first, followed by the data.

[0162] As long as the constraints on the NAL unit transmission order are followed as described above, the multiplexing unit 4802 may transmit the NAL units in any order. For example, sequence identification information may be defined, and the multiplexing unit 4802 may have the function of transmitting NAL units in multiple patterns of order. For example, the sequence identification information of the NAL units may be stored in the stream PS.

[0163] The three-dimensional data decoding device may perform decoding based on sequence identification information. The three-dimensional data decoding device may instruct the three-dimensional data encoding device to send a desired transmission order, and the three-dimensional data encoding device (multiplexing unit 4802) may control the transmission order according to the instructed transmission order.

[0164] Furthermore, the multiplexing unit 4802 may generate encoded data that merges multiple functions, as long as it adheres to the constraints of the transmission order, such as the transmission order of data integration. For example, as shown in Figure 18, the GOF header and the AU header may be merged, or AXPS and AYPS may be merged. In this case, pcc_nal_unit_type is defined as an identifier indicating that the data has multiple functions.

[0165] The following describes modifications of this embodiment. PS has levels, such as frame-level PS, sequence-level PS, and PCC sequence-level PS. If we consider the PCC sequence level as the higher level and the frame level as the lower level, the following method may be used to store the parameters.

[0166] The default PS value is shown in the higher-level PS. If the value of the lower-level PS differs from the value of the higher-level PS, the PS value is shown in the lower-level PS. Alternatively, the PS value is not listed in the higher-level PS, but is listed in the lower-level PS. Alternatively, information on whether the PS value is shown in the lower-level PS, the higher-level PS, or both is shown in either the lower-level PS or the higher-level PS, or both. Alternatively, the lower-level PS may be merged with the higher-level PS. Alternatively, if the lower-level PS and the higher-level PS overlap, the multiplexing unit 4802 may omit sending one of them.

[0167] The encoding unit 4801 or the multiplexing unit 4802 may divide the data into slices or tiles and send out the divided data. The divided data includes information for identifying the divided data, and the parameter set includes parameters used for decoding the divided data. In this case, pcc_nal_unit_type is defined as an identifier indicating that it is data that stores data or parameters related to tiles or slices.

[0168] (Embodiment 3) HEVC coding has data partitioning tools such as slicing or tiling to enable parallel processing in the decoding device, but PCC (Point Cloud Compression) coding does not yet have such tools.

[0169] In PCC, various data partitioning methods are possible depending on parallel processing, compression efficiency, and compression algorithms. This section explains the definitions of slices and tiles, data structures, and transmission / reception methods.

[0170] Figure 19 is a block diagram showing the configuration of a first encoding unit 4910 included in the three-dimensional data encoding device according to this embodiment. The first encoding unit 4910 generates encoded data (encoded stream) by encoding point cloud data using a first encoding method (GPCC (Geometry based PCC)). This first encoding unit 4910 includes a division unit 4911, a plurality of position information encoding units 4912, a plurality of attribute information encoding units 4913, an additional information encoding unit 4914, and a multiplexing unit 4915.

[0171] The division unit 4911 generates multiple divided data by dividing the point cloud data. Specifically, the division unit 4911 generates multiple divided data by dividing the space of the point cloud data into multiple subspaces. Here, a subspace is either a tile or a slice, or a combination of a tile and a slice. More specifically, the point cloud data includes location information, attribute information, and additional information. The division unit 4911 divides the location information into multiple divided location information and the attribute information into multiple divided attribute information. The division unit 4911 also generates additional information related to the division.

[0172] Multiple location information encoding units 4912 generate multiple encoded location information by encoding multiple divided location information. For example, multiple location information encoding units 4912 process multiple divided location information in parallel.

[0173] The multiple attribute information encoding unit 4913 generates multiple encoded attribute information by encoding multiple divided attribute information. For example, the multiple attribute information encoding unit 4913 processes multiple divided attribute information in parallel.

[0174] The additional information encoding unit 4914 generates encoded additional information by encoding the additional information contained in the point cloud data and the additional information related to data division generated by the division unit 4911 during division.

[0175] The multiplexing unit 4915 generates encoded data (encoded stream) by multiplexing multiple encoded position information, multiple encoded attribute information, and encoded additional information, and transmits the generated encoded data. The encoded additional information is used during decoding.

[0176] In Figure 19, an example is shown where there are two location information encoding units 4912 and two attribute information encoding units 4913. However, the number of location information encoding units 4912 and attribute information encoding units 4913 may be one or three or more. Furthermore, multiple divided data may be processed in parallel within the same chip, such as multiple cores in a CPU, or in parallel across the cores of multiple chips, or across multiple cores on multiple chips.

[0177] Figure 20 is a block diagram showing the configuration of the first decoding unit 4920. The first decoding unit 4920 restores point cloud data by decoding encoded data (encoded stream) generated when point cloud data is encoded using a first encoding method (GPCC). This first decoding unit 4920 includes a demultiplexing unit 4921, multiple location information decoding units 4922, multiple attribute information decoding units 4923, an additional information decoding unit 4924, and a coupling unit 4925.

[0178] The demultiplexing unit 4921 generates multiple encoded position information, multiple encoded attribute information, and encoded additional information by demultiplexing the encoded data (encoded stream).

[0179] The multiple location information decoding units 4922 generate multiple segmented location information by decoding multiple encoded location information. For example, the multiple location information decoding units 4922 process multiple encoded location information in parallel.

[0180] The multiple attribute information decoding unit 4923 generates multiple segmented attribute information by decoding multiple encoded attribute information. For example, the multiple attribute information decoding unit 4923 processes multiple encoded attribute information in parallel.

[0181] Multiple additional information decoding units 4924 generate additional information by decoding encoded additional information.

[0182] The coupling unit 4925 generates position information by combining multiple division position information using additional information. The coupling unit 4925 generates attribute information by combining multiple division attribute information using additional information.

[0183] In Figure 20, an example is shown where there are two location information decoding units 4922 and two attribute information decoding units 4923. However, the number of location information decoding units 4922 and attribute information decoding units 4923 may be one each, or three or more. Furthermore, multiple divided data may be processed in parallel within the same chip, such as multiple cores in a CPU, or in parallel across the cores of multiple chips, or across multiple cores of multiple chips.

[0184] Next, the configuration of the division section 4911 will be described. Figure 21 is a block diagram of the division section 4911. The division section 4911 includes a slice division section 4931, a geometry tile division section 4932, and an attribute tile division section 4933.

[0185] The slice division unit 4931 generates multiple slice position information by dividing position information (Position(Geometry)) into slices. The slice division unit 4931 also generates multiple slice attribute information by dividing attribute information (Attribute) into slices. Furthermore, the slice division unit 4931 outputs slice additional information (SliceMetaData) which includes information related to slice division and information generated during slice division.

[0186] The location information tile division unit 4932 generates multiple divided location information (multiple tile location information) by dividing multiple slice location information into tiles. The location information tile division unit 4932 also outputs location tile additional information (Geometry Tile MetaData) which includes information related to the tile division of the location information and information generated in the tile division of the location information.

[0187] The attribute information tile division unit 4933 generates multiple divided attribute information (multiple tile attribute information) by dividing multiple slice attribute information into tiles. The attribute information tile division unit 4933 also outputs attribute tile additional information (Attribute Tile MetaData) which includes information related to the tile division of attribute information and information generated in the tile division of attribute information.

[0188] The number of slices or tiles to be divided must be one or more. In other words, it is not necessary to divide the slices or tiles.

[0189] Furthermore, while this example shows tile division after slicing, slicing may also be performed after tile division. In addition to slicing and tiling, new division types may be defined, and division may be performed using three or more division types.

[0190] The following describes methods for dividing point cloud data. Figure 22 shows examples of slicing and tiling.

[0191] First, the method of slice division will be described. The division unit 4911 divides the three-dimensional point cloud data into arbitrary point clouds in units of slices. In slice division, the division unit 4911 does not divide the position information and attribute information that constitute a point, but divides the position information and attribute information together. That is, the division unit 4911 performs slice division so that the position information and attribute information at an arbitrary point belong to the same slice. According to these, the number of divisions and the division method may be any method. Also, the minimum unit of division is a point. For example, the number of divisions of the position information and the attribute information is the same. For example, the three-dimensional point corresponding to the position information after slice division and the three-dimensional point corresponding to the attribute information are included in the same slice.

[0192] Also, the division unit 4911 generates slice additional information, which is additional information related to the number of divisions and the division method during slice division. The slice additional information is the same for the position information and the attribute information. For example, the slice additional information includes information indicating the reference coordinate position, size, or side length of the bounding box after division. Also, the slice additional information includes information indicating the number of divisions, the division type, and the like.

[0193] Next, the method of tile division will be described. The division unit 4911 divides the data divided by slice into slice position information (G slice) and slice attribute information (A slice), and divides the slice position information and the slice attribute information into tiles respectively.

[0194] In FIG. 22, an example of division using an octree structure is shown, but the number of divisions and the division method may be any method.

[0195] Also, the division unit 4911 may divide the position information and the attribute information by different division methods, or by the same division method. Also, the division unit 4911 may divide a plurality of slices into tiles by different division methods, or by the same division method.

[0196] In addition, the splitting unit 4911 generates tile addition information related to the number of splits and the splitting method during tile splitting. The tile addition information (position tile addition information and attribute tile addition information) is independent of the position information and the attribute information. For example, the tile addition information includes information indicating the reference coordinate position, size, or side length of the bounding box after splitting. The tile addition information also includes information indicating the number of splits and the split type, etc.

[0197] Next, an example of a method for splitting point cloud data into slices or tiles will be described. The splitting unit 4911 may use a predetermined method as the method for slice or tile splitting, or may adaptively switch the method to be used according to the point cloud data.

[0198] During slice splitting, the splitting unit 4911 divides the three-dimensional space collectively for the position information and the attribute information. For example, the splitting unit 4911 determines the shape of the object and divides the three-dimensional space into slices according to the shape of the object. For example, the splitting unit 4911 extracts an object such as a tree or a building and performs splitting in object units. For example, the splitting unit 4911 performs slice splitting so that the whole of one or more objects is included in one slice. Or, the splitting unit 4911 divides one object into a plurality of slices.

[0199] In this case, the encoding device may, for example, change the encoding method for each slice. For example, the encoding device may use a high-quality compression method for a specific object or a specific part of an object. In this case, the encoding device may store information indicating the encoding method for each slice in the additional information (metadata).

[0200] In addition, the splitting unit 4911 may perform slice splitting so that each slice corresponds to a predetermined coordinate space based on the map information or the position information.

[0201] When dividing into tiles, the division unit 4911 divides the position information and attribute information independently. For example, the division unit 4911 divides a slice into tiles according to the amount of data or processing load. For example, the division unit 4911 determines whether the amount of data in a slice (for example, the number of three-dimensional points included in the slice) is greater than a predetermined threshold. If the amount of data in a slice is greater than the threshold, the division unit 4911 divides the slice into tiles. If the amount of data in a slice is less than the threshold, the division unit 4911 does not divide the slice into tiles.

[0202] For example, the division unit 4911 divides the slice into tiles so that the processing amount or processing time in the decoding device is within a certain range (less than or equal to a predetermined value). This ensures that the processing amount per tile in the decoding device is constant, facilitating distributed processing in the decoding device.

[0203] Furthermore, if the processing load differs between location information and attribute information, for example, if the processing load of location information is greater than that of attribute information, the division unit 4911 will increase the number of divisions for location information to be greater than the number of divisions for attribute information.

[0204] Furthermore, for example, if the decoding device may decode and display location information quickly and decode and display attribute information later at a slower pace depending on the content, the division unit 4911 may divide the location information into more divisions than the attribute information. This allows the decoding device to process location information in parallel, thus enabling faster processing of location information than of attribute information.

[0205] Furthermore, the decoding device does not necessarily need to process the sliced ​​or tiled data in parallel; it may decide whether or not to process them in parallel depending on the number or capacity of the decoding processing units.

[0206] By dividing the data in the manner described above, adaptive encoding can be achieved according to the content or object. Furthermore, parallel processing can be implemented in the decoding process. This improves the flexibility of the point cloud coding system or point cloud decoding system.

[0207] Figure 23 shows examples of slice and tile division patterns. In the figure, DU stands for Data Unit, representing data for a tile or slice. Each DU also includes a Slice Index and a Tile Index. The number in the upper right corner of the DU indicates the Slice Index, and the number in the lower left corner indicates the Tile Index.

[0208] In Pattern 1, the number of divisions and the division method are the same for G slices and A slices in slice partitioning. In tile partitioning, the number of divisions and the division method for G slices are different from those for A slices. Also, the same number of divisions and division method are used between multiple G slices. The same number of divisions and division method are used between multiple A slices.

[0209] In Pattern 2, the number of divisions and the division method are the same for G slices and A slices in slice partitioning. In tile partitioning, the number of divisions and the division method for G slices are different from those for A slices. Also, the number of divisions and the division method differ between multiple G slices. The number of divisions and the division method differ between multiple A slices.

[0210] Next, the method for encoding the divided data will be described. The three-dimensional data encoding device (first encoding unit 4910) encodes each of the divided data. When encoding attribute information, the three-dimensional data encoding device generates dependency information as additional information, indicating which configuration information (location information, additional information, or other attribute information) was used as the basis for encoding. In other words, the dependency information indicates, for example, the configuration information of the reference (dependent). In this case, the three-dimensional data encoding device generates the dependency information based on the configuration information corresponding to the division shape of the attribute information. Note that the three-dimensional data encoding device may generate dependency information based on configuration information corresponding to multiple division shapes.

[0211] Dependency information is generated by a three-dimensional data encoding device, and the generated dependency information may be sent to a three-dimensional data decoding device. Alternatively, the three-dimensional data decoding device may generate the dependency information, and the three-dimensional data encoding device may not need to send the dependency information. Also, the dependencies used by the three-dimensional data encoding device may be predetermined, and the three-dimensional data encoding device may not need to send the dependency information.

[0212] FIG. 24 is a diagram showing an example of the dependency of each data. The tip of the arrow in the figure indicates the dependent destination, and the origin of the arrow indicates the dependent source. The three-dimensional data decoding device decodes the data in the order from the dependent destination to the dependent source. Also, the data shown by the solid line in the figure is the data actually sent, and the data shown by the dotted line is the data not sent.

[0213] Also, in the same figure, G indicates position information, and A indicates attribute information. G s1 indicates the position information of slice number 1, G s2 indicates the position information of slice number 2. G s1t1 indicates the position information of slice number 1 and tile number 1, G s1t2 indicates the position information of slice number 1 and tile number 2, G s2t1 indicates the position information of slice number 2 and tile number 1, G s2t2 indicates the position information of slice number 2 and tile number 2. Similarly, A s1 indicates the attribute information of slice number 1, A s2 indicates the attribute information of slice number 2. A s1t1 indicates the attribute information of slice number 1 and tile number 1, A s1t2 indicates the attribute information of slice number 1 and tile number 2, A s2t1 indicates the attribute information of slice number 2 and tile number 1, A s2t2 indicates the attribute information of slice number 2 and tile number 2.

[0214] Mslice indicates slice addition information, MGtile indicates position tile addition information, and MAtile indicates attribute tile addition information. D s1t1 is the attribute information As1t1 shows the dependency information of D s2t1 is the attribute information A s2t1 shows the dependency information.

[0215] Also, the three-dimensional data encoding device may rearrange the data in the decoding order so that the three-dimensional data decoding device does not need to rearrange the data. Note that the data may be rearranged in the three-dimensional data decoding device, or the data may be rearranged in both the three-dimensional data encoding device and the three-dimensional data decoding device.

[0216] FIG. 25 is a diagram showing an example of the decoding order of data. In the example of FIG. 25, decoding is performed in order from the left data. The three-dimensional data decoding device decodes the dependent data first from the dependent data. For example, the three-dimensional data encoding device rearranges the data in advance and sends it out in this order. Note that any order may be used as long as the dependent data comes first. Also, the three-dimensional data encoding device may send the additional information and the dependency information before the data.

[0217] FIG. 26 is a flowchart showing the processing flow by the three-dimensional data encoding device. First, the three-dimensional data encoding device encodes the data of a plurality of slices or tiles as described above (S4901). Next, the three-dimensional data encoding device rearranges the data so that the dependent data comes first as shown in FIG. 25 (S4902). Next, the three-dimensional data encoding device multiplexes (NAL unitizes) the rearranged data (S4903).

[0218] Next, the configuration of the combining unit 4925 included in the first decoding unit 4920 will be described. FIG. 27 is a block diagram showing the configuration of the combining unit 4925. The combining unit 4925 includes a position information tile combining unit 4941 (Geometry Tile Combiner), an attribute information tile combining unit 4942 (Attribute Tile Combiner), and a slice combining unit (Slice Combiner).

[0219] The position information tile joining unit 4941 generates multiple slice position information by joining multiple divided position information using position tile additional information. The attribute information tile joining unit 4942 generates multiple slice attribute information by joining multiple divided attribute information using attribute tile additional information.

[0220] The slice joining unit 4943 generates position information by combining multiple slice position information using slice additional information. Furthermore, the slice joining unit 4943 generates attribute information by combining multiple slice attribute information using slice additional information.

[0221] The number of slices or tiles to be divided must be one or more. In other words, the slices or tiles do not need to be divided at all.

[0222] Furthermore, while this example shows tile division after slicing, slicing may also be performed after tile division. In addition to slicing and tiling, new division types may be defined, and division may be performed using three or more division types.

[0223] Next, the structure of the sliced ​​or tiled encoded data and the method of storing the encoded data in the NAL unit (multiplexing method) will be explained. Figure 28 shows the structure of the encoded data and the method of storing the encoded data in the NAL unit.

[0224] The encoded data (splitting position information and splitting attribute information) is stored in the NAL unit's payload.

[0225] Encoded data includes a header and a payload. The header includes identification information to identify the data contained in the payload. This identification information includes, for example, the type of slice or tile division (slice_type, tile_type), index information to identify the slice or tile (slice_idx, tile_idx), location information of the data (slice or tile), or the address of the data (address). Index information to identify a slice is also written as SliceIndex. Index information to identify a tile is also written as TileIndex. The type of division can be, for example, a method based on the object shape as described above, a method based on map information or location information, or a method based on the amount of data or processing amount.

[0226] Furthermore, all or part of the above information may be stored in one of the headers of the partitioned location information and the partitioned attribute information, but not in the other. For example, if the same partitioning method is used for both location information and attribute information, the partitioning type (slice_type, tile_type) and index information (slice_idx, tile_idx) will be the same for both location information and attribute information. Therefore, this information may be included in one of the headers of the location information or attribute information. For example, if attribute information depends on location information, the location information is processed first. Therefore, this information may be included in the header of the location information, but not in the header of the attribute information. In this case, the three-dimensional data decoding device will determine, for example, that the dependent attribute information belongs to the same slice or tile as the dependent location information slice or tile.

[0227] Furthermore, additional information related to slice or tile division (slice additional information, location tile additional information, or attribute tile additional information), and dependency information indicating dependencies, etc., may be stored in an existing parameter set (GPS, APS, location SPS, or attribute SPS, etc.) and transmitted. If the division method changes from frame to frame, information indicating the division method may be stored in the parameter set for each frame (GPS or APS, etc.). If the division method does not change within a sequence, information indicating the division method may be stored in the parameter set for each sequence (location SPS or attribute SPS). Moreover, if the same division method is used for location information and attribute information, information indicating the division method may be stored in the parameter set of the PCC stream (stream PS).

[0228] Furthermore, the above information may be stored in any of the parameter sets described above, or in multiple parameter sets. Alternatively, a parameter set for tiling or slicing may be defined, and the above information may be stored in that parameter set. Additionally, this information may be stored in the header of the encoded data.

[0229] Furthermore, the header of the encoded data includes identification information indicating dependencies. In other words, if there are dependencies between data, the header includes identification information for referencing the dependent data from the dependent data source. For example, the header of the dependent data includes identification information to identify that data. The header of the dependent data includes identification information indicating the dependent data. Note that if the identification information for identifying the data, additional information related to slicing or tiling, and identification information indicating dependencies can be identified or derived from other information, this information may be omitted.

[0230] Next, the flow of the point cloud data encoding and decoding processes according to this embodiment will be described. Figure 29 is a flowchart of the point cloud data encoding process according to this embodiment.

[0231] First, the three-dimensional data encoding device determines the division method to be used (S4911). This division method includes whether or not to perform slicing division or tiling division. The division method may also include the number of divisions if slicing or tiling division is performed, and the type of division. The type of division refers to methods based on object shape, methods based on map information or location information, or methods based on data volume or processing volume, as described above. The division method may also be predetermined.

[0232] If slice splitting is performed (Yes in S4912), the three-dimensional data encoding device generates multiple slice location information and multiple slice attribute information by splitting the location information and attribute information together (S4913). The three-dimensional data encoding device also generates slice addition information related to the slice splitting. The three-dimensional data encoding device may also split the location information and attribute information independently.

[0233] If tile division is performed (Yes in S4914), the three-dimensional data encoding device generates multiple division position information and multiple division attribute information by independently dividing multiple slice position information and multiple slice attribute information (or position information and attribute information) (S4915). The three-dimensional data encoding device also generates position tile addition information and attribute tile addition information related to tile division. The three-dimensional data encoding device may divide the slice position information and slice attribute information together.

[0234] Next, the three-dimensional data encoding device generates multiple encoded location information and multiple encoded attribute information by encoding each of the multiple division location information and multiple division attribute information (S4916). The three-dimensional data encoding device also generates dependency information.

[0235] Next, the three-dimensional data encoding device generates encoded data (encoded stream) by NAL unitizing (multiplexing) multiple encoded position information, multiple encoded attribute information, and additional information (S4917). The three-dimensional data encoding device then transmits the generated encoded data.

[0236] Figure 30 is a flowchart of the point cloud data decoding process according to this embodiment. First, the three-dimensional data decoding device determines the division method by analyzing the additional information related to the division method (slice additional information, position tile additional information, and attribute tile additional information) contained in the encoded data (encoded stream) (S4921). This division method includes whether or not to perform slice division and whether or not to perform tile division. The division method may also include the number of divisions and the type of division when slice division or tile division is performed.

[0237] Next, the three-dimensional data decoding device generates partitioning location information and partitioning attribute information by decoding multiple encoded position information and multiple encoded attribute information contained in the encoded data using dependency information contained in the encoded data (S4922).

[0238] If the additional information indicates that tile division has been performed (Yes in S4923), the three-dimensional data decoding device generates multiple slice position information and multiple slice attribute information by combining multiple division position information and multiple division attribute information in their respective methods, based on the position tile additional information and attribute tile additional information (S4924). The three-dimensional data decoding device may combine the multiple division position information and multiple division attribute information in the same method.

[0239] If the additional information indicates that slice division has been performed (Yes in S4925), the three-dimensional data decoding device generates position information and attribute information by combining multiple slice position information and multiple slice attribute information (multiple division position information and multiple division attribute information) in the same way based on the slice additional information (S4926). The three-dimensional data decoding device may combine the multiple slice position information and multiple slice attribute information in different ways.

[0240] Furthermore, attribute information for tiles or slices (identifier, area information, address information, and location information, etc.) may be stored not only in SEI but also in other control information. For example, attribute information may be stored in control information that shows the overall structure of the PCC data, or it may be stored in control information for each tile or slice.

[0241] Furthermore, when a three-dimensional data encoding device (three-dimensional data transmission device) transmits PCC data to another device, it may convert control information such as SEI into control information specific to the protocol of that system and present it accordingly.

[0242] For example, when a three-dimensional data encoding device converts PCC data containing attribute information to ISOBMFF (ISO Base Media File Format), it may store the SEI together with the PCC data in an "mdat box," or it may store it in a "track box" that contains control information related to the stream. In other words, the three-dimensional data encoding device may store the control information in a table for random access. Furthermore, when the three-dimensional data encoding device packets and transmits PCC data, it may store the SEI in the packet header. By making attribute information available at the system layer in this way, access to attribute information and tile data or slice data becomes easier, and the access speed can be improved.

[0243] In the configuration of the three-dimensional data decoding device, the memory management unit may determine in advance whether the information necessary for the decoding process is in memory, and if the information necessary for the decoding process is not available, it may obtain the information from storage or the network.

[0244] When a three-dimensional data decoding device acquires PCC data from storage or a network using Pull in a protocol such as MPEG-DASH, the memory management unit may identify the attribute information of the data necessary for decoding based on information from the localization unit or the like, request tiles or slices containing the identified attribute information, and acquire the necessary data (PCC stream). The identification of tiles or slices containing attribute information may be performed on the storage or network side, or by the memory management unit. For example, the memory management unit may acquire the SEI of all PCC data in advance and identify tiles or slices based on that information.

[0245] If all PCC data is transmitted from storage or the network using Push via the UDP protocol, the memory management unit may, based on information from the localization unit or the like, identify the data attribute information and tiles or slices necessary for decryption processing, and obtain the desired data by filtering the desired tiles or slices from the transmitted PCC data.

[0246] Furthermore, the three-dimensional data encoding device may determine, when acquiring data, whether the desired data exists, whether real-time processing is possible based on the data size, etc., or the communication status, etc. If the three-dimensional data encoding device determines, based on this determination result, that data acquisition is difficult, it may select and acquire a different slice or tile with a different priority or data volume.

[0247] Alternatively, the three-dimensional data decoding device may transmit information from the localization unit or other sources to a cloud server, which may then determine the necessary information based on that information.

[0248] (Embodiment 4) Next, we will explain the tile appending information. The three-dimensional data encoding device generates tile appending information, which is metadata about the tile division method, and transmits the generated tile appending information to the three-dimensional data decoding device.

[0249] Figure 31 shows an example of the syntax for tile metadata (TileMetaData). As shown in Figure 31, for example, tile metadata includes division method information (type_of_divide), shape information (topview_shape), overlap flag (tile_overlap_flag), overlap information (type_of_overlap), height information (tile_height), number of tiles (tile_number), and tile position information (global_position, relative_position).

[0250] The division method information (type_of_divide) indicates how the tiles are divided. For example, the division method information indicates whether the tiles are divided based on map information, i.e., based on a top view (top_view), or something else (other).

[0251] Shape information (topview_shape) is included in the tile information when, for example, the tile division method is based on a top view. Shape information indicates the shape of the tile when viewed from above. For example, this shape includes squares and circles. This shape may also include polygons other than ellipses, rectangles, or quadrilaterals, or other shapes. Furthermore, shape information is not limited to the shape of the tile when viewed from above, but may also indicate the three-dimensional shape of the tile (for example, cubes and cylinders).

[0252] The tile_overlap_flag indicates whether tiles overlap or not. For example, the tile overlap flag is included in the tile information when the tile division method is based on a top view. In this case, the tile overlap flag indicates whether tiles overlap in a top view. The tile overlap flag may also indicate whether tiles overlap in three-dimensional space.

[0253] The overlap information (type_of_overlap) is included in the tile information when tiles overlap, for example. The overlap information indicates how the tiles overlap, such as the size of the overlapping area.

[0254] The height information (tile_height) indicates the height of the tile. The height information may also include information indicating the shape of the tile. For example, if the tile's shape when viewed from above is rectangular, this information may indicate the lengths of the sides (vertical and horizontal lengths) of that rectangle. Alternatively, if the tile's shape when viewed from above is circular, this information may indicate the diameter or radius of that circle.

[0255] Furthermore, the height information may indicate the height of each tile, or it may indicate a common height for multiple tiles. Alternatively, multiple height types for roads and overpasses may be predefined, and the height information may indicate the height of each height type and the height type of each tile. Or, the height of each height type may be predefined, and the height information may indicate the height type of each tile. In other words, the height of each height type does not necessarily have to be indicated by the height information.

[0256] The tile_number indicates the number of tiles. Note that tile information may also include information indicating the spacing between tiles.

[0257] Tile position information (global_position, relative_position) is information used to identify the location of each tile. For example, tile position information indicates the absolute or relative coordinates of each tile.

[0258] Furthermore, some or all of the above information may be provided for each tile, or for multiple tiles (for example, for each frame or for multiple frames).

[0259] The three-dimensional data encoding device may include the tile addition information in the SEI (Supplemental Enhancement Information) and send it. Alternatively, the three-dimensional data encoding device may store the tile addition information in an existing parameter set (PPS, GPS, or APS, etc.) and send it.

[0260] For example, if the tile information changes from frame to frame, the tile information may be stored in a parameter set for each frame (such as GPS or APS). If the tile information does not change within a sequence, the tile information may be stored in a parameter set for each sequence (location SPS or attribute SPS). Furthermore, if the same tile division information is used for both location information and attribute information, the tile information may be stored in the parameter set of the PCC stream (stream PS).

[0261] Furthermore, tile information may be stored in any of the parameter sets described above, or in multiple parameter sets. Additionally, tile information may be stored in the header of the encoded data. Furthermore, tile information may be stored in the header of the NAL unit.

[0262] Furthermore, all or part of the tile addition information may be stored in one of the headers of the division location information and the division attribute information, but not in the other. For example, if the same tile addition information is used for both location information and attribute information, the tile addition information may be included in one of the headers of the location information or attribute information. For example, if attribute information depends on location information, the location information is processed first. Therefore, the header of the location information may contain this tile addition information, while the header of the attribute information may not. In this case, the three-dimensional data decoding device will determine, for example, that the attribute information of the dependency belongs to the same tile as the tile of the location information to which it depends.

[0263] The 3D data decoding device reconstructs the tiled point cloud data based on the tile information. If there is duplicate point cloud data, the 3D data decoding device identifies the multiple duplicate point cloud data, selects one, or merges the multiple point cloud data.

[0264] Furthermore, the three-dimensional data decoding device may perform decoding using tile-added information. For example, if multiple tiles overlap, the three-dimensional data decoding device may decode each tile, perform processing using the decoded data (e.g., smoothing or filtering), and generate point cloud data. This may enable highly accurate decoding.

[0265] Figure 32 shows an example of a system configuration including a three-dimensional data encoding device and a three-dimensional data decoding device. The tile division unit 5051 divides point cloud data, including position information and attribute information, into a first tile and a second tile. The tile division unit 5051 also sends tile addition information related to tile division to the decoding unit 5053 and the tile joining unit 5054.

[0266] The encoding unit 5052 generates encoded data by encoding the first tile and the second tile.

[0267] The decoding unit 5053 reconstructs the first and second tiles by decoding the encoded data generated by the encoding unit 5052. The tile joining unit 5054 reconstructs the point cloud data (position information and attribute information) by joining the first and second tiles using the tile addition information.

[0268] Next, we will explain slice appending information. The three-dimensional data encoding device generates slice appending information, which is metadata about the slice division method, and transmits the generated slice appending information to the three-dimensional data decoding device.

[0269] Figure 33 shows an example of the syntax for slice metadata (SliceMetaData). As shown in Figure 33, for example, slice metadata includes division method information (type_of_divide), overlap flag (slice_overlap_flag), overlap information (type_of_overlap), number of slices (slice_number), slice position information (global_position, relative_position), and slice size information (slice_bounding_box_size).

[0270] The division method information (type_of_divide) indicates how the slice is divided. For example, the division method information indicates whether the slice is divided based on object information as shown in Figure 50 (object). Note that the slice supplement information may also include information indicating how the object is divided. For example, this information indicates whether one object is divided into multiple slices or assigned to one slice. This information may also indicate the number of divisions if one object is divided into multiple slices.

[0271] The overlap flag (slice_overlap_flag) indicates whether or not the slices overlap. The overlap information (type_of_overlap) is included in the slice append information, for example, if the slices overlap. The overlap information indicates how the slices overlap, for example, the size of the overlapping area.

[0272] The slice_number indicates the number of slices.

[0273] Slice position information (global_position, relative_position) and slice size information (slice_bounding_box_size) are information about the region of the slice. Slice position information is information used to identify the position of each slice. For example, slice position information indicates the absolute or relative coordinates of each slice. Slice size information (slice_bounding_box_size) indicates the size of each slice. For example, slice size information indicates the size of the bounding box of each slice.

[0274] The three-dimensional data encoding device may include slice addition information in the SEI and send it out. Alternatively, the three-dimensional data encoding device may store the slice addition information in an existing parameter set (PPS, GPS, or APS, etc.) and send it out.

[0275] For example, if slice addition information changes from frame to frame, the slice addition information may be stored in a parameter set for each frame (such as GPS or APS). If the slice addition information does not change within a sequence, the slice addition information may be stored in a parameter set for each sequence (location SPS or attribute SPS). Furthermore, if the same slice division information is used for both location information and attribute information, the slice addition information may be stored in the parameter set of the PCC stream (stream PS).

[0276] Furthermore, slice addition information may be stored in any of the parameter sets mentioned above, or in multiple parameter sets. Also, slice addition information may be stored in the header of the encoded data. Additionally, slice addition information may be stored in the header of the NAL unit.

[0277] Furthermore, all or part of the slice addition information may be stored in one of the headers of the division location information and the division attribute information, but not in the other. For example, if the same slice addition information is used for both location information and attribute information, the slice addition information may be included in one of the headers of the location information or attribute information. For example, if attribute information depends on location information, the location information is processed first. Therefore, the header of the location information may contain this slice addition information, while the header of the attribute information may not. In this case, the three-dimensional data decoding device will determine, for example, that the attribute information that depends on the location information belongs to the same slice as the slice of the location information it depends on.

[0278] The 3D data decoding device reconstructs the sliced ​​point cloud data based on the slice addition information. If there is duplicate point cloud data, the 3D data decoding device identifies the multiple duplicate point cloud data, selects one, or merges the multiple point cloud data.

[0279] Furthermore, the three-dimensional data decoding device may perform decoding using slice-added information. For example, if multiple slices overlap, the three-dimensional data decoding device may decode each slice, perform processing (e.g., smoothing or filtering) using the decoded data, and generate point cloud data. This may enable highly accurate decoding.

[0280] Figure 34 is a flowchart of the three-dimensional data encoding process, including the generation of tile-added information, using the three-dimensional data encoding device according to this embodiment.

[0281] First, the three-dimensional data encoding device determines the method for dividing the tiles (S5031). Specifically, the three-dimensional data encoding device determines whether to use a top-view-based division method or another method. The three-dimensional data encoding device also determines the shape of the tiles when using the top-view-based division method. Furthermore, the three-dimensional data encoding device determines whether or not a tile overlaps with other tiles.

[0282] If the tile division method determined in step S5031 is a division method based on a top view (Yes in S5032), the three-dimensional data encoding device indicates in the tile addition information that the tile division method is a division method based on a top view (top_view) (S5033).

[0283] On the other hand, if the tile division method determined in step S5031 is other than the division method based on the top view (No in S5032), the three-dimensional data encoding device indicates in the tile addition information that the tile division method is other than the division method based on the top view (top_view) (S5034).

[0284] Furthermore, if the shape of the tile viewed from above, as determined in step S5031, is a square (square in S5035), the three-dimensional data encoding device records that the shape of the tile viewed from above is a square in the tile supplement information (S5036). On the other hand, if the shape of the tile viewed from above, as determined in step S5031, is a circle (circle in S5035), the three-dimensional data encoding device records that the shape of the tile viewed from above is a circle in the tile supplement information (S5037).

[0285] Next, the three-dimensional data encoding device determines whether a tile overlaps with another tile (S5038). If a tile overlaps with another tile (Yes in S5038), the three-dimensional data encoding device records that the tile overlaps in the tile information (S5039). On the other hand, if a tile does not overlap with another tile (No in S5038), the three-dimensional data encoding device records that the tile does not overlap in the tile information (S5040).

[0286] Next, the three-dimensional data encoding device divides the tiles based on the tile division method determined in step S5031, encodes each tile, and sends out the generated encoded data and tile additional information (S5041).

[0287] Figure 35 is a flowchart of the three-dimensional data decoding process using tile-added information by the three-dimensional data decoding device according to this embodiment.

[0288] First, the three-dimensional data decoding device analyzes the tile addition information contained in the bitstream (S5051).

[0289] If the tile information indicates that a tile does not overlap with other tiles (No in S5052), the 3D data decoding device generates point cloud data for each tile by decoding each tile (S5053). Next, the 3D data decoding device reconstructs point cloud data from the point cloud data of each tile based on the tile division method and tile shape indicated in the tile information (S5054).

[0290] On the other hand, if the tile addition information indicates that a tile overlaps with other tiles (Yes in S5052), the 3D data decoding device generates point cloud data for each tile by decoding each tile. The 3D data decoding device also identifies the overlapping portion of the tiles based on the tile addition information (S5055). The 3D data decoding device may use multiple pieces of overlapping information to perform the decoding process for the overlapping portion. Next, the 3D data decoding device reconstructs point cloud data from the point cloud data of each tile based on the tile division method, tile shape, and overlapping information indicated in the tile addition information (S5056).

[0291] The following describes variations related to slicing. The three-dimensional data encoding device may transmit information indicating the type of object (road, building, tree, etc.) or attributes (dynamic information, static information, etc.) as additional information. Alternatively, encoding parameters may be predetermined according to the object, and the three-dimensional data encoding device may notify the three-dimensional data decoding device of the encoding parameters by sending the type of object or attributes.

[0292] The following methods may be used for the encoding order and transmission order of slice data. For example, the 3D data encoding device may encode slice data in order from data that is easy to recognize or cluster. Alternatively, the 3D data encoding device may encode slice data in order from slice data that has been clustered first. The 3D data encoding device may also transmit the encoded slice data in order. Alternatively, the 3D data encoding device may transmit slice data in order of the decoding priority in the application. For example, if the decoding priority of dynamic information is high, the 3D data encoding device may transmit slice data in order from slices grouped by dynamic information.

[0293] Furthermore, if the order of encoded data differs from the order of decoding priority, the three-dimensional data encoding device may rearrange the encoded data before sending it out. Also, when storing encoded data, the three-dimensional data encoding device may rearrange the encoded data before storing it.

[0294] The application (3D data decoding device) requests the server (3D data encoding device) to send slices containing the desired data. The server sends the slice data required by the application, and does not need to send unnecessary slice data.

[0295] The application requests the server to send tiles containing the desired data. The server sends the tile data that the application needs, and does not need to send any tile data that is not needed.

[0296] (Embodiment 5) This section explains the encoding of location information (geometry). In encoding location information, the 3D data encoding device divides the region containing point cloud data using an octave tree and converts it into a set of point occupancy information for each node. Occupancy information is 8 bits of information indicating whether or not a point exists in each child node, with 0 or 1 indicating whether or not a point is contained in each child node.

[0297] There are several ways to divide an octave tree: using breadth-first search, which divides the nodes in ascending order of depth (depth value); and using depth-first search, which explores the points down to the lowest depth and then returns to the previous depth to search again. In encoding, the occupied information is encoded in the order described above.

[0298] Figure 36 shows the tree structure when a point cloud with a depth of 6 is divided using an octree. Figure 37 shows an example of the data structure of encoded data in an octree structure using breadth-first search. The encoded data of the point cloud includes a header and a payload. The payload contains information for each depth (depth #0 to #6) in order. Figure 38 shows an example of the payload syntax. The payload contains occupancy codes for each depth.

[0299] Next, let's explain the hierarchical structure. For example, as shown in Figure 36, we define multiple levels (also called hierarchical levels). Level 2 is a point cloud represented using point cloud data that has been divided into octave trees from depth=0 to the final depth (depth=6), Level 1 is a point cloud represented using point cloud data that has been divided into octave trees from depth=0 to depth=5, and Level 0 is a point cloud represented using point cloud data that has been divided into octave trees from depth=0 to depth=4. In other words, the resolution of the point cloud increases in the order of Level 0, Level 1, and Level 2. Conversely, the resolution of the point cloud decreases in the order of Level 2, Level 1, and Level 0. This can also be described as quantization being halved as the level decreases. In this way, by using levels, it is possible to represent the entire point cloud using all the data down to the lowest layer, or to represent low-resolution point cloud data using data from depth=0 to some of the higher layers. Depending on the resolution of the data being handled, or the amount of data, various combinations of levels can be set adaptively as needed.

[0300] Figure 39 is a flowchart of the decoding process for decoding encoded data containing location information at the desired resolution. First, the three-dimensional data decoding device determines the level (resolution) to be decoded and the depth corresponding to that level (S8801).

[0301] Next, the 3D data decoding device decodes the encoded data at the determined depth. Specifically, the 3D data decoding device decodes the first depth (depth 0 (depth=0)) (S8802). If the decoding of all depths to be decoded is not complete (No in S8803), the 3D data decoding device decodes the next depth (S8804). The 3D data decoding device may also decode the depth to be decoded using data from the previous level (or depth). If the decoding of all depths to be decoded is complete (Yes in S8803), the 3D data decoding device displays the obtained point cloud (S8805).

[0302] Furthermore, the three-dimensional data decoding device decodes the encoded data at a determined depth, and does not need to decode the encoded data at the remaining depths. Figure 40 shows the relationship between levels and the data to be decoded. Figure 41 is a schematic diagram showing the levels. For example, in Figure 40, decoding of depths #0 to #4 is necessary to decode the point cloud at level 0. Therefore, the three-dimensional data decoding device decodes up to encoded data A.

[0303] Furthermore, decoding of the Level 1 point cloud requires decoding of depth #0 to depth #5. Therefore, the 3D data decoding device decodes up to the encoded data of B. Also, decoding of the Level 2 (all) point cloud requires decoding of depth #0 to depth #6. Therefore, the 3D data decoding device decodes up to the encoded data of C.

[0304] Through the above process, the 3D data decoding device can decode low-resolution data. Therefore, the 3D data decoding device can reduce the amount of data or skip the decoding process when high-resolution point clouds are not required, thereby reducing the processing load.

[0305] Alternatively, a three-dimensional data decoding device can decode low-resolution data and then display the remaining data without waiting for the remaining data to be decoded, and then display the remaining data after decoding the high-resolution data. This reduces the initial delay between decoding and display.

[0306] Here, the three-dimensional data decoding device needs to determine that it has acquired data up to point A or B in order to decode the data up to a certain point, that is, it needs to determine the boundary between depths in the encoded data, or the depth information. Figure 42 shows an example of the header syntax. Figure 43 shows an example of the payload syntax.

[0307] For example, a three-dimensional data decoder may use the number of points (numPoint), the number of depths (depth) of the point cloud shown in the header, and the encoded data (occupancy_code) for each node or leaf stored for each depth shown in the payload, decode the encoded data sequentially from the beginning, and analyze the information of the decoded occupancy code to determine the boundary between depths or depth information.

[0308] This embodiment describes a data structure that facilitates partial decoding of such depth data, as well as data splitting and merging. Here, a new data structure that takes a hierarchical structure into consideration is defined. By using this data structure, data splitting and merging at the hierarchical data level becomes possible. The three-dimensional data encoding device or three-dimensional data decoding device can reduce the amount of data required for transmission by extracting specific necessary hierarchical data. Furthermore, the functionality of the three-dimensional data encoding device or three-dimensional data decoding device is improved by enabling data splitting or merging without decoding the encoded data.

[0309] Figure 44 shows the structure of encoded data containing all positional information from depth 0 to depth 6 (depth#0 to depth#6). This encoded data is also called the overall encoded data or bitstream (encoded bitstream). In this example, the overall encoded data does not contain information that explicitly indicates the boundaries between data at different depths. A three-dimensional data decoder can obtain the boundaries between depths, or depth information, by analyzing the occupancy code. Here, the number of points included in the header (numPoint) indicates the total number of points included in the overall encoded data. Also, the depth number, depth, which indicates the number of depths, is "7" in this example.

[0310] Figure 45 shows the structure of the overall encoded data. The overall encoded data shown in Figure 45 includes the structure shown in Figure 44, plus hierarchical structure metadata, which is metadata indicating the hierarchical structure. Figure 46 shows an example of the syntax of depth information (depth_info). Figure 47 shows an example of how depth information is stored in hierarchical structure metadata (layer_metadata).

[0311] Depth information includes the depth number and length information indicating the length of data for each depth. The length information, for example, indicates the difference in bytes or bits between the starting and ending positions of the encoded data (also referred to as depth data) for the corresponding depth.

[0312] Hierarchical metadata containing length information may be sent before or after the encoded data. The length information may also be stored in the header of the overall encoded data. Figure 48 shows an example of the header syntax in this case. The header includes the number of points (numPoint) and depth information (depth_info).

[0313] Thus, for example, data boundaries between depths are explicitly indicated in the hierarchical metadata or header. Note that the 3D data decoding device does not necessarily have to use the hierarchical metadata for decoding. In this case, the 3D data decoding device uses the hierarchical metadata when splitting or reconstructing the hierarchical data.

[0314] In the configurations shown in Figures 44 and 45, all depth data is continuous, and the context information used for entropy coding is also continuous and not initialized.

[0315] By using this structure, the three-dimensional data decoding device can easily divide the entire encoded data into data for each depth, thereby reducing the processing load. Furthermore, the transmission of the divided data can be reduced, thus reducing the amount of data transmitted.

[0316] Next, we will explain the case where the concept of layers is introduced. Figure 49 shows an example of the structure of the overall encoded data in this case. In this figure, depths 0 to 4 (depth#0 to depth#4) are defined as layer 0, depth 5 (depth#5) is defined as layer 1, and depth 6 (depth#6) is defined as layer 2. In other words, compared to the levels (layer levels) described above, levels are defined by multiple depths from depth 0 to a desired depth, whereas multiple layers are defined by depths that do not overlap with each other. For example, layer 0 includes the same depth as level 0, layer 1 corresponds to the difference in depth between level 1 and level 0, and layer 2 corresponds to the difference between level 2 and level 1.

[0317] In this case, hierarchical information (layer_info) is added to the overall encoded data. Figure 50 shows an example of the syntax of hierarchical information. Hierarchical information includes the number of layers (layer) and the number of layer depths (num_depth) indicating the number of depths contained in each layer. Figure 51 shows an example of the syntax of hierarchical structure metadata. For example, hierarchical structure metadata includes hierarchical information (layer_info). Figure 52 shows an example of the syntax of the header of the overall encoded data. For example, as shown in Figure 52, the header includes depth information (depth_info). Note that both hierarchical information and depth information may be included in either the header or the hierarchical structure metadata. Note that the syntax structure shown here is just an example and is not limited to it. The information included in the overall encoded data should be information that allows the three-dimensional data decoder to obtain the number of depths, the number of layers, the number of layer depths, and length information. For example, the overall encoded data may include information indicating the length of each layer.

[0318] Furthermore, if the layer structure is common across multiple processing units (e.g., frames), some or all of this information may be included in higher-level metadata, such as a sequence-level parameter set (e.g., SPS).

[0319] Furthermore, the overall encoded data may include a flag indicating whether or not it contains layer information (layer_info) and depth information (depth_info). If the flag is set to ON (e.g., value 1), the layer information and depth information may be included in the overall encoded data. Note that separate flags may be set for the layer information and depth information.

[0320] By using this structure, it becomes easy to split the main data into data for each depth, or into data for each hierarchy, thereby reducing the amount of processing required. Furthermore, the ability to transmit the split data reduces the amount of data that can be transmitted.

[0321] Next, we will explain the overall structure of the point cloud data. In addition to geometry, point cloud data may contain one or more attribute pieces of information, such as color or reflectance. Attribute information may also have a hierarchical structure, similar to geometry.

[0322] If attribute information has a hierarchical structure similar to location information, the 3D data encoding device stores the hierarchical structure information in hierarchical metadata or a data header, similar to location information. For example, the 3D data encoding device stores the hierarchical structure information in both the attribute information header and the location information header. When storing the hierarchical structure information in the hierarchical structure metadata, the 3D data encoding device may store the hierarchical structure metadata in separate parameter sets for location information such as GPS or APS and attribute information, or it may store it in a common parameter set such as SPS. Alternatively, the 3D data encoding device may store the hierarchical structure metadata in SEI or other metadata.

[0323] Figure 53 shows an example of a bitstream configuration where location information and attribute information each have a hierarchical structure, and the hierarchical structure information is stored in hierarchical metadata. By using this structure, it is easy to divide the main data into data for each layer, thereby reducing the amount of processing required. Furthermore, the transmission of divided data becomes possible, further reducing the amount of data transmitted. In addition, it becomes possible to similarly divide location information and attribute information into layers.

[0324] Figure 54 shows an example of the syntax for hierarchical metadata. This figure illustrates an example of applying hierarchical metadata common to both location information and attribute information.

[0325] Hierarchical metadata includes hierarchical information (layer_info), the number of components (component), and depth information for each component (depth_info). The number of components indicates the number of components such as location information and attribute information. For example, if point cloud data has color and reflectance in addition to location information, the number of components is 3. Note that the number of attribute information components may be indicated assuming that location information is always present. Also, if the number of attribute information components is indicated in the SPS, this information may be omitted. This reduces the amount of data.

[0326] The hierarchical information (layer_info) indicates the number of layers and the depth of each layer. For example, the hierarchical information is common to all components.

[0327] Depth information (depth_info) indicates the number of depths and the data length (length information) of each depth data. Depth information is set for each component, for example. Note that some or all of the depth information may be common to all components.

[0328] Furthermore, if the hierarchical structure is to be independent for each component, hierarchical information may be generated for each component.

[0329] While this explanation describes a method of hierarchical structure based on depth, hierarchical structure may also be performed based on temporal or spatial information. When hierarchical structure is used, the hierarchical structure metadata should be indicated using the method described above. Furthermore, configurations without a hierarchical structure may be generated. Information indicating whether or not the encoded data has a hierarchical structure may be included in the header or metadata. This allows for a mix of data with and without a hierarchical structure. For example, location information may have a hierarchical structure, while attribute information does not. Information indicating this may also be shown in the header, etc.

[0330] Next, we will explain the data reference relationships and dependencies between location information and attribute information. Figures 55, 56, and 57 show the reference relationships between location information and attribute information.

[0331] When encoded data has a hierarchical structure, Layer 0 is the base layer and can be decoded independently. On the other hand, Layer 1 cannot be decoded independently and is decoded in conjunction with the data in Layer 0. Similarly, Layer 2 cannot be decoded independently and is decoded in conjunction with the data in Layers 0 and 1. Furthermore, when using an octree-based encoding scheme, as shown in Figure 55, Layer 0 of attribute information is decoded by referencing Layer 0 of location information. For example, as shown in Figure 56, Layer 1 of attribute information is decoded by referencing Layer 0 of attribute information and Layers 0 and 1 of location information. When there is a reference relationship or dependency in decoding, the referenced data is sent first. By transmitting the referenced data first, it becomes possible for the 3D data decoding device to decode the data in the order it was acquired, enabling efficient decoding such as reducing the capacity of the receive buffer.

[0332] The following describes how to store encoded data in a file format such as ISOBMFF. Figure 58 shows an example of a bitstream configuration. Figure 59 shows an example of a three-dimensional data encoding device configuration. The three-dimensional data encoding device includes an encoding unit 8801 and a file conversion unit 8802. The encoding unit 8801 generates a bitstream containing encoded data and control information by encoding point cloud data. The file conversion unit 8802 converts the bitstream into a file format.

[0333] Figure 60 shows an example of the configuration of a three-dimensional data decoding device. The three-dimensional data decoding device includes a file inverse conversion unit 8811 and a decoding unit 8812. The file inverse conversion unit 8811 converts the file format into a bitstream containing encoded data and control information. The decoding unit 8812 generates point cloud data by decoding the bitstream.

[0334] Figure 61 shows the basic structure of ISOBMFF. Figure 62 is a protocol stack diagram when the NAL unit common to the PCC codec is stored in ISOBMFF. Here, the NAL unit of the PCC codec is stored in ISOBMFF.

[0335] NAL units include data NAL units and metadata NAL units. Data NAL units include geometry slice data and attribute slice data. Metadata NAL units include SPS, GPS, APS, and SEI.

[0336] ISOBMFF (ISO based media file format) is a file format standard defined in ISO / IEC 14496-12. It specifies a format that can store various media such as video, audio, and text in multiplexed formats, and is a media-independent standard.

[0337] The basic unit in ISOBMFF is the box. A box consists of type, length, and data, and a file is a collection of boxes of various types. A file mainly consists of boxes such as ftyp, which indicates the file's brand using 4CC, moov, which stores metadata such as control information, and mdat, which stores data.

[0338] The method for storing each type of media in ISOBMFF is specified separately; for example, the method for storing AVC video and HEVC video is specified in ISO / IEC 14496-15. Furthermore, it is conceivable to extend the functionality of ISOBMFF to store and transmit PCC encoded data.

[0339] When storing NAL units for metadata in ISOBMFF, the SEI may be stored in the "mdat box" along with the PCC data, or in the "track box" which contains control information about the stream. Furthermore, when transmitting data in packets, the SEI may be stored in the packet header. By indicating the SEI at the system layer, access to attribute information, tile, and slice data becomes easier, thus improving access speed.

[0340] Next, we will describe the first example of format conversion of PCC hierarchical data. The encoding scheme is a technique for compressing data. On the other hand, additional functions are provided in the system format and have a different role than the encoding scheme. Such additional functions are defined by standards different from the encoding scheme. To create the optimal format for providing these additional functions, the three-dimensional data encoding device converts the data. In doing so, the three-dimensional data encoding device pre-stores information in the encoded data that facilitates conversion. This reduces the amount of processing involved in the conversion.

[0341] The following describes how to convert data for each hierarchical level and hierarchical structure metadata containing hierarchical information into a file format. Slice data such as location information and attribute information are stored in the sample file format. The sample is stored in mdat. In addition, for access to the sample data, metadata such as a random access table stores information indicating the data structure, offset information indicating the data location, and data length information. Note that this information may be stored in a table different from the random access table.

[0342] The following describes the case where slice data is stored in a sample. Figure 63 is a diagram showing the conversion process from bitstream to file format. The three-dimensional data encoding device stores the position information slice and attribute information slice in a one-to-one correspondence with the sample. Here, each slice contains information from all layers (hierarchical data).

[0343] Location information samples belong to the Geometry Track, and attribute information samples belong to the Attribute Track. Hierarchical information is stored in the metadata for each frame belonging to the Meta Data Track. In addition, metadata samples may store information indicating that the location information sample and the attribute information sample belong to the same frame, information indicating whether the attribute information sample refers to the location information sample in the case of an octave-based encoding scheme, and timestamp information common to both the location information sample and the attribute information sample.

[0344] A frame unit that operates with a common timestamp may be called an access unit. Layer information may be stored in the moov. Here, layer information includes, for example, the layer information described above. Layer information may also include at least some of the other information contained in the layer structure metadata or header described above, such as depth information.

[0345] This method allows slices to be stored directly as samples, making processing easier.

[0346] Figure 64 is a flowchart of the format conversion process. First, the three-dimensional data encoding device starts the format conversion of the encoded data (S8811). Next, the three-dimensional data encoding device stores one slice containing multiple layers as one sample (S8812). The three-dimensional data encoding device also stores hierarchical information as metadata (S8813). The three-dimensional data encoding device forms a frame (AU: access unit) (S8814).

[0347] Next, we will explain the partial decoding method using the file format. Figure 65 is a flowchart of this decoding process. First, the three-dimensional data decoding device extracts the desired sample by random access (S8821). Specifically, the three-dimensional data decoding device uses metadata contained in the moov file and a random access table to identify the location of the desired sample and extracts the data of that sample.

[0348] The three-dimensional data decoder analyzes the hierarchical information metadata (S8822) and extracts layer boundary information within the sample (S8823). Specifically, the three-dimensional data decoder obtains layer boundary information from the hierarchical information metadata, specifically the number of depths contained in each layer within the sample and the data length of each depth. For example, the three-dimensional data decoder calculates the data length of each layer from the number of depths contained in each layer and the data length of each depth, and determines the layer data boundary based on the calculated data length of each layer.

[0349] Next, the three-dimensional data decoder divides the layers using layer boundary information and decodes the desired data (S8824). For example, the three-dimensional data decoder extracts a specific layer component from the sample.

[0350] In this way, by including hierarchical information in the metadata, a three-dimensional data decoding device can use the hierarchical information to extract specific information without decoding the encoded data.

[0351] The following describes a second example of PCC hierarchical data format conversion. When a three-dimensional data encoding device stores hierarchical data as a sample, it may store one hierarchical data as one sample or as one subsample.

[0352] Figure 66 shows the conversion process from bitstream to file format. The three-dimensional data encoding device stores the data for each layer of the position information slice and attribute information slice, with a one-to-one correspondence to the samples.

[0353] Location data samples belong to the Geometry Track, and attribute data samples belong to the Attribute Track. Hierarchical information is stored in the metadata for each frame, which belongs to the Meta Data Track. Furthermore, there is a track for each layer, and each track has its own set of samples. Having a track for each layer makes it easier to handle data on a layer-by-layer basis.

[0354] Furthermore, since the slice data in the encoded data bitstream contains all the hierarchical data, the three-dimensional data encoding device stores the data in samples while dividing it. If hierarchical information is shown in the bitstream, the three-dimensional data encoding device divides the slice data using information such as the data length of each hierarchical data. If hierarchical information is not shown in the bitstream, the three-dimensional data encoding device calculates the hierarchical information while decoding the encoded data. The three-dimensional data encoding device re-encodes and divides the data based on the obtained hierarchical information.

[0355] This process allows for the storage of layer-specific information in tracks and samples for each layer. Therefore, data can be extracted layer by layer in the 3D data decoding device, making it easier to handle data for each layer.

[0356] Figure 67 shows an example of the syntax for hierarchical metadata. Hierarchical metadata includes hierarchical information (layer_info), the number of components (component), and depth information for each component (depth_info).

[0357] Furthermore, when the 3D data encoding device divides a slice into hierarchical data, if the sample includes header information, it may copy and add the slice header to all divided data. Figure 68 schematically illustrates this division process. Alternatively, the 3D data encoding device may store the slice header in metadata instead of including it in the sample. Copying the header information reduces the processing required to create the header.

[0358] Furthermore, the 3D data encoding device may add an identifier to the file format indicating whether the data to be stored in the sample is hierarchical. Also, if the data is hierarchical, the 3D data encoding device may add an identifier to the file format indicating whether the data contains all the hierarchical data, or whether the data to be stored in the sample is hierarchical data. The 3D data encoding device may also indicate this information using a media type or a box type such as 4CC. This facilitates media identification.

[0359] Figure 69 is a flowchart of the conversion process using hierarchical information. First, the three-dimensional data encoding device starts format conversion of the encoded data (S8831). Next, the three-dimensional data encoding device divides the slice into layer-specific information using hierarchical information metadata (S8832). Next, the three-dimensional data encoding device stores each of the divided hierarchical data into one sample (S8833). Next, the three-dimensional data encoding device stores the hierarchical information in metadata (S8834). Next, the three-dimensional data encoding device constructs a frame (AU) (S8835).

[0360] Figure 70 is a flowchart of the conversion process that does not use hierarchical information. First, the three-dimensional data encoding device starts format conversion of the encoded data (S8841). Next, the three-dimensional data encoding device decodes the data and determines the boundaries of the hierarchical data (S8842). Next, the three-dimensional data encoding device re-encodes and divides the data (S8843). Next, the three-dimensional data encoding device stores each of the divided hierarchical data into one sample (S8844). Next, the three-dimensional data encoding device stores the hierarchical information in metadata (S8845). Next, the three-dimensional data encoding device constructs a frame (AU) (S8846).

[0361] Figure 71 is a flowchart of the decoding process for hierarchical data sample data. First, the three-dimensional data decoding device extracts the desired sample by random access (S8851). Next, the three-dimensional data decoding device decodes the data contained in the extracted sample (S8852).

[0362] Next, we will describe another example of a hierarchical data structure. Figures 72 and 73 show examples of the structure of fully encoded data (PCC data). The hierarchical structure in the examples shown in Figures 72 and 73 is the same as in Figure 49.

[0363] Figure 72 shows a case where one depth data is used as one slice data, and a slice header is assigned to each depth data. The slice header includes a depthId that identifies the hierarchy of the depth data, a layerId that indicates the hierarchy to which the depth belongs, and a length that indicates the length of the depth data. The slice header may also include a groupId that indicates that the data belongs to the same frame. In other words, the groupId indicates the frame (time) to which the data belongs.

[0364] If this information is included in the slice header, the overall encoded data does not need to have hierarchical metadata. Furthermore, the 3D data encoding device may store parameters common to all depths in the header of the slice transmitting the initial depth, or it may store them in a common header and place them before the data for depth#0. The 3D data encoding device may also store depthId and groupId in the slice header, and the number of depths, as well as the layerId and length for each depth, in the hierarchical metadata or common header.

[0365] Furthermore, depth#0 can be decoded independently, while depths other than depth#0 cannot be decoded independently and depend on other data. The 3D data decoding device determines that data other than depth#0 cannot be decoded independently and decodes the depth data to be decoded together with depth data that has the same groupId as the depth data to be decoded and has a smaller depthId than the depth data to be decoded.

[0366] Figure 73 shows a case where one hierarchical data is used as one slice data, and a slice header is assigned to each hierarchical data. The slice header includes the layerId, the number of depths included in the hierarchical (num_depth), and the length of the depth data (length). The slice header may also include a groupId indicating that the hierarchical data belongs to the same frame. Note that the slice header includes the layerId and groupId, and the number of hierarchicals, the number of depths included in each hierarchical, and the depth length information (length) may be included in the hierarchical structure metadata.

[0367] By using this structure, it becomes easy to divide the main data into data for each hierarchical level, thereby reducing the processing load during division. Furthermore, the ability to transmit the divided data reduces the amount of data transmitted. Additionally, location information and attribute information can be similarly divided hierarchically.

[0368] Next, we will describe a third example of PCC hierarchical data format conversion. When a three-dimensional data encoding device stores hierarchical data in samples, it may store one depth of data as one sample, or it may store one depth of data as one subsample.

[0369] Figure 74 shows the conversion process from bitstream to file format. The three-dimensional data encoding device stores the data for each layer of the position information slice and attribute information slice, with a one-to-one correspondence to the samples.

[0370] Location data samples belong to the Geometry Track, and attribute data samples belong to the Attribute Track. Hierarchical information is stored in the metadata for each frame, which belongs to the Meta Data Track. There is a track for each layer, and there are samples belonging to each track. Having a track for each layer makes it easy to handle data on a layer-by-layer basis.

[0371] Furthermore, since slice data is configured for each hierarchical data level in the encoded data bitstream, the 3D data encoding device can directly store the data as a sample. Therefore, the processing load can be reduced compared to cases where slice data is not configured for each hierarchical data level. Additionally, the hierarchical information is stored in metadata.

[0372] Figure 75 is a flowchart of the format conversion process. First, the three-dimensional data encoding device starts format conversion of the encoded data (S8861). Next, the three-dimensional data encoding device stores slice data for each hierarchical level in one sample (S8862). Next, the three-dimensional data encoding device stores hierarchical information in metadata (S8863). Next, the three-dimensional data encoding device constructs a frame (AU) (S8864).

[0373] Figure 76 is a flowchart of the decoding process. First, the three-dimensional data decoding device analyzes metadata to access specific hierarchical data and obtains the number of depths belonging to each layer (S8871). Next, the three-dimensional data decoding device uses the obtained information to calculate the starting position of the depth data at the beginning of the hierarchical data and the overall size of the layer (S8872). Next, the three-dimensional data decoding device decodes the hierarchical data (S8873).

[0374] Figure 77 shows an example of the syntax for depth information. Figure 78 shows an example of the syntax for the sample size box (sample_size_box:stsz). The three-dimensional data encoding device may store the size of each hierarchical data (entry_size) in the sample size box, which stores the size information for each sample.

[0375] Figure 79 shows an example of the syntax for hierarchical information (layer_info). Figure 80 shows an example of the syntax for PCCLayerStructureBox. For example, as shown in Figures 79 and 80, the three-dimensional data encoding device stores the number of layers (layer) and the number of depths included in the layers (num_depth) in a PCCLayerStructureBox. The three-dimensional data encoding device may store this information in the same box or in separate boxes.

[0376] Next, the process of extracting partial data from a file format will be described. The three-dimensional data decoding device randomly accesses partially decoded data from the file using the data structure and hierarchical structure metadata described in this embodiment and extracts the data. The three-dimensional data decoding device can access the data and extract the desired data based on the position information and attribute information contained in the metadata, such as frame, hierarchy, the data length of each, and the number of depths included in the hierarchy.

[0377] Figure 81 schematically illustrates this extraction operation. The transmission unit 8821 has a complete data file (file format) with Layer 0 and Layer 1, and the reception unit 8823 has a Layer 0 data file. In this state, if the reception unit 8823 wants to acquire Layer 1 data, it requests the transmission unit 8821 to send the Layer 1 file. The extraction unit 8822 included in the transmission unit 8821 extracts the Layer 1 file from the complete data file (file format) and provides the Layer 1 file (bitstream) to the reception unit 8823. The reception unit 8823 integrates the Layer 0 file and the Layer 1 file to generate a complete data file.

[0378] Figure 82 shows an example of a complete data file (file format). Figures 83, 84, and 85 show examples of bitstreams extracted by the extraction unit 8822. For example, as shown in Figure 83, the extraction unit 8822 may extract all data from the file format. Alternatively, as shown in Figure 84, the extraction unit 8822 may extract location information but not attribute information. Alternatively, as shown in Figure 85, the extraction unit 8822 may extract layer 0 but not layer 1. Alternatively, the extraction unit 8822 may transmit the data in a rearranged state, for example, although not shown.

[0379] By using the data structure and hierarchical metadata described in this embodiment, hierarchical data can be easily divided, enabling the acquisition of necessary data while avoiding unnecessary data. This reduces transmission bandwidth and transmission delay, improving the functionality of data transmission.

[0380] Next, we will explain the partial decoding process for direct mode data. Direct mode is a technique used in octree encoding where, for a given node, octree encoding is stopped, and the coordinates of the points in the leaf node are directly encoded. For example, direct mode is used when the points belonging to a node are sparse. By using direct mode, the amount of data can be reduced.

[0381] Figure 86 shows an example of direct mode. In the area enclosed by the dotted line in Figure 86, there are two leaf points (sparse). For example, if it is determined that the points are sparse at node A at depth=1, the coordinates of these two points are directly recorded in the data area at depth1. The coordinates of the two points are the coordinates from node A, and these coordinates have a resolution of depth=4. In other words, the data at depth=1 includes the occupancy code at depth=1 and the coordinate data (area B) in direct mode at depth=4, which is enclosed by the dotted line.

[0382] Next, we will explain how to decode such data. When decoding depth0 to depth4, the three-dimensional data decoding device uses a normal decoding method. On the other hand, when the three-dimensional data decoding device partially extracts depth0 to depth1 and decodes it, if it is necessary for all the decoded data to have the same resolution, the coordinates of depth4 included in depth1 may not be used when decoding. Also, if it is acceptable for high resolutions to be mixed, the three-dimensional data decoding device may use the coordinates of depth4 included in depth1 when decoding.

[0383] Next, we will explain data partitioning when direct mode is used. When the transmitting device extracts and transmits data from depth0 to depth1, it may or may not include direct mode (depth4) information in depth1.

[0384] Furthermore, the transmitting device may decide whether or not to include direct mode information depending on whether the receiving device needs that information. If the receiving device needs the information, it may include it in the bitstream; if it does not need it, it may omit it. This reduces the amount of data. For example, if the transmitting device sometimes decodes data with a resolution of depth 2 to depth 4 along with data with a resolution of depth 0 to depth 1, it may decide that direct mode information is necessary. If it does not decode data with a resolution of depth 2 to depth 4, it may decide that direct mode information is unnecessary.

[0385] As described above, the three-dimensional data encoding device according to this embodiment performs the processing shown in Figure 87. The three-dimensional data encoding device sets a hierarchical structure for the multiple positional information of multiple three-dimensional points included in the point cloud data, having multiple depths and multiple layers, each containing one or more depths (S8881). The three-dimensional data encoding device generates multiple first encoded data (e.g., depth data) for each depth by encoding the multiple positional information for each depth (S8882). The three-dimensional data encoding device generates a bitstream that includes multiple second encoded data, which are encoded data for each layer and include one or more first encoded data for one or more depths included in the corresponding layer (S8883). The bitstream includes first information indicating the data length of each of the multiple second encoded data.

[0386] According to this, a three-dimensional data decoding device that decodes a bitstream can easily access data at any level using the first information. Therefore, the three-dimensional data encoding device can reduce the processing load of the three-dimensional data decoding device.

[0387] For example, the first piece of information includes a second piece of information (e.g., num_depth) indicating the number of depths contained in each of the multiple layers, and a third piece of information (e.g., length) indicating the data length of each of the multiple first encoded data.

[0388] For example, a bitstream includes a first header common to multiple second encoded data (e.g., a hierarchical metadata or header as shown in Figure 49), and the first header contains first information.

[0389] For example, a bitstream includes multiple second headers for each second encoded data (e.g., slice headers shown in Figure 73), the first information includes multiple fourth pieces of information corresponding to one of the multiple second encoded data and indicating the data length of the corresponding second encoded data, and each of the multiple second headers includes fourth pieces of information indicating the data length of the second encoded data corresponding to that second header.

[0390] For example, a bitstream includes multiple third headers for each of the multiple first encoded data (e.g., slice headers shown in Figure 72), where the first information includes second information (e.g., num_depth) indicating the number of depths contained in each of the multiple layers, and fifth information (e.g., length) corresponding to each of the multiple first encoded data and indicating the data length of the corresponding first encoded data, and each of the multiple third headers includes fifth information indicating the data length of the first encoded data corresponding to that third header.

[0391] For example, the three-dimensional data encoding device further generates multiple third-encoded data for each depth by encoding multiple attribute information possessed by multiple three-dimensional points for each depth, the bitstream is encoded data for each hierarchy and includes multiple fourth-encoded data which include one or more third-encoded data for one or more depths included in the corresponding hierarchy, and the bitstream includes sixth information indicating the data length of each of the multiple fourth-encoded data.

[0392] For example, a three-dimensional data encoding device comprises a processor and memory, and the processor uses the memory to perform the above processing.

[0393] Furthermore, the three-dimensional data decoding device according to this embodiment performs the processing shown in Figure 88. The three-dimensional data decoding device obtains first information from a bitstream containing a plurality of second encoded data and first information indicating the data length of each of the plurality of second encoded data (S8886). The three-dimensional data decoding device uses the first information to obtain at least one of the plurality of second encoded data (S8887). The three-dimensional data decoding device decodes the obtained at least one second encoded data (S8888). The bitstream contains a plurality of positional information of a plurality of three-dimensional points included in point cloud data, and includes a plurality of first encoded data for each depth, which is generated by encoding a plurality of positional information for each depth, each of which has a hierarchical structure having a plurality of depths and a plurality of layers, each of which has a depth of 1 or more. Each of the plurality of second encoded data corresponds to one of the plurality of layers, and includes one or more first encoded data included in the layer corresponding to the second encoded data among the plurality of first encoded data.

[0394] According to this, the three-dimensional data decoding device can easily access data at any level using the first information. Therefore, the processing load of the three-dimensional data decoding device can be reduced.

[0395] For example, the first piece of information includes a second piece of information (e.g., num_depth) indicating the number of depths contained in each of the multiple layers, and a third piece of information (e.g., length) indicating the data length of each of the multiple first encoded data. For example, a three-dimensional data decoder uses the second and third pieces of information to calculate the data length of the second encoded data.

[0396] For example, a bitstream includes a first header common to multiple second encoded data (e.g., a hierarchical metadata or header as shown in Figure 49), and the first header contains first information.

[0397] For example, a bitstream includes multiple second headers for each second encoded data (e.g., slice headers shown in Figure 73), the first information includes multiple fourth pieces of information corresponding to one of the multiple second encoded data and indicating the data length of the corresponding second encoded data, and each of the multiple second headers includes fourth pieces of information indicating the data length of the second encoded data corresponding to that second header.

[0398] For example, a bitstream includes multiple third headers for each of the multiple first encoded data (e.g., slice headers shown in Figure 72), where the first information includes second information (e.g., num_depth) indicating the number of depths contained in each of the multiple layers, and fifth information (e.g., length) corresponding to each of the multiple first encoded data and indicating the data length of the corresponding first encoded data, and each of the multiple third headers includes fifth information indicating the data length of the first encoded data corresponding to that third header.

[0399] For example, the bitstream includes multiple third-encoded data for each depth, generated by encoding multiple attribute information of multiple three-dimensional points at each depth. The bitstream is encoded data for each hierarchy and includes multiple fourth-encoded data, which include one or more third-encoded data for one or more depths included in the corresponding hierarchy. The bitstream includes sixth-information indicating the data length of each of the multiple fourth-encoded data. The three-dimensional data decoding device further obtains the sixth-information from the bitstream, uses the sixth-information to obtain at least one of the multiple fourth-encoded data, and decodes the obtained at least one fourth-encoded data.

[0400] For example, a three-dimensional data decoding device comprises a processor and memory, and the processor uses the memory to perform the above processing.

[0401] (Embodiment 6) Embodiment 6 will now be described.

[0402] Conformance refers to a predetermined standard (e.g., a standard defined in a standard) that a three-dimensional point cloud encoded by a three-dimensional data encoding device, i.e., a bitstream, or a three-dimensional data decoding device that decodes the bitstream, must satisfy. Conformance is also expressed as a conformance point, conformance point, or conformance level. The three-dimensional data encoding device selects one conformance from a predetermined set of conformances and encodes the three-dimensional point cloud using a predetermined method based on the selected conformance. The three-dimensional data decoding device decides whether or not to decode the bitstream based on whether or not the bitstream conforms to the predetermined conformance and whether or not the three-dimensional data decoding device supports decoding bitstreams that conform to the predetermined conformance.

[0403] The three-dimensional data encoding device encodes the three-dimensional point cloud using any method (predetermined processing) such that the bitstream can satisfy a predetermined conformance. The three-dimensional data encoding device may perform any of the following methods on the three-dimensional point cloud: scaling, quantization, division into slices or tiles, offsetting of the divided space, table referencing (codebook), etc. The precision level of the three-dimensional point cloud may be adjusted to increase, maintain, or decrease the bit precision. The three-dimensional data encoding device generates a bitstream containing information indicating the conformance that the bitstream generated by encoding the three-dimensional point cloud will satisfy.

[0404] The 3D data decoder obtains conformance information from the bitstream (e.g., the syntax of the encoded data) and, based on this conformance information, determines whether the bitstream of the encoded 3D point cloud satisfies a predetermined conformance. If the decoder determines that the conformance is satisfied, it decodes the point cloud data.

[0405] A conformance combination (set) may include any of the following parameters: for example, the precision of the 3D point cloud encoding (i.e., the number of bits in the encoded data), the number of 3D points per partitioned data unit (slice or tile), the number of available processing cores in the 3D data decoder, the processor speed of the 3D data decoder, the application requirements of the 3D data decoder (e.g., real-time, low-power mode, remote server processing, etc.), lossless or lossy encoding, and information about the bounding box of the slice (size, etc.).

[0406] In this embodiment, bit precision refers to the number of bits. Bit precision refers to the precision in hardware processing.

[0407] Figure 89 is a block diagram showing an example of the configuration of a three-dimensional data encoding device according to Embodiment 6. Figure 90 is a flowchart showing a first example of a three-dimensional data encoding method according to Embodiment 6.

[0408] The three-dimensional data encoding device 9600 comprises a determination unit 9601, a conversion unit 9602, and an encoding unit 9603.

[0409] The 3D data encoding device 9600 receives point cloud data from a 3D point cloud. The 3D data encoding device 9600 then acquires the point cloud data.

[0410] The determination unit 9601 of the three-dimensional data encoding device 9600 determines the conformance of the encoded data, which is the data obtained after encoding the point cloud data of the three-dimensional point cloud (S9601). Here, the conformance is determined according to the performance of the corresponding equipment (three-dimensional data decoding device), the use case of the application, the type of three-dimensional point cloud being handled, etc. The conformance may be predetermined or determined adaptively. The three-dimensional data encoding device 9600 determines the conformance by selecting one conformance from a combination of conformances.

[0411] The determination unit 9601 determines whether the three-dimensional point cloud of the point cloud data satisfies the determined conformance (S9602).

[0412] If the determination unit 9601 determines that the three-dimensional point cloud data does not satisfy the determined conformance (No in S9602), the transformation unit 9602 performs predetermined processing to satisfy the determined conformance (S9603). The transformation unit 9602 may perform, for example, quantization or data partitioning as predetermined processing.

[0413] After step S9603, or if the determination unit 9601 determines that the three-dimensional point cloud of the point cloud data satisfies the determined conformance (Yes in S9602), the encoding unit 9603 generates metadata including a conformance index indicating the determined conformance (S9604). The conformance index is identification information for identifying one conformance among a combination of multiple conformances.

[0414] The encoding unit 9603 encodes the converted point cloud data in step S9603, or the unconverted point cloud data if Yes is determined in step S9602, and generates a bitstream containing the encoded point cloud data and metadata (S9605).

[0415] Figure 91 is a block diagram showing an example of the configuration of a three-dimensional data decoding device according to Embodiment 6. Figure 92 is a flowchart showing an example of a three-dimensional data decoding method according to Embodiment 6.

[0416] The three-dimensional data decoding device 9610 comprises a determination unit 9611 and a decoding unit 9612.

[0417] The three-dimensional data decoding device 9610 acquires a bitstream. The bitstream includes encoded point cloud data (encoded data) and metadata including a conformance index.

[0418] The three-dimensional data decoding device 9610 is a decoding device that conforms to at least one of a predetermined set of conformances, and can decode data if the bitstream conforms to a conformance supported by the three-dimensional data decoding device 9610.

[0419] The determination unit 9611 obtains a conformance index from the metadata (S9611).

[0420] The determination unit 9611 determines whether the conformance indicated by the conformance index is included in the conformance to which the three-dimensional data decoding device 9610 conforms (S9612). The conformance to which the three-dimensional data decoding device 9610 conforms is the decoding condition of the three-dimensional data decoding device 9610. The determination unit 9611 also determines whether the bitstream satisfies the conformance indicated by the conformance index (S9612). In other words, the determination unit 9611 determines whether the bitstream satisfies the decoding condition of the three-dimensional data decoding device 9610.

[0421] If the determination unit 9611 determines that the bitstream is included in the conformance to which the three-dimensional data decoding device 9610 conforms (Yes in S9612), the decoding unit 9612 decodes the encoded point cloud data included in the bitstream (S9613).

[0422] If the determination unit 9611 determines that the bitstream is not included in the conformance to which the three-dimensional data decoding device 9610 conforms (No in S9612), the decoding unit 9612 skips decoding the encoded point cloud data included in the bitstream and does not decode the point cloud data (S9614).

[0423] If the decoding unit 9612 determines "No" in step S9612, it may decode the point cloud data as is, or it may proceed to error-specific processing and perform predetermined processing. Here, the predetermined processing may, for example, determine the quality after decoding, and if there are no problems with the quality, display (output) the decoding result as is, or if the quality is poor, not display (output) the decoding result.

[0424] Next, we will explain the types of constraints that can be imposed by setting conformances.

[0425] Two objects can be considered as targets for conformance constraints (hereinafter referred to as "constraint targets"): the input point cloud and the segmented data.

[0426] The input point cloud is point cloud data representing the three-dimensional point cloud input to the three-dimensional coding device. The input point cloud is the original point cloud data before it is divided into slices or tiles. The input point cloud is equivalent to the point cloud data after integrating multiple divided data sets. If such an input point cloud is subject to constraints, the conformance includes the number of three-dimensional points in the input point cloud, bit precision (number of bits), etc. In this case, an input point cloud that conforms to the constraints may be generated by using a sensor that outputs detection results that conform to the constraints as an input point cloud. Alternatively, an input point cloud that conforms to the constraints may be generated by the three-dimensional data coding device performing predetermined processing on an existing input point cloud to make it conform to the constraints.

[0427] Instead of imposing constraints on the input point cloud as described above, the partitioned data may be the subject of the constraints. The partitioned data is the data obtained after dividing the input point cloud into slices or tiles. When the partitioned data is the subject of the constraints, the conformance includes the number of three-dimensional points or bit precision (number of bits) per partitioned data unit. In other words, the three-dimensional data encoding device may perform a process to partition the input point cloud such that the partitioned data after partitioning satisfies the conformance requirements.

[0428] Furthermore, conformance may be set for both the input point cloud and the partitioned data. In this case, if the input point cloud does not satisfy the conformance requirement, the 3D data encoding device will divide the input point cloud into multiple partitioned data so that the conformance requirement for each partitioned data unit is satisfied. On the other hand, if the input point cloud does satisfy the conformance requirement, the 3D data encoding device does not need to divide the input point cloud into multiple partitioned data. In other words, if the input point cloud satisfies the conformance requirement, the 3D data encoding device may encode the input point cloud as is without dividing it.

[0429] When constraints are placed on the input point cloud, the 3D data encoding device can use the conformance of the input point cloud (e.g., the number of 3D points in the input point cloud, the number of bits, etc.) as a specification when pre-generating an input point cloud that satisfies the conformance. On the other hand, when constraints are placed on the partitioned data, the 3D data encoding device can use this as a specification when partitioning an input point cloud that does not satisfy the conformance.

[0430] For example, the number of three-dimensional points or the distribution range of the three-dimensional point cloud used in map data varies depending on the size and density of the map, and there is no upper limit. For such large-scale three-dimensional point clouds, a method of imposing constraints on the divided data can be used. In other words, by imposing constraints on the divided data units, it becomes possible to decode large-scale point cloud data with no upper limit on the number of three-dimensional points or bits, even with a three-dimensional data decoding device that has limited resources (memory, processing power).

[0431] Furthermore, by imposing constraints on both the input point cloud and the segmented data, this encoding method can be applied to a variety of content.

[0432] Figure 93 is a flowchart showing a second example of a three-dimensional data encoding method according to Embodiment 6. This three-dimensional data encoding method switches processing depending on whether the constraint target is an input point cloud or segmented data.

[0433] The three-dimensional data encoding device determines whether the constraint target is an input point cloud or segmented data (S9621). Whether the constraint target is an input point cloud or segmented data may be predetermined by the user, or it may be determined according to the application of the three-dimensional point cloud.

[0434] When the 3D data encoding device determines that the constraint target is the input point cloud (input point cloud in S9621), it applies constraints to the input point cloud, either on the bit precision (number of bits) or the number of points (S9622). Specifically, the 3D data encoding device determines the bit precision or number of points as conformance for the input point cloud. The number of bits is the maximum number of bits in the encoded data after encoding the point cloud data of the input point cloud. The number of points is the range of the number of 3D points included in the input point cloud.

[0435] The three-dimensional data encoding device determines whether the input point cloud is within the specified range (S9623). In other words, the three-dimensional data encoding device determines whether the input point cloud satisfies the conformance determined in step S9622.

[0436] If the three-dimensional data encoding device determines that the input point cloud is not within the specified range (No in S9623), it processes the input point cloud to conform to the specified range (S9624). In other words, if the input point cloud does not satisfy the conformance requirements, the three-dimensional data encoding device performs predetermined processing on the input point cloud to satisfy the conformance requirements.

[0437] If the three-dimensional data encoding device determines that the input point cloud is within the specified range (Yes in S9623), it encodes the input point cloud that has been determined to be within the specified range (S9630). In addition, the three-dimensional data encoding device encodes the processed input point cloud that satisfies conformance after the predetermined processing in step S9624 (S9630).

[0438] Returning to step S9621, if the three-dimensional data encoding device determines that the constraint target is segmented data (segmented data in S9621), it targets the segmented data and constrains the bit precision (number of bits) or number of points for each segmented data (S9625). Specifically, the three-dimensional data encoding device determines the bit precision or number of points as conformance for each segmented data unit.

[0439] The three-dimensional data encoding device determines whether the input point cloud falls within the specified range (S9626). Step S9626 is the same process as step S9623.

[0440] If the three-dimensional data encoding device determines that the input point cloud is within the specified range (Yes in S9626), it encodes the input point cloud that was determined to be within the specified range (S9630).

[0441] Step S9626 does not necessarily have to be performed.

[0442] If the three-dimensional data encoding device determines that the input point cloud is not within the specified range (No in S9626), it divides the input point cloud into multiple divided data (S9627). The three-dimensional data encoding device may, for example, divide the three-dimensional space in which the input point cloud exists into multiple subspaces, and then determine the point cloud data representing the three-dimensional point cloud contained in each subspace as the divided data. The subspaces may be slices or tiles.

[0443] The three-dimensional data encoding device determines whether the divided data is within the specified range (S9628). In other words, the three-dimensional data encoding device determines whether the divided data satisfies the conformance determined in step S9625.

[0444] If the three-dimensional data encoding device determines that the divided data is not within the specified range (No in S9628), it processes the divided data to conform to the specified range (S9629). In other words, if the divided data does not satisfy the conformance requirements, the three-dimensional data encoding device performs predetermined processing on the divided data to satisfy the conformance requirements.

[0445] If the three-dimensional data encoding device determines that the divided data is within the specified range (Yes in S9628), it encodes the divided data that has been determined to be within the specified range (S9630). In addition, the three-dimensional data encoding device encodes the processed divided data that satisfies conformance after the predetermined processing in step S9629 (S9630).

[0446] Furthermore, the determination of conformance, or the predetermined processing to conform to the constraints, may be performed for each divided data. In other words, there may be a mix of divided data in which the predetermined processing to conform to the constraints is performed and divided data in which it is not performed. That is, the determination in step S9628 is performed for each of the multiple divided data, and the predetermined processing to conform to the constraints (S9629) is performed for divided data that does not satisfy the requirements, while the predetermined processing to conform to the constraints (S9629) does not need to be performed for divided data that does satisfy the requirements.

[0447] The three-dimensional data encoding device may, during encoding, add metadata to the bitstream that includes constraint information indicating whether the constraint target is set in the input point cloud or in the segmented data.

[0448] Figure 94 shows an example of a bounding box.

[0449] One way to constrain the bit precision of the divided data is to define the size of the bounding box for the divided data. For example, the size of this bounding box may be defined so that the height, width, and depth are all within a predetermined number of bits. Furthermore, an upper limit (maximum value) for the number of bits in each of the height, width, and depth may also be defined.

[0450] Figure 95 is a flowchart showing another example of the three-dimensional data decoding method according to Embodiment 6.

[0451] The three-dimensional data decoding device analyzes the metadata contained in the bitstream and obtains the conformance (constraints) contained in the metadata (S9631). This conformance is the conformance determined when the input point cloud or segmented data is encoded by the three-dimensional data encoding device.

[0452] The three-dimensional data decoding device determines whether the constraint target is an input point cloud or segmented data (S9632).

[0453] If the three-dimensional data decoder determines that the constraint is an input point cloud (input point cloud in S9632), it checks whether the bitstream for each input point cloud satisfies the decoding conditions of the three-dimensional data decoder (S9633). If the three-dimensional data decoder finds that the bitstream for each input point cloud satisfies the decoding conditions, it decodes the encoded input point cloud contained in the bitstream. If the bitstream for each input point cloud does not satisfy the decoding conditions, it does not need to decode the encoded input point cloud.

[0454] If the three-dimensional data decoding device determines that the constraint target is segmented data (segmented data in S9632), it checks whether the bitstream of each segmented data unit satisfies the decoding conditions (S9634). If the three-dimensional data decoding device confirms that the bitstream of each segmented data unit satisfies the decoding conditions, it decodes the encoded segmented data contained in the bitstream. If the bitstream of each segmented data unit does not satisfy the decoding conditions, it does not need to decode the encoded segmented data.

[0455] Next, we will explain the prescribed processing when specifying a constraint on the number of bits per slice.

[0456] Figure 96 is a flowchart showing a third example of the three-dimensional data encoding method according to Embodiment 6. This flowchart is an example where a bit reduction process is performed as a predetermined step. In the following explanation, the case where the constraint target is an input point cloud will be used as an example, but the same can be applied to segmented data. In other words, the input point cloud may be replaced with segmented data.

[0457] The three-dimensional data encoding device determines the conformance of the number of bits in the positional information of the input point cloud (S9641). In other words, the three-dimensional data encoding device determines the upper limit (first maximum number of bits) of the number of bits in the encoded data after encoding the input point cloud as a conformance. The first maximum number of bits relates to the number of bits in the data after encoding the positional information. The positional information included in the input point cloud indicates the position of each three-dimensional point in the input point cloud. The positional information is, for example, the coordinates of each three-dimensional point. The coordinates may be expressed in a Cartesian coordinate system or a polar coordinate system.

[0458] The three-dimensional data encoding device determines whether the number of bits of the position information in the input point cloud satisfies the conformance determined in step S9641 (S9642). In other words, the three-dimensional data encoding device determines whether the number of bits of the position information in the input point cloud satisfies the first maximum number of bits.

[0459] If the three-dimensional data encoding device determines that the number of bits of the position information in the input point cloud does not satisfy the determined conformance (No in S9642), that is, if it determines that it exceeds the determined conformance, it executes a predetermined process to reduce the number of bits of the position information in the input point cloud (S9643).

[0460] The three-dimensional data encoding device encodes the position information of the input point cloud (S9644) if it determines that the number of bits of the position information of the input point cloud satisfies the determined conformance (Yes in S9642). The three-dimensional data encoding device also encodes the position information of the processed input point cloud (S9644) which satisfies the conformance achieved by the predetermined processing in step S9643.

[0461] The three-dimensional data encoding device generates a bitstream by encoding the point cloud data to satisfy the determined conformance by executing steps S9642 to S9644. Specifically, the three-dimensional data encoding device generates a bitstream by encoding the position information of the input point cloud to satisfy the determined first maximum number of bits. The three-dimensional data encoding device adds conformance information indicating the determined conformance to the bitstream. The conformance information is, for example, first bit number information indicating the first maximum number of bits. The first bit number information may be the value of the first maximum number of bits itself, or it may be identification information for uniquely identifying the first maximum number of bits. The identification information may be, for example, a conformance index.

[0462] Next, the predetermined process in step S9643 of Figure 96 will be described.

[0463] Figure 97 shows an example of the process for reducing the number of bits according to Embodiment 6.

[0464] In a predetermined process, the number of bits may be reduced by quantizing the position coordinates (x,y,z). For example, as shown in Figure 97, if the maximum number of bits specified by conformance is 4 bits, and any of the coordinates (x,y,z) included in the position information is 6 bits, the number of bits in the position information does not meet the maximum number of bits specified by conformance. Therefore, as a predetermined process to be performed on the input point cloud to satisfy conformance, the number of bits in the position information is reduced by 2 bits by performing a 2-bit shift quantization. Here, the data subject to the 2-bit shift quantization may be the input bits, or it may be the bits substantially used by all three-dimensional points in the input point cloud.

[0465] Figure 98 shows another example of the bit reduction process according to Embodiment 6.

[0466] In a predetermined process, data partitioning, such as dividing the data into slices or tiles, may be used to reduce the bit precision per slice. Specifically, when a three-dimensional data encoding device partitions the input point cloud into slices, it may reduce the bit precision by shifting the coordinates of the origins of other slices to align with the origin of one slice.

[0467] When the maximum number of bits per slice is defined as conformance, if the number of bits per point cloud (the number of bits for the size, width, height, and depth of the bounding box including the slice) exceeds the conformance (i.e., the maximum number of bits per slice does not satisfy the conformance), the 3D data encoding device divides the point cloud so that the number of bits in the bounding box of the slice satisfies the conformance. The 3D data encoding device then reduces the number of bits and conforms to the conformance by shifting the origin of the bounding box of the divided slice (slice2) to match the origin of the bounding box of the divided slice (slice1) in Figure 98. For example, if there is a rule that the upper limit of the number of bits per slice is M bits, and the width of the bounding box constituting the input point cloud is N (>M) bits, the width of the bounding box may be divided into a number of int(N / M)+1, each with a size of M bits.

[0468] Although Figure 98 uses a diagram representing a two-dimensional space to explain a predetermined process, the predetermined process may also be applied to a three-dimensional space or to spaces of other dimensions.

[0469] Figure 99 is a flowchart showing a fourth example of a three-dimensional data encoding method according to Embodiment 6. This flowchart is an example where a process to increase the number of bits is performed as a predetermined process.

[0470] The three-dimensional data encoding device determines the conformance of the number of bits in the positional information of the input point cloud (S9651). In other words, the three-dimensional data encoding device determines the upper limit of the number of bits in the encoded data after encoding the input point cloud (first maximum number of bits) as the conformance.

[0471] The three-dimensional data encoding device determines whether the number of bits of the position information in the input point cloud satisfies the conformance determined in step S9651 (S9652). In other words, the three-dimensional data encoding device determines whether the number of bits of the position information in the input point cloud satisfies the first maximum number of bits.

[0472] If the three-dimensional data encoding device determines that the number of bits in the position information of the input point cloud does not satisfy the determined conformance (No in S9652), that is, if it determines that it is insufficient to meet the determined conformance, it executes a predetermined process to increase the number of bits in the position information of the input point cloud (S9653).

[0473] The three-dimensional data encoding device encodes the position information of the input point cloud (S9654) if it determines that the number of bits of the position information of the input point cloud satisfies the determined conformance (Yes in S9652). The three-dimensional data encoding device also encodes the position information of the processed input point cloud (S9654) which satisfies the conformance achieved by the predetermined processing performed in step S9653.

[0474] Next, the predetermined process in step S9653 of Figure 99 will be described.

[0475] Figure 100 shows an example of the process for increasing the number of bits according to Embodiment 6.

[0476] In the specified process, if the number of bits is insufficient, the number of bits may be increased by upsampling or other means. For example, in this method, the number of bits may be increased by padding the position coordinates (x, y, z).

[0477] Figure 101 shows another example of the process for increasing the number of bits according to Embodiment 6.

[0478] In a given process, the number of bits may be adjusted by bit shifting or shifting the origin of the point cloud. The number of bits per slice can be increased by inputting a shift value to a data partition such as a slice or tile. When combining multiple slices (multiple divided data), the number of bits may be increased by shifting the coordinates of different slices from different bitstreams. For example, the number of bits may be increased by shifting the origin of the bounding box of slice (slice2) to a position that does not overlap with the bounding box of the divided slice (slice1) in Figure 101. This allows for conformance adjustment.

[0479] Conformance may also be defined by a combination of the number of bits in the positional information and the number of three-dimensional points. This example is illustrated using Figure 102. Note that the number of three-dimensional points is sometimes referred to as the number of points in the point cloud.

[0480] Figure 102 is a flowchart showing a fifth example of the three-dimensional data encoding method according to Embodiment 6. In this flowchart, the conformance is determined by the number of bits of position information and the range of the number of three-dimensional points in the input point cloud. Note that the conformance is not limited to a combination of position accuracy and the range of the number of three-dimensional points, but may also be a combination of other parameters.

[0481] The three-dimensional data encoding device determines the conformance of the encoded data, which is the data obtained after encoding the point cloud data of the three-dimensional point cloud (S9661). The conformance determined here includes a first maximum number of bits that defines the number of bits of position information and a range of the number of three-dimensional points included in the input point cloud.

[0482] The three-dimensional data encoding device determines whether the input three-dimensional point cloud satisfies the determined number of bits for conformance (S9662). In other words, the three-dimensional data encoding device determines whether the number of bits of the positional information in the input three-dimensional point cloud satisfies (matches) the first maximum number of bits determined.

[0483] If the three-dimensional data encoding device determines that the input three-dimensional point cloud does not satisfy the conformance of the determined number of bits (No in S9662), it performs a predetermined process to satisfy the determined conformance (S9663). The predetermined process is, for example, one of the processes described using Figures 97, 98, 100, and 101.

[0484] If step S9663 is completed, or if the input three-dimensional point cloud satisfies the determined number of bits for conformance (Yes in S9662), the three-dimensional data encoding device executes the next step S9664.

[0485] The three-dimensional data encoding device determines whether the input three-dimensional point cloud satisfies conformance with respect to the determined range of three-dimensional points (S9664). In other words, the three-dimensional data encoding device determines whether the number of three-dimensional points in the input three-dimensional point cloud falls within the determined range of three-dimensional points.

[0486] If the three-dimensional data encoding device determines that the input three-dimensional point cloud does not satisfy the conformance regarding the determined range of three-dimensional points (No in S9664), it performs a predetermined process to satisfy the determined conformance (S9665). The predetermined process is, for example, one of the processes described using Figures 97, 98, 100, and 101.

[0487] If step S9665 is completed, or if the input three-dimensional point cloud is determined to satisfy the conformance of the determined range of three-dimensional points (Yes in S9664), the three-dimensional data encoding device executes the next step S9666.

[0488] The three-dimensional data encoding device generates metadata (S9666) that includes a conformance index indicating the conformance determined in step S9661. The conformance index is identification information for identifying one conformance from a combination of multiple conformances.

[0489] The three-dimensional data encoding device encodes the point cloud data after processing in step S9663, the point cloud data after processing in step S9665, the point cloud data after processing in steps S9663 and S9665, or the unprocessed point cloud data (i.e., point cloud data that has not undergone the predetermined processing), according to the determination results of steps S9662 and S9664, and generates a bitstream containing the encoded point cloud data and metadata (S9667).

[0490] The three-dimensional data encoding device generates a bitstream by encoding the point cloud data in a manner that satisfies both the determined first maximum number of bits and the range of the number of three-dimensional points, by executing steps S9662 to S9665. Here, the three-dimensional data encoding device adds conformance information indicating the determined conformance to the bitstream. The conformance information is, for example, first bit number information indicating the first maximum number of bits and range information indicating the range of the number of three-dimensional points. The range information may be the value of the range of the number of three-dimensional points itself, or it may be identification information for uniquely identifying the range of the number.

[0491] Figure 103 shows an example of a conformance combination according to Embodiment 6.

[0492] As shown in this diagram, conformance combinations may be represented by a combination of the number of bits in the location information and the range of the number of points in the point cloud. In this example, the number of bits in the location information is classified into two stages: an upper limit of 32 bits and an upper limit of 64 bits. The range of the number of points in the point cloud is classified into three stages: 10,000 or less, more than 10,000 and less than or equal to 100,000, and more than 100,000. As a result, the conformance combinations are classified into six conformance points, and each conformance is assigned a conformance index. In other words, by specifying a conformance index represented by a number from 1 to 6, the number of bits in the location information and the range of the number of points in the point cloud can be uniquely set.

[0493] The examples described in Figures 102 and 103 illustrate cases where both the number of bits for positional information in each divided data unit (e.g., slice unit) and the range of the number of points in each divided data unit are simultaneously constrained.

[0494] The three-dimensional data encoding device selects one of several conformance points included in a conformance combination and encodes the data to fit the selected conformance point. If the data does not fit the selected conformance point, the three-dimensional data encoding device performs quantization, data partitioning, or merger of partitioned data so that the number of bits in each partitioned data unit or the number of points in each partitioned data unit fits the conformance point.

[0495] Furthermore, when a three-dimensional data encoding device attempts to conform to one conformance—the number of bits in the positional information or the range of the number of points in the point cloud—it may fail to conform to the other conformance. For example, if a three-dimensional data encoding device divides the data to conform to the number of bits, the number of three-dimensional points in the point cloud is also divided, which may prevent it from fitting within the specified range of points.

[0496] In this case, the three-dimensional data encoding device may perform predetermined processing according to the specified priority. For example, compliance with the constraint on the number of bits of position information may be the highest priority (mandatory), while compliance with the constraint on the range of the number of points in the point cloud may be desirable as much as possible. In other words, the three-dimensional data encoding method may perform predetermined processing to satisfy the preferred specified conformance, but may not perform predetermined processing to satisfy the less preferred specified conformance.

[0497] In cases where processing speed is prioritized, such as in low-latency mode, the priority for adhering to the bit count constraint may be set higher than the constraint for the range of the number of points in the point cloud. From the perspective of coding efficiency, the priority for adhering to the constraint for the number of points in the point cloud may be set higher than the constraint for the bit count. Thus, the priority for adhering to the bit count and the range of the number of points in the point cloud may be set according to the purpose.

[0498] The conformance may be defined by the number of bits, the size of the bounding box of the divided data, the number of divided data, or a combination of these.

[0499] Conformance may also be defined by a combination of the number of bits in the location information and the frame rate. This example is illustrated using Figure 104.

[0500] Figure 104 is a flowchart showing a sixth example of the three-dimensional data encoding method according to Embodiment 6. In this flowchart, the number of bits for positional information and the frame rate of the input point cloud are determined as conformances.

[0501] The three-dimensional data encoding device determines the conformance of the encoded data, which is the data obtained after encoding the point cloud data of the three-dimensional point cloud (S9671). The conformance determined here includes a first maximum number of bits that defines the number of bits of position information and the frame rate of the input point cloud.

[0502] The three-dimensional data encoding device determines whether the input three-dimensional point cloud satisfies the determined number of bits for conformance (S9672). In other words, the three-dimensional data encoding device determines whether the number of bits for the positional information of the input three-dimensional point cloud satisfies (matches) the determined first maximum number of bits, and whether the frame rate of the input three-dimensional point cloud satisfies the determined frame rate.

[0503] If the three-dimensional data encoding device determines that the input three-dimensional point cloud does not satisfy the conformance of the determined number of bits (No in S9672), it performs a predetermined process to satisfy the determined conformance (S9673). The predetermined process is, for example, one of the processes described using Figures 97, 98, 100, and 101.

[0504] If step S9673 is completed, or if the input three-dimensional point cloud is determined to satisfy the conformance of the determined range of three-dimensional points (Yes in S9672), the three-dimensional data encoding device executes the next step S9674.

[0505] The three-dimensional data encoding device generates metadata including a conformance index indicating the conformance determined in step S9671 (S9674). The conformance index is identification information for identifying one conformance among a combination of multiple conformances.

[0506] The three-dimensional data encoding device encodes either the point cloud data processed in step S9673 or the unprocessed point cloud data (i.e., point cloud data that has not undergone the predetermined processing) according to the determination result in step S9672, and generates a bitstream containing the encoded point cloud data and metadata (S9675).

[0507] Thus, for example, conformance may be determined by hardware requirements that can be translated into application needs in terms of the number of frames per second (frame rate / fps) required by the hardware (three-dimensional data encoding / decoding device). In this case, one frame may be considered as a 360-degree capture of LiDAR. The combination of conformance may also be a combination of other parameters. Furthermore, conformance may be defined by the frame rate per segmented data (slice).

[0508] Figure 105 shows another example of conformance combinations according to Embodiment 6.

[0509] As shown in this diagram, conformance combinations may be represented by a combination of the number of bits in the location information and the frame rate. In this example, the number of bits in the location information is classified into three stages: 16 bits, 32 bits, and 64 bits. The frame rate is classified into three stages: less than 60 fps, less than 10 fps, and less than 1 fps. As a result, the conformance combinations are classified into nine conformance points, and each conformance is assigned a conformance index. In other words, by specifying a conformance index represented by a number from 1 to 9, the number of bits in the location information and the frame rate can be uniquely set.

[0510] Up to this point, we have described examples of setting conformance in the encoding and decoding of location information, but conformance may also be set in the encoding and decoding of attribute information in the same way as in the encoding and decoding of location information. The conformance applied to attribute information may not only specify attribute sub-information such as color or reflectance, but also specify information related to location information associated with the point cloud for the purpose of prediction or compression. Prediction of attribute information using hierarchical structures such as LoD or RAHT requires geometric location information of the three-dimensional point cloud in order to perform functions to search for subsamples or neighboring points within a predetermined distance. For this reason, the conformance used to define attribute information may include parameters based on this location information.

[0511] Furthermore, the conformance for defining attribute information may include an upper limit on the number of bits used to represent color. This conformance may be, for example, the number of bits in RGB information. The number of bits in RGB information indicates whether the point cloud color is represented using 8 bits, 12 bits, or 16 bits. A similar specification may be applied to reflectance.

[0512] Figure 106 shows another example of conformance combinations according to Embodiment 6.

[0513] As shown in this diagram, conformance combinations may be represented by a combination of the number of bits in the color and the attribute transformation parameters. In this example, the number of bits in the color is classified into three categories: 8 bits, 12 bits, and 16 bits. The attribute transformation parameters indicate, for example, the number of levels in the hierarchical structure of the LoD used during prediction. The attribute transformation parameters are classified into three categories: less than 10 levels, less than 5 levels, and 1 level. This classifies the conformance combinations into nine conformance points, and each conformance is assigned a conformance index. In other words, by specifying a conformance index represented by a number from 1 to 9, the number of bits in the color and the attribute transformation parameters can be uniquely set.

[0514] Conformance may be set for attribute information. That is, point cloud data may include attribute information for each three-dimensional point in the three-dimensional point cloud, in addition to positional information. The three-dimensional data encoding device determines a second maximum number of bits, which defines the number of bits after encoding the attribute information of the point cloud data of the three-dimensional point cloud, as the conformance of the encoded data. The three-dimensional data encoding device generates a bitstream by encoding the attribute information to satisfy the determined second maximum number of bits. The bitstream may include second bit information indicating the second maximum number of bits.

[0515] This explains the syntax of conformance indexes.

[0516] Figure 107 shows an example of an SPS (Sequence Parameter Set) according to Embodiment 6 (Example 1). Figure 108 shows an example of an SPS according to Embodiment 6 (Example 2). Figure 109 shows an example of a GPS (Geometry Parameter Set) according to Embodiment 6 (Example 3). Figure 110 shows the configuration of a bitstream according to Embodiment 6.

[0517] As shown in Figure 107, the conformance index may be included in SPS so as to be part of the available profile_idc, profile_compatibility_flags, or level_idc parameters.

[0518] Furthermore, as shown in Figure 108, depending on the description and usage scenario, the conformance index may be set as an independent parameter of the SPS to further expand the number of differential profiles, levels, and conformances for the G-PCC encoding and decoding process. That is, the conformance index may be set in header 9621 in Figure 110.

[0519] Furthermore, as shown in Figure 109, the conformance index may be set to be included in the header of each slice in the Geometry slice header (GPS). In other words, the conformance index may be set in headers 9622 and 9623 in Figure 110.

[0520] This means that each slice can have a different conformance index to accommodate different types of three-dimensional point clouds or data from different regions. It can also perform encoding or decoding operations that are compatible with different processor types, such as CPUs and GPU ASICs.

[0521] Figure 111 is a diagram illustrating an example of switching conformances according to the location of a three-dimensional point cloud, according to Embodiment 6.

[0522] Figure 111 shows an example where an indoor three-dimensional point cloud 9631 and an outdoor three-dimensional point cloud 9632 are obtained, and a portion of the three-dimensional region 9633 within the outdoor three-dimensional point cloud 9632 corresponds to the indoor three-dimensional point cloud 9631. Since the indoor three-dimensional point cloud 9631 and the outdoor three-dimensional point cloud 9632 are data acquired by different sensors, for example, their point cloud densities differ from each other.

[0523] The indoor 3D point cloud 9631 is a point cloud measured indoors, such as in an office, and therefore cannot be acquired by outdoor sensors. In other words, the outdoor 3D point cloud 9632 does not include the indoor 3D point cloud 9631.

[0524] Thus, the indoor three-dimensional point cloud 9631 and the outdoor three-dimensional point cloud 9632 are independent point cloud data generated with different accuracies and densities. For this reason, independent conformances may be set for each of the three-dimensional point clouds 9631 and 9632, and the conformance may be switched according to the three-dimensional point clouds 9631 and 9632.

[0525] The two sets of data can be combined by switching the precision of the outdoor 3D point cloud 9632 and scaling it up so that it accommodates the indoor 3D point cloud 9631. The indoor point cloud can also be shifted using a different slice origin value.

[0526] As described above, the three-dimensional data encoding device according to one aspect of this embodiment performs the processing shown in Figure 112. The three-dimensional data encoding device determines the first maximum number of bits in the encoded data after encoding at least one of the divided data units when the point cloud data representing the three-dimensional point cloud is divided into multiple parts, and the point cloud data units before division (S9681). The three-dimensional data encoding device generates a bitstream by encoding the multiple divided data obtained by dividing the point cloud data, or the point cloud data before division, to satisfy the determined first maximum number of bits (S9682). The bitstream includes first bit number information indicating the first maximum number of bits.

[0527] According to this, the three-dimensional data encoding method generates a bitstream containing first bit number information indicating the first maximum number of bits in the encoded data after encoding. Therefore, the three-dimensional data decoding device can determine whether it can properly decode the encoded data without analyzing the bitstream. This reduces the processing load on the three-dimensional data decoding device.

[0528] For example, the point cloud data includes position information for each three-dimensional point in the three-dimensional point cloud. The first maximum number of bits relates to the number of bits after encoding the position information. In the generation (S9682), the bitstream is generated by encoding the multiple divided data obtained by dividing the point cloud data, or the position information of the point cloud data before division, to satisfy the determined first maximum number of bits.

[0529] According to this, the three-dimensional data encoding method generates a bitstream that includes first bit information indicating the first maximum number of bits of the encoded position information. Therefore, the three-dimensional data decoding device can determine whether it can properly decode the position information without analyzing the bitstream.

[0530] For example, the three-dimensional data encoding device further determines the range of the number of three-dimensional points included in at least one of the divided data unit and the point cloud data unit. In the generation (S9682), the bitstream is generated by encoding the multiple divided data obtained by dividing the point cloud data, or the point cloud data before division, so as to satisfy the determined first maximum number of bits and the range of the number. The bitstream further includes range information indicating the range of the number.

[0531] According to this, the three-dimensional data encoding method generates a bitstream that includes range information indicating the range of the number of three-dimensional points in the encoded data. Therefore, the three-dimensional data decoding device can determine whether it can properly decode the position information without analyzing the bitstream. This reduces the processing load on the three-dimensional data decoding device.

[0532] For example, the point cloud data further includes attribute information for each three-dimensional point in the three-dimensional point cloud. The three-dimensional data encoding device further determines a second maximum number of bits after encoding the attribute information of at least one of the three-dimensional point clouds in the divided data unit and the point cloud data unit. In the generation (S9682), the bitstream is generated by (i) encoding the divided data from the point cloud data, or the position information of the point cloud data before division, to satisfy the first maximum number of bits determined, and (ii) encoding the attribute information of the divided data from the point cloud data, or the point cloud data before division, to satisfy the second maximum number of bits determined. The bitstream further includes second bit number information indicating the second maximum number of bits.

[0533] According to this, the three-dimensional data encoding method generates a bitstream that includes a second bit number information indicating the second maximum number of bits in the encoded attribute information. Therefore, the three-dimensional data decoding device can determine whether it can properly decode the attribute information without analyzing the bitstream. This reduces the processing load on the three-dimensional data decoding device.

[0534] For example, a three-dimensional data encoding device comprises a processor and memory, and the processor uses the memory to perform the above processing.

[0535] Furthermore, a three-dimensional data decoding device according to one aspect of this embodiment performs the processing shown in Figure 113. The three-dimensional data decoding device acquires a bitstream that includes encoded data, which is the encoded data of at least one of the divided data units when point cloud data representing a three-dimensional point cloud is divided into multiple parts, and the data of the point cloud data units before division, and first bit number information indicating the first maximum number of bits of the encoded data (S9691). The three-dimensional data decoding device determines whether the acquired bitstream satisfies the first maximum number of bits indicated by the first bit number information (S9692). If the three-dimensional data decoding device determines that the bitstream satisfies the first maximum number of bits, it decodes the encoded data (S9693).

[0536] According to this, the three-dimensional data decoding method obtains first bit number information, which indicates the first maximum number of bits in the encoded data after encoding, from the bitstream, and can appropriately decode the point cloud data based on the obtained first bit number information.

[0537] For example, in the decoding operation (S9693), the three-dimensional data decoding device will not decode the encoded data if it determines that the bitstream does not satisfy the first maximum number of bits.

[0538] According to this method, the decoding process for encoded data of bitstreams that cannot be properly decoded is not performed, thus reducing the processing load.

[0539] For example, the point cloud data includes position information for each three-dimensional point in the three-dimensional point cloud. The first maximum number of bits relates to the number of bits after encoding the position information.

[0540] According to this, the three-dimensional data decoding method obtains first bit number information indicating the first maximum number of bits of encoded position information from the bitstream, and can appropriately decode the point cloud data based on the obtained first bit number information.

[0541] For example, the bitstream further includes range information indicating the range of the number of three-dimensional points included in at least one of the divided data unit and the point cloud data unit. In the determination (S9692), the three-dimensional data decoder further determines whether the bitstream satisfies the range of the number indicated by the range information. In the decoding (S9693), if it is determined that the bitstream satisfies the first maximum number of bits and satisfies the range of the number, the encoded data is decoded; if it is determined that the bitstream does not satisfy the first maximum number of bits or does not satisfy the range of the number, the encoded data is not decoded.

[0542] According to this, the three-dimensional data decoding method obtains range information from the bitstream that indicates the range of the number of three-dimensional points in the encoded data, and can appropriately decode the point cloud data based on the obtained range information.

[0543] For example, the point cloud data further includes attribute information for each three-dimensional point in the three-dimensional point cloud. The bitstream further includes second bit number information indicating the second maximum number of bits after encoding of the attribute information of at least one of the three-dimensional point clouds in the divided data unit and the point cloud data unit. The determination (S9692) further determines whether the bitstream satisfies the second maximum number of bits indicated by the second bit number information. In the decoding (S9693), if it is determined that the bitstream satisfies both the first maximum number of bits and the second maximum number of bits, the encoded data is decoded; if it is determined that the bitstream does not satisfy either the first maximum number of bits or the second maximum number of bits, the encoded data is not decoded.

[0544] According to this, the three-dimensional data decoding method obtains second bit number information, which indicates the second maximum number of bits of the encoded attribute information, from the bitstream, and can appropriately decode the point cloud data based on the obtained second bit number information.

[0545] For example, a three-dimensional data decoding device comprises a processor and memory, and the processor uses the memory to perform the above processing.

[0546] (Embodiment 7) Next, the configuration of the three-dimensional data creation device 810 according to this embodiment will be described. Figure 114 is a block diagram showing an example configuration of the three-dimensional data creation device 810 according to this embodiment. This three-dimensional data creation device 810 is mounted on a vehicle, for example. The three-dimensional data creation device 810 transmits and receives three-dimensional data with an external traffic monitoring cloud, a preceding vehicle, or a following vehicle, and also creates and stores three-dimensional data.

[0547] The three-dimensional data creation device 810 includes a data receiving unit 811, a communication unit 812, a reception control unit 813, a format conversion unit 814, a plurality of sensors 815, a three-dimensional data creation unit 816, a three-dimensional data synthesis unit 817, a three-dimensional data storage unit 818, a communication unit 819, a transmission control unit 820, a format conversion unit 821, and a data transmission unit 822.

[0548] The data receiving unit 811 receives three-dimensional data 831 from a traffic monitoring cloud or a preceding vehicle. The three-dimensional data 831 includes information such as a point cloud, visible light images, depth information, sensor position information, or speed information, including areas that cannot be detected by the vehicle's sensors 815.

[0549] The communication unit 812 communicates with the traffic monitoring cloud or the preceding vehicle and sends data transmission requests and other messages to the traffic monitoring cloud or the preceding vehicle.

[0550] The receiving control unit 813 exchanges information such as the supported format with the communication destination via the communication unit 812 and establishes communication with the communication destination.

[0551] The format conversion unit 814 generates three-dimensional data 832 by performing format conversion on the three-dimensional data 831 received by the data reception unit 811. Furthermore, if the three-dimensional data 831 is compressed or encoded, the format conversion unit 814 performs decompression or decoding.

[0552] Multiple sensors 815 are a group of sensors that acquire information from outside the vehicle, such as LiDAR, visible light cameras, or infrared cameras, and generate sensor information 833. For example, if sensor 815 is a laser sensor such as LiDAR, the sensor information 833 is three-dimensional data such as a point cloud. Note that there are not necessarily multiple sensors 815.

[0553] The three-dimensional data creation unit 816 generates three-dimensional data 834 from the sensor information 833. The three-dimensional data 834 includes information such as a point cloud, visible light image, depth information, sensor position information, or velocity information.

[0554] The three-dimensional data synthesis unit 817 synthesizes three-dimensional data 835, which includes the space in front of the preceding vehicle that cannot be detected by the vehicle's sensors 815, by combining three-dimensional data 834 created based on the vehicle's sensor information 833 with three-dimensional data 832 created by the traffic monitoring cloud or the preceding vehicle.

[0555] The three-dimensional data storage unit 818 stores the generated three-dimensional data 835, etc.

[0556] The communication unit 819 communicates with the traffic monitoring cloud or following vehicles and sends data transmission requests, etc., to the traffic monitoring cloud or following vehicles.

[0557] The transmission control unit 820 exchanges information such as the supported format with the communication destination via the communication unit 819 and establishes communication with the communication destination. The transmission control unit 820 also determines the transmission area, which is the space of the three-dimensional data to be transmitted, based on the three-dimensional data construction information of the three-dimensional data 832 generated by the three-dimensional data synthesis unit 817 and the data transmission request from the communication destination.

[0558] Specifically, the transmission control unit 820 determines a transmission area that includes the space in front of its own vehicle that cannot be detected by the sensors of the following vehicle, in response to a data transmission request from the traffic monitoring cloud or a following vehicle. The transmission control unit 820 also determines the transmission area by determining whether the transmissionable space or the transmitted space has been updated based on the three-dimensional data construction information. For example, the transmission control unit 820 determines the transmission area to be the area specified in the data transmission request and in which the corresponding three-dimensional data 835 exists. The transmission control unit 820 then notifies the format conversion unit 821 of the format supported by the communication destination and the transmission area.

[0559] The format conversion unit 821 generates three-dimensional data 837 by converting the three-dimensional data 836 in the transmission area from the three-dimensional data 835 stored in the three-dimensional data storage unit 818 to a format supported by the receiving side. The format conversion unit 821 may also reduce the amount of data by compressing or encoding the three-dimensional data 837.

[0560] The data transmission unit 822 transmits three-dimensional data 837 to a traffic monitoring cloud or following vehicles. This three-dimensional data 837 includes, for example, information such as a point cloud in front of the vehicle, including areas that are blind spots for following vehicles, visible light images, depth information, or sensor position information.

[0561] Although this example describes a case where format conversion is performed by the format conversion units 814 and 821, format conversion is not required.

[0562] With this configuration, the three-dimensional data creation device 810 acquires three-dimensional data 831 from an external source for areas that cannot be detected by the vehicle's sensors 815, and generates three-dimensional data 835 by combining the three-dimensional data 831 with three-dimensional data 834 based on sensor information 833 detected by the vehicle's sensors 815. In this way, the three-dimensional data creation device 810 can generate three-dimensional data for areas that cannot be detected by the vehicle's sensors 815.

[0563] Furthermore, the three-dimensional data creation device 810 can transmit three-dimensional data, including the space in front of its own vehicle that cannot be detected by the sensors of the following vehicle, to the traffic monitoring cloud or following vehicle in response to a data transmission request from the traffic monitoring cloud or following vehicle.

[0564] Next, the procedure for transmitting three-dimensional data to a following vehicle using the three-dimensional data creation device 810 will be described. Figure 115 is a flowchart showing an example of the procedure for transmitting three-dimensional data to a traffic monitoring cloud or a following vehicle using the three-dimensional data creation device 810.

[0565] First, the three-dimensional data creation device 810 generates and updates three-dimensional data 835 of the space including the space on the road in front of the vehicle (S801). Specifically, the three-dimensional data creation device 810 constructs three-dimensional data 835 that includes the space in front of the vehicle ahead, which cannot be detected by the vehicle's sensors 815, by combining three-dimensional data 834 created based on the vehicle's sensor information 833 with three-dimensional data 831 created by the traffic monitoring cloud or the vehicle ahead.

[0566] Next, the three-dimensional data creation device 810 determines whether the three-dimensional data 835 contained in the transmitted space has changed (S802).

[0567] If a vehicle or person enters the transmitted space from the outside, causing a change in the three-dimensional data 835 contained in that space (Yes in S802), the three-dimensional data creation device 810 transmits the three-dimensional data, including the three-dimensional data 835 of the space that has been changed, to the traffic monitoring cloud or the following vehicle (S803).

[0568] The 3D data creation device 810 may transmit the 3D data of the space where the change has occurred in accordance with the transmission timing of the 3D data transmitted at predetermined intervals, or it may transmit it immediately after detecting the change. In other words, the 3D data creation device 810 may transmit the 3D data of the space where the change has occurred with priority over the 3D data transmitted at predetermined intervals.

[0569] Furthermore, the three-dimensional data creation device 810 may transmit all of the three-dimensional data of the space in which the change occurred, or it may transmit only the difference in the three-dimensional data (for example, information on three-dimensional points that have appeared or disappeared, or displacement information of three-dimensional points).

[0570] Furthermore, the three-dimensional data creation device 810 may transmit metadata related to its own vehicle's hazard avoidance actions, such as sudden braking warnings, to following vehicles prior to the three-dimensional data of the space where the change has occurred. This allows following vehicles to recognize sudden braking by the preceding vehicle earlier and initiate hazard avoidance actions such as deceleration earlier.

[0571] If no change has occurred in the three-dimensional data 835 contained in the transmitted space (No in S802), or after step S803, the three-dimensional data creation device 810 transmits the three-dimensional data contained in a space of a predetermined shape located at a distance L in front of its own vehicle to the traffic monitoring cloud or a following vehicle (S804).

[0572] Furthermore, for example, the processes in steps S801 to S804 are repeated at predetermined time intervals.

[0573] Furthermore, if there is no difference between the three-dimensional data 835 of the space to be transmitted and the three-dimensional map, the three-dimensional data creation device 810 does not need to transmit the three-dimensional data 837 of the space.

[0574] In this embodiment, the client device transmits sensor information obtained from the sensor to the server or another client device.

[0575] First, the system configuration according to this embodiment will be described. Figure 116 is a diagram showing the configuration of the three-dimensional map and sensor information transmission and reception system according to this embodiment. This system includes a server 901 and client devices 902A and 902B. When client devices 902A and 902B are not specifically distinguished, they will also be referred to as client device 902.

[0576] The client device 902 is, for example, an in-vehicle device mounted on a moving object such as a vehicle. The server 901 is, for example, a traffic monitoring cloud and is capable of communicating with multiple client devices 902.

[0577] Server 901 transmits a three-dimensional map composed of point clouds to client device 902. Note that the composition of the three-dimensional map is not limited to point clouds; it may also represent other three-dimensional data, such as a mesh structure.

[0578] The client device 902 transmits sensor information acquired by the client device 902 to the server 901. The sensor information includes, for example, at least one of the following: LiDAR acquisition information, visible light image, infrared image, depth image, sensor position information, and velocity information.

[0579] The data transmitted and received between the server 901 and the client device 902 may be compressed to reduce data size, or it may be left uncompressed to maintain data accuracy. When data is compressed, a three-dimensional compression method based on an octave structure, for example, can be used for point clouds. In addition, a two-dimensional image compression method can be used for visible light images, infrared images, and depth images. A two-dimensional image compression method is, for example, MPEG-4 AVC or HEVC, which are standardized by MPEG.

[0580] Furthermore, in response to a request from the client device 902 to send a 3D map, the server 901 sends a 3D map managed by the server 901 to the client device 902. The server 901 may also send a 3D map without waiting for a request from the client device 902. For example, the server 901 may broadcast a 3D map to one or more client devices 902 located in a predetermined space. Alternatively, the server 901 may send a 3D map appropriate to the location of the client device 902 at regular intervals after receiving a transmission request from the client device 902. The server 901 may also send a 3D map to the client device 902 whenever the 3D map managed by the server 901 is updated.

[0581] The client device 902 sends a request to the server 901 to send a three-dimensional map. For example, if the client device 902 wants to perform self-position estimation while driving, the client device 902 sends a request to the server 901 to send a three-dimensional map.

[0582] Furthermore, the client device 902 may request the server 901 to send a 3D map in the following cases: If the 3D map held by the client device 902 is outdated, the client device 902 may request the server 901 to send a 3D map. For example, if a certain period of time has elapsed since the client device 902 acquired the 3D map, the client device 902 may request the server 901 to send a 3D map.

[0583] Client device 902 may request server 901 to send the three-dimensional map to the server 901 a certain time before client device 902 leaves the space represented by the three-dimensional map held by client device 902. For example, client device 902 may request server 901 to send the three-dimensional map to the server 901 if it is within a predetermined distance from the boundary of the space represented by the three-dimensional map held by client device 902. Furthermore, if the movement path and speed of client device 902 are known, the time when client device 902 leaves the space represented by the three-dimensional map held by client device 902 may be predicted based on these.

[0584] If the error in the alignment between the three-dimensional data created by the client device 902 from sensor information and the three-dimensional map exceeds a certain level, the client device 902 may request the server 901 to send the three-dimensional map.

[0585] The client device 902 transmits sensor information to the server 901 in response to a request for transmission of sensor information sent from the server 901. The client device 902 may also send sensor information to the server 901 without waiting for a request for transmission of sensor information from the server 901. For example, once the client device 902 receives a request for transmission of sensor information from the server 901, it may periodically transmit sensor information to the server 901 for a certain period. Furthermore, if the error in the alignment between the three-dimensional data created by the client device 902 based on the sensor information and the three-dimensional map obtained from the server 901 exceeds a certain level, the client device 902 may determine that a change has occurred in the three-dimensional map around the client device 902 and transmit this information, along with the sensor information, to the server 901.

[0586] Server 901 requests client device 902 to transmit sensor information. For example, Server 901 receives location information of client device 902, such as GPS, from client device 902. Based on the location information of client device 902, if Server 901 determines that client device 902 is approaching an area with little information on the three-dimensional map managed by Server 901, it requests client device 902 to transmit sensor information in order to generate a new three-dimensional map. Server 901 may also request sensor information transmission if it wants to update the three-dimensional map, check road conditions during snowfall or disasters, check traffic congestion, or check incidents and accidents.

[0587] Furthermore, the client device 902 may set the amount of sensor information data to send to the server 901 depending on the communication status or bandwidth at the time of receiving the sensor information transmission request from the server 901. Setting the amount of sensor information data to send to the server 901 means, for example, increasing or decreasing the data itself, or selecting an appropriate compression method.

[0588] Figure 117 is a block diagram showing an example configuration of the client device 902. The client device 902 receives a three-dimensional map composed of a point cloud, etc., from the server 901, and estimates its own position from the three-dimensional data created based on the sensor information of the client device 902. The client device 902 also transmits the acquired sensor information to the server 901.

[0589] The client device 902 includes a data receiving unit 1011, a communication unit 1012, a reception control unit 1013, a format conversion unit 1014, a plurality of sensors 1015, a three-dimensional data creation unit 1016, a three-dimensional image processing unit 1017, a three-dimensional data storage unit 1018, a format conversion unit 1019, a communication unit 1020, a transmission control unit 1021, and a data transmission unit 1022.

[0590] The data receiving unit 1011 receives the three-dimensional map 1031 from the server 901. The three-dimensional map 1031 is data that includes point clouds such as WLD or SWLD. The three-dimensional map 1031 may contain either compressed or uncompressed data.

[0591] The communication unit 1012 communicates with the server 901 and sends data transmission requests (for example, a request to transmit a 3D map) to the server 901.

[0592] The receiving control unit 1013 exchanges information such as the supported format with the communication destination via the communication unit 1012 and establishes communication with the communication destination.

[0593] The format conversion unit 1014 generates a three-dimensional map 1032 by performing format conversion on the three-dimensional map 1031 received by the data reception unit 1011. Furthermore, if the three-dimensional map 1031 is compressed or encoded, the format conversion unit 1014 performs decompression or decoding. However, if the three-dimensional map 1031 is uncompressed data, the format conversion unit 1014 does not perform decompression or decoding.

[0594] Multiple sensors 1015 are a group of sensors that acquire external information from the vehicle on which the client device 902 is installed, such as LiDAR, visible light cameras, infrared cameras, or depth sensors, and generate sensor information 1033. For example, if sensor 1015 is a laser sensor such as LiDAR, the sensor information 1033 is three-dimensional data such as a point cloud (point cloud data). Note that there are not necessarily multiple sensors 1015.

[0595] The three-dimensional data creation unit 1016 creates three-dimensional data 1034 of the vehicle's surroundings based on the sensor information 1033. For example, the three-dimensional data creation unit 1016 uses information acquired by LiDAR and visible light images obtained by a visible light camera to create point cloud data with color information of the vehicle's surroundings.

[0596] The three-dimensional image processing unit 1017 uses the received three-dimensional map 1032, such as a point cloud, and the three-dimensional data 1034 of the vehicle's surroundings generated from sensor information 1033 to perform self-position estimation processing for the vehicle. Alternatively, the three-dimensional image processing unit 1017 may create three-dimensional data 1035 of the vehicle's surroundings by combining the three-dimensional map 1032 and the three-dimensional data 1034, and then perform self-position estimation processing using the created three-dimensional data 1035.

[0597] The three-dimensional data storage unit 1018 stores the three-dimensional map 1032, three-dimensional data 1034, and three-dimensional data 1035, etc.

[0598] The format conversion unit 1019 generates sensor information 1037 by converting the sensor information 1033 to a format supported by the receiving side. The format conversion unit 1019 may also reduce the amount of data by compressing or encoding the sensor information 1037. Furthermore, the format conversion unit 1019 may omit processing if format conversion is not necessary. The format conversion unit 1019 may also control the amount of data transmitted according to the specified transmission range.

[0599] The communication unit 1020 communicates with the server 901 and receives data transmission requests (sensor information transmission requests), etc., from the server 901.

[0600] The transmission control unit 1021 exchanges information such as the supported format with the communication destination via the communication unit 1020 and establishes communication.

[0601] The data transmission unit 1022 transmits sensor information 1037 to the server 901. The sensor information 1037 includes information acquired by multiple sensors 1015, such as information acquired by LiDAR, brightness images acquired by a visible light camera, infrared images acquired by an infrared camera, depth images acquired by a depth sensor, sensor position information, and velocity information.

[0602] Next, the configuration of server 901 will be described. Figure 118 is a block diagram showing an example configuration of server 901. Server 901 receives sensor information transmitted from client device 902 and creates three-dimensional data based on the received sensor information. Server 901 updates the three-dimensional map it manages using the created three-dimensional data. In addition, in response to a request from client device 902 to transmit the three-dimensional map, server 901 transmits the updated three-dimensional map to client device 902.

[0603] Server 901 comprises a data receiving unit 1111, a communication unit 1112, a reception control unit 1113, a format conversion unit 1114, a three-dimensional data creation unit 1116, a three-dimensional data synthesis unit 1117, a three-dimensional data storage unit 1118, a format conversion unit 1119, a communication unit 1120, a transmission control unit 1121, and a data transmission unit 1122.

[0604] The data receiving unit 1111 receives sensor information 1037 from the client device 902. The sensor information 1037 includes, for example, information acquired by LiDAR, brightness images acquired by a visible light camera, infrared images acquired by an infrared camera, depth images acquired by a depth sensor, sensor position information, and velocity information.

[0605] The communication unit 1112 communicates with the client device 902 and sends data transmission requests (for example, requests to transmit sensor information) to the client device 902.

[0606] The receiving control unit 1113 exchanges information such as the supported format with the communication destination via the communication unit 1112 and establishes communication.

[0607] The format conversion unit 1114 generates sensor information 1132 by decompressing or decoding the received sensor information 1037 if it is compressed or encoded. However, the format conversion unit 1114 does not perform decompression or decoding if the sensor information 1037 is uncompressed data.

[0608] The three-dimensional data creation unit 1116 creates three-dimensional data 1134 of the area around the client device 902 based on the sensor information 1132. For example, the three-dimensional data creation unit 1116 uses information acquired by LiDAR and visible light images obtained by a visible light camera to create point cloud data with color information of the area around the client device 902.

[0609] The three-dimensional data synthesis unit 1117 updates the three-dimensional map 1135 managed by the server 901 by synthesizing the three-dimensional data 1134, which was created based on the sensor information 1132, with the three-dimensional map 1135.

[0610] The three-dimensional data storage unit 1118 stores three-dimensional maps 1135, etc.

[0611] The format conversion unit 1119 generates a three-dimensional map 1031 by converting the three-dimensional map 1135 to a format supported by the receiving side. The format conversion unit 1119 may also reduce the amount of data by compressing or encoding the three-dimensional map 1135. Furthermore, the format conversion unit 1119 may omit processing if format conversion is not necessary. The format conversion unit 1119 may also control the amount of data transmitted according to the specified transmission range.

[0612] The communication unit 1120 communicates with the client device 902 and receives data transmission requests (such as requests to transmit a three-dimensional map) from the client device 902.

[0613] The transmission control unit 1121 exchanges information such as the supported format with the communication destination via the communication unit 1120 and establishes communication.

[0614] The data transmission unit 1122 transmits the three-dimensional map 1031 to the client device 902. The three-dimensional map 1031 is data that includes point clouds such as WLD or SWLD. The three-dimensional map 1031 may contain either compressed or uncompressed data.

[0615] Next, we will describe the operation flow of the client device 902. Figure 119 is a flowchart showing the operation of the client device 902 when acquiring a three-dimensional map.

[0616] First, the client device 902 requests the server 901 to transmit a three-dimensional map (such as a point cloud) (S1001). At this time, the client device 902 may also transmit its own location information obtained by GPS or the like, and request the server 901 to transmit a three-dimensional map related to that location information.

[0617] Next, the client device 902 receives a three-dimensional map from the server 901 (S1002). If the received three-dimensional map is compressed data, the client device 902 decodes the received three-dimensional map to generate an uncompressed three-dimensional map (S1003).

[0618] Next, the client device 902 creates three-dimensional data 1034 of the area around the client device 902 from sensor information 1033 obtained from multiple sensors 1015 (S1004). Then, the client device 902 estimates its own position using the three-dimensional map 1032 received from the server 901 and the three-dimensional data 1034 created from the sensor information 1033 (S1005).

[0619] Figure 120 is a flowchart illustrating the operation of the client device 902 when transmitting sensor information. First, the client device 902 receives a request to transmit sensor information from the server 901 (S1011). Upon receiving the transmission request, the client device 902 transmits sensor information 1037 to the server 901 (S1012). If the sensor information 1033 includes multiple pieces of information obtained from multiple sensors 1015, the client device 902 may generate sensor information 1037 by compressing each piece of information using a compression method suitable for each piece of information.

[0620] Next, the operation flow of server 901 will be described. Figure 121 is a flowchart showing the operation of server 901 when acquiring sensor information. First, server 901 requests client device 902 to send sensor information (S1021). Next, server 901 receives sensor information 1037 sent from client device 902 in response to the request (S1022). Next, server 901 creates three-dimensional data 1134 using the received sensor information 1037 (S1023). Next, server 901 reflects the created three-dimensional data 1134 in the three-dimensional map 1135 (S1024).

[0621] Figure 122 is a flowchart illustrating the operation of server 901 when transmitting a three-dimensional map. First, server 901 receives a request to transmit a three-dimensional map from client device 902 (S1031). Upon receiving the request to transmit a three-dimensional map, server 901 transmits the three-dimensional map 1031 to client device 902 (S1032). At this time, server 901 may extract a three-dimensional map of the vicinity of client device 902 according to its location information and transmit the extracted three-dimensional map. Alternatively, server 901 may compress the three-dimensional map composed of a point cloud using, for example, an octave tree compression method, and transmit the compressed three-dimensional map.

[0622] The following describes some modifications of this embodiment.

[0623] Server 901 uses sensor information 1037 received from client device 902 to create three-dimensional data 1134 of the area around client device 902. Next, server 901 calculates the difference between the created three-dimensional data 1134 and the three-dimensional map 1135 of the same area managed by server 901 by matching them. If the difference is greater than or equal to a predetermined threshold, server 901 determines that some kind of abnormality has occurred around client device 902. For example, when ground subsidence occurs due to a natural disaster such as an earthquake, a large difference may occur between the three-dimensional map 1135 managed by server 901 and the three-dimensional data 1134 created based on sensor information 1037.

[0624] The sensor information 1037 may include information indicating at least one of the following: the type of sensor, the performance of the sensor, and the model number of the sensor. Furthermore, a class ID corresponding to the sensor's performance may be added to the sensor information 1037. For example, if the sensor information 1037 is information acquired by a LiDAR, it is conceivable to assign identifiers to the sensor's performance, such as class 1 for sensors that can acquire information with accuracy in the millimeter range, class 2 for sensors that can acquire information with accuracy in the centimeter range, and class 3 for sensors that can acquire information with accuracy in the meter range. The server 901 may also estimate the sensor's performance information from the model number of the client device 902. For example, if the client device 902 is mounted in a vehicle, the server 901 may determine the sensor's specifications from the vehicle's make and model. In this case, the server 901 may have previously acquired information about the vehicle's make and model, or this information may be included in the sensor information. The server 901 may also use the acquired sensor information 1037 to switch the degree of correction applied to the three-dimensional data 1134 created using the sensor information 1037. For example, if the sensor performance is high precision (Class 1), the server 901 does not perform any correction on the three-dimensional data 1134. If the sensor performance is low precision (Class 3), the server 901 applies a correction to the three-dimensional data 1134 according to the accuracy of the sensor. For example, the lower the accuracy of the sensor, the stronger the degree (intensity) of the correction applied by the server 901.

[0625] Server 901 may simultaneously send requests for the transmission of sensor information to multiple client devices 902 located in a given space. When Server 901 receives multiple sensor information from multiple client devices 902, it is not necessary to use all of the sensor information to create the three-dimensional data 1134. For example, it may select which sensor information to use depending on the performance of the sensors. For example, when updating the three-dimensional map 1135, Server 901 may select high-precision sensor information (Class 1) from the multiple sensor information received and use the selected sensor information to create the three-dimensional data 1134.

[0626] Server 901 is not limited to servers such as traffic monitoring clouds, but may also be other client devices (in-vehicle). Figure 123 shows the system configuration in this case.

[0627] For example, client device 902C requests sensor information from a nearby client device 902A and obtains the sensor information from client device 902A. Then, client device 902C uses the obtained sensor information from client device 902A to create three-dimensional data and updates the three-dimensional map of client device 902C. In this way, client device 902C can generate a three-dimensional map of the space obtainable from client device 902A, taking advantage of the performance of client device 902C. For example, this case is likely to occur when client device 902C has high performance.

[0628] In this case, client device 902A, which provided the sensor information, is granted the right to acquire the high-precision three-dimensional map generated by client device 902C. Client device 902A receives the high-precision three-dimensional map from client device 902C in accordance with that right.

[0629] Furthermore, client device 902C may send requests for the transmission of sensor information to multiple nearby client devices 902 (client devices 902A and 902B). If the sensor of client device 902A or client device 902B is high-performance, client device 902C can create three-dimensional data using the sensor information obtained from this high-performance sensor.

[0630] Figure 124 is a block diagram showing the functional configuration of the server 901 and the client device 902. The server 901 includes, for example, a three-dimensional map compression / decoding processing unit 1201 that compresses and decodes three-dimensional maps, and a sensor information compression / decoding processing unit 1202 that compresses and decodes sensor information.

[0631] The client device 902 comprises a three-dimensional map decoding processing unit 1211 and a sensor information compression processing unit 1212. The three-dimensional map decoding processing unit 1211 receives encoded data of the compressed three-dimensional map, decodes the encoded data, and obtains the three-dimensional map. The sensor information compression processing unit 1212 compresses the sensor information itself instead of the three-dimensional data created from the acquired sensor information, and sends the encoded data of the compressed sensor information to the server 901. With this configuration, the client device 902 only needs to internally store a processing unit (device or LSI) that performs the processing of decoding the three-dimensional map (point cloud, etc.), and does not need to internally store a processing unit that performs the processing of compressing the three-dimensional data of the three-dimensional map (point cloud, etc.). This reduces the cost and power consumption of the client device 902.

[0632] As described above, the client device 902 according to this embodiment is mounted on a mobile body and creates three-dimensional data 1034 of the surrounding area of ​​the mobile body from sensor information 1033 indicating the surrounding conditions of the mobile body obtained by a sensor 1015 mounted on the mobile body. The client device 902 estimates the self-position of the mobile body using the created three-dimensional data 1034. The client device 902 transmits the acquired sensor information 1033 to the server 901 or another client device 902.

[0633] According to this, the client device 902 transmits sensor information 1033 to the server 901, etc. This may reduce the amount of data transmitted compared to transmitting three-dimensional data. In addition, since the client device 902 does not need to perform processing such as compression or encoding of three-dimensional data, the processing load on the client device 902 can be reduced. Therefore, the client device 902 can achieve a reduction in the amount of data transmitted or a simplification of the device configuration.

[0634] Furthermore, the client device 902 sends a request to the server 901 to send a three-dimensional map, and receives the three-dimensional map 1031 from the server 901. In estimating its own position, the client device 902 uses the three-dimensional data 1034 and the three-dimensional map 1032 to estimate its own position.

[0635] Furthermore, the sensor information 1033 includes at least one of the following: information obtained from the laser sensor, brightness image, infrared image, depth image, sensor position information, and sensor velocity information.

[0636] Furthermore, sensor information 1033 includes information indicating the performance of the sensor.

[0637] Furthermore, the client device 902 encodes or compresses the sensor information 1033, and when transmitting the sensor information, it sends the encoded or compressed sensor information 1037 to the server 901 or another client device 902. This allows the client device 902 to reduce the amount of data transmitted.

[0638] For example, the client device 902 includes a processor and memory, and the processor uses the memory to perform the above processing.

[0639] Furthermore, the server 901 according to this embodiment is capable of communicating with a client device 902 mounted on the mobile body, and receives sensor information 1037 from the client device 902 that indicates the surrounding conditions of the mobile body, obtained by a sensor 1015 mounted on the mobile body. The server 901 creates three-dimensional data 1134 of the surroundings of the mobile body from the received sensor information 1037.

[0640] According to this, the server 901 creates three-dimensional data 1134 using sensor information 1037 transmitted from the client device 902. This may reduce the amount of data transmitted compared to when the client device 902 transmits the three-dimensional data. In addition, since the client device 902 does not need to perform processing such as compression or encoding of the three-dimensional data, the processing load on the client device 902 can be reduced. Therefore, the server 901 can reduce the amount of data transmitted or simplify the configuration of the device.

[0641] Furthermore, the server 901 also sends a request to the client device 902 to transmit sensor information.

[0642] Furthermore, the server 901 updates the three-dimensional map 1135 using the created three-dimensional data 1134 and sends the three-dimensional map 1135 to the client device 902 in response to a request from the client device 902 to send the three-dimensional map 1135.

[0643] Furthermore, the sensor information 1037 includes at least one of the following: information obtained from the laser sensor, brightness image, infrared image, depth image, sensor position information, and sensor velocity information.

[0644] Furthermore, sensor information 1037 includes information indicating the performance of the sensor.

[0645] Furthermore, the server 901 corrects the three-dimensional data according to the performance of the sensor. This allows the three-dimensional data creation method to improve the quality of the three-dimensional data.

[0646] Furthermore, when receiving sensor information, the server 901 receives multiple pieces of sensor information 1037 from multiple client devices 902, and selects the sensor information 1037 to be used to create the three-dimensional data 1134 based on the multiple pieces of information indicating the performance of the sensors contained in the multiple pieces of sensor information 1037. In this way, the server 901 can improve the quality of the three-dimensional data 1134.

[0647] Furthermore, the server 901 decodes or decodes the received sensor information 1037 and creates three-dimensional data 1134 from the decoded or decoded sensor information 1132. This allows the server 901 to reduce the amount of data transmitted.

[0648] For example, server 901 is equipped with a processor and memory, and the processor uses the memory to perform the above processing.

[0649] The following describes some variations. Figure 125 is a diagram showing the configuration of the system according to this embodiment. The system shown in Figure 125 includes a server 2001, a client device 2002A, and a client device 2002B.

[0650] Client devices 2002A and 2002B are mounted on a moving object such as a vehicle and transmit sensor information to server 2001. Server 2001 transmits a three-dimensional map (point cloud) to client devices 2002A and 2002B.

[0651] Client device 2002A comprises a sensor information acquisition unit 2011, a storage unit 2012, and a data transmission feasibility determination unit 2013. The configuration of client device 2002B is similar. Furthermore, in the following description, unless otherwise specified, client device 2002A and client device 2002B will also be referred to as client device 2002.

[0652] Figure 126 is a flowchart showing the operation of the client device 2002 according to this embodiment.

[0653] The sensor information acquisition unit 2011 acquires various sensor information using sensors (sensor group) mounted on the mobile body. In other words, the sensor information acquisition unit 2011 acquires sensor information indicating the surrounding conditions of the mobile body obtained by sensors (sensor group) mounted on the mobile body. The sensor information acquisition unit 2011 also stores the acquired sensor information in the storage unit 2012. This sensor information includes at least one of LiDAR acquisition information, visible light images, infrared images, and depth images. The sensor information may also include at least one of sensor position information, velocity information, acquisition time information, and acquisition location information. Sensor position information indicates the position of the sensor that acquired the sensor information. Velocity information indicates the velocity of the mobile body when the sensor acquired the sensor information. Acquisition time information indicates the time when the sensor information was acquired by the sensor. Acquisition location information indicates the position of the mobile body or sensor when the sensor information was acquired by the sensor.

[0654] Next, the data transmission feasibility determination unit 2013 determines whether the mobile device (client device 2002) is in an environment where it can transmit sensor information to the server 2001 (S2002). For example, the data transmission feasibility determination unit 2013 may use information such as GPS to identify the location and time of the client device 2002 and determine whether data can be transmitted. Alternatively, the data transmission feasibility determination unit 2013 may determine whether data can be transmitted based on whether it can connect to a specific access point.

[0655] If the client device 2002 determines that the moving object is in an environment where it can transmit sensor information to the server 2001 (Yes in S2002), it transmits the sensor information to the server 2001 (S2003). In other words, as soon as the client device 2002 is in a situation where it can transmit sensor information to the server 2001, it transmits the sensor information it holds to the server 2001. For example, suppose a millimeter-wave access point capable of high-speed communication is installed at an intersection. When the client device 2002 enters the intersection, it transmits the sensor information it holds to the server 2001 at high speed using millimeter-wave communication.

[0656] Next, the client device 2002 deletes the sensor information already transmitted to the server 2001 from the storage unit 2012 (S2004). The client device 2002 may also delete sensor information that has not yet been transmitted to the server 2001 if it meets certain conditions. For example, the client device 2002 may delete the sensor information from the storage unit 2012 when the acquisition time of the stored sensor information becomes older than a certain time from the current time. In other words, the client device 2002 may delete the sensor information from the storage unit 2012 if the difference between the time the sensor information was acquired by the sensor and the current time exceeds a predetermined time. Furthermore, the client device 2002 may delete the sensor information from the storage unit 2012 when the acquisition location of the stored sensor information is more than a certain distance from the current location. In other words, the client device 2002 may delete the sensor information from the storage unit 2012 if the difference between the position of the moving object or sensor when the sensor information was acquired by the sensor and the current position of the moving object or sensor exceeds a predetermined distance. This makes it possible to reduce the capacity of the storage unit 2012 of the client device 2002.

[0657] If client device 2002 has not finished acquiring sensor information (No in S2005), client device 2002 repeats the processing from step S2001 onwards. If client device 2002 has finished acquiring sensor information (Yes in S2005), client device 2002 terminates processing.

[0658] Furthermore, the client device 2002 may select the sensor information to send to the server 2001 according to the communication status. For example, if high-speed communication is possible, the client device 2002 will prioritize sending sensor information with a large size stored in the memory unit 2012 (e.g., LiDAR acquisition information). If high-speed communication is difficult, the client device 2002 will send sensor information with a small size stored in the memory unit 2012 and a high priority (e.g., visible light images). This allows the client device 2002 to efficiently send the sensor information stored in the memory unit 2012 to the server 2001 according to the network status.

[0659] Furthermore, the client device 2002 may obtain time information indicating the current time and location information indicating the current location from the server 2001. The client device 2002 may also determine the acquisition time and location of sensor information based on the acquired time and location information. In other words, the client device 2002 may obtain time information from the server 2001 and generate acquisition time information using the acquired time information. Furthermore, the client device 2002 may obtain location information from the server 2001 and generate acquisition location information using the acquired location information.

[0660] For example, regarding time information, the server 2001 and client device 2002 synchronize their time using a mechanism such as NTP (Network Time Protocol) or PTP (Precision Time Protocol). This allows client device 2002 to obtain accurate time information. Furthermore, since the server 2001 can synchronize time between multiple client devices, the time in sensor information acquired by different client devices 2002 can be synchronized. Therefore, the server 2001 can handle sensor information that indicates the synchronized time. Note that any method other than NTP or PTP can be used for time synchronization. Also, GPS information may be used as the above time information and location information.

[0661] Server 2001 may obtain sensor information from multiple client devices 2002 by specifying the time or location. For example, in the event of an accident, to find clients that were nearby, Server 2001 broadcasts a sensor information transmission request to multiple client devices 2002, specifying the time and location of the accident. Client devices 2002 that have sensor information for the corresponding time and location then transmit the sensor information to Server 2001. In other words, Client device 2002 receives a sensor information transmission request from Server 2001 that includes specification information specifying the location and time. If Client device 2002 determines that the storage unit 2012 has stored sensor information obtained at the location and time indicated by the specification information, and that the moving object is in an environment where it can transmit sensor information to Server 2001, it transmits the sensor information obtained at the location and time indicated by the specification information to Server 2001. As a result, Server 2001 can obtain sensor information related to the occurrence of an accident from multiple client devices 2002 and use it for accident analysis, etc.

[0662] Furthermore, the client device 2002 may refuse to transmit sensor information when it receives a request from the server 2001 to transmit sensor information. Alternatively, the client device 2002 may pre-configure which of the multiple sensor information requests it can transmit. Or, the server 2001 may query the client device 2002 each time to determine whether or not to transmit sensor information.

[0663] Furthermore, client devices 2002 that transmit sensor information to server 2001 may be awarded points. These points can be used to pay for things like gasoline, electric vehicle (EV) charging fees, highway tolls, or rental car fees. Also, after acquiring sensor information, server 2001 may delete information that identifies the client device 2002 that sent the sensor information. For example, this information could be the network address of client device 2002. This anonymizes the sensor information, allowing users of client device 2002 to confidently transmit sensor information from client device 2002 to server 2001. Server 2001 may also consist of multiple servers. For example, by sharing sensor information among multiple servers, even if one server fails, other servers can communicate with client device 2002. This prevents service interruptions due to server failures.

[0664] Furthermore, the specified location in the sensor information transmission request indicates the location where the accident occurred, and may differ from the location of the client device 2002 at the specified time specified in the sensor information transmission request. Therefore, the server 2001 can request information acquisition from client devices 2002 located within a specified range, such as within XXm of the location, by specifying a range such as within XXm of the location. Similarly, for the specified time, the server 2001 may specify a range such as within N seconds before or after a certain time. This allows the server 2001 to acquire sensor information from client devices 2002 that were located within XXm of absolute position S at "time: tN to t+N". When the client device 2002 transmits three-dimensional data such as LiDAR, it may transmit data generated immediately after time t.

[0665] Furthermore, the server 2001 may separately specify information indicating the location of the client device 2002 from which sensor information is to be acquired, and the location where the sensor information is desired. For example, the server 2001 specifies that sensor information including at least the range YYm from absolute position S should be acquired from a client device 2002 located within XXm of absolute position S. When the client device 2002 selects the three-dimensional data to transmit, it selects one or more randomly accessible units of three-dimensional data so as to include at least the sensor information within the specified range. Also, when the client device 2002 transmits a visible light image, it may transmit multiple temporally consecutive image data, including at least the frame immediately before or after time t.

[0666] If the client device 2002 can utilize multiple physical networks, such as 5G, WiFi, or multiple modes in 5G, for transmitting sensor information, the client device 2002 may select the network to use according to the priority notified by the server 2001. Alternatively, the client device 2002 may select a network that can secure appropriate bandwidth based on the size of the data to be transmitted. Alternatively, the client device 2002 may select a network to use based on the cost of data transmission, etc. Furthermore, the transmission request from the server 2001 may include information indicating a transmission deadline, such as transmitting if the client device 2002 can start transmitting by time T. If sufficient sensor information is not obtained within the deadline, the server 2001 may issue another transmission request.

[0667] The sensor information may include compressed or uncompressed sensor data, along with header information indicating the characteristics of the sensor data. The client device 2002 may transmit the header information to the server 2001 via a different physical network or communication protocol than the sensor data. For example, the client device 2002 transmits the header information to the server 2001 prior to transmitting the sensor data. The server 2001 determines whether to acquire the sensor data from the client device 2002 based on the analysis results of the header information. For example, the header information may include information indicating the LiDAR point cloud acquisition density, elevation angle, or frame rate, or the resolution, signal-to-noise ratio, or frame rate of the visible light image. This allows the server 2001 to acquire sensor information from the client device 2002 that has sensor data of the determined quality.

[0668] As described above, the client device 2002 is mounted on the mobile body and acquires sensor information indicating the surrounding conditions of the mobile body obtained by sensors mounted on the mobile body, and stores the sensor information in the storage unit 2012. The client device 2002 determines whether the mobile body is in an environment where it can transmit sensor information to the server 2001, and if it determines that the mobile body is in an environment where it can transmit sensor information to the server, it transmits the sensor information to the server 2001.

[0669] Furthermore, the client device 2002 creates three-dimensional data of the surroundings of the moving object from the sensor information and uses the created three-dimensional data to estimate the self-position of the moving object.

[0670] Furthermore, the client device 2002 sends a request to the server 2001 to send a 3D map, and receives the 3D map from the server 2001. In estimating its own position, the client device 2002 uses the 3D data and the 3D map to estimate its own position.

[0671] Furthermore, the processing performed by the client device 2002 may be implemented as an information transmission method in the client device 2002.

[0672] Furthermore, the client device 2002 includes a processor and memory, and the processor may use the memory to perform the above processing.

[0673] Next, the sensor information collection system according to this embodiment will be described. Figure 127 is a diagram showing the configuration of the sensor information collection system according to this embodiment. As shown in Figure 127, the sensor information collection system according to this embodiment includes terminal 2021A, terminal 2021B, communication device 2022A, communication device 2022B, network 2023, data collection server 2024, map server 2025, and client device 2026. Note that terminal 2021A and terminal 2021B will also be referred to as terminal 2021 unless specifically distinguished. Similarly, communication device 2022A and communication device 2022B will also be referred to as communication device 2022 unless specifically distinguished.

[0674] The data collection server 2024 collects data such as sensor data obtained from sensors on terminal 2021 as position-related data that is associated with its position in three-dimensional space.

[0675] Sensor data refers to data acquired using sensors installed in terminal 2021, such as the surrounding conditions of terminal 2021 or the internal conditions of terminal 2021. Terminal 2021 transmits sensor data collected from one or more sensor devices located in a position where it can communicate directly with terminal 2021, or where it can communicate via one or more relay devices using the same communication method, to the data acquisition server 2024.

[0676] The location-related data may include, for example, information indicating the operating status of the terminal itself or the equipment installed on the terminal, operation logs, and service usage status. Furthermore, the location-related data may include information linking the identifier of terminal 2021 to the location or travel path of terminal 2021.

[0677] The location-related data contains location information that corresponds to location information in three-dimensional data, such as three-dimensional map data. Details of the location information will be described later.

[0678] Location-related data may include, in addition to location information which indicates a location, at least one of the following: time information as described above, and information indicating the attributes of the data included in the location-related data, or the type of sensor that generated the data (e.g., model number). Location information and time information may be stored in the header area of ​​the location-related data or in the header area of ​​the frame that stores the location-related data. Alternatively, location information and time information may be transmitted and / or stored separately from the location-related data as metadata associated with the location-related data.

[0679] The map server 2025 is connected to, for example, the network 2023 and transmits three-dimensional data, such as three-dimensional map data, in response to requests from other devices, such as the terminal 2021. Furthermore, as described in each of the embodiments above, the map server 2025 may also have a function to update the three-dimensional data using sensor information transmitted from the terminal 2021.

[0680] The data collection server 2024 is connected to network 2023, for example, and collects location-related data from other devices such as terminal 2021. It stores the collected location-related data in a storage device, either internally or on another server. The data collection server 2024 also transmits the collected location-related data or metadata of 3D map data generated based on the location-related data to terminal 2021 upon request from terminal 2021.

[0681] Network 2023 is a communication network, such as the Internet. Terminal 2021 is connected to Network 2023 via communication device 2022. Communication device 2022 communicates with terminal 2021 by switching between one or more communication methods. Communication device 2022 is, for example, (1) a base station such as LTE (Long Term Evolution), (2) an access point (AP) such as WiFi or millimeter wave communication, (3) a gateway for an LPWA (Low Power Wide Area) Network such as SIGFOX, LoRaWAN or Wi-SUN, or (4) a communication satellite that communicates using a satellite communication method such as DVB-S2.

[0682] The base station may communicate with terminal 2021 using a method classified as NB-IoT (Narrow Band-IoT) or LPWA such as LTE-M, or it may communicate with terminal 2021 while switching between these methods.

[0683] Here, we take an example where terminal 2021 has the function to communicate with communication device 2022 that uses two types of communication methods, and communicates with map server 2025 or data collection server 2024 using one of these communication methods, or by switching between multiple communication methods and the communication device 2022 that is the direct communication partner. However, the configuration of the sensor information collection system and terminal 2021 is not limited to this. For example, terminal 2021 may not have the function to communicate using multiple communication methods, but may have the function to communicate using one of the communication methods. Also, terminal 2021 may support three or more communication methods. Furthermore, each terminal 2021 may support different communication methods.

[0684] Terminal 2021 has the configuration of a client device 902, for example, as shown in Figure 117. Terminal 2021 performs position estimation, such as its own position, using the received three-dimensional data. Terminal 2021 also generates position-related data by associating sensor data acquired from sensors with position information obtained through position estimation processing.

[0685] The location information added to location-related data indicates, for example, the position in the coordinate system used in the three-dimensional data. For example, the location information is a coordinate value expressed as latitude and longitude. In this case, terminal 2021 may include information indicating the coordinate system on which the coordinate value is based, and the three-dimensional data used for position estimation, along with the coordinate value itself. The coordinate value may also include altitude information.

[0686] Furthermore, location information may be associated with data units or spatial units that can be used to encode the three-dimensional data described above. These units include, for example, WLD, GOS, SPC, VLM, or VXL. In this case, location information is represented by an identifier that identifies a data unit such as an SPC that corresponds to location-related data. In addition to the identifier that identifies a data unit such as an SPC, location information may also include information indicating three-dimensional data encoded from the three-dimensional space containing the data unit such as the SPC, or information indicating a detailed location within the SPC. Information indicating three-dimensional data is, for example, the file name of the three-dimensional data.

[0687] Thus, by generating location-related data associated with location information based on position estimation using three-dimensional data, this system can add more accurate location information to sensor information than when adding location information based on the self-position of a client device (terminal 2021) acquired using GPS. As a result, even when other devices use the location-related data in other services, it may be possible to more accurately identify the location corresponding to the location-related data in real space by performing position estimation based on the same three-dimensional data.

[0688] In this embodiment, the example of ...

Claims

1. A three-dimensional data encoding method performed by a processor, Encoded data is generated by encoding point cloud data containing multiple three-dimensional points. A bitstream is generated that includes the encoded data and conformance index, and satisfies conformance information corresponding to the conformance index. The conformance information includes at least one of the maximum number of bits for a slice corresponding to a portion of the encoded data and the maximum number of bits for the encoded data. Three-dimensional data encoding method.

2. The point cloud data includes positional information for each three-dimensional point, The maximum number of bits in the slice or the encoded data is related to the number of bits after encoding the position information. The three-dimensional data encoding method according to claim 1.

3. The conformance information further includes the maximum number of three-dimensional points in the slice. The three-dimensional data encoding method according to claim 1 or 2.

4. The conformance information includes the maximum number of bits in the slice, the maximum number of bits in the encoded data, and the maximum number of three-dimensional points in the slice. The three-dimensional data encoding method according to claim 2.

5. A three-dimensional data decoding method performed by a processor, A bitstream is obtained that includes encoded data in which point cloud data containing multiple three-dimensional points is encoded, and a conformance index, and that satisfies conformance information corresponding to the conformance index. The encoded data is decoded, The conformance information includes at least one of the maximum number of bits for a slice corresponding to a portion of the encoded data and the maximum number of bits for the encoded data. Three-dimensional data decoding method.

6. moreover, Determining whether the bitstream satisfies the conformance information. The method for decoding three-dimensional data according to claim 5.

7. The point cloud data includes positional information for each three-dimensional point, The aforementioned maximum number of bits relates to the number of bits after encoding the location information. The method for decoding three-dimensional data according to claim 5.

8. The conformance information further includes the maximum number of three-dimensional points in the slice. The method for decoding three-dimensional data according to claim 5.

9. The conformance information includes the maximum number of bits in the slice, the maximum number of bits in the encoded data, and the maximum number of three-dimensional points in the slice. The method for decoding three-dimensional data according to claim 8.

10. The slice represents a portion of the plurality of three-dimensional points in the point cloud data. A method for decoding three-dimensional data according to any one of claims 5 to 9.

11. The encoded data corresponds to the encoded point cloud frame. A method for decoding three-dimensional data according to any one of claims 5 to 9.

12. Each value of the information included in the conformance information differs depending on the conformance index. A method for decoding three-dimensional data according to any one of claims 5 to 9.

13. Processor and Equipped with memory, The processor uses the memory to: Encoded data is generated by encoding point cloud data containing multiple three-dimensional points. A bitstream is generated that includes the encoded data and conformance index, and satisfies conformance information corresponding to the conformance index. The conformance information includes at least one of the maximum number of bits for a slice corresponding to a portion of the encoded data and the maximum number of bits for the encoded data. Three-dimensional data encoding device.

14. Processor and Equipped with memory, The processor uses the memory to: A bitstream is obtained that includes encoded data in which point cloud data containing multiple three-dimensional points is encoded, and a conformance index, and that satisfies conformance information corresponding to the conformance index. The encoded data is decoded, The conformance information includes at least one of the maximum number of bits for a slice corresponding to a portion of the encoded data and the maximum number of bits for the encoded data. Three-dimensional data decoding device.

Images