Three-dimensional data encoding method, three-dimensional data decoding method, three-dimensional data encoding device, and three-dimensional data decoding device

Abstract

A three-dimensional data encoding method comprises dividing point cloud data into a plurality of sub-point cloud data (S7551), shifting each of the plurality of sub-point cloud data according to a predetermined amount of shift (S7552), and generating a bit stream by encoding the plurality of sub-point cloud data after shifting (S7553). The bit stream includes first control information that is common to the plurality of sub-point cloud data and includes first information regarding the amount of shift of the plurality of sub-point cloud data, and a plurality of second control information for each sub-point cloud data.

Description

Technology Field

[0001] The present disclosure relates to a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, and a three-dimensional data decoding device. Background Technology

[0002] In a wide range of fields, such as computer vision for the autonomous operation of automobiles or robots, map information, surveillance, infrastructure inspection, or image transmission, the widespread adoption of devices or services utilizing three-dimensional data is expected in the future. Three-dimensional data is acquired through various methods, including distance sensors such as rangefinders, stereo cameras, or combinations of multiple monocular cameras.

[0003] One method of representing three-dimensional data is called a point cloud, which represents the shape of a three-dimensional structure using a point cloud within three-dimensional space. In a point cloud, the location and color of the point cloud are stored. While point clouds are expected to become the mainstream method for representing three-dimensional data, the amount of data from a point cloud is very large. Therefore, for the storage or transmission of three-dimensional data, compression of the data volume through encoding becomes essential, just as with two-dimensional video (for example, MPEG-4 AVC or HEVC standardized by MPEG).

[0004] In addition, point cloud compression is partially supported by open libraries (Point Cloud Libraries) that perform point cloud-related processing.

[0005] In addition, a technology is known for searching for and displaying facilities located around a vehicle using three-dimensional map data (see, for example, Patent Document 1). Prior art literature

[0006] International Publication No. 2014 / 020663 The problem to be solved

[0007] In the encoding process of three-dimensional data, it is desired to be able to reduce the amount of encoding.

[0008] The present disclosure aims to provide a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, or a three-dimensional data decoding device capable of reducing the amount of encoding. means of solving the problem

[0009] A three-dimensional data encoding method according to one aspect of the present disclosure is a three-dimensional data encoding method for encoding point cloud data representing a plurality of three-dimensional positions in a three-dimensional space, wherein the point cloud data is divided into a plurality of sub-point cloud data by dividing the three-dimensional space into a plurality of sub-spaces, each of the plurality of sub-point cloud data is moved according to a predetermined amount of movement, and a bit stream is generated by encoding the plurality of sub-point cloud data after movement, and the bit stream includes first control information common to the plurality of sub-point cloud data and including first information regarding the amount of movement of the plurality of sub-point cloud data, and a plurality of second control information for each of the sub-point cloud data.

[0010] A three-dimensional data decoding method according to one aspect of the present disclosure comprises decoding a plurality of sub-point cloud data, in which point cloud data representing a plurality of three-dimensional positions is divided into a plurality of sub-spaces, from a bit stream, the plurality of sub-point cloud data after movement, which are moved according to a predetermined amount of movement, obtaining first information regarding the amount of movement of the plurality of sub-point cloud data from first control information common to the plurality of sub-point cloud data included in the bit stream, calculating a plurality of the said amount of movement of the plurality of sub-point cloud data using the said first information, and restoring the plurality of sub-point cloud data by moving each of the decoded sub-point cloud data after movement according to the said amount of movement corresponding to the sub-point cloud data. Effects of the invention

[0011] The present disclosure may provide a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, or a three-dimensional data decoding device capable of reducing the amount of encoding. Brief explanation of the drawing

[0012] FIG. 1 is a diagram showing the configuration of a three-dimensional data encoding and decoding system according to Embodiment 1. FIG. 2 is a diagram showing an example of the configuration of point cloud data according to Embodiment 1. FIG. 3 is a diagram showing an example of the configuration of a data file in which point cloud data information according to Embodiment 1 is described. FIG. 4 is a diagram showing the types of point cloud data according to Embodiment 1. FIG. 5 is a diagram showing the configuration of the first encoding unit according to embodiment 1. FIG. 6 is a block diagram of the first encoding unit according to embodiment 1. FIG. 7 is a diagram showing the configuration of the first decoding unit according to embodiment 1. FIG. 8 is a block diagram of the first decoding unit according to embodiment 1. FIG. 9 is a diagram showing the configuration of the second encoding unit according to embodiment 1. FIG. 10 is a block diagram of the second encoding unit according to embodiment 1. FIG. 11 is a drawing showing the configuration of the second decoding unit according to embodiment 1. FIG. 12 is a block diagram of the second decoding unit according to embodiment 1. FIG. 13 is a diagram showing a protocol stack related to PCC encoded data according to Embodiment 1. FIG. 14 is a drawing showing the basic structure of an ISOBMFF according to embodiment 2. FIG. 15 is a diagram showing a protocol stack according to embodiment 2. FIG. 16 is a diagram showing an example of saving a NAL unit according to embodiment 2 to a file for codec 1. FIG. 17 is a diagram showing an example of saving a NAL unit according to embodiment 2 to a file for codec 2. FIG. 18 is a drawing showing the configuration of the first multiplexer according to embodiment 2. FIG. 19 is a diagram showing the configuration of the first demultiplexer according to embodiment 2. FIG. 20 is a drawing showing the configuration of a second multiplexer according to embodiment 2. FIG. 21 is a diagram showing the configuration of a second demultiplexer according to embodiment 2. FIG. 22 is a flowchart of processing by the first multiplexer according to embodiment 2. FIG. 23 is a flowchart of processing by the second multiplexer according to embodiment 2. FIG. 24 is a flowchart of processing by the first demultiplexer and the first decoding unit according to embodiment 2. FIG. 25 is a flowchart of processing by the second demultiplexer and the second decoding unit according to embodiment 2. FIG. 26 is a diagram showing the configuration of an encoding unit and a third multiplexing unit according to embodiment 3. FIG. 27 is a diagram showing the configuration of the third demultiplexer and decoding unit according to embodiment 3. FIG. 28 is a flowchart of processing by the third multiplexer according to embodiment 3. FIG. 29 is a flowchart of processing by the third demultiplexer and decoding unit according to embodiment 3. FIG. 30 is a flowchart of processing by a three-dimensional data storage device according to embodiment 3. FIG. 31 is a flowchart of processing by a three-dimensional data acquisition device according to embodiment 3. FIG. 32 is a diagram showing the configuration of the encoding unit and the multiplexing unit according to embodiment 4. FIG. 33 is a diagram showing an example of the configuration of encoded data according to embodiment 4. FIG. 34 is a diagram showing an example of the configuration of encoding data and a NAL unit according to embodiment 4. FIG. 35 is a drawing showing an example of semantics of pcc_nal_unit_type according to embodiment 4. FIG. 36 is a diagram showing an example of the transmission sequence of a NAL unit according to embodiment 4. FIG. 37 is a drawing showing an example of dividing a slice and a tile according to embodiment 5. FIG. 38 is a drawing showing an example of a division pattern of slices and tiles according to embodiment 5. FIG. 39 is a block diagram of the first encoding unit according to embodiment 6. FIG. 40 is a block diagram of the first decoding unit according to embodiment 6. FIG. 41 is a drawing showing an example of the shape of a tile according to embodiment 6. FIG. 42 is a drawing showing an example of a tile and a slice according to embodiment 6. FIG. 43 is a block diagram of a divided section according to embodiment 6. FIG. 44 is a drawing showing an example of a map viewed from the top surface of point cloud data according to embodiment 6. FIG. 45 is a drawing showing an example of tile division according to embodiment 6. FIG. 46 is a drawing showing an example of tile division according to embodiment 6. FIG. 47 is a drawing showing an example of tile division according to embodiment 6. FIG. 48 is a diagram showing an example of tile data stored on a server according to embodiment 6. FIG. 49 is a drawing showing a system for tile division according to embodiment 6. FIG. 50 is a drawing showing an example of slice division according to embodiment 6. FIG. 51 is a drawing showing an example of a dependency relationship according to embodiment 6. FIG. 52 is a diagram showing an example of the decoding sequence of data according to embodiment 6. FIG. 53 is a diagram showing an example of encoding data of a tile according to embodiment 6. FIG. 54 is a block diagram of a joint portion according to embodiment 6. FIG. 55 is a diagram showing an example of the configuration of encoding data and a NAL unit according to embodiment 6. FIG. 56 is a flowchart of encoding processing according to embodiment 6. FIG. 57 is a flowchart of decoding processing according to embodiment 6. FIG. 58 is a drawing showing an example of the syntax of tile additional information according to embodiment 6. FIG. 59 is a block diagram of an encoding and decoding system according to embodiment 6. FIG. 60 is a drawing showing an example of the syntax of slice addition information according to embodiment 6. FIG. 61 is a flowchart of encoding processing according to embodiment 6. FIG. 62 is a flowchart of decoding processing according to embodiment 6. FIG. 63 is a flowchart of encoding processing according to embodiment 6. FIG. 64 is a flowchart of decoding processing according to embodiment 6. FIG. 65 is a drawing showing an example of a division method according to embodiment 7. FIG. 66 is a diagram showing an example of point cloud data division according to embodiment 7. FIG. 67 is a drawing showing an example of the syntax of tile additional information according to embodiment 7. FIG. 68 is a drawing showing an example of index information according to embodiment 7. FIG. 69 is a drawing showing an example of a dependency relationship according to embodiment 7. FIG. 70 is a drawing showing an example of transmission data according to embodiment 7. FIG. 71 is a drawing showing an example of the configuration of a NAL unit according to embodiment 7. FIG. 72 is a drawing showing an example of a dependency relationship according to Embodiment 7. FIG. 73 is a diagram showing an example of the decoding sequence of data according to embodiment 7. FIG. 74 is a drawing showing an example of a dependency relationship according to Embodiment 7. FIG. 75 is a diagram showing an example of the decoding sequence of data according to embodiment 7. FIG. 76 is a flowchart of encoding processing according to embodiment 7. FIG. 77 is a flowchart of decoding processing according to embodiment 7. FIG. 78 is a flowchart of encoding processing according to embodiment 7. FIG. 79 is a flowchart of encoding processing according to embodiment 7. FIG. 80 is a drawing showing an example of transmitted data and received data according to embodiment 7. FIG. 81 is a flowchart of decoding processing according to embodiment 7. FIG. 82 is a drawing showing an example of transmitted data and received data according to embodiment 7. FIG. 83 is a flowchart of decoding processing according to embodiment 7. FIG. 84 is a flowchart of encoding processing according to embodiment 7. FIG. 85 is a drawing showing an example of index information according to embodiment 7. FIG. 86 is a drawing showing an example of a dependency relationship according to embodiment 7. FIG. 87 is a drawing showing an example of transmission data according to embodiment 7. FIG. 88 is a drawing showing an example of transmitted data and received data according to embodiment 7. FIG. 89 is a flowchart of decoding processing according to embodiment 7. FIG. 90 is a flowchart of encoding processing according to embodiment 7. FIG. 91 is a flowchart of decoding processing according to embodiment 7. FIG. 92 is a block diagram showing an example of the configuration of a three-dimensional data encoding device according to embodiment 8. FIG. 93 is a diagram for explaining the schematic of an encoding method by a three-dimensional data encoding device according to embodiment 8. FIG. 94 is a drawing for explaining a first example of a position shift according to embodiment 8. FIG. 95 is a drawing for explaining a second example of a position shift according to embodiment 8. FIG. 96 is a flowchart showing an example of a encoding method according to embodiment 8. FIG. 97 is a flowchart showing an example of a decoding method according to embodiment 8. FIG. 98 is a drawing for explaining a third example of a position shift according to embodiment 8. FIG. 99 is a flowchart showing an example of a encoding method according to embodiment 8. FIG. 100 is a flowchart showing an example of a decoding method according to embodiment 8. FIG. 101 is a drawing for explaining a fourth example of a position shift according to embodiment 8. FIG. 102 is a flowchart showing an example of a encoding method according to embodiment 8. FIG. 103 is a drawing for explaining a fifth example of a position shift according to embodiment 8. FIG. 104 is a drawing for explaining an encoding method according to embodiment 8. FIG. 105 is a drawing showing an example of the syntax of a GPS according to embodiment 8. FIG. 106 is a drawing showing an example of the syntax of a header of position information according to embodiment 8. FIG. 107 is a flowchart illustrating an example of a encoding method that switches processing according to embodiment 8. FIG. 108 is a flowchart illustrating an example of a decoding method that switches processing according to embodiment 8. FIG. 109 is a diagram showing an example of the data configuration of a bit stream according to embodiment 8. FIG. 110 is an example of a case where the divided data of FIG. 109 according to embodiment 8 is framed. FIG. 111 is a drawing showing another example of a divided area according to embodiment 8. FIG. 112 is a drawing showing another example of a divided area according to embodiment 8. FIG. 113 is a drawing showing another example of a divided area according to embodiment 8. FIG. 114 is a drawing showing another example of a divided area according to embodiment 8. FIG. 115 is a drawing showing another example of data configuration according to embodiment 8. FIG. 116 is a drawing showing another example of data configuration according to embodiment 8. FIG. 117 is a drawing showing another example of data configuration according to embodiment 8. FIG. 118 is a drawing showing another example of data configuration according to embodiment 8. FIG. 119 is a drawing showing another example of data configuration according to embodiment 8. FIG. 120 is a flowchart of encoding processing according to embodiment 8. FIG. 121 is a flowchart of decoding processing according to embodiment 8. FIG. 122 is a drawing showing examples of individual position shift amounts and divided area position shift amounts according to embodiment 9. FIG. 123 is a flowchart of the shift amount calculation process according to embodiment 9. FIG. 124 is a diagram schematically illustrating the division process of minimum position coordinates according to embodiment 9. FIG. 125 is a diagram showing an example of calculating a shift amount in a three-dimensional data decoding device according to embodiment 9. FIG. 126 is a flowchart of the shift amount calculation process according to embodiment 9. FIG. 127 is a diagram schematically illustrating the division process of minimum position coordinates according to embodiment 9. FIG. 128 is a diagram showing an example of calculating a shift amount in a three-dimensional data decoding device according to embodiment 9. FIG. 129 is a diagram showing an example of setting a lower bit according to embodiment 9. FIG. 130 is a flowchart of the shift amount calculation process according to embodiment 9. FIG. 131 is a flowchart of the shift amount calculation process according to embodiment 9. FIG. 132 is a drawing showing an example of the syntax of a GPS according to embodiment 9. FIG. 133 is a drawing showing an example of the syntax of a position information header according to embodiment 9. FIG. 134 is a flowchart of the shift amount transmission processing according to embodiment 9. FIG. 135 is a flowchart of the shift amount calculation process according to embodiment 9. FIG. 136 is a flowchart of three-dimensional data encoding processing according to embodiment 9. FIG. 137 is a flowchart of three-dimensional data decoding processing according to embodiment 9. Specific details for implementing the invention

[0013] A three-dimensional data encoding method according to one aspect of the present disclosure is a three-dimensional data encoding method for encoding point cloud data representing a plurality of three-dimensional positions in a three-dimensional space, wherein the point cloud data is divided into a plurality of sub-point cloud data by dividing the three-dimensional space into a plurality of sub-spaces, each of the plurality of sub-point cloud data is moved according to a predetermined amount of movement, and a bit stream is generated by encoding the plurality of sub-point cloud data after movement, and the bit stream includes first control information common to the plurality of sub-point cloud data and including first information regarding the amount of movement of the plurality of sub-point cloud data, and a plurality of second control information for each of the sub-point cloud data.

[0014] According to this, the three-dimensional data encoding method can transmit a displacement amount to a three-dimensional data decoding device based on the first information. In addition, the amount of encoding can be reduced by integrating the first information into multiple sub-point cloud data.

[0015] For example, the above amount of movement may be a amount of movement based on either the sub-point cloud data or the sub-space containing the sub-point cloud data.

[0016] For example, the above displacement amount includes an upper bit and a lower bit, the above first information is information common to the plurality of sub-point cloud data and indicates the number of bits of the lower bit, and each of the plurality of second control information may include second information indicating the value of the upper bit included in the displacement amount for the sub-point cloud data corresponding to the second control information.

[0017] According to this, the three-dimensional data encoding method can transmit a displacement amount to a three-dimensional data decoding device by means of first information and second information. In addition, the amount of encoding can be reduced by integrating the first information into multiple sub-point cloud data.

[0018] For example, the first control information includes a flag indicating whether information indicating the number of bits of the lower bit is included in the first control information or in each of the plurality of second control informations, and when the flag indicates that information indicating the number of bits of the lower bit is included in the first control information, the first control information includes the first information, and each of the plurality of second control informations does not include information indicating the number of bits of the lower bit, and when the flag indicates that information indicating the number of bits of the lower bit is included in each of the plurality of second control informations, each of the plurality of second control informations includes third information indicating the number of bits of the lower bit included in the shift amount for the sub-point cloud data corresponding to the second control information, and the first control information does not have to include the first information.

[0019] According to this, the three-dimensional data encoding method can switch whether the first information is common to multiple sub-point cloud data, so appropriate encoding is performed.

[0020] For example, all bits included in the lower bits have a value of 0, and the bit stream does not need to include information indicating the value of the lower bits.

[0021] A three-dimensional data decoding method according to one aspect of the present disclosure comprises decoding a plurality of sub-point cloud data, in which point cloud data representing a plurality of three-dimensional positions is divided into a plurality of sub-spaces, from a bit stream, the plurality of sub-point cloud data after movement, which are moved according to a predetermined amount of movement, obtaining first information regarding the amount of movement of the plurality of sub-point cloud data from first control information common to the plurality of sub-point cloud data included in the bit stream, calculating a plurality of the said amount of movement of the plurality of sub-point cloud data using the said first information, and restoring the plurality of sub-point cloud data by moving each of the decoded sub-point cloud data after movement according to the said amount of movement corresponding to the sub-point cloud data.

[0022] According to this, the three-dimensional data decoding method can calculate the amount of movement based on the first information. In addition, since the first information is common to multiple sub-point cloud data, the amount of encoding is reduced.

[0023] For example, the above amount of movement may be a amount of movement based on either the sub-point cloud data or the sub-space containing the sub-point cloud data.

[0024] For example, the above displacement amount includes an upper bit and a lower bit, the above first information is information common to the plurality of sub-point cloud data and indicates the number of bits of the lower bit, and the above three-dimensional data decoding method may also obtain a plurality of second information indicating the value of the upper bit of the displacement amount of each of the plurality of sub-point cloud data from a plurality of second control information for each of the sub-point cloud data included in the bit stream, and in the calculation of the displacement amount, the plurality of displacement amounts of the plurality of sub-point cloud data may be calculated using the above first information and the above second information.

[0025] According to this, the three-dimensional data decoding method can calculate the amount of movement based on the first information and the second information. In addition, since the first information is common to multiple sub-point cloud data, the amount of encoding is reduced.

[0026] For example, the first control information may include a flag indicating whether information indicating the number of bits of the lower bit is included in the first control information or in each of the plurality of second control information, and when the flag indicates that information indicating the number of bits of the lower bit is included in the first control information, the first information may be obtained from the first control information, and the plurality of shift amounts of the plurality of sub-point cloud data may be calculated using the first information and the plurality of second information, and when the flag indicates that information indicating the number of bits of the lower bit is included in each of the plurality of second control information, the plurality of third information indicating the number of bits of the lower bit of the shift amount of each of the plurality of sub-point cloud data may be obtained from each of the plurality of second control information, and the plurality of shift amounts of the plurality of sub-point cloud data may be calculated using the plurality of third information and the plurality of second information.

[0027] According to this, the three-dimensional data decoding method can switch whether to convert into multiple sub-point cloud data based on the first information, so appropriate decoding is performed.

[0028] For example, in calculating the above amount of movement, all bits included in the lower bit may be set to a value of 0.

[0029] In addition, a three-dimensional data encoding device according to one aspect of the present disclosure is a three-dimensional data encoding device for encoding point cloud data representing a plurality of three-dimensional positions in a three-dimensional space, comprising a processor and a memory, wherein the processor uses the memory to divide the point cloud data into a plurality of sub-point cloud data by dividing the three-dimensional space into a plurality of sub-spaces, moves each of the plurality of sub-point cloud data according to a predetermined amount of movement, and generates a bit stream by encoding the plurality of sub-point cloud data after movement, wherein the bit stream includes first control information common to the plurality of sub-point cloud data and including first information regarding the amount of movement of the plurality of sub-point cloud data, and a plurality of second control information for each of the sub-point cloud data.

[0030] According to this, the three-dimensional data encoding device can transmit the amount of movement to the three-dimensional data decoding device by means of the first information. In addition, the amount of encoding can be reduced by integrating the first information into a plurality of sub-point cloud data.

[0031] In addition, a three-dimensional data decoding device according to one aspect of the present disclosure comprises a processor and a memory, wherein the processor uses the memory to decode a plurality of sub-point cloud data, which are divided into a plurality of three-dimensional positions by dividing a three-dimensional space into a plurality of sub-spaces, and then decodes a plurality of sub-point cloud data after movement according to a predetermined amount of movement from a bit stream, and obtains first information regarding the amount of movement of the plurality of sub-point cloud data from first control information common to the plurality of sub-point cloud data included in the bit stream, calculates a plurality of the said amount of movement of the plurality of sub-point cloud data using the first information, and restores the plurality of sub-point cloud data by moving each of the decoded sub-point cloud data after movement according to the said amount of movement corresponding to the sub-point cloud data.

[0032] According to this, the three-dimensional data decoding device can calculate the amount of movement based on the first information. In addition, since the first information is common to multiple sub-point cloud data, the amount of code is reduced.

[0033] In addition, these comprehensive or specific embodiments may be realized as a system, method, integrated circuit, computer program, or recording medium such as a computer-readable CD-ROM, or as any combination of a system, method, integrated circuit, computer program, and recording medium.

[0034] Hereinafter, embodiments will be described in detail with reference to the drawings. Furthermore, all embodiments described below represent specific examples of the present disclosure. The numerical values, shapes, materials, components, arrangement positions and connection types of components, steps, and sequence of steps shown in the following embodiments are examples and are not well known facts that limit the present disclosure. Additionally, regarding the components in the following embodiments, components not described in independent claims are described as arbitrary components.

[0035] (Embodiment 1)

[0036] When using encoded point cloud data in actual devices or services, it is desirable to transmit and receive necessary information according to the application to conserve network bandwidth. However, until now, such a function has not existed in the encoding structure of three-dimensional data, nor has there been an encoding method for it.

[0037] In this embodiment, a three-dimensional data encoding method and a three-dimensional data encoding device for providing a function to transmit and receive information necessary according to the application in three-dimensional point cloud encoded data, a three-dimensional data decoding method and a three-dimensional data decoding device for decoding said encoded data, a three-dimensional data multiplexing method for multiplexing said encoded data, and a three-dimensional data transmission method for transmitting said encoded data are described.

[0038] In particular, currently, a first encoding method and a second encoding method are being considered as encoding methods (encoding schemes) for point cloud data, but since the composition of the encoded data and the method for storing the encoded data in a system format are not defined, there is a problem that MUX processing (multiplexing) in the encoding unit, or transmission or storage, cannot be performed as is.

[0039] Also, there is no method to support a format in which two codecs, such as the first encoding method and the second encoding method, are mixed, such as PCC (Point Cloud Compression).

[0040] In this embodiment, the configuration of PCC encoded data in which two codecs of the first encoding method and the second encoding method are mixed, and a method for storing the encoded data in a system format are described.

[0041] First, the configuration of a three-dimensional data (point cloud data) encoding and decoding system according to the present embodiment will be explained. FIG. 1 is a diagram showing an example of the configuration of a three-dimensional data encoding and decoding system according to the present embodiment. As shown in FIG. 1, the three-dimensional data encoding and decoding system includes a three-dimensional data encoding system (4601), a three-dimensional data decoding system (4602), a sensor terminal (4603), and an external connection unit (4604).

[0042] The three-dimensional data encoding system (4601) generates encoded data or multiplexed data by encoding point cloud data, which is three-dimensional data. Additionally, the three-dimensional data encoding system (4601) may be a three-dimensional data encoding device realized by a single device, or a system realized by multiple devices. Furthermore, the three-dimensional data encoding device may include some of the multiple processing units included in the three-dimensional data encoding system (4601).

[0043] The three-dimensional data encoding system (4601) includes a point cloud data generation system (4611), a presentation unit (4612), an encoding unit (4613), a multiplexing unit (4614), an input / output unit (4615), and a control unit (4616). The point cloud data generation system (4611) includes a sensor information acquisition unit (4617) and a point cloud data generation unit (4618).

[0044] The sensor information acquisition unit (4617) acquires sensor information from the sensor terminal (4603) and outputs the sensor information to the point cloud data generation unit (4618). The point cloud data generation unit (4618) generates point cloud data from the sensor information and outputs the point cloud data to the encoding unit (4613).

[0045] The presentation unit (4612) presents sensor information or point cloud data to the user. For example, the presentation unit (4612) displays information or an image based on the sensor information or point cloud data.

[0046] The encoding unit (4613) encodes (compresses) point cloud data and outputs the obtained encoded data, control information obtained during the encoding process, and other additional information to the multiplexing unit (4614). The additional information includes, for example, sensor information.

[0047] The multiplexing unit (4614) generates multiplexed data by multiplexing the encoded data input from the encoding unit (4613), control information, and additional information. The format of the multiplexed data is, for example, a file format for storage or a packet format for transmission.

[0048] The input / output unit (4615) (e.g., a communication unit or an interface) outputs multiplexed data to the outside. Alternatively, the multiplexed data is stored in an accumulation unit such as an internal memory. The control unit (4616) (or application execution unit) controls each processing unit. That is, the control unit (4616) performs control such as encoding and multiplexing.

[0049] Additionally, sensor information may be input to the encoding unit (4613) or the multiplexing unit (4614). Also, the input / output unit (4615) may output point cloud data or encoded data to the outside as is.

[0050] The transmission signal (multiplexed data) output from the three-dimensional data encoding system (4601) is input to the three-dimensional data decoding system (4602) through the external connection unit (4604).

[0051] The three-dimensional data decoding system (4602) generates point cloud data, which is three-dimensional data, by decoding encoded data or multiplexed data. Additionally, the three-dimensional data decoding system (4602) may be a three-dimensional data decoding device realized by a single device, or a system realized by multiple devices. Furthermore, the three-dimensional data decoding device may include some of the multiple processing units included in the three-dimensional data decoding system (4602).

[0052] The three-dimensional data decoding system (4602) includes a sensor information acquisition unit (4621), an input / output unit (4622), a demultiplexer unit (4623), a decoding unit (4624), a presentation unit (4625), a user interface (4626), and a control unit (4627).

[0053] The sensor information acquisition unit (4621) acquires sensor information from the sensor terminal (4603).

[0054] The input / output unit (4622) acquires a transmission signal, decodes multiplexed data (file format or packet) from the transmission signal, and outputs the multiplexed data to the demultiplexer (4623).

[0055] The demultiplexer (4623) obtains encoded data, control information, and additional information from the multiplexed data, and outputs the encoded data, control information, and additional information to the decoder (4624).

[0056] The decoding unit (4624) reconstructs point cloud data by decoding the encoded data.

[0057] The presentation unit (4625) presents point cloud data to the user. For example, the presentation unit (4625) displays information or images based on the point cloud data. The user interface (4626) obtains instructions based on the user's operation. The control unit (4627) (or application execution unit) controls each processing unit. That is, the control unit (4627) performs control such as demultiplexing, decoding, and presentation.

[0058] Additionally, the input / output unit (4622) may acquire point cloud data or encoded data directly from the outside. Also, the presentation unit (4625) may acquire additional information such as sensor information and present information based on the additional information. Also, the presentation unit (4625) may perform presentation based on user instructions acquired from the user interface (4626).

[0059] The sensor terminal (4603) generates sensor information, which is information obtained from the sensor. The sensor terminal (4603) is a terminal equipped with a sensor or a camera, and, for example, may be a moving object such as a car, a flying object such as an airplane, a portable terminal, or a camera.

[0060] Sensor information obtainable from the sensor terminal (4603) is, for example, (1) the distance between the sensor terminal (4603) and an object, or the reflectance of the object, obtained from a LIDAR, millimeter-wave radar, or infrared sensor, (2) the distance between the camera and an object or the reflectance of the object, obtained from multiple monocular camera images or stereo camera images. In addition, the sensor information may include the attitude, direction, gyroscope (angular velocity), position (GPS information or altitude), speed, or acceleration of the sensor. In addition, the sensor information may include temperature, atmospheric pressure, humidity, or magnetic field.

[0061] The external connection part (4604) is realized by an integrated circuit (LSI or IC), an external storage part, communication with a cloud server via the internet, or broadcasting, etc.

[0062] Next, point cloud data will be described. Figure 2 is a diagram showing the structure of point cloud data. Figure 3 is a diagram showing an example of the structure of a data file in which information about point cloud data is described.

[0063] Point cloud data contains data from multiple points. The data for each point includes location information (three-dimensional coordinates) and attribute information regarding that location information. A collection of these points is called a point cloud. For example, a point cloud represents the three-dimensional shape of an object.

[0064] Position information, such as three-dimensional coordinates, is sometimes referred to as geometry. Additionally, the data for each point may include attribute information of multiple attribute types. Attribute types include, for example, color or reflectance.

[0065] One attribute information may be mapped to one location information, or one attribute information having multiple different attribute types may be mapped to one location information. Also, multiple attribute information of the same attribute type may be mapped to one location information.

[0066] The example of the data file configuration shown in Figure 3 is an example where location information and attribute information correspond one-to-one, and shows the location information and attribute information of N points constituting the point cloud data.

[0067] Position information is, for example, information on the three axes x, y, and z. Attribute information is, for example, color information in RGB. Representative data files include ply files.

[0068] Next, the types of point cloud data will be explained. Figure 4 is a diagram showing the types of point cloud data. As shown in Figure 4, point cloud data includes static objects and dynamic objects.

[0069] A static object is three-dimensional point cloud data at an arbitrary time (a specific point in time). A dynamic object is three-dimensional point cloud data that changes over time. From now on, the three-dimensional point cloud data at a specific time is referred to as a PCC frame, or simply a frame.

[0070] The object may be a point cloud with a limited area, like conventional image data, or a large-scale point cloud with an unrestricted area, like map information.

[0071] In addition, point cloud data of various densities may exist, and sparse and dense point cloud data may also be present.

[0072] Below, details of each processing unit will be described. Sensor information is acquired in various ways, such as distance sensors like LIDAR or rangefinders, stereo cameras, or combinations of multiple monocular cameras. The point cloud data generation unit (4618) generates point cloud data based on the sensor information obtained from the sensor information acquisition unit (4617). The point cloud data generation unit (4618) generates location information as point cloud data and adds attribute information regarding the location information to the location information.

[0073] The point cloud data generation unit (4618) may process the point cloud data when generating location information or adding attribute information. For example, the point cloud data generation unit (4618) may reduce the amount of data by deleting point clouds with overlapping locations. In addition, the point cloud data generation unit (4618) may transform the location information (position shift, rotation, or normalization, etc.) and render the attribute information.

[0074] In addition, in FIG. 1, the point cloud data generation system (4611) is included in the three-dimensional data encoding system (4601), but it may be installed independently outside the three-dimensional data encoding system (4601).

[0075] The encoding unit (4613) generates encoded data by encoding point cloud data according to a predetermined encoding method. There are two main types of encoding methods. The first is an encoding method using position information, and this encoding method will be referred to as the first encoding method. The second is an encoding method using a video codec, and this encoding method will be referred to as the second encoding method.

[0076] The decoding unit (4624) decodes point cloud data by decoding the encoded data according to a predetermined encoding method.

[0077] The multiplexer (4614) generates multiplexed data by multiplexing encoded data using an existing multiplexing method. The generated multiplexed data is transmitted or accumulated. In addition to PCC encoded data, the multiplexer (4614) multiplexes other media such as video, audio, subtitles, applications, files, or reference time information. Furthermore, the multiplexer (4614) may also multiplex attribute information related to sensor information or point cloud data.

[0078] Multiplexing methods or file formats include ISOBMFF, ISOBMFF-based transmission methods such as MPEG-DASH, MMT, MPEG-2 TS Systems, and RMP.

[0079] The demultiplexer (4623) extracts PCC encoded data, other media, and visual information, etc. from the multiplexed data.

[0080] The input / output unit (4615) transmits multiplexed data using a method suited to a transmitting medium or a storing medium, such as broadcasting or communication. The input / output unit (4615) may communicate with another device via the Internet or with a storage unit such as a cloud server.

[0081] As communication protocols, http, ftp, TCP, or UDP are used. PULL communication methods may be used, or PUSH communication methods may be used.

[0082] Either wired or wireless transmission may be used. For wired transmission, Ethernet (registered trademark), USB, RS-232C, HDMI (registered trademark), or coaxial cable, etc., are used. For wireless transmission, wireless LAN, Wi-Fi (registered trademark), Bluetooth (registered trademark), or millimeter wave, etc., are used.

[0083] In addition, broadcasting systems such as DVB-T2, DVB-S2, DVB-C2, ATSC3.0, or ISDB-S3 are used.

[0084] FIG. 5 is a diagram showing the configuration of a first encoding unit (4630), which is an example of an encoding unit (4613) that performs encoding of the first encoding method. FIG. 6 is a block diagram of the first encoding unit (4630). The first encoding unit (4630) generates encoded data (encoded stream) by encoding point cloud data using the first encoding method. This first encoding unit (4630) includes a position information encoding unit (4631), an attribute information encoding unit (4632), an additional information encoding unit (4633), and a multiplexing unit (4634).

[0085] The first encoding unit (4630) has the characteristic of performing encoding while being conscious of the three-dimensional structure. In addition, the first encoding unit (4630) has the characteristic that the attribute information encoding unit (4632) performs encoding using information obtained from the position information encoding unit (4631). The first encoding method is also called GPCC (Geometry based PCC).

[0086] Point cloud data is PCC point cloud data such as a PLY file, or PCC point cloud data generated from sensor information, and includes position information, attribute information, and other additional information (metadata). Position information is input to the position information encoding unit (4631), attribute information is input to the attribute information encoding unit (4632), and additional information is input to the additional information encoding unit (4633).

[0087] The location information encoding unit (4631) generates encoded location information (Compressed Geometry), which is encoded data, by encoding location information. For example, the location information encoding unit (4631) encodes location information using an N-ary tree structure such as an octal tree. Specifically, in an octal tree, the target space is divided into 8 nodes (subspaces), and 8-bit information (occupancy code) indicating whether a point cloud is included in each node is generated. Also, the node containing the point cloud is further divided into 8 nodes, and 8-bit information indicating whether a point cloud is included in each of the 8 nodes is generated. This process is repeated until the number of point clouds included in a predetermined hierarchy or node is below a threshold.

[0088] The attribute information encoding unit (4632) generates encoded attribute information (Compressed Attribute), which is encoded data, by encoding using the configuration information generated by the location information encoding unit (4631). For example, the attribute information encoding unit (4632) determines a reference point (reference node) to be referenced in encoding the target point (target node) of the processing target based on the octal tree structure generated by the location information encoding unit (4631). For example, the attribute information encoding unit (4632) references a node among the surrounding nodes or adjacent nodes where the parent node in the octal tree is the same as the target node. Furthermore, the method of determining the reference relationship is not limited to this.

[0089] Additionally, the encoding process of attribute information may include at least one of quantization processing, prediction processing, and arithmetic encoding processing. In this case, "reference" refers to using a reference node to calculate the predicted value of the attribute information, or using the state of a reference node (e.g., occupancy information indicating whether the reference node contains a point cloud) to determine the encoding parameters. For example, the encoding parameters are quantization parameters in the quantization processing, or context in arithmetic encoding, etc.

[0090] The additional information encoding unit (4633) generates encoded additional information (Compressed MetaData), which is encoded data, by encoding compressible data among the additional information.

[0091] The multiplexer (4634) generates a compressed stream, which is encoded data, by multiplexing the encoding location information, encoding attribute information, encoding additional information, and other additional information. The generated compressed stream is output to a processing unit of a system layer not shown.

[0092] Next, a first decoding unit (4640), which is an example of a decoding unit (4624) that performs decoding of the first encoding method, will be described. FIG. 7 is a diagram showing the configuration of the first decoding unit (4640). FIG. 8 is a block diagram of the first decoding unit (4640). The first decoding unit (4640) generates point cloud data by decoding encoded data (encoded stream) encoded by the first encoding method using the first encoding method. This first decoding unit (4640) includes a demultiplexer (4641), a position information decoding unit (4642), an attribute information decoding unit (4643), and an additional information decoding unit (4644).

[0093] A compressed stream, which is encoded data, is input to the first decoding unit (4640) from the processing unit of the system layer that is not shown in the city.

[0094] The demultiplexer (4641) separates the compressed geometry, compressed attribute, compressed metadata, and other additional information from the compressed data.

[0095] The position information decoding unit (4642) generates position information by decoding encoded position information. For example, the position information decoding unit (4642) restores position information of a point group represented in three-dimensional coordinates from encoded position information represented in an N-ary tree structure such as an octal tree.

[0096] The attribute information decoding unit (4643) decodes encoded attribute information based on the configuration information generated by the location information decoding unit (4642). For example, the attribute information decoding unit (4643) determines a reference point (reference node) to be referenced in decoding a target point (target node) of a processing target based on the octal tree structure obtained from the location information decoding unit (4642). For example, the attribute information decoding unit (4643) references a node among surrounding nodes or adjacent nodes where the parent node in the octal tree is the same as the target node. Furthermore, the method for determining the reference relationship is not limited to this.

[0097] Additionally, the decoding process of attribute information may include at least one of inverse quantization processing, prediction processing, and arithmetic decoding processing. In this case, "reference" refers to using a reference node to calculate the predicted value of the attribute information, or using the state of a reference node (e.g., occupancy information indicating whether the reference node contains a point cloud) to determine decoding parameters. For example, decoding parameters are quantization parameters in the inverse quantization processing, or context in arithmetic decoding, etc.

[0098] The additional information decoding unit (4644) generates additional information by decoding the encoded additional information. Additionally, the first decoding unit (4640) uses the additional information required for decoding the location information and attribute information during decoding, and outputs the additional information required for the application to the outside.

[0099] Next, a second encoding unit (4650), which is an example of an encoding unit (4613) that performs encoding of the second encoding method, will be described. FIG. 9 is a diagram showing the configuration of the second encoding unit (4650). FIG. 10 is a block diagram of the second encoding unit (4650).

[0100] The second encoding unit (4650) generates encoded data (encoded stream) by encoding point cloud data using a second encoding method. This second encoding unit (4650) includes an additional information generating unit (4651), a position image generating unit (4652), an attribute image generating unit (4653), an image encoding unit (4654), an additional information encoding unit (4655), and a multiplexing unit (4656).

[0101] The second encoding unit (4650) is characterized by generating a position image and an attribute image by projecting a three-dimensional structure onto a two-dimensional image, and encoding the generated position image and attribute image using a conventional image encoding method. The second encoding method is also called VPCC (Video based PCC).

[0102] Point cloud data is PCC point cloud data such as a PLY file, or PCC point cloud data generated from sensor information, and includes position information, attribute information, and other additional information (metadata).

[0103] The additional information generation unit (4651) generates map information of multiple two-dimensional images by projecting a three-dimensional structure onto a two-dimensional image.

[0104] The position image generation unit (4652) generates a position image (Geometry Image) based on position information and map information generated by the additional information generation unit (4651). This position image is, for example, a distance image in which depth is represented as a pixel value. In addition, this distance image may be an image in which multiple point groups are viewed from a single viewpoint (an image in which multiple point groups are projected onto a single two-dimensional plane), multiple images in which multiple point groups are viewed from multiple viewpoints, or a single image in which these multiple images are integrated.

[0105] The attribute image generation unit (4653) generates an attribute image based on attribute information and map information generated by the additional information generation unit (4651). This attribute image is, for example, an image in which attribute information (e.g., color (RGB)) is represented as a pixel value. In addition, this image may be an image in which multiple point groups are viewed from a single viewpoint (an image in which multiple point groups are projected onto a single two-dimensional plane), multiple images in which multiple point groups are viewed from multiple viewpoints, or a single image in which these multiple images are integrated.

[0106] The image encoding unit (4654) generates encoded data, namely a compressed geometry image and a compressed attribute image, by encoding the position image and the attribute image using an image encoding method. Additionally, any known encoding method may be used as the image encoding method. For example, the image encoding method may be AVC or HEVC.

[0107] The additional information encoding unit (4655) generates compressed additional information by encoding additional information and map information included in the point cloud data.

[0108] The multiplexer (4656) generates a compressed stream, which is encoded data, by multiplexing an encoded position image, an encoded attribute image, encoded additional information, and other additional information. The generated compressed stream is output to a processing unit of a system layer not shown.

[0109] Next, a second decoding unit (4660), which is an example of a decoding unit (4624) that performs decoding of the second encoding method, will be described. FIG. 11 is a diagram showing the configuration of the second decoding unit (4660). FIG. 12 is a block diagram of the second decoding unit (4660). The second decoding unit (4660) generates point cloud data by decoding encoded data (encoded stream) encoded by the second encoding method using the second encoding method. This second decoding unit (4660) includes a demultiplexer (4661), an image decoding unit (4662), an additional information decoding unit (4663), a location information generation unit (4664), and an attribute information generation unit (4665).

[0110] A compressed stream, which is encoded data, is input to the second decoding unit (4660) from the processing unit of the system layer that is not shown in the city.

[0111] The demultiplexer (4661) separates the compressed geometry image, the compressed attribute image, the compressed metadata, and other additional information from the encoded data.

[0112] The image decoder (4662) generates a position image and an attribute image by decoding the encoded position image and the encoded attribute image using an image encoding method. Additionally, any known encoding method may be used as the image encoding method. For example, the image encoding method may be AVC or HEVC.

[0113] The additional information decoding unit (4663) generates additional information including map information, etc. by decoding the encoded additional information.

[0114] The location information generation unit (4664) generates location information using location images and map information. The attribute information generation unit (4665) generates attribute information using attribute images and map information.

[0115] The second decoding unit (4660) uses additional information required for decoding during decoding and outputs additional information required for the application to the outside.

[0116] The following describes the challenges associated with the PCC encoding method. FIG. 13 is a diagram showing a protocol stack related to PCC encoded data. FIG. 13 illustrates an example of multiplexing data from other media, such as video (e.g., HEVC) or audio, with PCC encoded data, and transmitting or storing it.

[0117] Multiplexing schemes and file formats have the function of multiplexing various encoded data and transmitting or storing them. To transmit or store encoded data, the data must be converted into the format of the multiplexing scheme. For example, HEVC specifies a technique for storing encoded data in a data structure called a NAL unit and storing the NAL unit in an ISOBMFF.

[0118] Meanwhile, currently, a first encoding method (Codec1) and a second encoding method (Codec2) are being considered as encoding methods for point cloud data. However, since the composition of the encoded data and the method for storing the encoded data in a system format are not defined, there is a problem in that MUX processing (multiplexing), transmission, and storage in the encoding unit cannot be performed as they are.

[0119] In addition, if no specific encoding method is described thereafter, it shall be considered as either the first encoding method or the second encoding method.

[0120] (Embodiment 2)

[0121] In this embodiment, a method for saving NAL units to an ISOBMFF file is described.

[0122] ISOBMFF (ISO based media file format) is a file format specification defined in ISO / IEC 14496-12. ISOBMFF defines a format capable of storing various media, such as video, audio, and text, through multiplexing, and is a standard that is independent of the media.

[0123] This explains the basic structure (file) of ISOBMFF. The basic unit in ISOBMFF is the box. A box consists of type, length, and data, and a file is a collection of boxes of various types combined.

[0124] Figure 14 is a diagram showing the basic structure (file) of ISOBMFF. The ISOBMFF file mainly includes boxes such as ftyp, which represents the file brand in 4CC (4 character code), moov, which stores metadata such as control information, and mdat, which stores data.

[0125] Storage methods for ISOBMFF files for each medium are separately specified; for example, storage methods for AVC video and HEVC video are specified in ISO / IEC 14496-15. Here, one might consider extending the functions of ISOBMFF to accumulate or transmit PCC encoded data, but there are no regulations yet regarding storing PCC encoded data in ISOBMFF files. Therefore, in this embodiment, a method for storing PCC encoded data in an ISOBMFF file is described.

[0126] FIG. 15 is a diagram showing the protocol stack when a PCC codec common NAL unit is stored in an ISOBMFF file. Here, a PCC codec common NAL unit is stored in an ISOBMFF file. Although the NAL unit is PCC codec common, since multiple PCC codecs are stored in the NAL unit, it is desirable to define a storage method (Carriage of Codec1, Carriage of Codec2) according to each codec.

[0127] Next, a method for storing a common PCC NAL unit that supports multiple PCC codecs in an ISOBMFF file will be described. FIG. 16 is a diagram showing an example of storing a common PCC NAL unit in an ISOBMFF file of the storage method of Codec 1 (Carriage of Codec 1). FIG. 17 is a diagram showing an example of storing a common PCC NAL unit in an ISOBMFF file of the storage method of Codec 2 (Carriage of Codec 2).

[0128] Here, ftyp is important information for identifying the file format, and for ftyp, a different identifier is defined for each codec. When PCC encoded data encoded with the first encoding method (encoding scheme) is stored in a file, ftyp is set to pcc1. When PCC encoded data encoded with the second encoding method is stored in a file, ftyp is set to pcc2.

[0129] Here, pcc1 indicates that PCC codec 1 (first encoding method) is used. pcc2 indicates that PCC codec 2 (second encoding method) is used. That is, pcc1 and pcc2 indicate that the data is PCC (coded data of three-dimensional data (point cloud data)) and also represent PCC codecs (first encoding method and second encoding method).

[0130] The following describes how to save NAL units to an ISOBMFF file. The multiplexer interprets the NAL unit header, and if pcc_codec_type=Codec1, writes pcc1 to ftyp in ISOBMFF.

[0131] Also, the multiplexer interprets the NAL unit header, and if pcc_codec_type=Codec2, writes pcc2 to ftyp of ISOBMFF.

[0132] Additionally, if pcc_nal_unit_type is metadata, the multiplexer stores the NAL unit in a predetermined way, for example, moov or mdat. If pcc_nal_unit_type is data, the multiplexer stores the NAL unit in a predetermined way, for example, moov or mdat.

[0133] For example, the multiplexer may store the NAL unit size in the NAL unit, just like HEVC.

[0134] By the present storage method, by interpreting the ftyp included in the file in the demultiplexer (system layer), it becomes possible to determine whether PCC encoded data has been encoded by the first encoding method or the second encoding method. Furthermore, as described above, by determining whether PCC encoded data has been encoded by the first encoding method or the second encoding method, encoded data encoded by one of the encoding methods can be extracted from data containing a mixture of encoded data encoded by both encoding methods. By doing so, the amount of data transmitted can be reduced when transmitting encoded data. Additionally, by the present storage method, a common data format can be used in the first encoding method and the second encoding method without setting different data (file) formats.

[0135] In addition, if the codec identification information is indicated in the metadata of the system layer, such as ftyp in ISOBMFF, the multiplexer may save the NAL unit with pcc_nal_unit_type removed to the ISOBMFF file.

[0136] Next, the configuration and operation of the multiplexer provided by the three-dimensional data encoding system (three-dimensional data encoding device) according to the present embodiment, and the demultiplexer provided by the three-dimensional data decoding system (three-dimensional data decoding device) according to the present embodiment will be described.

[0137] FIG. 18 is a diagram showing the configuration of the first multiplexer (4710). The first multiplexer (4710) includes a file conversion unit (4711) that generates multiplexed data (file) by storing the encoded data and control information (NAL unit) generated by the first encoding unit (4630) in an ISOBMFF file. This first multiplexer (4710) is included, for example, in the multiplexer (4614) shown in FIG. 1.

[0138] FIG. 19 is a diagram showing the configuration of a first demultiplexer (4720). The first demultiplexer (4720) has a file inverse conversion unit (4721) that acquires encoded data and control information (NAL unit) from multiplexed data (file) and outputs the acquired encoded data and control information to a first decoding unit (4640). This first demultiplexer (4720) is included in, for example, the demultiplexer (4623) shown in FIG. 1.

[0139] FIG. 20 is a diagram showing the configuration of a second multiplexer (4730). The second multiplexer (4730) includes a file conversion unit (4731) that generates multiplexed data (file) by storing the encoded data and control information (NAL unit) generated by the second encoding unit (4650) in an ISOBMFF file. This second multiplexer (4730) is included in, for example, the multiplexer (4614) shown in FIG. 1.

[0140] FIG. 21 is a diagram showing the configuration of a second demultiplexer (4740). The second demultiplexer (4740) includes a file inverse conversion unit (4741) that acquires encoded data and control information (NAL unit) from multiplexed data (file) and outputs the acquired encoded data and control information to a second decoding unit (4660). This second demultiplexer (4740) is included in, for example, the demultiplexer (4623) shown in FIG. 1.

[0141] FIG. 22 is a flowchart of multiplexing processing by the first multiplexer (4710). First, the first multiplexer (4710) determines whether the codec being used is the first encoding method or the second encoding method by interpreting the pcc_codec_type included in the NAL unit header (S4701).

[0142] If pcc_codec_type indicates a second encoding method (second encoding method in S4702), the first multiplexer (4710) does not process the NAL unit (S4703).

[0143] Meanwhile, if pcc_codec_type indicates a second encoding method (the first encoding method in S4702), the first multiplexer (4710) writes pcc1 in ftyp (S4704). That is, the first multiplexer (4710) writes information in ftyp indicating that data encoded by the first encoding method is stored in a file.

[0144] Next, the first multiplexer (4710) interprets the pcc_nal_unit_type included in the NAL unit header and stores the data in a box (moov or mdat, etc.) in a predetermined way according to the data type indicated in the pcc_nal_unit_type (S4705). Then, the first multiplexer (4710) creates an ISOBMFF file including the ftyp and the box (S4706).

[0145] FIG. 23 is a flowchart of multiplexing processing by the second multiplexer (4730). First, the second multiplexer (4730) determines whether the codec being used is the first encoding method or the second encoding method by interpreting the pcc_codec_type included in the NAL unit header (S4711).

[0146] If pcc_unit_type indicates a second encoding method (second encoding method in S4712), the second multiplexer (4730) writes pcc2 in ftyp (S4713). That is, the second multiplexer (4730) writes information in ftyp indicating that data encoded by the second encoding method is stored in a file.

[0147] Next, the second multiplexer (4730) interprets the pcc_nal_unit_type included in the NAL unit header and stores the data in a box (moov or mdat, etc.) in a predetermined way according to the data type indicated in the pcc_nal_unit_type (S4714). Then, the second multiplexer (4730) creates an ISOBMFF file including the ftyp and the box (S4715).

[0148] Meanwhile, if pcc_unit_type represents the first encoding method (the first encoding method in S4712), the second multiplexer (4730) does not process the NAL unit (S4716).

[0149] Additionally, the above processing illustrates an example of encoding PCC data using either the first encoding method or the second encoding method. The first multiplexer (4710) and the second multiplexer (4730) save the desired NAL unit to a file by identifying the codec type of the NAL unit. Furthermore, if identification information of the PCC codec is included in addition to the NAL unit header, the first multiplexer (4710) and the second multiplexer (4730) may identify the codec type (the first encoding method or the second encoding method) by using the identification information of the PCC codec included in addition to the NAL unit header in steps S4701 and S4711.

[0150] Additionally, the first multiplexer (4710) and the second multiplexer (4730) may delete pcc_nal_unit_type from the NAL unit header and then save the data to the file when saving the data to the file in steps S4706 and S4714.

[0151] FIG. 24 is a flowchart illustrating processing by the first demultiplexer (4720) and the first decoding unit (4640). First, the first demultiplexer (4720) interprets ftyp included in the ISOBMFF file (S4721). If the codec indicated in ftyp is the second encoding method (pcc2) (second encoding method in S4722), the first demultiplexer (4720) determines that the data included in the payload of the NAL unit is data encoded by the second encoding method (S4723). Also, the first demultiplexer (4720) transmits the result of the determination to the first decoding unit (4640). The first decoding unit (4640) does not process the NAL unit (S4724).

[0152] Meanwhile, if the codec indicated in ftyp is the first encoding method (pcc1) (the first encoding method in S4722), the first demultiplexer (4720) determines that the data included in the payload of the NAL unit is data encoded by the first encoding method (S4725). Also, the first demultiplexer (4720) transmits the result of the determination to the first decoding unit (4640).

[0153] The first decoding unit (4640) identifies data by making pcc_nal_unit_type included in the NAL unit header an identifier of the NAL unit for the first encoding method (S4726). Then, the first decoding unit (4640) decodes PCC data using the decoding process of the first encoding method (S4727).

[0154] FIG. 25 is a flowchart illustrating processing by the second demultiplexer (4740) and the second decoding unit (4660). First, the second demultiplexer (4740) interprets ftyp included in the ISOBMFF file (S4731). If the codec indicated in ftyp is the second encoding method (pcc2) (the second encoding method in S4732), the second demultiplexer (4740) determines that the data included in the payload of the NAL unit is data encoded by the second encoding method (S4733). Also, the second demultiplexer (4740) transmits the result of the determination to the second decoding unit (4660).

[0155] The second decoding unit (4660) identifies data by making pcc_nal_unit_type included in the NAL unit header an identifier for the NAL unit for the second encoding method (S4734). Then, the second decoding unit (4660) decodes PCC data using the decoding process of the second encoding method (S4735).

[0156] Meanwhile, if the codec indicated in ftyp is the first encoding method (pcc1) (the first encoding method in S4732), the second demultiplexer (4740) determines that the data included in the payload of the NAL unit is data encoded by the first encoding method (S4736). Also, the second demultiplexer (4740) transmits the result of the determination to the second decoding unit (4660). The second decoding unit (4660) does not process the NAL unit (S4737).

[0157] In this way, for example, in the first demultiplexer (4720) or the second demultiplexer (4740), the codec type of the NAL unit can be identified in a rapid step by identifying the codec type. Additionally, the desired NAL unit can be input into the first decoding unit (4640) or the second decoding unit (4660), and unnecessary NAL units can be removed. In this case, the processing of interpreting the codec identification information in the first decoding unit (4640) or the second decoding unit (4660) may become unnecessary. Furthermore, the processing of interpreting the codec identification information by referring to the NAL unit type again in the first decoding unit (4640) or the second decoding unit (4660) may be performed.

[0158] Additionally, if pcc_nal_unit_type is deleted from the NAL unit header in the first multiplexer (4710) or the second multiplexer (4730), the first demultiplexer (4720) or the second demultiplexer (4740) may assign pcc_nal_unit_type to the NAL unit and then output it to the first decoding unit (4640) or the second decoding unit (4660).

[0159] (Embodiment 3)

[0160] In this embodiment, a multiplexer and a demultiplexer corresponding to the encoding unit (4670) and the decoding unit (4680) corresponding to the plurality of codecs described in Embodiment 1 are described. FIG. 26 is a diagram showing the configuration of the encoding unit (4670) and the third multiplexer (4750) according to this embodiment.

[0161] The encoding unit (4670) encodes point cloud data using either or both of the first encoding method and the second encoding method. The encoding unit (4670) may switch the encoding method (first encoding method and second encoding method) on a point cloud data unit or a frame unit. Additionally, the encoding unit (4670) may switch the encoding method on a coding-capable unit.

[0162] The encoding unit (4670) generates encoding data (encoding stream) containing identification information of the PCC codec.

[0163] The third multiplexer (4750) is equipped with a file conversion unit (4751). The file conversion unit (4751) converts the NAL unit output from the encoding unit (4670) into a PCC data file. The file conversion unit (4751) interprets the codec identification information included in the NAL unit header and determines whether the PCC encoded data is data encoded by the first encoding method, data encoded by the second encoding method, or data encoded by both methods. The file conversion unit (4751) writes a brand name that can identify the codec in ftyp. For example, if it indicates that it is encoded by both methods, pcc3 is written in ftyp.

[0164] Additionally, if the encoding unit (4670) includes identification information of a PCC codec in addition to the NAL unit, the file conversion unit (4751) may determine the PCC codec (encoding method) using said identification information.

[0165] FIG. 27 is a diagram showing the configuration of the third demultiplexer (4760) and the decoding unit (4680) according to the present embodiment.

[0166] The third demultiplexer (4760) is provided with a file inverse conversion unit (4761). The file inverse conversion unit (4761) interprets ftyp included in the file and determines whether the PCC encoded data is data encoded by the first encoding method, data encoded by the second encoding method, or data encoded by both methods.

[0167] When PCC encoded data is encoded using one of the encoding methods, data is input to the corresponding decoding unit among the first decoding unit (4640) and the second decoding unit (4660), and data is not input to the other decoding unit. When PCC encoded data is encoded using both encoding methods, data is input to the decoding unit (4680) corresponding to both methods.

[0168] The decoding unit (4680) decodes the PCC encoded data using either one or both of the first encoding method and the second encoding method.

[0169] FIG. 28 is a flowchart showing processing by the third multiplexer (4750) according to the present embodiment.

[0170] First, the third multiplexer (4750) determines whether the codec being used is the first encoding method, the second encoding method, or both the first encoding method and the second encoding method by interpreting the pcc_codec_type included in the NAL unit header (S4741).

[0171] When the second encoding method is being used (Yes in S4742, and also the second encoding method in S4743), the third multiplexer (4750) writes pcc2 in ftyp (S4744). That is, the third multiplexer (4750) writes information in ftyp indicating that data encoded by the second encoding method is stored in a file.

[0172] Next, the third multiplexer (4750) interprets the pcc_nal_unit_type included in the NAL unit header and stores the data in a box (moov or mdat, etc.) in a predetermined way according to the data type indicated in the pcc_unit_type (S4745). Then, the third multiplexer (4750) creates an ISOBMFF file including the ftyp and the box (S4746).

[0173] Meanwhile, if the first encoding method is being used (Yes in S4742, and also the first encoding method in S4743), the third multiplexer (4750) writes pcc1 in ftyp (S4747). That is, the third multiplexer (4750) writes information in ftyp indicating that data encoded by the first encoding method is stored in a file.

[0174] Next, the third multiplexer (4750) interprets the pcc_nal_unit_type included in the NAL unit header and stores the data in a box (moov or mdat, etc.) in a predetermined way according to the data type indicated in the pcc_unit_type (S4748). Then, the third multiplexer (4750) creates an ISOBMFF file including the ftyp and the box (S4746).

[0175] Meanwhile, when both the first encoding method and the second encoding method are used (No in S4742), the third multiplexer (4750) writes pcc3 in ftyp (S4749). That is, the third multiplexer (4750) writes information in ftyp indicating that data encoded by both encoding methods is stored in a file.

[0176] Next, the third multiplexer (4750) interprets the pcc_nal_unit_type included in the NAL unit header and stores the data in a box (moov or mdat, etc.) in a predetermined way according to the data type indicated in the pcc_unit_type (S4750). Then, the third multiplexer (4750) creates an ISOBMFF file including the ftyp and the box (S4746).

[0177] FIG. 29 is a flowchart illustrating processing by the third demultiplexer (4760) and the decoder (4680). First, the third demultiplexer (4760) interprets the ftyp included in the ISOBMFF file (S4761). If the codec indicated in the ftyp is the second encoding method (pcc2) (Yes in S4762, and the second encoding method in S4763), the third demultiplexer (4760) determines that the data included in the payload of the NAL unit is data encoded by the second encoding method (S4764). Also, the third demultiplexer (4760) transmits the result of the determination to the decoder (4680).

[0178] The decoding unit (4680) identifies the data by determining that the pcc_nal_unit_type included in the NAL unit header is an identifier of the NAL unit for the second encoding method (S4765). Then, the decoding unit (4680) decodes the PCC data using the decoding process of the second encoding method (S4766).

[0179] Meanwhile, if the codec indicated in ftyp is the first encoding method (pcc1) (Yes in S4762, and also the first encoding method in S4763), the third demultiplexer (4760) determines that the data included in the payload of the NAL unit is data encoded by the first encoding method (S4767). Also, the third demultiplexer (4760) transmits the result of the determination to the decoder (4680).

[0180] The decoding unit (4680) identifies the data by determining that the pcc_nal_unit_type included in the NAL unit header is an identifier of the NAL unit for the first encoding method (S4768). Then, the decoding unit (4680) decodes the PCC data using the decoding process of the first encoding method (S4769).

[0181] Meanwhile, if it is indicated that both encoding methods are being used in ftyp (pcc3) (No in S4762), the third demultiplexer (4760) determines that the data included in the payload of the NAL unit is data encoded using both the first encoding method and the second encoding method (S4770). Also, the third demultiplexer (4760) transmits the result of the determination to the decoding unit (4680).

[0182] The decoding unit (4680) identifies data by determining that pcc_nal_unit_type included in the NAL unit header is an identifier for the NAL unit for the codec listed in pcc_codec_type (S4771). Then, the decoding unit (4680) decodes PCC data using decoding processing of both encoding methods (S4772). That is, the decoding unit (4680) decodes data encoded by the first encoding method using decoding processing of the first encoding method, and decodes data encoded by the second encoding method using decoding processing of the second encoding method.

[0183] Variations of the present embodiment are described below. As for the types of brands shown in ftyp, the following types may be represented as identification information. In addition, combinations of multiple types shown below may be represented as identification information.

[0184] The identification information may indicate whether the object of the original data prior to PCC encoding is a point cloud with a limited area or a large point cloud without a limited area, such as map information.

[0185] The identification information may indicate whether the original data prior to PCC encoding is a static object or a dynamic object.

[0186] As described above, the identification information may indicate whether the PCC encoded data is data encoded by the first encoding method or data encoded by the second encoding method.

[0187] The identification information may represent the algorithm used in PCC encoding. Here, the algorithm is, for example, an encoding method usable in the first encoding method or the second encoding method.

[0188] The identification information may indicate the difference in the storage method for the ISOBMFF file of PCC encoded data. For example, the identification information may indicate whether the storage method used is a storage method for archiving or a storage method for real-time transmission, such as dynamic streaming.

[0189] In addition, in Embodiments 2 and 3, examples in which ISOBMFF is used as a file format were described, but other methods may be used. For example, when storing PCC encoded data in MPEG-2 TS Systems, MPEG-DASH, MMT, or RMP, the same method as in the present embodiment may be used.

[0190] In addition, the above description shows an example of storing metadata, such as identification information, in ftyp, but such metadata may be stored in a place other than ftyp. For example, such metadata may be stored in moov.

[0191] As described above, the three-dimensional data storage device (or three-dimensional data multiplexing device, or three-dimensional data encoding device) performs the processing shown in FIG. 30.

[0192] First, a three-dimensional data storage device (e.g., including a first multiplexer (4710), a second multiplexer (4730), or a third multiplexer (4750)) acquires one or more units (e.g., NAL units) in which a coded stream of point cloud data is encoded is stored (S4781). Next, the three-dimensional data storage device stores one or more units in a file (e.g., a file of ISOBMFF) (S4782). In addition, in the storage (S4782), the three-dimensional data storage device stores information (e.g., pcc1, pcc2, or pcc3) indicating that the data stored in the file is data encoded with point cloud data in the control information (e.g., ftyp) of the file.

[0193] According to this, a device processing a file generated by the said three-dimensional data storage device can determine early whether the data stored in the said file is encoded data of point cloud data by referring to the file's control information. Accordingly, it is possible to reduce the throughput of the said device or accelerate the processing.

[0194] For example, the above information also indicates the encoding method used for encoding the point cloud data among the first encoding method and the second encoding method. Additionally, the fact that the data stored in the file is data encoded from the point cloud data, and the encoding method used for encoding the point cloud data among the first encoding method and the second encoding method, may be represented as a single piece of information or as different pieces of information.

[0195] According to this, a device processing a file generated by the said three-dimensional data storage device can determine early on the codec used for the data stored in the said file by referring to the file's control information. Accordingly, it is possible to reduce the throughput of the said device or accelerate the processing.

[0196] For example, the first encoding method is a method (GPCC) that encodes position information representing the location of point cloud data as an N-ary tree (N is an integer greater than or equal to 2) and encodes attribute information using the position information, and the second encoding method is a method (VPCC) that generates a two-dimensional image from point cloud data and encodes the two-dimensional image using an image encoding method.

[0197] For example, the above file conforms to ISOBMFF (ISO based media file format).

[0198] For example, a three-dimensional data storage device is equipped with a processor and memory, and the processor uses the memory to perform the above processing.

[0199] In addition, as described above, the three-dimensional data acquisition device (or three-dimensional data demultiplexing device or three-dimensional data decoding device) performs the processing shown in FIG. 31.

[0200] A three-dimensional data acquisition device (including, for example, a first demultiplexer (4720), a second demultiplexer (4740), or a third demultiplexer (4760)) acquires a file (for example, an ISOBMFF file) in which one or more units (for example, NAL units) are stored, in which a encoded stream in which point cloud data is encoded is stored (S4791). Next, the three-dimensional data acquisition device acquires one or more units from the file (S4792). In addition, the control information of the file (for example, ftyp) includes information (for example, pcc1, pcc2, or pcc3) indicating that the data stored in the file is data in which point cloud data is encoded.

[0201] For example, the three-dimensional data acquisition device determines whether the data stored in the file is data encoded with point cloud data by referring to the above information. Also, if the three-dimensional data acquisition device determines that the data stored in the file is data encoded with point cloud data, it generates point cloud data by decoding the data encoded with point cloud data included in one or more units. Or, if the three-dimensional data acquisition device determines that the data stored in the file is data encoded with point cloud data, it outputs (notifies) information indicating that the data included in one or more units is data encoded with point cloud data to a subsequent processing unit (for example, a first decoding unit (4640), a second decoding unit (4660), or a decoding unit (4680)).

[0202] According to this, the three-dimensional data acquisition device can determine early whether the data stored in the file is encoded data of point cloud data by referring to the control information of the file. Accordingly, it is possible to reduce the throughput of the three-dimensional data acquisition device or the downstream device or to accelerate the processing.

[0203] For example, the above information also indicates the encoding method used for encoding among the first encoding method and the second encoding method. Additionally, the fact that the data stored in the file is data encoded from point cloud data, and the encoding method used for encoding the point cloud data among the first encoding method and the second encoding method, may be represented as a single piece of information or as different pieces of information.

[0204] According to this, the three-dimensional data acquisition device can determine early on the codec used for the data stored in the file by referring to the file control information. Accordingly, it is possible to reduce the throughput of the three-dimensional data acquisition device or the subsequent device, or to accelerate the processing.

[0205] For example, a three-dimensional data acquisition device acquires data encoded by one of the encoding methods from encoded point cloud data including data encoded by a first encoding method and data encoded by a second encoding method, based on the above information.

[0206] For example, the first encoding method is a method (GPCC) that encodes position information representing the location of point cloud data as an N-ary tree (N is an integer greater than or equal to 2) and encodes attribute information using the position information, and the second encoding method is a method (VPCC) that generates a two-dimensional image from point cloud data and encodes the two-dimensional image using an image encoding method.

[0207] For example, the above file conforms to ISOBMFF (ISO based media file format).

[0208] For example, a three-dimensional data acquisition device is equipped with a processor and memory, and the processor uses the memory to perform the above processing.

[0209] (Embodiment 4)

[0210] In this embodiment, the types of encoded data (position information (Geometry), attribute information (Attribute), additional information (Metadata)) generated by the first encoding unit (4630) or the second encoding unit (4650) described above, the method for generating additional information (metadata), and multiple processing in the multiplexing unit are explained. Additionally, additional information (metadata) may be referred to as a parameter set or control information.

[0211] In this embodiment, the dynamic object (time-varying three-dimensional point cloud data) described in FIG. 4 is used as an example, but the same method may be used for static objects (three-dimensional point cloud data at any given time).

[0212] FIG. 32 is a diagram showing the configuration of an encoding unit (4801) and a multiplexing unit (4802) included in a three-dimensional data encoding device according to the present embodiment. The encoding unit (4801) corresponds, for example, to the first encoding unit (4630) or the second encoding unit (4650) described above. The multiplexing unit (4802) corresponds to the multiplexing unit (4634 or 4656) described above.

[0213] The encoding unit (4801) encodes point cloud data of multiple PCC (Point Cloud Compression) frames and generates multiple compressed data of multiple location information, attribute information and additional information.

[0214] The multiplexing unit (4802) converts data of multiple data types (location information, attribute information and additional information) into a data configuration that takes into account data access in a decoding device by converting the data into NAL units.

[0215] FIG. 33 is a diagram showing an example of the configuration of encoded data generated by the encoding unit (4801). The arrows in the diagram represent dependency relationships based on the decoding of the encoded data, and the root of the arrow depends on the data at the end of the arrow. That is, the decoding device decodes the data at the end of the arrow and uses the decoded data to decode the data at the root of the arrow. In other words, dependency means that the data of the dependent source is referenced (used) in the processing (encoding or decoding, etc.) of the dependent source's data.

[0216] First, the process of generating encoded data of location information is explained. The encoding unit (4801) generates encoded location data (Compressed Geometry Data) for each frame by encoding the location information of each frame. Also, the encoded location data is denoted as G(i). i represents the frame number or the time of the frame, etc.

[0217] Additionally, the encoding unit (4801) generates a set of position parameters (GPS(i)) corresponding to each frame. The set of position parameters includes parameters that can be used to decode encoded position data. Also, the encoded position data for each frame depends on the corresponding set of position parameters.

[0218] Additionally, encoded position data consisting of multiple frames is defined as a position sequence (Geometry Sequence). The encoding unit (4801) generates a position sequence parameter set (Geometry Sequence PS: also referred to as Position SPS) that stores parameters commonly used for decoding processing for multiple frames within the position sequence. The position sequence depends on the Position SPS.

[0219] Next, the generation process of encoded attribute data is described. The encoding unit (4801) generates compressed attribute data for each frame by encoding the attribute information of each frame. Also, the compressed attribute data is denoted as A(i). Also, FIG. 33 shows an example in which attribute X and attribute Y exist, the compressed attribute data of attribute X is denoted as AX(i), and the compressed attribute data of attribute Y is denoted as AY(i).

[0220] Additionally, the encoding unit (4801) generates an attribute parameter set (APS(i)) corresponding to each frame. Also, the attribute parameter set of attribute X is denoted as AXPS(i), and the attribute parameter set of attribute Y is denoted as AYPS(i). The attribute parameter set includes parameters that can be used for decoding encoding attribute information. The encoding attribute data depends on the corresponding attribute parameter set.

[0221] Additionally, encoding attribute data consisting of multiple frames is defined as an attribute sequence. The encoding unit (4801) generates an attribute sequence parameter set (Attribute Sequence PS: also referred to as Attribute SPS) that stores parameters commonly used for decoding processing for multiple frames within the attribute sequence. The attribute sequence depends on the Attribute SPS.

[0222] Also, in the first encoding method, the encoding attribute data depends on the encoding position data.

[0223] Also, FIG. 33 shows an example where there are two types of attribute information (attribute X and attribute Y). When there are two types of attribute information, for example, data and metadata are generated by two encoding units. Also, for example, an attribute sequence is defined for each type of attribute information, and an attribute SPS is generated for each type of attribute information.

[0224] Additionally, FIG. 33 illustrates an example where there is one type of location information and two types of attribute information, but it is not limited to this; the attribute information may be one type or three or more types. In this case, encoded data can be generated in the same way. Also, in the case of point cloud data that does not have attribute information, attribute information may not be present. In that case, the encoding unit (4801) does not need to generate a parameter set related to attribute information.

[0225] Next, the process of generating additional information (metadata) will be described. The encoding unit (4801) generates a PCC stream PS (PCC Stream PS: also referred to as stream PS), which is a set of parameters for the entire PCC stream. The encoding unit (4801) stores parameters in stream PS that can be commonly used for decoding processing for one or more position sequences and one or more attribute sequences. For example, stream PS includes identification information indicating the codec of the point cloud data and information indicating the algorithm used for encoding. The position sequence and attribute sequence depend on stream PS.

[0226] Next, the access unit and GOF will be described. In this embodiment, the concepts of an access unit (Access Unit: AU) and a GOF (Group of Frame) are newly introduced.

[0227] An access unit is a basic unit for accessing data during decoding and consists of one or more data and one or more metadata. For example, an access unit consists of location information at the same time and one or more attribute information. A GOF is a random access unit and consists of one or more access units.

[0228] The encoding unit (4801) generates an access unit header (AU Header) as identification information indicating the beginning of the access unit. The encoding unit (4801) stores parameters related to the access unit in the access unit header. For example, the access unit header includes the configuration or information of the encoded data included in the access unit. In addition, the access unit header includes parameters commonly used for the data included in the access unit, such as parameters related to the decoding of the encoded data.

[0229] Additionally, the encoding unit (4801) may generate an access unit delimiter that does not include parameters according to the access unit, instead of an access unit header. This access unit delimiter is used as identification information indicating the beginning of the access unit. The decoding device identifies the beginning of the access unit by detecting the access unit header or the access unit delimiter.

[0230] Next, the generation of identification information for the GOF head is described. The encoding unit (4801) generates a GOF header as identification information indicating the GOF head. The encoding unit (4801) stores parameters related to the GOF in the GOF header. For example, the GOF header includes the composition or information of the encoded data included in the GOF. In addition, the GOF header includes parameters commonly used in the data included in the GOF, such as parameters related to the decoding of the encoded data.

[0231] Additionally, the encoding unit (4801) may generate a GOF delimiter that does not contain parameters related to the GOF instead of a GOF header. This GOF delimiter is used as identification information indicating the beginning of the GOF. The decoding device identifies the beginning of the GOF by detecting the GOF header or the GOF delimiter.

[0232] In PCC encoded data, for example, an access unit is defined as a PCC frame unit. A decoder accesses a PCC frame based on the identification information at the beginning of the access unit.

[0233] Also, for example, GOF is defined as one random access unit. The decoder accesses the random access unit based on the identification information at the beginning of the GOF. For example, if PCC frames have no dependency relationship and can be decoded independently, the PCC frame may be defined as a random access unit.

[0234] In addition, two or more PCC frames may be allocated to one access unit, and multiple random access units may be allocated to one GOF.

[0235] Additionally, the encoding unit (4801) may define and generate parameter sets or metadata other than those mentioned above. For example, the encoding unit (4801) may generate SEI (Supplemental Enhancement Information) that stores parameters (optional parameters) that may not necessarily be used during decoding.

[0236] Next, the structure of the encoded data and the method of storing the encoded data in the NAL unit will be explained.

[0237] For example, a data format is defined for each type of encoded data. FIG. 34 is a diagram showing an example of encoded data and a NAL unit.

[0238] For example, as shown in FIG. 34, the encoded data includes a header and a payload. Additionally, the encoded data may include length information indicating the length (amount of data) of the encoded data, the header, or the payload. Also, the encoded data may not include a header.

[0239] A header contains identification information to identify data, for example. This identification information indicates, for example, the data type or frame number.

[0240] A header contains identification information indicating a reference relationship, for example. This identification information is stored in the header when there is a dependency relationship between data, for example, and is information for referencing a reference location from a reference source. For example, the header of a reference location contains identification information to identify the data in question. The header of a reference source contains identification information indicating the reference location.

[0241] In addition, if the reference site or source can be identified or derived from other information, identification information for specifying data or identification information indicating a reference relationship may be omitted.

[0242] The multiplexer (4802) stores the encoded data in the payload of the NAL unit. The NAL unit header includes pcc_nal_unit_type, which is identification information of the encoded data. FIG. 35 is a diagram showing an example of the semantics of pcc_nal_unit_type.

[0243] As shown in FIG. 35, when pcc_codec_type is codec 1 (Codec1: first encoding method), values ​​0 to 10 of pcc_nal_unit_type are assigned to the encoding position data (Geometry), encoding attribute X data (AttributeX), encoding attribute Y data (AttributeY), position PS (Geom. PS), attribute XPS (AttrX. PS), attribute YPS (AttrX. PS), position SPS (Geometry Sequence PS), attribute XSPS (AttributeX Sequence PS), attribute YSPS (AttributeY Sequence PS), AU header (AU Header), and GOF header (GOF Header) in codec 1. Also, values ​​11 and above are assigned as reserves for codec 1.

[0244] When pcc_codec_type is Codec 2 (Codec2: second encoding method), values ​​of pcc_nal_unit_type 0 to 2 are assigned to the codec's data A (DataA), metadata A (MetaDataA), and metadata B (MetaDataB). Also, values ​​3 and above are assigned as reserves for Codec 2.

[0245] Next, the data transmission order will be explained. Below, the constraints on the transmission order of the NAL unit will be explained.

[0246] The multiplexer (4802) groups NAL units into GOF or AU units and transmits them. The multiplexer (4802) places a GOF header at the beginning of the GOF and places an AU header at the beginning of the AU.

[0247] Even if data is lost due to packet loss, etc., the multiplexer (4802) may place a sequence parameter set (SPS) for each AU so that the decoding device can decode from the next AU.

[0248] If there is a dependency relationship between the encoded data and the decoding, the decoding device decodes the data of the reference source after decoding the data of the reference location. In order to enable the decoding device to decode in the received order without rearranging the data, the multiplexer (4802) first transmits the data of the reference location.

[0249] FIG. 36 is a diagram showing an example of the transmission sequence of a NAL unit. FIG. 36 shows three examples of location information priority, parameter priority, and data integration.

[0250] The location-first transmission order is an example where location information and attribute information are bundled and transmitted separately. In this transmission order, the transmission of location information is completed faster than the transmission of attribute information.

[0251] For example, by using this transmission sequence, a decoder that does not decode attribute information may be able to form a time that is not processed by ignoring the decoding of attribute information. Also, for example, in the case of a decoder that wants to decode position information quickly, it may be possible to decode the position information more quickly by obtaining the encoded data of the position information quickly.

[0252] In addition, in Fig. 36, the attribute XSPS and attribute YSPS are combined and listed as the attribute SPS, but the attribute XSPS and attribute YSPS may be placed separately.

[0253] In a parameter set priority transmission order, the parameter set is transmitted first, and the data is transmitted next.

[0254] According to the constraints on the transmission order of NAL units as described above, the multiplexer (4802) may transmit NAL units in any order. For example, sequence identification information may be defined, and the multiplexer (4802) may have the function of transmitting NAL units in the order of multiple patterns. For example, the sequence identification information of NAL units is stored in stream PS.

[0255] The three-dimensional data decoding device may perform decoding based on sequence identification information. A desired transmission sequence is indicated from the three-dimensional data decoding device to the three-dimensional data encoding device, and the three-dimensional data encoding device (multiplexer (4802)) may control the transmission sequence according to the indicated transmission sequence.

[0256] Additionally, the multiplexer (4802) may generate encoded data that merges multiple functions, provided that the range is within the constraints of the transmission order, such as the transmission order of data integration. For example, as shown in FIG. 36, the GOF header and the AU header may be integrated, or the AXPS and AYPS may be integrated. In this case, an identifier indicating that the data has multiple functions is defined in pcc_nal_unit_type.

[0257] Variations of the present embodiment are described below. The PS has levels, such as frame-level PS, sequence-level PS, and PCC sequence-level PS. If the PCC sequence level is set as the upper level and the frame level as the lower level, the following method may be used for storing parameters.

[0258] The default PS value is displayed in the higher-level PS. Also, if the value of the lower-level PS is different from the value of the higher-level PS, the PS value is displayed in the lower-level PS. Alternatively, the PS value is not listed in the higher-level PS, but is listed in the lower-level PS. Alternatively, information on whether to display the PS value in the lower-level PS, the higher-level PS, or both is displayed in either the lower-level PS or the higher-level PS, or both. Alternatively, the lower-level PS may be merged into the higher-level PS. Alternatively, if the lower-level PS and the higher-level PS overlap, the multiplexer (4802) may omit the transmission of either one.

[0259] Additionally, the encoding unit (4801) or the multiplexing unit (4802) may divide the data into slices or tiles, and transmit the divided data. The divided data includes information for identifying the divided data, and parameters used for decoding the divided data are included in the parameter set. In this case, pcc_nal_unit_type is defined with an identifier indicating that it is data that stores data or parameters according to the tile or slice.

[0260] (Embodiment 5)

[0261] Below, a method for partitioning point cloud data is described. Figure 37 is a diagram showing an example of slice and tile partitioning.

[0262] First, the method of slice partitioning is explained. A three-dimensional data encoding device divides three-dimensional point cloud data into arbitrary point clouds in slice units. In slice partitioning, the three-dimensional data encoding device does not separate the position information and attribute information constituting the point, but rather divides the position information and attribute information together. That is, the three-dimensional data encoding device performs slice partitioning such that the position information and attribute information for any point belong to the same slice. Furthermore, according to these, the number of divisions and the division method may be any method. Also, the minimum unit of division is a point. For example, the number of divisions for position information and attribute information is the same. For example, a three-dimensional point corresponding to position information after slice partitioning and a three-dimensional point corresponding to attribute information are included in the same slice.

[0263] Additionally, the three-dimensional data encoding device generates slice supplementary information, which is additional information related to the number of divisions and the division method, when dividing into slices. The slice supplementary information is identical in position information and attribute information. For example, the slice supplementary information includes information indicating the reference coordinate position, size, or side length of the bounding box after division. Furthermore, the slice supplementary information includes information indicating the number of divisions and the division type, etc.

[0264] Next, the method of tile division is explained. The three-dimensional data encoding device divides the slice-divided data into slice position information (G slice) and slice attribute information (A slice), and divides the slice position information and slice attribute information into tile units.

[0265] In addition, Figure 37 shows an example of partitioning into an octal tree structure, and the number of partitions and the partitioning method can be any method.

[0266] In addition, the three-dimensional data encoding device may divide position information and attribute information using different division methods or using the same division method. In addition, the three-dimensional data encoding device may divide multiple slices into tiles using different division methods or using the same division method.

[0267] Additionally, the three-dimensional data encoding device generates additional tile information related to the number of divisions and the division method when dividing tiles. The additional tile information (position tile additional information and attribute tile additional information) is independent of the position information and attribute information. For example, the additional tile information includes information indicating the reference coordinate position, size, or side length of the bounding box after division. Furthermore, the additional tile information includes information indicating the number of divisions and the division type, etc.

[0268] Next, examples of methods for dividing point cloud data into slices or tiles are described. As a method for slicing or tiling, the three-dimensional data encoding device may use a predetermined method, or adaptively switch the method used depending on the point cloud data.

[0269] When performing slice division, the three-dimensional data encoding device divides the three-dimensional space in bulk based on position information and attribute information. For example, the three-dimensional data encoding device determines the shape of an object and divides the three-dimensional space into slices according to the shape of the object. For example, the three-dimensional data encoding device extracts objects such as trees or buildings and performs division on an object-by-object basis. For example, the three-dimensional data encoding device performs slice division such that one or more objects are included in a single slice. Alternatively, the three-dimensional data encoding device divides a single object into multiple slices.

[0270] In this case, the encoding device may change the encoding method for each slice, for example. For example, the encoding device may use a high-quality compression method for a specific object or a specific part of an object. In this case, the encoding device may store information indicating the encoding method for each slice in additional information (metadata).

[0271] In addition, the three-dimensional data encoding device may perform slice division based on map information or location information so that each slice corresponds to a predetermined coordinate space.

[0272] When dividing into tiles, the three-dimensional data encoding device divides position information and attribute information independently. For example, the three-dimensional data encoding device divides a slice into tiles based on the amount of data or throughput. For example, the three-dimensional data encoding device determines whether the amount of data in a slice (e.g., the number of three-dimensional points included in the slice) is greater than a predetermined threshold. If the amount of data in a slice is greater than the threshold, the three-dimensional data encoding device divides the slice into tiles. If the amount of data in a slice is less than the threshold, the three-dimensional data encoding device does not divide the slice into tiles.

[0273] For example, a three-dimensional data encoding device divides a slice into tiles so that the throughput or processing time in the decoder is within a certain range (less than or equal to a predetermined value). As a result, the throughput per tile in the decoder becomes constant, and distributed processing in the decoder becomes easier.

[0274] In addition, when the throughput of the position information and the attribute information differs, for example, when the throughput of the position information is greater than the throughput of the attribute information, the three-dimensional data encoding device makes the number of divisions of the position information greater than the number of divisions of the attribute information.

[0275] In addition, for example, depending on the content, if the decoder can quickly decode and display location information and then slowly decode and display attribute information, the three-dimensional data encoding device may have a greater number of divisions for location information than for attribute information. By doing so, the decoder can increase the number of parallels for location information, thereby speeding up the processing of location information compared to the processing of attribute information.

[0276] In addition, the decoding device does not necessarily need to process sliced ​​or tiled data in parallel, and may determine whether to process them in parallel based on the number or capability of the decoding processing units.

[0277] By dividing in the manner described above, adaptive encoding based on content or objects can be realized. Additionally, parallel processing during decoding can be achieved. This improves the flexibility of the point cloud encoding system or the point cloud decoding system.

[0278] FIG. 38 is a diagram illustrating an example of a pattern for slicing and tile division. In the diagram, DU stands for Data Unit and represents data for a tile or a slice. Additionally, each DU includes a Slice Index and a Tile Index. The numerical value in the upper right corner of the DU in the diagram represents the Slice Index, and the numerical value in the lower left corner of the DU represents the Tile Index.

[0279] In Pattern 1, regarding slice division, the number of divisions and the division method are the same for G slices and A slices. Regarding tile division, the number of divisions and the division method for G slices are different from the number of divisions and the division method for A slices. Also, the same number of divisions and the division method are used among multiple G slices. The same number of divisions and the division method are used among multiple A slices.

[0280] In Pattern 2, regarding slice division, the number of divisions and the division method are the same for G slices and A slices. Regarding tile division, the number of divisions and the division method for G slices are different from the number of divisions and the division method for A slices. Also, the number of divisions and the division method are different among multiple G slices. The number of divisions and the division method are different among multiple A slices.

[0281] (Embodiment 6)

[0282] The following describes an example of performing slice partitioning after tile partitioning. In autonomous applications, such as vehicle self-driving, point cloud data is required not for the entire area, but for the area surrounding the vehicle or the area in the vehicle's direction of travel. Here, tiles and slices can be used to selectively decode the original point cloud data. By partitioning three-dimensional point cloud data into tiles and further partitioning it into slices, improvements in encoding efficiency or parallel processing can be realized. When partitioning the data, additional information (metadata) is generated, and the generated additional information is sent to a multiplexer.

[0283] FIG. 39 is a block diagram showing the configuration of a first encoding unit (5010) included in a three-dimensional data encoding device according to the present embodiment. The first encoding unit (5010) generates encoded data (encoded stream) by encoding point cloud data using a first encoding method (GPCC (Geometry based PCC)). The first encoding unit (5010) includes a splitting unit (5011), a plurality of position information encoding units (5012), a plurality of attribute information encoding units (5013), an additional information encoding unit (5014), and a multiplexing unit (5015).

[0284] The dividing unit (5011) generates multiple divided data by dividing the point cloud data. Specifically, the dividing unit (5011) generates multiple divided data by dividing the space of the point cloud data into multiple sub-spaces. Here, a sub-space is either a tile or a slice, or a combination of a tile and a slice. More specifically, the point cloud data includes location information, attribute information, and additional information. The dividing unit (5011) divides the location information into multiple divided location information and divides the attribute information into multiple divided attribute information. In addition, the dividing unit (5011) generates additional information regarding the division.

[0285] For example, the dividing unit (5011) first divides the point group into tiles. Next, the dividing unit (5011) further divides the obtained tiles into slices.

[0286] A plurality of location information encoding units (5012) generate a plurality of encoded location information by encoding a plurality of divided location information. For example, the plurality of location information encoding units (5012) process the plurality of divided location information in parallel.

[0287] A plurality of attribute information encoding unit (5013) generates a plurality of encoded attribute information by encoding a plurality of divided attribute information. For example, the plurality of attribute information encoding unit (5013) processes a plurality of divided attribute information in parallel.

[0288] The additional information encoding unit (5014) generates encoded additional information by encoding additional information included in the point cloud data and additional information regarding data division generated at the division unit (5011).

[0289] The multiplexer (5015) generates encoded data (encoded stream) by multiplexing a plurality of encoding location information, a plurality of encoding attribute information, and encoding additional information, and transmits the generated encoded data. In addition, the encoding additional information is used during decoding.

[0290] Additionally, FIG. 39 illustrates an example in which the number of position information encoding units (5012) and attribute information encoding units (5013) are each 2, but the number of position information encoding units (5012) and attribute information encoding units (5013) may each be 1 or 3 or more. Also, multiple divided data may be processed in parallel within the same tip, such as multiple cores in a CPU, processed in parallel with cores of multiple tips, or processed in parallel with multiple cores of multiple tips.

[0291] Next, the decoding process will be described. FIG. 40 is a block diagram showing the configuration of the first decoding unit (5020). The first decoding unit (5020) restores point cloud data by decoding the encoded data (encoded stream) generated by encoding the point cloud data with the first encoding method (GPCC). This first decoding unit (5020) includes a demultiplexer (5021), a plurality of position information decoding units (5022), a plurality of attribute information decoding units (5023), an additional information decoding unit (5024), and a combination unit (5025).

[0292] The demultiplexer (5021) generates a plurality of encoding location information, a plurality of encoding attribute information, and encoding additional information by demultiplexing the encoding data (encoding stream).

[0293] A plurality of location information decoding units (5022) generate a plurality of divided location information by decoding a plurality of encoded location information. For example, the plurality of location information decoding units (5022) process a plurality of encoded location information in parallel.

[0294] A plurality of attribute information decoding unit (5023) generates a plurality of divided attribute information by decoding a plurality of encoded attribute information. For example, the plurality of attribute information decoding unit (5023) processes a plurality of encoded attribute information in parallel.

[0295] A plurality of additional information decoding units (5024) generate additional information by decoding the encoded additional information.

[0296] The combining unit (5025) generates location information by combining a plurality of segmented location information using additional information. The combining unit (5025) generates attribute information by combining a plurality of segmented attribute information using additional information. For example, the combining unit (5025) first generates point cloud data corresponding to a tile by combining decoded point cloud data for a slice using slice additional information. Next, the combining unit (5025) restores the original point cloud data by combining point cloud data corresponding to a tile using tile additional information.

[0297] Additionally, in FIG. 39, the number of location information decoding units (5022) and attribute information decoding units (5023) is shown as two, but the number of location information decoding units (5022) and attribute information decoding units (5023) may be one or three or more. Also, multiple divided data may be processed in parallel within the same tip, such as multiple cores in a CPU, processed in parallel with cores of multiple tips, or processed in parallel with multiple cores of multiple tips.

[0298] Next, we will explain the point cloud data partitioning method. In autonomous applications such as vehicle self-driving, point cloud data is required for the area around the vehicle or the region in the vehicle's direction of travel, rather than point cloud data for the entire area.

[0299] FIG. 41 is a drawing showing examples of tile shapes. As shown in FIG. 41, various shapes such as circles, rectangles, or ellipses may be used as tile shapes.

[0300] FIG. 42 is a diagram showing examples of tiles and slices. The configuration of the slices may differ between tiles. For example, the configuration of the tiles or slices may be optimized based on the amount of data. Alternatively, the configuration of the tiles or slices may be optimized based on the decoding speed.

[0301] In addition, tile division may be performed based on location information. In this case, attribute information is divided in the same way as the corresponding location information.

[0302] In addition, regarding slice division after tile division, position information and attribute information may be divided into slices by different methods. For example, the method of slice division for each tile may be selected according to the requirements of the application. Depending on the requirements of the application, different slice division methods or tile division methods may be used.

[0303] For example, the dividing unit (5011) divides the point cloud data into one or more tiles based on location information, such as map information, in a two-dimensional shape viewed from above. Then, the dividing unit (5011) divides each tile into one or more slices.

[0304] Additionally, the dividing section (5011) may divide the position information (Geometry) and attribute information (Attribute) into slices in the same way.

[0305] In addition, location information and attribute information may each be of one type or two or more types. Also, in the case of point cloud data that does not have attribute information, attribute information may be omitted.

[0306] FIG. 43 is a block diagram of a dividing unit (5011). The dividing unit (5011) includes a tile dividing unit (5031) (Tile Divider), a position information slice dividing unit (5032) (Geometry Slice Divider), and an attribute information slice dividing unit (5033) (Attribute Slice Divider).

[0307] The tile splitting unit (5031) generates multiple tile position information by dividing position information (Position(Geometry)) into tiles. Additionally, the tile splitting unit (5031) generates multiple tile attribute information by dividing attribute information (Attribute) into tiles. Additionally, the tile splitting unit (5031) outputs tile additional information (TileMetaData) including information regarding tile splitting and information generated in tile splitting.

[0308] The location information slice division unit (5032) generates multiple divided location information (multiple slice location information) by dividing multiple tile location information into slices. Additionally, the location information slice division unit (5032) outputs location slice additional information (Geometry Slice MetaData) including information regarding the slice division of location information and information generated in the slice division of location information.

[0309] The attribute information slice division unit (5033) generates multiple divided attribute information (multiple slice attribute information) by dividing multiple tile attribute information into slices. Additionally, the attribute information slice division unit (5033) outputs attribute slice additional information (Attribute Slice MetaData) including information regarding the slice division of attribute information and information generated in the slice division of attribute information.

[0310] Next, examples of tile shapes will be described. The entire three-dimensional map is divided into multiple tiles. The data of the multiple tiles is optionally transmitted to a three-dimensional data decoder. Alternatively, the data among the multiple tiles is transmitted to the three-dimensional data decoder in order, starting with the data of highest importance. Depending on the situation, the tile shapes may be selected from multiple shapes.

[0311] FIG. 44 is a diagram showing an example of a map viewed from above using point cloud data obtained by LiDAR. The example shown in FIG. 44 is point cloud data of a highway and includes a flyover section.

[0312] FIG. 45 is a diagram showing an example of dividing the point cloud data shown in FIG. 44 into square tiles. Such square division can be easily performed on a map server. Also, for normal roads, the height of the tiles is set low. In the case of grade-separated intersections, the height of the tiles is set higher than that of normal roads so that the tiles include the grade-separated intersections.

[0313] FIG. 46 is a diagram showing an example in which the point cloud data shown in FIG. 44 is divided into circular tiles. In this case, adjacent tiles may overlap when viewed from a plane. When the vehicle requires point cloud data of a surrounding area, the three-dimensional data encoding device transmits point cloud data of a cylindrical area (a circle when viewed from the top) around the vehicle to the vehicle.

[0314] Also, as in the example of Fig. 45, the height of the tile is set low for normal roads. In the grade-separated intersection, the height of the tile is set higher than that of normal roads so that the tile covers the grade-separated intersection.

[0315] A three-dimensional data encoding device may change the height of a tile according to, for example, the shape or height of a road or building. Additionally, a three-dimensional data encoding device may change the height of a tile according to location information or area information. Furthermore, a three-dimensional data encoding device may change the height of a tile for each tile. Alternatively, a three-dimensional data encoding device may change the height of a tile for each section containing multiple tiles. That is, a three-dimensional data encoding device may make the heights of multiple tiles within a section identical. Additionally, tiles of different heights may overlap when viewed from the top surface.

[0316] FIG. 47 is a drawing showing an example of tile division using tiles of various shapes, sizes, or heights. The shape of the tiles can be any shape, any size, or a combination thereof.

[0317] For example, in addition to the example of dividing into square tiles without overlap as described above and the example of dividing into circular tiles with overlap, the three-dimensional data encoding device may also perform division into square tiles with overlap. Also, the shape of the tiles does not have to be square or circular, and a polygon with three or more vertices may be used, or a shape without vertices may be used.

[0318] Also, the shape of the tile may be of two or more types, and tiles of different shapes may overlap. Also, there may be one or more types of tile shapes, and within the same shape being divided, shapes of different sizes may be combined, and they may overlap.

[0319] For example, in areas without objects such as roads, tiles larger than those in areas where objects exist are used. Additionally, the three-dimensional data encoding device may adaptively change the shape or size of the tiles depending on the object.

[0320] In addition, for example, since it is highly likely that a three-dimensional data encoding device will need to read tiles far ahead of the vehicle, which is the direction of travel, the tiles in the direction of travel can be set to a large size, and since it is unlikely that the vehicle will travel to the side, the tiles on the side can be set to a smaller size than the tiles in the direction of travel.

[0321] FIG. 48 is a diagram illustrating an example of tile data stored on a server. For example, point cloud data is pre-tile-divided and encoded, and the obtained encoded data is stored on a server. A user obtains the desired tile data from the server when needed. Alternatively, the server (three-dimensional data encoding device) may perform tile division and encoding to include the data desired by the user according to the user's instructions.

[0322] For example, if the moving speed of a moving object (vehicle) is fast, it can be considered that more extensive point cloud data is required. Accordingly, the server may determine the shape and size of the tiles based on the pre-estimated speed of the vehicle (e.g., the legal speed limit of the road, the speed of the vehicle that can be estimated from the width and shape of the road, or statistical speed, etc.) and perform tile division. Alternatively, as shown in FIG. 48, the server may pre-encode multiple tiles of shapes or sizes and store the obtained data. The moving object may acquire data of tiles of appropriate shapes and sizes according to the direction and speed of the moving object.

[0323] FIG. 49 is a diagram showing an example of a system for tile division. As shown in FIG. 49, the shape and area of ​​the tile may be determined based on the location of an antenna (base station) which is a communication means for transmitting point cloud data, or the communication area supported by the antenna. Alternatively, when point cloud data is generated by a sensor such as a camera, the shape and area of ​​the tile may be determined based on the location of the sensor or the target range (detection range) of the sensor.

[0324] One tile may be assigned to one antenna or sensor, or one tile may be assigned to multiple antennas or sensors. Multiple tiles may be assigned to one antenna or sensor. Antennaes or sensors may be fixed or movable.

[0325] For example, encoded data divided into tiles may be managed by a server connected to an antenna or sensor for the area assigned to the tile. The server may manage the encoded data of its own area and tile information of an adjacent area. Multiple encoded data of multiple tiles may be managed in a centralized management server (cloud) that manages multiple servers corresponding to each tile. Alternatively, the antenna or sensor may be directly connected to the centralized management server without installing a server corresponding to the tile.

[0326] In addition, the target range of the antenna or sensor may vary depending on the power of the radio waves, differences in the device, and installation conditions, so the shape and size of the tile may also be changed accordingly. Depending on the target range of the antenna or sensor, a slice may be assigned instead of a tile, or a PCC frame may be assigned.

[0327] Next, we will explain the method of dividing tiles into slices. Encoding efficiency can be improved by assigning similar objects to the same slice.

[0328] For example, a three-dimensional data encoding device may recognize objects (roads, buildings, trees, etc.) using features of point cloud data and perform slice partitioning by clustering point clouds for each object.

[0329] Alternatively, the three-dimensional data encoding device may perform slice partitioning by grouping objects having the same attributes and assigning slices to each group. Here, attributes refer to information regarding movement, for example, and grouping is performed by classifying objects into dynamic information such as pedestrians and vehicles, quasi-dynamic information such as accidents and traffic congestion, quasi-static information such as traffic regulations and road construction, and static information such as road surfaces and structures.

[0330] In addition, data may be duplicated across multiple slices. For example, when slicing for multiple object groups, any object may belong to one object group or to two or more object groups.

[0331] FIG. 50 is a drawing showing an example of this slice division. For example, in the example shown in FIG. 50, the tile is a rectangular shape. In addition, the tile may be cylindrical or have other shapes.

[0332] The point clouds contained in the tile are grouped into object groups, such as roads, buildings, and trees. Then, slicing is performed so that each object group is contained in a single slice. Subsequently, each slice is encoded individually.

[0333] Next, a method for encoding divided data will be described. A three-dimensional data encoding device (a first encoding unit (5010)) encodes each of the divided data. When encoding attribute information, the three-dimensional data encoding device generates dependency relationship information as additional information, which indicates which configuration information (location information, additional information, or other attribute information) was used for encoding. That is, the dependency relationship information represents, for example, the configuration information of a reference location (dependency location). In this case, the three-dimensional data encoding device generates dependency relationship information based on configuration information corresponding to the divided shape of the attribute information. Additionally, the three-dimensional data encoding device may generate dependency relationship information based on configuration information corresponding to multiple divided shapes.

[0334] Dependency relationship information may be generated by a three-dimensional data encoding device and transmitted to a three-dimensional data decoder. Alternatively, the three-dimensional data decoder may generate dependency relationship information, and the three-dimensional data encoding device may not transmit the dependency relationship information. Furthermore, the dependency relationships used by the three-dimensional data encoding device may be predetermined, and the three-dimensional data encoding device may not transmit the dependency relationship information.

[0335] FIG. 51 is a diagram illustrating an example of the dependency relationships of each data. The tip of the arrow in the diagram represents the dependency source, and the root of the arrow represents the source of dependency. The three-dimensional data decoding device decodes the data in the order from the dependency source to the source of dependency. Also, the data represented by solid lines in the diagram is the data that is actually transmitted, and the data represented by dotted lines is the data that is not transmitted.

[0336] Also, in the same drawing, G represents location information and A represents attribute information. G t1 Silver represents the location information of tile number 1, and G t2 represents the location information of tile number 2. Gt1s1 Silver represents the location information of tile number 1 and slice number 1, and G t1s2 represents the location information of tile number 1 and slice number 2, and G t2s1 Silver, tile number 2 also indicates the location information of slice number 1, and G t2s2 represents the location information of tile number 2 as well as slice number 2. Likewise, A t1 Silver represents the attribute information of tile number 1, and A t2 represents the attribute information of tile number 2. A t1s1 Silver represents the attribute information of tile number 1 and slice number 1, and A t1s2 represents the attribute information of tile number 1 and slice number 2, and A t2s1 Silver, tile number 2 also represents the attribute information of slice number 1, and A t2s2 represents the attribute information of tile number 2 and slice number 2.

[0337] Mtile represents tile additional information, MGslice represents position slice additional information, and MAslice represents attribute slice additional information. D t1s1 Attribute information A t1s1 Represents the dependency relationship information of, and D t2s1 Attribute information A t2s1 Represents the dependency relationship information.

[0338] In addition, depending on the application, different tile division or slice division structures may be used.

[0339] In addition, the three-dimensional data encoding device may rearrange the data in the decoding order so that there is no need to rearrange the data in the three-dimensional data decoding device. Furthermore, the data may be rearranged in the three-dimensional data decoding device, or the data may be rearranged in both the three-dimensional data encoding device and the three-dimensional data decoding device.

[0340] FIG. 52 is a diagram illustrating an example of the decoding order of data. In the example of FIG. 52, decoding is performed in order starting from the data on the left. When there is a dependency relationship between data, the three-dimensional data decoding device decodes the dependent data first. For example, the three-dimensional data encoding device rearranges the data in advance and transmits it in this order. Additionally, any order is acceptable as long as the dependent data comes first. Furthermore, the three-dimensional data encoding device may transmit additional information and dependency relationship information before the data.

[0341] Additionally, the three-dimensional data decoding device may optionally decode tiles based on a request from an application and information obtained from the NAL unit header. FIG. 53 is a diagram showing an example of encoded tile data. For example, the decoding order of the tiles is arbitrary. That is, there may be no dependency relationship between the tiles.

[0342] Next, the configuration of the combination unit (5025) included in the first decoding unit (5020) will be explained. FIG. 54 is a block diagram showing the configuration of the combination unit (5025). The combination unit (5025) includes a position information slice combination unit (5041) (Geometry Slice Combiner), an attribute information slice combination unit (5042) (Attribute Slice Combiner), and a tile combination unit (Tile Combiner).

[0343] The location information slice combining unit (5041) generates multiple tile location information by combining multiple divided location information using location slice additional information. The attribute information slice combining unit (5042) generates multiple tile attribute information by combining multiple divided attribute information using attribute slice additional information.

[0344] The tile combining unit (5043) generates location information by combining multiple tile location information using tile additional information. Additionally, the tile combining unit (5043) generates attribute information by combining multiple tile attribute information using tile additional information.

[0345] In addition, the number of slices or tiles to be divided is 1 or more. That is, the division of slices or tiles does not need to be performed.

[0346] Next, the composition of slice-divided or tile-divided encoded data and the method of storing the encoded data in a NAL unit (multiplexing method) will be described. FIG. 55 is a diagram showing the composition of encoded data and the method of storing the encoded data in a NAL unit.

[0347] Encoded data (segmentation location information and segmentation attribute information) is stored in the payload of the NAL unit.

[0348] Encoded data includes a header and a payload. The header includes identification information for identifying the data included in the payload. This identification information includes, for example, the type of slice division or tile division (slice_type, tile_type), index information for identifying a slice or tile (slice_idx, tile_idx), location information of the data (slice or tile), or the address of the data. Index information for identifying a slice is also referred to as SliceIndex. Index information for identifying a tile is also referred to as TileIndex. Furthermore, the type of division is, for example, a method based on object shape as described above, a method based on map information or location information, or a method based on data volume or throughput.

[0349] Additionally, the header of the encoded data includes identification information indicating dependency relationships. That is, if there is a dependency relationship between data, the header includes identification information for referencing the dependency from the dependency source. For example, the header of the data of the dependency includes identification information for specifying the data. The header of the data of the dependency includes identification information indicating the dependency. Furthermore, if the identification information for specifying data, additional information regarding slice partitioning or tile partitioning, and identification information indicating dependency relationships can be identified or derived from other information, these details may be omitted.

[0350] Next, the flow of encoding and decoding processing of point cloud data according to the present embodiment will be described. FIG. 56 is a flowchart of encoding processing of point cloud data according to the present embodiment.

[0351] First, the three-dimensional data encoding device determines the division method to be used (S5011). This division method includes whether to perform tile division or slice division. In addition, the division method may include the number of divisions when performing tile division or slice division, and the type of division. The type of division is a method based on object shape as described above, a method based on map information or location information, or a method based on data amount or throughput. In addition, the division method may be predetermined.

[0352] When tile division is performed (Yes in S5012), the three-dimensional data encoding device generates multiple tile location information and multiple tile attribute information by dividing the location information and attribute information in batches (S5013). Additionally, the three-dimensional data encoding device generates additional tile information regarding tile division. Furthermore, the three-dimensional data encoding device may divide the location information and attribute information independently.

[0353] When slice division is performed (Yes in S5014), the three-dimensional data encoding device generates multiple divided location information and multiple divided attribute information by independently dividing multiple tile location information and multiple tile attribute information (or location information and attribute information) (S5015). Additionally, the three-dimensional data encoding device generates additional location slice information and additional attribute slice information according to the slice division. Furthermore, the three-dimensional data encoding device may divide the tile location information and tile attribute information in a batch.

[0354] Next, the three-dimensional data encoding device generates multiple encoded location information and multiple encoded attribute information by encoding each of the multiple segmented location information and multiple segmented attribute information (S5016). In addition, the three-dimensional data encoding device generates dependency relationship information.

[0355] Next, the three-dimensional data encoding device generates encoded data (encoded stream) by unitizing (multiplexing) a plurality of encoding position information, a plurality of encoding attribute information and additional information into NAL units (S5017). In addition, the three-dimensional data encoding device transmits the generated encoded data.

[0356] FIG. 57 is a flowchart of the decoding process of point cloud data according to the present embodiment. First, the three-dimensional data decoding device determines a division method by interpreting additional information regarding a division method (tile additional information, position slice additional information and attribute slice additional information) included in the encoded data (encoded stream) (S5021). This division method includes whether to perform tile division and whether to perform slice division. In addition, the division method may include the number of divisions when performing tile division or slice division, and the type of division, etc.

[0357] Next, the three-dimensional data decoding device generates segmentation position information and segmentation attribute information by decoding a plurality of encoding position information and a plurality of encoding attribute information included in the encoded data using dependency relationship information included in the encoded data (S5022).

[0358] When it is indicated that slice division is performed by additional information (Yes in S5023), the three-dimensional data decoding device generates multiple tile location information and multiple tile attribute information by combining multiple division location information and multiple division attribute information in each way based on the location slice additional information and the attribute slice additional information (S5024). Additionally, the three-dimensional data decoding device may combine multiple division location information and multiple division attribute information in the same way.

[0359] When it is indicated that tile division is performed by additional information (Yes in S5025), the three-dimensional data decoding device generates position information and attribute information by combining a plurality of tile position information and a plurality of tile attribute information (a plurality of division position information and a plurality of division attribute information) in the same way based on the tile additional information (S5026). Additionally, the three-dimensional data decoding device may combine the plurality of tile position information and the plurality of tile attribute information in different ways.

[0360] Next, tile supplementary information is described. The three-dimensional data encoding device generates tile supplementary information, which is metadata regarding the tile division method, and transmits the generated tile supplementary information to the three-dimensional data decoding device.

[0361] FIG. 58 is a diagram showing an example of the syntax of tile additional information (TileMetaData). As shown in FIG. 58, for example, the tile additional information includes type of divide information, topview shape information, a tile overlap flag, type of overlap information, height information, tile number, and tile position information (global_position, relative_position).

[0362] The type_of_divide information indicates the method of dividing the tile. For example, the type_of_divide information indicates whether the method of dividing the tile is based on map information, that is, whether it is based on the top view (top_view) or other.

[0363] Shape information (topview_shape) is included in the tile supplementary information, for example, when the tile division method is based on the top view. Shape information represents the shape of the tile as seen from the top view. For example, this shape includes squares and circles. Additionally, this shape may include ellipses, rectangles, or polygons other than squares, and may include other shapes. Furthermore, shape information is not limited to the shape of the tile as seen from the top view and may represent the three-dimensional shape of the tile (e.g., cubes and cylinders).

[0364] The tile_overlap_flag indicates whether tiles overlap. For example, the tile_overlap_flag is included in the tile supplementary information when the tile partitioning method is based on the top view. In this case, the tile_overlap_flag indicates whether tiles overlap when viewed from the top view. Additionally, the tile_overlap_flag may indicate whether tiles overlap in three-dimensional space.

[0365] Overlap information (type_of_overlap) is included in the tile supplementary information, for example, when tiles overlap. Overlap information indicates the method of tile overlap, etc. For example, overlap information indicates the size of the overlapping area, etc.

[0366] The height information (tile_height) indicates the height of the tile. Additionally, the height information may include information indicating the shape of the tile. For example, if the shape of the tile when viewed from the top surface is a rectangle, the information may indicate the lengths of the sides (height and width) of the rectangle. Furthermore, if the shape of the tile when viewed from the top surface is a circle, the information may indicate the diameter or radius of the circle.

[0367] Additionally, the height information may represent the height of each individual tile or a common height across multiple tiles. Furthermore, multiple height types, such as roads and overpasses, may be pre-set, and the height of each height type and the height type of each tile may be represented by the height information. Alternatively, the height of each height type may be pre-defined, and the height type of each tile may be represented by the height information. In other words, the height of each height type does not need to be represented by the height information.

[0368] Tile_number indicates the number of tiles. Additionally, tile supplementary information may include information indicating the spacing between tiles.

[0369] Tile position information (global_position, relative_position) is information used to specify the location of each tile. For example, tile position information represents the absolute or relative coordinates of each tile.

[0370] In addition, some or all of the above information may be installed for each tile, or for each of multiple tiles (e.g., for each frame or for each of multiple frames).

[0371] The three-dimensional data encoding device may include tile supplementary information in SEI (Supplemental Enhancement Information) and transmit it. Alternatively, the three-dimensional data encoding device may store the tile supplementary information in an existing parameter set (PPS, GPS, or APS, etc.) and transmit it.

[0372] For example, if tile additional information changes per frame, the tile additional information may be stored in the parameter set for each frame (e.g., GPS or APS). If the tile additional information does not change within a sequence, the tile additional information may be stored in the parameter set for each sequence (e.g., Position SPS or Attribute SPS). Furthermore, if the same tile splitting information is used in both the position information and the attribute information, the tile additional information may be stored in the parameter set of the PCC stream (Stream PS).

[0373] Additionally, the tile additional information may be stored in any one of the above parameter sets or in multiple parameter sets. Additionally, the tile additional information may be stored in the header of the encoded data. Additionally, the tile additional information may be stored in the header of the NAL unit.

[0374] In addition, all or part of the tile additional information may be stored in either the header of the segmentation location information or the header of the segmentation attribute information, and not in the other. For example, if the same tile additional information is used in the location information and the attribute information, the tile additional information may be included in the header of either the location information or the attribute information. For example, if the attribute information depends on the location information, the location information is processed first. Therefore, the tile additional information may be included in the header of the location information, and the tile additional information may not be included in the header of the attribute information. In this case, the three-dimensional data decoding device determines, for example, that the attribute information of the dependent source belongs to the same tile as the tile of the dependent location information.

[0375] The three-dimensional data decoding device reconstructs tile-divided point cloud data based on tile additional information. If there is duplicate point cloud data, the three-dimensional data decoding device identifies multiple duplicate point cloud data and selects one of them or merges multiple point cloud data.

[0376] Additionally, the three-dimensional data decoding device may perform decoding using tile supplementary information. For example, if multiple tiles overlap, the three-dimensional data decoding device may perform decoding for each tile and generate point cloud data by performing processing (e.g., smoothing or filtering) using the multiple decoded data. This may enable high-precision decoding.

[0377] FIG. 59 is a diagram showing an example of a configuration of a system including a three-dimensional data encoding device and a three-dimensional data decoding device. A tile splitting unit (5051) splits point cloud data including location information and attribute information into a first tile and a second tile. Also, the tile splitting unit (5051) sends tile additional information resulting from the tile splitting to a decoding unit (5053) and a tile combining unit (5054).

[0378] The encoding unit (5052) generates encoded data by encoding the first tile and the second tile.

[0379] The decoding unit (5053) restores the first tile and the second tile by decoding the encoded data generated by the encoding unit (5052). The tile combining unit (5054) restores the point cloud data (location information and attribute information) by combining the first tile and the second tile using tile additional information.

[0380] Next, slice supplementary information is described. The three-dimensional data encoding device generates slice supplementary information, which is metadata regarding the slice division method, and transmits the generated slice supplementary information to the three-dimensional data decoding device.

[0381] FIG. 60 is a diagram showing an example of the syntax of slice additional information (SliceMetaData). As shown in FIG. 60, for example, the slice additional information includes type of divide information, a duplicate flag (slice_overlap_flag), duplicate information (type_of_overlap), number of slices (slice_number), slice position information (global_position, relative_position), and slice size information (slice_bounding_box_size).

[0382] The type of divide information indicates the method of dividing the slice. For example, the type of divide information indicates whether the method of dividing the slice is an object based on the object information shown in FIG. 50. Additionally, the slice additional information may include information indicating the method of dividing the object. For example, this information indicates whether one object is divided into multiple slices or assigned to one slice. Also, this information may indicate the number of divisions when one object is divided into multiple slices.

[0383] The overlap flag (slice_overlap_flag) indicates whether slices overlap. The overlap information (type_of_overlap) is included in the slice supplementary information, for example, if slices overlap. The overlap information indicates the method of slice overlap, etc. For example, the overlap information indicates the size of the overlapping area, etc.

[0384] slice_number represents the number of slices.

[0385] Slice position information (global_position, relative_position) and slice size information (slice_bounding_box_size) are information regarding the area of ​​a slice. Slice position information is information used to specify the location of each slice. For example, slice position information indicates the absolute or relative coordinates of each slice. Slice size information (slice_bounding_box_size) indicates the size of each slice. For example, slice size information indicates the size of the bounding box of each slice.

[0386] The three-dimensional data encoding device may include slice additional information in the SEI and transmit it. Alternatively, the three-dimensional data encoding device may store the slice additional information in an existing parameter set (PPS, GPS, or APS, etc.) and transmit it.

[0387] For example, if slice additional information changes per frame, it may be stored in the parameter set for each frame (e.g., GPS or APS). If slice additional information does not change within a sequence, it may be stored in the parameter set for each sequence (e.g., Position SPS or Attribute SPS). Additionally, if the same slice partitioning information is used in both position information and attribute information, it may be stored in the parameter set of the PCC stream (Stream PS).

[0388] Additionally, the slice additional information may be stored in any one of the above parameter sets or in multiple parameter sets. Additionally, the slice additional information may be stored in the header of the encoded data. Additionally, the slice additional information may be stored in the header of the NAL unit.

[0389] In addition, all or part of the slice additional information may be stored in either the header of the segmentation location information or the header of the segmentation attribute information, and not in the other. For example, if the same slice additional information is used in the location information and the attribute information, the slice additional information may be included in the header of either the location information or the attribute information. For example, if the attribute information depends on the location information, the location information is processed first. Therefore, the slice additional information may be included in the header of the location information, and the slice additional information may not be included in the header of the attribute information. In this case, the three-dimensional data decoder determines, for example, that the attribute information of the dependent source belongs to the same slice as the slice of the location information of the dependent source.

[0390] The three-dimensional data decoder reconstructs sliced ​​point cloud data based on slice addition information. If there is duplicate point cloud data, the three-dimensional data decoder identifies multiple duplicate point cloud data and selects one of them, or merges multiple point cloud data.

[0391] Additionally, the three-dimensional data decoder may perform decoding using slice supplementary information. For example, if multiple slices overlap, the three-dimensional data decoder may perform decoding for each slice and generate point cloud data by performing processing (e.g., smoothing or filtering) using the multiple decoded data. This makes it possible to achieve high-precision decoding.

[0392] FIG. 61 is a flowchart of a three-dimensional data encoding process including the generation of tile additional information by a three-dimensional data encoding device according to the present embodiment.

[0393] First, the three-dimensional data encoding device determines a method for dividing the tile (S5031). Specifically, the three-dimensional data encoding device determines whether to use a method based on the top view (top_view) or another method as a method for dividing the tile. In addition, the three-dimensional data encoding device determines the shape of the tile when using the method based on the top view. In addition, the three-dimensional data encoding device determines whether the tile overlaps with other tiles.

[0394] If the tile division method determined in step S5031 is a division method based on the top view (Yes in S5032), the three-dimensional data encoding device records in the tile additional information that the tile division method is a division method based on the top view (top_view) (S5033).

[0395] Meanwhile, if the tile division method determined in step S5031 is other than the division method based on the top view (No in S5032), the three-dimensional data encoding device records in the tile additional information that the tile division method is a method other than the division method based on the top view (top_view) (S5034).

[0396] Additionally, if the shape of the tile viewed from the top surface determined in step S5031 is square (square in S5035), the three-dimensional data encoding device records that the shape of the tile viewed from the top surface is square in the tile additional information (S5036). Meanwhile, if the shape of the tile viewed from the top surface determined in step S5031 is circle (circle in S5035), the three-dimensional data encoding device records that the shape of the tile viewed from the top surface is circle in the tile additional information (S5037).

[0397] Next, the three-dimensional data encoding device determines whether the tile overlaps with another tile (S5038). If the tile overlaps with another tile (Yes in S5038), the three-dimensional data encoding device records that the tile overlaps in the tile supplementary information (S5039). Meanwhile, if the tile does not overlap with another tile (No in S5038), the three-dimensional data encoding device records that the tile does not overlap in the tile supplementary information (S5040).

[0398] Next, the three-dimensional data encoding device divides the tiles based on the tile division method determined in step S5031, encodes each tile, and transmits the generated encoded data and tile additional information (S5041).

[0399] FIG. 62 is a flowchart of three-dimensional data decoding processing using tile additional information by a three-dimensional data decoding device according to the present embodiment.

[0400] First, the three-dimensional data decoder interprets tile additional information included in the bit stream (S5051).

[0401] If the additional tile information indicates that a tile does not overlap with other tiles (No in S5052), the three-dimensional data decoder generates point cloud data for each tile by decoding each tile (S5053). Next, the three-dimensional data decoder reconstructs point cloud data from the point cloud data of each tile based on the tile division method and the shape of the tile indicated in the additional tile information (S5054).

[0402] Meanwhile, if the additional tile information indicates that a tile overlaps with another tile (Yes in S5052), the three-dimensional data decoder generates point cloud data for each tile by decoding each tile. Additionally, the three-dimensional data decoder identifies the overlapping portion of the tile based on the additional tile information (S5055). Furthermore, the three-dimensional data decoder may perform decoding processing for the overlapping portion using multiple overlapping information. Next, the three-dimensional data decoder reconstructs point cloud data from the point cloud data of each tile based on the tile division method, tile shape, and overlap information indicated in the additional tile information (S5056).

[0403] Variations regarding slices, etc., are described below. A three-dimensional data encoding device may transmit information indicating the type of object (road, building, tree, etc.) or attributes (dynamic information, static information, etc.) as supplementary information. Alternatively, encoding parameters may be predetermined according to the object, and the three-dimensional data encoding device may notify the three-dimensional data decoding device of the encoding parameters by transmitting the type or attributes of the object.

[0404] Regarding the encoding and transmission order of slice data, the following methods may be used. For example, the three-dimensional data encoding device may encode slice data in order starting from data where object recognition or clustering is easy. Alternatively, the three-dimensional data encoding device may perform encoding in order starting from slice data that has been clustered quickly. Furthermore, the three-dimensional data encoding device may transmit the encoded slice data in order. Alternatively, the three-dimensional data encoding device may transmit slice data in order of high decoding priority in the application. For example, if dynamic information has a high decoding priority, the three-dimensional data encoding device may transmit slice data in order starting from slices grouped by dynamic information.

[0405] In addition, if the sequence number of the encoded data and the sequence number of the decoding priority are different, the three-dimensional data encoding device may transmit the encoded data after rearranging it. In addition, when accumulating encoded data, the three-dimensional data encoding device may accumulate the encoded data after rearranging it.

[0406] The application (three-dimensional data decoder) requests the server (three-dimensional data encoder) to transmit a slice containing the desired data. The server transmits the slice data required by the application and does not need to transmit unnecessary slice data.

[0407] The application requests the server to send tiles containing the desired data. The server sends the tile data required by the application and does not need to send unnecessary tile data.

[0408] As described above, the three-dimensional data encoding device according to the present embodiment performs the processing shown in FIG. 63. First, the three-dimensional data encoding device generates a plurality of encoded data by encoding a plurality of sub-spaces (e.g., tiles) that divide a target space containing a plurality of three-dimensional points (S5061). The three-dimensional data encoding device generates a bit stream including the plurality of encoded data and first information (e.g., topview_shape) representing the shape of the plurality of sub-spaces (S5062).

[0409] According to this, the three-dimensional data encoding device can select any shape from multiple types of sub-space shapes, thereby improving encoding efficiency.

[0410] For example, the above shape is the shape of the plurality of subspaces in two dimensions or in three dimensions. For example, the above shape is the shape of the plurality of subspaces viewed from above. That is, the first information represents the shape of the subspace viewed from a specific direction (e.g., upward). In other words, the first information represents the shape of the subspace viewed from below. For example, the above shape is a rectangle or a circle.

[0411] For example, the bit stream includes second information (e.g., tile_overlap_flag) indicating whether the plurality of sub-sections overlap.

[0412] According to this, the three-dimensional data encoding device can duplicate subspaces, so it can generate subspaces without complicating the shape of the subspaces.

[0413] For example, the bit stream includes third information (e.g., type_of_divide) indicating whether the method of dividing the plurality of sub-sections is a method of dividing when viewed from the top surface.

[0414] For example, the bit stream includes a fourth piece of information (e.g., tile_height) representing at least one of the height, width, depth, and radius of the plurality of sub-sections.

[0415] For example, the bit stream includes fifth information (e.g., global_position or relative_position) indicating the position of each of the plurality of sub-sections.

[0416] For example, the bit stream includes sixth information (e.g., tile_number) indicating the number of the plurality of sub-sections.

[0417] For example, the bit stream includes seventh information indicating the interval of the plurality of sub-intervals.

[0418] For example, a three-dimensional data encoding device is equipped with a processor and memory, and the processor uses the memory to perform the above processing.

[0419] In addition, the three-dimensional data decoding device according to the present embodiment performs the processing shown in FIG. 64. First, the three-dimensional data decoding device restores the plurality of subspaces by decoding the plurality of encoded data generated by dividing the target space containing the plurality of three-dimensional points included in the bit stream into a plurality of subspaces (e.g., tiles) (S5071). The three-dimensional data decoding device restores the target space by combining the plurality of subspaces using first information (e.g., topview_shape) representing the shape of the plurality of subspaces included in the bit stream (S5072). For example, the three-dimensional data decoding device can determine the location and range of each subspace within the target space by recognizing the shape of the plurality of subspaces using the first information. The three-dimensional data decoding device can combine the plurality of subspaces based on the location and range of the identified plurality of subspaces. By doing so, the three-dimensional data decoding device can correctly combine the plurality of subspaces.

[0420] For example, the above shape is a shape in two dimensions or a shape in three dimensions of the plurality of subspaces. For example, the above shape is a rectangle or a circle.

[0421] For example, the bit stream includes second information (e.g., tile_overlap_flag) indicating whether the plurality of sub-sections overlap. In the reconstruction of the target space, the three-dimensional data decoder further uses the second information to combine the plurality of sub-spaces. For example, the three-dimensional data decoder uses the second information to determine whether the sub-spaces overlap. If the sub-spaces overlap, the three-dimensional data decoder identifies the overlapping region and performs a predetermined response for the identified overlapping region.

[0422] For example, the bit stream includes third information (e.g., type_of_divide) indicating whether the method of dividing the plurality of sub-sections is a method of dividing when viewed from the top surface. When the third information indicates that the method of dividing the plurality of sub-sections is a method of dividing when viewed from the top surface, the three-dimensional data decoding device combines the plurality of sub-spaces using the first information.

[0423] For example, the bit stream includes fourth information (e.g., tile_height) representing at least one of the height, width, depth, and radius of the plurality of sub-sections. In the reconstruction of the target space, the three-dimensional data decoding device further uses the fourth information to combine the plurality of sub-spaces. For example, the three-dimensional data decoding device can determine the location and range of each sub-space within the target space by recognizing the height of the plurality of sub-spaces using the fourth information. The three-dimensional data decoding device can combine the plurality of sub-spaces based on the determined location and range of the plurality of sub-spaces.

[0424] For example, the bit stream includes fifth information (e.g., global_position or relative_position) indicating the position of each of the plurality of sub-sections. In the reconstruction of the target space, the three-dimensional data decoder further uses the fifth information to combine the plurality of sub-spaces. For example, the three-dimensional data decoder can determine the position of each sub-space within the target space by recognizing the positions of the plurality of sub-spaces using the fifth information. The three-dimensional data decoder can combine the plurality of sub-spaces based on the identified positions of the plurality of sub-spaces.

[0425] For example, the bit stream includes sixth information (e.g., tile_number) indicating the number of the plurality of sub-sections. In the restoration of the target space, the three-dimensional data decoding device further uses the sixth information to combine the plurality of sub-spaces.

[0426] For example, the bit stream includes seventh information indicating the interval of the plurality of sub-sections. In the restoration of the target space, the three-dimensional data decoding device further uses the seventh information to combine the plurality of sub-spaces. For example, the three-dimensional data decoding device can determine the location and range of each sub-space within the target space by recognizing the interval of the plurality of sub-spaces using the seventh information. The three-dimensional data decoding device can combine the plurality of sub-spaces based on the determined locations and ranges of the plurality of sub-spaces.

[0427] For example, a three-dimensional data decoding device is equipped with a processor and memory, and the processor uses the memory to perform the above processing.

[0428] (Embodiment 7)

[0429] In this embodiment, processing of a segmentation unit that does not include points (e.g., a tile or a slice) is described. First, a method for segmenting point cloud data is described.

[0430] In video encoding standards such as HEVC, since data exists for every pixel of a two-dimensional image, data exists in all data regions even when the two-dimensional space is divided into multiple data regions. On the other hand, in the encoding of three-dimensional point cloud data, the points themselves, which are the elements of the point cloud data, constitute the data, so there is a possibility that data does not exist in some regions.

[0431] There are various ways to spatially partition point cloud data, and partitioning methods can be classified based on whether the partitioning unit (e.g., a tile or slice), which is the partitioned data unit, always contains point data of 1 or more.

[0432] A partitioning method that includes one or more point data in all of multiple partitioning units is called a first partitioning method. As a first partitioning method, for example, there is a method of partitioning point cloud data with regard to the processing time of encoding or the size of the encoded data. In this case, the number of points in each partitioning unit is generally evenly distributed.

[0433] FIG. 65 is a diagram illustrating an example of a division method. For example, as a first division method, a method of dividing points belonging to the same space into two equal spaces may be used, as shown in FIG. 65 (a). Also, as shown in FIG. 65 (b), the space may be divided into a plurality of sub-spaces (division units) such that each division unit includes a point.

[0434] Since these methods are point-conscious divisions, every division unit always contains at least one point.

[0435] A partitioning method in which there is a possibility that at least one partitioning unit that does not contain point data is included among multiple partitioning units is called a second partitioning method. For example, as a second partitioning method, a method of dividing space equally can be used as shown in (c) of FIG. 65. In this case, it cannot be said that a point exists in the partitioning unit. That is, there are cases where a point does not exist in the partitioning unit.

[0436] When a three-dimensional data encoding device divides point cloud data, it may indicate in the division additional information (metadata) related to the division (e.g., tile additional information or slice additional information) whether (1) a division method including one or more point data in all of the multiple division units is used, (2) a division method including one or more division units that do not include point data is used among the multiple division units, or (3) a division method including one or more division units that do not include point data is used, and transmit the said division additional information.

[0437] Additionally, the three-dimensional data encoding device may represent the above information as a type of division method. Furthermore, the three-dimensional data encoding device may perform division using a predetermined division method and may not transmit additional division information. In that case, the three-dimensional data encoding device specifies in advance whether the division method is a first division method or a second division method.

[0438] Hereinafter, examples of the second division method and the generation and transmission of encoded data are described. Furthermore, as a method for dividing three-dimensional space, tile division is described as an example; however, it does not have to be tile division, and the following method can be applied to division methods with division units other than tiles. For example, tile division may be replaced with slice division.

[0439] FIG. 66 is a diagram illustrating an example of dividing point cloud data into six tiles. FIG. 66 illustrates an example where the minimum unit is a point, and shows an example of dividing geometry and attribute information together. In addition, the same applies when geometry and attribute information are divided by individual division methods or division numbers, when there is no attribute information, and when there is multiple attribute information.

[0440] In the example shown in FIG. 66, after tile division, there are tiles containing a dot (#1, #2, #4, #6) and tiles not containing a dot (#3, #5). Tiles not containing a dot are called null tiles.

[0441] In addition, the method of division is not limited to dividing into six tiles, and any method of division may be used. For example, the division unit may be a cube, or a non-cubic shape such as a rectangle or a cylinder. Multiple division units may have the same shape or may include different shapes. Also, as a method of division, a predetermined method may be used, or a different method may be used for each predetermined unit (e.g., a PCC frame).

[0442] In the present partitioning method, when point cloud data is partitioned into tiles, if there is no data within the tile, a bit stream is generated that includes information indicating that the tile is a null tile.

[0443] Next, a method for transmitting null tiles and a method for signaling null tiles will be described. A three-dimensional data encoding device may generate, for example, the following information as additional information (metadata) regarding data division, and transmit the generated information. FIG. 67 is a diagram showing an example of the syntax of tile additional information (TileMetaData). The tile additional information includes division method information (type_of_divide), division method null information (type_of_divide_null), number of tile divisions (number_of_tiles), and tile null flag (tile_null_flag).

[0444] The type_of_divide information is information regarding the division method or division type. For example, the type_of_divide information indicates one or more division methods or division types. For example, division methods include top-view division and equal division. Additionally, if there is only one definition of the division method, the type_of_divide information does not need to be included in the tile additional information.

[0445] The type_of_divide_null information indicates whether the division method used is the following first division method or second division method. Here, the first division method is a division method in which each of the multiple division units always contains at least one point data. The second division method is a division method in which there is at least one division unit that does not contain point data among the multiple division units, or a division method in which there is a possibility that there is at least one division unit that does not contain point data among the multiple division units.

[0446] Additionally, the tile supplementary information may include at least one of the following as tile division information: (1) information indicating the number of tile divisions (number_of_tiles), or information for specifying the number of tile divisions; (2) information indicating the number of null tiles, or information for specifying the number of null tiles; and (3) information indicating the number of tiles other than null tiles, or information for specifying the number of tiles other than null tiles. Additionally, the tile supplementary information may include information indicating the shape of the tile or whether the tiles are duplicated, as tile division information.

[0447] Additionally, the tile supplementary information indicates the division information for each tile in sequence. For example, the order of the tiles is predetermined for each division method and is known to the three-dimensional data encoding device and the three-dimensional data decoding device. Furthermore, if the order of the tiles is not predetermined, the three-dimensional data encoding device may transmit information indicating the order to the three-dimensional data decoding device.

[0448] The splitting information for each tile includes a tile null flag (tile_null_flag), which indicates whether data (points) exist within the tile. Additionally, if there is no data within the tile, the tile null flag may be included as the tile splitting information.

[0449] Additionally, if the tile is not a null tile, the additional tile information includes subdivision information for each tile (location information (e.g., origin coordinates (origin_x, origin_y, origin_z)), and height information of the tile, etc.). Also, if the tile is a null tile, the additional tile information does not include subdivision information for each tile.

[0450] For example, if the information on the slice division of each tile is stored in the division information for each tile, the three-dimensional data encoding device does not need to store the information on the slice division of null tiles in the additional information.

[0451] In addition, in this example, the number of tiles (number_of_tiles) represents the number of tiles including null tiles. FIG. 68 is a diagram showing an example of tile index information (idx). In the example shown in FIG. 68, index information is also assigned to null tiles.

[0452] Next, a data structure and transmission method for encoded data including null tiles will be described. FIGS. 69 to 71 are diagrams showing a data structure in which data does not exist in the 3rd and 5th tiles when location information and attribute information are divided into 6 tiles.

[0453] FIG. 69 is a diagram illustrating an example of the dependency relationships of each data. The tips of the arrows in the diagram represent the sources of dependency, and the roots represent the sources of dependency. Also, in the said diagram, G tn (n is 1~6) represents the location information of tile number n, and A tn represents the attribute information of tile number n. M tile Silver indicates additional tile information.

[0454] FIG. 70 is a diagram showing an example of the configuration of transmitted data, which is encoded data transmitted from a three-dimensional data encoding device. FIG. 71 is also a diagram showing the configuration of encoded data and a method of storing encoded data in a NAL unit.

[0455] As shown in FIG. 71, within the data headers of location information (divided location information) and attribute information (divided attribute information), the tile index information (tile_idx) is included, respectively.

[0456] In addition, as shown in Structure 1 of FIG. 70, the three-dimensional data encoding device does not need to transmit position information or attribute information constituting the null tile. Alternatively, as shown in Structure 2 of FIG. 70, the three-dimensional data encoding device may transmit information indicating that the tile is a null tile as null tile data. For example, the three-dimensional data encoding device may indicate that the type of the data is a null tile in the tile_type stored in the header of the NAL unit or in the header within the payload (nal_unit_payload) of the NAL unit, and transmit the header. Furthermore, the following explanation will be based on Structure 1.

[0457] In structure 1, when a null tile exists, the value of the tile index information (tile_idx) included in the header of the location information data or attribute information data in the transmission data is missing and not continuous.

[0458] Furthermore, the three-dimensional data encoding device transmits the data at the reference location so that it can be decoded before the data at the reference source when there is a dependency relationship between the data. Additionally, the tiles of attribute information have a dependency relationship with the tiles of position information. The same tile index number is assigned to the attribute information and position information that have a dependency relationship.

[0459] Additionally, tile supplementary information related to tile division may be stored in both the location information parameter set (GPS) and the attribute information parameter set (APS), or in either one. If tile supplementary information is stored in one of the GPS and APS, reference information indicating the GPS or APS of the reference location may be stored in the other of the GPS and APS. Furthermore, if the tile division method differs in the location information and attribute information, different tile supplementary information is stored in the GPS and APS, respectively. Also, if the tile division method is the same across the sequence (multiple PCC frames), tile supplementary information may be stored in the GPS, APS, or SPS (sequence parameter set).

[0460] For example, when tile supplementary information is stored in both GPS and APS, tile supplementary information for location information is stored in GPS, and tile supplementary information for attribute information is stored in APS. Additionally, when tile supplementary information is stored in common information such as SPS, tile supplementary information used commonly in both location information and attribute information may be stored, or tile supplementary information for location information and tile supplementary information for attribute information may be stored separately.

[0461] The following describes the combination of tile splitting and slice splitting. First, the data configuration and data transmission when tile splitting is performed after slice splitting will be explained.

[0462] FIG. 72 is a diagram showing an example of the dependency relationships of each data when tile division is performed after slice division. The tip of the arrow in the diagram indicates the place of dependency, and the root of the arrow indicates the source of dependency. Also, the data shown by solid lines in the diagram is the data that is actually transmitted, and the data shown by dotted lines is the data that is not transmitted.

[0463] Also, in the same drawing, G represents location information and A represents attribute information. Gs1 represents the location information of slice number 1, and G s2 represents the location information of slice number 2. G s1t1 Silver, slice number 1 also represents the location information of tile number 1, and G s2t2 represents the location information of slice number 2 as well as tile number 2. Likewise, A s1 represents the attribute information of slice number 1, and A s2 represents the attribute information of slice number 2. A s1t1 Silver represents the attribute information of slice number 1 and tile number 1, and A s2t1 Silver, slice number 2 also represents the attribute information of tile number 1.

[0464] Mslice represents slice additional information, MGtile represents position tile additional information, and MAtile represents attribute tile additional information. D s1t1 Attribute information A s1t1 Represents the dependency relationship information of, and D s2t1 Attribute information A s2t1 Represents the dependency relationship information.

[0465] The three-dimensional data encoding device does not need to generate and transmit position information and attribute information related to null tiles.

[0466] In addition, for all slices, even if the number of tile divisions is the same, the number of tiles generated and transmitted between slices may differ. For example, if the number of tile divisions for position information and attribute information is different, there may be cases where a null tile exists in one of the position information and attribute information and not in the other. In the example shown in FIG. 72, the position information (G of slice 1) s1 ) is, G s1t1 and G s1t2 It is divided into 2 tiles, and among them G s1t2is a null tile. Meanwhile, the attribute information of slice 1 (A s1 ) is not divided and is a single A s1t1 This exists, and null tiles do not exist.

[0467] Additionally, the three-dimensional data encoding device generates and transmits dependency relationship information of attribute information when data exists in at least the tile of attribute information, regardless of whether a null tile is included in the slice of position information. For example, when the three-dimensional data encoding device stores information on slice division for each tile in the division information for each slice included in the slice supplementary information related to slice division, it stores information on whether the corresponding tile is a null tile in this information.

[0468] FIG. 73 is a diagram illustrating an example of the decoding order of data. In the example of FIG. 73, decoding is performed sequentially starting from the data on the left. The three-dimensional data decoding device decodes the dependent data first among data that have a dependency relationship. For example, the three-dimensional data encoding device rearranges the data in advance and transmits it so that this order is achieved. Additionally, any order is acceptable as long as the dependent data comes first. Furthermore, the three-dimensional data encoding device may transmit additional information and dependency relationship information before the data.

[0469] Next, we will explain the data configuration and data transmission when performing slice division after tile division.

[0470] FIG. 74 is a diagram showing an example of the dependency relationships of each data when slice division is performed after tile division. The tip of the arrow in the diagram indicates the dependency, and the root of the arrow indicates the dependency. Also, the data shown by solid lines in the diagram is the data that is actually transmitted, and the data shown by dotted lines is the data that is not transmitted.

[0471] Also, in the same drawing, G represents location information and A represents attribute information. G t1 represents the location information of tile number 1. G t1s1 Silver represents the location information of tile number 1 and slice number 1, and G t1s2 represents the location information of tile number 1 as well as slice number 2. Likewise, A t1 Silver represents the attribute information of tile number 1, and A t1s1 It represents the attribute information of tile number 1 and slice number 1.

[0472] Mtile represents tile additional information, MGslice represents position slice additional information, and MAslice represents attribute slice additional information. D t1s1 Attribute information A t1s1 Represents the dependency relationship information of, and D t2s1 Attribute information A t2s1 Represents the dependency relationship information.

[0473] The three-dimensional data encoding device does not slice the null tile. Furthermore, it is not necessary to generate and transmit position information, attribute information, and dependency relationship information related to the null tile.

[0474] FIG. 75 is a diagram illustrating an example of the decoding order of data. In the example of FIG. 75, decoding is performed sequentially starting from the data on the left. When there is a dependency relationship between data, the three-dimensional data decoding device first decodes the data of the dependent source. For example, the three-dimensional data encoding device rearranges the data in advance and transmits it in this order. Additionally, any order is acceptable as long as the data of the dependent source comes first. Furthermore, the three-dimensional data encoding device may transmit additional information and dependency relationship information before the data.

[0475] Next, the flow of partitioning and combining point cloud data is explained. Additionally, while examples of tile partitioning and slice partitioning are described here, the same methods can be applied to partitioning other types of space.

[0476] FIG. 76 is a flowchart of a three-dimensional data encoding process including data partitioning processing by a three-dimensional data encoding device. First, the three-dimensional data encoding device determines the partitioning method to be used (S5101). Specifically, the three-dimensional data encoding device determines whether to use a first partitioning method or a second partitioning method. For example, the three-dimensional data encoding device may determine the partitioning method based on a designation from a user or an external device (e.g., a three-dimensional data decoder), or it may determine the partitioning method based on input point cloud data. In addition, the partitioning method to be used may be predetermined.

[0477] Here, the first division method is a division method in which at least one point data is always included in each of the plurality of division units (tiles or slices). The second division method is a division method in which at least one division unit that does not include point data exists among the plurality of division units, or a division method in which there is a possibility that at least one division unit that does not include point data exists among the plurality of division units.

[0478] If the determined division method is the first division method (the first division method in S5102), the three-dimensional data encoding device indicates that the division method used in the division additional information (e.g., tile additional information or slice additional information), which is metadata related to data division, is the first division method (S5103). Then, the three-dimensional data encoding device encodes all division units (S5104).

[0479] Meanwhile, if the determined division method is the second division method (the second division method in S5102), the three-dimensional data encoding device indicates that the division method used in the additional division information is the second division method (S5105). Then, the three-dimensional data encoding device encodes the division units among the plurality of division units, excluding the division units that do not include point data (e.g., null tiles) (S5106).

[0480] FIG. 77 is a flowchart of a three-dimensional data decoding process including data combination processing by a three-dimensional data decoding device. First, the three-dimensional data decoding device refers to additional division information included in the bit stream and determines whether the division method used is a first division method or a second division method (S5111).

[0481] If the division method used is the first division method (the first division method in S5112), the three-dimensional data decoder receives encoded data of all division units and generates decoded data of all division units by decoding the received encoded data (S5113). Next, the three-dimensional data decoder reconstructs a three-dimensional point cloud using the decoded data of all division units (S5114). For example, the three-dimensional data decoder reconstructs a three-dimensional point cloud by combining a plurality of division units.

[0482] Meanwhile, if the division method used is the second division method (the second division method in S5112), the three-dimensional data decoder receives encoded data of a division unit containing point data and encoded data of a division unit not containing point data, and generates decoded data by decoding the received encoded data of the division unit (S5115). Additionally, if a division unit not containing point data is not transmitted, the three-dimensional data decoder does not need to receive and decode a division unit not containing point cloud data. Next, the three-dimensional data decoder reconstructs a three-dimensional point cloud using the decoded data of the division unit containing point data (S5116). For example, the three-dimensional data decoder reconstructs a three-dimensional point cloud by combining a plurality of division units.

[0483] Below, other methods for partitioning point cloud data are described. As shown in FIG. 65 (c), when space is divided equally, there may be cases where no points exist in the partitioned space. In this case, the three-dimensional data encoding device combines the space where no points exist with another space where points exist. By doing so, the three-dimensional data encoding device can form a plurality of partition units such that every partition unit includes one or more points.

[0484] FIG. 78 is a flowchart of the data partitioning in this case. First, the three-dimensional data encoding device partitions the data using a specific method (S5121). For example, the specific method is the second partitioning method described above.

[0485] Next, the three-dimensional data encoding device determines whether a point is included in the target division unit, which is the division unit of the processing target (S5122). If a point is included in the target division unit (Yes in S5122), the three-dimensional data encoding device encodes the target division unit (S5123). On the other hand, if a point is not included in the target division unit (No in S5122), the three-dimensional data encoding device combines the target division unit with another division unit containing a point and encodes the division unit after combination (S5124). That is, the three-dimensional data encoding device encodes the target division unit together with another division unit containing a point.

[0486] In addition, although an example of performing determination and combination for each division unit has been described here, the processing method is not limited thereto. For example, a three-dimensional data encoding device may determine whether a point is included in each of a plurality of division units, perform combination so that division units not containing a point are eliminated, and then encode each of the plurality of division units after combination.

[0487] Next, a method for transmitting data including null tiles will be described. The three-dimensional data encoding device does not transmit data of the target tile when the target tile, which is the tile to be processed, is a null tile. FIG. 79 is a flowchart of the data transmission process.

[0488] First, the three-dimensional data encoding device determines a method for dividing tiles and divides point cloud data into tiles using the determined method (S5131).

[0489] Next, the three-dimensional data encoding device determines whether the target tile is a null tile (S5132). That is, the three-dimensional data encoding device determines whether there is no data within the target tile.

[0490] If the target tile is a null tile (Yes in S5132), the three-dimensional data encoding device indicates in the tile supplementary information that the target tile is a null tile and does not indicate the information of the target tile (such as the position and size of the tile) (S5133). Also, the three-dimensional data encoding device does not transmit the target tile (S5134).

[0491] Meanwhile, if the target tile is not a null tile (No in S5132), the three-dimensional data encoding device indicates in the tile supplementary information that the target tile is not a null tile and indicates information for each tile (S5135). Also, the three-dimensional data encoding device transmits the target tile (S5136).

[0492] In this way, by not including null tile information in the tile additional information, the amount of information in the tile additional information can be reduced.

[0493] Below, a decoding method for encoded data including null tiles is described. First, the processing for cases without packet loss is explained.

[0494] FIG. 80 is a diagram showing an example of transmitted data, which is encoded data transmitted from a three-dimensional data encoding device, and received data input to a three-dimensional data decoder. In addition, the case of a system environment without packet loss is assumed here, and the received data is the same as the transmitted data.

[0495] In the case of a system environment without packet loss, the three-dimensional data decoder receives all transmitted data. FIG. 81 is a flowchart of processing by the three-dimensional data decoder.

[0496] First, the three-dimensional data decoding device determines whether each tile is a null tile by referring to tile additional information (S5141) (S5142).

[0497] If the tile supplementary information indicates that the target tile is not a null tile (No in S5142), the three-dimensional data decoder determines that the target tile is not a null tile and decodes the target tile (S5143). Next, the three-dimensional data decoder obtains tile information (tile position information (origin coordinates, etc.) and size, etc.) from the tile supplementary information, and reconstructs three-dimensional data by combining a plurality of tiles using the obtained information (S5144).

[0498] Meanwhile, if the additional tile information indicates that the target tile is not a null tile (Yes in S5142), the three-dimensional data decoding device determines that the target tile is a null tile and does not decode the target tile (S5145).

[0499] In addition, the three-dimensional data decoder may determine that data with missing parts is a null tile by sequentially interpreting the index information indicated in the header of the encoded data. Furthermore, the three-dimensional data decoder may combine a determination method using tile additional information and a determination method using index information.

[0500] Next, the handling of cases involving packet loss will be described. FIG. 82 is a diagram showing examples of transmitted data sent from a three-dimensional data encoding device and received data input to a three-dimensional data decoder. Here, a case of a system environment with packet loss is assumed.

[0501] In a system environment with packet loss, there is a possibility that the three-dimensional data decoder may not be able to receive all of the transmitted data. In this example, G t2 and A t2 The packets are lost.

[0502] FIG. 83 is a flowchart of the processing of the three-dimensional data decoder in this case. First, the three-dimensional data decoder interprets the continuity of index information indicated in the header of the encoded data (S5151) and determines whether an index number of a target tile exists (S5152).

[0503] If the index number of the target tile exists (Yes in S5152), the three-dimensional data decoder determines that the target tile is not a null tile and performs decoding processing of the target tile (S5153). Next, the three-dimensional data decoder obtains tile information (tile position information (origin coordinates, etc.) and size, etc.) from tile supplementary information and reconstructs three-dimensional data by combining a plurality of tiles using the obtained information (S5154).

[0504] Meanwhile, if index information of the target tile does not exist (No in S5152), the three-dimensional data decoding device determines whether the target tile is a null tile by referring to tile additional information (S5155).

[0505] If the target tile is not a null tile (No in S5156), the three-dimensional data decoder determines that the target tile is lost (packet loss) and performs error decoding (S5157). Error decoding is a process that attempts to decode the original data by, for example, assuming that data existed. In this case, the three-dimensional data decoder may regenerate the three-dimensional data and perform reconstruction of the three-dimensional data (S5154).

[0506] Meanwhile, if the target tile is a null tile (Yes in S5156), the three-dimensional data decoding device assumes that the target tile is a null tile and does not perform decoding processing and reconstruction of the three-dimensional data (S5158).

[0507] Next, an encoding method for cases where null tiles are not specified is described. A three-dimensional data encoding device may generate encoded data and additional information using the following methods.

[0508] The three-dimensional data encoding device does not indicate information about null tiles in the tile supplementary information. The three-dimensional data encoding device assigns index numbers of tiles excluding null tiles to the data header. The three-dimensional data encoding device does not transmit null tiles.

[0509] In this case, the number of tile divisions (number_of_tiles) represents the number of divisions that do not include null tiles. Additionally, the three-dimensional data encoding device may store information indicating the number of separate null tiles in a bit stream. Furthermore, the three-dimensional data encoding device may indicate information regarding null tiles in the additional information, or indicate partial information regarding null tiles.

[0510] FIG. 84 is a flowchart of three-dimensional data encoding processing by a three-dimensional data encoding device in this case. First, the three-dimensional data encoding device determines a tile division method and divides point cloud data into tiles using the determined division method (S5161).

[0511] Next, the three-dimensional data encoding device determines whether the target tile is a null tile (S5162). That is, the three-dimensional data encoding device determines whether there is no data within the target tile.

[0512] If the target tile is not a null tile (No in S5162), the three-dimensional data encoding device assigns index information of tiles excluding null tiles to the data header (S5163). Then, the three-dimensional data encoding device transmits the target tile (S5164).

[0513] Meanwhile, if the target tile is a null tile (Yes in S5162), the three-dimensional data encoding device assigns index information of the target tile to the data header and does not transmit the target tile.

[0514] FIG. 85 is a diagram showing an example of index information (idx) added to a data header. As shown in FIG. 85, index information for null tiles is not added, and consecutive numbers are added to tiles other than null tiles.

[0515] FIG. 86 is a diagram illustrating an example of the dependency relationships of each data. The tips of the arrows in the diagram represent the sources of dependency, and the roots represent the sources of dependency. Also, in the said diagram, G tn (n is 1~4) represents the location information of tile number n, and A tn represents the attribute information of tile number n. M tile Silver indicates additional tile information.

[0516] FIG. 87 is a diagram showing an example of the configuration of transmitted data, which is encoded data transmitted from a three-dimensional data encoding device.

[0517] Below, a decoding method for cases where null tiles are not specified will be described. FIG. 88 is a diagram showing examples of transmitted data sent from a three-dimensional data encoding device and received data input to a three-dimensional data decoding device. Here, a case of a system environment with packet loss is assumed.

[0518] FIG. 89 is a flowchart of the processing of the three-dimensional data decoder in this case. First, the three-dimensional data decoder interprets the index information of the tiles indicated in the header of the encoded data and determines whether the index number of the target tile exists. Also, the three-dimensional data decoder obtains the number of tile divisions from the tile additional information (S5171).

[0519] If an index number of a target tile exists (Yes in S5172), the three-dimensional data decoder performs decoding processing of the target tile (S5173). Next, the three-dimensional data decoder obtains tile information (tile position information (origin coordinates, etc.) and size, etc.) from tile supplementary information, and reconstructs three-dimensional data by combining a plurality of tiles using the obtained information (S5175).

[0520] Meanwhile, if the index number of the target tile does not exist (No in S5172), the three-dimensional data decoder determines that the target tile is a packet loss and performs error decoding (S5174). Additionally, the three-dimensional data decoder determines that the space not present in the data is a null tile and reconstructs the three-dimensional data.

[0521] In addition, the three-dimensional data encoding device can properly determine that there is no point within the tile, rather than data loss or packet loss caused by measurement errors or data processing, by explicitly indicating a null tile.

[0522] Additionally, the three-dimensional data encoding device may use a method that explicitly indicates null packets and a method that does not explicitly indicate null packets. In that case, the three-dimensional data encoding device may indicate information regarding whether null packets are explicitly indicated in the tile supplementary information. Alternatively, depending on the type of partitioning method, the decision regarding whether null packets are explicitly indicated may be determined in advance, and the three-dimensional data encoding device may indicate whether null packets are explicitly indicated by indicating the type of partitioning method.

[0523] Also, in FIG. 67 and others, an example is shown in which information related to all tiles is displayed in the tile additional information. In the tile additional information, information of some tiles among a plurality of tiles may be displayed, and information of some tiles among a plurality of tiles may be displayed.

[0524] In addition, examples have been described in which information regarding partition data, such as whether partition data (tiles) exist, is stored in tile supplementary information. Some or all of this information may be stored in a parameter set or may be stored as data. When this information is stored as data, for example, a nal_unit_type representing information indicating whether partition data exists may be defined and this information may be stored in a NAL unit. In addition, this information may be stored in both supplementary information and data.

[0525] As described above, the three-dimensional data encoding device according to the present embodiment performs the processing shown in FIG. 90. First, the three-dimensional data encoding device generates a plurality of encoded data by encoding a plurality of subspaces (e.g., tiles or slices) that divide a target space containing a plurality of three-dimensional points (S5181). The three-dimensional data encoding device generates a bit stream containing the plurality of encoded data and a first information (e.g., tile_null_flag) corresponding to each of the plurality of subspaces (S5182). Each of the plurality of first information indicates whether a second information representing the structure of the corresponding subspace is included in the bit stream.

[0526] According to this, for example, the second information can be omitted for subspaces that do not include points, so the amount of data in the bit stream can be reduced.

[0527] For example, the second information includes information indicating the coordinates of the origin of the corresponding subspace. For example, the second information includes information indicating at least one of the height, width, and depth of the corresponding subspace.

[0528] According to this, the three-dimensional data encoding device can reduce the amount of data in the bit stream.

[0529] In addition, as shown in FIG. 78, the three-dimensional data encoding device may divide a target space containing a plurality of three-dimensional points into a plurality of subspaces (e.g., tiles or slices), combine the plurality of subspaces according to the number of three-dimensional points included in each subspace, and encode the subspaces after combination. For example, the three-dimensional data encoding device may combine the plurality of subspaces such that the number of three-dimensional points included in each of the plurality of subspaces after combination is greater than or equal to a predetermined number. For example, the three-dimensional data encoding device may combine a subspace that does not contain three-dimensional points with a subspace that contains three-dimensional points.

[0530] According to this, the three-dimensional data encoding device can suppress the generation of a subspace with a small number of points or one that does not contain points, thereby improving encoding efficiency.

[0531] For example, a three-dimensional data encoding device is equipped with a processor and memory, and the processor uses the memory to perform the above processing.

[0532] In addition, the three-dimensional data decoding device according to the present embodiment performs the processing shown in FIG. 91. First, the three-dimensional data decoding device obtains a plurality of first information (e.g., tile_null_flag) from a bit stream, each corresponding to a plurality of subspaces (e.g., tiles or slices) that divide a target space containing a plurality of three-dimensional points, and each indicating whether a second information representing the structure of the corresponding subspace is included in the bit stream (S5191). The three-dimensional data decoding device uses the plurality of first information to (i) restore a plurality of subspaces by decoding a plurality of encoded data generated by encoding a plurality of subspaces included in the bit stream, and (ii) restore a target space by combining the plurality of subspaces (S5192). For example, the three-dimensional data decoding device uses the first information to determine whether the second information is included in the bit stream, and if the second information is included in the bit stream, combines the decoded plurality of subspaces using the second information.

[0533] According to this, for example, the second information can be omitted for subspaces that do not include points, so the amount of data in the bit stream can be reduced.

[0534] For example, the second information includes information indicating the coordinates of the origin of the corresponding subspace. For example, the second information includes information indicating at least one of the height, width, and depth of the corresponding subspace.

[0535] According to this, the three-dimensional data decoding device can reduce the amount of data in the bit stream.

[0536] Additionally, the three-dimensional data decoding device may receive encoded data generated by dividing a target space containing a plurality of three-dimensional points into a plurality of subspaces (e.g., tiles or slices), combining the plurality of subspaces according to the number of three-dimensional points included in each subspace, and encoding the subspaces after combination, and may decode the received encoded data. For example, the encoded data may be generated by combining the plurality of subspaces such that the number of three-dimensional points included in each of the plurality of subspaces after combination is greater than or equal to a predetermined number. For example, the three-dimensional data may be generated by combining a subspace that does not contain three-dimensional points with a subspace that contains three-dimensional points.

[0537] According to this, the three-dimensional data device can decode encoded data with improved encoding efficiency by suppressing the generation of a subspace that has a small number of points or does not contain points.

[0538] For example, a three-dimensional data decoding device is equipped with a processor and memory, and the processor uses the memory to perform the above processing.

[0539] (Embodiment 8)

[0540] FIG. 92 is a block diagram showing an example of the configuration of a three-dimensional data encoding device according to the present embodiment. FIG. 93 is a diagram for explaining a schematic of an encoding method using a three-dimensional data encoding device according to the present embodiment.

[0541] A three-dimensional data encoding device (6800) generates segmented data in which point cloud data is divided into multiple parts, such as tiles or slices, and encodes each of the multiple segmented data. The segmented data is also called sub-point cloud data. The point cloud data is data representing multiple three-dimensional positions in a three-dimensional space. Also, the multiple segmented data are multiple sub-point cloud data in which the point cloud data is divided into multiple parts by dividing the three-dimensional space in which the point cloud data is placed into multiple sub-spaces. Also, the number of divisions, that is, the number of segmented data, may be 1, which indicates that it is not divided, or 2 or more.

[0542] Additionally, FIG. 92 describes the configuration of a three-dimensional data encoding device (6800) that divides into two parts. FIG. 93 shows an example of dividing point cloud data into four parts. FIG. 93 describes the space to be divided as a two-dimensional space, but it may be a one-dimensional space or a three-dimensional space.

[0543] The three-dimensional data encoding device (6800) comprises a division method determining unit (6801), a division unit (6802), a quantization unit (6803a, 6803b), a shift amount calculation unit (6804a, 6804b), a common position shift unit (6805a, 6805b), an individual position shift unit (6806a, 6806b), and an encoding unit (6807a, 6807b).

[0544] The partitioning method determining unit (6801) determines a partitioning method for point cloud data. The partitioning method determining unit (6801) outputs partitioning method information indicating the partitioning method to the partitioning unit (6802) and the shift amount calculation unit (6804a, 6804b). Specific examples of the partitioning method will be described later. The three-dimensional data encoding device (6800) may not have the partitioning method determining unit (6801). In this case, the three-dimensional data encoding device (6800) may partition the point cloud data into a plurality of partitioned data using a predetermined partitioning method.

[0545] The splitting unit (6802) splits the point cloud data into a plurality of split data according to the splitting method determined by the splitting method determining unit (6801). The plurality of split data split by the splitting unit (6802) are processed individually. Accordingly, the three-dimensional data encoding device (6800) is provided with a processing unit that performs subsequent processing for each split data. Specifically, the three-dimensional data encoding device (6800) is provided with a quantization unit (6803a), a shift amount calculation unit (6804a), a common position shift unit (6805a), an individual position shift unit (6806a), and an encoding unit (6807a) for processing the first split data. Additionally, the three-dimensional data encoding device (6800) comprises a quantization unit (6803b), a shift amount calculation unit (6804b), a common position shift unit (6805b), an individual position shift unit (6806b), and an encoding unit (6807b) for processing the second divided data. By doing so, the three-dimensional data encoding device (6800) can perform processing for each of the multiple divided data in parallel. Also, in FIG. 92, the three-dimensional data encoding device (6800) shows an example of a processing unit that processes two divided data in parallel, but it may have a processing unit that processes three or more divided data in parallel. Additionally, the three-dimensional data encoding device may be configured to process multiple divided data with separate processing units for each.

[0546] Each of the quantization units (6803a, 6803b) performs scaling (dividing position information into arbitrary values) and quantization on the corresponding divided data. Each of the quantization units (6803a, 6803b) may delete at least one of the duplicated points when multiple points overlap, or may not process the at least one point.

[0547] Each of the shift amount calculation units (6804a, 6804b) calculates at least one of a common position shift amount and an individual position shift amount for shifting, i.e., moving, the position of the corresponding divided data according to the division method determined by the division method determination unit (6801). Depending on the division method, the shift amount calculation units (6804a, 6804b) may calculate only the common position shift amount, only the individual position shift amount, or both the common position shift amount and the individual position shift amount.

[0548] A common position shift amount is a shift amount (movement amount) that moves the positions of multiple divided data in common. That is, the common position shift amount is the same among the multiple divided data. The common position shift amount includes the direction in which the positions of the multiple divided data are moved, and the distance in which they are moved. The common position shift amount is an example of a first movement amount.

[0549] The individual position shift amount is a shift amount (movement amount) that individually moves the position of each of the multiple divided data. The individual position shift amount is a shift amount determined in a one-to-one correspondence with each of the multiple divided data, and differs from one another among the multiple divided data in various cases. The individual position shift amount includes the direction of moving the position of the corresponding divided data and the distance of moving. The individual position shift amount is an example of the second movement amount.

[0550] Each of the common position shift units (6805a, 6805b) is a common position shift amount calculated by the shift amount calculation unit (6804a, 6804b), and performs a position shift on the corresponding divided data. Accordingly, the multiple divided data (6811~6814) from which the point cloud data (6810) shown in FIG. 93 (a) is divided move in the direction and distance represented by the common position shift amount, as shown in FIG. 93 (b).

[0551] Each of the individual position shift units (6806a, 6806b) is an individual position shift amount calculated by the shift amount calculation unit (6804a, 6804b), and performs a position shift on the corresponding divided data. Accordingly, the plurality of divided data (6811~6814) shown in (c) of FIG. 93 are moved in the direction and distance represented by the corresponding individual shift amount.

[0552] Each of the encoding units (6807a, 6807b) encodes the corresponding segment data among the plurality of segment data moved by the individual position shift units (6806a, 6806b).

[0553] Additionally, the processing order of the splitting unit (6802), the quantization unit (6803a, 6803b), the shift amount calculation unit (6804a, 6804b), the common position shift unit (6805a, 6805b), and the individual position shift unit (6806a, 6806b) may be changed. For example, the shift amount calculation unit (6804a, 6804b) and the common position shift unit (6805a, 6805b) may be processed before the splitting unit (6802). In this case, the shift amount calculation unit (6804a, 6804b) may be merged into one processing unit, and the common position shift unit (6805a, 6805b) may be merged into one processing unit. In addition, in this case, the shift amount calculation unit (6804a, 6804b) may calculate at least the common position shift amount among the common position shift amount and the individual position shift amount before the common position shift unit (6805a, 6805b), and the individual position shift amount may be calculated before the processing of the individual position shift unit (6806a, 6806b) is performed. That is, the processing unit that calculates the common position shift amount may be configured to calculate the common position shift amount before the common position shift unit, separately from the processing unit that calculates the individual position shift amount. Also, any two or more of the above processing units may be merged.

[0554] Next, examples of calculating a common position shift amount and individual position shift amounts will be explained using FIG. 94. FIG. 94 is a diagram for explaining a first example of position shifting. The first example is an example in which, after shifting point cloud data (6810) by a common position shift amount, a plurality of segmented data (6811 to 6814) are shifted by individual position shift amounts corresponding to each.

[0555] As shown in FIG. 94 (a), the three-dimensional data encoding device generates a bounding box (6820) of a size that includes all segmented data (6811–6814) of the point cloud data (6810), and calculates the minimum value point of the generated bounding box (6820). Then, the three-dimensional data encoding device calculates the direction and distance of a vector represented by the difference between the calculated minimum value point and the origin as a common position shift amount. Also, since the origin is 0, the difference is represented by the coordinates of the minimum value point. The origin may be a predetermined reference point that is not 0. The bounding box (6820) may be a rectangular area of ​​the minimum size that encloses all segmented data (6811–6814). The minimum value point of the bounding box (6820) is the point closest to the origin within the area of ​​the bounding box (6820). The bounding box (6820) is also called a common bounding box. Also, the bounding box is also called an encoding bounding box.

[0556] The three-dimensional data encoding device moves a plurality of segmented data (6811 to 6814) by a calculated common position shift amount as shown in FIG. 94 (b). Additionally, the three-dimensional data encoding device may move the point cloud data (6810) before segmentation by a common position shift amount.

[0557] Next, the three-dimensional data encoding device generates a bounding box (6821–6824) of a size containing the segmented data (6811–6814) for each of the multiple segmented data (6811–6814) after common position shifting, as shown in FIG. 94 (b), and calculates the minimum value point of each generated bounding box (6821–6824). Then, for each of the multiple segmented data (6811–6814), the three-dimensional data encoding device calculates the distance between the minimum value point of the bounding box corresponding to the segmented data and the origin as the individual position shift amount of the segmented data. Each bounding box (6821–6824) may be a rectangular area of ​​the minimum size surrounding the corresponding segmented data (6811–6814). The point of minimum value of the bounding boxes (6821–6824) is the point closest to the origin in each area of ​​the bounding boxes (6821–6824). Each bounding box (6821–6824) is also referred to as an individual bounding box.

[0558] The three-dimensional data encoding device moves each of the plurality of segmented data (6811 to 6814) by a calculated corresponding individual position shift amount, as shown in (c) of FIG. 94.

[0559] A three-dimensional data encoding device generates a bit stream by encoding each of the multiple segmented data (6811 to 6814) shifted by individual position shift amounts using corresponding bounding boxes (6821 to 6824). At this time, the three-dimensional data encoding device stores second bounding box information, which indicates the position and size of the minimum value point of each bounding box (6821 to 6824), in metadata included in the bit stream. Additionally, hereinafter, the bounding box is also referred to as an encoding bounding box (encoding BB).

[0560] Additionally, the first bounding box information, which indicates the common position shift amount and the position and size of the minimum value point of the bounding box (6820), is stored in the SPS of the data configuration of the bit stream shown in (e) of FIG. 94. Also, the individual position shift amount is stored in the header of the position information of the corresponding divided data. Also, the second bounding box information of the bounding boxes (6821~6824) used for encoding each divided data (6811~6814) is stored in the header of the position information of the corresponding divided data.

[0561] Here, if the common position shift amount is set to Shift_A and the individual position shift amount is set to Shift_B(i) (where i is the index of the partitioned data), the shift amount Shift(i) of the partitioned data (i) can be calculated using the following formula.

[0562] Shift(i)=Shift_A+Shift_B(i)

[0563] That is, as shown in (d) of FIG. 94, the total shift amount of each segmented data can be calculated by adding the common position shift amount and the corresponding individual position shift amount.

[0564] The three-dimensional data encoding device shifts the position of the point cloud data of the i-th segmented data by subtracting Shift(i) before encoding the point cloud data.

[0565] A three-dimensional data decoding device can obtain Shift_A and Shift_B(i) from the header of the SPS and the partitioned data, calculate Shift(i), and then return the partitioned data to its original position by adding Shift(i) to the decoded partitioned data (i). By doing so, multiple partitioned data can be correctly restored.

[0566] Next, a second example of a position shift in which a common position shift is performed and an individual position shift is not performed is explained using FIG. 95. In the second example, since the individual position shift amount is not transmitted, the amount of information in the bit stream can be reduced.

[0567] FIG. 95 is a diagram illustrating a second example of position shifting. In the second example, point cloud data (6810) is position shifted by a common position shift amount, and each segment data (6811 to 6814) is not position shifted by an individual position shift amount.

[0568] The three-dimensional data encoding device generates a bounding box (6820) of a size that includes all segmented data (6811 to 6814) of the point cloud data (6810), as shown in FIG. 95 (a), and calculates a common position shift amount using the generated bounding box (6820). The method for calculating the common position shift amount is the same as the method described using FIG. 94.

[0569] As shown in FIG. 95 (b), the three-dimensional data encoding device moves a plurality of segmented data (6811 to 6814) by a calculated common position shift amount and encodes using a bounding box (6820) containing all segmented data (6811 to 6814) after the common position shift.

[0570] In this way, in the second example, after dividing the point cloud data (6810), the individual position shift amount or bounding box information for each of the multiple divided data (6811 to 6814) is not calculated. Furthermore, in the second example, the three-dimensional data encoding device shifts the positions of the multiple divided data (6811 to 6814) by a common position shift amount and also encodes them using a common bounding box, but is not limited thereto. For example, the three-dimensional data encoding device may shift the positions of the multiple divided data (6811 to 6814) by a common position shift amount and encode them using the individual bounding boxes of each divided data (6811 to 6814). Alternatively, the three-dimensional data encoding device may shift each divided data (6811 to 6814) by an individual position shift amount and encode them using a common bounding box.

[0571] As shown in Fig. 95 (c), the total shift amount of each segmented data is the common position shift amount.

[0572] In addition, the first bounding box information, which indicates the position and size of the minimum value point of the bounding box (6820) containing the common position shift amount and all division data, is stored in the SPS among the data configurations of the bit stream shown in (d) of FIG. 95.

[0573] Meanwhile, the header of the position information of the corresponding segmented data does not store the individual position shift amount and the second bounding box information used for encoding each segmented data.

[0574] Additionally, a flag (identification information) indicating that the data was encoded using a bounding box containing a common position shift amount and all segmented data (6811~6814), and a flag (identification information) indicating that the individual position shift amount and the size information of the bounding box used to encode the segmented data are not stored in the position information header for each segmented data are stored in the SPS or GPS.

[0575] The three-dimensional data decoding device determines whether the information is encoded as common information or as individual information based on the flag stored in the SPS or GPS, and calculates the size of the position information and bounding box used for decoding.

[0576] In addition, BB represents a bounding box. Also, common information is a common position shift amount common to multiple partitioned data and first BB information. Also, the common position shift amount may be represented by a point of minimum value of the common bounding box. Also, individual information is an individual position shift amount for each partitioned data and second BB information of the bounding box for each partitioned data used for encoding. Also, the individual position shift amount may be represented by a point of minimum value of the bounding box for each partitioned data. Partition area information is information representing the partition boundary in the space when partitioning data, and may include a point of minimum value of BB and BB information representing the size of BB.

[0577] FIG. 96 is a flowchart showing an example of an encoding method when switching between the first example and the second example. FIG. 97 is a flowchart showing an example of a decoding method when switching between the first example and the second example.

[0578] As shown in FIG. 96, the three-dimensional data encoding device determines a common position shift amount of point cloud data (6810) and a common BB surrounding the point cloud data (6810) (S6801).

[0579] The three-dimensional data encoding device determines whether to shift each segmented data (6811 to 6814) individually using an individual position shift amount (S6802). The three-dimensional data encoding device may determine this based on the reduction amount of header information or the result of calculating the encoding efficiency.

[0580] The three-dimensional data encoding device determines that when each segmented data (6811 to 6814) is not shifted individually (No in S6802), it transmits a common position shift amount and a common BB size (S6803), and also determines that it does not transmit an individual position shift amount and a common BB size (S6804). Accordingly, the three-dimensional data encoding device generates a bit stream that includes a common position shift amount and a common BB size, and does not include an individual position shift amount and a individual BB size. Identification information indicating that it is not shifted individually may be stored in the bit stream.

[0581] Meanwhile, the three-dimensional data encoding device determines to transmit a common position shift amount and a common BB size when shifting each segmented data (6811 to 6814) individually (Yes in S6802) (S6805), and also determines to transmit an individual position shift amount and an individual BB size (S6806). Accordingly, the three-dimensional data encoding device generates a bit stream that includes the common position shift amount and the common BB size, and also includes the individual position shift amount and the individual BB size. Identification information indicating that the data is shifted individually may be stored in the bit stream. Additionally, in step S6806, the three-dimensional data encoding device may calculate the individual position shift amount and the individual BB size of each segmented data (6811 to 6814).

[0582] The three-dimensional data decoding device acquires identification information indicating whether individual shifts have been performed by acquiring a bit stream as shown in FIG. 97 (S6811).

[0583] The three-dimensional data decoding device uses identification information to determine whether individual shifts were performed during encoding (S6812).

[0584] If the three-dimensional data decoding device determines that individual shifts for each divided data (6811~6814) have not been performed (No in S6812), it obtains a common position shift amount and a common BB size from the bit stream (S6813).

[0585] Meanwhile, when the three-dimensional data decoding device determines that individual shifts have been performed for each divided data (6811 to 6814) (Yes in S6812), it obtains a common position shift amount and a common BB size from the bit stream (S6814), and also obtains an individual position shift amount and an individual BB size (S6815).

[0586] Next, a third example of position shifting by a position shift amount determined using a partitioned area of ​​the space where point cloud data exists will be explained using FIG. 98. In the third example, the amount of information in the position shift amount can also be reduced.

[0587] FIG. 98 is a diagram illustrating a third example of position shifting. In the third example, the total shift amount of each segmented data (6811 to 6814) is represented as a shift amount of three steps.

[0588] The three-dimensional data encoding device calculates a common position shift amount using a bounding box (6820) as shown in FIG. 98 (a). The method for calculating the common position shift amount is the same as the method described using FIG. 94. At this time, the three-dimensional data encoding device determines a plurality of division regions (6831 to 6834) (Fig. 98 (b)) for dividing the point cloud data (6810) into multiple parts, and divides the point cloud data (6810) into multiple division data (6811 to 6814) according to the determined plurality of division regions (6831 to 6834). Each of the plurality of division regions (6831 to 6834) is a region corresponding to the plurality of division data (6811 to 6814). Additionally, the plurality of division regions (6831 to 6834) are also referred to as division region bounding boxes.

[0589] The three-dimensional data encoding device calculates the direction and distance of a vector represented by the difference between the minimum value point of a bounding box (6820) containing all divided data as shown in FIG. 98 (b) and the minimum value point of each divided area (6831 to 6834) as the position shift amount of the divided area.

[0590] As shown in (c) of FIG. 98, the three-dimensional data encoding device generates a bounding box (6821 to 6824) of a size that includes the segmented data (6811 to 6814) for each of the segmented data (6811 to 6814), and calculates the minimum value point of each generated bounding box (6821 to 6824). Then, for each of the segmented data (6811 to 6814), the three-dimensional data encoding device calculates the direction and distance of a vector represented by the difference between the minimum value point of the bounding box corresponding to the segmented data and the minimum value point of the corresponding segmented area as the individual position shift amount of the segmented data.

[0591] The three-dimensional data encoding device stores bounding box information in a bit stream, which indicates a common position shift amount, an individual position shift amount for each divided data, and the size of the bounding box (6820).

[0592] Additionally, the first bounding box information, which indicates the common position shift amount and the position and size of the minimum point of the bounding box (6820), is stored in the SPS of the bit stream data configuration shown in (e) of FIG. 98. Also, the individual position shift amount is stored in the header of the position information of the corresponding divided data. Also, the divided area information, which includes the position shift amount for each divided area, is stored, for example, in a parameter set where the divided metadata is stored. Also, the second bounding box information of the bounding boxes (6821–6824) used for encoding each divided data (6811–6814) is stored in the header of the position information of the corresponding divided data. Here, each bounding box (6821–6824) used for encoding each divided data (6811–6814) is included in each divided area (6831–6834).

[0593] Here, if the common position shift amount is set to Shift_A, the individual position shift amount is set to Shift_B(i), and the position shift amount of the partitioned area is set to Shift_C(i) (where i is the index of the partitioned data), the shift amount Shift(i) of the partitioned data (i) can be calculated using the following formula.

[0594] Shift(i)=Shift_A+Shift_B(i)+Shift_C(i)

[0595] That is, as shown in (d) of FIG. 98, the total shift amount of each divided data can be calculated by adding three shift amounts: a common position shift amount, a position shift amount of the divided area, and an individual position shift amount.

[0596] The three-dimensional data encoding device shifts the position of the point cloud data of the i-th segmented data by subtracting Shift(i) before encoding the point cloud data.

[0597] A three-dimensional data decoding device can obtain Shift_A, Shift_B(i), and Shift_C(i) from the header of the SPS and the partitioned data, calculate Shift(i), and then return the partitioned data to its original position by adding Shift(i) to the decoded partitioned data (i). By doing so, multiple partitioned data can be correctly decoded.

[0598] This method has the effect of reducing the amount of information regarding the shift amount for each segmented data by representing the individual position shift amount as a difference from the segmented area when transmitting segmented area information.

[0599] Additionally, depending on whether or not the segmented area information is transmitted, the individual position shift amount may be switched between being the difference between the minimum value point of the common bounding box and the minimum value point of the individual bounding box, or the difference between the minimum value point of each segmented area and the minimum value point of the individual bounding box. In the latter case, the individual position shift amount is represented by the sum of the position shift amount of the segmented area and the calculated difference.

[0600] FIG. 99 is a flowchart showing an example of an encoding method in which the first example and the third example are switched when shifting positions individually. FIG. 100 is a flowchart showing an example of a decoding method in which the first example and the third example are switched when shifting positions individually.

[0601] The three-dimensional data encoding device determines a method for dividing point cloud data (6810) as shown in FIG. 99 (S6821). Specifically, the three-dimensional data encoding device determines whether to shift the point cloud data to a first example or to a third example.

[0602] The three-dimensional data encoding device determines whether the method of the third example using a divided region is based on the determined division method (S6822).

[0603] If the three-dimensional data encoding device determines that the method of the first example is the same (No in S6822), it sets the individual position shift amount as the difference between the point of the minimum value of the common BB and the point of the minimum value of the individual BB (S6823).

[0604] The three-dimensional data encoding device generates a bit stream containing common information and individual information (S6824). The bit stream may store identification information indicating that it is the method of the first example.

[0605] If the three-dimensional data encoding device determines that the method of the third example is correct (Yes in S6822), it sets the individual position shift amount to the difference between the point of the minimum value of the divided area BB and the point of the minimum value of the individual BB (S6825).

[0606] The three-dimensional data encoding device transmits a bit stream containing common information, individual information, and divided area information (S6826). The bit stream may store identification information indicating that it is the method of the third example.

[0607] A three-dimensional data decoding device determines whether the bit stream contains partitioned region information by acquiring a bit stream as shown in FIG. 100 (S6831). By doing so, the three-dimensional data decoding device determines whether the acquired bit stream contains point cloud data encoded in the first example or point cloud data encoded in the third example. Specifically, if the bit stream contains partitioned region information, it determines that the bit stream contains point cloud data encoded in the third example, and if it does not contain such information, it determines that the bit stream contains point cloud data encoded in the first example. Additionally, the three-dimensional data decoding device may determine whether the bit stream is encoded in the first example or the third example by acquiring identification information included in the bit stream.

[0608] The three-dimensional data decoding device obtains common information and individual information from the bit stream when the bit stream does not contain partition area information (No in S6831), that is, when encoded as the first example (S6832).

[0609] The three-dimensional data decoding device calculates the position shift amount for each segmented data, i.e., the position shift amount Shift(i) in the first example, the common BB, and the individual BB based on the acquired common information and individual information, and decodes the point cloud data using this information (S6833).

[0610] The three-dimensional data decoding device obtains common information, individual information, and partition information from the bit stream when the bit stream contains partition information (Yes in S6831), that is, when encoded as the third example (S6834).

[0611] The three-dimensional data decoding device calculates the position shift amount for each segmented data, i.e., the position shift amount Shift(i) in the third example, the common BB, the individual BB, and the segmented area based on the acquired common information, individual information, and segmented area information, and decodes the point cloud data using this information (S6835).

[0612] Next, a fourth example is described using FIG. 101, in which a partitioned area of ​​a space containing point cloud data is position-shifted by a position-shift amount determined as a bounding box for encoding. In the fourth example, since the partitioned area and the individual bounding box coincide, the amount of information in the bounding box can be reduced compared to the third example.

[0613] FIG. 101 is a diagram for explaining a fourth example of position shifting. In the fourth example, the total shift amount of each segmented data (6811 to 6814) is represented as a two-stage shift amount of a common position shift amount and a position shift amount of a segmented area.

[0614] The three-dimensional data encoding device calculates a common position shift amount using a bounding box (6820) as shown in FIG. 101 (a). The method for calculating the common position shift amount is the same as the method described using FIG. 94. At this time, the three-dimensional data encoding device divides the point cloud data (6810) into a plurality of segmented data (6811 to 6814). The method for dividing the point cloud data (6810) is the same as the method described using FIG. 98 (a).

[0615] The three-dimensional data encoding device calculates the position shift amount of the divided area as shown in (b) of FIG. 101. The method for calculating the position shift amount of the divided area is the same as the method described using (b) of FIG. 98.

[0616] The three-dimensional data encoding device calculates the position shift amount of the partitioned area as the individual position shift amount of each partitioned data (6811 to 6814). Accordingly, the three-dimensional data encoding device stores bounding box information, which indicates the size of the common position shift amount, the individual position shift amount (position shift amount of the partitioned area), and the bounding box (partitioned area), in a bit stream.

[0617] In addition, the first bounding box information, which indicates the common position shift amount and the position and size of the minimum point of the bounding box (6820), is stored in the SPS among the data configurations of the bit stream shown in (d) of FIG. 101.

[0618] Additionally, the individual position shift amount is stored in at least one of the header of the position information of the corresponding segmented data and the segmented metadata. When the individual position shift amount is stored in either the header of the position information or the segmented metadata, identification information indicating that the individual position shift amount is stored in the header of the position information of the segmented data, or identification information indicating that it is stored in the segmented metadata, may be stored in the GPS or SPS. Alternatively, the individual position shift amount may be stored, for example, in the header of the segmented data, and identification information (flag) indicating whether the position shift amount and bounding box information stored in the header of the segmented data match the segmented area may be stored in the GPS or SPS. Accordingly, the three-dimensional data decoding device determines that the segmented area information is stored in the header of the segmented data based on the flag, and can use the position shift amount and bounding box information stored in the header of the segmented data as information for the segmented area. Alternatively, the above flag may be stored in partition metadata, and the three-dimensional data decoding device may obtain partition area information by referring to the header of the partition data when the above flag indicates 1, that is, when it indicates that the partition area information is stored in the header of the partition data.

[0619] Here, if the common position shift amount is set as Shift_A, the individual position shift amount as Shift_B(i), and the position shift amount of the partitioned area as Shift_C(i) (where i is the index of the partitioned data), the shift amount Shift(i) of the partitioned data (i) can be calculated using the following formula.

[0620] Shift_B(i)=Shift_C(i)

[0621] Shift(i)=Shift_A+Shift_B(i)

[0622] That is, as shown in (c) of FIG. 101, the total shift amount of each divided data can be calculated by adding the common position shift amount and the position shift amount of the divided area (i.e., individual position shift amount).

[0623] The three-dimensional data encoding device shifts the position of the point cloud data of the i-th segmented data by subtracting Shift(i) before encoding the point cloud data.

[0624] A three-dimensional data decoding device can obtain Shift_A, Shift_B(i), or Shift_C(i) from the header of the SPS and the partitioned data, calculate Shift(i), and then return the partitioned data to its original position by adding Shift(i) to the decoded partitioned data (i). By doing so, multiple partitioned data can be correctly decoded.

[0625] This method has the effect of reducing the amount of information by making the individual position shift amount the position shift amount of the divided area, so that even if the transmission of divided area information is required, new divided area information does not need to be transmitted.

[0626] Additionally, depending on whether to transmit segmentation information or not, you may switch whether to use the individual position shift amount as the bounding box information of each segmented data's individual bounding box or as the segmentation information.

[0627] FIG. 102 is a flowchart showing an example of an encoding method in which the third example and the fourth example are switched when storing divided area information.

[0628] The three-dimensional data encoding device determines a method for dividing point cloud data (6810) as shown in FIG. 102 (S6841). Specifically, the three-dimensional data encoding device determines whether to shift the point cloud data to a third example or to a fourth example. The three-dimensional data encoding device may also determine this based on the amount of header information reduction or the result of calculating the encoding efficiency.

[0629] The three-dimensional data encoding device determines whether to perform position shifting using individual bounding boxes based on the determined division method (S6842). That is, the three-dimensional data encoding device determines whether to use the method of the third example or the method of the fourth example.

[0630] If the three-dimensional data encoding device determines that the method of the third example is correct (Yes in S6842), it sets the individual position shift amount to the difference between the point of the minimum value of the divided area BB and the point of the minimum value of the individual BB (S6843).

[0631] The three-dimensional data encoding device generates a bit stream including common information, individual information, and partitioned area information (S6844). The bit stream may store identification information indicating that it is the method of the third example.

[0632] If the three-dimensional data encoding device determines that the method of the fourth example is the same (No in S6842), it sets the individual position shift amount as the difference between the point of the minimum value of the common BB and the point of the minimum value of the divided area BB (S6845).

[0633] The three-dimensional data encoding device transmits a bit stream containing common information and partitioned area information (S6846). The bit stream may store identification information indicating that it is the method of the fourth example.

[0634] Next, a fifth example of position shifting using a position shift amount determined by using a partitioned area of ​​the space where point cloud data exists will be explained using FIG. 103. In the fifth example, the amount of information in the shift amount can be reduced.

[0635] FIG. 103 is a diagram illustrating a fifth example of position shift. In the fifth example, the division area of ​​the third example is different from the third example in that the division area is an equally divided area.

[0636] The three-dimensional data encoding device calculates a common position shift amount using a bounding box (6820) as shown in FIG. 103 (a). The method for calculating the common position shift amount is the same as the method described using FIG. 94. At this time, the three-dimensional data encoding device determines a plurality of division regions (6841 to 6844) (Fig. 103 (b)) for dividing the point cloud data (6810) into multiple parts, for example, based on a bounding box (6820) according to a predetermined rule. For example, when the bounding box (6820) is divided into N parts, the three-dimensional data encoding device can generate four division regions (6841 to 6844) of the same size when N=4. Here, for example, if it is determined that the area number is defined as a molton order as an identifier specifying each divided area (6841 to 6844), since the size of each divided area is the same, the position shift amount of the divided area can be calculated from the number of divisions and the area number. For this reason, the three-dimensional data encoding device may transmit identification information for identifying the area and the number of divisions instead of transmitting the position shift amount of the divided area. As described above, the identification information is a molton order corresponding to each of the plurality of divided areas (6841 to 6844).

[0637] The example of FIG. 103 (c) is an example in which the encoding area is made into individual bounding boxes for each of the multiple segmented data. In this case, the position shift amount of each segmented data is represented as the sum of a common position shift amount, a position shift amount of the corresponding segmented area, and an individual position shift amount from the reference position (position of the minimum point) of the corresponding segmented area. In this case, the corresponding segmented area does not have to include the entire area of ​​the encoding area (i.e., individual BB).

[0638] The example in FIG. 103 (d) is an example in which the encoding area of ​​each segmented data matches each segmented area. In this case, the position shift amount of each segmented data is represented as the sum of the common position shift amount and the position shift amount of the corresponding segmented area. In this case, since the corresponding segmented area matches the encoding area (i.e., individual BB), it includes the encoding area.

[0639] In this way, the three-dimensional data encoding device may or may not match the encoding area of ​​each segmented data with the segmentation area when segmenting the point cloud data. If the encoding area of ​​each segmented data and the segmentation area are matched, either method of FIG. 103 (c) or FIG. 103 (d) may be used. If the encoding area of ​​each segmented data and the segmentation area are not matched, method of FIG. 103 (c) may be used.

[0640] When using the methods of FIG. 103 (c) and FIG. 103 (d), bounding box information indicating the common position shift amount, the position and size of the minimum point of the bounding box (6820), is stored in the SPS among the data configurations of the bit stream. Also, information regarding the division method and the number of divisions, as division area information, are stored in the SPS or GPS, etc., as information common to all division data, and the number (identification information) in a predetermined order (Mortone order) of each division area is stored in the header of the position information of the division data as information of each division area.

[0641] Also, the individual position shift amount is stored in the header of the position information of the corresponding divided data. Also, in the case of (c) of FIG. 103, the individual position shift amount is also stored in the header of the position information of the divided data.

[0642] In addition, when aligning the partition area with the partition data (i.e., aligning the individual bounding boxes of the partition area with the partition data), the partition data number (Tile ID) included in the header of the partition data location information may be treated as a number in a predetermined order for each partition area. In this way, by representing the individual position shift amount as identification information (a number in a predetermined order) for each data, the amount of information in the header can be reduced.

[0643] In addition, when the three-dimensional data encoding device uses the division method of the fifth example, it may represent the position shift amount of the divided area as area information including the number of divisions and identification information of the divided area by using a division method that equally divides a common bounding box and a predetermined order, and when using a different division method, it may switch to representing the position shift amount in a way that does not use the above area information.

[0644] Here, if the common position shift amount is set to Shift_A, the individual position shift amount is set to Shift_B(i), and the position shift amount of the partitioned area is set to Shift_D(i) (where i is the index of the partitioned data), the shift amount Shift(i) of the partitioned data (i) can be calculated using the following formula.

[0645] Shift(i)=Shift_A+Shift_B(i)+Shift_D(i)

[0646] That is, the total shift amount of each partitioned data can be calculated by adding three shift amounts: a common position shift amount, a position shift amount of the partitioned area, and an individual position shift amount.

[0647] The three-dimensional data encoding device shifts the position of the point cloud data of the i-th segmented data by subtracting Shift(i) before encoding the point cloud data.

[0648] A three-dimensional data decoding device obtains Shift_A and Shift_B(i) from the header of the SPS and the divided data, and also obtains information regarding the division method, the number of divisions, and a number in a predetermined order for each divided area as division area information, derives a position shift amount Shift_D(i) by a predetermined method, calculates Shift(i), and then returns the divided data to its original position by adding Shift(i) to the decoded divided data (i). By doing so, multiple divided data can be correctly decoded.

[0649] A specific example of reducing the rules and header amount in the case of partitioning point cloud data into an octal tree is described. FIG. 104 is a diagram for explaining an encoding method in the case of partitioning a three-dimensional space into an octal tree.

[0650] First, the three-dimensional data encoding device may offset (shift position, move) the point cloud data in three-dimensional space by a common position shift amount and then divide it into an octal tree. The three-dimensional data encoding device divides the bounding box (6850) of the point cloud data into eight division regions using an octal tree and sets the number of divisions as the Depth of the octal tree. For example, the number of divisions for Depth is assigned as N=2^(Depth*3), the number of divisions when Depth=1 is 8, and the number of divisions when Depth=2 is 64. Also, the order of the division regions is in Morton order. The position information of the division regions can be calculated from the Morton order by applying the fifth example to three-dimensional space. Additionally, the Morton order has the characteristic that it can be calculated from the position information of the division regions in a predetermined way.

[0651] Bounding box information of point cloud data is stored in an SPS or GPS containing common metadata from multiple segmented data. The bounding box information includes the point (initial position) and size of the minimum value of the bounding box.

[0652] The three-dimensional data encoding device stores identification information indicating that the partitioning method is partitioned using an octal tree, and, in the case of octal tree partitioning, depth information indicating the depth of the octal tree, in the SPS or GPS of the data configuration of the bit stream shown in (e) of FIG. 104. In the header of each partitioned data, a number in the order of Moltons is stored as the partitioned data number. Also, in the case of the partitioning method being an octal tree, the position shift amount of the partitioned data and the bounding box information of the encoding are derived from the order of Moltons and are not stored in the bit stream.

[0653] That is, as shown in (f) of FIG. 104, the three-dimensional data encoding device calculates the location information of each of the multiple partitioned regions divided into an octal tree using a predetermined method, the identification information indicating that it is divided into an octal tree, the depth of the octal tree, and the Morton order. In the three-dimensional data decoding device, as shown in (g) of FIG. 104, the identification information indicating that it is divided into an octal tree, the depth of the octal tree, and the Morton order are obtained, and the location information of each of the multiple partitioned regions divided into an octal tree is restored using a predetermined method.

[0654] In addition, if there is no point cloud data within a partitioned area, the information regarding that partitioned area is not required. For example, if there is no point cloud data in the area where partition data number = 2, the partition data number is not transmitted, so the sequence of partition data numbers skips 2 and becomes 1, 3, and 4.

[0655] Additionally, the partition metadata may store partition data numbers, and may store all partition information, including partition regions without point cloud data. In this case, the partition information may indicate whether point cloud data exists in the corresponding partition region.

[0656] FIG. 105 is a diagram showing an example of GPS syntax.

[0657] octree_partition_flag is a flag indicating whether the partitioning method of the point cloud data is octree partitioning.

[0658] depth represents the depth of the octal tree partition when partitioning point cloud data into an octal tree.

[0659] gheader_BBmin_present_flag is a flag indicating whether the location information field of the bounding box of the point cloud data encoding exists in the location information header.

[0660] gheader_BBsize_present_flag is a flag indicating whether the size information field of the bounding box of the point cloud data encoding exists in the position information header.

[0661] Also, if octree_partition_flag=1, gheader_BBmin_present_flag and gheader_BBsize_present_flag are set to 0.

[0662] FIG. 106 is a diagram showing an example of the syntax of a header of location information.

[0663] partition_id represents identification information of the partitioned data. In the case of octal tree partitioning, partition_id represents a specific position in Morton order.

[0664] BBmin represents the amount of shift when encoding data or partitioned data.

[0665] BBsize represents the size of the bounding box when encoding data or segmented data.

[0666] In addition, although the partitioning of an octal tree was explained as an example, this method can be applied to other partitioning methods by determining a predetermined partitioning method, the order of partitioned data, or a method for calculating position information. For example, when viewing point cloud data from above and partitioning it into equal intervals in the xy plane, the partitioning method is the above partitioning method, and the number of partitions, the size of the partitions, and the order are determined and transmitted. Based on this information, both the three-dimensional data encoding device and the three-dimensional data decoding device can calculate position information or position shift amounts, and the amount of data can be reduced by not transmitting position information. Information on whether to partition using an octal tree, information on the partitioning plane, or information on partitioning using a quaternary tree may be transmitted.

[0667] FIG. 107 is a flowchart illustrating an example of an encoding method that switches processing depending on whether or not an octal tree is split. FIG. 108 is a flowchart illustrating an example of a decoding method that switches processing depending on whether or not an octal tree is split.

[0668] The three-dimensional data encoding device determines a method for partitioning point cloud data as shown in FIG. 107 (S6851).

[0669] The three-dimensional data encoding device determines whether to perform octal tree partitioning based on the determined partitioning method (S6852).

[0670] When the three-dimensional data encoding device determines that it does not divide the octal tree (No in S6852), it determines the individual position shift amount as the minimum value point of each individual BB for each divided data, shifts the position of the divided data, and encodes the divided data using the individual BB (S6853).

[0671] The three-dimensional data encoding device stores information of the common BB in the common metadata (S6854).

[0672] The three-dimensional data encoding device stores the individual position shift amount for each segmented data in the header of the segmented data (S6855).

[0673] When the three-dimensional data encoding device determines that it divides the octal tree (Yes in S6852), it determines the individual position shift amount as the minimum value point of the divided area of ​​the octal tree, shifts the position of the divided data, and encodes the divided data using the divided area of ​​the octal tree (S6856).

[0674] The three-dimensional data encoding device stores information of a common BB, identification information indicating an octal tree partition, and depth information in common metadata (S6857).

[0675] The three-dimensional data encoding device stores sequence information in the header of the divided data that indicates the sequence number of the molton order for specifying the individual position shift amount for each divided data (S6858).

[0676] The three-dimensional data decoding device obtains information indicating the partitioning method of the point cloud data from common metadata (S6861).

[0677] The three-dimensional data decoding device determines whether the division method is an octal tree division based on information indicating the acquired division method (S6862). Specifically, the three-dimensional data decoding device determines whether the division method is an octal tree based on identification information indicating whether the division method is an octal tree division.

[0678] When the partitioning method is not octal tree partitioning (No in S6862), the three-dimensional data decoding device obtains information on common BBs, individual position shift amounts, and information on individual BBs, and decodes the point cloud data (S6863).

[0679] When the partitioning method is octal tree partitioning (Yes in S6862), the three-dimensional data decoding device obtains information on common BBs, depth information, and individual order information, calculates individual position shift amounts and BB information of encoding, and decodes point cloud data (S6864).

[0680] With the multiple examples of methods described in this embodiment, it is possible to expect the possibility of reducing the code amount regardless of which method is used. It is possible to switch to any one of the multiple examples of methods using a predetermined method.

[0681] For example, a three-dimensional data encoding device may calculate a code amount and, based on the calculated code amount, decide to perform the first method among the plurality of methods above under predetermined conditions. Alternatively, in the case of irreversible compression, it may determine whether the quantization coefficient is greater than a predetermined value, whether the number of data divisions is greater than a predetermined number, or whether the number of point groups is less than a predetermined number, and switch from the first method among the plurality of methods above to the second method if there is a possibility that the overhead will change.

[0682] Additionally, in the header of each partitioned data, the difference with respect to common information is stored as individual information corresponding to the partitioned data; however, this is not limited to this, and the difference with the individual information of the partitioned data one preceding the partitioned data may also be stored as the individual information of the partitioned data.

[0683] FIG. 109 is a diagram showing an example of the data structure of a bit stream when partitioned data is classified into randomly accessible data A and non-randomly accessible data B. FIG. 110 is an example of the partitioned data of FIG. 109 being framed.

[0684] In this case, that is, when there are one or more random access units, different information may be stored in the data header of data A and the data header of data B, respectively. For example, the partition data of data A may store individual difference information from common information stored in GPS, and the partition data of data B may store difference information with respect to data A in the random access unit. Furthermore, when partition data of data B is included in multiple random access units, difference information with respect to data A may be stored in each of the partition data of multiple B data included in the same random access unit, and difference information with respect to data A or B data one preceding the said B data may be stored in the partition data of multiple B data.

[0685] Above, we explained partitioned data, and the same applies to frames.

[0686] In this embodiment, the partition area or partition boundary in data partitioning is mainly described using an example (Fig. 111) in which the area is partitioned by targeting the BB of all partitioned data, but even if the partition area is in a different case, the same effect of reducing the code amount can be expected by using the method of this embodiment.

[0687] As shown in FIG. 111, when dividing a BB containing all divided data, the three-dimensional data encoding device may make the point cloud data after the position shift the target of the division by shifting the position based on the minimum value of the BB. In addition, if the point cloud data is scaled or quantized, the three-dimensional data encoding device may make the point cloud data after scaling or quantization the target of the division, or may make the point cloud data before scaling or quantization the target of the division.

[0688] As shown in FIG. 112, the three-dimensional data encoding device may set the division area in the coordinate system of the input data of the point cloud data. In this case, the three-dimensional data encoding device does not shift the point of the minimum value of the BB of the point cloud data.

[0689] As shown in FIG. 113, the three-dimensional data encoding device may set the division area in a coordinate system above the point cloud data. For example, the three-dimensional data encoding device may set the division area based on GPS coordinates such as map data.

[0690] In this case, the three-dimensional data encoding device may store relative position information of the point cloud coordinate system with respect to the upper coordinate system in a bit stream and transmit it. For example, when a sensor such as an on-board Lidar senses point cloud data while moving, the three-dimensional data encoding device may transmit the sensor's position information (GPS coordinates, acceleration, velocity, distance traveled) as relative position information. In addition, in the case of point cloud data having a time-series frame structure, the three-dimensional data encoding device may store time-series sensor position information for each frame in a bit stream and transmit it. Furthermore, the upper coordinate system may be an absolute coordinate system or a relative coordinate system based on an absolute coordinate system. The upper coordinate system may be a coordinate system even higher than the upper coordinate system.

[0691] As shown in FIG. 114, the three-dimensional data encoding device may determine the segmentation area based on the object or data attribute of the point cloud data. For example, the three-dimensional data encoding device may determine the segmentation area based on the image recognition result. In this case, the segmentation area may include overlapping areas. The three-dimensional data encoding device may cluster each point of the point cloud based on a predetermined attribute, divide the data based on the number of points, or divide the area to have approximately the same number of point clouds.

[0692] A three-dimensional data encoding device signals the number of points to be encoded, and a three-dimensional data decoding device may decode point cloud data using the number of points.

[0693] The encoding target is set for each frame or for each segmented data segment of a frame, and the number of encoding target points is basically stored in each data header. For the number of encoding target points to be signaled, a reference value may be stored in common metadata, and the difference information with respect to the reference value may be stored in the data header.

[0694] A three-dimensional data encoding device may, for example, store a reference number of points A (i.e., reference value) in a GPS or SPS as shown in FIG. 115, and store a number of points B(i) (where i is the index of the segmented data) which is the difference from the reference number of points A in a respective data header. The number of points of the segmented data (i) is obtained by adding B(i) to A. Accordingly, a three-dimensional data decoding device calculates A+B(i) and decodes the calculated value as the number of points of the segmented data (i).

[0695] Alternatively, as shown in FIG. 116, the three-dimensional data encoding device may store difference information from the number of points of the segmented data prior to the segmented data in each data header. In this case, the number of points of the segmented data (i) is obtained by adding the number of segmented data B (1) to B (i) of the leading segmented data to the number A.

[0696] In addition, for example, in the case of time-series point cloud data having a plurality of frame structures as shown in FIG. 117, the three-dimensional data encoding device may store a reference value for the number of points to be encoded in a common SPS, and store the relative value of each frame relative to the reference value in the GPS or data header. In addition, the three-dimensional data encoding device may store the difference (relative value) relative to the reference value stored in the SPS in the GPS, and additionally store the difference (relative value) from the sum of the reference value of the SPS and the difference value of the GPS in the data header. By doing so, the effect of reducing the overhead encoding amount can be expected.

[0697] In addition, as a method for partitioning point cloud data, a method of partitioning such that the number of points becomes approximately equal can be considered. In that case, the same method as in Fig. 115 can be used.

[0698] For example, as shown in FIG. 118, in a method of dividing such that the difference between points included in the divided data is within 1, Δ may be represented as 1 bit of data, and when Δ=0, the difference information does not need to be represented.

[0699] For example, as shown in FIG. 119, the data may basically consist of a reference value recorded in the GPS as a division number and a remainder number. When Δ=0, the difference information does not need to be displayed.

[0700] Any of the methods shown in FIGS. 115 to 119 above can be used to achieve the effect of reducing the amount of overhead information.

[0701] As described above, the three-dimensional data encoding device according to the present embodiment performs the processing shown in FIG. 120. The three-dimensional data encoding device is a device that encodes point cloud data representing a plurality of three-dimensional positions in a three-dimensional space. The three-dimensional data encoding device shifts the point cloud data by a first shift amount (S6871). Next, the three-dimensional data encoding device divides the point cloud data into a plurality of sub-point cloud data by dividing the three-dimensional space into a plurality of sub-spaces (S6872). For each of the plurality of sub-point cloud data included in the point cloud data after shifting by the first shift amount, the three-dimensional data encoding device shifts the said sub-point cloud data by a second shift amount based on the position of the sub-space in which said sub-point cloud data is included (S6873). The three-dimensional data encoding device generates a bit stream by encoding the plurality of sub-point cloud data after shifting (S6874). The bit stream includes first movement information for calculating a first movement amount and a plurality of second movement information for calculating a plurality of second movement amounts, each of which is a plurality of sub-point cloud data. According to this, since the divided sub-point cloud data is encoded after being moved, the amount of information regarding the position information of each sub-point cloud data can be reduced, thereby improving encoding efficiency.

[0702] For example, multiple subspaces have the same size. Each of the multiple second movement information includes the number of multiple subspaces and first identification information for identifying the corresponding subspace. Because of this, the amount of information in the second movement information can be reduced, thereby improving encoding efficiency.

[0703] For example, the first identification information is a molton order corresponding to each of the plurality of subspaces.

[0704] For example, a plurality of subspaces are each spaces in which a single three-dimensional space is partitioned using an octal tree. The bit stream includes second identification information indicating that a plurality of subspaces are spaces partitioned using an octal tree, and depth information indicating the depth of the octal tree. Because of this, since point cloud data in a three-dimensional space is partitioned using an octal tree, the amount of location information of each subpoint cloud data can be reduced, and encoding efficiency can be improved.

[0705] For example, partitioning is performed after shifting the point cloud data by a first shift amount.

[0706] For example, a three-dimensional data encoding device is equipped with a processor and memory, and the processor uses the memory to perform the above processing.

[0707] In addition, the three-dimensional data decoding device according to the present embodiment performs the processing shown in FIG. 121. The three-dimensional data decoding device decodes from a bit stream the point cloud data representing multiple three-dimensional positions by dividing a three-dimensional space into multiple sub-spaces, the multiple sub-point cloud data each having been moved by a first movement amount and a corresponding second movement amount, first movement information for calculating the first movement amount, and multiple second movement information for calculating each of the multiple second movement amounts that moved the multiple sub-point cloud data (S6881). The three-dimensional data decoding device restores the point cloud data by moving each of the multiple sub-point cloud data by a movement amount obtained by adding the first movement amount and the corresponding second movement amount (S6882). According to this, the point cloud data can be correctly decoded using a bit stream with improved encoding efficiency.

[0708] For example, multiple subspaces have the same size. Each of the multiple second movement information includes the number of multiple subspaces and first identification information for identifying the corresponding subspace. Because of this, the amount of information in the second movement information can be reduced, thereby improving encoding efficiency.

[0709] For example, the first identification information is a molton order corresponding to each of the plurality of subspaces.

[0710] For example, a plurality of subspaces are each spaces in which a single three-dimensional space is partitioned using an octal tree. The bit stream includes second identification information indicating that a plurality of subspaces are spaces partitioned using an octal tree, and depth information indicating the depth of the octal tree. Because of this, since point cloud data in a three-dimensional space is partitioned using an octal tree, the amount of location information of each subpoint cloud data can be reduced, and encoding efficiency can be improved.

[0711] For example, a three-dimensional data decoding device is equipped with a processor and memory, and the processor uses the memory to perform the above processing.

[0712] (Embodiment 9)

[0713] In this embodiment, examples of methods for calculating individual position shift amounts and divided area position shift amounts using the above-described divided area information are explained.

[0714] FIG. 122 is a diagram illustrating examples of individual position shift amounts and segmented area position shift amounts. As shown in FIG. 122, the difference between the minimum coordinate of the bounding box of the point cloud data (three-dimensional data) and the origin is the common position shift amount. In addition, the bounding box of the point cloud data is divided into a plurality of segmented areas. In addition, the difference between the minimum coordinate of the bounding box of the point cloud data and the minimum coordinate of the segmented area is the segmented area position shift amount. In addition, the difference between the minimum coordinate of the segmented area and the minimum coordinate of the bounding box of the segmented data, which is the point cloud data included in the segmented area, is the individual position shift amount.

[0715] In addition, an example is shown here in which a common position shift amount and a divided area position shift amount are defined individually, but a common position shift amount may not exist. Also, the sum of the common position shift amount and the divided area position shift amount shown in FIG. 122 may be the divided area position shift amount. Also, the vector formed by adding the divided area position shift amount and the individual position shift amount shown in FIG. 122 may be the individual position shift amount.

[0716] FIG. 123 is a flowchart of the calculation process for the position shift amount of a divided area and the individual position shift amount according to the present embodiment. First, the three-dimensional data encoding device divides the point cloud data into slices, which are divided data (S7501). Additionally, any division method may be used. Also, multiple divided areas may not overlap, or may overlap.

[0717] Next, the three-dimensional data encoding device determines, for example, that the reference position (origin coordinates) of the slice encoding is set to the minimum value (minimum coordinates) of the bounding box of the point group (three-dimensional point) data constituting each slice (S7502).

[0718] Next, the three-dimensional data encoding device selects a target slice to be processed from a plurality of slices and starts processing for each slice (S7503). First, the three-dimensional data encoding device calculates the minimum position coordinates of the bounding box of the point cloud data (divided data) constituting the target slice (S7504). Here, the individual position shift amount is the difference between the minimum position coordinates of the bounding box of the point cloud data constituting the target slice and the minimum coordinates of the bounding box of the point cloud data (i.e., common position shift amount), and corresponds to the sum of the divided area position shift amount and the individual position shift amount shown in FIG. 122.

[0719] Next, the three-dimensional data encoding device divides the minimum position coordinates into division boundaries (S7505). For example, the division boundaries may be common to all axes (x, y, z) of all slices, or other boundaries may be used. Also, the division boundaries may be common within the slices, or common to two or more arbitrary axes.

[0720] FIG. 124 is a diagram schematically illustrating the division process of minimum position coordinates. As shown in FIG. 124, the minimum position coordinates are divided into upper bits and lower bits by the division boundary.

[0721] Also, for example, the number of upper bits representing the partition boundary is set to int(k*N). Here, N is the number of bits of the minimum position coordinate. Also, k (k <= 1) is a coefficient that is set either fixedly or adaptively. Accordingly, the partition boundary is determined based on the number of bits of the minimum position coordinate. In this way, by partitioning into upper bits and lower bits, the minimum position coordinate can be divided into a partition area position shift amount and an individual position shift amount.

[0722] Next, the three-dimensional data encoding device performs processing of the upper bits (S7506). Specifically, the three-dimensional data encoding device masks the lower bits to 0 (S7511). By doing so, the number of bits can be reduced. Next, the three-dimensional data encoding device extracts the upper bits and encodes the extracted upper bits (S7512).

[0723] Also, the three-dimensional data encoding device performs processing of the lower bits (S7507). Specifically, the three-dimensional data encoding device masks the upper bits to 0 (S7521). Next, the three-dimensional data encoding device extracts the lower bits and encodes the extracted lower bits (S7522).

[0724] If the processing of all slices is not completed (No in S7508), the three-dimensional data encoding device selects the next slice as the target slice (S7503) and performs the processing after step S7504 again. If the processing of all slices is completed (Yes in S7508), the three-dimensional data encoding device terminates the processing.

[0725] In addition, for example, the three-dimensional data encoding device moves (shifts) the positions of multiple three-dimensional points within a target slice according to the minimum position coordinates (the sum of the division area position shift amount and the individual position shift amount), and encodes the positions of the multiple three-dimensional points after the movement. That is, the three-dimensional data encoding device encodes the difference between the positions of multiple three-dimensional points within the target slice and the minimum position coordinates.

[0726] FIG. 125 is a diagram showing an example of calculating a shift amount in a three-dimensional data decoder. The three-dimensional data decoder decodes a lower bit and an upper bit from a bit stream and calculates a minimum position coordinate by concatenating the obtained lower bit and upper bit.

[0727] Furthermore, for example, the three-dimensional data decoder decodes the positions of multiple three-dimensional points after shifting from a bit stream. The three-dimensional data decoder shifts the positions of multiple three-dimensional points after shifting within a target slice according to the calculated minimum position coordinates (the sum of the division area position shift amount and the individual position shift amount) and restores the positions of the multiple three-dimensional points. Additionally, the shift performed by the three-dimensional data decoder is a shift in the reverse direction of the shift performed by the three-dimensional data encoding device.

[0728] Next, the method for calculating the individual position shift amount and the partition area position shift amount when using the octal tree partitioning method is explained. FIG. 126 is a flowchart of the process for calculating the partition area position shift amount and the individual position shift amount in this case. In addition, the process shown in FIG. 126 differs from the process shown in FIG. 123 in that S7501 is changed to S7501A and S7505 is changed to S7505A.

[0729] First, the three-dimensional data encoding device divides the point cloud data into slices using an octree partitioning method (S7501A). Additionally, steps S7502 to S7504 are the same as those in FIG. 123.

[0730] Next, the three-dimensional data encoding device divides the minimum position coordinates into division boundaries based on the depth bit (S7505A). For example, the division boundary is set at the depth bit position from the top. Here, the depth bit represents the depth of the hierarchy in the octal tree.

[0731] FIG. 127 is a diagram schematically illustrating the division process of the minimum position coordinates in this case. As shown in FIG. 127, the division boundary is set to the boundary of the depth bit from the upper.

[0732] For example, division is performed on all slices and all axes using division boundaries based on a common depth bit. Accordingly, the minimum position coordinates with the lower bit masked to a value of 0 represent the amount of position shift in the division area. Also, the value of the upper bit is equivalent to the Morton order representing the order of the division area. Furthermore, the probability of the lower bit becoming 0 is increased, and an improvement in the coding rate can be expected. In addition, the processing after step S7506 is the same as in FIG. 123.

[0733] In addition, the three-dimensional data encoding device may further divide either the upper bit or the lower bit, or both, into upper and lower bits and perform processing in multiple stages.

[0734] FIG. 128 is a diagram illustrating an example of calculating a shift amount in a three-dimensional data decoder. The three-dimensional data decoder decodes a lower bit and an upper bit from a bit stream and calculates a minimum position coordinate by concatenating the obtained lower bit and upper bit.

[0735] Next, a method for transmitting boundary information indicating the division boundary is explained. The boundary information may indicate the number of bits in the upper bit or the number of bits in the lower bit.

[0736] When individual partition boundaries (number of bits) are used in slices or axes, the boundary information represents each partition boundary (number of bits). When a common partition boundary is used, common boundary information representing the common partition boundary may be stored in a bit stream for multiple slices. Alternatively, boundary information representing the partition boundary of the corresponding slice or axis may be formed for each slice or axis, and the same partition boundary may be represented in multiple boundary information. Furthermore, which of these methods is used may be switched based on a flag.

[0737] In addition, the three-dimensional data encoding device may compare the number of bits of the upper bit and the lower bit, and the boundary information may represent the number of bits of the bit with the smaller number of bits between the upper bit and the lower bit. In that case, the three-dimensional data encoding device stores a flag in the bit stream indicating whether the number of bits represented in the boundary information is the number of bits of the upper bit or the number of bits of the lower bit. The three-dimensional data decoding device determines whether the number of bits represented in the boundary information is the number of bits of the upper bit or the number of bits of the lower bit based on the said flag, and switches the method of calculating the position shift amount.

[0738] In addition, when the three-dimensional data encoding device encodes and transmits lower bits, the three-dimensional data decoding device can calculate the number of lower bits based on the said lower bits, so it is not necessary to transmit the number of lower bits.

[0739] In addition, the three-dimensional data encoding device may switch the method of determining the division boundary (number of bits of the upper or lower bits) depending on the division method. For example, when octal tree division is used, the three-dimensional data encoding device uses depth bits, and when other division methods are used, for example, the number of bits of the upper bits is set to int(k*N).

[0740] The following describes another example of calculating the shift amount. A three-dimensional data encoding device may transmit the upper bits but not the lower bits. For example, when the three-dimensional data encoding device aligns the partitioned area with the en...

Claims

Claim 1 A three-dimensional data encoding method for encoding point cloud data representing multiple three-dimensional positions in a three-dimensional space, wherein the point cloud data is divided into multiple sub-point cloud data by dividing the three-dimensional space into multiple sub-spaces, each of the multiple sub-point cloud data is moved according to a predetermined amount of movement, and the bit stream is generated by encoding the multiple sub-point cloud data after movement, wherein the bit stream includes first control information that includes first information common to the amount of movement of each of the multiple sub-point cloud data, and multiple second control information for each sub-point cloud data. Claim 2 A three-dimensional data encoding method according to claim 1, wherein the displacement amount is a displacement amount based on either the sub-point cloud data or the sub-space containing the sub-point cloud data. Claim 3 A three-dimensional data encoding method according to claim 1 or claim 2, wherein the displacement amount includes an upper bit and a lower bit, the first information indicates the number of bits of the lower bit common to the displacement amount of each of the plurality of sub-point cloud data, and each of the plurality of second control information includes second information indicating the value of the upper bit included in the displacement amount for the sub-point cloud data corresponding to the second control information. Claim 4 A three-dimensional data encoding method according to claim 3, wherein the first control information includes a flag indicating whether information indicating the number of bits of the lower bit is included in the first control information or in each of the plurality of second control information, and when the flag indicates that information indicating the number of bits of the lower bit is included in the first control information, the first control information includes the first information, and each of the plurality of second control information does not include information indicating the number of bits of the lower bit, and when the flag indicates that information indicating the number of bits of the lower bit is included in each of the plurality of second control information, each of the plurality of second control information includes third information indicating the number of bits of the lower bit included in the shift amount for the sub-point cloud data corresponding to the second control information, and the first control information does not include the first information. Claim 5 A three-dimensional data encoding method according to claim 3, wherein all bits included in the lower bit have a value of 0, and the bit stream does not include information indicating the value of the lower bit. Claim 6 A three-dimensional data decoding method comprising: decoding from a bit stream multiple sub-point cloud data after movement, wherein point cloud data representing multiple three-dimensional positions is divided into multiple sub-spaces, and the multiple sub-point cloud data are divided into multiple sub-point cloud data according to a predetermined amount of movement; obtaining first information common to the amount of movement of each of the multiple sub-point cloud data from first control information common to the multiple sub-point cloud data included in the bit stream; calculating the multiple amounts of movement of the multiple sub-point cloud data using the first information; and restoring the multiple sub-point cloud data by moving each of the decoded multiple sub-point cloud data according to the amount of movement corresponding to the sub-point cloud data. Claim 7 A three-dimensional data decoding method according to claim 6, wherein the displacement amount is a displacement amount based on either the sub-point cloud data or the sub-space containing the sub-point cloud data. Claim 8 The three-dimensional data decoding method according to claim 6 or claim 7, wherein the displacement amount includes an upper bit and a lower bit, the first information indicates the number of bits of the lower bit common to the displacement amount of each of the plurality of sub-point cloud data, and the three-dimensional data decoding method further acquires a plurality of second information indicating the value of the upper bit of the displacement amount of each of the plurality of sub-point cloud data from a plurality of second control information for each of the sub-point cloud data included in the bit stream, and in the calculation of the displacement amount, calculates a plurality of the displacement amounts of the plurality of sub-point cloud data using the first information and the plurality of second information. Claim 9 A three-dimensional data decoding method according to claim 8, wherein the first control information includes a flag indicating whether information indicating the number of bits of the lower bit is included in the first control information or in each of the plurality of second control information, and when the flag indicates that information indicating the number of bits of the lower bit is included in the first control information, the first information is obtained from the first control information, and the plurality of shift amounts of the plurality of sub-point cloud data are calculated using the first information and the plurality of second information, and when the flag indicates that information indicating the number of bits of the lower bit is included in each of the plurality of second control information, the plurality of third information indicating the number of bits of the lower bit of the shift amount of each of the plurality of sub-point cloud data is obtained from each of the plurality of second control information, and the plurality of shift amounts of the plurality of sub-point cloud data are calculated using the plurality of third information and the plurality of second information. Claim 10 A three-dimensional data decoding method according to claim 8, wherein in calculating the amount of movement, all bits included in the lower bit are set to a value of 0. Claim 11 A three-dimensional data encoding device for encoding point cloud data representing multiple three-dimensional positions in a three-dimensional space, comprising a processor and a memory, wherein the processor uses the memory to divide the point cloud data into multiple sub-point cloud data by dividing the three-dimensional space into multiple sub-spaces, moves each of the multiple sub-point cloud data according to a predetermined amount of movement, and generates a bit stream by encoding the multiple sub-point cloud data after movement, wherein the bit stream includes first control information including first information common to the amount of movement of each of the multiple sub-point cloud data, and multiple second control information for each sub-point cloud data. Claim 12 A three-dimensional data decoding device comprising a processor and a memory, wherein the processor uses the memory to decode from a bit stream a plurality of sub-point cloud data after movement, wherein the portion of sub-point cloud data representing a plurality of three-dimensional positions is divided by dividing a three-dimensional space into a plurality of sub-spaces, and the portion of sub-point cloud data is moved according to a predetermined amount of movement, the processor obtains first information common to the amount of movement of each of the plurality of sub-point cloud data from first control information common to the plurality of sub-point cloud data included in the bit stream, calculates a plurality of the amount of movement of the plurality of sub-point cloud data using the first information, and restores the plurality of sub-point cloud data by moving each of the decoded portion of sub-point cloud data after movement according to the amount of movement corresponding to the sub-point cloud data.

Images