Three-dimensional data encoding methods, three-dimensional data decoding methods, three-dimensional data encoding devices, and three-dimensional data decoding devices.

Abstract

The invention relates to a method for encoding three-dimensional data comprising: acquiring an attribute information item of a three-dimensional point (S10621); performing an encoding operation which performs a transformation operation on the arithmetic value indicated by the attribute information item and encodes the attribute information item or encodes the attribute information item without performing a transformation operation, performing a transformation operation which performs at least one of the shift or scaling operations, performing a scaling operation which performs at least one of the multiplication and division or shift operations, performing a shift operation which performs addition and subtraction (S10622); and generating a bit stream which includes the encoded attribute information item and a transformation identifier item which indicates whether a transformation operation has been performed (S10623).

Description

Three-dimensional data encoding method, three-dimensional data decoding method, three-dimensional data encoding device, and three-dimensional data decoding device

[0001] The present disclosure relates to a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, and a three-dimensional data decoding device.

[0002] In the future, devices and services that utilize 3D data are expected to become widespread in a wide range of fields, including computer vision for autonomous operation of automobiles or robots, map information, surveillance, infrastructure inspection, video distribution, etc. 3D data can be acquired in a variety of ways, including distance sensors such as range finders, stereo cameras, or a combination of multiple monocular cameras.

[0003] One method of representing three-dimensional data is a representation called a point cloud, which represents the shape of a three-dimensional structure using a group of points in three-dimensional space. In a point cloud, the position and color of the points are stored. Point clouds are expected to become the mainstream method of representing three-dimensional data, but point clouds have a very large amount of data. Therefore, when storing or transmitting three-dimensional data, it is essential to compress the amount of data by encoding, just as with two-dimensional moving images (examples include MPEG-4 AVC or HEVC standardized by MPEG).

[0004] Furthermore, compression of point clouds is partially supported by a public library (Point Cloud Library) that performs processing related to point clouds.

[0005] Furthermore, a technique is known in which three-dimensional map data is used to search for and display facilities located around a vehicle (see, for example, Patent Document 1).

[0006] International Publication No. 2014 / 020663

[0007] It is desirable to be able to encode three-dimensional data appropriately.

[0008] An object of the present disclosure is to provide a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, or a three-dimensional data decoding device that can appropriately encode three-dimensional data.

[0009] A three-dimensional data encoding method according to one aspect of the present disclosure acquires attribute information of a three-dimensional point, performs a conversion process on a numerical value indicated by the attribute information, such as scaling, which involves performing at least one of multiplication / division and shift operations, and offsetting, which involves adding / subtracting, and then encodes the attribute information, or performs an encoding process that encodes the attribute information without performing the conversion process, and generates a bitstream that includes the encoded attribute information and conversion identification information that indicates whether the conversion process has been performed.

[0010] A three-dimensional data decoding method according to one aspect of the present disclosure acquires a bitstream including coded attribute information of three-dimensional points and coded transformation identification information indicating whether or not a transformation process has been performed on the attribute information, the transformation process performing scaling, which involves at least one of multiplication / division and shift operations, and offsetting, which involves addition / subtraction, and the like; decodes the coded attribute information; and decodes the coded transformation identification information.

[0011] The present disclosure can provide a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, or a three-dimensional data decoding device that can appropriately encode three-dimensional data.

[0012] FIG. 1 is a diagram showing a configuration of a three-dimensional data encoding / decoding system according to the first embodiment. FIG. 2 is a diagram showing an example of the configuration of point cloud data according to the first embodiment. FIG. 3 is a diagram showing an example of the configuration of a data file in which point cloud data information according to the first embodiment is described. FIG. 4 is a diagram showing types of point cloud data according to the first embodiment. FIG. 5 is a diagram showing a configuration of a first encoding unit according to the first embodiment. FIG. 6 is a block diagram of a first encoding unit according to the first embodiment. FIG. 7 is a diagram showing a configuration of a first decoding unit according to the first embodiment. FIG. 8 is a block diagram of a first decoding unit according to the first embodiment. FIG. 9 is a block diagram of a three-dimensional data encoding device according to the first embodiment. FIG. 10 is a diagram showing an example of position information according to the first embodiment. FIG. 11 is a diagram showing an example of an octree representation of position information according to the first embodiment. FIG. 12 is a block diagram of a three-dimensional data decoding device according to the first embodiment. FIG. 13 is a block diagram of an attribute information encoding unit according to the first embodiment. FIG. 14 is a block diagram of an attribute information decoding unit according to the first embodiment. FIG. 15 is a block diagram showing the configuration of an attribute information encoding unit according to the first embodiment. FIG. 16 is a block diagram of an attribute information encoding unit according to the first embodiment. FIG. 17 is a block diagram showing the configuration of an attribute information decoding unit according to the first embodiment. FIG. 18 is a block diagram of an attribute information decoding unit according to the first embodiment. FIG. 19 is a diagram showing the configuration of a second encoding unit according to the first embodiment. FIG. 20 is a block diagram of a second encoding unit according to the first embodiment. FIG. 21 is a diagram showing the configuration of a second decoding unit according to the first embodiment. FIG. 22 is a block diagram of a second decoding unit according to the first embodiment. FIG. 23 is a diagram showing a protocol stack related to PCC encoded data according to the first embodiment. FIG. 24 is a diagram showing the configurations of an encoding unit and a multiplexing unit according to the second embodiment. FIG. 25 is a diagram showing an example configuration of encoded data according to the second embodiment. FIG. 26 is a diagram showing example configurations of encoded data and NAL units according to the second embodiment. FIG. 27 is a diagram showing example semantics of pcc_nal_unit_type according to the second embodiment. FIG. 28 is a diagram showing an example transmission order of NAL units according to the second embodiment. FIG. 29 is a flowchart of processing by the 3D data encoding device according to the second embodiment.FIG. 30 is a flowchart of processing by a 3D data decoding device according to Embodiment 2. FIG. 31 is a flowchart of multiplexing processing according to Embodiment 2. FIG. 32 is a flowchart of demultiplexing processing according to Embodiment 2. FIG. 33 is a diagram showing an example of 3D points according to Embodiment 3. FIG. 34 is a diagram showing an example of LoD setting according to Embodiment 3. FIG. 35 is a diagram showing an example of thresholds used for setting LoD according to Embodiment 3. FIG. 36 is a diagram showing example attribute information used for predicted values ​​according to Embodiment 3. FIG. 37 is a diagram showing an example of an Exponential-Golomb code according to Embodiment 3. FIG. 38 is a diagram showing processing for the Exponential-Golomb code according to Embodiment 3. FIG. 39 is a diagram showing an example of the syntax of an attribute header according to Embodiment 3. FIG. 40 is a diagram showing an example of the syntax of attribute data according to Embodiment 3. FIG. 41 is a flowchart of 3D data encoding processing according to Embodiment 3. FIG. 42 is a flowchart of attribute information encoding processing according to Embodiment 3. FIG. 43 is a diagram showing processing for the Exponential-Golomb code according to Embodiment 3. FIG. 44 is a diagram showing an example of a reverse lookup table indicating the relationship between remaining codes and their values ​​according to the third embodiment. FIG. 45 is a flowchart of three-dimensional data decoding processing according to the third embodiment. FIG. 46 is a flowchart of attribute information decoding processing according to the third embodiment. FIG. 47 is a block diagram of a three-dimensional data encoding device according to the third embodiment. FIG. 48 is a block diagram of a three-dimensional data decoding device according to the third embodiment. FIG. 49 is a diagram for explaining encoding of attribute information using an RAHT according to the fourth embodiment. FIG. 50 is a diagram showing an example of setting a quantization scale for each layer according to the fourth embodiment. FIG. 51 is a diagram showing examples of a first code string and a second code string according to the fourth embodiment. FIG. 52 is a diagram showing an example of a truncated unary code according to the fourth embodiment. FIG. 53 is a diagram for explaining an inverse Haar transform according to the fourth embodiment. FIG. 54 is a diagram showing an example of syntax of attribute information according to the fourth embodiment. FIG. 55 is a diagram showing examples of coding coefficients and ZeroCnt according to the fourth embodiment. FIG. 56 is a flowchart of three-dimensional data encoding processing according to the fourth embodiment. FIG. 57 is a flowchart of attribute information encoding processing according to the fourth embodiment.FIG. 58 is a flowchart of coding coefficient coding processing according to the fourth embodiment. FIG. 59 is a flowchart of three-dimensional data decoding processing according to the fourth embodiment. FIG. 60 is a flowchart of attribute information decoding processing according to the fourth embodiment. FIG. 61 is a flowchart of coding coefficient decoding processing according to the fourth embodiment. FIG. 62 is a block diagram of an attribute information coding unit according to the fourth embodiment. FIG. 63 is a block diagram of an attribute information decoding unit according to the fourth embodiment. FIG. 64 is a block diagram showing the configuration of a three-dimensional data coding device according to the fifth embodiment. FIG. 65 is a block diagram showing the configuration of a three-dimensional data decoding device according to the fifth embodiment. FIG. 66 is a diagram showing a first example of SPS syntax according to the fifth embodiment. FIG. 67 is a diagram showing an example of the configuration of a bitstream according to the fifth embodiment. FIG. 68 is a diagram showing a second example of SPS syntax according to the fifth embodiment. FIG. 69 is a diagram showing a first example of transformation information syntax according to the fifth embodiment. FIG. 70 is a diagram showing a second example of transformation information syntax according to the fifth embodiment. FIG. 71 is a flowchart showing the processing procedure of a three-dimensional data coding device according to the fifth embodiment. FIG. 72 is a flowchart showing the processing steps of a three-dimensional data decoding device according to the fifth embodiment. FIG. 73 is a block diagram for explaining another example of processing of a three-dimensional data encoding device according to the fifth embodiment. FIG. 74 is a block diagram for explaining another example of processing of a three-dimensional data decoding device according to the fifth embodiment. FIG. 75 is a flowchart showing the processing steps of a three-dimensional data encoding device according to the fifth embodiment. FIG. 76 is a flowchart showing the processing steps of a three-dimensional data decoding device according to the fifth embodiment. FIG. 77 is a block diagram of a three-dimensional data creation device according to the sixth embodiment. FIG. 78 is a flowchart of a three-dimensional data creation method according to the sixth embodiment. FIG. 79 is a diagram showing the configuration of a system according to the sixth embodiment. FIG. 80 is a block diagram of a client device according to the sixth embodiment. FIG. 81 is a block diagram of a server according to the sixth embodiment. FIG. 82 is a flowchart of three-dimensional data creation processing by a client device according to the sixth embodiment. FIG. 83 is a flowchart of sensor information transmission processing by a client device according to the sixth embodiment.FIG. 84 is a flowchart of three-dimensional data creation processing by a server according to the sixth embodiment. FIG. 85 is a flowchart of three-dimensional map transmission processing by a server according to the sixth embodiment. FIG. 86 is a diagram illustrating a configuration of a modified example of a system according to the sixth embodiment. FIG. 87 is a diagram illustrating configurations of a server and a client device according to the sixth embodiment. FIG. 88 is a diagram illustrating configurations of a server and a client device according to the sixth embodiment. FIG. 89 is a flowchart of processing by a client device according to the sixth embodiment. FIG. 90 is a diagram illustrating a configuration of a sensor information collection system according to the sixth embodiment. FIG. 91 is a diagram illustrating an example of a system according to the sixth embodiment. FIG. 92 is a diagram illustrating a modified example of a system according to the sixth embodiment. FIG. 93 is a flowchart illustrating an example of application processing according to the sixth embodiment. FIG. 94 is a diagram illustrating sensor ranges of various sensors according to the sixth embodiment. FIG. 95 is a diagram illustrating an example of a configuration of an autonomous driving system according to the sixth embodiment. FIG. 96 is a diagram illustrating an example of a bitstream configuration according to the sixth embodiment. FIG. 97 is a flowchart of point cloud selection processing according to the sixth embodiment. FIG. 98 is a diagram illustrating an example screen of the point cloud selection processing according to the sixth embodiment. FIG. 99 is a diagram illustrating an example screen of the point cloud selection processing according to the sixth embodiment. FIG. 100 is a diagram showing an example of a screen for point cloud selection processing according to the sixth embodiment.

[0013] A three-dimensional data encoding method according to one aspect of the present disclosure acquires attribute information of a three-dimensional point, performs a conversion process on a numerical value indicated by the attribute information, such as scaling, which involves performing at least one of multiplication / division and shift operations, and offsetting, which involves adding / subtracting, and then encodes the attribute information, or performs an encoding process that encodes the attribute information without performing the conversion process, and generates a bitstream that includes the encoded attribute information and conversion identification information that indicates whether the conversion process has been performed.

[0014] Depending on the encoding unit that encodes the attribute information, it may be unable to process decimal points and / or negative numbers. Therefore, for example, by converting the attribute information using at least one of scaling and offsetting, the value indicated by the attribute information can be converted into a positive number. Therefore, according to the three-dimensional data encoding method disclosed herein, even if it is unable to encode decimal points and / or negative numbers, it is possible to appropriately convert and encode the attribute information and notify the user that the attribute information has been converted.

[0015] Also, for example, when the conversion process is performed in generating the bitstream, the bitstream is generated to further include conversion information including at least one of scale information indicating the value used for the scaling and offset information indicating the value used for the offset.

[0016] This allows the attribute information to be converted using an appropriate value to make the attribute information a positive number, for example, without having to predetermine the scale value and offset value, and the value used in the conversion process can be notified.

[0017] Also, for example, in acquiring the attribute information, a plurality of pieces of attribute information are acquired, and in the encoding process, the conversion process is performed for each piece of attribute information to encode the attribute information, or the attribute information is encoded without performing the conversion process, and in generating the bit stream, the bit stream is generated which includes the encoded plurality of pieces of attribute information and the conversion identification information for each piece of attribute information.

[0018] Depending on the 3D data, one 3D point may have multiple pieces of attribute information, or multiple 3D points having one piece of attribute information may be decoded. Even in such cases, the attribute information can be appropriately converted and encoded for each of the multiple pieces of attribute information, and it can be notified that the attribute information has been converted.

[0019] Also, for example, if the encoding process encodes the attribute information by performing the conversion process that performs both the scaling and the offsetting, then when generating the bitstream, the bitstream is generated, which further includes order information that indicates the order in which the scaling and the offsetting were performed.

[0020] This allows the attribute information to be converted in an appropriate order to make the attribute information a positive number, for example, without having to predetermine the order of scale and offset, and also makes it possible to notify the order.

[0021] Furthermore, for example, the transformation identification information indicates whether the scaling has been performed and whether the offset has been performed.

[0022] This allows the attribute information to be converted using an appropriate conversion process to make the attribute information a positive number, for example, without having to predetermine how to use at least one of the scale and offset, and the value used in the conversion process can be notified.

[0023] Also, for example, in generating the bitstream, the conversion identification information is stored in additional information common to the sequence.

[0024] In addition, a three-dimensional data decoding method according to one aspect of the present disclosure acquires a bitstream including coded attribute information of three-dimensional points and coded transformation identification information indicating whether or not a transformation process has been performed on the attribute information, the transformation process including scaling, which involves performing at least one of multiplication / division and shift operations, and offsetting, which involves addition / subtraction, and the like; decodes the coded attribute information; and decodes the coded transformation identification information.

[0025] Depending on the decoding unit that decodes the encoded attribute information, there is a possibility that decimal points and / or negative numbers cannot be processed. Therefore, for example, if the attribute information is encoded after being converted by at least one of scaling and offsetting, even such a decoding unit can decode the encoded attribute information. Furthermore, since it is possible to determine whether the value indicated by the attribute information has been converted based on the conversion identification information, it is possible to restore the decoded attribute information to its original appropriate value.

[0026] Furthermore, for example, a three-dimensional data decoding method according to one aspect of the present disclosure further performs an inverse transformation process, in which, when the transformation identification information indicates that the transformation process has been performed on the attribute information, the numerical value indicated by the decoded attribute information is subjected to inverse scaling, which involves performing at least one of multiplication / division and shift operations, and at least one of inverse offset, which involves addition / subtraction.

[0027] According to this, even if the value indicated by the attribute information is converted into a positive number based on the conversion identification information and then encoded, a conversion process that is reverse to the conversion process can be performed. Therefore, according to the three-dimensional data decoding method of the present disclosure, the encoded attribute information can be appropriately decoded, and the decoded attribute information can be appropriately converted (inversely converted) to return the attribute information to its original appropriate value.

[0028] Also, for example, the bitstream further includes transformation information including at least one of scale information indicating the value used for the scaling and offset information indicating the value used for the offset, and in the inverse transformation process, the inverse transformation process is performed based on at least one of the scale information and the offset information.

[0029] This allows the decoded attribute information to be restored to its original appropriate value by inverse conversion processing without having to predetermine the scale value and offset value.

[0030] Also, for example, the bit stream includes a plurality of encoded pieces of attribute information, and in the decoding, each of the plurality of encoded pieces of attribute information is decoded, and the conversion identification information indicates whether the conversion process has been performed for each piece of attribute information, and in the inverse conversion process, the inverse conversion process is performed on attribute information among the plurality of decoded pieces of attribute information for which the conversion identification information indicates that the conversion process has been performed.

[0031] Depending on the 3D data, one 3D point may have multiple pieces of attribute information, or multiple 3D points having one piece of attribute information may be decoded. Even in such cases, for each of the multiple pieces of attribute information, the decoded attribute information can be restored to its original appropriate value by inverse conversion processing.

[0032] Also, for example, the bitstream further includes order information indicating the order in which the scaling and offsetting are performed, and in the inverse transform process, the inverse scaling and inverse offsetting are performed in an order based on the order information.

[0033] This allows the decoded attribute information to be restored to its original appropriate value by inverse conversion processing without having to predetermine the order of scale and offset.

[0034] Furthermore, for example, the transformation identification information indicates whether the scaling has been performed and whether the offset has been performed.

[0035] This allows the decoded attribute information to be restored to its original appropriate value by inverse conversion processing without having to predetermine how to use at least one of the scale and the offset.

[0036] Also, for example, the bitstream has the conversion identification information stored in additional information common to the sequence.

[0037] In addition, a three-dimensional data encoding device according to one aspect of the present disclosure includes a processor and a memory, and the processor uses the memory to acquire attribute information of three-dimensional points, and performs a conversion process on the numerical values ​​indicated by the attribute information, performing scaling, which involves at least one of multiplication / division and shift operations, and offsetting, which involves addition / subtraction, before encoding the attribute information, or performs an encoding process that encodes the attribute information without performing the conversion process, and generates a bitstream that includes the encoded attribute information and conversion identification information that indicates whether the conversion process has been performed.

[0038] Depending on the encoding unit that encodes the attribute information, it may be unable to process decimal points and / or negative numbers. Therefore, for example, by converting the attribute information using at least one of scaling and offsetting, the value indicated by the attribute information can be converted into a positive number. Therefore, according to the three-dimensional data encoding device disclosed herein, even if it is unable to encode decimal points and / or negative numbers, it is possible to appropriately convert and encode the attribute information and notify the user that the attribute information has been converted.

[0039] In addition, a three-dimensional data decoding device according to one aspect of the present disclosure includes a processor and a memory, and the processor uses the memory to obtain a bitstream including coded attribute information of three-dimensional points and coded transformation identification information indicating whether or not a transformation process has been performed on the attribute information, the transformation process performing scaling, which involves at least one of multiplication / division and shift operations, and offsetting, which involves addition / subtraction, and the like; decodes the coded attribute information; and decodes the coded transformation identification information.

[0040] Depending on the decoding unit that decodes the encoded attribute information, there is a possibility that decimal points and / or negative numbers cannot be processed. Therefore, for example, if the attribute information is encoded after being converted by at least one of scaling and offsetting, even such a decoding unit can decode the encoded attribute information. Furthermore, since it is possible to determine whether the value indicated by the attribute information has been converted based on the conversion identification information, it is possible to restore the decoded attribute information to its original appropriate value.

[0041] These comprehensive or specific aspects may be realized as a system, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or may be realized as any combination of a system, a method, an integrated circuit, a computer program, and a recording medium.

[0042] Hereinafter, the embodiments will be described in detail with reference to the drawings. Note that each of the embodiments described below represents a specific example of the present disclosure. The numerical values, shapes, materials, components, component placement and connection configurations, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components not recited in independent claims will be described as optional components.

[0043] (First Embodiment) When using encoded data of a point cloud in an actual device or service, it is desirable to transmit and receive information required for the purpose in order to reduce network bandwidth. However, until now, such a function has not existed in the encoding structure of three-dimensional data, and no encoding method for this purpose has existed.

[0044] In this embodiment, we will describe a three-dimensional data encoding method and a three-dimensional data encoding device that provide the function of transmitting and receiving information required for the purpose in encoded data of a three-dimensional point cloud, as well as a three-dimensional data decoding method and a three-dimensional data decoding device that decodes the encoded data, a three-dimensional data multiplexing method that multiplexes the encoded data, and a three-dimensional data transmission method that transmits the encoded data.

[0045] In particular, currently, the first and second encoding methods are being considered as methods (encoding schemes) for encoding point cloud data, but the structure of the encoded data and the method for storing the encoded data in a system format have not been defined, and as things stand, there is a problem that MUX processing (multiplexing) in the encoding unit, or transmission or storage is not possible.

[0046] Furthermore, there has been no method so far that supports a format in which two codecs, a first encoding method and a second encoding method, are mixed, such as PCC (Point Cloud Compression).

[0047] In this embodiment, the structure of PCC encoded data in which two codecs, a first encoding method and a second encoding method, are mixed, and a method for storing the encoded data in a system format will be described.

[0048] First, the configuration of a three-dimensional data (point cloud data) encoding / decoding system according to this embodiment will be described. Fig. 1 is a diagram showing an example of the configuration of a three-dimensional data encoding / decoding system according to this embodiment. As shown in Fig. 1, the three-dimensional data encoding / decoding system includes a three-dimensional data encoding system 4601, a three-dimensional data decoding system 4602, a sensor terminal 4603, and an external connection unit 4604.

[0049] The three-dimensional data encoding system 4601 generates encoded data or multiplexed data by encoding point cloud data, which is three-dimensional data. Note that the three-dimensional data encoding system 4601 may be a three-dimensional data encoding device realized by a single device, or may be a system realized by multiple devices. Furthermore, the three-dimensional data encoding device may include some of the multiple processing units included in the three-dimensional data encoding system 4601.

[0050] The three-dimensional data encoding system 4601 includes a point cloud data generation system 4611, a presentation unit 4612, an encoding unit 4613, a multiplexing unit 4614, an input / output unit 4615, and a control unit 4616. The point cloud data generation system 4611 includes a sensor information acquisition unit 4617 and a point cloud data generation unit 4618.

[0051] The sensor information acquisition unit 4617 acquires sensor information from the sensor terminal 4603 and outputs the sensor information to the point cloud data generation unit 4618. The point cloud data generation unit 4618 generates point cloud data from the sensor information and outputs the point cloud data to the encoding unit 4613.

[0052] The presentation unit 4612 presents the sensor information or point cloud data to the user. For example, the presentation unit 4612 displays information or an image based on the sensor information or point cloud data.

[0053] The encoding unit 4613 encodes (compresses) the point cloud data, and outputs the obtained encoded data, control information obtained in the encoding process, and other additional information to the multiplexing unit 4614. The additional information includes, for example, sensor information.

[0054] The multiplexing unit 4614 generates multiplexed data by multiplexing the coded data input from the coding unit 4613, the control information, and the additional information. The format of the multiplexed data is, for example, a file format for storage or a packet format for transmission.

[0055] The input / output unit 4615 (e.g., a communication unit or an interface) outputs the multiplexed data to the outside. Alternatively, the multiplexed data is stored in a storage unit such as an internal memory. The control unit 4616 (or the application execution unit) controls each processing unit. In other words, the control unit 4616 controls encoding, multiplexing, etc.

[0056] The sensor information may be input to the encoding unit 4613 or the multiplexing unit 4614. The input / output unit 4615 may output the point cloud data or the encoded data directly to the outside.

[0057] The transmission signal (multiplexed data) output from the three-dimensional data encoding system 4601 is input to the three-dimensional data decoding system 4602 via the external connection unit 4604 .

[0058] The three-dimensional data decoding system 4602 generates point cloud data, which is three-dimensional data, by decoding the encoded data or multiplexed data. Note that the three-dimensional data decoding system 4602 may be a three-dimensional data decoding device realized by a single device, or may be a system realized by multiple devices. Furthermore, the three-dimensional data decoding device may include some of the multiple processing units included in the three-dimensional data decoding system 4602.

[0059] The three-dimensional data decoding system 4602 includes a sensor information acquisition unit 4621 , an input / output unit 4622 , a demultiplexing unit 4623 , a decoding unit 4624 , a presentation unit 4625 , a user interface 4626 , and a control unit 4627 .

[0060] The sensor information acquisition unit 4621 acquires sensor information from the sensor terminal 4603 .

[0061] The input / output unit 4622 acquires the transmission signal, decodes the multiplexed data (file format or packets) from the transmission signal, and outputs the multiplexed data to the demultiplexer 4623 .

[0062] The demultiplexing unit 4623 obtains the coded data, control information, and additional information from the multiplexed data, and outputs the coded data, control information, and additional information to the decoding unit 4624 .

[0063] The decoding unit 4624 reconstructs the point cloud data by decoding the encoded data.

[0064] The presentation unit 4625 presents the point cloud data to the user. For example, the presentation unit 4625 displays information or images based on the point cloud data. The user interface 4626 acquires instructions based on user operations. The control unit 4627 (or the application execution unit) controls each processing unit. In other words, the control unit 4627 controls demultiplexing, decoding, presentation, etc.

[0065] The input / output unit 4622 may acquire point cloud data or encoded data directly from the outside. The presentation unit 4625 may acquire additional information such as sensor information and present information based on the additional information. The presentation unit 4625 may present information based on a user instruction acquired by the user interface 4626.

[0066] The sensor terminal 4603 generates sensor information, which is information obtained by a sensor. The sensor terminal 4603 is a terminal equipped with a sensor or a camera, and examples of the sensor terminal 4603 include a mobile object such as an automobile, a flying object such as an airplane, a mobile terminal, or a camera.

[0067] Examples of sensor information that can be acquired by the sensor terminal 4603 include (1) the distance between the sensor terminal 4603 and an object, or the reflectance of the object, obtained from a LIDAR, millimeter-wave radar, or infrared sensor, and (2) the distance between a camera and an object, or the reflectance of the object, obtained from multiple monocular camera images or stereo camera images. The sensor information may also include the attitude, orientation, gyro (angular velocity), position (GPS information or altitude), speed, acceleration, etc. of the sensor. The sensor information may also include temperature, air pressure, humidity, magnetism, etc.

[0068] The external connection unit 4604 is realized by an integrated circuit (LSI or IC), an external storage unit, communication with a cloud server via the Internet, broadcasting, or the like.

[0069] Next, point cloud data will be described. Fig. 2 is a diagram showing the structure of point cloud data. Fig. 3 is a diagram showing an example of the structure of a data file in which information about point cloud data is written.

[0070] Point cloud data includes data on multiple points. Each point includes location information (three-dimensional coordinates) and attribute information for that location information. A collection of multiple points is called a point cloud. For example, a point cloud may represent the three-dimensional shape of an object.

[0071] Position information such as three-dimensional coordinates is sometimes called geometry. Data for each point may include attribute information of multiple attribute types. The attribute types may be, for example, color or reflectance.

[0072] One piece of attribute information may be associated with one piece of location information, or multiple pieces of attribute information having different attribute types may be associated with one piece of location information, or multiple pieces of attribute information of the same attribute type may be associated with one piece of location information.

[0073] The data file configuration example shown in FIG. 3 is an example in which position information and attribute information correspond one-to-one, and shows the position information and attribute information of N points that make up the point cloud data.

[0074] The position information is, for example, information on three axes, x, y, and z. The attribute information is, for example, RGB color information. A typical data file is a ply file.

[0075] Next, the types of point cloud data will be described. Fig. 4 is a diagram showing the types of point cloud data. As shown in Fig. 4, point cloud data includes static objects and dynamic objects.

[0076] A static object is 3D point cloud data at any time (a certain time). A dynamic object is 3D point cloud data that changes over time. Hereinafter, 3D point cloud data at a certain time will be referred to as a PCC frame or a frame.

[0077] The object may be a point cloud with a certain area restriction, such as ordinary video data, or a large-scale point cloud with no area restriction, such as map information.

[0078] Furthermore, there may be point cloud data of various densities, such as sparse point cloud data and dense point cloud data.

[0079] Each processing unit will be described in detail below. The sensor information is acquired by various methods, such as a distance sensor such as a LIDAR or a range finder, a stereo camera, or a combination of multiple monocular cameras. The point cloud data generation unit 4618 generates point cloud data based on the sensor information acquired by the sensor information acquisition unit 4617. The point cloud data generation unit 4618 generates position information as point cloud data, and adds attribute information for the position information to the position information.

[0080] The point cloud data generation unit 4618 may process the point cloud data when generating position information or adding attribute information. For example, the point cloud data generation unit 4618 may reduce the amount of data by deleting point clouds with overlapping positions. Furthermore, the point cloud data generation unit 4618 may convert (position shift, rotation, normalization, etc.) the position information or render the attribute information.

[0081] In FIG. 1, the point cloud data generation system 4611 is included in the three-dimensional data encoding system 4601, but it may be provided independently outside the three-dimensional data encoding system 4601.

[0082] The encoding unit 4613 generates encoded data by encoding the point cloud data based on a predetermined encoding method. There are two main types of encoding methods: the first is an encoding method that uses position information, and this encoding method will be referred to as the first encoding method hereinafter; and the second is an encoding method that uses a video codec, and this encoding method will be referred to as the second encoding method hereinafter.

[0083] The decoding unit 4624 decodes the coded data based on a predefined coding method to decode the point cloud data.

[0084] The multiplexing unit 4614 generates multiplexed data by multiplexing the encoded data using an existing multiplexing method. The generated multiplexed data is transmitted or stored. In addition to the PCC encoded data, the multiplexing unit 4614 multiplexes other media such as video, audio, subtitles, applications, and files, or reference time information. The multiplexing unit 4614 may further multiplex attribute information related to sensor information or point cloud data.

[0085] Multiplexing methods or file formats include ISOBMFF, and ISOBMFF-based transmission methods such as MPEG-DASH, MMT, MPEG-2 TS Systems, and RMP.

[0086] The demultiplexing unit 4623 extracts PCC encoded data, other media, time information, and the like from the multiplexed data.

[0087] The input / output unit 4615 transmits the multiplexed data using a method suited to the transmission medium or storage medium, such as broadcasting or communication. The input / output unit 4615 may communicate with other devices via the Internet, or may communicate with a storage unit such as a cloud server.

[0088] The communication protocol used may be http, ftp, TCP, UDP, etc. A pull-type communication method or a push-type communication method may be used.

[0089] Either wired transmission or wireless transmission may be used. For wired transmission, Ethernet (registered trademark), USB, RS-232C, HDMI (registered trademark), coaxial cable, etc. are used. For wireless transmission, wireless LAN, Wi-Fi (registered trademark), Bluetooth (registered trademark), millimeter waves, etc. are used.

[0090] As a broadcasting system, for example, DVB-T2, DVB-S2, DVB-C2, ATSC3.0, or ISDB-S3 is used.

[0091] Fig. 5 is a diagram showing the configuration of a first encoding unit 4630, which is an example of the encoding unit 4613 that performs encoding using the first encoding method. Fig. 6 is a block diagram of the first encoding unit 4630. The first encoding unit 4630 generates encoded data (encoded stream) by encoding point cloud data using the first encoding method. This first encoding unit 4630 includes a position information encoding unit 4631, an attribute information encoding unit 4632, an additional information encoding unit 4633, and a multiplexing unit 4634.

[0092] The first encoding unit 4630 is characterized in that it performs encoding with consideration given to a three-dimensional structure. The first encoding unit 4630 is also characterized in that the attribute information encoding unit 4632 performs encoding using information obtained from the position information encoding unit 4631. The first encoding method is also called GPCC (Geometry-based PCC).

[0093] The point cloud data is PCC point cloud data such as a PLY file, or PCC point cloud data generated from sensor information, and includes position information (Position), attribute information (Attribute), and other additional information (MetaData). The position information is input to a position information encoding unit 4631, the attribute information is input to an attribute information encoding unit 4632, and the additional information is input to an additional information encoding unit 4633.

[0094] The position information encoding unit 4631 encodes the position information to generate encoded position information (Compressed Geometry), which is encoded data. For example, the position information encoding unit 4631 encodes the position information using an N-ary tree structure such as an octree. Specifically, in an octree, the target space is divided into eight nodes (subspaces), and 8-bit information (occupancy code) indicating whether or not a point cloud is included in each node is generated. Furthermore, the node including the point cloud is further divided into eight nodes, and 8-bit information indicating whether or not a point cloud is included in each of the eight nodes is generated. This process is repeated until the number of point clouds included in a predetermined layer or node falls below a threshold.

[0095] The attribute information encoding unit 4632 generates encoded attribute information (Compressed Attribute) that is encoded data by encoding using the configuration information generated by the position information encoding unit 4631. For example, the attribute information encoding unit 4632 determines a reference point (reference node) to be referenced when encoding a target point (target node) to be processed, based on the octree structure generated by the position information encoding unit 4631. For example, the attribute information encoding unit 4632 references a peripheral node or adjacent node whose parent node in the octree is the same as that of the target node. Note that the method of determining the reference relationship is not limited to this.

[0096] Furthermore, the encoding process of the attribute information may include at least one of a quantization process, a prediction process, and an arithmetic coding process. In this case, the reference means using a reference node to calculate a predicted value of the attribute information, or using the state of the reference node (e.g., occupancy information indicating whether the reference node includes a point group) to determine an encoding parameter. For example, the encoding parameter may be a quantization parameter in a quantization process, or a context in arithmetic coding.

[0097] The additional information encoding unit 4633 generates encoded additional information (Compressed MetaData) that is encoded data by encoding compressible data from the additional information.

[0098] The multiplexing unit 4634 multiplexes the encoding position information, the encoding attribute information, the encoding additional information, and other additional information to generate an encoded stream (Compressed Stream) that is encoded data. The generated encoded stream is output to a processing unit of a system layer (not shown).

[0099] Next, a first decoding unit 4640, which is an example of the decoding unit 4624 that performs decoding using the first encoding method, will be described. FIG. 7 is a diagram showing the configuration of the first decoding unit 4640. FIG. 8 is a block diagram of the first decoding unit 4640. The first decoding unit 4640 generates point cloud data by decoding, using the first encoding method, coded data (coded stream). This first decoding unit 4640 includes a demultiplexing unit 4641, a position information decoding unit 4642, an attribute information decoding unit 4643, and an additional information decoding unit 4644.

[0100] A coded stream (compressed stream) that is coded data is input to the first decoding unit 4640 from a processing unit of a system layer (not shown).

[0101] The demultiplexer 4641 separates the encoded position information (Compressed Geometry), the encoded attribute information (Compressed Attribute), the encoded additional information (Compressed MetaData), and other additional information from the encoded data.

[0102] The position information decoding unit 4642 generates position information by decoding the encoded position information. For example, the position information decoding unit 4642 restores position information of a point group represented by three-dimensional coordinates from encoded position information represented by an N-ary tree structure such as an octree.

[0103] The attribute information decoding unit 4643 decodes the encoded attribute information based on the configuration information generated by the position information decoding unit 4642. For example, the attribute information decoding unit 4643 determines a reference point (reference node) to be referenced in decoding the target point (target node) to be processed based on the octree structure obtained by the position information decoding unit 4642. For example, the attribute information decoding unit 4643 references a peripheral node or adjacent node whose parent node in the octree is the same as that of the target node. Note that the method of determining the reference relationship is not limited to this.

[0104] Furthermore, the attribute information decoding process may include at least one of an inverse quantization process, a prediction process, and an arithmetic decoding process. In this case, the reference means using a reference node to calculate a predicted value of the attribute information, or using the state of the reference node (e.g., occupancy information indicating whether the reference node includes a point group) to determine a decoding parameter. For example, the decoding parameter is a quantization parameter in an inverse quantization process, or a context in an arithmetic decoding process.

[0105] The additional information decoding unit 4644 generates additional information by decoding the encoded additional information. The first decoding unit 4640 uses the additional information necessary for decoding the position information and attribute information during decoding, and outputs the additional information necessary for the application to the outside.

[0106] Next, an example of the configuration of the position information encoder will be described. Fig. 9 is a block diagram of a position information encoder 2700 according to this embodiment. The position information encoder 2700 includes an octree generator 2701, a geometric information calculator 2702, a coding table selector 2703, and an entropy encoder 2704.

[0107] The octree generation unit 2701 generates, for example, an occupancy tree from the input position information and generates an occupancy code for each node in the occupancy tree. The geometric information calculation unit 2702 acquires information indicating whether an adjacent node of the target node is an occupied node. For example, the geometric information calculation unit 2702 calculates the occupancy information of the adjacent node (information indicating whether the adjacent node is an occupied node) from the occupancy code of the parent node to which the target node belongs. The geometric information calculation unit 2702 may also store encoded nodes in a list and search for adjacent nodes from the list. The geometric information calculation unit 2702 may also switch adjacent nodes depending on the position of the target node within the parent node.

[0108] The coding table selection unit 2703 selects a coding table to be used for entropy coding of the target node using the occupancy information of the adjacent nodes calculated by the geometric information calculation unit 2702. For example, the coding table selection unit 2703 may generate a bit string using the occupancy information of the adjacent nodes, and select a coding table for an index number generated from the bit string.

[0109] The entropy coding unit 2704 generates the encoding position information and the metadata by entropy coding the occupancy code of the target node using the coding table of the selected index number. The entropy coding unit 2704 may add information indicating the selected coding table to the encoding position information.

[0110] The octree representation and the scanning order of position information will be described below. Position information (position data) is converted into an octree structure (octreeing) and then encoded. The octree structure is composed of nodes and leaves. Each node has eight nodes or leaves, and each leaf has voxel (VXL) information. FIG. 10 is a diagram showing an example of the structure of position information containing multiple voxels. FIG. 11 is a diagram showing an example of the position information shown in FIG. 10 converted into an octree structure. Here, of the leaves shown in FIG. 11, leaves 1, 2, and 3 represent voxels VXL1, VXL2, and VXL3 shown in FIG. 10, respectively, and represent a VXL containing a point cloud (hereinafter referred to as a valid VXL).

[0111] Specifically, node 1 corresponds to the entire space containing the position information in Figure 10. The entire space corresponding to node 1 is divided into eight nodes, and of the eight nodes, the node containing a valid VXL is further divided into eight nodes or leaves, and this process is repeated for each level of the tree structure. Here, each node corresponds to a subspace, and has information (occupancy code) as node information indicating the position of the next node or leaf after division. In addition, the block at the bottom level is set as a leaf, and the number of point clouds contained in the leaf, etc., is held as leaf information.

[0112] Next, an example configuration of the position information decoding unit will be described. Fig. 12 is a block diagram of the position information decoding unit 2710 according to this embodiment. The position information decoding unit 2710 includes an octree generation unit 2711, a geometric information calculation unit 2712, a coding table selection unit 2713, and an entropy decoding unit 2714.

[0113] The octree generator 2711 generates an octree of a certain space (node) using header information or metadata of a bitstream. For example, the octree generator 2711 generates a large space (root node) using the sizes of the space in the x-axis, y-axis, and z-axis directions added to the header information, and then generates an octree by dividing the space into two in the x-axis, y-axis, and z-axis directions to generate eight small spaces A (nodes A0 to A7). Nodes A0 to A7 are set in order as target nodes.

[0114] The geometric information calculation unit 2712 acquires occupancy information indicating whether an adjacent node of the target node is an occupied node. For example, the geometric information calculation unit 2712 calculates the occupancy information of the adjacent node from the occupancy code of the parent node to which the target node belongs. The geometric information calculation unit 2712 may also store decoded nodes in a list and search for adjacent nodes from the list. The geometric information calculation unit 2712 may also switch adjacent nodes depending on the position of the target node within the parent node.

[0115] The coding table selection unit 2713 selects a coding table (decoding table) to be used for entropy decoding of the target node using the occupancy information of the adjacent nodes calculated by the geometric information calculation unit 2712. For example, the coding table selection unit 2713 may generate a bit string using the occupancy information of the adjacent nodes, and select a coding table for an index number generated from the bit string.

[0116] The entropy decoding unit 2714 generates position information by entropy decoding the occupancy code of the target node using the selected coding table. Note that the entropy decoding unit 2714 may obtain information on the selected coding table by decoding it from the bitstream, and entropy decode the occupancy code of the target node using the coding table indicated by the information.

[0117] The configurations of the attribute information encoding unit and the attribute information decoding unit will be described below. Fig. 13 is a block diagram showing an example configuration of the attribute information encoding unit A100. The attribute information encoding unit may include multiple encoding units that execute different encoding methods. For example, the attribute information encoding unit may switch between the following two methods depending on the use case.

[0118] The attribute information encoding unit A100 includes an LoD attribute information encoding unit A101 and a conversion attribute information encoding unit A102. The LoD attribute information encoding unit A101 classifies each 3D point into multiple layers using the position information of the 3D point, predicts the attribute information of the 3D point belonging to each layer, and encodes the prediction residual. Here, each classified layer is called LoD (Level of Detail).

[0119] The transformed attribute information encoding unit A102 encodes the attribute information using RAHT (Region Adaptive Hierarchical Transform). Specifically, the transformed attribute information encoding unit A102 applies RAHT or Haar transform to each piece of attribute information based on position information of the three-dimensional points to generate high-frequency components and low-frequency components for each layer, and encodes these values ​​using quantization, entropy coding, or the like.

[0120] 14 is a block diagram showing an example configuration of the attribute information decoding unit A110. The attribute information decoding unit may include multiple decoding units that perform different decoding methods. For example, the attribute information decoding unit may switch between the following two methods for decoding based on information included in the header or metadata:

[0121] The attribute information decoding unit A110 includes an LoD attribute information decoding unit A111 and a converted attribute information decoding unit A112. The LoD attribute information decoding unit A111 classifies each 3D point into multiple hierarchical levels using position information of the 3D points, and decodes attribute values ​​while predicting attribute information of the 3D points belonging to each hierarchical level.

[0122] The transformed attribute information decoding unit A112 decodes the attribute information using RAHT (Region Adaptive Hierarchical Transform). Specifically, the transformed attribute information decoding unit A112 decodes the attribute values ​​by applying an inverse RAHT or an inverse Haar transform to the high-frequency components and low-frequency components of each attribute value based on the position information of the three-dimensional points.

[0123] FIG. 15 is a block diagram showing the configuration of an attribute information encoding unit 3140, which is an example of the LoD attribute information encoding unit A101.

[0124] The attribute information encoding unit 3140 includes an LoD generation unit 3141, a surrounding search unit 3142, a prediction unit 3143, a prediction residual calculation unit 3144, a quantization unit 3145, an arithmetic encoding unit 3146, an inverse quantization unit 3147, a decoded value generation unit 3148, and a memory 3149.

[0125] The LoD generation unit 3141 generates the LoD using the position information of the three-dimensional point.

[0126] The surrounding search unit 3142 searches for nearby 3D points adjacent to each 3D point using the LoD generation result by the LoD generation unit 3141 and distance information indicating the distance between each 3D point.

[0127] The prediction unit 3143 generates a predicted value of the attribute information of the target 3D point to be coded.

[0128] The prediction residual calculation unit 3144 calculates (generates) a prediction residual of the predicted value of the attribute information generated by the prediction unit 3143 .

[0129] The quantization unit 3145 quantizes the prediction residual of the attribute information calculated by the prediction residual calculation unit 3144 .

[0130] The arithmetic coding unit 3146 arithmetically codes the prediction residuals after being quantized by the quantization unit 3145. The arithmetic coding unit 3146 outputs a bit stream including the arithmetically coded prediction residuals to, for example, a three-dimensional data decoding device.

[0131] Note that the prediction residual may be binarized by, for example, the quantization unit 3145 before being arithmetically coded by the arithmetic coding unit 3146 .

[0132] Furthermore, for example, the arithmetic coding unit 3146 may initialize a coding table used for arithmetic coding before the arithmetic coding. The arithmetic coding unit 3146 may initialize a coding table used for arithmetic coding for each layer. Furthermore, the arithmetic coding unit 3146 may output information indicating the position of the layer for which the coding table has been initialized, by including it in the bitstream.

[0133] The inverse quantization unit 3147 inverse quantizes the prediction residuals quantized by the quantization unit 3145 .

[0134] The decoded value generation unit 3148 generates a decoded value by adding the predicted value of the attribute information generated by the prediction unit 3143 and the prediction residual after inverse quantization by the inverse quantization unit 3147 .

[0135] The memory 3149 is a memory that stores the decoded values ​​of the attribute information of each 3D point decoded by the decoded value generation unit 3148. For example, when generating a predicted value of a 3D point that has not yet been encoded, the prediction unit 3143 generates the predicted value by using the decoded value of the attribute information of each 3D point stored in the memory 3149.

[0136] 16 is a block diagram of an attribute information coding unit 6600, which is an example of the transformed attribute information coding unit A102. The attribute information coding unit 6600 includes a sorting unit 6601, a Haar transform unit 6602, a quantization unit 6603, an inverse quantization unit 6604, an inverse Haar transform unit 6605, a memory 6606, and an arithmetic coding unit 6607.

[0137] The sorting unit 6601 generates a Morton code using the position information of the 3D points and sorts the 3D points in Morton code order. The Haar transform unit 6602 generates coding coefficients by applying a Haar transform to the attribute information. The quantization unit 6603 quantizes the coding coefficients of the attribute information.

[0138] The inverse quantization unit 6604 inversely quantizes the quantized coding coefficients. The inverse Haar transform unit 6605 applies an inverse Haar transform to the coding coefficients. The memory 6606 stores values ​​of attribute information of multiple decoded 3D points. For example, the attribute information of the decoded 3D points stored in the memory 6606 may be used for predicting uncoded 3D points.

[0139] The arithmetic coding unit 6607 calculates ZeroCnt from the quantized coding coefficients and arithmetically codes the ZeroCnt. The arithmetic coding unit 6607 also arithmetically codes the quantized non-zero coding coefficients. The arithmetic coding unit 6607 may binarize the coding coefficients before arithmetic coding. The arithmetic coding unit 6607 may also generate and code various header information.

[0140] FIG. 17 is a block diagram showing the configuration of an attribute information decoding unit 3150, which is an example of the LoD attribute information decoding unit A111.

[0141] The attribute information decoding unit 3150 includes an LoD generation unit 3151 , a surrounding search unit 3152 , a prediction unit 3153 , an arithmetic decoding unit 3154 , an inverse quantization unit 3155 , a decoded value generation unit 3156 , and a memory 3157 .

[0142] The LoD generation unit 3151 generates LoD using the position information of the 3D points decoded by the position information decoding unit (not shown in FIG. 17).

[0143] The surrounding search unit 3152 searches for nearby 3D points adjacent to each 3D point using the LoD generation result by the LoD generation unit 3151 and distance information indicating the distance between each 3D point.

[0144] The prediction unit 3153 generates a predicted value of the attribute information of the target 3D point to be decoded.

[0145] The arithmetic decoding unit 3154 arithmetically decodes prediction residuals in the bitstream acquired from the attribute information coding unit 3140 shown in FIG. 15 . The arithmetic decoding unit 3154 may initialize a decoding table used for arithmetic decoding. The arithmetic decoding unit 3154 initializes a decoding table used for arithmetic decoding for a layer for which the arithmetic coding unit 3146 shown in FIG. 15 has performed coding processing. The arithmetic decoding unit 3154 may initialize a decoding table used for arithmetic decoding for each layer. The arithmetic decoding unit 3154 may also initialize the decoding table based on information included in the bitstream that indicates the position of the layer for which the coding table has been initialized.

[0146] The inverse quantization unit 3155 inverse quantizes the prediction residuals arithmetically decoded by the arithmetic decoding unit 3154 .

[0147] The decoded value generation unit 3156 generates a decoded value by adding the predicted value generated by the prediction unit 3153 and the prediction residual after inverse quantization by the inverse quantization unit 3155. The decoded value generation unit 3156 outputs the decoded attribute information data to another device.

[0148] The memory 3157 is a memory that stores the decoded values ​​of the attribute information of each 3D point decoded by the decoded value generation unit 3156. For example, when generating a predicted value of a 3D point that has not yet been decoded, the prediction unit 3153 generates the predicted value by using the decoded value of the attribute information of each 3D point stored in the memory 3157.

[0149] 18 is a block diagram of an attribute information decoding unit 6610, which is an example of the transformed attribute information decoding unit A112. The attribute information decoding unit 6610 includes an arithmetic decoding unit 6611, an inverse quantization unit 6612, an inverse Haar transform unit 6613, and a memory 6614.

[0150] The arithmetic decoding unit 6611 arithmetically decodes the ZeroCnt and the coding coefficients included in the bitstream. Note that the arithmetic decoding unit 6611 may also decode various types of header information.

[0151] The inverse quantization unit 6612 inverse quantizes the arithmetically decoded coding coefficients. The inverse Haar transform unit 6613 applies inverse Haar transform to the inversely quantized coding coefficients. The memory 6614 stores values ​​of attribute information of multiple decoded 3D points. For example, the attribute information of the decoded 3D points stored in the memory 6614 may be used to predict undecoded 3D points.

[0152] Next, a second encoding unit 4650, which is an example of the encoding unit 4613 that performs encoding using the second encoding method, will be described. Fig. 19 is a diagram showing the configuration of the second encoding unit 4650. Fig. 20 is a block diagram of the second encoding unit 4650.

[0153] The second encoding unit 4650 generates encoded data (encoded stream) by encoding the point cloud data using a second encoding method. The second encoding unit 4650 includes an additional information generation unit 4651, a position image generation unit 4652, an attribute image generation unit 4653, a video encoding unit 4654, an additional information encoding unit 4655, and a multiplexing unit 4656.

[0154] The second encoding unit 4650 has a feature of generating a position image and an attribute image by projecting a three-dimensional structure onto a two-dimensional image, and encoding the generated position image and attribute image using an existing video encoding method. The second encoding method is also called VPCC (Video-based PCC).

[0155] The point cloud data is PCC point cloud data such as a PLY file or PCC point cloud data generated from sensor information, and includes position information (Position), attribute information (Attribute), and other additional information (MetaData).

[0156] The additional information generating unit 4651 generates map information of a plurality of two-dimensional images by projecting a three-dimensional structure onto the two-dimensional images.

[0157] The position image generation unit 4652 generates a position image (geometry image) based on the position information and the map information generated by the additional information generation unit 4651. This position image is, for example, a distance image in which distance (depth) is indicated as pixel values. Note that this distance image may be an image in which multiple point clouds are viewed from a single viewpoint (an image in which multiple point clouds are projected onto a single two-dimensional plane), or may be multiple images in which multiple point clouds are viewed from multiple viewpoints, or may be a single image in which these multiple images are integrated.

[0158] The attribute image generation unit 4653 generates an attribute image based on the attribute information and the map information generated by the additional information generation unit 4651. This attribute image is, for example, an image in which attribute information (e.g., color (RGB)) is represented as pixel values. Note that this image may be an image in which multiple point clouds are viewed from a single viewpoint (an image in which multiple point clouds are projected onto a single two-dimensional plane), or multiple images in which multiple point clouds are viewed from multiple viewpoints, or a single image in which these multiple images are integrated.

[0159] The video encoding unit 4654 encodes the position image and the attribute image using a video encoding method to generate an encoded position image (Compressed Geometry Image) and an encoded attribute image (Compressed Attribute Image), which are encoded data. Note that any known encoding method may be used as the video encoding method. For example, the video encoding method is AVC, HEVC, or the like.

[0160] The additional information encoding unit 4655 generates encoded additional information (Compressed MetaData) by encoding the additional information and map information included in the point cloud data.

[0161] The multiplexing unit 4656 multiplexes the encoding position image, the encoding attribute image, the encoding additional information, and other additional information to generate an encoded stream (Compressed Stream) that is encoded data. The generated encoded stream is output to a processing unit of a system layer (not shown).

[0162] Next, a second decoding unit 4660, which is an example of the decoding unit 4624 that performs decoding using the second encoding method, will be described. FIG. 21 is a diagram showing the configuration of the second decoding unit 4660. FIG. 22 is a block diagram of the second decoding unit 4660. The second decoding unit 4660 generates point cloud data by decoding, using the second encoding method, coded data (coded stream). This second decoding unit 4660 includes a demultiplexing unit 4661, a video decoding unit 4662, an additional information decoding unit 4663, a position information generation unit 4664, and an attribute information generation unit 4665.

[0163] A coded stream (compressed stream) that is coded data is input to the second decoding unit 4660 from a processing unit of a system layer (not shown).

[0164] The demultiplexing unit 4661 separates the encoded position image (Compressed Geometry Image), the encoded attribute image (Compressed Attribute Image), the encoded additional information (Compressed MetaData), and other additional information from the encoded data.

[0165] The video decoding unit 4662 generates a position image and an attribute image by decoding the encoded position image and the encoded attribute image using a video encoding method. Note that any known encoding method may be used as the video encoding method. For example, the video encoding method may be AVC or HEVC.

[0166] The additional information decoding unit 4663 decodes the encoded additional information to generate additional information including map information and the like.

[0167] The position information generating unit 4664 generates position information using the position image and map information. The attribute information generating unit 4665 generates attribute information using the attribute image and map information.

[0168] The second decoding unit 4660 uses the additional information required for decoding during decoding, and outputs the additional information required for the application to the outside.

[0169] The following describes the problems with the PCC encoding method. Fig. 23 is a diagram showing a protocol stack related to PCC encoded data. Fig. 23 shows an example in which other media data such as video (e.g., HEVC) or audio are multiplexed onto the PCC encoded data and transmitted or stored.

[0170] The multiplexing method and file format have the function of multiplexing various coded data and transmitting or storing them. To transmit or store coded data, the coded data must be converted into a format suitable for the multiplexing method. For example, HEVC specifies a technique for storing coded data in a data structure called a NAL unit and storing the NAL unit in ISOBMFF.

[0171] On the other hand, currently, a first encoding method (Codec1) and a second encoding method (Codec2) are being considered as methods for encoding point cloud data, but the structure of the encoded data and the method for storing the encoded data in a system format have not been defined, which poses the problem that, as it stands, MUX processing (multiplexing), transmission, and storage in the encoding unit are not possible.

[0172] In the following description, unless a specific encoding method is specified, it refers to either the first encoding method or the second encoding method.

[0173] (Embodiment 2) In this embodiment, the types of coded data (position information (Geometry), attribute information (Attribute), additional information (Metadata)) generated by the above-described first coding unit 4630 or second coding unit 4650, a method for generating the additional information (metadata), and multiplexing processing in the multiplexing unit will be described. Note that the additional information (metadata) may also be referred to as a parameter set or control information.

[0174] In this embodiment, the dynamic object (three-dimensional point cloud data that changes over time) described in Figure 4 will be used as an example, but a similar method may also be used in the case of a static object (three-dimensional point cloud data at any time).

[0175] 24 is a diagram showing the configurations of an encoding unit 4801 and a multiplexing unit 4802 included in the three-dimensional data encoding device according to this embodiment. The encoding unit 4801 corresponds to, for example, the first encoding unit 4630 or the second encoding unit 4650 described above. The multiplexing unit 4802 corresponds to the multiplexing unit 4634 or 4656 described above.

[0176] The encoding unit 4801 encodes point cloud data of a plurality of PCC (Point Cloud Compression) frames, and generates encoded data (Multiple Compressed Data) of a plurality of pieces of position information, attribute information, and additional information.

[0177] The multiplexing unit 4802 converts data of multiple data types (position information, attribute information, and additional information) into NAL units, thereby converting the data into a data structure that takes into account data access in the decoding device.

[0178] 25 is a diagram showing an example of the structure of coded data generated by the coding unit 4801. The arrows in the diagram indicate dependencies related to the decoding of coded data, with the source of the arrow depending on the data at the end of the arrow. In other words, the decoding device decodes the data at the end of the arrow and uses the decoded data to decode the data at the end of the arrow. In other words, dependency means that the data on which the dependency is based is referenced (used) in the processing (encoding, decoding, etc.) of the data on which the dependency is based.

[0179] First, the process of generating encoded data of position information will be described. The encoding unit 4801 generates encoded position data (Compressed Geometry Data) for each frame by encoding the position information of each frame. The encoded position data is represented by G(i), where i indicates the frame number, the time of the frame, etc.

[0180] The encoding unit 4801 also generates a position parameter set (GPS(i)) corresponding to each frame. The position parameter set includes parameters that can be used to decode the encoded position data. Furthermore, the encoded position data for each frame depends on the corresponding position parameter set.

[0181] Furthermore, encoded position data consisting of multiple frames is defined as a position sequence (Geometry Sequence). The encoding unit 4801 generates a position sequence parameter set (Geometry Sequence PS: also referred to as Position SPS) that stores parameters commonly used in decoding processes for multiple frames in the position sequence. The position sequence depends on the Position SPS.

[0182] Next, a process for generating coded attribute data of attribute information will be described. The coding unit 4801 generates coded attribute data (Compressed Attribute Data) for each frame by coding the attribute information of each frame. The coded attribute data is represented by A(i). FIG. 25 shows an example in which an attribute X and an attribute Y exist, and the coded attribute data of attribute X is represented by AX(i) and the coded attribute data of attribute Y is represented by AY(i).

[0183] The encoding unit 4801 also generates an attribute parameter set (APS(i)) corresponding to each frame. The attribute parameter set for attribute X is represented by AXPS(i), and the attribute parameter set for attribute Y is represented by AYPS(i). The attribute parameter set includes parameters that can be used to decode encoded attribute information. The encoded attribute data depends on the corresponding attribute parameter set.

[0184] Furthermore, encoded attribute data consisting of multiple frames is defined as an attribute sequence. The encoding unit 4801 generates an attribute sequence parameter set (Attribute Sequence PS: also referred to as attribute SPS) that stores parameters commonly used in decoding processes for multiple frames in the attribute sequence. The attribute sequence depends on the attribute SPS.

[0185] In addition, in the first encoding method, the encoded attribute data depends on the encoded position data.

[0186] 25 shows an example in which two types of attribute information (attribute X and attribute Y) exist. When there are two types of attribute information, for example, two encoding units generate respective data and metadata. Also, for example, an attribute sequence is defined for each type of attribute information, and an attribute SPS is generated for each type of attribute information.

[0187] 25 shows an example in which there is one type of position information and two types of attribute information, but this is not limiting, and there may be one type of attribute information, or three or more types. In this case, encoded data can be generated using a similar method. Furthermore, in the case of point cloud data that does not have attribute information, the attribute information may not be necessary. In this case, the encoding unit 4801 does not need to generate a parameter set related to the attribute information.

[0188] Next, the process of generating additional information (metadata) will be described. The encoding unit 4801 generates a PCC stream PS (also referred to as stream PS), which is a parameter set for the entire PCC stream. The encoding unit 4801 stores, in the stream PS, parameters that can be used in common for decoding processes for one or more position sequences and one or more attribute sequences. For example, the stream PS includes identification information indicating the codec for the point cloud data, information indicating the algorithm used for encoding, and the like. The position sequence and the attribute sequence depend on the stream PS.

[0189] Next, the access unit and GOF will be explained. In this embodiment, the concepts of an access unit (AU) and a group of frames (GOF) are newly introduced.

[0190] An access unit is a basic unit for accessing data during decoding, and is composed of one or more pieces of data and one or more pieces of metadata. For example, an access unit is composed of position information at the same time and one or more pieces of attribute information. A GOF is a random access unit, and is composed of one or more access units.

[0191] The encoding unit 4801 generates an access unit header (AU Header) as identification information indicating the beginning of an access unit. The encoding unit 4801 stores parameters related to the access unit in the access unit header. For example, the access unit header includes the configuration or information of the coded data included in the access unit. The access unit header also includes parameters commonly used for the data included in the access unit, such as parameters related to decoding of the coded data.

[0192] Note that the encoding unit 4801 may generate an access unit delimiter that does not include parameters related to the access unit, instead of the access unit header. This access unit delimiter is used as identification information indicating the start of the access unit. The decoding device identifies the start of the access unit by detecting the access unit header or the access unit delimiter.

[0193] Next, generation of identification information for the start of a GOF will be described. The encoding unit 4801 generates a GOF header as identification information indicating the start of a GOF. The encoding unit 4801 stores parameters related to the GOF in the GOF header. For example, the GOF header includes the structure or information of the encoded data included in the GOF. The GOF header also includes parameters commonly used for the data included in the GOF, such as parameters related to decoding of the encoded data.

[0194] The encoding unit 4801 may generate a GOF delimiter that does not include parameters related to the GOF, instead of a GOF header. This GOF delimiter is used as identification information indicating the beginning of the GOF. The decoding device identifies the beginning of the GOF by detecting the GOF header or the GOF delimiter.

[0195] In PCC encoded data, for example, an access unit is defined as a PCC frame unit, and a decoding device accesses the PCC frame based on identification information at the beginning of the access unit.

[0196] Furthermore, for example, a GOF is defined as one random access unit. The decoding device accesses the random access unit based on identification information at the beginning of the GOF. For example, if PCC frames are not dependent on each other and can be decoded independently, the PCC frames may be defined as the random access unit.

[0197] It should be noted that two or more PCC frames may be assigned to one access unit, and multiple random access units may be assigned to one GOF.

[0198] Furthermore, the encoding unit 4801 may define and generate parameter sets or metadata other than those described above. For example, the encoding unit 4801 may generate SEI (Supplemental Enhancement Information) that stores parameters (optional parameters) that may not necessarily be used during decoding.

[0199] Next, the structure of the coded data and the method of storing the coded data in the NAL unit will be described.

[0200] For example, a data format is defined for each type of encoded data. Fig. 26 shows examples of encoded data and NAL units.

[0201] For example, as shown in Fig. 26, the coded data includes a header and a payload. The coded data may include length information indicating the length (amount of data) of the coded data, the header, or the payload. The coded data may not include a header.

[0202] The header includes, for example, identification information for identifying the data, such as the data type or frame number.

[0203] The header includes, for example, identification information indicating a reference relationship. This identification information is stored in the header when, for example, there is a dependency relationship between data, and is information for referencing the reference destination from the reference source. For example, the header of the reference destination includes identification information for identifying the data. The header of the reference source includes identification information indicating the reference destination.

[0204] If the reference destination or the reference source can be identified or derived from other information, the identification information for specifying the data or the identification information indicating the reference relationship may be omitted.

[0205] The multiplexing unit 4802 stores the coded data in the payload of the NAL unit. The NAL unit header includes pcc_nal_unit_type, which is coded data identification information. Figure 27 shows an example of the semantics of pcc_nal_unit_type.

[0206] As shown in FIG. 27, when pcc_codec_type is codec 1 (Codec1: first encoding method), the values ​​0 to 10 of pcc_nal_unit_type correspond to the encoded position data (Geometry), encoded attribute X data (AttributeX), encoded attribute Y data (AttributeY), position PS (Geom.PS), attribute XPS (AttrX.PS), attribute YPS (AttrX.PS), position SPS (Geometry Sequence PS), attribute XSPS (AttributeX Sequence PS), attribute YSPS (AttributeY Sequence PS), AU header (AU Header), and GOF header (GOF Values ​​11 and above are assigned as spares for Codec 1.

[0207] When pcc_codec_type is Codec 2 (Codec2: second encoding method), values ​​0 to 2 of pcc_nal_unit_type are assigned to codec data A (DataA), metadata A (MetaDataA), and metadata B (MetaDataB). Values ​​3 and above are assigned as spares for Codec 2.

[0208] Next, the data transmission order will be described. The following describes the restrictions on the transmission order of NAL units.

[0209] The multiplexing unit 4802 collectively transmits NAL units in units of GOF or AU. The multiplexing unit 4802 places a GOF header at the beginning of a GOF, and places an AU header at the beginning of an AU.

[0210] The multiplexing unit 4802 may allocate a sequence parameter set (SPS) for each AU so that the decoding device can decode from the next AU even if data is lost due to packet loss or the like.

[0211] If the coded data have a dependency relationship related to decoding, the decoding device decodes the referenced data first, and then decodes the referenced data. In order to enable the decoding device to decode the data in the order in which it was received without rearranging the data, the multiplexing unit 4802 sends the referenced data first.

[0212] 28 is a diagram showing examples of the transmission order of NAL units, showing three examples: position information priority, parameter priority, and data integration.

[0213] The location information priority transmission order is an example in which information related to location information and information related to attribute information are transmitted together. In this transmission order, the transmission of information related to location information is completed earlier than the transmission of information related to attribute information.

[0214] For example, by using this transmission order, a decoding device that does not decode attribute information may be able to set a time during which it does not process the attribute information by ignoring the decoding of the attribute information. Also, for example, in the case of a decoding device that wants to decode position information quickly, it may be able to decode the position information more quickly by obtaining the encoded data of the position information early.

[0215] In FIG. 28, the attributes XSPS and YSPS are combined and described as the attribute SPS, but the attributes XSPS and YSPS may be arranged separately.

[0216] In the parameter set priority sending order, the parameter sets are sent first and the data is sent later.

[0217] As described above, as long as the NAL unit transmission order constraints are met, the multiplexing unit 4802 may transmit the NAL units in any order. For example, order identification information may be defined, and the multiplexing unit 4802 may have the function of transmitting the NAL units in a plurality of order patterns. For example, the NAL unit order identification information may be stored in the stream PS.

[0218] The three-dimensional data decoding device may perform decoding based on the order identification information. The three-dimensional data decoding device may instruct the three-dimensional data encoding device on a desired transmission order, and the three-dimensional data encoding device (multiplexing unit 4802) may control the transmission order in accordance with the instructed transmission order.

[0219] The multiplexing unit 4802 may generate coded data that merges multiple functions, as long as the data transmission order is within the constraints of the data transmission order. For example, as shown in Fig. 28, the GOF header and the AU header may be merged, or the AXPS and the AYPS may be merged. In this case, an identifier indicating that the data has multiple functions is defined in pcc_nal_unit_type.

[0220] A modified example of this embodiment will be described below. PS has levels, such as a frame-level PS, a sequence-level PS, and a PCC sequence-level PS, and if the PCC sequence level is the higher level and the frame level is the lower level, the following method may be used to store parameters.

[0221] The default PS value is indicated in the higher PS. Also, if the value of the lower PS differs from the value of the higher PS, the PS value is indicated in the lower PS. Alternatively, the PS value is not written in the higher PS, but written in the lower PS. Alternatively, information on whether the PS value is to be written in the lower PS, the higher PS, or both is written in either or both of the lower PS and the higher PS. Alternatively, the lower PS may be merged into the higher PS. Alternatively, if the lower PS and the higher PS overlap, the multiplexing unit 4802 may omit sending one of them.

[0222] The encoding unit 4801 or the multiplexing unit 4802 may divide the data into slices or tiles and transmit the divided data. The divided data includes information for identifying the divided data, and parameters used to decode the divided data are included in the parameter set. In this case, an identifier indicating that the data stores data or parameters related to tiles or slices is defined in pcc_nal_unit_type.

[0223] The processing relating to the order identification information will be described below. Fig. 29 is a flowchart of processing by the three-dimensional data encoding device (the encoding unit 4801 and the multiplexing unit 4802) relating to the transmission order of NAL units.

[0224] First, the three-dimensional data encoding device determines the transmission order of NAL units (position information priority or parameter set priority) (S4801). For example, the three-dimensional data encoding device determines the transmission order based on a specification from a user or an external device (e.g., a three-dimensional data decoding device).

[0225] If the determined transmission order is positional information priority (positional information priority in S4802), the three-dimensional data encoding device sets the order identification information included in the stream PS to positional information priority (S4803). In other words, in this case, the order identification information indicates that the NAL units will be transmitted in the order in which positional information is priority. Then, the three-dimensional data encoding device transmits the NAL units in the order in which positional information is priority (S4804).

[0226] On the other hand, if the determined transmission order is parameter set priority (parameter set priority in S4802), the three-dimensional data encoding device sets the order identification information included in the stream PS to parameter set priority (S4805). That is, in this case, the order identification information indicates that the NAL units are to be transmitted in parameter set priority order. Then, the three-dimensional data encoding device transmits the NAL units in parameter set priority order (S4806).

[0227] 30 is a flowchart of processing by the 3D data decoding device relating to the transmission order of NAL units. First, the 3D data decoding device analyzes the order identification information included in the stream PS (S4811).

[0228] If the transmission order indicated by the order identification information is prioritized over position information (position information priority in S4812), the three-dimensional data decoding device decodes the NAL unit assuming that the transmission order of the NAL unit is prioritized over position information (S4813).

[0229] On the other hand, if the transmission order indicated by the order identification information is parameter set priority (parameter set priority in S4812), the three-dimensional data decoding device decodes the NAL unit assuming that the transmission order of the NAL unit is parameter set priority (S4814).

[0230] For example, if the three-dimensional data decoding device does not decode attribute information, in step S4813, it may not acquire all NAL units, but may instead acquire NAL units related to position information and decode the position information from the acquired NAL units.

[0231] Next, a process for generating AUs and GOFs will be described. Fig. 31 is a flowchart of a process performed by the three-dimensional data encoding device (the multiplexing unit 4802) for generating AUs and GOFs in multiplexing NAL units.

[0232] First, the three-dimensional data encoding device determines the type of encoded data (S4821). Specifically, the three-dimensional data encoding device determines whether the encoded data to be processed is the first data of an AU, the first data of a GOF, or other data.

[0233] If the encoded data is data at the beginning of a GOF (GOF beginning in S4822), the three-dimensional data encoding device places a GOF header and an AU header at the beginning of the encoded data belonging to the GOF and generates a NAL unit (S4823).

[0234] If the encoded data is data at the beginning of an AU (AU beginning in S4822), the three-dimensional data encoding device places an AU header at the beginning of the encoded data belonging to the AU and generates a NAL unit (S4824).

[0235] If the encoded data is neither the beginning of a GOF nor the beginning of an AU (other than the beginning of a GOF or an AU in S4822), the three-dimensional data encoding device places the encoded data after the AU header of the AU to which the encoded data belongs and generates a NAL unit (S4825).

[0236] Next, a process for accessing AUs and GOFs will be described. Fig. 32 is a flowchart of a process performed by a three-dimensional data decoding device for accessing AUs and GOFs during demultiplexing of NAL units.

[0237] First, the three-dimensional data decoding device determines the type of coded data included in the NAL unit by analyzing nal_unit_type included in the NAL unit (S4831). Specifically, the three-dimensional data decoding device determines whether the coded data included in the NAL unit is data at the beginning of an AU, data at the beginning of a GOF, or other data.

[0238] If the encoded data contained in the NAL unit is data at the start of the GOF (start of GOF in S4832), the three-dimensional data decoding device determines that the NAL unit is the start position of random access, accesses the NAL unit, and starts the decoding process (S4833).

[0239] On the other hand, if the encoded data contained in the NAL unit is data at the beginning of an AU (AU start at S4832), the three-dimensional data decoding device determines that the NAL unit is the beginning of an AU, accesses the data contained in the NAL unit, and decodes the AU (S4834).

[0240] On the other hand, if the coded data included in the NAL unit is neither the start of a GOF nor the start of an AU (other than the start of a GOF or an AU in S4832), the three-dimensional data decoding device does not process the NAL unit.

[0241] (Embodiment 3) Information on a 3D point cloud includes position information (geometry) and attribute information (attribute). The position information includes coordinates (x coordinate, y coordinate, z coordinate) based on a certain point. When encoding the position information, instead of directly encoding the coordinates of each 3D point, a method is used in which the position of each 3D point is expressed in an octtree representation and the octtree information is encoded to reduce the amount of code.

[0242] On the other hand, the attribute information includes information indicating color information (RGB, YUV, etc.) of each three-dimensional point, reflectance, normal vector, etc. For example, the three-dimensional data encoding device can encode the attribute information using an encoding method different from that for the position information.

[0243] In this embodiment, a method for encoding attribute information will be described. Note that in this embodiment, the attribute information values ​​will be described using integer values. For example, if each color component of RGB or YUV color information has 8-bit precision, each color component will take an integer value between 0 and 255. If the reflectance value has 10-bit precision, the reflectance value will take an integer value between 0 and 1023. Note that if the bit precision of the attribute information is decimal precision, the three-dimensional data encoding device may multiply the value by a scale value and then round it to an integer value so that the value of the attribute information becomes an integer value. Note that the three-dimensional data encoding device may add this scale value to the header of the bitstream, etc.

[0244] As a method for encoding the attribute information of a three-dimensional point, it is possible to calculate a predicted value of the attribute information of the three-dimensional point, and encode the difference (prediction residual) between the value of the original attribute information and the predicted value. For example, if the value of the attribute information of a three-dimensional point p is Ap and the predicted value is Pp, the three-dimensional data encoding device encodes the absolute difference Diffp = |Ap - Pp|. In this case, if the predicted value Pp can be generated with high accuracy, the value of the absolute difference Diffp will be small. Therefore, for example, the amount of code can be reduced by entropy encoding the absolute difference Diffp using a coding table in which the smaller the value, the smaller the number of generated bits.

[0245] One possible method for generating a predicted value of attribute information is to use attribute information of a reference 3D point, which is another 3D point surrounding the target 3D point to be encoded. Here, the reference 3D point is a 3D point within a predetermined distance range from the target 3D point. For example, if there are target 3D point p = (x1, y1, z1) and 3D point q = (x2, y2, z2), the 3D data encoding device calculates the Euclidean distance d(p, q) between 3D point p and 3D point q shown in (Equation A1).

[0246]

[0247] If the Euclidean distance d(p, q) is smaller than a predetermined threshold THd, the three-dimensional data encoding device determines that the position of the three-dimensional point q is close to the position of the target three-dimensional point p, and determines to use the value of the attribute information of the three-dimensional point q to generate a predicted value of the attribute information of the target three-dimensional point p. Note that other distance calculation methods may be used, such as Mahalanobis distance. The three-dimensional data encoding device may also determine that three-dimensional points outside a predetermined distance range from the target three-dimensional point are not used in the prediction process. For example, if a three-dimensional point r exists and the distance d(p, r) between the target three-dimensional point p and the three-dimensional point r is equal to or greater than a threshold THd, the three-dimensional data encoding device may determine that the three-dimensional point r is not used for prediction. Note that the three-dimensional data encoding device may add information indicating the threshold THd to a header of the bitstream, for example.

[0248] 33 is a diagram showing an example of a 3D point. In this example, the distance d(p, q) between the target 3D point p and the 3D point q is smaller than the threshold THd. Therefore, the 3D data encoding device determines that the 3D point q is the reference 3D point of the target 3D point p, and determines that the value of the attribute information Aq of the 3D point q will be used to generate the predicted value Pp of the attribute information Ap of the target 3D point p.

[0249] On the other hand, the distance d(p, r) between the target 3D point p and the 3D point r is equal to or greater than the threshold value THd. Therefore, the 3D data encoding device determines that the 3D point r is not a reference 3D point of the target 3D point p, and determines not to use the value of the attribute information Ar of the 3D point r to generate the predicted value Pp of the attribute information Ap of the target 3D point p.

[0250] Furthermore, when the 3D data encoding device encodes attribute information of a target 3D point using a predicted value, it uses a 3D point whose attribute information has already been encoded and decoded as a reference 3D point. Similarly, when the 3D data decoding device decodes attribute information of a target 3D point to be decoded using a predicted value, it uses a 3D point whose attribute information has already been decoded as a reference 3D point. This allows the same predicted value to be generated during encoding and decoding, so that the bit stream of 3D points generated during encoding can be correctly decoded on the decoding side.

[0251] Furthermore, when encoding attribute information of 3D points, it is possible to classify each 3D point into multiple layers using the position information of the 3D points and then encode them. Here, each classified layer is called LoD (Level of Detail). The method of generating LoD will be described with reference to FIG. 34.

[0252] First, the three-dimensional data encoding device selects an initial point a0 and assigns it to LoD0. Next, the three-dimensional data encoding device extracts point a1 whose distance from point a0 is greater than the LoD0 threshold Thres_LoD[0] and assigns it to LoD0. Next, the three-dimensional data encoding device extracts point a2 whose distance from point a1 is greater than the LoD0 threshold Thres_LoD[0] and assigns it to LoD0. In this way, the three-dimensional data encoding device configures LoD0 so that the distance between each point within LoD0 is greater than the threshold Thres_LoD[0].

[0253] Next, the three-dimensional data encoding device selects point b0, which has not yet been assigned an LoD, and assigns it to LoD1. Next, the three-dimensional data encoding device extracts point b1, whose distance from point b0 is greater than the threshold value Thres_LoD[1] for LoD1 and whose LoD has not been assigned, and assigns it to LoD1. Next, the three-dimensional data encoding device extracts point b2, whose distance from point b1 is greater than the threshold value Thres_LoD[1] for LoD1 and whose LoD has not been assigned, and assigns it to LoD1. In this way, the three-dimensional data encoding device configures LoD1 so that the distance between each point in LoD1 is greater than the threshold value Thres_LoD[1].

[0254] Next, the three-dimensional data encoding device selects point c0, which has not yet been assigned an LoD, and assigns it to LoD2. Next, the three-dimensional data encoding device extracts point c1, whose distance from point c0 is greater than the LoD2 threshold Thres_LoD[2] and whose LoD is not assigned, and assigns it to LoD2. Next, the three-dimensional data encoding device extracts point c2, whose distance from point c1 is greater than the LoD2 threshold Thres_LoD[2] and whose LoD is not assigned, and assigns it to LoD2. In this way, the three-dimensional data encoding device configures LoD2 so that the distance between each point within LoD2 is greater than the threshold Thres_LoD[2]. For example, as shown in FIG. 35, thresholds Thres_LoD[0], Thres_LoD[1], and Thres_LoD[2] for each LoD are set.

[0255] Furthermore, the three-dimensional data encoding device may add information indicating the threshold value of each LoD to the header of the bitstream, etc. For example, in the example shown in Fig. 35, the three-dimensional data encoding device may add threshold values ​​Thres_LoD[0], Thres_LoD[1], and Thres_LoD[2] to the header.

[0256] Furthermore, the three-dimensional data encoding device may assign all three-dimensional points to which an LoD has not been assigned to the lowest layer of the LoD. In this case, the three-dimensional data encoding device can reduce the amount of coding for the header by not adding the threshold value for the lowest layer of the LoD to the header. For example, in the example shown in FIG. 35, the three-dimensional data encoding device adds thresholds Thres_LoD[0] and Thres_LoD[1] to the header, but does not add Thres_LoD[2] to the header. In this case, the three-dimensional data decoding device may estimate the value of Thres_LoD[2] to be 0. Furthermore, the three-dimensional data encoding device may add the number of layers of the LoD to the header. This allows the three-dimensional data decoding device to determine the LoD of the lowest layer using the number of layers of the LoD.

[0257] In addition, by setting the threshold value of each LoD layer to a larger value as the layer becomes higher, the higher the layer (layer closer to LoD0), the greater the distance between three-dimensional points becomes, resulting in a sparse point cloud, and the lower the layer, the closer the distance between three-dimensional points becomes, resulting in a dense point cloud. In the example shown in Figure 35, LoD0 is the top layer.

[0258] Furthermore, the method of selecting the initial three-dimensional point when setting each LoD may depend on the encoding order when encoding the position information. For example, the three-dimensional data encoding device selects the three-dimensional point that was first encoded when encoding the position information as the initial point a0 of LoD0, and configures LoD0 by selecting points a1 and a2 using the initial point a0 as the base point. Then, the three-dimensional data encoding device may select, as the initial point b0 of LoD1, the three-dimensional point whose position information is encoded earliest among the three-dimensional points that do not belong to LoD0. In other words, the three-dimensional data encoding device may select, as the initial point n0 of LoDn, the three-dimensional point whose position information is encoded earliest among the three-dimensional points that do not belong to the upper layer (LoD0 to LoDn-1) of LoDn. As a result, the three-dimensional data decoding device can configure the same LoD as when encoding by using the same initial point selection method when decoding, thereby properly decoding the bitstream. Specifically, the three-dimensional data decoding device selects, as the initial point n0 of LoDn, the three-dimensional point whose position information has been decoded earliest among the three-dimensional points that do not belong to an upper layer of LoDn.

[0259] Below, a method for generating predicted values ​​of attribute information of three-dimensional points using LoD information will be described. For example, when encoding three-dimensional points in order starting from LoD0, a three-dimensional data encoding device generates a target three-dimensional point included in LoD1 using coded and decoded attribute information (hereinafter also simply referred to as "coded") included in LoD0 and LoD1. In this way, the three-dimensional data encoding device generates predicted values ​​of attribute information of three-dimensional points included in LoDn using coded attribute information included in LoDn' (n' <= n). In other words, the three-dimensional data encoding device does not use attribute information of three-dimensional points included in layers below LoDn to calculate predicted values ​​of attribute information of three-dimensional points included in LoDn.

[0260] For example, a three-dimensional data encoding device generates a predicted value of attribute information of a three-dimensional point by calculating the average of attribute values ​​of N or fewer three-dimensional points among encoded three-dimensional points surrounding a target three-dimensional point to be encoded. The three-dimensional data encoding device may also add the value of N to a header of a bitstream, etc. The three-dimensional data encoding device may also change the value of N for each three-dimensional point and add a value of N to each three-dimensional point. This allows an appropriate N to be selected for each three-dimensional point, thereby improving the accuracy of the predicted value. This reduces the prediction residual. The three-dimensional data encoding device may also add the value of N to a header of a bitstream and fix the value of N within the bitstream. This eliminates the need to encode or decode the value of N for each three-dimensional point, thereby reducing the amount of processing. The three-dimensional data encoding device may also encode the value of N separately for each LoD. This allows an appropriate N to be selected for each LoD, thereby improving coding efficiency.

[0261] Alternatively, the three-dimensional data encoding device may calculate a predicted value of the attribute information of a three-dimensional point as a weighted average value of the attribute information of N surrounding encoded three-dimensional points. For example, the three-dimensional data encoding device calculates the weight using distance information between the target three-dimensional point and each of the N surrounding three-dimensional points.

[0262] When a three-dimensional data encoding device encodes the value of N separately for each LoD, for example, the higher the LoD layer, the larger the value of N is set, and the lower the layer, the smaller the value of N is set. Since the distance between the three-dimensional points belonging to the higher LoD layer is greater, it is possible to improve prediction accuracy by setting the value of N to a large value and selecting and averaging multiple surrounding three-dimensional points. Furthermore, since the distance between the three-dimensional points belonging to the lower LoD layer is closer, it is possible to perform efficient prediction by setting the value of N to a small value and reducing the amount of averaging processing.

[0263] 36 is a diagram showing an example of attribute information used for predicted values. As described above, the predicted value of point P included in LoDN is generated using encoded surrounding points P' included in LoDN' (N'<=N). Here, surrounding points P' are selected based on the distance from point P. For example, the predicted value of attribute information of point b2 shown in FIG. 36 is generated using attribute information of points a0, a1, a2, b0, and b1.

[0264] The surrounding points that are selected vary depending on the value of N. For example, when N=5, a0, a1, a2, b0, and b1 are selected as surrounding points of point b2. When N=4, points a0, a1, a2, and b1 are selected based on distance information.

[0265] The predicted value is calculated by a distance-dependent weighted average. For example, in the example shown in FIG. 36, the predicted value a2p of point a2 is calculated by a weighted average of the attribute information of points a0 and a1, as shown in (Equation A2) and (Equation A3). ｉ is the value of the attribute information of point ai.

[0266]

[0267] The predicted value b2p of point b2 is calculated by the weighted average of the attribute information of points a0, a1, a2, b0, and b1, as shown in (Equation A4) to (Equation A6). ｉ is the value of the attribute information of point bi.

[0268]

[0269] The three-dimensional data encoding device may also calculate a difference (prediction residual) between the value of the attribute information of the three-dimensional point and a predicted value generated from the surrounding points, and quantize the calculated prediction residual. For example, the three-dimensional data encoding device performs quantization by dividing the prediction residual by a quantization scale (also called a quantization step). In this case, the smaller the quantization scale, the smaller the error that may occur due to quantization (quantization error). Conversely, the larger the quantization scale, the larger the quantization error.

[0270] The three-dimensional data encoding device may change the quantization scale used for each LoD. For example, the three-dimensional data encoding device may use a smaller quantization scale for higher layers and a larger quantization scale for lower layers. Since the values ​​of attribute information of three-dimensional points belonging to higher layers may be used as predicted values ​​of attribute information of three-dimensional points belonging to lower layers, the encoding efficiency can be improved by reducing the quantization scale for the higher layers to suppress quantization errors that may occur in the higher layers and increasing the accuracy of the predicted values. The three-dimensional data encoding device may add the quantization scale used for each LoD to a header or the like. This allows the three-dimensional data decoding device to correctly decode the quantization scale and therefore properly decode the bitstream.

[0271] Furthermore, the three-dimensional data encoding device may convert signed integer values ​​(signed quantized values), which are prediction residuals after quantization, into unsigned integer values ​​(unsigned quantized values). This eliminates the need to consider the occurrence of negative integers when entropy encoding the prediction residuals. Note that the three-dimensional data encoding device does not necessarily need to convert signed integer values ​​into unsigned integer values; for example, the sign bit may be separately entropy encoded.

[0272] The prediction residual is calculated by subtracting the predicted value from the original value. For example, the prediction residual a2r of point a2 is calculated by subtracting the value A of the attribute information of point a2 from the original value as shown in (Equation A7). ２ The prediction residual b2r of point b2 is calculated by subtracting the predicted value a2p of point a2 from the value B of the attribute information of point b2, as shown in (Equation A8). ２ is calculated by subtracting the predicted value b2p of point b2 from

[0273] a2r = A ２ -a2p...(Formula A7) b2r=B ２ -b2p...(Formula A8)

[0274] Furthermore, the prediction residual is quantized by dividing it by QS (Quantization Step). For example, the quantized value a2q of point a2 is calculated by (Equation A9). The quantized value b2q of point b2 is calculated by (Equation A10). Here, QS_LoD0 is the QS for LoD0, and QS_LoD1 is the QS for LoD1. That is, the QS may be changed according to the LoD.

[0275] a2q=a2r / QS_LoD0...(Formula A9) b2q=b2r / QS_LoD1...(Formula A10)

[0276] Furthermore, the three-dimensional data encoding device converts the signed integer value, which is the quantized value, into an unsigned integer value as follows: If the signed integer value a2q is less than 0, the three-dimensional data encoding device sets the unsigned integer value a2u to -1-(2×a2q). If the signed integer value a2q is 0 or greater, the three-dimensional data encoding device sets the unsigned integer value a2u to 2×a2q.

[0277] Similarly, the three-dimensional data encoding device sets the unsigned integer value b2u to −1−(2×b2q) when the signed integer value b2q is less than 0. The three-dimensional data encoding device sets the unsigned integer value b2u to 2×b2q when the signed integer value b2q is greater than or equal to 0.

[0278] Furthermore, the three-dimensional data encoding device may encode the quantized prediction residuals (unsigned integer values) by entropy coding. For example, the unsigned integer values ​​may be binarized and then subjected to binary arithmetic coding.

[0279] In this case, the three-dimensional data encoding device may switch the binarization method depending on the value of the prediction residual. For example, when the prediction residual pu is smaller than a threshold R_TH, the three-dimensional data encoding device binarizes the prediction residual pu with a fixed number of bits required to express the threshold R_TH. Furthermore, when the prediction residual pu is equal to or greater than the threshold R_TH, the three-dimensional data encoding device binarizes the binarized data of the threshold R_TH and the value of (pu-R_TH) using Exponential-Golomb or the like.

[0280] For example, the three-dimensional data encoding device binarizes the prediction residual pu in 6 bits when the threshold value R_TH is 63 and the prediction residual pu is smaller than 63. Furthermore, when the prediction residual pu is 63 or greater, the three-dimensional data encoding device performs arithmetic encoding by binarizing the binary data (111111) of the threshold value R_TH and (pu-63) using Exponential Golomb algorithm.

[0281] In a more specific example, when the prediction residual pu is 32, the three-dimensional data encoding device generates 6-bit binary data (100000) and arithmetically encodes this bit string. Also, when the prediction residual pu is 66, the three-dimensional data encoding device generates binary data (111111) of the threshold R_TH and a bit string (00100) that expresses the value 3 (66-63) in Exponential Golomb notation, and arithmetically encodes this bit string (111111+00100).

[0282] In this way, the three-dimensional data encoding device switches the binarization method depending on the size of the prediction residual, thereby enabling encoding while suppressing a sudden increase in the number of binarization bits when the prediction residual becomes large. Note that the three-dimensional data encoding device may add the threshold value R_TH to the header of the bitstream, etc.

[0283] For example, when encoding is performed at a high bit rate, that is, when the quantization scale is small, the quantization error is small and the prediction accuracy is high, which may result in a small prediction residual. Therefore, in this case, the three-dimensional data encoding device sets the threshold R_TH to a large value. This reduces the possibility of encoding binarized data at the threshold R_TH, improving encoding efficiency. Conversely, when encoding is performed at a low bit rate, that is, when the quantization scale is large, the quantization error is large and the prediction accuracy is poor, which may result in a large prediction residual. Therefore, in this case, the three-dimensional data encoding device sets the threshold R_TH to a small value. This prevents a sudden increase in the bit length of the binarized data.

[0284] Furthermore, the three-dimensional data encoding device may switch the threshold R_TH for each LoD and add the threshold R_TH for each LoD to a header or the like. That is, the three-dimensional data encoding device may switch the binarization method for each LoD. For example, in higher layers, the distance between three-dimensional points is large, which may result in poor prediction accuracy and large prediction residuals. Therefore, the three-dimensional data encoding device prevents a sudden increase in the bit length of the binarized data by setting a small threshold R_TH for higher layers. Furthermore, in lower layers, the distance between three-dimensional points is small, which may result in high prediction accuracy and small prediction residuals. Therefore, the three-dimensional data encoding device improves encoding efficiency by setting a large threshold R_TH for each layer.

[0285] 37 is a diagram showing an example of an exponential-Golomb code, illustrating the relationship between values ​​(multiple values) before binarization and bits (codes) after binarization. Note that 0 and 1 shown in FIG. 37 may be reversed.

[0286] Furthermore, the three-dimensional data encoding device applies arithmetic coding to the binary data of the prediction residual. This can improve encoding efficiency. Note that when applying arithmetic coding, the tendency of the occurrence probability of 0 and 1 for each bit may differ between the n-bit code (n-bit code), which is the portion of the binary data binarized with n bits, and the remaining code (remaining code), which is the portion binarized using Exponential Golomb Algorithm. Therefore, the three-dimensional data encoding device may switch the application method of arithmetic coding between the n-bit code and the remaining code.

[0287] For example, a three-dimensional data encoder performs arithmetic coding on an n-bit code using a different coding table (probability table) for each bit. In this case, the three-dimensional data encoder may change the number of coding tables used for each bit. For example, the three-dimensional data encoder performs arithmetic coding on the first bit b0 of an n-bit code using one coding table. The three-dimensional data encoder also uses two coding tables for the next bit b1. The three-dimensional data encoder also switches the coding table used for arithmetic coding of bit b1 depending on the value of b0 (0 or 1). Similarly, the three-dimensional data encoder uses four coding tables for the next bit b2. The three-dimensional data encoder also switches the coding table used for arithmetic coding of bit b2 depending on the values ​​of b0 and b1 (0 to 3).

[0288] In this way, the three-dimensional data encoding device performs arithmetic encoding on each bit bn-1 of the n-bit code. ｎ－１ The three-dimensional data encoding device uses coding tables. In addition, the three-dimensional data encoding device switches the coding table to be used depending on the value (occurrence pattern) of the bits before bn-1. This allows the three-dimensional data encoding device to use an appropriate coding table for each bit, thereby improving coding efficiency.

[0289] The three-dimensional data encoding device may reduce the number of encoding tables used for each bit. For example, when arithmetically encoding each bit bn-1, the three-dimensional data encoding device may use 2 or more encoding tables depending on the value (occurrence pattern) of the m bits (m<n-1) before bn-1. ｍ Alternatively, the encoding table may be switched among these. This reduces the number of encoding tables used for each bit, while improving encoding efficiency. The three-dimensional data encoding device may update the occurrence probabilities of 0 and 1 in each encoding table according to the value of the binary data that actually occurs. The three-dimensional data encoding device may also fix the occurrence probabilities of 0 and 1 in the encoding table for some bits. This reduces the number of updates to the occurrence probabilities, thereby reducing the amount of processing.

[0290] For example, if the n-bit code is b0b1b2...bn-1, there is one coding table for b0 (CTb0). There are two coding tables for b1 (CTb10, CTb11). The coding table to be used is switched according to the value of b0 (0 to 1). There are four coding tables for b2 (CTb20, CTb21, CTb22, CTb23). The coding table to be used is switched according to the values ​​of b0 and b1 (0 to 3). There are two coding tables for bn-1. ｎ－１ (CTbn0, CTbn1, ..., CTbn(2 ｎ－１ -1)). Also, the values ​​of b0b1...bn-2 (0 to 2 ｎ－１ -1) The encoding table to be used can be switched according to the above.

[0291] The three-dimensional data encoding device does not binarize the n-bit code, but encodes it as 0 to 2. ｎ Arithmetic coding by m-ary (m=2) ｎ ) may be applied. Furthermore, when the three-dimensional data encoding device performs m-ary arithmetic encoding of an n-bit code, the three-dimensional data decoding device may also restore the n-bit code by m-ary arithmetic decoding.

[0292] 38 is a diagram for explaining processing when, for example, the residual code is an exponential Golomb code. The residual code, which is the portion binarized using the exponential Golomb algorithm, includes a prefix portion and a suffix portion as shown in FIG. 38. For example, the three-dimensional data encoding device switches encoding tables between the prefix portion and the suffix portion. That is, the three-dimensional data encoding device arithmetically encodes each bit included in the prefix portion using a prefix encoding table, and arithmetically encodes each bit included in the suffix portion using a suffix encoding table.

[0293] The three-dimensional data encoding device may update the occurrence probabilities of 0 and 1 in each encoding table according to the value of the binary data that actually occurs. Alternatively, the three-dimensional data encoding device may fix the occurrence probabilities of 0 and 1 in one of the encoding tables. This reduces the number of updates to the occurrence probabilities, thereby reducing the amount of processing. For example, the three-dimensional data encoding device may update the occurrence probabilities for the prefix part and fix the occurrence probabilities for the suffix part.

[0294] Furthermore, the three-dimensional data encoding device decodes the quantized prediction residual by inverse quantization and reconstruction, and uses the decoded value, which is the decoded prediction residual, for prediction of the three-dimensional point to be encoded and beyond. Specifically, the three-dimensional data encoding device calculates an inverse quantization value by multiplying the quantized prediction residual (quantization value) by the quantization scale, and obtains a decoded value (reconstruction value) by adding the inverse quantization value and the prediction value.

[0295] For example, the inverse quantization value a2iq of point a2 is calculated by (Equation A11) using the quantization value a2q of point a2. The inverse quantization value b2iq of point b2 is calculated by (Equation A12) using the quantization value b2q of point b2. Here, QS_LoD0 is the QS for LoD0, and QS_LoD1 is the QS for LoD1. That is, the QS may be changed depending on the LoD.

[0296] a2iq=a2q×QS_LoD0...(Formula A11) b2iq=b2q×QS_LoD1...(Formula A12)

[0297] For example, the decoded value a2rec of point a2 is calculated by adding the predicted value a2p of point a2 to the inverse quantized value a2iq of point a2, as shown in (Equation A13). The decoded value b2rec of point b2 is calculated by adding the predicted value b2p of point b2 to the inverse quantized value b2iq of point b2, as shown in (Equation A14).

[0298] a2rec=a2iq+a2p...(Formula A13) b2rec=b2iq+b2p...(Formula A14)

[0299] An example of the syntax of a bitstream according to this embodiment will be described below. Fig. 39 is a diagram showing an example of the syntax of an attribute header (attribute_header) according to this embodiment. The attribute header is header information of attribute information. As shown in Fig. 39, the attribute header includes hierarchical number information (NumLoD), three-dimensional point number information (NumOfPoint[i]), hierarchical threshold (Thres_Lod[i]), surrounding point number information (NumNeighborPoint[i]), predicted threshold (THd[i]), quantization scale (QS[i]), and binarization threshold (R_TH[i]).

[0300] The number of layers information (NumLoD) indicates the number of layers of LoD to be used.

[0301] The three-dimensional point number information (NumOfPoint[i]) indicates the number of three-dimensional points belonging to layer i. Note that the three-dimensional data encoding device may add three-dimensional point total number information (AllNumOfPoint) indicating the total number of three-dimensional points to a separate header. In this case, the three-dimensional data encoding device does not need to add NumOfPoint[NumLoD-1] indicating the number of three-dimensional points belonging to the lowest layer to the header. In this case, the three-dimensional data decoding device can calculate NumOfPoint[NumLoD-1] using (Equation A15). This allows the amount of code in the header to be reduced.

[0302]

[0303] The layer threshold (Thres_Lod[i]) is a threshold used to set layer i. The three-dimensional data encoding device and the three-dimensional data decoding device configure LoDi so that the distance between each point in LoDi is greater than the threshold Thres_LoD[i]. Furthermore, the three-dimensional data encoding device does not need to add the value of Thres_Lod[NumLoD-1] (the lowest layer) to the header. In this case, the three-dimensional data decoding device estimates the value of Thres_Lod[NumLoD-1] to be 0. This allows the amount of code in the header to be reduced.

[0304] The surrounding point number information (NumNeighborPoint[i]) indicates the upper limit of the number of surrounding points used to generate a predicted value of a three-dimensional point belonging to layer i. If the number of surrounding points M is less than NumNeighborPoint[i] (M<NumNeighborPoint[i]), the three-dimensional data encoding device may calculate a predicted value using M surrounding points. Furthermore, if it is not necessary to separate the value of NumNeighborPoint[i] for each LoD, the three-dimensional data encoding device may add one surrounding point number information (NumNeighborPoint) used for all LoDs to the header.

[0305] The prediction threshold (THd[i]) indicates the upper limit of the distance between the target 3D point and surrounding 3D points used to predict the target 3D point to be encoded or decoded at layer i. The 3D data encoding device and the 3D data decoding device do not use 3D points that are farther away from the target 3D point than THd[i] for prediction. Note that if the 3D data encoding device does not need to separate the THd[i] values ​​for each LoD, it may add a single prediction threshold (THd) to the header that is used for all LoDs.

[0306] The quantization scale (QS[i]) indicates the quantization scale used in the quantization and dequantization of layer i.

[0307] The binarization threshold (R_TH[i]) is a threshold for switching the binarization method for the prediction residual of a three-dimensional point belonging to layer i. For example, if the prediction residual is smaller than the threshold R_TH, the three-dimensional data encoding device binarizes the prediction residual pu with a fixed number of bits, and if the prediction residual is equal to or greater than the threshold R_TH, the binarized data of the threshold R_TH and the value (pu-R_TH) are binarized using the exponential Golomb algorithm. Note that if it is not necessary to switch the value of R_TH[i] for each LoD, the three-dimensional data encoding device may add a single binarization threshold (R_TH) used for all LoDs to the header.

[0308] Note that R_TH[i] may be the maximum value that can be expressed in n bits. For example, in 6 bits, R_TH is 63, and in 8 bits, R_TH is 255. Furthermore, the three-dimensional data encoding device may encode the number of bits instead of encoding the maximum value that can be expressed in n bits as the binarization threshold. For example, the three-dimensional data encoding device may add a value of 6 to the header when R_TH[i] = 63, or a value of 8 to the header when R_TH[i] = 255. Furthermore, the three-dimensional data encoding device may define a minimum value (minimum number of bits) for the number of bits that represent R_TH[i], and add the number of bits relative to the minimum value to the header. For example, the three-dimensional data encoding device may add a value of 0 to the header when R_TH[i] = 63 and the minimum number of bits is 6, or add a value of 2 to the header when R_TH[i] = 255 and the minimum number of bits is 6.

[0309] Furthermore, the three-dimensional data encoding device may entropy-encode at least one of NumLoD, Thres_Lod[i], NumNeighborPoint[i], THd[i], QS[i], and R_TH[i] and add it to the header. For example, the three-dimensional data encoding device may binarize each value and arithmetically encode it. Furthermore, the three-dimensional data encoding device may encode each value at a fixed length to reduce the amount of processing.

[0310] Furthermore, the three-dimensional data encoding device may not add at least one of NumLod, Thres_Lod[i], NumNeighborPoint[i], THd[i], QS[i], and R_TH[i] to the header. For example, the value of at least one of these may be specified by a profile or level of a standard, etc. This allows the number of bits in the header to be reduced.

[0311] 40 is a diagram showing an example of the syntax of attribute data (attribute_data) according to this embodiment. This attribute data includes coded data of attribute information of multiple three-dimensional points. As shown in FIG. 40, the attribute data includes an n-bit code and a remaining code.

[0312] An n-bit code is coded data of the prediction residual of the attribute information value or a part thereof. The bit length of the n-bit code depends on the value of R_TH[i]. For example, if the value indicated by R_TH[i] is 63, the n-bit code is 6 bits, and if the value indicated by R_TH[i] is 255, the n-bit code is 8 bits.

[0313] The remaining code is the coded data of the prediction residual of the attribute information value that is coded using the exponential Golomb coding method. This remaining code is coded or decoded when the n-bit code is the same as R_TH[i]. The three-dimensional data decoding device also adds the value of the n-bit code and the value of the remaining code to decode the prediction residual. Note that if the n-bit code is not the same value as R_TH[i], the remaining code does not need to be coded or decoded.

[0314] The flow of processing in the three-dimensional data encoding device will be explained below. Fig. 41 is a flowchart of three-dimensional data encoding processing by the three-dimensional data encoding device.

[0315] First, the three-dimensional data encoding device encodes position information (geometry) (S3001). For example, the three-dimensional data is encoded using an octree representation.

[0316] If the position of a three-dimensional point changes due to quantization or the like after encoding the position information, the three-dimensional data encoding device reallocates the attribute information of the original three-dimensional point to the changed three-dimensional point (S3002). For example, the three-dimensional data encoding device performs the reallocation by interpolating the value of the attribute information according to the amount of change in position. For example, the three-dimensional data encoding device detects N pre-change three-dimensional points that are close to the changed three-dimensional position and calculates a weighted average of the attribute information values ​​of the N three-dimensional points. For example, the three-dimensional data encoding device determines a weight for the weighted average based on the distance from the changed three-dimensional position to each of the N three-dimensional points. The three-dimensional data encoding device then determines the value obtained by the weighted average as the value of the attribute information of the changed three-dimensional point. Furthermore, if two or more three-dimensional points change to the same three-dimensional position due to quantization or the like, the three-dimensional data encoding device may assign the average value of the attribute information of the two or more pre-change three-dimensional points as the value of the attribute information of the changed three-dimensional point.

[0317] Next, the three-dimensional data encoding device encodes the reallocated attribute information (Attribute) (S3003). For example, when encoding multiple types of attribute information, the three-dimensional data encoding device may encode the multiple types of attribute information in order. For example, when encoding color and reflectance as attribute information, the three-dimensional data encoding device may generate a bitstream in which the encoding result of reflectance is added after the encoding result of color. Note that the order in which the encoding results of the multiple attribute information are added to the bitstream is not limited to this order and may be any order.

[0318] The three-dimensional data encoding device may also add information indicating the start location of the encoded data for each piece of attribute information in the bitstream to a header or the like. This allows the three-dimensional data decoding device to selectively decode attribute information that requires decoding, thereby omitting the decoding process for attribute information that does not require decoding. This reduces the amount of processing required by the three-dimensional data decoding device. The three-dimensional data encoding device may also encode multiple types of attribute information in parallel and combine the encoding results into a single bitstream. This allows the three-dimensional data encoding device to encode multiple types of attribute information at high speed.

[0319] 42 is a flowchart of the attribute information encoding process (S3003). First, the three-dimensional data encoding device sets LoD (S3011). That is, the three-dimensional data encoding device assigns each three-dimensional point to one of a plurality of LoDs.

[0320] Next, the three-dimensional data encoding device starts a loop for each LoD (S3012). That is, the three-dimensional data encoding device repeatedly performs the processes of steps S3013 to S3021 for each LoD.

[0321] Next, the three-dimensional data encoding device starts a loop for each three-dimensional point (S3013). That is, the three-dimensional data encoding device repeatedly performs the processes of steps S3014 to S3020 for each three-dimensional point.

[0322] First, the three-dimensional data encoding device searches for multiple surrounding points, which are three-dimensional points that exist around the target three-dimensional point to be processed, and are used to calculate a predicted value of the target three-dimensional point (S3014). Next, the three-dimensional data encoding device calculates a weighted average of the attribute information values ​​of the multiple surrounding points and sets the obtained value as the predicted value P (S3015). Next, the three-dimensional data encoding device calculates a prediction residual, which is the difference between the attribute information of the target three-dimensional point and the predicted value (S3016). Next, the three-dimensional data encoding device calculates a quantized value by quantizing the prediction residual (S3017). Next, the three-dimensional data encoding device arithmetically encodes the quantized value (S3018).

[0323] The three-dimensional data encoding device also calculates inverse quantized values ​​by inverse quantizing the quantized values ​​(S3019). Next, the three-dimensional data encoding device generates decoded values ​​by adding predicted values ​​to the inverse quantized values ​​(S3020). Next, the three-dimensional data encoding device ends the loop for each three-dimensional point (S3021). The three-dimensional data encoding device also ends the loop for each LoD (S3022).

[0324] Hereinafter, a three-dimensional data decoding process in a three-dimensional data decoding device that decodes a bit stream generated by the above three-dimensional data encoding device will be described.

[0325] The three-dimensional data decoding device generates decoded binary data by arithmetically decoding the binary data of the attribute information in the bit stream generated by the three-dimensional data encoding device in the same manner as the three-dimensional data encoding device. Note that, in the three-dimensional data encoding device, if the application method of arithmetic coding is switched between the part binarized with n bits (n-bit code) and the part binarized using Exponential Golomb (remaining code), the three-dimensional data decoding device performs decoding accordingly when applying arithmetic decoding.

[0326] For example, in an arithmetic decoding method for an n-bit code, a three-dimensional data decoding device performs arithmetic decoding using a different coding table (decoding table) for each bit. In this case, the three-dimensional data decoding device may change the number of coding tables used for each bit. For example, arithmetic decoding is performed using one coding table for the first bit b0 of an n-bit code. The three-dimensional data decoding device also uses two coding tables for the next bit b1. The three-dimensional data decoding device also switches the coding table used for arithmetic decoding of bit b1 depending on the value of b0 (0 or 1). Similarly, the three-dimensional data decoding device uses four coding tables for the next bit b2. The three-dimensional data decoding device also switches the coding table used for arithmetic decoding of bit b2 depending on the values ​​of b0 and b1 (0 to 3).

[0327] In this way, the three-dimensional data decoding device performs arithmetic decoding on each bit bn-1 of the n-bit code. ｎ－１ The three-dimensional data decoding device uses coding tables. Furthermore, the three-dimensional data decoding device switches the coding table to be used depending on the value (occurrence pattern) of the bits before bn-1. This allows the three-dimensional data decoding device to use an appropriate coding table for each bit, thereby enabling appropriate decoding of a bitstream with improved coding efficiency.

[0328] The three-dimensional data decoding device may reduce the number of coding tables used for each bit. For example, when arithmetically decoding each bit bn-1, the three-dimensional data decoding device may use 2 or more coding tables depending on the value (occurrence pattern) of the m bits (m<n-1) before bn-1. ｍThe three-dimensional data decoding device may switch among the coding tables. This allows the three-dimensional data decoding device to appropriately decode a bitstream with improved coding efficiency while reducing the number of coding tables used for each bit. The three-dimensional data decoding device may update the occurrence probabilities of 0 and 1 in each coding table according to the value of the binary data that actually occurs. The three-dimensional data decoding device may also fix the occurrence probabilities of 0 and 1 in the coding table for some bits. This reduces the number of updates to the occurrence probabilities, thereby reducing the amount of processing.

[0329] For example, if the n-bit code is b0b1b2...bn-1, there is one coding table for b0 (CTb0). There are two coding tables for b1 (CTb10, CTb11). The coding table is switched depending on the value of b0 (0 to 1). There are four coding tables for b2 (CTb20, CTb21, CTb22, CTb23). The coding table is switched depending on the values ​​of b0 and b1 (0 to 3). There are two coding tables for bn-1 (CTb20, CTb21, CTb22, CTb23). ｎ－１ (CTbn0, CTbn1, ..., CTbn(2 ｎ－１ -1)). Also, the values ​​of b0b1...bn-2 (0 to 2 ｎ－１ -1) the encoding table is switched.

[0330] 43 is a diagram for explaining processing when, for example, the residual code is an exponential Golomb code. The portion (residual code) binarized and encoded by the three-dimensional data encoding device using the exponential Golomb algorithm includes a prefix section and a suffix section, as shown in FIG. 43. For example, the three-dimensional data decoding device switches encoding tables between the prefix section and the suffix section. That is, the three-dimensional data decoding device arithmetically decodes each bit included in the prefix section using the encoding table for the prefix, and arithmetically decodes each bit included in the suffix section using the encoding table for the suffix.

[0331] The three-dimensional data decoding device may update the occurrence probabilities of 0 and 1 in each encoding table according to the value of the binarized data generated during decoding. Alternatively, the three-dimensional data decoding device may fix the occurrence probabilities of 0 and 1 in one of the encoding tables. This reduces the number of updates to the occurrence probabilities, thereby reducing the amount of processing. For example, the three-dimensional data decoding device may update the occurrence probabilities for the prefix part and fix the occurrence probabilities for the suffix part.

[0332] Furthermore, the three-dimensional data decoding device decodes the quantized prediction residual (unsigned integer value) by multi-valuing the binary data of the arithmetically decoded prediction residual in accordance with the encoding method used in the three-dimensional data encoding device. The three-dimensional data decoding device first calculates the value of the decoded n-bit code by arithmetically decoding the binary data of the n-bit code. Next, the three-dimensional data decoding device compares the value of the n-bit code with the value of R_TH.

[0333] When the value of the n-bit code matches the value of R_TH, the three-dimensional data decoding device determines that a bit coded using the exponential-Golomb coding method exists next, and arithmetically decodes the remaining code, which is binary data coded using the exponential-Golomb coding method.The three-dimensional data decoding device then calculates the value of the remaining code from the decoded remaining code using a reverse lookup table showing the relationship between the remaining code and its value. Figure 44 is a diagram showing an example of a reverse lookup table showing the relationship between the remaining code and its value.Next, the three-dimensional data decoding device obtains a multi-valued post-quantization prediction residual by adding the value of the obtained remaining code to R_TH.

[0334] On the other hand, if the value of the n-bit code does not match the value of R_TH (the value is smaller than R_TH), the three-dimensional data decoding device determines the value of the n-bit code as the multi-valued post-quantization prediction residual. This allows the three-dimensional data decoding device to properly decode a bitstream generated by the three-dimensional data encoding device by switching the binarization method depending on the value of the prediction residual.

[0335] If the threshold value R_TH is added to a header or the like of a bitstream, the three-dimensional data decoding device may decode the value of the threshold value R_TH from the header and switch the decoding method using the decoded threshold value R_TH. Also, if the threshold value R_TH is added to a header or the like for each LoD, the three-dimensional data decoding device switches the decoding method using the decoded threshold value R_TH for each LoD.

[0336] For example, if the threshold value R_TH is 63 and the value of the decoded n-bit code is 63, the three-dimensional data decoding device obtains the value of the residual code by decoding the residual code using the exponential Golomb method. For example, in the example shown in FIG. 44 , the residual code is 00100, and the value of the residual code is 3. Next, the three-dimensional data decoding device obtains the value of the prediction residual, 66, by adding the value of the threshold value R_TH, 63, and the value of the residual code, 3.

[0337] Furthermore, if the value of the decoded n-bit code is 32, the three-dimensional data decoding device sets the value of the n-bit code, 32, as the value of the prediction residual.

[0338] Furthermore, the three-dimensional data decoding device converts the decoded quantized prediction residuals from unsigned integer values ​​to signed integer values, for example, by a process reverse to that performed by the three-dimensional data encoding device. This allows the three-dimensional data decoding device to appropriately decode the generated bitstream when entropy encoding the prediction residuals, without taking into account the occurrence of negative integers. Note that the three-dimensional data decoding device does not necessarily need to convert unsigned integer values ​​to signed integer values; for example, when decoding a bitstream generated by separately entropy encoding the sign bits, the sign bits may be decoded.

[0339] The three-dimensional data decoding device generates decoded values ​​by decoding the quantized prediction residuals converted into signed integer values ​​through inverse quantization and reconstruction. The three-dimensional data decoding device also uses the generated decoded values ​​for prediction of the three-dimensional point to be decoded and thereafter. Specifically, the three-dimensional data decoding device calculates an inverse quantization value by multiplying the quantized prediction residuals by the decoded quantization scale, and obtains a decoded value by adding the inverse quantization value and the prediction value.

[0340] The decoded unsigned integer value (unsigned quantized value) is converted into a signed integer value by the following process. If the LSB (least significant bit) of the decoded unsigned integer value a2u is 1, the three-dimensional data decoding device sets the signed integer value a2q to -((a2u+1)>>1). If the LSB of the unsigned integer value a2u is not 1, the three-dimensional data decoding device sets the signed integer value a2q to (a2u>>1).

[0341] Similarly, if the LSB of the decoded unsigned integer value b2u is 1, the three-dimensional data decoding device sets the signed integer value b2q to -((b2u+1)>>1). If the LSB of the unsigned integer value n2u is not 1, the three-dimensional data decoding device sets the signed integer value b2q to (b2u>>1).

[0342] Furthermore, the details of the inverse quantization and reconstruction processing by the three-dimensional data decoding device are the same as those of the inverse quantization and reconstruction processing by the three-dimensional data encoding device.

[0343] The flow of processing in the three-dimensional data decoding device will be described below. Figure 45 is a flowchart of three-dimensional data decoding processing by the three-dimensional data decoding device. First, the three-dimensional data decoding device decodes position information (geometry) from the bit stream (S3031). For example, the three-dimensional data decoding device performs decoding using an octree representation.

[0344] Next, the three-dimensional data decoding device decodes attribute information (Attribute) from the bitstream (S3032). For example, when decoding multiple types of attribute information, the three-dimensional data decoding device may decode the multiple types of attribute information in order. For example, when decoding color and reflectance as attribute information, the three-dimensional data decoding device decodes the encoding result of color and the encoding result of reflectance in the order in which they are added to the bitstream. For example, when the encoding result of reflectance is added after the encoding result of color in the bitstream, the three-dimensional data decoding device decodes the encoding result of color, and then decodes the encoding result of reflectance. Note that the three-dimensional data decoding device may decode the encoding results of attribute information added to the bitstream in any order.

[0345] The three-dimensional data decoding device may also obtain information indicating the start location of the encoded data for each piece of attribute information in the bitstream by decoding a header or the like. This allows the three-dimensional data decoding device to selectively decode attribute information that requires decoding, thereby omitting the decoding process for attribute information that does not require decoding. This reduces the amount of processing required by the three-dimensional data decoding device. The three-dimensional data decoding device may also decode multiple types of attribute information in parallel and integrate the decoding results into a single three-dimensional point cloud. This allows the three-dimensional data decoding device to decode multiple types of attribute information at high speed.

[0346] 46 is a flowchart of the attribute information decoding process (S3032). First, the three-dimensional data decoding device sets the LoD (S3041). That is, the three-dimensional data decoding device assigns each of a plurality of three-dimensional points having decoded position information to one of a plurality of LoDs. For example, this assignment method is the same as the assignment method used in the three-dimensional data encoding device.

[0347] Next, the three-dimensional data decoding apparatus starts a loop for each LoD (S3042). That is, the three-dimensional data decoding apparatus repeatedly performs the processes of steps S3043 to S3049 for each LoD.

[0348] Next, the three-dimensional data decoding device starts a loop for each three-dimensional point (S3043). That is, the three-dimensional data decoding device repeatedly performs the processes of steps S3044 to S3048 for each three-dimensional point.

[0349] First, the three-dimensional data decoding device searches for a plurality of surrounding points, which are three-dimensional points existing around the target three-dimensional point to be processed, and are used to calculate a predicted value of the target three-dimensional point (S3044). Next, the three-dimensional data decoding device calculates a weighted average of the values ​​of the attribute information of the plurality of surrounding points, and sets the obtained value as the predicted value P (S3045). Note that these processes are similar to those in the three-dimensional data encoding device.

[0350] Next, the 3D data decoding device arithmetically decodes the quantized value from the bitstream (S3046). The 3D data decoding device also calculates an inverse quantized value by inverse quantizing the decoded quantized value (S3047). The 3D data decoding device then generates a decoded value by adding a predicted value to the inverse quantized value (S3048). The 3D data decoding device then ends the loop in 3D point units (S3049). The 3D data decoding device also ends the loop in LoD units (S3050).

[0351] Next, the configurations of a three-dimensional data encoding device and a three-dimensional data decoding device according to this embodiment will be described. Fig. 47 is a block diagram showing the configuration of a three-dimensional data encoding device 3000 according to this embodiment. This three-dimensional data encoding device 3000 comprises a position information encoding unit 3001, an attribute information reallocation unit 3002, and an attribute information encoding unit 3003.

[0352] The attribute information encoding unit 3003 encodes position information (geometry) of multiple 3D points included in the input point cloud. The attribute information reallocation unit 3002 reallocates values ​​of attribute information of multiple 3D points included in the input point cloud using the results of encoding and decoding the position information. The attribute information encoding unit 3003 encodes the reallocated attribute information. In addition, the 3D data encoding device 3000 generates a bitstream including the encoded position information and the encoded attribute information.

[0353] 48 is a block diagram showing the configuration of a three-dimensional data decoding device 3010 according to this embodiment. This three-dimensional data decoding device 3010 includes a position information decoding unit 3011 and an attribute information decoding unit 3012.

[0354] The position information decoding unit 3011 decodes position information (geometry) of multiple 3D points from the bit stream. The attribute information decoding unit 3012 decodes attribute information (attribute) of multiple 3D points from the bit stream. The 3D data decoding device 3010 generates an output point cloud by combining the decoded position information and the decoded attribute information.

[0355] (Embodiment 4) Hereinafter, a method using RAHT (Region Adaptive Hierarchical Transform) will be described as another method for encoding attribute information of three-dimensional points. Fig. 49 is a diagram for explaining encoding of attribute information using RAHT.

[0356] First, the three-dimensional data encoding device generates a Morton code based on the position information of the three-dimensional points, and then sorts the attribute information of the three-dimensional points in Morton code order. For example, the three-dimensional data encoding device may sort in ascending order of the Morton codes. Note that the sorting order is not limited to the Morton code order, and other orders may be used.

[0357] Next, the three-dimensional data encoding device applies a Haar transform to the attribute information of two adjacent three-dimensional points in Morton code order to generate high-frequency components and low-frequency components of layer L. For example, the three-dimensional data encoding device may use a 2×2 matrix Haar transform. The generated high-frequency components are included in the encoding coefficients as high-frequency components of layer L, and the generated low-frequency components are used as input values ​​for layer L+1, which is higher than layer L.

[0358] After generating high-frequency components for layer L using the attribute information for layer L, the three-dimensional data encoding device continues processing for layer L+1. In the processing for layer L+1, the three-dimensional data encoding device generates high-frequency and low-frequency components for layer L+1 by applying a Haar transform to two low-frequency components obtained by Haar transforming the attribute information for layer L. The generated high-frequency components are included in the coding coefficients as high-frequency components for layer L+1, and the generated low-frequency components are used as input values ​​for layer L+2, which is higher than layer L+1.

[0359] The three-dimensional data encoding device repeats this hierarchical processing until it determines that it has reached the highest hierarchical level Lmax when only one low-frequency component is input to the hierarchical level. The three-dimensional data encoding device includes the low-frequency component of hierarchical level Lmax-1 input to hierarchical level Lmax in the encoding coefficients. The values ​​of the low-frequency or high-frequency components included in the encoding coefficients are then quantized and encoded using entropy encoding or the like.

[0360] In addition, when applying the Haar transform, if there is only one three-dimensional point as two adjacent three-dimensional points, the three-dimensional data encoding device may use the value of the attribute information of the one existing three-dimensional point as the input value of the higher hierarchy.

[0361] In this way, the three-dimensional data encoding device applies a hierarchical Haar transform to the input attribute information to generate high-frequency and low-frequency components of the attribute information, and then performs encoding by applying quantization, etc., which will be described later, thereby improving encoding efficiency.

[0362] When the attribute information is N-dimensional, the three-dimensional data encoding device may apply the Haar transform independently to each dimension to calculate the respective encoding coefficients. For example, when the attribute information is color information (RGB, YUV, etc.), the three-dimensional data encoding device applies the Haar transform to each component to calculate the respective encoding coefficients.

[0363] The three-dimensional data encoding device may apply the Haar transform in the order of layer L, L+1, ..., layer Lmax. The closer to layer Lmax, the more encoding coefficients are generated that include low-frequency components of the input attribute information.

[0364] w0 and w1 shown in Figure 49 are weights assigned to each three-dimensional point. For example, the three-dimensional data encoding device may calculate weights based on distance information between two adjacent three-dimensional points to which the Haar transform is applied. For example, the three-dimensional data encoding device may improve encoding efficiency by increasing the weight as the distance becomes shorter. Note that the three-dimensional data encoding device may calculate these weights using a different method, or may not use weights at all.

[0365] In the example shown in Fig. 49, the input attribute information is a0, a1, a2, a3, a4, and a5. Of the coding coefficients after Haar transform, Ta1, Ta5, Tb1, Tb3, Tc1, and d0 are coded. The other coding coefficients (b0, b2, c0, etc.) are intermediate values ​​and are not coded.

[0366] 49, a high-frequency component Ta1 and a low-frequency component b0 are generated by performing a Haar transform on a0 and a1. Here, when the weights w0 and w1 are equal, the low-frequency component b0 is the average value of a0 and a1, and the high-frequency component Ta1 is the difference between a0 and a1.

[0367] Since there is no attribute information paired with a2, a2 ​​is used as b1. Similarly, there is no attribute information paired with a3, so a3 is used as b2. Furthermore, by performing Haar transform on a4 and a5, a high-frequency component Ta5 and a low-frequency component b3 are generated.

[0368] At layer L+1, a Haar transform is performed on b0 and b1 to generate a high-frequency component Tb1 and a low-frequency component c0. Similarly, a Haar transform is performed on b2 and b3 to generate a high-frequency component Tb3 and a low-frequency component c1.

[0369] At layer Lmax-1, a Haar transform is performed on c0 and c1 to generate a high frequency component Tc1 and a low frequency component d0.

[0370] The three-dimensional data encoding device may quantize the encoding coefficients after applying the Haar transform before encoding. For example, the three-dimensional data encoding device performs quantization by dividing the encoding coefficients by a quantization scale (also called a quantization step (QS)). In this case, the smaller the quantization scale, the smaller the error (quantization error) that may occur due to quantization. Conversely, the larger the quantization scale, the larger the quantization error.

[0371] The three-dimensional data encoding device may change the value of the quantization scale for each layer. FIG. 50 is a diagram showing an example of setting the quantization scale for each layer. For example, the three-dimensional data encoding device may set a smaller quantization scale for higher layers and a larger quantization scale for lower layers. The coding coefficients of three-dimensional points belonging to higher layers contain more low-frequency components than lower layers, and are therefore likely to be important components in terms of human visual characteristics, etc. Therefore, by reducing the quantization scale for higher layers and suppressing quantization errors that may occur in the higher layers, visual degradation can be suppressed and encoding efficiency can be improved.

[0372] The three-dimensional data encoding device may add the quantization scale for each layer to a header, etc. This allows the three-dimensional data decoding device to correctly decode the quantization scale and appropriately decode the bitstream.

[0373] Furthermore, the three-dimensional data encoding device may adaptively switch the value of the quantization scale according to the importance of the target three-dimensional point to be encoded. For example, the three-dimensional data encoding device may use a small quantization scale for three-dimensional points of high importance and a large quantization scale for three-dimensional points of low importance. For example, the three-dimensional data encoding device may calculate the importance from weights used in the Haar transform. For example, the three-dimensional data encoding device may calculate the quantization scale using the sum of w0 and w1. By reducing the quantization scale for three-dimensional points of high importance in this way, the quantization error is reduced, and encoding efficiency can be improved.

[0374] Alternatively, the value of QS may be made smaller in higher layers, which increases the value of QW in higher layers and reduces the quantization error of the 3D points, thereby improving prediction efficiency.

[0375] Here, the coding coefficient Ta1q after quantization of the coding coefficient Ta1 of the attribute information a1 is expressed as Ta1 / QS_L. Note that QS may be the same value in all layers or in some layers.

[0376] QW (Quantization Weight) is a value representing the importance of the 3D point to be coded. For example, the sum of the above-mentioned w0 and w1 may be used as QW. As a result, the higher the layer, the larger the QW value, and the prediction efficiency can be improved by suppressing the quantization error of the 3D point.

[0377] For example, the three-dimensional data encoding device may initially initialize the QW values ​​of all three-dimensional points to 1, and then update the QW of each three-dimensional point using the values ​​of w0 and w1 during the Haar transform. Alternatively, the three-dimensional data encoding device may change the initial value according to the layer, rather than initializing the QW of all three-dimensional points to the value 1. For example, the quantization scale of the higher layer is reduced by setting a larger initial value for QW for the higher layer. This reduces prediction errors in the higher layer, thereby improving prediction accuracy in the lower layer and encoding efficiency. Note that the three-dimensional data encoding device does not necessarily have to use QW.

[0378] When QW is used, the quantized value Ta1q of Ta1 is calculated by (Equation K1) and (Equation K2).

[0379]

[0380] Furthermore, the three-dimensional data encoding device scans and encodes the quantized encoding coefficients (unsigned integer values) in a certain order. For example, the three-dimensional data encoding device encodes a plurality of three-dimensional points in order from the three-dimensional points included in the upper layer toward the lower layer.

[0381] For example, in the example shown in Figure 49, the three-dimensional data encoding device encodes multiple three-dimensional points included in the upper layer Lmax in the order of d0q to Tc1q, Tb1q, Tb3q, Ta1q, and Ta5q. Here, the lower the layer L, the more likely the encoding coefficients after quantization are to become 0. The following factors can be cited as reasons for this.

[0382] The coding coefficients of the lower layer L exhibit higher frequency components than those of the upper layer, and therefore tend to be 0 depending on the target 3D point. In addition, by switching the quantization scale according to the importance level and the like as described above, the quantization scale becomes larger in the lower layer, and the coding coefficients after quantization tend to be 0.

[0383] In this way, the lower the layer, the more likely the coding coefficient after quantization is to become 0, and the more likely it is that consecutive 0s will occur in the first code string. Fig. 51 is a diagram showing examples of the first code string and the second code string.

[0384] The three-dimensional data encoding device counts the number of times the value 0 occurs in the first code string, and encodes the number of times the value 0 occurs in succession instead of consecutive values ​​of 0. In other words, the three-dimensional data encoding device generates the second code string by replacing consecutive coding coefficients of the value 0 in the first code string with the number of consecutive 0s (ZeroCnt). As a result, when consecutive values ​​of 0 occur in coding coefficients after quantization, encoding efficiency can be improved by encoding the number of consecutive 0s rather than encoding a large number of 0s.

[0385] The three-dimensional data encoding device may also entropy-encode the value of ZeroCnt. For example, the three-dimensional data encoding device binarizes the value of ZeroCnt using a truncated unary code with a total number T of encoded three-dimensional points, and arithmetically encodes each bit after binarization. FIG. 52 is a diagram showing an example of a truncated unary code when the total number of encoded three-dimensional points is T. In this case, the three-dimensional data encoding device may improve encoding efficiency by using a different encoding table for each bit. For example, the three-dimensional data encoding device uses encoding table 1 for the first bit, encoding table 2 for the second bit, and encoding table 3 for the bits thereafter. In this way, the three-dimensional data encoding device can improve encoding efficiency by switching encoding tables for each bit.

[0386] Furthermore, the three-dimensional data encoding device may binarize ZeroCnt using Exponential-Golomb and then perform arithmetic coding. This allows for improved efficiency compared to binarized arithmetic coding using truncated unary codes when the value of ZeroCnt is likely to be large. The three-dimensional data encoding device may also add a flag to the header for switching between using truncated unary codes and using Exponential-Golomb. This allows the three-dimensional data encoding device to improve encoding efficiency by selecting the optimal binarization method. Furthermore, the three-dimensional data decoding device can switch binarization methods by referring to the flag included in the header, thereby correctly decoding the bitstream.

[0387] The three-dimensional data decoding device may convert the decoded quantized coding coefficients from unsigned integer values ​​to signed integer values ​​using a method reverse to that used by the three-dimensional data encoding device. This allows the three-dimensional data decoding device to properly decode the generated bitstream without considering the occurrence of negative integers when the coding coefficients are entropy coded. Note that the three-dimensional data decoding device does not necessarily have to convert the coding coefficients from unsigned integer values ​​to signed integer values. For example, when decoding a bitstream containing coded bits that have been separately entropy coded, the three-dimensional data decoding device may decode the coded bits.

[0388] The three-dimensional data decoding device decodes the quantized coding coefficients converted into signed integer values ​​by inverse quantization and inverse Haar transform. The three-dimensional data decoding device also uses the decoded coding coefficients for prediction of the three-dimensional point to be decoded and beyond. Specifically, the three-dimensional data decoding device calculates inverse quantized values ​​by multiplying the quantized coding coefficients by the decoded quantization scale. The three-dimensional data decoding device then obtains decoded values ​​by applying the inverse Haar transform (described later) to the inverse quantized values.

[0389] For example, the three-dimensional data decoding device converts the decoded unsigned integer value into a signed integer value in the following manner: If the least significant bit (LSB) of the decoded unsigned integer value a2u is 1, the signed integer value Ta1q is set to −((a2u+1)>>1). If the LSB of the decoded unsigned integer value a2u is not 1 (is 0), the signed integer value Ta1q is set to (a2u>>1).

[0390] The inverse quantization value of Ta1 is expressed as Ta1q×QS_L, where Ta1q is the quantization value of Ta1 and QS_L is the quantization step of layer L.

[0391] Furthermore, the QS may be the same value for all layers or for some layers. Furthermore, the three-dimensional data encoding device may add information indicating the QS to a header or the like. This allows the three-dimensional data decoding device to perform correct inverse quantization using the same QS as that used by the three-dimensional data encoding device.

[0392] Next, the inverse Haar transform will be described. Fig. 53 is a diagram for explaining the inverse Haar transform. The three-dimensional data decoding device decodes the attribute values ​​of the three-dimensional points by applying the inverse Haar transform to the coding coefficients after inverse quantization.

[0393] First, the three-dimensional data decoding device generates a Morton code based on the position information of the three-dimensional points, and sorts the three-dimensional points in Morton code order. For example, the three-dimensional data decoding device may sort the three-dimensional points in ascending order of the Morton codes. Note that the sorting order is not limited to the Morton code order, and other orders may be used.

[0394] Next, the three-dimensional data decoding device applies an inverse Haar transform to the coding coefficients including low-frequency components of layer L+1 and the coding coefficients including high-frequency components of layer L, thereby restoring attribute information of adjacent three-dimensional points in Morton code order in layer L. For example, the three-dimensional data decoding device may use an inverse Haar transform of a 2×2 matrix. The restored attribute information of layer L is used as input values ​​for the lower layer L−1.

[0395] The three-dimensional data decoding device repeats this hierarchical processing, and ends the processing when all the attribute information in the lowest layer has been decoded. Note that, if only one three-dimensional point exists as two adjacent three-dimensional points in layer L-1 when applying the inverse Haar transform, the three-dimensional data decoding device may substitute the value of the coded component of layer L for the attribute value of the one existing three-dimensional point. In this way, the three-dimensional data decoding device can apply the Haar transform to all values ​​of the input attribute information, and correctly decode a bitstream with improved coding efficiency.

[0396] When the attribute information is N-dimensional, the three-dimensional data decoding device may apply an inverse Haar transform to each dimension independently to decode each encoding coefficient. For example, when the attribute information is color information (RGB, YUV, etc.), the three-dimensional data decoding device applies an inverse Haar transform to the encoding coefficient for each component to decode each attribute value.

[0397] The three-dimensional data decoding device may apply the inverse Haar transform in the order of layers Lmax, L+1, ..., and layer L. Furthermore, w0 and w1 shown in FIG. 53 are weights assigned to each three-dimensional point. For example, the three-dimensional data decoding device may calculate weights based on distance information between two adjacent three-dimensional points to which the inverse Haar transform is applied. For example, the three-dimensional data encoding device may decode a bitstream with improved encoding efficiency by increasing the weight as the distance becomes shorter.

[0398] In the example shown in FIG. 53, the coding coefficients after inverse quantization are Ta1, Ta5, Tb1, Tb3, Tc1, and d0, and the decoded values ​​obtained are a0, a1, a2, a3, a4, and a5.

[0399] 54 is a diagram illustrating an example of the syntax of attribute information (attribute_data). The attribute information (attribute_data) includes the number of consecutive zeros (ZeroCnt), the number of attribute dimensions (attribute_dimension), and the coding coefficient (value[j][i]).

[0400] The number of consecutive zeros (ZeroCnt) indicates the number of consecutive 0s in the coding coefficient after quantization. Note that the three-dimensional data coding device may binarize ZeroCnt and then perform arithmetic coding.

[0401] 54, the three-dimensional data encoding device may determine whether the layer L (layer L) to which the encoding coefficient belongs is equal to or greater than a predetermined threshold TH_layer, and may switch the information to be added to the bitstream based on the determination result. For example, if the determination result is true, the three-dimensional data encoding device may add all of the encoding coefficients of the attribute information to the bitstream. Alternatively, if the determination result is false, the three-dimensional data encoding device may add only some of the encoding coefficients to the bitstream.

[0402] Specifically, if the determination result is true, the three-dimensional data encoding device adds the encoding result of the RGB or YUV three-dimensional color information to the bitstream. If the determination result is false, the three-dimensional data encoding device adds some of the color information, such as G or Y, to the bitstream, and does not need to add other components to the bitstream. In this way, the three-dimensional data encoding device can improve encoding efficiency by not adding some of the encoding coefficients of layers (layers lower than TH_layer) that include encoding coefficients that indicate high-frequency components whose degradation is less noticeable visually to the bitstream.

[0403] The number of attribute dimensions (attribute_dimension) indicates the number of dimensions of the attribute information. For example, if the attribute information is color information (RGB, YUV, etc.) of a three-dimensional point, the number of attribute dimensions is set to a value of 3 because color information is three-dimensional. If the attribute information is reflectance, the number of attribute dimensions is set to a value of 1 because reflectance is one-dimensional. Note that the number of attribute dimensions may be added to the header of the attribute information of the bitstream, etc.

[0404] The coding coefficient (value[j][i]) indicates the coding coefficient after quantization of the j-th dimension attribute information of the i-th 3D point. For example, if the attribute information is color information, value

[99] [1] indicates the coding coefficient of the second dimension (e.g., G value) of the 100th 3D point. Also, if the attribute information is reflectance information, value

[119] [0] indicates the coding coefficient of the first dimension (e.g., reflectance) of the 120th 3D point.

[0405] If the following condition is satisfied, the three-dimensional data encoding device may subtract the value 1 from value[j][i] and entropy encode the resulting value. In this case, the three-dimensional data decoding device restores the encoding coefficient by adding the value 1 to value[j][i] after entropy decoding.

[0406] The above conditions are (1) when attribute_dimension = 1, or (2) when attribute_dimension is 1 or greater and all dimension values ​​are equal. For example, if the attribute information is reflectance, attribute_dimension = 1, so the three-dimensional data encoder subtracts the value 1 from the encoding coefficient to calculate the value and encodes the calculated value. The three-dimensional data decoder adds the value 1 to the decoded value to calculate the encoding coefficient.

[0407] More specifically, for example, if the encoding coefficient for reflectance is 10, the three-dimensional data encoding device encodes the value 9 obtained by subtracting the value 1 from the encoding coefficient value 10. The three-dimensional data decoding device adds the value 1 to the decoded value 9 to calculate the encoding coefficient value 10.

[0408] Furthermore, since attribute_dimension = 3 when the attribute information is color, for example, if the quantized coding coefficients of each R, G, and B component are the same, the three-dimensional data coding device subtracts a value of 1 from each coding coefficient and encodes the resulting value. The three-dimensional data decoding device adds a value of 1 to the decoded value. More specifically, for example, if the R, G, and B coding coefficients = (1, 1, 1), the three-dimensional data coding device encodes (0, 0, 0). The three-dimensional data decoding device adds 1 to each component of (0, 0, 0) to calculate (1, 1, 1). Furthermore, if the R, G, and B coding coefficients = (2, 1, 2), the three-dimensional data coding device encodes (2, 1, 2) as is. The three-dimensional data decoding device uses the decoded (2, 1, 2) as is as the coding coefficient.

[0409] In this way, by providing ZeroCnt, a pattern in which all dimensions of value are 0 is not generated, and so it is possible to encode a value obtained by subtracting 1 from the value of value, thereby improving encoding efficiency.

[0410] Furthermore, value[0][i] shown in Fig. 54 indicates the coding coefficient after quantization of the first-dimensional attribute information of the i-th three-dimensional point. As shown in Fig. 54, if the layer L (layer L) to which the coding coefficient belongs is smaller than the threshold TH_layer, the amount of code may be reduced by adding the first-dimensional attribute information to the bit stream (without adding attribute information from the second dimension onwards to the bit stream).

[0411] The three-dimensional data encoding device may switch the calculation method for the ZeroCnt value depending on the value of attribute_dimension. For example, when attribute_dimension = 3, the three-dimensional data encoding device may count the number of times the values ​​of the encoding coefficients of all components (dimensions) are 0. Figure 55 is a diagram showing an example of the encoding coefficients and ZeroCnt in this case. For example, in the case of the color information shown in Figure 55, the three-dimensional data encoding device counts the number of consecutive encoding coefficients in which the R, G, and B components are all 0, and adds the counted number to the bitstream as ZeroCnt. This eliminates the need to encode ZeroCnt for each component, thereby reducing overhead. This improves encoding efficiency. Note that the three-dimensional data encoding device may calculate ZeroCnt for each dimension even when attribute_dimension is 2 or more, and add the calculated ZeroCnt to the bitstream.

[0412] 56 is a flowchart of three-dimensional data encoding processing according to this embodiment. First, the three-dimensional data encoding device encodes position information (geometry) (S6601). For example, the three-dimensional data encoding device performs encoding using an octree representation.

[0413] Next, the three-dimensional data encoding device converts the attribute information (S6602). For example, if the position of a three-dimensional point changes due to quantization or the like after encoding the position information, the three-dimensional data encoding device reassigns the attribute information of the original three-dimensional point to the changed three-dimensional point. The three-dimensional data encoding device may also perform the reassignment by interpolating the value of the attribute information according to the amount of change in position. For example, the three-dimensional data encoding device may detect N pre-change three-dimensional points that are close to the changed three-dimensional position, weight the values ​​of the attribute information of the N three-dimensional points based on the distance from the changed three-dimensional position to each of the N three-dimensional points, and set the obtained value as the value of the attribute information of the changed three-dimensional point. Furthermore, if two or more three-dimensional points change to the same three-dimensional position due to quantization or the like, the three-dimensional data encoding device may assign the average value of the attribute information of the two or more pre-change three-dimensional points as the value of the attribute information of the changed three-dimensional point.

[0414] Next, the three-dimensional data encoding device encodes the attribute information (S6603). For example, when encoding multiple pieces of attribute information, the three-dimensional data encoding device may encode the multiple pieces of attribute information in order. For example, when encoding color and reflectance as attribute information, the three-dimensional data encoding device generates a bitstream in which the encoding result of reflectance is added after the encoding result of color. Note that the encoding results of the multiple pieces of attribute information added to the bitstream may be in any order.

[0415] The three-dimensional data encoding device may also add information indicating the start location of the encoded data for each piece of attribute information in the bitstream to a header or the like. This allows the three-dimensional data decoding device to selectively decode attribute information that needs to be decoded, thereby omitting the decoding process for attribute information that does not need to be decoded. This reduces the amount of processing required by the three-dimensional data decoding device. The three-dimensional data encoding device may also encode multiple pieces of attribute information in parallel and combine the encoding results into a single bitstream. This allows the three-dimensional data encoding device to encode multiple pieces of attribute information at high speed.

[0416] 57 is a flowchart of the attribute information encoding process (S6603). First, the three-dimensional data encoding device generates encoding coefficients from the attribute information using Haar transform (S6611). Next, the three-dimensional data encoding device applies quantization to the encoding coefficients (S6612). Next, the three-dimensional data encoding device generates encoded attribute information (bit stream) by encoding the quantized encoding coefficients (S6613).

[0417] Furthermore, the three-dimensional data encoding device applies inverse quantization to the quantized coding coefficients (S6614). Next, the three-dimensional data decoding device decodes attribute information by applying an inverse Haar transform to the inverse quantized coding coefficients (S6615). For example, the decoded attribute information is referenced in subsequent encoding.

[0418] FIG. 58 is a flowchart of the coding coefficient coding process (S6613). First, the three-dimensional data coding device converts the coding coefficient from a signed integer value to an unsigned integer value (S6621). For example, the three-dimensional data coding device converts the signed integer value to an unsigned integer value as follows: If the signed integer value Ta1q is less than 0, the unsigned integer value is set to -1 - (2 x Ta1q). If the signed integer value Ta1q is 0 or greater, the unsigned integer value is set to 2 x Ta1q. Note that if the coding coefficient does not become a negative value, the three-dimensional data coding device may code the coding coefficient as an unsigned integer value as is.

[0419] If not all coding coefficients have been processed (No in S6622), the three-dimensional data coding device determines whether the value of the coding coefficient to be processed is zero (S6623). If the value of the coding coefficient to be processed is zero (Yes in S6623), the three-dimensional data coding device increments ZeroCnt by 1 (S6624) and returns to step S6622.

[0420] If the value of the coding coefficient to be processed is not zero (No in S6623), the three-dimensional data coding device encodes ZeroCnt and resets ZeroCnt to 0 (S6625). The three-dimensional data coding device also arithmetically codes the coding coefficient to be processed (S6626) and returns to step S6622. For example, the three-dimensional data coding device performs binary arithmetic coding. The three-dimensional data coding device may also subtract a value of 1 from the coding coefficient and code the resulting value.

[0421] Furthermore, the processes of steps S6623 to S6626 are repeated for each coding coefficient. Furthermore, if all coding coefficients have been processed (Yes in S6622), the three-dimensional data coding device ends the process.

[0422] 59 is a flowchart of three-dimensional data decoding processing according to this embodiment. First, the three-dimensional data decoding device decodes position information (geometry) from the bitstream (S6631). For example, the three-dimensional data decoding device performs decoding using an octree representation.

[0423] Next, the three-dimensional data decoding device decodes the attribute information from the bitstream (S6632). For example, when decoding multiple pieces of attribute information, the three-dimensional data decoding device may decode the multiple pieces of attribute information in order. For example, when decoding color and reflectance as attribute information, the three-dimensional data decoding device decodes the encoding result of color and the encoding result of reflectance in the order in which they are added to the bitstream. For example, when the encoding result of reflectance is added after the encoding result of color in the bitstream, the three-dimensional data decoding device decodes the encoding result of color, and then the encoding result of reflectance. Note that the three-dimensional data decoding device may decode the encoding results of the attribute information added to the bitstream in any order.

[0424] The three-dimensional data decoding device may also obtain information indicating the start location of the encoded data for each piece of attribute information in the bitstream by decoding a header or the like. This allows the three-dimensional data decoding device to selectively decode attribute information that requires decoding, thereby omitting the decoding process for attribute information that does not require decoding. This reduces the amount of processing required by the three-dimensional data decoding device. The three-dimensional data decoding device may also decode multiple pieces of attribute information in parallel and integrate the decoding results into a single three-dimensional point cloud. This allows the three-dimensional data decoding device to decode multiple pieces of attribute information at high speed.

[0425] 60 is a flowchart of the attribute information decoding process (S6632). First, the three-dimensional data decoding device decodes the coding coefficients from the bitstream (S6641). Next, the three-dimensional data decoding device applies inverse quantization to the coding coefficients (S6642). Next, the three-dimensional data decoding device decodes the attribute information by applying an inverse Haar transform to the coding coefficients after the inverse quantization (S6643).

[0426] 61 is a flowchart of the coding coefficient decoding process (S6641). First, the three-dimensional data decoding device decodes ZeroCnt from the bitstream (S6651). If not all coding coefficients have been processed (No in S6652), the three-dimensional data decoding device determines whether ZeroCnt is greater than 0 (S6653).

[0427] If ZeroCnt is greater than zero (Yes in S6653), the three-dimensional data decoding device sets the coding coefficient to be processed to 0 (S6654). Next, the three-dimensional data decoding device subtracts 1 from ZeroCnt (S6655) and returns to step S6652.

[0428] If ZeroCnt is zero (No in S6653), the three-dimensional data decoding device decodes the coding coefficient to be processed (S6656). For example, the three-dimensional data decoding device uses binary arithmetic decoding. The three-dimensional data decoding device may also add a value of 1 to the decoded coding coefficient.

[0429] Next, the three-dimensional data decoding device decodes ZeroCnt, sets the obtained value to ZeroCnt (S6657), and returns to step S6652.

[0430] Furthermore, the processes of steps S6653 to S6657 are repeated for each coding coefficient. Furthermore, if all coding coefficients have been processed (Yes in S6652), the three-dimensional data coding device converts the decoded coding coefficients from unsigned integer values ​​to signed integer values ​​(S6658). For example, the three-dimensional data decoding device may convert the decoded coding coefficients from unsigned integer values ​​to signed integer values ​​as follows: If the LSB (least significant bit) of the decoded unsigned integer value Ta1u is 1, the signed integer value Ta1q is set to -((Ta1u + 1) >> 1). If the LSB of the decoded unsigned integer value Ta1u is not 1 (if it is 0), the signed integer value Ta1q is set to (Ta1u >> 1). Note that if the coding coefficient does not become a negative value, the three-dimensional data decoding device may use the decoded coding coefficient as a signed integer value as is.

[0431] 62 is a block diagram of an attribute information encoding unit 6600 included in the three-dimensional data encoding device. The attribute information encoding unit 6600 includes a sorting unit 6601, a Haar transform unit 6602, a quantization unit 6603, an inverse quantization unit 6604, an inverse Haar transform unit 6605, a memory 6606, and an arithmetic encoding unit 6607.

[0432] The sorting unit 6601 generates a Morton code using the position information of the 3D points and sorts the 3D points in Morton code order. The Haar transform unit 6602 generates coding coefficients by applying a Haar transform to the attribute information. The quantization unit 6603 quantizes the coding coefficients of the attribute information.

[0433] The inverse quantization unit 6604 inversely quantizes the quantized coding coefficients. The inverse Haar transform unit 6605 applies an inverse Haar transform to the coding coefficients. The memory 6606 stores values ​​of attribute information of multiple decoded 3D points. For example, the attribute information of the decoded 3D points stored in the memory 6606 may be used for predicting uncoded 3D points.

[0434] The arithmetic coding unit 6607 calculates ZeroCnt from the quantized coding coefficients and arithmetically codes the ZeroCnt. The arithmetic coding unit 6607 also arithmetically codes the quantized non-zero coding coefficients. The arithmetic coding unit 6607 may binarize the coding coefficients before arithmetic coding. The arithmetic coding unit 6607 may also generate and code various header information.

[0435] 63 is a block diagram of an attribute information decoding unit 6610 included in the three-dimensional data decoding device. The attribute information decoding unit 6610 includes an arithmetic decoding unit 6611, an inverse quantization unit 6612, an inverse Haar transform unit 6613, and a memory 6614.

[0436] The arithmetic decoding unit 6611 arithmetically decodes the ZeroCnt and the coding coefficients included in the bitstream. Note that the arithmetic decoding unit 6611 may also decode various types of header information.

[0437] The inverse quantization unit 6612 inverse quantizes the arithmetically decoded coding coefficients. The inverse Haar transform unit 6613 applies inverse Haar transform to the inversely quantized coding coefficients. The memory 6614 stores values ​​of attribute information of multiple decoded 3D points. For example, the attribute information of the decoded 3D points stored in the memory 6614 may be used to predict undecoded 3D points.

[0438] In the above embodiment, an example is shown in which the three-dimensional points are coded from the lower layer to the upper layer as the coding order, but this is not necessarily limited to this. For example, a method of scanning the coding coefficients after Haar transform from the upper layer to the lower layer may be used. In this case, the three-dimensional data coding device may also code the number of consecutive zeros as ZeroCnt.

[0439] Furthermore, the three-dimensional data encoding device may switch whether or not to use the encoding method using ZeroCnt described in this embodiment on a WLD, SPC, or volume basis. In this case, the three-dimensional data encoding device may add information indicating whether or not the encoding method using ZeroCnt was applied to the header information. This allows the three-dimensional data decoding device to perform decoding appropriately. As an example of a switching method, for example, the three-dimensional data encoding device counts the number of occurrences of encoding coefficients with a value of 0 for one volume. If the count value exceeds a predetermined threshold, the three-dimensional data encoding device applies the method using ZeroCnt to the next volume; if the count value is equal to or less than the threshold, the three-dimensional data encoding device does not apply the method using ZeroCnt to the next volume. This allows the three-dimensional data encoding device to appropriately switch whether or not to apply the encoding method using ZeroCnt depending on the characteristics of the three-dimensional point to be encoded, thereby improving encoding efficiency.

[0440] (Embodiment 5) Fig. 64 is a block diagram showing the configuration of a three-dimensional data encoding device according to this embodiment. Note that in Fig. 64, an encoding unit that encodes position information, which is provided in the three-dimensional data encoding device, is not shown.

[0441] The three-dimensional data encoding device 10600 includes a transform unit 10610 and an encoding unit 10620 .

[0442] The conversion unit 10610 performs conversion processing on the input attribute information before inputting it to the encoding unit 10620. The conversion processing is, for example, at least one of offsetting (offset processing) and scaling (scaling processing) described below.

[0443] The conversion unit 10610 has a scale unit 10611 and an offset unit 10612. It is sufficient that the conversion unit 10610 has at least one of the offset unit 10612 and the scale unit 10611. For example, when the conversion unit 10610 only performs an offset on the attribute information, the conversion unit 10610 does not need to have the scale unit 10611.

[0444] The scale unit 10611 performs scaling (multiplication or division), which is an example of a conversion process, on the input attribute information and outputs a scale value (more specifically, scale information, which is information indicating the scale value, which is the value used for scaling).

[0445] The offset unit 10612 performs offset (addition or subtraction), which is another example of a conversion process, on the scaled attribute information and outputs an offset value (more specifically, offset information, which is information indicating the offset value, which is the value used for the offset).

[0446] The encoding unit 10620 encodes the attribute information converted by the conversion unit 10610 (post-conversion attribute information), and also encodes conversion information such as an offset value or a scale value as additional information (metadata).

[0447] The encoding unit 10620 includes an attribute information encoding unit 10621 and an additional information encoding unit 10622 .

[0448] The attribute information encoding unit 10621 encodes the converted attribute information, which is the attribute information converted by the conversion unit 10610 .

[0449] The additional information encoding unit 10622 encodes the additional information including the conversion information such as the scale value and offset value output by the conversion unit 10610 .

[0450] For example, if the encoding unit 10620 does not support encoding of negative values, or if it is specified that the encoding unit 10620 does not support encoding of negative values, when the format of the input attribute information has a negative value, the conversion unit 10610 adds an offset value to the attribute information and converts the attribute information to a positive value.

[0451] For example, if the encoding unit 10620 does not support decimals and floating points but supports integers, or if the encoding unit 10620 is specified to not support decimals and floating points but supports integers, when the format of the input attribute information is not an integer, the scaling unit 10611 multiplies the input attribute information (more specifically, the numerical value indicated by the input attribute information) by a scale value to convert the attribute information into a positive numerical value.

[0452] For example, if the encoding unit 10620 supports encoding of attribute information of 8-bit unsigned integer type (positive integer) and the input attribute information is a 32-bit signed floating point in the range of [-1, 1], the attribute information is first converted into a scaled_value, which is an 8-bit signed integer type value in the range of [-127, 128], by processing such as scaling, rounding, truncation, and rounding up.

[0453] Note that scaled_attribute=round(input_attribute×scale).

[0454] Here, scale is an example of a scale value, which is a value by which the value indicated by the attribute information is multiplied, for example, 2^(8 bits-1), that is, 2 to the power of 7=128.

[0455] Next, the scaled attribute information is converted into an 8-bit unsigned integer type in the range of [0, 255] by an offset.

[0456] Note that offset_attribute=scaled_attribute+offset.

[0457] Here, offset is an example of an offset value, which is a value to be added to the value indicated by the attribute information, and is, for example, 2^(8bit-1)-1=127.

[0458] The offset value and / or scale value, which are the conversion information used in the conversion, are input to the encoding unit 10620 and encoded as additional information.

[0459] The additional information encoding unit 10622 may encode the transformation information as it is as the additional information, or may encode information from which the transformation information can be derived as the additional information.

[0460] Furthermore, the information from which an offset value can be derived and the information from which a scale value can be derived may be shown independently, or may be shown as common information.

[0461] For example, in the above example, the offset value and scale value are determined in advance as offset = 2^(N-1) and scale = 2^(N-1) - 1. In this case, for example, the encoding unit 10620 stores the value of N (an integer equal to or greater than 1) in the additional information and encodes it, that is, encodes the additional information indicating the value of N as conversion information.

[0462] Furthermore, if the encoding unit supports 8-bit unsigned integers, N may be defined as the number of bits of the unsigned integer, and N=8.

[0463] Alternatively, N may be predetermined to indicate the number of bits of an unsigned integer type. In such a case, when information indicating that N indicates the number of bits of an unsigned integer type is stored in the coded stream (bit stream), the information indicating that N indicates the number of bits of an unsigned integer type does not need to be included in the additional information.

[0464] Furthermore, the conversion unit 10610 may determine the offset and scale (offset value and scale value), respectively, based on the values ​​and characteristics indicated by the attribute information constituting the three-dimensional point cloud.

[0465] After scaling the attribute information, the scale unit 10611 may round the value of the attribute information after scaling by rounding up, rounding down, rounding down, or the like.

[0466] Furthermore, if there is no need to convert the attribute information so that the value indicated by the attribute information is a positive integer, the conversion unit 10610 may not convert the attribute information. In this case, the conversion unit 10610 may not output the scale value and offset value, or may output information indicating that no conversion was performed as conversion information. In this case, for example, the encoding unit 10620 encodes the attribute information that has not been converted by the conversion unit 10610.

[0467] Fig. 65 is a block diagram showing the configuration of a three-dimensional data decoding device according to this embodiment. Note that Fig. 65 omits the illustration of a decoding unit that is provided in the three-dimensional data decoding device and that decodes encoded position information.

[0468] The three-dimensional data decoding device 10630 includes a decoding unit 10640 and an inverse transform unit 10650 .

[0469] The decoding unit 10640 receives the coded attribute information (coded attribute information) and the coded additional information (coded additional information) and decodes the coded attribute information and the coded additional information. The decoding unit 10640 includes an attribute information decoding unit 10641 and an additional information decoding unit 10642.

[0470] The attribute information decoding unit 10641 generates decoded attribute information by decoding the encoded attribute information.

[0471] The additional information decoding unit 10642 extracts conversion information indicating an offset value, a scale value, and the like by decoding the coded additional information.

[0472] The inverse transform unit 10650 performs inverse transform processing on the decoded attribute information based on the transform information. The inverse transform processing is at least one of inverse offset (inverse offset processing) and inverse scaling (inverse scaling processing), which will be described later. The inverse transform unit 10650 includes an inverse offset unit 10651 and an inverse scale unit 10652.

[0473] The reverse offset unit 10651 reverse-offsets the decoded attribute information, which is an example of reverse conversion processing, using an offset value extracted from the conversion information. That is, the reverse offset unit 10651 performs a conversion on the decoded attribute information that is the reverse of the conversion that the conversion unit 10610 (more specifically, the offset unit 10612) performed on the attribute information. For example, when the conversion unit 10610 added an offset value to a value indicated by the attribute information, the reverse offset unit 10651 subtracts the offset value from the value indicated by the decoded attribute information.

[0474] The inverse scale unit 10652 inversely scales the inversely offset decoded attribute information, which is another example of an inverse conversion process, using the scale value extracted from the additional information. That is, the inverse scale unit 10652 performs a conversion on the decoded attribute information that is the inverse of the conversion that the conversion unit 10610 (more specifically, the scale unit 10611) performed on the attribute information. For example, if the conversion unit 10610 multiplied the value indicated by the attribute information by the scale value, the inverse scale unit 10652 divides the value indicated by the decoded attribute information by the scale value.

[0475] For example, if the offset value extracted from the additional information is offset and the scale value is scale, the inversely offset attribute information is derived as offset_attribute=decoded_value-offset, and the inversely scaled attribute information is derived as scaled_attribute=offset_attribute / scale.

[0476] In scaling and inverse scaling, the amount of processing may be reduced by using a shift operation (bit shift) instead of multiplication and / or division by expressing the scale value as a power of 2. In other words, scaling and inverse scaling are processes that perform at least one of multiplication, division, and shift operations on values ​​indicated by attribute information.

[0477] With the above configuration, the inverse conversion unit 10650 provided in the three-dimensional data decoding device 10630 performs an inverse conversion process based on the conversion information contained in the encoded data, thereby making it possible to reproduce the attribute information before it was converted by the conversion unit 10610 provided in the three-dimensional data encoding device 10600.

[0478] The three-dimensional data decoding device 10630 does not necessarily have to perform the inverse transformation process, and may select whether or not to perform the inverse transformation process based on the application or use case.

[0479] In addition, although the conversion unit 10610 has been described as being configured so that the offset unit 10612 is located after (at a later stage of) the scale unit 10611, and the inverse conversion unit 10650 has been described as being configured so that the inverse scale unit 10652 is located after the inverse offset unit 10651, the configuration is not limited to this. For example, the conversion unit 10610 may be configured so that the scale unit 10611 is located after the offset unit 10612, and the inverse conversion unit 10650 may be configured so that the inverse offset unit 10651 is located after the inverse scale unit 10652.

[0480] Alternatively, the three-dimensional data encoding device 10600 may select which configuration to use, i.e., the order in which to scale and offset the attribute information, based on the type of attribute information (attribute_type), and store information indicating which configuration was used, i.e., information indicating the order in which the scaling and offsetting were performed (order information), in the additional information as, for example, a flag, and transmit this information to the three-dimensional data decoding device 10630. The three-dimensional data decoding device 10630 may select the order in which to perform the inverse scaling and inverse offsetting based on the order information, and perform the inverse transform process on the decoded attribute information in the selected order.

[0481] FIG. 66 is a diagram showing a first example of the syntax of an SPS (Sequence parameter set) according to this embodiment.

[0482] In this embodiment, the SPS indicates an SPS identifier (sps_idx) and additional information related to the entire sequence (common_information()). For example, for each attribute included in the bitstream, the SPS indicates additional information related to the attribute information, such as an attribute information identifier (attribute_type), the number of dimensions of the attribute information (num_dimension), an identifier for identifying instances of the same attribute type (instance_id), and other additional information related to the attribute information (attribute_info).

[0483] The three-dimensional point cloud data may not have attribute information corresponding to the position information, or may have one or more attribute information corresponding to the position information. For example, when the three-dimensional point cloud data has multiple pieces of attribute information corresponding to one piece of position information, the three-dimensional data encoding device 10600 generates conversion information corresponding to each piece of attribute information and stores the generated conversion information in the additional information (in other words, generates additional information including the conversion information).

[0484] The transform_flag is information (flag) indicating whether the additional information for each attribute information includes transformation information. For example, if the three-dimensional data encoding device 10600 includes transformation information in the additional information, it sets transform_flag = 1 and sets the transformation information (offset value and / or scale value). Also, for example, if the three-dimensional data encoding device 10600 does not include transformation information, it sets transform_flag = 0.

[0485] The number of dimensions of the attribute information is 1 or more, and when the number of dimensions is 2 or more, common conversion information may be used for all dimensions. Of course, even when the number of dimensions of the attribute information is 2 or more, individual conversion information may be used for all dimensions.

[0486] For example, if the attribute information is composed of three-dimensional color information and one-dimensional reflectance, common conversion information may be applied to R (Red), G (Green), and B (Blue) in the color information, and conversion information different from that of the color information may be applied to the reflectance.

[0487] Furthermore, the conversion information may be stored in the additional information common to a specific plurality of pieces of attribute information, assuming that the conversion information is common to the specific plurality of pieces of attribute information. Alternatively, for example, the conversion information may be shown for each type of attribute information, assuming that the conversion information common to a plurality of instances of the same attribute type (attribute_type) is used.

[0488] Alternatively, it may be predetermined based on the type of attribute that conversion information is not used and is not indicated in additional information for a specific attribute type.

[0489] FIG. 67 is a diagram showing an example of the structure of a bitstream according to this embodiment.

[0490] The conversion information is stored in, for example, the SPS.

[0491] Note that the transformation information does not have to be stored in the SPS, and may be stored, for example, in a parameter set (APS / Attribute Parameter Set) related to encoding of attribute information, or in other additional information such as a slice header.

[0492] Alternatively, the transformation information may be stored in additional information such as supplemental enhancement information (SEI).

[0493] Instance_id indicates an identifier of the same attribute type. For example, in the case of Color1, attribute_type=color and instance_id=0. Also, in the case of Color2, attribute_type=color and instance_id=1.

[0494] For example, if the attribute information is color information, no conversion information is generated.

[0495] Furthermore, for example, if the attribute information is information indicating a normal vector, conversion information is generated and included in the bitstream together with the SPS or APS.

[0496] For example, if Attribute0 included in the bitstream is color information (Color), which is an example of attribute information, flag information (transform_flag) indicating whether transformation information is included in the bitstream is set to transform_flag = 0, indicating that no transformation processing has been performed on Attribute0.

[0497] On the other hand, for example, if Attribute 1 included in the bitstream is normal vector information (Normal_Vector), which is another example of attribute information, setting transform_flag = 1 indicates that a transformation process has been performed on Attribute 1. In this case, for example, the bitstream includes information indicating the value used in the offset (offset value) and information indicating the value used in the scaling (scale value), such as offset = A and scale = B.

[0498] Note that transform_flag may be replaced with transform_information_type, and the combination indicating the offset value and scale value may be switched based on transform_information_type, or the method and / or format for indicating the offset and / or scale value may be switched. As described above, the bitstream includes information (transform identification information) indicating whether or not a transform process has been performed, such as transform flag information such as transform_flag and transform type information such as transform_information_type.

[0499] Fig. 68 is a diagram showing a second example of the syntax of the SPS according to this embodiment. Fig. 69 is a diagram showing a first example of the syntax of the conversion information according to this embodiment.

[0500] For example, as shown in Fig. 69, the method of indicating the scale value and offset value may be selectable from among showing the value as it is, showing it as a power of 2 (value calculated as y = log2(x)), showing it by a predetermined method, etc. For example, a flag indicating which syntax to use may be displayed, and the syntax may be switched by the flag.

[0501] For example, when transform_type==2, the three-dimensional data encoding device obtains offset_log2 and calculates offset=2^(offset_log2-X)-Y and scale=2^(scale_log2-Z) to calculate the offset value and scale value.

[0502] Note that X, Y, and Z are any integers. X, Y, and Z may be predetermined values, or information indicating these values ​​may be included in the bit stream and transmitted from the three-dimensional data encoding device 10600 to the three-dimensional data decoding device 10630.

[0503] In the above calculation, instead of offset_lo2 and scale_log2, values ​​obtained by subtracting 1 (for example, offset_log2_minus1 and scale_log2_minus_1) may be stored in the additional information.

[0504] This allows the number of bits to be reduced.

[0505] FIG. 70 is a diagram showing a second example of the syntax of the conversion information according to the present embodiment.

[0506] In the example shown in FIG. 70, the syntax is switched based on the type of attribute information (attribute_type), such as indicating either an offset value or a scale value, indicating both, or indicating neither.

[0507] Note that, when both offset and scale are executed, order information (transform_order) indicating the order of configuration of the offset unit 10612 and the scale unit 10611 of the transform unit 10610 provided in the three-dimensional data encoding device 10600 may be included in the bitstream as additional information, for example. This makes it possible for the three-dimensional data decoding device 10630 to extract predetermined syntax based on the additional information such as transform_type, attribute_type, and transform_order, and apply it to the inverse transform unit 10650.

[0508] FIG. 71 is a flowchart showing the processing procedure of the three-dimensional data encoding device according to this embodiment.

[0509] First, the three-dimensional data encoding device 10600 determines whether or not to convert the input attribute information (S10601).

[0510] When the three-dimensional data encoding device 10600 determines that the input attribute information is to be converted (Yes in S10601), the three-dimensional data encoding device 10600 performs a conversion process on the input attribute information (S10602). For example, the three-dimensional data encoding device 10600 performs offset and scale on the input attribute information.

[0511] Next, the three-dimensional data encoding device 10600 stores the transformation information in the additional information and sets transform_flag = 1 (S10603). For example, information indicating the offset value used for the offset and the scale value used for the scale is included in the additional information as the transformation information.

[0512] Next, the three-dimensional data encoding device 10600 encodes the additional information including the conversion information and the attribute information on which the conversion process has been performed (S10604). After step S10604, for example, the three-dimensional data encoding device 10600 generates a bit stream including the encoded information as encoded data and transmits it to the three-dimensional data decoding device 10630.

[0513] On the other hand, if the three-dimensional data encoding device 10600 determines that the input attribute information should not be converted (No in S10601), it does not perform conversion processing on the input attribute information, and therefore does not store the conversion information in the additional information and sets transform_flag = 0 (S10605).

[0514] Next, the three-dimensional data encoding device 10600 encodes the additional information that does not include conversion information and the attribute information for which conversion processing has not been performed, i.e., the input attribute information (S10606). After step S10606, for example, the three-dimensional data encoding device 10600 generates a bit stream that includes this encoded information as encoded data and transmits it to the three-dimensional data decoding device 10630.

[0515] FIG. 72 is a flowchart showing the processing procedure of the three-dimensional data decoding device according to this embodiment.

[0516] First, the three-dimensional data decoding device 10630 receives, for example, a bit stream transmitted by the three-dimensional data encoding device 10600, decodes the encoded data contained in the received bit stream, and analyzes the additional information contained in the decoded encoded data (S10611).

[0517] Next, the three-dimensional data decoding device 10630 decodes the attribute information of the coded data included in the bitstream (S10612).

[0518] Next, the three-dimensional data decoding device 10630 determines whether or not transform_flag included in the bitstream is set to 1 (S10613). In other words, the three-dimensional data decoding device 10630 determines whether or not transform_flag = 1, thereby determining whether or not transformation processing has been performed on the attribute information included in the bitstream.

[0519] If the three-dimensional data decoding device 10630 determines that the transform_flag included in the bitstream is set to 1 (Yes in S10613), it extracts transformation information from the additional information and performs an inverse transformation process on the attribute information based on the extracted transformation information (S10614).

[0520] On the other hand, if the three-dimensional data decoding device 10630 determines that the transform_flag included in the bitstream is not set to 1 (No in S10613), that is, if transform_flag = 0, it terminates the processing without performing the inverse transform processing.

[0521] In the above example, the data format of the attribute information input to the encoding unit 10620 is converted and the conversion information is included in the bitstream, but the present invention is not limited to this.

[0522] Fig. 73 is a block diagram for explaining another example of the processing of the three-dimensional data encoding device according to this embodiment. Fig. 74 is a block diagram for explaining another example of the processing of the three-dimensional data decoding device according to this embodiment.

[0523] For example, the three-dimensional data encoding device may store in additional information format information indicating the data format of the attribute information input to the conversion unit 10660 and the format of the attribute information to be encoded after conversion, and encode the additional information including the format information in the encoding unit 10670.

[0524] The three-dimensional data decoding device 10630 can reproduce the attribute information before the conversion process in the conversion unit 10660 provided in the three-dimensional data encoding device by performing an inverse conversion on the decoded attribute information in the inverse conversion unit 10690 based on the format information extracted by the decoding unit 10680.

[0525] Format information (data format information) is information indicating, for example, the data type, the number of bits, whether signed or unsigned, etc. For example, format information is information such as int8, uint16, float16, etc. The number such as 8 in int8 indicates the number of bits.

[0526] Alternatively, for example, the format information may indicate the file format of the point cloud data before the conversion process is performed (e.g., a ply file, a pcd file, a las file, a txt file, a csv file, etc.), or may indicate the file format of the point cloud data after the conversion process is performed.

[0527] Furthermore, although the offset and scale of the attribute information have been described above, the method described in this embodiment can also be applied to position information.

[0528] Before encoding the position information, conversion processing such as offset and scale may be performed, and after decoding the position information, inverse conversion processing may be performed. In this case, the conversion information or format information may be stored in additional information such as an SPS.

[0529] Furthermore, the three-dimensional data encoding device may have a conversion unit that performs conversion on at least one or both of the position information and the attribute information. Similarly, the three-dimensional data decoding device may have an inverse conversion unit that performs conversion on either one or both (i.e., at least one) of the position information and the attribute information. In such cases, the additional information may include conversion information for the position information and conversion information for the attribute information.

[0530] Although offsetting, scaling, and quantization have been described above as methods for converting input point cloud data, other conversion methods may be used instead. For example, the conversion may use a predetermined linear or nonlinear conversion means, such as a conversion or approximation using a predetermined function.

[0531] The additional information may include not only information indicating the format of the attribute information, but also information indicating the order of the point cloud data, information indicating the sort order, time stamp information, and the like.

[0532] As described above, the three-dimensional data encoding device according to this embodiment performs the processing shown in FIG.

[0533] FIG. 75 is a flowchart showing the processing procedure of the three-dimensional data encoding device according to this embodiment.

[0534] First, the three-dimensional data encoding device acquires attribute information of the three-dimensional point (S10621).

[0535] Next, the three-dimensional data encoding device performs a conversion process on the numerical value indicated by the acquired attribute information, such as scaling (which involves at least one of multiplication / division and shift operations) and offset (which involves addition / subtraction), and then encodes the attribute information, or performs an encoding process that encodes the attribute information without performing the conversion process (S10622). For example, the three-dimensional data encoding device determines whether the format of the attribute information is a predetermined format, and if it is, converts the attribute information into a positive value using a predetermined offset value and a predetermined scale value.

[0536] Next, the three-dimensional data encoding device generates a bitstream including the encoded attribute information and transformation identification information indicating whether or not a transformation process has been performed (S10623). The transformation identification information is, for example, information such as the above-mentioned transform_flag and transform_information_type. Note that the three-dimensional data encoding device may encode the transformation identification information and generate a bitstream including the encoded attribute information and the encoded transformation identification information.

[0537] Depending on the encoding unit (e.g., encoding unit 10620 or 10670) that encodes the attribute information, it may be unable to process decimal points and / or negative numbers. Therefore, for example, by converting the attribute information using at least one of scaling and offsetting, it is possible to convert the value indicated by the attribute information into a positive number. Therefore, according to the three-dimensional data encoding method disclosed herein, even if it is unable to encode decimal points and / or negative numbers, it is possible to appropriately convert and encode the attribute information and notify the user that the attribute information has been converted.

[0538] Also, for example, in generating a bitstream (S10623), if a conversion process is performed in step S10622, a bitstream is generated that further includes conversion information including at least one of scale information indicating the value used for scaling (scale value) and offset information indicating the value used for offset (offset value).

[0539] This allows the attribute information to be converted using an appropriate value to make the attribute information a positive number, for example, without having to predetermine the scale value and offset value, and the value used in the conversion process can be notified.

[0540] Also, for example, in the attribute information acquisition (S10621), multiple pieces of attribute information are acquired, and in the encoding process (S10622), the attribute information is encoded by performing the above-mentioned conversion process for each piece of attribute information, or the attribute information is encoded without performing the above-mentioned conversion process, and in the bitstream generation, a bitstream is generated that includes the encoded multiple pieces of attribute information and conversion identification information for each piece of attribute information.

[0541] Depending on the three-dimensional data (data indicating three-dimensional points), one three-dimensional point may have multiple pieces of attribute information, or multiple three-dimensional points having one piece of attribute information may be decoded. Even in such cases, the attribute information can be appropriately converted and encoded for each of the multiple pieces of attribute information, and it can be notified that the attribute information has been converted.

[0542] Also, for example, if attribute information is encoded using a conversion process that performs both scaling and offsetting during encoding processing, then when generating a bitstream, a bitstream is generated that further includes order information indicating the order in which the scaling and offsetting were performed.

[0543] This allows the attribute information to be converted in an appropriate order to make the attribute information a positive number, for example, without having to predetermine the order of scale and offset, and also makes it possible to notify the order.

[0544] Also, for example, the transformation identification information indicates whether scaling has been performed and whether offsetting has been performed.

[0545] This allows the attribute information to be converted using an appropriate conversion process to make the attribute information a positive number, for example, without having to predetermine how to use at least one of the scale and offset, and the converted value can be notified.

[0546] Also, for example, in generating a bitstream, the conversion identification information is stored in the additional information common to the sequence.

[0547] Furthermore, for example, the three-dimensional data encoding device includes a processor and a memory, and the processor performs the above-mentioned processing using the memory. The memory may store a control program for performing the above-mentioned processing.

[0548] As described above, the three-dimensional data decoding device according to this embodiment performs the processing shown in FIG.

[0549] FIG. 76 is a flowchart showing the processing procedure of the three-dimensional data decoding device according to this embodiment.

[0550] First, the three-dimensional data decoding device obtains a bitstream including attribute information of encoded three-dimensional points and encoded transformation identification information indicating whether or not a transformation process has been performed on the attribute information, which involves scaling, which involves at least one of multiplication / division and shift operations, and offsetting, which involves addition / subtraction (S10631).

[0551] Next, the three-dimensional data decoding device decodes the encoded attribute information (S10632).

[0552] Next, the three-dimensional data decoding device decodes the coded transformation identification information (S10633). Note that the three-dimensional data decoding device may obtain a bitstream including coded attribute information of the three-dimensional points and uncoded transformation identification information. In this case, the three-dimensional data decoding device may obtain the transformation identification information from the bitstream and decode the coded attribute information.

[0553] Depending on the decoding unit (e.g., decoding unit 10640 or decoding unit 10680) that decodes the encoded attribute information, it may be impossible to process decimal points and / or negative numbers. Therefore, for example, if the attribute information is encoded after being converted using at least one of scaling and offset, even such a decoding unit can decode the encoded attribute information. Furthermore, because it is possible to determine whether the value indicated by the attribute information has been converted based on the conversion identification information, it is possible to restore the decoded attribute information to its original appropriate value.

[0554] Furthermore, for example, the three-dimensional data decoding device according to the present disclosure further performs an inverse transformation process on the numerical value indicated by the decoded attribute information, in which, when the transformation identification information indicates that a transformation process has been performed on the attribute information, the device performs at least one of inverse scaling, which involves performing at least one of multiplication / division and shift operations, and inverse offset, which involves adding / subtracting.

[0555] According to this, even if the value indicated by the attribute information is converted into a positive number based on the conversion identification information and then encoded, a conversion process that is reverse to the conversion process can be performed. Therefore, according to the three-dimensional data decoding method of the present disclosure, the encoded attribute information can be appropriately decoded, and the decoded attribute information can be appropriately converted (inversely converted) to return the attribute information to its original appropriate value.

[0556] Also, for example, the bitstream further includes conversion information including at least one of scale information indicating the value used for scaling (scale value) and offset information indicating the value used for offset (offset value), and in the inverse conversion process, the inverse conversion process is performed based on at least one of the scale information and the offset information.

[0557] This allows the decoded attribute information to be restored to its original appropriate value by inverse conversion processing without having to predetermine the scale value and offset value.

[0558] Also, for example, the bit stream includes multiple pieces of coded attribute information, and in the decoding process (S10632), each of the multiple pieces of coded attribute information is decoded, and the conversion identification information indicates whether or not a conversion process has been performed for each piece of attribute information, and in the inverse conversion process, the inverse conversion process is performed on attribute information among the multiple pieces of decoded attribute information whose conversion identification information indicates that a conversion process has been performed.

[0559] Depending on the 3D data, one 3D point may have multiple pieces of attribute information, or multiple 3D points having one piece of attribute information may be decoded. Even in such cases, for each of the multiple pieces of attribute information, the decoded attribute information can be restored to its original appropriate value by inverse conversion processing.

[0560] Also, for example, the bitstream further includes order information indicating the order in which scaling and offsetting are performed, and in the inverse transform process, inverse scaling and inverse offsetting are performed in an order based on the order information.

[0561] This allows the decoded attribute information to be restored to its original appropriate value by inverse conversion processing without having to predetermine the order of scale and offset.

[0562] Furthermore, for example, the transformation identification information indicates whether scaling has been performed and whether offsetting has been performed. That is, for example, the transformation identification information indicates whether scaling has been performed and whether offsetting has been performed.

[0563] This allows the decoded attribute information to be restored to its original appropriate value by inverse conversion processing without having to predetermine how to use at least one of the scale and the offset.

[0564] Also, for example, in the bitstream, the conversion identification information is stored in the additional information common to the sequence.

[0565] Furthermore, for example, the three-dimensional data decoding device includes a processor and a memory, and the processor performs the above-mentioned processing using the memory. The memory may store a control program for performing the above-mentioned processing.

[0566] (Embodiment 6) Next, the configuration of a three-dimensional data creation device 810 according to this embodiment will be described. Fig. 77 is a block diagram showing an example configuration of a three-dimensional data creation device 810 according to this embodiment. This three-dimensional data creation device 810 is mounted on a vehicle, for example. The three-dimensional data creation device 810 transmits and receives three-dimensional data to and from an external traffic monitoring cloud, a leading vehicle, or a following vehicle, and also creates and stores three-dimensional data.

[0567] The three-dimensional data creation device 810 includes a data receiving unit 811, a communication unit 812, a receiving control unit 813, a format conversion unit 814, multiple sensors 815, a three-dimensional data creation unit 816, a three-dimensional data synthesis unit 817, a three-dimensional data storage unit 818, a communication unit 819, a transmission control unit 820, a format conversion unit 821, and a data transmission unit 822.

[0568] The data receiving unit 811 receives three-dimensional data 831 from the traffic monitoring cloud or a preceding vehicle. The three-dimensional data 831 includes, for example, information such as a point cloud, visible light image, depth information, sensor position information, or speed information, including areas that cannot be detected by the sensor 815 of the vehicle itself.

[0569] The communication unit 812 communicates with the traffic monitoring cloud or the vehicle ahead, and transmits data transmission requests and the like to the traffic monitoring cloud or the vehicle ahead.

[0570] The reception control unit 813 exchanges information such as compatible formats with the communication destination via the communication unit 812, and establishes communication with the communication destination.

[0571] The format conversion unit 814 generates three-dimensional data 832 by performing format conversion or the like on the three-dimensional data 831 received by the data receiving unit 811. Furthermore, if the three-dimensional data 831 is compressed or encoded, the format conversion unit 814 performs decompression or decoding processing.

[0572] The multiple sensors 815 are a group of sensors, such as LiDAR, a visible light camera, or an infrared camera, that acquire information about the outside of the vehicle, and generate sensor information 833. For example, if the sensor 815 is a laser sensor such as LiDAR, the sensor information 833 is three-dimensional data such as a point cloud (point cloud data). Note that the number of sensors 815 does not need to be multiple.

[0573] The three-dimensional data creation unit 816 generates three-dimensional data 834 from the sensor information 833. The three-dimensional data 834 includes information such as a point cloud, a visible light image, depth information, sensor position information, or velocity information.

[0574] The three-dimensional data synthesis unit 817 synthesizes three-dimensional data 834 created based on the vehicle's own sensor information 833 with three-dimensional data 832 created by a traffic monitoring cloud or a preceding vehicle, etc., to construct three-dimensional data 835 that includes the space in front of the preceding vehicle, which cannot be detected by the vehicle's own sensor 815.

[0575] The three-dimensional data storage unit 818 stores the generated three-dimensional data 835 and the like.

[0576] The communication unit 819 communicates with the traffic monitoring cloud or the following vehicle, and transmits data transmission requests and the like to the traffic monitoring cloud or the following vehicle.

[0577] The transmission control unit 820 establishes communication with the communication destination by exchanging information such as compatible formats with the communication destination via the communication unit 819. Furthermore, the transmission control unit 820 determines a transmission region, which is the space of the three-dimensional data to be transmitted, based on the three-dimensional data construction information of the three-dimensional data 832 generated by the three-dimensional data synthesis unit 817 and a data transmission request from the communication destination.

[0578] Specifically, in response to a data transmission request from the traffic monitoring cloud or a following vehicle, the transmission control unit 820 determines a transmission area that includes the space ahead of the vehicle that cannot be detected by the sensors of the following vehicle. The transmission control unit 820 also determines the transmission area by determining whether the transmittable space or the transmitted space has been updated based on the three-dimensional data construction information. For example, the transmission control unit 820 determines the transmission area to be an area that is specified in the data transmission request and in which corresponding three-dimensional data 835 exists. The transmission control unit 820 then notifies the format conversion unit 821 of the format supported by the communication destination and the transmission area.

[0579] The format conversion unit 821 converts three-dimensional data 836 of the transmission region, out of the three-dimensional data 835 stored in the three-dimensional data storage unit 818, into a format compatible with the receiving side, thereby generating three-dimensional data 837. Note that the format conversion unit 821 may reduce the amount of data by compressing or encoding the three-dimensional data 837.

[0580] The data transmission unit 822 transmits three-dimensional data 837 to the traffic monitoring cloud or the following vehicle. This three-dimensional data 837 includes, for example, information such as a point cloud ahead of the vehicle, including areas that are blind spots for the following vehicle, a visible light image, depth information, or sensor position information.

[0581] Although an example in which format conversion is performed by the format conversion units 814 and 821 has been described, format conversion does not necessarily have to be performed.

[0582] With this configuration, the three-dimensional data creation device 810 externally acquires three-dimensional data 831 of an area that cannot be detected by the sensor 815 of the vehicle, and generates three-dimensional data 835 by combining the three-dimensional data 831 with three-dimensional data 834 based on sensor information 833 detected by the sensor 815 of the vehicle. In this way, the three-dimensional data creation device 810 can generate three-dimensional data of an area that cannot be detected by the sensor 815 of the vehicle.

[0583] In addition, in response to a data transmission request from a traffic monitoring cloud or a following vehicle, the three-dimensional data creation device 810 can transmit three-dimensional data including the space in front of the vehicle that cannot be detected by the sensors of the following vehicle to the traffic monitoring cloud or the following vehicle, etc.

[0584] Next, a description will be given of the procedure for transmitting three-dimensional data to a following vehicle in the three-dimensional data creation device 810. Fig. 78 is a flowchart showing an example of the procedure for transmitting three-dimensional data to a traffic monitoring cloud or a following vehicle by the three-dimensional data creation device 810.

[0585] First, the three-dimensional data creation device 810 generates and updates three-dimensional data 835 of a space including the space on the road ahead of the vehicle (S801). Specifically, the three-dimensional data creation device 810 constructs three-dimensional data 835 including the space ahead of the vehicle ahead that cannot be detected by the sensor 815 of the vehicle ahead, for example, by combining three-dimensional data 834 created based on sensor information 833 of the vehicle ahead with three-dimensional data 831 created by a traffic monitoring cloud or a vehicle ahead.

[0586] Next, the three-dimensional data creation device 810 determines whether the three-dimensional data 835 included in the transmitted space has changed (S802).

[0587] If a change occurs in the three-dimensional data 835 contained in the space that has already been transmitted, for example, due to a vehicle or person entering the space from outside (Yes in S802), the three-dimensional data creation device 810 transmits three-dimensional data including the three-dimensional data 835 of the space where the change has occurred to the traffic monitoring cloud or a following vehicle (S803).

[0588] The three-dimensional data creation device 810 may transmit the three-dimensional data of the space where a change has occurred in accordance with the transmission timing of the three-dimensional data transmitted at predetermined intervals, or may transmit the data immediately after detecting the change. In other words, the three-dimensional data creation device 810 may transmit the three-dimensional data of the space where a change has occurred in priority over the three-dimensional data transmitted at predetermined intervals.

[0589] Furthermore, the three-dimensional data creation device 810 may transmit all of the three-dimensional data of the space in which a change has occurred as the three-dimensional data of the space in which a change has occurred, or may transmit only the differences in the three-dimensional data (for example, information on three-dimensional points that have appeared or disappeared, or displacement information of three-dimensional points, etc.).

[0590] Furthermore, the three-dimensional data creation device 810 may transmit metadata regarding danger avoidance actions of the vehicle, such as a sudden braking warning, to the following vehicle prior to transmitting the three-dimensional data of the space where a change has occurred. This allows the following vehicle to recognize the sudden braking of the vehicle in front at an early stage and to begin danger avoidance actions, such as deceleration, at an earlier stage.

[0591] If there is no change in the three-dimensional data 835 contained in the transmitted space (No in S802), or after step S803, the three-dimensional data creation device 810 transmits the three-dimensional data contained in a space of a predetermined shape at a distance L ahead of the vehicle to the traffic monitoring cloud or a following vehicle (S804).

[0592] Furthermore, for example, the processes of steps S801 to S804 are repeatedly performed at predetermined time intervals.

[0593] Furthermore, if there is no difference between the three-dimensional data 835 of the space currently being transmitted and the three-dimensional map, the three-dimensional data creation device 810 does not need to transmit the three-dimensional data 837 of the space.

[0594] In this embodiment, the client device transmits sensor information obtained by the sensor to a server or another client device.

[0595] First, the configuration of a system according to this embodiment will be described. Fig. 79 is a diagram showing the configuration of a system for transmitting and receiving 3D maps and sensor information according to this embodiment. This system includes a server 901 and client devices 902A and 902B. When there is no need to distinguish between the client devices 902A and 902B, they will also be referred to as client device 902.

[0596] The client device 902 is, for example, an in-vehicle device mounted on a mobile object such as a vehicle. The server 901 is, for example, a traffic monitoring cloud or the like, and is capable of communicating with a plurality of client devices 902.

[0597] The server 901 transmits a three-dimensional map composed of a point cloud to the client device 902. Note that the composition of the three-dimensional map is not limited to a point cloud, and may represent other three-dimensional data such as a mesh structure.

[0598] The client device 902 transmits sensor information acquired by the client device 902 to the server 901. The sensor information includes, for example, at least one of LiDAR acquisition information, a visible light image, an infrared image, a depth image, sensor position information, and velocity information.

[0599] Data transmitted between the server 901 and the client device 902 may be compressed to reduce data size, or may remain uncompressed to maintain data accuracy. When compressing data, a three-dimensional compression method based on an octree structure, for example, can be used for point clouds. Also, a two-dimensional image compression method can be used for visible light images, infrared images, and depth images. Examples of two-dimensional image compression methods include MPEG-4 AVC or HEVC, which are standardized by MPEG.

[0600] Furthermore, the server 901 transmits the three-dimensional map managed by the server 901 to the client device 902 in response to a request to transmit the three-dimensional map from the client device 902. Note that the server 901 may transmit the three-dimensional map without waiting for a request to transmit the three-dimensional map from the client device 902. For example, the server 901 may broadcast the three-dimensional map to one or more client devices 902 in a predetermined space. Furthermore, the server 901 may transmit a three-dimensional map appropriate for the position of the client device 902 to the client device 902 that has received a transmission request once, at regular intervals. Furthermore, the server 901 may transmit the three-dimensional map to the client device 902 every time the three-dimensional map managed by the server 901 is updated.

[0601] The client device 902 issues a request to send a three-dimensional map to the server 901. For example, when the client device 902 wants to estimate its own position while traveling, the client device 902 sends a request to send a three-dimensional map to the server 901.

[0602] In the following cases, the client device 902 may issue a request to send a three-dimensional map to the server 901. If the three-dimensional map held by the client device 902 is old, the client device 902 may issue a request to send a three-dimensional map to the server 901. For example, if a certain period of time has passed since the client device 902 acquired the three-dimensional map, the client device 902 may issue a request to send a three-dimensional map to the server 901.

[0603] The client device 902 may issue a request to send a three-dimensional map to the server 901 a certain time before the client device 902 leaves the space shown in the three-dimensional map held by the client device 902. For example, when the client device 902 is located within a predetermined distance from the boundary of the space shown in the three-dimensional map held by the client device 902, the client device 902 may issue a request to send a three-dimensional map to the server 901. Furthermore, when the movement path and movement speed of the client device 902 are known, the time when the client device 902 will leave the space shown in the three-dimensional map held by the client device 902 may be predicted based on these.

[0604] If the error in aligning the three-dimensional data created by the client device 902 from sensor information with the three-dimensional map is equal to or greater than a certain level, the client device 902 may issue a request to the server 901 to send the three-dimensional map.

[0605] The client device 902 transmits sensor information to the server 901 in response to a request to transmit sensor information transmitted from the server 901. Note that the client device 902 may send sensor information to the server 901 without waiting for a request to transmit sensor information from the server 901. For example, once the client device 902 receives a request to transmit sensor information from the server 901, the client device 902 may periodically transmit the sensor information to the server 901 for a certain period of time. Furthermore, if the error in aligning the three-dimensional data created by the client device 902 based on the sensor information with the three-dimensional map obtained from the server 901 is equal to or greater than a certain level, the client device 902 may determine that a change may have occurred in the three-dimensional map around the client device 902, and may transmit this information along with the sensor information to the server 901.

[0606] The server 901 issues a request to transmit sensor information to the client device 902. For example, the server 901 receives location information of the client device 902, such as GPS, from the client device 902. When the server 901 determines, based on the location information of the client device 902, that the client device 902 is approaching a space with little information in the three-dimensional map managed by the server 901, it issues a request to transmit sensor information to the client device 902 in order to generate a new three-dimensional map. The server 901 may also issue a request to transmit sensor information when it wants to update the three-dimensional map, when it wants to check road conditions during snowfall or a disaster, when it wants to check traffic congestion conditions, or when it wants to check incident and accident conditions, etc.

[0607] Furthermore, the client device 902 may set the amount of data of the sensor information to be transmitted to the server 901 depending on the communication state or bandwidth at the time of receiving a request to transmit the sensor information from the server 901. Setting the amount of data of the sensor information to be transmitted to the server 901 means, for example, increasing or decreasing the amount of the data itself or appropriately selecting a compression method.

[0608] 80 is a block diagram showing an example configuration of a client device 902. The client device 902 receives a three-dimensional map composed of a point cloud or the like from the server 901, and estimates the self-position of the client device 902 from three-dimensional data created based on sensor information of the client device 902. The client device 902 also transmits the acquired sensor information to the server 901.

[0609] The client device 902 includes a data receiving unit 1011, a communication unit 1012, a receiving control unit 1013, a format conversion unit 1014, multiple sensors 1015, a three-dimensional data creation unit 1016, a three-dimensional image processing unit 1017, a three-dimensional data storage unit 1018, a format conversion unit 1019, a communication unit 1020, a transmission control unit 1021, and a data transmission unit 1022.

[0610] The data receiving unit 1011 receives a three-dimensional map 1031 from the server 901. The three-dimensional map 1031 is data including a point cloud such as a WLD or SWLD. The three-dimensional map 1031 may include either compressed data or uncompressed data.

[0611] The communication unit 1012 communicates with the server 901 and transmits data transmission requests (for example, requests to transmit a three-dimensional map) to the server 901 .

[0612] The reception control unit 1013 exchanges information such as compatible formats with the communication destination via the communication unit 1012, and establishes communication with the communication destination.

[0613] The format conversion unit 1014 generates a three-dimensional map 1032 by performing format conversion or the like on the three-dimensional map 1031 received by the data receiving unit 1011. Furthermore, if the three-dimensional map 1031 is compressed or encoded, the format conversion unit 1014 performs decompression or decoding processing. Note that if the three-dimensional map 1031 is uncompressed data, the format conversion unit 1014 does not perform decompression or decoding processing.

[0614] The multiple sensors 1015 are a group of sensors, such as LiDAR, a visible light camera, an infrared camera, or a depth sensor, that acquire information about the outside of the vehicle in which the client device 902 is mounted, and generate sensor information 1033. For example, if the sensor 1015 is a laser sensor such as LiDAR, the sensor information 1033 is three-dimensional data such as a point cloud (point cloud data). Note that the number of sensors 1015 does not need to be multiple.

[0615] The three-dimensional data creation unit 1016 creates three-dimensional data 1034 of the surroundings of the vehicle based on the sensor information 1033. For example, the three-dimensional data creation unit 1016 creates point cloud data with color information of the surroundings of the vehicle using information acquired by LiDAR and visible light images acquired by a visible light camera.

[0616] The three-dimensional image processing unit 1017 performs a process of estimating the vehicle's own position using a three-dimensional map 1032 such as a received point cloud and three-dimensional data 1034 of the surroundings of the vehicle generated from the sensor information 1033. The three-dimensional image processing unit 1017 may generate three-dimensional data 1035 of the surroundings of the vehicle by combining the three-dimensional map 1032 and the three-dimensional data 1034, and perform a process of estimating the vehicle's own position using the generated three-dimensional data 1035.

[0617] The three-dimensional data storage unit 1018 stores a three-dimensional map 1032, three-dimensional data 1034, three-dimensional data 1035, and the like.

[0618] The format conversion unit 1019 generates sensor information 1037 by converting the sensor information 1033 into a format supported by the receiving side. The format conversion unit 1019 may reduce the amount of data by compressing or encoding the sensor information 1037. The format conversion unit 1019 may omit processing when format conversion is not necessary. The format conversion unit 1019 may also control the amount of data to be transmitted in accordance with a specified transmission range.

[0619] The communication unit 1020 communicates with the server 901 and receives data transmission requests (sensor information transmission requests) and the like from the server 901 .

[0620] The transmission control unit 1021 exchanges information such as compatible formats with the communication destination via the communication unit 1020 to establish communication.

[0621] The data transmission unit 1022 transmits the sensor information 1037 to the server 901. The sensor information 1037 includes information acquired by the multiple sensors 1015, such as information acquired by LiDAR, a luminance image acquired by a visible light camera, an infrared image acquired by an infrared camera, a depth image acquired by a depth sensor, sensor position information, and speed information.

[0622] Next, the configuration of the server 901 will be described. Fig. 81 is a block diagram showing an example configuration of the server 901. The server 901 receives sensor information transmitted from the client device 902 and creates three-dimensional data based on the received sensor information. The server 901 uses the created three-dimensional data to update the three-dimensional map managed by the server 901. In addition, in response to a request from the client device 902 to transmit the three-dimensional map, the server 901 transmits the updated three-dimensional map to the client device 902.

[0623] The server 901 includes a data receiving unit 1111, a communication unit 1112, a receiving control unit 1113, a format conversion unit 1114, a three-dimensional data creation unit 1116, a three-dimensional data synthesis unit 1117, a three-dimensional data storage unit 1118, a format conversion unit 1119, a communication unit 1120, a transmission control unit 1121, and a data transmission unit 1122.

[0624] The data receiving unit 1111 receives sensor information 1037 from the client device 902. The sensor information 1037 includes, for example, information acquired by LiDAR, a luminance image acquired by a visible light camera, an infrared image acquired by an infrared camera, a depth image acquired by a depth sensor, sensor position information, and speed information.

[0625] The communication unit 1112 communicates with the client device 902 and transmits data transmission requests (for example, requests to transmit sensor information) to the client device 902 .

[0626] The reception control unit 1113 exchanges information such as compatible formats with the communication destination via the communication unit 1112 to establish communication.

[0627] If the received sensor information 1037 is compressed or encoded, the format conversion unit 1114 performs decompression or decoding processing to generate the sensor information 1132. Note that if the sensor information 1037 is uncompressed data, the format conversion unit 1114 does not perform decompression or decoding processing.

[0628] The three-dimensional data creation unit 1116 creates three-dimensional data 1134 of the periphery of the client device 902 based on the sensor information 1132. For example, the three-dimensional data creation unit 1116 creates point cloud data with color information of the periphery of the client device 902 using information acquired by LiDAR and visible light images acquired by a visible light camera.

[0629] The three-dimensional data synthesis unit 1117 synthesizes three-dimensional data 1134 created based on the sensor information 1132 with a three-dimensional map 1135 managed by the server 901, thereby updating the three-dimensional map 1135.

[0630] The three-dimensional data storage unit 1118 stores a three-dimensional map 1135 and the like.

[0631] The format conversion unit 1119 generates the three-dimensional map 1031 by converting the three-dimensional map 1135 into a format supported by the receiving side. The format conversion unit 1119 may reduce the amount of data by compressing or encoding the three-dimensional map 1135. The format conversion unit 1119 may also omit processing if format conversion is not necessary. The format conversion unit 1119 may also control the amount of data to be transmitted in accordance with the specified transmission range.

[0632] The communication unit 1120 communicates with the client device 902 and receives data transmission requests (requests to transmit three-dimensional maps) and the like from the client device 902 .

[0633] The transmission control unit 1121 exchanges information such as compatible formats with the communication destination via the communication unit 1120 to establish communication.

[0634] The data transmission unit 1122 transmits the three-dimensional map 1031 to the client device 902. The three-dimensional map 1031 is data including a point cloud such as a WLD or SWLD. The three-dimensional map 1031 may include either compressed data or uncompressed data.

[0635] Next, a description will be given of the operation flow of the client device 902. Fig. 82 is a flowchart showing the operation of the client device 902 when acquiring a three-dimensional map.

[0636] First, the client device 902 requests the server 901 to transmit a three-dimensional map (such as a point cloud) (S1001). At this time, the client device 902 may also transmit location information of the client device 902 obtained by GPS or the like, thereby requesting the server 901 to transmit a three-dimensional map related to the location information.

[0637] Next, the client device 902 receives the three-dimensional map from the server 901 (S1002). If the received three-dimensional map is compressed data, the client device 902 decodes the received three-dimensional map to generate an uncompressed three-dimensional map (S1003).

[0638] Next, the client device 902 creates three-dimensional data 1034 of the surroundings of the client device 902 from sensor information 1033 obtained by the multiple sensors 1015 (S1004). Next, the client device 902 estimates its own position using the three-dimensional map 1032 received from the server 901 and the three-dimensional data 1034 created from the sensor information 1033 (S1005).

[0639] 83 is a flowchart showing the operation of the client device 902 when transmitting sensor information. First, the client device 902 receives a request to transmit sensor information from the server 901 (S1011). Upon receiving the transmission request, the client device 902 transmits sensor information 1037 to the server 901 (S1012). Note that, when the sensor information 1033 includes multiple pieces of information obtained by multiple sensors 1015, the client device 902 may generate the sensor information 1037 by compressing each piece of information using a compression method suitable for that piece of information.

[0640] Next, the operation flow of the server 901 will be described. FIG. 84 is a flowchart showing the operation of the server 901 when acquiring sensor information. First, the server 901 requests the client device 902 to transmit sensor information (S1021). Next, the server 901 receives the sensor information 1037 transmitted from the client device 902 in response to the request (S1022). Next, the server 901 creates three-dimensional data 1134 using the received sensor information 1037 (S1023). Next, the server 901 reflects the created three-dimensional data 1134 in the three-dimensional map 1135 (S1024).

[0641] 85 is a flowchart showing the operation of the server 901 when transmitting a three-dimensional map. First, the server 901 receives a request to transmit a three-dimensional map from the client device 902 (S1031). Having received the request to transmit the three-dimensional map, the server 901 transmits a three-dimensional map 1031 to the client device 902 (S1032). At this time, the server 901 may extract a three-dimensional map of the vicinity of the client device 902 based on its location information and transmit the extracted three-dimensional map. The server 901 may also compress the three-dimensional map composed of a point cloud using, for example, a compression method with an octree structure and transmit the compressed three-dimensional map.

[0642] A modification of this embodiment will now be described.

[0643] The server 901 uses the sensor information 1037 received from the client device 902 to create three-dimensional data 1134 of the vicinity of the position of the client device 902. Next, the server 901 matches the created three-dimensional data 1134 with a three-dimensional map 1135 of the same area managed by the server 901, thereby calculating the difference between the three-dimensional data 1134 and the three-dimensional map 1135. If the difference is equal to or greater than a predetermined threshold, the server 901 determines that some abnormality has occurred in the vicinity of the client device 902. For example, when ground subsidence or the like occurs due to a natural disaster such as an earthquake, a large difference may occur between the three-dimensional map 1135 managed by the server 901 and the three-dimensional data 1134 created based on the sensor information 1037.

[0644] The sensor information 1037 may include information indicating at least one of the sensor type, sensor performance, and sensor model number. Furthermore, a class ID or the like according to the sensor performance may be added to the sensor information 1037. For example, if the sensor information 1037 is information acquired by LiDAR, an identifier may be assigned to the sensor performance, such as Class 1 for a sensor that can acquire information with an accuracy of several millimeters, Class 2 for a sensor that can acquire information with an accuracy of several centimeters, and Class 3 for a sensor that can acquire information with an accuracy of several meters. Furthermore, the server 901 may estimate the sensor performance information, etc., from the model number of the client device 902. For example, if the client device 902 is installed in a vehicle, the server 901 may determine the sensor specification information from the vehicle model. In this case, the server 901 may acquire vehicle model information in advance, or the information may be included in the sensor information. Furthermore, the server 901 may use the acquired sensor information 1037 to switch the degree of correction for the three-dimensional data 1134 created using the sensor information 1037. For example, if the sensor performance is high accuracy (class 1), the server 901 does not perform correction on the three-dimensional data 1134. If the sensor performance is low accuracy (class 3), the server 901 applies correction according to the accuracy of the sensor to the three-dimensional data 1134. For example, the server 901 increases the degree (strength) of correction as the accuracy of the sensor decreases.

[0645] The server 901 may simultaneously issue requests to send sensor information to multiple client devices 902 in a certain space. When the server 901 receives multiple pieces of sensor information from multiple client devices 902, the server 901 does not need to use all of the sensor information to create the three-dimensional data 1134. For example, the server 901 may select the sensor information to use depending on the performance of the sensor. For example, when updating the three-dimensional map 1135, the server 901 may select high-precision sensor information (Class 1) from the multiple pieces of sensor information it has received and use the selected sensor information to create the three-dimensional data 1134.

[0646] The server 901 is not limited to a server such as a traffic monitoring cloud, but may be another client device (mounted in a vehicle). Fig. 86 is a diagram showing the system configuration in this case.

[0647] For example, client device 902C issues a request to send sensor information to nearby client device 902A and acquires the sensor information from client device 902A. Client device 902C then creates three-dimensional data using the acquired sensor information from client device 902A and updates the three-dimensional map of client device 902C. This allows client device 902C to generate a three-dimensional map of the space that can be acquired from client device 902A by taking advantage of the performance of client device 902C. For example, it is thought that such a case will occur when client device 902C has high performance.

[0648] In this case, the client device 902A that provided the sensor information is granted the right to obtain the high-precision 3D map generated by the client device 902C. The client device 902A receives the high-precision 3D map from the client device 902C in accordance with this right.

[0649] In addition, the client device 902C may issue a request to transmit sensor information to multiple nearby client devices 902 (client device 902A and client device 902B). If the sensor of the client device 902A or the client device 902B is high performance, the client device 902C can create three-dimensional data using the sensor information obtained by this high performance sensor.

[0650] 87 is a block diagram showing the functional configuration of the server 901 and the client device 902. The server 901 includes, for example, a 3D map compression / decoding processing unit 1201 that compresses and decodes 3D maps, and a sensor information compression / decoding processing unit 1202 that compresses and decodes sensor information.

[0651] The client device 902 includes a three-dimensional map decoding processing unit 1211 and a sensor information compression processing unit 1212. The three-dimensional map decoding processing unit 1211 receives encoded data of the compressed three-dimensional map and decodes the encoded data to acquire the three-dimensional map. The sensor information compression processing unit 1212 compresses the sensor information itself instead of three-dimensional data created from the acquired sensor information, and transmits the encoded data of the compressed sensor information to the server 901. With this configuration, the client device 902 only needs to internally include a processing unit (device or LSI) that performs processing to decode the three-dimensional map (point cloud, etc.), and does not need to internally include a processing unit that performs processing to compress the three-dimensional data of the three-dimensional map (point cloud, etc.). This allows the cost and power consumption of the client device 902 to be reduced.

[0652] As described above, the client device 902 according to this embodiment is mounted on a mobile body and generates three-dimensional data 1034 of the surroundings of the mobile body from sensor information 1033 indicating the surrounding conditions of the mobile body, which is obtained by the sensor 1015 mounted on the mobile body. The client device 902 estimates the self-position of the mobile body using the generated three-dimensional data 1034. The client device 902 transmits the acquired sensor information 1033 to the server 901 or another client device 902.

[0653] According to this, the client device 902 transmits the sensor information 1033 to the server 901 or the like. This may reduce the amount of data to be transmitted compared to when transmitting three-dimensional data. Furthermore, since the client device 902 does not need to perform processing such as compression or encoding of the three-dimensional data, the amount of processing by the client device 902 can be reduced. Therefore, the client device 902 can reduce the amount of data to be transmitted or simplify the device configuration.

[0654] Furthermore, the client device 902 further transmits a request to the server 901 to send a three-dimensional map, and receives a three-dimensional map 1031 from the server 901. The client device 902 estimates its own location using the three-dimensional data 1034 and the three-dimensional map 1032.

[0655] The sensor information 1033 also includes at least one of information obtained by a laser sensor, a luminance image, an infrared image, a depth image, sensor position information, and sensor speed information.

[0656] The sensor information 1033 also includes information indicating the performance of the sensor.

[0657] Furthermore, the client device 902 encodes or compresses the sensor information 1033, and transmits the encoded or compressed sensor information 1037 to the server 901 or another client device 902. This allows the client device 902 to reduce the amount of data to be transmitted.

[0658] For example, the client device 902 includes a processor and a memory, and the processor uses the memory to perform the above-described processing.

[0659] Furthermore, the server 901 according to this embodiment is capable of communicating with a client device 902 mounted on the mobile object, and receives sensor information 1037 indicating the surrounding conditions of the mobile object, obtained by a sensor 1015 mounted on the mobile object, from the client device 902. The server 901 creates three-dimensional data 1134 of the surroundings of the mobile object from the received sensor information 1037.

[0660] According to this, the server 901 creates three-dimensional data 1134 using the sensor information 1037 transmitted from the client device 902. This may reduce the amount of data to be transmitted compared to when the client device 902 transmits the three-dimensional data. Furthermore, since the client device 902 does not need to perform processes such as compression or encoding of the three-dimensional data, the amount of processing by the client device 902 can be reduced. Therefore, the server 901 can reduce the amount of data to be transmitted or simplify the device configuration.

[0661] Furthermore, the server 901 further transmits a request to the client device 902 to transmit sensor information.

[0662] The server 901 also updates a three-dimensional map 1135 using the created three-dimensional data 1134 and transmits the three-dimensional map 1135 to the client device 902 in response to a request from the client device 902 to transmit the three-dimensional map 1135.

[0663] The sensor information 1037 also includes at least one of information obtained by a laser sensor, a luminance image, an infrared image, a depth image, sensor position information, and sensor speed information.

[0664] The sensor information 1037 also includes information indicating the performance of the sensor.

[0665] Furthermore, the server 901 further corrects the three-dimensional data in accordance with the performance of the sensor. In this way, the three-dimensional data creation method can improve the quality of the three-dimensional data.

[0666] Furthermore, when receiving sensor information, the server 901 receives a plurality of pieces of sensor information 1037 from a plurality of client devices 902, and selects the sensor information 1037 to be used to create the three-dimensional data 1134 based on a plurality of pieces of information indicating the performance of the sensors included in the plurality of pieces of sensor information 1037. This allows the server 901 to improve the quality of the three-dimensional data 1134.

[0667] Furthermore, the server 901 decodes or decompresses the received sensor information 1037, and creates three-dimensional data 1134 from the decoded or decompressed sensor information 1132. This allows the server 901 to reduce the amount of data to be transmitted.

[0668] For example, the server 901 includes a processor and a memory, and the processor uses the memory to perform the above-mentioned processing.

[0669] A modified example will be described below. Fig. 88 is a diagram showing the configuration of a system according to this embodiment. The system shown in Fig. 88 includes a server 2001, a client device 2002A, and a client device 2002B.

[0670] Client device 2002A and client device 2002B are mounted on a moving body such as a vehicle, and transmit sensor information to server 2001. Server 2001 transmits a three-dimensional map (point cloud) to client device 2002A and client device 2002B.

[0671] The client device 2002A includes a sensor information acquisition unit 2011, a storage unit 2012, and a data transmission permission determination unit 2013. The client device 2002B has a similar configuration. In the following description, when there is no need to distinguish between the client device 2002A and the client device 2002B, they will also be referred to as the client device 2002.

[0672] FIG. 89 is a flowchart showing the operation of the client device 2002 according to this embodiment.

[0673] The sensor information acquisition unit 2011 acquires various sensor information using a sensor (sensor group) mounted on the mobile object. That is, the sensor information acquisition unit 2011 acquires sensor information indicating the surrounding conditions of the mobile object, obtained by the sensor (sensor group) mounted on the mobile object. The sensor information acquisition unit 2011 also stores the acquired sensor information in the storage unit 2012. This sensor information includes at least one of LiDAR acquisition information, visible light images, infrared images, and depth images. The sensor information may also include at least one of sensor position information, speed information, acquisition time information, and acquisition location information. The sensor position information indicates the position of the sensor that acquired the sensor information. The speed information indicates the speed of the mobile object when the sensor acquired the sensor information. The acquisition time information indicates the time when the sensor information was acquired by the sensor. The acquisition location information indicates the position of the mobile object or the sensor when the sensor information was acquired by the sensor.

[0674] Next, the data transmission possibility determination unit 2013 determines whether the mobile object (client device 2002) is in an environment where it can transmit sensor information to the server 2001 (S2002). For example, the data transmission possibility determination unit 2013 may use information such as GPS to identify the location and time of the client device 2002 and determine whether data transmission is possible. Alternatively, the data transmission possibility determination unit 2013 may determine whether data transmission is possible based on whether connection to a specific access point is possible.

[0675] When the client device 2002 determines that the mobile object is in an environment where it can transmit sensor information to the server 2001 (Yes in S2002), the client device 2002 transmits the sensor information to the server 2001 (S2003). In other words, when the client device 2002 is in a situation where it can transmit the sensor information to the server 2001, the client device 2002 transmits the sensor information it holds to the server 2001. For example, a millimeter wave access point capable of high-speed communication is installed at an intersection or the like. When the client device 2002 enters the intersection, it uses millimeter wave communication to transmit the sensor information it holds to the server 2001 at high speed.

[0676] Next, the client device 2002 deletes the sensor information that has been transmitted to the server 2001 from the storage unit 2012 (S2004). The client device 2002 may delete the sensor information that has not been transmitted to the server 2001 if the sensor information satisfies a predetermined condition. For example, the client device 2002 may delete the sensor information from the storage unit 2012 when the acquisition time of the retained sensor information becomes older than a predetermined time from the current time. That is, the client device 2002 may delete the sensor information from the storage unit 2012 when the difference between the time when the sensor information was acquired by the sensor and the current time exceeds a predetermined time. The client device 2002 may also delete the sensor information from the storage unit 2012 when the acquisition location of the retained sensor information becomes more than a predetermined distance away from the current location. That is, the client device 2002 may delete the sensor information from the storage unit 2012 when the difference between the position of the mobile object or sensor when the sensor information was acquired by the sensor and the current position of the mobile object or sensor exceeds a predetermined distance. This makes it possible to reduce the capacity of the storage unit 2012 of the client device 2002 .

[0677] If the client device 2002 has not yet finished acquiring the sensor information (No in S2005), the client device 2002 repeats the processes from step S2001 onward. If the client device 2002 has finished acquiring the sensor information (Yes in S2005), the client device 2002 ends the process.

[0678] Furthermore, the client device 2002 may select the sensor information to be transmitted to the server 2001 in accordance with the communication conditions. For example, when high-speed communication is possible, the client device 2002 transmits sensor information (e.g., LiDAR acquisition information) that is large in size and stored in the storage unit 2012 with priority. When high-speed communication is difficult, the client device 2002 transmits sensor information (e.g., visible light images) that is small in size and has high priority and stored in the storage unit 2012. This allows the client device 2002 to efficiently transmit the sensor information stored in the storage unit 2012 to the server 2001 in accordance with the network conditions.

[0679] The client device 2002 may also acquire time information indicating the current time and location information indicating the current location from the server 2001. The client device 2002 may also determine the acquisition time and acquisition location of the sensor information based on the acquired time information and location information. That is, the client device 2002 may acquire time information from the server 2001 and generate acquired time information using the acquired time information. The client device 2002 may also acquire location information from the server 2001 and generate acquired location information using the acquired location information.

[0680] For example, with regard to time information, the server 2001 and the client device 2002 perform time synchronization using a mechanism such as NTP (Network Time Protocol) or PTP (Precision Time Protocol). This allows the client device 2002 to acquire accurate time information. Furthermore, since time can be synchronized between the server 2001 and multiple client devices, the time in the sensor information acquired by different client devices 2002 can be synchronized. Therefore, the server 2001 can handle sensor information indicating synchronized time. Note that the time synchronization mechanism may be any method other than NTP or PTP. Furthermore, GPS information may be used as the time information and location information.

[0681] The server 2001 may acquire sensor information from multiple client devices 2002 by specifying a time or a location. For example, if an accident occurs, in order to search for clients who were nearby, the server 2001 broadcasts a sensor information transmission request to multiple client devices 2002 by specifying the time and location of the accident. Then, the client devices 2002 that have sensor information for the corresponding time and location transmit the sensor information to the server 2001. That is, the client device 2002 receives a sensor information transmission request from the server 2001, including specification information that specifies the location and time. If the client device 2002 determines that the sensor information obtained at the location and time indicated by the specification information is stored in the memory unit 2012 and that a mobile object is in an environment where it can transmit sensor information to the server 2001, the client device 2002 transmits the sensor information obtained at the location and time indicated by the specification information to the server 2001. In this way, the server 2001 can acquire sensor information related to the occurrence of an accident from multiple client devices 2002 and use it for accident analysis, etc.

[0682] The client device 2002 may refuse to transmit the sensor information when receiving a request to transmit sensor information from the server 2001. The client device 2002 may also set in advance which of the multiple pieces of sensor information are transmittable. Alternatively, the server 2001 may inquire of the client device 2002 each time whether or not it is possible to transmit the sensor information.

[0683] In addition, points may be awarded to the client device 2002 that transmits sensor information to the server 2001. These points can be used to pay for, for example, gasoline purchases, EV (electric vehicle) charging fees, highway tolls, or rental car fees. After acquiring the sensor information, the server 2001 may delete information for identifying the client device 2002 that transmitted the sensor information. For example, this information may be information such as the network address of the client device 2002. This allows the sensor information to be anonymized, so that the user of the client device 2002 can transmit the sensor information to the server 2001 with peace of mind. In addition, the server 2001 may be composed of multiple servers. For example, by sharing sensor information among multiple servers, even if one server fails, other servers can communicate with the client device 2002. This makes it possible to avoid service interruptions due to server failures.

[0684] Furthermore, the specified location specified in the sensor information transmission request indicates the location of an accident, etc., and may differ from the location of the client device 2002 at the specified time specified in the sensor information transmission request. Therefore, the server 2001 can specify a range, such as within XX meters of the specified location, to request information acquisition from the client device 2002 located within that range. Similarly, for the specified time, the server 2001 may specify a range, such as within N seconds before and after a certain time. This allows the server 2001 to acquire sensor information from the client device 2002 that was located "at a location: within XX meters of absolute position S, from time: t-N to t+N." When transmitting three-dimensional data such as LiDAR data, the client device 2002 may transmit data generated immediately after time t.

[0685] Furthermore, the server 2001 may separately specify, as the specified location, information indicating the location of the client device 2002 from which sensor information is to be acquired and the location from which the sensor information is desired. For example, the server 2001 specifies that sensor information including at least a range of YYm from the absolute position S is to be acquired from the client device 2002 that is located within XXm of the absolute position S. When selecting three-dimensional data to transmit, the client device 2002 selects one or more randomly accessible units of three-dimensional data so as to include at least the sensor information in the specified range. Furthermore, when transmitting visible light images, the client device 2002 may transmit multiple temporally consecutive image data including at least a frame immediately before or after time t.

[0686] If the client device 2002 can use multiple physical networks, such as 5G or Wi-Fi, or multiple modes in 5G, to transmit sensor information, the client device 2002 may select a network to use according to the priority order notified by the server 2001. Alternatively, the client device 2002 itself may select a network that can ensure an appropriate bandwidth based on the size of the data to be transmitted. Alternatively, the client device 2002 may select a network to use based on the cost of transmitting data, etc. Furthermore, the transmission request from the server 2001 may include information indicating a transmission deadline, such as a request that the client device 2002 transmit if it can start transmission by time T. If the server 2001 is unable to acquire sufficient sensor information within the deadline, the server 2001 may issue a transmission request again.

[0687] The sensor information may include header information indicating characteristics of the sensor data along with compressed or uncompressed sensor data. The client device 2002 may transmit the header information to the server 2001 via a physical network or communication protocol different from that for the sensor data. For example, the client device 2002 transmits the header information to the server 2001 prior to transmitting the sensor data. The server 2001 determines whether to acquire the sensor data of the client device 2002 based on an analysis result of the header information. For example, the header information may include information indicating the LiDAR point cloud acquisition density, elevation angle, or frame rate, or the resolution, signal-to-noise ratio, or frame rate of the visible light image. This allows the server 2001 to acquire sensor information from the client device 2002 having sensor data of the determined quality.

[0688] As described above, the client device 2002 is mounted on a mobile object, acquires sensor information indicating the surrounding conditions of the mobile object obtained by the sensors mounted on the mobile object, and stores the sensor information in the storage unit 2012. The client device 2002 determines whether the mobile object is in an environment where it can transmit the sensor information to the server 2001, and if it determines that the mobile object is in an environment where it can transmit the sensor information to the server, transmits the sensor information to the server 2001.

[0689] Furthermore, the client device 2002 creates three-dimensional data of the surroundings of the mobile object from the sensor information, and estimates the mobile object's own position using the created three-dimensional data.

[0690] Furthermore, the client device 2002 further transmits a request to the server 2001 to send a three-dimensional map, and receives the three-dimensional map from the server 2001. The client device 2002 estimates its own location using the three-dimensional data and the three-dimensional map.

[0691] The processing by the client device 2002 may be realized as an information transmission method in the client device 2002.

[0692] The client device 2002 may also include a processor and a memory, and the processor may use the memory to perform the above processing.

[0693] Next, a sensor information collection system according to this embodiment will be described. Fig. 90 is a diagram showing the configuration of the sensor information collection system according to this embodiment. As shown in Fig. 90, the sensor information collection system according to this embodiment includes a terminal 2021A, a terminal 2021B, a communication device 2022A, a communication device 2022B, a network 2023, a data collection server 2024, a map server 2025, and a client device 2026. Note that when there is no particular need to distinguish between the terminal 2021A and the terminal 2021B, they are also referred to as terminal 2021. When there is no particular need to distinguish between the communication devices 2022A and 2022B, they are also referred to as communication device 2022.

[0694] The data collection server 2024 collects data such as sensor data obtained by a sensor provided in the terminal 2021 as position-related data associated with a position in three-dimensional space.

[0695] The sensor data is, for example, data acquired by using a sensor provided in the terminal 2021, such as the state of the surroundings of the terminal 2021 or the state of the inside of the terminal 2021. The terminal 2021 transmits to the data collection server 2024 sensor data collected from on...

Claims

1. A three-dimensional data encoding method comprising: acquiring attribute information of a three-dimensional point; performing a conversion process on a numerical value indicated by the attribute information, such as scaling by performing at least one of multiplication / division and shift operations, and offsetting by performing addition / subtraction, and then encoding the attribute information; or performing an encoding process for encoding the attribute information without performing the conversion process; and generating a bit stream including the encoded attribute information and conversion identification information indicating whether the conversion process has been performed.

2. The three-dimensional data encoding method according to claim 1, wherein, in generating the bit stream, when the conversion process is executed, the bit stream is generated further including conversion information including at least one of scale information indicating the value used for the scaling and offset information indicating the value used for the offset.

3. The three-dimensional data encoding method according to claim 1 or 2, wherein the acquiring of the attribute information comprises acquiring a plurality of pieces of the attribute information; the encoding process comprises, for each piece of the attribute information, executing the conversion process to encode the attribute information or encoding the attribute information without executing the conversion process; and the generating of the bit stream comprises generating the bit stream including the encoded plurality of pieces of the attribute information and the conversion identification information for each piece of the attribute information.

4. A three-dimensional data encoding method according to any one of claims 1 to 3, wherein, in the encoding process, if the attribute information is encoded by performing the conversion process that performs both the scaling and the offsetting, then, in generating the bit stream, the bit stream is generated which further includes order information indicating the order in which the scaling and the offsetting were performed.

5. A three-dimensional data encoding method according to any one of claims 1 to 4, wherein the transformation identification information indicates whether or not the scaling has been performed and whether or not the offset has been performed.

6. The three-dimensional data encoding method according to any one of claims 1 to 5, wherein in generating the bit stream, the conversion identification information is stored in additional information common to a sequence.

7. A three-dimensional data decoding method, comprising: acquiring a bit stream including coded attribute information of a three-dimensional point; and coded transformation identification information indicating whether or not a transformation process has been performed on the attribute information, the transformation process including at least one of scaling, which involves at least one of multiplication / division and shift operations, and offset, which involves addition / subtraction; decoding the coded attribute information; and decoding the coded transformation identification information.

8. The three-dimensional data decoding method according to claim 7, further comprising the steps of: when the conversion identification information indicates that the conversion process has been performed on the attribute information, performing an inverse conversion process on the numerical value indicated by the decoded attribute information, which performs at least one of inverse scaling, which involves multiplication / division and / or shift operations, and at least one of inverse offset, which involves addition / subtraction.

9. The three-dimensional data decoding method according to claim 8, wherein the bit stream further includes transformation information including at least one of scale information indicating a value used for the scaling and offset information indicating a value used for the offset, and wherein the inverse transformation process is performed based on at least one of the scale information and the offset information.

10. A three-dimensional data decoding method as described in claim 8 or 9, wherein the bit stream includes a plurality of encoded pieces of attribute information, and in the decoding, each of the plurality of encoded pieces of attribute information is decoded, the transformation identification information indicates whether or not the transformation process has been performed for each piece of attribute information, and in the inverse transformation process, the inverse transformation process is performed on attribute information among the plurality of decoded pieces of attribute information for which the transformation identification information indicates that the transformation process has been performed.

11. A three-dimensional data decoding method according to any one of claims 8 to 10, wherein the bit stream further includes order information indicating an order in which the scaling and offsetting were performed, and in the inverse transformation process, the inverse scaling and the inverse offsetting are performed in an order based on the order information.

12. A three-dimensional data decoding method according to any one of claims 7 to 11, wherein the transformation identification information indicates whether or not the scaling has been performed and whether or not the offset has been performed.

13. The three-dimensional data decoding method according to any one of claims 7 to 12, wherein the bit stream has the conversion identification information stored in additional information common to a sequence.

14. A three-dimensional data encoding device comprising: a processor; and a memory, wherein the processor uses the memory to acquire attribute information of a three-dimensional point; perform a conversion process on a numerical value indicated by the attribute information, such as scaling, which involves at least one of multiplication / division and shift operations, and offset, which involves addition / subtraction, before encoding the attribute information; or perform an encoding process to encode the attribute information without performing the conversion process; and generate a bit stream including the encoded attribute information and conversion identification information indicating whether the conversion process has been performed.

15. A three-dimensional data decoding device comprising: a processor; and a memory, wherein the processor uses the memory to obtain a bit stream including coded attribute information of three-dimensional points and coded transformation identification information indicating whether or not a transformation process has been performed on the attribute information, the transformation process including scaling which performs at least one of multiplication / division and shift operations, and offset which performs addition / subtraction, the bit stream decoding the coded attribute information; and the coded transformation identification information decoding the coded transformation identification information.

Images