Three-dimensional data encoding method and three-dimensional data decoding method

Abstract

The invention relates to a three-dimensional data encoding method and a three-dimensional data decoding method. The three-dimensional data encoding method includes: (i) if the number of a plurality of reference values available for calculating a prediction value of a current value in one prediction mode selected from two or more prediction modes is greater than a prescribed number corresponding to the selected one prediction mode, encoding the current value in the one prediction mode, and if the number of the plurality of reference values is greater than the prescribed number corresponding to the selected one prediction mode, encoding the current value; (a) calculates the prediction value according to the selected one prediction mode, and (b) calculates a prediction residual, which is the difference between the current value and the calculated prediction value, (ii) if the number is equal to or less than the predetermined number, does not perform the calculation of the prediction value, and if the number is equal to or less than the predetermined number, does not perform the calculation of the prediction value. And generating a bitstream containing the prediction residual.

Description

[0001] This application is a divisional case of the invention patent application with the application date of June 14, 2019, application number 201980039596.7, and invention name “Three-dimensional data encoding method, three-dimensional data decoding method, three-dimensional data encoding device, and three-dimensional data decoding device”. Technical Field

[0002] The present disclosure relates to a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, and a three-dimensional data decoding device. Background Art

[0003] Devices and services that utilize three-dimensional data will become increasingly common in a wide range of fields, including computer vision for autonomous vehicles and robots, mapping, surveillance, infrastructure inspection, and image distribution. Three-dimensional data is acquired using various methods, including distance sensors such as rangefinders, stereo cameras, and combinations of multiple single-lens reflex cameras.

[0004] One method of representing three-dimensional data is called a point cloud, which uses a cluster of points within a three-dimensional space to represent the shape of a three-dimensional structure. The point cloud stores the position and color of the point cluster. While point clouds are expected to become a mainstream method for representing three-dimensional data, the amount of point cluster data is extremely large. Therefore, when storing or transmitting three-dimensional data, as with two-dimensional moving images (for example, MPEG-4 AVC and HEVC, which have been standardized in MPEG), data compression through encoding is required.

[0005] Furthermore, compression of point clouds is partially supported by a public library (PointCloud Library) that performs point cloud association processing.

[0006] Furthermore, there is a known technique for searching for facilities around a vehicle using three-dimensional map data and displaying the facilities (for example, see Patent Document 1).

[0007] Prior art literature

[0008] Patent Literature

[0009] Patent Document 1 International Publication No. 2014 / 020663 Summary of the Invention

[0010] Problems to be solved by the invention

[0011] It is desirable to correctly decode attribute information when decoding three-dimensional data.

[0012] An object of the present disclosure is to provide a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, or a three-dimensional data decoding device that can correctly decode attribute information during the decoding of three-dimensional data.

[0013] Means used to solve problems

[0014] A three-dimensional data encoding method according to one embodiment of the present invention includes: (i) when the number of multiple reference values ​​that can be used to calculate a prediction value of a current value in a prediction mode selected from two or more prediction modes is greater than a prescribed number corresponding to the selected prediction mode, (a) calculating the prediction value according to the selected prediction mode, and (b) calculating the difference between the current value and the calculated prediction value, i.e., the prediction residual, and (ii) when the number is the same as or less than the prescribed number, not performing the calculation of the prediction value and generating a bit stream containing the prediction residual.

[0015] A three-dimensional data decoding method according to one embodiment of the present invention includes: (i) when the number of multiple reference values ​​of a prediction value that can be used to calculate a current value in a prediction mode selected from two or more prediction modes is greater than a prescribed number corresponding to the selected prediction mode, (a) obtaining a bit stream including a prediction mode and a prediction residual, (b) calculating the prediction value according to the obtained prediction mode; and (c) calculating the current value by adding the prediction value to the prediction residual, and (ii) when the number is the same as or less than the prescribed number, not performing the calculation of the prediction value.

[0016] Effects of the Invention

[0017] The present disclosure can provide a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, or a three-dimensional data decoding device that can correctly decode attribute information during decoding of three-dimensional data. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 The structure of the encoded three-dimensional data in the first embodiment is shown.

[0019] Figure 2 An example of the prediction structure between SPCs belonging to the lowest layer of the GOS in the first embodiment is shown.

[0020] Figure 3 An example of the inter-layer prediction structure in the first embodiment is shown.

[0021] Figure 4 An example of the encoding order of the GOS in the first embodiment is shown.

[0022] Figure 5 An example of the encoding order of the GOS in the first embodiment is shown.

[0023] Figure 6 This is a block diagram of the three-dimensional data encoding device according to the first embodiment.

[0024] Figure 7 This is a flowchart of the encoding process in implementation mode 1.

[0025] Figure 8 This is a block diagram of the three-dimensional data decoding device according to the first embodiment.

[0026] Figure 9 This is a flowchart of the decoding process in the first embodiment.

[0027] Figure 10 An example of meta-information in the first embodiment is shown.

[0028] Figure 11 A configuration example of a SWLD according to the second embodiment is shown.

[0029] Figure 12 An operation example of the server and client according to the second embodiment is shown.

[0030] Figure 13 An operation example of the server and client according to the second embodiment is shown.

[0031] Figure 14 An operation example of the server and client according to the second embodiment is shown.

[0032] Figure 15 An operation example of the server and client according to the second embodiment is shown.

[0033] Figure 16 This is a block diagram of a three-dimensional data encoding device according to Embodiment 2.

[0034] Figure 17 This is a flowchart of the encoding process in implementation mode 2.

[0035] Figure 18 This is a block diagram of a three-dimensional data decoding device according to Embodiment 2.

[0036] Figure 19 This is a flowchart of the decoding process of implementation mode 2.

[0037] Figure 20 A configuration example of a WLD according to the second embodiment is shown.

[0038] Figure 21 An example of the octree structure of the WLD according to the second embodiment is shown.

[0039] Figure 22 A configuration example of a SWLD according to the second embodiment is shown.

[0040] Figure 23 An example of the octree structure of the SWLD according to the second embodiment is shown.

[0041] Figure 24 This is a block diagram of a three-dimensional data creation device according to a third embodiment.

[0042] Figure 25 This is a block diagram of a three-dimensional data transmitting device according to a third embodiment.

[0043] Figure 26 This is a block diagram of a three-dimensional information processing device according to a fourth embodiment.

[0044] Figure 27 This is a block diagram of a three-dimensional data creation device according to a fifth embodiment.

[0045] Figure 28 The configuration of the system according to the sixth embodiment is shown.

[0046] Figure 29 This is a block diagram of a client device according to a sixth embodiment.

[0047] Figure 30 This is a block diagram of a server in implementation mode 6.

[0048] Figure 31 This is a flowchart of the three-dimensional data creation process performed by the client device in the sixth embodiment.

[0049] Figure 32 This is a flowchart of the sensor information transmission process performed by the client device in the sixth embodiment.

[0050] Figure 33 This is a flowchart of the three-dimensional data creation process performed by the server in the sixth embodiment.

[0051] Figure 34 This is a flowchart of the three-dimensional map transmission processing performed by the server in the sixth embodiment.

[0052] Figure 35 The configuration of a modified example of the system of the sixth embodiment is shown.

[0053] Figure 36 The configuration of the server and client device according to the sixth embodiment is shown.

[0054] Figure 37 This is a block diagram of a three-dimensional data encoding device according to embodiment 7.

[0055] Figure 38 An example of the prediction residual in the seventh embodiment is shown.

[0056] Figure 39 An example of volume in embodiment 7 is shown.

[0057] Figure 40 An example of octree representation of volume in embodiment 7 is shown.

[0058] Figure 41 An example of a bit string of volume in embodiment 7 is shown.

[0059] Figure 42 An example of octree representation of volume in embodiment 7 is shown.

[0060] Figure 43 An example of volume in embodiment 7 is shown.

[0061] Figure 44 This is a diagram for explaining the intra-frame prediction processing of embodiment 7.

[0062] Figure 45 This is a diagram used to illustrate the rotation and translation processing of embodiment 7.

[0063] Figure 46 An example of the syntax of the RT application flag and RT information in the seventh embodiment is shown.

[0064] Figure 47 This is a diagram used to illustrate the inter-frame prediction processing of embodiment 7.

[0065] Figure 48 This is a block diagram of a three-dimensional data decoding device according to a seventh embodiment.

[0066] Figure 49 This is a flowchart of a three-dimensional data encoding process performed by the three-dimensional data encoding device according to the seventh embodiment.

[0067] Figure 50 This is a flowchart of a three-dimensional data decoding process performed by the three-dimensional data decoding device according to the seventh embodiment.

[0068] Figure 51 This is a diagram showing an example of three-dimensional points in the eighth embodiment.

[0069] Figure 52 This is a diagram showing an example of setting LoD in the eighth embodiment.

[0070] Figure 53 This is a diagram showing an example of a threshold value used in setting LoD in the eighth embodiment.

[0071] Figure 54 This is a diagram showing an example of attribute information used for prediction values ​​in the eighth embodiment.

[0072] Figure 55 This is a diagram showing an example of the Exponential Golomb code according to the eighth embodiment.

[0073] Figure 56 This is a diagram showing the processing of the Exponential Golomb code according to the eighth embodiment.

[0074] Figure 57 This is a diagram showing a syntax example of an attribute header according to the eighth embodiment.

[0075] Figure 58 This is a diagram showing a syntax example of attribute data in the eighth embodiment.

[0076] Figure 59 This is a flowchart of the three-dimensional data encoding process in the eighth embodiment.

[0077] Figure 60 This is a flowchart of the attribute information encoding process in the eighth embodiment.

[0078] Figure 61 This is a diagram showing the processing of the Exponential Golomb code according to the eighth embodiment.

[0079] Figure 62 This is a diagram showing an example of a reverse calculation table showing the relationship between the remaining codes and their values ​​according to the eighth embodiment.

[0080] Figure 63 This is a flowchart of three-dimensional data decoding processing in the eighth embodiment.

[0081] Figure 64 This is a flowchart of the attribute information decoding process in the eighth embodiment.

[0082] Figure 65 This is a block diagram of a three-dimensional data encoding device according to an eighth embodiment.

[0083] Figure 66 This is a block diagram of a three-dimensional data decoding device according to an eighth embodiment.

[0084] Figure 67 This is a flowchart of the three-dimensional data encoding process in the eighth embodiment.

[0085] Figure 68 This is a flowchart of three-dimensional data decoding processing in the eighth embodiment.

[0086] Figure 69 This is a diagram showing a first example of a table showing predicted values ​​calculated in each prediction mode according to the ninth embodiment.

[0087] Figure 70 This is a diagram showing an example of attribute information used for prediction values ​​in the ninth embodiment.

[0088] Figure 71This is a diagram showing a second example of a table showing predicted values ​​calculated in each prediction mode according to the ninth embodiment.

[0089] Figure 72 This is a diagram showing a third example of a table showing predicted values ​​calculated in each prediction mode according to the ninth embodiment.

[0090] Figure 73 This is a diagram showing a fourth example of a table showing predicted values ​​calculated in each prediction mode according to the ninth embodiment.

[0091] Figure 74 This is a diagram showing a fifth example of a table showing predicted values ​​calculated in each prediction mode according to the ninth embodiment.

[0092] Figure 75 This is a diagram showing a sixth example of a table showing predicted values ​​calculated in each prediction mode according to the ninth embodiment.

[0093] Figure 76 This is a diagram showing a seventh example of a table showing predicted values ​​calculated in each prediction mode according to the ninth embodiment.

[0094] Figure 77 This is a diagram showing an eighth example of a table showing predicted values ​​calculated in each prediction mode according to the ninth embodiment.

[0095] Figure 78 This is a diagram showing a first example of a binarization table when the prediction mode value in the ninth embodiment is binarized and encoded.

[0096] Figure 79 This is a diagram showing a second example of a binarization table when the prediction mode value in the ninth embodiment is binarized and encoded.

[0097] Figure 80 This is a diagram showing a third example of a binarization table when the prediction mode value in the ninth embodiment is binarized and encoded.

[0098] Figure 81 This is a diagram for explaining an example of encoding binary data in a binarization table when the prediction mode of the ninth embodiment is binarized and encoded.

[0099] Figure 82 This is a flowchart showing an example of encoding of a prediction mode value according to the ninth embodiment.

[0100] Figure 83 This is a flowchart showing an example of decoding of a prediction mode value according to the ninth embodiment.

[0101] Figure 84This is a diagram showing another example of a table showing predicted values ​​calculated in each prediction mode according to the ninth embodiment.

[0102] Figure 85 This is a diagram for explaining an example of encoding binary data in a binarization table when the prediction mode of the ninth embodiment is binarized and encoded.

[0103] Figure 86 This is a flowchart showing another example of encoding of prediction mode values ​​in embodiment 9.

[0104] Figure 87 This is a flowchart showing another example of decoding of prediction mode values ​​in the ninth embodiment.

[0105] Figure 88 This is a flowchart showing an example of processing for determining whether to fix the prediction mode value based on condition A during encoding in embodiment 9.

[0106] Figure 89 This is a flowchart showing an example of a process of determining whether to set the prediction mode value to a fixed value or to decode according to condition A during decoding in embodiment 9.

[0107] Figure 90 This is a diagram for explaining the maximum absolute difference value maxdiff in the ninth embodiment.

[0108] Figure 91 This is a diagram showing an example of the syntax of implementation mode 9.

[0109] Figure 92 This is a diagram showing an example of the syntax of implementation mode 9.

[0110] Figure 93 This is a flowchart of a three-dimensional data encoding process performed by the three-dimensional data encoding device according to the ninth embodiment.

[0111] Figure 94 This is a flowchart of attribute information encoding processing in the ninth embodiment.

[0112] Figure 95 This is a flowchart of the calculation process of the prediction value in the three-dimensional data encoding device of embodiment 9.

[0113] Figure 96 This is a flowchart of the prediction mode selection process in the ninth embodiment.

[0114] Figure 97 This is a flowchart of the process of selecting the prediction mode with the minimum cost in the ninth embodiment.

[0115] Figure 98 This is a flowchart of a three-dimensional data decoding process performed by the three-dimensional data decoding device according to the ninth embodiment.

[0116] Figure 99 This is a flowchart of the attribute information decoding process in the ninth embodiment.

[0117] Figure 100 This is a flowchart of a process for calculating a prediction value in the three-dimensional data decoding device according to the ninth embodiment.

[0118] Figure 101 This is a flowchart of the decoding process of the prediction mode in embodiment 9.

[0119] Figure 102 This is a block diagram showing the structure of an attribute information encoding unit included in the three-dimensional data encoding device according to the ninth embodiment.

[0120] Figure 103 This is a block diagram showing the configuration of an attribute information decoding unit included in the three-dimensional data decoding device according to the ninth embodiment.

[0121] Figure 104 This is a block diagram showing the structure of a three-dimensional data encoding device according to a ninth embodiment.

[0122] Figure 105 This is a block diagram showing the structure of a three-dimensional data decoding device according to a ninth embodiment.

[0123] Figure 106 This is a flowchart of the processing of the three-dimensional data encoding device according to the ninth embodiment.

[0124] Figure 107 This is a flowchart of the processing of the three-dimensional data decoding device according to the ninth embodiment.

[0125] Figure 108 This is a diagram showing an example of a table showing prediction values ​​calculated by the three-dimensional data encoding device in each prediction mode according to the tenth embodiment.

[0126] Figure 109 This is a diagram showing an example of a binarization table when the prediction mode value in embodiment 10 is binarized and encoded.

[0127] Figure 110 This is a diagram showing an example of a table showing prediction values ​​calculated by the three-dimensional data decoding device in each prediction mode according to the tenth embodiment.

[0128] Figure 111 This is a diagram showing an example of a table showing prediction values ​​calculated by the three-dimensional data encoding device in each prediction mode according to the tenth embodiment.

[0129] Figure 112 This is a diagram showing an example of a binarization table when the prediction mode value in embodiment 10 is binarized and encoded.

[0130] Figure 113 This is a diagram showing an example of a table showing prediction values ​​calculated by the three-dimensional data decoding device in each prediction mode according to the tenth embodiment.

[0131] Figure 114 An example of an initial table used in the process of assigning prediction values ​​to prediction modes in the tenth embodiment is shown.

[0132] Figure 115 This is a diagram showing an example of a table generated using the initial table of embodiment 10, which shows the prediction values ​​calculated by the three-dimensional data encoding device in each prediction mode.

[0133] Figure 116 This is a flowchart showing an example of a calculation process of a predicted value according to the tenth embodiment.

[0134] Figure 117 This is a flowchart of the prediction mode selection process in the tenth embodiment.

[0135] Figure 118 This is a flowchart of the process of selecting the prediction mode with the minimum cost in the tenth embodiment.

[0136] Figure 119 This is a flowchart of a three-dimensional data decoding process performed by the three-dimensional data decoding device according to the tenth embodiment.

[0137] Figure 120 This is a flowchart of the attribute information decoding process in the tenth embodiment.

[0138] Figure 121 This is a flowchart of the calculation process of the predicted value in the tenth embodiment.

[0139] Figure 122 This is a flowchart showing another example of the calculation process of the predicted value in the tenth embodiment.

[0140] Figure 123 This is a flowchart of the prediction mode selection process in the tenth embodiment.

[0141] Figure 124 This is a flowchart of the calculation process of the predicted value in the tenth embodiment.

[0142] Figure 125 This is a block diagram showing the structure of a three-dimensional data encoding device according to a tenth embodiment.

[0143] Figure 126 This is a block diagram showing the structure of a three-dimensional data decoding device according to a tenth embodiment.

[0144] Figure 127 This is a flowchart of the processing of the three-dimensional data encoding device according to the tenth embodiment.

[0145] Figure 128 This is a flowchart of the processing of the three-dimensional data decoding device according to the tenth embodiment. DETAILED DESCRIPTION

[0146] A three-dimensional data encoding method according to one embodiment of the present invention is a three-dimensional data encoding method for encoding multiple three-dimensional points. The method uses attribute information of one or more second three-dimensional points around a first three-dimensional point, selects one prediction mode from two or more prediction modes for calculating a prediction value of the attribute information of the first three-dimensional point, calculates the prediction value of the selected prediction mode, calculates the difference between the attribute information of the first three-dimensional point and the calculated prediction value, i.e., the prediction residual, and generates a bit stream including the prediction mode and the prediction residual. In the calculation of the prediction value, when the prediction value based on the attribute information of the second three-dimensional point cannot be assigned to the selected prediction mode, a prescribed fixed value is assigned as the prediction value of the prediction mode.

[0147] Thus, a predetermined fixed value is assigned as a prediction value to prediction modes that cannot be assigned more than one prediction value. Therefore, in the generated bitstream, a prediction residual is calculated based on the prediction value of the selected prediction mode, and includes the prediction mode and the prediction residual. Therefore, when the 3D data decoding device decodes the attribute information of the 3D point being processed from the obtained bitstream, similar to the 3D data encoding method, a predetermined fixed value is assigned as a prediction value to the prediction mode that cannot be assigned more than one prediction value. This allows the device to calculate a prediction value that is consistent with the prediction value calculated during encoding. Consequently, the 3D data decoding device can correctly decode the attribute information of the 3D point being processed.

[0148] In addition, it may also be the case that, in the calculation of the prediction value, the prediction value based on the attribute information of the second three-dimensional point cannot be assigned to the selected prediction mode because the number of the second three-dimensional points is less than the specified number and the prediction value cannot be assigned to the prediction mode.

[0149] Furthermore, the predetermined fixed value may be an initial value.

[0150] Alternatively, the predetermined fixed value may be 0.

[0151] In addition, it may also be that, in the calculation of the predicted value, in the first prediction mode among the two or more prediction modes, the average of the attribute information of the one or more second three-dimensional points is calculated as the predicted value, and in the second prediction mode among the two or more prediction modes, the attribute information of the second three-dimensional point is calculated as the predicted value.

[0152] In addition, it may also be that the two or more prediction modes are respectively represented by prediction mode values ​​of different values, and the prediction mode value of the prediction mode indicating that the average is assigned as the prediction value is smaller than the prediction mode value of the prediction mode indicating that the attribute information of the one or more second three-dimensional points is assigned as the prediction value.

[0153] In addition, it may also be that the two or more prediction modes are respectively represented by prediction mode values ​​of different values, and the prediction mode value of the prediction mode indicating that the attribute information of a second three-dimensional point is assigned as a prediction value is smaller than the prediction mode value of the prediction mode indicating that the attribute information of other second three-dimensional points located at a distance from the first three-dimensional point than the second three-dimensional point is assigned as a prediction value.

[0154] A three-dimensional data decoding method according to one embodiment of the present invention is a three-dimensional data decoding method for decoding a plurality of three-dimensional points, wherein a prediction mode and a prediction residual of a first three-dimensional point among the plurality of three-dimensional points are obtained by obtaining a bit stream, a prediction value of the obtained prediction mode is calculated, and the prediction value is added to the prediction residual to calculate the attribute information of the first three-dimensional point. In the calculation of the prediction value, when a prediction value based on the attribute information of a second three-dimensional point around the first three-dimensional point cannot be assigned to the obtained prediction mode, a prescribed fixed value is assigned as the prediction value of the prediction mode.

[0155] Thus, when decoding the attribute information of the 3D point being processed from the obtained bitstream, a predetermined fixed value is assigned as a prediction value to a prediction mode that cannot be assigned more than one prediction value, similar to the 3D data encoding method. This allows the calculation of a prediction value that matches the prediction value calculated during encoding. Consequently, the attribute information of the 3D point being processed can be correctly decoded.

[0156] In addition, it may also be the case that, in the calculation of the prediction value, the prediction value based on the attribute information of the second three-dimensional point cannot be assigned to the selected prediction mode because the number of the second three-dimensional points is less than the specified number and the prediction value cannot be assigned to the prediction mode.

[0157] Furthermore, the predetermined fixed value may be an initial value.

[0158] Alternatively, the predetermined fixed value may be 0.

[0159] In addition, it may also be that, in the calculation of the predicted value, in the first prediction mode among the two or more prediction modes, the average of the attribute information of the second three-dimensional point is calculated as the predicted value, and in the second prediction mode among the two or more prediction modes, the attribute information of the second three-dimensional point is calculated as the predicted value.

[0160] In addition, it may also be that the two or more prediction modes are respectively represented by prediction mode values ​​of different values, and the prediction mode value of the prediction mode indicating that the average is assigned as the prediction value is smaller than the prediction mode value of the prediction mode indicating that the attribute information of the one or more second three-dimensional points is assigned as the prediction value.

[0161] In addition, it may also be that more than two prediction modes are represented by prediction mode values ​​of different values, respectively, and the prediction mode value of the prediction mode indicating that the attribute information of a second three-dimensional point is assigned as a prediction value is smaller than the prediction mode value of the prediction mode indicating that the attribute information of other second three-dimensional points located at a distance from the first three-dimensional point than the second three-dimensional point is assigned as a prediction value.

[0162] In addition, a three-dimensional data encoding device of one embodiment of the present invention is a three-dimensional data encoding device that encodes multiple three-dimensional points, and is equipped with a processor and a memory. The processor uses the memory, uses the attribute information of one or more second three-dimensional points around the first three-dimensional point, selects one prediction mode from two or more prediction modes for calculating the prediction value of the attribute information of the first three-dimensional point, calculates the prediction value of the selected prediction mode, calculates the difference between the attribute information of the first three-dimensional point and the calculated prediction value, that is, the prediction residual, and generates a bit stream including the prediction mode and the prediction residual. In the calculation of the prediction value, when the prediction value based on the attribute information of the second three-dimensional point cannot be assigned to the selected prediction mode, a prescribed fixed value is assigned as the prediction value of the prediction mode.

[0163] Thus, a predetermined fixed value is assigned as a prediction value to prediction modes that cannot be assigned more than one prediction value. Therefore, in the generated bitstream, a prediction residual is calculated based on the prediction value of the selected prediction mode, and includes the prediction mode and the prediction residual. Therefore, when the 3D data decoding device decodes the attribute information of the 3D point being processed from the obtained bitstream, similar to the 3D data encoding method, a predetermined fixed value is assigned as a prediction value to the prediction mode that cannot be assigned more than one prediction value. This allows the device to calculate a prediction value that is consistent with the prediction value calculated during encoding. Consequently, the 3D data decoding device can correctly decode the attribute information of the 3D point being processed.

[0164] In addition, a three-dimensional data decoding device of one embodiment of the present invention is a three-dimensional data decoding device that decodes multiple three-dimensional points, and is equipped with a processor and a memory. The processor uses the memory to obtain a bit stream to obtain a prediction mode and a prediction residual of a first three-dimensional point among the multiple three-dimensional points, calculates a prediction value of the obtained prediction mode, adds the prediction value and the prediction residual, and thereby calculates the attribute information of the first three-dimensional point. In the calculation of the prediction value, when the obtained prediction mode cannot be assigned a prediction value based on the attribute information of the second three-dimensional point around the first three-dimensional point, a prescribed fixed value is assigned as the prediction value of the prediction mode.

[0165] Thus, when decoding the attribute information of the 3D point being processed from the obtained bitstream, a predetermined fixed value is assigned as a prediction value to a prediction mode that cannot be assigned more than one prediction value, similar to the 3D data encoding method. This allows the calculation of a prediction value that is consistent with the prediction value calculated during encoding. Consequently, the attribute information of the 3D point being processed can be accurately decoded.

[0166] In addition, these general or specific forms can be implemented by systems, methods, integrated circuits, computer programs or computer-readable recording media such as CD-ROMs, and can be implemented by any combination of systems, methods, integrated circuits, computer programs and recording media.

[0167] The following detailed description of the embodiments is given with reference to the accompanying drawings. In addition, the embodiments to be described below are all specific examples of the present disclosure. The numerical values, shapes, materials, components, configuration positions of components, connection forms, steps, order of steps, etc. shown in the following embodiments are all examples, and their purpose is not to limit the present disclosure. Moreover, among the components of the following embodiments, the components that are not recorded in the technical solution showing the highest concept are described as arbitrary components.

[0168] (Implementation 1)

[0169] First, the data structure of encoded three-dimensional data (hereinafter also referred to as encoded data) according to this embodiment will be described. Figure 1 The structure of the encoded three-dimensional data according to this embodiment is shown.

[0170] In this embodiment, the three-dimensional space is divided into spaces (SPC) equivalent to the spaces of pictures in the encoding of dynamic images, and the three-dimensional data is encoded in units of space. The space is further divided into volumes (VLM) equivalent to macroblocks in dynamic image encoding, and prediction and conversion are performed in units of VLM. The volume includes a plurality of voxels (VXL), which are the smallest units corresponding to position coordinates. In addition, prediction means generating predicted three-dimensional data similar to the processing unit of the processing object with reference to other processing units, and encoding the difference between the predicted three-dimensional data and the processing unit of the processing object, similar to the prediction performed in the two-dimensional image. Furthermore, the prediction includes not only spatial prediction with reference to other prediction units at the same time, but also temporal prediction with reference to prediction units at different times.

[0171] For example, when encoding a three-dimensional space represented by point cloud data, such as a point cloud, a three-dimensional data encoding device (hereinafter referred to as an encoding device) encodes each point in the point cloud or multiple points contained within a voxel, all at once, according to the size of the voxel. Subdividing the voxels allows for a highly accurate representation of the point cloud's three-dimensional shape, while increasing the voxel size allows for a coarser representation of the point cloud's three-dimensional shape.

[0172] In addition, although the following description uses the case where the three-dimensional data is a point cloud as an example, the three-dimensional data is not limited to point clouds and can also be three-dimensional data in any form.

[0173] Furthermore, voxels with a hierarchical structure can be utilized. In this case, within the nth level, it is possible to sequentially indicate whether a sampling point exists in the n-1th level or lower levels (the levels below the nth level). For example, when decoding only the nth level, if a sampling point exists in the n-1th level or lower levels, decoding can be performed as if the sampling point exists at the center of the voxel in the nth level.

[0174] Furthermore, the encoding device obtains point group data through a distance sensor, a stereo camera, a monocular camera, a gyroscope, or an inertial sensor.

[0175] Similar to video encoding, spaces are classified into at least one of the following three prediction structures: independently decodable intra-frame space (I-SPC), unidirectionally referenced prediction space (P-SPC), and bidirectionally referenced prediction space (B-SPC). Spaces also contain two types of time information: decoding time and display time.

[0176] And, as Figure 1As shown, as a processing unit including a plurality of spaces, there is a GOS (Group of Space) which is a random access unit. Furthermore, as a processing unit including a plurality of GOS, there is a world space (WLD).

[0177] The spatial area occupied by world space is associated with an absolute position on Earth using GPS or latitude and longitude information. This location information is stored as metadata. Metadata can be included in the encoded data or transmitted separately.

[0178] Furthermore, within the GOS, all SPCs may be three-dimensionally adjacent, or there may be SPCs that are not three-dimensionally adjacent to other SPCs.

[0179] In addition, below, the encoding, decoding, or referencing of three-dimensional data contained in a processing unit such as a GOS, SPC, or VLM will also be referred to simply as encoding, decoding, or referencing the processing unit. Furthermore, the three-dimensional data contained in the processing unit includes, for example, at least one pair of spatial positions such as three-dimensional coordinates and characteristic values ​​such as color information.

[0180] Next, the prediction structure of the SPC in the GOS will be described. Although multiple SPCs in the same GOS or multiple VLMs in the same SPC occupy different spaces, they have the same time information (decoding time and display time).

[0181] Furthermore, within a GOS, the first SPC in decoding order is the I-SPC. Furthermore, there are two types of GOS: closed GOS and open GOS. A closed GOS is one that can decode all SPCs within the GOS when decoding starts from the first I-SPC. In an open GOS, some SPCs within the GOS that are earlier than the display time of the first I-SPC refer to a different GOS and can only be decoded in that GOS.

[0182] In addition, in coded data such as map information, the WLD may be decoded in the reverse order of the coding order. If there is a dependency between GOS, it will be difficult to reproduce the data in the reverse order. Therefore, in this case, a closed GOS is basically used.

[0183] Furthermore, the GOS has a layer structure in the height direction, and encoding or decoding is performed sequentially starting from the SPC of the bottom layer.

[0184] Figure 2 An example of the prediction structure between SPCs belonging to the lowest layer of the GOS is shown. Figure 3 An example of an inter-layer prediction structure is shown.

[0185] There are one or more I-SPCs within a GOS. While objects such as people, animals, cars, bicycles, traffic lights, and buildings serving as land landmarks exist within a three-dimensional space, encoding small objects as I-SPCs is particularly effective. For example, a three-dimensional data decoding device (hereinafter also referred to as a decoding device) decodes only the I-SPCs within the GOS when decoding the GOS with low throughput or high speed.

[0186] Furthermore, the encoding device may switch the encoding interval or the appearance frequency of the I-SPC according to the density of objects within the WLD.

[0187] And, in Figure 3 In the configuration shown, the encoding device or decoding device encodes or decodes multiple layers sequentially starting from the lower layer (layer 1). This allows, for example, autonomous vehicles to prioritize data near the ground, which contains a large amount of information.

[0188] In addition, in the coded data used by drones, etc., coding or decoding can be performed sequentially starting from the SPC of the upper layer in the height direction within the GOS.

[0189] Furthermore, the encoding device or decoding device may encode or decode multiple layers in such a manner that the decoding device can roughly grasp the GOS and gradually increase the resolution. For example, the encoding device or decoding device may encode or decode layers in the order of 3, 8, 1, 9, etc.

[0190] Next, the corresponding method of static objects and dynamic objects is explained.

[0191] In three-dimensional space, there are static objects or scenes such as buildings and roads (hereinafter referred to as static objects), as well as dynamic objects such as vehicles and people (hereinafter referred to as dynamic objects). Object detection can also be performed by extracting feature points from point cloud data or images captured by stereo cameras. Here, an example of a method for encoding dynamic objects is described.

[0192] The first method is a method of encoding without distinguishing between static objects and dynamic objects. The second method is a method of distinguishing between static objects and dynamic objects using identification information.

[0193] For example, GOS is used as the identification unit. In this case, GOS including SPCs constituting static objects and GOS including SPCs constituting dynamic objects are distinguished within the coded data or by identification information stored separately from the coded data.

[0194] Alternatively, SPC is used as the identification unit. In this case, the SPC including only the VLM constituting the static object and the SPC including the VLM constituting the dynamic object are distinguished by the above-mentioned identification information.

[0195] Alternatively, VLM or VXL may be used as the identification unit. In this case, a VLM or VXL including a static object and a VLM or VXL including a dynamic object are distinguished by the above-mentioned identification information.

[0196] Furthermore, the encoding device may encode dynamic objects as one or more VLMs or SPCs, and may encode the VLMs or SPCs containing static objects and the SPCs containing dynamic objects as separate GOSs. Furthermore, if the size of the GOS is variable depending on the size of the dynamic object, the encoding device may store the size of the GOS separately as meta-information.

[0197] Furthermore, the encoding device encodes static objects and dynamic objects independently, allowing dynamic objects to be superimposed on the world space composed of static objects. In this case, a dynamic object is composed of one or more SPCs, each of which corresponds to one or more SPCs that constitute the static object with which it is superimposed. Furthermore, dynamic objects can be represented not by SPCs but by one or more VLMs or VXLs.

[0198] Furthermore, the encoding device may encode static objects and dynamic objects as different streams.

[0199] Furthermore, the encoding device may generate a GOS that includes one or more SPCs constituting a dynamic object. Furthermore, the encoding device may set the GOS (GOS_M) including the dynamic object and the GOS of the static object corresponding to the spatial region of the GOS_M to be of the same size (occupying the same spatial region). This enables overlapping processing on a GOS basis.

[0200] The P-SPC or B-SPC that constitutes a dynamic object can also refer to the SPC contained in a different coded GOS. When the position of a dynamic object changes over time and the same dynamic object is coded as a GOS at different times, cross-GOS reference is effective from the perspective of compression rate.

[0201] Furthermore, the first and second methods can be switched depending on the intended use of the encoded data. For example, when the encoded three-dimensional data is used as a map, separation from dynamic objects is desirable, so the encoding device uses the second method. Alternatively, when encoding three-dimensional data of events such as concerts or sports, where separation of dynamic objects is not necessary, the encoding device uses the first method.

[0202] Furthermore, the decoding time and display time of the GOS or SPC can be stored in the encoded data or as meta-information. Furthermore, the time information of static objects can all be the same. In this case, the actual decoding time and display time can be determined by the decoding device. Alternatively, different values ​​can be assigned to each GOS or SPC as the decoding time, and the same value can be assigned to all as the display time. Moreover, as shown in the decoder mode in dynamic image encoding such as HEVC's HRD (Hypothetical Reference Decoder), the decoder has a buffer of a specified size. As long as the bit stream is read at a specified bit rate according to the decoding time, a model that will not be destroyed and is guaranteed to be decodable can be imported.

[0203] Next, the configuration of GOS in the world space is explained. The coordinates of the three-dimensional space in the world space are represented by three mutually orthogonal coordinate axes (x-axis, y-axis, z-axis). By setting a prescribed rule in the encoding order of GOS, spatially adjacent GOS can be encoded continuously in the encoded data. For example, Figure 4 In the example shown, the GOS within the xz plane are encoded continuously. After encoding all GOS within a single xz plane, the y-axis value is updated. In other words, as encoding continues, world space extends in the y-axis direction. Furthermore, the GOS index numbers are set to the encoding order.

[0204] Here, the three-dimensional world space corresponds one-to-one to absolute geographic coordinates such as GPS or latitude and longitude. Alternatively, the three-dimensional space can be represented by relative positions relative to a predetermined reference position. The directions of the x-axis, y-axis, and z-axis of the three-dimensional space are represented as direction vectors determined based on latitude and longitude, and these direction vectors are stored as metadata along with the encoded data.

[0205] Furthermore, the size of the GOS is set to be fixed, and the encoding device stores this size as meta-information. Furthermore, the size of the GOS can be switched, for example, depending on whether the location is urban, indoors, or outdoors. In other words, the size of the GOS can be switched based on the quantity or nature of objects with informational value. Alternatively, within the same world space, the encoding device can appropriately switch the size of the GOS or the spacing of I-SPCs within the GOS based on, for example, the density of objects. For example, the encoding device can set the GOS size to be smaller and the spacing of I-SPCs within the GOS to be shorter when the density of objects is higher.

[0206] exist Figure 5In the example, in the area from the 3rd to the 10th GOS, the density of objects is high, so the GOS is subdivided to achieve fine-grained random access. In addition, the 7th to the 10th GOS exist on the back of the 3rd to the 6th GOS, respectively.

[0207] Next, the configuration and operation flow of the three-dimensional data encoding device according to this embodiment will be described. Figure 6 This is a block diagram of the three-dimensional data encoding device 100 according to this embodiment. Figure 7 3D data encoding apparatus 100 is a flowchart showing an example of its operation.

[0208] Figure 6 The three-dimensional data encoding device 100 shown generates encoded three-dimensional data 112 by encoding three-dimensional data 111. The three-dimensional data encoding device 100 includes an acquisition unit 101, an encoding region determination unit 102, a division unit 103, and an encoding unit 104.

[0209] like Figure 7 As shown, first, the acquisition unit 101 acquires three-dimensional data 111 as point cloud data ( S101 ).

[0210] Next, the coding region determination unit 102 determines a coding target region from the spatial region corresponding to the obtained point cloud data (S102). For example, the coding region determination unit 102 determines a spatial region around the position of the user or vehicle as the coding target region.

[0211] Next, the segmentation unit 103 segments the point cloud data contained in the encoding target region into processing units. Here, the processing units are the aforementioned GOS and SPCs, for example. Furthermore, the encoding target region corresponds to the aforementioned world space, for example. Specifically, the segmentation unit 103 segments the point cloud data into processing units based on the size of the pre-set GOS and the presence or size of dynamic objects (S103). Furthermore, the segmentation unit 103 determines the starting position of the SPC, which is the first in the encoding order, within each GOS.

[0212] Next, the encoding unit 104 generates the encoded three-dimensional data 112 by sequentially encoding the plurality of SPCs in each GOS ( S104 ).

[0213] In addition, although an example is shown here in which each GOS is encoded after the encoding target area is divided into GOS and SPC, the processing order is not limited to the above. For example, the order may be such that the composition of a GOS is determined, the GOS is encoded, and then the composition of the GOS is determined.

[0214] In this manner, the 3D data encoding device 100 generates encoded 3D data 112 by encoding the 3D data 111. Specifically, the 3D data encoding device 100 divides the 3D data into random access units, namely, first processing units (GOS) corresponding to respective 3D coordinates. The first processing units (GOS) are divided into a plurality of second processing units (SPCs), and the second processing units (SPCs) are divided into a plurality of third processing units (VLMs). Furthermore, each third processing unit (VLM) includes one or more voxels (VXLs), which are the smallest units corresponding to positional information.

[0215] Next, the 3D data encoding device 100 encodes each of the plurality of first processing units (GOS) to generate the encoded 3D data 112. Specifically, the 3D data encoding device 100 encodes each of the plurality of second processing units (SPC) in each first processing unit (GOS). Furthermore, the 3D data encoding device 100 encodes each of the plurality of third processing units (VLM) in each second processing unit (SPC).

[0216] For example, when the first processing unit (GOS) to be processed is a closed GOS, the three-dimensional data encoding device 100 encodes the second processing unit (SPC) to be processed included in the first processing unit (GOS) to be processed by referring to other second processing units (SPC) included in the first processing unit (GOS) to be processed. In other words, the three-dimensional data encoding device 100 does not refer to the second processing unit (SPC) included in the first processing unit (GOS) different from the first processing unit (GOS) to be processed.

[0217] Furthermore, when the first processing unit (GOS) of the processing object is an open GOS, the second processing unit (SPC) of the processing object included in the first processing unit (GOS) of the processing object is encoded with reference to other second processing units (SPC) included in the first processing unit (GOS) of the processing object, or a second processing unit (SPC) included in a first processing unit (GOS) different from the first processing unit (GOS) of the processing object.

[0218] Furthermore, the three-dimensional data encoding device 100 selects one as the type of the second processing unit (SPC) of the processing object from a first type (I-SPC) that does not refer to other second processing units (SPCs), a second type (P-SPC) that refers to one other second processing unit (SPC), and a third type that refers to two other second processing units (SPCs), and encodes the second processing unit (SPC) of the processing object according to the selected type.

[0219] Next, the configuration and operation flow of the three-dimensional data decoding device according to this embodiment will be described. Figure 8 This is a block diagram of a three-dimensional data decoding device 200 according to this embodiment. Figure 9 3D data decoding apparatus 200 is a flowchart showing an example of its operation.

[0220] Figure 8 The illustrated three-dimensional data decoding device 200 generates decoded three-dimensional data 212 by decoding encoded three-dimensional data 211. Here, the encoded three-dimensional data 211 is, for example, the encoded three-dimensional data 112 generated by the three-dimensional data encoding device 100. The three-dimensional data decoding device 200 includes an acquisition unit 201, a decoding start GOS determination unit 202, a decoding SPC determination unit 203, and a decoding unit 204.

[0221] First, the acquisition unit 201 acquires the encoded 3D data 211 (S201). Next, the decoding start GOS determination unit 202 determines the GOS to be decoded (S202). Specifically, the decoding start GOS determination unit 202 refers to metadata stored within or separately from the encoded 3D data 211 and determines the GOS that includes the SPC corresponding to the spatial position, object, or time at which decoding should start as the decoding start GOS.

[0222] Next, the decoded SPC determination unit 203 determines the type (I, P, B) of the SPC to be decoded within the GOS (S203). For example, the decoded SPC determination unit 203 determines (1) whether to decode only the I-SPC, (2) whether to decode both the I-SPC and the P-SPC, or (3) whether to decode all types. Alternatively, this step may be omitted if the type of SPC to be decoded is predetermined, such as when all SPCs are to be decoded.

[0223] Next, the decoding unit 204 obtains the first SPC in the decoding order (the same as the encoding order) within the GOS, and the address position at the beginning of the encoded 3D data 211. From this address position, it obtains the encoded data of the first SPC and decodes each SPC sequentially from the first SPC (S204). The address position is stored in meta-information, etc.

[0224] In this manner, the 3D data decoding device 200 decodes the decoded 3D data 212. Specifically, the 3D data decoding device 200 decodes each of the encoded 3D data 211 of the first processing unit (GOS) corresponding to the three-dimensional coordinates, thereby generating the decoded 3D data 212 of the first processing unit (GOS), which is a random access unit. More specifically, the 3D data decoding device 200 decodes each of the plurality of second processing units (SPC) for each first processing unit (GOS). Furthermore, the 3D data decoding device 200 decodes each of the plurality of third processing units (VLM) for each second processing unit (SPC).

[0225] The following describes the meta-information for random access. This meta-information is generated by the three-dimensional data encoding device 100 and included in the encoded three-dimensional data 112 (211).

[0226] In conventional random access of two-dimensional moving images, decoding starts from the first frame of the random access unit near a specified time. However, in world space, random access to (coordinates, objects, etc.) is also possible in addition to time.

[0227] Therefore, in order to realize random access to at least the three elements of coordinates, objects, and time, a table is prepared that associates each element with the index number of the GOS. Furthermore, the index number of the GOS is associated with the address of the I-SPC at the beginning of the GOS. Figure 10 An example of a table included in the meta information is shown. Figure 10 Of all the tables shown, at least one table may be used.

[0228] The following describes random access using coordinates as the starting point, as an example. When accessing the coordinates (x2, y2, z2), the coordinate-GOS table is first referenced, and it is determined that the location with coordinates (x2, y2, z2) is included in the second GOS. Next, the GOS address table is referenced, and since the address of the first I-SPC in the second GOS is known to be addr(2), the decoding unit 204 obtains data from this address and begins decoding.

[0229] In addition, the address can be an address in a logical format or a physical address of an HDD or memory. Furthermore, information identifying a file segment can be used instead of an address. For example, a file segment is a unit obtained by segmenting one or more GOSs.

[0230] Furthermore, if an object spans multiple GOSs, the object GOS table may also indicate the GOSs to which the multiple objects belong. If the multiple GOSs are closed, the encoding and decoding devices can perform encoding and decoding in parallel. Furthermore, if the multiple GOSs are open, the GOSs can reference each other, further improving compression efficiency.

[0231] Examples of objects include people, animals, cars, bicycles, traffic lights, and buildings that serve as landmarks on land. For example, when encoding in world space, the three-dimensional data encoding device 100 extracts feature points unique to an object from a three-dimensional point cloud, detects the object based on the feature points, and can set the detected object as a random access point.

[0232] In this manner, the three-dimensional data encoding device 100 generates first information indicating a plurality of first processing units (GOS) and the three-dimensional coordinates corresponding to each of the plurality of first processing units (GOS). The encoded three-dimensional data 112 (211) includes the first information. Furthermore, the first information further indicates at least one of an object, a time, and a data storage destination corresponding to each of the plurality of first processing units (GOS).

[0233] The three-dimensional data decoding device 200 obtains first information from the encoded three-dimensional data 211 , uses the first information to identify the first processing unit of encoded three-dimensional data 211 corresponding to the specified three-dimensional coordinates, object or time, and decodes the encoded three-dimensional data 211 .

[0234] Other examples of meta-information are described below. In addition to the meta-information for random access, the 3D data encoding device 100 can also generate and store the following meta-information. The 3D data decoding device 200 can also use this meta-information during decoding.

[0235] When using 3D data as map information, profiles are defined based on the intended use, and information indicating the profiles can be included in the meta-information. For example, profiles for urban areas or suburban areas, or for flying objects, can be defined, with the maximum and minimum sizes of the world space, SPC, or VLM defined for each. For example, in an urban area profile, more detailed information is required than in a suburban area, so the minimum size of the VLM is set smaller.

[0236] Meta-information may also include a tag value indicating the type of object. This tag value corresponds to the VLM, SPC, or GOS that constitutes the object. Tag values ​​can be set based on the object type, for example, a tag value of "0" representing a "person," a tag value of "1" representing a "car," and a tag value of "2" representing a "traffic light." Alternatively, when the object type is difficult or unnecessary to determine, a tag value indicating properties such as size or whether the object is dynamic or static can be used.

[0237] Furthermore, the meta-information may include information indicating the range of the spatial region occupied by the world space.

[0238] Furthermore, the meta-information may be used as header information common to the entire stream of coded data or a plurality of SPCs such as an SPC within a GOS, and may store the size of the SPC or VXL.

[0239] Furthermore, the meta-information may include identification information of a distance sensor, a camera, or the like used in generating the point cloud, or information indicating the positional accuracy of a point group within the point cloud.

[0240] Also, the meta information may include information showing whether the world space is composed of only static objects or contains dynamic objects.

[0241] Modifications of this embodiment will be described below.

[0242] The encoding device or decoding device can encode or decode two or more different SPCs or GOSs in parallel. The GOSs to be encoded or decoded in parallel can be determined based on meta-information indicating the spatial position of the GOSs.

[0243] When three-dimensional data is used as a spatial map for a moving vehicle or flying object, or when such a spatial map is generated, the encoding device or decoding device can encode or decode the GOS or SPC contained in a space determined based on GPS, path information, or zoom factor, etc.

[0244] Furthermore, the decoding device may also decode sequentially starting from spaces closest to its own position or path. The encoding device or decoding device may also prioritize spaces farther from its own position or path, lowering the priority of spaces closer to it, and perform encoding or decoding accordingly. Here, lowering the priority means reducing the processing order, reducing the resolution (post-filtering processing), or reducing image quality (to improve coding efficiency, such as increasing the quantization step size).

[0245] Furthermore, when decoding coded data that has been spatially hierarchically coded, the decoding device may decode only the lower layer.

[0246] Furthermore, the decoding device may start decoding from a lower layer according to the zoom ratio or purpose of the map.

[0247] Furthermore, in applications such as estimating the self-position of a car or robot during autonomous driving or identifying objects, the encoding device or decoding device may also reduce the resolution of areas outside the area within a specified height from the road surface (the area for identification) for encoding or decoding.

[0248] Furthermore, the encoding device may independently encode the point clouds representing indoor and outdoor spatial shapes. For example, by separating the GOS representing indoor spaces (indoor GOS) from the GOS representing outdoor spaces (outdoor GOS), the decoding device can select the GOS to decode based on the viewpoint position when using the encoded data.

[0249] Furthermore, the encoding device can encode indoor and outdoor GOSs that are close to each other in the coded stream. For example, the encoding device can associate identifiers between the two and store information indicating the corresponding identifiers within the coded stream or in separately stored meta-information. This allows the decoding device to identify the indoor and outdoor GOSs that are close to each other by referring to the information in the meta-information.

[0250] Furthermore, the encoding device may switch the size of the GOS or SPC between the indoor GOS and the outdoor GOS. For example, the encoding device may set the size of the GOS to be smaller indoors than outdoors. Furthermore, the encoding device may change the accuracy of feature point extraction from the point cloud or the accuracy of object detection between the indoor GOS and the outdoor GOS.

[0251] Furthermore, the encoding device can add information to the encoded data that allows the decoding device to distinguish between dynamic and static objects. This allows the decoding device to display dynamic objects in combination with a red frame or explanatory text. Alternatively, the decoding device can replace dynamic objects with a red frame or explanatory text alone. Furthermore, the decoding device can display more detailed object categories. For example, a car can be displayed with a red frame, while a person can be displayed with a yellow frame.

[0252] Furthermore, the encoding or decoding device may determine whether to encode or decode dynamic objects and static objects as different SPCs or GOSs based on the frequency of occurrence of dynamic objects or the ratio of static objects to dynamic objects. For example, if the frequency or ratio of dynamic objects exceeds a threshold, an SPC or GOS containing a mixture of dynamic and static objects is permitted. If the frequency or ratio of dynamic objects does not exceed the threshold, an SPC or GOS containing a mixture of dynamic and static objects is not permitted.

[0253] When dynamic objects are detected from 2D camera image information rather than point clouds, the encoding device can obtain information identifying the detection result (box or text, etc.) and the object position separately, encoding this information as part of the 3D encoded data. In this case, the decoding device displays auxiliary information representing the dynamic object (box or text) overlaid on the decoded result of the static object.

[0254] Furthermore, the encoding device may change the density of VXL or VLM based on factors such as the complexity of the shape of the static object. For example, the encoding device may set VXL or VLM to be denser when the shape of the static object is more complex. Furthermore, the encoding device may determine the quantization step size when quantizing spatial position or color information based on the density of VXL or VLM. For example, the encoding device may set the quantization step size to be smaller when the VXL or VLM is denser.

[0255] As described above, the encoding device or decoding device according to the present embodiment performs spatial encoding or decoding in spatial units having coordinate information.

[0256] Furthermore, the encoding device and the decoding device perform encoding or decoding in units of volume in space. The volume includes voxels, which are the minimum units corresponding to the position information.

[0257] Furthermore, the encoding device and decoding device perform encoding or decoding by creating a table that associates various elements of spatial information, including coordinates, objects, and time, with GOPs, or by creating a table that associates various elements, thereby establishing a correspondence between arbitrary elements. Furthermore, the decoding device uses the values ​​of the selected elements to determine coordinates, identifies a volume, voxel, or space based on the coordinates, and decodes the space including the volume or voxel, or the identified space.

[0258] Furthermore, the encoding device determines a volume, voxel, or space that can be selected by an element through feature point extraction or object recognition, and encodes it as a volume, voxel, or space that can be randomly accessed.

[0259] Spaces are divided into three types: I-SPC, which can be encoded or decoded by the space itself; P-SPC, which can be encoded or decoded with reference to any one processed space; and B-SPC, which can be encoded or decoded with reference to any two processed spaces.

[0260] One or more volumes correspond to static objects or dynamic objects. The space containing static objects and the space containing dynamic objects are encoded or decoded as different GOSs. That is, the SPC containing static objects and the SPC containing dynamic objects are assigned to different GOSs.

[0261] Dynamic objects are encoded or decoded for each object and correspond to one or more spaces containing only static objects. In other words, multiple dynamic objects are encoded separately, and the resulting encoded data of multiple dynamic objects corresponds to an SPC containing only static objects.

[0262] The encoding and decoding devices increase the priority of the I-SPC within the GOS to perform encoding or decoding. For example, the encoding device performs encoding to minimize degradation of the I-SPC (enabling more faithful reproduction of the original 3D data after decoding). Furthermore, the decoding device, for example, only decodes the I-SPC.

[0263] The encoding device can change the frequency of using I-SPCs to perform encoding according to the density or number of objects in world space. Specifically, the encoding device changes the frequency of selecting I-SPCs according to the number or density of objects contained in the three-dimensional data. For example, the encoding device may increase the frequency of using I-space as the density of objects in world space increases.

[0264] Furthermore, the encoding device sets a random access point in units of GOS, and stores information indicating a spatial region corresponding to the GOS in the header information.

[0265] The encoding device, for example, uses a default value as the spatial size of the GOS. Alternatively, the encoding device may change the size of the GOS based on the number (quantity) or density of objects or dynamic objects. For example, the encoding device may set the spatial size of the GOS to be smaller when the objects or dynamic objects are denser or more numerous.

[0266] Furthermore, the space or volume includes a cluster of feature points derived from information obtained by sensors such as depth sensors, gyroscopes, or cameras. The coordinates of the feature points are set to the center of the voxel. Furthermore, by subdividing the voxels, it is possible to achieve higher accuracy in positional information.

[0267] The feature point cluster is derived using multiple pictures. The multiple pictures have at least two types of time information: actual time information and spatially corresponding time information common to multiple pictures (for example, encoding time used for rate control, etc.).

[0268] Furthermore, encoding or decoding is performed in units of GOS including one or more spaces.

[0269] The encoding device and the decoding device refer to the space in the processed GOS to predict the P space or B space in the GOS to be processed.

[0270] Alternatively, the encoding device and the decoding device predict the P space or B space in the GOS to be processed by using the processed space in the GOS to be processed without referring to different GOSs.

[0271] Furthermore, the encoding device and the decoding device transmit or receive the encoded stream in units of a world space including one or more GOSs.

[0272] Furthermore, the GOS has a layer structure in at least one direction within world space, and the encoding and decoding devices perform encoding or decoding starting from the lowest layer. For example, the randomly accessible GOS belongs to the lowest layer. The GOS belonging to a higher layer only references the GOS belonging to the layers below it. In other words, the GOS is spatially divided in a predetermined direction and includes multiple layers, each having one or more SPCs. The encoding and decoding devices perform encoding or decoding for each SPC by referencing SPCs contained in layers in the same layer or layers below it.

[0273] Furthermore, the encoding device and decoding device continuously encode or decode the GOS within a world space unit that includes multiple GOS. The encoding device and decoding device write or read information indicating the order (direction) of encoding or decoding as metadata. In other words, the encoded data includes information indicating the order in which the multiple GOS were encoded.

[0274] Furthermore, the encoding device and the decoding device encode or decode two or more different spaces or GOSs in parallel.

[0275] Furthermore, the encoding device and the decoding device encode or decode the space information (coordinates, size, etc.) of the space or the GOS.

[0276] Furthermore, the encoding device and the decoding device encode or decode the space or GOS included in the specific space specified based on external information related to the own position and / or area size, such as GPS, route information, or magnification.

[0277] The encoding device or decoding device performs encoding or decoding by giving a lower priority to a space far from the own position than to a space close to the own position.

[0278] The encoding device sets a direction in the world space according to a magnification or application, and encodes a GOS having a layer structure in that direction. Furthermore, the decoding device decodes the GOS having a layer structure in the world space direction set according to the magnification or application, preferentially starting from the lower layer.

[0279] The encoding device varies the accuracy of feature point extraction and object recognition, the size of the spatial area, etc. in indoor and outdoor spaces. However, the encoding device and decoding device encode or decode indoor GOS and outdoor GOS with close coordinates adjacent to each other in world space, and also encode or decode these identifiers in correspondence.

[0280] (Implementation Method 2)

[0281] When using point cloud encoded data in actual devices or services, it is desirable to transmit and receive only the required information according to the application in order to reduce network bandwidth. However, existing 3D data encoding structures do not have this capability, and therefore no corresponding encoding method exists.

[0282] What will be described in this embodiment is a three-dimensional data encoding method and a three-dimensional data encoding device for providing the function of sending and receiving required information according to the purpose in the encoded data of a three-dimensional point cloud, as well as a three-dimensional data decoding method and a three-dimensional data decoding device for decoding the encoded data.

[0283] A voxel (VXL) having a certain feature value or more is defined as a feature voxel (FVXL), and a world space (WLD) formed by the FVXL is defined as a sparse world space (SWLD). Figure 11 This section shows an example of a sparse world space and its configuration. SWLD includes: FGOS, a GOS constructed from FVXL; FSPC, an SPC constructed from FVXL; and FVLM, a VLM constructed from FVXL. The data structures and prediction structures of FGOS, FSPC, and FVLM can be the same as those of GOS, SPC, and VLM.

[0284] Feature quantities represent the three-dimensional positional information of the VXL or visible light information at the VXL position. In particular, feature quantities are more likely to be detected at corners and edges of three-dimensional objects. Specifically, these feature quantities may be three-dimensional feature quantities or visible light feature quantities as described below, but any feature quantity may be used as long as it represents the position, brightness, or color information of the VXL.

[0285] As the three-dimensional feature, a SHOT feature (Signature of Histograms of Orientations), a PFH feature (Point Feature Histograms), or a PPF feature (Point Pair Feature) is used.

[0286] SHOT features are obtained by segmenting the area around the VXL, calculating the inner product between the reference point and the normal vector of the segmented area, and then histizing the result. These features are characterized by high dimensionality and high expressiveness.

[0287] PFH features are obtained by selecting multiple pairs of points near VXL, calculating normal vectors and other parameters from these two points, and then histogramming them. Since these PFH features are histogram features, they are robust to small amounts of noise and have high expressiveness.

[0288] The PPF feature is calculated based on the VXL of two points using the normal vector, etc. Since all VXLs are used in this PPF feature, it is insensitive to occlusion.

[0289] Furthermore, as feature quantities of visible light, SIFT (Scale-Invariant Feature Transform), SURF (Speeded Up Robust Features), or HOG (Histogram of Oriented Gradients) that utilize information such as brightness gradient information of an image can be used.

[0290] SWLD is generated by calculating the above-mentioned feature quantity from each VXL of WLD and extracting FVXL. Here, SWLD can be updated every time WLD is updated, or it can be updated regularly after a certain period of time regardless of the update timing of WLD.

[0291] SWLDs can be generated for each feature. For example, as shown in SWLD1 based on SHOT features and SWLD2 based on SIFT features, SWLDs can be generated for each feature and used according to the application. Furthermore, the calculated feature values ​​for each FVXL can be stored as feature value information in each FVXL.

[0292] Next, the method of using the sparse world space (SWLD) will be described. Since the SWLD only contains feature voxels (FVXL), the data size is generally smaller than that of the WLD which includes all VXLs.

[0293] In applications that utilize feature quantities to achieve a specific purpose, using SWLD information instead of WLD can reduce the time required to read data from the hard disk and reduce the bandwidth and transmission time during network transmission. For example, by pre-storing WLD and SWLD as map information on a server, switching the transmitted map information to WLD or SWLD based on client requests can reduce network bandwidth and transmission time. A specific example is shown below.

[0294] Figure 12 as well as Figure 13 The following shows the use of SWLD and WLD. Figure 12 As shown, when client 1, which is an in-vehicle device, needs map information for determining its own position, client 1 sends a request to the server to obtain map data for estimating its own position (S301). The server sends SWLD to client 1 in accordance with the acquisition request (S302). Client 1 uses the received SWLD to determine its own position (S303). At this time, client 1 obtains VXL information of the surrounding area of ​​client 1 through various methods such as distance sensors such as rangefinders, stereo cameras, or a combination of multiple monocular cameras, and estimates its own position information based on the obtained VXL information and SWLD. Here, the own position information includes the three-dimensional position information and orientation of client 1.

[0295] like Figure 13 As shown, when client 2, an in-vehicle device, requires map information for mapping purposes such as three-dimensional maps, client 2 sends a request to the server to obtain map data for mapping (S311). In response to this request, the server sends a WLD to client 2 (S312). Client 2 uses the received WLD to perform map rendering (S313). At this point, client 2 creates a conceptual image using, for example, images captured by a visible light camera and the WLD received from the server, and then renders the created image on a screen such as a car navigation system.

[0296] As described above, the server sends SWLD to the client for applications that primarily require the characteristic values ​​of each VXL, such as estimating its own position, and sends WLD to the client when detailed VXL information is required, such as for map drawing. This enables efficient transmission and reception of map data.

[0297] In addition, the client can determine whether it needs SWLD or WLD and request the server to send SWLD or WLD. Moreover, the server can determine whether to send SWLD or WLD based on the status of the client or the network.

[0298] Next, a method for switching between transmission and reception of the sparse world space (SWLD) and the world space (WLD) will be described.

[0299] The reception of WLD or SWLD can be switched according to the network bandwidth. Figure 14 An example of operation in this case is shown. For example, when a low-speed network with sufficient network bandwidth, such as in an LTE (Long Term Evolution) environment, is used, the client accesses the server via the low-speed network (S321) and obtains the SWLD (map information) from the server (S322). Alternatively, when a high-speed network with sufficient network bandwidth, such as in a WiFi environment, is used, the client accesses the server via the high-speed network (S323) and obtains the WLD from the server (S324). This allows the client to obtain appropriate map information based on the client's network bandwidth.

[0300] Specifically, the client receives SWLD via LTE outdoors, and acquires WLD via WiFi when entering a facility or other indoor location. This allows the client to acquire more detailed map information indoors.

[0301] In this way, the client can request WLD or SWLD from the server according to the frequency band of the network it is using. Alternatively, the client can send information indicating the frequency band of the network it is using to the server, and the server can send appropriate data (WLD or SWLD) to the client based on this information. Alternatively, the server can determine the network bandwidth of the client and send appropriate data (WLD or SWLD) to the client.

[0302] Furthermore, the reception of WLD or SWLD can be switched according to the moving speed. Figure 15 An example of operation in this case is shown. For example, when the client is moving at high speed (S331), the client receives SWLD from the server (S332). In addition, when the client is moving at low speed (S333), the client receives WLD from the server (S334). Accordingly, the client can suppress the network bandwidth and obtain map information according to the speed. Specifically, when the client is driving on a highway, by receiving SWLD with a small amount of data, the map information can be updated at a roughly appropriate speed. In addition, when the client is driving on a general road, by receiving WLD, more detailed map information can be obtained.

[0303] In this way, the client can request a WLD or SWLD from the server based on its own moving speed. Alternatively, the client can send information indicating its own moving speed to the server, and the server can send appropriate data (WLD or SWLD) to the client based on this information. Alternatively, the server can determine the client's moving speed and send appropriate data (WLD or SWLD) to the client.

[0304] Alternatively, the client can first obtain the SWLD from the server and then obtain the WLD of important areas within it. For example, when acquiring map data, the client can first use the SWLD to obtain general map information, filter out areas with a high incidence of features such as buildings, signs, or people, and then obtain the WLD of these filtered areas. This allows the client to reduce the amount of data received from the server while still obtaining detailed information about the desired area.

[0305] Alternatively, the server can create separate SWLDs for each object based on the WLD, and the client can receive them separately based on their intended use. This can reduce network bandwidth. For example, the server can pre-identify people and cars from the WLD and create SWLDs for people and cars. If the client wants to obtain information about people around it, it will receive the SWLDs for people; if it wants to obtain information about cars, it will receive the SWLDs for cars. Furthermore, the type of SWLD can be distinguished based on information attached to the header (such as a flag or type).

[0306] Next, the configuration and operation flow of the three-dimensional data encoding device (for example, a server) according to this embodiment will be described. Figure 16 This is a block diagram of a three-dimensional data encoding device 400 according to this embodiment. Figure 17 This is a flowchart of the three-dimensional data encoding process performed by the three-dimensional data encoding device 400.

[0307] Figure 16 The illustrated three-dimensional data encoding device 400 encodes input three-dimensional data 411 to generate encoded three-dimensional data 413 and 414 as encoded streams. Encoded three-dimensional data 413 corresponds to the WLD, and encoded three-dimensional data 414 corresponds to the SWLD. The three-dimensional data encoding device 400 includes an acquisition unit 401, a coding region determination unit 402, an SWLD extraction unit 403, a WLD encoding unit 404, and a SWLD encoding unit 405.

[0308] like Figure 17 As shown, first, the acquisition unit 401 acquires input three-dimensional data 411 which is point cloud data in a three-dimensional space ( S401 ).

[0309] Next, the coding region determination unit 402 determines a spatial region to be coded based on the spatial region where the point cloud data exists ( S402 ).

[0310] Next, the SWLD extraction unit 403 defines the spatial region to be encoded as a WLD and calculates features from each VXL contained in the WLD. Furthermore, the SWLD extraction unit 403 extracts VXLs whose features exceed a predetermined threshold, defines these VXLs as FVXLs, and appends these FVXLs to the SWLD to generate extracted three-dimensional data 412 (S403). Specifically, extracted three-dimensional data 412 with features exceeding the threshold is extracted from the input three-dimensional data 411.

[0311] Next, the WLD encoder 404 encodes the input 3D data 411 corresponding to the WLD to generate encoded 3D data 413 corresponding to the WLD (S404). At this time, the WLD encoder 404 appends information to the header of the encoded 3D data 413 that identifies the encoded 3D data 413 as a stream containing the WLD.

[0312] Then, the SWLD encoding unit 405 encodes the extracted 3D data 412 corresponding to the SWLD to generate encoded 3D data 414 corresponding to the SWLD (S405). At this time, the SWLD encoding unit 405 adds information to the header of the encoded 3D data 414 to distinguish that the encoded 3D data 414 is a stream containing the SWLD.

[0313] Furthermore, the order of the process of generating the encoded three-dimensional data 413 and the process of generating the encoded three-dimensional data 414 may be reversed. Furthermore, part or all of the above processes may be executed in parallel.

[0314] The information assigned to the header of the encoded three-dimensional data 413 and 414 is defined as a parameter such as "world_type". In the case of world_type = 0, it indicates that the stream contains WLD, and in the case of world_type = 1, it indicates that the stream contains SWLD. When more categories are defined, the assigned value can be increased, such as world_type = 2. In addition, a specific flag can be included in one of the encoded three-dimensional data 413 and 414. For example, the encoded three-dimensional data 414 can be assigned a flag indicating that the stream contains SWLD. In this case, the decoding device can determine whether it is a stream containing WLD or a stream containing SWLD based on the presence or absence of the flag.

[0315] Furthermore, the encoding method used by the WLD encoding unit 404 when encoding the WLD may be different from the encoding method used by the SWLD encoding unit 405 when encoding the SWLD.

[0316] For example, since data is decimated in SWLD, the correlation with surrounding data may be lower than that in WLD. Therefore, in the encoding method for SWLD, inter-frame prediction is prioritized over intra-frame prediction and inter-frame prediction in comparison with the encoding method for WLD.

[0317] Furthermore, the encoding method used for SWLD and the encoding method used for WLD may differ in the representation of the three-dimensional position. For example, the three-dimensional position of FVXL may be represented by three-dimensional coordinates in FWLD, while the three-dimensional position may be represented by an octree (described later) in WLD, or vice versa.

[0318] Furthermore, the SWLD encoding unit 405 encodes the SWLD encoded three-dimensional data 414 so that the data size is smaller than the WLD encoded three-dimensional data 413. For example, as described above, the correlation between data in SWLD and WLD may be lower. This may reduce the encoding efficiency, and the data size of the encoded three-dimensional data 414 may be larger than the data size of the WLD encoded three-dimensional data 413. Therefore, if the data size of the obtained encoded three-dimensional data 414 is larger than the data size of the WLD encoded three-dimensional data 413, the SWLD encoding unit 405 re-encodes the data to generate the encoded three-dimensional data 414 with a reduced data size.

[0319] For example, the SWLD extraction unit 403 generates extracted three-dimensional data 412 again with a reduced number of extracted feature points, and the SWLD encoding unit 405 encodes the extracted three-dimensional data 412. Alternatively, the degree of quantization in the SWLD encoding unit 405 may be coarsened. For example, in the octree structure described later, the degree of quantization can be coarsened by rounding the data in the lowest layer.

[0320] Furthermore, if the data size of the SWLD encoded three-dimensional data 414 cannot be made smaller than the data size of the WLD encoded three-dimensional data 413, the SWLD encoding unit 405 may not generate the SWLD encoded three-dimensional data 414. Alternatively, the WLD encoded three-dimensional data 413 may be copied to the SWLD encoded three-dimensional data 414. In other words, the WLD encoded three-dimensional data 413 may be used as the SWLD encoded three-dimensional data 414.

[0321] Next, the configuration and operation flow of the three-dimensional data decoding device (eg, client) according to this embodiment will be described. Figure 18 This is a block diagram of a three-dimensional data decoding device 500 according to this embodiment. Figure 19 3D data decoding processing performed by the 3D data decoding apparatus 500 is shown in FIG.

[0322] Figure 18 The illustrated three-dimensional data decoding apparatus 500 generates decoded three-dimensional data 512 or 513 by decoding encoded three-dimensional data 511. Here, the encoded three-dimensional data 511 is, for example, the encoded three-dimensional data 413 or 414 generated by the three-dimensional data encoding apparatus 400.

[0323] The three-dimensional data decoding device 500 includes an acquisition unit 501 , a header analysis unit 502 , a WLD decoding unit 503 , and a SWLD decoding unit 504 .

[0324] like Figure 19 As shown, first, the acquisition unit 501 obtains the encoded 3D data 511 (S501). Next, the header analysis unit 502 analyzes the header of the encoded 3D data 511 to determine whether the encoded 3D data 511 is a stream containing WLD or a stream containing SWLD (S502). For example, this determination is made by referring to the world_type parameter described above.

[0325] If the encoded 3D data 511 is a stream containing a WLD ("Yes" in S503), the WLD decoding unit 503 decodes the encoded 3D data 511 to generate decoded 3D WLD data 512 (S504). If the encoded 3D data 511 is a stream containing a SWLD ("No" in S503), the SWLD decoding unit 504 decodes the encoded 3D data 511 to generate decoded 3D SWLD data 513 (S505).

[0326] Furthermore, similar to the encoding device, the decoding method used by the WLD decoding unit 503 when decoding the WLD may be different from the decoding method used by the SWLD decoding unit 504 when decoding the SWLD. For example, in the decoding method for the SWLD, priority may be given to the inter-frame prediction of intra-frame prediction and inter-frame prediction over the decoding method for the WLD.

[0327] Furthermore, the three-dimensional position representation method can be different between the decoding method used for SWLD and the decoding method used for WLD. For example, the three-dimensional position of FVXL can be represented by three-dimensional coordinates in SWLD, while the three-dimensional position can be represented by an octree (described later) in WLD, or vice versa.

[0328] Next, octree representation as a method of representing three-dimensional positions will be described. VXL data included in three-dimensional data is converted into an octree structure and then encoded. Figure 20 An example of VXL of WLD is shown. Figure 21 Shown Figure 20 The octree structure of WLD is shown in Figure 2. Figure 20 In the example shown, there are three VXLs 1 to 3 that are VXLs (hereinafter referred to as valid VXLs) that include a point group. Figure 21 As shown, the octree structure consists of nodes and leaf nodes. Each node has a maximum of 8 nodes or leaf nodes. Each leaf node has VXL information. Here, Figure 21 Among the leaf nodes shown, leaf nodes 1, 2, and 3 represent Figure 20 VXL1, VXL2, and VXL3 shown.

[0329] Specifically, each node and leaf node corresponds to a three-dimensional position. Figure 20 The block corresponding to node 1 is divided into 8 blocks. Among the 8 blocks, the blocks containing valid VXLs are set as nodes, and the remaining blocks are set as leaf nodes. The block corresponding to the node is further divided into 8 nodes or leaf nodes. This process is repeated as many times as the number of levels in the tree structure. Finally, all the blocks at the bottom level are set as leaf nodes.

[0330] and, Figure 22 Shown from Figure 20 An example of SWLD generated by WLD is shown. Figure 20 The feature extraction results of VXL1 and VXL2 shown are determined to be FVXL1 and FVXL2 and are included in SWLD. In addition, VXL3 is not determined to be FVXL and is therefore not included in SWLD. Figure 23 Shown Figure 22 The octree structure of SWLD is shown in Figure 1. Figure 23 In the octree structure shown, Figure 21 The leaf node 3 shown, which is equivalent to VXL3, is deleted. Figure 21 Node 3 shown does not have a valid VXL and is changed to a leaf node. Generally speaking, the number of leaf nodes of SWLD is smaller than that of WLD, and the coded three-dimensional data of SWLD is also smaller than that of WLD.

[0331] Modifications of this embodiment will be described below.

[0332] For example, when a client such as a vehicle-mounted device estimates its own position, it receives SWLD from the server, uses SWLD to estimate its own position, and performs obstacle detection. It then uses various methods such as distance sensors such as rangefinders, stereo cameras, or a combination of multiple monocular cameras to perform obstacle detection based on the three-dimensional information of the surrounding area obtained by itself.

[0333] Furthermore, it's generally difficult to include VXL data for flat areas in SWLD. Therefore, the server maintains a subsampled world space (SubWLD) that downsamples the WLD for stationary obstacle detection and can send both the SWLD and SubWLD to the client. This reduces network bandwidth while enabling client-side position estimation and obstacle detection.

[0334] Furthermore, when clients need to quickly render 3D map data, a grid-like structure is often more convenient. Therefore, the server can generate a grid based on the WLD and store it in advance as a grid world space (MWLD). For example, if the client requires a coarse 3D rendering, it receives the MWLD; if it requires a detailed 3D rendering, it receives the WLD. This helps reduce network bandwidth.

[0335] Furthermore, while the server sets the VXL with a feature value above a threshold value from each VXL as the FVXL, the FVXL can also be calculated using different methods. For example, if the server determines that the VXL, VLM, SPC, or GOS that constitute a signal or intersection are necessary for self-position estimation, driving assistance, or autonomous driving, they can be included in the SWLD as FVXL, FVLM, FSPC, or FGOS. Furthermore, this determination can be made manually. In addition, the FVXL, etc. obtained by the above method can be added to the FVXL, etc. set based on the feature value. That is, the SWLD extraction unit 403 can further extract data corresponding to objects with predetermined attributes from the input three-dimensional data 411 as extracted three-dimensional data 412.

[0336] Furthermore, labels different from feature values ​​can be assigned to situations required for these purposes. The server can separately maintain FVXL, which is required for estimating its own position, such as signals or intersections, driving assistance, or autonomous driving, as a higher layer of SWLD (e.g., lane world space).

[0337] Furthermore, the server can also add attributes to the VXL within the WLD in random access units or specified units. Attributes include, for example, information indicating whether a location is required or not for estimating the location, or information indicating whether traffic information such as signals or intersections is important. Attributes can also include the correspondence between lane information (GDF: Geographic Data Files, etc.) and features (such as intersections or roads).

[0338] Furthermore, as a method of updating WLD or SWLD, the following method can be adopted.

[0339] Update information showing changes in people, construction, or street trees (trajectory orientation) is uploaded to the server as a point group or metadata. The server updates the WLD based on this upload, and then updates the SWLD using the updated WLD.

[0340] Furthermore, if the client detects a mismatch between the 3D information generated by itself when estimating its own position and the 3D information received from the server, it can send the generated 3D information to the server along with an update notification. In this case, the server uses the WLD to update the SWLD. If the SWLD is not updated, the server determines that the WLD itself is outdated.

[0341] Furthermore, while information distinguishing between WLD and SWLD is added to the coded stream header, if there are multiple world spaces, such as a grid world space or a lane world space, information distinguishing between them can be added to the header. Furthermore, if there are multiple SWLDs with different feature values, information distinguishing between them can also be added to the header.

[0342] Furthermore, while the SWLD consists of FVXLs, it can also include VXLs that are not identified as FVXLs. For example, the SWLD can include adjacent VXLs used to calculate the characteristics of the FVXLs. This allows the client to calculate the characteristics of the FVXLs upon receiving the SWLD, even if the FVXLs in the SWLD do not have feature information attached. Furthermore, the SWLD can include information to distinguish whether each VXL is an FVXL or a VXL.

[0343] As described above, the three-dimensional data encoding device 400 extracts extracted three-dimensional data 412 (second three-dimensional data) whose feature value is above a threshold value from the input three-dimensional data 411 (first three-dimensional data), and generates encoded three-dimensional data 414 (first encoded three-dimensional data) by encoding the extracted three-dimensional data 412.

[0344] Based on this, the three-dimensional data encoding device 400 generates encoded three-dimensional data 414 by encoding data with a feature value greater than or equal to a threshold value. This reduces the amount of data compared to directly encoding the input three-dimensional data 411. Therefore, the three-dimensional data encoding device 400 can reduce the amount of data required for transmission.

[0345] Furthermore, the three-dimensional data encoding device 400 further encodes the input three-dimensional data 411 to generate encoded three-dimensional data 413 (second encoded three-dimensional data).

[0346] With this, the three-dimensional data encoding device 400 can selectively transmit the encoded three-dimensional data 413 and the encoded three-dimensional data 414 according to the intended use, for example.

[0347] Furthermore, the extracted three-dimensional data 412 is encoded using a first encoding method, and the input three-dimensional data 411 is encoded using a second encoding method that is different from the first encoding method.

[0348] Accordingly, the three-dimensional data encoding device 400 can adopt appropriate encoding methods for the input three-dimensional data 411 and the extracted three-dimensional data 412 respectively.

[0349] Furthermore, in the first encoding method, compared with the second encoding method, priority is given to inter-frame prediction among intra-frame prediction and inter-frame prediction.

[0350] According to this, the three-dimensional data encoding apparatus 400 can increase the priority of inter-frame prediction for the extracted three-dimensional data 412 where the correlation between adjacent data tends to be low.

[0351] Furthermore, the first encoding method and the second encoding method use different methods to represent three-dimensional positions. For example, the second encoding method represents three-dimensional positions using an octree, while the first encoding method represents three-dimensional positions using three-dimensional coordinates.

[0352] With this, the three-dimensional data encoding device 400 can adopt a more appropriate three-dimensional position expression method for three-dimensional data having different data amounts (number of VXLs or FVXLs).

[0353] Furthermore, at least one of the coded 3D data 413 and 414 includes an identifier indicating whether the coded 3D data is obtained by encoding the input 3D data 411 or by encoding a portion of the input 3D data 411. In other words, the identifier indicates whether the coded 3D data is the coded 3D data 413 of the WLD or the coded 3D data 414 of the SWLD.

[0354] Based on this, the decoding device can easily determine whether the acquired encoded three-dimensional data is the encoded three-dimensional data 413 or the encoded three-dimensional data 414.

[0355] Furthermore, the three-dimensional data encoding device 400 encodes the extracted three-dimensional data 412 so that the data amount of the encoded three-dimensional data 414 is smaller than the data amount of the encoded three-dimensional data 413 .

[0356] As a result, the three-dimensional data encoding device 400 can make the data amount of the encoded three-dimensional data 414 smaller than the data amount of the encoded three-dimensional data 413 .

[0357] Furthermore, the three-dimensional data encoding device 400 extracts data corresponding to objects having predetermined attributes from the input three-dimensional data 411 as extracted three-dimensional data 412. For example, the objects having predetermined attributes may be objects required for self-position estimation, driving assistance, or autonomous driving, such as signals or intersections.

[0358] In this way, the three-dimensional data encoding device 400 can generate the encoded three-dimensional data 414 including the data required by the decoding device.

[0359] Furthermore, the three-dimensional data encoding device 400 (server) further transmits one of the encoded three-dimensional data 413 and 414 to the client according to the status of the client.

[0360] Thus, the three-dimensional data encoding device 400 can send appropriate data according to the status of the client.

[0361] Furthermore, the client status includes the client's communication status (eg, network bandwidth) or the client's moving speed.

[0362] Furthermore, the three-dimensional data encoding device 400 further transmits one of the encoded three-dimensional data 413 and 414 to the client according to the client's request.

[0363] Thus, the three-dimensional data encoding device 400 can send appropriate data according to the client's request.

[0364] Furthermore, the three-dimensional data decoding device 500 according to this embodiment decodes the encoded three-dimensional data 413 or 414 generated by the three-dimensional data encoding device 400 .

[0365] Specifically, the three-dimensional data decoding apparatus 500 decodes the encoded three-dimensional data 414 obtained by encoding the extracted three-dimensional data 412 whose feature quantity extracted from the input three-dimensional data 411 is greater than or equal to a threshold value using a first decoding method. Furthermore, the three-dimensional data decoding apparatus 500 decodes the encoded three-dimensional data 413 obtained by encoding the input three-dimensional data 411 using a second decoding method that is different from the first decoding method.

[0366] Thus, the 3D data decoding device 500 can selectively receive the encoded 3D data 414 and 413, which are obtained by encoding data with feature quantities exceeding a threshold value, according to, for example, the intended use. This allows the 3D data decoding device 500 to reduce the amount of data transmitted. Furthermore, the 3D data decoding device 500 can employ appropriate decoding methods for each of the input 3D data 411 and the extracted 3D data 412.

[0367] Furthermore, in the first decoding method, compared with the second decoding method, priority is given to inter prediction between intra prediction and inter prediction.

[0368] With this, the three-dimensional data decoding apparatus 500 can increase the priority of inter-frame prediction for extracting three-dimensional data in which the correlation between adjacent data tends to be low.

[0369] Furthermore, the first decoding method and the second decoding method use different methods to represent the three-dimensional position. For example, the second decoding method represents the three-dimensional position using an octree, while the first decoding method represents the three-dimensional position using three-dimensional coordinates.

[0370] Thus, the three-dimensional data decoding apparatus 500 can adopt a more appropriate three-dimensional position expression method for three-dimensional data having different data amounts (number of VXLs or FVXLs).

[0371] Furthermore, at least one of the encoded 3D data 413 and 414 includes an identifier indicating whether the encoded 3D data is obtained by encoding the input 3D data 411 or by encoding a portion of the input 3D data 411. The 3D data decoding apparatus 500 identifies the encoded 3D data 413 and 414 by referring to the identifier.

[0372] Based on this, the three-dimensional data decoding apparatus 500 can easily determine whether the obtained encoded three-dimensional data is the encoded three-dimensional data 413 or the encoded three-dimensional data 414 .

[0373] Furthermore, the 3D data decoding device 500 notifies the server of the status of the client (the 3D data decoding device 500 ) and receives one of the encoded 3D data 413 and 414 transmitted from the server according to the status of the client.

[0374] Thus, the three-dimensional data decoding apparatus 500 can receive appropriate data according to the status of the client.

[0375] Furthermore, the client status includes the client's communication status (eg, network bandwidth) or the client's moving speed.

[0376] Furthermore, the three-dimensional data decoding apparatus 500 further requests one of the encoded three-dimensional data 413 and 414 from the server, and receives one of the encoded three-dimensional data 413 and 414 transmitted from the server in accordance with the request.

[0377] This allows the three-dimensional data decoding apparatus 500 to receive appropriate data corresponding to the intended use.

[0378] (Implementation 3)

[0379] In this embodiment, a method for transmitting and receiving three-dimensional data between vehicles will be described. For example, three-dimensional data is transmitted and received between a vehicle and surrounding vehicles.

[0380] Figure 24 This is a block diagram of a three-dimensional data creation device 620 according to this embodiment. This three-dimensional data creation device 620 is included in, for example, a vehicle and creates denser third three-dimensional data 636 by combining received second three-dimensional data 635 with first three-dimensional data 632 created by the three-dimensional data creation device 620.

[0381] The three-dimensional data creation device 620 includes a three-dimensional data creation unit 621 , a request range determination unit 622 , a search unit 623 , a reception unit 624 , a decoding unit 625 , and a synthesis unit 626 .

[0382] First, the three-dimensional data creation unit 621 uses sensor information 631 detected by sensors in the vehicle to create first three-dimensional data 632. Next, the request range determination unit 622 determines a request range, which is a three-dimensional space range where the data in the created first three-dimensional data 632 is insufficient.

[0383] Next, the search unit 623 searches for surrounding vehicles that hold three-dimensional data for the requested range and sends request range information 633 indicating the requested range to the surrounding vehicles identified by the search. Next, the receiving unit 624 receives encoded three-dimensional data 634 as an encoded stream for the requested range from the surrounding vehicles (S624). Alternatively, the search unit 623 can indiscriminately issue requests to all vehicles within the identified range and receive encoded three-dimensional data 634 from any responding vehicles. Furthermore, the search unit 623 is not limited to vehicles and can also issue requests to objects such as traffic lights and signs and receive encoded three-dimensional data 634 from those objects.

[0384] Next, the decoder 625 decodes the received encoded three-dimensional data 634 to obtain second three-dimensional data 635. Next, the synthesizer 626 synthesizes the first three-dimensional data 632 and the second three-dimensional data 635 to create denser third three-dimensional data 636.

[0385] Next, the configuration and operation of the three-dimensional data transmitting device 640 according to this embodiment will be described. Figure 25 It is a block diagram of the three-dimensional data transmitting device 640.

[0386] The three-dimensional data sending device 640 is included in the above-mentioned surrounding vehicles, for example, and processes the fifth three-dimensional data 652 produced by the surrounding vehicles into the sixth three-dimensional data 654 requested by the own vehicle, and generates encoded three-dimensional data 634 by encoding the sixth three-dimensional data 654, and sends the encoded three-dimensional data 634 to the own vehicle.

[0387] The three-dimensional data transmitting device 640 includes a three-dimensional data creating unit 641 , a receiving unit 642 , an extracting unit 643 , an encoding unit 644 , and a transmitting unit 645 .

[0388] First, the three-dimensional data creation unit 641 uses sensor information 651 detected by sensors included in surrounding vehicles to create fifth three-dimensional data 652. Next, the receiving unit 642 receives the requested range information 633 transmitted from the own vehicle.

[0389] Next, the extraction unit 643 extracts the three-dimensional data within the requested range indicated by the requested range information 633 from the fifth three-dimensional data 652, and processes the fifth three-dimensional data 652 into sixth three-dimensional data 654. The encoding unit 644 then encodes the sixth three-dimensional data 654, generating encoded three-dimensional data 634 as an encoded stream. The transmission unit 645 then transmits the encoded three-dimensional data 634 to the vehicle itself.

[0390] In addition, although the example in which the own vehicle has the three-dimensional data creation device 620 and the surrounding vehicles have the three-dimensional data transmission device 640 is described here, each vehicle may also have the functions of the three-dimensional data creation device 620 and the three-dimensional data transmission device 640.

[0391] (Implementation 4)

[0392] In this embodiment, an operation related to abnormal situations in self-position estimation based on a three-dimensional map will be described.

[0393] Applications such as autonomous driving of vehicles and autonomous movement of mobile objects such as robots and drones are expected to expand in the future. One example of a method for achieving such autonomous movement is a method in which a mobile object estimates its own position within a three-dimensional map (self-position estimation) and drives according to the map.

[0394] Self-position estimation is achieved by matching the three-dimensional map with the three-dimensional information around the own vehicle obtained by sensors such as the rangefinder (LIDAR, etc.) or stereo camera installed on the own vehicle (hereinafter referred to as the own vehicle detection three-dimensional data), and estimating the own vehicle position within the three-dimensional map.

[0395] 3D maps, such as those offered by HERE's HD maps, are not just 3D point clouds; they also include 2D map data such as road and intersection shape information, as well as real-time information such as traffic jams and accidents. 3D maps are constructed from multiple layers of data, including 3D and 2D data and metadata that changes over time. Devices can retrieve only the data they need, or they can reference only the data they need.

[0396] The point cloud data may be the aforementioned SWLD, or may include point group data that is not a feature point. Furthermore, the transmission and reception of the point cloud data is basically performed in one or more random access units.

[0397] The following methods can be used to match a 3D map with the 3D data detected by the own vehicle. For example, the device compares the shapes of the point clusters in the respective point clouds and determines areas with high similarity between feature points as being at the same location. Furthermore, if the 3D map is constructed using SWLD, the device compares and matches the feature points that make up the SWLD with the 3D feature points extracted from the 3D data detected by the own vehicle.

[0398] To accurately estimate the vehicle's position, the following conditions (A) and (B) must be met: (A) a 3D map and 3D vehicle detection data must be available, and (B) their accuracy must meet a predetermined standard. However, in the following exceptional circumstances, either (A) or (B) may not be met.

[0399] (1) A three-dimensional map cannot be obtained through the communication path.

[0400] (2) There is no three-dimensional map, or the obtained three-dimensional map is damaged.

[0401] (3) The sensor of the own vehicle fails, or due to bad weather, the accuracy of the three-dimensional data generated by the own vehicle is insufficient.

[0402] The following describes the operations for dealing with these abnormal situations. Although the following operation is described using a vehicle as an example, the following method can also be applied to all moving objects that move autonomously, such as robots and drones.

[0403] The following describes the configuration and operation of the three-dimensional information processing device according to this embodiment for detecting abnormalities in three-dimensional data corresponding to a three-dimensional map or a vehicle. Figure 26 This is a block diagram showing a configuration example of a three-dimensional information processing device 700 according to this embodiment.

[0404] The three-dimensional information processing device 700 is mounted on a mobile object such as a motor vehicle. Figure 26As shown, the three-dimensional information processing device 700 includes a three-dimensional map acquisition unit 701 , a vehicle detection data acquisition unit 702 , an abnormality determination unit 703 , a countermeasure operation determination unit 704 , and an operation control unit 705 .

[0405] The 3D information processing device 700 may also include a camera that captures 2D images, or a sensor (not shown) that uses ultrasonic or laser-based 1D data to detect structures or moving objects around the vehicle. Furthermore, the 3D information processing device 700 may include a communication unit (not shown) that generates a 3D map via a mobile communication network such as 4G or 5G, or between vehicles or between roads and vehicles.

[0406] The three-dimensional map acquisition unit 701 acquires a three-dimensional map 711 near the driving route. For example, the three-dimensional map acquisition unit 701 acquires the three-dimensional map 711 through a mobile communication network, vehicle-to-vehicle communication, or road-to-vehicle communication.

[0407] Next, the vehicle detection data acquisition unit 702 acquires the vehicle detection three-dimensional data 712 based on the sensor information. For example, the vehicle detection data acquisition unit 702 generates the vehicle detection three-dimensional data 712 based on the sensor information acquired by the sensors included in the vehicle.

[0408] Next, abnormality determination unit 703 detects abnormalities by performing predetermined checks on at least one of the obtained three-dimensional map 711 and the vehicle's own three-dimensional detection data 712. Specifically, abnormality determination unit 703 determines whether at least one of the obtained three-dimensional map 711 and the vehicle's own three-dimensional detection data 712 is abnormal.

[0409] When an abnormality is detected, the countermeasure decision unit 704 decides a countermeasure for the abnormality. Next, the task control unit 705 controls the operations of various processing units required for executing the countermeasure, such as the three-dimensional map acquisition unit 701.

[0410] If no abnormality is detected, the three-dimensional information processing apparatus 700 ends the processing.

[0411] The 3D information processing device 700 estimates the vehicle's own position using the 3D map 711 and the own vehicle detection 3D data 712. The 3D information processing device 700 then uses the result of the own position estimation to enable the vehicle to autonomously drive.

[0412] Based on this, the three-dimensional information processing device 700 obtains map data (three-dimensional map 711) including the first three-dimensional position information via the channel. For example, the first three-dimensional position information is encoded in units of subspaces having three-dimensional coordinate information. The first three-dimensional position information includes multiple random access units, each of which is a collection of one or more subspaces and can be independently decoded. For example, the first three-dimensional position information is data (SWLD) encoded at feature points where the three-dimensional feature quantity exceeds a predetermined threshold.

[0413] The three-dimensional information processing device 700 generates second three-dimensional position information (vehicle detection three-dimensional data 712) based on the information detected by the sensor. The three-dimensional information processing device 700 then performs abnormality determination processing on the first three-dimensional position information or the second three-dimensional position information to determine whether the first three-dimensional position information or the second three-dimensional position information is abnormal.

[0414] When the three-dimensional information processing device 700 determines that the first three-dimensional position information or the second three-dimensional position information is abnormal, it determines a countermeasure for the abnormality and then executes control required for executing the countermeasure.

[0415] With this, the three-dimensional information processing device 700 can detect abnormality in the first three-dimensional position information or the second three-dimensional position information and perform a corresponding operation.

[0416] (Implementation 5)

[0417] In this embodiment, a method of transmitting three-dimensional data to a following vehicle will be described.

[0418] Figure 27 This is a block diagram illustrating an example configuration of a three-dimensional data creation device 810 according to this embodiment. This three-dimensional data creation device 810 is, for example, mounted on a vehicle. It transmits and receives three-dimensional data with external traffic cloud monitoring, preceding vehicles, and following vehicles, while also creating and storing the three-dimensional data.

[0419] The three-dimensional data production device 810 includes: a data receiving unit 811, a communication unit 812, a receiving control unit 813, a format conversion unit 814, multiple sensors 815, a three-dimensional data production unit 816, a three-dimensional data synthesis unit 817, a three-dimensional data storage unit 818, a communication unit 819, a sending control unit 820, a format conversion unit 821, and a data sending unit 822.

[0420] The data receiving unit 811 receives 3D data 831 from traffic cloud monitoring or the preceding vehicle. The 3D data 831 includes, for example, a point cloud containing information about areas that the vehicle's sensors 815 cannot detect, visible light images, depth information, sensor position information, or speed information.

[0421] The communication unit 812 communicates with the traffic cloud monitoring or the vehicle ahead, and sends a data transmission request or the like to the traffic cloud monitoring or the vehicle ahead.

[0422] The reception control unit 813 exchanges information such as a corresponding format with the communication partner via the communication unit 812 to establish communication with the communication partner.

[0423] The format conversion unit 814 generates three-dimensional data 832 by performing format conversion on the three-dimensional data 831 received by the data reception unit 811. Furthermore, if the three-dimensional data 831 is compressed or encoded, the format conversion unit 814 performs decompression or decoding processing.

[0424] Multiple sensors 815 are a group of sensors such as LiDAR, visible light cameras, or infrared cameras that obtain information about the exterior of the vehicle and generate sensor information 833. For example, if sensor 815 is a laser sensor such as LiDAR, sensor information 833 is three-dimensional data such as a point cloud (point cloud data). Furthermore, the number of sensors 815 does not need to be multiple.

[0425] The three-dimensional data generating unit 816 generates three-dimensional data 834 based on the sensor information 833. The three-dimensional data 834 includes, for example, point clouds, visible light images, depth information, sensor position information, or speed information.

[0426] The three-dimensional data synthesis unit 817 synthesizes the three-dimensional data 832 produced by traffic cloud monitoring or the vehicle in front, etc., into the three-dimensional data 834 produced based on the sensor information 833 of the own vehicle, thereby constructing three-dimensional data 835 that also includes the space in front of the vehicle in front that cannot be detected by the sensor 815 of the own vehicle.

[0427] The three-dimensional data accumulation unit 818 accumulates the generated three-dimensional data 835 and the like.

[0428] The communication unit 819 communicates with the traffic cloud monitoring system or the vehicle behind, and sends a data transmission request or the like to the traffic cloud monitoring system or the vehicle behind.

[0429] The transmission control unit 820 exchanges information such as the corresponding format with the communication partner via the communication unit 819 to establish communication with the communication partner. Furthermore, the transmission control unit 820 determines the spatial transmission area of ​​the three-dimensional data to be transmitted based on the three-dimensional data construction information of the three-dimensional data 832 generated by the three-dimensional data synthesis unit 817 and the data transmission request from the communication partner.

[0430] Specifically, based on data transmission requests from traffic cloud monitoring or following vehicles, the transmission control unit 820 determines a transmission area, including the space in front of the vehicle itself, which is not detectable by the following vehicle's sensors. Furthermore, the transmission control unit 820 determines the transmission area by determining, based on the three-dimensional data construction information, whether updates to the available transmission space or the already transmitted space have occurred. For example, the transmission control unit 820 determines as the transmission area an area that is both specified in the data transmission request and contains the corresponding three-dimensional data 835. Furthermore, the transmission control unit 820 notifies the format conversion unit 821 of the communication partner's supported format and the transmission area.

[0431] Format conversion unit 821 converts three-dimensional data 836 in the transmission area of ​​three-dimensional data 835 stored in three-dimensional data storage unit 818 into a format compatible with the receiving end, thereby generating three-dimensional data 837. Format conversion unit 821 may also compress or encode three-dimensional data 837 to reduce the data volume.

[0432] The data transmission unit 822 transmits the three-dimensional data 837 to the traffic cloud monitoring or the following vehicle. The three-dimensional data 837 includes, for example, a point cloud in front of the vehicle including information on areas that are blind spots for the following vehicle, visible light images, depth information, or sensor position information.

[0433] In addition, although the format conversion units 814 and 821 perform format conversion, the format conversion may not be performed.

[0434] With this configuration, three-dimensional data creation device 810 obtains three-dimensional data 831 from an external source for an area that is undetectable by the vehicle's sensors 815, and generates three-dimensional data 835 by combining three-dimensional data 831 with three-dimensional data 834 based on sensor information 833 detected by the vehicle's sensors 815. This allows three-dimensional data creation device 810 to generate three-dimensional data for an area that is undetectable by the vehicle's sensors 815.

[0435] In addition, the three-dimensional data production device 810 can send three-dimensional data of the space in front of its own vehicle that cannot be detected by the sensors of the rear vehicle to the traffic cloud monitoring or the rear vehicle, etc. according to the data sending request from the traffic cloud monitoring or the rear vehicle.

[0436] (Implementation 6)

[0437] In the fifth embodiment, a client device such as a vehicle transmits three-dimensional data to another vehicle or a server such as a traffic cloud monitoring system. In this embodiment, the client device transmits sensor information obtained by a sensor to a server or another client device.

[0438] First, the configuration of the system according to this embodiment will be described. Figure 28 The configuration of a system for transmitting and receiving three-dimensional maps and sensor information according to this embodiment is shown. The system includes a server 901 and client devices 902A and 902B. In addition, when client devices 902A and 902B are not specifically distinguished, they are also referred to as client device 902.

[0439] The client device 902 is, for example, an in-vehicle device mounted on a mobile object such as a vehicle. The server 901 is, for example, a traffic cloud monitoring system, and is capable of communicating with a plurality of client devices 902 .

[0440] The server 901 transmits a three-dimensional map composed of point clouds to the client device 902. In addition, the composition of the three-dimensional map is not limited to point clouds, and can also be expressed by other three-dimensional data such as a mesh structure.

[0441] The client device 902 sends sensor information obtained by the client device 902 to the server 901. The sensor information includes, for example, at least one of LiDAR information, visible light images, infrared images, depth images, sensor position information, and speed information.

[0442] The data sent and received between server 901 and client device 902 can be compressed to reduce data size, or not compressed to maintain data accuracy. When data compression is used, for example, a three-dimensional compression method based on an octree can be used for point clouds. Furthermore, two-dimensional image compression methods can be used for visible light images, infrared images, and depth images. Examples of two-dimensional image compression methods include MPEG-4 AVC and HEVC standardized by MPEG.

[0443] Furthermore, in response to a request from client device 902 to transmit the three-dimensional map, server 901 transmits the three-dimensional map managed by server 901 to client device 902. Alternatively, server 901 may transmit the three-dimensional map without waiting for a request from client device 902 to transmit the three-dimensional map. For example, server 901 may broadcast the three-dimensional map to one or more client devices 902 in a predetermined space. Furthermore, server 901 may transmit a three-dimensional map appropriate for the location of client device 902 at regular intervals to a client device 902 that has received a transmission request. Furthermore, server 901 may transmit the three-dimensional map to client device 902 whenever the three-dimensional map managed by server 901 is updated.

[0444] The client device 902 sends a three-dimensional map transmission request to the server 901. For example, when the client device 902 wants to estimate its own position while driving, the client device 902 sends the three-dimensional map transmission request to the server 901.

[0445] In addition, the client device 902 may also issue a request to transmit a three-dimensional map to the server 901 in the following situations. If the three-dimensional map held by the client device 902 is relatively old, the client device 902 may also issue a request to transmit the three-dimensional map to the server 901. For example, if a certain period of time has passed since the client device 902 acquired the three-dimensional map, the client device 902 may also issue a request to transmit the three-dimensional map to the server 901.

[0446] Alternatively, client device 902 may issue a three-dimensional map transmission request to server 901 a certain time before client device 902 is about to exit the space shown on the three-dimensional map held by client device 902. For example, client device 902 may issue a three-dimensional map transmission request to server 901 when client device 902 is within a predetermined distance from the boundary of the space shown on the three-dimensional map held by client device 902. Furthermore, if the movement path and speed of client device 902 are known, the time at which client device 902 will exit the space shown on the three-dimensional map held by client device 902 can be predicted based on the known movement path and speed.

[0447] When the error between the three-dimensional data generated by the client device 902 based on the sensor information and the position of the three-dimensional map is greater than a certain range, the client device 902 may send a request to the server 901 to send the three-dimensional map.

[0448] Client device 902 transmits sensor information to server 901 in response to a sensor information transmission request received from server 901. Alternatively, client device 902 may transmit sensor information to server 901 without waiting for a sensor information transmission request from server 901. For example, after receiving a sensor information transmission request from server 901, client device 902 may periodically transmit sensor information to server 901 over a predetermined period. Furthermore, if the position error between three-dimensional data generated by client device 902 based on sensor information and a three-dimensional map received from server 901 is within a predetermined range, client device 902 may determine that the three-dimensional map surrounding client device 902 may have changed, and transmit this determination result along with the sensor information to server 901.

[0449] Server 901 issues a sensor information transmission request to client device 902. For example, server 901 receives location information of client device 902, such as GPS, from client device 902. Based on the location information of client device 902, if server 901 determines that client device 902 is approaching a space with less information on the three-dimensional map managed by server 901, server 901 issues a sensor information transmission request to client device 902 in order to regenerate the three-dimensional map. Alternatively, server 901 may issue a sensor information transmission request to update the three-dimensional map, to check road conditions such as during snow accumulation or disasters, or to check traffic conditions or incidents.

[0450] Furthermore, the client device 902 may set the data volume of the sensor information to be transmitted to the server 901 according to the communication state or bandwidth at the time of receiving the sensor information transmission request received from the server 901. Setting the data volume of the sensor information to be transmitted to the server 901 may include, for example, increasing or decreasing the data itself or selecting an appropriate compression method.

[0451] Figure 29 This is a block diagram illustrating an example configuration of client device 902. Client device 902 receives a three-dimensional map composed of a point cloud or the like from server 901 and estimates its own position based on the three-dimensional data generated based on sensor information from client device 902. Client device 902 then transmits the obtained sensor information to server 901.

[0452] The client device 902 includes: a data receiving unit 1011, a communication unit 1012, a receiving control unit 1013, a format conversion unit 1014, multiple sensors 1015, a three-dimensional data production unit 1016, a three-dimensional image processing unit 1017, a three-dimensional data storage unit 1018, a format conversion unit 1019, a communication unit 1020, a sending control unit 1021, and a data sending unit 1022.

[0453] The data receiving unit 1011 receives a three-dimensional map 1031 from the server 901. The three-dimensional map 1031 is data including a point cloud such as WLD or SWLD. The three-dimensional map 1031 may include either compressed data or uncompressed data.

[0454] The communication unit 1012 communicates with the server 901 and transmits a data transmission request (for example, a transmission request for a three-dimensional map) and the like to the server 901 .

[0455] The reception control unit 1013 exchanges information such as the corresponding format with the communication partner via the communication unit 1012 and establishes communication with the communication partner.

[0456] The format conversion unit 1014 generates a three-dimensional map 1032 by performing format conversion on the three-dimensional map 1031 received by the data receiving unit 1011. Furthermore, if the three-dimensional map 1031 is compressed or encoded, the format conversion unit 1014 performs decompression or decoding. If the three-dimensional map 1031 is uncompressed data, the format conversion unit 1014 does not perform decompression or decoding.

[0457] Multiple sensors 1015 are a group of sensors mounted on client device 902, such as LiDAR, visible light cameras, infrared cameras, or depth sensors, for obtaining information outside the vehicle and generating sensor information 1033. For example, if sensor 1015 is a laser sensor such as LiDAR, sensor information 1033 is three-dimensional data such as a point cloud (point cloud data). Furthermore, there may not be multiple sensors 1015.

[0458] The three-dimensional data generation unit 1016 generates three-dimensional data 1034 of the surrounding area of ​​the own vehicle based on the sensor information 1033. For example, the three-dimensional data generation unit 1016 uses information obtained by LiDAR and visible light images obtained by a visible light camera to generate point cloud data with color information of the surrounding area of ​​the own vehicle.

[0459] The three-dimensional image processing unit 1017 uses the received three-dimensional map 1032, such as a point cloud, and three-dimensional data 1034 of the vehicle's surroundings generated based on the sensor information 1033 to perform processing such as estimating the vehicle's own position. Alternatively, the three-dimensional image processing unit 1017 may synthesize the three-dimensional map 1032 and the three-dimensional data 1034 to create three-dimensional data 1035 of the vehicle's surroundings, and use the created three-dimensional data 1035 to perform processing for estimating the vehicle's own position.

[0460] The three-dimensional data storage unit 1018 stores the three-dimensional map 1032 , the three-dimensional data 1034 , and the three-dimensional data 1035 .

[0461] The format conversion unit 1019 converts the sensor information 1033 into a format compatible with the receiving end to generate sensor information 1037. Furthermore, the format conversion unit 1019 can reduce the data volume by compressing or encoding the sensor information 1037. Furthermore, if format conversion is not necessary, the format conversion unit 1019 can omit the processing. Furthermore, the format conversion unit 1019 can control the amount of data transmitted according to the specified transmission range.

[0462] The communication unit 1020 communicates with the server 901 and receives a data transmission request (a sensor information transmission request) and the like from the server 901 .

[0463] The transmission control unit 1021 exchanges information such as the corresponding format with the communication partner via the communication unit 1020 to establish communication.

[0464] The data transmission unit 1022 transmits the sensor information 1037 to the server 901. The sensor information 1037 includes information obtained by the plurality of sensors 1015, such as information obtained by LiDAR, a brightness image (visible light image) obtained by a visible light camera, an infrared image obtained by an infrared camera, a depth image obtained by a depth sensor, sensor position information, and speed information.

[0465] Next, the configuration of the server 901 will be described. Figure 30 This is a block diagram illustrating an example configuration of server 901. Server 901 receives sensor information transmitted from client device 902 and generates three-dimensional data based on the received sensor information. Server 901 uses the generated three-dimensional data to update the three-dimensional map managed by server 901. Furthermore, server 901 transmits the updated three-dimensional map to client device 902 in response to a request for transmission of the three-dimensional map from client device 902.

[0466] The server 901 includes: a data receiving unit 1111, a communication unit 1112, a receiving control unit 1113, a format conversion unit 1114, a three-dimensional data production unit 1116, a three-dimensional data synthesis unit 1117, a three-dimensional data storage unit 1118, a format conversion unit 1119, a communication unit 1120, a sending control unit 1121, and a data sending unit 1122.

[0467] The data receiving unit 1111 receives sensor information 1037 from the client device 902. The sensor information 1037 includes, for example, information obtained by LiDAR, a brightness image (visible light image) obtained by a visible light camera, an infrared image obtained by an infrared camera, a depth image obtained by a depth sensor, sensor position information, and speed information.

[0468] The communication unit 1112 communicates with the client device 902 and transmits a data transmission request (for example, a sensor information transmission request) and the like to the client device 902 .

[0469] The reception control unit 1113 exchanges information such as the corresponding format with the communication partner via the communication unit 1112 to establish communication.

[0470] If the received sensor information 1037 is compressed or encoded, the format conversion unit 1114 performs decompression or decoding processing to generate the sensor information 1132. If the sensor information 1037 is uncompressed data, the format conversion unit 1114 does not perform decompression or decoding processing.

[0471] The three-dimensional data generation unit 1116 generates three-dimensional data 1134 of the surroundings of the client device 902 based on the sensor information 1132. For example, the three-dimensional data generation unit 1116 uses information obtained by LiDAR and visible light images obtained by a visible light camera to generate point cloud data with color information of the surroundings of the client device 902.

[0472] The three-dimensional data synthesis unit 1117 synthesizes the three-dimensional data 1134 generated based on the sensor information 1132 with the three-dimensional map 1135 managed by the server 901 , thereby updating the three-dimensional map 1135 .

[0473] The three-dimensional data storage unit 1118 stores the three-dimensional map 1135 and the like.

[0474] Format conversion unit 1119 generates three-dimensional map 1031 by converting three-dimensional map 1135 into a format compatible with the receiving end. Furthermore, format conversion unit 1119 can reduce the data volume by compressing or encoding three-dimensional map 1135. Furthermore, format conversion unit 1119 can omit this processing if format conversion is not necessary. Furthermore, format conversion unit 1119 can control the amount of data transmitted according to the specified transmission range.

[0475] The communication unit 1120 communicates with the client device 902 and receives a data transmission request (a request to transmit a three-dimensional map) and the like from the client device 902 .

[0476] The transmission control unit 1121 exchanges information such as the corresponding format with the communication partner via the communication unit 1120 to establish communication.

[0477] The data transmission unit 1122 transmits the three-dimensional map 1031 to the client device 902. The three-dimensional map 1031 is data including a point cloud such as WLD or SWLD. The three-dimensional map 1031 may include either compressed data or uncompressed data.

[0478] Next, the workflow of the client device 902 will be described. Figure 31 This is a flowchart showing the operation performed by the client device 902 when obtaining a three-dimensional map.

[0479] First, the client device 902 requests the server 901 to transmit a three-dimensional map (such as a point cloud) (S1001). At this time, the client device 902 also transmits its location information obtained by GPS, etc., and thus requests the server 901 to transmit a three-dimensional map related to the location information.

[0480] Next, the client device 902 receives a 3D map from the server 901 ( S1002 ). If the received 3D map is compressed data, the client device 902 decodes the received 3D map to generate an uncompressed 3D map ( S1003 ).

[0481] Next, the client device 902 creates three-dimensional data 1034 of the client device 902's surroundings based on the sensor information 1033 obtained from the plurality of sensors 1015 (S1004). Next, the client device 902 estimates its own position using the three-dimensional map 1032 received from the server 901 and the three-dimensional data 1034 created based on the sensor information 1033 (S1005).

[0482] Figure 321037 is a flowchart illustrating the operation of client device 902 when transmitting sensor information. First, client device 902 receives a request to transmit sensor information from server 901 (S1011). Upon receiving the request, client device 902 transmits sensor information 1037 to server 901 (S1012). If sensor information 1033 includes multiple pieces of information obtained by multiple sensors 1015, client device 902 compresses each piece of information using a compression method appropriate for that piece of information, thereby generating sensor information 1037.

[0483] Next, the workflow of the server 901 will be described. Figure 33 This is a flowchart illustrating the operation of server 901 when acquiring sensor information. First, server 901 requests client device 902 to transmit sensor information (S1021). Next, server 901 receives sensor information 1037 transmitted from client device 902 in response to the request (S1022). Next, server 901 uses received sensor information 1037 to create three-dimensional data 1134 (S1023). Next, server 901 reflects the created three-dimensional data 1134 on three-dimensional map 1135 (S1024).

[0484] Figure 34 1 is a flowchart showing the operation of the server 901 when transmitting a three-dimensional map. First, the server 901 receives a request to transmit a three-dimensional map from the client device 902 (S1031). The server 901, having received the request to transmit the three-dimensional map, transmits the three-dimensional map 1031 to the client device 902 (S1032). At this time, the server 901 can extract a three-dimensional map in the vicinity corresponding to the location information of the client device 902 and transmit the extracted three-dimensional map. Furthermore, the server 901 can compress the three-dimensional map composed of the point cloud, for example, using an octree compression method, and transmit the compressed three-dimensional map.

[0485] Hereinafter, modifications of this embodiment will be described.

[0486] Server 901 uses sensor information 1037 received from client device 902 to create three-dimensional data 1134 near the location of client device 902. Next, server 901 matches the created three-dimensional data 1134 with a three-dimensional map 1135 of the same area managed by server 901, calculating the difference between the three-dimensional data 1134 and the three-dimensional map 1135. If the difference exceeds a predetermined threshold, server 901 determines that some abnormality has occurred near client device 902. For example, if ground subsidence occurs due to a natural disaster such as an earthquake, a significant difference may occur between the three-dimensional map 1135 managed by server 901 and the three-dimensional data 1134 created based on the sensor information 1037.

[0487] Sensor information 1037 may also include at least one of the sensor type, sensor performance, and sensor model. Furthermore, a category ID corresponding to the sensor's performance may be added to sensor information 1037. For example, if sensor information 1037 is obtained by LiDAR, an identifier may be assigned based on sensor performance. For example, a sensor capable of obtaining information with an accuracy of several millimeters may be assigned category 1, a sensor capable of obtaining information with an accuracy of several centimeters may be assigned category 2, and a sensor capable of obtaining information with an accuracy of several meters may be assigned category 3. Furthermore, server 901 may estimate sensor performance information based on the model of client device 902. For example, if client device 902 is mounted in a vehicle, server 901 may determine sensor specifications based on the vehicle's model. In this case, server 901 may obtain the vehicle's model information in advance or include this information in the sensor information. Furthermore, server 901 may use the obtained sensor information 1037 to switch the degree of correction applied to three-dimensional data 1134 generated using sensor information 1037. For example, when the sensor performance is high accuracy (category 1), the server 901 does not perform correction on the three-dimensional data 1134. When the sensor performance is low accuracy (category 3), the server 901 applies correction appropriate to the sensor accuracy to the three-dimensional data 1134. For example, the lower the sensor accuracy, the greater the degree (intensity) of correction.

[0488] The server 901 can also simultaneously request sensor information to multiple client devices 902 located in a given space. When the server 901 receives multiple sensor information items from multiple client devices 902, it is not necessary to utilize all of the sensor information in creating the three-dimensional data 1134. For example, the server 901 can select sensor information to be used based on sensor performance. For example, when updating the three-dimensional map 1135, the server 901 can select highly accurate sensor information (category 1) from the multiple sensor information items received and use this information to create the three-dimensional data 1134.

[0489] The server 901 is not limited to servers such as traffic cloud monitoring, and can also be other client devices (vehicle-mounted). Figure 35 The system configuration in this case is shown.

[0490] For example, client device 902C requests sensor information from nearby client device 902A and receives the sensor information from client device 902A. Client device 902C then uses the sensor information received from client device 902A to create three-dimensional data and update the three-dimensional map on client device 902C. In this way, client device 902C can leverage the performance of client device 902C to generate a three-dimensional map of the space available from client device 902A. This is possible, for example, when client device 902C has high performance.

[0491] In this case, client device 902A, which has provided the sensor information, is granted the right to obtain the highly accurate three-dimensional map generated by client device 902C. Client device 902A receives the highly accurate three-dimensional map from client device 902C in accordance with this right.

[0492] Alternatively, client device 902C may request the transmission of sensor information to multiple nearby client devices 902 (client device 902A and client device 902B). If the sensor of client device 902A or client device 902B is high-performance, client device 902C can create three-dimensional data using the sensor information obtained by the high-performance sensor.

[0493] Figure 36 This is a block diagram showing the functional configuration of the server 901 and the client device 902. The server 901 includes, for example, a 3D map compression / decoding processing unit 1201 that compresses and decodes a 3D map, and a sensor information compression / decoding processing unit 1202 that compresses and decodes sensor information.

[0494] The client device 902 includes a three-dimensional map decoding processing unit 1211 and a sensor information compression processing unit 1212. The three-dimensional map decoding processing unit 1211 receives the compressed encoded data of the three-dimensional map, decodes the encoded data, and obtains the three-dimensional map. The sensor information compression processing unit 1212 does not compress the three-dimensional data generated by the obtained sensor information, but instead compresses the sensor information itself and transmits the compressed encoded data of the sensor information to the server 901. With this configuration, the client device 902 can internally retain the processing unit (device or LSI) for decoding the three-dimensional map (point cloud, etc.), without having to internally retain the processing unit for compressing the three-dimensional data of the three-dimensional map (point cloud, etc.). This can reduce the cost and power consumption of the client device 902.

[0495] As described above, the client device 902 according to this embodiment is mounted on a mobile object and generates three-dimensional data 1034 of the mobile object's surroundings based on sensor information 1033 indicating the surrounding conditions of the mobile object obtained by the sensor 1015 mounted on the mobile object. The client device 902 estimates the mobile object's own position using the generated three-dimensional data 1034. The client device 902 transmits the obtained sensor information 1033 to the server 901 or another mobile object 902.

[0496] Based on this, the client device 902 transmits the sensor information 1033 to the server 901, etc. This can potentially reduce the amount of data transmitted compared to when transmitting three-dimensional data. Furthermore, since the client device 902 does not need to perform processing such as compression or encoding of the three-dimensional data, the amount of processing required by the client device 902 can be reduced. Consequently, the client device 902 can reduce the amount of data transmitted and simplify the device configuration.

[0497] Furthermore, the client device 902 sends a request to send a three-dimensional map to the server 901, and receives the three-dimensional map 1031 from the server 901. The client device 902 estimates its own position using the three-dimensional data 1034 and the three-dimensional map 1032.

[0498] Furthermore, the sensor information 1033 includes at least one of information obtained by a laser sensor, a brightness image (visible light image), an infrared image, a depth image, position information of the sensor, and speed information of the sensor.

[0499] Also, the sensor information 1033 includes information showing the performance of the sensor.

[0500] Furthermore, the client device 902 encodes or compresses the sensor information 1033, and transmits the encoded or compressed sensor information 1037 to the server 901 or another mobile object 902. Thus, the client device 902 can reduce the amount of transmitted data.

[0501] For example, the client device 902 includes a processor and a memory, and the processor performs the above-described processing using the memory.

[0502] Furthermore, the server 901 according to this embodiment can communicate with a client device 902 mounted on a mobile object, and receives sensor information 1037 indicating the surrounding conditions of the mobile object obtained by a sensor 1015 mounted on the mobile object from the client device 902. Based on the received sensor information 1037, the server 901 creates three-dimensional data 1134 of the surroundings of the mobile object.

[0503] Based on this, server 901 uses sensor information 1037 transmitted from client device 902 to create three-dimensional data 1134. This potentially reduces the amount of data transmitted compared to when client device 902 transmits the three-dimensional data. Furthermore, since client device 902 does not need to perform processing such as compression or encoding of the three-dimensional data, the processing load on client device 902 can be reduced. This allows server 901 to reduce the amount of data transmitted and simplify the device configuration.

[0504] Furthermore, the server 901 further transmits a request to transmit sensor information to the client device 902 .

[0505] Furthermore, the server 901 further updates the three-dimensional map 1135 using the generated three-dimensional data 1134 , and transmits the three-dimensional map 1135 to the client device 902 in response to a transmission request for the three-dimensional map 1135 from the client device 902 .

[0506] Furthermore, the sensor information 1037 includes at least one of information obtained by a laser sensor, a brightness image (visible light image), an infrared image, a depth image, position information of the sensor, and speed information of the sensor.

[0507] Also, the sensor information 1037 includes information showing the performance of the sensor.

[0508] Furthermore, the server 901 calibrates the three-dimensional data according to the performance of the sensor.

[0509] Furthermore, when receiving sensor information, server 901 receives multiple sensor information 1037 from multiple client devices 902 and selects sensor information 1037 for use in creating three-dimensional data 1134 based on multiple pieces of information indicating sensor performance included in the multiple sensor information 1037. This allows server 901 to improve the quality of three-dimensional data 1134.

[0510] Furthermore, the server 901 decodes or decompresses the received sensor information 1037 and creates three-dimensional data 1134 based on the decoded or decompressed sensor information 1132. In this way, the server 901 can reduce the amount of data to be transmitted.

[0511] For example, the server 901 includes a processor and a memory, and the processor performs the above-mentioned processing using the memory.

[0512] (Implementation 7)

[0513] In this embodiment, a method for encoding and a method for decoding three-dimensional data using inter-frame prediction processing will be described.

[0514] Figure 37 3D data encoding device 1300 according to this embodiment is a block diagram. The 3D data encoding device 1300 encodes 3D data to generate an encoded bit stream (hereinafter also referred to as a bit stream) as an encoded signal. Figure 37 As shown, the three-dimensional data encoding device 1300 includes: a segmentation unit 1301, a subtraction unit 1302, a transformation unit 1303, a quantization unit 1304, an inverse quantization unit 1305, an inverse transformation unit 1306, an addition unit 1307, a reference volume memory 1308, an intra-frame prediction unit 1309, a reference space memory 1310, an inter-frame prediction unit 1311, a prediction control unit 1312, and an entropy coding unit 1313.

[0515] The segmentation unit 1301 segments each space (SPC) contained in the three-dimensional data into a plurality of volumes (VLMs) serving as coding units. Furthermore, the segmentation unit 1301 performs an octree representation (octree conversion) on the voxels within each volume. Alternatively, the segmentation unit 1301 may make the space and the volume the same size and perform an octree representation on the space. Furthermore, the segmentation unit 1301 may append information required for octree conversion (such as depth information) to the header of the bitstream.

[0516] The subtraction unit 1302 calculates the difference between the volume (coding target volume) output from the division unit 1301 and the prediction volume generated by intra prediction or inter prediction described later, and outputs the calculated difference as a prediction residual to the transformation unit 1303 . Figure 38An example of calculating the prediction residual is shown. The bit strings of the encoding target volume and the predicted volume shown here are, for example, position information indicating the positions of three-dimensional points (eg, point clouds) included in the volume.

[0517] The following describes octree representation and the voxel scanning order. The volume is converted to an octree structure (octreeing) and then encoded. The octree structure consists of nodes and leaf nodes. Each node has eight nodes or leaves, and each leaf node contains voxel (VXL) information. Figure 39 An example of the configuration of a volume including a plurality of voxels is shown. Figure 40 Shows the Figure 39 The volume shown is transformed into an example of an octree structure. Figure 40 The leaf nodes 1, 2, and 3 in the leaf nodes shown represent Figure 39 The voxels VXL1, VXL2, and VXL3 shown express the VXL (hereinafter referred to as effective VXL) including the point group.

[0518] For example, an octree is represented by a binary sequence of 0 and 1. For example, when a node or a valid VXL is set to 1 and the others are set to 0, a node or a leaf node is assigned Figure 40 The binary sequence shown in FIG. Then, the binary sequence is scanned in a width-first or depth-first scanning order. For example, in the case of a width-first scan, Figure 41 The binary sequence shown in A. When scanning in depth first, we get Figure 41 The binary sequence obtained by this scanning is encoded by entropy coding, thereby reducing the amount of information.

[0519] Next, we will explain the depth information in the octree representation. The depth in the octree representation is used to control the granularity of the point cloud information contained in the volume. If the depth is set to a large value, the point cloud information can be reproduced at a finer level, but the amount of data used to represent the nodes and leaf nodes will increase. Conversely, if the depth is set to a small value, the amount of data can be reduced, but multiple point cloud information with different positions and colors will be treated as the same position and the same color, thus losing the original information of the point cloud information.

[0520] For example, Figure 42 Shows the Figure 40 The example of an octree with depth = 2 shown is expressed as an octree with depth = 1. Figure 42 The octree shown is Figure 40 The octree shown has a small amount of data. Figure 42 The octree shown is the same as Figure 42Compared to the octree shown, the number of bits after binary serialization is less. Figure 40 The leaf nodes 1 and 2 shown in the figure become Figure 41 The leaf node 1 shown is shown. That is, Figure 40 The leaf node 1 and the leaf node 2 are shown as information in different positions.

[0521] Figure 43 Shown with Figure 42 The volume corresponding to the octree shown. Figure 39 VXL1 and VXL2 shown with Figure 43 In this case, the three-dimensional data encoding device 1300 corresponds to the VXL12 shown. Figure 39 The color information of VXL1 and VXL2 shown here generates Figure 43 For example, the three-dimensional data encoding device 1300 calculates the color information of VXL12 using the average, median, or weighted average of the color information of VXL1 and VXL2 as the color information of VXL12. In this way, the three-dimensional data encoding device 1300 can control the reduction of data volume by changing the depth of the octree.

[0522] The three-dimensional data encoding device 1300 may also set the depth information of the octree using any of world space units, space units, and volume units. Furthermore, in this case, the three-dimensional data encoding device 1300 may also attach the depth information to the world space header information, the space header information, or the volume header information. Furthermore, the same value may be used as depth information in all world spaces, spaces, and volumes at different times. In this case, the three-dimensional data encoding device 1300 may also attach the depth information to the header information that manages the world space at all times.

[0523] When the voxels contain color information, the transformation unit 1303 applies a frequency transform, such as an orthogonal transform, to the prediction residuals of the color information of the voxels within the volume. For example, the transformation unit 1303 scans the prediction residuals in a certain scanning order to create a one-dimensional array. The transformation unit 1303 then applies a one-dimensional orthogonal transform to the created one-dimensional array, thereby transforming the one-dimensional array into the frequency domain. Consequently, when the values ​​of the prediction residuals within the volume are close, the values ​​of the frequency components in the low-frequency band become larger, while the values ​​of the frequency components in the high-frequency band become smaller. Consequently, the quantization unit 1304 can more effectively reduce the amount of code.

[0524] Furthermore, the transform unit 1303 may utilize an orthogonal transform of two or more dimensions instead of a one-dimensional orthogonal transform. For example, the transform unit 1303 maps the prediction residual into a two-dimensional arrangement in a certain scanning order, and applies a two-dimensional orthogonal transform to the obtained two-dimensional arrangement. Furthermore, the transform unit 1303 may select the orthogonal transform method to be used from a plurality of orthogonal transform methods. In this case, the three-dimensional data encoding device 1300 appends information indicating which orthogonal transform method is used to the bitstream. Furthermore, the transform unit 1303 may select the orthogonal transform method to be used from a plurality of orthogonal transform methods with different dimensions. In this case, the three-dimensional data encoding device 1300 appends information indicating which dimensional orthogonal transform method is used to the bitstream.

[0525] For example, the transformation unit 1303 matches the scanning order of the prediction residuals with the scanning order (width-first or depth-first, etc.) in the octree within the volume. Accordingly, since it is not necessary to append information indicating the scanning order of the prediction residuals to the bitstream, it is possible to reduce additional overhead. Furthermore, the transformation unit 1303 may also apply a scanning order different from the scanning order of the octree. In this case, the three-dimensional data encoding device 1300 appends information indicating the scanning order of the prediction residuals to the bitstream. Accordingly, the three-dimensional data encoding device 1300 can efficiently encode the prediction residuals. Furthermore, the three-dimensional data encoding device 1300 may append information indicating whether the scanning order of the octree is applied (a flag, etc.) to the bitstream, and in the case where the scanning order of the octree is not applied, append information indicating the scanning order of the prediction residuals to the bitstream.

[0526] The conversion unit 1303 may convert not only the prediction residual of color information but also other attribute information of the voxel. For example, the conversion unit 1303 may convert and encode information such as reflectivity obtained when a point cloud is acquired by LiDAR or the like.

[0527] The conversion unit 1303 may skip processing when the space does not have attribute information such as color information. Furthermore, the three-dimensional data encoding device 1300 may add information (flag) indicating whether to skip processing by the conversion unit 1303 to the bitstream.

[0528] The quantization unit 1304 quantizes the frequency components of the prediction residual generated by the transformation unit 1303 using the quantization control parameters to generate quantization coefficients. This reduces the amount of information. The generated quantization coefficients are output to the entropy coding unit 1313. The quantization unit 1304 can control the quantization control parameters according to world space units, space units, or volume units. In this case, the three-dimensional data encoding device 1300 attaches the quantization control parameters to the respective header information, etc. In addition, the quantization unit 1304 can also change the weight according to the frequency components of each prediction residual to perform quantization control. For example, the quantization unit 1304 can perform fine quantization on the low-frequency components and coarse quantization on the high-frequency components. In this case, the three-dimensional data encoding device 1300 can attach parameters representing the weights of each frequency component to the header.

[0529] The quantization unit 1304 may skip the process if the space does not have attribute information such as color information. Furthermore, the three-dimensional data encoding apparatus 1300 may add information (flag) indicating whether the process of the quantization unit 1304 is skipped to the bitstream.

[0530] The inverse quantization unit 1305 inversely quantizes the quantization coefficients generated by the quantization unit 1304 using the quantization control parameters, thereby generating inverse quantization coefficients of the prediction residual, and outputs the generated inverse quantization coefficients to the inverse transformation unit 1306 .

[0531] The inverse transform unit 1306 applies inverse transform to the inverse quantized coefficients generated by the inverse quantization unit 1305 to generate a post-inverse transform prediction residual. Since this post-inverse transform prediction residual is a prediction residual generated after quantization, it may not completely match the prediction residual output by the transform unit 1303.

[0532] The adder 1307 adds the inverse-transformed prediction residual generated by the inverse-transformer 1306 to the prediction volume generated by intra-frame prediction or inter-frame prediction, which is used to generate the pre-quantized prediction residual, to generate a reconstructed volume. This reconstructed volume is stored in the reference volume memory 1308 or the reference space memory 1310.

[0533] The intra prediction unit 1309 generates a predicted volume of the encoding target volume using the attribute information of the adjacent volumes stored in the reference volume memory 1308. The attribute information includes the color information or reflectivity of the voxels. The intra prediction unit 1309 generates a predicted value of the color information or reflectivity of the encoding target volume.

[0534] Figure 44 1309 is a diagram for explaining the operation of the intra prediction unit 1309. For example, Figure 44As shown, the intra-frame prediction unit 1309 generates a predicted volume of the encoding target volume (volume idx = 3) based on the adjacent volume (volume idx = 0). Here, volume idx is the identifier information added to the volume in the space, and different values ​​are assigned to each volume. The order of assigning volume idx can be the same as the encoding order or different from the encoding order. For example, as Figure 44 The intra-frame prediction unit 1309 uses the average value of the color information of the voxels contained in volume idx=0, which is the adjacent volume, as the predicted value of the color information of the encoding target volume. In this case, the prediction residual is generated by subtracting the predicted value of the color information from the color information of each voxel contained in the encoding target volume. The processing subsequent to the transformation unit 1303 is performed on this prediction residual. In this case, the three-dimensional data encoding device 1300 also adds the adjacent volume information and prediction mode information to the bitstream. Here, the adjacent volume information is information indicating the adjacent volume used in the prediction, such as the volume idx indicating the adjacent volume used in the prediction. The prediction mode information indicates the mode used to generate the prediction volume. The mode may be, for example, an average value mode that generates a prediction value based on the average value of the voxels in the adjacent volume, or an average value mode that generates a prediction value based on the average value of the voxels in the adjacent volume.

[0535] The intra prediction unit 1309 can also generate a prediction volume based on multiple adjacent volumes. Figure 44 In the illustrated configuration, the intra-frame prediction unit 1309 generates prediction volume 0 based on volume idx = 0, and generates prediction volume 1 based on volume idx = 1. The intra-frame prediction unit 1309 then generates the final prediction volume by averaging the prediction volumes 0 and 1. In this case, the three-dimensional data encoding apparatus 1300 may also append multiple volume idx values ​​used in generating the prediction volumes to the bitstream.

[0536] Figure 45 The inter-frame prediction process involved in this embodiment is schematically shown. The inter-frame prediction unit 1311 performs encoding (inter-frame prediction) on the space (SPC) at a certain time T_Cur using the coded space at a different time T_LX. In this case, the inter-frame prediction unit 1311 applies rotation and translation processing to the coded space at different times T_LX to perform encoding.

[0537] Furthermore, the 3D data encoding apparatus 1300 appends RT information related to the spatial rotation and translation processing applied at a different time T_LX to the bitstream. For example, the different time T_LX is a time T_L0 prior to the certain time T_Cur. In this case, the 3D data encoding apparatus 1300 may also append RT information RT_L0 related to the spatial rotation and translation processing applied at time T_L0 to the bitstream.

[0538] Alternatively, the different time T_LX is, for example, time T_L1 after the certain time T_Cur. In this case, the three-dimensional data encoding apparatus 1300 may add RT information RT_L1 regarding the spatial rotation and translation processing applied at time T_L1 to the bitstream.

[0539] Alternatively, the inter-frame prediction unit 1311 performs encoding by referring to spaces at different times T_L0 and T_L1 (bi-prediction). In this case, the 3D data encoding apparatus 1300 may add RT information RT_L0 and RT_L1 regarding spatial rotation and translation, respectively, to the bitstream.

[0540] In addition, although T_L0 is set as the time before T_Cur and T_L1 is set as the time after T_Cur, the present invention is not limited to this. For example, T_L0 and T_L1 can both be the time before T_Cur. Alternatively, T_L0 and T_L1 can both be the time after T_Cur.

[0541] Furthermore, when encoding with reference to multiple spaces at different times, the three-dimensional data encoding device 1300 may append RT information related to the rotation and translation applied to each space to the bitstream. For example, the three-dimensional data encoding device 1300 manages the multiple referenced encoded spaces using two reference lists (L0 list and L1 list). When the first reference space in the L0 list is set to L0R0, the second reference space in the L0 list is set to L0R1, the first reference space in the L1 list is set to L1R0, and the second reference space in the L1 list is set to L1R1, the three-dimensional data encoding device 1300 appends RT information RT_L0R0 for L0R0, RT information RT_L0R1 for L0R1, RT information RT_L1R0 for L1R0, and RT information RT_L1R1 for L1R1 to the bitstream. For example, the three-dimensional data encoding device 1300 appends this RT information to the header of the bitstream.

[0542] Alternatively, when encoding with reference to multiple reference spaces at different times, the three-dimensional data encoding device 1300 may determine whether rotation and translation are applicable for each reference space. In this case, the three-dimensional data encoding device 1300 may append information indicating whether rotation and translation are applicable for each reference space (such as an RT applicable flag) to header information of the bitstream. For example, the three-dimensional data encoding device 1300 calculates RT information and an ICP error value using an ICP (Interactive Closest Point) algorithm for each reference space to be referenced based on the encoding target space. If the ICP error value is below a predetermined value, the three-dimensional data encoding device 1300 determines that rotation and translation are not necessary and sets the RT applicable flag to OFF (invalid). Furthermore, if the ICP error value is greater than the predetermined value, the three-dimensional data encoding device 1300 sets the RT applicable flag to ON (valid) and appends the RT information to the bitstream.

[0543] Figure 46 This section shows an example of a syntax for appending RT information and an RT applicable flag to a header. Furthermore, the number of bits allocated to each syntax can be determined based on the range of values ​​that the syntax can take. For example, if the number of reference spaces included in reference list L0 is 8, 3 bits can be allocated to MaxRefSpc_l0. The number of allocated bits can be varied based on the values ​​that each syntax can take, or can be fixed regardless of the values ​​that can be taken. When the number of allocated bits is fixed, the three-dimensional data encoding device 1300 can append this fixed number of bits to other header information.

[0544] Here, Figure 46 MaxRefSpc_10 indicates the number of reference spaces included in reference list L0. RT_flag_10[i] is the RT application flag for reference space i in reference list L0. When RT_flag_10[i] is 1, rotation and translation are applied to reference space i. When RT_flag_10[i] is 0, rotation and translation are not applied to reference space i.

[0545] R_l0[i] and T_l0[i] are the RT information for reference space i within reference list L0. R_l0[i] is the rotation information for reference space i within reference list L0. The rotation information indicates the content of the applied rotation processing, such as a rotation matrix or quaternion. T_l0[i] is the translation information for reference space i within reference list L0. The translation information indicates the content of the applied translation processing, such as a translation vector.

[0546] MaxRefSpc_l1 indicates the number of reference spaces included in reference list L1. RT_flag_l1[i] is the RT application flag for reference space i in reference list L1. When RT_flag_l1[i] is 1, rotation and translation are applied to reference space i. When RT_flag_l1[i] is 0, rotation and translation are not applied to reference space i.

[0547] R_l1[i] and T_l1[i] are the RT information for reference space i within reference list L1. R_l1[i] is the rotation information for reference space i within reference list L1. The rotation information indicates the content of the applied rotation processing, such as a rotation matrix or quaternion. T_l1[i] is the translation information for reference space i within reference list L1. The translation information indicates the content of the applied translation processing, such as a translation vector.

[0548] The inter-frame prediction unit 1311 generates a predicted volume for the target volume using information about the already-encoded reference space stored in the reference space memory 1310. As described above, before generating the predicted volume for the target volume, the inter-frame prediction unit 1311 uses the ICP (Interactive Closest Point) algorithm to determine RT information in the target space and the reference space to approximate the overall positional relationship between the target space and the reference space. The inter-frame prediction unit 1311 then uses the determined RT information to rotate and translate the reference space, thereby obtaining reference space B. Subsequently, the inter-frame prediction unit 1311 generates a predicted volume for the target volume within the target space using information within reference space B. The three-dimensional data encoding device 1300 appends the RT information used to obtain reference space B to, for example, header information of the target space.

[0549] In this way, the inter-frame prediction unit 1311 applies rotation and translation processing to the reference space, thereby approximating the overall positional relationship between the encoding target space and the reference space, and then uses the information in the reference space to generate a prediction volume. This improves the accuracy of the prediction volume. In addition, since the prediction residual can be suppressed, the amount of coding can be reduced. In addition, although an example of performing ICP using the encoding target space and the reference space is shown here, the present invention is not limited to this. For example, in order to reduce the processing amount, the inter-frame prediction unit 1311 may also perform ICP using at least one of the encoding target space from which the number of voxels or point clouds is extracted, and the reference space from which the number of voxels or point clouds is extracted, to obtain RT information.

[0550] Furthermore, if the ICP error value obtained from the ICP result is less than a predetermined first threshold, that is, for example, if the positional relationship between the encoding target space and the reference space is close, the inter-frame prediction unit 1311 may determine that rotation and translation are unnecessary and not perform rotation and translation. In this case, the three-dimensional data encoding device 1300 does not need to add RT information to the bitstream, thereby reducing overhead.

[0551] Furthermore, if the ICP error value exceeds a predetermined second threshold, the inter-frame prediction unit 1311 determines that the spatial shape change is significant and may apply intra-frame prediction to all volumes in the encoding target space. Hereinafter, the space to which intra-frame prediction is applied is referred to as the intra-frame space. Furthermore, the second threshold is a value greater than the first threshold described above. Furthermore, the method is not limited to ICP; any method that can determine RT information from two voxel sets or two point cloud sets may be applied.

[0552] Furthermore, when the three-dimensional data contains attribute information such as shape or color, the inter-frame prediction unit 1311 searches, for example, for a volume in the reference space that is closest to the shape or color attribute information of the target volume as a prediction volume for the target volume within the target space. Furthermore, this reference space is, for example, a reference space that has been subjected to the aforementioned rotation and translation processing. The inter-frame prediction unit 1311 generates a prediction volume based on the volume (reference volume) obtained through the search. Figure 47 is a diagram for explaining the generation of the prediction volume. Figure 47 When encoding the target volume (volume idx = 0) using inter-frame prediction, the reference volumes within the reference space are sequentially scanned, searching for the volume with the smallest prediction residual, the difference between the target volume and the reference volume. The inter-frame prediction unit 1311 selects the volume with the smallest prediction residual as the prediction volume. The prediction residual between the target volume and the prediction volume is encoded by the transformation unit 1303 and subsequent processing. Here, the prediction residual refers to the difference between the attribute information of the target volume and the attribute information of the prediction volume. Furthermore, the 3D data encoding device 1300 appends the volume idx of the reference volume within the reference space used as the prediction volume to the header of the bitstream.

[0553] exist Figure 47 In the example shown, the reference volume idx=4 in the reference space L0R0 is selected as the prediction volume of the encoding target volume. Then, the prediction residual between the encoding target volume and the reference volume and the reference volume idx=4 are encoded and added to the bitstream.

[0554] In addition, although the description here is given of an example of generating a predicted volume of attribute information, the same processing can be performed on a predicted volume of position information.

[0555] The prediction control unit 1312 controls whether intra-frame prediction or inter-frame prediction is used to encode the target volume. A mode that includes both intra-frame prediction and inter-frame prediction is referred to as a prediction mode. For example, the prediction control unit 1312 calculates an evaluation value using the prediction residual when the target volume is predicted using intra-frame prediction and the prediction residual when the target volume is predicted using inter-frame prediction, and selects the prediction mode with the lower evaluation value. Alternatively, the prediction control unit 1312 may apply orthogonal transform, quantization, and entropy coding to the prediction residual for intra-frame prediction and the prediction residual for inter-frame prediction, respectively, to calculate the actual code amount and use the calculated code amount as the evaluation value to select the prediction mode. Furthermore, overhead information other than the prediction residual (such as reference volume IDX information) may be added to the evaluation value. Furthermore, if the target space is predetermined to be encoded in intra-frame space, the prediction control unit 1312 may normally select intra-frame prediction.

[0556] The entropy coding unit 1313 generates a coded signal (coded bit stream) by performing variable-length coding on the quantized coefficients input from the quantization unit 1304. Specifically, the entropy coding unit 1313 binarizes the quantized coefficients and performs arithmetic coding on the resulting binary signal.

[0557] Next, a three-dimensional data decoding device that decodes the encoded signal generated by the three-dimensional data encoding device 1300 will be described. Figure 48 This is a block diagram of a three-dimensional data decoding device 1400 according to this embodiment. The three-dimensional data decoding device 1400 includes an entropy decoding unit 1401, an inverse quantization unit 1402, an inverse transform unit 1403, an addition unit 1404, a reference volume memory 1405, an intra-frame prediction unit 1406, a reference space memory 1407, an inter-frame prediction unit 1408, and a prediction control unit 1409.

[0558] The entropy decoding unit 1401 performs variable-length decoding on the coded signal (coded bit stream). For example, the entropy decoding unit 1401 performs arithmetic decoding on the coded signal to generate a binary signal, and generates quantization coefficients based on the generated binary signal.

[0559] The inverse quantization unit 1402 performs inverse quantization on the quantized coefficients input from the entropy decoding unit 1401 using a quantization parameter added to a bit stream or the like, thereby generating inverse quantized coefficients.

[0560] The inverse transform unit 1403 generates a prediction residual by performing an inverse transform on the inverse quantized coefficients input from the inverse quantization unit 1402. For example, the inverse transform unit 1403 generates a prediction residual by performing an inverse orthogonal transform on the inverse quantized coefficients based on information added to the bitstream.

[0561] The adder 1404 adds the prediction residual generated by the inverse transform unit 1403 to the prediction volume generated by intra-frame prediction or inter-frame prediction to generate a reconstructed volume. This reconstructed volume is output as decoded 3D data and stored in the reference volume memory 1405 or the reference space memory 1407.

[0562] The intra prediction unit 1406 generates a prediction volume through intra prediction using the reference volume in the reference volume memory 1405 and information appended to the bitstream. Specifically, the intra prediction unit 1406 obtains prediction mode information and neighboring volume information (e.g., volume idx) appended to the bitstream, and generates a prediction volume using the neighboring volumes indicated by the neighboring volume information using the mode indicated by the prediction mode information. The details of this processing are the same as those of the intra prediction unit 1309 described above, except that the information appended to the bitstream is used.

[0563] The inter-frame prediction unit 1408 generates a prediction volume through inter-frame prediction using the reference space in the reference space memory 1407 and the information attached to the bitstream. Specifically, the inter-frame prediction unit 1408 uses the RT information of each reference space attached to the bitstream to apply rotation and translation processing to the reference space, and generates a prediction volume using the applied reference space. In addition, when the RT application flag of each reference space is present in the bitstream, the inter-frame prediction unit 1408 applies rotation and translation processing to the reference space according to the RT application flag. In addition, the details of the above-mentioned processing are the same as those of the above-mentioned inter-frame prediction unit 1311, except that the information attached to the bitstream is used.

[0564] Whether the decoding target volume is decoded using intra-frame prediction or inter-frame prediction is controlled by the prediction control unit 1409. For example, the prediction control unit 1409 selects intra-frame prediction or inter-frame prediction based on information appended to the bitstream indicating the prediction mode to be used. Alternatively, if the decoding target space is predetermined to be decoded as an intra-frame space, the prediction control unit 1409 may normally select intra-frame prediction.

[0565] The following describes variations of this embodiment. While this embodiment describes the application of rotation and translation using spatial units as an example, rotation and translation can also be applied using finer units. For example, the three-dimensional data encoding device 1300 can divide the space into subspaces and apply rotation and translation in subspace units. In this case, the three-dimensional data encoding device 1300 generates RT information for each subspace and appends the generated RT information to the header of the bitstream. Alternatively, the three-dimensional data encoding device 1300 can apply rotation and translation using volume units as encoding units. In this case, the three-dimensional data encoding device 1300 generates RT information in encoded volume units and appends the generated RT information to the header of the bitstream. Furthermore, these methods can be combined. That is, the three-dimensional data encoding device 1300 can apply rotation and translation in large units and then apply rotation and translation in finer units. For example, the three-dimensional data encoding device 1300 can apply rotation and translation in spatial units and then apply different rotations and translations to each of the multiple volumes contained in the resulting space.

[0566] Furthermore, while this embodiment illustrates the application of rotation and translation to the reference space as an example, this is not limiting. For example, the three-dimensional data encoding device 1300 may apply scaling to change the size of the three-dimensional data. Furthermore, the three-dimensional data encoding device 1300 may also apply any one or both of rotation, translation, and scaling. Furthermore, as described above, when processing is applied in multiple stages using different units, the type of processing applied to each unit may be different. For example, rotation and translation may be applied to spatial units, while translation may be applied to volume units.

[0567] Note that these modifications are also applicable to the three-dimensional data decoding apparatus 1400 .

[0568] As described above, the three-dimensional data encoding device 1300 according to this embodiment performs the following processing. Figure 48 This is a flowchart of the inter-frame prediction process performed by the three-dimensional data encoding device 1300.

[0569] First, the 3D data encoding device 1300 generates predicted position information (e.g., predicted volume) using position information of 3D points included in target 3D data (e.g., encoding target space) and reference 3D data at different times (e.g., reference space) (S1301). Specifically, the 3D data encoding device 1300 generates predicted position information by applying rotation and translation processing to the position information of 3D points included in the reference 3D data.

[0570] Furthermore, the 3D data encoding device 1300 performs rotation and translation processing using a first unit (e.g., space) and generates predicted position information using a second unit (e.g., volume) that is smaller than the first unit. For example, the 3D data encoding device 1300 may search for a volume in the reference space after rotation and translation that minimizes the difference between the encoding target volume and the position information, and use the resulting volume as the predicted volume. Alternatively, the 3D data encoding device 1300 may perform rotation and translation processing and generation of predicted position information using the same unit.

[0571] Furthermore, the three-dimensional data encoding device 1300 may apply a first rotation and translation process to the position information of the three-dimensional points contained in the reference three-dimensional data using a first unit (e.g., space), and apply a second rotation and translation process to the position information of the three-dimensional points obtained by the first rotation and translation process using a second unit (e.g., volume) that is smaller than the first unit, thereby generating predicted position information.

[0572] Here, the position information of the three-dimensional point and the predicted position information are as follows: Figure 41 As shown, the octree structure is used for representation. For example, the 3D point position information and predicted position information are represented in a depth-first scan order of the octree structure. Alternatively, the 3D point position information and predicted position information are represented in a depth-first scan order of the octree structure.

[0573] And, as Figure 46 As shown, the three-dimensional data encoding device 1300 encodes an RT applicable flag indicating whether rotation and translation processing is applied to the position information of the three-dimensional point included in the reference three-dimensional data. That is, the three-dimensional data encoding device 1300 generates a coded signal (coded bit stream) including the RT applicable flag. Furthermore, the three-dimensional data encoding device 1300 encodes RT information indicating the content of the rotation and translation processing. That is, the three-dimensional data encoding device 1300 generates a coded signal (coded bit stream) including RT information. Alternatively, the three-dimensional data encoding device 1300 may encode the RT information when the RT applicable flag indicates that rotation and translation processing is applied, and may not encode the RT information when the RT applicable flag indicates that rotation and translation processing is not applied.

[0574] The 3D data includes, for example, position information of 3D points and attribute information (color information, etc.) of each 3D point. The 3D data encoding device 1300 generates predicted attribute information by using the attribute information of the 3D points included in the reference 3D data (S1302).

[0575] Next, the three-dimensional data encoding device 1300 uses the predicted position information to encode the position information of the three-dimensional points included in the object three-dimensional data. Figure 38 As shown, differential position information, which is a difference between the position information of the three-dimensional point included in the target three-dimensional data and the predicted position information, is calculated (S1303).

[0576] The 3D data encoding device 1300 then uses the predicted attribute information to encode the attribute information of the 3D points included in the target 3D data. For example, the 3D data encoding device 1300 calculates differential attribute information, which is the difference between the attribute information of the 3D points included in the target 3D data and the predicted attribute information (S1304). The 3D data encoding device 1300 then transforms and quantizes the calculated differential attribute information (S1305).

[0577] Finally, the 3D data encoding device 1300 encodes (eg, entropy encodes) the differential position information and the quantized differential attribute information ( S1306 ). That is, the 3D data encoding device 1300 generates an encoded signal (encoded bitstream) including the differential position information and the differential attribute information.

[0578] If the 3D data does not contain attribute information, the 3D data encoding device 1300 may not perform steps S1302, S1304, and S1305. Furthermore, the 3D data encoding device 1300 may only encode the 3D point position information or the 3D point attribute information.

[0579] and, Figure 49 The order of processing shown is merely an example and is not intended to be limiting. For example, since the processing of location information (S1301, S1303) and the processing of attribute information (S1302, S1304, S1305) are independent of each other, they can be executed in any order, or some of them can be processed in parallel.

[0580] As described above, in this embodiment, the 3D data encoding device 1300 generates predicted position information using the position information of 3D points contained in the target 3D data and reference 3D data at different times. The device then encodes the difference between the position information of the 3D points contained in the target 3D data and the predicted position information, i.e., the differential position information. This reduces the amount of data in the encoded signal, thereby improving encoding efficiency.

[0581] Furthermore, in this embodiment, the 3D data encoding device 1300 generates predicted attribute information by using attribute information of 3D points contained in the reference 3D data. The device then encodes the difference between the attribute information of the 3D points contained in the target 3D data and the predicted attribute information, i.e., the difference attribute information. This reduces the amount of data in the encoded signal, thereby improving encoding efficiency.

[0582] For example, the three-dimensional data encoding device 1300 includes a processor and a memory, and the processor performs the above-mentioned processing using the memory.

[0583] Figure 48 This is a flowchart of the inter-frame prediction process performed by the three-dimensional data decoding apparatus 1400 .

[0584] First, the three-dimensional data decoding apparatus 1400 decodes (eg, performs entropy decoding) the differential position information and the differential attribute information based on the encoded signal (encoded bit stream) ( S1401 ).

[0585] Furthermore, the 3D data decoding apparatus 1400 decodes, based on the coded signal, an RT application flag indicating whether rotation and translation processing is applied to the position information of the 3D points included in the reference 3D data. Furthermore, the 3D data decoding apparatus 1400 decodes RT information indicating the content of the rotation and translation processing. Furthermore, the 3D data decoding apparatus 1400 decodes the RT information if the RT application flag indicates that rotation and translation processing is applied, and does not decode the RT information if the RT application flag indicates that rotation and translation processing is not applied.

[0586] Next, the three-dimensional data decoding apparatus 1400 performs inverse quantization and inverse transformation on the decoded differential attribute information ( S1402 ).

[0587] Next, the 3D data decoding device 1400 generates predicted position information (e.g., predicted volume) using the position information of 3D points included in the target 3D data (e.g., the decoding target space) and the reference 3D data at different times (e.g., the reference space) (S1403). Specifically, the 3D data decoding device 1400 generates the predicted position information by applying rotation and translation processing to the position information of 3D points included in the reference 3D data.

[0588] More specifically, when the RT application flag indicates that rotation and translation processing is applicable, the three-dimensional data decoding apparatus 1400 applies the rotation and translation processing to the position information of the three-dimensional points included in the reference three-dimensional data indicated by the RT information. Furthermore, when the RT application flag indicates that rotation and translation processing is not applicable, the three-dimensional data decoding apparatus 1400 does not apply the rotation and translation processing to the position information of the three-dimensional points included in the reference three-dimensional data.

[0589] Furthermore, the three-dimensional data decoding apparatus 1400 may perform rotation and translation processing in a first unit (e.g., space), and may generate predicted position information in a second unit (e.g., volume) that is smaller than the first unit. Furthermore, the three-dimensional data decoding apparatus 1400 may perform rotation and translation processing and generation of predicted position information in the same unit.

[0590] Furthermore, the three-dimensional data decoding device 1400 may apply a first rotation and translation process using a first unit (e.g., space) to the position information of the three-dimensional points contained in the reference three-dimensional data, and may apply a second rotation and translation process using a second unit (e.g., volume) that is smaller than the first unit to the position information of the three-dimensional points obtained by the first rotation and translation process, thereby generating predicted position information.

[0591] Here, the position information of the three-dimensional point and the predicted position information are as follows: Figure 41 As shown, the octree structure is used for representation. For example, the 3D point position information and predicted position information are represented in a depth-first scan order between depth and width in the octree structure. Alternatively, the 3D point position information and predicted position information are represented in a depth-first scan order between depth and width in the octree structure.

[0592] The three-dimensional data decoding apparatus 1400 generates predicted attribute information using attribute information of three-dimensional points included in the reference three-dimensional data ( S1404 ).

[0593] Next, the 3D data decoding device 1400 decodes the encoded position information contained in the encoded signal using the predicted position information, thereby restoring the position information of the 3D points contained in the target 3D data. Here, the encoded position information is, for example, differential position information, and the 3D data decoding device 1400 restores the position information of the 3D points contained in the target 3D data by adding the differential position information to the predicted position information (S1405).

[0594] The 3D data decoding device 1400 then decodes the coded attribute information contained in the coded signal using the predicted attribute information, thereby restoring the attribute information of the 3D points contained in the target 3D data. Here, the coded attribute information is, for example, differential attribute information, and the 3D data decoding device 1400 restores the attribute information of the 3D points contained in the target 3D data by adding the differential attribute information to the predicted attribute information (S1406).

[0595] Alternatively, if the 3D data does not include attribute information, the 3D data decoding apparatus 1400 may not execute steps S1402, S1404, and S1406. Furthermore, the 3D data decoding apparatus 1400 may only decode the 3D point position information or the 3D point attribute information.

[0596] and, Figure 50 The order of processing shown is an example and is not limiting. For example, since the processing of location information (S1403, S1405) and the processing of attribute information (S1402, S1404, S1406) are independent of each other, they can be performed in any order, and some of them can be processed in parallel.

[0597] (Implementation 8)

[0598] 3D point group information includes position information (geometry) and attribute information (attribute). Position information includes coordinates (x, y, and z) relative to a particular point. When encoding position information, instead of directly encoding the coordinates of each 3D point, an octree representation is used to represent the position of each 3D point and encode the octree information, thereby reducing the amount of code.

[0599] On the other hand, attribute information includes information representing color information (RGB, YUV, etc.), reflectivity, normal vector, etc. of each 3D point. For example, the 3D data encoding device can encode attribute information using a different encoding method from that for position information.

[0600] In this embodiment, a method for encoding attribute information is described. In addition, in this embodiment, integer values ​​are used as the values ​​of the attribute information for description. For example, when each color component of the color information RGB or YUV has 8-bit precision, each color component takes an integer value from 0 to 255. When the reflectivity value has 10-bit precision, the reflectivity value takes an integer value from 0 to 1023. In addition, when the bit precision of the attribute information is decimal precision, the three-dimensional data encoding device may also multiply the value by a scaling value and round it to an integer value so that the value of the attribute information becomes an integer value. In addition, the three-dimensional data encoding device may also add the scaling value to the header of the bit stream, etc.

[0601] One possible method for encoding attribute information for a 3D point involves calculating a predicted value for the attribute information and encoding the difference between the original attribute information value and the predicted value (prediction residual). For example, if the attribute information value of a 3D point p is Ap and the predicted value is Pp, the 3D data encoding device encodes the absolute difference Diffp = |Ap-Pp|. In this case, if the predicted value Pp can be generated with high accuracy, the absolute difference Diffp becomes smaller. Therefore, entropy encoding the absolute difference Diffp using a coding table that generates fewer bits as the value decreases can reduce the amount of code.

[0602] As a method for generating predicted values ​​for attribute information, one approach is to use attribute information from other 3D points located around the target 3D point being encoded, namely reference 3D points. Reference 3D points are 3D points within a predetermined distance range from the target 3D point. For example, given a target 3D point p = (x1, y1, z1) and a 3D point q = (x2, y2, z2), the 3D data encoding device calculates the Euclidean distance d(p, q) between the 3D points p and q, as shown in (Equation A1).

[0603]

Formula 1

[0604]

[0605] When the Euclidean distance d(p, q) is less than a predetermined threshold THd, the 3D data encoding device determines that the position of the 3D point q is close to the position of the target 3D point p, and determines to use the value of the attribute information of the 3D point q in generating the predicted value of the attribute information of the target 3D point p. Alternatively, the distance calculation method may be another method, such as the Mahalanobis distance. Alternatively, the 3D data encoding device may determine not to use 3D points outside a predetermined distance range from the target 3D point for prediction. For example, if a 3D point r exists and the distance d(p, r) between the target 3D point p and 3D point r is greater than a threshold THd, the 3D data encoding device may determine not to use 3D point r for prediction. Alternatively, the 3D data encoding device may append information indicating the threshold THd to a header of the bitstream.

[0606] Figure 51 This figure shows an example of 3D points. In this example, the distance d(p, q) between the target 3D point p and the 3D point q is less than a threshold value THd. Therefore, the 3D data encoding device determines that the 3D point q is a reference 3D point for the target 3D point p and uses the value of the attribute information Aq of the 3D point q in generating the predicted value Pp of the attribute information Ap of the target 3D point p.

[0607] On the other hand, the distance d(p, r) between the target 3D point p and the 3D point r is greater than the threshold value THd. Therefore, the 3D data encoding device determines that the 3D point r is not a reference 3D point for the target 3D point p and determines not to use the value of the attribute information Ar of the 3D point r in generating the predicted value Pp of the attribute information Ap of the target 3D point p.

[0608] Furthermore, when using a predicted value to encode the attribute information of a target 3D point, the 3D data encoding device utilizes a 3D point whose attribute information has already been encoded and decoded as a reference 3D point. Similarly, when using a predicted value to decode the attribute information of a target 3D point to be decoded, the 3D data decoding device utilizes a 3D point whose attribute information has already been decoded as a reference 3D point. This allows the generation of the same predicted value during encoding and decoding, enabling the decoding side to accurately decode the bitstream of the 3D point generated by encoding.

[0609] In addition, when encoding the attribute information of a 3D point, it is considered to classify each 3D point into multiple levels using the position information of the 3D point and then encode it. Here, each level after classification is called LoD (Level of Detail). Figure 52 The method of generating LoD is described.

[0610] First, the 3D data encoding device selects an initial point a0 and assigns it to LoD0. Next, the 3D data encoding device extracts a point a1 whose distance from point a0 is greater than the LoD0 threshold Thres_LoD[0] and assigns it to LoD0. Next, the 3D data encoding device extracts a point a2 whose distance from point a1 is greater than the LoD0 threshold Thres_LoD[0] and assigns it to LoD0. In this way, the 3D data encoding device constructs LoD0 such that the distance between each point in LoD0 is greater than the threshold Thres_LoD[0].

[0611] Next, the 3D data encoding device selects point b0, which has not yet been assigned a LoD, and assigns it to LoD1. Next, the 3D data encoding device extracts point b1, which has not yet been assigned a LoD, and whose distance from point b0 is greater than the LoD1 threshold Thres_LoD[1], and which has not yet been assigned a LoD, and assigns it to LoD1. Next, the 3D data encoding device extracts point b2, which has not yet been assigned a LoD, and whose distance from point b1 is greater than the LoD1 threshold Thres_LoD[1], and which has not yet been assigned a LoD, and assigns it to LoD1. In this way, the 3D data encoding device constructs LoD1 so that the distance between each point in LoD1 is greater than the threshold Thres_LoD[1].

[0612] Next, the three-dimensional data encoding device selects point c0 that has not yet been assigned a LoD, and assigns it to LoD2. Next, the three-dimensional data encoding device extracts point c1 that has not yet been assigned a LoD and whose distance from point c0 is greater than the threshold value Thres_LoD[2] of LoD2, and which has not yet been assigned a LoD, and assigns it to LoD2. Next, the three-dimensional data encoding device extracts point c2 that has not yet been assigned a LoD and whose distance from point c1 is greater than the threshold value Thres_LoD[2] of LoD2, and which has not yet been assigned a LoD, and assigns it to LoD2. In this way, the three-dimensional data encoding device constructs LoD2 in such a way that the distance between each point in LoD2 is greater than the threshold value Thres_LoD[2]. For example, Figure 53 As shown, the threshold values ​​of each LoD, Thres_LoD[0], Thres_LoD[1], and Thres_LoD[2], are set.

[0613] In addition, the three-dimensional data encoding device may also add information indicating the threshold value of each LoD to the header of the bit stream. Figure 53 In the case of the example shown, the three-dimensional data encoding device may add threshold values ​​Thres_LoD[0], Thres_LoD[1], and Thres_LoD[2] to the header.

[0614] Alternatively, the 3D data encoding device may assign all 3D points that are not assigned LoD to the lowest layer of LoD. In this case, the 3D data encoding device can reduce the coding amount of the header by not adding the threshold of the lowest layer of LoD to the header. For example, Figure 53 In the example shown, the 3D data encoding device adds thresholds Thres_LoD[0] and Thres_LoD[1] to the header, but does not add Thres_LoD[2]. In this case, the 3D data decoding device may estimate the value of Thres_LoD[2] to be 0. Alternatively, the 3D data encoding device may add the number of LoD levels to the header. This allows the 3D data decoding device to determine the LoD of the lowest layer using the number of LoD levels.

[0615] In addition, if Figure 53 As shown in FIG, the threshold value of each LoD layer is set to be larger as it is closer to the upper layer, so that the upper layer (the layer closer to LoD0) becomes a sparse point group with a long distance between three-dimensional points, and the lower layer becomes a dense point group with a short distance between three-dimensional points. Figure 53 In the example shown, LoD0 is the highest layer.

[0616] Furthermore, the method for selecting the initial 3D points when setting each LoD can also depend on the encoding order used when encoding the position information. For example, the 3D data encoding device selects the first 3D point encoded during position information encoding as the initial point a0 of LoD0. Using initial point a0 as a base point, points a1 and a2 are selected to form LoD0. Furthermore, the 3D data encoding device can select the 3D point whose position information was encoded earliest among the 3D points not belonging to LoD0 as the initial point b0 of LoD1. In other words, the 3D data encoding device can select the 3D point whose position information was encoded earliest among the 3D points in the upper layer (LoD0 to LoDn-1) that do not belong to LoDn as the initial point n0 of LoDn. Thus, by using the same initial point selection method during decoding, the 3D data decoding device can construct the same LoD as during encoding, thereby enabling proper decoding of the bitstream. Specifically, the 3D data decoding device selects the 3D point whose position information was decoded earliest among the 3D points in the upper layer that do not belong to LoDn as the initial point n0 of LoDn.

[0617] The following describes a method for generating predicted values ​​for 3D point attribute information using LoD information. For example, when encoding the 3D points included in LoD0, the 3D data encoding device uses the encoded and decoded (hereinafter referred to as "encoded") attribute information included in LoD0 and LoD1 to generate the target 3D point included in LoD1. In this way, the 3D data encoding device uses the encoded attribute information included in LoDn' (n' <= n) to generate predicted values ​​for the attribute information of the 3D points included in LoDn. In other words, the 3D data encoding device does not use the attribute information of the 3D points included in layers below LoDn when calculating the predicted values ​​for the attribute information of the 3D points included in LoDn.

[0618] For example, the 3D data encoding device generates a predicted value for attribute information of a 3D point by calculating the average of the attribute values ​​of up to N 3D points among the already encoded 3D points surrounding the target 3D point. Furthermore, the 3D data encoding device may append the value of N to a header of the bitstream, for example. Alternatively, the 3D data encoding device may change the value of N for each 3D point, appending the value of N to each 3D point. This allows for selecting an appropriate value of N for each 3D point, thereby improving the accuracy of the predicted value and reducing the prediction residual. Alternatively, the 3D data encoding device may append the value of N to the header of the bitstream, fixing the value of N within the bitstream. This eliminates the need to encode or decode the value of N for each 3D point, thereby reducing the amount of processing. Alternatively, the 3D data encoding device may encode the value of N for each LoD. By selecting an appropriate value of N for each LoD, encoding efficiency can be improved.

[0619] Alternatively, the 3D data encoding device may calculate the predicted value of the attribute information of the 3D point by taking a weighted average of the attribute information of N surrounding 3D points that have already been encoded. For example, the 3D data encoding device may calculate the weight using the distance information between the target 3D point and each of the N surrounding 3D points.

[0620] When a 3D data encoding device encodes the value of N for each LoD, for example, the value of N is set to a larger value for higher LoD layers and a smaller value for lower LoD layers. In higher LoD layers, the distances between 3D points belonging to that layer are greater. Therefore, it is possible to improve prediction accuracy by setting N to a large value and selecting multiple surrounding 3D points for averaging. Furthermore, in lower LoD layers, the distances between 3D points belonging to that layer are closer, making it possible to achieve efficient prediction while reducing the amount of averaging processing by setting N to a small value.

[0621] Figure 54 This is a diagram showing an example of attribute information used in the prediction value. As described above, the predicted value of the point P included in LoDN' (N' <= N) is generated using the encoded surrounding points P' included in LoDN'. Here, the surrounding points P' are selected based on the distance from the point P. For example, the attribute information of points a0, a1, a2, b0, and b1 are used to generate Figure 54 The predicted value of the attribute information of point b2 is shown.

[0622] The selected surrounding points change depending on the value of N. For example, if N = 5, a0, a1, a2, b0, and b1 are selected as the surrounding points of point b2. If N = 4, a0, a1, a2, and b1 are selected based on distance information.

[0623] The predicted value is calculated by weighted averaging depending on the distance. For example, in Figure 54 In the example shown, the predicted value a2p of point a2 is calculated by taking the weighted average of the attribute information of points a0 and a1 as shown in (Formula A2) and (Formula A3). i It is the value of the attribute information of point ai.

[0624]

Formula 2

[0625]

[0626] In addition, the predicted value b2p of point b2 is calculated by taking the weighted average of the attribute information of points a0, a1, a2, b0, and b1 as shown in (Formula A4) to (Formula A6). i It is the value of the attribute information of point bi.

[0627]

Formula 3

[0628]

[0629] Alternatively, the 3D data encoding device may calculate the difference (prediction residual) between the attribute information of a 3D point and the predicted value generated from surrounding points, and quantize the calculated prediction residual. For example, the 3D data encoding device quantizes the prediction residual by dividing it by a quantization scale (also known as a quantization step size). In this case, the smaller the quantization scale, the smaller the error (quantization error) that may be caused by quantization. Conversely, the larger the quantization scale, the larger the quantization error.

[0630] Furthermore, the 3D data encoding device can also change the quantization scale used for each LoD. For example, the 3D data encoding device can reduce the quantization scale as the layer increases, and increase the quantization scale as the layer decreases. The attribute information values ​​of 3D points belonging to the upper layer can be used as predicted values ​​for the attribute information of 3D points belonging to the lower layer. This can reduce the quantization scale of the upper layer and suppress the quantization error that would occur in the upper layer. By improving the accuracy of the predicted value, the encoding efficiency can be improved. Furthermore, the 3D data encoding device can also attach the quantization scale used for each LoD to a header, for example. This allows the 3D data decoding device to correctly decode the quantization scale and thus properly decode the bitstream.

[0631] Furthermore, the 3D data encoding device may also convert the signed integer value (signed quantized value) of the quantized prediction residual into an unsigned integer value (unsigned quantized value). This eliminates the need to consider the generation of negative integers when entropy encoding the prediction residual. Furthermore, the 3D data encoding device does not necessarily need to convert the signed integer value into an unsigned integer value; for example, the sign bit may be separately entropy encoded.

[0632] The prediction residual is calculated by subtracting the predicted value from the original value. For example, as shown in (Equation A7), the prediction residual a2r of point a2 is calculated by subtracting the predicted value a2p of point a2 from the value A2 of the attribute information of point a2. As shown in (Equation A8), the prediction residual b2r of point b2 is calculated by subtracting the predicted value b2p of point b2 from the value B2 of the attribute information of point b2.

[0633] a2r=A2-a2p…(Formula A7)

[0634] b2r=B2-b2p...(Formula A8)

[0635] The prediction residual is quantized by dividing it by QS (quantization step). For example, the quantized value a2q at point a2 is calculated using (Equation A9). The quantized value b2q at point b2 is calculated using (Equation A10). Here, QS_LoD0 is the QS for LoD0, and QS_LoD1 is the QS for LoD1. In other words, QS can be changed depending on the LoD.

[0636] a2q=a2r / QS_LoD0…(Formula A9)

[0637] b2q=b2r / QS_LoD1…(Formula A10)

[0638] Furthermore, as described below, the three-dimensional data encoding device converts the signed integer value, which is the quantized value, into an unsigned integer value. If the signed integer value a2q is less than 0, the three-dimensional data encoding device sets the unsigned integer value a2u to -1-(2×a2q). If the signed integer value a2q is greater than or equal to 0, the three-dimensional data encoding device sets the unsigned integer value a2u to 2×a2q.

[0639] Similarly, when the signed integer value b2q is less than 0, the three-dimensional data encoding device sets the unsigned integer value b2u to -1-(2×b2q). When the signed integer value b2q is greater than or equal to 0, the three-dimensional data encoding device sets the unsigned integer value b2u to 2×b2q.

[0640] Furthermore, the three-dimensional data encoding device may encode the quantized prediction residual (unsigned integer value) by entropy coding. For example, the unsigned integer value may be binarized and then binary arithmetic coding may be applied.

[0641] Furthermore, in this case, the 3D data encoding device may also switch binarization methods based on the value of the prediction residual. For example, when the prediction residual pu is less than the threshold R_TH, the 3D data encoding device binarizes the prediction residual pu using a fixed number of bits required to represent the threshold R_TH. Furthermore, when the prediction residual pu is greater than the threshold R_TH, the 3D data encoding device binarizes the binarized data of the threshold R_TH and the value (pu - R_TH) using, for example, Exponential-Golomb.

[0642] For example, when the threshold R_TH is 63 and the prediction residual pu is less than 63, the 3D data encoding device binarizes the prediction residual pu using 6 bits. Furthermore, when the prediction residual pu is greater than 63, the 3D data encoding device binarizes the binary data (111111) and (pu-63) at the threshold R_TH using Exponential Golomb, thereby performing arithmetic coding.

[0643] In a more specific example, when the prediction residual pu is 32, the 3D data encoding device generates 6-bit binary data (100000) and performs arithmetic coding on this bit string. Alternatively, when the prediction residual pu is 66, the 3D data encoding device generates binary data (111111) representing the threshold value R_TH using Exponential Golomb coefficients and a bit string (00100) with a value of 3 (66-63), and performs arithmetic coding on this bit string (111111+00100).

[0644] In this way, the 3D data encoding device switches the binarization method according to the size of the prediction residual, thereby being able to perform encoding while suppressing a sharp increase in the number of binarization bits when the prediction residual becomes larger. In addition, the 3D data encoding device can also add the threshold R_TH to the header of the bitstream, etc.

[0645] For example, when encoding at a high bit rate, that is, when the quantization scale is small, the quantization error decreases, prediction accuracy increases, and the resulting prediction residual may not increase. Therefore, in this case, the three-dimensional data encoding device sets the threshold R_TH to a large value. This reduces the likelihood of encoding binary data at the threshold R_TH, improving encoding efficiency. Conversely, when encoding at a low bit rate, that is, when the quantization scale is large, the quantization error increases, prediction accuracy decreases, and the resulting prediction residual may increase. Therefore, in this case, the three-dimensional data encoding device sets the threshold R_TH to a small value. This prevents a sharp increase in the bit length of the binary data.

[0646] Alternatively, the 3D data encoding device may switch the threshold R_TH for each LoD and append the threshold R_TH for each LoD to a header, for example. In other words, the 3D data encoding device may switch the binarization method for each LoD. For example, in higher layers, the distances between 3D points are greater, resulting in poor prediction accuracy and a potentially large prediction residual. Therefore, the 3D data encoding device sets a smaller threshold R_TH for higher layers to prevent a dramatic increase in the bit length of the binarized data. Furthermore, in lower layers, the distances between 3D points are closer, resulting in high prediction accuracy and a potentially smaller prediction residual. Therefore, the 3D data encoding device improves encoding efficiency by setting a larger threshold R_TH for each layer.

[0647] Figure 55 This is a diagram showing an example of an Exponential Golomb code, and is a diagram showing the relationship between the value before binarization (multi-value) and the bit after binarization (code). Figure 55 The 0s and 1s shown are reversed.

[0648] Furthermore, the three-dimensional data encoding device applies arithmetic coding to the binarized data of the prediction residual. This improves coding efficiency. Furthermore, when arithmetic coding is applied, the probability of occurrence of 0 and 1 in each bit of the binarized data may differ between the n-bit code (the n-bit code) and the remaining code (the remaining code) binarized using the exponential Golomb algorithm. Therefore, the three-dimensional data encoding device can also switch the arithmetic coding application method between the n-bit code and the remaining code.

[0649] For example, for n-bit coding, the three-dimensional data coding device uses a different coding table (probability table) to perform arithmetic coding on each bit. In this case, the three-dimensional data coding device can also change the number of coding tables used for each bit. For example, the three-dimensional data coding device uses 1 coding table to perform arithmetic coding on the first bit b0 of the n-bit coding. In addition, the three-dimensional data coding device uses 2 coding tables for the next bit b1. In addition, the three-dimensional data coding device switches the coding table used in the arithmetic coding of the bit b1 according to the value of b0 (0 or 1). Similarly, the three-dimensional data coding device also uses 4 coding tables for the next bit b2. In addition, the three-dimensional data coding device switches the coding table used in the arithmetic coding of the bit b2 according to the values ​​of b0 and b1 (0 to 3).

[0650] Thus, the three-dimensional data encoding device uses 2 when performing arithmetic coding on each bit bn-1 of the n-bit code. n-1 In addition, the 3D data encoding device switches the encoding table to be used according to the value (generation pattern) of the bit before bn-1. As a result, the 3D data encoding device can use the appropriate encoding table for each bit, thereby improving encoding efficiency.

[0651] In addition, the three-dimensional data encoding device can also reduce the number of encoding tables used for each bit. For example, when performing arithmetic coding on each bit bn-1, the three-dimensional data encoding device can also switch between two encoding tables based on the value (generation pattern) of the m bits (m<n-1) before bn-1. m The 3D data encoding device can also update the probability of occurrence of 0 and 1 in each coding table based on the value of the binary data actually generated. Furthermore, the 3D data encoding device can also fix the probability of occurrence of 0 and 1 in the coding table for some bits. This can reduce the number of updates to the probability of occurrence, thereby reducing the amount of processing.

[0652] For example, when the n-bit code is b0b1b2…bn-1, there is one coding table for b0 (CTb0). There are two coding tables for b1 (CTb10, CTb11). In addition, the coding table used is switched according to the value of b0 (0 to 1). There are four coding tables for b2 (CTb20, CTb21, CTb22, CTb23). In addition, the coding table used is switched according to the values ​​of b0 and b1 (0 to 3). There are 2 coding tables for bn-1. n-1 (CTbn0, CTbn1, ..., CTbn(2 n-1 -1)). In addition, according to the value of b0b1…bn-2 (0~2 n-1 -1) to switch the encoding table used.

[0653] Furthermore, the three-dimensional data encoding device may also apply setting values ​​of 0 to 2 to the n-bit code without performing binarization. n -1 value m-ary arithmetic coding (m = 2 n ). In addition, when the three-dimensional data encoding device performs arithmetic encoding on the n-bit code using m-ary, the three-dimensional data decoding device can also restore the n-bit code by arithmetic decoding of m-ary.

[0654] Figure 56 : is a diagram for explaining the processing when the residual code is an Exp Golomb code. Figure 56 As shown, the portion binarized using the Exponential Golomb algorithm, i.e., the remaining code, includes the prefix portion and the suffix portion. For example, the 3D data encoding device switches between coding tables for the prefix portion and the suffix portion. Specifically, the 3D data encoding device performs arithmetic coding on each bit in the prefix portion using the coding table for the prefix portion, and performs arithmetic coding on each bit in the suffix portion using the coding table for the suffix portion.

[0655] Furthermore, the 3D data encoding device can update the occurrence probabilities of 0 and 1 in each code table based on the values ​​of the actual binary data generated. Alternatively, the 3D data encoding device can fix the occurrence probabilities of 0 and 1 in a particular code table. This reduces the number of updates to the occurrence probabilities, thereby reducing the amount of processing. For example, the 3D data encoding device can update the occurrence probabilities for the prefix portion and fix the occurrence probabilities for the suffix portion.

[0656] Furthermore, the 3D data encoding device decodes the quantized prediction residual through inverse quantization and reconstruction, and uses the decoded prediction residual, i.e., the decoded value, for subsequent prediction of the 3D point being encoded. Specifically, the 3D data encoding device calculates an inverse quantization value by multiplying the quantized prediction residual (quantized value) by the quantization scale, and then adds the inverse quantization value to the predicted value to obtain a decoded value (reconstructed value).

[0657] For example, the inverse quantized value a2iq at point a2 is calculated using the quantized value a2q at point a2 by (Equation A11). The inverse quantized value b2iq at point b2 is calculated using the quantized value b2q at point b2 by (Equation A12). Here, QS_LoD0 is the QS for LoD0, and QS_LoD1 is the QS for LoD1. In other words, the QS can be changed depending on the LoD.

[0658] a2iq=a2q×QS_LoD0…(Formula A11)

[0659] b2iq=b2q×QS_LoD1…(Formula A12)

[0660] For example, as shown in (Equation A13), the decoded value a2rec at point a2 is calculated by adding the predicted value a2p at point a2 to the inverse quantized value a2iq at point a2. As shown in (Equation A14), the decoded value b2rec at point b2 is calculated by adding the predicted value b2p at point b2 to the inverse quantized value b2iq at point b2.

[0661] a2rec=a2iq+a2p…(Formula A13)

[0662] b2rec=b2iq+b2p (Formula A14)

[0663] Hereinafter, an example of the syntax of the bit stream according to this embodiment will be described. Figure 57 This is a diagram showing an example of the syntax of the attribute header (attribute_header) of this embodiment. The attribute header is the header information of the attribute information. Figure 57 As shown, the attribute header includes the number of layers information (NumLoD), three-dimensional point information (NumOfPoint[i]), layer threshold (Thres_Lod[i]), surrounding point information (NumNeighorPoint[i]), prediction threshold (THd[i]), quantization scale (QS[i]), and binarization threshold (R_TH[i]).

[0664] The number of layers information (NumLoD) indicates the number of layers of LoD used.

[0665] The three-dimensional point number information (NumOfPoint[i]) represents the number of three-dimensional points belonging to level i. In addition, the three-dimensional data encoding device may also attach the three-dimensional point total number information (AllNumOfPoint) representing the total number of three-dimensional points to another header. In this case, the three-dimensional data encoding device may not attach NumOfPoint[NumLoD - 1], which represents the number of three-dimensional points belonging to the bottommost level, to the header. In this case, the three-dimensional data decoding device can calculate NumOfPoint[NumLoD - 1] through (Equation A15). Thereby, the encoding amount of the header can be reduced.

[0666]

Equation 4

[0667]

[0668] The level threshold (Thres_Lod[i]) is a threshold for setting level i. The three-dimensional data encoding device and the three-dimensional data decoding device form LoDi such that the distance between each point within LoDi is greater than the threshold Thres_LoD[i]. In addition, the three-dimensional data encoding device may not attach the value of Thres_Lod[NumLoD - 1] (the bottommost level) to the header. In this case, the three-dimensional data decoding device estimates the value of Thres_Lod[NumLoD - 1] as 0. Thereby, the encoding amount of the header can be reduced.

[0669] The surrounding point number information (NumNeighorPoint[i]) represents the upper limit value of the number of surrounding points used in the generation of the predicted value of the three-dimensional points belonging to level i. When the number of surrounding points M is less than NumNeighorPoint[i] (M < NumNeighorPoint[i]), the three-dimensional data encoding device may also use M surrounding points to calculate the predicted value. In addition, when it is not necessary to separate the value of NumNeighorPoint[i] for each LoD, the three-dimensional data encoding device may also attach one surrounding point number information (NumNeighorPoint) used in all LoDs to the header.

[0670] The prediction threshold (THd[i]) represents the upper limit value of the distance between the surrounding three-dimensional points and the object three-dimensional point used in the prediction of the object three-dimensional point to be encoded or decoded at level i. The three-dimensional data encoding device and the three-dimensional data decoding device do not use the three-dimensional points whose distance from the object three-dimensional point is farther than THd[i] for prediction. In addition, when it is not necessary to separate the value of THd[i] for each LoD, the three-dimensional data encoding device may also attach one prediction threshold (THd) used in all LoDs to the header.

[0671] The quantization scale (QS[i]) represents the quantization scale used in quantization and inverse quantization of level i.

[0672] The binarization threshold (R_TH[i]) is used to switch the binarization method for the prediction residual of a 3D point belonging to layer i. For example, when the prediction residual is less than the threshold R_TH, the 3D data encoding device binarizes the prediction residual pu with a fixed number of bits. When the prediction residual is greater than the threshold R_TH, the 3D data encoding device binarizes the binarized data of the threshold R_TH and the value (pu - R_TH) using Exponential Golomb. Alternatively, if the value of R_TH[i] does not need to be switched between LoDs, the 3D data encoding device may append a single binarization threshold (R_TH) to the header for all LoDs.

[0673] In addition, R_TH[i] may also be a maximum value represented by nbit. For example, in 6bit, R_TH is 63, and in 8bit, R_TH is 255. In addition, the three-dimensional data encoding device may encode the number of bits instead of encoding the maximum value represented by nbit as the binarization threshold. For example, the three-dimensional data encoding device may append the value 6 to the header when R_TH[i]=63, and append the value 8 to the header when R_TH[i]=255. In addition, the three-dimensional data encoding device may also define a minimum value (minimum number of bits) for the number of bits representing R_TH[i], and append the relative number of bits relative to the minimum value to the header. For example, the three-dimensional data encoding device may append the value 0 to the header when R_TH[i]=63 and the minimum number of bits is 6, and append the value 2 to the header when R_TH[i]=255 and the minimum number of bits is 6.

[0674] Alternatively, the 3D data encoding device may entropy encode at least one of NumLoD, Thres_Lod[i], NumNeighborPoint[i], THd[i], QS[i], and R_TH[i] and append the result to the header. For example, the 3D data encoding device may binarize each value and perform arithmetic encoding. Furthermore, to reduce processing overhead, the 3D data encoding device may encode each value with a fixed length.

[0675] Furthermore, the 3D data encoding device may not add at least one of NumLoD, Thres_Lod[i], NumNeighborPoint[i], THd[i], QS[i], and R_TH[i] to the header. For example, the value of at least one of these values ​​may be specified by a profile or level of a standard. This can reduce the number of bits in the header.

[0676] Figure 58 This is a diagram showing an example of the syntax of attribute data (attribute_data) in this embodiment. The attribute data includes encoded data of attribute information of a plurality of three-dimensional points. Figure 58 As shown, the attribute data includes n-bit code and remaining code.

[0677] The n-bit code is the coded data of the prediction residual of the attribute information value, or a portion thereof. The bit length of the n-bit code depends on the value of R_TH[i]. For example, if the value indicated by R_TH[i] is 63, the n-bit code is 6 bits, and if the value indicated by R_TH[i] is 255, the n-bit code is 8 bits.

[0678] The remaining code is the coded data of the prediction residual of the attribute information value after Exponential Golomb coding. If the n-bit code is the same as R_TH[i], the remaining code is encoded or decoded. Furthermore, the 3D data decoding device adds the value of the n-bit code to the value of the remaining code to decode the prediction residual. Furthermore, if the n-bit code is not the same as R_TH[i], the remaining code may not be encoded or decoded.

[0679] The following describes the flow of processing in the three-dimensional data encoding device. Figure 59 This is a flowchart of a three-dimensional data encoding process performed by a three-dimensional data encoding device.

[0680] First, the three-dimensional data encoding device encodes position information (geometry) (S3001). For example, the three-dimensional data encoding is performed using an octree representation.

[0681] After encoding the position information, if the position of a 3D point changes due to quantization or other factors, the 3D data encoding device reassigns the attribute information of the original 3D point to the changed 3D point (S3002). For example, the 3D data encoding device reassigns the attribute information by interpolating the value of the attribute information based on the amount of change in position. For example, the 3D data encoding device detects N 3D points before the change that are close to the changed 3D position and takes a weighted average of the attribute information values ​​of the N 3D points. For example, in the weighted average, the 3D data encoding device determines a weight based on the distance from the changed 3D position to each of the N 3D points. The 3D data encoding device then determines the value obtained by the weighted average as the attribute information value of the changed 3D point. Furthermore, if two or more 3D points change to the same 3D position due to quantization or other factors, the 3D data encoding device may assign the average of the attribute information of the two or more 3D points before the change as the attribute information value of the changed 3D point.

[0682] Next, the 3D data encoding device encodes the reallocated attribute information (S3003). For example, when encoding multiple types of attribute information, the 3D data encoding device may encode the multiple types of attribute information sequentially. For example, when encoding color and reflectivity as attribute information, the 3D data encoding device may generate a bitstream in which the encoding result of reflectivity is appended after the encoding result of color. Furthermore, the order of the multiple encoding results of the attribute information appended to the bitstream is not limited to this order and may be any order.

[0683] Furthermore, the 3D data encoding device may also append information indicating the starting location of the encoded data for each attribute information within the bitstream to a header, for example. This allows the 3D data decoding device to selectively decode attribute information that requires decoding, thereby omitting the decoding process for attribute information that does not require decoding. This reduces the processing load of the 3D data decoding device. Furthermore, the 3D data encoding device may also encode multiple types of attribute information in parallel and combine the encoded results into a single bitstream. This enables the 3D data encoding device to encode multiple types of attribute information at high speed.

[0684] Figure 60 3D data encoding process (S3003) First, the 3D data encoding device sets the LoD (S3011). That is, the 3D data encoding device assigns each 3D point to any one of a plurality of LoDs.

[0685] Next, the three-dimensional data encoding device starts a loop in units of LoDs (S3012). That is, the three-dimensional data encoding device repeats the processing of steps S3013 to S3021 for each LoD.

[0686] Next, the three-dimensional data encoding device starts a three-dimensional point-based loop (S3013). That is, the three-dimensional data encoding device repeatedly performs the processing of steps S3014 to S3020 for each three-dimensional point.

[0687] First, the 3D data encoding device searches for a plurality of surrounding points, three-dimensional points surrounding the target 3D point to be used in calculating the predicted value of the target 3D point being processed (S3014). Next, the 3D data encoding device calculates a weighted average of the attribute information values ​​of the plurality of surrounding points and sets the resulting value as the predicted value P (S3015). Next, the 3D data encoding device calculates the difference between the attribute information of the target 3D point and the predicted value, namely the prediction residual (S3016). Next, the 3D data encoding device quantizes the prediction residual to calculate a quantized value (S3017). Next, the 3D data encoding device performs arithmetic coding on the quantized value (S3018).

[0688] Furthermore, the 3D data encoding device calculates an inverse quantized value by inverse quantizing the quantized value (S3019). Next, the 3D data encoding device generates a decoded value by adding the predicted value to the inverse quantized value (S3020). The 3D data encoding device then terminates the loop for the 3D point unit (S3021). Furthermore, the 3D data encoding device terminates the loop for the LoD unit (S3022).

[0689] Hereinafter, a three-dimensional data decoding process in a three-dimensional data decoding device that decodes a bit stream generated by the above-described three-dimensional data encoding device will be described.

[0690] The 3D data decoding device generates decoded binary data by performing arithmetic decoding on the binary data of the attribute information within the bit stream generated by the 3D data encoding device using the same method as the 3D data encoding device. Furthermore, if the 3D data encoding device switches the arithmetic coding method between the portion binarized using n bits (n-bit coding) and the portion binarized using Exponential Golomb coding (residual coding), the 3D data decoding device performs decoding accordingly when arithmetic decoding is applied.

[0691] For example, in an arithmetic decoding method for n-bit encoding, a three-dimensional data decoding device uses a different encoding table (decoding table) for each bit to perform arithmetic decoding. In this case, the three-dimensional data decoding device may also change the number of encoding tables used for each bit. For example, one encoding table is used to perform arithmetic decoding on the first bit b0 of the n-bit encoding. In addition, the three-dimensional data decoding device uses two encoding tables for the next bit b1. In addition, the three-dimensional data decoding device switches the encoding table used for arithmetic decoding of bit b1 according to the value of b0 (0 or 1). Similarly, the three-dimensional data decoding device further uses four encoding tables for the next bit b2. In addition, the three-dimensional data decoding device switches the encoding table used for arithmetic decoding of bit b2 according to the values ​​of b0 and b1 (0 to 3).

[0692] In this way, the three-dimensional data decoding device uses 2 when performing arithmetic decoding on each bit bn-1 of the n-bit code. n-1 In addition, the 3D data decoding device switches the coding table to be used according to the value (occurrence pattern) of the bit before bn-1. As a result, the 3D data decoding device can use the appropriate coding table for each bit to appropriately decode the bit stream with improved coding efficiency.

[0693] In addition, the three-dimensional data decoding device may also reduce the number of coding tables used for each bit. For example, when performing arithmetic decoding on each bit bn-1, the three-dimensional data decoding device may switch between two encoding tables according to the value (generation pattern) of the m bits (m<n-1) before bn-1. m The 3D data decoding device can thus appropriately decode a bitstream with improved coding efficiency while reducing the number of coding tables used for each bit. Furthermore, the 3D data decoding device can update the occurrence probabilities of 0 and 1 in each coding table based on the values ​​of the actual binary data generated. Furthermore, the 3D data decoding device can also fix the occurrence probabilities of 0 and 1 in the coding tables for a portion of the bits. This reduces the number of updates to the occurrence probabilities, thereby reducing the amount of processing.

[0694] For example, when the n-bit code is b0b1b2…bn-1, there is one coding table for b0 (CTb0). There are two coding tables for b1 (CTb10, CTb11). In addition, the coding table is switched according to the value of b0 (0 to 1). There are four coding tables for b2 (CTb20, CTb21, CTb22, CTb23). In addition, the coding table is switched according to the values ​​of b0 and b1 (0 to 3). There are 2 coding tables for bn-1. n-1 (CTbn0, CTbn1, ..., CTbn(2 n-1 -1)). In addition, according to the value of b0b1…bn-2 (0~2 n-1-1) to switch the encoding table.

[0695] For example, Figure 61 : is a diagram for explaining the processing when the residual code is an Exp Golomb code. Figure 61 As shown, the portion encoded by the 3D data encoding device using Exponential Golomb binarization (the remaining code) includes the prefix and suffix parts. For example, the 3D data decoding device switches between coding tables for the prefix and suffix parts. Specifically, the 3D data decoding device performs arithmetic decoding on each bit in the prefix part using the coding table for the prefix part, and performs arithmetic decoding on each bit in the suffix part using the coding table for the suffix part.

[0696] Furthermore, the 3D data decoding device can update the occurrence probabilities of 0 and 1 in each code table based on the value of the binary data generated during decoding. Alternatively, the 3D data decoding device can fix the occurrence probabilities of 0 and 1 in a particular code table. This reduces the number of updates to the occurrence probabilities, thereby reducing the amount of processing. For example, the 3D data decoding device can update the occurrence probabilities for the prefix portion and fix the occurrence probabilities for the suffix portion.

[0697] The 3D data decoding device converts the binary data of the prediction residual obtained through arithmetically decoding into multi-valued data in accordance with the encoding method used by the 3D data encoding device, thereby decoding the quantized prediction residual (an unsigned integer value). The 3D data decoding device first calculates the value of the n-bit code obtained by arithmetically decoding the n-bit encoded binary data. Next, the 3D data decoding device compares the n-bit code value with the value of R_TH.

[0698] If the value of the n-bit code matches the value of R_TH, the 3D data decoding device determines that the next bit is coded using Exponential Golomb coding, and performs arithmetic decoding on the residual code, the binary data coded using Exponential Golomb coding. The 3D data decoding device then calculates the value of the residual code based on the decoded residual code using a back-calculation table that shows the relationship between the residual code and the value. Figure 62 : is a diagram showing an example of a back-estimation table indicating the relationship between the residual code and its value. Next, the three-dimensional data decoding apparatus adds the obtained residual code value to R_TH to obtain a multi-valued quantized prediction residual.

[0699] On the other hand, if the n-bit coded value does not match the value of R_TH (is smaller than R_TH), the 3D data decoding device directly determines the n-bit coded value as the multi-valued quantized prediction residual. This allows the 3D data decoding device to appropriately decode the bitstream generated by switching the binarization method in accordance with the value of the prediction residual in the 3D data encoding device.

[0700] Furthermore, when the threshold R_TH is appended to a header of the bitstream, for example, the 3D data decoding device may decode the value of the threshold R_TH from the header and use the decoded value of the threshold R_TH to switch the decoding method. Furthermore, when the threshold R_TH is appended to a header for each LoD, for example, the 3D data decoding device may use the decoded threshold R_TH to switch the decoding method for each LoD.

[0701] For example, when the threshold R_TH is 63 and the decoded n-bit code value is 63, the three-dimensional data decoding device decodes the remaining code using the Exponential Golomb method to obtain the value of the remaining code. Figure 62 In the example shown, the residual code is 00100, and the residual code value is 3. Next, the three-dimensional data decoding apparatus adds the value 63 of the threshold R_TH and the value 3 of the residual code to obtain a prediction residual value 66.

[0702] Furthermore, when the decoded n-bit code value is 32, the three-dimensional data decoding apparatus sets the n-bit code value 32 as the value of the prediction residual.

[0703] Furthermore, the 3D data decoding device converts the decoded quantized prediction residual from an unsigned integer value to a signed integer value, for example, by performing a process that is the inverse of the process performed in the 3D data encoding device. This allows the 3D data decoding device to appropriately decode a bitstream generated without regard to the occurrence of negative integers when entropy encoding the prediction residual. Furthermore, the 3D data decoding device does not necessarily need to convert unsigned integer values ​​to signed integer values; for example, when decoding a bitstream generated by separately entropy encoding a sign bit, the 3D data decoding device can also decode the sign bit.

[0704] The 3D data decoding device decodes the quantized prediction residual, which has been converted to a signed integer value, through inverse quantization and reconstruction to generate a decoded value. The 3D data decoding device then uses the generated decoded value for subsequent predictions of the 3D point being decoded. Specifically, the 3D data decoding device multiplies the quantized prediction residual by the decoded quantization scale to calculate an inverse quantization value, and then adds the inverse quantization value to the prediction value to obtain a decoded value.

[0705] The decoded unsigned integer value (unsigned quantized value) is converted to a signed integer value through the following processing. If the LSB (least significant bit) of the decoded unsigned integer value a2u is 1, the 3D data decoding device sets the signed integer value a2q to -((a2u+1)>>1). If the LSB of the unsigned integer value a2u is not 1, the 3D data decoding device sets the signed integer value a2q to (a2u>>1).

[0706] Similarly, when the LSB of the decoded unsigned integer value b2u is 1, the three-dimensional data decoding device sets the signed integer value b2q to -((b2u+1)>>1). When the LSB of the unsigned integer value n2u is not 1, the three-dimensional data decoding device sets the signed integer value b2q to (b2u>>1).

[0707] Note that details of the inverse quantization and reconstruction processing performed by the three-dimensional data decoding device are the same as those of the inverse quantization and reconstruction processing in the three-dimensional data encoding device.

[0708] The following describes the flow of processing in the three-dimensional data decoding device. Figure 63 3D data decoding processing performed by a 3D data decoding device. First, the 3D data decoding device decodes position information (geometry) from a bitstream (S3031). For example, the 3D data decoding device uses an octree representation for decoding.

[0709] Next, the three-dimensional data decoding device decodes the attribute information (Attribute) from the bit stream (S3032). For example, in the case of decoding multiple types of attribute information, the three-dimensional data decoding device may also decode the multiple types of attribute information in sequence. For example, in the case of decoding color and reflectivity as attribute information, the three-dimensional data decoding device decodes the color encoding result and the reflectivity encoding result in the order in which they are attached to the bit stream. For example, in the case where the reflectivity encoding result is attached after the color encoding result in the bit stream, the three-dimensional data decoding device decodes the color encoding result and then decodes the reflectivity encoding result. In addition, the three-dimensional data decoding device may decode the encoding results of the attribute information attached to the bit stream in any order.

[0710] Furthermore, the 3D data decoding device can also obtain information indicating the starting location of the encoded data for each attribute information within the bitstream by decoding the header. This allows the 3D data decoding device to selectively decode attribute information that requires decoding, thereby omitting the decoding process for attribute information that does not require decoding. This reduces the processing load of the 3D data decoding device. Furthermore, the 3D data decoding device can decode multiple types of attribute information in parallel and merge the decoded results into a single 3D point group. This enables the 3D data decoding device to decode multiple types of attribute information at high speed.

[0711] Figure 64 This is a flowchart of the attribute information decoding process (S3032). First, the 3D data decoding device sets the LoD (S3041). Specifically, the 3D data decoding device assigns each of the multiple 3D points with decoded position information to one of multiple LoDs. For example, this assignment method is the same as that used in the 3D data encoding device.

[0712] Next, the three-dimensional data decoding device starts a loop in units of LoDs ( S3042 ), that is, the three-dimensional data decoding device repeats the processing of steps S3043 to S3049 for each LoD.

[0713] Next, the three-dimensional data decoding device starts a three-dimensional point-based loop ( S3043 ), that is, the three-dimensional data decoding device repeats the processing of steps S3044 to S3048 for each three-dimensional point.

[0714] First, the 3D data decoding device searches for a plurality of surrounding points, three-dimensional points surrounding the target 3D point, used in calculating the predicted value of the target 3D point (S3044). Next, the 3D data decoding device calculates a weighted average of the attribute information values ​​of the plurality of surrounding points and sets the resulting average as the predicted value P (S3045). These processes are identical to those in the 3D data encoding device.

[0715] Next, the 3D data decoding device performs arithmetic decoding on the quantized value from the bitstream (S3046). Furthermore, the 3D data decoding device inversely quantizes the decoded quantized value to calculate an inverse quantized value (S3047). Next, the 3D data decoding device adds the predicted value to the inverse quantized value to generate a decoded value (S3048). The 3D data decoding device then terminates the loop for the 3D point unit (S3049). Furthermore, the 3D data decoding device terminates the loop for the LoD unit (S3050).

[0716] Next, the configurations of the three-dimensional data encoding device and the three-dimensional data decoding device according to this embodiment will be described. Figure 653D data encoding device 3000 of this embodiment is a block diagram showing the structure of the 3D data encoding device 3000. The 3D data encoding device 3000 includes a position information encoding unit 3001, an attribute information reallocation unit 3002, and an attribute information encoding unit 3003.

[0717] The attribute information encoding unit 3003 encodes the positional information (geometry) of the multiple 3D points included in the input point group. The attribute information reallocation unit 3002 uses the encoding and decoding results of the positional information to reallocate the attribute information values ​​of the multiple 3D points included in the input point group. The attribute information encoding unit 3003 encodes the reallocated attribute information. Furthermore, the 3D data encoding device 3000 generates a bitstream containing the encoded positional information and the encoded attribute information.

[0718] Figure 66 3D data decoding apparatus 3010 of this embodiment is a block diagram showing the structure of the 3D data decoding apparatus 3010. The 3D data decoding apparatus 3010 includes a position information decoding unit 3011 and an attribute information decoding unit 3012.

[0719] The position information decoding unit 3011 decodes the position information (geometry) of multiple 3D points from the bitstream. The attribute information decoding unit 3012 decodes the attribute information (attribute) of multiple 3D points from the bitstream. Furthermore, the 3D data decoding device 3010 combines the decoded position information with the decoded attribute information to generate an output point cloud.

[0720] As described above, the three-dimensional data encoding device of this embodiment performs Figure 67 The three-dimensional data encoding device encodes a three-dimensional point having attribute information. First, the three-dimensional data encoding device calculates a predicted value of the attribute information of the three-dimensional point (S3061). Next, the three-dimensional data encoding device calculates the difference between the attribute information of the three-dimensional point and the predicted value, that is, the prediction residual (S3062). Next, the three-dimensional data encoding device generates binary data by binarizing the prediction residual (S3063). Next, the three-dimensional data encoding device performs arithmetic coding on the binary data (S3064).

[0721] Thus, the three-dimensional data encoding device can reduce the code amount of the encoded data of the attribute information by calculating the prediction residual of the attribute information and binarizing and arithmetic encoding the prediction residual.

[0722] For example, in arithmetic coding ( S3064 ), the three-dimensional data coding apparatus uses a different coding table for each bit of the binary data, thereby improving coding efficiency.

[0723] For example, in arithmetic coding (S3064), the number of coding tables used increases as the lower bits of binary data increase.

[0724] For example, during arithmetic coding (S3064), the three-dimensional data coding apparatus selects a coding table for use in arithmetic coding of the target bit based on the value of the most significant bit of the target bit contained in the binary data. This allows the three-dimensional data coding apparatus to select a coding table based on the value of the most significant bit, thereby improving coding efficiency.

[0725] For example, during binarization (S3063), if the prediction residual is less than a threshold value (R_TH), the 3D data encoding device generates binary data by binarizing the prediction residual with a fixed number of bits. If the prediction residual is greater than the threshold value (R_TH), the 3D data encoding device generates binary data including a first code (n-bit code) having a fixed number of bits representing the threshold value (R_TH) and a second code (residual code) obtained by binarizing the value obtained by subtracting the threshold value (R_TH) from the prediction residual using Exponential Golomb method. During arithmetic coding (S3064), the 3D data encoding device uses different arithmetic coding methods for the first and second codes.

[0726] Thus, the three-dimensional data encoding device can perform arithmetic encoding on the first code and the second code using arithmetic encoding methods suitable for the first code and the second code, respectively, for example, thereby improving encoding efficiency.

[0727] For example, the 3D data encoding device quantizes the prediction residual and binarizes the quantized prediction residual in binarization (S3063). The threshold (R_TH) changes according to the quantization scale during quantization. This allows the 3D data encoding device to use an appropriate threshold value corresponding to the quantization scale, thereby improving encoding efficiency.

[0728] For example, the second code includes a prefix part and a suffix part. In arithmetic coding (S3064), the three-dimensional data coding device uses different coding tables for the prefix part and the suffix part. This can improve coding efficiency.

[0729] For example, the three-dimensional data encoding device includes a processor and a memory, and the processor performs the above-mentioned processing using the memory.

[0730] In addition, the three-dimensional data decoding device of this embodiment performs Figure 68The three-dimensional data decoding device decodes a three-dimensional point having attribute information. First, the three-dimensional data decoding device calculates a predicted value of the attribute information of the three-dimensional point (S3071). Next, the three-dimensional data decoding device generates binary data by performing arithmetic decoding on the encoded data contained in the bit stream (S3072). Next, the three-dimensional data decoding device generates a prediction residual by multi-valuedizing the binary data (S3073). Next, the three-dimensional data decoding device calculates a decoded value of the attribute information of the three-dimensional point by adding the predicted value to the prediction residual (S3074).

[0731] As a result, the three-dimensional data decoding device can calculate the prediction residual of the attribute information and appropriately decode the bit stream of the attribute information generated by binarizing and arithmetic coding the prediction residual.

[0732] For example, in arithmetic decoding ( S3072 ), the 3D data decoding apparatus uses a different coding table for each bit of binary data, thereby enabling the 3D data decoding apparatus to appropriately decode a bit stream with improved coding efficiency.

[0733] For example, in arithmetic decoding ( S3072 ), the number of coding tables used increases as the lower bits of binary data increase.

[0734] For example, during arithmetic decoding (S3072), the 3D data decoding apparatus selects a coding table for use in arithmetic decoding of the target bit based on the value of the most significant bit of the target bit contained in the binary data. This allows the 3D data decoding apparatus to appropriately decode a bitstream with improved coding efficiency.

[0735] For example, in multi-value conversion (S3073), the 3D data decoding device generates a first value by multi-valuedizing a first code (n-bit code) having a fixed number of bits included in the binary data. If the first value is less than a threshold value (R_TH), the 3D data decoding device determines the first value as the prediction residual. If the first value is greater than the threshold value (R_TH), the 3D data decoding device generates a second value by multi-valuedizing a second code (residual code) that is an Exponential Golomb code included in the binary data. The prediction residual is then generated by adding the first and second values. In arithmetic decoding (S3072), the 3D data decoding device uses different arithmetic decoding methods for the first and second codes.

[0736] As a result, the three-dimensional data decoding device can appropriately decode a bit stream with improved coding efficiency.

[0737] For example, the 3D data decoding device inversely quantizes the prediction residual and, in the addition operation (S3074), adds the prediction value to the inversely quantized prediction residual. The threshold (R_TH) is changed according to the quantization scale during inverse quantization. This allows the 3D data decoding device to appropriately decode a bitstream with improved coding efficiency.

[0738] For example, the second code includes a prefix part and a suffix part. During arithmetic decoding (S3072), the 3D data decoding device uses different code tables for the prefix part and the suffix part. This allows the 3D data decoding device to appropriately decode a bitstream with improved coding efficiency.

[0739] For example, the three-dimensional data decoding device includes a processor and a memory, and the processor performs the above-mentioned processing using the memory.

[0740] (Implementation 9)

[0741] The predicted value may be generated by a method different from that in Embodiment 8. Hereinafter, the 3D point to be encoded may be referred to as a first 3D point, and the 3D points around it may be referred to as a second 3D point.

[0742] For example, when generating a predicted value for 3D point attribute information, the attribute value of the closest 3D point among the already encoded and decoded surrounding 3D points of the target 3D point can be directly used as the predicted value. Alternatively, prediction mode information (PredMode) can be added to each 3D point to generate the predicted value, and a predicted value can be generated by selecting one of multiple predicted values. For example, among a total of M prediction modes, prediction mode 0 can be assigned the average value, prediction mode 1 can be assigned the attribute value of 3D point A, and so on, and prediction mode M-1 can be assigned the attribute value of 3D point Z. The prediction mode used for prediction can then be added to the bitstream for each 3D point. In this way, the first prediction mode value, representing the first prediction mode that calculates the predicted value using the average of the surrounding 3D point attribute information as the predicted value, can be smaller than the second prediction mode value, representing the second prediction mode that calculates the predicted value using the surrounding 3D point attribute information itself as the predicted value. Here, the predicted value calculated in prediction mode 0, or the "average value," is the average value of the attribute values ​​of the 3D points surrounding the target 3D point.

[0743] Figure 69 This is a diagram showing a first example of a table showing predicted values ​​calculated in each prediction mode according to the ninth embodiment. Figure 70 This is a diagram showing an example of attribute information used for prediction values ​​in the ninth embodiment. Figure 71 This is a diagram showing a second example of a table showing predicted values ​​calculated in each prediction mode according to the ninth embodiment.

[0744] The number of prediction modes M may be added to the bitstream. Alternatively, the number of prediction modes M may be defined by a standard profile or level, rather than being added to the bitstream. Furthermore, the number of prediction modes M may be calculated based on the number of three-dimensional points N used in the prediction. For example, the number of prediction modes M may be calculated as M = N + 1.

[0745] in addition, Figure 69 The table shown is an example where the number of 3D points used in prediction, N, is 4, and the number of prediction modes, M, is 5. The attribute information of points a0, a1, a2, and b1 can be used to generate a predicted value for the attribute information of point b2. When selecting a single prediction mode from multiple prediction modes, a prediction mode that generates attribute values ​​for each of points a0, a1, a2, and b1 as predicted values ​​can also be selected based on the distance information from point b2 to each of points a0, a1, a2, and b1. A prediction mode is assigned to each 3D point to be encoded. The predicted value is calculated based on the value corresponding to the assigned prediction mode.

[0746] Figure 71 The table shown is Figure 69 This example also uses N = 4 3D points used in prediction and M = 5 prediction modes. The predicted value for the attribute information of point a2 can be generated using the attribute information of points a0 and a1. When selecting a single prediction mode from multiple prediction modes, a prediction mode that generates the attribute values ​​of points a0 and a1 as predicted values ​​can be selected based on the distance information from point a2 to each of points a0 and a1. A prediction mode is assigned to each 3D point to be encoded. The predicted value is calculated based on the value corresponding to the assigned prediction mode.

[0747] Furthermore, when the number of adjacent points, that is, the number N of surrounding three-dimensional points is less than 4 as in the case of the point a2 described above, the prediction mode to which no prediction value is allocated in the table may be set as unavailable.

[0748] In addition, the allocation of prediction mode values ​​can also be determined in the order of distance from the 3D point of the encoding object. For example, the closer the distance from the 3D point of the encoding object to the surrounding 3D points with attribute information used as prediction values, the smaller the prediction mode value representing multiple prediction modes. Figure 69In the example, this indicates that the distances to point b2, the 3D point being encoded, are close in the order of points b1, a2, a1, and a0. For example, in calculating the predicted value, the attribute information of point b1 is calculated as the predicted value in the prediction mode indicated by the prediction mode value of "1" among two or more prediction modes, and the attribute information of point a2 is calculated as the predicted value in the prediction mode indicated by the prediction mode value of "2." Thus, the prediction mode value of the prediction mode indicating that the attribute information of point b1 is calculated as the predicted value is smaller than the prediction mode value of the prediction mode indicating that the attribute information of point a2 is calculated as the predicted value. Point a2 is located farther from point b2 than from point b1.

[0749] This allows for assigning a smaller prediction mode value to a point that is likely to be easily predicted and selected due to its proximity, reducing the number of bits used to encode the prediction mode value. Furthermore, smaller prediction mode values ​​can be preferentially assigned to 3D points that share the same LoD as the target 3D point.

[0750] Figure 72 This figure shows a third example of a table showing predicted values ​​calculated in each prediction mode in Embodiment 9. Specifically, Example 3 illustrates a case where the attribute information used in the predicted values ​​is based on the color information (YUV) of surrounding 3D points. Similarly, the attribute information used for the predicted values ​​may also be color information indicating the color of the 3D points.

[0751] like Figure 72 As shown, the predicted value calculated in the prediction mode with a prediction mode value of "0" is the average of the individual YUV components that define the YUV color space. Specifically, the predicted value includes: the weighted average Yave of the Y component values ​​corresponding to points b1, a2, a1, and a0, namely Yb1, Ya2, Ya1, and Ya0; the weighted average Uave of the U component values ​​corresponding to points b1, a2, a1, and a0, namely Ub1, Ua2, Ua1, and Ua0; and the weighted average Vave of the V component values ​​corresponding to points b1, a2, a1, and a0, namely Vb1, Va2, Va1, and Va0. Furthermore, the predicted value calculated in the prediction modes with prediction mode values ​​of "1" to "4" includes the color information of the surrounding three-dimensional points b1, a2, a1, and a0, respectively. Color information is represented by a combination of the values ​​of the Y, U, and V components.

[0752] In addition, Figure 72 In the RGB color space, color information is represented by values ​​defined by the YUV color space, but is not limited to the YUV color space. It can also be represented by values ​​defined by the RGB color space or by values ​​defined by other color spaces.

[0753] Thus, in calculating the predicted value, two or more averages or attribute information may be calculated as the predicted value of the prediction mode. In addition, the two or more averages or attribute information may each represent the values ​​of two or more components defining the color space.

[0754] In addition, for example, Figure 72 If the prediction mode value "2" is selected in the table, the Y component, U component, and V component of the attribute value of the 3D point to be encoded can also be used as the prediction values ​​Ya2, Ua2, and Va2, respectively. In this case, "2" as the prediction mode value is added to the bitstream.

[0755] Figure 73 This figure shows a fourth example of a table showing predicted values ​​calculated in each prediction mode in Embodiment 9. Specifically, Example 4 illustrates a case where the attribute information used in the predicted values ​​is based on reflectance information of surrounding three-dimensional points. For example, the reflectance information is information indicating reflectance R.

[0756] like Figure 73 As shown in the figure, the predicted value calculated in the prediction mode indicated by the prediction mode value "0" is the weighted average Rave of the reflectances Rb1, Ra2, Ra1, and Ra0 corresponding to points b1, a2, a1, and a0, respectively. Furthermore, the predicted values ​​calculated in the prediction modes indicated by the prediction mode values ​​"1" to "4" are the reflectances Rb1, Ra2, Ra1, and Ra0 of the surrounding three-dimensional points b1, a2, a1, and a0, respectively.

[0757] In addition, for example, Figure 73 When the prediction mode value "3" is selected in the table, the reflectivity of the attribute value of the 3D point to be encoded can also be used as the prediction value Ra1 for encoding. In this case, "3" as the prediction mode value is added to the bitstream.

[0758] like Figure 72 and Figure 73 As shown, the attribute information may include first attribute information and second attribute information of a different type from the first attribute information. The first attribute information may be, for example, color information. The second attribute information may be, for example, reflectance information. When calculating the predicted value, the first predicted value may be calculated using the first attribute information, and the second predicted value may be calculated using the second attribute information.

[0759] In the case where attribute information has multiple components, such as color information such as YUV color space and RGB color space, prediction values ​​can also be calculated in prediction modes separated for each component. For example, in the case of YUV space, each prediction value using the Y component, the U component, and the V component can be calculated by the prediction mode selected in each component. For example, prediction mode values ​​can also be selected in prediction mode Y for calculating the prediction value using the Y component, prediction mode U for calculating the prediction value using the U component, and prediction mode V for calculating the prediction value using the V component. In this case, the prediction mode value representing the prediction mode of each component is used using the prediction mode described later. Figures 74 to 76 These prediction mode values ​​can also be added to the bit stream respectively. In addition, in the above, the YUV color space is described, but the same can be applied to the RGB color space.

[0760] In addition, the prediction values ​​of two or more components among the multiple components containing attribute information can also be calculated using a common prediction mode. For example, in the case of the YUV color space, prediction mode values ​​can be selected from the prediction mode Y for calculating the prediction value using the Y component and the prediction mode UV for calculating the prediction value using the UV component. In this case, the prediction mode value indicating the prediction mode of each component is used using the prediction mode described later. Figure 74 as well as Figure 77 These prediction mode values ​​can also be attached to the bit stream separately.

[0761] Figure 74 This figure shows a fifth example of a table showing predicted values ​​calculated in each prediction mode in Embodiment 9. Specifically, the fifth example is an example in which the attribute information used for the predicted value is the value of the Y component of the color information of the surrounding three-dimensional points.

[0762] like Figure 74 As shown, the predicted value calculated in prediction mode Y indicated by a prediction mode value of "0" is the weighted average Yave of the Y component values ​​Yb1, Ya2, Ya1, and Ya0 corresponding to points b1, a2, a1, and a0, respectively. Furthermore, the predicted values ​​calculated in prediction modes indicated by prediction mode values ​​of "1" to "4" are the Y component values ​​Yb1, Ya2, Ya1, and Ya0 of the surrounding three-dimensional points b1, a2, a1, and a0, respectively.

[0763] In addition, for example, Figure 74 In the table, when prediction mode Y is selected, which is indicated by a prediction mode value of "2", the Y component of the attribute value of the 3D point to be encoded can also be used as the prediction value Ya2 for encoding. In this case, "2" as the prediction mode value is added to the bitstream.

[0764] Figure 75 This figure shows a sixth example of a table showing predicted values ​​calculated in each prediction mode in Embodiment 9. Specifically, the sixth example is an example in which the attribute information used for the predicted value is the value of the U component of the color information of the surrounding three-dimensional points.

[0765] like Figure 75 As shown, the predicted value calculated in the prediction mode U indicated by the prediction mode value "0" is the weighted average Uave of the U component values ​​corresponding to the points b1, a2, a1, and a0, namely Ub1, Ua2, Ua1, and Ua0. In addition, the predicted values ​​calculated in the prediction modes indicated by the prediction mode values ​​"1" to "4" are the U component values ​​Ub1, Ua2, Ua1, and U0 of the surrounding three-dimensional points b1, a2, a1, and a0, respectively.

[0766] In addition, for example, Figure 75 When prediction mode U is selected with a prediction mode value of "1" in the table, the U component of the attribute value of the 3D point to be encoded can also be used as the prediction value Ub1 for encoding. In this case, "1" as the prediction mode value is added to the bitstream.

[0767] Figure 76 This figure shows a seventh example of a table showing predicted values ​​calculated in each prediction mode in Embodiment 9. Specifically, the seventh example is an example in which the attribute information used for the predicted value is the value of the V component of the color information of the surrounding three-dimensional points.

[0768] like Figure 76 As shown, the predicted value calculated in prediction mode V, indicated by a prediction mode value of "0," is the weighted average Vave of the V component values ​​corresponding to points b1, a2, a1, and a0, namely, Vb1, Va2, Va1, and Va0. Furthermore, the predicted values ​​calculated in prediction modes indicated by prediction mode values ​​of "1" to "4" are the V component values ​​Vb1, Va2, Va1, and Va0 of the surrounding three-dimensional points b1, a2, a1, and a0, respectively.

[0769] In addition, for example, Figure 76 In the case where prediction mode V indicated by the prediction mode value "4" is selected in the table, the V component of the attribute value of the 3D point to be encoded can also be used as the prediction value Va0 for encoding. In this case, "4" as the prediction mode value is added to the bitstream.

[0770] Figure 77 This figure shows an eighth example of a table showing predicted values ​​calculated in each prediction mode in Embodiment 9. Specifically, the eighth example is an example in which the attribute information used in the predicted values ​​is the U component value and the V component value of the color information of the surrounding three-dimensional points.

[0771] like Figure 77 As shown, the predicted value calculated in prediction mode U, indicated by a prediction mode value of "0", includes: the weighted average Uave of the U component values ​​corresponding to points b1, a2, a1, and a0, namely Ub1, Ub2, Ua1, and Ua0; and the weighted average Vave of the V component values ​​corresponding to points b1, a2, a1, and a0, namely Vb1, Vb2, Va1, and Va0. In addition, the predicted value calculated in prediction modes indicated by prediction mode values ​​of "1" to "4" includes the U component values ​​and V component values ​​of the surrounding three-dimensional points b1, a2, a1, and a0, respectively.

[0772] In addition, for example, Figure 77 If the prediction mode UV indicated by the prediction mode value "1" is selected in the table, the U component and V component of the attribute value of the 3D point to be encoded can be used as the prediction values ​​Ub1 and Vb1, respectively. In this case, "1" as the prediction mode value is added to the bitstream.

[0773] The prediction mode during encoding can also be selected through RD optimization. For example, one can consider calculating the cost (P) when a certain prediction mode P is selected and selecting the prediction mode P with the lowest cost (P). The cost (P) can be calculated using Equation D1 using, for example, the prediction residual (P) for the predicted value of prediction mode P, the number of bits (P) required to encode prediction mode P, and the adjustment parameter λ.

[0774] cost(P)=abs(residual(P))+λ×bit(P)…(Formula D1)

[0775] abs(x) represents the absolute value of x.

[0776] Instead of abs(x), the square value of x may be used.

[0777] By using the above-mentioned formula D1, a prediction mode can be selected that takes into account the balance between the size of the prediction residual and the number of bits required to encode the prediction mode. In addition, the adjustment parameter λ can also be set to a different value depending on the value of the quantization scale. For example, when the quantization scale is small (at a high bit rate), by reducing the λ value, a prediction mode in which the prediction residual (P) becomes smaller can be selected to maximize the prediction accuracy. When the quantization scale is large (at a low bit rate), by increasing the λ value, an appropriate prediction mode can be selected while considering the number of bits (bit(P)) required to encode the prediction mode P.

[0778] Furthermore, a small quantization scale is, for example, smaller than the first quantization scale. A large quantization scale is, for example, larger than a second quantization scale that is greater than the first quantization scale. Furthermore, the smaller the quantization scale, the smaller the λ value may be set.

[0779] The prediction residual (P) is calculated by subtracting the prediction value of prediction mode P from the attribute value of the 3D point being encoded. Alternatively, instead of using the prediction residual (P) during cost calculation, the prediction residual (P) can be quantized and inverse-quantized, and then added to the prediction value to obtain a decoded value. The difference (coding error) between the original 3D point attribute value and the decoded value using prediction mode P can be reflected in the cost value. This allows the prediction mode with the smallest coding error to be selected.

[0780] The number of bits bit(P) required to encode the prediction mode P may be the number of bits after binarization, for example, when the prediction mode is binarized and encoded. For example, when the number of prediction modes M=5, Figure 78 As shown, the prediction mode value representing the prediction mode can also be binarized using a truncated unary code with a maximum value of 5, using the prediction mode number M. In this case, 1 bit is used for the prediction mode value "0," 2 bits are used for the prediction mode value "1," 3 bits are used for the prediction mode value "2," and 4 bits are used for the prediction mode values ​​"4" and "5," respectively, as the number of bits required to encode the prediction mode value (bits P). By using truncated unary coding, the smaller the prediction mode value, the fewer bits are used. Therefore, the amount of code required to encode the prediction mode value representing the prediction mode calculated using the average value of the prediction value calculated for the prediction mode value "0" or the 3D point attribute information calculated for the prediction mode value "1," i.e., the attribute information of the 3D point close to the encoding target 3D point, which is likely to have the smallest cost (P).

[0781] In addition, when the maximum value of the number of prediction modes is not determined, Figure 79 As shown in FIG, the prediction mode value representing the prediction mode may be binarized using a unary code. In addition, when the occurrence probabilities of the prediction modes are close, as shown in FIG. Figure 80 As shown, the amount of coding may be reduced by binarizing the prediction mode value indicating the prediction mode using fixed coding.

[0782] Alternatively, as the number of bits bit(P) required to encode the prediction mode value representing prediction mode P, the binary data representing the prediction mode value of prediction mode P may be arithmetic-coded, and the amount of code after the arithmetic coding may be set to the value bit(P). This allows the cost to be calculated using a more accurate number of bits bit(P), thereby enabling a more appropriate prediction mode to be selected.

[0783] in addition, Figure 78 This is a diagram showing a first example of a binarization table when binarizing and encoding prediction mode values ​​in Embodiment 9. Specifically, the first example is an example of binarizing prediction mode values ​​by truncated unary encoding when the number of prediction modes M=5.

[0784] in addition, Figure 79 This is a diagram showing a second example of a binarization table when binarizing and encoding prediction mode values ​​in Embodiment 9. Specifically, the second example is an example of binarizing prediction mode values ​​using unary encoding when the number of prediction modes M=5.

[0785] in addition, Figure 80 This is a diagram showing a third example of a binarization table for binarizing and encoding prediction mode values ​​in Embodiment 9. Specifically, the third example is an example of binarizing prediction mode values ​​using fixed encoding when the number of prediction modes M=5.

[0786] The prediction mode value representing the prediction mode (PredMode) may also be binarized, arithmetically coded, and appended to the bitstream. As described above, the prediction mode value may be binarized, for example, by truncated unary coding using the value of the prediction mode number M. In this case, the maximum number of bits after binarization of the prediction mode value is M-1.

[0787] In addition, the binary data after binarization can also be arithmetically coded using a coding table. In this case, for example, it is also possible to perform coding by switching the coding table for each bit of the binary data, thereby improving the coding efficiency. In addition, in order to reduce the number of coding tables, it is also possible to use the coding table A for one bit to encode the first bit of the binary data, and use the coding table B for the remaining bits (remaining code) to encode the remaining bits. For example, in Figure 81 When encoding the binary data "1110" with the prediction mode value "3" shown, the first bit "1" can be encoded using coding table A, and the remaining bits "110" can be encoded using coding table B.

[0788] in addition, Figure 81 This is a diagram for explaining an example of encoding binary data of a binarization table when the prediction mode of the ninth embodiment is binarized and encoded. Figure 81 The binarization table in is an example of binarizing the prediction mode value by truncated unary coding when the number of prediction modes M=5.

[0789] Thus, by switching the coding table according to the bit position of the binary data while reducing the number of coding tables, the coding efficiency can be improved. In addition, when encoding the remaining bit, the coding table can be further switched for each bit to perform arithmetic coding, or the coding table can be switched according to the result of arithmetic coding to perform decoding.

[0790] When encoding prediction mode values ​​by binarizing them using truncated unary coding using the prediction mode number M, the prediction mode number M used for the truncated unary coding may be added to a header of the bitstream, etc., to enable the prediction mode to be determined based on the decoded binary data at the decoding end. Furthermore, the possible value MaxM for the prediction mode number may be specified by a standard, etc., or the value MaxM-M (M <= MaxM) may be added to the header. Furthermore, the prediction mode number M may not be added to the stream, but may be specified by a profile or level of a standard, etc.

[0791] Furthermore, considering prediction mode values ​​binarized using truncated unary coding, arithmetic coding can be performed by switching the coding table between the one-bit portion and the remaining portion as described above. Furthermore, the occurrence probabilities of 0 and 1 in each coding table can be updated based on the values ​​of the binary data actually generated. Furthermore, the occurrence probabilities of 0 and 1 in any coding table can be fixed. This can also reduce the number of updates to the occurrence probabilities and the amount of processing. For example, the occurrence probability of the one-bit portion can be updated, while the occurrence probability of the remaining bit portion can be fixed.

[0792] Figure 82 This is a flowchart showing an example of encoding of a prediction mode value according to the ninth embodiment. Figure 83 This is a flowchart showing an example of decoding of a prediction mode value according to the ninth embodiment.

[0793] like Figure 82 As shown, in encoding of the prediction mode value, first, the prediction mode value is binarized by truncated unary encoding using the prediction mode number M ( S3401 ).

[0794] Next, arithmetic coding is performed on the truncated unary-coded binary data (S3402). Thus, the binary data is included in the bitstream as a prediction mode.

[0795] In addition, if Figure 83 As shown, in decoding the prediction mode value, first, the bit stream is arithmetically decoded using the prediction mode number M to generate truncated unary coded binary data (S3403).

[0796] Next, a prediction mode value is calculated based on the truncated unary-coded binary data ( S3404 ).

[0797] As a method of binarizing the prediction mode value representing the prediction mode (PredMode), an example of binarization using truncated unary coding of the value of the prediction mode number M is shown, but it is not necessarily limited to this. For example, the prediction mode value can also be binarized using truncated unary coding in which the prediction value is assigned to the number L (L<=M) of prediction modes. For example, in the case where the number of prediction modes M=5, when there are only two surrounding three-dimensional points that can be used in the prediction of a three-dimensional point of a certain encoding object, as shown in FIG. Figure 84 As shown in , there are 3 prediction modes available and the remaining 2 prediction modes are unavail...

Claims

1. A three-dimensional data encoding method, the three-dimensional data encoding method comprising: (i) when the number of a plurality of reference values ​​that can be used to calculate a predicted value of a current value in a prediction mode selected from two or more prediction modes is greater than a prescribed number corresponding to the selected prediction mode, (a) calculating the predicted value according to the selected prediction mode, and (b) calculating a difference between the current value and the calculated predicted value, i.e., a prediction residual, (ii) if the number is the same as or smaller than the prescribed number, not performing the calculation of the predicted value, and A bitstream including the prediction residual is generated.

2. The three-dimensional data encoding method according to claim 1, wherein: (ii) when the number is the same as or smaller than the prescribed number, allocating the current value as the prediction residual.

3. A three-dimensional data decoding method, the three-dimensional data decoding method comprising: (i) in a case where the number of a plurality of reference values ​​of a prediction value that can be used to calculate a current value in a prediction mode selected from two or more prediction modes is greater than a prescribed number corresponding to the selected one prediction mode, (a) obtaining a bit stream including a prediction mode and a prediction residual, (b) calculating the prediction value according to the obtained one prediction mode; and (c) calculating the current value by adding the prediction value to the prediction residual, and (ii) when the number is the same as or smaller than the prescribed number, the calculation of the predicted value is not performed.

4. The three-dimensional data decoding method according to claim 3, wherein: (ii) when the number is the same as or smaller than the prescribed number, allocating the prediction residual as the current value.

Images