Encoding method, decoding method, encoding device, and decoding device

Abstract

To improve encoding efficiency.SOLUTION: An encoding method determines whether a count value showing the number of adjacent nodes including a coefficient is equal to or more than a threshold, or less than the threshold, generates a plurality of first coefficients by performing first conversion processing including prediction processing using one or more adjacent modes to a plurality of coefficients of a plurality of child nodes if the count value is equal to or more than the threshold, and generates a plurality of second coefficients by performing second conversion processing including no prediction processing using one or more adjacent nodes to the plurality of coefficients of the plurality of child nodes if the count value is less than the threshold.SELECTED DRAWING: Figure 108

Description

[Technical field]

[0001] The present disclosure relates to a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, and a three-dimensional data decoding device. [Background technology]

[0002] In the future, devices and services that utilize 3D data are expected to become widespread in a wide range of fields, such as computer vision for autonomous operation of automobiles or robots, map information, monitoring, infrastructure inspection, video distribution, etc. 3D data is acquired in various ways, such as distance sensors such as range finders, stereo cameras, or a combination of multiple monocular cameras.

[0003] One method of expressing three-dimensional data is a method called a point cloud, which represents the shape of a three-dimensional structure using a group of points in a three-dimensional space. In a point cloud, the positions and colors of the points are stored. Point clouds are expected to become the mainstream method of expressing three-dimensional data, but point clouds have a very large amount of data. Therefore, when storing or transmitting three-dimensional data, it is essential to compress the amount of data by encoding, just like two-dimensional video images (examples include MPEG-4 AVC or HEVC standardized by MPEG).

[0004] In addition, compression of point clouds is partially supported by public libraries (Point Cloud Library) that perform point cloud-related processing.

[0005] Furthermore, a technique is known that uses three-dimensional map data to search for and display facilities located around a vehicle (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0006] [Patent Document 1] International Publication No. 2014 / 020663 Summary of the Invention [Problem to be solved by the invention]

[0007] In the three-dimensional data encoding process and the three-dimensional data decoding process, it is desirable to be able to improve the encoding efficiency.

[0008] An object of the present disclosure is to provide a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, or a three-dimensional data decoding device that can improve encoding efficiency. [Means for solving the problem]

[0009] An encoding method according to one embodiment of the present disclosure determines whether a count value indicating the number of adjacent nodes containing coefficients is greater than or equal to a threshold value or less than the threshold value, and if the count value is greater than or equal to the threshold value, generates a plurality of first coefficients by performing a first conversion process including a prediction process using one or more adjacent nodes on a plurality of coefficients of a plurality of child nodes, and if the count value is less than the threshold value, generates a plurality of second coefficients by performing a second conversion process that does not include the prediction process using the one or more adjacent nodes on a plurality of coefficients of a plurality of child nodes.

[0010] A decoding method according to one embodiment of the present disclosure determines whether a count value indicating the number of adjacent nodes containing coefficients is greater than or equal to a threshold value or less than the threshold value, and if the count value is greater than or equal to the threshold value, generates a plurality of first inverse transform coefficients by performing a first inverse transform process including a prediction process using one or more adjacent nodes on a plurality of first coefficients, and if the count value is less than the threshold value, generates a plurality of second inverse transform coefficients by performing a second inverse transform process not including the prediction process using the one or more adjacent nodes on a plurality of first coefficients. Effect of the Invention

[0011] The present disclosure can provide a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, or a three-dimensional data decoding device that can improve encoding efficiency. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram showing a structure of encoded three-dimensional data according to the first embodiment. [Diagram 2] FIG. 2 is a diagram showing an example of a prediction structure between SPCs belonging to the lowest layer of a GOS according to the first embodiment. [Diagram 3] FIG. 3 is a diagram showing an example of an inter-layer prediction structure according to the first embodiment. [Figure 4] FIG. 4 is a diagram showing an example of the coding order of GOS according to the first embodiment. [Diagram 5] FIG. 5 is a diagram showing an example of the coding order of GOS according to the first embodiment. [Figure 6] FIG. 6 is a diagram showing an example of meta information according to the first embodiment. [Figure 7] FIG. 7 is a schematic diagram showing transmission and reception of three-dimensional data between vehicles according to the second embodiment. [Figure 8] FIG. 8 is a diagram illustrating an example of three-dimensional data transmitted between vehicles according to the second embodiment. [Figure 9] FIG. 9 is a diagram for explaining a transmission process of three-dimensional data according to the third embodiment. [Figure 10] FIG. 10 is a diagram showing a configuration of a system according to the fourth embodiment. [Figure 11] FIG. 11 is a block diagram of a client device according to the fourth embodiment. [Figure 12] FIG. 12 is a block diagram of a server according to the fourth embodiment. [Figure 13] FIG. 13 is a flowchart of a three-dimensional data creation process by a client device according to the fourth embodiment. [Figure 14] FIG. 14 is a flowchart of a sensor information transmission process by a client device according to the fourth embodiment. [Figure 15] FIG. 15 is a flowchart of three-dimensional data creation processing by the server according to the fourth embodiment. [Figure 16] FIG. 16 is a flowchart of a 3D map transmission process performed by the server according to the fourth embodiment. [Figure 17] FIG. 17 is a diagram showing a configuration of a modified example of the system according to the fourth embodiment. In FIG. [Figure 18] FIG. 18 is a diagram showing configurations of a server and a client device according to the fourth embodiment. [Figure 19] FIG. 19 is a diagram illustrating the configurations of a server and a client device according to the fifth embodiment. [Figure 20] FIG. 20 is a flowchart of a process performed by the client device according to the fifth embodiment. [Figure 21] FIG. 21 is a diagram illustrating a configuration of a sensor information collecting system according to the fifth embodiment. As shown in FIG. [Figure 22] FIG. 22 is a diagram illustrating an example of a volume according to the sixth embodiment. [Diagram 23] FIG. 23 is a diagram showing an example of an octree representation of a volume according to the sixth embodiment. [Figure 24] FIG. 24 is a diagram showing an example of a bit string of a volume according to the sixth embodiment. [Diagram 25] FIG. 25 is a diagram showing an example of an octree representation of a volume according to the sixth embodiment. [Figure 26] FIG. 26 is a diagram illustrating an example of a volume according to the sixth embodiment. [Figure 27] FIG. 27 is a diagram for explaining encoding of attribute information using RAHT according to the seventh embodiment. [Figure 28] FIG. 28 is a diagram showing an example of setting a quantization scale for each layer according to the seventh embodiment. In FIG. [Figure 29] FIG. 29 is a diagram showing examples of the first code string and the second code string according to the seventh embodiment. In FIG. [Diagram 30] FIG. 30 is a diagram showing an example of a truncated unary code according to the seventh embodiment. [Diagram 31] FIG. 31 is a diagram for explaining the inverse Haar transform according to the seventh embodiment. [Diagram 32] FIG. 32 is a diagram illustrating an example of the syntax of attribute information according to the seventh embodiment. [Diagram 33] FIG. 33 is a diagram showing an example of coding coefficients and ZeroCnt according to the seventh embodiment. [Diagram 34] FIG. 34 is a flowchart of three-dimensional data encoding processing according to the seventh embodiment. [Diagram 35] FIG. 35 is a flowchart of the attribute information encoding process according to the seventh embodiment. [Diagram 36] FIG. 36 is a flowchart of a coding coefficient coding process according to the seventh embodiment. [Figure 37] FIG. 37 is a flowchart of three-dimensional data decoding processing according to the seventh embodiment. [Figure 38] FIG. 38 is a flowchart of the attribute information decoding process according to the seventh embodiment. [Figure 39] FIG. 39 is a flowchart of the coding coefficient decoding process according to the seventh embodiment. [Diagram 40] FIG. 40 is a block diagram of an attribute information encoding unit according to the seventh embodiment. As shown in FIG. [Diagram 41] FIG. 41 is a block diagram of an attribute information decoding unit according to the seventh embodiment. [Diagram 42] FIG. 42 is a diagram showing an example of a first code string and a second code string according to a modification of the seventh embodiment. In FIG. [Diagram 43] FIG. 43 is a diagram illustrating an example of the syntax of attribute information according to a modification of the seventh embodiment. [Diagram 44] FIG. 44 is a diagram showing examples of coding coefficients, ZeroCnt, and TotalZeroCnt according to a modification of the seventh embodiment. [Diagram 45] FIG. 45 is a flowchart of a coding coefficient coding process according to a modification of the seventh embodiment. [Diagram 46] FIG. 46 is a flowchart of a coding coefficient decoding process according to a modification of the seventh embodiment. [Figure 47] FIG. 47 is a diagram illustrating an example of the syntax of attribute information according to a modification of the seventh embodiment. [Figure 48] FIG. 48 is a diagram showing a configuration of a three-dimensional data encoding device according to the eighth embodiment. [Figure 49] FIG. 49 is a diagram showing a configuration of a three-dimensional data decoding device according to the eighth embodiment. [Figure 50] FIG. 50 is a diagram for explaining a RAHT according to the eighth embodiment. [Figure 51] FIG. 51 is a diagram for explaining integer-to-integer conversion according to the eighth embodiment. In FIG. [Figure 52] FIG. 52 is a diagram for explaining the hierarchical conversion process according to the eighth embodiment. In FIG. [Figure 53] FIG. 53 is a block diagram of a three-dimensional data encoding device according to the eighth embodiment. [Figure 54] FIG. 54 is a block diagram of a lossless attribute information encoding unit according to the eighth embodiment. [Figure 55] FIG. 55 is a block diagram of an integer conversion unit according to the eighth embodiment. [Figure 56] FIG. 56 is a block diagram of a three-dimensional data decoding device according to the eighth embodiment. [Figure 57] FIG. 57 is a block diagram of a lossless attribute information decoding unit according to the eighth embodiment. [Figure 58] FIG. 58 is a block diagram of an inverse integer transform unit according to the eighth embodiment. [Figure 59] FIG. 59 is a flowchart of a lossless attribute information encoding process according to the eighth embodiment. [Figure 60] FIG. 60 is a flowchart of a lossless attribute information decoding process according to the eighth embodiment. [Figure 61] FIG. 61 is a diagram illustrating an example of the configuration of an integer Haar transform unit according to the eighth embodiment. As shown in FIG. [Figure 62] FIG. 62 is a diagram illustrating an example of the configuration of an inverse integer Haar transform unit according to the eighth embodiment. As shown in FIG. [Figure 63] FIG. 63 is a diagram showing a configuration of a three-dimensional data encoding device according to the eighth embodiment. [Figure 64]FIG. 64 is a diagram showing a configuration of a three-dimensional data decoding device according to the eighth embodiment. [Figure 65] FIG. 65 is a diagram showing a configuration of a three-dimensional data encoding device according to the eighth embodiment. As shown in FIG. [Figure 66] FIG. 66 is a diagram showing a configuration of a three-dimensional data decoding device according to the eighth embodiment. [Figure 67] FIG. 67 is a diagram showing an example of a bitstream configuration according to the eighth embodiment. [Figure 68] FIG. 68 is a diagram showing an example of a configuration of a bit stream according to the eighth embodiment. [Figure 69] FIG. 69 is a diagram showing a configuration of a three-dimensional data encoding device according to the ninth embodiment. [Figure 70] FIG. 70 is a diagram showing a configuration of a three-dimensional data decoding device according to the ninth embodiment. [Figure 71] FIG. 71 is a diagram showing an example of a configuration of a bit stream according to the ninth embodiment. [Figure 72] FIG. 72 is a diagram illustrating an example of the configuration of an integer RAHT-Haar transform unit according to the ninth embodiment. [Figure 73] FIG. 73 is a diagram illustrating an example of the configuration of an inverse integer RAHT-Haar transform unit according to the ninth embodiment. [Figure 74] FIG. 74 is a flowchart of three-dimensional data encoding processing according to the ninth embodiment. [Figure 75] FIG. 75 is a flowchart of three-dimensional data decoding processing according to the ninth embodiment. [Figure 76] FIG. 76 is a diagram illustrating a prediction process according to the tenth embodiment. [Figure 77] FIG. 77 is a diagram showing the relationship between the nodes according to the tenth embodiment. In FIG. [Figure 78] FIG. 78 is a diagram showing a first example of an encoding method according to the tenth embodiment. In FIG. [Figure 79] FIG. 79 is a diagram showing a first example of a decoding method according to the tenth embodiment. In FIG. [Figure 80]FIG. 80 is a diagram showing a second example of the encoding method according to the tenth embodiment. In FIG. [Figure 81] FIG. 81 is a diagram showing a second example of the decoding method according to the tenth embodiment. In FIG. [Figure 82] FIG. 82 is a diagram showing a third example of the encoding method according to the tenth embodiment. In FIG. [Figure 83] FIG. 83 is a diagram showing a third example of the decoding method according to the tenth embodiment. In FIG. [Figure 84] FIG. 84 is a diagram showing a fourth example of the encoding method according to the tenth embodiment. In FIG. [Figure 85] FIG. 85 is a diagram showing a fourth example of the decoding method according to the tenth embodiment. In FIG. [Figure 86] FIG. 86 is a diagram showing a fifth example of the encoding method according to the tenth embodiment. In FIG. [Figure 87] FIG. 87 is a diagram showing a fifth example of the decoding method according to the tenth embodiment. In FIG. [Figure 88] FIG. 88 is a flowchart of three-dimensional data encoding processing according to the tenth embodiment. [Figure 89] FIG. 89 is a flowchart of three-dimensional data decoding processing according to the tenth embodiment. [Figure 90] FIG. 90 is a diagram illustrating a prediction process according to the eleventh embodiment. [Figure 91] FIG. 91 is a diagram showing a first example of the encoding process according to the eleventh embodiment. As shown in FIG. [Figure 92] FIG. 92 is a diagram illustrating a first example of the decoding process according to the eleventh embodiment. In FIG. [Figure 93] FIG. 93 is a diagram illustrating an example of the syntax of an attribute information header according to the eleventh embodiment. [Figure 94] FIG. 94 is a diagram illustrating an example of the syntax of an attribute information header according to the eleventh embodiment. [Figure 95] FIG. 95 is a flowchart of three-dimensional data encoding processing according to the eleventh embodiment. [Figure 96] FIG. 96 is a flowchart of three-dimensional data decoding processing according to the eleventh embodiment. [Figure 97] FIG. 97 is a diagram showing a second example of the encoding process according to the eleventh embodiment. As shown in FIG. [Figure 98] FIG. 98 is a diagram illustrating an example of the syntax of an attribute information header according to the eleventh embodiment. [Figure 99] FIG. 99 is a diagram showing an example of the syntax of an attribute information header according to the eleventh embodiment. [Figure 100] FIG. 100 is a flowchart of three-dimensional data encoding processing according to the eleventh embodiment. [Figure 101] FIG. 101 is a flowchart of three-dimensional data decoding processing according to the eleventh embodiment. [Figure 102] FIG. 102 is a diagram showing a third example of the encoding process according to the eleventh embodiment. [Figure 103] FIG. 103 is a diagram illustrating an example of the syntax of an attribute information header according to the eleventh embodiment. [Figure 104] FIG. 104 is a diagram illustrating an example of the syntax of an attribute information header according to the eleventh embodiment. [Figure 105] FIG. 105 is a flowchart of three-dimensional data encoding processing according to the eleventh embodiment. [Figure 106] FIG. 106 is a flowchart of three-dimensional data decoding processing according to the eleventh embodiment. [Figure 107] FIG. 107 is a diagram showing an example of changing the reference range and the reference count according to the eleventh embodiment. [Figure 108] FIG. 108 is a flowchart of three-dimensional data encoding processing according to the eleventh embodiment. [Fig. 109] FIG. 109 is a flowchart of three-dimensional data decoding processing according to the eleventh embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] A three-dimensional data encoding method according to one embodiment of the present disclosure determines whether a first effective node number, which is the number of effective nodes that are nodes including three-dimensional points and are included in multiple first nodes belonging to a higher layer than a target node in an N-ary tree structure of multiple three-dimensional points included in point cloud data, is greater than or equal to a predetermined first threshold, and if the first effective node number is greater than or equal to the first threshold, a first encoding including a prediction process using multiple second nodes that include a parent node of the target node and belong to the same layer as the parent node is performed on attribute information of the target node, and if the first effective node number is less than the first threshold, a second encoding that does not include the prediction process using the multiple second nodes is performed on the attribute information of the target node.

[0014] According to this, the three-dimensional data encoding method can appropriately select whether or not to use the first encoding including the prediction process, thereby improving the encoding efficiency.

[0015] For example, the plurality of first nodes may include the parent node and a plurality of nodes that belong to the same hierarchical level as the parent node.

[0016] For example, the plurality of first nodes may include a grandparent node of the target node and a plurality of nodes that belong to the same hierarchical level as the grandparent node.

[0017] For example, in the second encoding, a predicted value of the attribute information of the target node may be set to zero.

[0018] For example, the three-dimensional data encoding method may further generate a bitstream including encoded attribute information of the target node and first information indicating whether the first encoding is applicable.

[0019] For example, the three-dimensional data encoding method may further generate a bit stream including encoded attribute information of the target node and second information indicating the first threshold value.

[0020] For example, it may be determined whether a second effective node number, which is the number of effective nodes included in a plurality of second nodes including a grandparent node of the target node and a plurality of nodes belonging to the same hierarchical level as the grandparent node, is greater than or equal to a predetermined second threshold, and if the first effective node number is greater than the first threshold and the second effective node number is greater than or equal to the second threshold, the first encoding is performed on the attribute information of the target node, and if the first effective node number is less than the first threshold or the second effective node number is less than the second threshold, the second encoding is performed on the attribute information of the target node.

[0021] A three-dimensional data decoding method according to one embodiment of the present disclosure determines whether a first effective node number, which is the number of effective nodes that are nodes including three-dimensional points and are included in multiple first nodes belonging to a higher layer than a target node in an N-ary tree structure of multiple three-dimensional points included in point cloud data, is greater than or equal to a predetermined first threshold, and if the first effective node number is greater than or equal to the first threshold, performs a first decoding on attribute information of the target node, which includes a parent node of the target node and includes a prediction process using multiple second nodes that belong to the same layer as the parent node, and if the first effective node number is less than the first threshold, performs a second decoding on the attribute information of the target node that does not include the prediction process using the multiple second nodes.

[0022] According to this, the three-dimensional data decoding method can appropriately select whether or not to use the first decoding including the prediction process, thereby improving the coding efficiency.

[0023] For example, the plurality of first nodes may include the parent node and a plurality of nodes that belong to the same hierarchical level as the parent node.

[0024] For example, the plurality of first nodes may include a grandparent node of the target node and a plurality of nodes that belong to the same hierarchical level as the grandparent node.

[0025] For example, in the second decoding, a predicted value of the attribute information of the target node may be set to zero.

[0026] For example, the three-dimensional data decoding method may further include acquiring first information indicating whether the first decoding is applicable or not from a bit stream including encoded attribute information of the target node.

[0027] For example, the three-dimensional data decoding method may further include acquiring second information indicating the first threshold value from a bit stream including encoded attribute information of the target node.

[0028] For example, it may be determined whether a second valid node count, which is the number of valid nodes included in a plurality of second nodes including a grandparent node of the target node and a plurality of nodes belonging to the same hierarchical level as the grandparent node, is greater than or equal to a predetermined second threshold, and if the first valid node count is greater than the first threshold and the second valid node count is greater than or equal to the second threshold, the first decryption is performed on the attribute information of the target node, and if the first valid node count is less than the first threshold or the second valid node count is less than the second threshold, the second decryption is performed on the attribute information of the target node.

[0029] Moreover, a three-dimensional data encoding device according to one embodiment of the present disclosure includes a processor and a memory, and the processor uses the memory to determine whether a first effective node number, which is the number of effective nodes that are nodes including three-dimensional points and are included in a plurality of first nodes belonging to a higher layer than a target node in an N-ary tree structure of a plurality of three-dimensional points included in point cloud data, is greater than or equal to a predetermined first threshold, and if the first effective node number is greater than or equal to the first threshold, a first encoding is performed on attribute information of the target node, the first encoding including a prediction process using a plurality of second nodes that include a parent node of the target node and belong to the same layer as the parent node, and if the first effective node number is less than the first threshold, a second encoding is performed on the attribute information of the target node that does not include the prediction process using the plurality of second nodes.

[0030] According to this, the three-dimensional data encoding device can appropriately select whether or not to use the first encoding including the prediction process, thereby improving the encoding efficiency.

[0031] Moreover, a three-dimensional data decoding device according to one embodiment of the present disclosure includes a processor and a memory, and the processor uses the memory to determine whether a first effective node number, which is the number of effective nodes that are nodes including three-dimensional points and are included in a plurality of first nodes belonging to a higher layer than a target node in an N-ary tree structure of a plurality of three-dimensional points included in point cloud data, is greater than or equal to a predetermined first threshold, and if the first effective node number is greater than or equal to the first threshold, a first decoding is performed on attribute information of the target node, the first decoding including a prediction process using a plurality of second nodes that include a parent node of the target node and belong to the same layer as the parent node, and if the first effective node number is less than the first threshold, a second decoding is performed on the attribute information of the target node, the second decoding not including the prediction process using the plurality of second nodes.

[0032] According to this, the three-dimensional data decoding device can appropriately select whether or not to use the first decoding including the prediction process, thereby improving the encoding efficiency.

[0033] These comprehensive or specific aspects may be realized as a system, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM, or may be realized as any combination of a system, a method, an integrated circuit, a computer program, and a recording medium.

[0034] Hereinafter, the embodiments will be described in detail with reference to the drawings. Note that each of the embodiments described below shows a specific example of the present disclosure. The numerical values, shapes, materials, components, arrangement and connection forms of the components, steps, and order of steps shown in the following embodiments are merely examples and are not intended to limit the present disclosure. In addition, among the components in the following embodiments, components that are not described in the independent claims will be described as optional components.

[0035] (Embodiment 1) First, the data structure of encoded three-dimensional data (hereinafter, also referred to as encoded data) according to this embodiment will be described. Fig. 1 is a diagram showing the structure of encoded three-dimensional data according to this embodiment.

[0036] In this embodiment, the three-dimensional space is divided into spaces (SPC) corresponding to pictures in video coding, and the three-dimensional data is coded using the spaces as units. The space is further divided into volumes (VLM) corresponding to macroblocks in video coding, and prediction and conversion are performed using the VLM as units. The volume includes a plurality of voxels (VXL), which are the smallest units to which position coordinates are associated. Note that prediction refers to, as with prediction performed on two-dimensional images, generating predicted three-dimensional data similar to the processing unit to be processed by referring to other processing units, and coding the difference between the predicted three-dimensional data and the processing unit to be processed. In addition, this prediction includes not only spatial prediction that refers to other prediction units at the same time, but also temporal prediction that refers to prediction units at different times.

[0037] For example, when a three-dimensional data encoding device (hereinafter also referred to as an encoding device) encodes a three-dimensional space represented by point cloud data such as a point cloud, it encodes each point of the point cloud or a group of points contained in a voxel according to the size of the voxel. By subdividing the voxel, the three-dimensional shape of the point cloud can be expressed with high accuracy, and by increasing the size of the voxel, the three-dimensional shape of the point cloud can be expressed roughly.

[0038] In the following, an example will be described in which the three-dimensional data is a point cloud; however, the three-dimensional data is not limited to a point cloud, and may be three-dimensional data in any format.

[0039] Also, voxels with a hierarchical structure may be used. In this case, in the nth layer, it may be indicated in order whether a sample point exists in the n-1th or lower layer (a layer below the nth layer). For example, when decoding only the nth layer, if a sample point exists in the n-1th or lower layer, the sample point can be decoded by considering it to exist at the center of the voxel in the nth layer.

[0040] In addition, the encoding device acquires the point cloud data using a distance sensor, a stereo camera, a monocular camera, a gyro, an inertial sensor, or the like.

[0041] As in video coding, the space is classified into at least three prediction structures, including an independently decodable intra space (I-SPC), a unidirectionally referable predictive space (P-SPC), and a bidirectionally referable bidirectional space (B-SPC). The space also has two types of time information: the decoding time and the display time.

[0042] As shown in Fig. 1, there is a random access unit called a Group Of Space (GOS) which is a processing unit including multiple spaces. Furthermore, there is a World (WLD) which is a processing unit including multiple GOS.

[0043] The spatial region that the world occupies is associated with an absolute position on the earth by GPS or latitude and longitude information, etc. This position information is stored as meta information. Note that the meta information may be included in the encoded data or may be transmitted separately from the encoded data.

[0044] Furthermore, within a GOS, all SPCs may be three-dimensionally adjacent, or there may be SPCs that are not three-dimensionally adjacent to other SPCs.

[0045] In the following, the process of encoding, decoding, referencing, etc. of three-dimensional data included in a processing unit such as a GOS, SPC, or VLM will also be simply referred to as encoding, decoding, referencing, etc. of a processing unit. Also, the three-dimensional data included in a processing unit includes at least one pair of a spatial position such as three-dimensional coordinates and a characteristic value such as color information.

[0046] Next, the prediction structure of SPCs in a GOS will be explained. Multiple SPCs in the same GOS, or multiple VLMs in the same SPC, occupy different spaces from each other, but have the same time information (decoding time and display time).

[0047] In addition, the first SPC in a GOS in decoding order is the I-SPC. There are two types of GOS: closed GOS and open GOS. A closed GOS is a GOS in which all SPCs in the GOS can be decoded when decoding starts from the first I-SPC. In an open GOS, some SPCs in the GOS whose display time precedes the first I-SPC refer to a different GOS, and cannot be decoded using only that GOS.

[0048] In addition, in coded data such as map information, the WLD may be decoded in the reverse order from the coding order, and if there is a dependency between the GOS, reverse playback is difficult. Therefore, in such cases, closed GOS is basically used.

[0049] Furthermore, the GOS has a layer structure in the height direction, and encoding or decoding is performed in order starting from the SPC in the lower layer.

[0050] Fig. 2 is a diagram showing an example of a prediction structure between SPCs belonging to the lowest layer of a GOS, and Fig. 3 is a diagram showing an example of a prediction structure between layers.

[0051] There are one or more I-SPCs in a GOS. Objects such as humans, animals, cars, bicycles, traffic lights, and landmark buildings exist in a three-dimensional space, and it is effective to encode small objects as I-SPCs. For example, a three-dimensional data decoding device (hereinafter also referred to as a decoding device) decodes only the I-SPCs in the GOS when decoding the GOS with low processing load or at high speed.

[0052] Furthermore, the encoding device may switch the encoding interval or occurrence frequency of I-SPC depending on the density of objects in the WLD.

[0053] In addition, in the configuration shown in Fig. 3, the encoding device or decoding device encodes or decodes multiple layers in order from the lower layer (layer 1). This allows, for example, an autonomous vehicle to increase the priority of data near the ground, which contains more information.

[0054] In addition, in the case of encoded data used by drones, etc., encoding or decoded may be performed in order from the SPC of the upper layer in the height direction within the GOS.

[0055] Alternatively, the encoding device or decoding device may encode or decode multiple layers so that the decoding device can roughly grasp the GOS and gradually increase the resolution. For example, the encoding device or decoding device may encode or decode layers 3, 8, 1, 9, etc. in that order.

[0056] Next, how to handle static and dynamic objects will be described.

[0057] In a three-dimensional space, there exist static objects or scenes such as buildings or roads (hereinafter collectively referred to as static objects), and dynamic objects such as cars or people (hereinafter referred to as dynamic objects). Object detection is performed separately by extracting feature points from point cloud data or camera images such as a stereo camera. Here, an example of a method for encoding dynamic objects will be described.

[0058] The first method is to encode the static objects without distinguishing between the static objects and the dynamic objects, and the second method is to distinguish between the static objects and the dynamic objects by using identification information.

[0059] For example, a GOS is used as an identification unit. In this case, a GOS including SPCs constituting static objects and a GOS including SPCs constituting dynamic objects are distinguished by identification information stored in the coded data or separately from the coded data.

[0060] Alternatively, the SPC may be used as the identification unit. In this case, the SPC including the VLM constituting the static object and the SPC including the VLM constituting the dynamic object are distinguished by the above-mentioned identification information.

[0061] Alternatively, the VLM or VXL may be used as the identification unit. In this case, the VLM or VXL including a static object is distinguished from the VLM or VXL including a dynamic object by the above-mentioned identification information.

[0062] The encoding device may also encode a dynamic object as one or more VLMs or SPCs, and encode a VLM or SPC including a static object and an SPC including a dynamic object as different GOSs. In addition, when the size of the GOS varies depending on the size of the dynamic object, the encoding device stores the size of the GOS separately as meta information.

[0063] The encoding device may also encode the static object and the dynamic object independently of each other, and overlay the dynamic object on the world composed of the static objects. In this case, the dynamic object is composed of one or more SPCs, and each SPC is associated with one or more SPCs constituting the static object on which the SPC is overlaid. Note that the dynamic object may be represented by one or more VLMs or VXLs instead of SPCs.

[0064] The encoding device may also encode static objects and dynamic objects as different streams.

[0065] The encoding device may also generate a GOS including one or more SPCs that constitute a dynamic object. Furthermore, the encoding device may set the GOS including the dynamic object (GOS_M) and the GOS of the static object corresponding to the spatial area of ​​GOS_M to the same size (occupy the same spatial area). This allows the superimposition process to be performed on a GOS-by-GOS basis.

[0066] A P-SPC or B-SPC constituting a dynamic object may refer to an SPC included in a different encoded GOS. In cases where the position of a dynamic object changes over time and the same dynamic object is encoded as a GOS at a different time, referencing across GOS is effective in terms of compression ratio.

[0067] Also, the first method and the second method may be switched depending on the use of the encoded data. For example, when the encoded three-dimensional data is used as a map, it is desirable to be able to separate dynamic objects, so the encoding device uses the second method. On the other hand, when encoding three-dimensional data of an event such as a concert or sporting event, if there is no need to separate dynamic objects, the encoding device uses the first method.

[0068] Furthermore, the decode time and display time of the GOS or SPC can be stored in the encoded data or as meta information. Furthermore, all time information of static objects may be the same. In this case, the actual decode time and display time may be determined by the decoding device. Alternatively, a different value may be assigned as the decode time for each GOS or SPC, and the same value may be assigned as the display time for all. Furthermore, a model may be introduced in which the decoder has a buffer of a predetermined size, and guarantees that decoding can be performed without failure if a bitstream is read at a predetermined bit rate according to the decode time, as in a decoder model in video coding such as the HRD (Hypothetical Reference Decoder) of HEVC.

[0069] Next, the arrangement of GOS in the world will be described. The coordinates of the three-dimensional space in the world are expressed by three mutually orthogonal coordinate axes (x-axis, y-axis, z-axis). By setting a predetermined rule for the coding order of GOS, coding can be performed so that spatially adjacent GOS are continuous in the coded data. For example, in the example shown in FIG. 4, GOS in the xz plane are coded continuously. After coding of all GOS in a certain xz plane is completed, the value of the y axis is updated. In other words, as coding progresses, the world extends in the y axis direction. Also, the index numbers of GOS are set in the coding order.

[0070] Here, the three-dimensional space of the world is associated one-to-one with absolute geographical coordinates such as GPS or latitude and longitude. Alternatively, the three-dimensional space may be expressed by a relative position from a preset reference position. The directions of the x-axis, y-axis, and z-axis of the three-dimensional space are expressed as direction vectors determined based on the latitude and longitude, and the direction vectors are stored as meta information together with the encoded data.

[0071] Moreover, the size of the GOS is fixed, and the encoding device stores the size as meta information. The size of the GOS may be switched depending on, for example, whether it is an urban area or not, or whether it is indoors or outdoors. That is, the size of the GOS may be switched depending on the amount or nature of objects that have information value. Alternatively, the encoding device may adaptively switch the size of the GOS or the interval of I-SPCs in the GOS depending on the density of objects within the same world. For example, the higher the density of objects, the smaller the size of the GOS and the shorter the interval of I-SPCs in the GOS.

[0072] In the example of Figure 5, the third to tenth GOS areas have a high density of objects, so the GOS are subdivided to achieve fine-grained random access. Note that the seventh to tenth GOS are located behind the third to sixth GOS, respectively.

[0073] (Embodiment 2) In this embodiment, a method for transmitting and receiving three-dimensional data between vehicles will be described.

[0074] FIG. 7 is a schematic diagram showing the transmission and reception of three-dimensional data 607 between a vehicle 600 and a nearby vehicle 601.

[0075] When acquiring three-dimensional data using a sensor (such as a distance sensor such as a range finder, a stereo camera, or a combination of multiple monocular cameras) mounted on the vehicle 600, an area where three-dimensional data cannot be created (hereinafter, this area is referred to as an occlusion area 604) occurs, even though it is within the sensor detection range 602 of the vehicle 600, due to obstacles such as surrounding vehicles 601. Furthermore, the accuracy of autonomous operation increases as the space for acquiring three-dimensional data becomes larger, but the sensor detection range of the vehicle 600 alone is limited.

[0076] A sensor detection range 602 of the host vehicle 600 includes an area 603 from which three-dimensional data can be acquired and an occlusion area 604. The area from which the host vehicle 600 wishes to acquire three-dimensional data includes the sensor detection range 602 of the host vehicle 600 and other areas. In addition, a sensor detection range 605 of a nearby vehicle 601 includes the occlusion area 604 and an area 606 that is not included in the sensor detection range 602 of the host vehicle 600.

[0077] The surrounding vehicle 601 transmits information detected by the surrounding vehicle 601 to the host vehicle 600. The host vehicle 600 can acquire three-dimensional data 607 of an occlusion region 604 and a region 606 outside the sensor detection range 602 of the host vehicle 600 by acquiring information detected by the surrounding vehicle 601 such as a vehicle ahead. The host vehicle 600 uses the information acquired by the surrounding vehicle 601 to complement the three-dimensional data of the occlusion region 604 and the region 606 outside the sensor detection range.

[0078] The three-dimensional data in the autonomous operation of a vehicle or a robot is used for self-location estimation, detection of surrounding conditions, or both. For example, for self-location estimation, three-dimensional data generated by the vehicle 600 based on sensor information of the vehicle 600 is used. For detection of surrounding conditions, in addition to the three-dimensional data generated by the vehicle 600, three-dimensional data acquired from a surrounding vehicle 601 is also used.

[0079] The surrounding vehicle 601 that transmits the three-dimensional data 607 to the host vehicle 600 may be determined according to the state of the host vehicle 600. For example, the surrounding vehicle 601 is a vehicle ahead when the host vehicle 600 travels straight, an oncoming vehicle when the host vehicle 600 turns right, and a vehicle behind when the host vehicle 600 reverses. Also, the driver of the host vehicle 600 may directly specify the surrounding vehicle 601 that transmits the three-dimensional data 607 to the host vehicle 600.

[0080] Furthermore, the vehicle 600 may search for a surrounding vehicle 601 that possesses three-dimensional data of an area that is included in the space in which the three-dimensional data 607 is to be acquired but cannot be acquired by the vehicle 600. The area that the vehicle 600 cannot acquire is an occlusion area 604 or an area 606 outside the sensor detection range 602, etc.

[0081] Furthermore, the host vehicle 600 may identify the occlusion region 604 based on sensor information of the host vehicle 600. For example, the host vehicle 600 identifies a region included in the sensor detection range 602 of the host vehicle 600, for which three-dimensional data cannot be created, as the occlusion region 604.

[0082] An example of operation will be described below when it is the vehicle ahead that transmits the three-dimensional data 607. Fig. 8 is a diagram showing an example of the three-dimensional data transmitted in this case.

[0083] 8, the three-dimensional data 607 transmitted from the vehicle in front is, for example, a sparse world of point cloud (SWLD). That is, the vehicle in front creates three-dimensional data (point cloud) of the WLD from information detected by the sensor of the vehicle in front, and creates three-dimensional data of the SWLD (point cloud) by extracting data whose feature amount is equal to or greater than a threshold value from the three-dimensional data of the WLD. Then, the vehicle in front transmits the created three-dimensional data of the SWLD to the vehicle 600.

[0084] The host vehicle 600 receives the SWLD and merges the received SWLD with the point cloud created by the host vehicle 600.

[0085] The transmitted SWLD has information on absolute coordinates (the position of the SWLD in the coordinate system of the three-dimensional map). The vehicle 600 can realize the merging process by overwriting the point cloud generated by the vehicle 600 based on the absolute coordinates.

[0086] The SWLD transmitted from the nearby vehicle 601 may be the SWLD of an area 606 outside the sensor detection range 602 of the host vehicle 600 and within the sensor detection range 605 of the nearby vehicle 601, or the SWLD of an occlusion area 604 for the host vehicle 600, or both. In addition, the transmitted SWLD may be the SWLD of an area used by the nearby vehicle 601 to detect the surrounding situation among the above SWLDs.

[0087] Furthermore, the surrounding vehicle 601 may change the density of the transmitted point cloud according to the available communication time based on the speed difference between the vehicle itself 600 and the surrounding vehicle 601. For example, when the speed difference is large and the available communication time is short, the surrounding vehicle 601 may lower the density (amount of data) of the point cloud by extracting three-dimensional points with large feature values ​​from the SWLD.

[0088] In addition, detecting the surrounding conditions means determining the presence or absence of people, vehicles, road construction equipment, etc., identifying their types, and detecting their positions, movement direction, movement speed, etc.

[0089] Furthermore, the host vehicle 600 may acquire braking information of the surrounding vehicle 601 instead of or in addition to the three-dimensional data 607 generated by the surrounding vehicle 601. Here, the braking information of the surrounding vehicle 601 is, for example, information indicating that the accelerator or brake of the surrounding vehicle 601 has been depressed or the degree of depression.

[0090] In addition, in the point cloud generated by each vehicle, the three-dimensional space is subdivided into random access units in consideration of low-latency communication between vehicles. On the other hand, in the case of three-dimensional maps, which are map data downloaded from a server, the three-dimensional space is divided into larger random access units compared to the case of vehicle-to-vehicle communication.

[0091] Data for areas that are likely to become occlusion areas, such as the area in front of a leading vehicle or the area behind a trailing vehicle, is divided into small random access units as data for low latency.

[0092] Since the importance of the view in front increases when driving at high speeds, each vehicle creates SWLDs with a narrower field of view in fine random access units when driving at high speeds.

[0093] If the SWLD created by the vehicle in front for transmission includes an area from which the vehicle 600 can acquire point clouds, the vehicle in front may reduce the amount of transmission by removing the point clouds from that area.

[0094] (Embodiment 3) In this embodiment, a method of transmitting three-dimensional data to a following vehicle, etc. will be described. Fig. 9 is a diagram showing an example of a target space of three-dimensional data to be transmitted to a following vehicle, etc.

[0095] Vehicle 801 transmits three-dimensional data such as a point cloud (group of points) contained in a rectangular space 802 of width W, height H, and depth D at a distance L from the vehicle 801 in front of the vehicle 801 to a traffic monitoring cloud that monitors road conditions or a following vehicle at time intervals of Δt.

[0096] If a change occurs in the three-dimensional data contained in the space 802 that has been previously transmitted, such as when a vehicle or person enters the space 802 from the outside, the vehicle 801 also transmits the three-dimensional data of the space where the change has occurred.

[0097] Although FIG. 9 shows an example in which the shape of the space 802 is a rectangular parallelepiped, the space 802 does not necessarily have to be a rectangular parallelepiped as long as it includes the space on the road ahead that is in a blind spot for following vehicles.

[0098] It is desirable to set the distance L to a distance at which the following vehicle that has received the three-dimensional data can safely stop. For example, the distance L is set to the sum of the distance the following vehicle travels while it takes to receive the three-dimensional data, the distance the following vehicle travels before it starts to decelerate in response to the received data, and the distance the following vehicle requires to safely stop after starting to decelerate. Since these distances change depending on the speed, the distance L may change depending on the speed V of the vehicle, such as L = a x V + b (a and b are constants).

[0099] The width W is set to a value at least larger than the width of the lane in which the vehicle 801 is traveling. More desirably, the width W is set to a size that includes adjacent spaces such as left and right lanes or side strips.

[0100] The depth D may be a fixed value, or may vary according to the vehicle speed V, such as D=c×V+d (c and d are constants). By setting D so that D>V×Δt, the space to be transmitted can overlap with the space that has already been transmitted in the past. This allows the vehicle 801 to more reliably transmit the space on the road to the following vehicles, etc. without omissions.

[0101] In this way, by limiting the three-dimensional data transmitted by vehicle 801 to a space that is useful for following vehicles, the volume of three-dimensional data to be transmitted can be effectively reduced, thereby achieving low communication latency and low cost.

[0102] (Embodiment 4) In the third embodiment, an example has been described in which a client device such as a vehicle transmits three-dimensional data to another vehicle or a server such as a traffic monitoring cloud. In the present embodiment, the client device transmits sensor information obtained by a sensor to the server or another client device.

[0103] First, the configuration of the system according to this embodiment will be described. Fig. 10 is a diagram showing the configuration of a transmission / reception system for a 3D map and sensor information according to this embodiment. This system includes a server 901 and client devices 902A and 902B. When there is no particular need to distinguish between the client devices 902A and 902B, they are also referred to as client devices 902.

[0104] The client device 902 is, for example, an in-vehicle device mounted on a moving object such as a vehicle. The server 901 is, for example, a traffic monitoring cloud or the like, and is capable of communicating with a plurality of client devices 902.

[0105] The server 901 transmits a three-dimensional map composed of a point cloud to the client device 902. Note that the composition of the three-dimensional map is not limited to a point cloud, and may represent other three-dimensional data such as a mesh structure.

[0106] The client device 902 transmits sensor information acquired by the client device 902 to the server 901. The sensor information includes, for example, at least one of LiDAR acquisition information, a visible light image, an infrared image, a depth image, sensor position information, and speed information.

[0107] Data transmitted between the server 901 and the client device 902 may be compressed to reduce data, or may be left uncompressed to maintain data accuracy. When compressing data, a three-dimensional compression method based on an octree structure, for example, can be used for the point cloud. Also, a two-dimensional image compression method can be used for the visible light image, the infrared image, and the depth image. The two-dimensional image compression method is, for example, MPEG-4 AVC or HEVC standardized by MPEG.

[0108] Furthermore, the server 901 transmits the three-dimensional map managed by the server 901 to the client device 902 in response to a transmission request for the three-dimensional map from the client device 902. Note that the server 901 may transmit the three-dimensional map without waiting for a transmission request for the three-dimensional map from the client device 902. For example, the server 901 may broadcast the three-dimensional map to one or more client devices 902 in a predetermined space. Furthermore, the server 901 may transmit a three-dimensional map suitable for the position of the client device 902 at regular intervals to the client device 902 that has received a transmission request once. Furthermore, the server 901 may transmit the three-dimensional map to the client device 902 every time the three-dimensional map managed by the server 901 is updated.

[0109] The client device 902 issues a request for transmitting a three-dimensional map to the server 901. For example, when the client device 902 wants to estimate its own position while driving, the client device 902 transmits a request for transmitting a three-dimensional map to the server 901.

[0110] In the following cases, the client device 902 may issue a transmission request for a three-dimensional map to the server 901. If the three-dimensional map held by the client device 902 is old, the client device 902 may issue a transmission request for a three-dimensional map to the server 901. For example, if a certain period of time has passed since the client device 902 acquired the three-dimensional map, the client device 902 may issue a transmission request for a three-dimensional map to the server 901.

[0111] The client device 902 may issue a request to send a three-dimensional map to the server 901 a certain time before the client device 902 leaves the space shown in the three-dimensional map held by the client device 902. For example, when the client device 902 is within a predetermined distance from the boundary of the space shown in the three-dimensional map held by the client device 902, the client device 902 may issue a request to send a three-dimensional map to the server 901. In addition, when the movement route and movement speed of the client device 902 are known, the time when the client device 902 will leave the space shown in the three-dimensional map held by the client device 902 may be predicted based on these.

[0112] If the error in aligning the three-dimensional data created by the client device 902 from sensor information with the three-dimensional map is equal to or greater than a certain level, the client device 902 may issue a request to the server 901 to transmit the three-dimensional map.

[0113] The client device 902 transmits the sensor information to the server 901 in response to a request to transmit the sensor information transmitted from the server 901. The client device 902 may transmit the sensor information to the server 901 without waiting for a request to transmit the sensor information from the server 901. For example, once the client device 902 receives a request to transmit the sensor information from the server 901, the client device 902 may transmit the sensor information to the server 901 periodically for a certain period of time. In addition, when an error occurs in aligning three-dimensional data created by the client device 902 based on the sensor information with the three-dimensional map obtained from the server 901, the client device 902 may determine that a change may have occurred in the three-dimensional map around the client device 902 and transmit a message to that effect along with the sensor information to the server 901.

[0114] The server 901 issues a transmission request for sensor information to the client device 902. For example, the server 901 receives location information of the client device 902, such as a GPS, from the client device 902. When the server 901 determines, based on the location information of the client device 902, that the client device 902 is approaching a space with little information in the three-dimensional map managed by the server 901, it issues a transmission request for sensor information to the client device 902 in order to generate a new three-dimensional map. The server 901 may also issue a transmission request for sensor information when it is desired to update the three-dimensional map, when it is desired to check road conditions such as during snowfall or a disaster, when it is desired to check traffic congestion conditions, or when it is desired to check incident and accident conditions, etc.

[0115] Furthermore, the client device 902 may set the amount of data of the sensor information to be transmitted to the server 901 depending on the communication state or bandwidth at the time of receiving a request to transmit the sensor information from the server 901. Setting the amount of data of the sensor information to be transmitted to the server 901 means, for example, increasing or decreasing the amount of the data itself or appropriately selecting a compression method.

[0116] 11 is a block diagram showing an example of the configuration of a client device 902. The client device 902 receives a three-dimensional map composed of a point cloud or the like from the server 901, and estimates the self-position of the client device 902 from three-dimensional data created based on sensor information of the client device 902. In addition, the client device 902 transmits the acquired sensor information to the server 901.

[0117] The client device 902 includes a data receiving unit 1011, a communication unit 1012, a reception control unit 1013, a format conversion unit 1014, multiple sensors 1015, a three-dimensional data creation unit 1016, a three-dimensional image processing unit 1017, a three-dimensional data storage unit 1018, a format conversion unit 1019, a communication unit 1020, a transmission control unit 1021, and a data transmission unit 1022.

[0118] The data receiving unit 1011 receives a three-dimensional map 1031 from the server 901. The three-dimensional map 1031 is data including a point cloud such as a WLD or a SWLD. The three-dimensional map 1031 may include either compressed data or uncompressed data.

[0119] The communication unit 1012 communicates with the server 901, and transmits data transmission requests (for example, requests to transmit a three-dimensional map) to the server 901.

[0120] The reception control unit 1013 exchanges information such as compatible formats with the communication destination via the communication unit 1012, and establishes communication with the communication destination.

[0121] The format conversion unit 1014 generates a three-dimensional map 1032 by performing format conversion and the like on the three-dimensional map 1031 received by the data receiving unit 1011. Furthermore, if the three-dimensional map 1031 is compressed or encoded, the format conversion unit 1014 performs decompression or decoding processing. Note that if the three-dimensional map 1031 is uncompressed data, the format conversion unit 1014 does not perform decompression or decoding processing.

[0122] The multiple sensors 1015 are a group of sensors, such as LiDAR, a visible light camera, an infrared camera, or a depth sensor, that acquire information about the outside of the vehicle in which the client device 902 is mounted, and generate sensor information 1033. For example, when the sensor 1015 is a laser sensor such as LiDAR, the sensor information 1033 is three-dimensional data such as a point cloud (point cloud data). Note that the number of sensors 1015 does not need to be multiple.

[0123] The three-dimensional data creation unit 1016 creates three-dimensional data 1034 of the surroundings of the host vehicle based on the sensor information 1033. For example, the three-dimensional data creation unit 1016 creates point cloud data with color information of the surroundings of the host vehicle using information acquired by the LiDAR and a visible light image acquired by a visible light camera.

[0124] The three-dimensional image processing unit 1017 performs self-position estimation processing of the vehicle using a three-dimensional map 1032 such as a received point cloud and three-dimensional data 1034 of the surroundings of the vehicle generated from the sensor information 1033. The three-dimensional image processing unit 1017 may generate three-dimensional data 1035 of the surroundings of the vehicle by combining the three-dimensional map 1032 and the three-dimensional data 1034, and perform self-position estimation processing using the generated three-dimensional data 1035.

[0125] The three-dimensional data storage unit 1018 stores a three-dimensional map 1032, three-dimensional data 1034, and three-dimensional data 1035, and the like.

[0126] The format conversion unit 1019 generates sensor information 1037 by converting the sensor information 1033 into a format supported by the receiving side. The format conversion unit 1019 may reduce the amount of data by compressing or encoding the sensor information 1037. Furthermore, the format conversion unit 1019 may omit processing when format conversion is not necessary. Furthermore, the format conversion unit 1019 may control the amount of data to be transmitted in accordance with a designated transmission range.

[0127] The communication unit 1020 communicates with the server 901, and receives data transmission requests (sensor information transmission requests) and the like from the server 901.

[0128] The transmission control unit 1021 exchanges information such as compatible formats with the communication destination via the communication unit 1020, and establishes communication.

[0129] The data transmission unit 1022 transmits the sensor information 1037 to the server 901. The sensor information 1037 includes information acquired by a plurality of sensors 1015, such as information acquired by a LiDAR, a luminance image acquired by a visible light camera, an infrared image acquired by an infrared camera, a depth image acquired by a depth sensor, sensor position information, and speed information.

[0130] Next, the configuration of the server 901 will be described. Fig. 12 is a block diagram showing an example of the configuration of the server 901. The server 901 receives sensor information transmitted from the client device 902, and creates three-dimensional data based on the received sensor information. The server 901 updates a three-dimensional map managed by the server 901 using the created three-dimensional data. In addition, the server 901 transmits the updated three-dimensional map to the client device 902 in response to a transmission request for the three-dimensional map from the client device 902.

[0131] The server 901 includes a data receiving unit 1111, a communication unit 1112, a receiving control unit 1113, a format conversion unit 1114, a three-dimensional data creation unit 1116, a three-dimensional data synthesis unit 1117, a three-dimensional data storage unit 1118, a format conversion unit 1119, a communication unit 1120, a transmission control unit 1121, and a data transmission unit 1122.

[0132] The data receiving unit 1111 receives sensor information 1037 from the client device 902. The sensor information 1037 includes, for example, information acquired by a LiDAR, a luminance image acquired by a visible light camera, an infrared image acquired by an infrared camera, a depth image acquired by a depth sensor, sensor position information, and speed information.

[0133] The communication unit 1112 communicates with the client device 902 and transmits a data transmission request (for example, a request to transmit sensor information) to the client device 902 .

[0134] The reception control unit 1113 exchanges information such as compatible formats with the communication destination via the communication unit 1112, and establishes communication.

[0135] When the received sensor information 1037 is compressed or encoded, the format conversion unit 1114 performs decompression or decoding processing to generate the sensor information 1132. Note that, when the sensor information 1037 is non-compressed data, the format conversion unit 1114 does not perform decompression or decoding processing.

[0136] The three-dimensional data creation unit 1116 creates three-dimensional data 1134 of the periphery of the client device 902 based on the sensor information 1132. For example, the three-dimensional data creation unit 1116 creates point cloud data with color information of the periphery of the client device 902 using information acquired by the LiDAR and visible light images acquired by a visible light camera.

[0137] A three-dimensional data synthesis unit 1117 synthesizes three-dimensional data 1134 created based on the sensor information 1132 with a three-dimensional map 1135 managed by the server 901, thereby updating the three-dimensional map 1135.

[0138] The three-dimensional data storage unit 1118 stores a three-dimensional map 1135 and the like.

[0139] The format conversion unit 1119 generates the three-dimensional map 1031 by converting the three-dimensional map 1135 into a format supported by the receiving side. The format conversion unit 1119 may reduce the amount of data by compressing or encoding the three-dimensional map 1135. Furthermore, the format conversion unit 1119 may omit processing when format conversion is not necessary. Furthermore, the format conversion unit 1119 may control the amount of data to be transmitted in accordance with a designated transmission range.

[0140] The communication unit 1120 communicates with the client device 902 and receives a data transmission request (a request to transmit a three-dimensional map) or the like from the client device 902 .

[0141] A transmission control unit 1121 exchanges information such as compatible formats with a communication destination via a communication unit 1120, and establishes communication.

[0142] The data transmission unit 1122 transmits the three-dimensional map 1031 to the client device 902. The three-dimensional map 1031 is data including a point cloud such as a WLD or a SWLD. The three-dimensional map 1031 may include either compressed data or uncompressed data.

[0143] Next, a description will be given of the operation flow of the client device 902. Fig. 13 is a flowchart showing the operation of the client device 902 when acquiring a three-dimensional map.

[0144] First, the client device 902 requests the server 901 to transmit a three-dimensional map (such as a point cloud) (S1001). At this time, the client device 902 may also transmit position information of the client device 902 obtained by GPS or the like, thereby requesting the server 901 to transmit a three-dimensional map related to the position information.

[0145] Next, the client device 902 receives the three-dimensional map from the server 901 (S1002). If the received three-dimensional map is compressed data, the client device 902 decodes the received three-dimensional map to generate an uncompressed three-dimensional map (S1003).

[0146] Next, the client device 902 creates three-dimensional data 1034 of the periphery of the client device 902 from sensor information 1033 obtained by the multiple sensors 1015 (S1004). Next, the client device 902 estimates its own position using the three-dimensional map 1032 received from the server 901 and the three-dimensional data 1034 created from the sensor information 1033 (S1005).

[0147] 14 is a flowchart showing an operation of the client device 902 when transmitting sensor information. First, the client device 902 receives a request to transmit sensor information from the server 901 (S1011). The client device 902 that has received the transmission request transmits sensor information 1037 to the server 901 (S1012). Note that, when the sensor information 1033 includes a plurality of pieces of information obtained by a plurality of sensors 1015, the client device 902 may generate the sensor information 1037 by compressing each piece of information using a compression method suitable for each piece of information.

[0148] Next, the operation flow of the server 901 will be described. Fig. 15 is a flowchart showing the operation when the server 901 acquires sensor information. First, the server 901 requests the client device 902 to transmit sensor information (S1021). Next, the server 901 receives the sensor information 1037 transmitted from the client device 902 in response to the request (S1022). Next, the server 901 creates three-dimensional data 1134 using the received sensor information 1037 (S1023). Next, the server 901 reflects the created three-dimensional data 1134 in the three-dimensional map 1135 (S1024).

[0149] 16 is a flowchart showing the operation of the server 901 when transmitting a three-dimensional map. First, the server 901 receives a request to transmit a three-dimensional map from the client device 902 (S1031). The server 901 that has received the request to transmit the three-dimensional map transmits a three-dimensional map 1031 to the client device 902 (S1032). At this time, the server 901 may extract a three-dimensional map of the vicinity according to the position information of the client device 902, and transmit the extracted three-dimensional map. The server 901 may also compress the three-dimensional map composed of a point cloud, for example, using a compression method based on an octet tree structure, and transmit the compressed three-dimensional map.

[0150] A modification of this embodiment will now be described.

[0151] The server 901 creates three-dimensional data 1134 of the vicinity of the position of the client device 902 using the sensor information 1037 received from the client device 902. Next, the server 901 calculates the difference between the three-dimensional data 1134 and the three-dimensional map 1135 by matching the created three-dimensional data 1134 with a three-dimensional map 1135 of the same area managed by the server 901. If the difference is equal to or greater than a predetermined threshold, the server 901 determines that some abnormality has occurred in the vicinity of the client device 902. For example, when ground subsidence or the like occurs due to a natural disaster such as an earthquake, it is considered that a large difference occurs between the three-dimensional map 1135 managed by the server 901 and the three-dimensional data 1134 created based on the sensor information 1037.

[0152] The sensor information 1037 may include information indicating at least one of the type of sensor, the performance of the sensor, and the model number of the sensor. In addition, a class ID or the like according to the performance of the sensor may be added to the sensor information 1037. For example, when the sensor information 1037 is information acquired by LiDAR, it is possible to assign an identifier to the performance of the sensor, such as class 1 for a sensor that can acquire information with an accuracy of several millimeters, class 2 for a sensor that can acquire information with an accuracy of several centimeters, and class 3 for a sensor that can acquire information with an accuracy of several meters. In addition, the server 901 may estimate the performance information of the sensor from the model number of the client device 902. For example, when the client device 902 is mounted on a vehicle, the server 901 may determine the spec information of the sensor from the model of the vehicle. In this case, the server 901 may acquire information on the model of the vehicle in advance, or the information may be included in the sensor information. In addition, the server 901 may use the acquired sensor information 1037 to switch the degree of correction for the three-dimensional data 1134 created using the sensor information 1037. For example, when the sensor performance is high accuracy (class 1), the server 901 does not perform correction on the three-dimensional data 1134. When the sensor performance is low accuracy (class 3), the server 901 applies correction according to the accuracy of the sensor to the three-dimensional data 1134. For example, the server 901 increases the degree (strength) of correction as the accuracy of the sensor decreases.

[0153] The server 901 may simultaneously issue requests to send sensor information to multiple client devices 902 in a certain space. When the server 901 receives multiple pieces of sensor information from multiple client devices 902, the server 901 does not need to use all of the sensor information to create the three-dimensional data 1134, and may select the sensor information to be used according to the performance of the sensor, for example. For example, when updating the three-dimensional map 1135, the server 901 may select high-precision sensor information (class 1) from the multiple pieces of sensor information received, and create the three-dimensional data 1134 using the selected sensor information.

[0154] The server 901 is not limited to a server such as a traffic monitoring cloud, but may be another client device (mounted in a vehicle). Fig. 17 is a diagram showing the system configuration in this case.

[0155] For example, client device 902C issues a request to send sensor information to nearby client device 902A and acquires the sensor information from client device 902A. Client device 902C then creates three-dimensional data using the acquired sensor information of client device 902A and updates the three-dimensional map of client device 902C. This allows client device 902C to generate a three-dimensional map of the space that can be acquired from client device 902A by taking advantage of the performance of client device 902C. For example, it is considered that such a case occurs when client device 902C has high performance.

[0156] In this case, the client device 902A that provided the sensor information is given the right to obtain the high-precision 3D map generated by the client device 902C. The client device 902A receives the high-precision 3D map from the client device 902C in accordance with the right.

[0157] In addition, the client device 902C may issue a request to transmit sensor information to multiple nearby client devices 902 (client device 902A and client device 902B). If the sensor of the client device 902A or the client device 902B is high performance, the client device 902C can create three-dimensional data using the sensor information obtained by the high performance sensor.

[0158] 18 is a block diagram showing the functional configuration of a server 901 and a client device 902. The server 901 includes, for example, a 3D map compression / decoding processing unit 1201 that compresses and decodes a 3D map, and a sensor information compression / decoding processing unit 1202 that compresses and decodes sensor information.

[0159] The client device 902 includes a three-dimensional map decoding processing unit 1211 and a sensor information compression processing unit 1212. The three-dimensional map decoding processing unit 1211 receives encoded data of the compressed three-dimensional map and decodes the encoded data to obtain the three-dimensional map. The sensor information compression processing unit 1212 compresses the sensor information itself instead of the three-dimensional data created from the acquired sensor information, and transmits the encoded data of the compressed sensor information to the server 901. With this configuration, the client device 902 only needs to internally hold a processing unit (device or LSI) that performs processing to decode the three-dimensional map (point cloud, etc.), and does not need to internally hold a processing unit that performs processing to compress the three-dimensional data of the three-dimensional map (point cloud, etc.). This makes it possible to reduce the cost and power consumption of the client device 902.

[0160] As described above, the client device 902 according to this embodiment is mounted on a moving body, and creates three-dimensional data 1034 of the surroundings of the moving body from sensor information 1033 indicating the surrounding conditions of the moving body, obtained by a sensor 1015 mounted on the moving body. The client device 902 estimates the self-position of the moving body using the created three-dimensional data 1034. The client device 902 transmits the acquired sensor information 1033 to the server 901 or another moving body 902.

[0161] According to this, the client device 902 transmits the sensor information 1033 to the server 901 or the like. This may reduce the amount of data to be transmitted compared to the case of transmitting three-dimensional data. Also, since the client device 902 does not need to perform processing such as compression or encoding of the three-dimensional data, the amount of processing by the client device 902 can be reduced. Therefore, the client device 902 can reduce the amount of data to be transmitted or simplify the device configuration.

[0162] Moreover, the client device 902 further transmits a request to the server 901 to transmit a three-dimensional map, and receives a three-dimensional map 1031 from the server 901. The client device 902 estimates its own location using the three-dimensional data 1034 and the three-dimensional map 1032.

[0163] The sensor information 1033 includes at least one of information obtained by a laser sensor, a luminance image, an infrared image, a depth image, sensor position information, and sensor speed information.

[0164] Furthermore, the sensor information 1033 includes information indicating the performance of the sensor.

[0165] Furthermore, the client device 902 encodes or compresses the sensor information 1033, and transmits the encoded or compressed sensor information 1037 to the server 901 or another mobile body 902. This allows the client device 902 to reduce the amount of data to be transmitted.

[0166] For example, the client device 902 includes a processor and a memory, and the processor uses the memory to perform the above-mentioned processing.

[0167] Furthermore, the server 901 according to this embodiment is capable of communicating with a client device 902 mounted on the mobile object, and receives sensor information 1037 indicating the surrounding conditions of the mobile object, obtained by a sensor 1015 mounted on the mobile object, from the client device 902. The server 901 creates three-dimensional data 1134 of the surroundings of the mobile object from the received sensor information 1037.

[0168] According to this, the server 901 creates three-dimensional data 1134 using the sensor information 1037 transmitted from the client device 902. This may reduce the amount of data transmitted compared to when the client device 902 transmits three-dimensional data. In addition, since the client device 902 does not need to perform processing such as compression or encoding of the three-dimensional data, the amount of processing by the client device 902 can be reduced. Therefore, the server 901 can reduce the amount of data transmitted or simplify the device configuration.

[0169] Moreover, the server 901 further transmits a request to the client device 902 to transmit the sensor information.

[0170] Furthermore, the server 901 further updates a three-dimensional map 1135 using the created three-dimensional data 1134 , and transmits the three-dimensional map 1135 to the client device 902 in response to a transmission request for the three-dimensional map 1135 from the client device 902 .

[0171] The sensor information 1037 includes at least one of information obtained by a laser sensor, a luminance image, an infrared image, a depth image, sensor position information, and sensor speed information.

[0172] Additionally, the sensor information 1037 includes information indicating the performance of the sensor.

[0173] Moreover, the server 901 further corrects the three-dimensional data in accordance with the performance of the sensor, thereby enabling the three-dimensional data creation method to improve the quality of the three-dimensional data.

[0174] Furthermore, in receiving the sensor information, the server 901 receives a plurality of pieces of sensor information 1037 from a plurality of client devices 902, and selects the sensor information 1037 to be used for creating the three-dimensional data 1134 based on a plurality of pieces of information indicating the performance of the sensors included in the plurality of pieces of sensor information 1037. This allows the server 901 to improve the quality of the three-dimensional data 1134.

[0175] Furthermore, the server 901 decodes or expands the received sensor information 1037, and creates three-dimensional data 1134 from the decoded or expanded sensor information 1132. This allows the server 901 to reduce the amount of data to be transmitted.

[0176] For example, the server 901 includes a processor and a memory, and the processor uses the memory to perform the above-mentioned processing.

[0177] (Embodiment 5) In this embodiment, a modification of the above-mentioned embodiment 4 will be described. Fig. 19 is a diagram showing a configuration of a system according to this embodiment. The system shown in Fig. 19 includes a server 2001, a client device 2002A, and a client device 2002B.

[0178] Client device 2002A and client device 2002B are mounted on a moving object such as a vehicle, and transmit sensor information to server 2001. Server 2001 transmits a three-dimensional map (point cloud) to client device 2002A and client device 2002B.

[0179] The client device 2002A includes a sensor information acquisition unit 2011, a storage unit 2012, and a data transmission permission determination unit 2013. The client device 2002B has a similar configuration. In the following description, the client device 2002A and the client device 2002B will also be referred to as the client device 2002 when there is no need to distinguish between them.

[0180] FIG. 20 is a flowchart showing the operation of client device 2002 according to the present embodiment.

[0181] The sensor information acquisition unit 2011 acquires various sensor information using a sensor (sensor group) mounted on the moving object. That is, the sensor information acquisition unit 2011 acquires sensor information indicating the surrounding situation of the moving object, acquired by a sensor (sensor group) mounted on the moving object. The sensor information acquisition unit 2011 also stores the acquired sensor information in the storage unit 2012. This sensor information includes at least one of LiDAR acquisition information, a visible light image, an infrared image, and a depth image. The sensor information may also include at least one of sensor position information, speed information, acquisition time information, and acquisition location information. The sensor position information indicates the position of the sensor that acquired the sensor information. The speed information indicates the speed of the moving object when the sensor acquired the sensor information. The acquisition time information indicates the time when the sensor information was acquired by the sensor. The acquisition location information indicates the position of the moving object or the sensor when the sensor information was acquired by the sensor.

[0182] Next, the data transmission possibility determination unit 2013 determines whether the mobile object (client device 2002) is in an environment where it can transmit sensor information to the server 2001 (S2002). For example, the data transmission possibility determination unit 2013 may use information such as GPS to identify the location and time of the client device 2002 and determine whether data can be transmitted. The data transmission possibility determination unit 2013 may also determine whether data can be transmitted based on whether it can connect to a specific access point.

[0183] When the client device 2002 determines that the moving object is in an environment in which it can transmit sensor information to the server 2001 (Yes in S2002), it transmits the sensor information to the server 2001 (S2003). That is, when the client device 2002 is in a situation in which it can transmit the sensor information to the server 2001, the client device 2002 transmits the sensor information it holds to the server 2001. For example, a millimeter wave access point capable of high-speed communication is installed at an intersection or the like. When the client device 2002 enters the intersection, it transmits the sensor information it holds to the server 2001 at high speed using millimeter wave communication.

[0184] Next, the client device 2002 deletes the sensor information already transmitted to the server 2001 from the storage unit 2012 (S2004). The client device 2002 may delete the sensor information when the sensor information not transmitted to the server 2001 satisfies a predetermined condition. For example, the client device 2002 may delete the sensor information from the storage unit 2012 when the acquisition time of the sensor information held by the client device 2002 becomes older than a certain time before the current time. That is, the client device 2002 may delete the sensor information from the storage unit 2012 when the difference between the time when the sensor information was acquired by the sensor and the current time exceeds a predetermined time. The client device 2002 may also delete the sensor information from the storage unit 2012 when the acquisition location of the sensor information held by the client device 2002 becomes more distant from the current location by a certain distance. That is, the client device 2002 may delete the sensor information from the storage unit 2012 when the difference between the position of the moving object or the sensor when the sensor information was acquired by the sensor and the current position of the moving object or the sensor exceeds a predetermined distance. This makes it possible to reduce the capacity of the storage unit 2012 of the client device 2002 .

[0185] If the client device 2002 has not finished acquiring the sensor information (No in S2005), the client device 2002 performs the processes from step S2001 onward again. If the client device 2002 has finished acquiring the sensor information (Yes in S2005), the client device 2002 ends the process.

[0186] Furthermore, the client device 2002 may select the sensor information to be transmitted to the server 2001 in accordance with the communication conditions. For example, when high-speed communication is possible, the client device 2002 transmits sensor information having a large size stored in the storage unit 2012 (e.g., LiDAR acquisition information, etc.) with priority. When high-speed communication is difficult, the client device 2002 transmits sensor information having a small size stored in the storage unit 2012 with high priority (e.g., visible light image). This allows the client device 2002 to efficiently transmit the sensor information stored in the storage unit 2012 to the server 2001 in accordance with the network conditions.

[0187] Furthermore, the client device 2002 may acquire time information indicating the current time and location information indicating the current location from the server 2001. Furthermore, the client device 2002 may determine the acquisition time and acquisition location of the sensor information based on the acquired time information and location information. That is, the client device 2002 may acquire time information from the server 2001 and generate acquisition time information using the acquired time information. Furthermore, the client device 2002 may acquire location information from the server 2001 and generate acquisition location information using the acquired location information.

[0188] For example, regarding time information, the server 2001 and the client device 2002 perform time synchronization using a mechanism such as NTP (Network Time Protocol) or PTP (Precision Time Protocol). This allows the client device 2002 to obtain accurate time information. In addition, since time can be synchronized between the server 2001 and multiple client devices, the time in the sensor information obtained by the different client devices 2002 can be synchronized. Therefore, the server 2001 can handle the sensor information indicating the synchronized time. Note that the time synchronization mechanism may be any method other than NTP or PTP. Furthermore, GPS information may be used as the above-mentioned time information and location information.

[0189] The server 2001 may acquire sensor information from a plurality of client devices 2002 by specifying a time or a location. For example, when an accident occurs, the server 2001 broadcasts a sensor information transmission request to a plurality of client devices 2002 by specifying the time and location of the accident in order to search for a client who was in the vicinity. Then, the client device 2002 having the sensor information of the corresponding time and location transmits the sensor information to the server 2001. That is, the client device 2002 receives a sensor information transmission request including designation information designating a location and time from the server 2001. When the client device 2002 determines that the sensor information obtained at the location and time indicated by the designation information is stored in the storage unit 2012 and that the mobile object is in an environment in which the sensor information can be transmitted to the server 2001, the client device 2002 transmits the sensor information obtained at the location and time indicated by the designation information to the server 2001. In this way, the server 2001 can acquire sensor information related to the occurrence of an accident from a plurality of client devices 2002 and use it for accident analysis, etc.

[0190] Note that the client device 2002 may refuse to transmit the sensor information when receiving a sensor information transmission request from the server 2001. Also, the client device 2002 may set in advance which of a plurality of pieces of sensor information is transmittable. Alternatively, the server 2001 may inquire of the client device 2002 each time whether or not the sensor information can be transmitted.

[0191] Also, points may be given to the client device 2002 that transmits the sensor information to the server 2001. The points may be used to pay for, for example, gasoline purchase costs, EV (Electric Vehicle) charging costs, highway tolls, rental car costs, and the like. Also, after acquiring the sensor information, the server 2001 may delete information for identifying the client device 2002 that transmitted the sensor information. For example, this information is information such as the network address of the client device 2002. This allows the sensor information to be anonymized, so that the user of the client device 2002 can transmit the sensor information from the client device 2002 to the server 2001 with peace of mind. Also, the server 2001 may be composed of multiple servers. For example, by sharing the sensor information among multiple servers, even if one server breaks down, the other servers can communicate with the client device 2002. This makes it possible to avoid a service interruption due to a server failure.

[0192] Furthermore, the designated location specified in the sensor information transmission request indicates the location where an accident occurred, etc., and may differ from the location of the client device 2002 at the designated time specified in the sensor information transmission request. Therefore, the server 2001 can request information acquisition from the client device 2002 that exists within the designated location by specifying a range, such as within XX m of the surrounding area. Similarly, for the designated time, the server 2001 may specify a range, such as within N seconds before and after a certain time. This allows the server 2001 to acquire sensor information from the client device 2002 that exists in "location: within XX m from absolute position S at time: tN to t+N." When transmitting three-dimensional data such as LiDAR data, the client device 2002 may transmit data generated immediately after time t.

[0193] Furthermore, the server 2001 may separately specify, as the specified location, information indicating the location of the client device 2002 from which sensor information is to be acquired and a location from which sensor information is desired. For example, the server 2001 specifies that sensor information including at least a range of YYm from the absolute position S is to be acquired from the client device 2002 that exists within XXm from the absolute position S. When selecting three-dimensional data to be transmitted, the client device 2002 selects one or more randomly accessible units of three-dimensional data so as to include at least the sensor information in the specified range. Furthermore, when transmitting a visible light image, the client device 2002 may transmit a plurality of temporally consecutive image data including at least a frame immediately before or after time t.

[0194] When the client device 2002 can use multiple physical networks, such as 5G or WiFi, or multiple modes in 5G, for transmitting sensor information, the client device 2002 may select a network to be used according to the priority order notified by the server 2001. Alternatively, the client device 2002 itself may select a network that can secure an appropriate bandwidth based on the size of the transmission data. Alternatively, the client device 2002 may select a network to be used based on the cost of data transmission, etc. Also, the transmission request from the server 2001 may include information indicating a transmission deadline, such as "transmission will be performed if the client device 2002 can start transmission by time T." The server 2001 may issue a transmission request again if sufficient sensor information cannot be acquired within the deadline.

[0195] The sensor information may include header information indicating characteristics of the sensor data along with compressed or uncompressed sensor data. The client device 2002 may transmit the header information to the server 2001 via a physical network or communication protocol different from that of the sensor data. For example, the client device 2002 transmits the header information to the server 2001 prior to transmitting the sensor data. The server 2001 determines whether to acquire the sensor data of the client device 2002 based on an analysis result of the header information. For example, the header information may include information indicating the point cloud acquisition density, elevation angle, or frame rate of the LiDAR, or the resolution, signal-to-noise ratio, or frame rate of the visible light image. This allows the server 2001 to acquire the sensor information from the client device 2002 having the sensor data of the determined quality.

[0196] As described above, the client device 2002 is mounted on a moving object, acquires sensor information indicating the surrounding conditions of the moving object obtained by a sensor mounted on the moving object, and stores the sensor information in the storage unit 2012. The client device 2002 determines whether the moving object is in an environment where the sensor information can be transmitted to the server 2001, and transmits the sensor information to the server 2001 when it is determined that the moving object is in an environment where the sensor information can be transmitted to the server.

[0197] Furthermore, the client device 2002 further creates three-dimensional data of the surroundings of the moving object from the sensor information, and estimates the self-position of the moving object using the created three-dimensional data.

[0198] Moreover, the client device 2002 further transmits a request to the server 2001 to transmit a three-dimensional map, and receives the three-dimensional map from the server 2001. The client device 2002 estimates its own location using the three-dimensional data and the three-dimensional map.

[0199] The above-described processing by client device 2002 may be realized as an information transmission method in client device 2002.

[0200] The client device 2002 may also include a processor and a memory, and the processor may use the memory to perform the above-mentioned processes.

[0201] Next, a sensor information collection system according to the present embodiment will be described. FIG. 21 is a diagram showing the configuration of the sensor information collection system according to the present embodiment. As shown in FIG. 21, the sensor information collection system according to the present embodiment includes a terminal 2021A, a terminal 2021B, a communication device 2022A, a communication device 2022B, a network 2023, a data collection server 2024, a map server 2025, and a client device 2026. When the terminal 2021A and the terminal 2021B are not particularly distinguished from each other, they are also referred to as the terminal 2021. When the communication device 2022A and the communication device 2022B are not particularly distinguished from each other, they are also referred to as the communication device 2022.

[0202] The data collection server 2024 collects data such as sensor data obtained by a sensor included in the terminal 2021 as position-related data associated with a position in a three-dimensional space.

[0203] The sensor data is, for example, data acquired by using a sensor provided in the terminal 2021, such as the state of the surroundings of the terminal 2021 or the state of the inside of the terminal 2021. The terminal 2021 transmits to the data collection server 2024 sensor data collected from one or more sensor devices that are located in positions that can directly communicate with the terminal 2021 or can communicate with the terminal 2021 via one or more relay devices using the same communication method.

[0204] The data included in the location-related data may include, for example, information indicating the operating state of the terminal itself or a device included in the terminal, an operating log, a service usage status, etc. Furthermore, the data included in the location-related data may include information associating an identifier of the terminal 2021 with the location or a movement route of the terminal 2021, etc.

[0205] The information indicating a position included in the position-related data is associated with information indicating a position in three-dimensional data such as three-dimensional map data, etc. The information indicating a position will be described in detail later.

[0206] The position related data may include at least one of the above-mentioned time information and information indicating an attribute of the data included in the position related data or the type of the sensor that generated the data (e.g., model number, etc.) in addition to the position information that indicates a position. The position information and the time information may be stored in a header area of ​​the position related data or a header area of ​​a frame that stores the position related data. In addition, the position information and the time information may be transmitted and / or stored separately from the position related data as metadata associated with the position related data.

[0207] The map server 2025 is connected to the network 2023, for example, and transmits three-dimensional data such as three-dimensional map data in response to a request from another device such as the terminal 2021. As described in the above-mentioned embodiments, the map server 2025 may also have a function of updating the three-dimensional data using sensor information transmitted from the terminal 2021.

[0208] The data collection server 2024 is connected to the network 2023, for example, and collects location-related data from other devices such as the terminal 2021, and stores the collected location-related data in a storage device inside the server or in another server. The data collection server 2024 also transmits the collected location-related data or metadata of three-dimensional map data generated based on the location-related data to the terminal 2021 in response to a request from the terminal 2021.

[0209] The network 2023 is a communication network such as the Internet. The terminal 2021 is connected to the network 2023 via a communication device 2022. The communication device 2022 communicates with the terminal 2021 by switching one communication method or a plurality of communication methods. The communication device 2022 is, for example, (1) a base station such as LTE (Long Term Evolution), (2) an access point (AP) such as WiFi or millimeter wave communication, (3) a gateway of an LPWA (Low Power Wide Area) Network such as SIGFOX, LoRaWAN, or Wi-SUN, or (4) a communication satellite that communicates by using a satellite communication method such as DVB-S2.

[0210] In addition, the base station may communicate with the terminal 2021 using a method classified as LPWA, such as NB-IoT (Narrow Band-IoT) or LTE-M, or may communicate with the terminal 2021 by switching between these methods.

[0211] Here, the terminal 2021 has a function of communicating with a communication device 2022 using two types of communication methods, and communicates with the map server 2025 or the data collection server 2024 using one of these communication methods, or by switching between these multiple communication methods and the communication device 2022 that is the direct communication partner, but the configuration of the sensor information collection system and the terminal 2021 is not limited to this. For example, the terminal 2021 may not have a communication function for multiple communication methods, but may have a function of communicating using any one of the communication methods. The terminal 2021 may also support three or more communication methods. Each terminal 2021 may support a different communication method.

[0212] The terminal 2021 has, for example, the configuration of the client device 902 shown in Fig. 11. The terminal 2021 performs position estimation such as its own position using the received three-dimensional data. The terminal 2021 also generates position-related data by associating sensor data acquired from a sensor with position information obtained by position estimation processing.

[0213] The location information added to the location-related data indicates, for example, a location in a coordinate system used in three-dimensional data. For example, the location information is a coordinate value expressed by latitude and longitude values. At this time, the terminal 2021 may include, in the location information, information indicating the coordinate system that is the reference for the coordinate value together with the coordinate value and the three-dimensional data used for the location estimation. The coordinate value may also include altitude information.

[0214] The position information may also be associated with a data unit or a space unit that can be used to encode the above-mentioned three-dimensional data. The unit may be, for example, WLD, GOS, SPC, VLM, or VXL. In this case, the position information is expressed by an identifier for identifying a data unit such as an SPC corresponding to the position-related data. Note that the position information may include, in addition to an identifier for identifying a data unit such as an SPC, information indicating three-dimensional data obtained by encoding a three-dimensional space including the data unit such as the SPC, or information indicating a detailed position within the SPC. The information indicating the three-dimensional data may be, for example, a file name of the three-dimensional data.

[0215] In this way, the system generates location-related data associated with location information based on location estimation using three-dimensional data, and thereby can add location information to the sensor information with higher accuracy than when location information based on the self-location of the client device (terminal 2021) acquired using GPS is added to the sensor information. As a result, even when the location-related data is used by another device for another service, it may be possible to more accurately identify the location corresponding to the location-related data in real space by performing location estimation based on the same three-dimensional data.

[0216] In the present embodiment, the data transmitted from the terminal 2021 is location-related data, but the data transmitted from the terminal 2021 may be data that is not associated with location information. That is, the three-dimensional data or sensor data described in the other embodiments may be transmitted and received via the network 2023 described in the present embodiment.

[0217] Next, different examples of location information indicating a position in a three-dimensional or two-dimensional real space or map space will be described. The location information added to the location-related data may be information indicating a relative position with respect to a feature point in the three-dimensional data. Here, the feature point serving as the reference of the location information is, for example, a feature point encoded as SWLD and notified to the terminal 2021 as three-dimensional data.

[0218] The information indicating the relative position with respect to the feature point may be, for example, represented by a vector from the feature point to the point indicated by the position information, and may be information indicating the direction and distance from the feature point to the point indicated by the position information. Alternatively, the information indicating the relative position with respect to the feature point may be information indicating the amount of displacement on each of the X-axis, Y-axis, and Z-axis from the feature point to the point indicated by the position information. Furthermore, the information indicating the relative position with respect to the feature point may be information indicating the distance from each of three or more feature points to the point indicated by the position information. Note that the relative position may not be the relative position of the point indicated by the position information expressed with each feature point as a reference, but may be the relative position of each feature point expressed with the point indicated by the position information as a reference. An example of the position information based on the relative position with respect to the feature point includes information for identifying the reference feature point and information indicating the relative position of the point indicated by the position information with respect to the feature point. Furthermore, when the information indicating the relative position with respect to the feature point is provided separately from the three-dimensional data, the information indicating the relative position with respect to the feature point may include the coordinate axis used to derive the relative position, information indicating the type of three-dimensional data, and / or information indicating the size per unit amount (such as scale) of the value of the information indicating the relative position.

[0219] Furthermore, the position information may include information indicating a relative position with respect to each of a plurality of feature points. When the position information is expressed as a relative position with respect to a plurality of feature points, the terminal 2021 that attempts to identify a position indicated by the position information in the real space may calculate a candidate point of the position indicated by the position information from the position of the feature point estimated from the sensor data for each feature point, and may determine that a point obtained by averaging the calculated candidate points is the point indicated by the position information. According to this configuration, the influence of an error when estimating the position of the feature point from the sensor data can be reduced, so that the estimation accuracy of the point indicated by the position information in the real space can be improved. Furthermore, when the position information includes information indicating a relative position with respect to a plurality of feature points, even if there is a feature point that cannot be detected due to restrictions such as the type or performance of the sensor provided in the terminal 2021, it is possible to estimate the value of the point indicated by the position information as long as any one of the plurality of feature points can be detected.

[0220] A point that can be identified from sensor data can be used as a feature point. An example of a point that can be identified from sensor data is a point or a point within an area that satisfies a predetermined condition for detecting a feature point, such as the above-mentioned three-dimensional feature amount or the feature amount of visible light data being equal to or greater than a threshold value.

[0221] Also, markers installed in the real space may be used as feature points. In this case, the markers may be detected and their positions specified from data acquired using a sensor such as LiDAR or a camera. For example, the markers are expressed by changes in color or brightness (reflectance), or three-dimensional shapes (such as unevenness). Also, coordinate values ​​indicating the position of the marker, or two-dimensional codes or barcodes generated from the identifiers of the markers may be used.

[0222] Also, a light source that transmits an optical signal may be used as a marker. When a light source of an optical signal is used as a marker, not only information for acquiring a position such as a coordinate value or an identifier, but also other data may be transmitted by the optical signal. For example, the optical signal may include information indicating the content of a service corresponding to the position of the marker, an address such as a URL for acquiring the content, or an identifier of a wireless communication device for receiving the service, and a wireless communication method for connecting to the wireless communication device. By using an optical communication device (light source) as a marker, it becomes easier to transmit data other than information indicating a position, and it becomes possible to dynamically switch the data.

[0223] The terminal 2021 grasps the correspondence between feature points between different data by using, for example, an identifier commonly used between the data, or information or a table indicating the correspondence between feature points between the data. Furthermore, when there is no information indicating the correspondence between feature points, the terminal 2021 may determine that the feature points that are closest when the coordinates of a feature point in one three-dimensional data are converted to a position in the three-dimensional data space of the other are corresponding feature points.

[0224] When using the position information based on the relative positions described above, even between terminals 2021 or services using different three-dimensional data, it is possible to identify or estimate the position indicated by the position information based on common feature points included in or associated with each three-dimensional data. As a result, it is possible to identify or estimate the same position with higher accuracy between terminals 2021 or services using different three-dimensional data.

[0225] Furthermore, even when using map data or three-dimensional data expressed using different coordinate systems, the effect of errors associated with coordinate system conversion can be reduced, making it possible to link services based on more accurate location information.

[0226] Below, an example of a function provided by the data collection server 2024 will be described. The data collection server 2024 may transfer the received location-related data to another data server. When there are multiple data servers, the data collection server 2024 determines to which data server the received location-related data should be transferred, and transfers the location-related data to the data server determined as the transfer destination.

[0227] The data collection server 2024 determines the destination of transfer, for example, based on a determination rule for the destination server that is set in advance in the data collection server 2024. The determination rule for the destination server is set, for example, in a transfer destination table that associates an identifier associated with each terminal 2021 with a destination data server.

[0228] The terminal 2021 adds an identifier associated with the terminal 2021 to the location related data to be transmitted, and transmits the data to the data collection server 2024. The data collection server 2024 identifies a destination data server corresponding to the identifier added to the location related data based on a destination server determination rule using a destination table or the like, and transmits the location related data to the identified data server. The destination server determination rule may be specified by a determination condition using the time or place at which the location related data was acquired. Here, the identifier associated with the above-mentioned transmission source terminal 2021 is, for example, an identifier unique to each terminal 2021, or an identifier indicating a group to which the terminal 2021 belongs.

[0229] In addition, the destination table does not have to directly associate the identifier associated with the terminal of the source with the data server of the destination. For example, the data collection server 2024 holds a management table storing tag information given to each unique identifier of the terminal 2021, and a destination table in which the tag information is associated with the data server of the destination. The data collection server 2024 may determine the data server of the destination based on the tag information using the management table and the destination table. Here, the tag information is, for example, control information for management or control information for providing services given to the type, model number, owner, group to which the terminal 2021 corresponding to the identifier, or other identifiers. In addition, the destination table may use an identifier unique to each sensor instead of the identifier associated with the terminal 2021 of the source. In addition, the determination rule of the destination server may be set from the client device 2026.

[0230] The data collection server 2024 may determine a plurality of data servers as transfer destinations and transfer the received location-related data to the plurality of data servers. With this configuration, for example, when the location-related data is automatically backed up, or when the location-related data needs to be sent to a data server for providing each service in order to use the location-related data in common for different services, the data can be transferred as intended by changing the settings for the data collection server 2024. As a result, compared to the case where the destination of the location-related data is set in each individual terminal 2021, the number of steps required for building and modifying the system can be reduced.

[0231] In response to a transfer request signal received from a data server, the data collection server 2024 may register the data server specified in the transfer request signal as a new transfer destination, and transfer subsequently received location-related data to that data server.

[0232] The data collection server 2024 may store the location-related data received from the terminal 2021 in a recording device, and in response to a transmission request signal received from the terminal 2021 or the data server, may transmit the location-related data specified in the transmission request signal to the requesting terminal 2021 or data server.

[0233] The data collection server 2024 may determine whether or not to provide the location-related data to the requesting data server or terminal 2021, and if it is determined that the location-related data can be provided, may transfer or transmit the location-related data to the requesting data server or terminal 2021.

[0234] When a request for current location-related data is received from client device 2026, even if it is not the timing for terminal 2021 to transmit location-related data, data collection server 2024 may request terminal 2021 to transmit location-related data, and terminal 2021 may transmit the location-related data in response to the transmission request.

[0235] In the above explanation, it is described that the terminal 2021 transmits location information data to the data collection server 2024. However, the data collection server 2024 may be provided with a function necessary for collecting location-related data from the terminal 2021, such as a function for managing the terminal 2021, or a function used when collecting location-related data from the terminal 2021.

[0236] The data collection server 2024 may have a function of transmitting a data request signal to the terminal 2021 to request the transmission of location information data, and collecting location-related data.

[0237] Management information such as an address for communicating with the terminal 2021 from which data is to be collected or an identifier unique to the terminal 2021 is registered in advance in the data collection server 2024. The data collection server 2024 collects location-related data from the terminal 2021 based on the registered management information. The management information may include information such as the type of sensor included in the terminal 2021, the number of sensors included in the terminal 2021, and a communication method supported by the terminal 2021.

[0238] The data collection server 2024 may collect information from the terminal 2021, such as the operating status or current location of the terminal 2021.

[0239] The management information may be registered from the client device 2026, or the terminal 2021 may transmit a registration request to the data collection server 2024 to start the registration process. The data collection server 2024 may have a function of controlling communication with the terminal 2021.

[0240] The communication between the data collection server 2024 and the terminal 2021 may be a dedicated line provided by a service provider such as an MNO (Mobile Network Operator) or an MVNO (Mobile Virtual Network Operator), or a virtual dedicated line configured by a VPN (Virtual Private Network). With this configuration, the communication between the terminal 2021 and the data collection server 2024 can be performed safely.

[0241] The data collection server 2024 may have a function of authenticating the terminal 2021 or a function of encrypting data transmitted and received between the terminal 2021. Here, the authentication process of the terminal 2021 or the encryption process of the data is performed using an identifier unique to the terminal 2021 or an identifier unique to a terminal group including a plurality of terminals 2021, which is shared in advance between the data collection server 2024 and the terminal 2021. This identifier is, for example, an International Mobile Subscriber Identity (IMSI), which is a unique number stored in a Subscriber Identity Module (SIM) card. The identifier used in the authentication process and the identifier used in the data encryption process may be the same or different.

[0242] The authentication or data encryption process between the data collection server 2024 and the terminal 2021 can be provided if both the data collection server 2024 and the terminal 2021 have a function for performing the process, and does not depend on the communication method used by the relay communication device 2022. Therefore, a common authentication or encryption process can be used without considering the communication method used by the terminal 2021, improving the convenience of the user's system construction. However, not depending on the communication method used by the relay communication device 2022 means that it is not necessary to change depending on the communication method. In other words, for the purpose of improving transmission efficiency or ensuring safety, the authentication or data encryption process between the data collection server 2024 and the terminal 2021 may be switched depending on the communication method used by the relay device.

[0243] The data collection server 2024 may provide the client device 2026 with a UI for managing data collection rules such as the type of location-related data to be collected from the terminal 2021 and the schedule for data collection. This allows the user to specify the terminal 2021 from which data is to be collected using the client device 2026, as well as the time and frequency of data collection. The data collection server 2024 may also specify an area on a map from which data is to be collected, and collect location-related data from the terminal 2021 included in that area.

[0244] When managing data collection rules for each terminal 2021, the client device 2026 presents, for example, a list of the terminals 2021 or sensors to be managed on a screen. The user sets the necessity of data collection, a collection schedule, etc. for each item in the list.

[0245] When specifying an area on a map from which data is to be collected, the client device 2026 presents, for example, a two-dimensional or three-dimensional map of the area to be managed on a screen. The user selects an area from which data is to be collected on the displayed map. The area selected on the map may be a circular or rectangular area centered on a point specified on the map, or may be a circular or rectangular area that can be specified by a dragging operation. The client device 2026 may also select an area in a preset unit such as a city, an area within a city, a block, or a major road. Instead of specifying an area using a map, an area may be set by inputting numerical values ​​of latitude and longitude, or an area may be selected from a list of candidate areas derived based on input text information. The text information is, for example, the name of a region, a city, or a landmark.

[0246] Furthermore, the user may specify one or more terminals 2021 and set conditions such as within a range of 100 meters around the terminal 2021, so that data collection may be performed while dynamically changing the specified area.

[0247] Also, when the client device 2026 is equipped with a sensor such as a camera, an area on a map may be specified based on the position of the client device 2026 in real space obtained from the sensor data. For example, the client device 2026 may estimate its own position using the sensor data, and specify an area within a predetermined distance from a point on the map corresponding to the estimated position or a distance specified by the user as an area for collecting data. Also, the client device 2026 may specify a sensing area of ​​the sensor, i.e., an area corresponding to the acquired sensor data, as an area for collecting data. Alternatively, the client device 2026 may specify an area based on a position corresponding to the sensor data specified by the user as an area for collecting data. The area on the map or the position corresponding to the sensor data may be estimated by the client device 2026 or the data collection server 2024.

[0248] When designating an area on a map, the data collection server 2024 may identify the terminal 2021 in the designated area by collecting current position information of each terminal 2021, and may request the identified terminal 2021 to transmit position-related data. Alternatively, instead of the data collection server 2024 identifying the terminal 2021 in the area, the data collection server 2024 may transmit information indicating the designated area to the terminal 2021, and the terminal 2021 may determine whether or not it is within the designated area, and transmit the position-related data if it is determined that it is within the designated area.

[0249] The data collection server 2024 transmits data such as a list or a map for providing the above-mentioned UI (User Interface) in an application executed by the client device 2026 to the client device 2026. The data collection server 2024 may transmit not only data such as a list or a map but also an application program to the client device 2026. The above-mentioned UI may be provided as content created in HTML or the like that can be displayed in a browser. Note that some data such as map data may be provided from a server other than the data collection server 2024, such as a map server 2025.

[0250] When the user presses a setting button or performs other input to notify completion of input, the client device 2026 transmits the input information as setting information to the data collection server 2024. The data collection server 2024 transmits a signal to each terminal 2021, based on the setting information received from the client device 2026, requesting location-related data or notifying a collection rule for location-related data, and collects location-related data.

[0251] Next, an example of controlling the operation of the terminal 2021 based on additional information added to three-dimensional or two-dimensional map data will be described.

[0252] In this configuration, object information indicating the position of a power supply unit such as a wireless power supply antenna or a power supply coil buried in a road or parking lot is included in or associated with the three-dimensional data and provided to a terminal 2021, such as a car or a drone.

[0253] The vehicle or drone that has acquired the object information for charging moves the vehicle's own position by autonomous driving so that the position of the charging part, such as a charging antenna or charging coil, equipped on the vehicle faces the area indicated by the object information, and starts charging. In the case of a vehicle or drone that does not have an autonomous driving function, the direction to move or the operation to be performed is presented to the driver or operator using images or audio displayed on the screen. Then, when it is determined that the position of the charging part calculated based on the estimated self-position is within the area indicated by the object information or within a predetermined distance from the area, the image or audio presented is switched to one that stops driving or operation, and charging starts.

[0254] In addition, the object information may not be information indicating the position of the power supply unit, but may be information indicating an area in which a charging efficiency equal to or higher than a predetermined threshold can be obtained when the charging unit is placed in the area. The position of the object information may be represented by a point at the center of the area indicated by the object information, or may be represented by an area or line in a two-dimensional plane, or an area, line, or plane in a three-dimensional space.

[0255] According to this configuration, the position of the power supply antenna, which cannot be grasped by the sensing data of the LiDER or the image captured by the camera, can be grasped, so that the position of the wireless charging antenna equipped in the terminal 2021 such as a car and the wireless power supply antenna buried in the road or the like can be aligned with high accuracy. As a result, the charging speed during wireless charging can be shortened and the charging efficiency can be improved.

[0256] The object information may be an object other than the power supply antenna. For example, the three-dimensional data includes the position of an AP for millimeter wave wireless communication as object information. This allows the terminal 2021 to know the position of the AP in advance, and therefore the terminal 2021 can start communication by directing the beam direction in the direction of the object information. As a result, it is possible to improve communication quality, such as improving the transmission speed, shortening the time until communication starts, and extending the period during which communication is possible.

[0257] The object information may include information indicating a type of object corresponding to the object information. The object information may also include information indicating a process to be performed by the terminal 2021 when the terminal 2021 is included in an area in real space corresponding to the position of the object information in the three-dimensional data, or within a predetermined distance from the area.

[0258] The object information may be provided from a server different from the server providing the three-dimensional data. When the object information is provided separately from the three-dimensional data, object groups in which object information used in the same service is stored may be provided as separate data according to the type of target service or target device.

[0259] The three-dimensional data used in combination with the object information may be point cloud data of the WLD or feature point data of the SWLD.

[0260] (Embodiment 6) The octree representation and the voxel scan order will be described below. The volume is converted (octreeized) into an octree structure and then encoded. The octree structure is composed of nodes and leaves. Each node has eight nodes or leaves, and each leaf has voxel (VXL) information. FIG. 22 is a diagram showing an example of the structure of a volume including a plurality of voxels. FIG. 23 is a diagram showing an example of the volume shown in FIG. 22 converted into an octree structure. Here, among the leaves shown in FIG. 23, leaves 1, 2, and 3 respectively represent voxels VXL1, VXL2, and VXL3 shown in FIG. 22, and represent a VXL including a point cloud (hereinafter, effective VXL).

[0261] An octree is represented by a binary sequence of, for example, 0 and 1. For example, if a node or a valid VXL is set to value 1 and the rest to value 0, then the binary sequence shown in FIG. 23 is assigned to each node and leaf. Then, this binary sequence is scanned according to the scan order of breadth-first or depth-first. For example, when breadth-first scanning is performed, the binary sequence shown in FIG. 24A is obtained. When depth-first scanning is performed, the binary sequence shown in FIG. 24B is obtained. The binary sequence obtained by this scan is coded by entropy coding to reduce the amount of information.

[0262] Next, the depth information in the octree representation will be explained. The depth in the octree representation is used to control the granularity of the point cloud information contained in the volume to be retained. If the depth is set large, the point cloud information can be reproduced to a finer level, but the amount of data required to represent the nodes and leaves increases. Conversely, if the depth is set small, the amount of data decreases, but since multiple point cloud information with different positions and colors is considered to be in the same position and with the same color, the information contained in the original point cloud information will be lost.

[0263] For example, Fig. 25 is a diagram showing an example in which the octree with depth=2 shown in Fig. 23 is expressed as an octree with depth=1. The octree shown in Fig. 25 has a smaller amount of data than the octree shown in Fig. 23. In other words, the octree shown in Fig. 25 has a smaller number of bits after binary string conversion than the octree shown in Fig. 25. Here, leaf 1 and leaf 2 shown in Fig. 23 are expressed as leaf 1 shown in Fig. 24. In other words, the information that leaf 1 and leaf 2 shown in Fig. 23 were in different positions is lost.

[0264] Fig. 26 is a diagram showing a volume corresponding to the octree shown in Fig. 25. VXL1 and VXL2 shown in Fig. 22 correspond to VXL12 shown in Fig. 26. In this case, the three-dimensional data encoding device generates color information of VXL12 shown in Fig. 26 from color information of VXL1 and VXL2 shown in Fig. 22. For example, the three-dimensional data encoding device calculates an average value, a median value, a weighted average value, or the like of the color information of VXL1 and VXL2 as the color information of VXL12. In this way, the three-dimensional data encoding device may control the reduction of the amount of data by changing the depth of the octree.

[0265] The three-dimensional data encoding device may set the depth information of the octree in any unit of world, space, or volume. In addition, in this case, the three-dimensional data encoding device may add the depth information to the header information of the world, the header information of the space, or the header information of the volume. Also, the same value may be used as the depth information for all worlds, spaces, and volumes of different times. In this case, the three-dimensional data encoding device may add the depth information to the header information that manages the worlds of all times.

[0266] (Embodiment 7) Hereinafter, a method using RAHT (Region Adaptive Hierarchical Transform) will be described as another method for encoding attribute information of three-dimensional points. Fig. 27 is a diagram for explaining encoding of attribute information using RAHT.

[0267] First, the three-dimensional data encoding device generates a Morton code based on the position information of the three-dimensional point, and sorts the attribute information of the three-dimensional point in the order of the Morton code. For example, the three-dimensional data encoding device may sort in ascending order of the Morton code. Note that the sorting order is not limited to the Morton code order, and other orders may be used.

[0268] Next, the three-dimensional data encoding device applies a Haar transform to the attribute information of two adjacent three-dimensional points in the Morton code order to generate high-frequency components and low-frequency components of layer L. For example, the three-dimensional data encoding device may use a 2×2 matrix Haar transform. The generated high-frequency components are included in the encoding coefficients as high-frequency components of layer L, and the generated low-frequency components are used as input values ​​for layer L+1, which is a layer above layer L.

[0269] After generating high-frequency components for layer L using the attribute information for layer L, the three-dimensional data encoding device continues to process layer L+1. In processing layer L+1, the three-dimensional data encoding device generates high-frequency and low-frequency components for layer L+1 by applying a Haar transform to two low-frequency components obtained by the Haar transform of the attribute information for layer L. The generated high-frequency components are included in the encoding coefficients as the high-frequency components for layer L+1, and the generated low-frequency components are used as input values ​​for layer L+2, which is higher than layer L+1.

[0270] The three-dimensional data encoding device repeats such hierarchical processing, and when only one low-frequency component is input to the hierarchy, it determines that the highest hierarchy Lmax has been reached. The three-dimensional data encoding device includes the low-frequency component of hierarchy Lmax-1 input to hierarchy Lmax in the encoding coefficients. The values ​​of the low-frequency or high-frequency components included in the encoding coefficients are then quantized and encoded using entropy encoding or the like.

[0271] In addition, when only one three-dimensional point exists as two adjacent three-dimensional points when applying the Haar transform, the three-dimensional data encoding device may use the value of the attribute information of the one existing three-dimensional point as the input value of the higher hierarchy.

[0272] In this way, the three-dimensional data encoding device applies a hierarchical Haar transform to the input attribute information to generate high-frequency and low-frequency components of the attribute information, and performs encoding by applying quantization, etc., which will be described later. This makes it possible to improve the encoding efficiency.

[0273] When the attribute information is N-dimensional, the three-dimensional data encoding device may apply a Haar transform to each dimension independently and calculate each encoding coefficient. For example, when the attribute information is color information (RGB or YUV, etc.), the three-dimensional data encoding device applies a Haar transform to each component and calculates each encoding coefficient.

[0274] The three-dimensional data encoding device may apply the Haar transform in the order of layer L, L+1, ..., layer Lmax. The closer to layer Lmax, the more encoding coefficients are generated that contain low-frequency components of the input attribute information.

[0275] w0 and w1 shown in Fig. 27 are weights assigned to each three-dimensional point. For example, the three-dimensional data encoding device may calculate the weight based on distance information between two adjacent three-dimensional points to which the Haar transform is applied. For example, the three-dimensional data encoding device may increase the weight as the distance becomes closer, thereby improving the encoding efficiency. Note that the three-dimensional data encoding device may calculate this weight by a different method, or may not use a weight.

[0276] In the example shown in Fig. 27, the input attribute information is a0, a1, a2, a3, a4, and a5. Among the coding coefficients after Haar transform, Ta1, Ta5, Tb1, Tb3, Tc1, and d0 are coded. The other coding coefficients (b0, b2, c0, etc.) are intermediate values ​​and are not coded.

[0277] Specifically, in the example shown in Fig. 27, a0 and a1 are subjected to Haar transform to generate a high-frequency component Ta1 and a low-frequency component b0. Here, when the weights w0 and w1 are equal, the low-frequency component b0 is the average value of a0 and a1, and the high-frequency component Ta1 is the difference between a0 and a1.

[0278] Since there is no attribute information paired with a2, a2 ​​is used as b1 as is. Similarly, since there is no attribute information paired with a3, a3 is used as b2 as is. Furthermore, a4 and a5 are subjected to Haar transform to generate a high frequency component Ta5 and a low frequency component b3.

[0279] In layer L+1, a high frequency component Tb1 and a low frequency component c0 are generated by performing a Haar transform on b0 and b1. Similarly, a high frequency component Tb3 and a low frequency component c1 are generated by performing a Haar transform on b2 and b3.

[0280] In the layer Lmax-1, a Haar transform is performed on c0 and c1 to generate a high-frequency component Tc1 and a low-frequency component d0.

[0281] The three-dimensional data encoding device may quantize the encoding coefficients after applying the Haar transform before encoding. For example, the three-dimensional data encoding device performs quantization by dividing the encoding coefficients by a quantization scale (also called a quantization step (QS (Quantization Step))). In this case, the smaller the quantization scale, the smaller the error (quantization error) that may occur due to quantization. Conversely, the larger the quantization scale, the larger the quantization error.

[0282] The three-dimensional data encoding device may change the value of the quantization scale for each layer. FIG. 28 is a diagram showing an example of setting the quantization scale for each layer. For example, the three-dimensional data encoding device sets a smaller quantization scale for higher layers and a larger quantization scale for lower layers. The coding coefficients of three-dimensional points belonging to higher layers contain more low-frequency components than lower layers, and are therefore likely to be important components in terms of human visual characteristics, etc. Therefore, by reducing the quantization scale for higher layers and suppressing quantization errors that may occur in higher layers, visual deterioration can be suppressed and coding efficiency can be improved.

[0283] The three-dimensional data encoding device may add a quantization scale for each layer to a header, etc. This allows the three-dimensional data decoding device to correctly decode the quantization scale and appropriately decode the bit stream.

[0284] Furthermore, the three-dimensional data encoding device may adaptively switch the value of the quantization scale according to the importance of the target three-dimensional point to be encoded. For example, the three-dimensional data encoding device may use a small quantization scale for a three-dimensional point with high importance, and a large quantization scale for a three-dimensional point with low importance. For example, the three-dimensional data encoding device may calculate the importance from the weighting during Haar transform, etc. For example, the three-dimensional data encoding device may calculate the quantization scale using the sum of w0 and w1. In this way, by reducing the quantization scale of a three-dimensional point with high importance, the quantization error is reduced, and the encoding efficiency can be improved.

[0285] In addition, the value of QS may be made smaller in higher layers, which increases the value of QW in higher layers and improves prediction efficiency by suppressing the quantization error of the 3D points.

[0286] Here, the coding coefficient Ta1q after quantization of the coding coefficient Ta1 of the attribute information a1 is expressed as Ta1 / QS_L. Note that QS may be the same value in all layers or in some layers.

[0287] QW (Quantization Weight) is a value that indicates the importance of the 3D point to be coded. For example, the sum of the above-mentioned w0 and w1 may be used as QW. This makes the value of QW larger in the higher layers, and the prediction efficiency can be improved by suppressing the quantization error of the 3D point.

[0288] For example, the three-dimensional data encoding device may first initialize the QW values ​​of all three-dimensional points to 1, and update the QW of each three-dimensional point using the values ​​of w0 and w1 at the time of Haar transformation. Alternatively, the three-dimensional data encoding device may change the initial value according to the layer, without initializing the QW of all three-dimensional points to the value 1. For example, the quantization scale of the upper layer is reduced by setting the initial value of QW larger for the higher layer. This makes it possible to suppress prediction errors in the upper layer, thereby improving the prediction accuracy of the lower layer and improving the encoding efficiency. It should be noted that the three-dimensional data encoding device does not necessarily have to use the QW.

[0289] When QW is used, the quantized value Ta1q of Ta1 is calculated by (Equation K1) and (Equation K2).

[0290]

number

[0291] In addition, the three-dimensional data encoding device scans and encodes the quantized encoding coefficients (unsigned integer values) in a certain order. For example, the three-dimensional data encoding device encodes a plurality of three-dimensional points from the three-dimensional points included in the upper layer toward the lower layer.

[0292] For example, in the case of the example shown in Fig. 27, the three-dimensional data encoding device encodes a plurality of three-dimensional points in the order of d0q, Tc1q, Tb1q, Tb3q, Ta1q, and Ta5q included in the upper layer Lmax. Here, the lower the layer L, the more likely the encoding coefficients after quantization are to become 0. The following factors can be cited as reasons for this.

[0293] The coding coefficients of the lower layer L show higher frequency components than those of the upper layer, and therefore tend to be 0 depending on the target 3D point. Also, due to the switching of the quantization scale according to the importance level and the like described above, the quantization scale becomes larger in the lower layer, and the coding coefficients after quantization tend to be 0.

[0294] In this way, the lower the layer, the more likely it is that the coding coefficient after quantization will be 0, and the more likely it is that consecutive 0s will occur in the first code string. Fig. 29 is a diagram showing examples of the first code string and the second code string.

[0295] The three-dimensional data encoding device counts the number of times the value 0 occurs in the first code string, and encodes the number of times the value 0 occurs in succession, instead of consecutive values ​​of 0. In other words, the three-dimensional data encoding device generates a second code string by replacing the encoding coefficients of consecutive values ​​of 0 in the first code string with the number of consecutive 0s (ZeroCnt). As a result, when the encoding coefficients after quantization have consecutive values ​​of 0, encoding the number of consecutive 0s can improve encoding efficiency rather than encoding a large number of 0s.

[0296] Also, the three-dimensional data encoding device may entropy-encode the value of ZeroCnt. For example, the three-dimensional data encoding device binarizes the value of ZeroCnt with a truncated unary code with a total number T of encoded three-dimensional points, and arithmetically encodes each bit after binarization. FIG. 30 is a diagram showing an example of a truncated unary code when the total number of encoded three-dimensional points is T. In this case, the three-dimensional data encoding device may improve the encoding efficiency by using a different encoding table for each bit. For example, the three-dimensional data encoding device uses encoding table 1 for the first bit, encoding table 2 for the second bit, and encoding table 3 for the bits thereafter. In this way, the three-dimensional data encoding device can improve the encoding efficiency by switching the encoding table for each bit.

[0297] Furthermore, the three-dimensional data encoding device may perform arithmetic coding after binarizing ZeroCnt using Exponential-Golomb. This can improve efficiency compared to binarized arithmetic coding using truncated unary code when the value of ZeroCnt is likely to become large. The three-dimensional data encoding device may add a flag to the header for switching between using truncated unary code or exponential-Golomb. This allows the three-dimensional data encoding device to improve encoding efficiency by selecting an optimal binarization method. Furthermore, the three-dimensional data decoding device can switch binarization methods by referring to the flag included in the header, and correctly decode the bit stream.

[0298] The three-dimensional data decoding device may convert the decoded quantized coding coefficients from unsigned integer values ​​to signed integer values ​​in a manner opposite to that performed by the three-dimensional data encoding device. This allows the three-dimensional data decoding device to properly decode the generated bit stream without considering the occurrence of negative integers when the coding coefficients are entropy coded. Note that the three-dimensional data decoding device does not necessarily need to convert the coding coefficients from unsigned integer values ​​to signed integer values. For example, when decoding a bit stream including coding bits that have been separately entropy coded, the three-dimensional data decoding device may decode the code bits.

[0299] The three-dimensional data decoding device decodes the quantized coding coefficients converted into signed integer values ​​by inverse quantization and inverse Haar transform. The three-dimensional data decoding device also uses the decoded coding coefficients for prediction of the three-dimensional point to be decoded and thereafter. Specifically, the three-dimensional data decoding device calculates an inverse quantized value by multiplying the quantized coding coefficients by the decoded quantization scale. Next, the three-dimensional data decoding device obtains a decoded value by applying an inverse Haar transform, which will be described later, to the inverse quantized value.

[0300] For example, the three-dimensional data decoding device converts the decoded unsigned integer value into a signed integer value in the following manner: If the LSB (least significant bit) of the decoded unsigned integer value a2u is 1, then the signed integer value Ta1q is set to -((a2u+1)>>1). If the LSB of the decoded unsigned integer value a2u is not 1 (is 0), then the signed integer value Ta1q is set to (a2u>>1).

[0301] The inverse quantization value of Ta1 is expressed as Ta1q×QS_L, where Ta1q is the quantization value of Ta1, and QS_L is the quantization step of the layer L.

[0302] Also, the QS may be the same value for all layers or for some layers. Also, the three-dimensional data encoding device may add information indicating the QS to a header, etc. This allows the three-dimensional data decoding device to perform correct inverse quantization using the same QS as that used by the three-dimensional data encoding device.

[0303] Next, the inverse Haar transform will be described. Fig. 31 is a diagram for explaining the inverse Haar transform. The three-dimensional data decoding device decodes the attribute values ​​of the three-dimensional points by applying the inverse Haar transform to the coding coefficients after the inverse quantization.

[0304] First, the three-dimensional data decoding device generates a Morton code based on the position information of the three-dimensional points, and sorts the three-dimensional points in the order of the Morton codes. For example, the three-dimensional data decoding device may sort the three-dimensional points in the ascending order of the Morton codes. Note that the sorting order is not limited to the Morton code order, and other orders may be used.

[0305] Next, the three-dimensional data decoding device applies an inverse Haar transform to the coding coefficients including low-frequency components of layer L+1 and the coding coefficients including high-frequency components of layer L to restore attribute information of adjacent three-dimensional points in Morton code order in layer L. For example, the three-dimensional data decoding device may use an inverse Haar transform of a 2×2 matrix. The restored attribute information of layer L is used as an input value for the lower layer L−1.

[0306] The three-dimensional data decoding device repeats such hierarchical processing, and ends the processing when all the attribute information in the lowest layer is decoded. Note that, when applying the inverse Haar transform, if only one three-dimensional point exists as two adjacent three-dimensional points in layer L-1, the three-dimensional data decoding device may substitute the value of the coding component of layer L for the attribute value of the one existing three-dimensional point. In this way, the three-dimensional data decoding device can apply the Haar transform to all values ​​of the input attribute information, and correctly decode a bitstream with improved coding efficiency.

[0307] When the attribute information is N-dimensional, the three-dimensional data decoding device may apply an inverse Haar transform to each dimension independently to decode each encoding coefficient. For example, when the attribute information is color information (RGB or YUV, etc.), the three-dimensional data decoding device applies an inverse Haar transform to the encoding coefficient for each component to decode each attribute value.

[0308] The three-dimensional data decoding device may apply the inverse Haar transform in the order of layers Lmax, L+1, ..., layer L. Furthermore, w0 and w1 shown in FIG. 31 are weights assigned to each three-dimensional point. For example, the three-dimensional data decoding device may calculate a weight based on distance information between two adjacent three-dimensional points to which the inverse Haar transform is applied. For example, the three-dimensional data encoding device may decode a bitstream with improved encoding efficiency by increasing the weight as the distance becomes closer.

[0309] In the example shown in FIG. 31, the coding coefficients after inverse quantization are Ta1, Ta5, Tb1, Tb3, Tc1, and d0, and the decoded values ​​obtained are a0, a1, a2, a3, a4, and a5.

[0310] 32 is a diagram illustrating an example of the syntax of attribute information (attribute_data). The attribute information (attribute_data) includes the number of consecutive zeros (ZeroCnt), the number of attribute dimensions (attribute_dimension), and the coding coefficient (value[j][i]).

[0311] The number of consecutive zeros (ZeroCnt) indicates the number of consecutive values ​​of 0 in the coding coefficient after quantization. Note that the three-dimensional data coding device may binarize ZeroCnt and then perform arithmetic coding.

[0312] Also, as shown in Fig. 32, the three-dimensional data encoding device may determine whether a layer L (layer L) to which the encoding coefficient belongs is equal to or greater than a predetermined threshold TH_layer, and may switch information to be added to the bit stream depending on the determination result. For example, if the determination result is true, the three-dimensional data encoding device may add all the encoding coefficients of the attribute information to the bit stream. Also, if the determination result is false, the three-dimensional data encoding device may add some of the encoding coefficients to the bit stream.

[0313] Specifically, if the determination result is true, the three-dimensional data encoding device adds the encoding result of the three-dimensional information of RGB or YUV of the color information to the bit stream. If the determination result is false, the three-dimensional data encoding device adds some of the color information, such as G or Y, to the bit stream, and does not need to add other components to the bit stream. In this way, the three-dimensional data encoding device can improve encoding efficiency by not adding some of the encoding coefficients of a layer (a layer smaller than TH_layer) that includes encoding coefficients indicating high-frequency components whose degradation is less noticeable visually to the bit stream.

[0314] The number of attribute dimensions (attribute_dimension) indicates the number of dimensions of the attribute information. For example, if the attribute information is color information (RGB or YUV, etc.) of a three-dimensional point, the number of attribute dimensions is set to a value of 3 since color information is three-dimensional. If the attribute information is reflectance, the number of attribute dimensions is set to a value of 1 since reflectance is one-dimensional. Note that the number of attribute dimensions may be added to the header of the attribute information of the bitstream, etc.

[0315] The coding coefficient (value[j][i]) indicates the coding coefficient after quantization of the j-th dimension attribute information of the i-th 3D point. For example, if the attribute information is color information, value

[99] [1] indicates the coding coefficient of the second dimension (e.g., G value) of the 100th 3D point. Also, if the attribute information is reflectance information, value

[0119] [0] indicates the coding coefficient of the first dimension (e.g., reflectance) of the 120th 3D point.

[0316] If the following condition is satisfied, the three-dimensional data encoding device may subtract the value 1 from value[j][i] and entropy encode the resulting value. In this case, the three-dimensional data decoding device restores the encoding coefficient by adding the value 1 to value[j][i] after entropy decoding.

[0317] The above conditions are (1) when attribute_dimension = 1, or (2) when attribute_dimension is 1 or greater and all dimension values ​​are equal. For example, when the attribute information is reflectance, attribute_dimension = 1, so the three-dimensional data encoding device subtracts the value 1 from the encoding coefficient to calculate the value, and encodes the calculated value. The three-dimensional data decoding device adds the value 1 to the decoded value to calculate the encoding coefficient.

[0318] More specifically, for example, when the encoding coefficient of reflectance is 10, the three-dimensional data encoding device encodes the value 9 obtained by subtracting the value 1 from the encoding coefficient value 10. The three-dimensional data decoding device adds the value 1 to the decoded value 9 to calculate the encoding coefficient value 10.

[0319] Furthermore, since attribute_dimension=3 when the attribute information is color, for example, if the coding coefficients after quantization of each component of R, G, and B are the same, the three-dimensional data coding device subtracts a value of 1 from each coding coefficient and codes the obtained value. The three-dimensional data decoding device adds a value of 1 to the decoded value. More specifically, for example, if the coding coefficients of R, G, and B=(1, 1, 1), the three-dimensional data coding device codes (0, 0, 0). The three-dimensional data decoding device adds 1 to each component of (0, 0, 0) to calculate (1, 1, 1). Also, if the coding coefficients of R, G, and B=(2, 1, 2), the three-dimensional data coding device codes (2, 1, 2) as it is. The three-dimensional data decoding device uses the decoded (2, 1, 2) as it is as a coding coefficient.

[0320] In this way, by providing ZeroCnt, a pattern in which all dimensions of value are 0 is not generated, so that the value obtained by subtracting 1 from the value of value can be coded. This improves coding efficiency.

[0321] Also, value[0][i] shown in Fig. 32 indicates the coding coefficient after quantization of the first-dimension attribute information of the i-th three-dimensional point. If the layer L (layerL) to which the coding coefficient belongs is smaller than the threshold TH_layer as shown in Fig. 32, the first-dimension attribute information may be added to the bit stream (attribute information of the second or subsequent dimensions is not added to the bit stream) to reduce the amount of code.

[0322] The three-dimensional data encoding device may switch the calculation method of the value of ZeroCnt depending on the value of attribute_dimension. For example, when attribute_dimension=3, the three-dimensional data encoding device may count the number of times that the values ​​of the encoding coefficients of all components (dimensions) become 0. FIG. 33 is a diagram showing an example of the encoding coefficients and ZeroCnt in this case. For example, in the case of the color information shown in FIG. 33, the three-dimensional data encoding device counts the number of consecutive encoding coefficients in which the R, G, and B components are all 0, and adds the counted number to the bit stream as ZeroCnt. This eliminates the need to encode ZeroCnt for each component, and can reduce overhead. This can improve the encoding efficiency. Note that the three-dimensional data encoding device may calculate ZeroCnt for each dimension even when attribute_dimension is 2 or more, and add the calculated ZeroCnt to the bit stream.

[0323] 34 is a flowchart of a three-dimensional data encoding process according to this embodiment. First, the three-dimensional data encoding device encodes position information (geometry) (S6601). For example, the three-dimensional data encoding device performs encoding using an octree representation.

[0324] Next, the three-dimensional data encoding device converts the attribute information (S6602). For example, when the position of the three-dimensional point changes due to quantization or the like after encoding the position information, the three-dimensional data encoding device reallocates the attribute information of the original three-dimensional point to the changed three-dimensional point. The three-dimensional data encoding device may perform reallocation by interpolating the value of the attribute information according to the amount of change in the position. For example, the three-dimensional data encoding device detects N three-dimensional points before the change that are close to the changed three-dimensional position, and weights and averages the values ​​of the attribute information of the N three-dimensional points based on the distance from the changed three-dimensional position to each of the N three-dimensional points, and sets the obtained value as the value of the attribute information of the changed three-dimensional point. Furthermore, when two or more three-dimensional points change to the same three-dimensional position due to quantization or the like, the three-dimensional data encoding device may assign the average value of the attribute information of the two or more three-dimensional points before the change as the value of the attribute information after the change.

[0325] Next, the three-dimensional data encoding device encodes the attribute information (S6603). For example, when encoding multiple pieces of attribute information, the three-dimensional data encoding device may encode the multiple pieces of attribute information in order. For example, when encoding color and reflectance as attribute information, the three-dimensional data encoding device generates a bitstream in which the encoding result of reflectance is added after the encoding result of color. Note that the encoding results of the multiple pieces of attribute information added to the bitstream may be in any order.

[0326] Furthermore, the three-dimensional data encoding device may add information indicating the start location of the encoded data of each piece of attribute information in the bit stream to a header or the like. This allows the three-dimensional data decoding device to selectively decode attribute information that needs to be decoded, and therefore omits the decoding process of attribute information that does not need to be decoded. This reduces the amount of processing by the three-dimensional data decoding device. Furthermore, the three-dimensional data encoding device may encode multiple pieces of attribute information in parallel and integrate the encoding results into one bit stream. This allows the three-dimensional data encoding device to encode multiple pieces of attribute information at high speed.

[0327] 35 is a flowchart of the attribute information encoding process (S6603). First, the three-dimensional data encoding device generates encoding coefficients from the attribute information by Haar transform (S6611). Next, the three-dimensional data encoding device applies quantization to the encoding coefficients (S6612). Next, the three-dimensional data encoding device generates encoded attribute information (bit stream) by encoding the quantized encoding coefficients (S6613).

[0328] In addition, the three-dimensional data encoding device applies inverse quantization to the quantized coding coefficients (S6614). Next, the three-dimensional data decoding device decodes the attribute information by applying an inverse Haar transform to the inverse quantized coding coefficients (S6615). For example, the decoded attribute information is referred to in subsequent encoding.

[0329] FIG. 36 is a flowchart of the coding coefficient coding process (S6613). First, the three-dimensional data coding device converts the coding coefficient from a signed integer value to an unsigned integer value (S6621). For example, the three-dimensional data coding device converts the signed integer value to an unsigned integer value as follows. If the signed integer value Ta1q is smaller than 0, the unsigned integer value is set to -1-(2×Ta1q). If the signed integer value Ta1q is 0 or greater, the unsigned integer value is set to 2×Ta1q. Note that if the coding coefficient does not become a negative value, the three-dimensional data coding device may code the coding coefficient as an unsigned integer value as it is.

[0330] If all the coding coefficients have not been processed (No in S6622), the three-dimensional data coding device determines whether the value of the coding coefficient to be processed is zero (S6623). If the value of the coding coefficient to be processed is zero (Yes in S6623), the three-dimensional data coding device increments ZeroCnt by 1 (S6624) and returns to step S6622.

[0331] If the value of the coding coefficient to be processed is not zero (No in S6623), the three-dimensional data coding device codes ZeroCnt and resets ZeroCnt to 0 (S6625). The three-dimensional data coding device also arithmetically codes the coding coefficient to be processed (S6626) and returns to step S6622. For example, the three-dimensional data coding device performs binary arithmetic coding. The three-dimensional data coding device may also subtract a value of 1 from the coding coefficient and code the obtained value.

[0332] Furthermore, the processes of steps S6623 to S6626 are repeated for each coding coefficient. If all coding coefficients have been processed (Yes in S6622), the three-dimensional data coding device ends the process.

[0333] 37 is a flowchart of a three-dimensional data decoding process according to this embodiment. First, the three-dimensional data decoding device decodes position information (geometry) from the bit stream (S6631). For example, the three-dimensional data decoding device performs decoding using an octree representation.

[0334] Next, the three-dimensional data decoding device decodes the attribute information from the bit stream (S6632). For example, when decoding a plurality of pieces of attribute information, the three-dimensional data decoding device may decode the plurality of pieces of attribute information in order. For example, when decoding color and reflectance as attribute information, the three-dimensional data decoding device decodes the encoding result of color and the encoding result of reflectance in the order in which they are added to the bit stream. For example, when the encoding result of reflectance is added after the encoding result of color in the bit stream, the three-dimensional data decoding device decodes the encoding result of color, and then decodes the encoding result of reflectance. Note that the three-dimensional data decoding device may decode the encoding results of the attribute information added to the bit stream in any order.

[0335] Furthermore, the three-dimensional data decoding device may obtain information indicating the start location of the encoded data of each piece of attribute information in the bit stream by decoding a header or the like. This allows the three-dimensional data decoding device to selectively decode attribute information that needs to be decoded, and therefore omits the decoding process of attribute information that does not need to be decoded. This allows the amount of processing by the three-dimensional data decoding device to be reduced. Furthermore, the three-dimensional data decoding device may decode multiple pieces of attribute information in parallel and integrate the decoding results into one three-dimensional point cloud. This allows the three-dimensional data decoding device to decode multiple pieces of attribute information at high speed.

[0336] 38 is a flowchart of the attribute information decoding process (S6632). First, the three-dimensional data decoding device decodes the coding coefficients from the bit stream (S6641). Next, the three-dimensional data decoding device applies inverse quantization to the coding coefficients (S6642). Next, the three-dimensional data decoding device decodes the attribute information by applying an inverse Haar transform to the coding coefficients after the inverse quantization (S6643).

[0337] 39 is a flowchart of the coding coefficient decoding process (S6641). First, the three-dimensional data decoding device decodes ZeroCnt from the bit stream (S6651). If all coding coefficients have not been processed (No in S6652), the three-dimensional data decoding device determines whether ZeroCnt is greater than 0 (S6653).

[0338] If ZeroCnt is greater than zero (Yes in S6653), the three-dimensional data decoding device sets the coding coefficient to be processed to 0 (S6654). Next, the three-dimensional data decoding device subtracts 1 from ZeroCnt (S6655) and returns to step S6652.

[0339] If ZeroCnt is zero (No in S6653), the three-dimensional data decoding device decodes the coding coefficient to be processed (S6656). For example, the three-dimensional data decoding device uses binary arithmetic decoding. In addition, the three-dimensional data decoding device may add a value of 1 to the decoded coding coefficient.

[0340] Next, the three-dimensional data decoding device decodes ZeroCnt, sets the obtained value to ZeroCnt (S6657), and returns to step S6652.

[0341] Furthermore, the processes of steps S6653 to S6657 are repeated for each coding coefficient. Furthermore, if all coding coefficients have been processed (Yes in S6652), the three-dimensional data decoding device converts the decoded coding coefficients from unsigned integer values ​​to signed integer values ​​(S6658). For example, the three-dimensional data decoding device may convert the decoded coding coefficients from unsigned integer values ​​to signed integer values ​​as follows. If the LSB (least significant bit) of the decoded unsigned integer value Ta1u is 1, the signed integer value Ta1q is set to -((Ta1u+1)>>1). If the LSB of the decoded unsigned integer value Ta1u is not 1 (is 0), the signed integer value Ta1q is set to (Ta1u>>1). Note that if the coding coefficient does not become a negative value, the three-dimensional data decoding device may use the decoded coding coefficient as it is as a signed integer value.

[0342] 40 is a block diagram of an attribute information encoding unit 6600 included in the three-dimensional data encoding device. The attribute information encoding unit 6600 includes a sorting unit 6601, a Haar transform unit 6602, a quantization unit 6603, an inverse quantization unit 6604, an inverse Haar transform unit 6605, a memory 6606, and an arithmetic encoding unit 6607.

[0343] The sorting unit 6601 generates a Morton code using the position information of the 3D points, and sorts the multiple 3D points in Morton code order. The Haar transform unit 6602 generates coding coefficients by applying a Haar transform to the attribute information. The quantization unit 6603 quantizes the coding coefficients of the attribute information.

[0344] The inverse quantization unit 6604 inversely quantizes the quantized coding coefficients. The inverse Haar transform unit 6605 applies an inverse Haar transform to the coding coefficients. The memory 6606 stores values ​​of attribute information of multiple decoded 3D points. For example, the attribute information of the decoded 3D points stored in the memory 6606 may be used for predicting uncoded 3D points.

[0345] The arithmetic coding unit 6607 calculates ZeroCnt from the coding coefficients after quantization, and arithmetically codes the ZeroCnt. Furthermore, the arithmetic coding unit 6607 arithmetically codes the non-zero coding coefficients after quantization. The arithmetic coding unit 6607 may binarize the coding coefficients before arithmetic coding. Furthermore, the arithmetic coding unit 6607 may generate and code various header information.

[0346] 41 is a block diagram of the attribute information decoding unit 6610 included in the three-dimensional data decoding device. The attribute information decoding unit 6610 includes an arithmetic decoding unit 6611, an inverse quantization unit 6612, an inverse Haar transform unit 6613, and a memory 6614.

[0347] The arithmetic decoding unit 6611 arithmetically decodes the ZeroCnt and the coding coefficients included in the bit stream. Note that the arithmetic decoding unit 6611 may also decode various types of header information.

[0348] The inverse quantization unit 6612 inverse quantizes the arithmetically decoded coding coefficients. The inverse Haar transform unit 6613 applies an inverse Haar transform to the coding coefficients after the inverse quantization. The memory 6614 stores values ​​of attribute information of multiple decoded 3D points. For example, the attribute information of the decoded 3D points stored in the memory 6614 may be used to predict undecoded 3D points.

[0349] In the above embodiment, the example of coding 3D points from lower layer to upper layer is shown, but the coding order is not necessarily limited to this.For example, the coding coefficient after Haar transform may be scanned from upper layer to lower layer.In this case, the 3D data coding device may also code the number of consecutive values ​​of 0 as ZeroCnt.

[0350] In addition, the three-dimensional data encoding device may switch whether or not to use the encoding method using ZeroCnt described in the present embodiment on a WLD, SPC, or volume basis. In this case, the three-dimensional data encoding device may add information indicating whether or not the encoding method using ZeroCnt has been applied to the header information. This allows the three-dimensional data decoding device to perform appropriate decoding. As an example of the switching method, for example, the three-dimensional data encoding device counts the number of occurrences of an encoding coefficient with a value of 0 for one volume. If the count value exceeds a predetermined threshold, the three-dimensional data encoding device applies the method using ZeroCnt to the next volume, and if the count value is equal to or less than the threshold, does not apply the method using ZeroCnt to the next volume. This allows the three-dimensional data encoding device to switch whether or not to apply the encoding method using ZeroCnt appropriately according to the characteristics of the three-dimensional point to be encoded, thereby improving encoding efficiency.

[0351] Another method (variation) of this embodiment will be described below. The three-dimensional data encoding device scans and encodes the quantized encoding coefficients (unsigned integer values) in a certain order. For example, the three-dimensional data encoding device encodes a plurality of three-dimensional points in order from the three-dimensional points included in the lower layer toward the upper layer.

[0352] Fig. 42 is a diagram showing an example of the first code string and the second code string when this technique is used for the attribute information shown in Fig. 27. In this example, the three-dimensional data encoding device encodes multiple coding coefficients in the order of Ta1q to Ta5q, Tb1q, Tb3q, Tc1q, and d0q included in the lower layer L. Here, the lower the layer, the more likely the coding coefficients after quantization are to become 0. The following factors can be cited as reasons for this.

[0353] The coding coefficients of the lower layer L show higher frequency components than those of the upper layer, and therefore tend to be 0 depending on the 3D point to be coded. Also, due to the switching of the quantization scale according to the importance level and the like described above, the quantization scale becomes larger in the lower layer, and the coding coefficients after quantization tend to be 0.

[0354] In this way, the lower the layer, the more likely it is that the coding coefficient after quantization will be 0, and the more likely it is that the value 0 will occur consecutively in the first code string. The three-dimensional data coding device counts the number of times the value 0 occurs in the first code string, and encodes the number of times the value 0 occurs consecutively (ZeroCnt) instead of consecutive values ​​of 0. This makes it possible to improve coding efficiency when the coding coefficient after quantization has consecutive values ​​of 0 by coding the number of consecutive 0s rather than coding a large number of 0s.

[0355] Furthermore, the three-dimensional data encoding device may encode information indicating the total number of occurrences of the value 0. This can reduce the overhead of encoding ZeroCnt and improve encoding efficiency.

[0356] For example, the three-dimensional data encoding device encodes the total number of encoding coefficients with a value of 0 as TotalZeroCnt. As a result, in the example shown in FIG. 42, when the three-dimensional data decoding device decodes the second ZeroCnt (value 1) included in the second code string, the total number of decoded ZeroCnt becomes N+1 (=TotalZeroCnt). Therefore, the three-dimensional data decoding device can grasp that no more 0s will be generated. Therefore, the three-dimensional data encoding device does not need to encode ZeroCnt for each value thereafter, and the amount of code can be reduced.

[0357] Furthermore, the three-dimensional data encoding device may entropy-encode TotalZeroCnt. For example, the three-dimensional data encoding device binarizes the value of TotalZeroCnt using a truncated unary code of the total number T of encoded three-dimensional points, and arithmetically encodes each bit after binarization. At this time, the three-dimensional data encoding device may improve the encoding efficiency by using a different encoding table for each bit. For example, the three-dimensional data encoding device uses encoding table 1 for the first bit, encoding table 2 for the second bit, and encoding table 3 for the bits thereafter. In this way, the three-dimensional data encoding device can improve the encoding efficiency by switching the encoding table for each bit.

[0358] Furthermore, the three-dimensional data encoding device may perform arithmetic coding after binarizing TotalZeroCnt using exponential Golomb. This can improve efficiency compared to binarized arithmetic coding using truncated unary code when the value of TotalZeroCnt is likely to become large. The three-dimensional data encoding device may add a flag to the header for switching between using truncated unary code or exponential Golomb. This allows the three-dimensional data encoding device to improve encoding efficiency by selecting an optimal binarization method. Furthermore, the three-dimensional data decoding device can switch binarization methods by referring to the flag included in the header, and correctly decode the bit stream.

[0359] Fig. 43 is a diagram showing an example of the syntax of attribute information (attribute_data) in this modification. The attribute information (attribute_data) shown in Fig. 43 further includes a total number of zeros (TotalZeroCnt) in addition to the attribute information shown in Fig. 32. Note that other information is the same as in Fig. 32. The total number of zeros (TotalZeroCnt) indicates the total number of coding coefficients with a value of 0 after quantization.

[0360] Also, the three-dimensional data encoding device may switch the calculation method of the values ​​of TotalZereCnt and ZeroCnt depending on the value of attribute_dimension. For example, when attribute_dimension=3, the three-dimensional data encoding device may count the number of times that the values ​​of the encoding coefficients of all components (dimensions) become 0. FIG. 44 is a diagram showing an example of the encoding coefficients, ZeroCnt, and TotalZeroCnt in this case. For example, in the case of the color information shown in FIG. 44, the three-dimensional data encoding device counts the number of consecutive encoding coefficients in which the R, G, and B components are all 0, and adds the counted numbers to the bit stream as TotalZeroCnt and ZeroCnt. This eliminates the need to encode TotalZeroCnt and ZeroCnt for each component, and can reduce overhead. This can improve the encoding efficiency. Note that the three-dimensional data encoding device may calculate TotalZeoCnt and ZeroCnt for each dimension even when attribute_dimension is 2 or more, and add the calculated TotalZeoCnt and ZeroCnt to the bit stream.

[0361] 45 is a flowchart of the coding coefficient coding process (S6613) in this modified example. First, the three-dimensional data coding device converts the coding coefficient from a signed integer value to an unsigned integer value (S6661). Next, the three-dimensional data coding device codes TotalZeroCnt (S6662).

[0362] If all the coding coefficients have not been processed (No in S6663), the three-dimensional data coding device determines whether the value of the coding coefficient to be processed is zero (S6664). If the value of the coding coefficient to be processed is zero (Yes in S6664), the three-dimensional data coding device increments ZeroCnt by 1 (S6665) and returns to step S6663.

[0363] If the value of the coding coefficient to be processed is not zero (No in S6664), the three-dimensional data coding device determines whether TotalZeroCnt is greater than 0 (S6666). If TotalZeroCnt is greater than 0 (Yes in S6666), the three-dimensional data coding device codes ZeroCnt and sets TotalZeroCnt to TotalZeroCnt-ZeroCnt (S6667).

[0364] After step S6667, or if TotalZeroCnt is 0 (No in S6666), the three-dimensional data encoding device encodes the encoding coefficients, resets ZeroCnt to 0 (S6668), and returns to step S6663. For example, the three-dimensional data encoding device performs binary arithmetic encoding. Alternatively, the three-dimensional data encoding device may subtract a value of 1 from the encoding coefficients and encode the obtained value.

[0365] Furthermore, the processes of steps S6664 to S6668 are repeated for each coding coefficient. If all coding coefficients have been processed (Yes in S6663), the three-dimensional data coding device ends the process.

[0366] 46 is a flowchart of the coding coefficient decoding process (S6641) in this modified example. First, the three-dimensional data decoding device decodes TotalZeroCnt from the bit stream (S6671). Next, the three-dimensional data decoding device decodes ZeroCnt from the bit stream and sets TotalZeroCnt to TotalZeroCnt-ZeroCnt (S6672).

[0367] If all the encoding coefficients have not been processed (No in S6673), the three-dimensional data encoding device determines whether ZeroCnt is greater than 0 (S6674).

[0368] If ZeroCnt is greater than zero (Yes in S6674), the three-dimensional data decoding device sets the coding coefficient to be processed to 0 (S6675). Next, the three-dimensional data decoding device subtracts 1 from ZeroCnt (S6676) and returns to step S6673.

[0369] If ZeroCnt is zero (No in S6674), the three-dimensional data decoding device decodes the coding coefficient to be processed (S6677). For example, the three-dimensional data decoding device uses binary arithmetic decoding. Also, the three-dimensional data decoding device may add a value of 1 to the decoded coding coefficient.

[0370] Next, the three-dimensional data decoding device judges whether TotalZeroCnt is greater than 0 (S6678). If TotalZeroCnt is greater than 0 (Yes in S6678), the three-dimensional data decoding device decodes ZeroCnt, sets the obtained value to ZeroCnt, sets TotalZeroCnt to TotalZeroCnt-ZeroCnt (S6679), and returns to step S6673. If TotalZeroCnt is 0 (No in S6678), the three-dimensional data decoding device returns to step S6673.

[0371] Furthermore, the processes of steps S6674 to S6679 are repeated for each coding coefficient. Furthermore, if all coding coefficients have been processed (Yes in S6673), the three-dimensional data coding device converts the decoded coding coefficients from unsigned integer values ​​to signed integer values ​​(S6680).

[0372] Fig. 47 is a diagram showing another syntax example of attribute information (attribute_data). The attribute information (attribute_data) shown in Fig. 47 includes value[j][i]_greater_zero_flag, value[j][i]_greater_one_flag, and value[j][i] instead of the coding coefficient (value[j][i]) shown in Fig. 32. Note that other information is the same as in Fig. 32.

[0373] Value[j][i]_greater_zero_flag indicates whether the value of the coding coefficient (value[j][i]) is greater than 0 (is greater than or equal to 1). In other words, value[j][i]_greater_zero_flag indicates whether the value of the coding coefficient (value[j][i]) is 0 or not.

[0374] For example, if the value of the coding coefficient is greater than 0, value[j][i]_greater_zero_flag is set to the value 1, and if the value of the coding coefficient is 0, value[j][i]_greater_zero_flag is set to the value 0. If the value of value[j][i]_greater_zero_flag is 0, the three-dimensional data coding device may not add value[j][i] to the bitstream. In this case, the three-dimensional data decoding device may determine that the value of value[j][i] is 0. This makes it possible to reduce the amount of coding.

[0375] Value[j][i]_greater_one_flag indicates whether the value of the coding coefficient (value[j][i]) is greater than 1 (2 or more). In other words, value[j][i]_greater_one_flag indicates whether the value of the coding coefficient (value[j][i]) is 1 or not.

[0376] For example, if the value of the coding coefficient is greater than 1, value[j][i]_greater_one_flag is set to the value 1. Otherwise (if the value of the coding coefficient is 1 or less), value[j][i]_greater_one_flag is set to the value 0. If the value of value[j][i]_greater_one_flag is 0, the three-dimensional data coding device may not add value[j][i] to the bitstream. In this case, the three-dimensional data decoding device may determine that the value of value[j][i] is 1.

[0377] value[j][i] indicates the quantized coding coefficient of the j-th attribute information of the i-th 3D point. For example, if the attribute information is color information, value

[99] [1] indicates the coding coefficient of the second dimension (e.g., G value) of the 100th 3D point. Also, if the attribute information is reflectance information, value

[0119] [0] indicates the coding coefficient of the first dimension (e.g., reflectance) of the 120th 3D point.

[0378] The three-dimensional data encoding device may add value[j][i] to the bitstream when value[j][i]_greater_zero_flag=1 and value[j][i]_greater_one_flag=1. The three-dimensional data encoding device may also add a value obtained by subtracting 2 from value[j][i] to the bitstream. In this case, the three-dimensional data decoding device calculates the encoding coefficient by adding the value 2 to the decoded value[j][i].

[0379] The three-dimensional data encoding device may entropy-encode value[j][i]_greater_zero_flag and value[j][i]_greater_one_flag. For example, binary arithmetic encoding and binary arithmetic decoding may be used. This can improve the encoding efficiency.

[0380] (Embodiment 8) In this embodiment, lossless attribute coding will be described. In order to achieve high compression, attribute information included in PCC (Point Cloud Compression) data is transformed using multiple methods such as Lifting, RAHT (Region Adaptive Hierarchical Transform), or other transformation methods. Here, Lifting is one of the transformation methods using LoD (Level of Detail).

[0381] Since important signal information tends to be contained in low frequency components, the amount of code is reduced by quantizing high frequency components, i.e. the transform has strong energy compaction properties.

[0382] On the other hand, lossless compression is needed to preserve the original information while reducing the number of bits. Existing transforms such as lifting or RAHT cannot achieve lossless compression because the transform contains division and square root operators. To achieve efficient and effective lossless compression, a low-complexity integer-to-integer transform is needed.

[0383] Fig. 48 is a diagram showing the configuration of a three-dimensional data encoding device. As shown in Fig. 48, the three-dimensional data encoding device includes an integer transform unit 8301 and an entropy encoding unit 8302. The integer transform unit 8301 generates coefficient values ​​by performing integer transform on input point cloud data. The entropy encoding unit 8302 generates a bit stream by entropy encoding the coefficient values.

[0384] Fig. 49 is a diagram showing the configuration of a three-dimensional data decoding device. As shown in Fig. 49, the three-dimensional data decoding device includes an entropy decoding unit 8303 and an inverse integer transform unit 8304. The entropy decoding unit 8303 obtains coefficient values ​​by decoding the bit stream. The inverse integer transform unit 8304 generates output point cloud data by performing an inverse integer transform on the coefficient values.

[0385] The RAHT will be described below. RAHT is an example of a transformation process applied to a three-dimensional point. FIG. 50 is a diagram for explaining RAHT. Also, the m-th low-frequency component L l,m and the high frequency component H l,m are the two low-frequency components C l+1,2m and C l+1,2m+1 Therefore, it is expressed by the following (Equation O1). In other words, the low frequency component L l,m is expressed by (Equation O2), and the high frequency component H l,m is expressed by (Equation O3).

[0386] Moreover, the high frequency components are coded by quantization and entropy coding. The low frequency components are used in the next layer as shown in (Equation O4). Moreover, the coefficients α and β are updated for each layer. The coefficients α and β are expressed by (Equation O5) and (Equation O6). Moreover, the weights w l,m is expressed as (Equation O7).

[0387]

number

[0388] Next, we will explain integer-to-integer conversion. The RAHT process includes square root and division operations. This means that RAHT cannot achieve lossless compression because it loses information. On the other hand, integer-to-integer conversion can achieve lossless compression.

[0389] FIG. 51 is a diagram for explaining the integer-to-integer transform. In the integer-to-integer transform, fixed values ​​are used as coefficients in the RAHT. For example, the non-normalized Haar transform shown in the following (Equation O8) is used. That is, the low-frequency component L l,m is expressed by (Equation O9), and the high frequency component H l,m is expressed by (Equation O10).

[0390] Moreover, the high frequency components are coded by quantization and entropy coding, and the low frequency components are used in the next layer as shown in (Equation O11).

[0391]

number

[0392] Moreover, the unnormalized Haar transform can be rewritten as (Equation O12) and (Equation O13).

[0393]

number

[0394] The integer Haar transform is realized by (Equation O14) and (Equation O15), and the inverse integer Haar transform is realized by (Equation O16) and (Equation O17). Here, || is a floor function. In addition, |H l,m / 2| is included, the integer Haar transform and the inverse integer Haar transform can be used to transform this |H l,m / 2| is cancelled out. Thus, a lossless transformation is realized. Here, C i,j is defined as an integer, so H i,j and L i,j is also an integer.

[0395]

number

[0396] Therefore, efficient implementation can be achieved by the following (Equation O18) to (Equation O21). In other words, the conversion can be achieved by one addition, one subtraction, and one right shift (shift down).

[0397]

number

[0398] A recursive integer-integer transformation will now be described. FIG. 52 is a diagram for explaining hierarchical transformation processing. When applying the Haar transformation to an image, a data pair is required to perform a transformation suitable for pixel transformation. On the other hand, in the Haar transformation for a 3D point cloud, if a 3D point pair, which is a pair of point clouds, can be formed, integer Haar is applied, and if a 3D point pair cannot be used, the data of the 3D point is moved to the next layer (level). This processing is then performed recursively.

[0399] Next, the configuration of the three-dimensional data encoding device will be described. Fig. 53 is a block diagram of a three-dimensional data encoding device 8310. The three-dimensional data encoding device 8310 generates encoded data (encoded stream) by encoding point group data (point cloud). This three-dimensional data encoding device 8310 includes a position information encoding unit 8311, a lossless attribute information encoding unit 8312, an additional information encoding unit 8313, and a multiplexing unit 8314.

[0400] The position information encoding unit 8311 generates encoded position information by encoding the position information. For example, the position information encoding unit 8311 encodes the position information using an N-ary tree structure such as an octet tree. Specifically, in the octet tree, the target space is divided into eight nodes (subspaces), and 8-bit information (occupancy code) indicating whether or not a point cloud is included in each node is generated. In addition, the node including the point cloud is further divided into eight nodes, and 8-bit information indicating whether or not a point cloud is included in each of the eight nodes is generated. This process is repeated until the number of point clouds included in a predetermined hierarchy or node is equal to or less than a threshold value.

[0401] The lossless attribute information encoding unit 8312 generates encoded attribute information, which is encoded data, by encoding the attribute information using the configuration information generated by the position information encoding unit 8311.

[0402] The additional information encoding unit 8313 generates encoded additional information by encoding the additional information included in the point cloud data.

[0403] The multiplexing unit 8314 multiplexes the encoding position information, the encoding attribute information, and the encoding additional information to generate encoded data (encoded stream), and transmits the generated encoded data. In addition, the encoded additional information is used during decoding.

[0404] 54 is a block diagram of the lossless attribute information encoding unit 8312. The lossless attribute information encoding unit 8312 includes an integer conversion unit 8321 and an entropy encoding unit 8322. The integer conversion unit 8321 generates coefficient values ​​by performing integer conversion (e.g., integer Haar transform) on the attribute information. The entropy encoding unit 8322 generates encoded attribute information by entropy encoding the coefficient values.

[0405] 55 is a block diagram of the integer converter 8321. The integer converter 8321 includes a sorting unit 8323 and an integer Haar converter 8324. The sorting unit 8323 sorts the attribute information based on the position information. For example, the sorting unit 8323 sorts the attribute information into Morton order. The integer Haar converter 8324 generates coefficient values ​​by performing an integer Haar transform on the sorted attribute information.

[0406] Next, the configuration of the three-dimensional data decoding device according to this embodiment will be described. Fig. 56 is a block diagram showing the configuration of a three-dimensional data decoding device 8330. The three-dimensional data decoding device 8330 restores the point cloud data by decoding the coded data (coded stream) generated by coding the point cloud data. This three-dimensional data decoding device 8330 includes a demultiplexing unit 8331, a plurality of position information decoding units 8332, a plurality of lossless attribute information decoding units 8333, and an additional information decoding unit 8334.

[0407] The demultiplexing unit 8331 demultiplexes the coded data (coded stream) to generate coded position information, coded attribute information, and coded additional information.

[0408] The position information decoding unit 8332 generates position information by decoding the encoded position information. The lossless attribute information decoding unit 8333 generates attribute information by decoding the encoded attribute information. For example, the lossless attribute information decoding unit 8333 generates attribute information by performing an inverse integer transform (e.g., an inverse integer Haar transform) on the encoded attribute information. The additional information decoding unit 8334 generates additional information by decoding the encoded additional information.

[0409] 57 is a block diagram of the lossless attribute information decoding unit 8333. The lossless attribute information decoding unit 8333 includes an entropy decoding unit 8341 and an inverse integer transform unit 8342. The entropy decoding unit 8341 generates coefficient values ​​by entropy decoding the encoded attribute information. The inverse integer transform unit 8342 generates attribute information by performing an inverse integer transform (for example, an inverse integer Haar transform) on the coefficient values.

[0410] 58 is a block diagram of the inverse integer transform unit 8342. The inverse integer transform unit 8342 includes a sorting unit 8343 and an inverse integer Haar transform unit 8344. The sorting unit 8343 sorts the coefficient values ​​based on the position information. For example, the sorting unit 8343 sorts the coefficient values ​​into Morton order. The inverse integer Haar transform unit 8344 generates attribute information by performing an inverse integer Haar transform on the sorted coefficient values.

[0411] Furthermore, the three-dimensional data encoding device may add information indicating whether lossless encoding or lossy encoding has been used to the header of the bitstream or the like. For example, the three-dimensional data encoding device adds lossless_enable_flag to the header. When lossless_enable_flag=1, the three-dimensional data decoding device applies an inverse integer Haar transform to decode the losslessly encoded bitstream. When lossless_enable_flag=0, the three-dimensional data decoding device applies an inverse RAHT to decode the lossy encoded bitstream. In this way, the three-dimensional data decoding device can correctly decode the bitstream by switching the inverse transform process according to the value of lossless_enable_flag.

[0412] Note that the information indicating whether the bitstream is encoded by lossless encoding or lossy encoding is not necessarily limited to this, and may be, for example, the value of the quantization parameter QP or the quantization step Qstep. For example, if the value of the quantization parameter or the quantization step is a specific value (e.g., QP=4 or Qstep=1), the three-dimensional data decoding device may determine that the bitstream is encoded by lossless encoding, and may apply the inverse integer Haar transform to decode the losslessly encoded bitstream. Also, for example, if the value of the quantization parameter or the quantization step is greater than a specific value (e.g., QP=4 or Qstep=1), the three-dimensional data decoding device may determine that the bitstream is encoded by lossy encoding, and may apply the inverse RAHT to decode the lossy encoded bitstream.

[0413] Next, the lossless attribute information encoding process will be described with reference to FIG 59, which is a flowchart of the lossless attribute information encoding process.

[0414] First, the three-dimensional data encoding device rearranges the attribute information of the three-dimensional point group (S8301). For example, the three-dimensional data encoding device rearranges the attribute information of the three-dimensional point group into Morton order.

[0415] Next, the three-dimensional data encoding device selects a target point to be processed from the three-dimensional point cloud (S8302). Specifically, the three-dimensional data encoding device selects the top three-dimensional point from the three-dimensional point cloud sorted in Morton order.

[0416] Next, the three-dimensional data encoding device determines whether or not a three-dimensional point pair (Point Pair), which is a three-dimensional point located next to the target three-dimensional point in Morton order, exists (S8303). If a three-dimensional point pair exists (Yes in S8304), the three-dimensional data encoding device performs an integer Haar transform using the three-dimensional point pair to generate coefficient values ​​including high-frequency components and low-frequency components (S8305). Next, the three-dimensional data encoding device encodes (e.g., entropy encodes) the generated high-frequency components, and stores the encoded high-frequency components in a bit stream (S8306). In addition, the three-dimensional data encoding device stores the low-frequency components in a memory or the like for processing in the next layer (S8307).

[0417] On the other hand, if a 3D point pair does not exist (No in S8304), the 3D data encoding device stores the attribute information of the target 3D point in a memory or the like for the next layer (S8307).

[0418] If the target 3D point is not the last 3D point in the target hierarchy, which is the hierarchy to be processed (No in S8308), the 3D data encoding device selects the next 3D point in Morton order as the target 3D point (S8302), and performs processing from step S8303 onwards for the selected target 3D point. Note that the next 3D point in Morton order is the next 3D point of the 3D point pair if a 3D point pair exists, and is the next 3D point of the target 3D point if no 3D point pair exists.

[0419] If the target 3D point is the last 3D point in the target layer (Yes in S8308), the 3D data encoding device starts processing the next layer (the layer one level up) (S8309). If the original target layer is not the last layer (the top layer) (No in S8310), the 3D data encoding device selects the first 3D point in Morton order in the next layer as the target 3D point (S8302), and performs the processes in steps S8303 and after on the selected target 3D point.

[0420] Furthermore, if the original target layer is the last layer (Yes in S8310), the three-dimensional data encoding device encodes (e.g., entropy encodes) the low-frequency components generated in the last layer (top layer) and stores the encoded low-frequency components in the bitstream (S8311). Through the above processing, encoding attribute information is generated that includes the encoded high-frequency components for each three-dimensional point pair included in each layer and the encoded low-frequency components of the top layer.

[0421] Next, the lossless attribute information decoding process will be described with reference to FIG 60, which is a flowchart of the lossless attribute information decoding process.

[0422] First, the three-dimensional data decoding device decodes coefficient values ​​from the bit stream (S8321). The coefficient values ​​include high-frequency components for each three-dimensional point pair included in each layer and low-frequency components in the top layer. Next, the three-dimensional data decoding device sorts the obtained coefficient values ​​(S8322). For example, the three-dimensional data decoding device sorts multiple high-frequency components into Morton order.

[0423] Next, the three-dimensional data decoding device acquires the low-frequency component to be processed and the high-frequency component to be processed, which are the low-frequency component and the high-frequency component of the three-dimensional point pair to be processed (S8323 and S8324). Specifically, the low-frequency component to be processed is the low-frequency component decoded from the bit stream in the top layer, and in layers other than the top layer, it is the low-frequency component obtained by inverse transform processing in the layer one level above. Also, the high-frequency component to be processed is the leading high-frequency component among the high-frequency components sorted in Morton order in the top layer. Note that if there is no three-dimensional point pair, there is no high-frequency component to be processed.

[0424] Next, if a three-dimensional point pair exists (Yes in S8325), that is, if a high-frequency component to be processed exists, the three-dimensional data decoding device performs an inverse integer Haar transform using the low-frequency component to be processed and the high-frequency component to be processed to generate a low-frequency component of the next lower layer (S8326). Note that if the target layer is the bottom layer, attribute information is generated by the inverse integer Haar transform.

[0425] Next, the three-dimensional data decoding device stores the generated low-frequency components in a memory or the like for processing at the next layer (S8327).

[0426] On the other hand, if a 3D point pair does not exist (No in S8325), the 3D data decoding device stores the low frequency components to be processed in a memory or the like for the next layer (S8327).

[0427] If the coefficient value (three-dimensional point pair) to be processed is not the last coefficient value in the target hierarchy (No in S8328), the three-dimensional data decoding device selects the next three-dimensional point pair in Morton order as the three-dimensional point to be processed, and performs processing from step S8323 onwards on the selected three-dimensional point pair.

[0428] If the coefficient value to be processed is the last coefficient value in the target layer (Yes in S8328), the three-dimensional data decoding device starts processing the next layer (one layer below) (S8329). Also, if the original target layer is not the last layer (lowest layer) (No in S8330), the three-dimensional data decoding device selects the first 3D point pair in Morton order in the next layer as the 3D point pair to be processed, and performs the processes from step S8323 onwards on the selected 3D point pair.

[0429] Furthermore, the 3D data decoding device ends the process if the original target layer is the last layer (Yes in S8330).By the above process, attribute information of all 3D points is obtained.

[0430] Next, a configuration example of the integer Haar transform unit 8324 and the inverse integer Haar transform unit 8344 will be described. FIG. 61 is a diagram showing a configuration example of the integer Haar transform unit 8324. As shown in FIG. 61, the integer Haar transform unit 8324 includes a subtraction unit 8351, a right shift unit 8352, and an addition unit 8353. Here, C1 and C2 are attribute information of the 3D point pair in the lowest layer, and are low-frequency components of the 3D point pair obtained in the layer one level lower in layers other than the lowest layer. Also, H is a high-frequency component of the 3D point pair, and L is a low-frequency component of the 3D point pair. The configuration in the same figure realizes the calculations shown in (Equation O18) and (Equation O19).

[0431] Fig. 62 is a diagram showing an example configuration of the inverse integer Haar transform unit 8344. As shown in Fig. 62, the inverse integer Haar transform unit 8344 includes a right shift unit 8354, a subtraction unit 8355, and an addition unit 8356. The configuration in this figure realizes the calculations shown in (Equation O20) and (Equation O21).

[0432] In at least one of the forward transformation and the inverse transformation, the input data may be divided into a plurality of data pieces of a predetermined unit, and the divided data pieces may be processed in parallel, thereby speeding up the processing.

[0433] Next, an example of switching between lossless coding (integer Haar transform) and lossy coding (RAHT) will be described. Fig. 63 is a diagram showing the configuration of a three-dimensional data coding device in this case. The three-dimensional data coding device selectively performs lossless coding (lossless compression) and lossy coding (lossy compression). In addition, the three-dimensional data coding device may indicate the coding mode by a flag or QP.

[0434] The three-dimensional data encoding device shown in FIG. 63 includes a rearrangement unit 8361, a switching unit 8362, a RAHT unit 8363, a quantization unit 8364, an integer conversion unit 8365, and an entropy encoding unit 8366.

[0435] The sorting unit 8361 sorts the attribute information based on the position information, for example, into Morton order. The switching unit 8362 outputs the sorted attribute information to the RAHT unit 8363 or the integer conversion unit 8365. For example, the switching unit 8362 switches between using RAHT or integer Haar transform based on LOSSLESS_FLAG. Here, LOSSLESS_FLAG is a flag indicating whether to use RAHT (lossy coding) or integer Haar transform (lossless coding), and when LOSSLESS_FLAG is on (for example, value 1), integer Haar transform (lossless coding) is used, and when LOSSLESS_FLAG is off (for example, value 0), RAHT (lossy coding) is used.

[0436] Alternatively, the three-dimensional data encoding device may determine that the encoding is lossless when the value of the quantization parameter QP is a certain value α. Here, the value α is, for example, a quantization value calculated from the QP value, or a value that makes the value of the quantization step Qstep 1. For example, when QP=4, and Qstep=1, then α=4.

[0437] Alternatively, the switching between RAHT and integer Haar transform may be performed in any manner, not limited to LOSSLESS_FLAG or QP value. For example, the three-dimensional data encoding device may add an Enable_Integer_Haar_Transform flag to a header or the like, and apply the integer Haar transform when Enable_Integer_Haar_Transform=1, and apply the RAHT when Enable_Integer_Haar_Transform=0.

[0438] The RAHT unit 8363 generates coefficient values ​​by applying RAHT to the attribute information. The quantization unit 8364 quantizes the attribute information to generate quantized coefficients. The integer transform unit 8365 generates coefficient values ​​by applying integer Haar transform to the attribute information. The entropy coding unit 8366 entropy codes the quantized values ​​generated by the quantization unit 8364 or the coefficient values ​​generated by the integer transform unit 8365 to generate coded attribute information.

[0439] Figure 64 is a diagram showing the configuration of a three-dimensional data decoding device corresponding to the three-dimensional data encoding device shown in Figure 63. The three-dimensional data decoding device shown in Figure 64 includes an entropy decoding unit 8371, a rearrangement unit 8372, a switching unit 8373, an inverse quantization unit 8374, an inverse RAHT unit 8375, and an inverse integer transform unit 8376.

[0440] The entropy decoding unit 8371 generates coefficient values ​​(or quantized coefficients) by entropy decoding the encoding attribute information. The sorting unit 8372 sorts the coefficient values, for example, into Morton order, based on the position information. The switching unit 8373 outputs the sorted coefficient values ​​to the inverse quantization unit 8374 or the inverse integer transform unit 8376. For example, the switching unit 8373 switches between using RAHT or integer Haar transform based on LOSSLESS_FLAG. The switching method of the switching unit 8373 is the same as the switching method in the switching unit 8362 described above. The three-dimensional data decoding device acquires LOSSLESS_FLAG, a QP value, or an Enable_Integer_Haar_Transform flag from the bitstream.

[0441] The inverse quantization unit 8374 generates coefficient values ​​by inverse quantizing the quantized coefficients. The inverse RAHT unit 8375 generates attribute information by applying inverse RAHT to the coefficient values. The inverse integer transform unit 8376 generates attribute information by applying inverse integer Haar transform to the coefficient values.

[0442] In the examples shown in Figures 63 and 64, quantization is not performed when the integer Haar transform is applied, but quantization may be performed when the integer Haar transform is applied. Figure 65 is a diagram showing the configuration of a three-dimensional data encoding device in this case. Figure 66 is a diagram showing the configuration of a three-dimensional data decoding device.

[0443] As shown in FIG. 65, the quantization unit 8364A quantizes the coefficient values ​​generated by the RAHT unit 8363 and the coefficient values ​​generated by the integer conversion unit 8365 to generate quantized coefficients.

[0444] 66, the inverse quantization unit 8374A generates coefficient values ​​by inverse quantizing the quantized coefficients. The switching unit 8373A outputs the coefficient values ​​to the inverse RAHT unit 8375 or the inverse integer transform unit 8376.

[0445] Figures 67 and 68 are diagrams showing configuration examples of a bitstream (encoding attribute information) generated by a three-dimensional data encoding device. For example, as shown in Figure 67, LOSSLESS_FLAG is stored in the header of the bitstream. Alternatively, as shown in Figure 68, a QP value is included in the header of the bitstream. When the QP value is a predetermined value α, lossless encoding is applied.

[0446] (Embodiment 9) In this embodiment, an integer RAHT, which is a lossy transformation closer to lossless than normal RAHT, will be described. In order to facilitate hardware implementation, a fixed point RAHT can be introduced. The fixed point RAHT can be implemented using the following (Equation O22) and (Equation O23). Here, l is a low-frequency component, and h is a high-frequency component. c1 and c2 are attribute information of the 3D point pair at the lowest layer, and at layers other than the lowest layer, they are low-frequency components of the 3D point pair obtained at the layer one level lower. In addition, the transformation is orthonormal, and (Equation O24) holds.

[0447]

number

[0448] Furthermore, the updated weight w is expressed as w=w1+w2 when c1 and c2 are a 3D point pair, and as w=w1 when c1 and c2 are not paired.

[0449] Moreover, the above (Equation O22) is transformed into the following (Equations O25) to (Equations O29).

[0450]

number

[0451] Therefore, the forward transformation is expressed by (Equation O30) to (Equation O32).

[0452]

number

[0453] Moreover, the inverse transformation is expressed by (Equation O33) to (Equation O34).

[0454]

number

[0455] The adjusted quantization step (Aqs) is expressed by (Equation O36) based on (Equation O35). Therefore, (Equation O37) holds. In this way, the integer RAHT is expressed by b 2 This can be achieved by fixed point mounting.

[0456]

number

[0457] The relationship between the integer RAHT and the integer Haar transform is explained below. The integer RAHT and the integer Haar transform can be realized using a common process. Specifically, the integer Haar transform can be realized by setting the weights in all layers in the RAHT to w1 = w2 = 1.

[0458] In other words, the forward transformation in integer RAHT is expressed by (equation O38) to (equation O40), and the inverse transformation is expressed by (equation O41) to (equation O42). In addition, (equation O43) holds.

[0459]

number

[0460] If we set w1 = w2 = 1 in (Equation O38) to (Equation O43), the forward transform is expressed by (Equation O44) to (Equation O45), and the inverse transform is expressed by (Equation O46) to (Equation O47). In other words, the integer Haar transform is realized.

[0461]

number

[0462] Next, an example of switching between lossy coding (RAHT), nearly lossless lossy coding (integer RAHT), and lossless coding (integer Haar transform) will be described. Fig. 69 is a diagram showing the configuration of a three-dimensional data coding device in this case. The three-dimensional data coding device selectively performs lossy coding (RAHT), nearly lossless lossy coding (integer RAHT), and lossless coding (integer Haar transform). This switching is performed based on a flag or a QP value.

[0463] The three-dimensional data encoding device shown in FIG. 69 includes a rearrangement unit 8401, an integer RAHT-Haar transform unit 8402, a quantization unit 8403, and an entropy encoding unit 8404.

[0464] The sorting unit 8401 sorts the attribute information based on the position information, for example, into Morton order. The integer RAHT-Haar transform unit 8402 converts the attribute information by selectively using lossy coding (RAHT), near-lossless lossy coding (integer RAHT), and lossless coding (integer Haar transform) to generate coefficient values.

[0465] Specifically, the three-dimensional data encoding device uses lossless encoding (integer Haar transform) when the value of the quantization parameter QP is a certain value α and RAHT-HAAR_FLAG=HAAR. Here, the value α is, for example, a quantization value calculated from the QP value, or a value that makes the value of the quantization step Qstep 1. For example, when QP=4, and Qstep=1, α=4. Also, the value of α may be different between RAHT and Haar.

[0466] For example, when RAHT-HAAR_FLAG=RAHT and QP is greater than α, lossy coding (RAHT) is used. When RAHT-HAAR_FLAG=RAHT and QP=α, near-lossless lossy coding (integer RAHT) is used. When RAHT-HAAR_FLAG=HAAR and QP=α, lossless coding (integer Haar transform) is used. Also, when RAHT-HAAR_FLAG=HAAR and QP is greater than α, lossy coding (RAHT) may be used.

[0467] Furthermore, when RAHT-HAAR_FLAG=HAAR, the integer RAHT-Haar transform unit 8402 executes the Haar transform by setting w1=w2=1.

[0468] The quantization unit 8403 quantizes the coefficient value using the QP to generate quantized coefficients. The entropy coding unit 8404 entropy codes the quantized coefficients to generate coding attribute information.

[0469] Fig. 70 is a diagram showing the configuration of a three-dimensional data decoding device corresponding to the three-dimensional data encoding device shown in Fig. 69. The three-dimensional data decoding device shown in Fig. 70 includes an entropy decoding unit 8411, an inverse quantization unit 8412, a rearrangement unit 8413, and an inverse integer RAHT-Haar transform unit 8414.

[0470] The entropy decoding unit 8411 generates quantized coefficients by entropy decoding the coding attribute information. The inverse quantization unit 8412 generates coefficient values ​​by inverse quantizing the quantized coefficients using the QP. The sorting unit 8413 sorts the coefficient values ​​based on the position information, for example, in Morton order.

[0471] The inverse integer RAHT-Haar transform unit 8414 generates attribute information by selectively using lossy coding (RAHT), lossy coding close to lossless (integer RAHT), and lossless coding (integer Haar transform) to inversely transform coefficient values. Note that this switching method is the same as that of the integer RAHT-Haar transform unit 8402 described above. Note that the three-dimensional data decoding device acquires the LOSSLESS_FLAG and the QP value from the bitstream.

[0472] Fig. 71 is a diagram showing a configuration example of a bitstream (encoding attribute information) generated by a three-dimensional data encoding device. For example, as shown in Fig. 71, RAHT-HAAR_FLAG and a QP value are stored in the header of the bitstream. RAHT-HAAR_FLAG is a flag indicating whether to use lossy encoding (RAHT), near-lossless lossy encoding (integer RAHT), or lossless encoding (integer Haar transform). Note that RAHT-HAAR_FLAG may indicate whether to use lossless encoding (integer Haar transform), lossy encoding (RAHT), or near-lossless lossy encoding (integer RAHT).

[0473] Below, we will explain an example of the implementation of the configuration for integer RAHT. Integer RAHT can be implemented as follows. B is b 2 It indicates the integer precision of and is expressed by (Equation O48).

[0474]

number

[0475] kBit represents the precision of B. For example, in the case of 8-bit precision, kBit = 8. Also, kHal = (1 << (kBit-1)) represents the precision that supports rounding (such as truncation or rounding off). The adjusted quantization step (Aqs) can be implemented by (Equation O49).

[0476]

number

[0477] Here, QS represents the quantization step. In addition, the forward transform is expressed by (Equation O50) to (Equation O51).

[0478]

number

[0479] The quantized high frequency components are expressed by (Equation O52). Also, the inverse quantization of the high frequency components is expressed by (Equation O53).

[0480]

number

[0481] The inverse transformation is expressed by (Equation O54) to (Equation O55).

[0482]

number

[0483] Below, an example is shown in which, in the implementation of integer RAHT, a conditional flag is used to set the rounding bit precision to 0 and Aqs is set to realize integer Haar in RAHT. When integer Haar is applied, the three-dimensional data encoding device sets the weight to 1 (w1 = w2 = 1). In addition, the three-dimensional data encoding device sets kHalf to 0 (kHalf = 0). In addition, the three-dimensional data encoding device switches Aqs as follows. When using integer Haar transform, the three-dimensional data encoding device sets Aqs = QS. In addition, when using integer RAHT, the three-dimensional data encoding device sets Aqs = sqrt_integer (((QS * QS) * (w1 + w2)) / (w1 * w2)). Here, sqrt_integer (n) indicates the integer part of the square root of n. Therefore, (Equation O56) holds.

[0484]

number

[0485] The integer Haar forward transform in RATH is expressed by (Equation O57) to (Equation O59).

[0486]

number

[0487] When lossless encoding is used, QS is set to 1, so that (Equation O60) holds. Also, when lossless encoding is used, quantization and dequantization may be skipped.

[0488]

number

[0489] The inverse quantization of the high frequency components is expressed by (Equation O61). The inverse integer transform is expressed by (Equation O62) to (Equation O63).

[0490]

number

[0491] As another example of implementation of lossless encoding, the following bit shift operations may be used. These operations are performed on integer data types (fixed-point arithmetic). The attribute information is shifted up with kBit precision before the conversion process.

[0492] When integer Haar is applied, the three-dimensional data encoding device sets the weights to 1 (w1 = w2 = 1). The three-dimensional data encoding device also sets kHalf to 0 (kHalf = 0). The three-dimensional data encoding device also switches Aqs as follows. When integer Haar transform is used, the three-dimensional data encoding device sets Aqs = QS. When integer RAHT is used, the three-dimensional data encoding device sets Aqs = sqrt_integer(((QS * QS) * (w1 + w2)) / (w1 * w2)). Therefore, (Equation O64) holds.

[0493]

number

[0494] As shown in (Equation O65), the attribute information is shifted up to kBit precision before the conversion process.

[0495]

number

[0496] The integer Haar forward transform in RATH is expressed by (Equation O66) to (Equation O67) and is performed with k-bit accuracy.

[0497]

number

[0498] In order to remove the floating-point precision of B for the floor function, the kBit precision of the low-frequency components is removed. Therefore, the low-frequency components are expressed by (Equation O68).

[0499]

number

[0500] The inverse integer transform is expressed by (Equation O69) to (Equation O71). As a result, the changes from the integer RAHT can be reduced.

[0501]

number

[0502] An example of the configuration of the forward transform will be described below. Fig. 72 is a diagram showing an example of the configuration of the integer RAHT-Haar transform unit 8402. The integer RAHT-Haar transform unit 8402 includes left shift units 8421, 8422, and 8430, a subtraction unit 8423, a division unit 8424, right shift units 8425, 8427, and 8429, a multiplication unit 8426, a switching unit 8428, and an addition unit 8431.

[0503] If c1 and c2 are original signals of attribute information (signals located at the lowest layer of RAHT), the left shift units 8421 and 8422 shift c1 and c2 up by kBits (left shift). This increases the bit precision of the original signals, improving the calculation precision during conversion processing. This improves the coding efficiency. Also, if c1 and c2 are signals located above the lowest layer of RAHT, the kBit shift up does not need to be applied.

[0504] In addition, when the three-dimensional data encoding device applies the integer Haar transform and QS=1 (lossless encoding), it is not necessary to apply a k-bit shift-up to the original signal of the attribute information located at the lowest layer of the RAHT. This makes it possible to realize lossless encoding while reducing the amount of processing.

[0505] The subtraction unit 8423 subtracts the shift-up c1 from the shift-up c2. The division unit 8424 divides the value obtained by the subtraction unit 8423 by Aqs. Here, Aqs is expressed by (Equation O72). integer_square_root(n) indicates the integer part of the square root of n. That is, Aqs depends on QS (quantization step) and weight. Also, when the integer Haar transform is used, Aqs=QS is set.

[0506]

number

[0507] A right shift unit 8425 generates a high frequency component h by shifting down the value obtained by the division unit 8424. A multiplication unit 8426 multiplies the value obtained by the subtraction unit 8423 by B. B is expressed by (Equation O73). Furthermore, when the integer Haar transform is used, w1=w2=1 is set.

[0508]

number

[0509] The right shift unit 8427 shifts down the value obtained by the multiplication unit 8426. The switching unit 8428 outputs the value obtained by the right shift unit 8427 to the right shift unit 8429 when the integer Haar transform is used, and outputs the value obtained by the right shift unit 8427 to the addition unit 8431 when the integer Haar transform is not used.

[0510] When the integer Haar transform is applied, the right shift unit 8429 and the left shift unit 8430 shift the value obtained by the right shift unit 8427 down by k bits and up by k bits. This sets the value of the lower k bits to 0. In other words, since it is possible to delete the decimal point values ​​generated by division by the value 2 when the integer Haar transform is applied, it is possible to realize a truncation process (floor process) of the value. Note that any method may be applied as long as it can realize the truncation process of the value.

[0511] In addition, when QS>1 (lossy coding) in the integer Haar transform, the kBit shift-down and kBit shift-up do not need to be applied to the value obtained by the right shift unit 8427. This makes it possible to maintain the precision after the decimal point that occurs due to division by the value 2 when applying the integer Haar transform. This improves the calculation precision and improves the coding efficiency.

[0512] In addition, when the original signal of the attribute information (located at the bottom layer of the RAHT) is not shifted up by kBits, the kBit shift-down and kBit shift-up do not need to be applied to the value obtained by the right shift unit 8427. This can reduce the amount of processing.

[0513] The adder 8431 generates a low-frequency component l by adding the value obtained by the left shifter 8430 or the right shifter 8427 to the value obtained by the left shifter 8421. Note that in the top-layer calculation, the obtained low-frequency component l is shifted down by kBits.

[0514] Next, a configuration example of the inverse transform will be described. Fig. 73 is a diagram showing a configuration example of the inverse integer RAHT-Haar transform unit 8414. The inverse integer RAHT-Haar transform unit 8414 includes left shift units 8441 and 8447, multiplication units 8442, 8443, and 8449, right shift units 8444, 8446, 8450, and 8452, a switching unit 8445, and subtraction units 8448 and 8451.

[0515] The left shift unit 8441 shifts up (left-shifts) the high frequency component h by k bits. The multiplication unit 8442 multiplies the value obtained by the left shift unit 8441 by Aqs. Note that Aqs is the same as in the case of FIG. 72 described above. The multiplication unit 8443 multiplies the value obtained by the multiplication unit 8442 by B. Note that B is the same as in the case of FIG. 72 described above.

[0516] The right shift unit 8444 shifts down the value obtained by the multiplication unit 8443. The switching unit 8445 outputs the value obtained by the right shift unit 8444 to the right shift unit 8446 when the integer Haar transform is used, and outputs the value obtained by the right shift unit 8444 to the subtraction unit 8448 when the integer Haar transform is not used.

[0517] When the integer Haar transform is applied, the right shift unit 8446 and the left shift unit 8447 shift the value obtained by the right shift unit 8444 down by k bits and shift it up by k bits. This sets the value of the lower k bits to 0. In other words, since it is possible to delete the decimal point values ​​generated by division by the value 2 when the integer Haar transform is applied, it is possible to realize a truncation process (floor process) of the value. Note that any method may be applied as long as it can realize the truncation process of the value.

[0518] In addition, when QS>1 (lossy coding) in the integer Haar transform, the kBit shift-down and kBit shift-up do not need to be applied to the value obtained by the right shift unit 8444. This makes it possible to maintain the decimal precision that occurs due to division by the value 2 when applying the integer Haar transform. Therefore, the calculation precision is improved, and a bitstream with improved coding efficiency can be correctly decoded.

[0519] Furthermore, if the decoded signal of the attribute information (located at the bottom layer of the RAHT) is not shifted down by kBits, the kBit shift-down and kBit shift-up do not need to be applied to the value obtained by the right shift unit 8444. This can reduce the amount of processing.

[0520] The subtraction unit 8448 subtracts from the low-frequency component l the value obtained by the left shift unit 8447 or the right shift unit 8444. In the calculation of the top layer, the low-frequency component l is shifted up by k bits, and the subtraction unit 8448 subtracts the value obtained by the left shift unit 8447 or the right shift unit 8444 from the low-frequency component l after the shift up.

[0521] The multiplication unit 8449 multiplies the value obtained by the subtraction unit 8448 by -1. The right shift unit 8450 shifts down the value obtained by the multiplication unit 8449 by kBits. The subtraction unit 8451 subtracts the value obtained by the subtraction unit 8448 from the value obtained by the multiplication unit 8442. The right shift unit 8452 shifts down the value obtained by the subtraction unit 8451 by kBits. This returns the bit precision of c1 and c2 to the original. This process makes it possible to obtain a decoded result with the original bit precision while improving the calculation precision during the conversion process. In addition, the three-dimensional data decoding device does not need to apply kBit shift down to signals higher than the lowest layer of the RAHT.

[0522] In addition, when the three-dimensional data decoding device applies the integer Haar transform and QS=1 (lossless coding), it is not necessary to apply a kBit shift-up to the decoded signal of the attribute information located at the bottom layer of the RAHT. This allows the lossless coded bitstream to be correctly decoded while reducing the amount of processing.

[0523] As described above, the three-dimensional data encoding device according to this embodiment performs the process shown in FIG. 74. The three-dimensional data encoding device converts a plurality of pieces of attribute information of a plurality of three-dimensional points included in point cloud data into a plurality of coefficient values ​​(S8401), and generates a bit stream by encoding the plurality of coefficient values ​​(S8402). In the conversion (S8401), the three-dimensional data encoding device performs a weighting operation hierarchically to separate each of the plurality of pieces of attribute information into high-frequency components and low-frequency components, thereby generating a plurality of coefficient values ​​belonging to any of a plurality of hierarchies. In the weighting operation, the three-dimensional data encoding device performs the weighting operation using a plurality of weights that are fixed or not fixed in a plurality of hierarchies. The bit stream includes first information (at least one of RAHT-HAAR_FLAG and QP) indicating whether or not a plurality of weights are fixed in a plurality of hierarchies.

[0524] According to this, the three-dimensional data encoding device can reduce loss due to conversion by fixing weights in multiple layers, thereby improving accuracy.

[0525] For example, when multiple weights are fixed in multiple hierarchical layers, the multiple weights are set to 1.

[0526] For example, as shown in FIG. 72, in a weighting calculation, a three-dimensional data encoding device calculates a first value by subtracting first attribute information (e.g., c1) included in a plurality of attribute information from second attribute information (e.g., c2) included in the plurality of attribute information, and calculates a high-frequency component (e.g., h) by dividing the first value by a first coefficient (e.g., Aqs), where the first coefficient (e.g., Aqs) depends on the quantization step (e.g., QS) and weights (e.g., w1 and w2).

[0527] For example, as shown in FIG. 72, in a weighting calculation, a three-dimensional data encoding device calculates a second value by multiplying the first value by a second coefficient (e.g., B) that depends on the weight, calculates a third value by shifting the second value down by a predetermined number of bits and then shifting it up by a predetermined number of bits, and calculates a low-frequency component (e.g., l) by adding the third value to the first attribute information.

[0528] For example, the three-dimensional data encoding device includes a processor and a memory, and the processor uses the memory to perform the above-mentioned processing.

[0529] Moreover, the three-dimensional data decoding device according to this embodiment performs the process shown in FIG. 75. The three-dimensional data decoding device acquires first information (at least one of RAHT-HAAR_FLAG and QP) indicating whether or not a plurality of weights are fixed in a plurality of layers from the bit stream (S8411), decodes a plurality of coefficient values ​​from the bit stream (S8412), and generates a plurality of attribute information of a plurality of three-dimensional points in the point cloud data by inverse transforming the plurality of coefficient values ​​(S8413). The plurality of coefficient values ​​belong to any of a plurality of layers. In the inverse transform, the three-dimensional data decoding device generates a plurality of attribute information by performing an inverse weighting operation that combines a high-frequency component and a low-frequency component on the plurality of coefficient values. In the inverse weighting operation, the three-dimensional data decoding device performs a weighting operation using a plurality of weights that are fixed or not fixed in a plurality of layers according to the first information.

[0530] According to this, the three-dimensional data decoding device can reduce loss due to conversion by fixing weights in multiple layers, thereby improving accuracy.

[0531] For example, when multiple weights are fixed in multiple hierarchical layers, the multiple weights are set to 1.

[0532] For example, as shown in FIG. 73, in an inverse weighting operation, a three-dimensional data decoding device calculates a first value by multiplying the high-frequency component by a first coefficient (e.g., Aqs), calculates first attribute information (e.g., c1) included in the multiple attribute information from a second value based on the low-frequency component (e.g., l), and calculates second attribute information (e.g., c2) included in the multiple attribute information by subtracting the second value from the first value, where the first coefficient (e.g., Aqs) depends on the quantization step (e.g., QS) and weights (e.g., w1 and w2).

[0533] For example, as shown in FIG. 73, in a weighting calculation, a three-dimensional data decoding device calculates a third value by multiplying the first value by a second coefficient (e.g., B) that depends on the weight, calculates a fourth value by shifting the third value down by a predetermined number of bits and then shifting it up by a predetermined number of bits, and calculates the second value by subtracting the low-frequency component from the fourth value.

[0534] For example, the three-dimensional data decoding device includes a processor and a memory, and the processor uses the memory to perform the above-mentioned processing.

[0535] (Embodiment 10) In this embodiment, a RAHT (Region Adaptive Hierarchical Transform) or Haar transform process using prediction will be described. FIG. 76 is a diagram for explaining the prediction process, showing a hierarchical structure in the RAHT or Haar transform process. The three-dimensional data encoding device may predict an attribute value (attribute information) of a lower layer from an attribute value of a higher layer during hierarchical encoding by RAHT or Haar transform, and encode a difference value between the attribute value and a predicted value obtained by prediction. For example, when encoding an attribute value of each layer by repeating hierarchical encoding from a higher layer to a lower layer, the three-dimensional data encoding device predicts an attribute value of a lower layer (attribute value of a child node group) using an attribute value (attribute value of a parent node group) calculated in the higher layer. The three-dimensional data encoding device may encode a difference value obtained by subtracting a predicted value from the attribute value of each child node, instead of the attribute value of each child node. In this way, the difference value of the attribute value of the lower layer can be reduced by generating an appropriate predicted value from the attribute value of the higher layer, and the encoding efficiency can be improved. Note that a similar prediction process may be performed in the three-dimensional data decoding device.

[0536] FIG. 77 is a diagram showing an example of the relationship between nodes in an octree structure based on position information of three-dimensional points. As shown in FIG. 77, the three-dimensional data encoding device predicts an attribute value of a child node from an attribute value of a parent node or a neighboring parent node. Here, a plurality of nodes in the same hierarchy as a parent node are called a parent node group. A plurality of child nodes of a parent node are called a child node group. A neighboring parent node is a node different from a parent node included in a parent node group and is adjacent to a parent node. Note that a parent node group may include some of the plurality of nodes in the same hierarchy as a parent node. For example, a parent node group may include a parent node and a plurality of neighboring parent nodes. Or, a parent node group may include a node within a predetermined distance from a parent node (or a target node).

[0537] Furthermore, the attribute information of a higher-level node such as a parent node is calculated, for example, from the attribute information of the lower-level node. For example, the attribute information of a parent node is the average value of the attribute information of the child nodes of the parent node. Note that the method of calculating the attribute information of the higher-level node is not limited to the average, and other calculation methods such as a weighted average may be used.

[0538] For example, the three-dimensional data encoding device may calculate the distance d in three-dimensional space between a target node, which is a child node to be encoded, and a parent node or a neighboring parent node, and use a weighted average value using the distance d as a predicted value. For example, the predicted value cp of a child node c may be calculated using the following (Equation P1) and (Equation P2). Note that Ai is the value of the attribute information of the parent node pi, and d(c, pi) is the distance between the child node c and the parent node pi, for example, the Euclidean distance. n is the total number of parent nodes and neighboring parent nodes used to generate the predicted value.

[0539]

number

[0540] Furthermore, the three-dimensional data encoding device may use the attribute value of the parent node or the neighboring parent node as the predicted value as is. For example, the three-dimensional data encoding device may use the attribute value Ap of the parent node as the predicted value of the child node, or may use the attribute value Anp of the neighboring parent node as the predicted value of the child node. Furthermore, the three-dimensional data encoding device may select whether to use a calculated value (e.g., a weighted average value) calculated from attribute information of multiple nodes included in the parent node group, or to use the attribute value of the parent node or the neighboring parent node as is.

[0541] At this time, the three-dimensional data encoding device may add information (prediction mode) indicating which of the calculated value and the attribute value of the parent node or the neighboring parent node is used as the predicted value, for example, to the bit stream for each child node group. This allows the three-dimensional data encoding device to select an appropriate prediction mode for each child node group, thereby improving encoding efficiency. Also, by adding the prediction mode to the bit stream, the three-dimensional data decoding device can generate a predicted value using the prediction mode selected by the three-dimensional data encoding device. Therefore, the three-dimensional data decoding device can appropriately decode the bit stream.

[0542] In addition, the prediction mode may not be added to each child node group, but may be added to a unit larger than the child node group, or a unit smaller than the child node group. For example, the three-dimensional data encoding device can improve the encoding efficiency while suppressing the overhead of encoding the prediction mode by adding a prediction mode to each N child node group (N is an integer equal to or greater than 1). In addition, the three-dimensional data encoding device may add the prediction mode to a header such as an APS. Here, the APS is a parameter set of attribute information for each frame.

[0543] Next, a first example of a coding method using prediction will be described below. Figure 78 is a diagram showing the first example of the coding method.

[0544] When calculating the difference between an attribute value and a predicted value, the three-dimensional data encoding device calculates a transform coefficient of the attribute value and a transform coefficient of the predicted value by applying RAHT or Haar transform to each of the attribute value and the predicted value. The three-dimensional data encoding device obtains the difference between the transform coefficient of the attribute value and the transform coefficient of the predicted value. This makes it possible to reduce the difference value to be encoded, thereby improving encoding efficiency.

[0545] In addition, when the three-dimensional data encoding device selects a predicted value from the calculation value and the attribute value of the parent node or the neighboring parent node (when adding a prediction mode to a bit stream), for example, the prediction value of each prediction mode is used to calculate a difference value of the transform coefficient, and the absolute value sum of the difference values ​​is used to obtain a cost value, and the prediction mode that minimizes the cost may be selected. This allows the prediction mode that minimizes the difference value to be appropriately selected, and the encoding efficiency to be improved. For example, the three-dimensional data encoding device may calculate the cost value cost using the following (Equation P3).

[0546]

number

[0547] Here, m is the number of child nodes included in the child node group. λ is an adjustment parameter. Predbit is the amount of bits for encoding the prediction mode. Ti is the transformation coefficient of the attribute value, and PTi is the transformation coefficient of the predicted value. Note that the method of selecting the prediction mode is not limited to this, and the prediction mode may be selected based on other conditions or external instructions.

[0548] As shown in Fig. 78, the three-dimensional data encoding device applies RAHT or Haar transform to the attribute values ​​of the child node group to calculate the transform coefficients Ti of the attribute values ​​(S9101). The three-dimensional data encoding device also generates predicted values ​​of the child node group by predicting the attribute values ​​of the child node group from the attribute values ​​of the parent node group (S9102). Next, the three-dimensional data encoding device applies RAHT or Haar transform to the predicted values ​​to calculate the transform coefficients PTi of the predicted values ​​(S9103).

[0549] Next, the three-dimensional data encoding device calculates a difference value that is a difference between the transform coefficient Ti of the attribute value and the transform coefficient PTi of the predicted value (S9104). Next, the three-dimensional data encoding device quantizes the difference value (S9105) and arithmetically codes the quantized difference value (S9106) to generate coded data (bit stream). Note that, when lossless coding is used, the three-dimensional data encoding device may skip the quantization (S9105).

[0550] Next, a first example of a decoding method for decoding the coded data (bit stream) generated by the above-mentioned first example of the coding method will be described. Fig. 79 is a diagram showing the first example of the decoding method.

[0551] First, the three-dimensional data decoding device arithmetically decodes the encoded data (bit stream) (S9111), and inversely quantizes the resulting signal (S9112) to generate differential values ​​of the transform coefficients of the child nodes. Note that, when using lossless decoding (when lossless encoding is used), the three-dimensional data decoding device may skip inverse quantization (S9112).

[0552] The three-dimensional data decoding device also generates a predicted value by predicting the attribute values ​​of the child nodes from the attribute values ​​of the parent nodes (S9113). In addition, in the three-dimensional data encoding device, when it is selected to use either the calculated value or the attribute value of the parent node or the neighboring parent node for generating the predicted value (when the prediction mode is added to the bit stream), the three-dimensional data decoding device generates the predicted value using the decoded prediction mode. Next, the three-dimensional data decoding device calculates the transform coefficient PTi of the predicted value by applying the RAHT or Haar transform to the predicted value (S9114).

[0553] Next, the three-dimensional data decoding device calculates the transform coefficients Ti of the child nodes by adding the transform coefficients PTi of the predicted values ​​to the difference values ​​of the transform coefficients of the child nodes (S9115). Next, the three-dimensional data decoding device generates decoded values ​​of the attribute values ​​of the child nodes by applying inverse RAHT or inverse Haar transform to the transform coefficients Ti of the child nodes (S9116). In this way, the three-dimensional data decoding device can appropriately decode the bit stream.

[0554] Next, a second example of the encoding method will be described. In this second example, an integer Haar transform is used instead of the RATH or Haar transform. Figure 80 is a diagram showing the second example of the encoding method.

[0555] When calculating the difference between the attribute value and the predicted value, the three-dimensional data encoding device may apply an integer Haar transform to each of the attribute value and the predicted value to calculate a transform coefficient of the attribute value and a transform coefficient of the predicted value, and obtain the difference between the transform coefficient of the attribute value and the transform coefficient of the predicted value. This allows the three-dimensional data encoding device to reduce the difference value to be encoded, thereby improving encoding efficiency.

[0556] In addition, when the three-dimensional data encoding device selects a predicted value from the calculated value and the attribute value of the parent node or the neighboring parent node (when adding a prediction mode to a bit stream), for example, the prediction value of each prediction mode may be used to calculate a difference value of the transform coefficient, and the absolute sum of the difference values ​​may be used to obtain a cost value, and the prediction mode that minimizes the cost may be selected. This allows the three-dimensional data encoding device to appropriately select a prediction mode that minimizes the difference value, thereby improving the encoding efficiency. For example, the three-dimensional data encoding device may calculate the cost value cost using the above (Equation P3).

[0557] 80, the three-dimensional data encoding device calculates a transform coefficient Ti of the attribute value by applying an integer Haar transform to the attribute value of the child node group (S9101A). In addition, the three-dimensional data encoding device generates a predicted value of the child node group by predicting the attribute value of the child node group from the attribute value of the parent node group (S9102).

[0558] When applying the integer Haar transform, the three-dimensional data encoding device truncates the value after the decimal point of the predicted value (S9107). Alternatively, the three-dimensional data encoding device may apply rounding or the like to the predicted value and set the value after the decimal point of the predicted value to 0. For example, when kBit represents the decimal part of the bits of the attribute value, the three-dimensional data encoding device applies kBit shift down and kBit shift up to the predicted value, thereby performing a value truncation process (floor process) and setting the value of the decimal part of kBit to 0. In this way, by making the predicted value an integer value before applying the integer Haar transform, the transform coefficient of the predicted value after the integer Haar transform also becomes an integer value. As a result, the difference value between the transform coefficient Ti of the attribute value of the child node group and the transform coefficient PTi of the predicted value also becomes an integer value. Therefore, lossless encoding using the integer Haar transform can be realized. Note that any method may be applied as long as it is a method for setting the value of the decimal part of the predicted value to 0.

[0559] Furthermore, the three-dimensional data encoding device stores a flag in the bit stream indicating whether or not to apply the integer Haar transform, and when applying the integer Haar transform, sets the decimal part of the predicted value to 0, and when not applying the integer Haar transform, does not need to set the decimal part to 0. Note that, when QS>1 (lossy encoding) in the integer Haar transform, the three-dimensional data encoding device does not need to apply the process of setting the decimal part of the predicted value to 0. This improves calculation accuracy by maintaining the decimal precision of the predicted value when applying the integer Haar transform. This improves encoding efficiency.

[0560] Next, the three-dimensional data encoding device calculates a transform coefficient PTi of the predicted value by applying an integer Haar transform to the predicted value after the truncation process (S9103A). Next, the three-dimensional data encoding device calculates a difference value that is a difference between the transform coefficient Ti of the attribute value and the transform coefficient PTi of the predicted value (S9104).

[0561] Here, the differential values ​​of the transform coefficients may be converted to integer values ​​and then arithmetically coded. In this case, information in the decimal part is lost. Therefore, in order to support lossless coding, the three-dimensional data coding device sets the value of the decimal part of the differential values ​​of the transform coefficients to a value of 0 in advance. This allows for lossless coding without losing information even when the differential values ​​are converted to integer values ​​before arithmetic coding.

[0562] Next, the three-dimensional data encoding device quantizes the difference value (S9105) and arithmetically encodes the quantized difference value (S9106) to generate encoded data (bit stream). Note that, when lossless encoding is used, the three-dimensional data encoding device may skip the quantization (S9105).

[0563] Next, a second example of a decoding method for decoding the coded data (bit stream) generated by the second example of the coding method will be described. Fig. 81 is a diagram showing the second example of the decoding method.

[0564] First, the three-dimensional data decoding device arithmetically decodes the encoded data (bit stream) (S9111), and inversely quantizes the resulting signal (S9112) to generate differential values ​​of the transform coefficients of the child nodes. Note that, when using lossless decoding (when lossless encoding is used), the three-dimensional data decoding device may skip inverse quantization (S9112).

[0565] The three-dimensional data decoding device also generates a predicted value by predicting the attribute values ​​of the child nodes from the attribute values ​​of the parent nodes (S9113). Note that in the three-dimensional data encoding device, when the use of either the calculated value or the attribute value of the parent node or the neighboring parent node is selected for generating the predicted value (when the prediction mode is added to the bit stream), the three-dimensional data decoding device generates the predicted value using the decoded prediction mode.

[0566] When applying the integer Haar transform, the three-dimensional data decoding device truncates the value after the decimal point of the predicted value (S9117). Alternatively, the three-dimensional data decoding device may apply rounding or the like to the predicted value and set the value after the decimal point of the predicted value to 0. For example, when kBit represents a decimal part of the bits of the attribute value, the three-dimensional data decoding device may set the value of the decimal part of kBit to 0 by performing a value truncation process (floor process) by applying kBit shift down and kBit shift up to the predicted value. In this way, by making the predicted value an integer value before applying the integer Haar transform, the transform coefficient of the predicted value after the integer Haar transform also becomes an integer value. As a result, the sum of the transform coefficient Ti of the attribute value of the child node group and the transform coefficient PTi of the predicted value also becomes an integer value. Therefore, the three-dimensional data decoding device can appropriately decode the bit stream of lossless coding using the integer Haar transform. Note that any method may be applied as long as it is a method for setting the value of the decimal part of the predicted value to 0.

[0567] Furthermore, the three-dimensional data decoding device obtains a flag indicating whether or not to apply the integer Haar transform from the bitstream, and when applying the integer Haar transform, sets the decimal part of the predicted value to 0, and when not applying the integer Haar transform, does not have to set the decimal part to 0. Note that, when QS>1 (lossy decoding) in the integer Haar transform, the three-dimensional data decoding device does not have to apply the process of setting the decimal part of the predicted value to 0. In this way, the three-dimensional data decoding device maintains the precision after the decimal point of the predicted value when applying the integer Haar transform, thereby improving calculation accuracy and enabling the three-dimensional data decoding device to appropriately decode a bitstream with improved coding efficiency.

[0568] Next, the three-dimensional data decoding device calculates a transform coefficient PTi of the predicted value by applying an integer Haar transform to the predicted value after the truncation process (S9114A). Next, the three-dimensional data decoding device calculates a transform coefficient Ti of the child node group by adding the transform coefficient PTi of the predicted value to a difference value of the transform coefficients of the child node group (S9115). Next, the three-dimensional data decoding device generates a decoded value of the attribute value of the child node group by applying an inverse integer Haar transform to the transform coefficient Ti of the child node group (S9116A). In this way, the three-dimensional data decoding device can appropriately decode the bit stream.

[0569] Next, a third example of the encoding method will be described. In this third example, the integer Haar transform is used as in the second example. Also, this example differs from the second example in that the timing of the truncation process (S9108) is after application of the integer Haar transform (S9103A).

[0570] 82 is a diagram showing a third example of an encoding method. Note that the following mainly describes the differences from the second example, and overlapping descriptions will be omitted.

[0571] 82, the three-dimensional data encoding device calculates a transform coefficient Ti of the attribute value by applying an integer Haar transform to the attribute value of the child node group (S9101A). In addition, the three-dimensional data encoding device generates a predicted value of the child node group by predicting the attribute value of the child node group from the attribute value of the parent node group (S9102).

[0572] Next, the three-dimensional data encoding device calculates the transform coefficients PTi of the predicted values ​​by applying an integer Haar transform to the predicted values ​​(S9103A). Next, the three-dimensional data encoding device performs a rounding process on the transform coefficients PTi to set the decimal parts of the transform coefficients to 0 (S9108). Note that the details of this rounding process are the same as those of step S9107 described above, although they differ depending on whether the signal to be processed is a predicted value or a transform coefficient PTi.

[0573] Next, the three-dimensional data encoding device calculates a difference value which is the difference between the transformation coefficient Ti of the attribute value and the transformation coefficient PTi of the predicted value after the truncation process (S9104).

[0574] Next, a third example of a decoding method for decoding the coded data (bit stream) generated by the third example of the coding method will be described. Figure 83 is a diagram showing the third example of the decoding method.

[0575] First, the three-dimensional data decoding device arithmetically decodes the encoded data (bit stream) (S9111), and inversely quantizes the resulting signal (S9112) to generate differential values ​​of the transform coefficients of the child nodes. Note that, when using lossless decoding (when lossless encoding is used), the three-dimensional data decoding device may skip inverse quantization (S9112).

[0576] Furthermore, the three-dimensional data decoding device generates predicted values ​​by predicting the attribute values ​​of the child node group from the attribute values ​​of the parent node group (S9113).

[0577] Next, the three-dimensional data decoding device calculates the transform coefficients PTi of the predicted values ​​by applying an integer Haar transform to the predicted values ​​(S9114A). Next, the three-dimensional data decoding device performs a rounding process on the transform coefficients PTi to set the decimal parts of the transform coefficients to 0 (S9118). Note that the details of this rounding process are the same as those of step S9117 described above, although they differ depending on whether the signal to be processed is a predicted value or a transform coefficient PTi.

[0578] Next, the three-dimensional data decoding device calculates the transform coefficients Ti of the child nodes by adding the transform coefficients PTi of the predicted values ​​after the truncation process to the difference values ​​of the transform coefficients of the child nodes (S9115). Next, the three-dimensional data decoding device generates decoded values ​​of the attribute values ​​of the child nodes by applying an inverse integer Haar transform to the transform coefficients Ti of the child nodes (S9116A). In this way, the three-dimensional data decoding device can appropriately decode the bit stream.

[0579] Next, a fourth example of the encoding method will be described. In the first to third examples, a transform process is applied to an attribute value and a predicted value, and a difference value between a transform coefficient of the generated attribute value and a transform coefficient of the predicted value is calculated. In the fourth example, a three-dimensional data encoding device calculates a difference value between an attribute value and a predicted value, and calculates a transform coefficient of the difference value by applying a RAHT or Haar transform to the difference value. This allows the three-dimensional data encoding device to reduce the transform coefficient of the difference value to be encoded while suppressing the amount of processing, thereby improving encoding efficiency.

[0580] In addition, when the three-dimensional data encoding device selects a predicted value from the calculated value and the attribute value of the parent node or the neighboring parent node (when adding a prediction mode to a bit stream), for example, the predicted value of each prediction mode may be used to calculate a difference value of the attribute value, and the absolute sum of the difference values ​​may be used to obtain the following cost value, and the prediction mode that minimizes the cost may be selected. This allows the three-dimensional data encoding device to appropriately select the prediction mode that minimizes the difference value, thereby improving the encoding efficiency. For example, the three-dimensional data encoding device may calculate the cost value cost using the following (Equation P4).

[0581]

number

[0582] Here, m is the number of child nodes included in the child node group, λ is an adjustment parameter, Predbit is the amount of bits for encoding the prediction mode, and Di is the difference between the attribute value and the predicted value.

[0583] The three-dimensional data encoding device may calculate the cost value cost using the following (Equation P5). This allows the three-dimensional data encoding device to select a prediction mode that reduces the value of the transform coefficient of the difference value, thereby improving the encoding efficiency.

[0584]

number

[0585] Here, Ti is a transform coefficient of the difference value. Note that the method of selecting a prediction mode is not limited to this, and a prediction mode may be selected based on other conditions or an external instruction, or the like.

[0586] Fig. 84 is a diagram showing a fourth example of the encoding method. As shown in Fig. 84, the three-dimensional data encoding device generates a predicted value Pi by predicting an attribute value of a child node group from an attribute value of a parent node group (S9121). Next, the three-dimensional data encoding device calculates a difference value Di that is the difference between the attribute value Ai of the child node group and the predicted value Pi (S9122). Next, the three-dimensional data encoding device calculates a transform coefficient Ti of the difference value by applying RAHT or Haar transform to the difference value Di (S9123).

[0587] Next, the three-dimensional data encoding device quantizes the transform coefficients Ti of the difference values ​​(S9124), and arithmetically encodes the quantized transform coefficients Ti (S9125) to generate encoded data (bit stream). Note that, when using lossless encoding, the three-dimensional data encoding device may skip the quantization (S9124).

[0588] Next, a fourth example of a decoding method for decoding the coded data (bit stream) generated by the fourth example of the coding method will be described. Figure 85 is a diagram showing the fourth example of the decoding method.

[0589] First, the three-dimensional data decoding device arithmetically decodes the encoded data (bit stream) (S9131), and inversely quantizes the obtained signal (S9132) to generate a transform coefficient Ti of a difference value of a child node group. Note that the three-dimensional data decoding device may skip the inverse quantization (S9132) when using lossless decoding (when lossless encoding is used). Next, the three-dimensional data decoding device applies an inverse RAHT or inverse Haar transform to the transform coefficient Ti to generate a difference value Di of an attribute value (S9133).

[0590] The three-dimensional data decoding device also generates a predicted value by predicting the attribute values ​​of the child nodes from the attribute values ​​of the parent nodes (S9134). Note that in the three-dimensional data encoding device, when a calculation value or an attribute value of a parent node or a neighboring parent node is selected to be used for generating a predicted value (when a prediction mode is added to the bit stream), the three-dimensional data decoding device generates a predicted value using the decoded prediction mode.

[0591] Next, the three-dimensional data decoding device generates a decoded value of the attribute value of the child node group by adding the difference value Di and the predicted value Pi (S9135). In this way, the three-dimensional data decoding device can appropriately decode the bit stream.

[0592] Next, a fifth example of the encoding method will be described. This fifth example is different from the fourth example in that an integer Haar transform is used instead of the RATH or Haar transform. Figure 86 is a diagram showing the fifth example of the encoding method.

[0593] The three-dimensional data encoding device generates a predicted value Pi by predicting the attribute values ​​of the child node group from the attribute values ​​of the parent node group (S9121). Next, the three-dimensional data encoding device performs a rounding process on the predicted value Pi to set the decimal part of the predicted value to 0 (S9126). Note that the details of this process are the same as, for example, step S9107 shown in FIG. 80.

[0594] Next, the three-dimensional data encoding device calculates a difference value Di, which is the difference between the attribute value Ai of the child node group and the predicted value Pi after the truncation process (S9122). Next, the three-dimensional data encoding device calculates a transform coefficient Ti of the difference value by applying an integer Haar transform to the difference value Di (S9123A).

[0595] Next, the three-dimensional data encoding device quantizes the transform coefficients Ti of the difference values ​​(S9124), and arithmetically encodes the quantized transform coefficients Ti (S9125) to generate encoded data (bit stream). Note that, when using lossless encoding, the three-dimensional data encoding device may skip the quantization (S9124).

[0596] Here, the transform coefficient of the difference value may be converted to an integer value and then arithmetically coded, in which case the information of the decimal part of the transform coefficient is lost. In response to this, the three-dimensional data coding device sets the value of the decimal part of the transform coefficient of the difference value to 0 in advance. This allows information to be lost even if the transform coefficient Ti of the difference value is converted to an integer value before arithmetic coding, and thus allows lossless coding to be realized.

[0597] Next, a fifth example of a decoding method for decoding the coded data (bit stream) generated by the fifth example of the coding method will be described. Fig. 87 is a diagram showing the fifth example of the decoding method.

[0598] First, the three-dimensional data decoding device arithmetically decodes the coded data (bit stream) (S9131), and inversely quantizes the obtained signal (S9132) to generate a transform coefficient Ti of the difference value of the child node group. Note that the three-dimensional data decoding device may skip the inverse quantization (S9132) when using lossless decoding (when lossless coding is used). Next, the three-dimensional data decoding device applies an inverse integer Haar transform to the transform coefficient Ti to generate a difference value Di of the attribute value (S9133A).

[0599] The three-dimensional data decoding device also generates a predicted value by predicting the attribute values ​​of the child node group from the attribute values ​​of the parent node group (S9134). Next, the three-dimensional data decoding device performs a rounding process on the predicted value Pi to set the decimal part of the predicted value to 0 (S9136). Note that the details of this process are the same as, for example, step S9117 shown in FIG. 81.

[0600] Next, the three-dimensional data decoding device generates a decoded value of the attribute value of the child node group by adding the difference value Di and the predicted value Pi after the truncation process (S9135). In this way, the three-dimensional data decoding device can appropriately decode the bit stream.

[0601] As described above, the three-dimensional data encoding device according to this embodiment performs the process shown in FIG. 88. First, the three-dimensional data encoding device generates a predicted value of attribute information of a target node in an N-ary tree structure (e.g., an 8-ary tree structure) of a plurality of three-dimensional points included in point cloud data (S9141). Next, the three-dimensional data encoding device encodes the attribute information of the target node using the predicted value and a transform process (e.g., RAHT, Haar transform, or integer Haar transform) that hierarchically repeats an operation of dividing each of a plurality of input signals into high-frequency components and low-frequency components (S9142). In generating the predicted value (S9141), the three-dimensional data encoding device selects which of a plurality of first nodes (e.g., parent node group) that include the parent node of the target node and belong to the same layer as the parent node, the attribute information of which node is to be used for generating the predicted value of the target node.

[0602] This allows the three-dimensional data encoding device to appropriately select attribute information used to generate a predicted value, thereby improving encoding efficiency.

[0603] For example, the three-dimensional data encoding device selects whether to use attribute information of a second node included in the plurality of first nodes as a predicted value as is, or to calculate a predicted value from a plurality of attribute information of the plurality of first nodes. For example, the second node is a parent node.

[0604] For example, the three-dimensional data encoding device generates a plurality of predicted values ​​of a plurality of third nodes (e.g., a group of child nodes) that include a target node and belong to the same layer as the target node. In the encoding (S9142), the three-dimensional data encoding device generates a plurality of first transform coefficients by performing a conversion process on a plurality of attribute information of a plurality of third nodes (e.g., S9101 in FIG. 78), generates a plurality of second transform coefficients by performing a conversion process on a plurality of predicted values ​​of a plurality of third nodes (e.g., S9103), calculates a difference value between the corresponding first transform coefficient and second transform coefficient (e.g., S9104), and encodes the calculated difference value (e.g., S9106).

[0605] For example, the transform process is an integer-integer transform (e.g., an integer Haar transform). In generating a plurality of second transform coefficients, the three-dimensional data encoding device performs a process of truncating the decimal parts of a plurality of predicted values ​​of a plurality of third nodes (e.g., S9107 in FIG. 80), and performs a transform process on the plurality of predicted values ​​after the process to generate a plurality of second transform coefficients (e.g., S9103A).

[0606] For example, the transform process is an integer-integer transform (e.g., an integer Haar transform). In calculating the difference value, the three-dimensional data encoding device performs a process of truncating the decimal part of the second transform coefficients (e.g., S9108 in FIG. 82), and calculates the difference value using the second transform coefficients after the process (S9104).

[0607] For example, the three-dimensional data encoding device includes a processor and a memory, and the processor uses the memory to perform the above-mentioned processing.

[0608] Also, the three-dimensional data decoding device according to this embodiment performs the process shown in FIG. 89. First, the three-dimensional data decoding device acquires a difference value between the attribute information of the target node in the N (N is an integer equal to or greater than 2)-ary tree structure (e.g., 8-ary tree structure) of the multiple three-dimensional points included in the point cloud data from the bit stream and the predicted value (S9151), generates a predicted value (S9152), and decodes the attribute information of the target node using the difference value, the predicted value, and an inverse transform process (e.g., RAHT, Haar transform, or integer Haar transform) that hierarchically repeats an operation of dividing each of the multiple input signals into high-frequency components and low-frequency components (S9153). In generating the predicted value (S9152), the three-dimensional data decoding device selects which of the multiple first nodes (e.g., parent node group) that include the parent node of the target node and belong to the same layer as the parent node, the attribute information of which node is to be used for generating the predicted value of the target node.

[0609] This allows the three-dimensional data decoding device to appropriately select attribute information used to generate a predicted value, thereby improving coding efficiency.

[0610] For example, the three-dimensional data decoding device selects whether to use attribute information of a second node included in the plurality of first nodes as a predicted value as is, or to calculate a predicted value from a plurality of attribute information of the plurality of first nodes. For example, the second node is a parent node.

[0611] For example, the three-dimensional data decoding device obtains a plurality of difference values ​​of a plurality of third nodes (e.g., a group of child nodes) that include a target node and belong to the same layer as the target node from the bit stream (e.g., S9111 in FIG. 79), and generates a plurality of predicted values ​​of the plurality of third nodes (S9113). In the decoding (S9153), the three-dimensional data decoding device performs a conversion process on the plurality of predicted values ​​of the plurality of third nodes to generate a plurality of second conversion coefficients (e.g., S9114), generates a plurality of first conversion coefficients by adding the corresponding difference values ​​and second conversion coefficients to the plurality of difference values ​​and the plurality of second conversion coefficients (e.g., S9115), and generates a plurality of attribute information of the plurality of third nodes by performing an inverse conversion process on the first conversion coefficients (e.g., S9116).

[0612] For example, the transform process is an integer-integer transform (e.g., an integer Haar transform). In generating the second transform coefficients, the three-dimensional data decoding device performs a process of truncating the decimal parts of the prediction values ​​of the third nodes (e.g., S9117 in FIG. 81), and performs a transform process on the prediction values ​​after the process to generate the second transform coefficients (e.g., S9114A).

[0613] For example, the transform process is an integer-integer transform (for example, an integer Haar transform). In generating a plurality of first transform coefficients, the three-dimensional data decoding device performs a process of truncating a fractional part of a plurality of second transform coefficients (for example, S9118 in FIG. 83), and calculates a plurality of second transform coefficients using the plurality of second transform coefficients after the process (for example, S9115).

[0614] For example, the three-dimensional data decoding device includes a processor and a memory, and the processor uses the memory to perform the above-mentioned processing.

[0615] (Embodiment 11) In this embodiment, RAHT (Region Adaptive Hierarchical Transform) or Haar transform processing using prediction will be described. Fig. 90 is a diagram for explaining the prediction processing, and is a diagram showing a hierarchical structure in RAHT or Haar transform processing.

[0616] The three-dimensional data encoding device performs hierarchical prediction encoding in which, during hierarchical encoding by RAHT or Haar transform, an attribute value (attribute information) of a lower layer is predicted from an attribute value of a higher layer, and a difference value between the attribute information and a predicted value obtained by the prediction is encoded. In addition, the three-dimensional data encoding device adaptively switches whether or not to perform this hierarchical prediction encoding based on a certain condition. For example, the following condition may be used as the certain condition.

[0617] Condition 1 is that the number of effective nodes is greater than a predetermined threshold (THnode). When the number of effective nodes is greater than the threshold, the three-dimensional data encoding device applies hierarchical predictive encoding, and when the number of effective nodes is equal to or less than the threshold, the three-dimensional data encoding device does not apply hierarchical predictive encoding.

[0618] Here, a valid node is a node having an attribute value to be used for prediction among multiple nodes (parent node and neighboring parent node) included in the parent node group of the child node group to be coded. In other words, a valid node is a node that includes a three-dimensional point in that node or in a node lower than that node (descendant node). The child node group includes multiple nodes (child nodes) to be coded. The parent node group also includes a parent node and multiple neighboring parent nodes. A neighboring parent node is a node that belongs to the same hierarchical level as the parent node and is adjacent to the parent node.

[0619] For example, in the example shown in FIG. 90, the number of effective nodes, which is the total number of effective nodes included in the parent node group, is 11. For example, when THnode=5, the number of effective nodes (=11)>THnode, so the three-dimensional data encoding device encodes the child node group using hierarchical predictive encoding. When THnode=12, the number of effective nodes (=11)<=THnode, so the three-dimensional data encoding device does not encode the child node group using hierarchical predictive encoding. As a result, when the number of effective nodes is greater than the threshold, the three-dimensional data encoding device can generate a highly accurate predicted value using the attribute value of the parent node or a neighboring parent node, so that the encoding efficiency can be improved by applying hierarchical predictive encoding. Furthermore, when the number of effective nodes is equal to or less than the threshold, the three-dimensional data encoding device can reduce the amount of processing by not applying hierarchical predictive encoding.

[0620] When hierarchical predictive coding is not applied, the three-dimensional data coding device applies, for example, RAHT or Haar transform to the attribute values ​​of the child nodes, and entropy codes the obtained transform coefficients.

[0621] Fig. 91 is a diagram showing a first example of the encoding process. When calculating a difference value between an attribute value and a predicted value, the three-dimensional data encoding device calculates a transform coefficient of the attribute value and a transform coefficient of the predicted value by applying RAHT or Haar transform to each of the attribute value and the predicted value. The three-dimensional data encoding device obtains a difference value between the transform coefficient of the attribute value and the transform coefficient of the predicted value. This makes it possible to reduce the difference value to be encoded, thereby improving encoding efficiency.

[0622] In addition, when the three-dimensional data encoding device selects a predicted value from the calculation value and the attribute value of the parent node or the neighboring parent node (when adding a prediction mode to a bitstream), for example, the prediction value of each prediction mode may be used to calculate a difference value of the transform coefficient, and the absolute sum of the difference values ​​may be used to obtain a cost value, and the prediction mode that minimizes the cost value may be selected. This allows the prediction mode that minimizes the difference value to be appropriately selected, and the encoding efficiency to be improved. For example, the three-dimensional data encoding device may calculate the cost value cost using the following (Equation R1).

[0623]

number

[0624] Here, m is the number of child nodes included in the child node group. λ is an adjustment parameter. Predbit is the amount of bits for encoding the prediction mode. Ti is the transformation coefficient of the attribute value, and PTi is the transformation coefficient of the predicted value. Note that the method of selecting the prediction mode is not limited to this, and the prediction mode may be selected based on other conditions or external instructions.

[0625] As shown in FIG. 91, the three-dimensional data encoding device calculates the transformation coefficients Ti of the attribute values ​​by applying the RAHT or Haar transformation to the attribute values ​​of the child node group (S9501).

[0626] In the example shown in Fig. 91, the number of effective nodes is 2, and the number of effective nodes (=2) <= THnode (for example, 11). In this case, the three-dimensional data encoding device does not generate a predicted value. For example, the three-dimensional data encoding device uses a predicted value = 0.

[0627] Next, the three-dimensional data encoding device calculates a difference value (S9502) which is the difference between the transformation coefficient Ti of the attribute value and the predicted value = 0. In other words, the three-dimensional data encoding device outputs the transformation coefficient Ti as it is as the difference value.

[0628] Next, the three-dimensional data encoding device quantizes the difference value (transform coefficient Ti) (S9503), and arithmetically encodes the quantized difference value (S9504) to generate encoded data (bit stream). Note that, when lossless encoding is used, the three-dimensional data encoding device may skip the quantization (S9504).

[0629] If the number of effective nodes is greater than THnode, the three-dimensional data encoding device generates predicted values ​​of the child node group by predicting the attribute values ​​of the child node group from the attribute values ​​of the parent node group. Next, the three-dimensional data encoding device calculates a transform coefficient PTi of the predicted value by applying RAHT or Haar transform to the predicted value. In addition, in step S9502, the three-dimensional data encoding device calculates a difference value that is the difference between the transform coefficient Ti of the attribute value and the transform coefficient PTi of the predicted value.

[0630] Next, an example of a decoding process for decoding the coded data (bit stream) generated by the first example of the coding process will be described. Fig. 92 is a diagram showing the first example of the decoding process.

[0631] First, the three-dimensional data decoding device arithmetically decodes the encoded data (bit stream) (S9511), and inversely quantizes the resulting signal (S9512) to generate differential values ​​of the transform coefficients of the child nodes. Note that, when using lossless decoding (when lossless encoding is used), the three-dimensional data decoding device may skip inverse quantization (S9512).

[0632] In the example shown in Fig. 92, the number of valid nodes is 2, and the number of valid nodes (=2) <= THnode (for example, 11). In this case, the three-dimensional data decoding device does not generate a predicted value. For example, the three-dimensional data decoding device uses a predicted value = 0.

[0633] Next, the three-dimensional data decoding device calculates the transform coefficients Ti of the child node group by adding the predicted value=0 to the difference values ​​of the transform coefficients of the child node group (S9513). In other words, the three-dimensional data decoding device outputs the difference values ​​as they are as the transform coefficients Ti.

[0634] Next, the three-dimensional data decoding device generates decoded values ​​of the attribute values ​​of the child nodes by applying inverse RAHT or inverse Haar transform to the transform coefficients Ti of the child nodes (S9514). In this way, the three-dimensional data decoding device can appropriately decode the bit stream.

[0635] In addition, when the number of effective nodes>THnode, the three-dimensional data decoding device generates a predicted value by predicting the attribute value of the child node group from the attribute value of the parent node group. In addition, in the three-dimensional data encoding device, when it is selected to use either the calculated value or the attribute value of the parent node or the neighboring parent node for generating the predicted value (when the prediction mode is added to the bit stream), the three-dimensional data decoding device generates a predicted value using the decoded prediction mode. Next, the three-dimensional data decoding device calculates a transform coefficient PTi of the predicted value by applying RAHT or Haar transform to the predicted value. In addition, in step S9513, the three-dimensional data decoding device calculates a transform coefficient Ti of the child node group by adding the transform coefficient PTi of the predicted value to the difference value of the transform coefficient of the child node group.

[0636] 93 is a diagram showing an example of the syntax of an attribute information header (attribute_header). The attribute information header is a header of attribute information included in a bitstream, and is, for example, a header for each frame or for each set of frames.

[0637] As shown in Fig. 93, the attribute information header includes RAHTPredictionFlag (hierarchical predictive coding flag) and THnode (first threshold information). RAHTPredictionFlag is information for switching whether or not to apply hierarchical predictive coding that predicts an attribute value of a lower layer from an attribute value of a higher layer when performing hierarchical coding by RAHT or Haar. RAHTPredictionFlag = 1 indicates that hierarchical predictive coding is applied. RAHTPredictionFlag = 0 indicates that hierarchical predictive coding is not applied.

[0638] THnode is information for switching whether or not to apply hierarchical coding for each child node group. THnode is added to the attribute information header when RAHTPredictionFlag = 1, and is not added to the attribute information header when RAHTPredictionFlag = 0. If the number of effective nodes in the parent node group is greater than THnode, hierarchical predictive coding is applied, and if the number of effective nodes is equal to or less than THnode, hierarchical predictive coding is not applied.

[0639] Fig. 94 is a diagram showing another syntax example of the attribute information header. The attribute information header shown in Fig. 94 does not include RAHTPredictionFlag, but includes THnode. When the minimum value of the number of valid nodes of a parent node is 1, the three-dimensional data encoding device can always apply hierarchical predictive encoding to each child node group by setting THnode=0. Therefore, in this case, RAHTPredictionFlag can be omitted. This allows the data size of the header to be reduced.

[0640] The three-dimensional data encoding device may entropy-encode RAHTPredictionFlag or THnode before adding it to the header. For example, the three-dimensional data encoding device may binarize and arithmetically encode each value. The three-dimensional data encoding device may also encode each value in a fixed length to reduce the amount of processing.

[0641] Also, RAHTPredictionFlag or THnode does not necessarily have to be added to the header. For example, the value of RAHTPredictionFlag or THnode may be specified in a profile or level of a standard, etc. This can reduce the amount of bits in the header.

[0642] FIG. 95 is a flowchart of a three-dimensional data encoding process (switching process of hierarchical predictive encoding). First, the three-dimensional data encoding device calculates the number of effective nodes of a parent node group (S9521). Next, the three-dimensional data encoding device determines whether the number of effective nodes is greater than THnode (S9522). If the number of effective nodes is greater than THnode (Yes in S9522), the three-dimensional data encoding device performs hierarchical predictive encoding of the attribute values ​​of the child node group (S9523). On the other hand, if the number of effective nodes is equal to or less than THnode (No in S9522), the three-dimensional data encoding device performs hierarchical non-predictive encoding of the attribute values ​​of the child node group (S9524). Hierarchical non-predictive encoding is encoding that does not use hierarchical predictive encoding, for example, encoding that does not include a predictive process.

[0643] 96 is a flowchart of a three-dimensional data decoding process (switching process of hierarchical prediction decoding). First, the three-dimensional data decoding device calculates the number of effective nodes of the parent node group (S9531). Next, the three-dimensional data decoding device determines whether the number of effective nodes is greater than THnode (S9532).

[0644] If the number of valid nodes is greater than THnode (Yes in S9532), the three-dimensional data decoding device performs hierarchical prediction decoding on the attribute values ​​of the child node group (S9533). Here, hierarchical prediction decoding is a process of decoding a signal generated by the abo...

Claims

1. determining whether a count value indicating the number of adjacent nodes including the coefficient is equal to or greater than a threshold or is less than the threshold; if the count value is equal to or greater than the threshold, performing a first transformation process including a prediction process using one or more adjacent nodes on a plurality of coefficients of a plurality of child nodes to generate a plurality of first coefficients; If the count value is less than the threshold, a second transformation process not including the prediction process using the one or more adjacent nodes is performed on a plurality of coefficients of a plurality of child nodes to generate a plurality of second coefficients. Encoding method.

2. determining whether a count value indicating the number of adjacent nodes including the coefficient is equal to or greater than a threshold or is less than the threshold; if the count value is equal to or greater than the threshold, performing a first inverse transform process on a plurality of first coefficients, the first inverse transform process including a prediction process using one or more adjacent nodes, to generate a plurality of first inverse transform coefficients; If the count value is less than the threshold, a second inverse transform process that does not include the prediction process using the one or more adjacent nodes is performed on a plurality of second coefficients to generate a plurality of second inverse transform coefficients. Decryption method.

3. A processor; A memory. The processor uses the memory to: determining whether a count value indicating the number of adjacent nodes including the coefficient is equal to or greater than a threshold or is less than the threshold; if the count value is equal to or greater than the threshold, performing a first transformation process including a prediction process using one or more adjacent nodes on a plurality of coefficients of a plurality of child nodes to generate a plurality of first coefficients; If the count value is less than the threshold, a second transformation process not including the prediction process using the one or more adjacent nodes is performed on a plurality of coefficients of a plurality of child nodes to generate a plurality of second coefficients. Encoding device.

4. A processor; A memory. The processor uses the memory to: determining whether a count value indicating the number of adjacent nodes including the coefficient is equal to or greater than a threshold or is less than the threshold; if the count value is equal to or greater than the threshold, performing a first inverse transform process on a plurality of first coefficients, the first inverse transform process including a prediction process using one or more adjacent nodes, to generate a plurality of first inverse transform coefficients; If the count value is less than the threshold, a second inverse transform process that does not include the prediction process using the one or more adjacent nodes is performed on a plurality of second coefficients to generate a plurality of second inverse transform coefficients. Decryption device.

Images