Three-dimensional data encoding method, three-dimensional data decoding method, three-dimensional data encoding device, and three-dimensional data decoding device
By using attribute information to select prediction modes for three-dimensional data encoding, the method improves encoding efficiency, reducing data volume and enhancing storage/transmission capabilities.
Patent Information
- Application Number
- KR1020267035133
- Authority / Receiving Office
- KR · KR
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2018-06-13
- Filing Date
- 2019-06-13
- Publication Date
- 2026-11-09
AI Technical Summary
Existing three-dimensional data encoding methods are inefficient, leading to high data volume and storage/transmission challenges.
A method that utilizes attribute information of surrounding three-dimensional points to select prediction modes for calculating predicted values and generates a bit stream with prediction residuals, improving encoding efficiency.
Enhances encoding efficiency by reducing the amount of data required for three-dimensional data representation, facilitating effective storage and transmission.
Smart Images

Figure PAT00128_ABST
Abstract
Description
Technology Field
[0001] The present disclosure relates to a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, and a three-dimensional data decoding device. Background Technology
[0002] In a wide range of fields, such as computer vision for the autonomous operation of automobiles or robots, map information, surveillance, infrastructure inspection, and image transmission, the widespread adoption of devices or services utilizing three-dimensional data is expected in the future. Three-dimensional data is acquired through various methods, including distance sensors such as rangefinders, stereo cameras, or combinations of multiple monocular cameras.
[0003] As a method of representing three-dimensional data, there is a representation method called a point cloud, which displays the shape of a three-dimensional structure using a point cloud within three-dimensional space. In a point cloud, the location and color of the point cloud are stored. Although point clouds are expected to become the mainstream method for representing three-dimensional data, the amount of data from a point cloud is very large. Therefore, for the storage or transmission of three-dimensional data, compression of the data volume through encoding becomes essential, just as with two-dimensional video (for example, MPEG-4 AVC or HEVC standardized by MPEG).
[0004] In addition, point cloud compression is partially supported by open libraries (Point Cloud Libraries) that perform point cloud-related processing.
[0005] In addition, a technology is known for using three-dimensional map data to search for and display facilities located around a vehicle (see, for example, Patent Document 1). Prior art literature
[0006] International Publication No. 2014 / 020663 The problem to be solved
[0007] In the encoding of three-dimensional data, it is desired to be able to improve encoding efficiency.
[0008] The present disclosure aims to provide a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, or a three-dimensional data decoding device capable of improving encoding efficiency. means of solving the problem
[0009] A three-dimensional data encoding method related to one aspect of the present disclosure is a three-dimensional data encoding method for encoding a plurality of three-dimensional points, wherein the method uses attribute information of one or more second three-dimensional points around a first three-dimensional point to select one of two or more prediction modes for calculating a predicted value of the attribute information of the first three-dimensional point, calculates a predicted value of the selected prediction mode, calculates a prediction residual which is the difference between the attribute information of the first three-dimensional point and the calculated prediction value, and generates a bit stream including the prediction mode and the prediction residual.
[0010] A three-dimensional data decoding method related to one aspect of the present disclosure is a three-dimensional data decoding method for decoding a plurality of three-dimensional points, wherein the method acquires a prediction mode and a prediction residual of a first three-dimensional point among the plurality of three-dimensional points by acquiring a bit stream, calculates a prediction value of the acquired prediction mode, and calculates attribute information of the first three-dimensional point by adding the prediction value and the prediction residual. Effects of the invention
[0011] The present disclosure may provide a three-dimensional data encoding method, a three-dimensional data decoding method, a three-dimensional data encoding device, or a three-dimensional data decoding device capable of improving encoding efficiency. Brief explanation of the drawing
[0012] FIG. 1 is a diagram showing the configuration of encoded three-dimensional data related to embodiment 1. FIG. 2 is a diagram showing an example of a prediction structure between SPCs belonging to the lowest layer of the GOS related to Embodiment 1. FIG. 3 is a diagram showing an example of a prediction structure between layers related to Embodiment 1. FIG. 4 is a diagram showing an example of the encoding sequence of GOS related to Embodiment 1. FIG. 5 is a diagram showing an example of the encoding sequence of GOS related to Embodiment 1. FIG. 6 is a block diagram of a three-dimensional data encoding device related to Embodiment 1. FIG. 7 is a flowchart of the encoding process related to Embodiment 1. FIG. 8 is a block diagram of a three-dimensional data decoding device related to embodiment 1. FIG. 9 is a flowchart of the decoding process related to Embodiment 1. FIG. 10 is a drawing showing an example of meta-information related to Embodiment 1. FIG. 11 is a drawing showing an example of the configuration of a SWLD related to embodiment 2. FIG. 12 is a diagram showing an example of the operation of a server and a client related to embodiment 2. FIG. 13 is a diagram showing an example of the operation of a server and a client related to embodiment 2. FIG. 14 is a diagram showing an example of the operation of a server and a client related to embodiment 2. FIG. 15 is a diagram showing an example of the operation of a server and a client related to embodiment 2. FIG. 16 is a block diagram of a three-dimensional data encoding device related to embodiment 2. FIG. 17 is a flowchart of the encoding process related to embodiment 2. FIG. 18 is a block diagram of a three-dimensional data decoding device related to embodiment 2. FIG. 19 is a flowchart of the decoding process related to embodiment 2. FIG. 20 is a drawing showing an example of the configuration of a WLD related to embodiment 2. FIG. 21 is a diagram showing an example of an octal tree structure of a WLD related to embodiment 2. FIG. 22 is a drawing showing an example of the configuration of a SWLD related to embodiment 2. FIG. 23 is a diagram showing an example of an octal tree structure of a SWLD related to embodiment 2. FIG. 24 is a block diagram of a three-dimensional data creation device related to embodiment 3. FIG. 25 is a block diagram of a three-dimensional data transmission device related to embodiment 3. FIG. 26 is a block diagram of a three-dimensional information processing device related to embodiment 4. FIG. 27 is a block diagram of a three-dimensional data creation device related to embodiment 5. FIG. 28 is a diagram showing the configuration of a system related to embodiment 6. FIG. 29 is a block diagram of a client device related to embodiment 6. FIG. 30 is a block diagram of a server related to embodiment 6. FIG. 31 is a flowchart of the three-dimensional data creation process by a client device related to embodiment 6. FIG. 32 is a flowchart of the sensor information transmission processing by a client device related to embodiment 6. FIG. 33 is a flowchart of the three-dimensional data creation process by the server related to Embodiment 6. FIG. 34 is a flowchart of the three-dimensional map transmission processing by the server related to embodiment 6. FIG. 35 is a diagram showing the configuration of a modified example of a system related to embodiment 6. FIG. 36 is a diagram showing the configuration of a server and client device related to embodiment 6. FIG. 37 is a block diagram of a three-dimensional data encoding device related to embodiment 7. FIG. 38 is a diagram showing an example of a predicted residual related to Embodiment 7. FIG. 39 is a drawing showing an example of a volume related to embodiment 7. FIG. 40 is a drawing showing an example of an octal tree representation of a volume related to embodiment 7. FIG. 41 is a diagram showing an example of a bit sequence of a volume related to embodiment 7. FIG. 42 is a drawing showing an example of an octal tree representation of a volume related to embodiment 7. FIG. 43 is a drawing showing an example of a volume related to embodiment 7. FIG. 44 is a diagram illustrating intra-predictive processing related to embodiment 7. FIG. 45 is a drawing for explaining rotation and translation processing related to embodiment 7. FIG. 46 is a diagram showing an example of the syntax of the RT application flag and RT information related to embodiment 7. FIG. 47 is a diagram illustrating the inter-prediction processing related to embodiment 7. FIG. 48 is a block diagram of a three-dimensional data decoding device related to embodiment 7. FIG. 49 is a flowchart of three-dimensional data encoding processing by a three-dimensional data encoding device related to embodiment 7. FIG. 50 is a flowchart of three-dimensional data decoding processing by a three-dimensional data decoding device related to embodiment 7. FIG. 51 is a diagram showing reference relationships in an octal tree structure related to embodiment 8. FIG. 52 is a drawing showing reference relationships in a spatial region related to embodiment 8. FIG. 53 is a drawing showing an example of an adjacent reference node related to embodiment 8. FIG. 54 is a diagram showing the relationship between a parent node and a node related to embodiment 8. FIG. 55 is a diagram showing an example of an occupancy code of a parent node related to embodiment 8. FIG. 56 is a block diagram of a three-dimensional data encoding device related to embodiment 8. FIG. 57 is a block diagram of a three-dimensional data decoding device related to embodiment 8. FIG. 58 is a flowchart of three-dimensional data encoding processing related to embodiment 8. FIG. 59 is a flowchart of three-dimensional data decoding processing related to embodiment 8. FIG. 60 is a drawing showing a conversion example of an encoding table related to embodiment 8. FIG. 61 is a drawing showing a reference relationship in a spatial region related to Variation 1 of Embodiment 8. FIG. 62 is a drawing showing a syntax example of header information related to Variation Example 1 of Embodiment 8. FIG. 63 is a drawing showing a syntax example of header information related to Variation Example 1 of Embodiment 8. FIG. 64 is a drawing showing an example of an adjacent reference node related to Variation Example 2 of Embodiment 8. FIG. 65 is a drawing showing an example of a target node and an adjacent node related to Variation Example 2 of Embodiment 8. FIG. 66 is a diagram showing reference relationships in an octal tree structure related to Variation Example 3 of Embodiment 8. FIG. 67 is a drawing showing a reference relationship in a spatial region related to Variation Example 3 of Embodiment 8. FIG. 68 is a drawing showing an example of a three-dimensional point related to embodiment 9. FIG. 69 is a drawing showing an example of setting the LoD related to embodiment 9. FIG. 70 is a diagram showing an example of a threshold used for setting the LoD related to embodiment 9. FIG. 71 is a diagram showing an example of attribute information used for the predicted value related to embodiment 9. FIG. 72 is a drawing showing an example of an exponential Colombe symbol related to embodiment 9. FIG. 73 is a diagram showing the processing of the exponential Colombe sign related to embodiment 9. FIG. 74 is a drawing showing an example of the syntax of an attribute header related to embodiment 9. FIG. 75 is a drawing showing an example of the syntax of attribute data related to embodiment 9. FIG. 76 is a flowchart of three-dimensional data encoding processing related to embodiment 9. FIG. 77 is a flowchart of the attribute information encoding process related to embodiment 9. FIG. 78 is a diagram showing the processing of the exponential Colombe sign related to embodiment 9. FIG. 79 is a drawing showing an example of a reverse lookup table representing the relationship between the remaining symbols and their values related to embodiment 9. FIG. 80 is a flowchart of three-dimensional data decoding processing related to embodiment 9. FIG. 81 is a flowchart of attribute information decoding processing related to embodiment 9. FIG. 82 is a block diagram of a three-dimensional data encoding device related to embodiment 9. FIG. 83 is a block diagram of a three-dimensional data decoding device related to embodiment 9. FIG. 84 is a flowchart of three-dimensional data encoding processing related to embodiment 9. FIG. 85 is a flowchart of three-dimensional data decoding processing related to embodiment 9. FIG. 86 is a drawing showing a first example of a table showing predicted values calculated in each prediction mode related to embodiment 10. FIG. 87 is a diagram showing an example of attribute information used for a predicted value related to embodiment 10. FIG. 88 is a drawing showing a second example of a table showing predicted values calculated in each prediction mode related to embodiment 10. FIG. 89 is a drawing showing a third example of a table showing predicted values calculated in each prediction mode related to embodiment 10. FIG. 90 is a drawing showing a fourth example of a table showing predicted values calculated in each prediction mode related to embodiment 10. FIG. 91 is a drawing showing a fifth example of a table showing predicted values calculated in each prediction mode related to embodiment 10. FIG. 92 is a drawing showing a sixth example of a table showing predicted values calculated in each prediction mode related to embodiment 10. FIG. 93 is a drawing showing a seventh example of a table showing the predicted values calculated in each prediction mode related to embodiment 10. FIG. 94 is a drawing showing the eighth example of a table showing the predicted values calculated in each prediction mode related to embodiment 10. FIG. 95 is a diagram showing a first example of a binarization table in which a prediction mode value related to embodiment 10 is binarized and encoded. FIG. 96 is a diagram showing a second example of a binarization table in which a prediction mode value related to embodiment 10 is binarized and encoded. FIG. 97 is a diagram showing a third example of a binarization table in which a prediction mode value related to embodiment 10 is binarized and encoded. FIG. 98 is a diagram illustrating an example of encoding binary data of a binary table when encoding a prediction mode related to embodiment 10 by binary encoding. FIG. 99 is a flowchart illustrating an example of encoding of a prediction mode value related to embodiment 10. FIG. 100 is a flowchart showing an example of decoding a prediction mode value related to embodiment 10. FIG. 101 is a drawing showing another example of a table showing the predicted values calculated in each prediction mode related to embodiment 10. FIG. 102 is a diagram for explaining an example of encoding binary data of a binary table when encoding a prediction mode related to embodiment 10 by binary encoding. FIG. 103 is a flowchart illustrating another example of encoding of a prediction mode value related to embodiment 10. FIG. 104 is a flowchart showing another example of decoding a prediction mode value related to embodiment 10. FIG. 105 is a flowchart illustrating an example of a process for determining whether to fix the prediction mode value according to condition A during encoding related to embodiment 10. FIG. 106 is a flowchart illustrating an example of a process for determining whether to set the prediction mode value to a fixed value or to decode it according to condition A during decoding related to embodiment 10. FIG. 107 is a diagram illustrating the maximum absolute difference value maxdiff related to embodiment 10. FIG. 108 is a drawing showing an example of a syntax related to embodiment 10. FIG. 109 is a drawing showing an example of a syntax related to embodiment 10. FIG. 110 is a flowchart of three-dimensional data encoding processing by a three-dimensional data encoding device related to embodiment 10. FIG. 111 is a flowchart of attribute information encoding processing related to embodiment 10. FIG. 112 is a flowchart of the calculation process of a predicted value in a three-dimensional data encoding device related to embodiment 10. FIG. 113 is a flowchart of the selection processing of the prediction mode related to embodiment 10. FIG. 114 is a flowchart of the process for selecting a prediction mode that minimizes the cost associated with embodiment 10. FIG. 115 is a flowchart of three-dimensional data decoding processing by a three-dimensional data decoding device related to embodiment 10. FIG. 116 is a flowchart of attribute information decoding processing related to embodiment 10. FIG. 117 is a flowchart of the calculation process of a predicted value in a three-dimensional data decoding device related to embodiment 10. FIG. 118 is a flowchart of the decoding process of the prediction mode related to embodiment 10. FIG. 119 is a block diagram showing the configuration of an attribute information encoding unit equipped with a three-dimensional data encoding device related to embodiment 10. FIG. 120 is a block diagram showing the configuration of an attribute information decoding unit equipped with a three-dimensional data decoding device related to embodiment 10. FIG. 121 is a block diagram showing the configuration of a three-dimensional data encoding device related to embodiment 10. FIG. 122 is a block diagram showing the configuration of a three-dimensional data decoding device related to embodiment 10. FIG. 123 is a flowchart of the processing of a three-dimensional data encoding device related to embodiment 10. FIG. 124 is a flowchart of the processing of a three-dimensional data decoding device related to embodiment 10. Specific details for implementing the invention
[0013] A three-dimensional data encoding method related to one aspect of the present disclosure is a three-dimensional data encoding method for encoding a plurality of three-dimensional points, wherein the method uses attribute information of one or more second three-dimensional points around a first three-dimensional point to select one of two or more prediction modes for calculating a predicted value of the attribute information of the first three-dimensional point, calculates a predicted value of the selected prediction mode, calculates a prediction residual which is the difference between the attribute information of the first three-dimensional point and the calculated prediction value, and generates a bit stream including the prediction mode and the prediction residual.
[0014] According to this, since attribute information can be encoded using the predicted value of one of two or more prediction modes, the encoding efficiency of attribute information can be improved.
[0015] In addition, in calculating the above-mentioned prediction value, in the first prediction mode among the above-mentioned two or more prediction modes, the average of the attribute information of the above-mentioned one or more second three-dimensional points may be calculated as the above-mentioned prediction value, and in the second prediction mode among the above-mentioned two or more prediction modes, the attribute information of the above-mentioned second three-dimensional points may be calculated as the above-mentioned prediction value.
[0016] Additionally, the first prediction mode value representing the first prediction mode is smaller than the second prediction mode value representing the second prediction mode, and the bit stream may include the prediction mode value representing the selected prediction mode as the prediction mode.
[0017] In addition, in calculating the above prediction value, two or more of the above average or attribute information may be calculated as the prediction value of the above prediction mode.
[0018] In addition, the above attribute information is color information representing the color of a three-dimensional point, and the above two or more average or attribute information may each represent the values of two or more components that define a color space.
[0019] In addition, in the calculation of the above prediction value, in the third prediction mode among the two or more prediction modes, attribute information of one second three-dimensional point is calculated as the prediction value, and in the fourth prediction mode among the two or more prediction modes, attribute information of another second three-dimensional point located at a distance from the first three-dimensional point farther than that of the one second three-dimensional point is calculated as the prediction value, and the third prediction mode value representing the third prediction mode is smaller than the fourth prediction mode value representing the fourth prediction mode, and the bit stream may include the prediction mode value representing the selected prediction mode as the prediction mode.
[0020] According to this, the amount of code can be reduced by making the prediction mode value, which represents a prediction mode that produces a prediction value that is easy to select, a small value.
[0021] Additionally, the above attribute information includes a first attribute information and a second attribute information of a different type from the first attribute information, and in calculating the predicted value, the first predicted value may be calculated using the first attribute information and the second predicted value may also be calculated using the second attribute information.
[0022] In addition, when selecting the prediction mode, if the maximum absolute difference value of the attribute information of the one or more second three-dimensional points is less than a predetermined threshold, a prediction mode for calculating the average of the attribute information of the one or more second three-dimensional points as the prediction value may be selected, and if the maximum absolute difference value is greater than or equal to the predetermined threshold, one of the two or more prediction modes may be selected.
[0023] According to this, throughput can be reduced when the maximum absolute difference value is below a predetermined threshold.
[0024] A three-dimensional data decoding method related to one aspect of the present disclosure is a three-dimensional data decoding method for decoding a plurality of three-dimensional points, wherein the method acquires a prediction mode and a prediction residual of a first three-dimensional point among the plurality of three-dimensional points by acquiring a bit stream, calculates a prediction value of the acquired prediction mode, and calculates attribute information of the first three-dimensional point by adding the prediction value and the prediction residual.
[0025] According to this, encoded attribute information can be properly decoded using the prediction value of one of two or more prediction modes.
[0026] In addition, in calculating the above-mentioned prediction value, in the first prediction mode among the above-mentioned two or more prediction modes, the average of the attribute information of one or more second three-dimensional points around the first three-dimensional point may be calculated as the above-mentioned prediction value, and in the second prediction mode among the above-mentioned two or more prediction modes, the attribute information of the second three-dimensional point may be calculated as the above-mentioned prediction value.
[0027] Additionally, the first prediction mode value representing the first prediction mode is smaller than the second prediction mode value representing the second prediction mode, and the bit stream may include the prediction mode value representing the selected prediction mode as the prediction mode.
[0028] In addition, in calculating the above prediction value, two or more of the above average or attribute information may be calculated as the prediction value of the above prediction mode.
[0029] In addition, the above attribute information is color information representing the color of a three-dimensional point, and the above two or more average or attribute information may each represent the values of two or more components that define a color space.
[0030] In addition, in the calculation of the above prediction value, in the third prediction mode among the two or more prediction modes, attribute information of one of the two or more second three-dimensional points around the first three-dimensional point is calculated as the prediction value, and in the fourth prediction mode among the two or more prediction modes, attribute information of another second three-dimensional point located at a distance from the first three-dimensional point farther than that of the one second three-dimensional point is calculated as the prediction value, and the third prediction mode value representing the third prediction mode is smaller than the fourth prediction mode value representing the fourth prediction mode, and the bit stream may include the prediction mode value representing the selected prediction mode as the prediction mode.
[0031] According to this, the time required for decoding attribute information can be reduced by making the prediction mode value, which represents a prediction mode that produces a prediction value that is easy to select, a small value.
[0032] Additionally, the above attribute information includes a first attribute information and a second attribute information of a different type from the first attribute information, and in calculating the predicted value, the first predicted value may be calculated using the first attribute information and the second predicted value may also be calculated using the second attribute information.
[0033] In addition, in calculating the above prediction value, if the maximum absolute difference value of the attribute information of one or more second three-dimensional points around the first three-dimensional point is less than a predetermined threshold, the average of the attribute information of the one or more second three-dimensional points is calculated as the above prediction value, and if the maximum absolute difference value is greater than or equal to the predetermined threshold, the prediction value of one of the two or more prediction modes may be calculated as the above prediction value.
[0034] According to this, throughput can be reduced when the maximum absolute difference value is below a predetermined threshold.
[0035] In addition, a three-dimensional data encoding device related to one aspect of the present disclosure is a three-dimensional data encoding device that encodes a plurality of three-dimensional points, and comprises a processor and a memory. The processor uses the memory to select one of two or more prediction modes for calculating a predicted value of attribute information of a first three-dimensional point using attribute information of one or more second three-dimensional points around a first three-dimensional point, calculates a predicted value of the selected prediction mode, calculates a prediction residual which is the difference between the attribute information of the first three-dimensional point and the calculated prediction value, and generates a bit stream including the prediction mode and the prediction residual.
[0036] According to this, since attribute information can be encoded using the predicted value of one of two or more prediction modes, the encoding efficiency of attribute information can be improved.
[0037] In addition, a three-dimensional data decoding device related to one aspect of the present disclosure is a three-dimensional data decoding device for decoding a plurality of three-dimensional points, comprising a processor and a memory, wherein the processor acquires a prediction mode and a prediction residual of a first three-dimensional point among the plurality of three-dimensional points by acquiring a bit stream using the memory, calculates a prediction value of the acquired prediction mode, and calculates attribute information of the first three-dimensional point by adding the prediction value and the prediction residual.
[0038] According to this, throughput can be reduced when the maximum absolute difference value is below a predetermined threshold.
[0039] In addition, comprehensive or specific embodiments thereof may be realized as a system, method, integrated circuit, computer program, or recording medium such as a computer-readable CD-ROM, or as any combination of a system, method, integrated circuit, computer program, and recording medium.
[0040] Hereinafter, embodiments will be described in detail with reference to the drawings. Furthermore, all embodiments described below represent specific embodiments of the present disclosure. The numerical values, shapes, materials, components, arrangement positions and connection types of components, steps, and sequence of steps described in the following embodiments are examples and are not well known facts that limit the present disclosure. Additionally, regarding the components in the following embodiments, any components not described in the independent claim representing the highest concept are described as arbitrary components.
[0041] (Embodiment 1)
[0042] First, the data structure of the encoded three-dimensional data (hereinafter also referred to as encoded data) related to the present embodiment will be explained. FIG. 1 is a diagram showing the configuration of the encoded three-dimensional data related to the present embodiment.
[0043] In this embodiment, the three-dimensional space is divided into spaces (SPCs) corresponding to pictures in video encoding, and three-dimensional data is encoded using the spaces as units. The spaces are also divided into volumes (VLMs) corresponding to macroblocks, etc. in video encoding, and prediction and transformation are performed using the VLMs as units. The volumes include a plurality of voxels (VXLs), which are the smallest units corresponding to position coordinates. Furthermore, prediction involves generating predicted three-dimensional data similar to the processing unit of the target processing by referencing another processing unit, just as prediction is performed on a two-dimensional image, and encoding the difference between said predicted three-dimensional data and the processing unit of the target processing. In addition, this prediction includes not only spatial prediction referencing another prediction unit at the same time, but also temporal prediction referencing a prediction unit at a different time.
[0044] For example, when a three-dimensional data encoding device (hereinafter also referred to as an encoding device) encodes a three-dimensional space represented by point cloud data, such as a point cloud, it groups each point of the point cloud or multiple points contained within a voxel according to the size of the voxel. Subdividing the voxel allows the three-dimensional shape of the point cloud to be represented with high precision, while increasing the size of the voxel allows the three-dimensional shape of the point cloud to be represented approximately.
[0045] In addition, the following explanation uses the case where the three-dimensional data is a point cloud as an example, but the three-dimensional data is not limited to a point cloud and can be any format of three-dimensional data.
[0046] Additionally, a hierarchical voxel structure may be used. In this case, for the n-th order layer, whether a sample point exists in the n-1th order layer or lower (the layer below the n-th order layer) may be indicated in order. For example, when decoding only the n-th order layer, if a sample point exists in the n-1th order layer or lower, decoding can be performed by assuming that the sample point exists at the center of the voxel of the n-th order layer.
[0047] In addition, the encoding device acquires point cloud data using a distance sensor, a stereo camera, a monocular camera, a gyroscope, or an inertial sensor, etc.
[0048] The space is classified into at least three prediction structures, including an intra-space (I-SPC) that can be decoded independently, a predictive space (P-SPC) that can only be referenced unidirectionally, and a bidirectional space (B-SPC) that can be referenced bidirectionally, just like video encoding. In addition, the space has two types of time information: a decoding time and a display time.
[0049] Also, as shown in FIG. 1, there exists a Group of Space (GOS), which is a random access unit, as a processing unit including multiple spaces. In addition, there exists a World (WLD) as a processing unit including multiple GOSs.
[0050] The spatial area occupied by the world corresponds to an absolute location on Earth based on GPS or latitude and longitude information. This location information is stored as metadata. Additionally, the metadata may be included in the encoded data or transmitted separately from the encoded data.
[0051] Additionally, within GOS, all SPCs may be three-dimensionally adjacent, and there may exist SPCs that are not three-dimensionally adjacent to other SPCs.
[0052] Furthermore, below, processing such as encoding, decoding, or referencing of three-dimensional data included in a processing unit such as a GOS, SPC, or VLM is also simply described as encoding, decoding, or referencing the processing unit. Additionally, the three-dimensional data included in the processing unit includes, for example, at least one set of spatial positions such as three-dimensional coordinates and characteristic values such as color information.
[0053] Next, the prediction structure of the SPC in the GOS is explained. Multiple SPCs within the same GOS, or multiple VLMs within the same SPC, occupy different spaces but have the same time information (decoding time and display time).
[0054] Also, the SPC that is the leading in the decoding order within the GOS is the I-SPC. Additionally, there are two types of GOS: closed GOS and open GOS. A closed GOS is a GOS that can decode all SPCs within the GOS when decoding is started from the leading I-SPC. In an open GOS, some SPCs within the GOS whose display time is earlier than the leading I-SPC refer to a different GOS, and decoding cannot be performed using only that GOS.
[0055] Furthermore, in encoded data such as map information, WLDs are sometimes decoded in the reverse order of encoding, and if there are dependencies between GOSs, reverse regeneration becomes difficult. Therefore, in such cases, a closed GOS is basically used.
[0056] In addition, the GOS has a layered structure in the height direction, and encoding or decoding is performed sequentially starting from the SPC of the layer below.
[0057] Figure 2 is a diagram showing an example of a prediction structure between SPCs belonging to the lowest layer of GOS. Figure 3 is a diagram showing an example of a prediction structure between layers.
[0058] There is one or more I-SPCs within the GOS. In three-dimensional space, there are objects such as people, animals, cars, bicycles, signals, or landmark buildings, but objects that are particularly small in size are valid when encoded as I-SPCs. For example, when a three-dimensional data decoder (hereinafter also referred to as a decoder) decodes the GOS at low throughput or high speed, it decodes only the I-SPCs within the GOS.
[0059] Additionally, the encoding device may switch the encoding interval or occurrence frequency of I-SPC according to the density of objects within the WLD.
[0060] In addition, in the configuration shown in FIG. 3, the encoding device or the decoding device encodes or decodes a plurality of layers in order from the lower layer (Layer 1). By doing so, for example, in the case of an autonomous vehicle, the priority of data near the ground, which has a larger amount of information, can be increased.
[0061] In addition, for encoded data used in drones, etc., encoding or decoding may be performed sequentially from the SPC of the upper layer in the height direction within the GOS.
[0062] In addition, the encoding device or the decoding device may encode or decode multiple layers so that the decoding device can roughly determine the GOS and gradually increase the resolution. For example, the encoding device or the decoding device may encode or decode in the order of layers 3, 8, 1, 9…
[0063] Next, the handling methods for static and dynamic objects will be explained.
[0064] In three-dimensional space, there exist static objects or scenes, such as buildings or roads (hereinafter collectively referred to as static objects), and dynamic objects, such as cars or people (hereinafter referred to as dynamic objects). Object detection is performed separately by extracting feature points from point cloud data or camera images, such as those from stereo cameras. Here, an example of an encoding method for dynamic objects is described.
[0065] The first method is a method of encoding without distinguishing between static objects and dynamic objects. The second method is a method of distinguishing between static objects and dynamic objects based on identification information.
[0066] For example, a GOS is used as an identification unit. In this case, a GOS containing an SPC that constitutes a static object and a GOS containing an SPC that constitutes a dynamic object are distinguished by identification information stored within the encoded data or separately from the encoded data.
[0067] Alternatively, an SPC may be used as an identification unit. In this case, an SPC containing a VLM constituting a static object and an SPC containing a VLM constituting a dynamic object are distinguished by the identification information.
[0068] Alternatively, VLM or VXL may be used as an identification unit. In this case, a VLM or VXL containing static objects and a VLM or VXL containing dynamic objects are distinguished by the identification information.
[0069] In addition, the encoding device may encode a dynamic object as one or more VLMs or SPCs, and may encode the VLM or SPC containing the static object and the SPC containing the dynamic object as different GOSs. In addition, if the size of the GOS varies depending on the size of the dynamic object, the encoding device stores the size of the GOS separately as metadata.
[0070] In addition, the encoding device may encode static objects and dynamic objects independently of each other, and may superimpose dynamic objects on a world composed of static objects. In this case, the dynamic object is composed of one or more SPCs, and each SPC corresponds to one or more SPCs that constitute the static object to which the said SPC is superimposed. In addition, the dynamic object may be represented by one or more VLMs or VXLs instead of SPCs.
[0071] Additionally, the encoding device may encode static objects and dynamic objects as different streams.
[0072] Additionally, the encoding device may generate a GOS containing one or more SPCs that constitute a dynamic object. Furthermore, the encoding device may set the GOS containing the dynamic object (GOS_M) and the GOS of the static object corresponding to the spatial area of GOS_M to be of the same size (occupy the same spatial area). By doing so, overlapping processing can be performed on a GOS basis.
[0073] P-SPCs or B-SPCs constituting a dynamic object may refer to SPCs included in different GOSs that have been encoded. In cases where the location of a dynamic object changes over time and the same dynamic object is encoded as a GOS at a different time, referencing across GOSs becomes valid in terms of compression ratio.
[0074] Additionally, depending on the application of the encoded data, the first method and the second method described above may be switched. For example, when the encoded three-dimensional data is used as a map, the encoding device uses the second method because it is desirable to be able to separate dynamic objects. On the other hand, when the encoding device is encoding three-dimensional data for events such as concerts or sports, if there is no need to separate dynamic objects, the first method is used.
[0075] In addition, the decoding time and display time of the GOS or SPC may be stored within the encoded data or as metadata. Also, the time information of the static object may all be identical. In this case, the actual decoding time and display time may be determined by the decoder. Alternatively, different values may be assigned to each GOS or SPC as the decoding time, and the same value may be assigned to all as the display time. Furthermore, a model may be introduced that guarantees that the decoder can decode without corruption by having a buffer of a predetermined size and reading the bit stream at a predetermined bit rate according to the decoding time, such as the decoder model in motion encoding like HEVC’s HRD (Hypothetical Reference Decoder).
[0076] Next, the arrangement of GOS within the world is described. The coordinates of the three-dimensional space within the world are represented by three mutually orthogonal coordinate axes (x-axis, y-axis, and z-axis). By setting a predetermined rule for the encoding order of GOS, encoding can be performed so that spatially adjacent GOS are continuous within the encoded data. For example, in the example shown in Fig. 4, GOS within the xz plane are encoded continuously. After the encoding of all GOS within a given xz plane is completed, the value of the y-axis is updated. That is, as encoding proceeds, the world expands in the y-axis direction. In addition, the index numbers of the GOS are set according to the encoding order.
[0077] Here, the three-dimensional space of the world is corresponded one-to-one with geographical absolute coordinates such as GPS, latitude, and longitude. Alternatively, the three-dimensional space may be represented by a relative position from a pre-set reference position. The directions of the x-axis, y-axis, and z-axis of the three-dimensional space are represented as direction vectors determined based on latitude and longitude, and said direction vectors are stored as meta-information along with encoded data.
[0078] In addition, the size of the GOS is fixed, and the encoding device stores the said size as metadata. Furthermore, the size of the GOS may be changed depending, for example, whether it is a city area, or whether it is indoors or outdoors. In short, the size of the GOS may be changed depending on the quantity or nature of objects that have informational value. Alternatively, the encoding device may adaptively change the size of the GOS or the interval of the I-SPC within the GOS according to the density of objects within the same world. For example, the encoding device reduces the size of the GOS and shortens the interval of the I-SPC within the GOS as the density of objects increases.
[0079] In the example of Fig. 5, the GOS is subdivided to realize random access at fine particle size because the object density is high in the regions of the 3rd to 10th GOS. Additionally, the 7th to 10th GOS each exist inside the 3rd to 6th GOS.
[0080] Next, the configuration and operation flow of the three-dimensional data encoding device related to the present embodiment will be described. FIG. 6 is a block diagram of the three-dimensional data encoding device (100) related to the present embodiment. FIG. 7 is a flowchart showing an example of the operation of the three-dimensional data encoding device (100).
[0081] The three-dimensional data encoding device (100) shown in FIG. 6 generates encoded three-dimensional data (112) by encoding three-dimensional data (111). The three-dimensional data encoding device (100) comprises an acquisition unit (101), an encoding area determination unit (102), a division unit (103), and an encoding unit (104).
[0082] As shown in FIG. 7, first, the acquisition unit (101) acquires three-dimensional data (111), which is point cloud data (S101).
[0083] Next, the encoding area determination unit (102) determines the area to be encoded among the spatial areas corresponding to the acquired point cloud data (S102). For example, the encoding area determination unit (102) determines the spatial area surrounding the location of the user or vehicle as the area to be encoded according to the location of the user or vehicle.
[0084] Next, the division unit (103) divides the point cloud data included in the area of the encoding target into each processing unit. Here, the processing unit is the GOS and SPC described above. Also, the area of the encoding target corresponds, for example, to the world described above. Specifically, the division unit (103) divides the point cloud data into processing units based on the size of the GOS set in advance, or the presence or size of dynamic objects (S103). Also, the division unit (103) determines the starting position of the SPC that is the leading in the encoding order for each GOS.
[0085] Next, the encoding unit (104) generates encoded three-dimensional data (112) by sequentially encoding a plurality of SPCs within each GOS (S104).
[0086] In addition, although an example is shown here in which the region to be encoded is divided into GOS and SPC and then each GOS is encoded, the order of processing is not limited to the above. For example, a sequence may be used in which the configuration of one GOS is determined, that GOS is encoded, and then the configuration of the next GOS is determined.
[0087] In this way, the three-dimensional data encoding device (100) generates encoded three-dimensional data (112) by encoding three-dimensional data (111). Specifically, the three-dimensional data encoding device (100) divides the three-dimensional data into a first processing unit (GOS), which is a random access unit and each corresponds to a three-dimensional coordinate, divides the first processing unit (GOS) into a plurality of second processing units (SPC), and divides the second processing units (SPC) into a plurality of third processing units (VLM). In addition, the third processing unit (VLM) includes one or more voxels (VXL), which are the minimum units to which position information corresponds.
[0088] Next, the three-dimensional data encoding device (100) generates encoded three-dimensional data (112) by encoding each of a plurality of first processing units (GOS). Specifically, the three-dimensional data encoding device (100) encodes each of a plurality of second processing units (SPC) in each first processing unit (GOS). In addition, the three-dimensional data encoding device (100) encodes each of a plurality of third processing units (VLM) in each second processing unit (SPC).
[0089] For example, when the first processing unit (GOS) of the processing target is a closed GOS, the three-dimensional data encoding device (100) encodes the second processing unit (SPC) of the processing target included in the first processing unit (GOS) of the processing target by referring to another second processing unit (SPC) included in the first processing unit (GOS) of the processing target. In short, the three-dimensional data encoding device (100) does not refer to the second processing unit (SPC) included in the first processing unit (GOS) that is different from the first processing unit (GOS) of the processing target.
[0090] Meanwhile, if the first processing unit (GOS) of the processing target is an open GOS, the second processing unit (SPC) of the processing target included in the first processing unit (GOS) of the processing target is encoded by referring to another second processing unit (SPC) included in the first processing unit (GOS) of the processing target, or a second processing unit (SPC) included in a first processing unit (GOS) that is different from the first processing unit (GOS) of the processing target.
[0091] Additionally, the three-dimensional data encoding device (100) selects one of the following types of a second processing unit (SPC) of a processing target: a first type (I-SPC) that does not refer to another second processing unit (SPC), a second type (P-SPC) that refers to another second processing unit (SPC), and a third type that refers to two other second processing units (SPC), and encodes the second processing unit (SPC) of the processing target according to the selected type.
[0092] Next, the configuration and operation flow of the three-dimensional data decoding device related to the present embodiment will be described. FIG. 8 is a block diagram of the blocks of the three-dimensional data decoding device (200) related to the present embodiment. FIG. 9 is a flowchart showing an example of the operation of the three-dimensional data decoding device (200).
[0093] The three-dimensional data decoding device (200) shown in FIG. 8 generates decoded three-dimensional data (212) by decoding encoded three-dimensional data (211). Here, the encoded three-dimensional data (211) is, for example, encoded three-dimensional data (112) generated by a three-dimensional data encoding device (100). The three-dimensional data decoding device (200) comprises an acquisition unit (201), a decoding start GOS determination unit (202), a decoding SPC determination unit (203), and a decoding unit (204).
[0094] First, the acquisition unit (201) acquires encoded three-dimensional data (211) (S201). Next, the decoding start GOS determination unit (202) determines the GOS of the decoding target (S202). Specifically, the decoding start GOS determination unit (202) determines the GOS of the decoding target by referring to metadata stored within the encoded three-dimensional data (211) or separately from the encoded three-dimensional data, and includes an SPC corresponding to the spatial location, object, or time at which decoding is started.
[0095] Next, the decoding SPC determination unit (203) determines the type (I, P, B) of the SPC to be decoded within the GOS (S203). For example, the decoding SPC determination unit (203) determines whether to (1) decode only the I-SPC, (2) decode the I-SPC and P-SPC, or (3) decode all types. Additionally, if the type of SPC to be decoded is determined in advance, such as decoding all SPCs, this step does not need to be performed.
[0096] Next, the decoding unit (204) obtains an address position that the SPC leading in the decoding order (same as the encoding order) within the GOS starts in the encoded three-dimensional data (211), obtains the encoded data of the leading SPC from the said address position, and sequentially decodes each SPC in order starting from the said leading SPC (S204). In addition, the said address position is stored in meta information, etc.
[0097] In this way, the three-dimensional data decoding device (200) decodes the decoded three-dimensional data (212). Specifically, the three-dimensional data decoding device (200) generates the decoded three-dimensional data (212) of the first processing unit (GOS) by decoding each of the encoded three-dimensional data (211) of the first processing unit (GOS), each of which is a random access unit and corresponds to a three-dimensional coordinate. More specifically, the three-dimensional data decoding device (200) decodes each of the plurality of second processing units (SPC) in each first processing unit (GOS). In addition, the three-dimensional data decoding device (200) decodes each of the plurality of third processing units (VLM) in each second processing unit (SPC).
[0098] Hereinafter, meta-information for random access is described. This meta-information is generated by a three-dimensional data encoding device (100) and is included in encoded three-dimensional data (112) (211).
[0099] In conventional random access for two-dimensional video images, decoding was initiated from the leading frame of a random access unit that is near a specified time. On the other hand, in the case of the world, random access to space (coordinates or objects, etc.) is assumed in addition to time.
[0100] Therefore, in order to realize random access to at least three elements—coordinates, objects, and time—a table is prepared that corresponds each element to an index number of the GOS. Additionally, an index number of the GOS is corresponded to the address of the I-SPC that is the leader of the GOS. FIG. 10 is a diagram showing an example of a table included in meta-information. Furthermore, it is not necessary to use all the tables shown in FIG. 10, and at least one table needs to be used.
[0101] Hereinafter, as an example, random access based on coordinates will be described. When accessing coordinates (x2, y2, z2), first, by referring to the coordinate-GOS table, it can be seen that the point where the coordinates are (x2, y2, z2) is included in the second GOS. Next, by referring to the GOS address table, it can be seen that the address of the leading I-SPC in the second GOS is addr (2), so the decoding unit (204) obtains data from this address and starts decoding.
[0102] In addition, the address may be an address in a logical format, or a physical address of an HDD or memory. Also, information specifying a file segment may be used instead of an address. For example, a file segment is a unit segmented from one or more GOSs.
[0103] Additionally, if an object spans multiple GOSs, the object-GOS table may list multiple GOSs to which the object belongs. If the multiple GOSs are closed GOSs, the encoding device and the decoding device can perform encoding or decoding in parallel. On the other hand, if the multiple GOSs are open GOSs, compression efficiency can be further improved by the multiple GOSs referencing each other.
[0104] Examples of objects include people, animals, cars, bicycles, signals, or buildings that serve as landmarks. For example, a three-dimensional data encoding device (100) can extract characteristic points unique to an object from a three-dimensional point cloud, etc., during world encoding, detect an object based on said characteristic points, and set the detected object as a random access point.
[0105] In this way, the three-dimensional data encoding device (100) generates first information representing a plurality of first processing units (GOS) and three-dimensional coordinates corresponding to each of the plurality of first processing units (GOS). Also, the encoded three-dimensional data (112 (211)) includes this first information. Also, the first information represents at least one of an object, a time, and a data storage location corresponding to each of the plurality of first processing units (GOS).
[0106] A three-dimensional data decoding device (200) obtains first information from encoded three-dimensional data (211), uses the first information to identify the encoded three-dimensional data (211) of a first processing unit corresponding to a specified three-dimensional coordinate, object, or time, and decodes the encoded three-dimensional data (211).
[0107] Examples of other meta-information are described below. In addition to meta-information for random access, the three-dimensional data encoding device (100) may generate and store meta-information as follows. Also, the three-dimensional data decoding device (200) may use this meta-information during decoding.
[0108] When three-dimensional data is used as map information, a profile is defined according to the application, and information representing said profile may be included in the metadata. For example, profiles for urban areas, suburbs, or flying objects are defined, and for each, the maximum or minimum sizes of the World, SPC, or VLM are defined. For instance, since more detailed information is required for urban areas than for suburbs, the minimum size of the VLM is set smaller.
[0109] Meta information may include tag values indicating the type of object. These tag values correspond to the VLM, SPC, or GOS that constitute the object. For example, tag values may be set for each type of object, such as tag value "0" representing a "person," tag value "1" representing a "vehicle," and tag value "2" representing a "signal." Alternatively, if it is difficult to determine the type of object or if determination is unnecessary, tag values indicating properties such as size or whether it is a dynamic or static object may be used.
[0110] Additionally, the meta-information may include information indicating the range of the spatial area occupied by the world.
[0111] Additionally, the metadata may store the size of the SPC or VXL as common header information for multiple SPCs, such as the entire stream of encoded data or the SPC within the GOS.
[0112] Additionally, the metadata may include identification information, such as distance sensors or cameras used to generate the point cloud, or information indicating the positional precision of the point cloud within the point cloud.
[0113] Additionally, the meta-information may include information indicating whether the world consists only of static objects or includes dynamic objects.
[0114] Variations of the present embodiment will be described below.
[0115] The encoding device or the decoding device may encode or decode two or more different SPCs or GOSs in parallel. The GOSs to be encoded or decoded in parallel may be determined based on meta-information indicating the spatial location of the GOS.
[0116] In cases where three-dimensional data is used as a spatial map when a vehicle or flying object moves, or when such a spatial map is generated, the encoding device or the decoding device may encode or decode the GOS or SPC included in the space specified based on GPS, route information, or zoom magnification, etc.
[0117] Additionally, the decoder may perform decoding sequentially starting from the space close to its own location or the driving path. The encoding device or the decoder may encode or decode the space far from its own location or the driving path with lower priority compared to the space close to it. Here, lowering priority means lowering the processing order, lowering the resolution (filtering out processing), or lowering the image quality (increasing encoding efficiency; for example, increasing the quantization step).
[0118] In addition, when the decoding device decodes encoded data that is hierarchically encoded within a space, it may decode only the lower layer.
[0119] In addition, the decoding device may prioritize decoding from lower layers depending on the map's zoom level or application.
[0120] In addition, for applications such as self-position estimation or object recognition performed during autonomous driving of a vehicle or robot, the encoding device or the decoding device may perform encoding or decoding with reduced resolution in areas other than the area within a specific height from the road surface (the area where recognition is performed).
[0121] Additionally, the encoding device may encode point clouds representing indoor and outdoor spatial shapes separately. For example, by separating the GOS representing the indoor area (Indoor GOS) and the GOS representing the outdoor area (Outdoor GOS), the decoding device can select the GOS to decode based on the time position when using the encoded data.
[0122] Additionally, the encoding device may encode an indoor GOS and an outdoor GOS with close coordinates so that they are adjacent within the encoding stream. For example, the encoding device maps the identifiers of both and stores information representing the mapped identifiers within the encoding stream or in metadata stored separately. Thus, the decoding device can identify an indoor GOS and an outdoor GOS with close coordinates by referring to the information in the metadata.
[0123] Additionally, the encoding device may switch the size of the GOS or SPC between the indoor GOS and the outdoor GOS. For example, the encoding device sets the size of the GOS to be smaller indoors compared to outdoors. Furthermore, the encoding device may change the precision when extracting feature points from a point cloud or the precision of object detection between the indoor GOS and the outdoor GOS.
[0124] Additionally, the encoding device may add information to the encoded data to enable the decoding device to distinguish and display dynamic objects from static objects. By doing so, the decoding device can display dynamic objects together with red frames or explanatory characters. Alternatively, the decoding device may display only red frames or explanatory characters instead of dynamic objects. Furthermore, the decoding device may display more detailed object types. For example, a red frame may be used for cars, and a yellow range may be used for people.
[0125] Additionally, the encoding device or the decoding device may determine whether to encode or decode dynamic objects and static objects using different SPCs or GOSs based on the frequency of occurrence of dynamic objects or the ratio of static objects to dynamic objects. For example, if the frequency or ratio of occurrence of dynamic objects exceeds a threshold, an SPC or GOS containing a mixture of dynamic and static objects is permitted, and if the frequency or ratio of occurrence of dynamic objects does not exceed a threshold, an SPC or GOS containing a mixture of dynamic and static objects is not permitted.
[0126] When detecting dynamic objects from two-dimensional image information of a camera rather than a point cloud, the encoding device may separately acquire information (frames or characters, etc.) for identifying the detection result and the object location, and encode these information as part of three-dimensional encoded data. In this case, the decoder displays auxiliary information (frames or characters) representing the dynamic object by superimposing it on the decoding result of the static object.
[0127] Additionally, the encoding device may change the density of VXL or VLM in SPC according to the complexity of the shape of the static object, etc. For example, the encoding device sets VXL or VLM to be denser as the shape of the static object becomes more complex. Furthermore, the encoding device may determine the quantization step, etc., when quantizing spatial position or color information, according to the density of VXL or VLM. For example, the encoding device sets the quantization step to be smaller as VXL or VLM becomes denser.
[0128] As described above, the encoding device or decoding device related to the present embodiment performs encoding or decoding of space in space units having coordinate information.
[0129] In addition, the encoding device and the decoding device perform encoding or decoding on a volume basis within the space. The volume includes a voxel, which is the smallest unit to which position information corresponds.
[0130] In addition, the encoding device and the decoding device perform encoding or decoding by mapping arbitrary elements using a table mapping each element of spatial information, including coordinates, objects, and time, to a GOP, or a table mapping elements to each other. In addition, the decoding device determines coordinates using the value of a selected element, specifies a volume, voxel, or space from the coordinates, and decodes the space containing the said volume or voxel, or the specified space.
[0131] In addition, the encoding device determines a volume, voxel, or space selectable by an element through feature point extraction or object recognition, and encodes it as a randomly accessible volume, voxel, or space.
[0132] Spaces are classified into three types: I-SPC, which can be encoded or decoded as a space group; P-SPC, which is encoded or decoded by referencing any one processed space; and B-SPC, which is encoded or decoded by referencing any two processed spaces.
[0133] One or more volumes correspond to static objects or dynamic objects. Spaces containing static objects and spaces containing dynamic objects are encoded or decoded as different GOSs. In short, SPCs containing static objects and SPCs containing dynamic objects are assigned to different GOSs.
[0134] Dynamic objects are encoded or decoded for each object and correspond to one or more spaces containing static objects. In short, multiple dynamic objects are encoded individually, and the encoded data of the multiple dynamic objects obtained corresponds to an SPC containing static objects.
[0135] The encoding device and the decoding device increase the priority of the I-SPC within the GOS and perform encoding or decoding. For example, the encoding device performs encoding to minimize I-SPC degradation (so that the original three-dimensional data is more faithfully reproduced after decoding). Also, the decoding device decodes, for example, only the I-SPC.
[0136] The encoding device may perform encoding by changing the frequency of using I-SPC depending on the density or number (quantity) of objects in the world. In short, the encoding device changes the frequency of selecting I-SPC according to the number or density of objects included in the three-dimensional data. For example, the encoding device increases the frequency of using I-space as the number of objects in the world becomes denser.
[0137] In addition, the encoding device sets random access points in GOS units and stores information indicating the spatial region corresponding to the GOS in the header information.
[0138] The encoding device uses, for example, a default value as the space size of the GOS. Additionally, the encoding device may change the size of the GOS according to the number (quantity) or density of objects or dynamic objects. For example, the encoding device reduces the space size of the GOS as the objects or dynamic objects become denser or more numerous.
[0139] In addition, the space or volume includes a group of feature points derived using information obtained from sensors such as a depth sensor, a gyroscope, or a camera. The coordinates of the feature points are set at the center position of the voxel. In addition, high precision of position information can be realized by subdividing the voxel.
[0140] A feature point group is derived using multiple pictures. The multiple pictures have at least two types of time information, including actual time information and the same time information in multiple pictures corresponding to the space (e.g., encoded time used for rate control, etc.).
[0141] In addition, encoding or decoding is performed in GOS units containing one or more spaces.
[0142] The encoding device and the decoding device refer to the space within the GOS that has been processed and perform prediction of the P-space or B-space within the GOS of the processing target.
[0143] Alternatively, the encoding device and the decoding device perform prediction of the P-space or B-space within the GOS of the processing target using the processed space within the GOS of the processing target without referring to a different GOS.
[0144] In addition, the encoding device and the decoding device transmit or receive an encoded stream in world units including one or more GOS.
[0145] In addition, the GOS has a layer structure in at least one direction within the world, and the encoding device and the decoding device perform encoding or decoding from the lower layer. For example, a randomly accessible GOS belongs to the lowest layer. A GOS belonging to an upper layer refers to a GOS belonging to the same layer or lower. In short, the GOS includes a plurality of layers that are spatially divided in a predetermined direction and each includes one or more SPCs. The encoding device and the decoding device encode or decode each SPC by referring to an SPC included in the same layer as the said SPC or in a layer lower than the said SPC.
[0146] Additionally, the encoding device and the decoding device sequentially encode or decode GOS within a world unit containing a plurality of GOS. The encoding device and the decoding device record or read out information indicating the order (direction) of encoding or decoding as metadata. In short, the encoding data includes information indicating the encoding order of a plurality of GOS.
[0147] In addition, the encoding device and the decoding device encode or decode two or more different spaces or GOS in parallel.
[0148] In addition, the encoding device and the decoding device encode or decode spatial information (coordinates, size, etc.) of space or GOS.
[0149] In addition, the encoding device and the decoding device encode or decode a space or GOS included in a specific space specified based on external information regarding its location or / and area size, such as GPS, route information, or scale.
[0150] An encoding device or a decoding device encodes or decodes spaces far from its location with lower priority compared to spaces close to it.
[0151] The encoding device sets one direction of the world according to the scale or purpose and encodes a GOS having a layer structure in that direction. In addition, the decoding device decodes the GOS having a layer structure in one direction of the world set according to the scale or purpose, starting from the lower layer.
[0152] The encoding device changes the extraction of feature points included in the space, the precision of object recognition, or the size of the spatial region in indoor and outdoor environments. However, the encoding device and the decoding device encode or decode indoor GOS and outdoor GOS with close coordinates adjacently within the world, and encode or decode by matching these identifiers.
[0153] (Embodiment 2)
[0154] When using encoded point cloud data in actual devices or services, it is desirable to transmit and receive necessary information according to the application to conserve network bandwidth. However, until now, such a function has not existed in the encoding structure of three-dimensional data, nor has there been an encoding method for it.
[0155] In this embodiment, a three-dimensional data encoding method and a three-dimensional data encoding device for providing a function to transmit and receive only necessary information according to the application in encoded data of a three-dimensional point cloud, and a three-dimensional data decoding method and a three-dimensional data decoding device for decoding said encoded data are described.
[0156] A voxel (VXL) having a feature quantity greater than a certain amount is defined as a feature voxel (FVXL), and a world (WLD) composed of FVXL is defined as a sparse world (SWLD). Figure 11 is a diagram showing an example of the configuration of a sparse world and a world. SWLD includes FGOS, which is a GOS composed of FVXL, FSPC, which is an SPC composed of FVXL, and FVLM, which is a VLM composed of FVXL. The data structure and prediction structure of FGOS, FSPC, and FVLM may be the same as those of GOS, SPC, and VLM.
[0157] A feature quantity is a feature quantity that represents the three-dimensional position information of VXL or the visible light information of the VXL position, and is a feature quantity that is frequently detected, especially at corners and edges of three-dimensional objects. Specifically, this feature quantity is a three-dimensional feature quantity or a visible light feature quantity as described below, but any other feature quantity that represents the position, luminance, or color information of VXL may be used.
[0158] As three-dimensional features, SHOT features (Signature of Histograms of Orientations), PFH features (Point Feature Histograms), or PPF features (Point Pair Feature) are used.
[0159] SHOT features are obtained by dividing the area around VXL, calculating the inner product of the reference point and the normal vectors of the divided regions, and histogramming it. These SHOT features are characterized by high dimensionality and high feature representation power.
[0160] PFH features are obtained by selecting multiple sets of two points near VXL, calculating normal vectors and the like from those two points, and histogramming them. Since these PFH features are histogram features, they are robust against some disturbances and have high feature representation power.
[0161] The PPF feature is a feature calculated using normal vectors, etc., for every two points of VXL. Since the entire VXL is used for this PPF feature, it is robust against occlusion.
[0162] In addition, SIFT (Scale-Invariant Feature Transform), SURF (Speeded Up Robust Features), or HOG (Histogram of Oriented Gradients), which use information such as image luminance gradient information as visible light features, can be used.
[0163] SWLD is generated by calculating the above feature quantity from each VXL of WLD and extracting FVXL. Here, SWLD may be updated whenever WLD is updated, or it may be updated periodically after a certain period of time, regardless of the timing of WLD updates.
[0164] SWLDs may be generated for each feature. For example, different SWLDs may be generated for each feature, such as SWLD1 based on SHOT features and SWLD2 based on SIFT features, and SWLDs may be distinguished and used according to their purpose. Additionally, the calculated features of each FVXL may be retained in each FVXL as feature information.
[0165] Next, we will explain how to use Sparse Worlds (SWLD). Since SWLDs contain only feature voxels (FVXLs), they generally have a smaller data size compared to WLDs, which contain all VXLs.
[0166] In applications that achieve a specific purpose using feature quantities, the read output time from the hard disk and the bandwidth and transmission time during network transmission can be reduced by using SWLD information instead of WLD. For example, WLD and SWLD can be maintained on a server as map information, and network bandwidth and transmission time can be reduced by converting the transmitted map information into WLD or SWLD upon a request from a client. Specific examples are provided below.
[0167] FIGS. 12 and FIGS. 13 are drawings illustrating examples of the use of SWLD and WLD. As shown in FIG. 12, when a client (1), which is an on-board device, requires map information for self-position determination purposes, the client (1) sends a request to the server to acquire map data for self-position estimation (S301). The server transmits SWLD to the client (1) in accordance with the acquisition request (S302). The client (1) performs self-position determination using the received SWLD (S303). At this time, the client (1) acquires VXL information around the client (1) using various methods such as a distance sensor like a rangefinder, a stereo camera, or a combination of multiple monocular cameras, and estimates self-position information from the obtained VXL information and SWLD. Here, self-position information includes the client (1)'s three-dimensional position information and direction, etc.
[0168] As shown in FIG. 13, when a client (2), which is a vehicle-mounted device, requires map information for map drawing purposes such as a three-dimensional map, the client (2) sends a request to the server to acquire map data for map drawing (S311). The server transmits a WLD to the client (2) in accordance with the acquisition request (S312). The client (2) performs map drawing using the received WLD (S313). At this time, the client (2) creates a rendered image using, for example, an image taken by itself with a visible light camera, etc., and the WLD acquired from the server, and draws the created image on a screen such as a car navigation system.
[0169] As described above, the server transmits SWLD to the client for applications that primarily require the feature quantities of each VXL, such as self-location estimation, and transmits WLD to the client when detailed VXL information is required, such as map rendering. This makes it possible to efficiently transmit and receive map data.
[0170] Furthermore, the client may independently determine whether it requires SWLD or WLD and request the server to transmit either SWLD or WLD. Additionally, the server may decide whether to transmit SWLD or WLD based on the client or network conditions.
[0171] Next, we will explain how to switch transmission between Sparse Worlds (SWLD) and Worlds (WLD).
[0172] Depending on the network band, it may be possible to switch whether to receive WLD or SWLD. FIG. 14 is a diagram showing an example of operation in this case. For example, when a low-speed network with limited available network bands, such as in an LTE (Long Term Evolution) environment, is used, the client accesses the server via the low-speed network (S321) and obtains SWLD as map information from the server (S322). On the other hand, when a high-speed network with available network bands, such as in a Wi-Fi (registered trademark) environment, is used, the client accesses the server via the high-speed network (S323) and obtains WLD from the server (S324). Thus, the client can obtain appropriate map information according to the network band of the client.
[0173] Specifically, the client receives SWLD via LTE outdoors, and acquires WLD via Wi-Fi (registered trademark) when entering indoors, such as within a facility. This enables the client to acquire more detailed map information of the indoor area.
[0174] In this way, the client may request WLD or SWLD from the server depending on the network range it uses. Alternatively, the client may transmit information indicating the network range it uses to the server, and the server may transmit data (WLD or SWLD) suitable for the client based on said information. Alternatively, the server may determine the client's network range and transmit data (WLD or SWLD) suitable for the client.
[0175] In addition, the reception of WLD or SWLD may be switched depending on the speed of movement. FIG. 15 is a diagram illustrating an example of operation in this case. For example, when the client is moving at high speed (S331), the client receives SWLD from the server (S332). On the other hand, when the client is moving at low speed (S333), the client receives WLD from the server (S334). By doing so, the client can acquire map information suitable for the speed while suppressing network bandwidth. Specifically, when the client is driving on a high-speed road, it can update approximate map information at an appropriate speed by receiving SWLD, which has a small amount of data. On the other hand, when the client is driving on a general road, it becomes possible to acquire more detailed map information by receiving WLD.
[0176] In this way, the client may request WLD or SWLD from the server according to its movement speed. Alternatively, the client may transmit information indicating its movement speed to the server, and the server may transmit data (WLD or SWLD) suitable for the client according to said information. Alternatively, the server may determine the client's movement speed and transmit data (WLD or SWLD) suitable for the client.
[0177] Furthermore, the client may acquire SWLD from the server first, and then acquire WLD of important areas from it. For example, when acquiring map data, the client may first obtain approximate map information as SWLD, narrow down the area where features such as buildings, signs, or people frequently appear, and subsequently acquire WLD of the narrowed area. By doing so, the client can acquire detailed information about necessary areas while limiting the amount of data received from the server.
[0178] Furthermore, the server may generate a different SWLD for each object from the WLD, and the client may receive each according to its purpose. This allows for the reduction of network bandwidth. For example, the server recognizes people or cars in advance from the WLD and generates SWLDs for people and cars. If the client wants to obtain information about nearby people, it receives the SWLD for people; if it wants to obtain information about cars, it receives the SWLD for cars. Additionally, the types of such SWLDs may be distinguished by information (such as flags or types) attached to headers.
[0179] Next, the configuration and operation flow of a three-dimensional data encoding device (e.g., a server) related to the present embodiment will be described. FIG. 16 is a block diagram of a three-dimensional data encoding device (400) related to the present embodiment. FIG. 17 is a flowchart of three-dimensional data encoding processing by the three-dimensional data encoding device (400).
[0180] The three-dimensional data encoding device (400) shown in FIG. 16 generates encoded three-dimensional data (413 and 414), which are encoded streams, by encoding input three-dimensional data (411). Here, the encoded three-dimensional data (413) is encoded three-dimensional data corresponding to WLD, and the encoded three-dimensional data (414) is encoded three-dimensional data corresponding to SWLD. The three-dimensional data encoding device (400) comprises an acquisition unit (401), an encoding region determination unit (402), a SWLD extraction unit (403), a WLD encoding unit (404), and a SWLD encoding unit (405).
[0181] As shown in FIG. 17, first, the acquisition unit (401) acquires input three-dimensional data (411), which is point cloud data in three-dimensional space (S401).
[0182] Next, the encoding area determination unit (402) determines the spatial area of the encoding target based on the spatial area where the point cloud data exists (S402).
[0183] Next, the SWLD extraction unit (403) defines the spatial region of the encoding target as WLD and calculates feature quantities from each VXL included in the WLD. Then, the SWLD extraction unit (403) extracts VXLs whose feature quantities are greater than or equal to a predetermined threshold, defines the extracted VXLs as FVXLs, and generates extracted three-dimensional data (412) by adding the FVXLs to the SWLD (S403). In short, extracted three-dimensional data (412) whose feature quantities are greater than or equal to the threshold is extracted from the input three-dimensional data (411).
[0184] Next, the WLD encoding unit (404) generates encoded three-dimensional data (413) corresponding to the WLD by encoding the input three-dimensional data (411) corresponding to the WLD (S404). At this time, the WLD encoding unit (404) adds information to the header of the encoded three-dimensional data (413) to distinguish whether the encoded three-dimensional data (413) is a stream containing the WLD.
[0185] Additionally, the SWLD encoding unit (405) generates encoded three-dimensional data (414) corresponding to the SWLD by encoding the extracted three-dimensional data (412) corresponding to the SWLD (S405). At this time, the SWLD encoding unit (405) adds information to the header of the encoded three-dimensional data (414) to distinguish that the encoded three-dimensional data (414) is a stream containing the SWLD.
[0186] Additionally, the processing order of the process for generating encoded three-dimensional data (413) and the process for generating encoded three-dimensional data (414) may be the opposite of the above. Also, some or all of these processes may be performed in parallel.
[0187] As information assigned to the header of the encoded three-dimensional data (413 and 414), a parameter named "world_type" is defined, for example. If world_type=0, it indicates that the stream contains WLD, and if world_type=1, it indicates that the stream contains SWLD. In addition, if multiple other types are defined, the assigned value may be increased, such as world_type=2. Also, a specific flag may be included on one side of the encoded three-dimensional data (413 and 414). For example, a flag indicating that the stream contains SWLD may be assigned to the encoded three-dimensional data (414). In this case, the decoder can determine whether the stream contains WLD or SWLD based on the presence or absence of the flag.
[0188] In addition, the encoding method used by the WLD encoding unit (404) when encoding the WLD and the encoding method used by the SWLD encoding unit (405) when encoding the SWLD may be different.
[0189] For example, in SWLD, since data is filtered out, there is a possibility that the correlation with surrounding data will be lower compared to WLD. Therefore, in the encoding method used for SWLD, inter-prediction may be prioritized over intra-prediction compared to the encoding method used for WLD.
[0190] In addition, the encoding method used for SWLD and the encoding method used for WLD may differ in the method of representing three-dimensional positions. For example, in SWLD, the three-dimensional position of FVXL may be represented by three-dimensional coordinates, and in WLD, the three-dimensional position may be represented by an octal tree as described later, or vice versa.
[0191] Additionally, the SWLD encoding unit (405) performs encoding such that the data size of the SWLD encoded three-dimensional data (414) becomes smaller than the data size of the WLD encoded three-dimensional data (413). For example, as described above, SWLD may have lower correlation between data compared to WLD. Consequently, the encoding efficiency is reduced, and the data size of the encoded three-dimensional data (414) may become larger than the data size of the WLD encoded three-dimensional data (413). Therefore, if the data size of the obtained encoded three-dimensional data (414) is larger than the data size of the WLD encoded three-dimensional data (413), the SWLD encoding unit (405) performs re-coding to regenerate the encoded three-dimensional data (414) with reduced data size.
[0192] For example, the SWLD extraction unit (403) regenerates extracted three-dimensional data (412) with reduced number of extracted feature points, and the SWLD encoding unit (405) encodes the extracted three-dimensional data (412). Alternatively, the degree of quantization in the SWLD encoding unit (405) may be made rougher. For example, in the octal tree structure described later, the degree of quantization can be made rougher by rounding the data of the lowest layer.
[0193] Additionally, if the SWLD encoding unit (405) cannot make the data size of the SWLD encoded three-dimensional data (414) smaller than the data size of the WLD encoded three-dimensional data (413), it is not necessary to generate the SWLD encoded three-dimensional data (414). Alternatively, the WLD encoded three-dimensional data (413) may be copied to the SWLD encoded three-dimensional data (414). In short, the WLD encoded three-dimensional data (413) may be used as is as the SWLD encoded three-dimensional data (414).
[0194] Next, the configuration and operation flow of a three-dimensional data decoding device (e.g., a client) related to the present embodiment will be described. FIG. 18 is a block diagram of a three-dimensional data decoding device (500) related to the present embodiment. FIG. 19 is a flowchart of three-dimensional data decoding processing by the three-dimensional data decoding device (500).
[0195] The three-dimensional data decoding device (500) shown in FIG. 18 generates decoded three-dimensional data (512 or 513) by decoding encoded three-dimensional data (511). Here, the encoded three-dimensional data (511) is, for example, encoded three-dimensional data (413 or 414) generated by the three-dimensional data encoding device (400).
[0196] This three-dimensional data decoding device (500) comprises an acquisition unit (501), a header interpretation unit (502), a WLD decoding unit (503), and a SWLD decoding unit (504).
[0197] As shown in FIG. 19, first, the acquisition unit (501) acquires encoded three-dimensional data (511) (S501). Next, the header interpretation unit (502) interprets the header of the encoded three-dimensional data (511) to determine whether the encoded three-dimensional data (511) is a stream containing WLD or a stream containing SWLD (S502). For example, the determination is performed by referencing the world_type parameter described above.
[0198] If the encoded three-dimensional data (511) is a stream containing WLD (Yes in S503), the WLD decoder (503) generates the decoded three-dimensional data (512) of WLD by decoding the encoded three-dimensional data (511) (S504). Meanwhile, if the encoded three-dimensional data (511) is a stream containing SWLD (No in S503), the SWLD decoder (504) generates the decoded three-dimensional data (513) of SWLD by decoding the encoded three-dimensional data (511) (S505).
[0199] In addition, similar to the encoding device, the decoding method used by the WLD decoding unit (503) when decoding WLD and the decoding method used by the SWLD decoding unit (504) when decoding SWLD may be different. For example, in the decoding method used for SWLD, the inter prediction among intra prediction and inter prediction may be prioritized over the decoding method used for WLD.
[0200] In addition, the method of representing three-dimensional positions may differ between the decoding method used in SWLD and the decoding method used in WLD. For example, in SWLD, the three-dimensional position of FVXL may be represented by three-dimensional coordinates, and in WLD, the three-dimensional position may be represented by an octal tree as described later, or vice versa.
[0201] Next, the octal tree representation, which is a method for representing three-dimensional positions, will be explained. VXL data included in three-dimensional data is converted into an octal tree structure and then encoded. FIG. 20 is a diagram showing an example of VXL in WLD. FIG. 21 is a diagram showing the octal tree structure of WLD shown in FIG. 20. In the example shown in FIG. 20, there are three (VXL1 to 3) VXLs (hereinafter referred to as effective VXLs) containing a point cloud. As shown in FIG. 21, the octal tree structure consists of nodes and leaves. Each node has a maximum of eight nodes or leaves. Each leaf contains VXL information. Here, among the leaves shown in FIG. 21, leaves 1, 2, and 3 represent VXL1, VXL2, and VXL3 shown in FIG. 20, respectively.
[0202] Specifically, each node and leaf corresponds to a three-dimensional position. Node 1 corresponds to the entire block shown in FIG. 20. The block corresponding to Node 1 is divided into eight blocks, and among the eight blocks, the block containing the valid VXL is set as a node, and the other blocks are set as leaves. The block corresponding to the node is again divided into eight nodes or leaves, and this process is repeated for as many levels of the tree structure. Also, all blocks in the lowest level are set as leaves.
[0203] Also, FIG. 22 is a diagram showing an example of SWLD generated from the WLD shown in FIG. 20. VXL1 and VXL2 shown in FIG. 20 are determined to be FVXL1 and FVXL2 as a result of feature extraction and are added to the SWLD. Meanwhile, VXL3 is not determined to be FVXL and is not included in the SWLD. FIG. 23 is a diagram showing the octal tree structure of the SWLD shown in FIG. 22. In the octal tree structure shown in FIG. 23, Leaf 3, which corresponds to VXL3 shown in FIG. 21, is deleted. As a result, Node 3 shown in FIG. 21 no longer has a valid VXL and is changed to a leaf. In this way, the number of leaves in the SWLD is generally smaller than the number of leaves in the WLD, and the encoded three-dimensional data of the SWLD is also smaller than the encoded three-dimensional data of the WLD.
[0204] Variations of the present embodiment will be described below.
[0205] For example, when a client such as an on-board device performs self-position estimation, it receives SWLD from a server and performs self-position estimation using SWLD. When performing obstacle detection, it may perform obstacle detection based on surrounding three-dimensional information acquired by itself using various methods such as a distance sensor such as a rangefinder, a stereo camera, or a combination of multiple monocular cameras.
[0206] Furthermore, it is generally difficult to include VXL data for flat areas in SWLDs. Therefore, for the detection of static obstacles, the server may maintain a subsampled world (subWLD) obtained by subsampling the WLD and transmit both the SWLD and the subWLD to the client. This allows for self-location estimation and obstacle detection on the client side while conserving network bandwidth.
[0207] In addition, when a client draws three-dimensional map data at high speed, it is sometimes more convenient for the map information to have a mesh structure. Therefore, the server may generate a mesh from the WLD and maintain it in advance as a mesh world (MWLD). For example, the client receives the MWLD when it requires a rough three-dimensional drawing, and receives the WLD when it requires a detailed three-dimensional drawing. This allows network bandwidth to be reduced.
[0208] Additionally, the server may set the VXL among each VXL whose feature quantity is above the threshold as FVXL, but it is not necessary to calculate FVXL using a different method. For example, the server may determine that VXL, VLM, SPC, or GOS constituting a signal or intersection, etc., are necessary for self-position estimation, driving assistance, or automatic driving, and may include them in SWLD as FVXL, FVLM, FSPC, FGOS. Also, the above determination may be performed manually. Additionally, FVXL obtained by the above method may be added to FVXL, etc., set based on the feature quantity. In short, the SWLD extraction unit (403) may also extract data corresponding to an object having a predetermined attribute from the input three-dimensional data (411) as extraction three-dimensional data (412).
[0209] In addition, the purpose required for their use may be labeled separately from the feature quantity. Also, the server may separately maintain FVXL required for self-position estimation of signals or intersections, driving assistance, or autonomous driving as an upper layer of SWLD (e.g., lane world).
[0210] Additionally, the server may add attributes to the VXL within the WLD at random access units or at predetermined units. The attributes include, for example, information indicating whether they are necessary or unnecessary for self-location estimation, or information indicating whether they are important as traffic information such as signals or intersections. Additionally, the attributes may include a correspondence relationship with features (intersections or roads, etc.) in lane information (GDF: Geographic Data Files, etc.).
[0211] In addition, the following methods may be used as a method for updating WLD or SWLD.
[0212] Update information indicating changes in people, construction, or street trees (for trucks), etc., is uploaded to the server as a point cloud or metadata. The server updates the WLD based on the upload, and then updates the SWLD using the updated WLD.
[0213] Additionally, if the client detects a discrepancy between the 3D information it generated and the 3D information received from the server during self-position estimation, it may send the 3D information it generated to the server along with an update notification. In this case, the server updates the SWLD using the WLD. If the SWLD is not updated, the server determines that the WLD itself is outdated.
[0214] Also, as header information for the encoded stream, information distinguishing whether it is a WLD or a SWLD is added, but if there are multiple types of worlds, such as mesh worlds or rain worlds, information distinguishing them may be added to the header information. Also, if there are multiple SWLDs with different feature quantities, information distinguishing each of them may be added to the header information.
[0215] Additionally, although the SWLD is composed of FVXLs, it may include VXLs that are not determined to be FVXLs. For example, the SWLD may include adjacent VXLs used when calculating the feature quantity of an FVXL. Thus, even if feature quantity information is not attached to each FVXL of the SWLD, the client can calculate the feature quantity of an FVXL when receiving the SWLD. Furthermore, in that case, the SWLD may include information to distinguish whether each VXL is an FVXL or a VXL.
[0216] As described above, the three-dimensional data encoding device (400) extracts extracted three-dimensional data (412) (second three-dimensional data) in which the feature quantity is greater than or equal to a threshold from input three-dimensional data (411) (first three-dimensional data), and generates encoded three-dimensional data (414) (first encoded three-dimensional data) by encoding the extracted three-dimensional data (412).
[0217] According to this, the three-dimensional data encoding device (400) generates encoded three-dimensional data (414) in which the feature quantity is greater than or equal to the threshold. By doing so, the amount of data can be reduced compared to the case where the input three-dimensional data (411) is encoded as is. Therefore, the three-dimensional data encoding device (400) can reduce the amount of data transmitted.
[0218] Additionally, the three-dimensional data encoding device (400) also generates encoded three-dimensional data (413) (second encoded three-dimensional data) by encoding the input three-dimensional data (411).
[0219] According to this, the three-dimensional data encoding device (400) can selectively transmit encoded three-dimensional data (413) and encoded three-dimensional data (414) depending on the intended use, for example.
[0220] Also, the extracted three-dimensional data (412) is encoded by a first encoding method, and the input three-dimensional data (411) is encoded by a second encoding method different from the first encoding method.
[0221] According to this, the three-dimensional data encoding device (400) can use an encoding method suitable for the input three-dimensional data (411) and the extracted three-dimensional data (412), respectively.
[0222] Also, in the first encoding method, among intra prediction and inter prediction, inter prediction takes precedence over the second encoding method.
[0223] According to this, the three-dimensional data encoding device (400) can increase the priority of inter-prediction for extracted three-dimensional data (412) where the correlation between adjacent data is likely to be low.
[0224] In addition, the method of representing three-dimensional positions differs between the first encoding method and the second encoding method. For example, in the second encoding method, the three-dimensional position is represented by an octal tree, and in the first encoding method, the three-dimensional position is represented by three-dimensional coordinates.
[0225] According to this, the three-dimensional data encoding device (400) can use a more suitable method of representing three-dimensional positions for three-dimensional data with different data numbers (number of VXL or FVXL).
[0226] Also, at least one of the encoded three-dimensional data (413 and 414) includes an identifier indicating whether the encoded three-dimensional data is encoded three-dimensional data obtained by encoding the input three-dimensional data (411) or encoded three-dimensional data obtained by encoding a portion of the input three-dimensional data (411). In short, the identifier indicates whether the encoded three-dimensional data is encoded three-dimensional data (413) of WLD or encoded three-dimensional data (414) of SWLD.
[0227] According to this, the decoding device can easily determine whether the acquired encoded three-dimensional data is encoded three-dimensional data (413) or encoded three-dimensional data (414).
[0228] Additionally, the three-dimensional data encoding device (400) encodes the extracted three-dimensional data (412) such that the amount of data of the encoded three-dimensional data (414) becomes smaller than the amount of data of the encoded three-dimensional data (413).
[0229] According to this, the three-dimensional data encoding device (400) can make the amount of data of the encoded three-dimensional data (414) smaller than the amount of data of the encoded three-dimensional data (413).
[0230] Additionally, the three-dimensional data encoding device (400) also extracts data corresponding to an object having predetermined attributes from the input three-dimensional data (411) as extracted three-dimensional data (412). For example, an object having predetermined attributes is an object required for self-position estimation, driving assistance, or automatic driving, and is a signal or an intersection.
[0231] According to this, the three-dimensional data encoding device (400) can generate encoded three-dimensional data (414) containing data required by the decoding device.
[0232] Also, the three-dimensional data encoding device (400) (server) also transmits one side of the encoded three-dimensional data (413 and 414) to the client depending on the client state.
[0233] According to this, the three-dimensional data encoding device (400) can transmit appropriate data depending on the state of the client.
[0234] In addition, the client's status includes the client's communication status (e.g., network bandwidth) or the client's movement speed.
[0235] Additionally, the three-dimensional data encoding device (400) also transmits one of the encoded three-dimensional data (413 and 414) to the client according to the client's request.
[0236] According to this, the three-dimensional data encoding device (400) can transmit appropriate data according to the client's request.
[0237] Also, the three-dimensional data decoding device (500) related to the present embodiment decodes encoded three-dimensional data (413 or 414) generated by the three-dimensional data encoding device (400).
[0238] In summary, the three-dimensional data decoding device (500) decodes encoded three-dimensional data (414) obtained by encoding extracted three-dimensional data (412), in which the feature quantity extracted from input three-dimensional data (411) is greater than or equal to a threshold, using a first decoding method. Additionally, the three-dimensional data decoding device (500) decodes encoded three-dimensional data (413) obtained by encoding input three-dimensional data (411), using a second decoding method different from the first decoding method.
[0239] According to this, the three-dimensional data decoding device (500) can selectively receive encoded three-dimensional data (414) and encoded three-dimensional data (413), which are encoded data in which the feature quantity is greater than or equal to a threshold, for example, depending on the intended use. By doing so, the three-dimensional data decoding device (500) can reduce the amount of data transmitted. Additionally, the three-dimensional data decoding device (500) can use a decoding method suitable for the input three-dimensional data (411) and the extracted three-dimensional data (412), respectively.
[0240] Also, in the first decoding method, among intra prediction and inter prediction, inter prediction takes precedence over the second decoding method.
[0241] According to this, the three-dimensional data decoding device (500) can increase the priority of inter-prediction for extracted three-dimensional data where the correlation between adjacent data is likely to be low.
[0242] In addition, the method of representing three-dimensional positions differs between the first decoding method and the second decoding method. For example, in the second decoding method, the three-dimensional position is represented by an octal tree, while in the first decoding method, the three-dimensional position is represented by three-dimensional coordinates.
[0243] According to this, the three-dimensional data decoding device (500) can use a more suitable method of representing three-dimensional positions for three-dimensional data with different data numbers (number of VXL or FVXL).
[0244] Additionally, at least one of the encoded three-dimensional data (413 and 414) includes an identifier indicating whether the encoded three-dimensional data is encoded three-dimensional data obtained by encoding the input three-dimensional data (411) or encoded three-dimensional data obtained by encoding a portion of the input three-dimensional data (411). The three-dimensional data decoding device (500) identifies the encoded three-dimensional data (413 and 414) by referring to the identifier.
[0245] According to this, the three-dimensional data decoding device (500) can easily determine whether the acquired encoded three-dimensional data is encoded three-dimensional data (413) or encoded three-dimensional data (414).
[0246] Additionally, the three-dimensional data decoding device (500) also notifies the server of the status of the client (three-dimensional data decoding device (500)). Depending on the status of the client, the three-dimensional data decoding device (500) receives one of the encoded three-dimensional data (413 and 414) transmitted from the server.
[0247] According to this, the three-dimensional data decoding device (500) can receive appropriate data depending on the state of the client.
[0248] In addition, the client's status includes the client's communication status (e.g., network bandwidth) or the client's movement speed.
[0249] Additionally, the three-dimensional data decoding device (500) also requests one side of the encoded three-dimensional data (413 and 414) from the server, and receives one side of the encoded three-dimensional data (413 and 414) transmitted from the server in accordance with the request.
[0250] According to this, the three-dimensional data decoding device (500) can receive appropriate data according to the purpose.
[0251] (Embodiment 3)
[0252] In this embodiment, a method for transmitting and receiving three-dimensional data between vehicles is described. For example, the transmission and reception of three-dimensional data between a self-vehicle and surrounding vehicles is performed.
[0253] FIG. 24 is a block diagram of a three-dimensional data creation device (620) related to the present embodiment. This three-dimensional data creation device (620) is included in, for example, a vehicle, and creates a denser third three-dimensional data (636) by combining the received second three-dimensional data (635) with the first three-dimensional data (632) created by the three-dimensional data creation device (620).
[0254] This three-dimensional data creation device (620) comprises a three-dimensional data creation unit (621), a request range determination unit (622), a search unit (623), a receiving unit (624), a decoding unit (625), and a synthesis unit (626).
[0255] First, the three-dimensional data creation unit (621) creates first three-dimensional data (632) using sensor information (631) detected by a sensor equipped with the vehicle. Next, the required range determination unit (622) determines a required range, which is a three-dimensional spatial range where data is insufficient, among the created first three-dimensional data (632).
[0256] Next, the search unit (623) searches for surrounding vehicles that possess three-dimensional data of the required range and transmits required range information (633) indicating the required range to specific surrounding vehicles through the search. Next, the receiving unit (624) receives encoded three-dimensional data (634), which is an encoded stream of the required range, from surrounding vehicles (S624). Additionally, the search unit (623) may indiscriminately send requests to all vehicles existing within a specific range and receive encoded three-dimensional data (634) from those that responded. Furthermore, the search unit (623) may not be limited to vehicles but may send requests to objects such as signal lights or markers and receive encoded three-dimensional data (634) from said objects.
[0257] Next, the decoding unit (625) obtains second three-dimensional data (635) by decoding the received encoded three-dimensional data (634). Next, the synthesis unit (626) creates a denser third three-dimensional data (636) by synthesizing the first three-dimensional data (632) and the second three-dimensional data (635).
[0258] Next, the configuration and operation of the three-dimensional data transmission device (640) related to the present embodiment will be described. FIG. 25 is a block diagram of the three-dimensional data transmission device (640).
[0259] A three-dimensional data transmission device (640) is, for example, included in the surrounding vehicle described above, processes the fifth three-dimensional data (652) created by the surrounding vehicle into the sixth three-dimensional data (654) required by the vehicle, generates encoded three-dimensional data (634) by encoding the sixth three-dimensional data (654), and transmits the encoded three-dimensional data (634) to the vehicle.
[0260] The three-dimensional data transmission device (640) comprises a three-dimensional data creation unit (641), a receiving unit (642), an extraction unit (643), an encoding unit (644), and a transmitting unit (645).
[0261] First, the three-dimensional data creation unit (641) creates the fifth three-dimensional data (652) using sensor information (651) detected by a sensor equipped with a surrounding vehicle. Next, the receiving unit (642) receives the request range information (633) transmitted from the vehicle.
[0262] Next, the extraction unit (643) processes the fifth three-dimensional data (652) into the sixth three-dimensional data (654) by extracting the three-dimensional data of the required range indicated in the required range information (633) from the fifth three-dimensional data (652). Next, the encoding unit (644) generates encoded three-dimensional data (634), which is an encoded stream, by encoding the sixth three-dimensional data (654). Then, the transmission unit (645) transmits the encoded three-dimensional data (634) to the vehicle.
[0263] Additionally, although an example is described here in which a vehicle is equipped with a three-dimensional data creation device (620) and a surrounding vehicle is equipped with a three-dimensional data transmission device (640), each vehicle may have the functions of a three-dimensional data creation device (620) and a three-dimensional data transmission device (640).
[0264] (Embodiment 4)
[0265] In this embodiment, the operation of the ideal system in self-position estimation based on a three-dimensional map is described.
[0266] Applications such as the autonomous driving of vehicles, robots, or flying vehicles like drones are expected to expand in the future. As an example of a means to realize such autonomous movement, there is a method in which a moving object estimates its own position within a three-dimensional map (self-position estimation) and drives according to the map.
[0267] Self-position estimation can be achieved by matching a three-dimensional map with three-dimensional information around the vehicle (hereinafter referred to as self-detected three-dimensional data) acquired by sensors such as a rangefinder (LiDAR, etc.) or a stereo camera mounted on the vehicle, and estimating the position of the vehicle within the three-dimensional map.
[0268] The three-dimensional map may include not only three-dimensional point clouds, such as the HD map proposed by HERE, but also two-dimensional map data such as shape information of roads and intersections, or real-time changing information such as traffic congestion and accidents. The three-dimensional map may be constructed from multiple layers, such as three-dimensional data, two-dimensional data, and real-time changing metadata, and the device may acquire or reference only the necessary data.
[0269] The point cloud data may be the SWLD described above, or may include point cloud data that is not a feature point. In addition, the transmission and reception of the point cloud data is performed based on one or multiple random access units.
[0270] The following method may be used as a matching method between a three-dimensional map and three-dimensional data for self-deviation detection. For example, the device compares the shapes of point clouds in each other's point clouds and determines that areas with high similarity between feature points are the same location. Additionally, if the three-dimensional map is composed of SWLD, the device performs matching by comparing the feature points constituting the SWLD with the three-dimensional feature points extracted from the self-deviation detection three-dimensional data.
[0271] Here, in order to perform magnetic position estimation with high precision, it is necessary to (A) acquire a three-dimensional map and magnetic vehicle detection three-dimensional data, and also (B) ensure that their precision satisfies a predetermined standard. However, in the following abnormal cases, (A) or (B) cannot be satisfied.
[0272] (1) A three-dimensional map cannot be obtained via communication.
[0273] (2) The three-dimensional map does not exist, or the three-dimensional map was obtained but is damaged.
[0274] (3) The precision of the three-dimensional data generated from the vehicle detection is insufficient because the vehicle's sensor is malfunctioning or due to bad weather.
[0275] Actions to handle these abnormal cases are described below. Although the actions are described below using a car as an example, the following methods can be applied to all autonomously moving animal bodies, such as robots or drones.
[0276] Hereinafter, the configuration and operation of a three-dimensional information processing device related to the present embodiment for responding to abnormal cases in a three-dimensional map or self-detection three-dimensional data will be described. FIG. 26 is a block diagram showing an example of the configuration of a three-dimensional information processing device (700) related to the present embodiment.
[0277] The three-dimensional information processing device (700) is mounted on an animal body, such as a car, for example. As shown in FIG. 26, the three-dimensional information processing device (700) comprises a three-dimensional map acquisition unit (701), a self-vehicle detection data acquisition unit (702), an abnormal case determination unit (703), a response operation determination unit (704), and an operation control unit (705).
[0278] Additionally, the three-dimensional information processing device (700) may be equipped with an unillustrated two-dimensional or one-dimensional sensor for detecting structures or animal bodies around the vehicle, such as a camera for acquiring two-dimensional images or a sensor for one-dimensional data using ultrasound or a laser. Additionally, the three-dimensional information processing device (700) may be equipped with a communication unit (not illustrated) for acquiring a three-dimensional map via a mobile communication network such as 4G or 5G, or via vehicle-to-vehicle communication or road-to-vehicle communication.
[0279] The three-dimensional map acquisition unit (701) acquires a three-dimensional map (711) near the driving path. For example, the three-dimensional map acquisition unit (701) acquires the three-dimensional map (711) via a mobile communication network, or communication between vehicles or communication between a road and a vehicle.
[0280] Next, the vehicle detection data acquisition unit (702) acquires vehicle detection three-dimensional data (712) based on sensor information. For example, the vehicle detection data acquisition unit (702) generates vehicle detection three-dimensional data (712) based on sensor information acquired by a sensor equipped with the vehicle.
[0281] Next, the abnormal case determination unit (703) detects an abnormal case by performing a predetermined check on at least one of the acquired three-dimensional map (711) and the self-vehicle detection three-dimensional data (712). In short, the abnormal case determination unit (703) determines whether at least one of the acquired three-dimensional map (711) and the self-vehicle detection three-dimensional data (712) is abnormal.
[0282] When an abnormal case is detected, the response operation determination unit (704) determines a response operation for the abnormal case. Next, the operation control unit (705) controls the operation of each processing unit required for the implementation of the response operation, such as the three-dimensional map acquisition unit (701).
[0283] Meanwhile, if no abnormal case is detected, the three-dimensional information processing device (700) terminates the processing.
[0284] Additionally, the three-dimensional information processing device (700) performs self-position estimation of the vehicle having the three-dimensional information processing device (700) using the three-dimensional map (711) and self-vehicle detection three-dimensional data (712). Next, the three-dimensional information processing device (700) automatically drives the vehicle using the result of the self-position estimation.
[0285] In this way, the three-dimensional information processing device (700) acquires map data (three-dimensional map (711)) containing first three-dimensional position information through a communication channel. For example, the first three-dimensional position information is encoded using a subspace having three-dimensional coordinate information as a unit, and each includes a plurality of random access units, each of which is a set of one or more subspaces and each can be independently decoded. For example, the first three-dimensional position information is data (SWLD) in which feature points are encoded such that the three-dimensional feature quantity is greater than or equal to a predetermined threshold.
[0286] Additionally, the three-dimensional information processing device (700) generates second three-dimensional position information (self-detected three-dimensional data (712)) from information detected by a sensor. Next, the three-dimensional information processing device (700) determines whether the first three-dimensional position information or the second three-dimensional position information is abnormal by performing an abnormality determination process on the first three-dimensional position information or the second three-dimensional position information.
[0287] The three-dimensional information processing device (700) determines a countermeasure action for the abnormality when the first three-dimensional position information or the second three-dimensional position information is determined to be abnormal. Next, the three-dimensional information processing device (700) performs control necessary for the execution of the countermeasure action.
[0288] Thus, the three-dimensional information processing device (700) can detect an abnormality in the first three-dimensional position information or the second three-dimensional position information and perform a countermeasure operation.
[0289] (Embodiment 5)
[0290] In this embodiment, a method for transmitting three-dimensional data to a subsequent vehicle, etc., is described.
[0291] FIG. 27 is a block diagram showing an example of the configuration of a three-dimensional data creation device (810) related to the present embodiment. This three-dimensional data creation device (810) is, for example, mounted on a vehicle. The three-dimensional data creation device (810) creates and accumulates three-dimensional data while transmitting and receiving three-dimensional data with an external traffic monitoring cloud, a preceding vehicle, or a following vehicle.
[0292] The three-dimensional data creation device (810) comprises a data receiving unit (811), a communication unit (812), a receiving control unit (813), a format conversion unit (814), a plurality of sensors (815), a three-dimensional data creation unit (816), a three-dimensional data synthesis unit (817), a three-dimensional data accumulation unit (818), a communication unit (819), a transmission control unit (820), a format conversion unit (821), and a data transmission unit (822).
[0293] The data receiving unit (811) receives three-dimensional data (831) from a traffic monitoring cloud or a utility vehicle. The three-dimensional data (831) includes information such as a point cloud, visible light image, depth information, sensor location information, or speed information, for example, including an area that cannot be detected by the vehicle's sensor (815).
[0294] The communication unit (812) communicates with the traffic monitoring cloud or the utility vehicle and transmits data transmission requests, etc., to the traffic monitoring cloud or the utility vehicle.
[0295] The receiving control unit (813) exchanges information such as a corresponding format with the communication location through the communication unit (812) and establishes communication with the communication location.
[0296] The format conversion unit (814) generates three-dimensional data (832) by performing format conversion, etc. on the three-dimensional data (831) received by the data receiving unit (811). Additionally, if the three-dimensional data (831) is compressed or encoded, the format conversion unit (814) performs expansion or decoding processing.
[0297] A plurality of sensors (815) are a group of sensors that acquire external information of the vehicle, such as LiDAR, a visible light camera, or an infrared camera, and generate sensor information (833). For example, if the sensor (815) is a laser sensor such as LiDAR, the sensor information (833) is three-dimensional data such as a point cloud (point cloud data). Additionally, the sensors (815) do not have to be multiple.
[0298] The three-dimensional data generation unit (816) generates three-dimensional data (834) from sensor information (833). The three-dimensional data (834) includes information such as, for example, a point cloud, a visible light image, depth information, sensor position information, or velocity information.
[0299] The three-dimensional data synthesis unit (817) synthesizes three-dimensional data (832) created by a traffic monitoring cloud or a utility vehicle, etc., into three-dimensional data (834) created based on the sensor information (833) of the vehicle, thereby constructing three-dimensional data (835) that includes the space in front of the utility vehicle that cannot be detected by the sensor (815) of the vehicle.
[0300] The three-dimensional data accumulation unit (818) accumulates the generated three-dimensional data (835), etc.
[0301] The communication unit (819) communicates with the traffic monitoring cloud or the subsequent vehicle and transmits data transmission requests, etc., to the traffic monitoring cloud or the subsequent vehicle.
[0302] The transmission control unit (820) establishes communication with the communication source by exchanging information such as a corresponding format through the communication unit (819). Additionally, the transmission control unit (820) determines a transmission area, which is the space of the three-dimensional data to be transmitted, based on the three-dimensional data construction information of the three-dimensional data (832) generated by the three-dimensional data synthesis unit (817) and the data transmission request from the communication source.
[0303] Specifically, the transmission control unit (820) determines a transmission area including the space in front of the vehicle that cannot be detected by the sensors of the subsequent vehicle, in accordance with a data transmission request from a traffic monitoring cloud or a subsequent vehicle. Additionally, the transmission control unit (820) determines the transmission area by determining whether there is an update to the transmission-possible space or the space where transmission has been completed, based on three-dimensional data construction information. For example, the transmission control unit (820) determines the transmission area as the area designated by the data transmission request and the area where the corresponding three-dimensional data (835) exists. Then, the transmission control unit (820) notifies the format corresponding to the communication location and the transmission area to the format conversion unit (821).
[0304] The format conversion unit (821) generates three-dimensional data (837) by converting the three-dimensional data (836) of the transmission area among the three-dimensional data (835) accumulated in the three-dimensional data accumulation unit (818) into a format corresponding to the receiving side. Additionally, the format conversion unit (821) may reduce the amount of data by compressing or encoding the three-dimensional data (837).
[0305] The data transmission unit (822) transmits three-dimensional data (837) to a traffic monitoring cloud or a subsequent vehicle. This three-dimensional data (837) includes information such as a point cloud in front of the vehicle, visible light images, depth information, or sensor location information, for example, including an area that is blind spot of the subsequent vehicle.
[0306] In addition, although an example is described here in which format conversion is performed in the format conversion unit (814 and 821), format conversion is not required.
[0307] With this configuration, the three-dimensional data creation device (810) acquires three-dimensional data (831) from an external source in an area that cannot be detected by the sensor (815) of the vehicle, and generates three-dimensional data (835) by synthesizing the three-dimensional data (831) and three-dimensional data (834) based on sensor information (833) detected by the sensor (815) of the vehicle. Thus, the three-dimensional data creation device (810) can generate three-dimensional data in an area that cannot be detected by the sensor (815) of the vehicle.
[0308] Additionally, the three-dimensional data creation device (810) can transmit three-dimensional data including the space in front of the vehicle that cannot be detected by the sensors of the subsequent vehicle, in response to a data transmission request from the traffic monitoring cloud or the subsequent vehicle, to the traffic monitoring cloud or the subsequent vehicle.
[0309] (Embodiment 6)
[0310] In Embodiment 5, an example was described in which a client device, such as a vehicle, transmits three-dimensional data to another vehicle or a server, such as a traffic monitoring cloud. In this embodiment, the client device transmits sensor information obtained from a sensor to a server or another client device.
[0311] First, the configuration of the system related to the present embodiment will be described. FIG. 28 is a diagram showing the configuration of a system for transmitting and receiving three-dimensional map and sensor information related to the present embodiment. This system includes a server (901) and client devices (902A and 902B). Additionally, if the client devices (902A and 902B) are not specifically distinguished, they are also referred to as client devices (902).
[0312] The client device (902) is, for example, an onboard device mounted on a mobile body such as a vehicle. The server (901) is, for example, a traffic monitoring cloud and can communicate with multiple client devices (902).
[0313] The server (901) transmits a three-dimensional map composed of a point cloud to the client device (902). Additionally, the composition of the three-dimensional map is not limited to a point cloud and may display other three-dimensional data, such as a mesh structure.
[0314] The client device (902) transmits sensor information acquired by the client device (902) to the server (901). The sensor information includes, for example, at least one of LiDAR acquisition information, visible light image, infrared image, depth image, sensor position information, and velocity information.
[0315] Data transmitted and received between the server (901) and the client device (902) may be compressed to reduce data, or may remain uncompressed to maintain data precision. When compressing data, a three-dimensional compression method based on, for example, an octal tree structure may be used for point clouds. Additionally, a two-dimensional image compression method may be used for visible light images, infrared images, and depth images. A two-dimensional image compression method is, for example, MPEG-4 AVC or HEVC standardized by MPEG.
[0316] Additionally, the server (901) transmits a three-dimensional map managed by the server (901) to the client device (902) in response to a request for transmission of a three-dimensional map from the client device (902). Furthermore, the server (901) may transmit the three-dimensional map without waiting for a request for transmission of the three-dimensional map from the client device (902). For example, the server (901) may broadcast the three-dimensional map to one or more client devices (902) located in a predetermined space. Additionally, the server (901) may transmit a three-dimensional map suitable for the location of the client device (902) at regular intervals to the client device (902) that has received a transmission request once. Additionally, the server (901) may transmit the three-dimensional map to the client device (902) whenever the three-dimensional map managed by the server (901) is updated.
[0317] The client device (902) sends a request to transmit a three-dimensional map to the server (901). For example, if the client device (902) wants to perform self-position estimation while driving, the client device (902) sends a request to transmit a three-dimensional map to the server (901).
[0318] Additionally, in the following cases, the client device (902) may send a request to the server (901) to transmit a three-dimensional map. If the three-dimensional map maintained by the client device (902) is old, the client device (902) may send a request to the server (901) to transmit a three-dimensional map. For example, if a certain period of time has elapsed since the client device (902) acquired the three-dimensional map, the client device (902) may send a request to the server (901) to transmit a three-dimensional map.
[0319] Before a certain time when the client device (902) comes out of the space represented by the three-dimensional map maintained by the client device (902), the client device (902) may send a request to transmit the three-dimensional map to the server (901). For example, if the client device (902) is within a predetermined distance from the boundary of the space represented by the three-dimensional map maintained by the client device (902), the client device (902) may send a request to transmit the three-dimensional map to the server (901). Also, if the movement path and movement speed of the client device (902) are known, the time when the client device (902) comes out of the space represented by the three-dimensional map maintained by the client device (902) may be predicted based on these.
[0320] When the error in position alignment between the three-dimensional data created by the client device (902) from sensor information and the three-dimensional map is greater than a certain amount, the client device (902) may send a request to transmit the three-dimensional map to the server (901).
[0321] The client device (902) transmits sensor information to the server (901) in response to a request for transmission of sensor information transmitted from the server (901). Additionally, the client device (902) may send sensor information to the server (901) without waiting for a request for transmission of sensor information from the server (901). For example, once the client device (902) receives a request for transmission of sensor information from the server (901), it may periodically transmit sensor information to the server (901) for a certain period. Furthermore, if the error in position alignment between the three-dimensional data created by the client device (902) based on sensor information and the three-dimensional map obtained from the server (901) exceeds a certain level, the client device (902) may determine that there is a possibility of a change occurring in the three-dimensional map around the client device (902) and transmit sensor information to the server (901) along with this information.
[0322] The server (901) sends a request to transmit sensor information to the client device (902). For example, the server (901) receives location information of the client device (902), such as GPS, from the client device (902). When the server (901) determines, based on the location information of the client device (902), that the client device (902) is approaching a space with little information in the three-dimensional map managed by the server (901), it sends a request to transmit sensor information to the client device (902) to create a new three-dimensional map. Additionally, the server (901) may send a request to transmit sensor information when it wants to update the three-dimensional map, when it wants to check road conditions such as snowfall or disasters, or when it wants to check traffic congestion or accident situations.
[0323] Additionally, the client device (902) may set the amount of data of sensor information to be transmitted to the server (901) according to the communication status or bandwidth at the time of receiving the request to transmit sensor information received from the server (901). Setting the amount of data of sensor information to be transmitted to the server (901) means, for example, increasing or decreasing the data itself, or appropriately selecting a compression method.
[0324] FIG. 29 is a block diagram showing an example of the configuration of a client device (902). The client device (902) receives a three-dimensional map, such as a point cloud, from a server (901) and estimates the self-position of the client device (902) from three-dimensional data created based on sensor information of the client device (902). In addition, the client device (902) transmits the acquired sensor information to the server (901).
[0325] The client device (902) comprises a data receiving unit (1011), a communication unit (1012), a receiving control unit (1013), a format conversion unit (1014), a plurality of sensors (1015), a three-dimensional data creation unit (1016), a three-dimensional image processing unit (1017), a three-dimensional data accumulation unit (1018), a format conversion unit (1019), a communication unit (1020), a transmission control unit (1021), and a data transmission unit (1022).
[0326] The data receiving unit (1011) receives a three-dimensional map (1031) from the server (901). The three-dimensional map (1031) is data including a point cloud such as a WLD or SWLD. The three-dimensional map (1031) may include either compressed data or uncompressed data.
[0327] The communication unit (1012) communicates with the server (901) and transmits a data transmission request (e.g., a transmission request for a three-dimensional map) to the server (901).
[0328] The receiving control unit (1013) exchanges information such as a corresponding format with the communication unit (1012) through the communication unit (1012) and establishes communication with the communication unit.
[0329] The format conversion unit (1014) generates a three-dimensional map (1032) by performing format conversion, etc., on the three-dimensional map (1031) received by the data receiving unit (1011). Additionally, if the three-dimensional map (1031) is compressed or encoded, the format conversion unit (1014) performs expansion or decoding processing. Furthermore, if the three-dimensional map (1031) is uncompressed data, the format conversion unit (1014) does not perform expansion or decoding processing.
[0330] A plurality of sensors (1015) are a group of sensors that acquire external information of a vehicle equipped with a client device (902), such as LiDAR, a visible light camera, an infrared camera, or a depth sensor, and generate sensor information (1033). For example, if the sensor (1015) is a laser sensor such as LiDAR, the sensor information (1033) is three-dimensional data such as a point cloud (point cloud data). Additionally, the sensors (1015) do not have to be multiple.
[0331] The three-dimensional data creation unit (1016) creates three-dimensional data (1034) around the vehicle based on sensor information (1033). For example, the three-dimensional data creation unit (1016) creates point cloud data with color information around the vehicle using information acquired by LiDAR and a visible light image obtained by a visible light camera.
[0332] The three-dimensional image processing unit (1017) performs self-position estimation processing of the vehicle using a three-dimensional map (1032), such as a received point cloud, and three-dimensional data (1034) around the vehicle generated from sensor information (1033). Additionally, the three-dimensional image processing unit (1017) may create three-dimensional data (1035) around the vehicle by synthesizing the three-dimensional map (1032) and the three-dimensional data (1034), and perform self-position estimation processing using the created three-dimensional data (1035).
[0333] The three-dimensional data accumulation unit (1018) accumulates a three-dimensional map (1032), three-dimensional data (1034), and three-dimensional data (1035), etc.
[0334] The format conversion unit (1019) generates sensor information (1037) by converting sensor information (1033) into a format corresponding to the receiving side. Additionally, the format conversion unit (1019) may reduce the amount of data by compressing or encoding the sensor information (1037). Additionally, the format conversion unit (1019) may omit processing if format conversion is not necessary. Additionally, the format conversion unit (1019) may control the amount of data transmitted according to the specification of the transmission range.
[0335] The communication unit (1020) communicates with the server (901) and receives data transmission requests (requests for transmission of sensor information), etc. from the server (901).
[0336] The transmission control unit (1021) establishes communication by exchanging information such as a corresponding format with a communication partner through the communication unit (1020).
[0337] The data transmission unit (1022) transmits sensor information (1037) to the server (901). The sensor information (1037) includes information obtained by a plurality of sensors (1015), such as information obtained by LiDAR, a luminance image obtained by a visible light camera, an infrared image obtained by an infrared camera, a depth image obtained by a depth sensor, sensor position information, and speed information.
[0338] Next, the configuration of the server (901) is described. FIG. 30 is a block diagram showing an example of the configuration of the server (901). The server (901) receives sensor information transmitted from the client device (902) and creates three-dimensional data based on the received sensor information. The server (901) updates the three-dimensional map managed by the server (901) using the created three-dimensional data. In addition, the server (901) transmits the updated three-dimensional map to the client device (902) in response to a request for transmission of the three-dimensional map from the client device (902).
[0339] The server (901) is equipped with a data receiving unit (1111), a communication unit (1112), a receiving control unit (1113), a format conversion unit (1114), a three-dimensional data creation unit (1116), a three-dimensional data synthesis unit (1117), a three-dimensional data accumulation unit (1118), a format conversion unit (1119), a communication unit (1120), a transmission control unit (1121), and a data transmission unit (1122).
[0340] The data receiving unit (1111) receives sensor information (1037) from the client device (902). The sensor information (1037) includes, for example, information acquired by LiDAR, a luminance image acquired by a visible light camera, an infrared image acquired by an infrared camera, a depth image acquired by a depth sensor, sensor position information, and speed information.
[0341] The communication unit (1112) communicates with the client device (902) and transmits a data transmission request (e.g., a request to transmit sensor information) to the client device (902).
[0342] The receiving control unit (1113) establishes communication by exchanging information such as a corresponding format with the communication source through the communication unit (1112).
[0343] The format conversion unit (1114) generates sensor information (1132) by performing expansion or decoding processing when the received sensor information (1037) is compressed or encoded. Additionally, the format conversion unit (1114) does not perform expansion or decoding processing when the sensor information (1037) is uncompressed data.
[0344] The three-dimensional data creation unit (1116) creates three-dimensional data (1134) around the client device (902) based on sensor information (1132). For example, the three-dimensional data creation unit (1116) creates point cloud data with color information around the client device (902) using information acquired by LiDAR and a visible light image obtained by a visible light camera.
[0345] The three-dimensional data synthesis unit (1117) updates the three-dimensional map (1135) by synthesizing the three-dimensional data (1134) created based on the sensor information (1132) into the three-dimensional map (1135) managed by the server (901).
[0346] The three-dimensional data accumulation unit (1118) accumulates three-dimensional maps (1135), etc.
[0347] The format conversion unit (1119) generates a three-dimensional map (1031) by converting the three-dimensional map (1135) into a format corresponding to the receiving side. Additionally, the format conversion unit (1119) may reduce the amount of data by compressing or encoding the three-dimensional map (1135). Also, the format conversion unit (1119) may omit processing if there is no need to perform format conversion. Additionally, the format conversion unit (1119) may control the amount of data transmitted according to the specification of the transmission range.
[0348] The communication unit (1120) communicates with the client device (902) and receives data transmission requests (such as a three-dimensional map transmission request) from the client device (902).
[0349] The transmission control unit (1121) establishes communication by exchanging information such as a corresponding format with a communication partner through the communication unit (1120).
[0350] The data transmission unit (1122) transmits a three-dimensional map (1031) to a client device (902). The three-dimensional map (1031) is data including a point cloud such as a WLD or SWLD. The three-dimensional map (1031) may include either compressed data or uncompressed data.
[0351] Next, the operation flow of the client device (902) will be described. FIG. 31 is a flowchart showing the operation when acquiring a three-dimensional map by the client device (902).
[0352] First, the client device (902) requests the server (901) to transmit a three-dimensional map (point cloud, etc.) (S1001). At this time, the client device (902) may request the server (901) to transmit a three-dimensional map related to the location information by transmitting the location information of the client device (902) obtained from GPS, etc. together.
[0353] Next, the client device (902) receives a three-dimensional map from the server (901) (S1002). If the received three-dimensional map is compressed data, the client device (902) decodes the received three-dimensional map to generate an uncompressed three-dimensional map (S1003).
[0354] Next, the client device (902) creates three-dimensional data (1034) around the client device (902) from sensor information (1033) obtained from a plurality of sensors (1015) (S1004). Next, the client device (902) estimates the self-position of the client device (902) using the three-dimensional map (1032) received from the server (901) and the three-dimensional data (1034) created from the sensor information (1033) (S1005).
[0355] FIG. 32 is a flowchart illustrating the operation of transmitting sensor information by a client device (902). First, the client device (902) receives a request to transmit sensor information from a server (901) (S1011). Upon receiving the transmission request, the client device (902) transmits sensor information (1037) to the server (901) (S1012). Additionally, if the sensor information (1033) includes multiple pieces of information obtained from multiple sensors (1015), the client device (902) may generate the sensor information (1037) by compressing each piece of information using a compression method suitable for each piece of information.
[0356] Next, the operation flow of the server (901) is described. FIG. 33 is a flowchart showing the operation when acquiring sensor information by the server (901). First, the server (901) requests the transmission of sensor information from the client device (902) (S1021). Next, the server (901) receives the sensor information (1037) transmitted from the client device (902) in accordance with the request (S1022). Next, the server (901) creates three-dimensional data (1134) using the received sensor information (1037) (S1023). Next, the server (901) reflects the created three-dimensional data (1134) on the three-dimensional map (1135) (S1024).
[0357] FIG. 34 is a flowchart illustrating the operation of transmitting a three-dimensional map by a server (901). First, the server (901) receives a request to transmit a three-dimensional map from a client device (902) (S1031). Upon receiving the request to transmit a three-dimensional map, the server (901) transmits a three-dimensional map (1031) to the client device (902) (S1032). At this time, the server (901) may extract a three-dimensional map in the vicinity of the client device (902) based on its location information and transmit the extracted three-dimensional map. Alternatively, the server (901) may compress a three-dimensional map composed of a point cloud using, for example, a compression method based on an octal tree structure and transmit the three-dimensional map after compression.
[0358] Variations of the present embodiment will be described below.
[0359] The server (901) creates three-dimensional data (1134) near the location of the client device (902) using sensor information (1037) received from the client device (902). Next, the server (901) calculates the difference between the three-dimensional data (1134) and the three-dimensional map (1135) of the same area managed by the server (901) by matching the created three-dimensional data (1134) with the three-dimensional map (1135) of the same area managed by the server (901). If the difference is greater than a predetermined threshold, the server (901) determines that some abnormality has occurred in the vicinity of the client device (902). For example, when ground subsidence occurs due to natural disasters such as earthquakes, a large difference may occur between the three-dimensional map (1135) managed by the server (901) and the three-dimensional data (1134) created based on the sensor information (1037).
[0360] Sensor information (1037) may include information indicating at least one of the type of sensor, the performance of the sensor, and the model number of the sensor. Additionally, a class ID or the like according to the performance of the sensor may be added to the sensor information (1037). For example, if the sensor information (1037) is information acquired by LiDAR, it is conceivable to assign an identifier to the performance of the sensor, such as class 1 for a sensor capable of acquiring information with a precision of several millimeters, class 2 for a sensor capable of acquiring information with a precision of several centimeters, and class 3 for a sensor capable of acquiring information with a precision of several meters. Additionally, the server (901) may estimate the performance information of the sensor, etc., from the model number of the client device (902). For example, if the client device (902) is mounted on a vehicle, the server (901) may determine the specification information of the sensor from the vehicle model of the vehicle. In this case, the server (901) may acquire information on the vehicle type in advance, or the information may be included in the sensor information. Additionally, the server (901) may use the acquired sensor information (1037) to change the degree of correction for the three-dimensional data (1134) created using the sensor information (1037). For example, if the sensor performance is high precision (Class 1), the server (901) does not perform correction on the three-dimensional data (1134). If the sensor performance is low precision (Class 3), the server (901) applies correction to the three-dimensional data (1134) according to the precision of the sensor. For example, the server (901) makes the degree (strength) of correction stronger as the precision of the sensor decreases.
[0361] The server (901) may simultaneously send a request for transmission of sensor information to multiple client devices (902) located in a space. When the server (901) receives multiple sensor information from multiple client devices (902), it is not necessary to use all sensor information for creating three-dimensional data (1134), and may select the sensor information to use according to, for example, the performance of the sensor. For example, when updating the three-dimensional map (1135), the server (901) may select high-precision sensor information (Class 1) from among the multiple sensor information received, and create three-dimensional data (1134) using the selected sensor information.
[0362] The server (901) is not limited to a server such as a traffic monitoring cloud, but may be another client device (vehicle-mounted). FIG. 35 is a diagram showing the system configuration in this case.
[0363] For example, a client device (902C) sends a request to transmit sensor information to a nearby client device (902A) and obtains sensor information from the client device (902A). Then, the client device (902C) creates three-dimensional data using the obtained sensor information from the client device (902A) and updates the three-dimensional map of the client device (902C). In this way, the client device (902C) can generate a three-dimensional map of space obtainable from the client device (902A) by utilizing the performance of the client device (902C). For example, it is thought that such a case occurs when the performance of the client device (902C) is high.
[0364] Also, in this case, the client device (902A) that provided the sensor information is granted the right to acquire a high-precision three-dimensional map generated by the client device (902C). The client device (902A) receives the high-precision three-dimensional map from the client device (902C) in accordance with that right.
[0365] Additionally, the client device (902C) may send a request to transmit sensor information to a plurality of nearby client devices (902) (client device (902A) and client device (902B)). If the sensor of the client device (902A) or client device (902B) is high-performance, the client device (902C) may create three-dimensional data using the sensor information obtained from this high-performance sensor.
[0366] FIG. 36 is a block diagram showing the functional configuration of a server (901) and a client device (902). The server (901) is equipped with, for example, a three-dimensional map compression / decoding processing unit (1201) that compresses and decodes a three-dimensional map, and a sensor information compression / decoding processing unit (1202) that compresses and decodes sensor information.
[0367] The client device (902) is equipped with a three-dimensional map decoding processing unit (1211) and a sensor information compression processing unit (1212). The three-dimensional map decoding processing unit (1211) receives encoded data of a compressed three-dimensional map and decodes the encoded data to obtain a three-dimensional map. The sensor information compression processing unit (1212) compresses the sensor information itself instead of the three-dimensional data created from the obtained sensor information and transmits the encoded data of the compressed sensor information to the server (901). With this configuration, the client device (902) only needs to internally maintain a processing unit (device or LSI) that performs processing to decode a three-dimensional map (point cloud, etc.), and does not need to internally maintain a processing unit that performs processing to compress the three-dimensional data of a three-dimensional map (point cloud, etc.). This allows the cost and power consumption of the client device (902) to be suppressed.
[0368] As described above, the client device (902) related to the present embodiment is mounted on the mobile body and creates three-dimensional data (1034) around the mobile body from sensor information (1033) representing the surrounding conditions of the mobile body obtained by a sensor (1015) mounted on the mobile body. The client device (902) estimates the self-position of the mobile body using the created three-dimensional data (1034). The client device (902) transmits the acquired sensor information (1033) to a server (901) or another mobile body (902).
[0369] According to this, the client device (902) transmits sensor information (1033) to the server (901), etc. As a result, it is possible to reduce the amount of data transmitted compared to the case where three-dimensional data is transmitted. In addition, since there is no need to perform processing such as compression or encoding of the three-dimensional data at the client device (902), the processing load of the client device (902) can be reduced. Therefore, the client device (902) can realize a reduction in the amount of data transmitted or a simplification of the device configuration.
[0370] Additionally, the client device (902) also transmits a request to transmit a three-dimensional map to the server (901) and receives a three-dimensional map (1031) from the server (901). In estimating its own position, the client device (902) estimates its own position using three-dimensional data (1034) and a three-dimensional map (1032).
[0371] Also, the sensor information (1033) includes at least one of information obtained from a laser sensor, a luminance image, an infrared image, a depth image, position information of the sensor, and speed information of the sensor.
[0372] Also, the sensor information (1033) includes information indicating the performance of the sensor.
[0373] Additionally, the client device (902) encodes or compresses sensor information (1033), and in the transmission of sensor information, transmits the sensor information (1037) after encoding or compression to the server (901) or another mobile body (902). Accordingly, the client device (902) can reduce the amount of data transmitted.
[0374] For example, a client device (902) is equipped with a processor and memory, and the processor uses memory to perform the above processing.
[0375] Additionally, the server (901) related to the present embodiment is capable of communicating with a client device (902) mounted on a mobile body and receives sensor information (1037) indicating the surrounding conditions of the mobile body, obtained by a sensor (1015) mounted on the mobile body, from the client device (902). The server (901) creates three-dimensional data (1134) of the surroundings of the mobile body from the received sensor information (1037).
[0376] According to this, the server (901) creates three-dimensional data (1134) using sensor information (1037) transmitted from the client device (902). By doing so, it is possible to reduce the amount of transmitted data compared to when the client device (902) transmits the three-dimensional data. In addition, since there is no need to perform processing such as compression or encoding of the three-dimensional data at the client device (902), the processing load of the client device (902) can be reduced. Therefore, the server (901) can realize a reduction in the amount of transmitted data or a simplification of the device configuration.
[0377] Additionally, the server (901) also transmits a request for transmission of sensor information to the client device (902).
[0378] Additionally, the server (901) updates the three-dimensional map (1135) using the created three-dimensional data (1134) and transmits the three-dimensional map (1135) to the client device (902) in response to a request to transmit the three-dimensional map (1135) from the client device (902).
[0379] Also, the sensor information (1037) includes at least one of information obtained from a laser sensor, a luminance image, an infrared image, a depth image, position information of the sensor, and speed information of the sensor.
[0380] Also, the sensor information (1037) includes information indicating the performance of the sensor.
[0381] Additionally, the server (901) also corrects the three-dimensional data according to the performance of the sensor. According to this, the method for creating the three-dimensional data can improve the quality of the three-dimensional data.
[0382] Additionally, the server (901) receives sensor information from a plurality of client devices (902) and selects sensor information (1037) to be used for creating three-dimensional data (1134) based on a plurality of information indicating the performance of the sensors included in the plurality of sensor information (1037). Accordingly, the server (901) can improve the quality of the three-dimensional data (1134).
[0383] Additionally, the server (901) decodes or expands the received sensor information (1037) and creates three-dimensional data (1134) from the sensor information (1132) after decoding or expanding. Accordingly, the server (901) can reduce the amount of data transmitted.
[0384] For example, the server (901) is equipped with a processor and memory, and the processor uses the memory to perform the above processing.
[0385] (Embodiment 7)
[0386] In this embodiment, a method for encoding and decoding three-dimensional data using inter-prediction processing is described.
[0387] FIG. 37 is a block diagram of a three-dimensional data encoding device (1300) related to the present embodiment. The three-dimensional data encoding device (1300) generates an encoded bit stream (hereinafter also simply referred to as a bit stream) which is an encoded signal by encoding three-dimensional data. As shown in FIG. 37, the three-dimensional data encoding device (1300) comprises a splitting unit (1301), a subtraction unit (1302), a conversion unit (1303), a quantization unit (1304), an inverse quantization unit (1305), an inverse conversion unit (1306), an addition unit (1307), a reference volume memory (1308), an intra prediction unit (1309), a reference space memory (1310), an inter prediction unit (1311), a prediction control unit (1312), and an entropy encoding unit (1313).
[0388] The partitioning unit (1301) divides each space (SPC) included in the three-dimensional data into multiple volumes (VLM) that are encoding units. In addition, the partitioning unit (1301) converts the voxels within each volume into octal tree representations. In addition, the partitioning unit (1301) may make the space and the volume the same size and convert the space into an octal tree representation. In addition, the partitioning unit (1301) may add information necessary for octal tree conversion (such as depth information) to the header of the bit stream, etc.
[0389] The subtraction unit (1302) calculates the difference between the volume (encoding target volume) output from the division unit (1301) and the prediction volume generated by intra prediction or inter prediction described later, and outputs the calculated difference to the conversion unit (1303) as a prediction residual. FIG. 38 is a diagram showing an example of calculating a prediction residual. In addition, the bit sequences of the encoding target volume and the prediction volume shown here are, for example, location information indicating the location of a three-dimensional point (e.g., a point cloud) included in the volume.
[0390] The octal tree representation and the scan order of voxels will be described below. A volume is converted into an octal tree structure (octalization) and then encoded. An octal tree structure consists of nodes and leaves. Each node has eight nodes or leaves, and each leaf has voxel (VXL) information. FIG. 39 is a diagram showing an example of a volume structure containing multiple voxels. FIG. 40 is a diagram showing an example of converting the volume shown in FIG. 39 into an octal tree structure. Here, among the leaves shown in FIG. 40, leaves 1, 2, and 3 each represent the voxels VXL1, VXL2, and VXL3 shown in FIG. 39, and represent VXL containing a point cloud (hereinafter, effective VXL).
[0391] An octal tree is represented, for example, by a sequence of 0s and 1s. For example, if a node or valid VXL is set to the value 1 and everything else to the value 0, the sequence shown in FIG. 40 is assigned to each node and leaf. Then, this sequence is scanned according to a width-first or depth-first scan order. For example, if scanned in width-first order, the sequence shown in FIG. 41A is obtained. If scanned in depth-first order, the sequence shown in FIG. 41B is obtained. The sequence obtained by this scan is encoded by entropy encoding, and the amount of information is reduced.
[0392] Next, depth information in octal tree representation is explained. In octal tree representation, depth is used to control the granularity of point cloud information contained within a volume. Setting a high depth allows for the reproduction of point cloud information to a finer level, but increases the amount of data required to represent nodes and leaves. Conversely, setting a low depth reduces the amount of data, but because multiple point cloud information with different locations and colors is treated as having the same location and the same color, the information contained in the original point cloud information is lost.
[0393] For example, FIG. 42 is a diagram showing an example of representing the octal tree of depth=2 shown in FIG. 40 as an octal tree of depth=1. The octal tree shown in FIG. 42 has less data than the octal tree shown in FIG. 40. In short, the octal tree shown in FIG. 42 has fewer bits after decoding than the octal tree shown in FIG. 42. Here, Leaf 1 and Leaf 2 shown in FIG. 40 are represented as Leaf 1 shown in FIG. 41. In short, the information that Leaf 1 and Leaf 2 shown in FIG. 40 were at different locations is lost.
[0394] FIG. 43 is a drawing showing a volume corresponding to the octal tree shown in FIG. 42. VXL1 and VXL2 shown in FIG. 39 correspond to VXL12 shown in FIG. 43. In this case, the three-dimensional data encoding device (1300) generates color information of VXL12 shown in FIG. 43 from the color information of VXL1 and VXL2 shown in FIG. 39. For example, the three-dimensional data encoding device (1300) calculates the average value, median value, or weighted average value of the color information of VXL1 and VXL2 as the color information of VXL12. In this way, the three-dimensional data encoding device (1300) may control the reduction of the amount of data by changing the depth of the octal tree.
[0395] The three-dimensional data encoding device (1300) may set the depth information of the octal tree to any of the world unit, space unit, and volume unit. In that case, the three-dimensional data encoding device (1300) may add depth information to the header information of the world, the header information of the space, or the header information of the volume. In addition, the same value may be used as depth information for all worlds, spaces, and volumes with different times. In this case, the three-dimensional data encoding device (1300) may add depth information to the header information managing the world of the entire time.
[0396] When color information is included in the voxel, the conversion unit (1303) applies a frequency transformation, such as an orthogonal transformation, to the predicted residual of the color information of the voxel within the volume. For example, the conversion unit (1303) creates a one-dimensional array by scanning the predicted residual in a certain scanning order. Then, the conversion unit (1303) converts the one-dimensional array into the frequency domain by applying a one-dimensional orthogonal transformation to the created one-dimensional array. As a result, when the values of the predicted residuals within the volume are close, the value of the low-frequency component increases and the value of the high-frequency component decreases. Therefore, the code amount can be reduced more efficiently in the quantization unit (1304).
[0397] Additionally, the conversion unit (1303) may use an orthogonal transformation of two dimensions or more, rather than a one-dimensional one. For example, the conversion unit (1303) maps the predicted residuals to a two-dimensional array in a certain scan order and applies a two-dimensional orthogonal transformation to the obtained two-dimensional array. Additionally, the conversion unit (1303) may select an orthogonal transformation method to use from a plurality of orthogonal transformation methods. In this case, the three-dimensional data encoding device (1300) adds information indicating which orthogonal transformation method was used to the bit stream. Additionally, the conversion unit (1303) may select an orthogonal transformation method to use from a plurality of orthogonal transformation methods of different dimensions. In this case, the three-dimensional data encoding device (1300) adds information indicating which dimension of orthogonal transformation method was used to the bit stream.
[0398] For example, the conversion unit (1303) matches the scan order of the predicted residual to the scan order (width-first or depth-first, etc.) in the octal tree within the volume. By doing so, there is no need to add information indicating the scan order of the predicted residual to the bit stream, thereby reducing overhead. Additionally, the conversion unit (1303) may apply a scan order different from the scan order of the octal tree. In this case, the three-dimensional data encoding device (1300) adds information indicating the scan order of the predicted residual to the bit stream. By doing so, the three-dimensional data encoding device (1300) can efficiently encode the predicted residual. Additionally, the three-dimensional data encoding device (1300) may add information (flags, etc.) indicating whether to apply the scan order of the octal tree to the bit stream, and when the scan order of the octal tree is not applied, add information indicating the scan order of the predicted residual to the bit stream.
[0399] The conversion unit (1303) may convert not only the predicted residual of color information but also other attribute information possessed by the voxel. For example, the conversion unit (1303) may convert and encode information such as reflectance obtained when acquiring a point cloud with LiDAR, etc.
[0400] The conversion unit (1303) may skip processing if the space does not have attribute information such as color information. Additionally, the three-dimensional data encoding device (1300) may add information (flags) indicating whether to skip processing by the conversion unit (1303) to the bit stream.
[0401] The quantization unit (1304) generates quantization coefficients by performing quantization on the frequency components of the prediction residual generated by the conversion unit (1303) using quantization control parameters. This reduces the amount of information. The generated quantization coefficients are output to the entropy encoding unit (1313). The quantization unit (1304) may control the quantization control parameters in world units, space units, or volume units. In that case, the three-dimensional data encoding device (1300) adds the quantization control parameters to the respective header information, etc. Also, the quantization unit (1304) may perform quantization control by changing the weights for each frequency component of the prediction residual. For example, the quantization unit (1304) may finely quantize low-frequency components and roughly quantize high-frequency components. In this case, the three-dimensional data encoding device (1300) may add parameters indicating the weights of each frequency component to the header.
[0402] The quantization unit (1304) may skip processing if the space does not have attribute information such as color information. Additionally, the three-dimensional data encoding device (1300) may add information (flags) indicating whether to skip processing by the quantization unit (1304) to the bit stream.
[0403] The inverse quantization unit (1305) generates inverse quantization coefficients of the predicted residual by performing inverse quantization on the quantization coefficients generated in the quantization unit (1304) using quantization control parameters, and outputs the generated inverse quantization coefficients to the inverse transformation unit (1306).
[0404] The inverse transform unit (1306) generates a predicted residual after applying the inverse transform by applying the inverse transform to the inverse quantization coefficients generated in the inverse quantization unit (1305). Since this predicted residual after applying the inverse transform is a predicted residual generated after quantization, it does not have to be exactly the same as the predicted residual output by the transform unit (1303).
[0405] The adder (1307) generates a reconstruction volume by adding the prediction residual after applying the inverse transformation generated by the inverse transformation unit (1306) and the prediction volume generated by the intra prediction or inter prediction described later, which was used to generate the prediction residual before quantization. This reconstruction volume is stored in the reference volume memory (1308) or the reference space memory (1310).
[0406] The intra prediction unit (1309) generates a predicted volume of the volume to be encoded using attribute information of an adjacent volume stored in the reference volume memory (1308). The attribute information includes color information or reflectance of a voxel. The intra prediction unit (1309) generates a predicted value of the color information or reflectance of the volume to be encoded.
[0407] FIG. 44 is a diagram illustrating the operation of the intra prediction unit (1309). For example, the intra prediction unit (1309) generates a predicted volume of the encoding target volume (volume idx=3) shown in FIG. 44 from an adjacent volume (volume idx=0). Here, volume idx is identifier information added to a volume within a space, and a different value is assigned to each volume. The order of assignment of volume idx may be the same as the encoding order or may be different from the encoding order. For example, the intra prediction unit (1309) uses the average value of the color information of the voxels included in the adjacent volume, volume idx=0, as the predicted value of the color information of the encoding target volume shown in FIG. 44. In this case, a prediction residual is generated by subtracting the predicted value of the color information from the color information of each voxel included in the encoding target volume. Processing after the conversion unit (1303) is performed on this prediction residual. In addition, in this case, the three-dimensional data encoding device (1300) adds adjacent volume information and prediction mode information to the bit stream. Here, adjacent volume information is information indicating the adjacent volume used for prediction, and for example, indicates the volume index of the adjacent volume used for prediction. Also, prediction mode information indicates the mode used for generating the prediction volume. The mode is, for example, an average value mode that generates a prediction value from the average value of a voxel within the adjacent volume, or an intermediate value mode that generates a prediction value from the intermediate value of a voxel within the adjacent volume.
[0408] The intra prediction unit (1309) may generate a prediction volume from a plurality of adjacent volumes. For example, in the configuration shown in FIG. 44, the intra prediction unit (1309) generates a prediction volume 0 from a volume with volume idx=0 and generates a prediction volume 1 from a volume with volume idx=1. Then, the intra prediction unit (1309) generates the average of prediction volume 0 and prediction volume 1 as the final prediction volume. In this case, the three-dimensional data encoding device (1300) may add a plurality of volume idx of the plurality of volumes used to generate the prediction volume to the bit stream.
[0409] FIG. 45 is a diagram schematically illustrating the inter prediction processing related to the present embodiment. The inter prediction unit (1311) encodes (inter predicts) the space (SPC) of time T_Cur using the space where encoding is completed at different time T_LX. In this case, the inter prediction unit (1311) performs encoding processing by applying rotation and translation processing to the space where encoding is completed at different time T_LX.
[0410] Additionally, the three-dimensional data encoding device (1300) adds RT information related to rotation and translation processing applied to the space of a different time T_LX to the bit stream. A different time T_LX is, for example, a time T_L0 preceding the aforementioned time T_Cur. At this time, the three-dimensional data encoding device (1300) may add RT information RT_L0 related to rotation and translation processing applied to the space of time T_L0 to the bit stream.
[0411] Alternatively, a different time T_LX is, for example, a time T_L1 that is later than any of the above times T_Cur. In this case, the three-dimensional data encoding device (1300) may add RT information RT_L1, which is related to rotation and translation processing applied to the space of time T_L1, to the bit stream.
[0412] Alternatively, the inter-prediction unit (1311) performs encoding (pair prediction) by referring to both spaces of different times T_L0 and T_L1. In this case, the three-dimensional data encoding device (1300) may add both RT information RT_L0 and RT_L1 related to rotation and translation applied to each space to the bit stream.
[0413] Additionally, although T_L0 was defined as a time before T_Cur and T_L1 as a time after T_Cur in the above, it is not necessarily limited to this. For example, both T_L0 and T_L1 may be a time before T_Cur. Alternatively, both T_L0 and T_L1 may be a time after T_Cur.
[0414] Additionally, when the three-dimensional data encoding device (1300) performs encoding by referencing multiple different time spaces, RT information related to rotation and translation applied to each space may be added to the bit stream. For example, the three-dimensional data encoding device (1300) manages multiple encoded spaces referenced in two reference lists (L0 list and L1 list). When the first reference space in the L0 list is set to L0R0, the second reference space in the L0 list is set to L0R1, the first reference space in the L1 list is set to L1R0, and the second reference space in the L1 list is set to L1R1, the three-dimensional data encoding device (1300) adds RT information RT_L0R0 of L0R0, RT information RT_L0R1 of L0R1, RT information RT_L1R0 of L1R0, and RT information RT_L1R1 of L1R1 to the bit stream. For example, the three-dimensional data encoding device (1300) adds these RT information to the header of the bit stream, etc.
[0415] Additionally, when the three-dimensional data encoding device (1300) performs encoding by referencing multiple different reference spaces, it determines whether to apply rotation and translation for each reference space. At that time, the three-dimensional data encoding device (1300) may add information (such as an RT application flag) indicating whether rotation and translation have been applied for each reference space to the header information of the bit stream. For example, the three-dimensional data encoding device (1300) calculates RT information and an ICP error value using an Interactive Closest Point (ICP) algorithm for each reference space referenced from the encoding target space. If the ICP error value is less than or equal to a predetermined value, the three-dimensional data encoding device (1300) determines that there is no need to perform rotation and translation and sets the RT application flag to off. On the other hand, if the ICP error value is greater than the above predetermined value, the three-dimensional data encoding device (1300) sets the RT application flag to on and adds RT information to the bit stream.
[0416] FIG. 46 is a diagram illustrating an example of a syntax that adds RT information and an RT application flag to a header. Additionally, the number of bits allocated to each syntax may be determined within the range that the syntax can take. For example, if the number of reference spaces included in reference list L0 is 8, 3 bits may be allocated to MaxRefSpc_l0. The number of bits allocated may be variable depending on the value that each syntax can take, or fixed regardless of the value that can be taken. If the number of bits allocated is fixed, the three-dimensional data encoding device (1300) may add the fixed number of bits to other header information.
[0417] Here, MaxRefSpc_l0 shown in FIG. 46 represents the number of reference spaces included in reference list L0. RT_flag_l0[i] is the RT application flag for reference space i in reference list L0. If RT_flag_l0[i] is 1, rotation and translation are applied to reference space i. If RT_flag_l0[i] is 0, rotation and translation are not applied to reference space i.
[0418] R_l0[i] and T_l0[i] are RT information for reference space i in reference list L0. R_l0[i] is rotation information for reference space i in reference list L0. Rotation information represents the content of the applied rotation processing, for example, a rotation matrix or a quaternion. T_l0[i] is translation information for reference space i in reference list L0. Translation information represents the content of the applied translation processing, for example, a translation vector.
[0419] MaxRefSpc_l1 indicates the number of reference spaces included in reference list L1. RT_flag_l1[i] is the RT application flag for reference space i in reference list L1. If RT_flag_l1[i] is 1, rotation and translation are applied to reference space i. If RT_flag_l1[i] is 0, rotation and translation are not applied to reference space i.
[0420] R_l1[i] and T_l1[i] are RT information for reference space i in reference list L1. R_l1[i] is rotation information for reference space i in reference list L1. Rotation information represents the content of the applied rotation processing, for example, a rotation matrix or a quaternion. T_l1[i] is translation information for reference space i in reference list L1. Translation information represents the content of the applied translation processing, for example, a translation vector.
[0421] The inter prediction unit (1311) generates a predicted volume of the encoding target volume using information of the encoding completed reference space stored in the reference space memory (1310). As described above, before generating the predicted volume of the encoding target volume, the inter prediction unit (1311) obtains RT information using the Interactive Closest Point (ICP) algorithm in the encoding target space and the reference space to bring the overall positional relationship between the encoding target space and the reference space closer. Then, the inter prediction unit (1311) obtains reference space B by applying rotation and translation processing to the reference space using the obtained RT information. After that, the inter prediction unit (1311) generates a predicted volume of the encoding target volume within the encoding target space using information within reference space B. Here, the three-dimensional data encoding device (1300) adds the RT information used to obtain reference space B to the header information of the encoding target space, etc.
[0422] In this way, the inter prediction unit (1311) can improve the precision of the prediction volume by applying rotation and translation processing to the reference space to bring the overall positional relationship between the encoding target space and the reference space closer, and then generating the prediction volume using the information of the reference space. In addition, since the prediction residual can be suppressed, the amount of encoding can be reduced. Also, although an example of performing ICP using the encoding target space and the reference space is shown here, it is not necessarily limited to this. For example, in order to reduce the throughput, the inter prediction unit (1311) may obtain RT information by performing ICP using at least one of the encoding target space from which the number of voxels or point clouds has been filtered and the reference space from which the number of voxels or point clouds has been filtered.
[0423] Additionally, the inter prediction unit (1311) determines that rotation and translation processing is not necessary when the ICP error value obtained as a result of the ICP is smaller than a predetermined first threshold, that is, for example, when the positional relationship between the encoding target space and the reference space is close, and thus does not need to perform rotation and translation. In this case, the three-dimensional data encoding device (1300) may suppress overhead by not adding RT information to the bit stream.
[0424] Additionally, the inter prediction unit (1311) may determine that the shape change between spaces is large when the ICP error value is greater than a predetermined second threshold, and may apply intra prediction to all volumes of the space to be encoded. Hereinafter, the space to which intra prediction is applied is referred to as the intra space. Also, the second threshold is a value greater than the first threshold. Also, not limited to ICP, any method that obtains RT information from a set of two voxels or a set of two point clouds may be applied.
[0425] Additionally, if the three-dimensional data includes attribute information such as shape or color, the inter prediction unit (1311) searches for a volume that is closest to the shape or color attribute information of the target volume in the reference space as a predicted volume of the target volume in the target volume in the target volume in the target volume in the target volume in the target volume in the target volume in the target volume in the target volume in the target volume in the reference space, for example. Also, this reference space is, for example, the reference space after the rotation and translation processing described above has been performed. The inter prediction unit (1311) generates a predicted volume from the volume (reference volume) obtained by the search. FIG. 47 is a diagram for explaining the operation of generating a predicted volume. When the target volume in the target volume (volume idx=0) shown in FIG. 47 is encoded using inter prediction, the inter prediction unit (1311) searches for the volume with the smallest prediction residual, which is the difference between the target volume in the target volume and the reference volume, while scanning the reference volumes in the reference space in order. The inter prediction unit (1311) selects the volume with the smallest prediction residual as the predicted volume. The prediction residual of the volume to be encoded and the prediction volume is encoded by processing after the conversion unit (1303). Here, the prediction residual is the difference between the attribute information of the volume to be encoded and the attribute information of the prediction volume. Also, the three-dimensional data encoding device (1300) adds the volume idx of the reference volume within the reference space referenced as the prediction volume to the header of the bit stream, etc.
[0426] In the example shown in FIG. 47, the reference volume of volume idx=4 in reference space L0R0 is selected as the predicted volume of the volume to be encoded. Then, the predicted residuals of the volume to be encoded and the reference volume, and the reference volume idx=4 are encoded and added to the bit stream.
[0427] In addition, although an example of generating a predicted volume of attribute information was described here, the same processing may be performed on a predicted volume of location information.
[0428] The prediction control unit (1312) controls whether to encode the volume to be encoded using either intra prediction or inter prediction. Here, a mode including intra prediction and inter prediction is called a prediction mode. For example, the prediction control unit (1312) calculates the prediction residual when the volume to be encoded is predicted by intra prediction and the prediction residual when predicted by inter prediction as evaluation values, and selects the prediction mode with the smaller evaluation value. Additionally, the prediction control unit (1312) may calculate the actual code amount by applying orthogonal transformation, quantization, and entropy encoding to the prediction residual of intra prediction and the prediction residual of inter prediction, respectively, and select the prediction mode using the calculated code amount as an evaluation value. Additionally, overhead information other than the prediction residual (such as reference volume index information) may be added to the evaluation value. Furthermore, the prediction control unit (1312) may always select intra prediction when it is predetermined that the space to be encoded is encoded as an intra space.
[0429] The entropy encoding unit (1313) generates an encoded signal (encoded bit stream) by encoding the quantization coefficients, which are inputs from the quantization unit (1304), into variable lengths. Specifically, the entropy encoding unit (1313), for example, binarizes the quantization coefficients and performs arithmetic encoding on the obtained binary signal.
[0430] Next, a three-dimensional data decoding device that decodes an encoded signal generated by a three-dimensional data encoding device (1300) will be described. FIG. 48 is a block diagram of a three-dimensional data decoding device (1400) related to the present embodiment. This three-dimensional data decoding device (1400) comprises an entropy decoding unit (1401), an inverse quantization unit (1402), an inverse transformation unit (1403), an adder (1404), a reference volume memory (1405), an intra prediction unit (1406), a reference space memory (1407), an inter prediction unit (1408), and a prediction control unit (1409).
[0431] The entropy decoder (1401) decodes the encoded signal (encoded bit stream) for a variable length. For example, the entropy decoder (1401) arithmetic decodes the encoded signal to generate a binary signal and generates quantization coefficients from the generated binary signal.
[0432] The inverse quantization unit (1402) generates inverse quantization coefficients by inversely quantizing the quantization coefficients input from the entropy decoding unit (1401) using quantization parameters added to the bit stream, etc.
[0433] The inverse transform unit (1403) generates a predicted residual by inversely transforming the inverse quantization coefficients input from the inverse quantization unit (1402). For example, the inverse transform unit (1403) generates a predicted residual by inversely transforming the inverse quantization coefficients based on information added to the bit stream.
[0434] The adder (1404) generates a reconstruction volume by adding the prediction residual generated by the inverse transformation unit (1403) and the prediction volume generated by intra prediction or inter prediction. This reconstruction volume is output as decoded three-dimensional data and is stored in the reference volume memory (1405) or reference space memory (1407).
[0435] The intra prediction unit (1406) generates a predicted volume by intra prediction using information added to the reference volume and bit stream within the reference volume memory (1405). Specifically, the intra prediction unit (1406) obtains adjacent volume information (e.g., volume idx) added to the bit stream and prediction mode information, and generates a predicted volume by the mode represented by the prediction mode information using the adjacent volume represented by the adjacent volume information. Furthermore, the details of these processes are identical to the processing by the intra prediction unit (1309) described above, except that information added to the bit stream is used.
[0436] The inter prediction unit (1408) generates a prediction volume by inter prediction using information added to the reference space and bit stream within the reference space memory (1407). Specifically, the inter prediction unit (1408) applies rotation and translation processing to the reference space using RT information for each reference space added to the bit stream, and generates a prediction volume using the reference space after application. Additionally, if an RT application flag for each reference space exists within the bit stream, the inter prediction unit (1408) applies rotation and translation processing to the reference space according to the RT application flag. Furthermore, the details of these processes are identical to the processing by the inter prediction unit (1311) described above, except that information assigned to the bit stream is used.
[0437] The prediction control unit (1409) controls whether to decode a volume to be decoded using intra prediction or inter prediction. For example, the prediction control unit (1409) selects intra prediction or inter prediction based on information indicating the prediction mode to be used, which is added to the bit stream. Additionally, the prediction control unit (1409) may always select intra prediction if it is predetermined that the space to be decoded is to be decoded as an intra space.
[0438] Variations of the present embodiment are described below. In the present embodiment, an example in which rotation and translation are applied in space units has been described, but rotation and translation may be applied in more fine units. For example, the three-dimensional data encoding device (1300) may divide the space into subspaces and apply rotation and translation in subspace units. In this case, the three-dimensional data encoding device (1300) generates RT information for each subspace and adds the generated RT information to the header of the bit stream, etc. Also, the three-dimensional data encoding device (1300) may apply rotation and translation in volume units, which are encoding units. In this case, the three-dimensional data encoding device (1300) generates RT information in encoding volume units and adds the generated RT information to the header of the bit stream, etc. Furthermore, the above may be combined. In short, the three-dimensional data encoding device (1300) may apply rotation and translation in large units and then apply rotation and translation in fine units. For example, the three-dimensional data encoding device (1300) may apply rotation and translation in space units, and apply different rotation and translation to each of the multiple volumes included in the obtained space.
[0439] In addition, although an example of applying rotation and translation to a reference space has been described in this embodiment, it is not necessarily limited to this. For example, the three-dimensional data encoding device (1300) may change the size of the three-dimensional data by applying, for example, a scale processing. Also, the three-dimensional data encoding device (1300) may apply one or two of rotation, translation, and scale. Furthermore, when processing is applied to different units in multiple stages as described above, the type of processing applied to each unit may be different. For example, rotation and translation may be applied to the space unit, and translation may be applied to the volume unit.
[0440] In addition, these variations can be applied in the same way to a three-dimensional data decoding device (1400).
[0441] As described above, the three-dimensional data encoding device (1300) related to the present embodiment performs the following processing. FIG. 48 is a flowchart of the inter-prediction processing by the three-dimensional data encoding device (1300).
[0442] First, the three-dimensional data encoding device (1300) generates predicted position information (e.g., predicted volume) using position information of three-dimensional points included in reference three-dimensional data (e.g., reference space) at a different time from the target three-dimensional data (e.g., encoding target space) (S1301). Specifically, the three-dimensional data encoding device (1300) generates predicted position information by applying rotation and translation processing to the position information of three-dimensional points included in the reference three-dimensional data.
[0443] Additionally, the three-dimensional data encoding device (1300) may perform rotation and translation processing in a first unit (e.g., a space) and generate predicted position information in a second unit (e.g., a volume) that is more detailed than the first unit. For example, the three-dimensional data encoding device (1300) searches for a volume among a plurality of volumes included in a reference space after rotation and translation processing that minimizes the difference between the position information and the volume to be encoded included in the encoding target space, and uses the obtained volume as a predicted volume. Additionally, the three-dimensional data encoding device (1300) may perform rotation and translation processing and generate predicted position information in the same unit.
[0444] Additionally, the three-dimensional data encoding device (1300) may generate predicted position information by applying a first rotation and translation processing to the position information of a three-dimensional point included in the reference three-dimensional data in a first unit (e.g., space), and applying a second rotation and translation processing to the position information of the three-dimensional point obtained by the first rotation and translation processing in a second unit (e.g., volume) that is more fine than the first unit.
[0445] Here, the position information and predicted position information of the three-dimensional point are represented in an octal tree structure, for example, as shown in FIG. 41. For example, the position information and predicted position information of the three-dimensional point are displayed in the scan order prioritizing width over depth in the octal tree structure. Alternatively, the position information and predicted position information of the three-dimensional point are displayed in the scan order prioritizing depth over width in the octal tree structure.
[0446] Also, as shown in FIG. 46, the three-dimensional data encoding device (1300) encodes an RT application flag indicating whether rotation and translation processing is applied to the position information of a three-dimensional point included in the reference three-dimensional data. In short, the three-dimensional data encoding device (1300) generates an encoding signal (encoded bit stream) containing the RT application flag. Also, the three-dimensional data encoding device (1300) encodes RT information indicating the contents of rotation and translation processing. In short, the three-dimensional data encoding device (1300) generates an encoding signal (encoded bit stream) containing RT information. Additionally, the three-dimensional data encoding device (1300) encodes RT information when rotation and translation processing is applied according to the RT application flag, and does not encode RT information when rotation and translation processing is not applied according to the RT application flag.
[0447] In addition, the three-dimensional data includes, for example, location information of three-dimensional points and attribute information (color information, etc.) of each three-dimensional point. The three-dimensional data encoding device (1300) generates predicted attribute information using the attribute information of the three-dimensional points included in the reference three-dimensional data (S1302).
[0448] Next, the three-dimensional data encoding device (1300) encodes the position information of a three-dimensional point included in the target three-dimensional data using predicted position information. For example, the three-dimensional data encoding device (1300) calculates differential position information, which is the difference between the position information of a three-dimensional point included in the target three-dimensional data and the predicted position information, as shown in FIG. 38 (S1303).
[0449] Additionally, the three-dimensional data encoding device (1300) encodes attribute information of a three-dimensional point included in the target three-dimensional data using predicted attribute information. For example, the three-dimensional data encoding device (1300) calculates difference attribute information, which is the difference between the attribute information of a three-dimensional point included in the target three-dimensional data and the predicted attribute information (S1304). Next, the three-dimensional data encoding device (1300) performs conversion and quantization on the calculated difference attribute information (S1305).
[0450] Finally, the three-dimensional data encoding device (1300) encodes the difference position information and the difference attribute information after quantization (e.g., entropy encoding) (S1306). In short, the three-dimensional data encoding device (1300) generates an encoding signal (encoded bit stream) containing the difference position information and the difference attribute information.
[0451] Additionally, if the three-dimensional data does not include attribute information, the three-dimensional data encoding device (1300) does not need to perform steps S1302, S1304, and S1305. Also, the three-dimensional data encoding device (1300) may perform only one of encoding the position information of the three-dimensional point and encoding the attribute information of the three-dimensional point.
[0452] Also, the order of processing shown in FIG. 49 is an example and is not limited thereto. For example, since the processing of location information (S1301, S1303) and the processing of attribute information (S1302, S1304, S1305) are independent of each other, they may be performed in any order, and some of them may be processed in parallel.
[0453] As described above, the three-dimensional data encoding device (1300) of the present embodiment generates predicted position information using position information of a three-dimensional point included in reference three-dimensional data at a different time from the target three-dimensional data, and encodes differential position information, which is the difference between the position information of the three-dimensional point included in the target three-dimensional data and the predicted position information. By doing so, the amount of data of the encoding signal can be reduced, thereby improving encoding efficiency.
[0454] In addition, the three-dimensional data encoding device (1300) of the present embodiment generates predicted attribute information using attribute information of a three-dimensional point included in reference three-dimensional data, and encodes difference attribute information, which is the difference between the attribute information of a three-dimensional point included in target three-dimensional data and the predicted attribute information. By doing so, the amount of data of the encoding signal can be reduced, thereby improving encoding efficiency.
[0455] For example, a three-dimensional data encoding device (1300) is equipped with a processor and a memory, and the processor uses the memory to perform the above processing.
[0456] FIG. 48 is a flowchart of inter-prediction processing by a three-dimensional data decoding device (1400).
[0457] First, the three-dimensional data decoding device (1400) decodes differential position information and differential attribute information from the encoded signal (encoded bit stream) (e.g., entropy decoding) (S1401).
[0458] Additionally, the three-dimensional data decoder (1400) decodes an RT application flag from the encoded signal indicating whether rotation and translation processing should be applied to the position information of a three-dimensional point included in the reference three-dimensional data. Additionally, the three-dimensional data decoder (1400) decodes RT information indicating the contents of the rotation and translation processing. Furthermore, the three-dimensional data decoder (1400) decodes RT information when rotation and translation processing is applied according to the RT application flag, and does not need to decode RT information when rotation and translation processing is not applied according to the RT application flag.
[0459] Next, the three-dimensional data decoder (1400) performs inverse quantization and inverse transformation on the decoded differential attribute information (S1402).
[0460] Next, the three-dimensional data decoder (1400) generates predicted position information (e.g., predicted volume) using position information of three-dimensional points included in reference three-dimensional data (e.g., reference space) at a different time from the target three-dimensional data (e.g., decoder target space) (S1403). Specifically, the three-dimensional data decoder (1400) generates predicted position information by applying rotation and translation processing to the position information of three-dimensional points included in the reference three-dimensional data.
[0461] More specifically, the three-dimensional data decoder (1400) applies rotation and translation processing to the position information of a three-dimensional point included in the reference three-dimensional data represented by RT information when the application of rotation and translation processing is indicated by the RT application flag. On the other hand, when the application of rotation and translation processing is not indicated by the RT application flag, the three-dimensional data decoder (1400) does not apply rotation and translation processing to the position information of a three-dimensional point included in the reference three-dimensional data.
[0462] Additionally, the three-dimensional data decoder (1400) may perform rotation and translation processing as a first unit (e.g., space) and generate predicted position information as a second unit (e.g., volume) that is more detailed than the first unit. Additionally, the three-dimensional data decoder (1400) may perform rotation and translation processing and generate predicted position information as the same unit.
[0463] Additionally, the three-dimensional data decoding device (1400) may generate predicted position information by applying a first rotation and translation processing to the position information of a three-dimensional point included in the reference three-dimensional data in a first unit (e.g., space), and applying a second rotation and translation processing to the position information of the three-dimensional point obtained by the first rotation and translation processing in a second unit (e.g., volume) that is more fine than the first unit.
[0464] Here, the position information and predicted position information of the three-dimensional point are represented in an octal tree structure, for example, as shown in FIG. 41. For example, the position information and predicted position information of the three-dimensional point are displayed in the scan order prioritizing width over depth in the octal tree structure. Alternatively, the position information and predicted position information of the three-dimensional point are displayed in the scan order prioritizing depth over width in the octal tree structure.
[0465] The three-dimensional data decoding device (1400) generates predicted attribute information using attribute information of three-dimensional points included in the reference three-dimensional data (S1404).
[0466] Next, the three-dimensional data decoding device (1400) restores the location information of a three-dimensional point included in the target three-dimensional data by decoding the encoded location information included in the encoded signal using the predicted location information. Here, the encoded location information is, for example, differential location information, and the three-dimensional data decoding device (1400) restores the location information of a three-dimensional point included in the target three-dimensional data by adding the differential location information and the predicted location information (S1405).
[0467] Additionally, the three-dimensional data decoding device (1400) restores attribute information of a three-dimensional point included in the target three-dimensional data by decoding the encoding attribute information included in the encoding signal using the predicted attribute information. Here, the encoding attribute information is, for example, differential attribute information, and the three-dimensional data decoding device (1400) restores attribute information of a three-dimensional point included in the target three-dimensional data by adding the differential attribute information and the predicted attribute information (S1406).
[0468] Additionally, if the three-dimensional data does not include attribute information, the three-dimensional data decoding device (1400) does not need to perform steps S1402, S1404, and S1406. Also, the three-dimensional data decoding device (1400) may perform only one of the decoding of the position information of the three-dimensional point and the decoding of the attribute information of the three-dimensional point.
[0469] Also, the order of processing shown in FIG. 50 is an example and is not limited thereto. For example, since the processing of location information (S1403, S1405) and the processing of attribute information (S1402, S1404, S1406) are independent of each other, they may be performed in any order, and some of them may be processed in parallel.
[0470] (Embodiment 8)
[0471] In this embodiment, a method for controlling references during the encoding of an accusation code is described. Additionally, while the operation of a three-dimensional data encoding device is mainly described below, the same processing may also be performed in a three-dimensional data decoding device.
[0472] FIGS. 51 and FIGS. 52 are drawings showing reference relationships related to the present embodiment, FIG. 51 is a drawing showing reference relationships in an octal tree structure, and FIG. 52 is a drawing showing reference relationships in a spatial domain.
[0473] In this embodiment, when the three-dimensional data encoding device encodes the encoding information of a node to be encoded (hereinafter referred to as the target node), it refers to the encoding information of each node within the parent node to which the target node belongs. However, it does not refer to the encoding information of each node within another node in the same layer as the parent node (hereinafter referred to as the parent adjacent node). In short, the three-dimensional data encoding device sets the referencing of the parent adjacent node to be disabled or prohibits the referencing.
[0474] Additionally, the three-dimensional data encoding device may allow reference to encoding information within the parent node to which the parent node belongs (hereinafter referred to as the grandparent node). In short, the three-dimensional data encoding device may encode the encoding information of a target node by referencing the encoding information of the parent node and the grandparent node to which the target node belongs.
[0475] Here, encoding information refers to, for example, an occupancy code. When the three-dimensional data encoding device encodes the occupancy code of a target node, it refers to information indicating whether a point cloud is included in each node within the parent node to which the target node belongs (hereinafter referred to as occupancy information). In other words, when the three-dimensional data encoding device encodes the occupancy code of a target node, it refers to the occupancy code of the parent node. On the other hand, the three-dimensional data encoding device does not refer to the occupancy information of each node within the parent adjacent node. In short, the three-dimensional data encoding device does not refer to the occupancy code of the parent adjacent node. Furthermore, the three-dimensional data encoding device may refer to the occupancy information of each node within the grandfather node. In short, the three-dimensional data encoding device may refer to the occupancy information of the parent node and the parent adjacent node.
[0476] For example, when encoding the occupancy code of a target node, the three-dimensional data encoding device switches the encoding table used when entropy encoding the target node's occupancy code by using the occupancy code of the parent node or grandfather node to which the target node belongs. Further details regarding this will be described later. In this case, the three-dimensional data encoding device does not need to refer to the occupancy code of the parent adjacent node. Thus, when encoding the target node's occupancy code, the three-dimensional data encoding device can appropriately switch the encoding table according to the information of the occupancy code of the parent node or grandfather node, thereby improving encoding efficiency. Furthermore, by not referring to the parent adjacent node, the three-dimensional data encoding device can reduce the processing of verifying information about the parent adjacent node and the memory capacity required to store it. Additionally, it becomes easier to encode the occupancy code of each node of the octal tree by scanning in depth priority order.
[0477] Below, an example of encoding table conversion using the occupancy code of a parent node is described. FIG. 53 is a diagram showing an example of a target node and an adjacent reference node. FIG. 54 is a diagram showing the relationship between a parent node and a node. FIG. 55 is a diagram showing an example of the occupancy code of a parent node. Here, an adjacent reference node is a node that is referenced during the encoding of the target node among nodes that are spatially adjacent to the target node. In the example shown in FIG. 53, the adjacent node is a node belonging to the same layer as the target node. Also, as reference adjacent nodes, node X adjacent to the x-direction, node Y adjacent to the y-direction, and node Z adjacent to the z-direction of the target block are used. In short, one adjacent block is set as a reference adjacent block for each of the x, y, and z directions.
[0478] In addition, the node number shown in FIG. 54 is an example, and the relationship between the node number and the position of the node is not limited to this. Also, in FIG. 55, node 0 is assigned to the lower bit and node 7 is assigned to the upper bit, but the assignment may be performed in the reverse order. Also, each node may be assigned to any bit.
[0479] The three-dimensional data encoding device determines the encoding table when entropy encoding the accusation code of the target node, for example, by the following equation.
[0480] CodingTable=(FlagX<<2)+(FlagY<<1)+(FlagZ)
[0481] Here, CodingTable represents a coding table for the occupancy code of the target node and represents any one of values from 0 to 7. FlagX is occupancy information of adjacent node X, and represents 1 if adjacent node X contains a point cloud (occupies), and 0 otherwise. FlagY is occupancy information of adjacent node Y, and represents 1 if adjacent node Y contains a point cloud (occupies), and 0 otherwise. FlagZ is occupancy information of adjacent node Z, and represents 1 if adjacent node Z contains a point cloud (occupies), and 0 otherwise.
[0482] In addition, since information indicating whether an adjacent node is occupied is included in the occupancy code of the parent node, the three-dimensional data encoding device may select an encoding table using the value appearing in the occupancy code of the parent node.
[0483] Based on the above, the three-dimensional data encoding device can improve encoding efficiency by switching the encoding table using information indicating whether a point cloud is included in an adjacent node of a target node.
[0484] In addition, the three-dimensional data encoding device may switch adjacent reference nodes according to the spatial position of the target node within the parent node, as shown in FIG. 53. In short, the three-dimensional data encoding device may switch the referenced adjacent node among a plurality of adjacent nodes according to the spatial position of the target node within the parent node.
[0485] Next, an example of the configuration of a three-dimensional data encoding device and a three-dimensional data decoding device will be described. FIG. 56 is a block diagram of a three-dimensional data encoding device (2100) related to the present embodiment. The three-dimensional data encoding device (2100) shown in FIG. 56 comprises an octal tree generation unit (2101), a geometric information calculation unit (2102), an encoding table selection unit (2103), and an entropy encoding unit (2104).
[0486] The octal tree generation unit (2101) generates, for example, an octal tree from input three-dimensional points (point cloud) and generates an occupancy code for each node included in the octal tree. The geometric information calculation unit (2102) obtains occupancy information indicating whether the adjacent reference node of the target node is occupied. For example, the geometric information calculation unit (2102) obtains the occupancy information of the adjacent reference node from the occupancy code of the parent node to which the target node belongs. Additionally, the geometric information calculation unit (2102) may switch the adjacent reference node according to its position within the parent node of the target node, as shown in FIG. 53. Also, the geometric information calculation unit (2102) does not refer to the occupancy information of each node within the parent adjacent node.
[0487] The encoding table selection unit (2103) selects an encoding table to be used for entropy encoding of the occupancy code of a target node using the occupancy information of an adjacent reference node calculated by the geometric information calculation unit (2102). The entropy encoding unit (2104) generates a bit stream by entropy encoding the occupancy code using the selected encoding table. Additionally, the entropy encoding unit (2104) may add information representing the selected encoding table to the bit stream.
[0488] FIG. 57 is a block diagram of a three-dimensional data decoding device (2110) related to the present embodiment. The three-dimensional data decoding device (2110) shown in FIG. 57 comprises an octal tree generation unit (2111), a geometric information calculation unit (2112), an encoding table selection unit (2113), and an entropy decoding unit (2114).
[0489] The octal tree generation unit (2111) generates an octal tree of a space (node) using header information of a bit stream, etc. For example, the octal tree generation unit (2111) generates a large space (root node) using the size of the x-axis, y-axis, and z-axis directions of the space added to the header information, and generates an octal tree by creating eight small spaces A (nodes A0 to A7) by dividing the space into two in the x-axis, y-axis, and z-axis directions, respectively. Also, nodes A0 to A7 are set in order as target nodes.
[0490] The geometric information calculation unit (2112) obtains occupancy information indicating whether the adjacent reference node of the target node is occupied. For example, the geometric information calculation unit (2112) obtains occupancy information of the adjacent reference node from the occupancy code of the parent node to which the target node belongs. Additionally, the geometric information calculation unit (2112) may switch the adjacent reference node according to its position within the parent node of the target node, as shown in FIG. 53. Also, the geometric information calculation unit (2112) does not refer to the occupancy information of each node within the parent adjacent node.
[0491] The encoding table selection unit (2113) selects an encoding table (decoding table) to be used for entropy decoding of the occupancy code of a target node using the occupancy information of an adjacent reference node calculated by the geometric information calculation unit (2112). The entropy decoding unit (2114) generates a three-dimensional point by entropy decoding the occupancy code using the selected encoding table. Additionally, the encoding table selection unit (2113) may decode and obtain information of the selected encoding table added to the bit stream, and the entropy decoding unit (2114) may use the encoding table represented by the obtained information.
[0492] Each bit of the occupancy code (8 bits) included in the bit stream indicates whether a point cloud is included in each of the 8 subspaces A (nodes A0 to A7). Additionally, the three-dimensional data decoding device divides the subspace node A0 into 8 subspaces B (nodes B0 to B7) to generate an octal tree, and obtains information indicating whether a point cloud is included in each node of the subspace B by decoding the occupancy code. In this way, the three-dimensional data decoding device generates an octal tree from the large space to the small space and decodes the occupancy code of each node.
[0493] The flow of processing by the three-dimensional data encoding device and the three-dimensional data decoding device is described below. FIG. 58 is a flowchart of three-dimensional data encoding processing in the three-dimensional data encoding device. First, the three-dimensional data encoding device determines (defines) a space (target node) that includes some or all of the input three-dimensional point cloud (S2101). Next, the three-dimensional data encoding device divides the target node into eight subspaces (nodes) to generate eight subspaces (nodes) (S2102). Next, the three-dimensional data encoding device generates an occlusion code for the target node based on whether the point cloud is included in each node (S2103).
[0494] Next, the three-dimensional data encoding device calculates (obtains) occupancy information of the adjacent reference node of the target node from the occupancy code of the parent node of the target node (S2104). Next, the three-dimensional data encoding device selects an encoding table to be used for entropy encoding based on the determined occupancy information of the adjacent reference node of the target node (S2105). Next, the three-dimensional data encoding device entropy-encodes the occupancy code of the target node using the selected encoding table (S2106).
[0495] In addition, the three-dimensional data encoding device divides each node into 8 parts and repeats the process of encoding the occupancy code of each node until the node can no longer be divided (S2107). In short, the process from steps S2102 to S2106 is repeated recursively.
[0496] FIG. 59 is a flowchart of a three-dimensional data decoding method in a three-dimensional data decoding device. First, the three-dimensional data decoding device determines (defines) a space to be decoded (target node) using header information of a bit stream (S2111). Next, the three-dimensional data decoding device divides the target node into eight sub-spaces (nodes) to create eight sub-spaces (nodes) (S2112). Next, the three-dimensional data decoding device calculates (obtains) occupancy information of adjacent reference nodes of the target node from the occupancy code of the parent node of the target node (S2113).
[0497] Next, the three-dimensional data decoder selects an encoding table to be used for entropy decoding based on the occupancy information of an adjacent reference node (S2114). Next, the three-dimensional data decoder entropy decodes the occupancy code of the target node using the selected encoding table (S2115).
[0498] In addition, the three-dimensional data decoding device divides each node into 8 parts and repeats the process of decoding the occlusion code of each node until the node can no longer be divided (S2116). In short, the process from steps S2112 to S2115 is repeated recursively.
[0499] Next, an example of a conversion of the encoding table is explained. FIG. 60 is a diagram illustrating an example of a conversion of the encoding table. For example, as shown in encoding table 0 in FIG. 60, the same context model may be applied to multiple occupancy codes. Alternatively, a different context model may be assigned to each occupancy code. By doing so, the context model can be assigned according to the probability of occurrence of the occupancy code, thereby improving encoding efficiency. Furthermore, a context model that updates the probability table according to the frequency of occurrence of the occupancy code may be used. Alternatively, a context model with a fixed probability table may be used.
[0500] Hereinafter, a variation of the present embodiment 1 will be described. FIG. 61 is a diagram showing the reference relationship in the present variation. In the above embodiment, the three-dimensional data encoding device does not refer to the occupancy code of the parent adjacent node, but the decision of whether to refer to the occupancy code of the parent adjacent node may be switched according to specific conditions.
[0501] For example, when a three-dimensional data encoding device performs encoding while scanning an octal tree in a width-first manner, it refers to the occupancy information of nodes within parent adjacent nodes to encode the occupancy code of the target node. On the other hand, when a three-dimensional data encoding device performs encoding while scanning an octal tree in a depth-first manner, it prohibits the referencing of the occupancy information of nodes within parent adjacent nodes. In this way, by switching appropriately referenceable nodes according to the scanning order (encoding order) of the nodes in the octal tree, it is possible to realize improved encoding efficiency and suppression of processing load.
[0502] In addition, the three-dimensional data encoding device may add information to the header of the bit stream, such as whether the octet tree was encoded in width-first or depth-first. FIG. 62 is a diagram showing an example of the syntax of the header information in this case. The octree_scan_order shown in FIG. 62 is encoding order information (encoding order flag) indicating the encoding order of the octet tree. For example, if octree_scan_order is 0, it indicates width-first, and if it is 1, it indicates depth-first. Thus, the three-dimensional data decoding device can determine whether the bit stream was encoded in width-first or depth-first by referring to octree_scan_order, and thus can properly decode the bit stream.
[0503] Additionally, the three-dimensional data encoding device may add information indicating whether to prohibit referencing a parent adjacent node to the header information of the bit stream. FIG. 63 is a diagram showing an example of the syntax of the header information in this case. limit_refer_flag is prohibition switching information (prohibition switching flag) indicating whether to prohibit referencing a parent adjacent node. For example, if limit_refer_flag is 1, it indicates that referencing a parent adjacent node is prohibited, and if it is 0, it indicates no reference restriction (referencing a parent adjacent node is permitted).
[0504] In short, the three-dimensional data encoding device determines whether to prohibit the reference of a parent adjacent node, and switches whether to prohibit or allow the reference of a parent adjacent node based on the result of the above decision. In addition, the three-dimensional data encoding device generates a bit stream including prohibition switching information indicating whether to prohibit the reference of a parent adjacent node as a result of the above decision.
[0505] In addition, the three-dimensional data decoding device obtains prohibition switching information from a bit stream indicating whether to prohibit the reference of a parent adjacent node, and switches whether to prohibit or allow the reference of a parent adjacent node based on the prohibition switching information.
[0506] Thus, the three-dimensional data encoding device can generate a bit stream by controlling the reference of a parent adjacent node. Additionally, the three-dimensional data decoding device can obtain information from the header of the bit stream indicating whether the reference of a parent adjacent node is prohibited.
[0507] In addition, in this embodiment, the encoding process of an occupancy code is described as an example of an encoding process that prohibits referencing parent adjacent nodes, but it is not necessarily limited to this. For example, the same method can be applied when encoding other information of an octal tree node. For example, the method of this embodiment may be applied when encoding other attribute information such as color, normal vector, or reflectance attached to a node. In addition, the same method can be applied when encoding an encoding table or a predicted value.
[0508] Next, a variation of the present embodiment 2 will be described. In the above description, an example in which three reference adjacent nodes are used was shown as illustrated in FIG. 53, but four or more reference adjacent nodes may be used. FIG. 64 is a diagram showing an example of a target node and a reference adjacent node.
[0509] For example, a three-dimensional data encoding device calculates an encoding table when entropy-coding the accusation code of a target node shown in FIG. 64, for example, by the following equation.
[0510] CodingTable=(FlagX0<<3)+(FlagX1<<2)+(FlagY<<1)+(FlagZ)
[0511] Here, CodingTable represents a coding table for the occupancy code of the target node and represents any one of the values 0 to 15. FlagXN is occupancy information of adjacent node XN (N=0..1), and represents 1 if adjacent node XN is (occupied) containing a point cloud, and 0 otherwise. FlagY is occupancy information of adjacent node Y, and represents 1 if adjacent node Y is (occupied) containing a point cloud, and 0 otherwise. FlagZ is occupancy information of adjacent node Z, and represents 1 if adjacent node Z is (occupied) containing a point cloud, and 0 otherwise.
[0512] In this case, if an adjacent node, for example, adjacent node X0 in FIG. 64, is not referenced (reference prohibited), the three-dimensional data encoding device may use a fixed value such as 1 (occupied) or 0 (unoccupied) as a substitute value.
[0513] FIG. 65 is a diagram illustrating examples of a target node and an adjacent node. As shown in FIG. 65, if the adjacent node is not referenced (reference prohibited), the occupancy information of the adjacent node may be calculated by referencing the occupancy code of the grandfather node of the target node. For example, the three-dimensional data encoding device may calculate FlagX0 of the above equation using the occupancy information of the adjacent node G0 instead of the adjacent node X0 shown in FIG. 65, and determine the value of the encoding table using the calculated FlagX0. Additionally, the adjacent node G0 shown in FIG. 65 is an adjacent node that can determine whether it is occupied by the occupancy code of the grandfather node. The adjacent node X1 is an adjacent node that can determine whether it is occupied by the occupancy code of the parent node.
[0514] Hereinafter, a variation of the present embodiment 3 will be described. FIGS. 66 and FIGS. 67 are drawings showing reference relationships related to the present variation, FIG. 66 is a drawing showing reference relationships in an octal tree structure, and FIG. 67 is a drawing showing reference relationships in a spatial domain.
[0515] In the present variation, when the three-dimensional data encoding device encodes the encoding information of a node to be encoded (hereinafter referred to as target node 2), it refers to the encoding information of each node within the parent node to which target node 2 belongs. In short, the three-dimensional data encoding device allows the reference of information (e.g., occupancy information) of a child node of a first node among a plurality of adjacent nodes, where the target node and the parent node are the same. For example, when the three-dimensional data encoding device encodes the occupancy code of target node 2 shown in FIG. 66, it refers to a node existing within the parent node to which target node 2 belongs, for example, the occupancy code of the target node shown in FIG. 66. The occupancy code of the target node shown in FIG. 66 indicates whether each node within the target node adjacent to target node 2 is occupied, for example, as shown in FIG. 67. Therefore, the three-dimensional data encoding device can switch the encoding table of the occupancy code of target node 2 according to the finer shape of the target node, thereby improving encoding efficiency.
[0516] The three-dimensional data encoding device may calculate the encoding table when entropy-coding the accusation code of target node 2, for example, by the following equation.
[0517] CodingTable=(FlagX1<<5)+(FlagX2<<4)+(FlagX3<<3)+(FlagX4<<2)+(FlagY<<1)+(FlagZ)
[0518] Here, CodingTable represents a coding table for the occupancy code of target node 2 and represents any one of the values 0 to 63. FlagXN is occupancy information of adjacent node XN (N=1..4), and represents 1 if adjacent node XN is (occupied) containing a point cloud, and 0 otherwise. FlagY is occupancy information of adjacent node Y, and represents 1 if adjacent node Y is (occupied) containing a point cloud, and 0 otherwise. FlagZ is occupancy information of adjacent node Y, and represents 1 if adjacent node Z is (occupied) containing a point cloud, and 0 otherwise.
[0519] In addition, the three-dimensional data encoding device may change the method of calculating the encoding table according to the node position of the target node 2 within the parent node.
[0520] Additionally, the three-dimensional data encoding device may refer to the encoding information of each node within the parent adjacent node if the referencing of the parent adjacent node is not prohibited. For example, if the referencing of the parent adjacent node is not prohibited, the referencing of the information (e.g., occupancy information) of the child node of a third node that is different from the target node and the parent node is permitted. For example, in the example shown in FIG. 65, the three-dimensional data encoding device obtains the occupancy information of the child node of the adjacent node X0 by referring to the occupancy code of the adjacent node X0 that is different from the target node and the parent node. Based on the occupancy information of the child node of the adjacent node X0 obtained, the three-dimensional data encoding device switches the encoding table used for entropy encoding of the occupancy code of the target node.
[0521] As described above, the three-dimensional data encoding device related to the present embodiment encodes information (e.g., occupancy code) of a target node included in an N (N is an integer greater than or equal to 2) binary tree structure of a plurality of three-dimensional points included in the three-dimensional data. As shown in FIGS. 51 and 52, in the encoding, the three-dimensional data encoding device permits the reference of information (e.g., occupancy information) of a first node among a plurality of adjacent nodes spatially adjacent to the target node, where the target node and the parent node are the same, and prohibits the reference of information (e.g., occupancy information) of a second node where the target node and the parent node are different. In other words, in the encoding, the three-dimensional data encoding device permits the reference of information (e.g., occupancy code) of the parent node and prohibits the reference of information (e.g., occupancy code) of another node (parent adjacent node) in the same layer as the parent node.
[0522] According to this, the three-dimensional data encoding device can improve encoding efficiency by referencing information of a first node, in which the target node and the parent node are the same, among a plurality of adjacent nodes spatially adjacent to the target node. In addition, the three-dimensional data encoding device can reduce throughput by not referencing information of a second node, in which the target node and the parent node are different, among a plurality of adjacent nodes. In this way, the three-dimensional data encoding device can improve encoding efficiency and reduce throughput.
[0523] For example, the three-dimensional data encoding device also determines whether to prohibit the referencing of information of the second node, and in the encoding, switches whether to prohibit or allow the referencing of information of the second node based on the result of the determination. The three-dimensional data encoding device also generates a bit stream including prohibition switching information (e.g., limit_refer_flag shown in FIG. 63) indicating whether to prohibit the referencing of information of the second node as a result of the determination.
[0524] According to this, the three-dimensional data encoding device can switch whether to prohibit the reference of information of the second node. In addition, the three-dimensional data decoding device can perform decoding processing appropriately using the prohibition switching information.
[0525] For example, the information of the target node is information indicating whether a three-dimensional point exists in each of the child nodes belonging to the target node (e.g., an occupancy code), the information of the first node is information indicating whether a three-dimensional point exists in the first node (occupancy information of the first node), and the information of the second node is information indicating whether a three-dimensional point exists in the second node (occupancy information of the second node).
[0526] For example, a three-dimensional data encoding device selects an encoding table based on whether a three-dimensional point exists at a first node in the encoding, and entropy-encodes information of a target node (e.g., an accusation code) using the selected encoding table.
[0527] For example, in the encoding, the three-dimensional data encoding device allows the reference of information (e.g., occupancy information) of a child node of a first node among a plurality of adjacent nodes, as shown in FIGS. 66 and 67.
[0528] According to this, the three-dimensional data encoding device can improve encoding efficiency because it can refer to more detailed information of adjacent nodes.
[0529] For example, as shown in FIG. 53, the three-dimensional data encoding device switches the referenced adjacent node among a plurality of adjacent nodes according to the spatial position within the parent node of the target node in the encoding.
[0530] According to this, the three-dimensional data encoding device can refer to an appropriate adjacent node based on the spatial location within the parent node of the target node.
[0531] For example, a three-dimensional data encoding device is equipped with a processor and memory, and the processor uses the memory to perform the above processing.
[0532] In addition, the three-dimensional data decoding device related to the present embodiment decodes information of a target node (e.g., occupancy code) included in a truth structure of N (where N is an integer greater than or equal to 2) of a plurality of three-dimensional points included in the three-dimensional data. As shown in FIGS. 51 and 52, the three-dimensional data decoding device, in the decoding, permits the reference of information of a first node (e.g., occupancy information) among a plurality of adjacent nodes spatially adjacent to the target node, where the target node and the parent node are the same, and prohibits the reference of information of a second node (e.g., occupancy information) where the target node and the parent node are different. In other words, in the decoding, the three-dimensional data decoding device permits the reference of information of a parent node (e.g., occupancy code) and prohibits the reference of information of another node (parent adjacent node) in the same layer as the parent node (e.g., occupancy code).
[0533] According to this, the three-dimensional data decoding device can improve encoding efficiency by referencing information of a first node, in which the target node and the parent node are the same, among a plurality of adjacent nodes spatially adjacent to the target node. In addition, the three-dimensional data decoding device can reduce throughput by not referencing information of a second node, in which the target node and the parent node are different, among a plurality of adjacent nodes. In this way, the three-dimensional data decoding device can improve encoding efficiency and reduce throughput.
[0534] For example, the three-dimensional data decoding device also obtains prohibition switching information (e.g., limit_refer_flag shown in FIG. 63) from the bit stream indicating whether to prohibit the referencing of information of the second node, and in the decoding, switches whether to prohibit or allow the referencing of information of the second node based on the prohibition switching information.
[0535] According to this, the three-dimensional data decoding device can perform decoding processing appropriately using prohibition switching information.
[0536] For example, the information of the target node is information indicating whether a three-dimensional point exists in each of the child nodes belonging to the target node (e.g., an occupancy code), the information of the first node is information indicating whether a three-dimensional point exists in the first node (occupancy information of the first node), and the information of the second node is information indicating whether a three-dimensional point exists in the second node (occupancy information of the second node).
[0537] For example, in the decoding, the three-dimensional data decoding device selects an encoding table based on whether a three-dimensional point exists in the first node, and uses the selected encoding table to entropy decode information of the target node (e.g., accusation code).
[0538] For example, in the decoding, the three-dimensional data decoding device allows the reference of information (e.g., occupancy information) of a child node of a first node among a plurality of adjacent nodes, as shown in FIGS. 66 and 67.
[0539] According to this, the three-dimensional data decoding device can improve encoding efficiency because it can refer to more detailed information of adjacent nodes.
[0540] For example, as shown in FIG. 53, the three-dimensional data decoding device switches the referenced adjacent node among a plurality of adjacent nodes according to the spatial position within the parent node of the target node during decoding.
[0541] According to this, the three-dimensional data decoding device can refer to an appropriate adjacent node based on the spatial location within the parent node of the target node.
[0542] For example, a three-dimensional data decoding device is equipped with a processor and memory, and the processor uses the memory to perform the above processing.
[0543] (Embodiment 9)
[0544] The information of a three-dimensional point group includes location information (geometry) and attribute information. The location information includes coordinates (x-coordinate, y-coordinate, z-coordinate) based on a certain point. When encoding location information, instead of directly encoding the coordinates of each three-dimensional point, a method is used to reduce the amount of encoding by representing the location of each three-dimensional point as an octal tree representation and encoding the information of the octal tree.
[0545] Meanwhile, attribute information includes information representing the color information (RGB, YUV, etc.), reflectance, and normal vector of each three-dimensional point. For example, a three-dimensional data encoding device may encode the attribute information using an encoding method different from that of the position information.
[0546] In this embodiment, a method for encoding attribute information is described. Additionally, in this embodiment, integer values are used as the values of the attribute information. For example, when each color component of color information RGB or YUV has 8-bit precision, each color component takes an integer value from 0 to 255. When the reflectance value has 10-bit precision, the reflectance value takes an integer value from 0 to 1023. Furthermore, when the bit precision of the attribute information is fractional precision, the three-dimensional data encoding device may multiply the value by a scale value and then round to an integer value so that the attribute information value becomes an integer value. Additionally, the three-dimensional data encoding device may add this scale value to a header of a bit stream, etc.
[0547] As a method for encoding attribute information of three-dimensional points, one can consider calculating a predicted value of the attribute information of the three-dimensional points and encoding the difference (prediction residual) between the original attribute information value and the predicted value. For example, if the attribute information value of a three-dimensional point p is Ap and the predicted value is Pp, the three-dimensional data encoding device encodes the absolute difference value Diffp = |Ap - Pp|. In this case, if the predicted value Pp can be generated with high precision, the value of the absolute difference value Diffp becomes smaller. Therefore, the amount of encoding can be reduced by entropy encoding the absolute difference value Diffp using, for example, an encoding table where the number of generated bits decreases as the value becomes smaller.
[0548] As a method for generating predicted values of attribute information, one can consider using attribute information of a reference point, which is another three-dimensional point located around the target three-dimensional point of the encoding target. Here, a reference point is a three-dimensional point located within a predetermined distance range from the target three-dimensional point. For example, if there exists a target three-dimensional point p=(x1, y1, z1) and a three-dimensional point q=(x2, y2, z2), the three-dimensional data encoding device calculates the Euclidean distance d(p, q) between the three-dimensional point p and the three-dimensional point q, represented by (Equation A1).
[0549]
[0550] A three-dimensional data encoding device determines that if the Euclidean distance d(p, q) is smaller than a predetermined threshold THd, the location of three-dimensional point q is close to the location of the target three-dimensional point p, and determines that the value of the attribute information of three-dimensional point q is used to generate the predicted value of the attribute information of the target three-dimensional point p. Additionally, the distance calculation method may be any other method, and for example, the Mahalanobis distance may be used. Additionally, the three-dimensional data encoding device may determine that three-dimensional points outside the predetermined distance range from the target three-dimensional point are not used in the prediction process. For example, if a three-dimensional point r exists and the distance d(p, r) between the target three-dimensional point p and the three-dimensional point r is greater than or equal to the threshold THd, the three-dimensional data encoding device may determine that the three-dimensional point r is not used in the prediction. Additionally, the three-dimensional data encoding device may add information indicating the threshold THd to the header of a bit stream, etc.
[0551] FIG. 68 is a diagram showing an example of a three-dimensional point. In this example, the distance d(p, q) between the target three-dimensional point p and the three-dimensional point q is smaller than the threshold THd. Therefore, the three-dimensional data encoding device determines that the three-dimensional point q is a reference three-dimensional point of the target three-dimensional point p, and determines that the value of the attribute information Aq of the three-dimensional point q is used to generate the predicted value Pp of the attribute information Ap of the target three-dimensional point p.
[0552] Meanwhile, the distance d(p, r) between the target three-dimensional point p and the three-dimensional point r is greater than or equal to the threshold THd. Therefore, the three-dimensional data encoding device determines that the three-dimensional point r is not a reference three-dimensional point of the target three-dimensional point p, and determines that the value of the attribute information Ar of the three-dimensional point r is not used to generate the predicted value Pp of the attribute information Ap of the target three-dimensional point p.
[0553] Furthermore, when the three-dimensional data encoding device encodes attribute information of a target three-dimensional point using a predicted value, it uses a three-dimensional point for which attribute information has already been encoded and decoded as a reference three-dimensional point. Likewise, when the three-dimensional data decoding device decodes attribute information of a target three-dimensional point using a predicted value, it uses a three-dimensional point for which attribute information has already been decoded as a reference three-dimensional point. As a result, since the same predicted value can be generated during both encoding and decoding, the bit stream of the three-dimensional point generated by encoding can be correctly decoded on the decoding side.
[0554] In addition, when encoding attribute information of three-dimensional points, one can consider classifying each three-dimensional point into multiple layers using the positional information of the three-dimensional points before encoding. Here, each classified layer is called a Level of Detail (LoD). The method for generating LoDs is explained using Fig. 69.
[0555] First, the three-dimensional data encoding device selects an initial point a0 and assigns it to LoD0. Next, the three-dimensional data encoding device extracts a point a1, whose distance from point a0 is greater than the threshold Thres_LoD[0] of LoD0, and assigns it to LoD0. Next, the three-dimensional data encoding device extracts a point a2, whose distance from point a1 is greater than the threshold Thres_LoD[0] of LoD0, and assigns it to LoD0. In this way, the three-dimensional data encoding device configures LoD0 such that the distance between each point within LoD0 becomes greater than the threshold Thres_LoD[0].
[0556] Next, the three-dimensional data encoding device selects point b0, which is not yet assigned to LoD, and assigns it to LoD1. Next, the three-dimensional data encoding device extracts point b1, which is not assigned to LoD and whose distance from point b0 is greater than the threshold Thres_LoD[1] of LoD1, and assigns it to LoD1. Next, the three-dimensional data encoding device extracts point b2, which is not assigned to LoD and whose distance from point b1 is greater than the threshold Thres_LoD[1] of LoD1, and assigns it to LoD1. In this way, the three-dimensional data encoding device configures LoD1 such that the distance between each point within LoD1 is greater than the threshold Thres_LoD[1].
[0557] Next, the three-dimensional data encoding device selects point c0, which is not yet assigned to LoD, and assigns it to LoD2. Next, the three-dimensional data encoding device extracts point c1, which is not assigned to LoD and whose distance from point c0 is greater than the threshold Thres_LoD[2] of LoD2, and assigns it to LoD2. Next, the three-dimensional data encoding device extracts point c2, which is not assigned to LoD and whose distance from point c1 is greater than the threshold Thres_LoD[2] of LoD2, and assigns it to LoD2. In this way, the three-dimensional data encoding device configures LoD2 such that the distance between each point within LoD2 is greater than the threshold Thres_LoD[2]. For example, as shown in FIG. 70, the thresholds Thres_LoD[0], Thres_LoD[1], and Thres_LoD[2] of each LoD are set.
[0558] Additionally, the three-dimensional data encoding device may add information indicating the threshold of each LoD to the header of the bit stream, etc. For example, in the case of the example shown in FIG. 70, the three-dimensional data encoding device may add the thresholds Thres_LoD[0], Thres_LoD[1], and Thres_LoD[2] to the header.
[0559] Additionally, the three-dimensional data encoding device may assign all three-dimensional points that are not assigned to the bottom layer of the LoD. In this case, the three-dimensional data encoding device may reduce the amount of code in the header by not adding the threshold of the bottom layer of the LoD to the header. For example, in the example shown in FIG. 70, the three-dimensional data encoding device adds the thresholds Thres_LoD[0] and Thres_LoD[1] to the header, but does not add Thres_LoD[2] to the header. In this case, the three-dimensional data decoding device may estimate the value of Thres_LoD[2] to be 0. Additionally, the three-dimensional data encoding device may add the number of layers of the LoD to the header. Thus, the three-dimensional data decoding device can determine the LoD of the bottom layer using the number of layers of the LoD.
[0560] In addition, as shown in FIG. 70, by setting the threshold values of each layer of LoD to be larger for higher layers, the upper layers (layers closer to LoD0) become sparse, with distances between three-dimensional points becoming wider, while the lower layers become dense, with distances between three-dimensional points becoming closer. Also, in the example shown in FIG. 70, LoD0 is the uppermost layer.
[0561] Furthermore, the method for selecting the initial three-dimensional point when setting each LoD may depend on the encoding order during position information encoding. For example, the three-dimensional data encoding device selects the three-dimensional point encoded first during position information encoding as the initial point a0 of LoD0, and constructs LoD0 by selecting points a1 and a2 starting from the initial point a0. Then, the three-dimensional data encoding device may select the three-dimensional point that was encoded the earliest among the three-dimensional points not belonging to LoD0 as the initial point b0 of LoD1. In short, the three-dimensional data encoding device may select the three-dimensional point that was encoded the earliest among the three-dimensional points not belonging to the upper layer (LoD0 to LoDn-1) of LoDn as the initial point n0 of LoDn. Thus, the three-dimensional data decoding device can construct the same LoD as during encoding by using the same initial point selection method during decoding, and thus can properly decode the bit stream. Specifically, the three-dimensional data decoding device selects, as the initial point n0 of LoDn, the three-dimensional point whose position information was decoded the fastest among the three-dimensional points that do not belong to the upper layer of LoDn.
[0562] The following describes a method for generating predicted values of attribute information for three-dimensional points using information from the LoD. For example, when a three-dimensional data encoding device encodes sequentially starting from the three-dimensional points included in LoD0, it generates the target three-dimensional points included in LoD1 using attribute information in the state where encoding and decoding are completed (hereinafter simply referred to as "encoding completed") included in LoD0 and LoD1. In this way, the three-dimensional data encoding device generates predicted values of attribute information for three-dimensional points included in LoDn using attribute information that has been encoded completed and included in LoDn' (n' <= n). In short, the three-dimensional data encoding device does not use attribute information of three-dimensional points included in the lower layer of LoDn to calculate the predicted values of attribute information for three-dimensional points included in LoDn.
[0563] For example, a three-dimensional data encoding device generates a predicted value of attribute information for a three-dimensional point by calculating the average of the attribute values of N or fewer three-dimensional points among the encoded three-dimensional points surrounding the target three-dimensional point. Additionally, the three-dimensional data encoding device may append the value of N to the header of a bit stream, etc. Furthermore, the three-dimensional data encoding device may change the value of N for each three-dimensional point and append the value of N for each three-dimensional point. By doing so, an appropriate N can be selected for each three-dimensional point, thereby improving the precision of the predicted value. Consequently, the prediction residual can be reduced. Alternatively, the three-dimensional data encoding device may append the value of N to the header of the bit stream and fix the value of N within the bit stream. This eliminates the need to encode or decode the value of N for each three-dimensional point, thereby reducing throughput. Additionally, the three-dimensional data encoding device may encode the value of N separately for each LoD. By doing so, encoding efficiency can be improved by selecting an appropriate N for each LoD.
[0564] Alternatively, the three-dimensional data encoding device may calculate the predicted value of the attribute information of a three-dimensional point based on the weighted average of the attribute information of N surrounding three-dimensional points that have completed encoding. For example, the three-dimensional data encoding device calculates weights using the respective distance information between the target three-dimensional point and the N surrounding three-dimensional points.
[0565] When a three-dimensional data encoding device encodes the value of N separately for each LoD, for example, the value of N is set larger for the upper layers of the LoD and smaller for the lower layers. In the upper layers of the LoD, since the distance between the three-dimensional points is greater, there is a possibility of improving prediction accuracy by setting a large value of N and selecting and averaging multiple surrounding three-dimensional points. Furthermore, in the lower layers of the LoD, since the distance between the three-dimensional points is closer, it becomes possible to perform efficient prediction by setting a small value of N and suppressing the throughput of averaging.
[0566] FIG. 71 is a diagram showing an example of attribute information used for a predicted value. As described above, the predicted value of point p included in LoDN is generated using encoded surrounding point p' included in LoDN' (N'<=N). Here, the surrounding point p' is selected based on the distance from point p. For example, the predicted value of the attribute information of point b2 shown in FIG. 71 is generated using the attribute information of points a0, a1, a2, b0, and b1.
[0567] Depending on the value of N described above, the selected surrounding points change. For example, when N=5, a0, a1, a2, b0, and b1 are selected as the surrounding points of point b2. When N=4, points a0, a1, a2, and b1 are selected based on distance information.
[0568] The predicted value is calculated by the weighted average of the distance dependence. For example, in the example shown in Fig. 71, the predicted value a2p of point a2 is calculated by the weighted average of the attribute information of points a0 and a1, as indicated by Equation A2 and Equation A3. Also, A i is the value of the attribute information of point ai.
[0569]
[0570] Also, the predicted value b2p of point b2 is calculated by the weighted average of the attribute information of points a0, a1, a2, b0, and b1, as shown in Equations A4 to A6. Also, B i is the value of the attribute information of point bi.
[0571]
[0572] Additionally, the three-dimensional data encoding device may calculate the difference between the attribute information value of a three-dimensional point and the predicted value generated from surrounding points (prediction residual), and quantize the calculated prediction residual. For example, the three-dimensional data encoding device performs quantization by dividing the prediction residual by a quantization scale (also called a quantization step). In this case, the smaller the quantization scale, the smaller the error that may occur due to quantization (quantization error). Conversely, the larger the quantization scale, the larger the quantization error.
[0573] Furthermore, the three-dimensional data encoding device may change the quantization scale used for each LoD. For example, the three-dimensional data encoding device may make the quantization scale smaller for higher layers and larger for lower layers. Since the attribute information values of three-dimensional points belonging to higher layers may be used as predicted values for the attribute information of three-dimensional points belonging to lower layers, encoding efficiency can be improved by reducing the quantization scale of higher layers to suppress quantization errors that may occur in higher layers and increase the precision of the predicted values. Additionally, the three-dimensional data encoding device may add the quantization scale used for each LoD to a header, etc. As a result, the three-dimensional data decoder can correctly decode the quantization scale, thereby enabling proper decoding of the bit stream.
[0574] Furthermore, the three-dimensional data encoding device may convert the signed integer value (signed quantization value), which is the prediction residual after quantization, into an unsigned integer value (unsigned quantization value). This eliminates the need to consider the occurrence of negative integers when entropy-coding the prediction residual. Additionally, the three-dimensional data encoding device is not required to convert the signed integer value into an unsigned integer value; for example, the sign bit may be entropy-coded separately.
[0575] Predicted residuals are calculated by subtracting the predicted value from the original value. For example, the predicted residual a2r for point a2 is calculated by subtracting the predicted value a2p for point a2 from the attribute information value A2 for point a2, as shown in Equation A7. The predicted residual b2r for point b2 is calculated by subtracting the predicted value b2p for point b2 from the attribute information value B2 for point b2, as shown in Equation A8.
[0576] a2r=A2-a2p ···(Equation A7)
[0577] b2r=B2-b2p ···(Equation A8)
[0578] Additionally, the prediction residual is quantized by dividing by the QS (Quantization Step). For example, the quantized value a2q of point a2 is calculated by (Equation A9). The quantized value b2q of point b2 is calculated by (Equation A10). Here, QS_LoD0 is the QS for LoD0, and QS_LoD1 is the QS for LoD1. In short, the QS may be changed according to LoD.
[0579] a2q=a2r / QS_LoD0 ···(Equation A9)
[0580] b2q=b2r / QS_LoD1 ···(Equation A10)
[0581] In addition, the three-dimensional data encoding device converts the quantized value, which is a signed integer value, into an unsigned integer value as follows. If the signed integer value a2q is less than 0, the three-dimensional data encoding device sets the unsigned integer value a2u to -1-(2×a2q). If the signed integer value a2q is 0 or greater, the three-dimensional data encoding device sets the unsigned integer value a2u to 2×a2q.
[0582] Likewise, the three-dimensional data encoding device sets the unsigned integer value b2u to -1-(2×b2q) when the signed integer value b2q is less than 0. The three-dimensional data encoding device sets the unsigned integer value b2u to 2×b2q when the signed integer value b2q is 0 or greater.
[0583] Additionally, the three-dimensional data encoding device may encode the prediction residual (unsigned integer value) after quantization by entropy encoding. For example, after binarizing the unsigned integer value, binary arithmetic encoding may be applied.
[0584] In addition, in this case, the three-dimensional data encoding device may switch the binarization method according to the value of the predicted residual. For example, when the predicted residual pu is smaller than the threshold R_TH, the three-dimensional data encoding device binarizes the predicted residual pu with a fixed number of bits required to represent the threshold R_TH. Also, when the predicted residual pu is greater than or equal to the threshold R_TH, the three-dimensional data encoding device binarizes the value of (pu-R_TH) and the binarized data of the threshold R_TH using Exponential-Golomb, etc.
[0585] For example, if the threshold R_TH is 63 and the predicted residual pu is less than 63, the three-dimensional data encoding device binarizes the predicted residual pu into 6 bits. Also, if the predicted residual pu is 63 or greater, the three-dimensional data encoding device performs arithmetic encoding by binarizing the binary data (111111) of the threshold R_TH and (pu-63) using an exponential guloom.
[0586] In a more specific example, when the predicted residual pu is 32, the three-dimensional data encoding device generates 6 bits of binary data (100000) and arithmetic-encodes this bit sequence. Also, when the predicted residual pu is 66, the three-dimensional data encoding device generates binary data (111111) of the threshold R_TH and a bit sequence (00100) representing the value 3 (66-63) as an exponential gulom, and arithmetic-encodes this bit sequence (111111+00100).
[0587] In this way, the three-dimensional data encoding device makes it possible to encode while suppressing a rapid increase in the number of binarization bits when the prediction residual increases, by switching the binarization method according to the magnitude of the prediction residual. Additionally, the three-dimensional data encoding device may add a threshold R_TH to the header of the bit stream, etc.
[0588] For example, when encoding is performed at a high bit rate—in other words, when the quantization scale is small—the quantization error is small, which increases prediction accuracy, and consequently, there is a possibility that the prediction residual will not increase. Therefore, in this case, the three-dimensional data encoding device sets the threshold R_TH to a large value. This reduces the likelihood of encoding binarized data with a threshold R_TH, thereby improving encoding efficiency. Conversely, when encoding is performed at a low bit rate—in other words, when the quantization scale is large—the quantization error is large, which decreases prediction accuracy, and consequently, there is a possibility that the prediction residual will increase. Therefore, in this case, the three-dimensional data encoding device sets the threshold R_TH to a small value. This prevents a rapid increase in the bit length of the binarized data.
[0589] Furthermore, the three-dimensional data encoding device may switch the threshold R_TH for each LoD and append the threshold R_TH for each LoD to headers, etc. In short, the three-dimensional data encoding device may switch the binarization method for each LoD. For example, in the upper layer, since the distance between three-dimensional points is large, there is a possibility that the prediction accuracy is poor and consequently the prediction residuals will increase. Therefore, the three-dimensional data encoding device prevents a rapid increase in the bit length of the binarized data by setting the threshold R_TH small for the upper layer. Also, in the lower layer, since the distance between three-dimensional points is close, there is a possibility that the prediction accuracy is high and consequently the prediction residuals will decrease. Therefore, the three-dimensional data encoding device improves encoding efficiency by setting the threshold R_TH large for the lower layer.
[0590] FIG. 72 is a diagram showing an example of an exponential Colombe code, illustrating the relationship between a value before binarization (multi-valued) and a bit after binarization (code). Additionally, the 0s and 1s shown in FIG. 72 may be inverted.
[0591] In addition, the three-dimensional data encoding device applies arithmetic encoding to the binarized data of the predicted residual. This can improve encoding efficiency. Furthermore, when applying arithmetic encoding, the trends in the probability of occurrence of 0 and 1 for each bit may differ between the n-bit code, which is the part of the binarized data binarized with n bits, and the remaining code, which is the part binarized using exponential chromosome. Therefore, the three-dimensional data encoding device may switch the method of applying arithmetic encoding between the n-bit code and the remaining code.
[0592] For example, a three-dimensional data encoding device performs arithmetic encoding for an n-bit code using a different encoding table (probability table) for each bit. At this time, the three-dimensional data encoding device may change the number of encoding tables used for each bit. For example, the three-dimensional data encoding device performs arithmetic encoding for the leading bit b0 of an n-bit code using one encoding table. Also, the three-dimensional data encoding device uses two encoding tables for the next bit b1. Also, the three-dimensional data encoding device switches the encoding table used for arithmetic encoding of bit b1 according to the value of b0 (0 or 1). Likewise, the three-dimensional data encoding device also uses four encoding tables for the next bit b2. Also, the three-dimensional data encoding device switches the encoding table used for arithmetic encoding of bit b2 according to the values of b0 and b1 (0 to 3).
[0593] In this way, when the three-dimensional data encoding device arithmetic-codes each bit bn-1 of an n-bit code, 2 n-1It uses a number of encoding tables. In addition, the three-dimensional data encoding device switches the encoding table used according to the value (occurrence pattern) of the bit preceding bn-1. By doing so, the three-dimensional data encoding device can use an appropriate encoding table for each bit, thereby improving encoding efficiency.
[0594] In addition, the three-dimensional data encoding device may reduce the number of encoding tables used for each bit. For example, when the three-dimensional data encoding device arithmetic-encodes each bit bn-1, the m bits preceding bn-1 (m <n-1)의 값(발생 패턴)에 따라 2 m The encoding tables may be switched. This allows for improved encoding efficiency while suppressing the number of encoding tables used for each bit. Additionally, the three-dimensional data encoding device may update the occurrence probabilities of 0 and 1 in each encoding table based on the values of the binarized data that actually occurred. Furthermore, the three-dimensional data encoding device may fix the occurrence probabilities of 0 and 1 in some bits' encoding tables. This allows for a reduction in throughput as the number of updates to the occurrence probabilities can be suppressed.
[0595] For example, if the n-bit code is b0 b1 b2… -1, there is one encoding table (CTb0) for b0. There are two encoding tables (CTb10, CTb11) for b1. Additionally, the encoding table used switches depending on the value of b0 (0 to 1). There are four encoding tables (CTb20, CTb21, CTb22, CTb23) for b2. Additionally, the encoding table used switches depending on the values of b0 and b1 (0 to 3). The encoding table for bn-1 is 2 n-1 Dog(CTbn0, CTbn1,…CTbn(2 n-1 It is -1). Also, the value of b0b1…bn-2(0~2 n-1 The encoding table used is switched according to -1).
[0596] In addition, the three-dimensional data encoding device, for n-bit codes, does not binarize but 0 to 2 n Arithmetic encoding by m-ary setting the value to -1 (m=2 n ) may be applied. Also, if the three-dimensional data encoding device arithmetic-encodes an n-bit code into m-ary, the three-dimensional data decoding device may also recover the n-bit code by arithmetic-encoding of m-ary.
[0597] FIG. 73 is a diagram illustrating the processing of, for example, the case where the remainder code is an exponential Golomb code. The remainder code, which is a part binarized using an exponential Golomb, includes a prefix part and a suffix part as shown in FIG. 73. For example, a three-dimensional data encoding device switches the encoding table in the prefix part and the suffix part. In short, the three-dimensional data encoding device arithmetic-encodes each bit included in the prefix part using the encoding table for the prefix, and arithmetic-encodes each bit included in the suffix part using the encoding table for the suffix.
[0598] In addition, the three-dimensional data encoding device may update the occurrence probabilities of 0 and 1 in each encoding table according to the values of the binarized data that actually occurred. Alternatively, the three-dimensional data encoding device may fix the occurrence probabilities of 0 and 1 in either encoding table. By doing so, the number of updates to the occurrence probabilities can be suppressed, thereby reducing throughput. For example, the three-dimensional data encoding device may update the occurrence probability for the prefix part and fix the occurrence probability for the suffix part.
[0599] In addition, the three-dimensional data encoding device decodes the prediction residual after quantization by inverse quantization and reconstruction, and uses the decoded value, which is the decoded prediction residual, for prediction after the three-dimensional point of the encoding target. Specifically, the three-dimensional data encoding device calculates an inverse quantization value by multiplying the prediction residual after quantization (quantization value) by a quantization scale, and obtains a decoded value (reconstruction value) by adding the inverse quantization value and the prediction value.
[0600] For example, the inverse quantization value a2iq of point a2 is calculated by (Equation A11) using the quantization value a2q of point a2. The inverse quantization value b2iq of point b2 is calculated by (Equation A12) using the quantization value b2q of point b2. Here, QS_LoD0 is the QS for LoD0, and QS_LoD1 is the QS for LoD1. In short, the QS may change depending on the LoD.
[0601] a2iq=a2q×QS_LoD0 ···(Equation A11)
[0602] b2iq=b2q×QS_LoD1 ···(Equation A12)
[0603] For example, the decoding value a2rec of point a2 is calculated by adding the predicted value a2p of point a2 to the inverse quantization value a2iq of point a2, as shown in Equation A13. The decoding value b2rec of point b2 is calculated by adding the predicted value b2p of point b2 to the inverse quantization value b2iq of point b2, as shown in Equation A14.
[0604] a2rec=a2iq+a2p ···(Equation A13)
[0605] b2rec=b2iq+b2p ···(Equation A14)
[0606] Hereinafter, an example of the syntax of a bit stream related to the present embodiment is described. FIG. 74 is a diagram showing an example of the syntax of an attribute header related to the present embodiment. The attribute header is header information of attribute information. As shown in FIG. 74, the attribute header includes layer number information (NumLoD), three-dimensional score information (NumOfPoint[i]), layer threshold (Thres_Lod[i]), neighbor score information (NumNeighorPoint[i]), prediction threshold (THd[i]), quantization scale (QS[i]), and binarization threshold (R_TH[i]).
[0607] NumLoD information indicates the number of layers of the LoD used.
[0608] The three-dimensional point information (NumOfPoint[i]) represents the number of three-dimensional points belonging to layer i. Additionally, the three-dimensional data encoding device may add three-dimensional point total count information (AllNumOfPoint), which represents the total number of three-dimensional points, to another header. In this case, the three-dimensional data encoding device does not need to add NumOfPoint[NumLoD-1], which represents the number of three-dimensional points belonging to the lowest layer, to the header. In this case, the three-dimensional data decoding device can calculate NumOfPoint[NumLoD-1] by (Equation A15). This allows for a reduction in the amount of code in the header.
[0609]
[0610] The layer threshold (Thres_Lod[i]) is the threshold used to set layer i. The three-dimensional data encoder and the three-dimensional data decoder configure LoDi such that the distance between each point within LoDi is greater than the threshold Thres_LoD[i]. Additionally, the three-dimensional data encoder does not need to add the value of Thres_Lod[NumLoD-1] (lowest layer) to the header. In this case, the three-dimensional data decoder estimates the value of Thres_Lod[NumLoD-1] to 0. This allows for a reduction in the amount of code in the header.
[0611] The neighbor score information (NumNeighorPoint[i]) represents the upper bound of the neighbor scores used to generate the predicted value of a three-dimensional point belonging to layer i. The three-dimensional data encoding unit [is] when the neighbor score M does not reach NumNeighorPoint[i] (M <NumNeighorPoint[i])에는, M개의 주위의 점수를 사용하여 예측값을 산출해도 된다. 또, 삼차원 데이터 부호화 장치는, 각 LoD에서 NumNeighorPoint[i]의 값을 나눌 필요가 없는 경우에는, 모든 LoD에서 사용되는 1개의 주위 점수 정보(NumNeighorPoint)를 헤더에 부가해도 된다.
[0612] The prediction threshold (THd[i]) represents the upper limit of the distance between a target 3D point and surrounding 3D points used for prediction of the target 3D point of the encoding or decoding target in layer i. The 3D data encoding device and the 3D data decoding device do not use 3D points for prediction that are further away from the target 3D point than THd[i]. Additionally, if the 3D data encoding device does not need to divide the value of THd[i] in each LoD, it may add a single prediction threshold (THd) used in all LoDs to the header.
[0613] The quantization scale (QS[i]) represents the quantization scale used in the quantization and inverse quantization of layer i.
[0614] The binarization threshold (R_TH[i]) is a threshold for switching the binarization method of the prediction residuals of the three-dimensional points belonging to layer i. For example, if the prediction residual is smaller than the threshold R_TH, the three-dimensional data encoding device binarizes the prediction residual pu with a fixed number of bits, and if the prediction residual is greater than or equal to the threshold R_TH, it binarizes the binarized data of the threshold R_TH and the value of (pu-R_TH) using exponential gulom. Additionally, if there is no need to switch the value of R_TH[i] in each LoD, the three-dimensional data encoding device may add a single binarization threshold (R_TH) used in all LoDs to the header.
[0615] In addition, R_TH[i] may be the maximum value that can be represented by n bits. For example, R_TH is 63 for 6 bits, and R_TH is 255 for 8 bits. Also, instead of encoding the maximum value that can be represented by n bits as the binarization threshold, the three-dimensional data encoding device may encode the number of bits. For example, the three-dimensional data encoding device may add the value 6 to the header when R_TH[i]=63, and the value 8 when R_TH[i]=255. Also, the three-dimensional data encoding device may define the minimum value of the number of bits representing R_TH[i] (minimum number of bits) and add the number of bits relative to the minimum value to the header. For example, the three-dimensional data encoding device may add the value 0 to the header when R_TH[i]=63 and the minimum number of bits is 6, and add the value 2 to the header when R_TH[i]=255 and the minimum number of bits is 6.
[0616] Additionally, the three-dimensional data encoding device may entropy-encode at least one of NumLoD, Thres_Lod[i], NumNeighborPoint[i], THd[i], QS[i], and R_TH[i] and append it to the header. For example, the three-dimensional data encoding device may binary-encode each value and perform arithmetic encoding. Additionally, the three-dimensional data encoding device may encode each value with a fixed length to suppress throughput.
[0617] In addition, the three-dimensional data encoding device does not need to add at least one of NumLoD, Thres_Lod[i], NumNeighborPoint[i], THd[i], QS[i], and R_TH[i] to the header. For example, at least one of these values may be defined as a profile or level, etc., in a specification. This allows the amount of bits in the header to be reduced.
[0618] FIG. 75 is a diagram showing an example of the syntax of attribute data related to the present embodiment. This attribute data includes encoded data of attribute information of a plurality of three-dimensional points. As shown in FIG. 75, the attribute data includes an n-bit code and a remaining code.
[0619] An n-bit code is encoded data or part thereof of the predicted residual of the attribute information value. The bit length of the n-bit code depends on the value of R_TH[i]. For example, if the value represented by R_TH[i] is 63, the n-bit code is 6 bits, and if the value represented by R_TH[i] is 255, the n-bit code is 8 bits.
[0620] The remaining code is the encoded data encoded in exponential Golomb among the encoded data of the predicted residual of the attribute information value. This remaining code is encoded or decoded when the n-bit code is identical to R_TH[i]. Additionally, the three-dimensional data decoder decodes the predicted residual by adding the value of the n-bit code and the value of the remaining code. Furthermore, if the n-bit code is not identical to R_TH[i], the remaining code does not need to be encoded or decoded.
[0621] The flow of processing in a three-dimensional data encoding device is described below. FIG. 76 is a flowchart of three-dimensional data encoding processing by a three-dimensional data encoding device.
[0622] First, the three-dimensional data encoding device encodes position information (geometry) (S3001). For example, three-dimensional data encoding is performed using an octal tree representation.
[0623] After encoding position information, if the position of a three-dimensional point changes due to quantization or the like, the three-dimensional data encoding device reassigns the attribute information of the original three-dimensional point to the three-dimensional point after the change (S3002). For example, the three-dimensional data encoding device performs the reassignment by interpolating the value of the attribute information according to the amount of change in position. For example, the three-dimensional data encoding device detects N three-dimensional points before the change that are close to the three-dimensional position after the change and weights the value of the attribute information of the N three-dimensional points. For example, in the weighted average, the three-dimensional data encoding device determines the weight based on the distance from the three-dimensional position after the change to each of the N three-dimensional points. Then, the three-dimensional data encoding device determines the value obtained by the weighted average as the value of the attribute information of the three-dimensional point after the change. In addition, if two or more three-dimensional points change to the same three-dimensional position due to quantization or the like, the three-dimensional data encoding device may assign the average value of the attribute information of two or more three-dimensional points before the change as the value of the attribute information of the three-dimensional point after the change.
[0624] Next, the three-dimensional data encoding device encodes attribute information after reallocation (S3003). For example, if the three-dimensional data encoding device encodes multiple types of attribute information, it may encode the multiple types of attribute information in order. For example, if the three-dimensional data encoding device encodes color and reflectance as attribute information, it may generate a bit stream in which the encoding result of reflectance is added after the encoding result of color. Furthermore, the order of the multiple encoding results of attribute information added to the bit stream is not limited to this order and may be any order.
[0625] In addition, the three-dimensional data encoding device may add information indicating the start location of the encoded data for each attribute information within the bit stream to a header, etc. Thus, the three-dimensional data decoding device can selectively decode attribute information that requires decoding, thereby omitting the decoding process of attribute information that does not require decoding. Consequently, the throughput of the three-dimensional data decoding device can be reduced. In addition, the three-dimensional data encoding device may encode multiple types of attribute information in parallel and integrate the encoding results into a single bit stream. Thus, the three-dimensional data encoding device can encode multiple types of attribute information at high speed.
[0626] FIG. 77 is a flowchart of attribute information encoding processing (S3003). First, the three-dimensional data encoding device sets the LoD (S3011). In short, the three-dimensional data encoding device assigns each three-dimensional point to one of a plurality of LoDs.
[0627] Next, the three-dimensional data encoding device initiates a loop in LoD units (S3012). In short, the three-dimensional data encoding device repeats the processing of steps S3013 to S3021 for each LoD.
[0628] Next, the three-dimensional data encoding device initiates a loop at the three-dimensional point level (S3013). In short, the three-dimensional data encoding device repeats the processing of steps S3014 to S3020 for each three-dimensional point.
[0629] First, the three-dimensional data encoding device searches for a plurality of surrounding points, which are three-dimensional points existing around the target three-dimensional point, to be used for calculating the predicted value of the target three-dimensional point of the processing target (S3014). Next, the three-dimensional data encoding device calculates a weighted average of the attribute information values of the plurality of surrounding points and sets the obtained value as the predicted value P (S3015). Next, the three-dimensional data encoding device calculates a prediction residual, which is the difference between the attribute information of the target three-dimensional point and the predicted value (S3016). Next, the three-dimensional data encoding device calculates a quantized value by quantizing the prediction residual (S3017). Next, the three-dimensional data encoding device performs arithmetic encoding on the quantized value (S3018).
[0630] Additionally, the three-dimensional data encoding device calculates an inverse quantization value by inversely quantizing the quantization value (S3019). Next, the three-dimensional data encoding device generates a decoded value by adding a predicted value to the inverse quantization value (S3020). Next, the three-dimensional data encoding device terminates a loop at the three-dimensional point unit level (S3021). Additionally, the three-dimensional data encoding device terminates a loop at the LoD unit level (S3022).
[0631] Hereinafter, three-dimensional data decoding processing in a three-dimensional data decoding device that decodes a bit stream generated by the above-described three-dimensional data encoding device will be described.
[0632] The three-dimensional data decoding device generates decoded binary data by performing arithmetic decoding on the binary data of attribute information within the bit stream generated by the three-dimensional data encoding device in the same manner as the three-dimensional data encoding device. Furthermore, in the case where the application method of arithmetic encoding is switched between the portion binaryized by n bits (n-bit code) and the portion binaryized using an exponentiation (remainder code) in the three-dimensional data encoding device, the three-dimensional data decoding device performs decoding in accordance with that method when applying arithmetic decoding.
[0633] For example, in an arithmetic decoding method for an n-bit code, a three-dimensional data decoder performs arithmetic decoding using a different encoding table (decoding table) for each bit. At this time, the three-dimensional data decoder may change the number of encoding tables used for each bit. For example, arithmetic decoding is performed using one encoding table for the leading bit b0 of an n-bit code. Also, the three-dimensional data decoder uses two encoding tables for the next bit b1. Also, the three-dimensional data decoder switches the encoding table used for arithmetic decoding of bit b1 according to the value of b0 (0 or 1). Likewise, the three-dimensional data decoder also uses four encoding tables for the next bit b2. Also, the three-dimensional data decoder switches the encoding table used for arithmetic decoding of bit b2 according to the values of b0 and b1 (0 to 3).
[0634] Thus, when the three-dimensional data decoding device performs arithmetic decoding of each bit bn-1 of an n-bit code, 2 n-1 It uses a number of encoding tables. In addition, the three-dimensional data decoder switches the encoding table used according to the value (occurrence pattern) of the bit preceding bn-1. By doing so, the three-dimensional data decoder can properly decode a bit stream with improved encoding efficiency by using an appropriate encoding table for each bit.
[0635] In addition, the three-dimensional data decoder may reduce the number of encoding tables used for each bit. For example, when the three-dimensional data decoder performs arithmetic decoding of each bit bn-1, the m bits preceding bn-1 (m <n-1)의 값(발생 패턴)에 따라 2 mThe encoding tables may be switched. By doing so, the three-dimensional data decoder can properly decode a bit stream with improved encoding efficiency while suppressing the number of encoding tables used for each bit. Additionally, the three-dimensional data decoder may update the occurrence probabilities of 0 and 1 in each encoding table according to the values of the binarized data that actually occurred. Furthermore, the three-dimensional data decoder may fix the occurrence probabilities of 0 and 1 in some bits' encoding tables. By doing so, the number of updates to the occurrence probabilities can be suppressed, thereby reducing throughput.
[0636] For example, if the n-bit code is b0 b1 b2… bn-1, there is one encoding table (CTb0) for b0. There are two encoding tables (CTb10, CTb11) for b1. Additionally, the encoding tables are switched according to the value of b0 (0–1). There are four encoding tables (CTb20, CTb21, CTb22, CTb23) for b2. Additionally, the encoding tables are switched according to the values of b0 and b1 (0–3). The encoding table for bn-1 is 2 n-1 Dog(CTbn0, CTbn1,…CTbn(2 n-1 It is -1). Also, the value of b0b1…bn-2(0~2 n-1 The encoding table is switched according to -1).
[0637] FIG. 78 is a diagram illustrating the processing of, for example, the case where the remainder code is an exponential Golomb code. The portion (remainder code) encoded by the three-dimensional data encoding device using exponential Golomb includes a prefix section and a suffix section as shown in FIG. 78. For example, the three-dimensional data decoding device switches the encoding table in the prefix section and the suffix section. In short, the three-dimensional data decoding device performs arithmetic decoding of each bit included in the prefix section using the encoding table for the prefix, and performs arithmetic decoding of each bit included in the suffix section using the encoding table for the suffix.
[0638] In addition, the three-dimensional data decoder may update the occurrence probabilities of 0 and 1 in each encoding table according to the values of the binarized data generated during decoding. Alternatively, the three-dimensional data decoder may fix the occurrence probabilities of 0 and 1 in either encoding table. By doing so, the number of updates to the occurrence probabilities can be suppressed, thereby reducing throughput. For example, the three-dimensional data decoder may update the occurrence probabilities for the prefix part and fix the occurrence probabilities for the suffix part.
[0639] Additionally, the three-dimensional data decoder decodes the predicted residual after quantization (unsigned integer value) by multi-valuing the binary data of the arithmetically decoded predicted residual according to the encoding method used in the three-dimensional data encoding device. The three-dimensional data decoder first calculates the value of the decoded n-bit code by arithmetically decoding the binary data of the n-bit code. Next, the three-dimensional data decoder compares the value of the n-bit code with the value of R_TH.
[0640] When the value of an n-bit code matches the value of R_TH, the three-dimensional data decoder determines that the next bit encoded in exponential Golomb is present, and performs arithmetic decoding of the remainder code, which is binary data encoded in exponential Golomb. Then, the three-dimensional data decoder calculates the value of the remainder code from the decoded remainder code using a reverse lookup table that represents the relationship between the remainder code and its value. FIG. 79 is a diagram showing an example of a reverse lookup table that represents the relationship between the remainder code and its value. Next, the three-dimensional data decoder obtains the predicted residual after multi-valued quantization by adding the obtained remainder code value to R_TH.
[0641] Meanwhile, if the value of the n-bit code and the value of R_TH do not match (the value is smaller than R_TH), the three-dimensional data decoder determines the value of the n-bit code as is as the predicted residual after multi-valued quantization. By doing so, the three-dimensional data decoder can properly decode the bit stream generated by switching the binarization method according to the value of the predicted residual in the three-dimensional data encoding device.
[0642] Additionally, if the threshold R_TH is appended to the header of the bit stream, the three-dimensional data decoder may decode the value of the threshold R_TH from the header and switch the decoding method using the decoded value of the threshold R_TH. Furthermore, if the threshold R_TH is appended to the header for each LoD, the three-dimensional data decoder switches the decoding method for each LoD using the decoded threshold R_TH.
[0643] For example, when the threshold R_TH is 63 and the value of the decoded n-bit code is 63, the three-dimensional data decoder obtains the value of the remainder code by decoding the remainder code with an exponentiation colom. For example, in the example shown in FIG. 79, the remainder code is 00100 and 3 is obtained as the value of the remainder code. Next, the three-dimensional data decoder obtains the value of the predicted residual 66 by adding the value of the threshold R_TH 63 and the value of the remainder code 3.
[0644] Also, when the value of the decoded n-bit code is 32, the three-dimensional data decoding device sets the value of the n-bit code, 32, as the value of the predicted residual.
[0645] Furthermore, the three-dimensional data decoder converts the predicted residual after decoding from an unsigned integer value to a signed integer value by, for example, processing that is the opposite of the processing in the three-dimensional data encoder. Thus, the three-dimensional data decoder can properly decode a generated bit stream without considering the occurrence of negative integers when entropy-coding the predicted residual. Additionally, the three-dimensional data decoder does not necessarily need to convert the unsigned integer value to a signed integer value; for example, when decoding a bit stream generated by entropy-coding the sign bit separately, the sign bit may be decoded.
[0646] A three-dimensional data decoder generates a decoded value by decoding the predicted residual after quantization, converted into a signed integer value, through inverse quantization and reconstruction. Additionally, the three-dimensional data decoder uses the generated decoded value for prediction after the three-dimensional point of the decoding target. Specifically, the three-dimensional data decoder calculates an inverse quantization value by multiplying the predicted residual after quantization by the decoded quantization scale, and obtains a decoded value by adding the inverse quantization value and the predicted value.
[0647] The decoded unsigned integer value (unsigned quantized value) is converted into a signed integer value by the following process. If the least significant bit (LSB) of the decoded unsigned integer value a2u is 1, the three-dimensional data decoding device sets the signed integer value a2q to -((a2u+1)>>1). If the LSB of the unsigned integer value a2u is not 1, the three-dimensional data decoding device sets the signed integer value a2q to (a2u>>1).
[0648] Likewise, the three-dimensional data decoding device sets the signed integer value b2q to -((b2u+1)>>1) when the LSB of the decoded unsigned integer value b2u is 1. The three-dimensional data decoding device sets the signed integer value b2q to (b2u>>1) when the LSB of the unsigned integer value n2u is not 1.
[0649] In addition, the details of the inverse quantization and reconstruction processing by the three-dimensional data decoding device are the same as the inverse quantization and reconstruction processing in the three-dimensional data encoding device.
[0650] The flow of processing in a three-dimensional data decoding device is described below. FIG. 80 is a flowchart of three-dimensional data decoding processing by a three-dimensional data decoding device. First, the three-dimensional data decoding device decodes position information (geometry) from a bit stream (S3031). For example, the three-dimensional data decoding device performs decoding using an octal tree representation.
[0651] Next, the three-dimensional data decoder decodes attribute information from the bit stream (S3032). For example, when the three-dimensional data decoder decodes multiple types of attribute information, it may decode the multiple types of attribute information in order. For example, when the three-dimensional data decoder decodes color and reflectance as attribute information, it decodes the encoded result of the color and the encoded result of the reflectance according to the order in which they are attached to the bit stream. For example, if the encoded result of the reflectance is attached after the encoded result of the color in the bit stream, the three-dimensional data decoder decodes the encoded result of the color and then decodes the encoded result of the reflectance. Additionally, the three-dimensional dat...
Claims
Claim 1 A three-dimensional data encoding method for encoding a plurality of three-dimensional points, wherein one prediction mode is selected from two or more prediction modes used to calculate a predicted value of attribute information of each three-dimensional point, and based on the selected prediction mode, a predicted value of attribute information of a first three-dimensional point is calculated according to attribute information of one or more second three-dimensional points, and a bit stream including an index representing the selected prediction mode is generated. Claim 2 A three-dimensional data decoding method for decoding a plurality of three-dimensional points, comprising acquiring a bit stream including a prediction residual and an index representing one prediction mode selected from two or more prediction modes used to calculate a prediction value of attribute information of a three-dimensional point, and, based on the acquired prediction mode, calculating a prediction value of attribute information of a first three-dimensional point according to attribute information of one or more second three-dimensional points, and calculating attribute information of the first three-dimensional point by adding the prediction value and the prediction residual.
