Information processing device and method

Abstract

The present disclosure relates to an information processing device and method that are capable of suppressing a decrease in encoding efficiency. The present disclosure describes: generating a predicted residual by decoding encoded data of the predicted residual, the predicted residual being a difference between the intensity of reflected light, which is attribute data of a point cloud representing a three-dimensional object as a set of points, and a predicted value of the intensity of reflected light generated by using a light reflection model for the surface of the object; deriving a coefficient of the light reflection model; deriving a prediction value by performing prediction processing using the light reflection model and the coefficient; and generating a reflected light intensity by adding the predicted residual, which is obtained by decoding the encoded data, to the derived prediction value. The present disclosure can be applied to an information processing device, electronic equipment, image processing method, and program, for example.

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device and method, and more particularly to an information processing device and method that can suppress a decrease in coding efficiency. [Background technology]

[0002] Conventionally, there is LiDAR (Light Detection and Ranging), a sensing technology that irradiates a real space with laser light to detect the distance to an object, the properties of the object, etc. This sensing technology can obtain 3D data with a three-dimensional structure, such as the reflected light intensity for each three-dimensional position (i.e., the reflected light intensity distribution in 3D space), as sensor data. It has been considered to represent this reflected light intensity distribution as attribute data (attribute information) of a point cloud that represents a three-dimensional object as a collection of points (see, for example, Non-Patent Document 1).

[0003] Such 3D data generally contains a large amount of information, and therefore requires compression (encoding) for recording, playback, transmission, etc. For example, in the case of G-PCC (Geometry-based Point Cloud Compression), attribute data is encoded by predictive coding that utilizes correlation with neighboring points (see, for example, Non-Patent Document 2). [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] A. Tatoglu and K. Pochiraju, "Point cloud segmentation with LIDAR reflection intensity behavior," 2012 IEEE International Conference on Robotics and Automation, 2012, pp. 786-790 [Non-patent document 2] "G-PCC Future Enhancements", ISO / IEC 23090-9:2019(E), ISO / IEC JTC 1 / SC 29 / WG 11 N18887, 2019-12-20 Summary of the Invention [Problem to be solved by the invention]

[0005] However, for example, when the inclination of the object surface varies widely, the reflected light intensity varies greatly even if the distance between points is close, which may reduce prediction accuracy and reduce coding efficiency.

[0006] The present disclosure has been made in light of such circumstances, and makes it possible to suppress a decrease in coding efficiency. [Means for solving the problem]

[0007] An information processing device according to one aspect of the present technology is an information processing device that includes: a decoding unit that decodes encoded data of a prediction residual that is the difference between reflected light intensity, which is attribute data of a point cloud that represents a three-dimensional object as a collection of points, and a predicted value of the reflected light intensity generated using a light reflection model on the surface of the object, to generate the prediction residual; a coefficient derivation unit that derives coefficients of the reflection model; a prediction unit that performs a prediction process using the reflection model and the coefficients to derive the predicted value; and a generation unit that generates the reflected light intensity by adding the prediction residual obtained by the decoding unit and the predicted value derived by the prediction unit.

[0008] An information processing method according to one aspect of the present technology is an information processing method that generates the prediction residual by decoding encoded data of a prediction residual that is the difference between reflected light intensity, which is attribute data of a point cloud that represents a three-dimensional object as a collection of points, and a predicted value of the reflected light intensity generated using a light reflection model on the surface of the object, derives coefficients of the reflection model, performs a prediction process using the reflection model and the coefficients to derive the predicted value, and generates the reflected light intensity by adding the prediction residual obtained by decoding the encoded data to the derived predicted value.

[0009] An information processing device according to another aspect of the present technology is an information processing device including: a coefficient derivation unit that derives coefficients of a light reflection model on the surface of a three-dimensional object; a prediction unit that performs a prediction process using the reflection model and the coefficients to derive a predicted value of reflected light intensity, which is attribute data of a point cloud that represents the object as a collection of points; a generation unit that generates a prediction residual, which is the difference between the reflected light intensity and the predicted value derived by the prediction unit; and an encoding unit that encodes the prediction residual generated by the generation unit.

[0010] An information processing method according to another aspect of the present technology is an information processing method that derives coefficients of a light reflection model on the surface of a three-dimensional object, performs prediction processing using the reflection model and the coefficients to derive a predicted value of reflected light intensity, which is attribute data of a point cloud that represents the object as a collection of points, generates a prediction residual, which is the difference between the reflected light intensity and the derived predicted value, and encodes the generated prediction residual.

[0011] In an information processing device and method according to one aspect of the present technology, encoded data of a prediction residual, which is the difference between reflected light intensity, which is attribute data of a point cloud that represents a three-dimensional object as a collection of points, and a predicted value of reflected light intensity generated using a light reflection model on the surface of the object, is decoded to generate a prediction residual, coefficients of the reflection model are derived, a prediction process is performed using the reflection model and its coefficients to derive the predicted value, and the predicted residual obtained by decoding the encoded data and the derived predicted value are added together to generate the reflected light intensity.

[0012] In an information processing device and method according to another aspect of the present technology, coefficients of a light reflection model on the surface of a three-dimensional object are derived, a prediction process is performed using the reflection model and its coefficients, a predicted value of reflected light intensity, which is attribute data of a point cloud that represents the object as a collection of points, a prediction residual, which is the difference between the reflected light intensity and the derived predicted value, is generated, and the generated prediction residual is encoded. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 10 is a diagram illustrating an example of predictive coding using correlation with neighboring points. [Figure 2] FIG. 10 is a diagram illustrating an example of predictive decoding using a reflection model. [Figure 3] FIG. 10 is a diagram illustrating an example of predictive decoding using a reflection model. [Figure 4] FIG. 10 is a diagram illustrating an example of predictive coding using a reflection model. [Figure 5] FIG. 10 is a diagram illustrating an example of predictive coding using a reflection model. [Figure 6] FIG. 10 is a diagram illustrating an example of a reflection model. [Figure 7] FIG. 10 is a diagram illustrating an example of a diffuse reflection model. [Figure 8]10A and 10B are diagrams illustrating an example of predictive decoding using a reflection model when estimating the diffuse reflection coefficient of a target point based on the diffuse reflection coefficient of a neighboring point. [Figure 9] FIG. 10 is a diagram illustrating an example of neighboring points. [Figure 10] FIG. 10 is a diagram illustrating an example of a model of light attenuation over distance. [Figure 11] FIG. 10 is a diagram illustrating an example of a method for calculating an incident angle. [Figure 12] FIG. 10 is a diagram illustrating an example of predictive decoding using a reflection model. [Figure 13] FIG. 10 is a diagram illustrating an example of predictive decoding using a reflection model when a transmitted diffuse reflection coefficient is applied. [Figure 14] 10A and 10B are diagrams illustrating an example of predictive decoding using a reflection model when switching the method of deriving the diffuse reflection coefficient. [Figure 15] FIG. 10 is a diagram illustrating an example of predictive decoding using a reflection model when a transmitted normal vector is applied. [Figure 16] FIG. 10 is a diagram illustrating an example of predictive decoding when a specular reflection model is applied. [Figure 17] 10A and 10B are diagrams illustrating examples of specular reflection model application conditions. [Figure 18] FIG. 1 is a block diagram illustrating an example of the main configuration of an encoding device. [Figure 19] 10 is a block diagram showing an example of the main configuration of an attribute data encoding unit. FIG. [Figure 20] 10 is a flowchart illustrating an example of the flow of an encoding process. [Figure 21] 10 is a flowchart illustrating an example of the flow of an attribute data encoding process. [Figure 22] 10 is a flowchart illustrating an example of the flow of an attribute data encoding process. [Figure 23] 10 is a flowchart illustrating an example of the flow of an attribute data encoding process. [Figure 24] FIG. 2 is a block diagram illustrating an example of the main configuration of a decoding device. [Figure 25]FIG. 10 is a block diagram showing an example of the main configuration of an attribute data decoding unit. [Figure 26] 10 is a flowchart illustrating an example of the flow of a decoding process. [Figure 27] 10 is a flowchart illustrating an example of the flow of an attribute data decoding process. [Figure 28] 10 is a flowchart illustrating an example of the flow of an attribute data decoding process. [Figure 29] 10 is a flowchart illustrating an example of the flow of an attribute data decoding process. [Figure 30] FIG. 1 is a block diagram illustrating an example of the main configuration of a computer. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, modes for carrying out the present disclosure (hereinafter referred to as embodiments) will be described in the following order. 1. Encoding and decoding of LiDAR data 2. Predictive coding and decoding using a reflection model 3. First embodiment (encoding device) 4. Second embodiment (decoding device) 5. Additional Notes

[0015] <1. Encoding and decoding of LiDAR data> <References supporting technical content and technical terminology> The scope of disclosure of the present technology includes not only the contents described in the embodiments but also the contents described in the following non-patent documents that were publicly known at the time of filing, as well as the contents of other documents referenced in the following non-patent documents.

[0016] Non-patent document 1: (mentioned above) Non-patent document 2: (mentioned above)

[0017] In other words, the contents of the above-mentioned non-patent documents and the contents of other documents referenced in the above-mentioned non-patent documents are also used as the basis for determining the support requirements.

[0018] <LiDARデータ> Conventionally, there has been LiDAR (Light Detection and Ranging), a sensing technology that irradiates a real space with laser light to detect the distance to an object, the properties of the object, etc. LiDAR can obtain information such as the reflected light intensity for each three-dimensional position (i.e., the reflected light intensity distribution in 3D space) as sensor data (also referred to as LiDAR data). Therefore, as described in Non-Patent Document 1, for example, it has been considered to represent this reflected light intensity distribution as attribute data (attribute information) of a point cloud that represents a three-dimensional object as a collection of points.

[0019] <Point Cloud> Point cloud data (also referred to as point cloud data) consists of position information (also referred to as geometry) and attribute information (also referred to as attributes) for each point. Attributes can include any information. For example, by using the reflected light intensity of the above-mentioned LiDAR data (reflected light intensity distribution) as an attribute and its position information as geometry, the LiDAR data (reflected light intensity distribution) can be expressed by a point cloud. As mentioned above, point clouds have a relatively simple data structure and can represent any three-dimensional structure with sufficient accuracy by using a sufficient number of points. In other words, they can easily represent LiDAR data (reflected light intensity distribution). Furthermore, by increasing the resolution, LiDAR data (reflected light intensity distribution) can also be represented with high accuracy.

[0020] <Compression of point cloud data> Such 3D data generally contains a large amount of information, and therefore requires compression (encoding) for recording, playback, transmission, etc. One method of encoding point cloud data is, for example, G-PCC (Geometry-based Point Cloud Compression), which is described in Non-Patent Document 2. In this G-PCC, attribute data is encoded by predictive coding that utilizes correlation with neighboring points.

[0021] For example, a reference relationship between neighboring points is constructed using the distance between the points, the attributes of the neighboring points are used as predicted values, and the difference between the attribute of the target point and that predicted value (also called a prediction residual) is coded. Generally, the closer the distance between points, the higher the correlation between the attributes. Therefore, by performing predictive coding as described above using the attributes of neighboring points, coding efficiency can be improved compared to coding the attributes of each point directly.

[0022] For example, suppose there is point cloud data with reflected light intensity as an attribute, as shown in Figure 1A. In Figure 1A, the black circles indicate the geometry (position information) of the points, and the numbers next to them indicate the reflected light intensity at each point. When encoding the reflected light intensity at each point, each value in the center column of the table shown in Figure 1C is encoded.

[0023] In the case of G-PCC predictive coding, as shown in Figure 1B, the reflected light intensity of each point is coded as the difference from the reflected light intensity of its neighboring points. For example, the points are coded starting from the top, with the previously processed point being the neighboring point. In this case, the first point from the top has no neighboring points, so its reflected light intensity is coded as is (intra). In other words, the reflected light intensity of "120" is coded. In contrast, the reflected light intensity of the second point from the top is coded with the first point from the top as its neighbor. In other words, the difference between them, "10", is coded. The differences (prediction residuals) are coded in the same way for the third and subsequent points from the top. In other words, the values ​​in the rightmost column of the table shown in Figure 1C are coded.

[0024] Therefore, predictive coding using G-PCC can reduce the coding value compared to coding each point individually, which means that the increase in the amount of code and the decrease in coding efficiency can be suppressed.

[0025] However, for example, when the inclination of an object surface varies widely, the reflected light intensity may vary significantly even if the distance between points is close. For example, in B of Figure 1, the difference in reflected light intensity between the third and fourth points from the top is larger than the differences between the other points. In other words, in such cases, there is a risk of reduced prediction accuracy and reduced coding efficiency. Additionally, the more complex the three-dimensional shape of the object, the more likely it is that the prediction accuracy and coding efficiency will be reduced.

[0026] 2. Predictive Coding and Decoding Using Reflection Models Therefore, predictive coding and predictive decoding are performed using a light reflection model. That is, as shown in the top row of the table in Fig. 2, a predicted value of reflected light intensity is derived using the light reflection model during decoding.

[0027] For example, in an information processing method, encoded data of a prediction residual, which is the difference between reflected light intensity, which is attribute data of a point cloud that represents a three-dimensional object as a collection of points, and a predicted value of reflected light intensity generated using a light reflection model on the surface of the object, is decoded to generate the prediction residual, coefficients of the reflection model are derived, a prediction process is performed using the reflection model and coefficients to derive a predicted value, and the prediction residual obtained by decoding the encoded data is added to the derived predicted value to generate the reflected light intensity.

[0028] For example, an information processing device may include a decoding unit that decodes encoded data of a prediction residual, which is the difference between reflected light intensity, which is attribute data of a point cloud that represents a three-dimensional object as a collection of points, and a predicted value of reflected light intensity generated using a light reflection model on the surface of the object, to generate the prediction residual; a coefficient derivation unit that derives coefficients of the reflection model; a prediction unit that performs a prediction process using the reflection model and coefficients to derive a predicted value; and a generation unit that generates the reflected light intensity by adding the prediction residual obtained by the decoding unit and the predicted value derived by the prediction unit.

[0029] Furthermore, as shown in the top row of the table in FIG. 4, a light reflection model is used to derive a predicted value of the reflected light intensity during encoding.

[0030] For example, in an information processing method, coefficients of a light reflection model on the surface of a three-dimensional object are derived, and a prediction process is performed using the reflection model and coefficients to derive a predicted value of reflected light intensity, which is attribute data of a point cloud that represents the object as a collection of points.A prediction residual, which is the difference between the reflected light intensity and the derived predicted value, is generated, and the generated prediction residual is encoded.

[0031] For example, an information processing device may include a coefficient derivation unit that derives coefficients of a light reflection model on the surface of a three-dimensional object, a prediction unit that performs prediction processing using the reflection model and coefficients to derive a predicted value of reflected light intensity, which is attribute data of a point cloud that represents an object as a collection of points, a generation unit that generates a prediction residual, which is the difference between the reflected light intensity and the predicted value derived by the prediction unit, and an encoding unit that encodes the prediction residual generated by the generation unit.

[0032] By doing so, it is possible to encode and decode the difference (prediction residual) between the reflected light intensity of the target point and the reflected light intensity derived using a reflection model at that target point, rather than the difference (prediction residual) between the reflected light intensity of the target point and a neighboring point. This makes it possible to suppress the effect of the object surface orientation on the prediction residual. Therefore, it is possible to suppress a decrease in prediction accuracy, i.e., a decrease in encoding efficiency, caused by changes in the object surface orientation.

[0033] <Diffuse reflection model> Next, reflection models will be explained. First, a general reflection model will be explained. For example, as shown in A of FIG. 6, when light is incident on a specific surface at an angle α (vector L) with respect to the normal vector N, the light is reflected in various directions. This is called diffuse reflection. The Lambert reflection model, which is one of the light reflection models, is a diffuse reflection model that models this type of diffuse reflection. In the Lambert reflection model, the reflected light intensity IR of diffuse reflection such as that shown in A of FIG. 6 can be expressed as in the following equations (1) and (2).

[0034] TIFF0007782563000001.tif757...(1) TIFF0007782563000002.tif1937...(2) where IR is the reflected light intensity, Ia is the ambient light intensity, Iin is the incident light intensity, kd is the diffuse reflection coefficient, N is the normal to the surface (normal vector), and L is the incident direction of light (incident vector).

[0035] In the case of LiDAR data, the incident light is laser light, which ideally can be kept constant (Iin = 1). It is also less affected by ambient light components, so ideally it can be considered 0 (Ia = 0). Laser light also attenuates over distance. In other words, the reflected light intensity depends on the shape, material, and distance of the object surface from which the laser light is reflected.

[0036] As shown in FIG. 7, the material of the object surface can be expressed by the diffuse reflection coefficient kd. The distance to the object surface can be expressed by the distance attenuation Zatt of the laser light, and the shape of the object surface can be expressed by the incident angle θ of the laser light with respect to (the normal to) the object surface.

[0037] That is, in the case of LiDAR data, the reflected light intensity R of the diffuse reflection can be expressed as in the following equations (3) and (4).

[0038] TIFF0007782563000003.tif962...(3) TIFF0007782563000004.tif1939...(4)

[0039] As shown in the third row from the top of the table in Fig. 2, when deriving a predicted value in predictive decoding of the reflected light intensity of diffuse reflection, a diffuse reflection model such as that shown in Equation (3) and Equation (4) may be applied as a reflection model. To derive the predicted value, the following coefficients of the diffuse reflection model may be derived: a diffuse reflection coefficient representing the material of the surface of the object, a distance attenuation representing the attenuation of the laser light irradiated on the object over the distance, and an incident angle of the laser light with respect to the surface of the object.

[0040] 4, when deriving a predicted value in predictive coding of the reflected light intensity of diffuse reflection, a diffuse reflection model such as that shown in Equation (3) and Equation (4) may be applied as a reflection model. To derive the predicted value, the following coefficients of the diffuse reflection model may be derived: a diffuse reflection coefficient representing the material of the surface of the object, a distance attenuation representing the attenuation of the laser light irradiated on the object over the distance, and an incident angle of the laser light with respect to the surface of the object.

[0041] <Predictive decoding> In this case, in predictive decoding of the reflected light intensity of diffuse reflection, the decoder restores the reflected light intensity R, for example, as shown in the following equation (5).

[0042] TIFF0007782563000005.tif1047...(5) where Rres is the prediction residual, and R' is the predicted value of the reflected light intensity R.

[0043] That is, the decoder obtains the prediction residual Rres from the encoder, derives a predicted value R', and adds them together as shown in equation (5) to obtain the reflected light intensity R. Here, the decoder derives the predicted value R' using a diffuse reflection model, i.e., the above equations (3) and (4). That is, the decoder derives the predicted value R' by calculating the coefficients of this diffuse reflection model: the diffuse reflection coefficient kd, the distance attenuation Zatt, and the incident angle θ.

[0044] <Predictive coding> In predictive coding of the reflected light intensity of diffuse reflection, the reflected light intensity R is known, and the prediction residual Rres is derived from the reflected light intensity R. For this reason, the encoder derives the predicted value R' in the same way as the decoder.

[0045] <Estimation of the diffuse reflectance coefficient kd> Next, we will explain how to derive each coefficient. First, we will explain the diffuse reflection coefficient kd. The diffuse reflection coefficient kd is a parameter that indicates the properties related to the diffusion of light from an object. In other words, it corresponds to the material of the object's surface. Details will be explained later, but the distance attenuation Zatt and the incident angle θ can be derived from the geometry, the LiDAR position (the position where the laser light is irradiated), etc. On the other hand, it is difficult to derive the diffuse reflection coefficient kd from this information, but it can be derived using equation (3). In other words, if the reflected light intensity R is known, the diffuse reflection coefficient kd can be calculated. However, the reflected light intensity R is unknown to the decoder.

[0046] Therefore, as shown in the sixth row from the top of the table in Fig. 2, the decoder may estimate the diffuse reflection coefficient kd of the target point based on the diffuse reflection coefficient kd of a neighboring point located near the target point. That is, as shown in Fig. 8, the decoder may calculate the diffuse reflection coefficient kd of the neighboring point using the reflected light intensity of the decoded neighboring point and equation (3), estimate the diffuse reflection coefficient kd of the target point based on the diffuse reflection coefficient kd of the neighboring point, calculate the distance attenuation Zatt and the incident angle θ, derive a predicted value R' using these coefficients and equation (3), and add the prediction residual Rres transmitted from the encoder to this predicted value R' to derive the reflected light intensity R of the target point.

[0047] In this case, any method can be used to estimate the diffuse reflection coefficient kd of the target point based on the diffuse reflection coefficient kd of the neighboring point. For example, the decoder may make the diffuse reflection coefficient kd of the target point the same as the diffuse reflection coefficient kd of the neighboring point (i.e., it may duplicate the diffuse reflection coefficient kd of the neighboring point). If the target point and the neighboring point are close to each other, it is highly likely that the materials of the objects at both points are the same, so it is also possible to consider the diffuse reflection coefficients kd of the neighboring point and the target point to be the same. Alternatively, the decoder may perform a predetermined calculation on the diffuse reflection coefficient kd of the neighboring point and use the result of the calculation as the diffuse reflection coefficient kd of the target point.

[0048] To enable the decoder to restore the reflected light intensity R in this manner, the encoder derives a predicted value R' using a similar technique and derives a prediction residual R' using the predicted value R'. That is, the encoder may estimate the diffuse reflection coefficient kd of the target point based on the diffuse reflection coefficients kd of neighboring points located near the target point, as shown in the sixth row from the top of the table in Figure 4. That is, as in the case of Figure 8, the encoder may calculate the diffuse reflection coefficient kd of the neighboring point processed before the target point using the reflected light intensity of the neighboring point and Equation (3), estimate the diffuse reflection coefficient kd of the target point based on the diffuse reflection coefficient kd of the neighboring point, calculate the distance attenuation Zatt and the incident angle θ, derive a predicted value R' using these coefficients and Equation (3), and then add the prediction residual R' transmitted from the encoder to the predicted value R' to derive the reflected light intensity R of the target point.

[0049] In this case, the method of estimating the diffuse reflection coefficient kd of the target point based on the diffuse reflection coefficient kd of the neighboring points is arbitrary, as in the case of the decoder. For example, the encoder may replicate the diffuse reflection coefficient kd of the neighboring points and use it as the diffuse reflection coefficient kd of the target point, or may perform a predetermined calculation on the diffuse reflection coefficient kd of the neighboring points and use the calculation result as the diffuse reflection coefficient kd of the target point. However, the same method as the decoder must be applied.

[0050] <nearby points> In this way, when the diffuse reflection coefficient kd of the target point is estimated based on the diffuse reflection coefficient kd of the neighboring points, the definition of the neighboring points is arbitrary.

[0051] For example, the decoder may determine the nearest point, which is the point closest to the target point, as the neighboring point, as shown in the seventh row from the top of the table in Figure 2. In other words, the decoder may estimate the diffuse reflection coefficient kd of the target point based on the diffuse reflection coefficient kd of the nearest point (the point closest to the target point among the decoded points). In this case, the estimation method is arbitrary, as described above.

[0052] To enable the decoder to restore the reflected light intensity R in this way, the encoder also estimates the diffuse reflection coefficient kd of the target point using a similar method. That is, the encoder may set the nearest point, which is the point closest to the target point, as the neighboring point, as shown in the seventh row from the top of the table in FIG. 4. That is, the encoder may estimate the diffuse reflection coefficient kd of the target point based on the diffuse reflection coefficient kd of the nearest point (the point closest to the target point among the points processed before the target point). In this case, the estimation method may be any method similar to that used by the decoder, as described above.

[0053] Furthermore, for example, the decoder may determine, as neighboring points, decoded points located within a radius r of the target point as shown in the eighth row from the top of the table in Fig. 2. In other words, the decoder may estimate the diffuse reflection coefficient kd of the target point based on the diffuse reflection coefficients kd of neighboring points located within a range of a predetermined distance from the target point.

[0054] For example, if a black circle is the target point in Figure 9, the decoder considers the gray points located within a radius r from the black circle as neighboring points. In this case, multiple points can be considered neighboring points, as shown in Figure 9.

[0055] In this case, any estimation method may be used, as described above. Furthermore, if there are multiple neighboring points, the decoder may estimate the diffuse reflection coefficient kd of the target point by duplicating the diffuse reflection coefficient kd of any of the neighboring points. Alternatively, the decoder may perform a predetermined calculation on the diffuse reflection coefficient kd of any of the multiple neighboring points and use the calculation result as the diffuse reflection coefficient kd of the target point. Furthermore, the decoder may use the calculation result using the diffuse reflection coefficients kd of multiple neighboring points (for example, a (weighted) average of the diffuse reflection coefficients kd of multiple neighboring points) as the diffuse reflection coefficient kd of the target point.

[0056] In this case, the encoder also estimates the diffuse reflection coefficient kd of the target point using a similar method. That is, the encoder may determine, as neighboring points, points that are located within a radius r from the target point as their center and that are processed before the target point, as shown in the eighth row from the top of the table in Fig. 4. That is, the encoder may estimate the diffuse reflection coefficient kd of the target point based on the diffuse reflection coefficients kd of neighboring points that are located within a range of a predetermined distance from the target point.

[0057] In this case, the estimation method may be any method similar to that used in the decoder, as described above. Furthermore, when there are multiple neighboring points, the encoder may copy the diffuse reflection coefficient kd of one of the neighboring points to estimate the diffuse reflection coefficient kd of the target point. Alternatively, the encoder may perform a predetermined calculation on the diffuse reflection coefficient kd of one of the multiple neighboring points and use the calculation result as the diffuse reflection coefficient kd of the target point. Furthermore, the encoder may use the calculation result using the diffuse reflection coefficients kd of multiple neighboring points (for example, a (weighted) average of the diffuse reflection coefficients kd of multiple neighboring points) as the diffuse reflection coefficient kd of the target point.

[0058] Furthermore, for example, the decoder may determine that points close to the target point in a PredTree relationship are neighboring points, as shown in the ninth row from the top of the table in FIG. 2. A PredTree is a tree structure that indicates the reference relationship of attributes when encoding attributes of a point cloud in G-PCC or the like. Points close to the target point in such a reference relationship may be determined as neighboring points. In other words, the decoder may estimate the diffuse reflection coefficient kd of the target point based on the diffuse reflection coefficient kd of a neighboring point in a predetermined reference relationship. The estimation method in this case is arbitrary, as described above.

[0059] In this case, the encoder also estimates the diffuse reflection coefficient kd of the target point using a similar method. That is, the encoder may determine that a point close to the target point in a PredTree relationship is a neighboring point, as shown in the ninth row from the top of the table in FIG. 4. That is, the encoder may estimate the diffuse reflection coefficient kd of the target point based on the diffuse reflection coefficient kd of a neighboring point in a predetermined reference relationship. In this case, the estimation method may be any method similar to that used by the decoder, as described above.

[0060] <Distance attenuation derivation> Next, distance attenuation Zatt will be described. Distance attenuation Zatt is a parameter corresponding to the distance from the LiDAR position (the position where the laser light is irradiated) to the object surface. That is, as shown in Fig. 8, distance attenuation Zatt can be derived using information such as the LiDAR position (the position where the laser light is irradiated) and the geometry of the object (target point).

[0061] For example, the decoder may derive the distance attenuation Zatt corresponding to the distance from the LiDAR position to the object surface according to a model of light attenuation over distance, as shown in the 18th row from the top of the table in Figure 2. The laser light of the LiDAR is attenuated by the beam expansion and the medium in the air. Such an attenuation model may be applied.

[0062] Any light attenuation model may be used. For example, as shown in FIG. 10, a model in which light attenuates in proportion to the inverse square of the distance may be applied. Also, a model in which distance attenuation Zatt is calculated using the light source area, diffusion angle, etc. in addition to the distance may be applied. Furthermore, a model that takes into account attenuation by the medium in the air may be applied.

[0063] The encoder also estimates the distance attenuation Zatt of the target point using a similar method. That is, the encoder may derive the distance attenuation Zatt corresponding to the distance from the LiDAR position to the object surface according to a light attenuation model according to the distance, as shown in the 18th row from the top of the table in Figure 4. The attenuation model in this case can be any model as long as it is the same as the one applied by the decoder.

[0064] Also, for example, the decoder may derive the distance attenuation Zatt corresponding to the distance from the position of the LiDAR to the object surface based on predetermined table information prepared in advance, as shown in the bottom row of the table in Fig. 2. In other words, the table information may be applied instead of the attenuation model.

[0065] This table information may be of any type. For example, it may be table information indicating the relationship between distance and attribute (distance attenuation Zatt). Alternatively, multiple types of tables may be prepared in the decoder, and the encoder may transmit a table ID or the like to the decoder, thereby allowing the decoder to specify the table to apply. For example, table information corresponding to different external environments such as weather may be prepared, and a table corresponding to the external environment may be selected.

[0066] The encoder also estimates the distance attenuation Zatt of the target point using a similar method. That is, the encoder may derive the distance attenuation Zatt corresponding to the distance from the LiDAR position to the object surface based on predetermined table information prepared in advance, for example, as shown in the bottom row of the table in Figure 4. In this case, the table information can be any information as long as it is the same as that applied by the decoder.

[0067] <Derivation of incident angle> Next, the incident angle θ will be explained. The incident angle θ is a parameter corresponding to the shape of the object (angle of the object surface). In other words, as shown in Figure 8, the distance attenuation Zatt can be derived using information such as the position of the LiDAR (the position where the laser light is irradiated) and the geometry of the object (target point).

[0068] As shown in A of Fig. 11, the incident angle θ indicates the incident angle of the laser light with respect to the normal to the object surface. That is, for example, as shown in the second row from the top of the table in Fig. 3, the decoder may derive the incident angle θ using the normal vector N of the object surface and the incident vector L indicating the incident direction of the light.

[0069] The encoder also estimates the incident angle θ of the target point using a similar method. That is, the encoder may derive the incident angle θ using the normal vector N of the object surface and the incident vector L indicating the incident direction of light, as shown in the second row from the top of the table in Fig. 5, for example.

[0070] <incident vector> The position of the laser beam (the position of the LiDAR) is known to the encoder. However, the decoder does not know the position of the laser beam (the position of the LiDAR). Therefore, the position of the laser beam (the position of the LiDAR) may be transmitted from the encoder to the decoder. For example, information indicating the position of the laser beam (the position of the LiDAR) may be included in an attribute. It may also be transmitted separately as metadata, etc.

[0071] The decoder may then derive the incident vector L based on the information indicating the irradiation position of the laser light (the position of the LiDAR) transmitted from the encoder and the geometry of the target point, as shown in the fourth row from the top of the table in Fig. 3. Note that the incident vector L may be transmitted from the encoder to the decoder instead of the irradiation position of the laser light (the position of the LiDAR). The incident angle may then be derived using the incident vector L.

[0072] The encoder may derive the incident vector L using known information (the irradiation position of the laser light), for example, as shown in the fourth row from the top of the table in FIG. 5. The encoder may encode the irradiation position of the laser light (the position of the LiDAR) and transmit it to the decoder. The encoder may also encode the incident vector L instead of the irradiation position of the laser light (the position of the LiDAR) and transmit it to the decoder.

[0073] <Normal vector> Next, the normal vector N will be described. The decoder may estimate the normal vector N from the geometry, for example, as shown in the sixth row from the top of the table in Fig. 3. For example, the decoder may estimate the normal vector of the target point based on the geometry of the target point, and derive the incident angle θ using the estimated normal vector.

[0074] The encoder also estimates the incident angle θ of the target point using a similar method. That is, the encoder may estimate the normal vector N from the geometry, for example, as shown in the sixth row from the top of the table in Fig. 5. For example, the encoder may estimate the normal vector of the target point based on the geometry of the target point, and derive the incident angle θ using the estimated normal vector.

[0075] The decoder may also estimate the normal vector N from the geometry of the decoded point, as shown in the seventh row from the top of the table in Figure 3. For example, the decoder may estimate the normal vector of the target point based on the geometry of neighboring points located near the target point, and derive the incident angle using the estimated normal vector.

[0076] The encoder also estimates the incident angle θ of the target point using a similar method. That is, the encoder may estimate the normal vector N from the geometry of a point processed before the target point, as shown in the seventh row from the top of the table in Fig. 5. For example, the encoder may estimate the normal vector of the target point based on the geometry of neighboring points located near the target point, and derive the incident angle using the estimated normal vector.

[0077] For example, the decoder may estimate the normal vector N from the geometries of k decoded neighboring points, as shown in the eighth row from the top of the table in Figure 3. For example, when k = 5, the normal vector N is estimated as shown in B of Figure 11. That is, the decoder may estimate the normal vector of the target point based on the geometries of a predetermined number of neighboring points, and derive the incident angle θ using the estimated normal vector.

[0078] The encoder also estimates the incident angle θ of the target point using a similar method. That is, the encoder may estimate the normal vector N from the geometries of k points processed before the target point, as shown in the eighth row from the top of the table in Fig. 5. That is, the encoder may estimate the normal vector of the target point based on the geometries of a predetermined number of neighboring points, and derive the incident angle θ using the estimated normal vector.

[0079] For example, the decoder may estimate the normal vector N from the geometry of neighboring points according to a PredTree relationship, as shown in the ninth row from the top of the table in Fig. 3. That is, the decoder may estimate the normal vector of the target point based on the geometry of neighboring points in a predetermined reference relationship, and derive the incident angle using the estimated normal vector.

[0080] The encoder also estimates the incident angle θ of the target point using a similar method. That is, the encoder may estimate the normal vector N from the geometry of neighboring points according to a PredTree relationship, as shown in the ninth row from the top of the table in Fig. 5. That is, the encoder may estimate the normal vector of the target point based on the geometry of neighboring points in a predetermined reference relationship, and derive the incident angle using the estimated normal vector.

[0081] As shown in A of FIG. 12, by deriving a predicted value R' of reflected light intensity using a diffuse reflection model, it is possible to obtain a prediction residual such as that shown in B of FIG. 12. That is, in this case, the prediction residual having a value such as that shown in the rightmost column of the table in C of FIG. 12 is encoded. Therefore, compared to the example of FIG. 1, the absolute value of the prediction residual to be encoded can be reduced. Furthermore, even if the angle of the object surface changes significantly, the reduction in prediction accuracy can be suppressed. Therefore, it is possible to suppress a reduction in prediction accuracy due to a change in the orientation of the object surface, i.e., a reduction in encoding efficiency.

[0082] <Application of the transmitted diffuse reflectance coefficient kd> In <Estimation of Diffuse Reflection Coefficient kd>, the diffuse reflection coefficient kd is estimated from information on neighboring points, but this is not limiting, and for example, the diffuse reflection coefficient kd may be transmitted from the encoder to the decoder.

[0083] That is, as shown in FIG. 13, the decoder may apply the diffuse reflection coefficient kd transmitted from the encoder as the diffuse reflection coefficient kd of the target point, calculate the distance attenuation Zatt and the incident angle θ, derive the predicted value R′ using these coefficients and equation (3), and add the prediction residual Rres transmitted from the encoder to the predicted value R′ to derive the reflected light intensity R of the target point.

[0084] For example, the decoder may apply the diffuse reflection coefficient kd of the transmitted target point as shown in the tenth row from the top of the table in Figure 2. In other words, the decoder may apply the diffuse reflection coefficient kd used in generating the prediction residual of the target point in the encoder.

[0085] The encoder may also derive the diffuse reflection coefficient kd using a similar method. That is, the encoder may transmit the diffuse reflection coefficient kd of the target point, for example, as shown in the tenth row from the top of the table in Fig. 4. That is, the encoder may derive the diffuse reflection coefficient kd of the target point based on the reflected light intensity R of the target point, derive a predicted value R' using the derived diffuse reflection coefficient, and encode the derived diffuse reflection coefficient kd.

[0086] By deriving the predicted value R' by applying the diffuse reflection coefficient kd thus transmitted, the decoder can suppress a decrease in prediction accuracy.

[0087] In this case, the diffuse reflection coefficient kd may be constantly transmitted (for each point) as shown in the eleventh row from the top of the table in Fig. 2 or the eleventh row from the top of the table in Fig. 4. In other words, the decoder may update the diffuse reflection coefficient for each point. The encoder may then derive the diffuse reflection coefficient kd for each point, derive a predicted value for the target point using the diffuse reflection coefficient of the target point, and encode the diffuse reflection coefficient for each point.

[0088] Furthermore, as shown in the twelfth row from the top of the table in Fig. 2 and the twelfth row from the top of the table in Fig. 4, the diffuse reflection coefficient kd may be transmitted at regular intervals (every multiple points). In other words, the decoder may update the diffuse reflection coefficient for each multiple points. The encoder may then derive a diffuse reflection coefficient for each multiple points, derive a predicted value for the target point using the latest diffuse reflection coefficient at that time, and encode the derived diffuse reflection coefficient.

[0089] 2 and 4, the diffuse reflection coefficient kd may be transmitted when a predetermined condition is met. That is, the decoder may update the diffuse reflection coefficient kd when the predetermined condition is met. The encoder may then derive the diffuse reflection coefficient kd when the predetermined condition is met, derive the predicted value for the target point using the latest diffuse reflection coefficient at that time, and encode the derived diffuse reflection coefficient.

[0090] For example, the decoder may update the diffuse reflection coefficient kd when the change in the geometry of the target point is equal to or greater than a predetermined threshold, and the encoder may derive the diffuse reflection coefficient when the change in the geometry of the target point is equal to or greater than a predetermined threshold.

[0091] Alternatively, as shown in the 14th row from the top of the table in FIG. 2 and the 14th row from the top of the table in FIG. 4, the diffuse reflection coefficient kd may be transmitted only for the first target point. That is, only the initial value may be transmitted. That is, the decoder may apply the same diffuse reflection coefficient kd to all points. Then, the encoder may derive the diffuse reflection coefficient kd for the first target point, derive predicted values ​​using the same diffuse reflection coefficient kd for all points, and encode the derived diffuse reflection coefficient kd.

[0092] <Switching the method for deriving the diffuse reflectance coefficient kd> As shown in the 15th row from the top of the table in Figure 2, the decoder may switch between estimating the diffuse reflection coefficient kd and applying the transmitted diffuse reflection coefficient kd (transmitting the applied diffuse reflection coefficient kd) as described above depending on the conditions.

[0093] That is, as shown in FIG. 14, the decoder may estimate the diffuse reflection coefficient or apply the diffuse reflection coefficient kd transmitted from the encoder as the diffuse reflection coefficient kd of the target point to determine the distance attenuation Zatt and the incident angle θ, derive a predicted value R' using these coefficients and equation (3), and add the prediction residual Rres transmitted from the encoder to the predicted value R' to derive the reflected light intensity R of the target point.

[0094] This makes it possible to transmit the diffuse reflection coefficient kd from the encoder to the decoder when, for example, the distance to the neighboring point is far and there is a high possibility that the object material is different between the neighboring point and the target point, and to estimate the diffuse reflection coefficient kd based on the diffuse reflection coefficient of the neighboring point in other cases. This makes it possible to suppress a decrease in coding efficiency.

[0095] For example, the decoder may estimate the diffuse reflection coefficient kd of a target point based on the diffuse reflection coefficient kd of a neighboring point located near the target point, or may apply the diffuse reflection coefficient kd used to generate the prediction residual of the target point, based on predetermined conditions.

[0096] Furthermore, for example, the decoder may estimate a neighboring point if one exists within the radius r, and may apply the transmitted one if one does not, as shown in the 16th row from the top of the table in Fig. 2. For example, if a neighboring point exists within a range of a predetermined distance from the target point, the decoder may estimate the diffuse reflection coefficient kd of the target point based on the diffuse reflection coefficient kd of the neighboring point, and if no neighboring point exists within that range, it may apply the diffuse reflection coefficient kd used to generate the prediction residual of the target point.

[0097] Furthermore, as shown in the 15th row from the top of the table in FIG. 4, the encoder may switch between estimating the diffuse reflection coefficient kd and transmitting the applied diffuse reflection coefficient kd according to the conditions.

[0098] For example, the encoder may estimate the diffuse reflection coefficient kd of a target point based on the diffuse reflection coefficient kd of a neighboring point located near the target point, or may derive the diffuse reflection coefficient kd of the target point based on the reflected light intensity of the target point, based on predetermined conditions.

[0099] Furthermore, for example, as shown in the 16th row from the top of the table in Figure 4, if a neighboring point exists within a range less than a predetermined distance from the target point, the encoder may estimate the diffuse reflection coefficient kd of the target point based on the diffuse reflection coefficient kd of the neighboring point, and if no neighboring point exists within that range, the encoder may derive the diffuse reflection coefficient kd of the target point based on the reflected light intensity of the target point.

[0100] <Normal vector transmission> Also, the normal vector N may be transmitted from the encoder to the decoder as shown in Fig. 15. That is, the decoder may derive the incident angle θ using the normal vector N transmitted from the encoder.

[0101] For example, the decoder may derive the incident angle θ by applying the normal vector used to generate the prediction residual of the target point in the encoder. Alternatively, the encoder may encode the normal vector N used to derive the incident angle θ. If normal information is signaled, it can be used directly to restore the reflected light intensity, thereby reducing the computational cost of normal estimation.

[0102] <Specular reflection model> For example, as shown in Figure 6B, when light (vector L) is incident at an angle α with respect to the normal vector N of a given surface, the light may be strongly reflected in a certain direction. This is called specular reflection. The Phong reflection model, which is one of the light reflection models, is a specular reflection model that includes this type of specular reflection. In the Phong reflection model, the reflected light intensity IP of specular reflection such as that shown in Figure 6B can be expressed as in the following equations (6) and (7).

[0103] TIFF0007782563000006.tif957...(6) TIFF0007782563000007.tif2037...(7) Here, ks is the specular reflection coefficient, R is the reflection direction (reflection vector), and V is the line of sight direction (line of sight vector).

[0104] In the case of LiDAR data, the incident light is laser light, which ideally can be kept constant (Iin = 1). It is also less affected by ambient light components, so ideally it can be considered 0 (Ia = 0). Laser light also attenuates over distance. In other words, the reflected light intensity depends on the shape, material, and distance of the object surface from which the laser light is reflected.

[0105] That is, in the case of LiDAR data, the reflected light intensity R of specular reflection can be expressed as the following equation (8).

[0106] TIFF0007782563000008.tif946...(8)

[0107] A reflection model that takes such specular reflection into consideration may be applied, as shown in the 11th row from the top of the table in Fig. 3 and the 11th row from the top of the table in Fig. 5. In other words, the reflection model is a specular reflection model, and the decoder or encoder may derive, as coefficients of the specular reflection model, a specular reflection coefficient and a diffuse reflection coefficient that represent the material of the surface of the object, distance attenuation that represents the attenuation of light over distance, and the angle of incidence of light with respect to the surface of the object.

[0108] In this case, in order to derive the predicted value R', it is necessary to derive the diffuse reflection coefficient kd and the specular reflection coefficient ks. That is, as shown in Fig. 16, the decoder may calculate the diffuse reflection coefficient kd and the specular reflection coefficient ks of the neighboring point using the reflected light intensity of the decoded neighboring point and Equation (8), estimate the diffuse reflection coefficient kd and the specular reflection coefficient ks of the target point based on the diffuse reflection coefficient kd and the specular reflection coefficient ks of the neighboring point, calculate the distance attenuation Zatt and the incident angle θ, derive the predicted value R' using these coefficients and Equation (3), and add the prediction residual Rres transmitted from the encoder to the predicted value R' to derive the reflected light intensity R of the target point.

[0109] By applying such a specular reflection model, it is possible to suppress a decrease in coding efficiency even when specular reflection occurs on the surface of an object.

[0110] <Specular reflection coefficient ks> The specular reflection coefficient ks may be a constant, as shown in the 13th row from the top of the table in FIG. 3 or the 13th row from the top of the table in FIG.

[0111] Furthermore, for example, as shown in the 14th row from the top of the table in Fig. 3 and the 14th row from the top of the table in Fig. 5, the specular reflection coefficient ks may be estimated by formulating two simultaneous equations from two neighboring points. For example, a decoder or encoder may estimate the specular reflection coefficient and diffuse reflection coefficient of a neighboring point using the reflected light intensities of multiple neighboring points located near the target point, and then estimate the specular reflection coefficient and diffuse reflection coefficient of the target point using the estimated specular reflection coefficients and diffuse reflection coefficients of the neighboring points.

[0112] Also, for example, as shown in the 15th row from the top of the table in Fig. 3, the decoder may apply the specular reflection coefficient ks transmitted from the encoder as the specular reflection coefficient ks of the target point. In other words, the decoder may apply the specular reflection coefficient and diffuse reflection coefficient used in generating the prediction residual of the target point as the specular reflection coefficient and diffuse reflection coefficient of the target point.

[0113] 5, the encoder may transmit the specular reflection coefficient ks of the target point to the decoder. In other words, the encoder may encode the specular reflection coefficient and the diffuse reflection coefficient of the target point.

[0114] <Model switching> For example, the reflection model to be applied may be switched based on a predetermined condition, as shown in the 17th row from the top of the table in Fig. 3 or the 17th row from the top of the table in Fig. 5. For example, the reflection model to be applied may be switched between the diffuse reflection model and the specular reflection model described above.

[0115] For example, the decoder or encoder may derive coefficients of a specular reflection model if a predetermined condition is met, and derive coefficients of a diffuse reflection model if the condition is not met.

[0116] For example, the reflection model to be applied may be selected based on whether the angle of incidence is close to the angle of the normal vector, as shown in the 18th row from the top of the table in Fig. 3 and the 18th row from the top of the table in Fig. 5. For example, as shown by the thick arrow in Fig. 17, if the incident vector is close to the normal vector, the specular reflection model may be applied, and in other cases, the diffuse reflection model may be applied.

[0117] In other words, if the angular difference between the angle of incidence of light on the surface of the object and the normal vector of the surface of the object is greater than or equal to a predetermined threshold, the coefficients of the diffuse reflection model may be derived, and if the angular difference is less than the threshold, the coefficients of the specular reflection model may be derived.

[0118] Also, as shown in the bottom row of the table in Fig. 3, the decoder may select the reflection model to be applied based on a control flag transmitted from the encoder. Also, as shown in the bottom row of the table in Fig. 5, the encoder may transmit to the decoder a control flag that specifies the reflection model to be applied by the decoder.

[0119] That is, the decoder may derive coefficients of the reflection model, either a diffuse reflection model or a specular reflection model, that is specified by the transmitted control flag, and the encoder may derive coefficients of the reflection model, either a diffuse reflection model or a specular reflection model, that is specified by the control flag, and encode the control flag.

[0120] 3. First Embodiment <Encoding device> Fig. 18 is a block diagram showing an example of the configuration of an encoding device, which is one aspect of an information processing device to which the present technology is applied. The encoding device 300 shown in Fig. 18 is a device that encodes a point cloud (3D data) in which the above-mentioned LiDAR data is used as attribute data. The above-mentioned present technology can be applied to the encoding device 300.

[0121] Note that Fig. 18 shows the main processing units, data flows, etc., and does not necessarily show everything. That is, in encoding device 300, there may be processing units that are not shown as blocks in Fig. 18, and there may be processing and data flows that are not shown as arrows, etc. in Fig. 18.

[0122] As shown in FIG. 18, the encoding device 300 includes a geometry data encoding unit 301 , a geometry data decoding unit 302 , a point cloud generating unit 303 , an attribute data encoding unit 304 , and a bitstream generating unit 305 .

[0123] The geometry data encoding unit 301 encodes position information of the point cloud (3D data) input to the encoding device 300, and generates encoded data of the geometry data. This encoding method is arbitrary. For example, processing such as filtering and quantization for noise suppression (denoising) may be performed. The geometry data encoding unit 301 supplies the generated encoded data to the geometry data decoding unit 302 and the bitstream generation unit 305.

[0124] The geometry data decoding unit 302 acquires the coded data supplied from the geometry data encoding unit 301. The geometry data decoding unit 302 decodes the coded data to generate geometry data. Any decoding method may be used as long as it is compatible with the coding performed by the geometry data encoding unit 301. For example, processing such as filtering or inverse quantization for denoising may be performed. The geometry data decoding unit 302 supplies the generated geometry data (decoded results) to the point cloud generation unit 303.

[0125] The point cloud generation unit 303 acquires attribute data of the point cloud input to the encoding device 300 and geometry data (decoded result) supplied from the geometry data decoding unit 302. The point cloud generation unit 303 performs processing (recolor processing) to match the attribute data with the geometry data (decoded result). The point cloud generation unit 303 supplies the attribute data associated with the geometry data (decoded result) to the attribute data encoding unit 304.

[0126] The attribute data encoding unit 304 acquires the point cloud data (geometry data (decoded result) and attribute data) supplied from the point cloud generation unit 303. The attribute data encoding unit 304 uses the geometry data (decoded result) to encode the attribute data and generate coded data of the attribute data. The attribute data encoding unit 304 supplies the generated coded data to the bit stream generation unit 105.

[0127] The bitstream generation unit 305 acquires coded data of geometry data supplied from the geometry data encoding unit 301. The bitstream generation unit 305 also acquires coded data of attribute data supplied from the attribute data encoding unit 304. The bitstream generation unit 305 multiplexes these coded data and generates a bitstream including these coded data. The bitstream generation unit 305 outputs the generated bitstream to the outside of the encoding device 300. This bitstream is supplied to a decoding side device (for example, a decoding device described later) via, for example, any communication medium or any storage medium.

[0128] In such an encoding device 300, the present technology described in the previous chapter (<2. Predictive encoding and predictive decoding using a reflection model>) can be applied to the attribute data encoding unit 304. That is, in this case, the attribute data encoding unit 204 encodes the attribute data by a method to which the present technology described in the previous chapter (<2. Predictive encoding and predictive decoding using a reflection model>) is applied.

[0129] With this configuration, the encoding device 300 can encode the difference (prediction residual) between the reflected light intensity of the target point and the reflected light intensity derived using a reflection model at that target point, rather than the difference (prediction residual) between the reflected light intensity of the target point and the reflected light intensity of a neighboring point. This allows the encoding device 300 to suppress the effect of the object surface orientation on the prediction residual. Therefore, the encoding device 300 can suppress a decrease in prediction accuracy, i.e., a decrease in encoding efficiency, caused by changes in the object surface orientation.

[0130] These processing units (the geometry data encoding unit 301 to the bitstream generation unit 305) may have any configuration. For example, each processing unit may be configured with a logic circuit that realizes the above-described processing. Furthermore, each processing unit may have, for example, a central processing unit (CPU), read-only memory (ROM), random access memory (RAM), etc., and may execute a program using these to realize the above-described processing. Of course, each processing unit may have both of these configurations, and may implement some of the above-described processing using a logic circuit and other processing by executing a program. The configurations of the processing units may be independent of each other. For example, some processing units may implement some of the above-described processing using a logic circuit, other processing units may implement the above-described processing by executing a program, and still other processing units may implement the above-described processing by both a logic circuit and by executing a program.

[0131] <Attribute data encoding part> Fig. 19 is a block diagram showing an example of the main configuration of the attribute data encoding unit 304 (Fig. 18). Note that Fig. 19 shows the main processing units, data flows, etc., and does not necessarily show everything. In other words, the attribute data encoding unit 304 may have processing units that are not shown as blocks in Fig. 19, or may have processing or data flows that are not shown as arrows, etc. in Fig. 19.

[0132] As shown in FIG. 19, the attribute data encoding unit 304 includes a control unit 321 , a coefficient derivation unit 331 , a prediction unit 332 , a prediction residual generation unit 333 , and an encoding unit 334 .

[0133] The control unit 321 performs processing related to control of encoding of attribute data. For example, the control unit 321 controls the operations of each processing unit, namely, the control unit 321, the coefficient derivation unit 331, the prediction unit 332, the prediction residual generation unit 333, and the encoding unit 334. In this case, the control unit 321 can perform this control by applying the present technology described above in the previous chapter (<2. Predictive encoding and predictive decoding using a reflection model>).

[0134] The coefficient derivation unit 331 is controlled by the control unit 321 and derives coefficients of an applicable reflection model (for example, a diffuse reflection model or a specular reflection model). At this time, the coefficient derivation unit 331 can derive the coefficients by applying the present technology described in the previous chapter (<2. Predictive coding and predictive decoding using a reflection model>). In other words, the coefficient derivation unit 331 derives coefficients of a light reflection model on the surface of a three-dimensional object.

[0135] The prediction unit 332 is controlled by the control unit 321 and derives a predicted value R' of the reflected light intensity R of the target point using the coefficient derived by the coefficient derivation unit 331. In doing so, the prediction unit 332 can derive the predicted value R' by applying the present technology described above in the previous chapter (<2. Predictive encoding and predictive decoding using a reflection model>). In other words, the prediction unit 332 performs prediction processing using a light reflection model on the surface of a three-dimensional object and the coefficient derived by the coefficient derivation unit 331, and derives a predicted value of the reflected light intensity, which is attribute data of a point cloud that represents the object as a collection of points.

[0136] The prediction residual generation unit 333 is controlled by the control unit 321 and derives a prediction residual Rres, which is the difference between the reflected light intensity R of the target point and the predicted value R' derived by the prediction unit 332. In this case, the prediction residual generation unit 333 can derive the prediction residual Rres by applying the present technology described in the previous chapter (<2. Predictive Coding and Predictive Decoding Using a Reflection Model>). In other words, the prediction residual generation unit 333 generates a prediction residual, which is the difference between the reflected light intensity, which is attribute data of a point cloud that represents a three-dimensional object as a collection of points, and the predicted value derived by the prediction unit 332. In other words, the prediction residual generation unit 333 can also be said to be a generation unit that generates the prediction residual.

[0137] The encoding unit 334 is controlled by the control unit 321 and encodes the prediction residual Rres generated by the prediction residual generation unit 333 to generate the encoded data. Any encoding method may be used. In this case, the encoding unit 334 may encode the prediction residual Rres by applying the present technology described in the previous chapter (<2. Predictive encoding and predictive decoding using a reflection model>) to generate the encoded data.

[0138] The encoding unit 334 supplies the generated encoded data of the prediction residual Rres to the bitstream generation unit 305 (FIG. 18). The bitstream generation unit 305 generates a bitstream including the generated encoded data of the prediction residual Rres, and outputs it to the outside of the encoding device 300. This bitstream is transmitted to a decoder via, for example, a storage medium or a transmission medium.

[0139] With this configuration, the attribute data encoding unit 304 can encode the difference (prediction residual) between the reflected light intensity of the target point and the reflected light intensity derived using a reflection model at that target point, rather than the difference (prediction residual) between the reflected light intensity of the target point and the reflected light intensity of a neighboring point. This allows the encoding device 300 to suppress the effect of the object surface orientation on the prediction residual. Therefore, the encoding device 300 can suppress a decrease in prediction accuracy, i.e., a decrease in encoding efficiency, caused by changes in the object surface orientation.

[0140] The reflection model may be a diffuse reflection model. The coefficient derivation unit 331 may be configured to derive, as coefficients, a diffuse reflection coefficient representing the material of the object's surface, a distance attenuation representing the attenuation of light over distance, and an incident angle of light with respect to the object's surface. In this case, the coefficient derivation unit 331 may estimate the diffuse reflection coefficient of the target point based on the diffuse reflection coefficient of a neighboring point located near the target point. For example, the coefficient derivation unit 331 may estimate the diffuse reflection coefficient of the target point based on the diffuse reflection coefficient of a nearest point located nearest to the target point. Alternatively, the coefficient derivation unit 331 may estimate the diffuse reflection coefficient of the target point based on the diffuse reflection coefficient of a neighboring point located within a range of a predetermined distance from the target point. Alternatively, the coefficient derivation unit 331 may estimate the diffuse reflection coefficient of the target point based on the diffuse reflection coefficient of a neighboring point in a predetermined reference relationship.

[0141] Furthermore, the coefficient derivation unit 331 may derive a diffuse reflection coefficient of the target point based on the reflected light intensity of the target point. Then, the prediction unit 332 may derive a predicted value using the diffuse reflection coefficient derived by the coefficient derivation unit 331. Then, the encoding unit 334 may encode the diffuse reflection coefficient derived by the coefficient derivation unit 331. For example, the coefficient derivation unit 331 may derive a diffuse reflection coefficient for each point. In that case, the prediction unit 332 may derive a predicted value for the target point using the diffuse reflection coefficient of the target point. Then, the encoding unit 334 may encode the diffuse reflection coefficient for each point. Furthermore, the coefficient derivation unit 331 may derive a diffuse reflection coefficient for each of multiple points. In that case, the prediction unit 332 may derive a predicted value for the target point using the latest diffuse reflection coefficient. Then, when the coefficient derivation unit 331 derives a diffuse reflection coefficient, the encoding unit 334 may encode the diffuse reflection coefficient.

[0142] Alternatively, the coefficient derivation unit 331 may derive the diffuse reflection coefficient when a predetermined condition is satisfied. In this case, the prediction unit 332 may derive a predicted value of the target point using the latest diffuse reflection coefficient. Then, when the coefficient derivation unit 331 derives the diffuse reflection coefficient, the encoding unit 334 may encode the diffuse reflection coefficient. For example, the coefficient derivation unit 331 may derive the diffuse reflection coefficient when the amount of change in the geometry of the target point is equal to or greater than a predetermined threshold.

[0143] Alternatively, the coefficient derivation unit 331 may derive a diffuse reflection coefficient for the first target point. In this case, the prediction unit 332 may derive predicted values ​​using the same diffuse reflection coefficient for all points. Then, when the coefficient derivation unit 331 derives the diffuse reflection coefficient, the encoding unit 334 may encode the diffuse reflection coefficient.

[0144] Furthermore, the coefficient derivation unit 331 may estimate the diffuse reflection coefficient of the target point based on the diffuse reflection coefficients of neighboring points located near the target point, or may derive the diffuse reflection coefficient of the target point based on the reflected light intensity of the target point, based on predetermined conditions. For example, if a neighboring point exists within a range of a predetermined distance from the target point, the coefficient derivation unit 331 may estimate the diffuse reflection coefficient of the target point based on the diffuse reflection coefficients of the neighboring points. Furthermore, if a neighboring point does not exist within that range, the coefficient derivation unit 331 may derive the diffuse reflection coefficient of the target point based on the reflected light intensity of the target point.

[0145] The coefficient derivation unit 331 may derive the distance attenuation in accordance with a model of attenuation of light over distance, or may derive the distance attenuation based on predetermined table information.

[0146] The coefficient derivation unit 331 may derive the incident angle using a normal vector of the object surface and an incident vector that indicates the incident direction of light. In this case, the encoding unit 334 may encode the incident vector used to derive the incident angle.

[0147] Furthermore, the coefficient derivation unit 331 may estimate a normal vector of the target point based on the geometry of the target point, and derive the angle of incidence using the estimated normal vector.

[0148] Alternatively, the coefficient derivation unit 331 may estimate a normal vector of the target point based on the geometries of neighboring points located near the target point, and derive the angle of incidence using the estimated normal vector. For example, the coefficient derivation unit 331 may estimate a normal vector of the target point based on the geometries of a predetermined number of neighboring points, and derive the angle of incidence using the estimated normal vector. Furthermore, the coefficient derivation unit 331 may estimate a normal vector of the target point based on the geometry of neighboring points in a predetermined reference relationship, and derive the angle of incidence using the estimated normal vector.

[0149] The encoding unit may encode the normal vector used to derive the angle of incidence.

[0150] The reflection model may be a specular reflection model. The coefficient derivation unit 331 may be configured to derive, as coefficients, a specular reflection coefficient and a diffuse reflection coefficient representing the material of the object's surface, a distance attenuation representing the attenuation of light over distance, and an incident angle of light with respect to the object's surface. For example, the coefficient derivation unit 331 may apply a predetermined constant as the specular reflection coefficient. The coefficient derivation unit 331 may also estimate the specular reflection coefficient and the diffuse reflection coefficient of the target point using the reflected light intensities of multiple neighboring points located near the target point, and then estimate the specular reflection coefficient and the diffuse reflection coefficient of the target point using the estimated specular reflection coefficients and diffuse reflection coefficients of the neighboring points. The encoding unit 334 may also encode the specular reflection coefficient and the diffuse reflection coefficient derived by the coefficient derivation unit 331.

[0151] The coefficient derivation unit 331 may derive coefficients of a specular reflection model if a predetermined condition is satisfied, and may derive coefficients of a diffuse reflection model if the condition is not satisfied. For example, the coefficient derivation unit 331 may derive coefficients of a diffuse reflection model if the angular difference between the angle of incidence of light with respect to the surface of an object and the normal vector of the surface of the object is equal to or greater than a predetermined threshold, and may derive coefficients of a specular reflection model if the angular difference is smaller than the threshold.

[0152] The coefficient derivation unit 331 may derive coefficients for the reflection model specified by the control flag, either the diffuse reflection model or the specular reflection model, and the encoding unit 334 may then encode the control flag.

[0153] These processing units (the control unit 321, and the coefficient derivation unit 331 to the encoding unit 334) may have any configuration. For example, each processing unit may be configured with a logic circuit that realizes the above-mentioned processing. Furthermore, each processing unit may have, for example, a CPU, a ROM, a RAM, etc., and may execute a program using these to realize the above-mentioned processing. Of course, each processing unit may have both of these configurations, and may implement some of the above-mentioned processes using logic circuits and others by executing programs. The configurations of the processing units may be independent of each other, and for example, some processing units may implement some of the above-mentioned processes using logic circuits, other processing units may implement the above-mentioned processes by executing programs, and still other processing units may implement the above-mentioned processes by both logic circuits and by executing programs.

[0154] <Encoding process flow> An example of the flow of the encoding process executed by the encoding device 300 will be described with reference to the flowchart of FIG.

[0155] When the encoding process starts, in step S301, the geometry data encoding unit 301 of the encoding device 300 encodes the geometry data of the input point cloud to generate encoded data of the geometry data.

[0156] In step S302, the geometry data decoding unit 302 decodes the coded data generated in step S301 to generate geometry data.

[0157] In step S303, the point cloud generation unit 303 performs recolor processing using the attribute data of the input point cloud and the geometry data (decoded result) generated in step S302, and associates the attribute data with the geometry data.

[0158] In step S304, the attribute data encoding unit 304 performs attribute data encoding processing to encode the attribute data that has been recolored in step S303, and generate encoded data of the attribute data.

[0159] In step S305, the bitstream generation unit 305 generates and outputs a bitstream including the coded data of the geometry data generated in step S301 and the coded data of the attribute data generated in step S304.

[0160] When the process of step S305 is completed, the encoding process ends.

[0161] <Attribute data encoding process flow 1> Next, an example of the flow of the attribute data encoding process executed in step S304 of Fig. 20 will be described with reference to the flowchart of Fig. 21. Note that the flowchart of Fig. 21 corresponds to the example of Fig. 8. That is, an example of the flow of the attribute data encoding process when deriving the diffuse reflection coefficient kd of the target point based on the attribute data of neighboring points will be described.

[0162] When the attribute data encoding process starts, the control unit 321 stores the first node in the stack in step S321.

[0163] In step S322, the control unit 321 extracts the node to be processed from the stack.

[0164] In step S323, the control unit 321 determines whether or not a node exists in the vicinity. If it is determined that a nearby point exists, the process proceeds to step S324.

[0165] In step S324, the coefficient derivation unit 331 estimates a normal vector based on the geometry.

[0166] In step S325, the coefficient derivation unit 331 derives the incident angle θ from the normal vector.

[0167] In step S326, the coefficient derivation unit 331 calculates the distance to the sensor based on the geometry.

[0168] In step S327, the coefficient derivation unit 331 refers to the attribute data of the neighboring points and estimates the diffuse reflection coefficient kd.

[0169] In step S328, the prediction unit 332 derives a predicted value R' based on the derived diffuse reflection coefficient, incident angle, and distance.

[0170] In step S329, the prediction residual generation unit 333 derives the prediction residual Rres. When the process of step S329 ends, the process proceeds to step S331. On the other hand, if it is determined in step S323 that no neighboring point exists, the process proceeds to step S330.

[0171] In step S330, the prediction residual generation unit 333 derives the prediction residual Rres by setting the predicted value R' to 0. When the process of step S330 ends, the process proceeds to step S331.

[0172] In step S331, the encoding unit 334 encodes the prediction residual Rres to generate encoded data.

[0173] In step S332, the control unit 321 stores the next node in the stack.

[0174] In step S333, the control unit 321 determines whether the stack is empty. If it is determined that the stack is not empty, the process returns to step S322, and the subsequent processes are repeated. Also, if it is determined that the stack is empty in step S333, the attribute data encoding process ends, and the process returns to FIG. 20.

[0175] <Attribute data encoding process flow 2> Next, an example of the flow of the attribute data encoding process executed in step S304 of Fig. 20 will be described with reference to the flowchart of Fig. 22. Note that the flowchart of Fig. 22 corresponds to the example of Fig. 13. That is, an example of the flow of the attribute data encoding process will be described when the diffuse reflection coefficient kd of the target point is transmitted from the encoder to the decoder.

[0176] When the attribute data encoding process starts, the control unit 321 stores the first node in the stack in step S351.

[0177] In step S352, the control unit 321 extracts the node to be processed from the stack.

[0178] In step S353, the coefficient derivation unit 331 estimates a normal vector based on the geometry.

[0179] In step S354, the coefficient derivation unit 331 derives the incident angle θ from the normal vector.

[0180] In step S355, the coefficient derivation unit 331 calculates the distance to the sensor based on the geometry.

[0181] In step S356, the coefficient derivation unit 331 estimates the diffuse reflection coefficient kd based on the reflected light intensity R of the target point, the incident angle θ, and the distance.

[0182] In step S357, the encoding unit 334 encodes the diffuse reflection coefficient kd to generate encoded data.

[0183] In step S358, the prediction unit 332 derives a predicted value R' based on the derived diffuse reflection coefficient, incident angle, and distance.

[0184] In step S359, the prediction residual generation unit 333 derives the prediction residual Rres.

[0185] In step 360, the encoding unit 334 encodes the prediction residual Rres to generate encoded data.

[0186] In step S361, the control unit 321 stores the next node in the stack.

[0187] In step S362, the control unit 321 determines whether the stack is empty. If it is determined that the stack is not empty, the process returns to step S352, and the subsequent processes are repeated. Also, if it is determined that the stack is empty in step S333, the attribute data encoding process ends, and the process returns to FIG. 20.

[0188] <Attribute data encoding process flow 3> Next, an example of the flow of the attribute data encoding process executed in step S304 of Fig. 20 will be described with reference to the flowchart of Fig. 23. Note that the flowchart of Fig. 23 corresponds to the example of Fig. 14. That is, an example of the flow of the attribute data encoding process when switching the method of deriving the diffuse reflection coefficient kd will be described.

[0189] When the attribute data encoding process starts, the control unit 321 stores the first node in the stack in step S381.

[0190] In step S382, the control unit 321 extracts the node to be processed from the stack.

[0191] In step S383, the coefficient derivation unit 331 estimates a normal vector based on the geometry.

[0192] In step S384, the coefficient derivation unit 331 derives the incident angle θ from the normal vector.

[0193] In step S385, the coefficient derivation unit 331 calculates the distance to the sensor based on the geometry.

[0194] In step S386, the control unit 321 determines whether or not to transmit the diffuse reflection coefficient kd to the decoder. If it is determined that the diffuse reflection coefficient kd should be transmitted, the process proceeds to step S387.

[0195] In step S387, the coefficient derivation unit 331 estimates the diffuse reflection coefficient kd based on the reflected light intensity R of the target point, the incident angle θ, and the distance.

[0196] In step S388, the encoding unit 334 encodes the diffuse reflection coefficient kd to generate the encoded data. When the process of step S388 ends, the process proceeds to step S390.

[0197] Also, if it is determined in step S386 that the diffuse reflection coefficient kd is not to be transmitted to the decoder, the process proceeds to step S389.

[0198] In step S389, the coefficient derivation unit 331 estimates the diffuse reflection coefficient kd of the target point by referring to the attribute data of the neighboring points. When the process of step S389 ends, the process proceeds to step S390.

[0199] In step S390, the prediction unit 332 derives a predicted value R' based on the derived diffuse reflection coefficient, incident angle, and distance.

[0200] In step S391, the prediction residual generation unit 333 derives the prediction residual Rres.

[0201] In step 392, the encoding unit 334 encodes the prediction residual Rres to generate encoded data.

[0202] In step S393, the control unit 321 stores the next node in the stack.

[0203] In step S394, the control unit 321 determines whether the stack is empty. If it is determined that the stack is not empty, the process returns to step S382, and the subsequent processes are repeated. Also, if it is determined in step S394 that the stack is empty, the attribute data encoding process ends, and the process returns to FIG. 20.

[0204] By performing each process as described above, the encoding device 300 can encode the difference (prediction residual) between the reflected light intensity of the target point and the reflected light intensity derived using a reflection model at that target point, rather than the difference (prediction residual) between the reflected light intensity of the target point and the reflected light intensity of a neighboring point. This allows the encoding device 300 to suppress the effect of the object surface orientation on the prediction residual. Therefore, the encoding device 300 can suppress a decrease in prediction accuracy, i.e., a decrease in encoding efficiency, caused by changes in the object surface orientation.

[0205] 4. Second Embodiment <Decryption device> Fig. 24 is a block diagram showing an example of the configuration of a decoding device, which is one aspect of an information processing device to which the present technology is applied. The decoding device 500 shown in Fig. 24 is a device that decodes encoded data of a point cloud (3D data). The present technology described above can be applied to the decoding device 500.

[0206] Note that Fig. 24 shows the main processing units, data flows, etc., and does not necessarily show everything. That is, in decoding device 500, there may be processing units that are not shown as blocks in Fig. 24, and there may be processing and data flows that are not shown as arrows, etc. in Fig. 24.

[0207] As shown in FIG. 24, the decoding device 500 includes an encoded data extraction unit 501, a geometry data decoding unit 502, an attribute data decoding unit 503, and a point cloud generation unit 504.

[0208] The coded data extraction unit 501 extracts coded data of geometry data and attribute data from the bit stream input to the decoding device 500. The coded data extraction unit 501 supplies the coded data of the extracted geometry data to the geometry data decoding unit 502. The coded data extraction unit 501 supplies the coded data of the extracted attribute data to the attribute data decoding unit 503.

[0209] The geometry data decoding unit 502 acquires the coded data of the geometry data supplied from the coded data extraction unit 501. The geometry data decoding unit 502 decodes the coded data to generate geometry data (decoded result). This decoding method may be any method similar to that used by the geometry data decoding unit 302 of the encoding device 300. The geometry data decoding unit 502 supplies the generated geometry data (decoded result) to the attribute data decoding unit 503 and the point cloud generation unit 504.

[0210] The attribute data decoding unit 503 acquires the coded data of the attribute data supplied from the coded data extraction unit 501. The attribute data decoding unit 503 acquires the geometry data (decoded result) supplied from the geometry data decoding unit 502. The attribute data decoding unit 503 decodes the encoded data using the geometry data (decoded result) to generate attribute data (decoded result). The attribute data decoding unit 503 supplies the generated attribute data (decoded result) to the point cloud generation unit 504.

[0211] The point cloud generation unit 504 acquires the geometry data (decoded result) supplied from the geometry data decoding unit 502. The point cloud generation unit 504 acquires the attribute data (decoded result) supplied from the attribute data decoding unit 503. The point cloud generation unit 504 associates the geometry data (decoded result) with the attribute data (decoded result) to generate point cloud data (decoded result). The point cloud generation unit 504 outputs the generated point cloud data (decoded result) to the outside of the decoding device 500.

[0212] In such a decoding device 500, the present technology described in the previous chapter (<2. Predictive coding and predictive decoding using a reflection model>) can be applied to the attribute data decoding unit 503. That is, in this case, the attribute data decoding unit 503 decodes the coded data of the attribute data by the method to which the present technology described in the previous chapter (<2. Predictive coding and predictive decoding using a reflection model>) is applied, and generates attribute data (decoded result).

[0213] With this configuration, the decoding device 500 can encode the attribute data so that it can be decoded independently for each slice, thereby more reliably decoding the attribute data.

[0214] These processing units (the encoded data extraction unit 501 to the point cloud generation unit 504) may have any configuration. For example, each processing unit may be configured with a logic circuit that realizes the above-mentioned processing. Furthermore, each processing unit may have, for example, a CPU, a ROM, a RAM, etc., and may execute a program using these to realize the above-mentioned processing. Of course, each processing unit may have both of these configurations, and may implement some of the above-mentioned processes using logic circuits and others by executing programs. The configurations of the processing units may be independent of each other, and for example, some processing units may implement some of the above-mentioned processes using logic circuits, other processing units may implement the above-mentioned processes by executing programs, and still other processing units may implement the above-mentioned processes by both logic circuits and by executing programs.

[0215] <Attribute data decoding part> Fig. 25 is a block diagram showing an example of the main configuration of the attribute data decoding unit 503 (Fig. 24). Note that Fig. 25 shows the main processing units, data flows, etc., and does not necessarily show everything. In other words, the attribute data decoding unit 503 may have processing units that are not shown as blocks in Fig. 25, or may have processing or data flows that are not shown as arrows, etc. in Fig. 25.

[0216] As shown in FIG. 25, the attribute data decoding unit 503 includes a control unit 521, a decoding unit 531, a coefficient derivation unit 532, a prediction unit 533, and a generation unit 534.

[0217] The control unit 521 performs processing related to control of decoding of attribute data. For example, the control unit 521 controls the operations of each processing unit, namely, the decoding unit 531, the coefficient derivation unit 532, the prediction unit 533, and the generation unit 534. In this case, the control unit 521 can perform the control by applying the present technology described in the previous chapter (<2. Predictive coding and predictive decoding using a reflection model>).

[0218] The decoding unit 531 acquires coded data of attribute data supplied from the coded data extraction unit 501 (FIG. 24). The decoding unit 531 decodes the coded data. This decoding results in a prediction residual Rres of the attribute data (reflected light intensity). At this time, the decoding unit 531 may decode the coded data by applying the present technology described in the previous chapter (<2. Predictive coding and predictive decoding using a reflection model>). That is, the decoding unit 531 generates a prediction residual by decoding coded data of the prediction residual, which is the difference between the reflected light intensity, which is attribute data of a point cloud representing a three-dimensional object as a set of points, and a predicted value of the reflected light intensity generated using a light reflection model on the surface of the object. Note that this decoding method may be any method as long as it is compatible with the coding method used by the coding unit 334 (FIG. 19) of the coding device 300. The decoding unit 531 supplies the generated prediction residual Rres to the coefficient derivation unit 532.

[0219] The coefficient derivation unit 532 is controlled by the control unit 521 and derives coefficients of an applied reflection model (for example, a diffuse reflection model or a specular reflection model). At this time, the coefficient derivation unit 532 can derive the coefficients by applying the present technology described in the chapter before last (<2. Predictive coding and predictive decoding using a reflection model>).

[0220] The prediction unit 533 is controlled by the control unit 521 and derives a predicted value R' of the reflected light intensity R of the target point using the coefficient derived by the coefficient derivation unit 532. At this time, the prediction unit 533 can derive the predicted value R' by applying the present technology described in the previous chapter (<2. Predictive encoding and predictive decoding using a reflection model>). That is, the prediction unit 533 performs prediction processing using a light reflection model on the surface of a three-dimensional object and the coefficients derived by the coefficient derivation unit 532, and derives a predicted value of the reflected light intensity, which is attribute data of a point cloud that is generated using the light reflection model on the surface of the object and represents the object as a collection of points.

[0221] The generation unit 534 adds the prediction residual Rres obtained by the decoding unit 531 and the predicted value R' obtained by the prediction unit 533 to generate the reflected light intensity R of the target point. In this case, the generation unit 534 may generate the reflected light intensity R of the target point by applying the present technology described in the previous chapter (<2. Predictive coding and predictive decoding using a reflectance model>). In other words, the generation unit 534 adds the prediction residual obtained by the decoding unit 531 and the predicted value derived by the prediction unit 533 to generate the reflected light intensity, which is attribute data of a point cloud that represents a three-dimensional object as a collection of points. The generation unit 534 supplies the generated reflected light intensity R to the point cloud generation unit 504 ( FIG. 24 ) as attribute data.

[0222] With this configuration, the attribute data decoding unit 503 can derive the reflected light intensity from the difference (prediction residual) between the reflected light intensity at the target point and the reflected light intensity derived using a reflection model at that target point, rather than from the difference (prediction residual) between the reflected light intensity at that target point. This allows the decoding device 500 to suppress the effect of the object surface orientation on the prediction residual. Therefore, the decoding device 500 can suppress a decrease in prediction accuracy, i.e., a decrease in encoding efficiency, caused by changes in the object surface orientation.

[0223] The reflection model may be a diffuse reflection model. The coefficient derivation unit 532 may be configured to derive, as coefficients, a diffuse reflection coefficient representing the material of the surface of the object, a distance attenuation representing the attenuation of light over distance, and an incident angle of light with respect to the surface of the object. In this case, the coefficient derivation unit 532 may estimate the diffuse reflection coefficient of the target point based on the diffuse reflection coefficients of neighboring points located near the target point.

[0224] For example, the coefficient derivation unit 532 may estimate the diffuse reflection coefficient of the target point based on the diffuse reflection coefficient of a nearest point located nearest to the target point. Furthermore, for example, the coefficient derivation unit 532 may estimate the diffuse reflection coefficient of the target point based on the diffuse reflection coefficient of a neighboring point located within a range of a predetermined distance from the target point. Furthermore, for example, the coefficient derivation unit 532 may estimate the diffuse reflection coefficient of the target point based on the diffuse reflection coefficient of a neighboring point in a predetermined reference relationship.

[0225] Furthermore, the coefficient derivation unit 532 may apply the diffuse reflection coefficient used to generate the prediction residual of the target point. For example, the coefficient derivation unit 532 may update the diffuse reflection coefficient for each point. For example, the coefficient derivation unit 532 may update the diffuse reflection coefficient when a predetermined condition is satisfied. For example, the coefficient derivation unit 532 may update the diffuse reflection coefficient when the amount of change in the geometry of the target point is equal to or greater than a predetermined threshold. For example, the coefficient derivation unit 532 may apply the same diffuse reflection coefficient to all points.

[0226] The coefficient derivation unit 532 may estimate the diffuse reflection coefficient of the target point based on the diffuse reflection coefficients of neighboring points located near the target point, or may apply the diffuse reflection coefficient used to generate the prediction residual of the target point, based on a predetermined condition. For example, if a neighboring point exists within a range of a predetermined distance from the target point, the coefficient derivation unit 532 may estimate the diffuse reflection coefficient of the target point based on the diffuse reflection coefficients of the neighboring points, and if no neighboring points exist within that range, may apply the diffuse reflection coefficient used to generate the prediction residual of the target point.

[0227] The coefficient derivation unit 532 may derive the distance attenuation according to a model of light attenuation over distance.

[0228] The coefficient derivation unit 532 may derive the distance attenuation based on predetermined table information.

[0229] The coefficient derivation unit 532 may derive the angle of incidence using a normal vector of the object's surface and an incident vector indicating the direction of light incidence. In this case, the coefficient derivation unit 532 may derive the angle of incidence using the incident vector used to generate the prediction residual of the target point. Alternatively, the coefficient derivation unit 532 may estimate the normal vector of the target point based on the geometry of the target point, and derive the angle of incidence using the estimated normal vector.

[0230] Alternatively, the coefficient derivation unit 532 may estimate a normal vector of the target point based on the geometries of neighboring points located near the target point, and derive the angle of incidence using the estimated normal vector. For example, the coefficient derivation unit 532 may estimate a normal vector of the target point based on the geometries of a predetermined number of neighboring points, and derive the angle of incidence using the estimated normal vector. Furthermore, the coefficient derivation unit 532 may estimate the normal vector of the target point based on the geometry of nearby points in a predetermined reference relationship, and derive the angle of incidence using the estimated normal vector.

[0231] The coefficient derivation unit 532 may derive the incident angle by applying the normal vector used to generate the prediction residual of the target point.

[0232] The reflection model may be a specular reflection model. The coefficient derivation unit 532 may be configured to derive, as coefficients, a specular reflection coefficient and a diffuse reflection coefficient that represent the material of the surface of the object, distance attenuation that represents the attenuation of light over distance, and an incident angle of light with respect to the surface of the object.

[0233] In this case, the coefficient derivation unit 532 may apply a predetermined constant as the specular reflection coefficient. Alternatively, the coefficient derivation unit 532 may estimate the specular reflection coefficient and diffuse reflection coefficient of the target point using the reflected light intensities of multiple neighboring points located near the target point, and estimate the specular reflection coefficient and diffuse reflection coefficient of the target point using the estimated specular reflection coefficients and diffuse reflection coefficients of the neighboring points. Alternatively, the coefficient derivation unit 532 may apply the specular reflection coefficient and diffuse reflection coefficient used in generating the prediction residual of the target point.

[0234] The coefficient derivation unit 532 may derive coefficients of a specular reflection model if a predetermined condition is satisfied, and may derive coefficients of a diffuse reflection model if the condition is not satisfied. For example, the coefficient derivation unit 532 may derive coefficients of a diffuse reflection model if the angular difference between the angle of incidence of light with respect to the surface of an object and the normal vector of the surface of the object is equal to or greater than a predetermined threshold, and may derive coefficients of a specular reflection model if the angular difference is smaller than the threshold.

[0235] The coefficient derivation unit 532 may derive the coefficients of the reflection model specified by the transmitted control flag, out of the diffuse reflection model and the specular reflection model.

[0236] These processing units (the control unit 521 and the decoding unit 531 to the generation unit 534) may have any configuration. For example, each processing unit may be configured with a logic circuit that realizes the above-described processing. Furthermore, each processing unit may have, for example, a CPU, a ROM, a RAM, etc., and may realize the above-described processing by executing a program using these. Of course, each processing unit may have both of these configurations, and may realize part of the above-described processing by a logic circuit and other parts by executing a program. The configurations of the processing units may be independent of each other. For example, some processing units may realize part of the above-described processing by a logic circuit, other processing units may realize the above-described processing by executing a program, and still other processing units may realize the above-described processing by both a logic circuit and by executing a program.

[0237] <Decryption process flow> Next, an example of the flow of the decoding process executed by the decoding device 500 will be described with reference to the flowchart of FIG.

[0238] When the decoding process starts, in step S501, the coded data extraction unit 501 of the decoding device 500 acquires and stores the bitstream, and extracts coded data of geometry data and attribute data.

[0239] In step S502, the geometry data decoding unit 502 decodes the coded data extracted in step S501 to generate geometry data (decoded result).

[0240] In step S503, the attribute data decoding unit 503 executes attribute data decoding processing to decode the coded data extracted in step S501 and generate attribute data (reflected light intensity R).

[0241] In step S504, the point cloud generation unit 504 executes a point cloud generation process and generates a point cloud (decoding result) by associating the geometry data generated in step S502 with the attribute data (reflected light intensity R) generated in step S503.

[0242] When the process of step S504 ends, the decoding process ends.

[0243] The present technology described above can be applied to the attribute data decoding process executed in step S504 of such a decoding process. That is, in this case, the attribute data decoding unit 503 executes the attribute data decoding process by a method to which the present technology described above is applied, and decodes the encoded data of the attribute data.

[0244] <Attribute data decoding process flow 1> Next, an example of the flow of the attribute data decoding process executed in step S503 of Fig. 26 will be described with reference to the flowchart of Fig. 27. Note that the flowchart of Fig. 27 corresponds to the example of Fig. 8. That is, an example of the flow of the attribute data decoding process when deriving the diffuse reflection coefficient kd of the target point based on the attribute data of neighboring points will be described.

[0245] When the attribute data decoding process starts, the control unit 521 stores the first node in the stack in step S521.

[0246] In step S522, the control unit 521 extracts the node to be processed from the stack.

[0247] In step S523, the decoding unit 531 decodes the coded data to obtain a prediction residual.

[0248] In step S524, the control unit 521 determines whether or not a node exists in the vicinity. If it is determined that a nearby point exists, the process proceeds to step S525.

[0249] In step S525, the coefficient derivation unit 532 estimates a normal vector based on the geometry.

[0250] In step S526, the coefficient derivation unit 532 derives the incident angle θ from the normal vector.

[0251] In step S527, the coefficient derivation unit 532 calculates the distance to the sensor based on the geometry.

[0252] In step S528, the coefficient derivation unit 532 refers to the attribute data of the neighboring points and estimates the diffuse reflection coefficient kd.

[0253] In step S529, the prediction unit 533 derives a predicted value R' based on the derived diffuse reflection coefficient, incident angle, and distance. When the process of step S529 ends, the process proceeds to step S531. Also, if it is determined in step S524 that no neighboring point exists, the process proceeds to step S530.

[0254] In step S530, the prediction unit 533 sets the predicted value R' to 0. When the process of step S530 ends, the process proceeds to step S531.

[0255] In step S531, the generation unit 534 adds the derived predicted value R' and the prediction residual Rres to generate the reflected light intensity R of the target point.

[0256] In step S532, the control unit 521 stores the next node in the stack.

[0257] In step S533, the control unit 521 determines whether the stack is empty. If it is determined that the stack is not empty, the process returns to step S522, and the subsequent processes are repeated. Also, if it is determined that the stack is empty in step S533, the attribute data encoding process ends, and the process returns to FIG. 27.

[0258] <Attribute data decryption process flow 2> Next, an example of the flow of the attribute data decoding process executed in step S503 of Fig. 26 will be described with reference to the flowchart of Fig. 28. Note that the flowchart of Fig. 28 corresponds to the example of Fig. 13. That is, an example of the flow of the attribute data decoding process when the diffuse reflection coefficient kd of the target point is transmitted from the encoder to the decoder will be described.

[0259] When the attribute data decoding process starts, the control unit 521 stores the first node in the stack in step S551.

[0260] In step S552, the control unit 521 extracts the node to be processed from the stack.

[0261] In step S553, the decoding unit 531 decodes the coded data to obtain the diffuse reflection coefficients transmitted from the encoder.

[0262] In step S554, the decoding unit 531 decodes the coded data to obtain the prediction residual Rres transmitted from the encoder.

[0263] In step S555, the coefficient derivation unit 532 estimates a normal vector based on the geometry.

[0264] In step S556, the coefficient derivation unit 532 derives the incident angle θ from the normal vector.

[0265] In step S557, the coefficient derivation unit 532 calculates the distance to the sensor based on the geometry.

[0266] In step S558, the prediction unit 533 derives a predicted value R' based on the diffuse reflection coefficient, the incident angle, and the distance transmitted from the encoder.

[0267] In step S559, the generation unit 534 adds the derived predicted value R' and the prediction residual Rres to generate the reflected light intensity R of the target point.

[0268] In step S560, the control unit 521 stores the next node in the stack.

[0269] In step S561, the control unit 521 determines whether the stack is empty. If it is determined that the stack is not empty, the process returns to step S552, and the subsequent processes are repeated. Also, if it is determined that the stack is empty in step S561, the attribute data encoding process ends, and the process returns to FIG. 27.

[0270] <Attribute data decryption process flow 3> Next, an example of the flow of the attribute data decoding process executed in step S503 in Fig. 26 will be described with reference to the flowchart in Fig. 29. Note that the flowchart in Fig. 29 corresponds to the example in Fig. 14. That is, an example of the flow of the attribute data decoding process when the method of deriving the diffuse reflection coefficient kd is switched will be described.

[0271] When the attribute data decoding process starts, the control unit 521 stores the first node in the stack in step S581.

[0272] In step S582, the control unit 521 extracts the node to be processed from the stack.

[0273] In step S583, the control unit 521 determines whether or not to decode the diffuse reflection coefficients. If it is determined that the diffuse reflection coefficients are to be decoded, the process proceeds to step S584.

[0274] In step S584, the decoding unit 531 decodes the encoded data to obtain the diffuse reflection coefficients transmitted from the encoder. When the process of step S584 ends, the process proceeds to step S586.

[0275] Furthermore, if it is determined in step S583 that the diffuse reflection coefficients are not to be decoded, the process proceeds to step S585. In step S585, the coefficient derivation unit 532 refers to nearby attribute data and estimates the diffuse reflection coefficients. When the process of step S505 ends, the process proceeds to step S586.

[0276] In step S586, the decoding unit 531 decodes the coded data to obtain the prediction residual Rres transmitted from the encoder.

[0277] In step S587, the coefficient derivation unit 532 estimates a normal vector based on the geometry.

[0278] In step S588, the coefficient derivation unit 532 derives the incident angle θ from the normal vector.

[0279] In step S589, the coefficient derivation unit 532 calculates the distance to the sensor based on the geometry.

[0280] In step S590, the prediction unit 533 derives a predicted value R' based on the diffuse reflection coefficient, the incident angle, and the distance transmitted from the encoder.

[0281] In step S591, the generation unit 534 adds the derived predicted value R' and the prediction residual Rres to generate the reflected light intensity R of the target point.

[0282] In step S592, the control unit 521 stores the next node in the stack.

[0283] In step S593, the control unit 521 determines whether the stack is empty. If it is determined that the stack is not empty, the process returns to step S582, and the subsequent processes are repeated. Also, if it is determined that the stack is empty in step S593, the attribute data encoding process ends, and the process returns to FIG. 27.

[0284] By performing each process as described above, the decoding device 500 can derive the reflected light intensity from the difference (prediction residual) between the reflected light intensity of the target point and the reflected light intensity derived using a reflection model at that target point, rather than from the difference (prediction residual) between the reflected light intensity of the target point and the reflected light intensity of a neighboring point. This allows the decoding device 500 to suppress the effect of the object surface orientation on the prediction residual. Therefore, the decoding device 500 can suppress a decrease in prediction accuracy, i.e., a decrease in encoding efficiency, caused by changes in the object surface orientation.

[0285] <5. Notes> <3D data> This technology can be applied to the encoding and decoding of 3D data of any standard. In other words, as long as it does not conflict with the above-mentioned technology, various processes such as encoding and decoding methods, and specifications of various data such as 3D data and metadata, are arbitrary. Furthermore, as long as it does not conflict with the above-mentioned technology, some of the above-mentioned processes and specifications may be omitted.

[0286] <Computer> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs constituting the software are installed on a computer. Here, the term "computer" includes computers built into dedicated hardware, and general-purpose personal computers, etc., that can execute various functions by installing various programs.

[0287] FIG. 30 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0288] In a computer 900 shown in FIG. 30, a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903 are interconnected via a bus 904.

[0289] An input / output interface 910 is also connected to the bus 904. To the input / output interface 910, an input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915 are connected.

[0290] The input unit 911 includes, for example, a keyboard, a mouse, a microphone, a touch panel, an input terminal, etc. The output unit 912 includes, for example, a display, a speaker, an output terminal, etc. The storage unit 913 includes, for example, a hard disk, a RAM disk, a non-volatile memory, etc. The communication unit 914 includes, for example, a network interface. The drive 915 drives removable media 921 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0291] In a computer configured as above, the CPU 901 performs the above-described series of processes by, for example, loading a program stored in the storage unit 913 into the RAM 903 via the input / output interface 910 and the bus 904 and executing the program. The RAM 903 also stores data necessary for the CPU 901 to execute various processes as appropriate.

[0292] The program executed by the computer can be applied by recording it on removable media 921 such as package media, for example. In this case, the program can be installed in storage unit 913 via input / output interface 910 by inserting removable media 921 into drive 915.

[0293] This program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, digital satellite broadcasting, etc. In this case, the program can be received by the communication unit 914 and installed in the storage unit 913.

[0294] Alternatively, this program can be installed in advance in the ROM 902 or the storage unit 913 .

[0295] <Applicable targets of this technology> The present technology can be applied to any configuration, for example, various electronic devices.

[0296] Furthermore, for example, the present technology can also be implemented as a part of an apparatus, such as a processor (e.g., a video processor) as a system LSI (Large Scale Integration), a module (e.g., a video module) using multiple processors, a unit (e.g., a video unit) using multiple modules, or a set in which other functions are added to a unit (e.g., a video set).

[0297] Furthermore, for example, the present technology can also be applied to a network system configured with multiple devices. For example, the present technology may be implemented as cloud computing in which multiple devices share and collaborate on processing via a network. For example, the present technology may be implemented in a cloud service that provides image (video)-related services to any terminal, such as a computer, AV (Audio Visual) equipment, a portable information processing terminal, or an IoT (Internet of Things) device.

[0298] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0299] <Fields and applications where this technology can be applied> Systems, devices, processing units, etc. to which the present technology is applied can be used in any field, such as transportation, medical care, crime prevention, agriculture, livestock farming, mining, beauty, factories, home appliances, weather, and nature monitoring. In addition, the applications thereof are also arbitrary.

[0300] <Other> In this specification, various types of information (metadata, etc.) related to encoded data (bitstream) may be transmitted or recorded in any form as long as they are associated with the encoded data. Here, the term "associate" means, for example, that one piece of data can be used (linked) when processing the other piece of data. In other words, pieces of associated data may be combined into one piece of data or may be individual pieces of data. For example, information associated with encoded data (image) may be transmitted over a transmission path separate from that of the encoded data (image). Furthermore, for example, information associated with encoded data (image) may be recorded on a recording medium separate from that of the encoded data (image) (or on a different recording area of ​​the same recording medium). Note that this "association" may refer to only a portion of the data, rather than the entire data. For example, an image and information corresponding to that image may be associated with each other in any unit, such as multiple frames, one frame, or a portion of a frame.

[0301] In this specification, terms such as "composite," "multiplex," "add," "integrate," "include," "store," "embed," "insert," and the like refer to combining multiple items into one, such as combining encoded data and metadata into one piece of data, and refer to one method of "associating" as described above.

[0302] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present technology.

[0303] For example, a configuration described as one device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, configurations described above as multiple devices (or processing units) may be combined and configured as one device (or processing unit). Of course, configurations other than those described above may be added to the configuration of each device (or each processing unit). Furthermore, as long as the configuration and operation of the entire system are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0304] Furthermore, for example, the above-described program may be executed in any device, as long as the device has the necessary functions (functional blocks, etc.) and can obtain the necessary information.

[0305] Also, for example, each step of a single flowchart may be executed by one device, or may be shared and executed by multiple devices. Furthermore, when one step includes multiple processes, the multiple processes may be executed by one device, or may be shared and executed by multiple devices. In other words, multiple processes included in one step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as one step.

[0306] For example, the steps of a program executed by a computer may be executed in chronological order in the order described herein, or may be executed in parallel or individually at the required timing, such as when a call is made. In other words, as long as no contradiction occurs, the steps may be executed in an order different from the order described above. Furthermore, the steps of this program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.

[0307] Furthermore, for example, multiple technologies related to the present technology can be implemented independently and independently, as long as no contradiction occurs. Of course, any multiple technologies can also be implemented in combination. For example, part or all of the present technology described in any embodiment can be implemented in combination with part or all of the present technology described in another embodiment. Furthermore, part or all of any of the above-described present technologies can be implemented in combination with other technologies not described above.

[0308] The present technology can also be configured as follows. (1) a decoding unit that decodes encoded data of a prediction residual, which is a difference between a reflected light intensity that is attribute data of a point cloud that represents a three-dimensional object as a set of points and a predicted value of the reflected light intensity generated using a light reflection model on the surface of the object, to generate the prediction residual; and a coefficient derivation unit that derives coefficients of the reflection model; a prediction unit that performs a prediction process using the reflection model and the coefficients to derive the predicted value; a generation unit that generates the reflected light intensity by adding the prediction residual obtained by the decoding unit and the predicted value derived by the prediction unit; An information processing device comprising: (2) the reflection model is a diffuse reflection model; The coefficient derivation unit is configured to derive, as the coefficients, a diffuse reflection coefficient representing the material of the surface of the object, a distance attenuation representing the attenuation of the light over distance, and an incident angle of the light with respect to the surface of the object. The information processing device described in (1). (3) The coefficient derivation unit estimates the diffuse reflection coefficient of the target point based on the diffuse reflection coefficient of a neighboring point located in the vicinity of the target point. (2) An information processing device according to the present invention. (4) The coefficient derivation unit estimates the diffuse reflection coefficient of the target point based on the diffuse reflection coefficient of a nearest neighbor point located nearest to the target point. (3) An information processing device according to the present invention. (5) The coefficient derivation unit estimates the diffuse reflection coefficient of the target point based on the diffuse reflection coefficient of the neighboring point located within a range of a predetermined distance from the target point. An information processing device according to (3) or (4). (6) The coefficient derivation unit estimates the diffuse reflection coefficient of the target point based on the diffuse reflection coefficient of the neighboring point in a predetermined reference relationship. An information processing device according to any one of (3) to (5). (7) The coefficient derivation unit applies the diffuse reflection coefficient used to generate the prediction residual of the target point. An information processing device according to any one of (2) to (6). (8) The coefficient derivation unit updates the diffuse reflection coefficient for each point. (7) An information processing device according to (7). (9) The coefficient derivation unit updates the diffuse reflection coefficient for each of a plurality of points. An information processing device according to (7) or (8). (10) The coefficient derivation unit updates the diffuse reflection coefficient when a predetermined condition is satisfied. An information processing device according to any one of (7) to (9). (11) The coefficient derivation unit updates the diffuse reflection coefficient when a change in geometry of the target point is equal to or greater than a predetermined threshold. (10) An information processing device according to (10). (12) The coefficient derivation unit applies the same diffuse reflection coefficient to all points. (7) An information processing device according to (7). (13) The coefficient derivation unit estimates the diffuse reflection coefficient of the target point based on the diffuse reflection coefficient of a neighboring point located near the target point, or applies the diffuse reflection coefficient used to generate the prediction residual of the target point, based on a predetermined condition. An information processing device according to any one of (2) to (12). (14) When the neighboring point exists within a range of a predetermined distance from the target point, the coefficient derivation unit estimates the diffuse reflection coefficient of the target point based on the diffuse reflection coefficient of the neighboring point, and when the neighboring point does not exist within the range, the coefficient derivation unit applies the diffuse reflection coefficient used to generate the prediction residual of the target point. (13) An information processing device according to (13). (15) The coefficient derivation unit derives the distance attenuation according to a model of attenuation of light over distance. An information processing device according to any one of (2) to (14). (16) The information processing device according to any one of (2) to (15), wherein the coefficient derivation unit derives the distance attenuation based on predetermined table information. (17) The coefficient derivation unit derives the incident angle using a normal vector of the surface of the object and an incident vector indicating an incident direction of the light. An information processing device according to any one of (2) to (16). (18) The coefficient derivation unit derives the incident angle by using the incident vector used to generate the prediction residual of the target point. (17) An information processing device according to (17). (19) The coefficient derivation unit estimates the normal vector of the target point based on a geometry of the target point, and derives the incident angle using the estimated normal vector. The information processing device according to (17) or (18). (20) The coefficient derivation unit estimates the normal vector of the target point based on the geometry of a neighboring point located near the target point, and derives the incident angle using the estimated normal vector. An information processing device according to any one of (17) to (19). (21) The coefficient derivation unit estimates the normal vector of the target point based on geometries of a predetermined number of the neighboring points, and derives the incident angle using the estimated normal vector. (20) An information processing device according to (20). (22) The coefficient derivation unit estimates the normal vector of the target point based on a geometry of the neighboring point in a predetermined reference relationship, and derives the incident angle using the estimated normal vector. The information processing device according to (20) or (21). (twenty three) The coefficient derivation unit derives the incident angle by applying the normal vector used to generate the prediction residual of the target point. An information processing device according to any one of (17) to (22). (24) The reflection model is a specular reflection model, The coefficient derivation unit is configured to derive, as the coefficients, a specular reflection coefficient and a diffuse reflection coefficient that represent the material of the surface of the object, a distance attenuation that represents the attenuation of the light over distance, and an incident angle of the light with respect to the surface of the object. An information processing device according to any one of (1) to (23). (25) The coefficient derivation unit applies a predetermined constant as the specular reflection coefficient. (24) An information processing device according to (24). (26) The coefficient derivation unit estimates the specular reflection coefficient and the diffuse reflection coefficient of the target point using the reflected light intensities of a plurality of neighboring points located near the target point, and estimates the specular reflection coefficient and the diffuse reflection coefficient of the target point using the estimated specular reflection coefficient and the diffuse reflection coefficient of the neighboring points. The information processing device according to (24) or (25). (27) The coefficient derivation unit applies the specular reflection coefficient and the diffuse reflection coefficient used to generate the prediction residual of the target point. An information processing device according to any one of (24) to (26). (28) The coefficient derivation unit derives the coefficients of a specular reflection model when a predetermined condition is satisfied, and derives the coefficients of a diffuse reflection model when the condition is not satisfied. An information processing device according to any one of (1) to (27). (29) The coefficient derivation unit derives the coefficients of the diffuse reflection model when an angular difference between an incident angle of the light with respect to the surface of the object and a normal vector of the surface of the object is equal to or greater than a predetermined threshold, and derives the coefficients of the specular reflection model when the angular difference is smaller than the threshold. (28) An information processing device according to (28). (30) The coefficient derivation unit derives the coefficients of the reflection model designated by the transmitted control flag, out of a diffuse reflection model and a specular reflection model. An information processing device according to any one of (1) to (29). (31) Decoding encoded data of a prediction residual, which is a difference between a reflected light intensity, which is attribute data of a point cloud that represents a three-dimensional object as a set of points, and a predicted value of the reflected light intensity generated using a light reflection model on the surface of the object, to generate the prediction residual; deriving coefficients of the reflection model; performing a prediction process using the reflection model and the coefficients to derive the predicted value; The reflected light intensity is generated by adding the prediction residual obtained by decoding the encoded data and the derived predicted value. Information processing methods.

[0309] (41) A coefficient derivation unit that derives coefficients of a light reflection model on the surface of a three-dimensional object; a prediction unit that performs a prediction process using the reflection model and the coefficients to derive a predicted value of reflected light intensity, which is attribute data of a point cloud that represents the object as a set of points; a generation unit that generates a prediction residual that is a difference between the reflected light intensity and the predicted value derived by the prediction unit; a coding unit that codes the prediction residual generated by the generation unit; An information processing device comprising: (42) The reflection model is a diffuse reflection model, The coefficient derivation unit is configured to derive, as the coefficients, a diffuse reflection coefficient representing the material of the surface of the object, a distance attenuation representing the attenuation of the light over distance, and an incident angle of the light with respect to the surface of the object. (41) An information processing device according to (41). (43) The coefficient derivation unit estimates the diffuse reflection coefficient of the target point based on the diffuse reflection coefficient of a neighboring point located in the vicinity of the target point. (42) An information processing device according to (42). (44) The coefficient derivation unit estimates the diffuse reflection coefficient of the target point based on the diffuse reflection coefficient of a nearest neighbor point located nearest to the target point. (43) An information processing device according to (43). (45) The coefficient derivation unit estimates the diffuse reflection coefficient of the target point based on the diffuse reflection coefficient of the neighboring point located within a range of a predetermined distance from the target point. The information processing device according to (43) or (44). (46) The coefficient derivation unit estimates the diffuse reflection coefficient of the target point based on the diffuse reflection coefficient of the neighboring point in a predetermined reference relationship. An information processing device according to any one of (43) to (45). (47) The coefficient derivation unit derives the diffuse reflection coefficient of the target point based on the reflected light intensity of the target point; the prediction unit derives the predicted value using the diffuse reflection coefficient derived by the coefficient derivation unit; The encoding unit encodes the diffuse reflection coefficients derived by the coefficient derivation unit. An information processing device according to any one of (42) to (46). (48) The coefficient derivation unit derives the diffuse reflection coefficient for each point, the prediction unit derives the predicted value of the target point using the diffuse reflection coefficient of the target point; The encoding unit encodes the diffuse reflection coefficient for each point. (47) An information processing device according to (47). (49) The coefficient derivation unit derives the diffuse reflection coefficient for each of a plurality of points, the prediction unit derives the predicted value of the target point using the latest diffuse reflection coefficient; The encoding unit encodes the diffuse reflection coefficient when the coefficient derivation unit derives the diffuse reflection coefficient. The information processing device according to (47) or (48). (50) The coefficient derivation unit derives the diffuse reflection coefficient when a predetermined condition is satisfied, the prediction unit derives the predicted value of the target point using the latest diffuse reflection coefficient; The encoding unit encodes the diffuse reflection coefficient when the coefficient derivation unit derives the diffuse reflection coefficient. An information processing device according to any one of (47) to (49). (51) The coefficient derivation unit derives the diffuse reflection coefficient when a change in geometry of the target point is equal to or greater than a predetermined threshold. (50) An information processing device according to (50). (52) The coefficient derivation unit derives the diffuse reflection coefficient for a first target point. The prediction unit derives the predicted value using the same diffuse reflection coefficient for all points. The encoding unit encodes the diffuse reflection coefficient when the coefficient derivation unit derives the diffuse reflection coefficient. (47) An information processing device according to (47). (53) The coefficient derivation unit estimates the diffuse reflection coefficient of the target point based on the diffuse reflection coefficient of a neighboring point located near the target point, or derives the diffuse reflection coefficient of the target point based on the reflected light intensity of the target point, based on a predetermined condition. An information processing device according to any one of (42) to (52). (54) When the neighboring point exists within a range of a predetermined distance from the target point, the coefficient derivation unit estimates the diffuse reflection coefficient of the target point based on the diffuse reflection coefficient of the neighboring point, and when the neighboring point does not exist within the range, derives the diffuse reflection coefficient of the target point based on the reflected light intensity of the target point. (53) An information processing device according to (53). (55) The coefficient derivation unit derives the distance attenuation according to a model of attenuation of light over distance. An information processing device according to any one of (42) to (54). (56) The information processing device according to any one of (42) to (55), wherein the coefficient derivation unit derives the distance attenuation based on predetermined table information. (57) The coefficient derivation unit derives the incident angle using a normal vector of the surface of the object and an incident vector indicating an incident direction of the light. An information processing device according to any one of (42) to (56). (58) The encoding unit encodes the incident vector used to derive the incident angle. (57) An information processing device according to (57). (59) The coefficient derivation unit estimates the normal vector of the target point based on a geometry of the target point, and derives the incident angle using the estimated normal vector. The information processing device according to (57) or (58). (60) The coefficient derivation unit estimates the normal vector of the target point based on the geometry of a neighboring point located near the target point, and derives the incident angle using the estimated normal vector. An information processing device according to any one of (57) to (59). (61) The coefficient derivation unit estimates the normal vector of the target point based on geometries of a predetermined number of the neighboring points, and derives the incident angle using the estimated normal vector. (60) An information processing device according to (60). (62) The coefficient derivation unit estimates the normal vector of the target point based on a geometry of the neighboring point in a predetermined reference relationship, and derives the incident angle using the estimated normal vector. The information processing device according to (60) or (61). (63) The encoding unit encodes the normal vector used to derive the incident angle. An information processing device according to any one of (57) to (62). (64) The reflection model is a specular reflection model, The coefficient derivation unit is configured to derive, as the coefficients, a specular reflection coefficient and a diffuse reflection coefficient that represent the material of the surface of the object, a distance attenuation that represents the attenuation of the light over distance, and an incident angle of the light with respect to the surface of the object. An information processing device according to any one of (41) to (63). (65) The coefficient derivation unit applies a predetermined constant as the specular reflection coefficient. (64) An information processing device according to (64). (66) The coefficient derivation unit estimates the specular reflection coefficient and the diffuse reflection coefficient of the target point using the reflected light intensities of a plurality of neighboring points located near the target point, and estimates the specular reflection coefficient and the diffuse reflection coefficient of the target point using the estimated specular reflection coefficient and the diffuse reflection coefficient of the neighboring points. The information processing device according to (64) or (65). (67) The encoding unit encodes the specular reflection coefficient and the diffuse reflection coefficient derived by the coefficient derivation unit. An information processing device according to any one of (64) to (66). (68) The coefficient derivation unit derives the coefficients of a specular reflection model when a predetermined condition is satisfied, and derives the coefficients of a diffuse reflection model when the condition is not satisfied. An information processing device according to any one of (41) to (67). (69) The coefficient derivation unit derives the coefficients of the diffuse reflection model when an angular difference between an incident angle of the light with respect to the surface of the object and a normal vector of the surface of the object is equal to or greater than a predetermined threshold, and derives the coefficients of the specular reflection model when the angular difference is smaller than the threshold. (68) An information processing device according to (68). (70) The coefficient derivation unit derives the coefficients of the reflection model designated by a control flag from among a diffuse reflection model and a specular reflection model; The encoding unit encodes the control flag. An information processing device according to any one of (41) to (69). (71) Deriving coefficients of a light reflection model on the surface of a three-dimensional object, and performing a prediction process using the reflection model and the coefficients to derive a predicted value of reflected light intensity, which is attribute data of a point cloud that represents the object as a set of points; generating a prediction residual that is a difference between the reflected light intensity and the derived predicted value; Encoding the generated prediction residual Information processing methods. [Explanation of symbols]

[0310] 300 encoding device, 301 geometry data encoding unit, 302 geometry data decoding unit, 303 point cloud generation unit, 304 attribute data decoding unit, 305 bitstream generation unit, 321 control unit, 331 coefficient derivation unit, 332 prediction unit, 333 prediction residual generation unit, 334 encoding unit, 500 decoding device, 501 encoded data extraction unit, 502 geometry data decoding unit, 503 attribute data decoding unit, 504 point cloud generation unit, 521 control unit, 531 decoding unit, 532 coefficient derivation unit, 533 prediction unit, 534 generation unit, 900 computer

Claims

1. a decoding unit that decodes encoded data of a prediction residual that is a difference between a reflected light intensity, which is attribute data of a point cloud that represents a three-dimensional object as a set of points, and a predicted value of the reflected light intensity generated using a light reflection model on the surface of the object, to generate the prediction residual; and a coefficient derivation unit that derives coefficients of the reflection model; a prediction unit that performs a prediction process using the reflection model and the coefficients to derive the predicted value; a generation unit that generates the reflected light intensity by adding the prediction residual obtained by the decoding unit and the predicted value derived by the prediction unit; An information processing device comprising:

2. the reflection model is a diffuse reflection model, The coefficient derivation unit is configured to derive, as the coefficients, a diffuse reflection coefficient representing the material of the surface of the object, a distance attenuation representing the attenuation of the light over distance, and an incident angle of the light with respect to the surface of the object. The information processing device according to claim 1 .

3. The coefficient derivation unit estimates the diffuse reflection coefficient of the target point based on the diffuse reflection coefficient of a neighboring point located in the vicinity of the target point. The information processing device according to claim 2 .

4. The coefficient derivation unit estimates the diffuse reflection coefficient of the target point based on the diffuse reflection coefficient of a nearest neighbor point located nearest to the target point. The information processing device according to claim 3 .

5. The coefficient derivation unit estimates the diffuse reflection coefficient of the target point based on the diffuse reflection coefficient of the neighboring point located within a range of a predetermined distance from the target point. The information processing device according to claim 3 .

6. The coefficient derivation unit estimates the diffuse reflection coefficient of the target point based on the diffuse reflection coefficient of the neighboring point in a predetermined reference relationship. The information processing device according to claim 3 .

7. The coefficient derivation unit applies the diffuse reflection coefficient used to generate the prediction residual of the target point. The information processing device according to claim 2 .

8. The coefficient derivation unit estimates the diffuse reflection coefficient of the target point based on the diffuse reflection coefficient of a neighboring point located near the target point, or applies the diffuse reflection coefficient used to generate the prediction residual of the target point, based on a predetermined condition. The information processing device according to claim 2 .

9. The coefficient derivation unit derives the incident angle using a normal vector of the surface of the object and an incident vector indicating the incident direction of the light. The information processing device according to claim 2 .

10. The coefficient derivation unit derives the incident angle by applying the normal vector used to generate the prediction residual of the target point. The information processing device according to claim 9 .

11. the reflection model is a specular reflection model, The coefficient derivation unit is configured to derive, as the coefficients, a specular reflection coefficient and a diffuse reflection coefficient that represent the material of the surface of the object, a distance attenuation that represents the attenuation of the light over distance, and an incident angle of the light with respect to the surface of the object. The information processing device according to claim 1 .

12. The coefficient derivation unit estimates the specular reflection coefficient and the diffuse reflection coefficient of the target point using the reflected light intensities of a plurality of neighboring points located near the target point, and estimates the specular reflection coefficient and the diffuse reflection coefficient of the target point using the estimated specular reflection coefficient and the diffuse reflection coefficient of the neighboring points. The information processing device according to claim 11.

13. The coefficient derivation unit applies the specular reflection coefficient and the diffuse reflection coefficient used to generate the prediction residual of the target point. The information processing device according to claim 11.

14. The coefficient derivation unit derives the coefficients of a specular reflection model when a predetermined condition is satisfied, and derives the coefficients of a diffuse reflection model when the condition is not satisfied. The information processing device according to claim 1 .

15. generating a prediction residual by decoding encoded data of a prediction residual that is a difference between a reflected light intensity, which is attribute data of a point cloud that represents a three-dimensional object as a set of points, and a predicted value of the reflected light intensity generated using a light reflection model on the surface of the object; deriving coefficients of the reflection model; performing a prediction process using the reflection model and the coefficients to derive the predicted value; The reflected light intensity is generated by adding the prediction residual obtained by decoding the encoded data and the derived predicted value. Information processing methods.

16. a coefficient derivation unit that derives coefficients of a light reflection model on the surface of a three-dimensional object; a prediction unit that performs a prediction process using the reflection model and the coefficients to derive a predicted value of reflected light intensity, which is attribute data of a point cloud that represents the object as a set of points; a generation unit that generates a prediction residual that is a difference between the reflected light intensity and the predicted value derived by the prediction unit; a coding unit that codes the prediction residual generated by the generation unit; An information processing device comprising:

17. the reflection model is a diffuse reflection model, The coefficient derivation unit is configured to derive, as the coefficients, a diffuse reflection coefficient representing the material of the surface of the object, a distance attenuation representing the attenuation of the light over distance, and an incident angle of the light with respect to the surface of the object. The information processing device according to claim 16.

18. the coefficient derivation unit derives the diffuse reflection coefficient of the target point based on the reflected light intensity of the target point; the prediction unit derives the predicted value using the diffuse reflection coefficient derived by the coefficient derivation unit; The encoding unit encodes the diffuse reflection coefficients derived by the coefficient derivation unit. The information processing device according to claim 17.

19. the reflection model is a specular reflection model, The coefficient derivation unit is configured to derive, as the coefficients, a specular reflection coefficient and a diffuse reflection coefficient that represent the material of the surface of the object, a distance attenuation that represents the attenuation of the light over distance, and an incident angle of the light with respect to the surface of the object. The information processing device according to claim 16.

20. Deriving coefficients of a light reflection model on the surface of a three-dimensional object; performing a prediction process using the reflection model and the coefficients to derive a predicted value of reflected light intensity, which is attribute data of a point cloud that represents the object as a set of points; generating a prediction residual that is a difference between the reflected light intensity and the derived predicted value; Encoding the generated prediction residual Information processing methods.

Images