Encoding device, encoding method, and program

Abstract

This encoding device comprises: an acquisition unit that acquires point group data indicating the three-dimensional position of a subject in a spherical coordinate system; an angle data encoding unit for performing frequency conversion of scan angle data that is time-series data of the scan angles of points included in the point group data and for quantizing and encoding a coefficient; and a distance data encoding unit for predicting the distance, for each of the points, from the position of a measurement source to an encoding target point on the basis of the distance to a neighboring point from the position of the measurement source in a distance map in which the scan angle is defined as an axis, and for encoding distance data indicating the predicted distance.

Description

[Technical Field]

[0001] The present invention relates to an encoding device, an encoding method, and a program. [Background technology]

[0002] G-PCC (Geometry-based Point Cloud Compression) is an international standard currently under study for the compression and encoding of point cloud data generated using LiDAR (Light Detection and Ranging), etc. G-PCC employs two encoding methods for geometry data, which is the coordinate information of point clouds: Octree Geometry, which is based on an octree representation, and Predictive Geometry, which is based on a predictive tree representation.

[0003] In the Octree Geometry method, the space encompassing the input point cloud is divided into an octree, and if a point exists in each divided region, further division is performed to determine the octree structure. Leaf nodes with no children correspond to each of the input point cloud. In this method, the division status of each node is coded in depth order starting from the root node. The division status of each node can be predicted and coded from surrounding nodes and parent nodes.

[0004] In the Predictive Geometry method, a prediction tree structure is determined for the input point cloud. Each node in the tree corresponds to one of the input points. In this method, the number of children and the position of each child are coded for each vertex, starting from the root node. When coding the child's position, it is predicted from the parent's position, and the prediction mode and prediction residual are coded.

[0005] Furthermore, in addition to the Cartesian coordinate system represented by orthogonal linear coordinates (x, y, z), predictive coding using a spherical coordinate system represented by spherical coordinates (r, θ, φ) can improve coding efficiency. For example, in the Predictive Geometry method, for a point cloud measured by a LiDAR device whose scanning pattern is a ring (rotating by φ), an offset can be applied to the φ of the root node to use as the predicted value of φ. Summary of the Invention [Problem to be solved by the invention]

[0006] Point clouds measured by devices such as LiDAR are generally sparsely distributed spatially, and the scanning pattern is often a simple shape such as a circular ring (due to the laser being driven by a single-axis, uniform rotation). In this case, they can be efficiently encoded using the Predictive Geometry method. However, in the case of point clouds acquired by LiDAR, which has a complex, intersecting scanning pattern, there is an issue that expressing them using a Predictive Tree results in countless branches, which reduces encoding efficiency.

[0007] In view of the above circumstances, an object of the present invention is to provide a technique that can efficiently encode point cloud data. [Means for solving the problem]

[0008] One aspect of the present invention is an encoding device that includes an acquisition unit that acquires point cloud data that indicates the three-dimensional position of a subject in a spherical coordinate system; an angle data encoding unit that frequency-converts scanning angle data, which is time-series data of the scanning angle of each point included in the point cloud data, and quantizes and encodes the coefficients; and a distance data encoding unit that predicts, for each of the points, the distance from the position of the measurement source to the point to be encoded based on the distance from the position of the measurement source to a neighboring point in a distance map whose axis is the scanning angle, and encodes distance data that indicates the predicted distance.

[0009] Another aspect of the present invention is an imaging apparatus including: an acquisition step of acquiring point cloud data indicating a three-dimensional position of a subject in a spherical coordinate system; This encoding method includes an angle data encoding step of frequency-converting scanning angle data, which is time-series data of the scanning angle of each point included in the point cloud data, and quantizing and encoding the coefficients; and a distance data encoding step of predicting, for each point, the distance from the position of the measurement source to the encoding target point based on the distance from the position of the measurement source to a neighboring point in a distance map whose axis is the scanning angle, and encoding distance data indicating the predicted distance.

[0010] Another aspect of the present invention is a program for causing a computer to function as the encoding device described above. [Effects of the Invention]

[0011] The present invention allows for efficient encoding of point cloud data. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a diagram illustrating the overall configuration of a point cloud data encoding system 1 according to an embodiment of the present invention. [Figure 2] 1 is a block diagram showing a functional configuration of an encoding device 10 according to an embodiment of the present invention. [Figure 3] 1 is a flowchart showing the operation of the encoding device 10 according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, a point cloud data encoding system according to an embodiment of the present invention will be described with reference to the drawings.

[0014] [Configuration of point cloud data encoding system] The configuration of a point cloud data encoding system 1 according to an embodiment of the present invention will be described below. The point cloud data encoding system 1 according to the embodiment described below scans the surface of an object and measures the three-dimensional position of each point on a scanning line to generate point cloud data. For the generated point cloud data, the point cloud data encoding system 1 converts the trajectory of the scanning line in a spherical coordinate system into time-series data of angle (scanning angle) and distance over time, represents and encodes the time-series data of angle in the frequency domain, and predictively encodes the time-series data of distance on a two-dimensional map with the angle component as its axis. The point cloud data encoding system 1 decodes the point cloud data compressed by the above-described predictive encoding.

[0015] 1 is a diagram showing the overall configuration of a point cloud data encoding system 1 according to an embodiment of the present invention. As shown in FIG. 1, the point cloud data encoding system 1 includes an encoding device 10, a decoding device 20, and a measurement device 30.

[0016] The measurement device 30 scans the surface of the object, measures the three-dimensional position of each point on the scan line, and generates point cloud data. The measurement device 30 has a measurement function such as LiDAR that can measure the three-dimensional position of each point on the scan line by scanning the surface of the object using, for example, a MEMS (Micro Electro Mechanical Systems) mirror or the like to continuously move a laser beam. The measurement device 30 outputs the generated point cloud data to the encoding device 10.

[0017] Encoding device 10 acquires point cloud data output from measurement device 30. For the acquired point cloud data, encoding device 10 converts the trajectory of a scanning line in a spherical coordinate system into time-series data of angle and distance over time, represents and encodes the angle time-series data in the frequency domain, and predictively encodes the distance time-series data on a two-dimensional map with the angle component as its axis. Encoding device 10 outputs the encoded data compressed by the above-mentioned predictive encoding to decoding device 20.

[0018] The decoding device 20 obtains the coded data output from the coding device 10. The decoding device 20 decodes the obtained coded data.

[0019] [Encoder configuration] The configuration of the encoding device 10 in this embodiment will be described in more detail below.

[0020] Fig. 2 is a block diagram showing the functional configuration of an encoding device 10 according to an embodiment of the present invention. As shown in Fig. 2, the encoding device 10 includes a point cloud data input unit 101, a point cloud data storage unit 102, an angle data encoding unit 103, an angle residual encoding unit 104, a distance prediction unit 105, a distance data encoding unit 106, and an encoded data output unit 107.

[0021] The point cloud data input unit 101 (acquisition unit) accepts input of point cloud data indicating three-dimensional positions of the surface of the subject measured by the measurement device 30. The point cloud data input unit 101 stores the input point cloud data in the point cloud data storage unit 102. Note that the point cloud data storage unit 102 may be configured to be provided in an external device other than the encoding device 10.

[0022] The point cloud data has, as attribute information for each point, the value of the scanning angle of the measurement laser when that point was measured. Alternatively, the point cloud data may have, as attribute information for each point, information indicating the three-dimensional position of each point and the attitude of the measurement device 30, and the point cloud data input unit 101 may be configured to be able to derive the scanning angle based on that information. Note that the laser rotates on two axes, and the scanning angle has two values, θ and φ.

[0023] Furthermore, the point cloud data has a timestamp (time information) of the time when each point was measured as attribute information of that point. Alternatively, the scanning time of each point may be unknown, but the scanning speed may be constant, the difference in scanning time between points may be known, and the point cloud data input unit 101 may be configured to set the scanning time of one point to 0 and derive the relative scanning times of other points.

[0024] If the timestamp accuracy is insufficient and multiple points scanned close to each other have the same timestamp, the point cloud data input unit 101 assumes that, for a group of points with the same or similar timestamps, when the points are projected onto a curved surface of radius r in a spherical coordinate system, the points will be arranged on a curve, and rearranges the order of the points from the start point of the curve to the end point.

[0025] If the scanning time of each point is unknown, the point cloud data input unit 101 may determine the order of points so that the amount of angular change between points is constant, assuming that the scanning speed is constant, and assign relative times to that order.

[0026] If there are multiple measuring devices 30, or if the measuring device 30 is equipped with multiple lasers and performs two or more scans simultaneously, the point cloud data input unit 101 separates the input point cloud data into lasers corresponding to the point cloud data, and stores them in the point cloud data storage unit 102 as point cloud groups.

[0027] Note that any method for classifying point cloud data may be used. For example, if the point cloud data is assigned an ID for identifying the laser as attribute information, the point cloud data may be classified based on the ID. Alternatively, some kind of clustering process may be performed. For example, points measured consecutively by each laser may be grouped by determining, in scanning order, whether to add each point to an already classified group or to generate a new group based on the distance from the already classified point.

[0028] Furthermore, if the posture of the measuring device 30 changes during scanning, the angle change becomes complex, which deteriorates the encoding efficiency in encoding the angle data described below, so the point cloud data may be separated into different point cloud groups before and after the change.

[0029] The angle data encoding unit 103 encodes the angle data. When there are multiple measurement devices 30, or when the measurement device 30 is equipped with multiple lasers and performs two or more scans simultaneously, the angle data encoding unit 103 encodes the angle data for each point cloud group. For the point cloud data stored in the point cloud data storage unit 102, the angle data encoding unit 103 arranges the angle information in the order of scanning, and generates time-series angle data.

[0030] If the scanning speed is not constant, the angle data encoding unit 103 generates time-series angle data in which each element is arranged at regular time intervals by interpolation from the scanning angle and time of the point cloud data. In this case, the angle data encoding unit 103 may also encode the time data corresponding to each point, and perform sampling in the decoding device 20 to reconstruct the original angle data. In the case of lossy encoding, the angle data encoding unit 103 does not need to perform sampling.

[0031] The angle data encoding unit 103 frequency-converts the time-series angle data and encodes the coefficients. Any method of frequency conversion may be used. For example, when a laser in the measurement device 30 is driven by a MEMS device, the angle often changes according to a sine wave, so the angle data encoding unit 103 can efficiently express the data using a DCT (Discrete Cosine Transform) transformation.

[0032] At this time, the angle data encoding unit 103 may allow a certain degree of error when truncating or quantizing the coefficients. The angle data encoding unit 103 may evaluate this error not only in terms of the angle but also in terms of the error in the coordinates in a rectangular coordinate system that are ultimately decoded. Furthermore, the angle data encoding unit 103 may make a decision by evaluating a trade-off with the amount of code.

[0033] Alternatively, instead of explicitly encoding the frequency components, the angle data encoding unit 103 may use an appropriate one of predetermined angle data patterns and encode information for identifying the pattern. In this case, the angle data encoding unit 103 may encode the frequency components of each pattern as additional information common to all groups. Alternatively, the angle data encoding unit 103 may express the angle data as a function for an arbitrary time and encode its parameters.

[0034] Note that the angle data encoding unit 103 may use the same coefficients as those in another group. For example, if the measurement device 30 is equipped with multiple lasers and the value of the scanning angle φ is common to all the lasers and only the value of the scanning angle θ differs, the angle data encoding unit 103 can use, for the angle φ, a coefficient from a group that corresponds to another laser that has already been encoded. Also, if the difference in the scanning angle θ between two different lasers is constant, one can be represented by adding an offset to the decoded result of the other.

[0035] The angular residual coding unit 104 codes the residual of the coded and decoded angle data. Note that the angular residual coding unit 104 may not perform coding when the residual is small, and may instead code a flag indicating whether decoding is necessary.

[0036] The distance prediction unit 105 encodes the distance data for each point. The distance prediction unit 105 encodes the input point cloud data in ascending order, starting with the point with the earliest scanning time. The distance prediction unit 105 predicts the distance between the position of the laser irradiation source (measurement source) and the encoding target point. Note that the distance prediction unit 105 may use the distance of the most recently encoded point as the predicted value, or may use the distance of a point with a closest scanning angle among the previously encoded points.

[0037] The distance prediction unit 105 generates a two-dimensional map of distance r (hereinafter referred to as "distance map") with the scanning angles θ and φ as axes. The distance prediction unit 105 stores the distance data of the points that have been encoded up to that point in the point cloud data storage unit 102, and performs prediction by referring to the distance map.

[0038] The distance prediction unit 105 may generate a distance map for each group of point cloud data, or may use a common distance map for all groups. Alternatively, the distance prediction unit 105 may refer to distance maps of other groups. Alternatively, the distance prediction unit 105 may select a specific distance map from multiple distance maps and encode information indicating the selected distance map as additional information.

[0039] If there is a change in the posture of the measuring device 30 or movement of the subject, the distance prediction unit 105 may update the map to reflect the change in posture or movement when using a distance map generated by one group in another group.

[0040] Alternatively, the distance prediction unit 105 may generate a new distance map from the decoded point cloud information. For example, the distance prediction unit 105 may convert the absolute coordinates in the Cartesian coordinate system of the decoded point cloud into relative spherical coordinates in the position and orientation of the measurement device 30 in the encoding control group, and generate a distance map based on the conversion result.

[0041] The distance prediction unit 105 may determine a predicted value from the distances to multiple neighboring points using an average value or other calculation method. For example, the distance prediction unit 105 may use the median of the distances to neighboring points as the predicted value. For example, the distance prediction unit 105 may perform weighting when making a prediction, and determine the weight based on the distance at time. Furthermore, if the distance prediction unit 105 has previously performed a prediction using the same neighboring point on the distance map, it may calculate the reliability of the prediction from the amount of prediction residual and determine the weight based on the calculation result.

[0042] The distance prediction unit 105 may select a specific prediction method from among several prediction methods to generate a predicted value, and may also encode information indicating the selected prediction method as additional information.

[0043] Distance data encoding unit 106 encodes a prediction residual value for a predicted value for the distance from the position of the laser irradiation source (measurement source) to the encoding target point. Note that distance data encoding unit 106 may quantize the prediction residual value before encoding. Distance data encoding unit 106 outputs the encoded data to encoded data output unit 107.

[0044] The coded data output unit 107 acquires the coded data output from the distance data coding unit 106. The coded data output unit 107 outputs the acquired coded data to the decoding device 20.

[0045] [Encoder operation] An example of the operation of the encoding device 10 will be described below. Fig. 3 is a flowchart showing the operation of the encoding device 10 in an embodiment of the present invention. The operation of the encoding device 10 shown in the flowchart of Fig. 3 starts, for example, when point cloud data is input from the measurement device 30 to the encoding device 10.

[0046] First, the point cloud data input unit 101 receives input of point cloud data indicating three-dimensional positions on the surface of the subject measured by the measurement device 30 (step S01).

[0047] Next, the angle data encoding unit 103 encodes the angle data. The angle data encoding unit 103 arranges angle information for the input point cloud data in scanning order to generate time-series angle data (step S02). The angle data encoding unit 103 frequency-converts the time-series angle data and encodes the coefficients (step S03).

[0048] Next, the distance prediction unit 105 encodes the distance data for each point. The distance prediction unit 105 encodes the input point cloud data in ascending order, starting with the point with the earliest scanning time. The distance prediction unit 105 predicts the distance between the position of the laser irradiation source (measurement source) and the encoding target point. The distance prediction unit 105 generates a two-dimensional distance map with the scanning angles θ and φ as axes (step S04). The distance prediction unit 105 stores the distance data of the points that have been encoded up to that point in the point cloud data storage unit 102, and performs prediction by referring to the distance map (step S05).

[0049] Next, distance data encoding unit 106 encodes a prediction residual value for a predicted value for the distance between the position of the laser irradiation source (measurement source) and the encoding target point (step S06).

[0050] Next, the coded data output unit 107 outputs the coded data to the decoding device 20 (step S07). This completes the operation of the coding device 10 shown in the flowchart of FIG.

[0051] As described above, the encoding device 10 in this embodiment frequency-converts the scanning angle data, which uses the scanning angle as time-series data, and quantizes and encodes the coefficients. For each point, the encoding device 10 uses a distance map with the scanning angle as its axis based on the decoding result to predict and encode the distance to the encoding target point from neighboring points on the distance map, and stores the distance based on the decoding result in the distance map.

[0052] The decoding device 20 corresponding to the encoding device 10 described above has the following configuration. The decoding device 20 decodes the scanning angle data. For each point, the decoding device 20 uses a distance map with the scanning angle as its axis based on the decoding result to predict the distance to the point to be decoded from nearby points on the map, and reconstructs the three-dimensional coordinates from the scanning angle and the distance.

[0053] (Variation) The encoding device 10 may encode attribute information such as color or reflection intensity in the same manner as the distance described above. For example, when encoding attribute information together with distance information point by point, the encoding device 10 may evaluate the reliability of a point to be used for prediction reference using the attribute information and use the evaluation result for selection. For example, the encoding device 10 may ignore points that have unusual intensity compared to neighboring points. Furthermore, for example, the encoding device 10 may perform evaluation using the number of responses to a pulse wave and the response waveform.

[0054] In addition, if the subject is semi-transparent and part of the laser light passes through the subject to reach another subject, it may be possible to observe multiple points with a single scan. In this case, the encoding device 10 can efficiently perform compression by using the same scanning time and angle values ​​for multiple points that have the same angle and different distances for the same input scanning time, and encoding the multiple distance values.

[0055] In the processing of each point, instead of encoding in scanning order, the encoding device 10 may, for example, first encode some points on a scanning line and then, when encoding the remaining points, reference points that have already been encoded in the same group and are located before and after them in scanning order. For example, when encoding an angular residual, if there are very few angular residuals at both the forward and backward points, the encoding device 10 may skip encoding the angular residual and a flag indicating whether encoding is necessary. Alternatively, the encoding device 10 may use a weighted sum of the angular residuals of both points as a predicted value of the angular residual of the point to be encoded and encode the predicted residual. Alternatively, when predicting distance information, the encoding device 10 may use a weighted sum of distance information at forward and backward points as a predicted value.

[0056] As described above, the encoding device 10 according to an embodiment of the present invention performs compression encoding of point cloud data. For example, the encoding device 10 encodes a point cloud acquired by a LiDAR device that continuously moves a laser beam using a MEMS mirror or the like to scan the surface of a subject and measure the three-dimensional position of each point on a scanning line. The encoding device 10 expresses and encodes the angle time-series data in the frequency domain as time-series data of angle and distance over time, representing the trajectory of the scanning line in a spherical coordinate system, and predictively encodes the distance time-series data on a two-dimensional distance map with the angle component as the axis.

[0057] As a result, the encoding device 10 according to the embodiment of the present invention can efficiently encode point clouds acquired by LiDAR with a complex intersecting scanning pattern. Furthermore, the encoding device 10 according to the embodiment of the present invention can perform predictive encoding with high accuracy even in scenes where the distance corresponding to the same scanning angle varies depending on the time, such as dynamic scenes where the position of the subject changes depending on the time.

[0058] According to the above-described embodiment, the encoding device includes an acquisition unit, an angle data encoding unit, and a distance data encoding unit. For example, the encoding device is the encoding device 10 in the embodiment, the acquisition unit is the point cloud data input unit 101 in the embodiment, the angle data encoding unit is the angle data encoding unit 103 in the embodiment, and the distance data encoding unit is the distance data encoding unit 106 in the embodiment.

[0059] The acquisition unit acquires point cloud data indicating the three-dimensional position of the subject in a spherical coordinate system. The angle data encoding unit frequency-converts scan angle data, which is time-series data of the scan angle of each point included in the point cloud data, and quantizes and encodes the coefficients. The distance data encoding unit predicts, for each point, the distance from the measurement source position to the encoding target point based on the distance from the measurement source position to a neighboring point in a distance map whose axis is the scan angle, and encodes distance data indicating the predicted distance.

[0060] In the encoding device described above, the distance map may be a two-dimensional map with axes corresponding to two scanning angles in a spherical coordinate system. For example, the two scanning angles are the values ​​of θ and φ in the embodiment.

[0061] In the encoding device described above, the point cloud data may include time information indicating the time at which the position of each point was measured. For example, the time information is a timestamp in the embodiment. In this case, the angle data encoding unit may generate the scan angle data by rearranging the scan angles of each point based on the time information.

[0062] In the encoding device, the distance data encoding unit may encode the distance data in order from the point with the earliest time based on the time information.

[0063] In the encoding device described above, if the point cloud data is data generated by multiple scans, the acquisition unit may group the point cloud data by scan. For example, the case where the point cloud data is data generated by multiple scans refers to a case in the embodiment where there are multiple measurement devices 30, or where the measurement device 30 is equipped with multiple lasers and performs two or more scans simultaneously. In this case, the angle data encoding unit may encode the scan angle data for each group.

[0064] In the encoding device described above, the distance data encoding unit may predict the distance from the position of the measurement source to the encoding target point based on the average or median of the distances from the position of the measurement source to multiple nearby points in the distance map.

[0065] A portion of the encoding device 10 in each of the above-described embodiments may be implemented by a computer. In this case, a program for implementing this function may be recorded on a computer-readable recording medium, and the program may be loaded into a computer system and executed. Note that the term "computer system" as used herein includes hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into a computer system. Furthermore, the term "computer-readable recording medium" may also include devices that dynamically store programs for a short period of time, such as communication lines used when transmitting programs via networks such as the Internet or telephone lines, or devices that store programs for a fixed period of time, such as volatile memory within a computer system serving as a server or client. The program may be for implementing a portion of the above-described functions, or may be capable of implementing the above-described functions in combination with a program already stored in the computer system, or may be implemented using a programmable logic device such as an FPGA (Field Programmable Gate Array).

[0066] Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention. [Explanation of symbols]

[0067] 1...point cloud data encoding system, 10...encoding device, 20...decoding device, 30...measuring device, 101...point cloud data input unit, 102...point cloud data storage unit, 103...angle data encoding unit, 104...angle residual encoding unit, 105...distance prediction unit, 106...distance data encoding unit, 107...encoded data output unit

Claims

1. an acquisition unit that acquires point cloud data indicating a three-dimensional position of a subject in a spherical coordinate system; an angle data encoding unit that performs frequency conversion on scanning angle data, which is time-series data of the scanning angles of each point included in the point cloud data, and quantizes and encodes the coefficients; a distance data encoding unit that predicts, for each of the points, a distance from a measurement source position to an encoding target point based on a distance map with the scanning angle as an axis, and encodes distance data indicating the predicted distance; Equipped with the distance map is generated based on the distance data of the points previously encoded; When there is a change in posture of a measuring device that measures the point cloud data or when there is a movement of the subject, the distance data encoding unit selects a prediction method for predicting the distance by reflecting the change in posture or the movement in the distance map, and encodes information indicating the selected prediction method as additional information. Encoding device.

2. The distance map is a two-dimensional map with axes corresponding to two scan angles in the spherical coordinate system. The encoding device according to claim 1 .

3. the point cloud data includes time information indicating the time when the position of each of the points was measured; The angle data encoding unit generates the scanning angle data by rearranging the scanning angles of the points based on the time information.

3. The encoding device according to claim 1 or 2.

4. The distance data encoding unit encodes the distance data in order from the point with the earliest time based on the time information. The encoding device according to claim 3 .

5. When the point cloud data is data generated by a plurality of scans, the acquisition unit: grouping the point cloud data by the scan; The angle data encoding unit encodes the scanning angle data for each group. Encoding device according to any one of claims 1 to 4.

6. an acquisition step of acquiring point cloud data indicating a three-dimensional position of a subject in a spherical coordinate system; an angle data encoding step of frequency-converting scanning angle data, which is time-series data of the scanning angles of each point included in the point cloud data, and quantizing and encoding the coefficients; a distance data encoding step of predicting, for each of the points, a distance from a measurement source position to an encoding target point based on a distance map with the scanning angle as an axis, and encoding distance data indicating the predicted distance; and the distance map is generated based on the distance data of the points previously encoded; In the distance data encoding step, if there is a change in posture of a measuring device that measures the point cloud data or a movement of the subject, a prediction method for predicting the distance by reflecting the change in posture or the movement in the distance map is selected, and information indicating the selected prediction method is encoded as additional information. Encoding method.

7. A program for causing a computer to function as the encoding device according to any one of claims 1 to 5.

Images