Point cloud processing device, method, and program

Abstract

A point cloud processing device according to an embodiment includes: a point cloud input unit for accepting input of a point cloud measured by measuring a measurement target object by means of a first three-dimensional measuring device and a second three-dimensional measuring device; an end point input unit for accepting input of position information relating to a point cloud on a coordinate system of the first and second three-dimensional measuring devices related to at least three end points of the measurement target object, the point cloud having been being input using the point cloud input unit; a positional relationship determining unit for determining a positional relationship between the first and second three-dimensional measuring devices and the measurement target object on the basis of the position information input using the end point input unit; and a calculating unit for calculating an external parameter for converting the coordinate system of the first and second three-dimensional measuring devices into a reference coordinate system on the basis of the positional relationship determined by the positional relationship determining unit.

Description

Point cloud processing device, method, and program

[0001] An embodiment of the present invention relates to a point cloud processing device, method, and program.

[0002] In recent years, in many fields, including architecture and surveying, industrial applications have been made in which 3D measurement devices, such as laser radars, e.g., LiDAR, are used to measure real space and acquire and use 3D point cloud data.

[0003] However, with a 3D measurement device fixed in one location, it is difficult to measure 3D data of the entire measurement space without omission due to obstructions by objects in the measurement space or limitations on the measurable distance.

[0004] Therefore, 3D point cloud data of the entire space to be measured is generated by combining 3D point cloud data measured from a total of multiple locations by moving a single 3D measurement device, and 3D point cloud data measured from multiple locations by multiple 3D measurement devices.

[0005] Furthermore, it has also become common to measure continuous three-dimensional point cloud data using a plurality of three-dimensional measurement devices and generate moving point cloud data, rather than video data, with reduced occlusion.

[0006] Here, each 3D point cloud data is expressed in a coordinate system based on the 3D measuring device that measured the data, and in order to combine these data, external parameters are required to convert each different coordinate system into the reference coordinate system.

[0007] The most direct method for obtaining the external parameters is to measure the direction and distance from a reference 3D measurement device to another measurement device, but this method is not generally applicable because it has the constraint that the two 3D measurement devices must be located in positions where there is a clear line of sight between them and the 3D measurement devices must have the shape of the measurement target.

[0008] Another method for obtaining the external parameters is to perform iterative calculations so that the sum of the differences in the positions of pairs of points selected from both point clouds is minimized when the two point clouds are superimposed, such as ICP (Iterative Closest Point) (see, for example, Non-Patent Document 1). However, this method presupposes that the two point clouds are sufficiently close to each other, and therefore has the problem of requiring an enormous amount of calculation unless the two 3D measurement devices actually perform spatial measurements at positions sufficiently close to each other, or unless approximate values ​​of the external parameters of the two 3D measurement devices are given.

[0009] Therefore, the most common method for obtaining external parameters is to place a measurement target of known size at a position that can be measured from both of the two 3D measurement devices, and calculate the external parameters between the two 3D measurement devices by triangulation-like geometric calculations based on the measurement results of this target.

[0010] In this method, a plate with a known size, called a calibration board, is generally used as a target (see, for example, Non-Patent Document 2).

[0011] BESL, Paul J.; MCKAY, Neil D. A Method for registration of 3-D shapes. In: Sensor fusion IV: control paradigms and data structures. Spie, 1992. p. 586-606.LEE, Hyunsuk; CHUNG, Woojin. Extrinsic calibration of multiple 3D LiDAR sensors by the use of planar objects. Sensors, 2022, 22.19: 7234.

[0012] When the calibration board is measured with a 3D measurement device to acquire 3D point cloud data, a point cloud is acquired for the surface of the calibration board facing the 3D measurement device. For both the reference 3D measurement device and the other 3D measurement device, the extrinsic parameters of the two 3D measurement devices can be calculated using the positions on the coordinate systems of the 3D measurement devices of at least three corners of the point cloud of the calibration board.

[0013] In this case, due to the limitations of the spatial resolution of the three-dimensional measuring device, the point in the acquired point cloud that is closest to the corner of the calibration board does not exactly match the coordinates of that corner, and in consideration of the random error that is a measurement error, a method may be used in which a plane is calculated using a method such as the least squares method from the point clouds measured by each measuring device that are contained in the reflective surface of the calibration board, and external parameters are calculated to match the two planes calculated from each point cloud.

[0014] However, when using this method, if the same surface of the calibration board is not facing the two 3D measurement devices with respect to the positional relationship between the two 3D measurement devices and the calibration board, the thickness of the calibration board as a physical entity will be introduced as an error in the calculation, which creates a problem in that the positional relationship during measurement is subject to restrictions other than line of sight.

[0015] In particular, when two three-dimensional measuring devices are installed relatively far apart, it is preferable to use a relatively large plate as the calibration board, and in order to impart rigidity to such a relatively large calibration board so that it does not distort, the thickness of the calibration board must be made relatively large, which exacerbates the problems associated with the above limitations.

[0016] Furthermore, although this method can statistically reduce the random errors in the depth direction of each point of the 3D measuring device, it cannot reduce the errors resulting from the fact that the point closest to the corner of the calibration board does not exactly match the coordinates of that corner, as described above.In order to statistically reduce these errors, it is necessary to calculate the external parameters from the point cloud of the measurement results of multiple calibration boards, rather than calculating them based on the measurement results of a single calibration board, and this also poses the issue of restricting the installation location of the calibration board.

[0017] The present invention has been made in light of the above-mentioned circumstances, and its purpose is to provide a point cloud processing device, method, and program that enables appropriate measurement of a measurement object using a three-dimensional measurement device.

[0018] A point cloud processing device according to one aspect of the present invention includes: a point cloud input unit that inputs a point cloud measured by measuring a measurement object using a first three-dimensional measuring device and a second three-dimensional measuring device; an endpoint input unit that inputs positional information of the point cloud input by the point cloud input unit, the point cloud on the coordinate systems of the first and second three-dimensional measuring devices relating to at least three endpoints of the measurement object; a positional relationship determination unit that determines a positional relationship between the first and second three-dimensional measuring devices and the measurement object based on the positional information input by the endpoint input unit; and a calculation unit that calculates external parameters that convert the coordinate systems of the first and second three-dimensional measuring devices into a reference coordinate system based on the positional relationship determined by the positional relationship determination unit.

[0019] A point cloud processing method according to one aspect of the present invention is a method performed by a point cloud processing device, and includes: inputting, via a point cloud input unit of the point cloud processing device, a point cloud measured by measuring a measurement object using a first three-dimensional measurement device and a second three-dimensional measurement device; inputting, via an endpoint input unit of the point cloud processing device, positional information on the point clouds on the coordinate systems of the first and second three-dimensional measurement devices that relate to at least three endpoints of the measurement object, the point clouds input by the point cloud input unit; determining, via a positional relationship determination unit of the point cloud processing device, a positional relationship between the first and second three-dimensional measurement devices and the measurement object, based on the positional information input by the endpoint input unit; and calculating, via a calculation unit of the point cloud processing device, external parameters for converting the coordinate systems of the first and second three-dimensional measurement devices into a reference coordinate system, based on the positional relationship determined by the positional relationship determination unit.

[0020] According to the present invention, it is possible to perform appropriate measurement of a measurement object using a three-dimensional measurement device.

[0021] FIG. 1 is a diagram showing an application example of a point cloud data processing device according to an embodiment of the present invention. FIG. 2 is a flowchart showing an example of a procedure for processing operations by the point cloud data processing device. FIG. 3 is a diagram explaining a first example of calculation of external parameters. FIG. 4 is a diagram explaining a second example of calculation of external parameters. FIG. 5 is a diagram showing an example of determining the surface that a target faces toward a three-dimensional measuring device. FIG. 6 is a diagram explaining a third example of calculation of external parameters. FIG. 7 is a diagram showing an example of the distance between a target and the three-dimensional measuring device and the angle formed between the target and the three-dimensional measuring device. FIG. 8 is a block diagram showing an example of the hardware configuration of a point cloud data processing device according to an embodiment of the present invention.

[0022] An embodiment of the present invention will now be described. Fig. 1 is a diagram showing an application example of a point cloud data processing device according to an embodiment of the present invention. Fig. 2 is a flowchart showing an example of the procedure of processing operations by the point cloud data processing device. As shown in Fig. 1, a point cloud data processing device (point cloud processing device) 100 according to an embodiment of the present invention includes a point cloud input unit 10, an end point input unit 20, a positional relationship determination unit 30, a storage unit 40, and an external parameter calculation unit 50.

[0023] In this embodiment, the operator places the calibration board in a position where it has a good view from two three-dimensional measurement devices (sometimes referred to as LiDARs), and a point cloud including the measurement results of the calibration board is measured using each of the first three-dimensional measurement device (LiDAR 1) L1 and the second three-dimensional measurement device (LiDAR 2) L2, and point cloud data D1 is acquired by the first three-dimensional measurement device L1, and second point cloud data D2 is acquired by the second three-dimensional measurement device L2.

[0024] The acquired point cloud data D1 and D2 are input to the point cloud input unit 10 of the point cloud data processing device 100 (S10). The specific input method depends on the point cloud data output method used by the three-dimensional measurement devices L1 and L2, but is generally a method in which information output and saved as a data file is input by file specification, and this method is not particularly limited.

[0025] Data indicating the size and thickness of the calibration board is input by an operator through an operation on a GUI (Graphical User Interface) and is stored in advance in the storage unit 40 of the point cloud data processing device 100. The method for achieving this storage is not particularly limited. For example, the size and thickness data may be set on the premise of a default calibration board and stored in advance in the storage unit 40 of the point cloud data processing device 100, or the size and thickness data may be input by the operator by some means as appropriate and stored in the storage unit 40.

[0026] Then, position information on the coordinate system of each of the three-dimensional measuring devices L1 and L2, which corresponds to at least three corners of the calibration board among the point cloud data D1 and D2, is input by operating the GUI, and is input to the endpoint input unit 20 together with the point cloud data D1 and D2 as the endpoints of the target (S20).

[0027] The method for inputting the position information is not particularly limited. One example is a method in which point cloud data is projected onto a two-dimensional display and position information is specified on the projected screen using a pointing device such as a mouse.

[0028] The positional relationship determination unit 30 uses data indicating the size and thickness of the calibration board stored in the memory unit 40 to calculate the approximate positional relationship between each of the three-dimensional measuring devices L1 and L2 and the calibration board by triangulation-like geometric calculations based on the coordinate values ​​of each orthogonal coordinate of the corresponding endpoints specified for the point cloud data D1 and D2.

[0029] Based on the calculation results, the positional relationship determination unit 30 determines the relationship between the direction in which the calibration board faces the first three-dimensional measurement device L1 and the direction in which the calibration board faces the second three-dimensional measurement device L2, for example, whether the calibration board faces the same side or opposite sides relative to the first three-dimensional measurement device L1 and the second three-dimensional measurement device L2. The positional relationship determination unit 30 outputs the result of this determination together with the calculation results of the positional relationship to the external parameter calculation unit 50 (S30). The above-mentioned "facing the same side" means that the side of the calibration board facing the first three-dimensional measurement device L1 and the side of the calibration board facing the second three-dimensional measurement device L2 are the same. Furthermore, the above-mentioned "facing opposite sides" means that the side of the calibration board facing the first three-dimensional measurement device L1 and the side of the calibration board facing the second three-dimensional measurement device L2 are different.

[0030] The external parameter calculation unit 50 calculates the external parameters between the first three-dimensional measurement device L1 and the second three-dimensional measurement device L2 based on the result of the determination by the positional relationship determination unit 30 (S40).

[0031] A first example of the external parameter is an external parameter that, when the calibration board faces the same side relative to the first three-dimensional measurement device L1 and the second three-dimensional measurement device L2, causes a plane fitted to the point cloud data forming the surface of the calibration board in the point cloud data D1 to completely coincide with a plane fitted to the point cloud data forming the surface of the calibration board in the point cloud data D2. A second example of the external parameter is an external parameter that, when the calibration board faces opposite sides relative to the first three-dimensional measurement device L1 and the second three-dimensional measurement device L2, causes a plane fitted to the point cloud data forming the surface of the calibration board in the point cloud data D1 to be parallel to a plane fitted to the point cloud data forming the surface of the calibration board in the point cloud data D2, separated by a distance equal to the thickness of the calibration board, as stored in storage unit 40.

[0032] The point cloud data processing device according to this embodiment can contribute to the accurate calculation of extrinsic parameters between two 3D measurement devices by using sets of point cloud data measured by arranging multiple calibration boards with a high degree of freedom, or by moving and arranging calibration boards with a high degree of freedom and using multiple sets of measured point cloud data.

[0033] FIG. 3 is a diagram illustrating a first example of extrinsic parameter calculation. The three-dimensional measurement devices and targets shown in FIG. 3 , including target T1 in this example, are actually arranged in three-dimensional space, but are depicted in plan views for ease of understanding. The same applies to FIGS. 4 to 7 , which will be described later. Also, in FIG. 3 , for ease of understanding, a line connecting the first three-dimensional measurement device L1 and the second three-dimensional measurement device L2 is depicted as the horizontal axis. The same applies to FIGS. 4 , 5 , and 7 , which will be described later. As shown in FIG. 3 , when target T1, which is a calibration board, faces the same side toward first three-dimensional measurement device L1 and second three-dimensional measurement device L2, the extrinsic parameter calculation unit 50 calculates, for both first three-dimensional measurement device L1 and second three-dimensional measurement device L2, a plane represented by the least square error of point clouds located within a certain distance from a longitudinal plane spanned by corner points of target T1, and calculates extrinsic parameters that allow these calculated planes to overlap.

[0034] This calculation makes it possible to determine the rotational parameters among the external parameters, and it is expected that the translational component values ​​calculated by geometric calculation from the coordinate values ​​of the corners of the target given to each three-dimensional measurement device will also improve.

[0035] 4 is a diagram illustrating a second example of the calculation of the extrinsic parameters. As shown in Fig. 4, when a target T2, which is a calibration board, faces different sides toward the first three-dimensional measurement device L1 and the second three-dimensional measurement device L2, the extrinsic parameter calculation unit 50 calculates, for both the first three-dimensional measurement device L1 and the second three-dimensional measurement device L2, planes represented by the least square error of point clouds existing within a certain distance from a longitudinal plane spanned by corner points of the target T1, and calculates extrinsic parameters that make these calculated planes parallel to each other and spaced apart by the thickness of the target T2, i.e., the length in the shorter direction.

[0036] 5 is a diagram showing an example of determining the surface of the target facing the three-dimensional measurement device. Whether the target T shown in FIG. 5 faces the same surface or different surfaces with respect to the first three-dimensional measurement device L1 and the second three-dimensional measurement device L2 can be determined by geometric calculation by the positional relationship determination unit 30 when the measurement coordinates of points nearest to the corners of the target T among the point clouds measured by the first three-dimensional measurement device L1 and the second three-dimensional measurement device L2 are given.

[0037] FIG. 6 illustrates a third example of the calculation of extrinsic parameters. In FIG. 6 , the x- and y-coordinate axes of the first three-dimensional measurement device L1 are indicated by solid lines, and the x- and y-coordinate axes of the second three-dimensional measurement device L2 are indicated by dotted lines. As shown in FIG. 6 , when multiple targets T1, T2, T3, and T4 are given, the accuracy of the extrinsic parameters can be improved by averaging the extrinsic parameters calculated for each target. Furthermore, by averaging in this manner, the accuracy of the translational components of the extrinsic parameters, which remain ambiguous in the calculation for matching planes calculated for a single target as described above, can also be improved. It is also preferable that the extrinsic parameter calculation unit 50 calculates values ​​reflecting the quality of the measurement conditions based on the positional relationships between each target and the first and second three-dimensional measurement devices L1 and L2 shown in FIG. 6 , and then calculates a weighted average incorporating the calculated values.

[0038] 7 is a diagram showing an example of the distance between the target and the three-dimensional measurement device and the angle formed between the target and the three-dimensional measurement device. It is more desirable that the target be positioned in a position relatively close to the first three-dimensional measurement device L1 and the second three-dimensional measurement device L2, as in the position of target T2 shown in FIG. 7, rather than in a position relatively far from the first three-dimensional measurement device L1 and the second three-dimensional measurement device L2, as in the position of target T1 shown in FIG. 7. It is also desirable that the normal to the target with respect to the vector connecting the first three-dimensional measurement device L1 and the second three-dimensional measurement device L2 be positioned in a position relatively close to the first three-dimensional measurement device L1 and the second three-dimensional measurement device L2, as in the position of target T2 shown in FIG. 7, so that the angle between the normal to the target and the vector connecting the first three-dimensional measurement device L1 and the second three-dimensional measurement device L2 is small. On the other hand, as shown in FIG. 7 , for target T3 where the first and second conditions are optimized, the first 3D measurement device L1 and the second 3D measurement device L2 each measure different surfaces of target T3, and therefore the thickness of target T3 is introduced into the error.

[0039] In the point cloud data processing device according to the present embodiment described above, the calibration board can be installed in any position as long as it is visible from both of the two 3D measuring devices, and point cloud data for calculating the external parameters can be acquired by installing the calibration board in that position, and good external parameters can be calculated.

[0040] 8 is a block diagram showing an example of the hardware configuration of a point cloud data processing device according to an embodiment of the present invention. In the example shown in FIG. 8, the point cloud data processing device 100 according to the embodiment is configured, for example, by a server computer or a personal computer, and has a hardware processor 111A such as a CPU (Central Processing Unit). A program memory 111B, a data memory 112, an input / output interface 113, and a communication interface 114 are connected to this hardware processor 111A via a bus 115.

[0041] The communication interface 114 includes, for example, one or more wireless communication interface units, and enables transmission and reception of information to and from a communication network. As the wireless interface, for example, an interface that adopts a low-power wireless data communication standard such as a wireless LAN (Local Area Network) is used.

[0042] An input device 200 and an output device 300, which are attached to the point cloud data processing device 100 and used by a user or the like, are connected to the input / output interface 113. The input / output interface 113 can take in operation data input by a user or the like through the input device 200, such as a keyboard, touch panel, touchpad, or mouse, and can output output data to an output device 300, which includes a display device using a liquid crystal or organic electroluminescence (EL) display, for display. The input device 200 and the output device 300 may be devices built into the point cloud data processing device 100, or may be input devices and output devices of other information terminals that can communicate with the point cloud data processing device 100 via a network.

[0043] The program memory 111B is a non-transitory tangible storage medium that is a combination of a non-volatile memory that can be written to and read from at any time, such as a hard disk drive (HDD) or a solid state drive (SSD), and a non-volatile memory such as a read only memory (ROM), and can store programs necessary to execute various control processes, etc., according to one embodiment.

[0044] The data memory 112 is a tangible storage medium that is, for example, a combination of the above-mentioned nonvolatile memory and a volatile memory such as RAM (Random Access Memory), and can be used to store various data or information acquired and created during various processing steps.

[0045] A point cloud data processing device 100 according to an embodiment of the present invention can be configured as a data processing device having the units shown in FIG. 1 as software-based processing function units.

[0046] The information storage unit used as a work memory or the like by each unit of the point cloud data processing device 100 can be configured by using the data memory 112 shown in Fig. 8. However, these configured storage areas are not essential components within the point cloud data processing device 100, and may be areas provided in, for example, an external storage medium such as a USB (Universal Serial Bus) memory, or a storage device such as a database server located in the cloud.

[0047] The processing function units in each of the above units can be realized by reading and executing a program stored in the program memory 111B by the hardware processor 111A. Note that some or all of these processing function units may be realized in various other forms, including integrated circuits such as an application specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0048] The methods described in each embodiment can be stored as a program (software means) that can be executed by a computer on a recording medium such as a magnetic disk (floppy disk, hard disk, etc.), optical disk (CD-ROM, DVD, MO, etc.), or semiconductor memory (ROM, RAM, flash memory, etc.), and can also be distributed by transmitting it via a communication medium. The program stored on the medium also includes a configuration program that configures the software means (including not only execution programs but also tables or data structures) that the computer executes. The computer that realizes this device reads the program stored on the recording medium and, in some cases, configures the software means using the configuration program, and executes the above-mentioned processing by controlling the operation of this software means. The term "recording medium" as used herein is not limited to a storage medium for distribution, but also includes a storage medium such as a magnetic disk or semiconductor memory installed inside the computer or in a device connected via a network.

[0049] The present invention is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the spirit of the invention. Furthermore, the embodiments may be implemented in appropriate combinations, in which case the combined effects can be obtained. Furthermore, the above-described embodiments include various inventions, and various inventions can be extracted by combining selected elements from the disclosed elements. For example, if the problem can be solved and the desired effect can be obtained even if some elements are deleted from all elements shown in the embodiments, the configuration from which these elements are deleted can be extracted as an invention.

[0050] REFERENCE SIGNS LIST 100... Point cloud data processing device 10... Point cloud input unit 20... End point input unit 30... Positional relationship determination unit 40... Storage unit 50... External parameter calculation unit

Claims

1. A point cloud processing device comprising: a point cloud input unit that inputs a point cloud measured by measuring a measurement object using a first three-dimensional measurement device and a second three-dimensional measurement device; an endpoint input unit that inputs positional information of the point cloud input by the point cloud input unit, the point cloud on the coordinate systems of the first and second three-dimensional measurement devices relating to at least three endpoints of the measurement object; a positional relationship determination unit that determines the positional relationship between the first and second three-dimensional measurement devices and the measurement object based on the positional information input by the endpoint input unit; and a calculation unit that calculates external parameters that convert the coordinate systems of the first and second three-dimensional measurement devices into a reference coordinate system based on the positional relationship determined by the positional relationship determination unit.

2. The point cloud processing device according to claim 1, wherein the calculation unit calculates the external parameters such that, when the surface of the measurement object facing the first three-dimensional measurement device and the surface of the measurement object facing the second three-dimensional measurement device are the same, a plane represented by the least square error of the point cloud measured by the first three-dimensional measurement device and input by the point cloud input unit, within a certain distance from the surface of the measurement object, indicated by the endpoints input by the endpoint input unit, and a plane represented by the least square error of the point cloud measured by the second three-dimensional measurement device and input by the point cloud input unit, within a certain distance from the surface of the measurement object, indicated by the endpoints input by the endpoint input unit, overlap.

3. The point cloud processing device according to claim 1, wherein the calculation unit calculates the external parameters such that, when the surface of the measurement object facing the first three-dimensional measurement device and the surface of the measurement object facing the second three-dimensional measurement device are different, a first plane represented by the least square error of the point cloud measured by the first three-dimensional measurement device and input by the point cloud input unit, within a certain distance from the surface of the measurement object, indicated by the endpoints input by the endpoint input unit, and a second plane represented by the least square error of the point cloud measured by the second three-dimensional measurement device and input by the point cloud input unit, within a certain distance from the surface of the measurement object, indicated by the endpoints input by the endpoint input unit, are spaced apart by the thickness of the measurement object between the first and second planes.

4. A method performed by a point cloud processing device, comprising: inputting, via a point cloud input unit of the point cloud processing device, a point cloud measured by measuring an object to be measured using a first three-dimensional measuring device and a second three-dimensional measuring device; inputting, via an endpoint input unit of the point cloud processing device, positional information on the point clouds on the coordinate systems of the first and second three-dimensional measuring devices, which are input via the point cloud input unit and relate to at least three endpoints of the object to be measured; determining, via a positional relationship determination unit of the point cloud processing device, the positional relationship between the first and second three-dimensional measuring devices and the object to be measured, based on the positional information input via the endpoint input unit; and calculating, via a calculation unit of the point cloud processing device, external parameters for converting the coordinate systems of the first and second three-dimensional measuring devices into a reference coordinate system, based on the positional relationship determined by the positional relationship determination unit.

5. A point cloud processing program that causes a processor to function as each part of the point cloud processing device according to any one of claims 1 to 3.

Images