Estimation device, estimation method, and estimation program

Abstract

The present invention provides an estimation device including: an estimation unit that estimates, with respect to point cloud data to be modeled, a point cloud which represents an object and which is represented by at least one planar model; a removal unit that removes, from the point cloud data to be modeled, a point cloud representing the object and estimated by the estimation unit; and an iteration determination unit that causes the estimation by the estimation unit and the removal by the removal unit to be repeated until a predetermined iteration end condition is met.

Description

Estimation device, estimation method, and estimation program

[0001] The disclosed technology relates to an estimation device, an estimation method, and an estimation program.

[0002] In recent years, mobile 3D point cloud collection systems using vehicles or drones equipped with high-resolution 3D measurement devices such as LiDAR have been developed. Large-scale point cloud data has attracted attention as a digital twin of the real world, and its applications are expanding to urban development, infrastructure maintenance, and natural disaster prevention measures.

[0003] Generally, a model fitting algorithm called Random Sample Consensus (RANSAC) is often used to find a coherent shape from scattered point cloud data. Point Cloud Library (PCL), which implements RANSAC for 3D point clouds, is also well known (see Non-Patent Document 1).

[0004] Radu Bogdan Rusu and Steve Cousins. 3D is here: Point Cloud Library (PCL). In IEEE International Conference on Robotics and Automation (ICRA), Shanghai, China, May 9-13 2011. IEEE.

[0005] When dealing with 3D point clouds, the enormous data size makes them extremely difficult to handle. This is especially true for visualization and structural analysis, where the amount of calculation required is large and high speed is required.

[0006] The implementation in Non-Patent Document 1 can handle a few types of simple shapes such as straight lines and planes, but a sophisticated estimation procedure is required to obtain detailed shapes for complex point clouds.

[0007] In addition, in real-world scenes, it is necessary to detect multiple objects. For example, when distinguishing between roads and sidewalks, or when determining passable areas based on steps, it is necessary to expand the detection targets to multiple objects.

[0008] The disclosed technology has been made in consideration of the above points, and aims to provide an estimation device, an estimation method, and an estimation program that can estimate a detailed shape model representing multiple objects based on a huge amount of point cloud data.

[0009] A first aspect of the present disclosure is an estimation device including: an estimation unit that estimates, for point cloud data to be modeled, a point cloud that represents an object and is represented by at least one planar model; a removal unit that removes the point cloud that represents the object estimated by the estimation unit from the point cloud data to be modeled; and an iteration determination unit that causes the estimation by the estimation unit and the removal by the removal unit to be repeated until a predetermined iteration termination condition is satisfied.

[0010] A second aspect of the present disclosure is an estimation method, in which a computer executes a process of estimating a point cloud representing an object and represented by at least one planar model for point cloud data to be modeled, removing the estimated point cloud representing the object from the point cloud data to be modeled, and repeating the estimation of the point cloud representing the object and the removal of the point cloud representing the object until a predetermined iteration termination condition is satisfied.

[0011] The disclosed technology makes it possible to estimate a detailed shape model representing multiple objects from a huge amount of point cloud data.

[0012] FIG. 1 is a diagram showing an example of a detection result of a first object. FIG. 2 is a diagram showing an example of a detection result of a second object. FIG. 3 is a block diagram showing the hardware configuration of an estimation device. FIG. 4 is a block diagram showing the configuration of an estimation device of this embodiment. FIG. 5 is a block diagram showing the configuration of an estimation device of this embodiment. FIG. 6 is a flowchart showing the flow of estimation processing by the estimation device. FIG. 7 is a subroutine showing the flow of processing by the estimation unit to detect an object. FIG. 8 is a subroutine of hierarchical RANSAC executed by the hierarchical estimation unit.

[0013] An example of an embodiment of the disclosed technology will be described below with reference to the drawings. Note that the same or equivalent components and parts in each drawing are given the same reference numerals. Also, the dimensional proportions in the drawings are exaggerated for the convenience of explanation and may differ from the actual proportions.

[0014] First, an overview of the embodiments of the present disclosure and underlying technologies will be described.

[0015] We consider the automatic detection of road surfaces from point cloud data of a city. In the point cloud of an entire city, buildings and other structures have uneven heights, while road surfaces can be considered as gently undulating curved surfaces. Multiple plane models are used to approximate these curved surfaces. An outline of this detection algorithm is given below.

[0016] [Outline of detection algorithm] The entire point cloud data is represented by Q. Q is arranged in an octree structure, and the voxel set at layer n is denoted by {V ｎ The object to be subjected to RANSAC is represented by {W ｎ}, and in the initial stage, ｎ}={V ｎ}. The plane model of layer n is P ｎ The detection algorithm consists of the following two routines:

[0017] (Routine A): Object {W ｎ}, and then RANSAC is performed to obtain the plane model P ｎ =RANSAC({W ｎ}).

[0018] (Routine B): P ｎ A voxel map (VoxelMap) to be described later is constructed using the above, and {W ｎ} into multiple regions. From layer n+1 onwards, routine A is executed for each divided region.

[0019] Next, the details of this detection algorithm will be described below.

[0020] First, the hierarchical RANSAC algorithm is explained as follows.

[0021] [Hierarchical RANSAC Algorithm] Let the layer of the octree structure be n, and let the voxels of this layer be a set {V ｎ The n+1th layer is a higher resolution layer, and is represented by {V ｎ＋１} is {V ｎ} is obtained by dividing it into eight layers. ｎ＋１}=D({V ｎ}) The voxel group {V ｎ}, and the plane model estimation is performed using RANSAC. ｎ The plane model estimation is expressed by the following equation: P ｎ =RANSAC({V ｎ )

[0022] plane P ｎ is the plane for level n of the octree structure. ｎ The voxels that intersect with are {C ｎ}, and the process of finding the intersection is expressed by the following equation (1-1): ｎ is the plane P ｎ and the point cloud W to be estimated ｎ The voxels of the intersection of point cloud W ｎ＋１ is the point cloud estimated for layer n+1. The processing flow of the hierarchical RANSAC algorithm is as follows.

[0023] Step A1: Let the initial layer be n, and let the point group to be estimated be {W ｎ}={V ｎ Step A2: P ｎ =RANSAC({W ｎ}) Step A3: {C ｎ} is calculated using the following formula (1-1). The right side is the plane P ｎ and point cloud W ｎ Represents an operation to find the intersection with ... (1-1) Step A4: {C ｎ＋１}=D({C ｎ}) Step A5: {W ｎ＋１} is calculated using the following formula (1-2). ... (1-2) Step A6: Update the layer from n to n+1, and repeat the process from step A2 a specified number of times. The specified number of times is specified according to the required accuracy of the required number of samples. In the case of a planar model, the required number of samples is "3", etc.

[0024] In addition, D({C ｎ}) corresponds to the process of obtaining a group of voxels one layer deeper according to the definition of the octree. ｎ is divided into eight, and the presence or absence of point coordinates is determined for the voxels in layer n+1. ｎ＋１} is obtained. ｎ＋１ is C ｎ It is a set of partial voxels of layer n+1 obtained by dividing the voxels of layer n into eight, and is composed of a voxel group made up of voxels that are determined to have point coordinates.

[0025] Since the octree hierarchy corresponds to the spatial quantization of a 3D point cloud, the hierarchical RANSAC algorithm significantly reduces the amount of RANSAC calculations by gradually sampling a large point cloud from a coarse granularity to a finer granularity, and the desired planar model can be obtained quickly. However, since the model obtained in this way is merely a simple estimation result, a procedure for obtaining a more detailed shape is required.

[0026] [Issues and Considerations of Hierarchical RANSAC] As the resolution of actual road surface points increases, they are not always distributed on a single plane due to undulations. However, in hierarchical RANSAC, the point cloud to be estimated {W ｎ}The whole is one plane model P ｎ Therefore, we considered extending the detection algorithm so that it could also handle high-resolution road surface estimation. ｎ} into multiple child sets and perform multiple plane model estimations, it was assumed that the undulations could be approximately represented.

[0027] However, the point group {W ｎ} is an octree structure, the adjacency relationship between voxels is not directly stored.

[0028] Therefore, to divide the voxel array based on spatial adjacency, it is necessary to calculate the distance between voxels in advance. Efficient distance calculation methods for octrees have been studied, including in Reference 1, but they are more complicated and require a larger amount of calculation than accessing adjacent elements in the general raster scan method.

[0029] [Reference 1] Bertram H. Drost and Slobodan Ilic. Almost constant-time 3d nearest-neighbor lookup using implicit octrees. Machine Vision and Applications, Vol. 29, No. 2,pp. 299-311, Feb 2018.

[0030] [Division Estimation Algorithm] The division estimation algorithm of this embodiment will now be described. In the detection algorithm of this embodiment, the point group {W ｎ} is close to a plane, and the center coordinates of the voxel are estimated on the plane P ｎ It is assumed that a two-dimensional map of voxels can be constructed by projecting the image onto the voxel. This is called a voxel map. In a voxel map, pixels and voxels correspond one-to-one, and the adjacency relationship between pixels can be expressed as the adjacency relationship between voxels. In other words, the point group {W ｎ} can be easily decomposed.

[0031] Point cloud {W ｎ} voxel map M ｎ And M ｎ The process of generating M is expressed by the following equation: ｎ = p ({W ｎ}, P ｎ )

[0032] M ｎ The set of partial images divided into {r ｎｉ}, and the image division process is expressed by the following formula: ｎｉ}=s(M ｎ )

[0033] i is the division unit, and the partial image set r ｎｉThe set of voxels corresponding to ｎｉ Let the point group {W ｎ} is a set of spatially divided sets {w ｎｉ The process of obtaining {w ｎｉ}=f({r ｎｉ}) = f(s(M ｎ ))

[0034] The hierarchical RANSAC algorithm using voxel maps is extended as follows.

[0035] Step B1: Let the initial layer be n, and let the point group to be estimated be {W ｎ}={V ｎ Step B2: P ｎ =RANSAC({W ｎ}) Step B3: M ｎ = p ({W ｎ}, P ｎ ) Step B4: {w ｎｉ}=f(s(Mn)) Step B5: Set w ｎｉ For all elements of the symbol n, the hierarchical RANSAC of steps A2 to A6 is performed. ｉ corresponds to n in steps A2 to A6.

[0036] In addition, steps A2 to A6 executed in step B5 are performed using the point cloud W ｎ Set group w ｎｉ By step B5, each w ｎｉ Layer estimation is performed independently for each layer, and a plane is estimated for each layer.

[0037] Although the voxel map has been described above using a planar model as an example, it can also be applied to other three-dimensional models. For example, in the case of a cylindrical model, if a voxel map is constructed on a two-dimensional plane obtained by expanding the cylindrical model, spatial division of a three-dimensional point cloud can be similarly performed. For other models, the above algorithm can be applied by following the texture mapping method used in the CG field and performing planar expansion.

[0038] In the above detection algorithm, the detection algorithm is executed for a single object and point clouds other than that object are excluded. However, in real-world scenes, the detection of multiple objects is required. For example, when distinguishing between roads and sidewalks, or when determining passable areas based on steps, it is necessary to expand the detection target to multiple objects.

[0039] <Multiple Object Estimation Algorithm> In this embodiment, as an initial process, the above-described detection algorithm is first executed for Q, and the set of points included in the detected model is represented by P. Due to the nature of RANSAC, P can be said to occupy an area with high distribution density. This is the first object, and is represented by k=1. Q ０ = Q, P ０ = P. By repeating the following process, multiple objects can be detected, provided that an upper limit for k is given.

[0040] Step C1: Q ｋ =Q ｋ－１ -P ｋ－１ P ｋ－１ Step C2: Calculate the points not included in Q ｋ , run the detection algorithm to detect new objects, and let k=k+1.

[0041] Examples of detection results are shown in Figures 1 and 2. Figure 1 shows an example in which a ceiling surface is detected as the first object, while Figure 2 shows an example in which a floor surface is detected as the second object.

[0042] The configuration and operation of this embodiment will be described below.

[0043] Fig. 3 is a block diagram showing the hardware configuration of the estimating device 100. As shown in Fig. 3, the estimating device 100 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input unit 15, a display unit 16, and a communication interface (I / F) 17. Each component is connected to each other via a bus 19 so as to be able to communicate with each other.

[0044] The CPU 11 is a central processing unit that executes various programs and controls each part. That is, the CPU 11 reads programs from the ROM 12 or the storage 14 and executes the programs using the RAM 13 as a work area. The CPU 11 controls the above components and performs various arithmetic processing in accordance with the programs stored in the ROM 12 or the storage 14. In this embodiment, an estimation program is stored in the ROM 12 or the storage 14.

[0045] The ROM 12 stores various programs and various data. The RAM 13 temporarily stores programs or data as a working area. The storage 14 is configured by a storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and stores various programs including an operating system and various data.

[0046] The input unit 15 includes a pointing device such as a mouse and a keyboard, and is used to input various types of information.

[0047] The display unit 16 is, for example, a liquid crystal display, and displays various information. The display unit 16 may be a touch panel type and function as the input unit 15.

[0048] The communication interface 17 is an interface for communicating with other devices such as terminals, etc. For this communication, for example, a wired communication standard such as Ethernet (registered trademark) or FDDI, or a wireless communication standard such as 4G, 5G, or Wi-Fi (registered trademark) is used.

[0049] Next, a description will be given of each functional configuration of the estimation device 100. Fig. 4 is a block diagram showing the configuration of the estimation device of this embodiment. Each functional configuration is realized by the CPU 11 reading out an estimation program stored in the ROM 12 or the storage 14, expanding it in the RAM 13, and executing it.

[0050] As shown in FIG. 4, the estimation device 100 includes a point cloud data storage unit 102 , an estimation unit 110 , a removal unit 112 , and a repetition determination unit 114 .

[0051] The point cloud data storage unit 102 stores point cloud data to be modeled.

[0052] The estimation unit 110 estimates a point cloud that represents an object and is expressed by at least one planar model from the point cloud data to be modeled.

[0053] The removal unit 112 removes the point cloud representing the object estimated by the estimation unit 110 from the point cloud data to be modeled.

[0054] The repetition determining unit 114 causes the estimation unit 110 to repeat the estimation and the removal unit 112 to repeat the removal until a predetermined repetition end condition is met.

[0055] As shown in FIG. 5, the estimation unit 110 includes a preprocessing unit 120 , a voxel map generating unit 122 , a map dividing unit 124 , and a layer estimation unit 126 .

[0056] The preprocessing unit 120 sets an initial layer n of the octree structure for the point cloud data to be estimated in the point cloud data storage unit 102, and calculates a set V of voxels in the layer. ｎ From the point cloud W to be estimated for the layer ｎ (corresponding to step B1). The preprocessing unit 120 estimates the plane P ｎ is calculated as an estimation result (corresponding to step B2).

[0057] The voxel map generator 122 generates a point cloud W ｎ The center coordinates of the plane P ｎ By projecting onto the voxel map M ｎ (corresponding to step B3).

[0058] The map division unit 124 divides the generated voxel map M ｎ The partial image set r divided into ｎｉ From the point cloud W to be estimated ｎ The set w obtained by dividing the space ｎｉ (corresponding to step B4). ｎｉ is the point group Wｎ is represented as an element of the partitioned set.

[0059] The layer estimation unit 126 estimates the spatially divided set w ｎｉ For each element of the division unit (set group w ｎｉ The layer estimation unit 126 performs hierarchical RANSAC on each element (each i in the element) (corresponding to step B5). As will be described below, the layer estimation unit 126 performs hierarchical RANSAC on each element (each i in the element) (corresponding to step B6). ｎ By dividing the image into layers, a point cloud at a deeper level is estimated.

[0060] The layer estimation unit 126 estimates a point cloud at layer n+1, which is one layer deeper than layer n, by using a set of points w ｎｉ For all elements of the element, the following hierarchical RANSAC is performed for each element in the division unit. ｎ By estimating the plane model for the plane P ｎ The layer estimation unit 126 updates the plane P ｎ and point cloud W ｎ Voxel group C of the intersection set with ｎ The layer estimation unit 126 creates the created voxel C ｎ is divided, and the presence or absence of point coordinates is determined for the divided area of ​​layer n+1, which is one layer deeper than layer n, to obtain a voxel group C consisting of voxels of layer n+1. ｎ＋１ The layer estimation unit 126 also estimates the plane P ｎ and the voxel group C of layer n+1 ｎ＋１ The intersection set of level n+1 points W ｎ＋１ The layer estimation unit 126 updates the layers and estimates the plane P ｎ Update of voxel group C ｎ Creation of layer n+1 voxel group C ｎ＋１ and the point cloud W ｎ＋１ Repeat each process of estimating.

[0061] Next, the operation of the estimation device 100 will be described.

[0062] 6 is a flowchart showing the flow of estimation processing by the estimation device 100. The estimation processing is performed by the CPU 11 reading out an estimation program from the ROM 12 or the storage 14, expanding it into the RAM 13, and executing it. The estimation processing is an example of an estimation method. The estimation device 100 receives point cloud data of an estimation target from the point cloud data storage unit 102 as input, and performs the following processing.

[0063] In step S90, the CPU 11, as the estimation unit 110, calculates a point cloud P that represents an object and is expressed by at least one planar model from the point cloud data to be modeled. ｋ Here, the point group P ｋ is the plane model P of the deepest layer n ｎ For example, the point group P ｋ is at least one plane model P ｎ is a set of points whose distance from each of the points is within a threshold.

[0064] In step S92, the CPU 11, functioning as the removal unit 112, removes the point cloud P representing the object estimated in step S90 from the point cloud data to be modeled. ｋ Remove.

[0065] In step S94, the CPU 11, functioning as the iteration determination unit 114, determines whether a predetermined iteration end condition is satisfied. For example, if the number of iterations k has reached its upper limit, it determines that the iteration end condition is satisfied, and the process proceeds to step S96. On the other hand, if the number of iterations k has not reached its upper limit, it determines that the iteration end condition is not satisfied, and the process returns to step S90.

[0066] In step S96, the CPU 11 outputs the estimation result to the display unit 16 or the like as a shape model of a plurality of point clouds.

[0067] The above step S90 is realized by a subroutine shown in FIG.

[0068] In step S100, the CPU 11, as the pre-processing unit 120, sets an initial layer n of the octree structure for the point cloud data to be estimated, and calculates a set V of voxels in the layer. ｎ From the point cloud W to be estimated for the layer ｎ Set.

[0069] In step S102, the CPU 11, as the pre-processing unit 120, calculates the point cloud W ｎ By estimating the plane model for ｎ is obtained as the estimated result.

[0070] In step S104, the CPU 11, functioning as the voxel map generating unit 122, generates the point cloud W ｎ The center coordinates of the plane P ｎ By projecting onto the voxel map M ｎ Generate.

[0071] In step S106, the CPU 11, functioning as the map dividing unit 124, divides the generated voxel map M ｎ The partial image set r divided into ｎｉ From the point cloud W to be estimated ｎ The set w is a spatial division of ｎｉ Ask for.

[0072] In step S108, the CPU 11, functioning as the layer estimation unit 126, calculates the spatially divided set group w ｎｉ For the set w ｎｉ Execute hierarchical RANSAC for all elements of . The hierarchical RANSAC will be described as a subroutine.

[0073] In step S110, the CPU 11 functions as the layer estimation unit 126 and outputs the estimation result.

[0074] 8 shows a subroutine of hierarchical RANSAC executed by the layer estimation unit 126 in step S108. The CPU 11, as the layer estimation unit 126, calculates a set of w ｎｉ The following process is performed for each element of all elements of ｉ corresponds to n in the following subroutine.

[0075] In step S200, the CPU 11 calculates the point cloud W ｎ The plane model estimation for the hierarchical plane P ｎ Then, the process returns to step S200 and repeats the process.

[0076] In step S202, the CPU 11 calculates the plane P ｎ and point cloud W ｎ Voxel group C of the intersection set with ｎ Create a.

[0077] In step S204, the created voxel group C ｎ is divided, and the presence or absence of point coordinates is determined for the divided area of ​​layer n+1, which is one layer deeper than layer n, to obtain a voxel group C consisting of voxels of layer n+1. ｎ＋１ Estimate.

[0078] In step S206, the CPU 11 calculates the plane P ｎ and the voxel group C of layer n+1 ｎ＋１ The intersection set of level n+1 points W ｎ＋１ Estimate.

[0079] In step S208, the CPU 11 determines whether the depth of the hierarchy satisfies a predetermined condition (a specified number of times). If the condition is met, the subroutine ends, and if the condition is not met, the process proceeds to step S210.

[0080] In step S210, the CPU 11 updates the hierarchy from n to n+1.

[0081] As described above, the estimation device 100 of this embodiment can estimate detailed shape models representing multiple objects at high speed from a huge amount of point cloud data.

[0082] In the above embodiment, the estimation process executed by the CPU by reading the software (program) may be executed by various processors other than the CPU. Examples of processors in this case include a PLD (Programmable Logic Device), a GPU (Graphics Processing Unit), and an ASIC (Application Specific Integrated Circuit), which are processors having a circuit configuration designed specifically for executing specific processes, such as a field-programmable gate array (FPGA), whose circuit configuration can be changed after manufacture. The estimation process may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (for example, multiple FPGAs, a combination of a CPU and an FPGA, etc.). Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit that combines circuit elements such as semiconductor elements.

[0083] In the above embodiment, the estimation program is pre-stored (installed) in the storage 14, but the present invention is not limited to this. The program may be provided in a form stored in a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The program may also be downloaded from an external device via a network.

[0084] The following additional notes are provided regarding the above-described embodiments.

[0085] (Supplementary Item 1) An estimation device including: a memory; and at least one processor connected to the memory, wherein the processor is configured to: estimate a point cloud that represents an object and is represented by at least one planar model for point cloud data to be modeled; remove the estimated point cloud that represents the object from the point cloud data to be modeled; and repeat the estimation of the point cloud that represents the object and the removal of the point cloud that represents the object until a predetermined iteration termination condition is satisfied.

[0086] (Supplementary Item 2) A non-transitory storage medium storing a program executable by a computer to execute an estimation process, the non-transitory storage medium comprising: for point cloud data to be modeled, estimating a point cloud that represents an object and that is represented by at least one planar model; removing the estimated point cloud that represents the object from the point cloud data to be modeled; and repeating the estimation of the point cloud that represents the object and the removal of the point cloud that represents the object until a predetermined iteration termination condition is satisfied.

[0087] REFERENCE SIGNS LIST 11 CPU 14 Storage 100 Estimation device 102 Point cloud data storage unit 110 Estimation unit 112 Removal unit 114 Iteration determination unit 120 Preprocessing unit 122 Voxel map generation unit 124 Map division unit 126 Hierarchical estimation unit

Claims

1. An estimation device including: an estimation unit that estimates a point cloud that represents an object from point cloud data to be modeled, the point cloud being represented by at least one planar model; a removal unit that removes the point cloud that represents the object estimated by the estimation unit from the point cloud data to be modeled; and an iteration determination unit that causes the estimation by the estimation unit and the removal by the removal unit to be repeated until a predetermined iteration termination condition is satisfied.

2. The estimation device according to claim 1, wherein the estimation unit includes: a pre-processing unit that sets an initial hierarchy of an octree structure for point cloud data to be modeled, sets a point cloud to be estimated for that hierarchy from a set of voxels in that hierarchy, and obtains a plane of that hierarchy as an estimation result by plane model estimation for the point cloud to be estimated; a voxel map generation unit that generates a voxel map by projecting the center coordinates of the point cloud to be estimated onto the plane of that hierarchy; a map division unit that obtains a set group obtained by spatially dividing the point cloud to be estimated from a set of partial images divided for the generated voxel map; and a hierarchical estimation unit that estimates a point cloud of a hierarchy that deepens the hierarchy by dividing the voxels in the element for each division unit for the spatially divided set group.

3. An estimation method performed by a computer, comprising the steps of: estimating a point cloud that represents an object and is represented by at least one planar model for point cloud data to be modeled; removing the estimated point cloud that represents the object from the point cloud data to be modeled; and repeating the estimation of the point cloud that represents the object and the removal of the point cloud that represents the object until a predetermined iteration termination condition is satisfied.

4. An estimation program for causing a computer to function as each part of the estimation device according to claim 1 or 2.

Images