Information processing device and distance measuring system

Abstract

This allows for labor-saving and highly accurate conversion from the device coordinate system to the world coordinate system. [Solution] The information processing device includes: a first extraction unit that extracts three-dimensional feature point cloud data from three-dimensional point cloud data in a first coordinate system; a second extraction unit that extracts three or more specific points by matching the three-dimensional feature point cloud data with a three-dimensional model; and an alignment unit that aligns the three or more specific points extracted by the second extraction unit in the first coordinate system with three or more specific points in the second coordinate system.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus and a distance measurement system.

Background Art

[0002] Three-dimensional point cloud data acquired using a distance measurement device may be converted from data in the device coordinate system to data in a predetermined reference coordinate system (hereinafter also referred to as the world coordinate system).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The present disclosure provides an information processing apparatus and a distance measurement system capable of streamlining and improving the accuracy of the conversion from the device coordinate system to the world coordinate system.

Means for Solving the Problems

[0005] In order to solve the above problems, according to the present disclosure, a first extraction unit that extracts three-dimensional feature point group data from three-dimensional point cloud data in a first coordinate system, a second extraction unit that extracts three or more specific points by matching the three-dimensional feature point group data with a solid model, and an alignment unit that aligns the three or more specific points extracted by the second extraction unit in the first coordinate system with three or more specific points in a second coordinate system. An information processing apparatus is provided. An information processing apparatus is provided.

Brief Description of the Drawings

[0006] [Figure 1] A block diagram showing the configuration of a distance measurement system according to a first embodiment of the present disclosure. [Figure 2] A diagram illustrating the alignment method according to the first embodiment of this disclosure. [Figure 3] A diagram showing an example of the first point cloud data in the device coordinate system. [Figure 4] A figure showing a three-dimensional model according to the first embodiment of this disclosure. [Figure 5] A figure showing an example of second-point cloud data in the world coordinate system. [Figure 6] A flowchart illustrating the operation of the information processing device according to the first embodiment of this disclosure. [Figure 7] A flowchart illustrating the alignment process according to the first embodiment of this disclosure. [Figure 8] This figure shows an example of first-point cloud data subject to mechanical error correction. [Figure 9] A diagram showing an example of the reference second-point cloud data. [Figure 10] A block diagram showing the configuration of a distance measuring system in one modified example. [Figure 11] A block diagram showing the configuration of a distance measuring system according to a second embodiment of the present disclosure. [Figure 12] A figure showing an example of the first image data. [Figure 13] A diagram showing an example of an operation screen displayed to the user. [Figure 14] A flowchart illustrating the operation of the information processing device according to the second embodiment of this disclosure. [Modes for carrying out the invention]

[0007] Embodiments of the present invention will be described below with reference to the drawings. The following description does not exclude any components or functions not shown or described.

[0008] (First Embodiment) Figure 1 is a block diagram showing the configuration of a distance measuring system 1 equipped with an information processing device 10 according to a first embodiment of the present disclosure. The distance measuring system 1 has distance measuring devices 2a and 2b. Distance measuring devices 2a and 2b measure the distance to an object 3. Distance measuring devices 2a and 2b are, for example, LiDAR (Light Detection & Ranging).

[0009] The distance measuring devices 2a and 2b perform distance measurement using, for example, the Time of Flight (ToF) method. The distance measuring devices 2a and 2b emit light L1 towards the object to be measured 3 and receive reflected light L2 that is reflected by the object to be measured 3. Based on the emitted light L1 and the received reflected light L2, the distance measuring devices 2a and 2b can obtain the distance to the object to be measured 3. Note that the distance measuring devices 2a and 2b may receive noise light (ambient light) L3.

[0010] ToF (Time-of-Flight) methods include, for example, dToF (direct ToF) and iToF (indirect ToF). In the dToF method, the distance to the object to be measured 3 can be measured based on the time difference between the emission timing of the emitted light L1 and the reception timing of the reflected light L2. In the iToF method, the distance to the object to be measured 3 can be measured based on the phase difference between the emission phase of the emitted light L1 and the reception phase of the reflected light L2. Either the dToF method or the iToF method may be applied to the distance measuring devices 2a and 2b, or other distance measuring methods may be applied.

[0011] The distance measuring devices 2a and 2b are positioned at different locations from each other. Furthermore, the distance measuring devices 2a and 2b are positioned such that one or more common distance measuring objects 3 are included in each other's field of view (FoV).

[0012] The distance measuring device 2a outputs first point cloud data 20a based on the distance measurement result of the distance measurement object 3. The first point cloud data 20a has a plurality of point data including distance information from the distance measurement object 3 plotted in the device coordinate system (first coordinate system) of the distance measuring device 2a. The first point cloud data 20a is data generated based on the received light signal of the received light including the reflected light L2 received by the distance measuring device 2a, or data designed by simulating the received light signal. Each point data of the first point cloud data 20a represents, for example, the position where the distance measurement object 3 reflects light.

[0013] Similarly, the distance measuring device 2b outputs second point cloud data 20b. The second point cloud data 20b has a plurality of point data including distance information from the distance measurement object 3 plotted in the device coordinate system (second coordinate system) of the distance measuring device 2b. The second point cloud data 20b is data generated based on the received light signal of the received light including the reflected light L2 received by the distance measuring device 2b, or data designed by simulating the received light signal. Also, the second point cloud data 20b may be data generated based on the received light signal of the received light including the reflected light L2 received by the distance measuring device 2a with a different FoV from the first point cloud data 20a, or data designed by simulating the received light signal, as will be described later.

[0014] The device coordinate systems of the distance measuring devices 2a and 2b are, for example, three-dimensional coordinate systems. In this case, the distance measuring devices 2a and 2b can output the first point cloud data 20a and the second point cloud data 20b, which are three-dimensional point cloud data (hereinafter also referred to as 3D point cloud data).

[0015] The information processing device 10 performs alignment of the first point cloud data 20a and the second point cloud data 20b. Hereinafter, in this specification, an example in which the device coordinate system of the distance measuring device 2b is used as the world coordinate system will be described. That is, in this specification, an example of converting the data of the first point cloud data 20a into the data of the device coordinate system (world coordinate system) of the distance measuring device 2b will be described.

[0016] The information processing device 10 includes a 3D point cloud data acquisition unit (acquisition unit) 11, a feature point cloud extraction unit (first extraction unit) 12, a 3D model matching unit 13, a specific point extraction unit (second extraction unit) 14, a transformation matrix estimation unit (generation unit) 15, and a 3D point cloud coordinate transformation unit (alignment unit) 16.

[0017] The 3D point cloud data acquisition unit 11 acquires the first point cloud data 20a and the second point cloud data 20b from the distance measuring devices 2a and 2b.

[0018] The feature point cloud extraction unit 12 extracts three-dimensional feature point cloud data (hereinafter also referred to as 3D feature point cloud data). The three-dimensional feature point cloud data is, for example, a collection of multiple point data that constitute the surface of a characteristic object (distance measurement target object 3) or reference surface (for example, a road surface or floor surface) included in a predetermined distance measurement range.

[0019] The feature point cloud extraction unit 12 extracts one or more first feature point cloud data from a portion of the first point cloud data 20a. Similarly, the feature point cloud extraction unit 12 extracts one or more second feature point cloud data from the second point cloud data 20b. The first feature point cloud data and the second feature point cloud data are the three-dimensional feature point cloud data described above.

[0020] The 3D model matching unit 13 extracts one or more first 3D models by matching one or more first feature point cloud data with a predetermined 3D model. Similarly, the 3D model matching unit 13 extracts one or more second 3D models from one or more second feature point cloud data.

[0021] The specific point extraction unit 14 extracts multiple first specific points and multiple second specific points from one or more first three-dimensional models and one or more second three-dimensional models, respectively. Details of the first and second specific points will be described later.

[0022] The transformation matrix estimation unit 15 generates a transformation matrix from the coordinate information of a plurality of first specific points and the coordinate information of a plurality of second specific points. For example, suppose that the first specific points extracted are first specific point P0=(Px0,Py0,Pz0), first specific point P1=(Px1,Py1,Pz1), and first specific point P2=(Px2,Py2,Pz2). Also, suppose that the second specific points extracted are second specific points Q0=(Qx0,Qy0,Qz0), second specific point Q1=(Qx1,Qy1,Qz1), and second specific point Q2=(Qx2,Qy2,Qz2). A specific point matrix P can be formed using the first specific points P0 to P2. A specific point matrix Q can be formed using the second specific points Q0 to Q2. In this case, the transformation matrix estimation unit 15 can estimate a transformation matrix A that satisfies the following equations (1) and (2).

number

[0023] The above transformation matrix A is, for example, J=(Q-AP) 2 The transformation matrix A can be estimated by solving a least-squares problem that minimizes the given expression. For example, a method using singular value decomposition can be applied to estimate the transformation matrix A. Alternatively, the transformation matrix A can be estimated using a quaternion-based method.

[0024] The above example shows that three first and three second specific points are acquired. Note that four or more first and second specific points may be acquired. Even in this case, the transformation matrix estimation unit 15 can estimate the transformation matrix A in the same way as in equation (1). Furthermore, by acquiring more first and second specific points, the information processing device 10 can improve the estimation accuracy of the transformation matrix A and thus improve the alignment accuracy.

[0025] The 3D point cloud coordinate transformation unit 16 uses the estimated transformation matrix A to perform relative alignment of the 3D point cloud data in the first coordinate system and the 3D point cloud data in the second coordinate system. The 3D point cloud coordinate transformation unit 16 can convert the 3D point cloud data acquired from the distance measuring device 2a into 3D point cloud data in the world coordinate system (in the example herein, the device coordinate system of the distance measuring device 2b). For example, any point data Pn=(Px,Py,Pz) acquired by the distance measuring device 2a can be converted into point data Qn=(Qx,Qy,Qz) in the world coordinate system, as shown in equation (3) below.

number

[0026] The transformation matrix A that can be calculated using equation (1) above is, for example, a rotation matrix. However, it is not limited to this; the transformation matrix A may also be a rigid body transformation matrix. In this case as well, the transformation matrix A can be calculated using the coordinate information of the first specific points P0 to P2 and the second specific points Q0 to Q2, in the same way as in the case of a rotation matrix.

[0027] Equation (2) above describes an example in which the specific point matrices P and Q are composed of the coordinates of specific points. However, the specific point matrices P and Q may also include vectors. For example, the specific point matrix P may include a vector of constant magnitude (e.g., a unit vector) that has information about the orientation from the first specific point P0 to the first specific point P1. The information processing device 10 can use the transformation matrix A calculated from the specific point matrices P and Q, which include vectors, to convert any vector extracted from the 3D point cloud data of the distance measuring device 2a into a vector in the world coordinate system.

[0028] Hereafter, this specification will describe an example in which the distance measuring system 1 has two distance measuring devices (i.e., distance measuring devices 2a and 2b). However, the distance measuring system 1 may be configured to have three or more distance measuring devices. In this case, the distance measuring system 1 can use 3D point cloud data acquired by any one of the distance measuring devices as second point cloud data 20b. This allows 3D point cloud data acquired by other distance measuring devices to be converted into 3D point cloud data in the world coordinate system. In other words, the distance measuring system 1 can align three or more specific points extracted in other coordinate systems to three or more specific points extracted in any one of a plurality of coordinate systems, including the first coordinate system and the second coordinate system.

[0029] Furthermore, the distance measuring system 1 does not necessarily need to be equipped with two distance measuring devices 2a and 2b. The distance measuring system 1 may use 3D point cloud data previously acquired by distance measuring device 2a as the second point cloud data 20b in the world coordinate system. In this case, distance measuring device 2b can be omitted. For example, the mechanical error (such as camera shake) of distance measuring device 2a may be corrected using 3D point cloud data previously acquired by distance measuring device 2a. In addition, if the field of view or position of distance measuring device 2a can be dynamically changed, 3D point cloud data acquired by distance measuring device 2a at a predetermined field of view and position may be used as the second point cloud data 20b in the world coordinate system.

[0030] The information processing device 10 may have a configuration that includes a CPU 17. The operation of each of the functions described above—the 3D point cloud data acquisition unit 11, the feature point cloud extraction unit 12, the 3D model matching unit 13, the specific point extraction unit 14, the transformation matrix estimation unit 15, and the 3D point cloud coordinate transformation unit 16—is performed, for example, by the CPU 17.

[0031] The information processing device 1 may be configured to include a memory 18. The memory 18 can store the above-mentioned 3D point cloud data, 3D feature point cloud data, a first or second 3D model, a first or second specific point, a transformation matrix A, or 3D point cloud data aligned to the 3D point cloud coordinate transformation unit 16. In addition, the memory 18 may store a predetermined 3D model used in the matching process of the 3D model matching unit 13, a threshold for matching with a 3D model, a threshold for extracting 3D feature point cloud data by the feature point cloud extraction unit 12, and the like. The memory 18 may be configured as temporary memory.

[0032] Figure 2 is a diagram illustrating the alignment method according to the first embodiment of this disclosure. Figure 2 shows distance measuring devices 2a and 2b, an object to be measured 3a, a reference surface 4, and a side wall 5.

[0033] The object to be measured 3a is placed on an arbitrary reference plane 4. The side wall 5 has, for example, a surface perpendicular to the reference plane 4. The reference plane 4 is, for example, a floor or the ground. The side wall 5 is, for example, a room wall or a side wall of a building, etc. In the example in Figure 2, the object to be measured 3a is a cylindrical object such as a lighting pole. The shape of the object to be measured 3a is arbitrary.

[0034] The distance measuring device 2a has a device coordinate system Xm-Ym-Zm. The distance measuring device 2b has a device coordinate system (world coordinate system) XYZ. Coordinate axis Xm intersects coordinate axis X, coordinate axis Ym intersects coordinate axis Y, and coordinate axis Zm intersects coordinate axis Z. At least one of coordinate axes Xm, Ym, or Zm may be parallel to the corresponding coordinate axis X, Y, or Z.

[0035] The distance measuring device 2b is positioned, for example, so that the coordinate axis Z is aligned with the direction normal to the reference plane 4. Furthermore, both the distance measuring devices 2a and 2b are positioned so that the object to be measured 3a, the reference plane 4, and the side wall 5 are included in the field of view (FoV).

[0036] The information processing device 10 extracts, for example, specific points 6a, 6b, and 6c from the object to be measured 3a. Specific point 6a is, for example, the intersection point of the central axis of the object to be measured 3a and the reference plane 4. Specific point 6b is a point located at a distance from specific point 6a in the normal direction V of the reference plane 4. Specific point 6b is, for example, the endpoint of a unit normal vector Ve originating from specific point 6a. Specific point 6c is a point located at a distance from specific point 6a in the normal direction H of the side wall 5 (i.e., the direction horizontal to the reference plane 4). Specific point 6c is, for example, the endpoint of a unit normal vector He originating from specific point 6a.

[0037] The information processing device 10 may extract other specific points from the object to be measured 3a. For example, specific points may be extracted from the vertices or centroid of the object to be measured 3a. Alternatively, the information processing device 10 may extract a normal vector or the like that enables the extraction of a new specific point by adding it to other specific points of the object to be measured 3a.

[0038] The distance measuring devices 2a and 2b can estimate the position coordinates of the extracted specific points (specific points 6a to 6c in the example in Figure 2). For example, distance measuring device 2a can obtain the coordinates of the first specific points P0 to P2 from the coordinates of specific points 6a to 6c as seen from distance measuring device 2a. Similarly, distance measuring device 2b can obtain the coordinates of the second specific points Q0 to Q2 from the coordinates of specific points 6a to 6c as seen from distance measuring device 2b.

[0039] Figure 3 shows an example of the first point cloud data 20a. Figure 3 also shows the first feature point cloud data 21a, 22a, 23a, and 25a.

[0040] As described above, the distance measuring device 2a has a device coordinate system Xm-Ym-Zm. In addition, each point data of the first point cloud data 20a is plotted on the device coordinate system Xm-Ym-Zm.

[0041] The first feature point cloud data 25a in Figure 3 is, for example, point cloud data of a reference plane. The reference plane is, for example, normal to the Z-axis of the world coordinate system. The first feature point cloud data 21a to 23a are, for example, point cloud data of objects placed on the reference plane.

[0042] Figure 4 shows a three-dimensional model according to the first embodiment of this disclosure. The feature point cloud extraction unit 12 can extract first feature point cloud data 21a to 23a and first feature point cloud data 25a from the first point cloud data 20a in Figure 3. The three-dimensional model matching unit 13 approximates the first feature point cloud data 21a to 23a and first feature point cloud data 25a to a first three-dimensional model (hereinafter also simply referred to as a three-dimensional model) such as a cone, cylinder, or plane. For example, RANSAC is used as the approximation method.

[0043] For example, the first feature point cloud data 21a to 23a in Figure 3 can be approximated to a cone 30, as shown in Figure 4. From the approximated cone 30, the vertices 31 and the central axis 32 can be extracted. In other words, the cone 30 contains two pieces of information: information about the position of the vertices 31 and information about the vector of the central axis 32.

[0044] The specific point extraction unit 14 can extract vertex 31 as a first specific point (hereinafter also simply referred to as a specific point). The specific point extraction unit 14 can also extract a point obtained by adding the vector of the central axis 32 to vertex 31, etc., as a specific point. As described above, by extracting at least two specific points, the specific point extraction unit 14 can extract two pieces of information: information about the position of vertex 31 and information about the vector of the central axis 32.

[0045] The specific point extraction unit 14 can approximate the first feature point cloud data 25a to a plane. The three-dimensional model of the plane includes a normal vector to the plane. For example, the specific point extraction unit 14 may extract a specific point by adding a normal vector of a fixed magnitude to the plane (e.g., a unit normal vector) to the vertex 31 in Figure 4. This makes it possible to extract a specific point that includes information about the normal vector.

[0046] Furthermore, from the first feature point cloud data, which approximates a cylinder, a specific point can be extracted by adding the vector of the cylinder's central axis to a specific point extracted from another distance-measuring object. This allows for the extraction of a specific point that contains information about the cylinder's central axis vector.

[0047] In addition to the above, the first feature point cloud data may be approximated to three-dimensional models such as polygonal pyramids, polygonal prisms, frustums of cones, frustums of polygons, spheres, polyhedra, or hemispheres. For example, from a polygonal pyramid, at least two specific points containing information about the vertices and central axis vectors can be extracted, similar to the case of a cone. From a polygonal prism, frustum of a cone, and frustums of polygons, at least one specific point containing information about the central axis vector can be extracted, similar to the case of a cylinder. From a sphere and a polyhedron, for example, at least one specific point containing information about the centroid can be extracted. From a hemisphere, for example, at least one specific point containing information about the center can be extracted. Note that the specific points may include not only uniquely identified locations in three-dimensional space, but also uniquely identified vectors in three-dimensional space (for example, axis vectors or normal vectors of the three-dimensional model).

[0048] In addition, the 3D model matching unit 13 may match the first feature point cloud data to the shape of a predetermined member or other object (for example, a gear or a screw). The specific point extraction unit 14 may extract one or more specific points according to the shape of the member or other object. The shape of the member or other object and the position information of the specific points extracted from the member or other object may be stored in an internal or external storage device (for example, memory 18) of the information processing device 10.

[0049] The specific point extraction unit 14 may extract specific points from the point data included in the first point cloud data 20a. Alternatively, the specific point extraction unit 14 may extract specific points that are not included in the point data within the first point cloud data 20a. In other words, the specific point extraction unit 14 may extract virtual points estimated from a three-dimensional model (for example, vertices of the three-dimensional model, or points on the central axis, etc.) as specific points.

[0050] Figure 5 shows an example of the second point cloud data 20b. The second point cloud data 20b has second feature point cloud data 21b, 22b, 23b, and 25b, which correspond to the first feature point cloud data 21a, 22a, 23a, and 25a in Figure 3, respectively. The 3D model matching unit 13 approximates the second feature point cloud data 21b to 23b to a second 3D model such as a cone, cylinder, frustum of a cone, polygonal pyramid, polygonal prism, frustum of a polygon, sphere, polyhedron, hemisphere, or plane. The specific point extraction unit 14 extracts a second specific point from the second 3D model that corresponds to the first specific point, similar to the first 3D model.

[0051] If the first and second specified points contain information about the central axis vector of the three-dimensional model (object), the 3D point cloud coordinate transformation unit 16 can align the first point cloud data 20a and the second point cloud data 20b using the transformation matrix A so that the inclination of the central axis of the object is approximately the same. Similarly, if the first and second specified points contain information about the normal vector to the centroid, vertex, or reference plane of the three-dimensional model, it becomes possible to align the 3D point cloud data so that the inclination of the normal vector to the centroid position, vertex position, or reference plane of the object is approximately the same.

[0052] Figure 6 is a flowchart illustrating the operation of the information processing device 10 according to the first embodiment of this disclosure. Below, an example of extracting a first specific point from the first point cloud data 20a will be described. The method for extracting a second specific point from the second point cloud data 20b is the same as the example described below.

[0053] First, the 3D point cloud data acquisition unit 11 acquires 3D point cloud data (step S1).

[0054] Next, as shown in Figure 3, the feature point cloud extraction unit 12 searches the first point cloud data 20a for first feature point cloud data to be matched to the first 3D model. If the feature point cloud extraction unit 12 finds first feature point cloud data, it extracts it (step S2).

[0055] As shown in Figure 4, the 3D model matching unit 13 approximates the first feature point cloud data extracted in step S2 to the first 3D model (3D model matching). This allows the 3D model matching unit 13 to extract the first 3D model (step S3).

[0056] As shown in Figure 4, the specific point extraction unit 14 extracts one or more first specific points from the first three-dimensional model generated in step S3 (step S4).

[0057] Next, the information processing device 10 determines whether the extraction of the first feature point cloud data has been completed (step S5). In step S5, for example, the feature point cloud extraction unit 12 determines whether there is any first feature point cloud data in the first point cloud data 20a that has not yet been extracted. If there is any first feature point cloud data that has not been extracted, the feature point cloud extraction unit 12 may extract the first feature point cloud data in step S2. The information processing device 10 may also determine whether a sufficient number of first specific points have been extracted at the time of step S5. If a sufficient number of first specific points have been extracted, the information processing device 10 does not need to extract any new first feature point cloud data.

[0058] After performing the processes in steps S2 to S5 once or more times, the information processing device 10 determines whether or not it was able to obtain three or more first specific points (step S6). If three or more first specific points were obtained, it may be determined that the extraction of the first specific points was successful (step S7).

[0059] If three or more first specific points cannot be obtained, it may be determined that the extraction of the first specific points has failed (step S8). In this case, the first point cloud data 20a may be reacquired by changing the field of view of the distance measuring device 2a, or the extraction process or extraction threshold of the first feature point cloud data in step S2 may be changed. Alternatively, the processing or threshold when approximating to the first three-dimensional model in step S3 may be changed.

[0060] The threshold for the determination in step S6 may be adjusted according to the number of unknowns in the transformation matrix A. For example, if the transformation matrix A is a rigid body transformation matrix, step S6 may determine whether or not four or more first specific points have been obtained. Furthermore, the threshold for the determination in step S6 may be arbitrarily adjusted based on the required alignment accuracy. That is, if four or more first specific points cannot be obtained, or if five or more cannot be obtained, the information processing device 10 may determine that the extraction of the first specific points has failed.

[0061] Figure 7 is a flowchart illustrating the alignment process according to the first embodiment of this disclosure. First, the information processing device 10 acquires a specific point in the device coordinate system (i.e., a first specific point) using the method described in Figure 6 (step S11). Similarly, the information processing device 10 acquires a specific point in the world coordinate system (i.e., a second specific point) (step S12).

[0062] The transformation matrix estimation unit 15 estimates the transformation matrix A based on the first and second specific points obtained in steps S11 and S12 (step S13). Specifically, the transformation matrix A can be estimated by solving equation (1) above using a method such as singular value decomposition.

[0063] The 3D point cloud coordinate transformation unit 16 can transform the coordinates of the 3D point cloud data output by the distance measuring device 2a using the transformation matrix A estimated in step S13 (step S14). Specifically, it can convert the 3D point cloud data of the distance measuring device 2a into 3D point cloud data in the world coordinate system (i.e., align it) using equation (3) or the like described above.

[0064] The above alignment makes it possible to combine, combine, or compare the 3D point cloud data from distance measuring devices 2a and 2b. Alternatively, 3D point cloud data from multiple distance measuring devices converted to the world coordinate system may be combined or otherwise processed.

[0065] The distance measuring system 1 can be used for various purposes. For example, the distance measuring system 1 may be used to correct mechanical errors in the distance measuring device 2a (e.g., image stabilization). That is, the mechanical position error of the first point cloud data 20a may be corrected using the second point cloud data 20b previously acquired by the distance measuring device 2a. The above-mentioned second point cloud data 20b previously acquired may be stored in memory 18, for example.

[0066] Figure 8 shows an example of the first point cloud data 20c that is subject to mechanical error correction. Figure 9 shows an example of the second point cloud data 20d which serves as a reference. The first point cloud data 20c is, for example, 3D point cloud data acquired by the distance measuring device 2a. The second point cloud data 20d is, for example, 3D point cloud data acquired by the distance measuring device 2a before the first point cloud data 20c. The distance measuring device 2a has a shift in FoV between the time of acquisition of the first point cloud data 20c and the time of acquisition of the second point cloud data 20d.

[0067] From the first point cloud data 20c in Figure 8, the first feature point cloud data 41a, 42a, and 43a can be extracted, similar to the first point cloud data 20a in Figure 3. From the second point cloud data 20d in Figure 9, the second feature point cloud data 41b, 42b, and 43b, corresponding to the first feature point cloud data 41a, 42a, and 43a, can be extracted. This allows for correction of the mechanical error in the first point cloud data 20c using the methods shown in Figures 6 and 7.

[0068] As a comparative example, let's consider an example of correcting the mechanical error of the first point cloud data 20c using ICP (Iterative Closest Point). First, as the first comparative example, let's consider the case where ICP is applied to the entire point cloud of both the first point cloud data 20c and the second point cloud data 20d. In this case, the nearest neighbor transformation matrix is ​​searched for between almost all point data in the first point cloud data 20c and almost all point data in the second point cloud data 20d. Therefore, the computational complexity is large in the first comparative example.

[0069] Furthermore, in the method of the first comparative example, the boundary points of the FoV of the distance measuring device 2a may affect the accuracy of the alignment. Specifically, Figure 8 shows the boundary points 46a, 47a, and 48a of the FoV. Figure 9 also shows the boundary points 46b, 47b, and 48b corresponding to boundary points 46a, 47a, and 48a. In the method of the first comparative example, weighting is applied to the alignment of boundary points 46a~48a and boundary points 46b~48b, which may reduce the weighting of the alignment between the first feature point cloud data 41a~43a and the second feature point cloud data 41b~43b. As a result, the accuracy of the alignment between the first feature point cloud data 41a~43a and the second feature point cloud data 41b~43b may decrease.

[0070] Compared to the first comparative example, the methods in Figures 6 and 7 only require searching for the transformation matrix A that minimizes the difference between the extracted first and second specific points. Therefore, the methods in Figures 6 and 7 can reduce the computational complexity compared to the first comparative example.

[0071] Furthermore, in the methods shown in Figures 6 and 7, the boundary points 46a-48a and 46b-48b do not affect the calculation of the transformation matrix A. Therefore, the methods shown in Figures 6 and 7 can align the first feature point cloud data 41a-43a and the second feature point cloud data 41b-43b with higher accuracy than the first comparative example.

[0072] As a second comparative example, we consider a method of applying ICP to pre-extracted first feature point cloud data 41a-43a and second feature point cloud data 41b-43b (hereinafter collectively referred to as feature point cloud data). In the method of the second comparative example, the accuracy of ICP may decrease if the point data within the feature point cloud data is sparse.

[0073] Furthermore, in feature point cloud data, the point data on the face of the object facing the distance measuring device (hereinafter also referred to as the front) may be dense. On the other hand, the point data on the face opposite the front of the object (hereinafter also referred to as the back) may be sparse. In ICP, if the point data on the back is sparse, the accuracy of object alignment may decrease.

[0074] In contrast to the second comparative example, the methods shown in Figures 6 and 7 allow for the application of a more accurate method (e.g., RANSAC) than ICP for matching feature point cloud data to a 3D model. This enables high-accuracy matching of feature point cloud data to a 3D model, even when the point data in the feature point cloud is sparse, or when the point data on the back of an object is sparse.

[0075] Furthermore, the methods shown in Figures 6 and 7 acquire specific points from the central axis of the 3D model. Even if information about the shape of a part of the object (for example, the shape of the back) is missing from the feature point cloud data, the central axis can sometimes be estimated from the shape of the front of the object. For this reason, the methods in Figures 6 and 7 can acquire specific points with high accuracy, meaning that alignment can be performed with higher accuracy than in the second comparative example.

[0076] As described above, the methods shown in Figures 6 and 7 can perform alignment with higher accuracy and lower computational complexity than the ICP-based method described in one comparative example.

[0077] Figure 10 is a block diagram showing the configuration of a distance measuring system 1a according to one modified example. The distance measuring system 1a in Figure 10 includes edge devices 51 and 52 and a server 53. Edge device 51 has a displacement correction unit 10a. Edge device 52 has a displacement correction unit 10b. Server 53 has an alignment unit 10c. The displacement correction units 10a, 10b, and the alignment unit 10c have the same configuration as the information processing device 10 in Figure 1.

[0078] The misalignment correction unit 10a corrects the mechanical error of the 3D point cloud data output by the distance measuring device 2a. Specifically, the misalignment correction unit 10a corrects the first point cloud data 20a acquired by the distance measuring device 2a using the second point cloud data 20b previously acquired by the distance measuring device 2a. The misalignment correction unit 10a may mechanically drive the distance measuring device 2a based on the estimated transformation matrix A to correct its position or FoV. Similarly, the misalignment correction unit 10b corrects the mechanical position error of the distance measuring device 2b. The misalignment correction units 10a and 10b may, for example, correct the mechanical error of the 3D point cloud data each time the distance measuring devices 2a and 2b output 3D point cloud data.

[0079] The alignment unit 10c aligns multiple distance measuring devices, including distance measuring devices 2a and 2b. The alignment unit 10c uses, for example, the device coordinate system of distance measuring device 2b as the world coordinate system. This allows multiple distance measuring devices, such as distance measuring device 2a, to be aligned with the world coordinate system. The alignment unit 10c may align the multiple distance measuring devices when the distance measuring system 1a is initialized. Alternatively, it may align the multiple distance measuring devices at regular intervals.

[0080] The alignment unit 10c may acquire the first point cloud data 20a and the second point cloud data 20b from the edge devices 51 and 52, respectively, or from the distance measuring devices 2a and 2b, respectively.

[0081] The misalignment correction unit 10a may acquire or store the transformation matrix A estimated by the alignment unit 10c. The misalignment correction unit 10a may use the transformation matrix A estimated by the alignment unit 10c to align multiple first point cloud data 20a acquired by the distance measuring device 2a (for example, at regular intervals) to the world coordinate system.

[0082] Several variations of the distance measuring system 1 are possible. For example, the distance measuring system 1 may be configured to include a storage device (e.g., memory 18) that stores the second point cloud data 20b previously acquired by the distance measuring device 2a, or a communication device that acquires the second point cloud data 20b from an external storage device, etc. These storage devices or communication devices, etc., may be located inside the information processing device 10.

[0083] The second point cloud data 20b may be output by devices other than the distance measuring devices 2a and 2b. For example, the second point cloud data 20b may be 3D point cloud data formed from survey results or design data of known feature points, or 3D point cloud data formed by technologies such as digital twins.

[0084] Furthermore, the second specific point does not necessarily have to be extracted by the specific point extraction unit 14. For example, the specific point extraction unit 14 may acquire the second specific point stored in a predetermined storage device (e.g., memory 18). The second specific point may be a specific point previously acquired by the specific point extraction unit 14, or it may be formed from known survey results or design data. In this case, the process of acquiring the second point cloud data 20b, the second 3D model, and the second feature point cloud data may be omitted. Similarly, the second 3D model or the second feature point cloud data may be data acquired from a predetermined storage device (e.g., memory 18), or data formed from known survey results or design data.

[0085] Thus, the information processing device 10 according to the first embodiment of this disclosure can reduce the effort and improve the accuracy of the conversion from the device coordinate system to the world coordinate system.

[0086] Specifically, the information processing device 10 extracts some feature point cloud data from the 3D point cloud data acquired by the distance measuring device. The information processing device 10 approximates the extracted feature point cloud data to a 3D model and estimates specific points from the vertices or central axes of the 3D model. The information processing device 10 estimates a transformation matrix A from the specific points and uses it to align the distance measuring device.

[0087] The number of point data points for the specific points mentioned above is significantly less than the number of point data points in the 3D point cloud data acquired by the distance measuring device. By searching for the transformation matrix A using the specific points, the information processing device 10 can significantly reduce the computational load compared to a method that iteratively searches for the transformation matrix using the 3D point cloud data.

[0088] Furthermore, the information processing device 10 can estimate specific points from the central axis of the 3D model. The information processing device 10 can estimate specific points with high accuracy without being affected by the density of point data of the 3D model or by missing data such as the back surface shape of the 3D model. In addition, the information processing device 10 is not affected by the boundary points of the FoV. For this reason, the information processing device 10 can perform high-precision alignment.

[0089] (Second embodiment) In step S2 of Figure 6, feature point cloud data is extracted from the 3D point cloud data. Feature point cloud data can be extracted, for example, by determining whether the 3D point cloud data can be comprehensively matched to a 3D model, but this method is computationally intensive. In the second embodiment of this disclosure, a method for extracting feature point cloud data with less computation is described.

[0090] Figure 11 is a block diagram showing the configuration of the distance measuring system 1b according to the second embodiment of this disclosure. The information processing device 10d in Figure 11 includes, in addition to the configuration of Figure 1, a 2D image data acquisition unit 61 and a 2D image feature point cloud extraction unit (third extraction unit) 62.

[0091] In Figure 11, the CPU 17 and memory 18 are not shown. Note that the functions of the 2D image data acquisition unit 61 and the 2D image feature point cloud extraction unit 62 are performed, for example, by the CPU 17 in Figure 1.

[0092] The 2D image data acquisition unit 61 acquires two-dimensional image data (hereinafter also referred to as 2D image data) consisting of first image data 70a corresponding to first point cloud data 20a and second image data 70b corresponding to second point cloud data 20b. The first image data 70a can be acquired, for example, from the distance measuring device 2a. The second image data 70b can be acquired, for example, from the distance measuring device 2b. That is, the first point cloud data 20a and the first image data 70a are the same FoV data and are the same first coordinate system data. Also, the second point cloud data 20b and the second image data 70b are the same FoV data and are the same second coordinate system data. The distance measuring devices 2a and 2b can detect the distance to an object based on at least one of the 3D point cloud data or the 2D image data. The 2D image data may be stored, for example, in the memory 18 in Figure 1.

[0093] The first image data 70a has multiple pixel data corresponding to multiple point data in the first point cloud data 20a. For example, each point data in the first point cloud data 20a corresponds one-to-one with each pixel data in the first image data 70a. For example, the distance measuring device 2a may output multiple pixel signals based on the received reflected light L2. Based on one pixel signal, one point data in the first point cloud data 20a and one pixel data in the first image data 70a may be generated. That is, the first image data 70a is data generated based on the brightness information of the received signal of the received light, including the reflected light L2 received by the distance measuring device 2a, or data designed to simulate the received signal.

[0094] Similarly, the second image data 70b has multiple pixel data corresponding to multiple point data of the second point cloud data 20b. The second image data 70b is, for example, data generated based on the brightness information of the received signal of received light, which includes reflected light L2 received by the distance measuring device 2b (or received by the distance measuring device 2a at a FoV different from the first image data 70a), or data designed to simulate the received signal.

[0095] Figure 12 shows an example of the first image data 70a. The first image data 70a in Figure 12 corresponds, for example, to the first point cloud data 20a in Figure 3. Note that for illustrative purposes, the overall brightness and contrast of the actual first image data 70a in Figure 12 have been adjusted.

[0096] As shown in Figure 12, each of the multiple pixel data in the first image data 70a contains luminance information of the received reflected light L2. In other words, the first image data 70a has a luminance distribution.

[0097] Furthermore, the first image data 70a in Figure 12 has feature point cloud data 71, 72, and 73 with different brightness levels from the surrounding area. Feature point cloud data 71, 72, and 73 correspond to the first feature point cloud data 21a, 22a, and 23a in Figure 3, respectively.

[0098] The 2D image feature point cloud extraction unit 62 in Figure 11 extracts feature point cloud data (for example, feature point cloud data 71-73 in Figure 12) from the first image data 70a through image processing. The feature point cloud extraction unit 12 in Figure 11 can also extract the first feature point cloud data by extracting point data corresponding to the feature point cloud data extracted by the 2D image feature point cloud extraction unit 62 from multiple point data in the first point cloud data 20a. Similarly, the 2D image feature point cloud extraction unit 62 and the feature point cloud extraction unit 12 in Figure 11 can extract the second feature point cloud data from the second point cloud data 20b based on the feature point cloud data extracted from the second image data 70b.

[0099] As described above, the 2D image feature point cloud extraction unit 62 obtains feature point cloud data of 2D image data (hereinafter also referred to as 2D feature point cloud data) by image processing of 2D image data acquired from the distance measuring device. Based on the 2D feature point cloud data, the feature point cloud extraction unit 12 can extract 3D feature point cloud data from 3D point cloud data with high accuracy and low computational cost.

[0100] The feature point cloud extraction unit 12 may extract a portion of the 3D point cloud data corresponding to the 2D feature point cloud data as 3D feature point cloud data. Alternatively, the feature point cloud extraction unit 12 may extract the 3D point cloud data corresponding to the 2D feature point cloud data and the 3D point cloud data arranged around it as 3D feature point cloud data.

[0101] As a comparative example, consider the case where each point in the 3D point cloud data contains luminance information. Even in this case, 3D feature point cloud data can be extracted from the 3D point cloud data based on the luminance information.

[0102] However, in the method of the comparative example, it can be difficult to distinguish between point cloud data of a reference plane (e.g., the ground) and point cloud data that can be matched to a 3D model. In contrast, in the method of the second embodiment of this disclosure, the influence of the reference plane can be eliminated by image processing of 2D image data, and 2D feature point cloud data and 3D feature point cloud data can be extracted. Note that the method of the comparative example described above may also be applied to the alignment method according to the second embodiment of this disclosure.

[0103] The information processing device 10d in Figure 11 may have a configuration that includes a 2D image misalignment detection unit 63. The 2D image misalignment detection unit 63 compares 2D image data acquired by the 2D image data acquisition unit 61 from a distance measuring device (for example, distance measuring device 2a) (hereinafter also referred to as first image data) with 2D image data previously acquired from the same distance measuring device (hereinafter also referred to as second image data). The second image data may be stored in, for example, the memory 18 in Figure 1.

[0104] The 2D image misalignment detection unit 63 can detect the misalignment of pixel positions between the second image data and the first image data, for example, by processing such as image stabilization. That is, the 2D image misalignment detection unit 63 can extract 2D feature point cloud data (hereinafter also referred to as the first 2D feature point cloud data) from the 2D feature point cloud data (hereinafter also referred to as the second 2D feature point cloud data) extracted from the second image data, based on the misalignment of pixel positions. This allows the first 2D feature point cloud data to be extracted with less computational effort than directly extracting the first 2D feature point cloud data from the first image data.

[0105] 2D feature point cloud data may be extracted by user operation. For example, the information processing device 10d in Figure 11 may have a configuration comprising a 2D image display unit 64 and a 2D image feature point specification unit 65. The 2D image display unit 64 displays 2D image data acquired from the distance measuring device to the user. The 2D image feature point specification unit 65 specifies the 2D feature point cloud data to be extracted based on user input.

[0106] Figure 13 shows an example of an operation screen 80 displayed to the user by the 2D image display unit 64. The operation screen 80 displays either one or both (for example, both) of the first image data 70a and the second image data 70b. The operation screen 80 also has a cropping brightness specification unit 81.

[0107] The first image data 70a in Figure 13 includes an object 75a and a reference surface 76a with a different luminance from object 75a. The second image data 70b includes an object 75b having approximately the same luminance as object 75a and a reference surface 76b having approximately the same luminance as reference surface 76a.

[0108] In the cropping brightness specification unit 81, for example, an upper brightness limit 81a and a lower brightness limit 81b can be specified. In other words, the cropping brightness specification unit 81 can specify the brightness range to be cropped. Note that the cropping brightness specification unit 81 may be configured to specify either the upper brightness limit 81a or the lower brightness limit 81b.

[0109] The user can adjust the brightness range of the extraction brightness specification unit 81, for example, so that the brightness of objects 75a and 75b is included in the brightness range, but the brightness of reference surfaces 76a and 76b is not included in the brightness range. This allows the user to extract objects 75a and 75b from the first image data 70a and the second image data 70b, respectively, as 2D feature point cloud data (hereinafter also referred to as the extraction target).

[0110] The user may also directly extract the target to be cut out (for example, objects 75a and 75b) from the first image data 70a and the second image data 70b by clicking or selecting a range.

[0111] The operation screen 80 may display either or both (for example, both) of the first point cloud data 20a and the second point cloud data 20b. The first point cloud data 20a and the second point cloud data 20b may also display the extraction ranges 82a and 82b corresponding to the extraction target specified by the user. The feature point cloud extraction unit 12 can extract the point cloud data within the extraction ranges 82a and 82b as 3D feature point cloud data.

[0112] The operation screen 80 may have a configuration that includes a cropping size specification section 83. The cropping size specification section 83 allows the user to specify the sizes of the cropping ranges 82a and 82b.

[0113] The user may directly specify or change the cropping ranges 82a and 82b by clicking or dragging. This allows the user to directly specify the 3D feature point cloud data. In this case, the first image data 70a, the second image data 70b, and the cropping brightness specification section 81 may be omitted from the operation screen 80.

[0114] The operation screen 80 may be displayed on a display device (e.g., a display) located inside or outside the information processing device 10d. Alternatively, the user may obtain the operation screen 80 from the information processing device 10d using any device (e.g., a laptop computer, a smartphone, or the edge device 51 in Figure 10). In other words, the operation screen 80 may be displayed on any device. The processing of the operation screen 80 may be performed by the CPU 17 in Figure 1. Furthermore, the configuration information (config) of the operation screen 80 may be stored in the memory 18.

[0115] As described above, the 2D image feature point cloud extraction unit 62 can extract 2D feature point cloud data based on luminance information of 2D image data or user instruction information, or the 2D image feature point cloud extraction unit 62 may extract 2D feature point cloud data based on design information of 2D image data, etc. Furthermore, the feature point cloud extraction unit 12 can extract 3D feature point cloud data based on luminance information of 2D feature point cloud data or user instruction information, or the 3D feature point cloud data may be extracted based on design information of 3D point cloud data, or the 3D feature point cloud data may be extracted based on design information of 3D point cloud data, etc.

[0116] Figure 14 is a flowchart illustrating the operation of the information processing device 10d according to the second embodiment of this disclosure. Below, an example of extracting 3D feature point cloud data from the first point cloud data 20a is described. The method for extracting 3D feature point cloud data from the second point cloud data 20b is the same as the example described below.

[0117] First, the 2D image data acquisition unit 61 acquires the first image data 70a from the distance measuring device 2a (step S21).

[0118] Next, the 2D image data acquisition unit 61 determines whether or not there are past frames of the first image data 70a, that is, whether or not the distance measuring device 2a has output the first image data 70a in the past (step S22).

[0119] If past frames exist, the 2D image shift detection unit 63 performs a process to detect the amount of pixel position shift between the first image data 70a acquired in step S1 and the past frames (for example, image stabilization processing) (step S23).

[0120] Next, the 2D image misalignment detection unit 63 estimates the pixel positions of the 2D feature point cloud data in the first image data 70a from the misalignment amount detected in step S23 (step S24). For example, the pixel positions of the 2D feature point cloud data in the first image data 70a can be estimated by adding the misalignment amount detected in step S23 to the pixel positions of the 2D feature point cloud data detected from past frames.

[0121] In step S22, if no past frames are available, the information processing device 10d estimates the pixel positions of the 2D feature point cloud data from user-specified data or design data (step S25). In the case of user-specified data, for example, the operation screen 80 in Figure 13 may be used. An example of a situation where no past frames are available is when the distance measuring system 1b is being initialized.

[0122] Next, the 2D image feature point cloud extraction unit 62 extracts 2D feature point cloud data based on the pixel positions of the 2D feature point cloud data estimated in step S24 or S25 (step S26).

[0123] Next, the feature point cloud extraction unit 12 extracts 3D feature point cloud data corresponding to the 2D feature point cloud data extracted in step S26 (step S27). The feature point cloud extraction unit 12 can, for example, extract the coordinates of each point data in the 3D feature point cloud data from the pixel position information of each point data in the 2D feature point cloud data. Alternatively, the correspondence between each point data in the 2D feature point cloud data and each point data in the 3D feature point cloud data may be stored in a predetermined storage device (e.g., memory 18). Furthermore, at least one of the point data in the 2D feature point cloud data or the point data in the 3D feature point cloud data may contain information about the corresponding point data in the other.

[0124] Next, the information processing device 10d determines whether or not the extraction of 3D feature point cloud data has been completed (step S28). In step S28, for example, the 2D image feature point cloud extraction unit 62 determines whether or not it can still extract 2D feature point cloud data from the first image data 70a. If it can extract 2D feature point cloud data, the 2D image feature point cloud extraction unit 62 extracts the 2D feature point cloud data in step S26.

[0125] Once the extraction of 3D feature point cloud data is complete, a 3D model matching of the extracted 3D feature point cloud data is performed in step S3, etc., as shown in Figure 6.

[0126] As shown in Figure 14, if past frames exist, the information processing device 10d can extract 2D feature point cloud data with low computational complexity based on those past frames. This reduces the computational complexity of the process in step S2 of Figure 6. In the initial state, when no past frames exist, the information processing device 10d can acquire the 2D feature point cloud through user operation or other means.

[0127] In addition to the methods in steps S21 to S25, 2D feature point cloud data may also be extracted by image processing of the first image data 70a (for example, a process to extract feature points from the brightness distribution).

[0128] As described above, the information processing device 10d according to the second embodiment of this disclosure acquires 2D image data corresponding to 3D point cloud data. The information processing device 10d also extracts 2D feature point cloud data based on the brightness distribution of the 2D image data. By extracting 3D feature point cloud data corresponding to 2D feature point cloud data from 3D point cloud data, the information processing device 10d can extract 3D feature point cloud data with high accuracy and low computational cost.

[0129] 2D feature point cloud data can also be extracted based on previously acquired 2D image data. That is, by comparing the 2D image data output by the distance measuring device with 2D image data previously output by the same device, the amount of pixel position shift can be detected through processing such as image stabilization. The information processing device 10d can extract 2D feature point cloud data with less computational effort based on the amount of pixel position shift and the 2D feature point cloud data extracted from the previously acquired 2D image data.

[0130] Furthermore, 2D and 3D feature point cloud data can be acquired manually or semi-manually by the user. For example, by specifying a luminance range to be extracted from the luminance distribution of 2D image data, objects that can be matched to a 3D model can be efficiently extracted from the 2D image data.

[0131] At least a portion of the information processing devices 10 and 10a to 10d (hereinafter also simply referred to as information processing device 10) described in the above-described embodiments may be configured as hardware or as software. In the case of software configuration, a program that realizes at least a portion of the functions of the information processing device 10 may be stored on a recording medium such as a flexible disk or CD-ROM, and loaded into a computer for execution. The recording medium is not limited to removable ones such as magnetic disks or optical disks, but may also be a fixed recording medium such as a hard disk drive or memory.

[0132] Furthermore, a program that implements at least some of the functions of the information processing device 10 may be distributed via communication lines such as the Internet (including wireless communication). In addition, the program may be encrypted, modulated, or compressed and distributed via wired or wireless lines such as the Internet, or stored on a recording medium.

[0133] Furthermore, this technology can take the following configuration. [Item 1] A first extraction unit extracts three-dimensional feature point cloud data from three-dimensional point cloud data in a first coordinate system, A second extraction unit extracts three or more specific points by matching the three-dimensional feature point cloud data with a three-dimensional model. The system includes an alignment unit that aligns three or more specific points extracted by the second extraction unit in the first coordinate system with three or more specific points in the second coordinate system, Information processing device. [Item 2] The alignment unit performs relative alignment of the three-dimensional point cloud data in the first coordinate system and the three-dimensional point cloud data in the second coordinate system based on three or more specific points extracted by the second extraction unit in the first coordinate system and three or more specific points in the second coordinate system. The information processing device described in item 1. [Item 3] Align three or more specific points extracted in one of the multiple coordinate systems, including the first and second coordinate systems, with three or more specific points extracted in another coordinate system. The information processing device described in item 1. [Item 4] The alignment unit estimates a transformation matrix from the first coordinate system to the second coordinate system based on three or more specific points extracted by the second extraction unit in the first coordinate system and three or more specific points in the second coordinate system. An information processing device as described in any one of items 1 to 3. [Item 5] The first extraction unit extracts the three-dimensional feature point cloud data based on brightness information, design information, or user instruction information of the three-dimensional point cloud data. An information processing device as described in any one of items 1 through 4. [Item 6] The second extraction unit defines three or more uniquely identified locations or vectors in three-dimensional space as the three or more identified points. An information processing device as described in any one of items 1 through 5. [Item 7] The three-dimensional model includes at least one of a plane, polyhedron, sphere, hemisphere, prism, pyramid, or frustum existing in the three-dimensional space. The second extraction unit extracts the three or more specific points based on at least one of a specific coordinate position in the three-dimensional model, a normal vector at the specific coordinate position, or an axis vector of the three-dimensional model. The information processing device described in item 6. [Item 8] The three or more specified points include any of the following: a point on the normal to the plane, the centroid of the polyhedron or sphere, the center of the hemisphere, the vertex of the cone or a point obtained by adding the axial vector of the cone to the vertex, or a point obtained by adding the axial vector of the prism or frustum to another specified point. The information processing device described in item 7. [Item 9] The system includes a third extraction unit that extracts feature points contained in the two-dimensional image data in the first coordinate system, The first extraction unit extracts the three-dimensional feature point cloud data, which includes points corresponding to the feature points extracted by the third extraction unit. An information processing device as described in any one of items 1 through 8. [Item 10] The aforementioned two-dimensional image data is data with the same field of view as the aforementioned three-dimensional point cloud data. The information processing device described in item 9. [Item 11] The third extraction unit extracts the feature points based on the brightness information of the two-dimensional image data. The information processing device described in item 9 or 10. [Item 12] The third extraction unit extracts the feature points based on the misalignment of the plurality of two-dimensional image data. An information processing device as described in any one of items 9 through 11. [Item 13] The third extraction unit detects the displacement of the multiple two-dimensional image data by image stabilization processing. The information processing device described in item 12. [Item 14] The third extraction unit extracts the feature points based on the design information of the two-dimensional image data or the user's instruction information. An information processing device as described in any one of items 9 through 11. [Item 15] The three-dimensional point cloud data is data generated based on a received light signal, or data designed to simulate the received light signal. The aforementioned received light signal includes the received light signal of reflected light from an object. An information processing device as described in any one of items 1 through 14. [Item 16] The three-dimensional point cloud data is data generated based on a received light signal, or data designed to simulate the received light signal. The two-dimensional image data is data generated based on the brightness information of the received light signal, or data designed to simulate the received light signal. The aforementioned received light signal includes the received light signal of reflected light from an object. An information processing device as described in any one of items 9 through 14. [Item 17] An information processing device described in any one of items 1 to 16, The system includes a distance measuring device that generates three-dimensional point cloud data in the first coordinate system and detects the distance of an object based on the three-dimensional point cloud data, Distancing system. [Item 18] An information processing device as described in any one of items 9 through 14 and 16, The system includes a distance measuring device that generates three-dimensional point cloud data and a two-dimensional image in the first coordinate system, and detects the distance of an object based on at least one of the three-dimensional point cloud data or the two-dimensional image. Distancing system. [Item 19] From the three-dimensional point cloud data in the first coordinate system, three-dimensional feature point cloud data is extracted. By matching the aforementioned three-dimensional feature point cloud data with the 3D model, three or more specific points are extracted. Align three or more specific points in the extracted first coordinate system with three or more specific points in the second coordinate system. Information processing methods. [Item 20] Information processing device, The steps include: extracting three-dimensional feature point cloud data from three-dimensional point cloud data in the first coordinate system; The steps include: extracting three or more specific points by matching the three-dimensional feature point cloud data with the 3D model; The process includes the step of aligning three or more specific points in the extracted first coordinate system with three or more specific points in the second coordinate system, program.

[0134] The aspects of this disclosure are not limited to the individual embodiments described above, but include various modifications that a person skilled in the art could conceive, and the effects of this disclosure are not limited to those described above. In other words, various additions, modifications, and partial deletions are possible, as long as they do not depart from the conceptual idea and spirit of this disclosure derived from the claims and their equivalents. [Explanation of Symbols]

[0135] 1, 1a, 1b Distance measuring system, 2a, 2b Distance measuring device, 3, 3a Object to be measured, 4 Reference plane, 5 Side wall, 6a, 6b, 6c Specific point, 10, 10d Information processing device, 10a, 10b Shift correction unit, 10c Alignment unit, 11 3D point cloud data acquisition unit, 12 Feature point cloud extraction unit, 13 3D model matching unit, 14 Specific point extraction unit, 15 Transformation matrix estimation unit, 16 3D point cloud coordinate transformation unit, 17 CPU, 18 Memory, 20a, 20c First point cloud data, 20b, 20d Second point cloud data, 21a, 22a, 23a, 25a, 41a, 41a, 43a First feature point cloud data, 21b, 22b, 23b, 25b, 41b, 41b, 43b Second feature point cloud data, 30 Cone, 31 Vertex, 32 Central axis, 46a, 46b, 47a, 47b, 48a, 48b Boundary points, 51, 52 Edge device, 53 Server, 61 2D image data acquisition unit, 62 2D image feature point cloud extraction unit, 63 2D image displacement detection unit, 64 2D image display unit, 65 2D image feature point specification unit, 70a First image data, 70b Second image data, 71, 72, 73 Feature point cloud data, 75a, 75b Object, 76a, 76b Reference plane, 80 Operation screen, 81 Cutout brightness specification unit, 81a Upper limit brightness, 81b Lower limit brightness, 82a, 82b Cutout range, 83 Cutout size specification unit

Claims

1. A first extraction unit extracts three-dimensional feature point cloud data from three-dimensional point cloud data in a first coordinate system, A second extraction unit extracts three or more specific points by matching the three-dimensional feature point cloud data with a three-dimensional model. The system includes an alignment unit that aligns three or more specific points extracted by the second extraction unit in the first coordinate system with three or more specific points in the second coordinate system, Information processing device.

2. The alignment unit estimates a transformation matrix from the first coordinate system to the second coordinate system based on three or more specific points extracted by the second extraction unit in the first coordinate system and three or more specific points in the second coordinate system. The information processing apparatus according to claim 1.

3. The aforementioned three-dimensional model includes at least one of a plane, polyhedron, sphere, hemisphere, prism, pyramid, or frustum existing in three-dimensional space. The second extraction unit extracts three or more specific points based on at least one of a specific coordinate position in the three-dimensional model, a normal vector at the specific coordinate position, or an axis vector of the three-dimensional model. The information processing apparatus according to claim 1.

4. The three or more specified points include any of the following: a point on the normal to the plane, the centroid of the polyhedron or sphere, the center of the hemisphere, the vertex of the cone or a point obtained by adding the axial vector of the cone to the vertex, or a point obtained by adding the axial vector of the prism or frustum to another specified point. The information processing apparatus according to claim 3.

5. The system includes a third extraction unit that extracts feature points contained in the two-dimensional image data in the first coordinate system, The first extraction unit extracts the three-dimensional feature point cloud data, which includes points corresponding to the feature points extracted by the third extraction unit. The information processing apparatus according to claim 1.

6. The aforementioned two-dimensional image data is data with the same field of view as the aforementioned three-dimensional point cloud data. The information processing apparatus according to claim 5.

7. The third extraction unit extracts the feature points based on the brightness information of the two-dimensional image data. The information processing apparatus according to claim 5.

8. An information processing device according to any one of claims 1 to 7, The system includes a distance measuring device that generates three-dimensional point cloud data in the first coordinate system and detects the distance of an object based on the three-dimensional point cloud data, Distancing system.

9. An information processing device according to any one of claims 5 to 7, The system includes a distance measuring device that generates three-dimensional point cloud data and a two-dimensional image in the first coordinate system, and detects the distance of an object based on at least one of the three-dimensional point cloud data or the two-dimensional image. Distancing system.

Images